Schedule demo
Home

Amazon Kinesis Data Streams Monitoring

Amazon Kinesis Data Streams - Overview

Amazon Kinesis Data Streams is a fully managed, serverless streaming data service that enables real-time collection, processing, and analysis of large volumes of data records. It allows applications to ingest data from multiple sources simultaneously, with support for configurable retention periods, multiple consumer applications, and enhanced fan-out for high-throughput parallel data processing.

Monitoring Amazon Kinesis Data Streams is essential for ensuring high-throughput data pipelines remain healthy and within provisioned capacity limits. Applications Manager's Amazon Kinesis Data Streams monitoring tool provides real-time visibility into stream-level metrics such as incoming traffic, record throughput, iterator age, latency, and PutRecords performance. With proactive alerting and historical trend analysis, the tool helps identify throttling events, consumer lag, and data pipeline bottlenecks before they impact downstream applications.

Creating a new Amazon Kinesis Data Streams monitor

To learn how to create a new Amazon Kinesis Data Streams monitor, refer here.

Monitored Parameters

Go to the Monitors Category View by clicking the Monitors tab. Click on the Kinesis Data Streams instance available under Amazon in the Cloud Apps section. Displayed below is the Amazon Kinesis Data Streams bulk configuration view distributed into three tabs:

  • Availability tab gives the availability history for the past 24 hours or 30 days.
  • Performance tab gives the health status and events for the past 24 hours or 30 days.
  • List view tab enables you to perform bulk admin configurations.

By clicking a monitor from the list, you'll be taken to the Amazon Kinesis Data Streams dashboard which includes the following tabs:

Performance Overview

ParameterDescription
STREAM INFORMATION
Stream StatusThe current status of the data stream.
Possible values: CREATING, DELETING, ACTIVE, UPDATING.
Number of Open ShardsThe number of open shards in the data stream at the time of polling.
Number of ConsumersThe number of enhanced fan-out consumers registered with the stream.
EXCEEDED PROVISIONED THROUGHPUT
Exceeded Read ThroughputThe average number of requests throttled for GetRecords operations for the stream between the poll interval.
Exceeded Write ThroughputThe average number of requests throttled for PutRecord and PutRecords operations for the stream between the poll interval.
ITERATOR AGE
Iterator AgeThe peak age of the last GetRecords operation made against all shards in the stream at the time of polling (in ms).
LATENCY
GetRecords LatencyThe average time taken per GetRecords operation for the stream between the poll interval (in ms).
PutRecord LatencyThe average time taken per PutRecord operation for the stream between the poll interval (in ms).
PutRecords LatencyThe average time taken per PutRecords operation for the stream between the poll interval (in ms).
I/O THROUGHPUT
GetRecords Data RateThe total amount of data retrieved per second using GetRecords operations for the stream between the poll interval (in MB/s).
PutRecord Data RateThe total amount of data sent per second using PutRecord operations for the stream between the poll interval (in MB/s).
PutRecords Data RateThe total amount of data sent per second using PutRecords operations for the stream between the poll interval (in MB/s).
GetRecords DataThe total amount of data retrieved using GetRecords operations for the stream between the poll interval (in MB).
PutRecord DataThe total amount of data written using PutRecord operations for the stream between the poll interval (in MB).
PutRecords DataThe total amount of data written using PutRecords operations for the stream between the poll interval (in MB).
INCOMING TRAFFIC
Incoming Traffic RateThe total amount of data successfully sent to the stream per second between the poll interval (in MB/s).
Incoming TrafficThe total amount of data successfully sent to the stream between the poll interval (in MB).
INCOMING RECORDS
Incoming Records RateThe total number of records successfully sent to the stream per minute between the poll interval (in records/min).
Incoming RecordsThe total number of records successfully sent to the stream between the poll interval.
RECORD TRAFFIC
GetRecords Records RateThe total number of records retrieved per minute using GetRecords operations for the stream between the poll interval (in records/min).
PutRecords RateThe total number of records sent in PutRecords operations per minute for the stream between the poll interval (in records/min).
GetRecords RequestsThe total number of records retrieved using GetRecords operations for the stream between the poll interval.
PutRecords RequestsThe total number of records sent in PutRecords operations for the stream between the poll interval.
PUTRECORDS BREAKDOWN
Failed PutRecordsThe total number of records that failed due to internal service errors in PutRecords operations for the stream between the poll interval.
Throttled PutRecordsThe total number of records rejected due to throttling in a PutRecords operation for the stream between the poll interval.
Successful PutRecordsThe total number of successful records in PutRecords operations for the stream between the poll interval.
FAILED PUTRECORDS PERCENTAGE
Failed PutRecords PercentageThe percentage of records that failed due to internal service errors in PutRecords operations for the stream between the poll interval. Calculated as (PutRecords Failed Records / PutRecords Total Records) * 100 (in %).
THROTTLED PUTRECORDS PERCENTAGE
Throttled PutRecords PercentageThe percentage of records rejected due to throttling in PutRecords operations for the stream between the poll interval. Calculated as (PutRecords Throttled Records / PutRecords Total Records) * 100 (in %).
SUCCESSFUL RECORDS
GetRecords Success RateThe total number of successful GetRecords operations per minute for the stream between the poll interval (in records/min).
PutRecord Success RateThe total number of successful PutRecord operations per minute for the stream between the poll interval (in records/min).
PutRecords Success RateThe total number of PutRecords requests in which at least one record was successfully written per minute for the stream between the poll interval (in records/min).

Consumers

Note: The Consumer Metrics are mapped under Performance Polling and disabled by default. To enable or disable metric collection, or to configure the polling interval, navigate to Settings → Performance Polling. Under the Optimize Data Collection tab, select Amazon Kinesis Data Streams as the Monitor Type, choose Consumer Metrics as the Metric Name, and update the required polling status and interval.

This tab displays the following Top 5 charts for enhanced fan-out consumers registered with the stream:

  • Top 5 Consumers by Subscription Processing Lag
  • Top 5 Consumers by Subscription Throttle Rate
  • Top 5 Consumers by Subscription Outgoing Traffic
  • Top 5 Consumers by Subscription Outgoing Records
  • Top 5 Consumers by Subscription Success Rate
  • Top 5 Consumers by Subscription Successful Delivery
ParameterDescription
CONSUMERS
Consumer NameThe name of the enhanced fan-out consumer.
Consumer StatusThe current status of the enhanced fan-out consumer.
Creation TimeThe date and time when the consumer was created.
Subscription Processing LagThe number of milliseconds the consumer is behind the most recent record in the shard at the time of polling (in ms).
Subscription Throttle RateThe number of times a SubscribeToShard request was throttled for the consumer between the poll interval.
Subscription Outgoing TrafficThe total amount of data sent to the consumer via SubscribeToShard events per second between the poll interval (in MB/s).
Subscription Outgoing RecordsThe total number of records sent to the consumer via SubscribeToShard events per minute between the poll interval (in records/min).
Subscription Success RateThe total number of successful SubscribeToShard requests for the consumer between the poll interval.
Subscription Successful DeliveryThe total number of successful SubscribeToShard event streaming calls for the consumer between the poll interval.

Configuration

ParameterDescription
CONFIGURATION
Stream StatusThe current status of the data stream.
Possible values: CREATING, DELETING, ACTIVE, UPDATING.
Stream Capacity ModeThe capacity mode of the data stream.
Number of Open ShardsThe number of open shards in the data stream.
Number of ConsumersThe number of enhanced fan-out consumers registered with the stream.
Retention PeriodThe current retention period of the stream (in hrs).
Maximum Record SizeThe maximum size of a data record written to the stream (in KiB).
Creation TimeThe date and time when the stream was created.
SECURITY CONFIGURATION
Encryption TypeThe type of encryption used on the stream.
Encryption KeyThe ID or ARN of the KMS key used to encrypt the data stream.

Loved by customers all over the world

"Standout Tool With Extensive Monitoring Capabilities"

★★★★★

It allows us to track crucial metrics such as response times, resource utilization, error rates, and transaction performance. The real-time monitoring alerts promptly notify us of any issues or anomalies, enabling us to take immediate action.

Reviewer Role: Research and Development

carlos-rivero
"I like Applications Manager because it helps us to detect issues present in our servers and SQL databases."
Carlos Rivero

Tech Support Manager, Lexmark

Trusted by thousands of leading businesses globally