Schedule demo

Amazon Data Firehose Monitoring


Amazon Data Firehose - Overview

Amazon Data Firehose is a fully managed service that reliably loads streaming data into data lakes, data stores, and analytics services. It can capture, transform, and deliver streaming data to destinations such as Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, Snowflake, and custom HTTP endpoints - without requiring you to write applications or manage infrastructure. Firehose automatically scales to match the throughput of your data and requires no ongoing administration.

Monitoring Amazon Data Firehose is critical for maintaining reliable data pipeline performance and ensuring timely delivery of streaming data to downstream destinations. Applications Manager's Amazon Data Firehose monitoring tool tracks key metrics across the entire delivery lifecycle, including incoming data volume and throughput, delivery success rates, backup operations, data freshness, transformation processing, KMS encryption health, and detailed configuration settings for each delivery stream. With proactive alerts and trend analysis, the tool helps identify delivery failures, throttling issues, and transformation errors before they disrupt downstream analytics workflows.

Creating a new Amazon Data Firehose monitor

To learn how to create a new Amazon Data Firehose monitor, refer here.

Monitored Parameters

Go to the Monitors Category View by clicking the Monitors tab. Click on the Data Firehose instance available under Amazon in the Cloud Apps section. Displayed below is the Amazon Data Firehose bulk configuration view, distributed into three tabs:

  • Availability tab gives the availability history for the past 24 hours or 30 days.
  • Performance tab gives the health status and events for the past 24 hours or 30 days.
  • List view tab enables you to perform bulk admin configurations.

By clicking a monitor from the list, you'll be taken to the Amazon Data Firehose dashboard, which includes the following tabs:

Performance Overview

ParameterDescription
DELIVERY STREAM INFORMATION
Delivery Stream StatusThe current status of the delivery stream.
Possible values: CREATING, DELETING, ACTIVE, CREATING_FAILED, DELETING_FAILED.
Delivery Stream TypeThe type of data source for the delivery stream.
Possible values: DirectPUT, Kinesis Data Stream, MSK.
Stream DestinationThe target destination platform selected to store or index data emitted by this stream lifecycle at the time of polling.
Failure TypeThe general category or class of the most recent delivery pipeline failure condition recorded at the time of polling.
Failure DetailsThe detailed contextual error messages and system logs explaining the primary root cause of the last failed pipeline operation at the time of polling.
DATA FRESHNESS
Data FreshnessThe maximum age of the oldest record in the delivery stream at the time of polling (in seconds).
DELIVERY OPERATIONS
Successful Delivery OperationsThe total number of successful delivery operations between the poll interval.
INCOMING DATA
Incoming DataThe total amount of data ingested into the delivery stream between the poll interval (in MB).
INCOMING RECORDS
Incoming RecordsThe total number of records ingested into the delivery stream between the poll interval.
DELIVERY DATA
Delivery DataThe total amount of data delivered to the destination between the poll interval (in MB).
DELIVERY RECORDS
Delivery RecordsThe total number of records delivered to the destination between the poll interval.
BACKUP DATA
Backup DataThe total amount of data delivered to S3 for backup between the poll interval (in MB).
BACKUP RECORDS
Backup RecordsThe total number of records delivered to S3 for backup between the poll interval.
Successful Backup OperationsThe total number of successful S3 put commands for backup between the poll interval.
BACKUP DATA FRESHNESS
Backup Data FreshnessThe maximum age of the oldest record in the Firehose backup stream at the time of polling (in seconds).
LATENCY
PutRecord LatencyThe average time taken per PutRecord operation between the poll interval (in ms).
PutRecordBatch LatencyThe average time taken per PutRecordBatch operation between the poll interval (in ms).

Data Ingestion

Note: Data Ingestion group metrics are mapped under Performance Polling and are disabled by default. To enable data collection, navigate to Settings → Performance Polling → Optimize Data Collection. Select Amazon Data Firehose as the Monitor Type, choose Data Ingestion Metrics as the Metric Name, and set the desired polling status.

ParameterDescription
STREAM LATENCY
Describe Stream LatencyThe average time taken per DescribeDeliveryStream operation between the poll interval (in ms).
Update Stream LatencyThe average time taken per UpdateDeliveryStream operation between the poll interval (in ms).
REQUESTS
Describe Stream RequestsThe total number of DescribeDeliveryStream requests between the poll interval.
Update Stream RequestsThe total number of UpdateDeliveryStream requests between the poll interval.
PutRecord RequestsThe total number of PutRecord requests between the poll interval.
PutRecordBatch RequestsThe total number of PutRecordBatch requests between the poll interval.
PUTRECORD DATA
PutRecord DataThe total amount of data written using PutRecord between the poll interval (in MB).
PutRecordBatch DataThe total amount of data written using PutRecordBatch between the poll interval (in MB).
PUTRECORDBATCH RECORDS
PutRecordBatch RecordsThe total number of records from PutRecordBatch operations between the poll interval.
INCOMING PUT REQUESTS
Incoming Put RequestsThe total number of successful PutRecord and PutRecordBatch requests between the poll interval.
Throttled RecordsThe total number of records was throttled because data ingestion exceeded stream limits between the poll interval.
SOURCE INCOMING DATA
Source Incoming DataThe total amount of data read from the source stream between the poll interval (in MB).
SOURCE INCOMING RECORDS
Source Incoming RecordsThe total number of records read from the source stream between the poll interval.
CONSUMER LAG
Consumer LagThe maximum lag indicating how far the consumer is behind the latest available record at the time of polling (in ms).
INGESTION THROUGHPUT LIMIT
Ingestion Throughput LimitThe maximum data throughput the delivery stream can ingest before throttling occurs at the time of polling (in MB/s).
INGESTION RECORDS LIMIT
Ingestion Records LimitThe maximum records per second limit that the delivery stream can ingest before throttling at the time of polling (in records/s).

Data Transformation

Note: Data Transformation group metrics are mapped under Performance Polling and are disabled by default. To enable data collection, navigate to Settings → Performance Polling → Optimize Data Collection. Select Amazon Data Firehose as the Monitor Type, choose Data Transformation Metrics as the Metric Name, and set the desired polling status.

ParameterDescription
LAMBDA PROCESSING DURATION
Lambda Processing DurationThe average time taken for each Lambda function invocation between the poll interval (in ms).
LAMBDA PROCESSING SUCCESS
Lambda Processing SuccessThe total number of successful Lambda function invocations between the poll interval.
PROCESSED RECORDS
Processed RecordsThe total number of successfully processed records between the poll interval.
PROCESSED DATA
Processed DataThe total amount of data successfully processed between the poll interval (in MB).
CONVERTED RECORDS
Converted RecordsThe total number of successfully converted records between the poll interval.
Conversion Failed RecordsThe total number of records that could not be converted between the poll interval.
CONVERTED DATA
Converted DataThe total amount of data from successfully converted records between the poll interval (in MB).
Conversion Failed DataThe total amount of data from records that could not be converted between the poll intervals (in MB).
DECOMPRESSED DATA
Decompressed DataThe total amount of successfully decompressed data between the poll interval (in MB).
DECOMPRESSED RECORDS
Decompressed RecordsThe total number of successfully decompressed records between the poll interval.
ACTIVE DYNAMIC PARTITIONING COUNT
Active Dynamic Partition CountThe maximum number of active partitions at the time of polling.
S3 OBJECTS DELIVERED
S3 Objects DeliveredThe total number of objects delivered to the S3 bucket between the poll interval.

Configuration

The Configuration tab provides detailed information about the delivery stream, including its general configuration, encryption settings, source configuration, and destination-specific delivery settings.

Note: The configuration details displayed vary based on the delivery stream's source and destination types.

 
ParameterDescription
CONFIGURATION
Version IDThe internal system revision sequence identifier assigned to track the active configuration layout snapshot of this stream asset at the time of polling.
Destination IDThe technical topology identifier matches the exact endpoint infrastructure definition within your stream configurations at the time of polling.
Creation TimeThe localized system timestamp logging when this delivery stream instance was originally provisioned inside the region at the time of polling.
Last Update TimeThe precise system timestamp tracking when this pipeline's metadata rules or configuration parameters were last modified at the time of polling.
Buffering SizeThe configured buffer size before data is delivered to the destination (in MiB).
Buffering IntervalThe buffer interval before data is delivered to the destination (in seconds).
SECURITY CONFIGURATION
KMS Key ARNThe fully qualified Amazon Resource Name of the AWS KMS key deployed to handle data block encryption at the time of polling.
KMS Key TypeThe architectural strategy deployed to handle resource layer cryptographic validation, distinguishing between customer-managed and AWS-owned master keys at the time of polling.
KMS Key StatusThe present validation availability and lifecycle performance phase of your deployed cryptographic key component at the time of polling.
Encryption Failure TypeThe exact diagnostic error category generated during the last server-side cryptographic processing failure loop at the time of polling.
Encryption Failure DetailsThe specific low-level system context logs the failure mechanics during an encryption or decryption breakdown at the time of polling.

KINESIS DATA STREAM SOURCE CONFIGURATION

Note: This section is displayed only when the delivery stream source is Kinesis Data Streams.

Kinesis Stream ARNThe precise global Amazon Resource Name identifying the primary Kinesis Data Stream serving as the ingress source queue for this delivery pipeline at the time of polling.
Delivery Start TimeThe initial timestamp marking when this delivery stream pipeline began processing data blocks out of the source queue at the time of polling.
IAM Role ARNThe target Identity and Access Management execution role provides the delivery stream engine with secure ingress permissions to pull from the source stream at the time of polling.

MANAGED STREAMING FOR KAFKA (MSK) SOURCE CONFIGURATION

Note: This section is displayed only when the delivery stream source is Amazon MSK.

MSK Cluster ARNThe fully qualified global Amazon Resource Name tracking the parent Managed Streaming for Apache Kafka infrastructure serving data to this stream layout at the time of polling.
MSK Topic NameThe target string key denotes the exact Kafka message topic workspace monitored for automated ingress processing at the time of polling.
Delivery Start TimeThe localized timestamp marking when the stream wrapper initiated consumer client network connections to the target Kafka topic interface at the time of polling.
Read Start TimeThe genesis offset position timestamp or log index layout used to fetch entries from the target source Kafka queue at the time of polling.

TARGET DESTINATION INFRASTRUCTURE SETTINGS

Note: The destination configuration displayed depends on the destination configured for the delivery stream.

EndpointThe connection server URL string is deployed to securely direct egress pipeline elements to your target cluster node at the time of polling.
Index NameThe active cluster collection label is deployed to receive, categorize, and archive processed ingestion records inside your storage workspace at the time of polling.
Index RotationThe active configuration setting manages how frequently your target cluster switches into fresh index collections at the time of polling.
Domain ARNThe global Amazon Resource Name denoting the exact storage domain instance linked to this active pipeline route at the time of polling.
Type NameThe active indexing schema layout maps individual data attributes directly into target cluster fields at the time of polling.
Document ID FormatThe identifier pattern rule used to serialize and map document entries during storage processing at the time of polling.
VPC IDThe technical identifier matching the isolated virtual private network infrastructure containing your protected destination cluster node at the time of polling.
Extended S3 BucketThe global Amazon Resource Name locates the primary target storage container handling data stream outputs at the time of polling.
File ExtensionThe customized trailing string suffix is appended to generated data file assets before they are flushed onto the storage system at the time of polling.
Dynamic PartitioningThe configuration setting controls whether inline data structures are parsed to dynamically separate storage output paths at the time of polling.
Data Format ConversionThe operational enablement phase tracks whether incoming unstructured data lines are dynamically converted into a structured format at the time of polling.
Cluster JDBC URLThe exact network string URL deployed to handle transactional database communication routes targeting your analytics cluster layout at the time of polling.
Target TableThe explicit table collection label mapping incoming record entries directly inside your destination database layout at the time of polling.
Catalog ARNThe unique technical identifier pointing to the active data metadata catalog repository linked to this pipeline target at the time of polling.
Catalog WarehouseThe physical data warehouse folder hierarchy directory location containing your catalog layout at the time of polling.
Database NameThe formal system label defining the target tabular warehouse instance within your metadata catalog at the time of polling.
Table NameThe target relational data row table is selected to store processed log blocks inside the catalog database at the time of polling.
Snowflake Account URLThe corporate web connection domain is deployed to authenticate stream payloads with your cloud data warehouse workspace at the time of polling.
Snowflake DatabaseThe specialized database catalog partition selected to hold stream output logs at the time of polling.
Snowflake Data LoadingThe precise loading strategy is configured to sync data blocks into storage rows within the cloud data warehouse at the time of polling.
Snowflake MetadataThe data structure column profile maps transactional stream metadata properties directly into storage indexes at the time of polling.
HEC EndpointThe primary HTTP Event Collector interface address is configured to aggregate event payloads inside your ingestion environment at the time of polling.
HEC Endpoint TypeThe transmission protocol pattern was deployed across the collector gateway interface at the time of polling.
HEC TokenThe bearer string credential token assigned to authenticate pipeline traffic with the event gateway environment at the time of polling.
HEC Acknowledgement TimeoutThe maximum allowable timeout constraint is set before a non-acknowledged event collector packet drops out of the queue processing loops at the time of polling.
S3 Backup ModeThe logical policy tracks whether raw input payloads are mirrored down to a fallback recovery bucket before target indexing loops execute at the time of polling.
S3 BucketThe dedicated Amazon S3 bucket name configured as the safe target for raw data backstops or transaction crash dumps at the time of polling.
S3 CompressionThe designated system encoding algorithm used to shrink data payloads before finalizing data dumps into the recovery bucket layout at the time of polling.
Buffering IntervalThe buffer interval before data is delivered to the destination (in seconds).
Retry DurationThe configuration parameter managing the active timeout boundary allowed for delivery operations to retry after initial downstream transport faults at the time of polling (in seconds).
Data ProcessingThe active operational enablement phase tracks your inline data modification or micro-lambda transformation engines at the time of polling.
CloudWatch LoggingThe active state tracks whether internal event exceptions and operational log lines are piped to CloudWatch Logs groups at the time of polling.
CloudWatch Log GroupThe dedicated CloudWatch system group destination receives pipeline telemetry data at the time of polling.
CloudWatch Log StreamThe active log stream track receives downstream interface log telemetry details within the target group container at the time of polling.
Secrets Manager ARNThe secure Amazon Resource Name that locates the managed Secrets Manager vault holding client credentials for the network gateway at the time of polling.
UsernameThe target system administrative identifier used to validate interface credentials during pipeline transport operations when polling.

Loved by customers all over the world

"Standout Tool With Extensive Monitoring Capabilities"

It allows us to track crucial metrics such as response times, resource utilization, error rates, and transaction performance. The real-time monitoring alerts promptly notify us of any issues or anomalies, enabling us to take immediate action.

Reviewer Role: Research and Development

carlos-rivero
"I like Applications Manager because it helps us to detect issues present in our servers and SQL databases."
Carlos Rivero

Tech Support Manager, Lexmark

Trusted by thousands of leading businesses globally