# Amazon Data Firehose Monitoring

- [Amazon Data Firehose - Overview](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#Overview)
- [Creating a new Amazon Data Firehose monitor](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#NewMonitor)
- [Monitored Parameters](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#MonitoredParameters)
  - [Performance Overview](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#performance-overview)
  - [Data Ingestion](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#DataIngestion)
  - [Data Transformation](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#DataTransformation)
  - [Configuration](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#configuration)

## Amazon Data Firehose - Overview

Amazon Data Firehose is a fully managed service that reliably loads streaming data into data lakes, data stores, and analytics services. It can capture, transform, and deliver streaming data to destinations such as Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, Snowflake, and custom HTTP endpoints - without requiring you to write applications or manage infrastructure. Firehose automatically scales to match the throughput of your data and requires no ongoing administration.

Monitoring Amazon Data Firehose is critical for maintaining reliable data pipeline performance and ensuring timely delivery of streaming data to downstream destinations. Applications Manager's Amazon Data Firehose monitoring tool tracks key metrics across the entire delivery lifecycle, including incoming data volume and throughput, delivery success rates, backup operations, data freshness, transformation processing, KMS encryption health, and detailed configuration settings for each delivery stream. With proactive alerts and trend analysis, the tool helps identify delivery failures, throttling issues, and transformation errors before they disrupt downstream analytics workflows.

## Creating a new Amazon Data Firehose monitor

To learn how to create a new Amazon Data Firehose monitor, [refer here](https://www.manageengine.com/products/applications_manager/help/aws-monitoring-tools.html#NewMonitor).

## Monitored Parameters

Go to the **Monitors Category View** by clicking the **Monitors** tab. Click on the **Data Firehose** instance available under **Amazon** in the **Cloud Apps** section. Displayed below is the Amazon Data Firehose bulk configuration view, distributed into three tabs:

- **Availability** tab gives the availability history for the past 24 hours or 30 days.
- **Performance** tab gives the health status and events for the past 24 hours or 30 days.
- **List view** tab enables you to perform [bulk admin configurations](https://www.manageengine.com/products/applications_manager/help/bulk-config.html).

By clicking a monitor from the list, you'll be taken to the Amazon Data Firehose dashboard, which includes the following tabs:

- [Performance Overview](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#performance-overview)
- [Data Ingestion](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#DataIngestion)
- [Data Transformation](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#DataTransformation)
- [Configuration](https://www.manageengine.com/products/applications_manager/help/aws-data-firehose-monitoring-tools.html#configuration)

### Performance Overview

| Parameter | Description |
|---|---|
| **DELIVERY STREAM INFORMATION** |  |
| Delivery Stream Status | The current status of the delivery stream.<br>**Possible values:** CREATING, DELETING, ACTIVE, CREATING_FAILED, DELETING_FAILED. |
| Delivery Stream Type | The type of data source for the delivery stream.<br>**Possible values:** DirectPUT, Kinesis Data Stream, MSK. |
| Stream Destination | The target destination platform selected to store or index data emitted by this stream lifecycle at the time of polling. |
| Failure Type | The general category or class of the most recent delivery pipeline failure condition recorded at the time of polling. |
| Failure Details | The detailed contextual error messages and system logs explaining the primary root cause of the last failed pipeline operation at the time of polling. |
| **DATA FRESHNESS** |  |
| Data Freshness | The maximum age of the oldest record in the delivery stream at the time of polling (in seconds). |
| **DELIVERY OPERATIONS** |  |
| Successful Delivery Operations | The total number of successful delivery operations between the poll interval. |
| **INCOMING DATA** |  |
| Incoming Data | The total amount of data ingested into the delivery stream between the poll interval (in MB). |
| **INCOMING RECORDS** |  |
| Incoming Records | The total number of records ingested into the delivery stream between the poll interval. |
| **DELIVERY DATA** |  |
| Delivery Data | The total amount of data delivered to the destination between the poll interval (in MB). |
| **DELIVERY RECORDS** |  |
| Delivery Records | The total number of records delivered to the destination between the poll interval. |
| **BACKUP DATA** |  |
| Backup Data | The total amount of data delivered to S3 for backup between the poll interval (in MB). |
| **BACKUP RECORDS** |  |
| Backup Records | The total number of records delivered to S3 for backup between the poll interval. |
| Successful Backup Operations | The total number of successful S3 put commands for backup between the poll interval. |
| **BACKUP DATA FRESHNESS** |  |
| Backup Data Freshness | The maximum age of the oldest record in the Firehose backup stream at the time of polling (in seconds). |
| **LATENCY** |  |
| PutRecord Latency | The average time taken per PutRecord operation between the poll interval (in ms). |
| PutRecordBatch Latency | The average time taken per PutRecordBatch operation between the poll interval (in ms). |

### Data Ingestion

**Note:** Data Ingestion group metrics are mapped under Performance Polling and are disabled by default. To enable data collection, navigate to **Settings → Performance Polling → Optimize Data Collection**. Select **Amazon Data Firehose** as the **Monitor Type**, choose **Data Ingestion Metrics** as the **Metric Name**, and set the desired polling status.

| Parameter | Description |
|---|---|
| **STREAM LATENCY** |  |
| Describe Stream Latency | The average time taken per DescribeDeliveryStream operation between the poll interval (in ms). |
| Update Stream Latency | The average time taken per UpdateDeliveryStream operation between the poll interval (in ms). |
| **REQUESTS** |  |
| Describe Stream Requests | The total number of DescribeDeliveryStream requests between the poll interval. |
| Update Stream Requests | The total number of UpdateDeliveryStream requests between the poll interval. |
| PutRecord Requests | The total number of PutRecord requests between the poll interval. |
| PutRecordBatch Requests | The total number of PutRecordBatch requests between the poll interval. |
| **PUTRECORD DATA** |  |
| PutRecord Data | The total amount of data written using PutRecord between the poll interval (in MB). |
| PutRecordBatch Data | The total amount of data written using PutRecordBatch between the poll interval (in MB). |
| **PUTRECORDBATCH RECORDS** |  |
| PutRecordBatch Records | The total number of records from PutRecordBatch operations between the poll interval. |
| **INCOMING PUT REQUESTS** |  |
| Incoming Put Requests | The total number of successful PutRecord and PutRecordBatch requests between the poll interval. |
| Throttled Records | The total number of records was throttled because data ingestion exceeded stream limits between the poll interval. |
| **SOURCE INCOMING DATA** |  |
| Source Incoming Data | The total amount of data read from the source stream between the poll interval (in MB). |
| **SOURCE INCOMING RECORDS** |  |
| Source Incoming Records | The total number of records read from the source stream between the poll interval. |
| **CONSUMER LAG** |  |
| Consumer Lag | The maximum lag indicating how far the consumer is behind the latest available record at the time of polling (in ms). |
| **INGESTION THROUGHPUT LIMIT** |  |
| Ingestion Throughput Limit | The maximum data throughput the delivery stream can ingest before throttling occurs at the time of polling (in MB/s). |
| **INGESTION RECORDS LIMIT** |  |
| Ingestion Records Limit | The maximum records per second limit that the delivery stream can ingest before throttling at the time of polling (in records/s). |

### Data Transformation

**Note:** Data Transformation group metrics are mapped under Performance Polling and are disabled by default. To enable data collection, navigate to **Settings → Performance Polling → Optimize Data Collection**. Select **Amazon Data Firehose** as the **Monitor Type**, choose **Data Transformation Metrics** as the **Metric Name**, and set the desired polling status.

| Parameter | Description |
|---|---|
| **LAMBDA PROCESSING DURATION** |  |
| Lambda Processing Duration | The average time taken for each Lambda function invocation between the poll interval (in ms). |
| **LAMBDA PROCESSING SUCCESS** |  |
| Lambda Processing Success | The total number of successful Lambda function invocations between the poll interval. |
| **PROCESSED RECORDS** |  |
| Processed Records | The total number of successfully processed records between the poll interval. |
| **PROCESSED DATA** |  |
| Processed Data | The total amount of data successfully processed between the poll interval (in MB). |
| **CONVERTED RECORDS** |  |
| Converted Records | The total number of successfully converted records between the poll interval. |
| Conversion Failed Records | The total number of records that could not be converted between the poll interval. |
| **CONVERTED DATA** |  |
| Converted Data | The total amount of data from successfully converted records between the poll interval (in MB). |
| Conversion Failed Data | The total amount of data from records that could not be converted between the poll intervals (in MB). |
| **DECOMPRESSED DATA** |  |
| Decompressed Data | The total amount of successfully decompressed data between the poll interval (in MB). |
| **DECOMPRESSED RECORDS** |  |
| Decompressed Records | The total number of successfully decompressed records between the poll interval. |
| **ACTIVE DYNAMIC PARTITIONING COUNT** |  |
| Active Dynamic Partition Count | The maximum number of active partitions at the time of polling. |
| **S3 OBJECTS DELIVERED** |  |
| S3 Objects Delivered | The total number of objects delivered to the S3 bucket between the poll interval. |

### Configuration

The **Configuration** tab provides detailed information about the delivery stream, including its general configuration, encryption settings, source configuration, and destination-specific delivery settings.

**Note:** The configuration details displayed vary based on the delivery stream's source and destination types.

| Parameter | Description |
|---|---|
| **CONFIGURATION** |  |
| Version ID | The internal system revision sequence identifier assigned to track the active configuration layout snapshot of this stream asset at the time of polling. |
| Destination ID | The technical topology identifier matches the exact endpoint infrastructure definition within your stream configurations at the time of polling. |
| Creation Time | The localized system timestamp logging when this delivery stream instance was originally provisioned inside the region at the time of polling. |
| Last Update Time | The precise system timestamp tracking when this pipeline's metadata rules or configuration parameters were last modified at the time of polling. |
| Buffering Size | The configured buffer size before data is delivered to the destination (in MiB). |
| Buffering Interval | The buffer interval before data is delivered to the destination (in seconds). |
| **SECURITY CONFIGURATION** |  |
| KMS Key ARN | The fully qualified Amazon Resource Name of the AWS KMS key deployed to handle data block encryption at the time of polling. |
| KMS Key Type | The architectural strategy deployed to handle resource layer cryptographic validation, distinguishing between customer-managed and AWS-owned master keys at the time of polling. |
| KMS Key Status | The present validation availability and lifecycle performance phase of your deployed cryptographic key component at the time of polling. |
| Encryption Failure Type | The exact diagnostic error category generated during the last server-side cryptographic processing failure loop at the time of polling. |
| Encryption Failure Details | The specific low-level system context logs the failure mechanics during an encryption or decryption breakdown at the time of polling. |
| **KINESIS DATA STREAM SOURCE CONFIGURATION** |  |
| Kinesis Stream ARN | The precise global Amazon Resource Name identifying the primary Kinesis Data Stream serving as the ingress source queue for this delivery pipeline at the time of polling. |
| Delivery Start Time | The initial timestamp marking when this delivery stream pipeline began processing data blocks out of the source queue at the time of polling. |
| IAM Role ARN | The target Identity and Access Management execution role provides the delivery stream engine with secure ingress permissions to pull from the source stream at the time of polling. |
| **MANAGED STREAMING FOR KAFKA (MSK) SOURCE CONFIGURATION** |  |
| MSK Cluster ARN | The fully qualified global Amazon Resource Name tracking the parent Managed Streaming for Apache Kafka infrastructure serving data to this stream layout at the time of polling. |
| MSK Topic Name | The target string key denotes the exact Kafka message topic workspace monitored for automated ingress processing at the time of polling. |
| Delivery Start Time | The localized timestamp marking when the stream wrapper initiated consumer client network connections to the target Kafka topic interface at the time of polling. |
| Read Start Time | The genesis offset position timestamp or log index layout used to fetch entries from the target source Kafka queue at the time of polling. |
| **TARGET DESTINATION INFRASTRUCTURE SETTINGS** |  |
| Endpoint | The connection server URL string is deployed to securely direct egress pipeline elements to your target cluster node at the time of polling. |
| Index Name | The active cluster collection label is deployed to receive, categorize, and archive processed ingestion records inside your storage workspace at the time of polling. |
| Index Rotation | The active configuration setting manages how frequently your target cluster switches into fresh index collections at the time of polling. |
| Domain ARN | The global Amazon Resource Name denoting the exact storage domain instance linked to this active pipeline route at the time of polling. |
| Type Name | The active indexing schema layout maps individual data attributes directly into target cluster fields at the time of polling. |
| Document ID Format | The identifier pattern rule used to serialize and map document entries during storage processing at the time of polling. |
| VPC ID | The technical identifier matching the isolated virtual private network infrastructure containing your protected destination cluster node at the time of polling. |
| Extended S3 Bucket | The global Amazon Resource Name locates the primary target storage container handling data stream outputs at the time of polling. |
| File Extension | The customized trailing string suffix is appended to generated data file assets before they are flushed onto the storage system at the time of polling. |
| Dynamic Partitioning | The configuration setting controls whether inline data structures are parsed to dynamically separate storage output paths at the time of polling. |
| Data Format Conversion | The operational enablement phase tracks whether incoming unstructured data lines are dynamically converted into a structured format at the time of polling. |
| Cluster JDBC URL | The exact network string URL deployed to handle transactional database communication routes targeting your analytics cluster layout at the time of polling. |
| Target Table | The explicit table collection label mapping incoming record entries directly inside your destination database layout at the time of polling. |
| Catalog ARN | The unique technical identifier pointing to the active data metadata catalog repository linked to this pipeline target at the time of polling. |
| Catalog Warehouse | The physical data warehouse folder hierarchy directory location containing your catalog layout at the time of polling. |
| Database Name | The formal system label defining the target tabular warehouse instance within your metadata catalog at the time of polling. |
| Table Name | The target relational data row table is selected to store processed log blocks inside the catalog database at the time of polling. |
| Snowflake Account URL | The corporate web connection domain is deployed to authenticate stream payloads with your cloud data warehouse workspace at the time of polling. |
| Snowflake Database | The specialized database catalog partition selected to hold stream output logs at the time of polling. |
| Snowflake Data Loading | The precise loading strategy is configured to sync data blocks into storage rows within the cloud data warehouse at the time of polling. |
| Snowflake Metadata | The data structure column profile maps transactional stream metadata properties directly into storage indexes at the time of polling. |
| HEC Endpoint | The primary HTTP Event Collector interface address is configured to aggregate event payloads inside your ingestion environment at the time of polling. |
| HEC Endpoint Type | The transmission protocol pattern was deployed across the collector gateway interface at the time of polling. |
| HEC Token | The bearer string credential token assigned to authenticate pipeline traffic with the event gateway environment at the time of polling. |
| HEC Acknowledgement Timeout | The maximum allowable timeout constraint is set before a non-acknowledged event collector packet drops out of the queue processing loops at the time of polling. |
| S3 Backup Mode | The logical policy tracks whether raw input payloads are mirrored down to a fallback recovery bucket before target indexing loops execute at the time of polling. |
| S3 Bucket | The dedicated Amazon S3 bucket name configured as the safe target for raw data backstops or transaction crash dumps at the time of polling. |
| S3 Compression | The designated system encoding algorithm used to shrink data payloads before finalizing data dumps into the recovery bucket layout at the time of polling. |
| Buffering Interval | The buffer interval before data is delivered to the destination (in seconds). |
| Retry Duration | The configuration parameter managing the active timeout boundary allowed for delivery operations to retry after initial downstream transport faults at the time of polling (in seconds). |
| Data Processing | The active operational enablement phase tracks your inline data modification or micro-lambda transformation engines at the time of polling. |
| CloudWatch Logging | The active state tracks whether internal event exceptions and operational log lines are piped to CloudWatch Logs groups at the time of polling. |
| CloudWatch Log Group | The dedicated CloudWatch system group destination receives pipeline telemetry data at the time of polling. |
| CloudWatch Log Stream | The active log stream track receives downstream interface log telemetry details within the target group container at the time of polling. |
| Secrets Manager ARN | The secure Amazon Resource Name that locates the managed Secrets Manager vault holding client credentials for the network gateway at the time of polling. |
| Username | The target system administrative identifier used to validate interface credentials during pipeline transport operations when polling. |