Microsoft Azure Data Explorer Cluster Monitoring
Microsoft Azure Data Explorer Cluster - An Overview
Azure Data Explorer (Kusto) is a fast, fully managed data analytics service for real-time analysis on large volumes of streaming data. Applications Manager's Azure Data Explorer Cluster monitoring tracks CPU and cache utilization, ingestion latency and volume, batch processing health, query performance, streaming ingestion, continuous export, and materialized view health.
Creating a new Microsoft Azure Data Explorer Cluster Monitor
To learn how to create a new Microsoft Azure Data Explorer Cluster monitor, click here.
Monitored Parameters
Navigate to the Category View by clicking the Monitors tab. Hover over 'Child Monitors' under Microsoft Azure in the Cloud Apps table, and then select the Data Explorer Clusters monitor from the displayed tooltip. This action will display the bulk configuration view for Microsoft Azure Data Explorer Cluster in three tabs:
- Availability tab gives the availability history for the past 24 hours or 30 days.
- Performance tab gives the health status and events for the past 24 hours or 30 days.
- List view tab enables you to perform bulk admin configurations.
Following are the list of metrics monitored in Microsoft Azure Data Explorer Cluster Monitoring in their corresponding tabs:
Performance Overview
| Parameter | Description |
|---|
| UTILIZATION |
| CPU | The average CPU utilization level in the Azure Data Explorer cluster at the time of polling (in %). |
| Ingestion Utilization | The average ratio of used ingestion slots in the Azure Data Explorer cluster between the poll intervals (in %). |
| Export Utilization | The maximum export utilization of the Azure Data Explorer cluster between the poll intervals (in %). |
| Cache Utilization Factor | The average cache utilization factor of the Azure Data Explorer cluster between the poll intervals (in %). |
| QUERY PERFORMANCE |
| Total Concurrent Queries | The maximum number of concurrent queries of the Azure Data Explorer cluster between the poll intervals. |
| Total Throttled Queries | The maximum number of throttled queries of the Azure Data Explorer cluster between the poll intervals. |
| Total Throttled Commands | The total number of throttled commands of the Azure Data Explorer cluster between the poll intervals. |
| INGESTION |
| Keep Alive | The average sanity check indicates the cluster responds to queries in the Azure Data Explorer cluster between the poll intervals. |
| Total Extents | The average data extents in the Azure Data Explorer cluster between the poll intervals. |
| Instance | The total number of instances in the Azure Data Explorer cluster between the poll intervals. |
| LATENCY |
| Follower Latency | The average follower databases synchronize changes in the leader databases of the Azure Data Explorer cluster between the poll intervals (in ms). |
Query Performance
| Parameter | Description |
|---|
| QUERY RESULT |
| Query Result | The number of concurrent queries of the Azure Data Explorer cluster between the poll intervals. |
| QUERY DURATION |
| Query Duration | The total number of query duration of the Azure Data Explorer cluster between the poll intervals (in ms). |
| LATENCY |
| Weak Consistency Latency | The average latency between the previous metadata sync and the next one of the Azure Data Explorer cluster between the poll intervals (in s). |
Note: Query Performance metrics are mapped under Performance Polling. To adjust the polling interval, navigate to Settings → Performance Polling. In the Optimize Data Collection tab, select Azure Data Explorer Cluster as the Monitor Type and Query Performance as the Metric Name. Then set Default Polling Status as required, by default opted as Never collect data.
Ingestion Health
| Parameter | Description |
|---|
| INGESTION RESULT |
| Ingestion Result | The total number of sources that either failed or succeeded to be ingested in the Azure Data Explorer cluster between the poll intervals. |
| LATENCY |
| Ingestion Latency | The average latency of data ingested from the time it is received by the cluster until it is ready for querying in the Azure Data Explorer cluster between the poll intervals (in s). |
| Stage Latency | The average cumulative time from when a message is discovered until it is received by the reporting component for processing in the Azure Data Explorer cluster between the poll intervals (in s). |
| Discovery Latency | The average time from when a message is enqueued or an event is created until it is discovered by the data connection of the Azure Data Explorer cluster between the poll intervals (in s). |
| Batch Duration | The average duration of the aggregation phase in the ingestion flow of the Azure Data Explorer cluster between the poll intervals (in s). |
| DATA SIZE |
| Ingestion Volume | The total volume of ingested data to the Azure Data Explorer cluster between the poll intervals (in MB). |
| Batch Size | The average uncompressed expected data size of an aggregated batch for ingestion in the Azure Data Explorer cluster between the poll intervals (in MB). |
| Received Data Size | The total number of size data received by data connection for the Azure Data Explorer cluster between the poll intervals (in MB). |
| BATCH PROCESSING |
| Batches Processed | The total number of batches aggregated for ingestion in the Azure Data Explorer cluster between the poll intervals. |
| Batch Blobs | The average data sources in an aggregated batch for ingestion in the Azure Data Explorer cluster between the poll intervals. |
| BLOB INGESTION |
| Blobs Received | The total number of blobs received from input stream by a component in the Azure Data Explorer cluster between the poll intervals. |
| Blobs Processed | The total number of blobs processed by a component in the Azure Data Explorer cluster between the poll intervals. |
| Blobs Dropped | The total number of blobs permanently rejected by a component in the Azure Data Explorer cluster between the poll intervals. |
| EVENTS |
| Events Received | The total number of events received by data connection in the Azure Data Explorer cluster between the poll intervals. |
| Events Processed | The total number of events processed by the Azure Data Explorer cluster between the poll intervals. |
| Events Dropped | The total number of events dropped permanently by data connection in the Azure Data Explorer cluster between the poll intervals. |
| QUEUE LENGTH |
| Queue Length | The average pending messages in a component's queue of the Azure Data Explorer cluster between the poll intervals. |
| QUEUE OLDEST MESSAGE |
| Queue Oldest Message | The average time from when the oldest message in queue was inserted of the Azure Data Explorer cluster between the poll intervals (in s). |
Note: Ingestion Health metrics are mapped under Performance Polling. To adjust the polling interval, navigate to Settings → Performance Polling. In the Optimize Data Collection tab, select Azure Data Explorer Cluster as the Monitor Type and Ingestion Health as the Metric Name. Then set Default Polling Status as required, by default opted as Never collect data.
Materialized View & Export Health
| Parameter | Description |
|---|
| CONTINUOUS EXPORT |
| Continuous Exported Records | The total number of records exported, fired for every storage artifact written during the export operation in the Azure Data Explorer cluster between the poll intervals. |
| Pending Continuous Export | The maximum pending continuous export jobs ready for execution in the Azure Data Explorer cluster between the poll intervals. |
| Continuous Export Result | The average time for follower databases to synchronize changes from leader databases in the Azure Data Explorer cluster between the poll intervals. |
| LATENESS CONTINUOUS EXPORT |
| Lateness Continuous Export | The maximum lateness reported by the continuous export jobs in the Azure Data Explorer cluster between the poll intervals (in minutes). |
| MATERIALIZED VIEW HEALTH |
| Materialized View Health | Indicates the Materialized View Health for the Azure Data Explorer Cluster between the poll intervals (Aggregation Type: Average). Interpreted from numeric values: . |
| MATERIALIZED VIEWS |
| Materialized View Result | The average result of materialized view process in the Azure Data Explorer cluster between the poll intervals. |
| Materialized View Records | The average records in the non-materialized part of the view in the Azure Data Explorer cluster between the poll intervals. |
| Materialized View Extents Rebuild | The average extents rebuild by the Azure Data Explorer cluster between the poll intervals. |
| Materialized View Data Loss | The maximum indicates potential data loss in materialized view of the Azure Data Explorer cluster between the poll intervals. |
| MATERIALIZED VIEW AGE |
| Materialized View Age | The maximum materialized view age of the Azure Data Explorer cluster between the poll intervals (in s). |
Note: Materialized View & Export Health metrics are mapped under Performance Polling. To adjust the polling interval, navigate to Settings → Performance Polling. In the Optimize Data Collection tab, select Azure Data Explorer Cluster as the Monitor Type and Materialized View & Export Health as the Metric Name. Then set Default Polling Status as required, by default opted as Never collect data.
Streaming Ingestion
| Parameter | Description |
|---|
| STREAMING INGEST RESULTS |
| Streaming Ingest Results | The number of streaming ingestion results in the Azure Data Explorer cluster between the poll intervals. |
| STREAMING INGEST DATA RATE |
| Streaming Ingest Data Rate | The average streaming ingestion data rate of the Azure Data Explorer cluster between the poll intervals (in MB). |
| STREAMING INGEST DURATION |
| Streaming Ingest Duration | The maximum streaming ingestion duration of the Azure Data Explorer cluster between the poll intervals (in ms). |
Note: Streaming Ingestion metrics are mapped under Performance Polling. To adjust the polling interval, navigate to Settings → Performance Polling. In the Optimize Data Collection tab, select Azure Data Explorer Cluster as the Monitor Type and Streaming Ingestion as the Metric Name. Then set Default Polling Status as required, by default opted as Never collect data.
Configuration
| Parameter | Description |
|---|
| GENERAL CONFIGURATION |
| Resource Group Name | The name of the Azure resource group that contains the Data Explorer cluster. |
| Location | The Azure region where the Data Explorer cluster is deployed. |
| Provisioning State | The provisioning state of the Data Explorer cluster. Possible values: Running, Creating, Deleting, Succeeded, Failed, Moving, Canceled. |
| Status | The current state of the Data Explorer cluster. Possible values: Creating, Unavailable, Running, Deleting, Deleted, Stopping, Stopped, Starting, Updating, Migrated. |
| Public Network Access | Indicates whether public network access to the Data Explorer cluster is enabled. Possible values: Enabled, Disabled, SecuredByPerimeter. |
| SKU DETAILS |
| SKU Name | The SKU name of the Data Explorer cluster. |
| SKU Tier | The SKU tier of the Data Explorer cluster. Possible values: Basic, Standard. |
| CLUSTER DETAILS |
| Data Ingestion URI | The URI used to ingest data into the Data Explorer cluster. |
| Disk Encryption Enabled | Indicates whether disk encryption is enabled for the cluster. |
| Purge Enabled | Indicates whether purge operations are enabled for the cluster. |
| Streaming Ingestion Enabled | Indicates whether streaming ingestion is enabled for the cluster. |
| Engine Type | The engine type of the Data Explorer cluster. Possible values: V2, V3. |
| Cluster URI | The URI of the Data Explorer cluster. |
