Microsoft Azure Data Factory
Microsoft Azure Data Factory - An Overview
Microsoft Azure Data Factory (ADF) is a cloud-based data integration service that enables organizations to orchestrate and automate the movement and transformation of data across hybrid and cloud environments. Applications Manager’s Azure Data Factory Monitoring helps you track pipeline runs, activity performance, and failure trends to optimize data integration processes and ensure seamless workflow orchestration in your Azure environment.
Creating a new Azure Data Factory Monitor
To learn how to create a new Microsoft Azure Data Factory Monitor, click here.
Monitored Parameters
Navigate to the Category View by clicking the Monitors tab. Hover over 'Child Monitors' under Microsoft Azure in the Cloud Apps table, and then select the Data Factories monitor from the displayed tooltip. The bulk configuration view will display metrics categorized under the following tabs:
- Availability tab displays the availability history for the past 24 hours or 30 days.
- Performance tab shows the Health Status and relevant events.
- List View supports bulk admin configurations.
The Microsoft Azure monitor provides a brief detail of the Azure Data Factory under the given subscription. Following are the list of metrics monitored in Azure Data Factory Monitoring in their corresponding tabs:
Performance Overview
| Parameter | Description |
|---|
| SUCCEEDED PIPELINE RUNS |
| Succeeded Pipeline Runs | The total number of pipeline runs that completed successfully between the poll interval. |
| FAILED PIPELINE RUNS |
| Failed Pipeline Runs | The total number of pipeline runs that failed execution between the poll interval. |
| CANCELLED PIPELINE RUNS |
| Cancelled Pipeline Runs | The total number of pipeline runs that were cancelled while they were running between the poll interval. |
| ELAPSED PIPELINE RUNS |
| Elapsed Pipeline Runs | The total number of pipeline runs that exceeded the configured elapsed time value between the poll interval. |
| SUCCEEDED TRIGGER RUNS |
| Succeeded Trigger Runs | The total number of trigger runs that successfully initiated pipelines between the poll interval. |
| FAILED TRIGGER RUNS |
| Failed Trigger Runs | The total number of trigger runs that failed to initiate pipelines between the poll interval. |
| CANCELLED TRIGGER RUNS |
| Cancelled Trigger Runs | The total number of trigger runs that were cancelled before starting a pipeline between the poll interval. |
| SUCCEEDED ACTIVITY RUNS |
| Succeeded Activity Runs | The total number of activity runs that completed successfully between the poll interval. |
| FAILED ACTIVITY RUNS |
| Failed Activity Runs | The total number of activity runs that failed execution between the poll interval. |
| CANCELLED ACTIVITY RUNS |
| Cancelled Activity Runs | The total number of activity runs that were cancelled while running between the poll interval. |
Run Statistics
| Parameter | Description |
|---|
| PIPELINE SUCCESS RATE |
| Pipeline Success Rate | The percentage of successful pipeline runs based on the runs displayed. |
| TRIGGER SUCCESS RATE |
| Trigger Success Rate | The percentage of successful trigger runs based on the runs displayed. |
| Pipeline Runs |
| Pipeline Run Identifier | Identifier of a pipeline run. |
| Pipeline Name | The name of the pipeline. |
| Status | Status of the pipeline run. Possible values:Queued, InProgress, Succeeded, Failed, Canceling, Cancelled. |
| Start Time | The start time of the pipeline run. |
| End Time | The end time of the pipeline run. |
| Total Duration (sec) | The total execution duration of the pipeline run (in seconds). |
| Trigger Run Identifier | Identifier of the trigger that initiated the pipeline run. |
| Trigger Name | Name of the trigger. |
| Trigger Runs |
| Trigger Run Identifier | Identifier of the trigger run. |
| Trigger Name | Name of the trigger. |
| Trigger Type | Type of the trigger. |
| Status | Status of the trigger run. Possible values:Succeeded, Failed, InProgress. |
| Scheduled Time | The time the trigger was scheduled to run. |
| Triggered Time | The time the trigger executed. |
| Delay | The delay before the trigger execution. |
| No. of Pipelines | The number of pipelines triggered by the trigger run. |
Notes:- The Pipeline Runs and Trigger Runs metrics are categorized under Performance Polling.
- To modify the polling interval, navigate to Settings → Performance Polling. In the Optimize Data Collection tab, select Azure Data Factory as the Monitor Type and Pipeline Runs or Trigger Runs as the Metric Name. Set the Default Polling Status as required (preferred interval: 1—24 hours).
- If a customized polling interval greater than 24 hours is configured, only the runs from the last 24 hours will be fetched.
- By default, Pipeline Runs and Trigger Runs are disabled. When enabled, the tables display the most recent 200 runs for the configured polling interval.
- To modify this limit, navigate to Settings → Performance Polling → Optimize Data Collection, select Azure Data Factory as the Monitor Type, and choose Pipeline Runs or Trigger Runs as the Metric Name. Specify a value between 50 and 500 for the Number of Pipeline/Trigger Runs to Fetch.
Integration Runtime
| Parameter | Description |
|---|
| Integration Runtime CPU Utilization | The average percentage of CPU utilized by the Integration Runtime at the time of polling. |
| Integration Runtime Available Memory | The average amount of available memory on the IR host machine at the time of polling (in MB). |
| Integration Runtime Available Nodes | The average number of available nodes capable of processing data integration activities at the time of polling. |
| Integration Runtime Queue Duration | The average time taken for an activity to begin execution after entering the queue between the poll interval (in seconds). |
| Integration Runtime Queue Length | The average number of activities waiting in the queue between the poll interval. |
Configuration
| Parameter | Description |
|---|
| Resource Group Name | The name of the Resource Group containing the Data Factory. |
| Location | The location of the Azure Data Factory. |
| Provisioning State | The current provisioning state of the Data Factory. |
| Creation Time | The time when the Data Factory was created. |
| Public Network Access | Indicates whether public network access is enabled or disabled. |
