Microsoft Azure Machine Learning Monitoring
Microsoft Azure Machine Learning - An Overview
Azure Machine Learning is a cloud platform for training, deploying, and managing machine learning models at scale. Applications Manager's Azure Machine Learning monitoring provides visibility into CPU and GPU utilization, memory, disk I/O, workspace run lifecycle, cluster node and core allocation, model deployment and registration activity, InfiniBand network traffic, and AI agent metrics.
Creating a new Microsoft Azure Machine Learning Monitor
To learn how to create a new Microsoft Azure Machine Learning monitor, click here.
Monitored Parameters
Navigate to the Category View by clicking the Monitors tab. Hover over 'Child Monitors' under Microsoft Azure in the Cloud Apps table, and then select the Machine Learning monitor from the displayed tooltip. This action will display the bulk configuration view for Microsoft Azure Machine Learning in three tabs:
- Availability tab gives the availability history for the past 24 hours or 30 days.
- Performance tab gives the health status and events for the past 24 hours or 30 days.
- List view tab enables you to perform bulk admin configurations.
Following are the list of metrics monitored in Microsoft Azure Machine Learning Monitoring in their corresponding tabs:
Performance Overview
| Parameter | Description |
|---|
| CPU USAGE |
| CPU Usage | The average of utilization of a CPU node in millicores for the Azure Machine Learning at the time of polling (in millicores). |
| CPU Capacity Usage | The average of capacity of a CPU node in millicores for the Azure Machine Learning at the time of polling (in millicores). |
| CPU MEMORY |
| CPU Memory Capacity Usage | The average of maximum memory utilization of a CPU node for the Azure Machine Learning at the time of polling (in MB). |
| CPU Memory Usage | The average of memory utilization of a CPU node for the Azure Machine Learning at the time of polling (in MB). |
| CPU UTILIZATION |
| CPU Utilization | The average of utilization on a CPU node for the Azure Machine Learning at the time of polling (in %). |
| CPU Memory Utilization | The average of memory utilization of a CPU node for the Azure Machine Learning at the time of polling (in %). |
| GPU UTILIZATION |
| GPU Utilization | The average utilization on a GPU node for the Azure Machine Learning at the time of polling (in %). |
| GPU Memory Utilization | The average memory utilization of a GPU device for the Azure Machine Learning at the time of polling (in %). |
| GPU MEMORY |
| GPU Memory Capacity Usage | The average of memory capacity of a GPU device for the Azure Machine Learning at the time of polling (in MB). |
| GPU Memory Usage | The average of memory utilization of a GPU device for the Azure Machine Learning at the time of polling (in MB). |
| GPU USAGE |
| GPU Usage | The average of utilization of a GPU device in milli-GPUs for the Azure Machine Learning at the time of polling (in milli-GPUs). |
| GPU Capacity | The average capacity of a GPU device in milli-GPUs of the Azure Machine Learning at the time of polling (in milli-GPUs). |
| GPU ENERGY |
| GPU Energy | The total number of interval energy in Joules on a GPU node for the Azure Machine Learning between the poll intervals. |
| BLOB STORAGE API CALLS |
| Storage API Success | The total number of Azure Blob Storage API calls success in the Azure Machine Learning between the poll intervals. |
| Storage API Failure | The total number of Azure Blob Storage API calls failure in the Azure Machine Learning between the poll intervals. |
| DISK USAGE |
| Disk Available | The average of available disk space for the Azure Machine Learning at the time of polling (in MB). |
| Disk Used | The average of disk space used by the Azure Machine Learning at the time of polling (in MB). |
| Disk Read | The average of data read from disk for the Azure Machine Learning at the time of polling (in MB). |
| Disk Write | The average of data written into disk for the Azure Machine Learning at the time of polling (in MB). |
| QUOTA UTILIZATION |
| Quota Utilization | The average of quota utilized for the Azure Machine Learning workspace at the time of polling (in %). |
Workspace Runs
| Parameter | Description |
|---|
| RUN STATUS |
| Completed Runs | The total number of runs completed successfully for this workspace in the Azure Machine Learning between the poll intervals. |
| Failed Runs | The total number of runs failed for this workspace in the Azure Machine Learning between the poll intervals. |
| Cancelled Runs | The total number of runs cancelled for this workspace in the Azure Machine Learning between the poll intervals. |
| Started Runs | The total number of runs running for this workspace in the Azure Machine Learning between the poll intervals. |
| Cancel Requested Runs | The total number of runs where cancel was requested for this workspace in the Azure Machine Learning between the poll intervals. |
| Provisioning Runs | The total number of runs that are provisioning in the Azure Machine Learning between the poll intervals. |
| ACTIVE RUNS |
| Preparing Runs | The total number of runs that are preparing for this workspace in the Azure Machine Learning between the poll intervals. |
| Finalizing Runs | The total number of runs entered finalizing state for this workspace in the Azure Machine Learning between the poll intervals. |
| Starting Runs | The total number of runs started for this workspace in the Azure Machine Learning between the poll intervals. |
| Queued Runs | The total number of runs that are queued for this workspace in the Azure Machine Learning between the poll intervals. |
| Not Responding Runs | The total number of runs not responding for this workspace in the Azure Machine Learning between the poll intervals. |
| Not Started Runs | The total number of runs in Not Started state for this workspace in the Azure Machine Learning between the poll intervals. |
| ERRORS AND WARNINGS |
| Errors | The total number of run errors in this workspace for the Azure Machine Learning between the poll intervals. |
| Warnings | The total number of run warnings in this workspace in the Azure Machine Learning between the poll intervals. |
Note: Workspace Runs metrics are mapped under Performance Polling. To adjust the polling interval, navigate to Settings → Performance Polling. In the Optimize Data Collection tab, select Azure Machine Learning as the Monitor Type and Workspace Runs as the Metric Name. Then set Default Polling Status as required, by default opted as Never collect data.
Nodes & Cores
| Parameter | Description |
|---|
| NODES |
| Total Nodes | The total number of nodes includes some of Active Nodes, Idle Nodes, Unusable Nodes, Preempted Nodes, Leaving Nodes in the Azure Machine Learning between the poll intervals. |
| Active Nodes | The total number of active nodes which are actively running a job in the Azure Machine Learning between the poll intervals. |
| Idle Nodes | The total number of idle nodes which are not running any jobs but can accept new job if available in the Azure Machine Learning between the poll intervals. |
| Leaving Nodes | The total number of leaving nodes which just finished processing a job and will go to Idle state in the Azure Machine Learning between the poll intervals. |
| Unusable Nodes | The total number of unusable nodes which are not functional due to some unresolvable issue in the Azure Machine Learning between the poll intervals. |
| Preempted Nodes | The total number of preempted nodes are the low priority nodes which are taken away from the available node pool in the Azure Machine Learning between the poll intervals. |
| CORES |
| Total Cores | The total number of cores in the Azure Machine Learning between the poll intervals. |
| Active Cores | The total number of active cores in the Azure Machine Learning between the poll intervals. |
| Idle Cores | The total number of idle cores in the Azure Machine Learning between the poll intervals. |
| Unusable Cores | The total number of unusable cores in the Azure Machine Learning between the poll intervals. |
| Preempted Cores | The total number of preempted cores in the Azure Machine Learning between the poll intervals. |
| Leaving Cores | The total number of leaving cores in the Azure Machine Learning between the poll intervals. |
Note: Nodes & Cores metrics are mapped under Performance Polling. To adjust the polling interval, navigate to Settings → Performance Polling. In the Optimize Data Collection tab, select Azure Machine Learning as the Monitor Type and Nodes & Cores as the Metric Name. Then set Default Polling Status as required, by default opted as Never collect data.
Network & Models
| Parameter | Description |
|---|
| NETWORK TRAFFIC |
| Network Input | The total amount of Network data received for the Azure Machine Learning workspace between the poll intervals (in MB). |
| Network Output | The total amount of Network data sent for the Azure Machine Learning workspace between the poll intervals (in MB). |
| INFINIBAND TRAFFIC |
| InfiniBand Received | The total amount of network data received over InfiniBand of the Azure Machine Learning between the poll intervals (in MB). |
| InfiniBand Sent | The total amount of network data sent over InfiniBand of the Azure Machine Learning between the poll intervals (in MB). |
| MODEL STATUS |
| Model Deploy Succeeded | The total number of model deployments that succeeded in this workspace in the Azure Machine Learning between the poll intervals. |
| Model Deploy Failed | The total number of model deployments that failed in this workspace in the Azure Machine Learning between the poll intervals. |
| Model Deploy Started | The total number of model deployments started in this workspace in the Azure Machine Learning between the poll intervals. |
| Model Register Succeeded | The total number of model registrations that succeeded in this workspace in the Azure Machine Learning between the poll intervals. |
| Model Register Failed | The total number of model registrations that failed in this workspace in the Azure Machine Learning between the poll intervals. |
Note: Network & Models metrics are mapped under Performance Polling. To adjust the polling interval, navigate to Settings → Performance Polling. In the Optimize Data Collection tab, select Azure Machine Learning as the Monitor Type and Network & Models as the Metric Name. Then set Default Polling Status as required, by default opted as Never collect data.
AI Agent
| Parameter | Description |
|---|
| AGENTS |
| Agents | The total number of events for AI Agents in the Azure Machine Learning between the poll intervals. |
| RUNS |
| Runs | The total number of runs by AI Agents in this workspace in the Azure Machine Learning between the poll intervals. |
| TOKENS |
| Tokens | The total number of tokens by AI Agents in the Azure Machine Learning between the poll intervals. |
| THREADS |
| Threads | The total number of events for AI Agent threads in the Azure Machine Learning between the poll intervals. |
| MESSAGES |
| Messages | The total number of events for AI Agent messages in the Azure Machine Learning between the poll intervals. |
| INDEXED FILES |
| Indexed Files | The total number of files indexed for file search in the Azure Machine Learning between the poll intervals. |
| TOOLBOX CALLS |
| Toolbox Calls | The total number of tool calls made by AI Agents in this workspace in the Azure Machine Learning between the poll intervals. |
| Toolbox Call Tool Calls | The total number of toolbox call_tool call events in the Azure Machine Learning between the poll intervals. |
| TOOLBOX TOOLS |
| Toolbox Tools Deferred | The total number of tools deferred by Tool Search in toolbox tools/list responses in the Azure Machine Learning between the poll intervals. |
| Toolbox Tools Visible | The total number of tools visible to the model in toolbox tools/list responses in the Azure Machine Learning between the poll intervals. |
| TOOLBOX SEARCH |
| Toolbox Tool Search Calls | The total number of toolbox tool_search call events in the Azure Machine Learning between the poll intervals. |
| Toolbox Tool Search Enabled | The total number of tool Search-enabled toolbox version creation events in the Azure Machine Learning between the poll intervals. |
| Toolbox Tool Search Results | The total number of tools returned by successful toolbox tool_search calls in the Azure Machine Learning between the poll intervals. |
Note: AI Agent metrics are mapped under Performance Polling. To adjust the polling interval, navigate to Settings → Performance Polling. In the Optimize Data Collection tab, select Azure Machine Learning as the Monitor Type and AI Agent as the Metric Name. Then set Default Polling Status as required, by default opted as Never collect data.
Configuration
| Parameter | Description |
|---|
| GENERAL CONFIGURATION |
| Resource Group Name | The name of the Azure resource group that contains the Machine Learning workspace. |
| Location | The Azure region where the Machine Learning workspace is deployed. |
| Provisioning State | The provisioning state of the Machine Learning workspace. Possible values: Unknown, Updating, Creating, Deleting, Succeeded, Failed, Canceled. |
| SKU Name | The SKU name associated with the Machine Learning workspace. |
| Kind | The kind of Machine Learning workspace. |
| Public Network Access | Indicates whether public network access is enabled for the Machine Learning workspace. Possible values: Enabled, Disabled. |
| RESOURCE METADATA |
| Creation Time | The timestamp when the Machine Learning workspace was created. |
| Created By | The identity that created the Machine Learning workspace. |
| Creator Identity Type | The type of identity that created the workspace. Possible values: User, Application, ManagedIdentity, Key. |
| Last Modified Time | The timestamp when the Machine Learning workspace was last modified. |
| Last Modified By | The identity that last modified the Machine Learning workspace. |
| Last Modifier Identity Type | The type of identity that last modified the workspace. Possible values: User, Application, ManagedIdentity, Key. |
| SECURITY AND NETWORK |
| Allow Role Assignment on RG | Indicates whether role assignments can be created at the resource group level for the Machine Learning workspace. |
| HBI Workspace | Indicates whether the workspace is designated as an HBI (High Business Impact) workspace. |
| Data Isolation | Indicates whether data associated with the workspace is isolated. |
| Service Side CMK Encryption | Indicates whether service-side Customer Managed Key (CMK) encryption is enabled for the workspace. Possible values: Enabled, Disabled. |
| Software Bill of Materials | Indicates whether Software Bill of Materials (SBOM) generation is enabled for the workspace. |
| WORKSPACE SETTINGS |
| Provision Network Now | Indicates whether the managed virtual network is provisioned immediately. |
| Soft Delete | Indicates whether soft delete is enabled for the workspace. |
| Storage HNS | Indicates whether the storage associated with the workspace has hierarchical namespace (HNS) enabled. |
| System Datastores Auth Mode | The authentication mode used to access the workspace's system datastores. |
| V1 Legacy Mode | Indicates whether V1 legacy mode is enabled for the workspace. |
