AWS Glue Monitoring
AWS Glue - Overview
AWS Glue is a fully managed, serverless data integration service that simplifies discovering, preparing, moving, and integrating data from multiple sources for analytics, machine learning, and application development. It provides ETL (Extract, Transform, Load) capabilities through Glue Jobs - which run scripts to process and transform data - and data cataloging through Glue Crawlers, which automatically scan data sources, infer schemas, and populate the AWS Glue Data Catalog.
Monitoring AWS Glue is essential for maintaining efficient ETL pipeline performance and data quality. Applications Manager's AWS Glue monitoring tool tracks key metrics for both Glue Jobs and Crawlers, including CPU and memory utilization, S3 data movement rates, task execution statistics, job run state distribution, and crawler runtime and table activity. With proactive alerts and historical trend analysis, the tool helps identify job failures, performance bottlenecks, and resource inefficiencies before they impact downstream data processing workflows.
Creating a new AWS Glue Job/ AWS Glue Crawler monitor
To learn how to create a new AWS Glue monitor, refer here.
Monitored Parameters
Go to the Monitors Category View by clicking the Monitors tab. Click on the Glue instance available under Amazon in the Cloud Apps section. Displayed below is the AWS Glue bulk configuration view, distributed into three tabs:
- Availability tab gives the availability history for the past 24 hours or 30 days.
- Performance tab gives the health status and events for the past 24 hours or 30 days.
- List view tab enables you to perform bulk admin configurations.
Click on the monitor name to see all the AWS Glue Crawler metrics listed under the following tabs:
Click on the monitor name to see all the AWS Glue Job metrics listed under the following tabs:
AWS Glue Crawler Metrics
Performance Overview
| Parameter | Description |
|---|
| LAST CRAWL DETAILS |
| Start Time | The date and time at which the crawler's most recent crawl execution began. |
| Log Group Name | The CloudWatch log group associated with the last crawler execution, where execution logs are stored. |
| Log Stream Name | The specific CloudWatch log stream containing the log events for the last crawler execution. |
| Crawl Error Message | The detailed error message or exception description is returned if the last crawl encountered an issue or failed. |
| Last Crawl Status | The current operational status of the most recent crawl run.
Possible values: SUCCEEDED, CANCELLED, FAILED. |
| FAILED RUN PERCENTAGE |
| Failed Run Percentage | The percentage of runs that failed at the time of polling (in %). |
| STOPPED RUN PERCENTAGE |
| Stopped Run Percentage | The percentage of runs that were stopped before finishing at the time of polling (in %). |
| TABLE ACTIVITY |
| Created Tables | The total number of new tables added by the crawler at the time of polling. |
| Updated Tables | The total number of existing tables modified by the crawler at the time of polling, where high numbers mean your existing data definitions are changing frequently. |
| Deleted Tables | The total number of tables removed by the crawler at the time of polling, showing when previously cataloged data paths are no longer available. |
| RUNTIME ACTIVITY |
| Last Runtime | The total execution time of the crawler's most recent run at the time of polling, where a sudden increase shows that processing took longer than usual (in seconds). |
| Median Runtime | The median runtime value of all successful runs is calculated at the time of polling, providing a reliable baseline to check if current performance is normal (in seconds). |
| ESTIMATED REMAINING TIME |
| Estimated Remaining Time | The estimated time left before an active crawl finishes execution at the time of polling, where stable or increasing numbers indicate an extended processing cycle (in seconds). |
| RUN STATE DISTRIBUTION |
| Failed Runs | The total number of runs that ended in failure at the time of polling. |
| Stopped Runs | The total number of runs that were stopped before finishing at the time of polling. |
| Completed Runs | The total number of runs that finished successfully at the time of polling. |
| Total Runs | The total number of times this crawler has run at the time of polling. |
Crawler Runs
Note: The Crawler Runs data collection is mapped under Performance Polling and is enabled by default with a default polling interval of 6 hours. To enable or disable it, navigate to Settings → Performance Polling, select the Optimize Data Collection tab, choose Glue Crawler as the monitor type, select Crawler Runs as the metric name, and choose the desired Default Polling Status.
The Crawler Runs tab displays a list of recent crawl executions for this crawler. Each row represents a single crawl run with its status and timing details.
| Parameter | Description |
|---|
| TABLE CRAWLER RUNS |
| Crawl ID | The unique identifier for the crawl run. |
| Log Group Name | The CloudWatch log group associated with this crawl execution. |
| Log Stream Name | The specific CloudWatch log stream for this crawl execution. |
| Start Time | The date and time when this crawl run started. |
| End Time | The date and time when this crawl run ended. |
| Duration | The total execution time of this crawl run (in seconds). |
| DPU Hour | The number of DPU-hours consumed by this crawl run (in hours). |
| Crawl Error Message | The error message is returned if this crawl run encountered an issue or failed. |
| Crawl Status | The final status of this crawl run.
Possible values: SUCCEEDED, CANCELLED, FAILED, RUNNING. |
Classifiers
Note: The Classifiers data collection is mapped under Performance Polling and disabled by default. To enable/disable it, navigate to Settings → Performance Polling, select the Optimize Data Collection tab, choose Glue Crawler as the monitor type, select Classifiers as the metric name, and choose the desired Default Polling Status.
The Classifiers tab lists the custom classifiers configured for this crawler, which are used to determine the schema and format of the data sources being crawled.
| Parameter | Description |
|---|
| TABLE CLASSIFIERS |
| Classifier Name | The name of the classifier. |
| Classifier Type | The type of the classifier (for example, GrokClassifier, XMLClassifier, JsonClassifier, CsvClassifier). |
| Creation Time | The date and time when the classifier was created. |
Crawler Source
The Crawler Source tab lists the data source targets configured for this crawler, showing where the crawler scans for data to catalog.
| Parameter | Description |
|---|
| TABLE CRAWLER SOURCE |
| Source Name | The name of the data source target. |
| Connection Name | The name of the connection associated with this data source, if applicable. |
| Source Path | The path or location of the data source (for example, an S3 bucket path or JDBC table). |
| Tables Count | The number of tables included from this data source. |
Configuration
| Parameter | Description |
|---|
| CRAWLER CONFIGURATION |
| Crawler State | The current operational state of the crawler.
Possible values: READY, RUNNING, STOPPING. |
| Version | The version of the crawler definition. |
| Description | A short description of the crawler. |
| Database Name | The name of the target AWS Glue Data Catalog database. |
| Created Time | The date and time when the crawler was created. |
| Crawler Role | The IAM role used by the crawler to access data stores and write metadata to the Data Catalog. |
| Schedule Expression | The cron expression that defines the schedule for running the crawler. |
| Schedule State | The current state of the scheduled crawler trigger. |
| Still Estimating | Indicates whether the crawler is currently estimating the remaining execution time for the active crawl run. |
AWS Glue Job Metrics
Performance Overview
| Parameter | Description |
|---|
| SYSTEM LOAD |
| CPU System Load | The average percentage of CPU system load across all executors at the time of polling (in %). |
| Driver System Load | The average percentage of CPU system load on the driver node at the time of polling (in %). |
| HEAP USAGE |
| JVM Heap Usage | The average percentage of JVM heap memory used across all executors at the time of polling (in %). |
| Driver JVM Heap Usage | The average percentage of JVM heap memory used on the driver node at the time of polling (in %). |
| ETL DATA MOVEMENT |
| S3 Data Read Rate | The rate at which data is read from Amazon S3 per second between the poll interval (in MB/s). |
| S3 Data Written Rate | The rate at which data is being written to S3 between the poll interval (in MB/s). |
| S3 Data Read | The total amount of data read from S3 between the poll interval (in MB). |
| S3 Data Written | The total amount of data written to S3 between the poll interval (in MB). |
| DRIVER DATA MOVEMENT |
| Driver S3 Data Read Rate | The rate at which the driver is reading data from S3 per second between the poll interval (in MB/s). |
| Driver S3 Data Written Rate | The rate at which the driver is writing data to S3 per second between the poll interval (in MB/s). |
| Driver S3 Data Read | The total amount of data read from S3 by the driver between the poll interval (in MB). |
| Driver S3 Data Written | The total amount of data written to S3 by the driver between the poll interval (in MB). |
| DATA SHUFFLE ACROSS EXECUTORS |
| Shuffle Data Read Rate | The rate at which shuffle data is being read per second between the poll interval (in MB/s). |
| Shuffle Data Written Rate | The rate at which shuffle data is being written per second between the poll interval (in MB/s). |
| Shuffle Data Read | The total amount of shuffle data read during inter-executor data transfer between the poll interval (in MB). |
| Shuffle Data Written | The total amount of shuffle data written during inter-executor data transfer between the poll interval (in MB). |
| TASKS ACTIVITY |
| Failed Tasks | The total number of tasks that failed during the job run between the poll interval. |
| Killed Tasks | The total number of tasks that were killed during the job run between the poll interval. |
| Completed Tasks | The total number of tasks completed during the job run between the poll interval. |
| Completed Stages | The total number of stages completed during the job run between the poll interval. |
| DATA READ RATE |
| Data Read Rate | The rate at which data is being read between the poll interval (in MB/s). |
| Data Read | The total amount of data read during the job run between the poll interval (in MB). |
| RECORDS READ RATE |
| Records Read Rate | The rate at which records are being read per minute between the poll interval (in records/min). |
| Records Read | The total number of records read during the job run between the poll interval (in records/min). |
| MEMORY PROFILE |
| Driver Memory Used | The average amount of memory used by the driver node at the time of polling (in MB). |
| Executor Memory Used | The average amount of memory used by all executor nodes at the time of polling (in MB). |
| DISK SPACE USED |
| Disk Space Used | The amount of disk space used by the job at the time of polling (in MB). |
| ELAPSED TIME |
| Elapsed Time | The total elapsed time of the job run between the poll interval (in ms). |
| ACTIVE EXECUTORS |
| Active Executors | The average number of currently active executors running tasks at the time of polling. |
| Max Needed Executors | The maximum number of executors needed for the job based on workload at the time of polling. |
| DRIVER STREAMING RECORDS |
| Driver Streaming Records | The total number of records processed by the driver during streaming jobs at the time of polling. |
| DRIVER STREAMING BATCH PROCESSING TIME |
| Driver Streaming Batch Processing Time | The average time taken to process each batch during streaming jobs at the time of polling (in ms). |
Job Runs
Note: The Job Runs data collection is mapped under Performance Polling and disabled by default. To enable/disable it, navigate to Settings → Performance Polling, select the Optimize Data Collection tab, choose Glue Job as the monitor type, select Job Runs as the metric name, and choose the desired Default Polling Status.
| Parameter | Description |
|---|
| JOB RUN PERCENTAGE |
| Completed Runs Percentage | The percentage of job runs that completed successfully at the time of polling (in %). |
| Failed Runs Percentage | The percentage of job runs that failed at the time of polling (in %). |
| Waiting Runs Percentage | The percentage of job runs that are waiting to be executed at the time of polling (in %). |
| Canceled Runs Percentage | The percentage of job runs that were canceled/stopped by the user at the time of polling (in %). |
| Error Runs Percentage | The percentage of job runs that encountered an error at the time of polling (in %). |
| Timeout Runs Percentage | The percentage of job runs that timed out at the time of polling (in %). |
| TABLE JOB RUNS |
| Job Run ID | The unique identifier for the job run. |
| Start Time | The date and time when this job run started. |
| End Time | The date and time when this job run ended. |
| Duration(s) | The total execution time of this job run (in seconds). |
| Attempt | The attempt number for this job run indicates how many times it has been retried. |
| Log Group Name | The CloudWatch log group associated with this job run. |
| Error Message | The error message is returned if this job run failed or encountered an error. |
| State | The final state of the job run.
Possible values: STARTING, RUNNING, STOPPING, STOPPED, SUCCEEDED, FAILED, TIMEOUT, ERROR, WAITING, EXPIRED. |
| JOB RUN STATE DISTRIBUTION |
| Failed Runs | The number of job runs that failed at the time of polling. |
| Canceled Runs | The number of job runs that were canceled/stopped by the user at the time of polling. |
| Error Runs | The number of job runs that encountered an error at the time of polling. |
| Timeout Runs | The number of job runs that timed out at the time of polling. |
| Completed Runs | The total number of job runs that completed successfully at the time of polling. |
| Waiting Runs | The number of job runs waiting to be executed at the time of polling. |
| Total Runs | The total number of job runs at the time of polling. |
Triggers
The Triggers tab lists the triggers associated with this Glue Job.
| Parameter | Description |
|---|
| TABLE TRIGGERS |
| Trigger Name | The name of the trigger associated with this job. |
| Trigger Type | The type of trigger.
Possible values: SCHEDULED, CONDITIONAL, ON_DEMAND, EVENT. |
| Description | A short description of the trigger. |
| Workflow Name | The name of the workflow that this trigger is associated with. |
| State | The current state of the trigger.
Possible values: CREATING, CREATED, ACTIVATING, ACTIVATED, DEACTIVATING, DEACTIVATED, DELETING, UPDATING. |
Configuration
| Parameter | Description |
|---|
| JOB CONFIGURATION |
| Job Type | The type of command that executes the job.
Possible values: glueetl, pythonshell, spark, ray. |
| Job Mode | The job authoring mode.
Possible values: SCRIPT, VISUAL, NOTEBOOK. |
| Glue Version | The version of AWS Glue used by the job (for example, 4.0). |
| Description | A short description of the job. |
| Script Location | The S3 path of the script to be executed by the job. |
| Creation Time | The date and time when the job was created. |
| Last Updation Time | The date and time when the job was last updated. |
| WORKER CONFIGURATION |
| Worker Type | The type of predefined worker allocated for the job.
Possible values: Standard, G.1X, G.2X, G.025X, Z.2X. |
| Number of Workers | The number of workers allocated for the job run. |
| Max Capacity | The maximum number of AWS Glue data processing units (DPUs) that can be allocated to the job. |
| Execution Class | The execution class for the job.
Possible values: FLEX, STANDARD. |
| Max Retries | The maximum number of times AWS Glue will retry a job run if it fails. |
| Max Concurrent Runs | The maximum number of concurrent runs allowed for the job. |
| Maintenance Window | The maintenance window is configured for the job. |
