Amazon Web Services Monitoring



Amazon Web Services (AWS) - Overview

Amazon Web Services (AWS) is a comprehensive cloud computing platform from Amazon. AWS offers an array of cloud computing services to build on from data centers spread across availability zones in regions across the world. However, you would still need a proper AWS monitoring software to be able to monitor and understand what is going on when you deploy and consume such a wide-ranging set of services.

From your infrastructure and applications to your end-user experience, Applications Manager's AWS monitoring provides robust monitoring of several core Amazon Web Services under your Amazon account. With all-in-one AWS cloud monitoring tools like Applications Manager in place, users can obtain a single pane view over various AWS services like:

Supported Services :

In this help document, you will learn how to get started with AWS performance monitoring, along with the list of parameters that are monitored using Applications Manager.

Supported Regions

You can collect metrics, visualize resource usage and alert on operational behavior for all supported services across your AWS cloud environment for the following regions:

Creating a new Amazon monitor

Prerequisites for setting up Amazon performance monitoring: Click here.

Using the REST API to add a new Amazon monitor: Click here

To create a new Amazon monitor, follow the steps given below:

  1. Click on New Monitor link.
  2. Select Amazon under Cloud Apps category.
  3. Specify the Display Name of the Amazon server
  4. Select the Amazon Services you wish to discover and monitor from the drop-down menu.
  5. Enter the Access Key Id of the AWS for accessing the AWS through the API. The access key has 20 alpha-numeric characters.
  6. Enter the Secret Access Key of the AWS. The secret key should be 40 alpha-numeric characters long.
  7. Choose the Account Type. The default account type when adding a new Amazon monitor is AWS Global. Choose AWS China as the Account Type to monitor services in the AWS China regions, or choose AWS Gov Cloud to monitor services in the AWS GovCloud regions.
  8. Based on the Account Type chosen, select the regions to monitor from either Global Regions, China Regions, or Gov Cloud Regions drop-down menu.*
  9. Select a Child Discovery option using any of the following option:*
    • Do not discover: Adds amazon monitor and discovery will not happen.
    • Discover and unmanage: Discovers and unmanage all the child monitors.
    • Discover and monitor child monitors: Discovers and monitors all the child monitors.
  10. If you enable the Advanced Settings option, specify the following details: *
    • Action on Deleted Resources: Select the action you want to perform on deleted resources using any one of the following options:
        • Continue Alerting: The option will keep the deleted instances in the console and deleted alerts will be shown in the child and parent monitor.
        • Stop Alerting: The option will not affect the health of the deleted instances.
        • Unmanage Resource: The option will unmanage the deleted resources and stop the polling for the same.
        • Delete Resource: The option will delete the monitors which are deleted in the AWS console.
      Note: EC2 instance will be deleted or unmanaged only after it is deleted in the AWS console.
    • Regex to filter EC2 instances name: Specify the input for regex to filter EC2 instances and add the instances that match the regex alone. Regex may contain alphanumeric characters, hyphen(-), underscore(_) and period(.). To ignore characters, specify .* at the beginning/end/middle one or more times.
    • Stop discovery: Enable this option to prevent you from discovering resources under a particular service. Use Services to stop discovery option to choose the services to stop discovering.
  11. Specify the Polling interval in minutes.
  12. If you are adding a new monitor from Admin server, select a Managed server.
  13. Choose the Monitor Group with which you want to associate the Amazon monitor to, from the combo box (optional). You can choose multiple groups to associate your monitor.
  14. Click Add Monitor(s). This discovers the Amazon server from the network and starts monitoring it.

Note: Make sure you have a working internet connecion to add or fetch data from the cloud services. If your environment requires the use of a proxy server to access external Websites, you can configure the proxy settings under the Settings tab.

Note: Parameters marked with a * have been added from Applications Manager version 15180 onwards.

Amazon Data Collection Settings

You can configure data collection settings for your Amazon monitor by going to the Settings tab, selecting Performance Polling under Discovery and then the Amazon tab. Here are the settings that you can customize for AWS performance monitoring in Applications Manager:

  • You can set the Unique Identifier for EC2 instances to be either the instance ID or a Instance name/Tag name.

    Caution: Once set, changing the Unique Identifier will result in the existing monitors being deleted. In AWS Console, if a new instance is launched with same name as of an already terminated instance, it will be merged with the existing monitor in Applications Manager. While merging, existing unmanaged EC2 instances will be automatically changed to "Manage" status, from Applications Manager Version 16360.

  • Enabling the Collect S3 Storage Statistics via CloudWatch API option allows you to fetch S3 Storage metrics using CloudWatch APIs.

Note: The option Tag name will be retained as it is for Applications Manager Versions till v16350. Renaming as Instance name will be applicable only from Applications Manager Version 16360.

For Applications Manager versions till 15170:

Option Description
Alert for Terminated EC2 Instance Option to receive notification when an EC2 instance is stopped.
Remove terminated EC2 Instances & Lambda Functions Option to completely remove terminated EC2 Instances and Lambda functions from the Applications Manager console in the consecutiv next poll.
Remove deleted DynamoDB tables Option to completely remove deleted DynamoDB tables from the Applications Manager console in the next poll.
Remove deleted Elastic Load Balancers Option to completely remove terminated Elastic Beanstalk Environments from the Applications Manager console in the next poll if the environment is terminated.
Remove inactive ECS clusters Option to completely remove terminated/deleted ECS clusters from the Applications Manager console in the next poll if the cluster is terminated/deleted.

For Applications Manager versions 15180 onwards:

Option Description
Enable Auto Scaling alerts for EC2 instances Option to receive alerts whenever an EC2 Instance is created/deleted.
Set health severity as Option to set Amazon monitor's health severity as Critical or Warning whenever a new EC2 Instance is created or deleted.
Raise autoscaling alerts when EC2 instances are Option to configure when an alert has to be recieved (When an EC2 Instance is created/deleted/both)

Monitored Parameters

Amazon accounts are monitored based on the parameters or the attributes listed below. These attributes provide detailed information about the functioning of the Amazon account. You can also configure thresholds to the numerical attributes based on these details and get notified when the thresholds are violated.

Go to the Monitors Category View by clicking the Monitors tab. Click on the Amazon monitor under the Cloud Apps Table. Displayed is the Amazon bulk configuration view distributed into three tabs:

  • Availability tab displays the Availability history for the past 24 hours or 30 days.
  • Performance tab displays the Health Status and events for the past 24 hours or 30 days along with key performance indicators of the Amazon account such as Total EC2 Instances running and Total RDS Instances along with heat charts for these attributes.
  • List view tab enables you to perform bulk admin configurations and displays all the Amazon instances along with an overall idea of their availability and health status.

AWS monitoring software like Applications Manager provide complete visibility into your Amazon cloud environments altogether in a single window based on the following tabs:

Note:
    • Each Amazon monitor instance is considered a basic monitor.
    • Each service instances (except S3, SNS and Billing information) running in your Amazon monitor is considered a basic monitor.

For more information, refer here.

Overview

Parameter Description
COMPUTE RESOURCES
Total EC2 Instances Running The number of EC2 instances running in the account.
Number of Application Load Balancers The total number of Application Load Balancers present in the account.
Number of Network Load Balancers The total number of Network Load Balancers present in the account.
Number of Lambda Functions The total number of Lambda functions present in the account.
Number of Elastic Beanstalk Environments The total number of Elastic Beanstalk environments present in the account.
STORAGE RESOURCES
Total EBS Volumes in use The number of EBS storage volumes currently in use.
Total S3 Buckets The total number of S3 buckets present in the account.
DATABASE RESOURCES
Total RDS Instances The total number of RDS instances present in the account.
Total DynamoDB tables The total number of DynamoDB tables present in the account.
APPLICATION INTEGRATION
Number of SQS Queues The total number of SQS Queues present in the account.
Number of SNS Topics The total number of SNS topics present in the account.
CONTAINERS
Number of ECS Clusters The total number of ECS clusters present in the account.
Number of EKS Clusters The total number of EKS clusters present in the account.

Compute

Following are the list of services that are shown in the 'Compute' tab:

EC2 Instances

This table provides details about the EC2 instances present in the Amazon account. Know more about EC2 Monitoring

Parameter Description
EC2 Instances
Instance ID The unique identifier of the EC2 instance
Region Name The region where this EC2 instance is running. To know the list of regions supported, refer here.
State The current state of the instance. The values include running, stopped, shutdown and terminated
Platform The OS on which the instance is running
Monitoring Indicates whether monitoring is enabled for the instance.
Public DNS Name The DNS name associated with the instance.
System Log Displays the system log of the instance.

You can perform the following admin actions on the EC2 instances:

  • Delete: Delete the EC2 instance from Applications Manager.
  • Assign Platform: Assign platforms such as Windows, Mac OS, etc. to instances.
  • Start Instances: Option to start the EC2 instance from Applications Manager.
  • Stop Instances: Option to stop EC2 instances from Applications Manager.
  • Reboot Instances: Option to reboot the EC2 instances from within Applications Manager.

Lambda

This table provides details about the Lambda functions present in the Amazon account. Know more about Lambda functions

Parameter Description
LAMBDA
Function Name of the Lambda function.
Region Name The region where this Lambda function is running. To know the list of regions supported, refer here.
Version Version of the Lambda function.
Runtime The runtime environment for the Lambda function.
Deployed Code Size (MB) Size of deployment package of the function (in MB).
Timeout The amount of time that Lambda allows a function to run before stopping it (in seconds).
Last Modified Time Date and time at which the function was last modified.

Elastic Beanstalk Environments

This table provides details about the Elastic Beanstalk environments present in the Amazon account. Know more about Elastic Beanstalk Environments

Parameter Description
Elastic Beanstalk Environments
Environment Name Name of the Elastic Beanstalk environment.
Region Code The region where the Elastic Beanstalk environment is running. To know the list of regions supported, refer here.
Tier Name Name of the tier. Indicates whether it is a WebServer or a Worker environment.
Environment Status Status of the environment. (Ready, Launching, Updating, Terminating, or Terminated)
Environment Health Status Shows the health status of the environment. (OK, Info, Unknown, No data, Warning, Degraded, or Severe)

Containers

ECS Clusters

This tab provides details about the ECS clusters present in the Amazon account. Know more about ECS clusters

Parameter Description
ECS Clusters
ClusterName Name of the ECS cluster.
Status The status of the Cluster. (ACTIVE, PROVISIONING, DEPROVISIONING, FAILED, INACTIVE)
Registered Container Instances The number of containers instances registered to the cluster.
Active Services The number of active services running in the cluster.
Running Tasks The number of tasks that are in RUNNING state.

EKS Clusters

This tab provides details about the EKS clusters present in the Amazon account. Know more about EKS clusters

Parameter Description
EKS Clusters
ClusterName Name of the EKS cluster.
Status The status of the cluster. (CREATING, ACTIVE, DELETING, FAILED, UPDATING)

Database

Following are the list of services that are shown in the 'Database' tab:

RDS Instances

This tab provides details about the RDS Instances present in the Amazon account. Know more about RDS Instances

Parameter Description
Instance ID The unique identifier of the instance.
Region Name The region in which the RDS instance is running. To know the list of regions supported, refer here.
State The current state of this instance. The possible values for this field are available, creating, failed, rebooting, etc.
DB Engine Name The name of the database engine associated with this instance
Allocated Storage The storage space allocated to this instance in Giga Bytes
Aurora Cluster Name The name of the Aurora DB cluster.
Cluster Instance Role The role that is associated with the Aurora DB instance.
Cluster Group status The current state of this DB cluster group.

DynamoDB

In this table, you can gather data for performance metrics of you Amazon DynamoDB like latency, request throughput and throttling errors. Optimize resource usage and improve application performance of your Amazon Dynamodb database. Know more about DynamoDB monitoring

Parameter Description
Tables
Name Name of the DynamoDB table.
Region Name of the AWS region where the current DynamoDB table resides. To know the list of regions supported, refer here.
Total Provisioned Read CU Total number of provisioned read capacity units for a table.
Total Provisioned Write CU Total number of provisioned write capacity units for a table.
Storage Size (GB) Total size of the specified table (in GB).
Table Status Current status of the table.
Creation Date The date and time at which the table was created.
DynamoDB Account Limits
Region The ID of AWS region where the current DynamoDB account resides.
Region Name Name of the region at which the DynamoDB account is located.
Read Limit Region wise maximum read capacity unit limit.
Provisioned Read Number of provisioned read capacity units.
Write Limit Region wise maximum write capacity unit limit.
Provisioned Write Number of provisioned write capacity units.
Table Read Limit Table-wise maximum read capacity units that can be provisioned.
Table Write Limit Table-wise maximum write capacity units that can be provisioned.

Note: DynamoDB Account Limits can be monitored only when DynamoDB is chosen for discovery. Also data collection can be performed only for the selected regions.

Amazon Elastic Load Balancing

Applications Manager's AWS ELB monitoring gives you the ability to monitor your Load Balancers, analyze traffic patterns and troubleshoot issues with your load balancers and targets for optimized performance. Know more about AWS ELB monitoring

Application Load Balancers

Parameter Description
Name The name of the Application Load Balancer
Region The AWS region in which the load balancer is configured. To know the list of regions supported, refer here.
State The state of the load balancer - active | provisioning | active_impaired | failed.
Availability zones The Availability Zones for the load balancer.
VPC ID The ID of the VPC for the load balancer.
Creation time The date and time the load balancer was created.

Network Load Balancers

Parameter Description
Name The name of the Network Load Balancer
Region The AWS region in which the load balancer is configured. To know the list of regions supported, refer here.
State The state of the load balancer - active | provisioning | active_impaired | failed.
Availability zones The Availability Zones for the load balancer.
VPC ID The ID of the VPC for the load balancer.
Creation time The date and time the load balancer was created.

SNS Services

In this tab you can monitor region and topic-wise metrics like notification status (delivery and failure), SMS deliveries, messages throughput - sent and failed counts, subscription count and more. Know more about SNS Service monitoring

Parameter Description
Total Messages Published The total number of messages published throughout all the AWS supported regions.
Total Notifications Delivered The total number of messages delivered throughout all the AWS supported regions.
Total Notifications Failed The total number of messages failed throughout all the AWS supported regions.
Total Published Size (KB) The total size of messages published throughout all the AWS supported regions.
Notifications Delivered (%) The percentage of total delivered notifications.
Notifications Failed (%) The percentage of total failed notifications.
SMS Month-To-Date charges The total charges accrued since the start of the current calendar month for sending SMS messages in AWS supported regions. Unit - USD

Network

CloudFront Distributions

This tab provides details about the CloudFront Distributions present in the Amazon account. Know more about CloudFront Distributions.

Parameter Description
CloudFront Distributions
Distribution ID The distribution ID of the cloudfront.
Status The status of the cloudfront.
State The current state of the cloudfront. Possible value: Enabled/ Disabled.

Storage

Following are the list of services that are shown in the 'Storage' tab:

Elastic File Systems

This table provides details about the Elastic File Systems present in the Amazon account. Know more about EFS Monitoring

Parameter Description
Elastic File Systems
File System ID The ID of the file system.
Name The name of the file system.
File System State The current state of the file system.
Number of Mount Targets Total number of mount targets present in the file system.

S3 Buckets

This tab provides details about the S3 buckets present in the Amazon account.

Parameter Description
S3 Buckets Storage Statistics
Bucket Name The unique name of the S3 bucket.
Bucket Size The size of the S3 bucket in megabytes.
Bucket Location The geographical region where Amazon has stored this bucket. To know the list of regions supported, refer here.
Creation Time The time when the bucket was created.
Virtual Folders The number of folders present in this S3 bucket.
Number of Objects The number of objects stored in this S3 bucket.
S3 Buckets Request Statistics
Bucket Name The unique name of the S3 bucket.
Total Requests The total number of HTTP requests made to the bucket.
Requests/min Number of requests made per minute to the bucket.
Get Requests The number of HTTP GET requests made for objects in the bucket.
Put Requests The number of HTTP PUT requests made for objects in the bucket.
First Byte Latency The per-request time (in milliseconds) from the complete request being received by the bucket to when the response starts to be returned.
Total Request Latency The elapsed per-request time (in milliseconds) from the first byte received to the last byte sent to the bucket. This includes the time taken to receive the request body and send the response body, which is not included in First Byte Latency.
Bytes Downloaded The number of bytes downloaded (in MB) for requests made to the bucket, where the response includes a body.
Bytes Uploaded The number of bytes uploaded (in MB) that contain a request body, made to the bucket.
Client Errors The number of HTTP 4xx client error status code requests made to the bucket.
Server Errors The number of HTTP 5xx server error status code requests made to the bucket.

You can perform the following admin actions on the S3 buckets from within Applications Manager.

  • Disable: Disables the monitoring of S3 bucket.
  • Enable: Enables the monitoring of S3 bucket.
  • Delete: Deletes the S3 bucket from Applications Manager console. Please note that the required S3 bucket instance should first be deleted from your AWS account before performing this action. Otherwise, Applications Manager will rediscover that S3 bucket and start monitoring again.

You can also view comparison reports based on attributes such as bucket size, virtual folders and number of object

Note:
  • 'Virtual Folder' metric is not supported upon using CloudWatch API to fetch Storage metrics.
  • To monitor S3 Bucket Request metrics, the same should be enabled in AWS console. We recommend using the name EntireBucket for a filter that applies to all objects. For more information, refer here.

SQS Services

In this tab, you can monitor all the available queues that are running within the SQS service. Know more about SQS service monitoring

Parameter Description
Queue Name Name of the queue.
Queue Type Type of the queue. (Standard/FIF0)
Region Shows the region where the queue was created. To know the list of regions supported, refer here.
Arrival Rate The rate at which the messages are arrived in the queue per minute.
Completion Rate The rate at which the messages are processed and deleted in the queue per minute.
Created Time Displays the time when the queue was first created.

Billing

Parameters Description
Total Expenditure And Forecast
Month-to-Date Expenditure The total expenditure in the current month for an AWS account.
Current Month Forecast The cost forecast for the current month.
Services And Tax
Month-to-Date Services Cost The total cost in USD incurred by all the services [Excluding tax].
Month-to-Date Tax The total cost in USD by tax.
Month-to-Date Top Services by Spend
Month-to-Date Costs (USD) The Month-to-Date Spend by Service graph shows the top services that you use most and the proportion of your costs(USD) that that service contributed to [excluding forecasting and *tax*].
Month-to-Date Spend by Service
AWS Service The AWS services that you use.
Costs (USD) The cost in USD incurred for the month to date by the services.

Note: The Billing data will be available only for AWS Global accounts. The polling interval for the AWS Billing Statistics can be changed by going to the Settings tab, clicking Performance Polling under Discovery and Data Collection, and navigateing to the Optimize Data Collection tab. Choose Amazon for the Monitor Type & AWS Billing Dashboards for the Metric Name and change the Default Polling Status.(Default and preferred time interval is 24 hours).


Supported Regions

You can collect metrics, visualize resource usage and alert on operational behavior for all supported services across your AWS cloud environment for the following regions:

AWS Global Regions

Region Name Region Code
Following regions are supported from Applications Manager v13600:
Asia Pacific (Mumbai) ap-south-1
Asia Pacific (Seoul) ap-northeast-2
Asia Pacific (Singapore) ap-southeast-1
Asia Pacific (Sydney) ap-southeast-2
Asia Pacific (Tokyo) ap-northeast-1
Canada (Central) ca-central-1
EU (Frankfurt) eu-central-1
EU (Ireland) eu-west-1
EU (London) eu-west-2
EU (Paris) eu-west-3
EU (Stockholm) eu-north-1
South America (Sao Paulo) sa-east-1
US East (N.Virginia) us-east-1
US East (Ohio) us-east-2
US West (N.California) us-west-1
US West (Oregon) us-west-2
Following regions are supported from Applications Manager v14780:
Africa (Cape Town) af-south-1
Asia Pacific (Hong Kong) ap-east-1
Asia Pacific (Osaka-Local) ap-northeast-3
Europe (Milan) eu-south-1
Middle East (Bahrain) me-south-1
Following region is supported from Applications Manager v16010:
Asia Pacific (Jakarta) ap-southeast-3
Following region is supported from Applications Manager v16340:
Asia Pacific (Hyderabad) ap-south-2
Europe (Spain) eu-south-2
Europe (Zurich) eu-central-2
Middle East (UAE) me-central-1
Following region is supported from Applications Manager v16700:
Asia Pacific (Melbourne) ap-southeast-4
Israel (Tel Aviv) il-central-1

In case you want to monitor AWS in all the above-mentioned regions, choose the All Regions option under Global Regions in the Add/Edit monitor page. By default, this option will be selected.

Note: Do not specify other regions when All Regions option is selected. If other regions are chosen along with All regions, only the chosen regions will be considered while adding/updating the monitor.

AWS China Regions

The China regions are isolated from all other AWS (Global) regions and operated by AWS's Chinese partners separately from all other AWS regions. However, the cloud services offered in both AWS China regions are the same as those available in other AWS (Global) regions. From Version 13620, we support monitoring of Amazon Web Services in AWS China regions. Here is a list of regions available in the AWS China account:

  1. China (Beijing)
  2. China (Ningxia)

The default Account Type when adding a new Amazon monitor is AWS Global. Choose AWS China as the Account Type to monitor services in the AWS China regions.

AWS Gov Cloud Regions

AWS Gov Cloud (US) gives government customers and their partners the flexibility to architect secure cloud solutions. This is designed to host sensitive data, regulated workloads, and address the most stringent US government security and compliance requirements. AWS Gov Cloud regions are operated by employees who are US citizens on US soil, and are accessible only to US entities and root account holders who pass a screening process.

From version 15390 onwards, we support monitoring of Amazon Web Services in AWS Gov Cloud regions. Following are the list of regions available in the AWS Gov Cloud account:

  1. US East (US-Gov-East)
  2. US West (US-Gov-West)

The default Account Type when adding a new Amazon monitor is AWS Global. Choose AWS Gov Cloud as the Account Type to monitor services in the AWS Gov Cloud (US) regions.


Troubleshooting

1. Some instances of EC2, RDS, S3, SNS are not being discovered for data collection

There are two ways of authenticating information in the requests made to AWS to collect data:
    • Signature version 2 (S2) and
    • Signature version 4 (S4)

In the following regions, AWS supports only signature version 4 signing process: [US East (Ohio), Canada (Central), Asia Pacific (Mumbai), Asia Pacific (Seoul), EU (Frankfurt), EU (London)]. If you are using AWS services from the regions which supports only S4, Applications Manager will not discover them as they require S4 signing process.

Solution: Upgrade to Applications Manager Version 13600 (or newer). We have upgraded to the latest aws jar and support both signing processes in AWS. From Version 13600, we also support all 15 regions available currently. (Know more)

Note: Make sure you have a working internet connecion to add or fetch data from the cloud services. If your environment requires the use of a proxy server to access external Websites, you can configure the proxy settings under the Settings tab.

2. When child monitors are not discovered or when no data in AWS and its child monitors.

  • Check the list of APIs mentioned in this page and make sure the IAM user has all the required API permissions.
  • Check the region in which the user has created the resources in AWS and make sure the same region is supported/discovered in Applications Manager. Know more

3. No data in EC2 Instance in OS level metrics.

Most of the times, the issues occurs due to the wrong configuration in agent configuration file, using which the Cloudwatch agent fetches the data from the EC2 instances.

Applications Manager can fetch the data from CloudWatch based on the following:

  • Namespace - AWS allows the user to have any namespace, but only 'CWAgent' as namespace is supported in Applications Manager
  • Dimensions - Only 'InstanceId' as dimension is supported in Applications Manager. If any additional dimensions are configured in agent configuration file, then it is not supported. It looks for only the 'instanceid' dimension.

In every data collection of EC2 instance monitor, first we look for 'Memory' metrics in Windows/Linux instances, and then we check for 'Disk' metrics. Make sure users are configured similar to the metrics mentioned in this page sample configuration files.