Amazon Web Services Monitoring



Amazon Web Services (AWS) - Overview

Amazon Web Services (AWS) is a comprehensive cloud computing platform from Amazon. AWS offers an array of cloud computing services to build on from data centers spread across availability zones in regions across the world. However, you would still need a proper AWS monitoring software to be able to monitor and understand what is going on when you deploy and consume such a wide-ranging set of services.

From your infrastructure and applications to your end-user experience, Applications Manager's AWS performance monitoring performs robust monitoring of several core Amazon Web Services under your Amazon account. With all-in-one AWS cloud monitoring tools like Applications Manager in place, users can obtain a single pane view over various AWS services like:


Creating a new Amazon monitor

Prerequisites for setting up Amazon performance monitoring: Click here.

Using the REST API to add a new Amazon monitor: Click here

To create a new Amazon monitor, follow the steps given below:

  1. Click on New Monitor link.
  2. Select Amazon under Cloud Apps category.
  3. Specify the Display Name of the Amazon server
  4. Select the Amazon Services you wish to discover and monitor from the drop-down menu.
  5. Enter the Access Key Id of the AWS for accessing the AWS through the API. The access key has 20 alpha-numeric characters.
  6. Enter the Secret Access Key of the AWS. The secret key should be 40 alpha-numeric characters long.
  7. Choose the Account Type. The default Account Type when adding a new Amazon monitor is AWS Global. Choose AWS China as the Account Type to monitor services in the AWS China regions.
  8. Specify the Polling Interval in minutes.
  9. If you are adding a new monitor from an Admin Server, select a Managed Server.
  10. Choose the Monitor Group with which you want to associate the Amazon monitor to, from the combo box (optional). You can choose multiple groups to associate your monitor.
  11. Click Add Monitor(s). This discovers the Amazon server from the network and starts monitoring it.
Note: Under the Admin → Performance Polling → Amazon → "On Modifying Amazon Services" option, you can choose to either 'Stop discovering' or 'delete' the existing added child monitors (Amazon EC2, RDS, DynamoDB etc) when a user edits and removes existing services. By default, this option is set to 'Stop discovering'

Amazon Data Collection Settings

You can configure data collection settings for your Amazon monitor by going to the Admin tab, selecting Performance Polling under Discovery and then the Amazon tab. Here are the settings that you can customize:

  • You can set the Unique Identifier for EC2 instances to be either the instance ID or a tag name.

    Caution: Once set, changing the Unique Identifier will result in the existing monitors being deleted. If a new instance with Tag name of an already terminated instance is launched, it will be merged with the former instance.

  • Enable the Alert for Terminated EC2 Instance option to receive notification when an instance is stopped.
  • Enabling the Remove terminated EC2 Instances & Lambda Functions option completely removes terminated EC2 Instances and Lambda functions from the Applications Manager console in the next poll. By default, this option is disabled. Enable it if you find a terminated EC2 instance/Lambda function in the Applications Manager console, after it was removed from the AWS console (If a user terminates an EC2 instance/Lambda function it will be removed from the AWS console after 60 minutes.)
  • Enabling the Remove deleted DynamoDB tables option completely removes deleted DynamoDB tables from the Applications Manager console in the next poll. By default, this option is disabled. Enable it if you find a deleted table in the Applications Manager console, after it was removed from the AWS console.
  • Enabling the Remove deleted Elastic Load Balancers option completely removes deleted Load Balancers from the Applications Manager console in the next poll. By default, this option is disabled. Enable it if you find a deleted load-balancer in the Applications Manager console, after it was removed from the AWS console. (If a user terminates an instance it will be removed from the AWS console after 60 minutes.
  • On modifying 'Amazon Services': Using this option, you can choose the action to be performed when a configured Amazon Service is removed. Go to Admin → Performance Polling → Amazon → On modifying 'Amazon Services'. Choose Delete if you want to delete the existing child monitors for the removed AWS service or Stop discovering to stop further discovering of child monitors for the removed AWS service.
Note: Make sure you have a working internet connecion to add or fetch data from the cloud services. If your environment requires the use of a proxy server to access external Websites, you can configure the proxy settings under the 'Admin' tab.

Monitored Parameters

Amazon accounts are monitored based on the parameters or the attributes listed below. These attributes provide detailed information about the functioning of the Amazon account. You can also configure thresholds to the numerical attributes based on these details and get notified when the thresholds are violated.

Go to the Monitors Category View by clicking the Monitors tab. Click on the Amazon monitor under the Cloud Apps Table. Displayed is the Amazon bulk configuration view distributed into three tabs:

  • Availability tab displays the Availability history for the past 24 hours or 30 days.
  • Performance tab displays the Health Status and events for the past 24 hours or 30 days along with key performance indicators of the Amazon account such as Total EC2 Instances running and Total RDS Instances along with heat charts for these attributes.
  • List view tab enables you to perform bulk admin configurations and displays all the Amazon instances along with an overall idea of their availability and health status.

AWS monitoring software like Applications Manager provide complete visibility into your Amazon cloud environments altogether in a single window based on the following tabs:

Overview

Parameter Description
COMPUTE RESOURCES
Total EC2 Instances Running The number of EC2 instances running in the account.
Number of Application Load Balancers The total number of Application Load Balancers present in the account.
Number of Network Load Balancers The total number of Network Load Balancers present in the account.
Number of SQS Queues The total number of SQS Queues present in the account.
Number of Lambda Functions The total number of Lambda functions present in the account.
STORAGE RESOURCES
Total EBS Volumes in use The number of EBS storage volumes currently in use.
Total S3 Buckets The total number of S3 buckets present in the account.
DATABASE RESOURCES
Total RDS Instances The total number of RDS instances present in the account.
Total DynamoDB tables The total number of DynamoDB tables present in the account.

Compute

EC2 Instances

This table provides details about the EC2 instances present in the Amazon account. Know more about EC2 Monitoring

Parameter Description
EC2 Instances
Instance ID The unique identifier of the EC2 instance
Region Name The region where this EC2 instance is running. To know the list of regions supported, refer here.
State The current state of the instance. The values include running, stopped, shutdown and terminated
Platform The OS on which the instance is running
Monitoring Indicates whether monitoring is enabled for the instance.
Public DNS Name The DNS name associated with the instance.
System Log Displays the system log of the instance.

You can perform the following admin actions on the EC2 instances:

  • Delete: Delete the EC2 instance from Applications Manager.
  • Assign Platform: Assign platforms such as Windows, Mac OS, etc. to instances.
  • Start Instances: Option to start the EC2 instance from Applications Manager.
  • Stop Instances: Option to stop EC2 instances from Applications Manager.
  • Reboot Instances: Option to reboot the EC2 instances from within Applications Manager.

Lambda

This table provides details about the Lambda functions present in the Amazon account. Know more about Lambda functions

Parameter Description
LAMBDA
Function Name of the Lambda function.
Region Name The region where this Lambda function is running. To know the list of regions supported, refer here.
Version Version of the Lambda function.
Runtime The runtime environment for the Lambda function.
Deployed Code Size (MB) Size of deployment package of the function (in MB).
Timeout The amount of time that Lambda allows a function to run before stopping it (in seconds).
Last Modified Time Date and time at which the function was last modified.

Amazon Elastic Load Balancing

Applications Manager gives you the ability to monitor your Load Balancers, analyze traffic patterns and troubleshoot issues with your load balancers and targets for optimized performance. Know more about AWS ELB monitoring

Application Load Balancers

Parameter Description
Name The name of the Application Load Balancer
Region The AWS region in which the load balancer is configured. To know the list of regions supported, refer here.
State The state of the load balancer - active | provisioning | active_impaired | failed.
Availability zones The Availability Zones for the load balancer.
VPC ID The ID of the VPC for the load balancer.
Creation time The date and time the load balancer was created.

Network Load Balancers

Parameter Description
Name The name of the Network Load Balancer
Region The AWS region in which the load balancer is configured. To know the list of regions supported, refer here.
State The state of the load balancer - active | provisioning | active_impaired | failed.
Availability zones The Availability Zones for the load balancer.
VPC ID The ID of the VPC for the load balancer.
Creation time The date and time the load balancer was created.

RDS Instances

This tab provides details about the RDS Instances present in the Amazon account. Know more about RDS Instances

Parameter Description
Instance ID The unique identifier of the instance.
Region Name The region in which the RDS instance is running. To know the list of regions supported, refer here.
State The current state of this instance. The possible values for this field are available, creating, failed, rebooting, etc.
DB Engine Name The name of the database engine associated with this instance
Allocated Storage The storage space allocated to this instance in Giga Bytes
Aurora Cluster Name The name of the Aurora DB cluster.
Cluster Instance Role The role that is associated with the Aurora DB instance.
Cluster Group status The current state of this DB cluster group.

S3 Buckets

This tab provides details about the S3 buckets present in the Amazon account.

Parameter Description
Bucket Name The unique name of the S3 bucket.
Bucket Size The size of the S3 bucket in megabytes.
Bucket Location The geographical region where Amazon has stored this bucket. To know the list of regions supported, refer here.
Creation Time The time when the bucket was created.
Virtual Folders The number of folders present in this S3 bucket.
Number of Objects The number of objects stored in this S3 bucket.

You can perform the following admin actions on the S3 buckets from within Applications Manager.

  • Disable: Disable the monitoring of S3 bucket.
  • Enable: Enable the monitoring of S3 bucket.
  • Delete: Delete the S3 bucket from Applications Manager.Use this option when S3 bucket is deleted from your Amazon account.

You can also view comparison reports based on attributes such as bucket size, virtual folders and number of object

SNS Services

In this tab you can monitor region and topic-wise metrics like notification status (delivery and failure), SMS deliveries, messages throughput - sent and failed counts, subscription count and more. Know more about SNS Service monitoring

Parameter Description
Total Messages Published The total number of messages published throughout all the AWS supported regions.
Total Notifications Delivered The total number of messages delivered throughout all the AWS supported regions.
Total Notifications Failed The total number of messages failed throughout all the AWS supported regions.
Total Published Size (KB) The total size of messages published throughout all the AWS supported regions.
Notifications Delivered (%) The percentage of total delivered notifications.
Notifications Failed (%) The percentage of total failed notifications.
SMS Month-To-Date charges The total charges accrued since the start of the current calendar month for sending SMS messages in AWS supported regions. Unit - USD

SQS Services

In this tab, you can monitor all the available queues that are running within the SQS service. Know more about SQS service monitoring

Parameter Description
Queue Name Name of the queue.
Queue Type Type of the queue. (Standard/FIF0)
Region Shows the region where the queue was created. To know the list of regions supported, refer here.
Arrival Rate The rate at which the messages are arrived in the queue per minute.
Completion Rate The rate at which the messages are processed and deleted in the queue per minute.
Created Time Displays the time when the queue was first created.

DynamoDB

In this tab, you can gather data for performance metrics of you Amazon DynamoDB like latency, request throughput and throttling errors. Optimize resource usage and improve application performance of your Amazon Dynamodb database. Know more about DynamoDB monitoring

Parameter Description
Tables
Name Name of the DynamoDB table.
Region Name of the AWS region where the current DynamoDB table resides. To know the list of regions supported, refer here.
Total Provisioned Read CU Total number of provisioned read capacity units for a table.
Total Provisioned Write CU Total number of provisioned write capacity units for a table.
Storage Size (GB) Total size of the specified table (in GB).
Table Status Current status of the table.
Creation Date The date and time at which the table was created.
DynamoDB Account Limits
Region The ID of AWS region where the current DynamoDB account resides.
Region Name Name of the region at which the DynamoDB account is located.
Read Limit Region wise maximum read capacity unit limit.
Provisioned Read Number of provisioned read capacity units.
Write Limit Region wise maximum write capacity unit limit.
Provisioned Write Number of provisioned write capacity units.
Table Read Limit Table-wise maximum read capacity units that can be provisioned.
Table Write Limit Table-wise maximum write capacity units that can be provisioned.

Billing

Parameters Description
Total Expenditure And Forecast
Month-to-Date Expenditure The total expenditure in the current month for an AWS account.
Current Month Forecast The cost forecast for the current month.
Services And Tax
Month-to-Date Services Cost The total cost in USD incurred by all the services [Excluding tax].
Month-to-Date Tax The total cost in USD by tax.
Month-to-Date Top Services by Spend
Month-to-Date Costs (USD) The Month-to-Date Spend by Service graph shows the top services that you use most and the proportion of your costs(USD) that that service contributed to [excluding forecasting and *tax*].
Month-to-Date Spend by Service
AWS Service The AWS services that you use.
Costs (USD) The cost in USD incurred for the month to date by the services.

Note: The Billing data will be available only for AWS Global accounts. The polling interval for the AWS Billing Statistics can be changed by going to the Admin tab, clicking Performance Polling under Discovery and Data Collection, and navigateing to the Optimize Data Collection tab. Choose Amazon for the Monitor Type & AWS Billing Dashboards for the Metric Name and change the Default Polling Status.(Default and preferred time interval is 24 hours).


Supported Regions

AWS Global Regions

From Version 13600 and above, we support these global AWS regions:

  1. US East (N. Virginia)
  2. US East (Ohio)
  3. US West (N. California)
  4. US West (Oregon)
  5. Canada (Central)
  6. South America (Sao Paulo)
  7. EU (Frankfurt)
  8. EU (Ireland)
  9. EU (London)
  10. EU (Paris)
  11. EU (Stockholm)
  12. Asia Pacific (Tokyo)
  13. Asia Pacific (Seoul)
  14. Asia Pacific (Mumbai)
  15. Asia Pacific (Singapore)
  16. Asia Pacific (Sydney)
  17. Asia Pacific (Hong Kong)
  18. Asia Pacific (Osaka-Local)
  19. Middle East (Bahrain)
  20. Africa (Cape Town)
  21. Europe (Milan)

AWS China Regions

The China regions are isolated from all other AWS (Global) regions and operated by AWS's Chinese partners separately from all other AWS regions. While the cloud services offered in both AWS China regions are the same as those available in other AWS (Global) regions. From Version 13620, we support monitoring of Amazon Web Services in AWS China regions. Here is a list of regions available in the AWS China account:

  1. China (Beijing)
  2. China (Ningxia)

The default Account Type when adding a new Amazon monitor is AWS Global. Choose AWS China as the Account Type to monitor services in the AWS China regions.


Troubleshooting

1. Some instances of EC2, RDS, S3, SNS are not being discovered for data collection

There are two ways of authenticating information in the requests made to AWS to collect data:
    • Signature version 2 (S2) and
    • Signature version 4 (S4)

In the following regions, AWS supports only signature version 4 signing process: [US East (Ohio), Canada (Central), Asia Pacific (Mumbai), Asia Pacific (Seoul), EU (Frankfurt), EU (London)]. If you are using AWS services from the regions which supports only S4, Applications Manager will not discover them as they require S4 signing process.

Solution: Upgrade to Applications Manager Version 13600 (or newer). We have upgraded to the latest aws jar and support both signing processes in AWS. From Version 13600, we also support all 15 regions available currently. (Know more)

Note: Make sure you have a working internet connecion to add or fetch data from the cloud services. If your environment requires the use of a proxy server to access external Websites, you can configure the proxy settings under the 'Admin' tab.

2. When child monitors are not discovered or when no data in AWS and it's child monitors.

  • Check the list of APIs mentioned in this page and make sure the IAM user has all the required API permissions.
  • Check the region in which the user has created the resources in AWS and make sure the same region is supported/discovered in Applications Manager. Know more

3. No data in EC2 Instance in OS level metrics.

Most of the times, the issues occurs due to the wrong configuration in agent configuration file, using which the Cloudwatch agent fetches the data from the EC2 instances.

Applications Manager can fetch the data from CloudWatch based on the following:

  • Namespace - AWS allows the user to have any namespace, but only 'CWAgent' as namespace is supported in Applications Manager
  • Dimensions - Only 'InstanceId' as dimension is supported in Applications Manager. If any additional dimensions are configured in agent configuration file, then it is not supported. It looks for only the 'instanceid' dimension.

In every data collection of EC2 instance monitor, first we look for 'Memory' metrics in Windows/Linux instances, and then we check for 'Disk' metrics. Make sure users are configured similar to the metrics mentioned in this page sample configuration files.