Elasticsearch Monitoring


Elasticsearch - An Overview

Elasticsearch is a highly scalable, distributed, open source RESTful search and analytics engine. It is multitenant-capable with an HTTP web interface and schema-free JSON documents. Based on Apache Lucene, Elasticsearch is one of the most popular enterprise search engines today and is capable of solving a growing number of use cases like log analytics, real-time application monitoring, and click stream analytics.

Monitoring Elasticsearch - What we do

Let’s take a look at what you need to see to monitor Elasticsearch, the performance metrics to gather and how you can ensure that your search server is up and operating as expected with Applications Manager's Elasticsearch monitoring:

  • Resource Utilization Details - Applications Manager automatically discovers Elasticsearch servers, monitors memory and CPU and notifies you of changes in resource consumption of thread pool queues.
  • Real-Time Data - You get up-to-the-second insight into cluster runtime metrics, individual cluster nodes, real-time threads and configurations.
  • Cluster and Node Monitoring - Stay on top of your cluster and node health in real-time with fine-grained statistics of performance from Disk I/O Java to Memory usage metrics.
  • Search and Indexing Performance - Gain complete control of your indexes and mappings. Monitor query latency, file system cache usage and request rates and take action if it surpasses a threshold.
  • Fix Performance Problems Faster - Get instant notifications when there are performance issues. Become aware of performance bottlenecks and take quick remedial actions before your end users experience issues.

Creating a new Elasticsearch monitor

Using the REST API to add a new Elasticsearch monitor: Click here

To create an Elasticsearch Monitor, follow the steps given below:

  • Click on New Monitor link. Choose ElasticsearchCluster.
  • Specify the Display Name of the Elasticsearch monitor.
  • Enter the HostName or IP Address of the host where Elasticsearch Cluster runs.
  • Enter the Port of the Elasticsearch Cluster. By default, it will be 9200.
  • Enter the polling interval time in minutes.
  • Click Test Credentials button, if you want to test the access to Elasticsearch server.
  • Choose the Monitor Group from the combo box with which you want to associate Elasticsearch Monitor (optional). You can choose multiple groups to associate your monitor.
  • Click Add Monitor(s). This discovers Elasticsearch from the network and starts monitoring.

Demo

Monitored Parameters

Go to the Monitors Category View by clicking the Monitors tab. Click on the Elasticsearch and ElasticsearchCluster monitors under the Web Server/Services Table. Displayed is the Elasticsearch or the ElasticsearchCluster bulk configuration view distributed into three tabs:

  • Availability tab displays the Availability history for the past 24 hours or 30 days.
  • Performance tab displays the Health Status and events for the past 24 hours or 30 days.
  • List view enables you to perform bulk admin configurations.

Click on the monitor name to see all the server details listed under the following tabs:

Elasticsearch Cluster

Overview

Parameter Description
Node Details
Cluster Name The name of the cluster in which the node is located.
Node Name The name of the node
Node Type The type of the node (Client or Data or Master-Eligible or Master-Data)
Avg Query Time The first phase of search operation is Query. The time taken to process the query in all shards
Avg Fetch Time The second phase of search operation is Fetch. The time taken to retrieve the query result, only from the shards which have the requested data.
CLUSTER OVERVIEW
Cluster Status The status of the cluster depending on the replicas of the cluster.
Total Nodes The total number of nodes in the cluster.
Total Indices The total number of indices in the cluster.
Total Shards The total number of shards in the cluster.
Total Docs The total number of documents present in the cluster.
Total Pending Tasks The total number of pending tasks in the cluster.

Cluster Details

Parameter Description
NODES SPLITUP
Client Node The total number of Client Nodes in the cluster.
Data Node The total number of Data Nodes in the cluster.
Master Node The total number of Master Eligible Nodes in the cluster.
Data-Master Node The total number of Data Nodes, which also acts as Master Eligible Nodes in the cluster.
SHARDS COUNT
Active Shards The number of Active Shards present in the cluster.
Active Primary Shards The number of Primary Shards that are Active in the cluster.
Relocating Shards The number of Relocating Shards present in the cluster.
Initializing Shards The number of Initializing Shards present in the cluster.
Unassigned Shards The number of Unassigned Shards present in the cluster.
Delayed Unassigned Shards The number of Delayed Unassigned Shards present in the cluster.
Total Shards The number of Shards present in the cluster.
Active Shards Percent The percentage of Active Shards
Active Primary Shards Percent The percentage of Active Primary Shards Shards
Relocating Shards Percent The percentage of Relocating Shards
Initializing Shards Percent The percentage of Initializing Shards
Unassigned Shards Percent The percentage of Unassigned Shards
Delayed Unassigned Shards Percent The percentage of Delayed Unassigned Shards
PENDING TASKS
Insert Order The order of the task in which it is inserted into the queue
Priority The priority assigned for the particular Task
Source The Source for the Pending Task
Time in Queue The total waiting time of the Task in that queue.

Indices

PARAMETER DESCRIPTION
Indices Overview
Index Name The name of the index representing a collection of documents.
Documents Indicates the number of documents that are available in the particular index.
Indexing Latency Amount of time taken to index a document in the particular index (in millisecond).
Indexing Rate The number of documents that are indexed per second.
Query Latency Amount of time taken to process the query in the particular index (in millisecond).
Query Rate The number of queries that are processed by the index per second.
Fetch Latency Amount of time taken to run the query and retrieve the data in the particular index (in millisecond).
Fetch Rate The number of queries that are run and retrieved data by the index per second.
Current Merges Indicates the number of merges that have occurred in the particular index.
Merge Time Amount of time taken to merge segments in the particular index (in millisecond).
Flush Time Amount of time taken to flush one or more indices to disk (in millisecond).
Refresh Time Amount of time taken to refresh an index (in millisecond).

Configuration

PARAMETER DESCRIPTION
CONFIGURATION DETAILS
Cluster Name The name of the cluster.
Cluster Status The status of the cluster.
Total Nodes The total number of nodes in the cluster.
Master Node Name The name of the Master Node in the cluster.
Master Node Port The port on which the Master node of Elasticsearch runs.
Master Node IP The IP address in which the Master Node runs.
Publish Port The publish port of the cluster.

Elasticsearch

Overview

PARAMETER DESCRIPTION
AVERAGE SYSTEM LOAD
Avg. System Load The average value of the amount of load that is being processed by the system (in the last 1 minute, 5 minutes, and 15 minutes).
CPU UTILIZATION
CPU Utilization Amount of CPU currently being utilized by the node (in %). 
SEARCH TIME
Average Query Time The first phase of search operation is Query. The time taken to process the query in all shards
Average Fetch Time The second phase of search operation is Fetch. The time taken to retrieve the query result, only from the shards which have the requested data.
SEGMENT TIME
Average Merge Time The average time taken for segment merging in a node. (A shard in elasticsearch is a Lucene index, broken down into segments. Segments are, periodically, merged into larger segments to keep the index size at bay and expunge deletes.)
Average Refresh Time The average time spent in refreshing an index. (Refresh time increases with the number of file operations for the Lucene index).
INDEXING TIME
Average Index Time The average time taken to index a document. (Documents are indexed i.e stored and made searchable.)
Average Delete Time The average time taken to delete an existing index.
Indexed Count The number of documents indexed.
Deleted Count The number of deleted documents.
Indexing Rate The number of documents that are indexed per second.
GET TIME
Average Get Time The average time taken to retrieve information about one or more indexes
Existing Count The number of get requests that were present.
Missing Count The number of get requests that were missing.
FLUSH TIME
Average Flush Time The average time taken to flush one or more indices to disk. (The flush process of an index basically frees memory from the index by flushing data to the index storage and clearing the internal transaction log.)
WARMER TIME
Average Warmer Time The average time taken to perform a warmup search on an index. (Index warming allows to run registered search requests to warm up the index before it is available for search.)
PERCOLATE TIME
Average Percolate Time The average time spent running percolator queries. (One of Elasticsearch's core feature is the ability to do search in reverse with the percolator. The percolator automatically indexes the query terms with the percolator queries. This allows the percolator to percolate documents more quickly.)

Memory Details

The total space used in the Direct Buffer pool.

PARAMETER DESCRIPTION
HEAP MEMORY
Used Heap Percent The percentage of JVM heap currently in use.
Free Heap Percent The percentage of JVM heap currently free
NON-HEAP MEMORY
Used Non-Heap Percent The percentage of non-heap memory currently in use.
Free Non-Heap Percent The percentage of non-heap memory currently free.
GARBAGE COLLECTION
GC Time - Young The total time spent on young-generation garbage collections.
GC Time - Old The total time spent on old-generation garbage collections.
GC Count - Young The total number of young-generation garbage collections.
GC Count - Old The total number of old-generation garbage collections.
BUFFER POOLS
Direct Buffer Space Used The total space used in the Direct Buffer pool.
Mapped Buffer Space Used The total space used in the Mapped Buffer pool.
Direct Buffer Connection Count The total connections to Direct Buffer pool.
Mapped Buffer Connection Count The total connections to Mapped Buffer pool.

I/O Details

PARAMETER DESCRIPTION
DISK I/O COUNT
Disk Read Count The number of read ( from the disk) requests by Elasticsearch.
Disk Write Count The number of write ( to the disk) requests by Elasticsearch.
DISK I/O SIZE
Disk Read Size The total size of read requests ( from the disk) by Elasticsearch.
Disk Write Size The total size of write requests ( to the disk) by Elasticsearch.
CACHE DETAILS
Cache Name The name of the cache.
Total Size (MB) The size of the cache.
Evictions The number of evictions from the filter cache.
BREAKER DETAILS
Breaker Name The name of the Circuit Breaker. (Circuit breakers are designed to deal with situations when request processing needs more memory than available. This would mean OOM (OutOfMemoryException). So sometimes it is better to fail a query instead of getting OOM, because when OOM appears JVM becomes not responsive.)
Limit Size (MB) The limit size of the particular Breaker.
Used Size (MB) The used size of the particular Breaker.
Tripped The total number of times the breaker circuit tripped.

Thread Pools

PARAMETER DESCRIPTION
THREAD DETAILS
Thread Name The name of the thread.
Configured Threads The number of threads of current configured type.
Queue The number of thread of current type in queue.
Active The number of active threads of current type.
Rejected The number of rejected threads of current type.
Largest The number of largest threads of current type.

Network

PARAMETER DESCRIPTION
TRANSPORT
Transmitted Bytes The number of bytes sent by the network. (Transport metrics about cluster communication)
Received Bytes The number of bytes received by the network. (Transport metrics about cluster communication)
Transmitted Packets The number of data packets sent by the network. (Transport metrics about cluster communication)
Received Packets The number of data packets sent by the network. (Transport metrics about cluster communication)
TCP CONNECTOR
Active Connections The number of active TCP connections.
Passive Connections The number of passive TCP connections.
HTTP CONNECTOR
Current Connections The number of http connections currently active.
Total Connections The total number of http connections.

Configuration

PARAMETER DESCRIPTION
CONFIGURATION DETAILS
Cluster Name The name of the cluster.
Node Name The name of the node in the cluster.
Node Type The type of the node (Client/Data/Master-Eligible/Data-Master).
Host The IP address of the Host.
ElasticSearch Version The version of the installed Elasticsearch.
Port The port in which Elasticsearch runs.
ElasticSearch Home The home directory of Elasticsearch.
Total Processors The total number of processors in the current node
Java Version The version of Java running in the node.
Java Vendor The Java vendor.