Apache HBase Monitoring

Apache HBase - An Overview

Apache HBase is a distributed, scalable, data store. Being open-sourced and non-relational, HBase is the database for you when you need random, real-time read/write access to your Big Data. Apache HBase provides Google's Bigtable-like capabilities on top of Hadoop and HDFS.

Monitoring Apache HBase - What we do.

Applications Manager aims to help Apache HBase administrators monitor the cluster in terms of region distribution. Let’s take a look at what you need to see to monitor Apache HBase, the performance metrics to gather and how you can ensure that your HBase database is up and operating as expected with Applications Manager:

  • Track HBase cluster metrics - Monitor HBase clusters to make sure your HBase cluster is up and running and is continuously operating as expected. Track the average load, requests, and the number of dead, live and overall number of region servers.
  • Memory and JVM usage - Track memory usage details such as available memory, free and total swap space size, the committed virtual memory size, etc. Keep track of JVM metrics to correlate server performance.
  • Guage Region servers performance - Get metrics about the block cache, the cache hit and miss counts and the hit ratio. Look at compaction metrics and and the total size of storage files that have been compacted.
  • Fix performance problems faster - Get instant notifications when there are performance issues with the components of Apache HBase components. Become aware of performance bottlenecks and take quick remedial actions before your end users experience issues.  

Apache HBase - Adding a new monitor

Supported versions: HBase 2.4.x and older versions

Prerequisites for monitoring Apache HBase:

JMX MBeans are used to collect HBase metrics. To enable JMX, open HBase/conf/HBase-env.sh and uncomment/add the below JVM arguments.

export HBase_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

export HBase_MASTER_OPTS="$HBase_MASTER_OPTS $HBase_JMX_BASE -Dcom.sun.management.jmxremote.port=10101"

export HBase_REGIONSERVER_OPTS="$HBase_REGIONSERVER_OPTS $HBase_JMX_BASE -Dcom.sun.management.jmxremote.port=10102"

Using the REST API to add a new Apache HBase monitor: Click here

  1. To create an Apache HBase Monitor, follow the steps given below: 
    • Click on New Monitor link. Choose Apache HBase.
    • Enter Display Name of the monitor.
    • Enter the IP Address or hostname of the host in which HBase is running.
    • Enter the JMX Port in the JMX Port field. For default installations of HBase, the JMX port number is 10101 for Master and 10102 for RegionServer.
    • Enter the credential details like user name, password and JNDIPath or select credentials from a Credential Manager list. If no authentication is required, then leave the fields blank.
    • Enter the polling interval time in minutes.
    • Click Test Credentials button, if you want to test the access to Apache HBase Server.
    • Choose the Monitor Group from the combo box with which you want to associate Apache HBase Monitor (optional). You can choose multiple groups to associate your monitor.
    • Click Add Monitor(s). This discovers Apache HBase from the network and starts monitoring.

Note:
In case you are unable to add the monitor even after enabling JMX, try providing the below argument:
  -Djava.rmi.server.hostname=[YOUR_IP]

Monitored Parameters

Go to the Monitors Category View by clicking the Monitors tab. Click on HBase under the Database Table. Displayed is the HBase bulk configuration view distributed into three tabs:

  • Availability tab, gives the Availability history for the past 24 hours or 30 days.
  • Performance tab gives the Health Status and events for the past 24 hours or 30 days.
  • List view enables you to perform bulk admin configurations.

Click on the monitor name to see all the server details listed under the following tabs:

Overview

Parameter Description
HBase MASTER OVERVIEW
Average Load The average number of regions served by each region server.
Dead Region Servers Count The number of dead region servers.
Region Servers Count The number of online region servers.
Cluster Requests / Min The total number of requests from all region servers to a cluster per minute.
HBase JVM METRICS
Total Heap Memory Calculates the total heap memory in MB.
Used Heap Memory Calculates the heap memory used in MB.
Committed Heap Memory Calculates the heap memory committed in MB.
Total Garbage Collections Total JVM garbage collection.
MEMORY DETAILS
Free Physical Memory Size The amount of free physical memory in Megabytes.
Free Swap Space Size Total amount of free swap space in Megabytes.
Total Physical Memory Size Total amount of physical memory in Megabytes.
Total Swap Space Size Total amount of swap space in Megabytes.
Committed Virtual Memory Size The amount of virtual memory that is guaranteed to be available to the running process in Megabytes.
THREAD DETAILS
Timed Waiting Threads Count The number of threads that are waiting for another thread to perform an action for up to a specified waiting time.
Waiting Threads Count The number of threads waiting for another thread to perform a particular action.
Blocked Threads Count The number of threads that are blocked waiting for a monitor lock.
Runnable Threads Count The number of threads that are being executed in the JVM

IPC

Parameter Description
IPC DETAILS
Number of Open Connections The number of open connections at the RPC layer.
Total Call Time The total call time, including both queued and processing time.
Number of Active RPC Handlers The number of RPC handlers actively servicing requests.
Number of Calls In Replication The number of currently enqueued operations received from replication.
DATA TRANSFERRED
Sent Data The number of bytes of sent data in the call queues.
Received Data The number of bytes of recieved data in the call queues.
EXCEPTIONS
Out of Order Scanner Exception  
Unknown Scanner Exception  
Region Too Busy Exception The region server is too busy exception.

Master - RegionServer

Parameter Description
REGIONS IN TRANSITION
Regions in Transition Counts the number of regions in transition.
Regions in Transition over Threshold Counts the number of regions in transition that exceed the threshold as defined by the property rit.metrics.threshold.time.
REGION SERVERS
Name The name of the region server.
Start Time The HBase master start time.

Region Server - Region Details

Parameter Description
STORE FILES
Store File Count The number of store files in the RegionServer.
Store File Size The total size of the store files (data files) in the RegionServer.
MEMSTORE
Memstore Size The total Memstore memory size of the RegionServer.
COMPACTION DETAILS
Compactions Completed Count Total number of compactions completed since startup by the nodes.
Num Bytes Compacted Count The number of bytes compacted across all this entity's hregions.
Num Files Compacted Count The number of files compacted across all this entity's hregions.
HLog COUNT
Delete Count The total number of Delete requests in the RegionServer.
Increment Count The total number of Increment requests in the RegionServer.
Append Count The total number of Append requests in the RegionServer.
Mutate Count The total number of Mutate requests in the RegionServer.
OPERATIONAL LATENCY - SCAN NEXT
Scan Next Num Operations The numer of Scan Next operations across all this entity's hregions.
Scan Next Min The minimum latency for Scan operation in the RegionServer.
Scan Next Max The maximum latency for Scan operation in the RegionServer.
Scan Next Mean The mean latency for Scan operation in the RegionServer.
Scan Next Median The median latency for Scan operation in the RegionServer.
OPERATIONAL LATENCY - GET
Get Num Operations The total number of Get requests per second in the RegionServer.
Get Min The minimum latency for Get operation in the RegionServer.
Get Max The maximum latency for Get operation in the RegionServer.
Get Mean The mean latency for Get operation in the RegionServer.
Get Median The median latency for Get operation in the RegionServer.

Region Server Details

Parameter Description
CACHE DETAILS
Block Cache Count The number of Store Files cached in the block cache.
Block Cache Eviction Count The total number of blocks that have been evicted from the block cache.
Block Cache Free Size The number of bytes that are free in the block cache.
Block Cache Express Hit Percent The block cache hit percent for requests where caching was turned on.
Block Cache Hit Count The total number of block cache hits for requests, regardless of caching setting.
Block Cache Count Hit Percent The block cache hit percent for all requests regardless of the caching setting.
Block Cache Miss Count The total number of block cache misses for requests, regardless of caching setting.
Block Cache Size The number of bytes used by cached blocks.
SLOW OPERATIONS COUNT
Slow Append Count The number of appends that took more than 1000 ms to complete.
Slow Get Count The number of gets that took more than 1000 ms to complete.
Slow Put Count The number of puts that took more than 1000 ms to complete.
Slow Increment Count The number of increments that took more than 1000 ms to complete.
Slow Delete Count The number of deletes that took more than 1000 ms to complete.

HLog

Parameter Description
HLog FILES SPLIT TIME
HLog Split Time Mean The average time to split the total size of a write-ahead log file.
HLog Split Time Min The minimum time to split the total size of a write-ahead log file.
HLog Split Time Max The maximum time to split the write-ahead log file after a restart.
HLog Split Time Num Operations The time to split write-ahead log files.
HLog FILE SPLIT SIZE
HLog Split Size Mean The average time to split the total size of an HLog file.
HLog Split Size Min The minimum time to split the total size of an HLog file.
HLog Split Size Max The maximum time to split the total size of an HLog file.
HLog Split Size Num Operations The size of write-ahead log files that were split.

Configuration

Parameter Description
CONFIGURATION
VM Name The Java virtual machine implementation name.
Boot Class Path The boot class path that is used by the bootstrap class loader to search for class files.
VM Vendor The Java virtual machine implementation vendor.
Spec Version The version of the JMX specification implemented by this product.