Istio Monitoring


Overview

Istio is an open source independent service mesh that provides the fundamentals you need to successfully run a distributed microservice architecture. As organizations increasingly adopt cloud platforms, developers have to architect for portability using microservices, while operators have to manage large distributed deployments that span hybrid and multi-cloud deployments. Istio reduces complexity of managing microservice deployments by providing a uniform way to secure, connect, and monitor microservices.

Creating a new Istio monitor

Prerequisites for monitoring Istio metrics: Click here

Using the REST API to add a new Istio monitor: Click here

Follow the steps given below to create a new Istio monitor:

  1. Click on New Monitor link.
  2. Select Istio under Services category.
  3. Specify the Display Name of the Istio server.
  4. Enter the Cluster Host Name/IP address of the server where Istio is running.
  5. Enter the credential details like user name and password for authentication, or select the required credentials from the Credential Manager list after enabling the Select from Credential list option.
  6. Check the box to enable Public Key Authentication (Supported for SSH2 only), the SSH Key for SSH authentication.
  7. Specify the command prompt value, which is the last character in your command prompt. Default value is $ and possible values are >, #, etc.
  8. Enter the SSH port. Default SSH port used is 22.
  9. Specify the Timeout in seconds. Default value is 40 seconds.
  10. Specify the Polling Interval in minutes.
  11. Choose the Monitor Group with which you want to associate the Istio to, from the combo box (optional). You can choose multiple groups to associate your monitor.
  12. Click Add Monitor(s). This discovers Istio from the network and starts monitoring it.

Monitored Parameters

Go to the Monitors Category View by clicking the Monitors tab. Click on Istio under the Services table. Displayed is the Istio bulk configuration view distributed into three tabs:

  • Availability tab gives the Availability history for the past 24 hours or 30 days.
  • Performance tab gives the Health Status and events for the past 24 hours or 30 days.
  • List view enables you to perform bulk admin configurations.

On clicking a monitor from the list, you'll be taken to the Istio monitor dashboard. It has 8 tabs -

Note: Tabs/Metrics marked with an * will be shown only for Istio versions below 1.5.0.

Overview

Parameter Description
ISTIO DETAILS
Namespace Namespace of the Istio server.
Status Status of the Istio namespace. (Active or Inactive)
No. of Istio Pods Number of pods running under Istio namespace.
Version Version of the Istio namespace.
ISTIO COMPONENT DETAILS *
Citadel Count Number of Citadel instances running per component.
Galley Count Number of Galley instances running per component.
Mixer Count Number of Mixer instances running per component.
Pilot Count Number of Pilot instances running per component.
ISTIOD COMPONENT DETAILS
Istiod count Number of instances (replicas) running per component.
Istio Pod Details
Pod Name Name of Istio pod.
Pod Type Type of the Istio pod.
Pod status Status of the Istio pod.
Deployed Node Node (host) of the deployed Istio pod.
ISTIO MCP RECONNECTIONS
Avg. sink reconnections Number of times the sink has reconnected.
ISTIO MCP RECEIVE FAILURES
Receive Failures in Source Number of mesh connection receive failures in the source.
ISTIO MCP RECEIVED ACKS
Request acks. received by source No. of request acknowledgments received by the source.

Citadel

Parameter Description
Citadel Certificate Details
Pod Name Name of Istio pod.
Certificates created for service account creation Number of certificates created for service account creation.
Root Cert. Expiry Date and time at which the root certificate will expire.
Days to expire Number of days remaining for the certificate to expire.
CSR Count Number of Certificate Signing Requests (CSR) made by the certificate.
gRPC Response Handled
gRPC Code Error status codes raised by the gRPC.
Response Handled Number of response types handled by the gRPC.

Galley

Parameter Description
Galley Instances *
Instance URL The URL of the instance.
Instances per URL Number of instances running per URL.
Webhook Configuration and Validation Details
Pod Name Name of Istio pod.
Webhook Cert. Updates Number of Webhook certificate updates received.
Config. loads Number of Webhook configuration loads tried.
Config. updates Number of Webhook configuration updates applied.
Config. errors Number of Webhook configuration errors obtained while applying.
Passed Validation Number of user configuration validation that has been passed.
Failed Validation Number of user configuration validation that has been failed.
Galley Runtime Details
Pod Name Name of Istio pod.
Events processed Number of events that have been processed.
Timer Resets Number of times the timer has been reset.
Successful Event Handles Number of times a Kubernetes source has successfully handled an event.
OnChange called Number of configuration changes called by a Galley strategy.
Snapshots published Number of snapshots published by a Galley strategy.

Mixer *

Parameter Description
Mixer Configurations *
Pod Name Name of Istio pod.
Known Adapter Configs. Number of known adapters encountered during processing of the adapter info configuration.
Known Instance Configs. Number of known instance encountered during processing of the instance configuration.
Known Rule Configs. Number of known rules encountered during processing of the rule configuration.
Known Template Config. Number of known template encountered during processing of the template configuration.
Known Handler Configs. Number of known handler encountered during processing of the handler configuration.
Mixer Config. Errors *
Pod Name Name of Istio pod.
Adapter Config. Errors Number of errors encountered during processing of the adapter info configuration.
Instance Config. Errors Number of errors encountered during processing of the instance configuration.
Template Config. Errors Number of errors encountered during processing of the template configuration.
Reused Handlers Number of handlers that were reused.
Handler Validation Errors Number of errors encountered because handler validation returned error.
Mixer Handlers *
Pod Name Name of Istio pod.
Total Mixer Handler Daemons Total number of Mixer handler daemons available.
Handlers failed at creation Number of handlers that were failed at creation.
Handlers failed at closing Number of handlers that were failed at closing.
Newly Created Handlers Number of handlers that were newly created.
Reused Handlers Number of handlers that were reused.

Pilot

Parameter Description
Inbound Listener Conflict Details
Pod Name Name of Istio pod.
Inbound Listener Conflicts Number of inbound port conflicts found.
Outbound Listener Conflict Details
Pod Name Name of Istio pod.
HTTP over current TCP Number of conflicting wildcard HTTP listeners with current wildcard TCP listener.
HTTP over HTTPS Number of conflicting HTTP listeners with well-known HTTPS ports.
TCP over current HTTP Number of conflicting wildcard TCP listeners with current wildcard HTTP listener.
TCP over current TCP Number of conflicting TCP listeners with current TCP listener.
Pilot Service Details
Pod Name Name of Istio pod.
Pilot services known Number of services known to Pilot.
Virtual services known Number of virtual services known to Pilot.
Services with duplicated domains Number of virtual services with duplicated domains.
Endpoints Connected using XDS Number of endpoints connected to Pilot using xDS protocol.
PILOT XDS PUSHES
Cluster Discovery Service Number of configuration pushes for Cluster discovery services.
Endpoint Discovery Service Number of configuration pushes for Endpoint discovery services.
Listener Discovery Service Number of configuration pushes for Listener discovery services.
Route Discovery Service Number of configuration pushes for Route discovery services.
AVG PROXY CONVERGENCE DELAY
Avg. delay between config. change Avg. delay between a configuration change and a proxy receiving all required configuration (in milliseconds).
Clusters and Endpoint Details
Pod Name Name of Istio pod.
Duplicate Envoy Clusters Number of Duplicate envoy clusters caused by service entries with same host name.
Cluster without instances Number of clusters without instances.
Unready endpoints Number of endpoints found in unready state.

Go Stats

Parameter Description
Go Memory Details
Pod Name Name of Istio pod.
Allocated Memory Amount of memory allocated for an Istio pod (in MB).
Heap Available Amount of heap memory available for use (in MB).
Heap In Use Amount of heap memory currently in use (in MB).
Cache Available Amount of cache memory available for use (in KB).
Cache In Use Amount of cache memory currently in use (in KB).
Go Routines Number of Go Routines that are running.
Go Threads Number of OS Threads that are created.
Memory used for GC Amount of memory used for Garbage Collection (in MB)
Memory targeted for next GC Amount of memory targeted for next Garbage Collection (in MB)
Last Garbage Collection Date and time at which Garbage Collection was last performed.

Processes

Parameter Description
Process Details
Pod Name Name of Istio pod.
Opened File Descriptors Number of open file descriptors in use.
Maximum File Descriptors Maximum number of file descriptors available.
Free File Descriptors Number of file descriptors that are unused.
Virtual Memory Size Amount of virtual memory available (in MB).
Process Start Time Date and time at which the process was started.
Total Running days Total number of the days for which the process was running.

Services

Parameter Description
Incoming Traffic Details
Workload name Name of the workload.
Request count Number of requests received between two intervals.
Failed requests Number of failed requests received by the workload.
Avg. Request size Avg. size of request received by the workload (in KB).
Avg. Response size Avg. size of response sent by a workload (in KB).
Avg. Request duration Avg. duration between a request and response (in seconds).