Pod level monitoring in Kubernetes

Essential pod metrics to track

Effective Kubernetes pod monitoring begins with the identification of key performance indicators. These metrics reflect both the health of your infrastructure and the behavior of your applications. By tracking the right data, you can ensure your Kubernetes environment remains stable and efficient.

1. Core infrastructure and resource metrics

Monitoring resource consumption is the first step toward stability. It helps you understand if your pods have enough power to perform their tasks.

CPU and memory utilization

CPU utilization measures the percentage of allocated cores that pod containers consume. This metric helps you detect CPU throttling or overprovisioning. For example, if usage stays above 80%, it often signals that you should use a horizontal pod autoscaler.

Memory usage tracks the resident set size and the working set. This is critical to prevent out-of-memory kills. These kills happen when Kubernetes evicts pods that exceed their set limits. If you monitor the ratio between requests and limits, you can ensure balanced scheduling across your nodes.

Network and storage indicators

Network metrics reveal bottlenecks in service meshes or internal traffic. You should track ingress and egress bytes as well as packet drops. Additionally, keep an eye on disk I/O for ephemeral volumes. If you run machine learning workloads, you must also monitor GPU utilization.

2. Pod lifecycle and stability

Infrastructure data alone does not show the full picture. You must also look at how pods behave within the cluster.

Restarts and status changes

The pod restart count is a clear indicator of instability. Frequent restarts often stem from application crashes, evictions, or failed liveness and readiness probes.

You should also monitor container status and phase transitions. Pods typically move through several phases:

• Pending
• Running
• Succeeded
• Failed
• Unknown

Lifecycle insights help you troubleshoot specific problems. For instance, if a pod stays in a Pending state, you likely have insufficient node resources or an image pull failure.

3. Application level performance

To see beyond the infrastructure, you must collect application-specific data. This information is often pulled through Prometheus exporters that are embedded in your pods.

• HTTP Request Latency: Measures how long it takes for your app to respond.
• Error Rates: Tracks the frequency of 4xx and 5xx status codes.
• Throughput: Monitors the number of requests per second.

Workload types determine your target thresholds. Stateless web apps can usually tolerate higher performance spikes. In contrast, databases require steady performance to remain reliable.

Native Kubernetes monitoring tools

Kubernetes includes lightweight, native mechanisms for monitoring pods. These tools are ideal for quick diagnostics because they do not require external dependencies. While they lack the long-term storage of enterprise platforms, they provide immediate insights into cluster health.

Basic command line diagnostics

The command line is the first place to look when a pod behaves unexpectedly. These commands provide a snapshot of the current state of your resources.

• Status overview: Use kubectl get pods to see a basic list of pod statuses.
• Deep dives: Use kubectl describe pod <name> to view detailed events, conditions, and probe failures.
• Log inspection: Use kubectl logs <pod> to see stdout and stderr output. This is the primary way to debug application-level errors.
• Direct access: Use kubectl port-forward to gain ephemeral access to specific pod ports for testing.

Real-time resource monitoring with Metrics Server

To see resource usage in real-time, you must enable the Metrics Server. You can do this by applying the components.yaml file from the cluster add-on site.

Understanding Metrics Server

The Metrics Server collects CPU and memory data from Kubelet summaries via cAdvisor. It then exposes this data through a specific API endpoint. This endpoint is used by the Horizontal Pod Autoscaler (HPA) to make scaling decisions.

While Metrics Server is fast, it has limitations. It does not store historical data and does not provide deep breakdowns of specific container processes.

Advanced cluster introspection

For issues that go beyond a single pod, you can look at the events and infrastructure components that manage the cluster.

The Events API

The Events API reveals why a pod might fail to start. By running kubectl get events --sort-by=.metadata.creationTimestamp, you can see scheduling issues. These often include:

• Node taints that prevent pods from landing on a node.
• Affinity violations where pods cannot find a compatible neighbor.
• Insufficient resource quotas.

Proxy and DNS metrics

You can gain indirect insights into pod health through the API server proxies. Monitoring CoreDNS and kube-proxy metrics helps you understand if networking issues are affecting pod communication.

Production limitations

Native tools work well for development or small-scale troubleshooting. However, they often fail in production environments for several reasons:

• Lack of Persistence: Data disappears once a pod or node is deleted.
• No Native Alerting: These tools show you the current state but cannot notify you when things go wrong.
• No Multi-cluster Support: Managing visibility across several clusters using only the command line is difficult and slow.

Deploying Prometheus for Kubernetes pod monitoring

Prometheus is the industry standard for Kubernetes observability. It provides deep granularity for pods through automated service discovery and data relabeling.

The following table outlines the key components and processes for a standard Prometheus deployment:

Scaling for large clusters

In clusters with over 1,000 pods, a single monitoring instance can become overwhelmed. You should shard your Prometheus setup and use "remote write" to send data to long-term storage like Thanos. Sampling data (focusing on the 95th percentile) can also prevent your monitoring system from crashing under the weight of too many metrics.

Integrating logging and tracing for total visibility

To truly understand your pods, you must combine metrics with logs and traces. This holistic approach ensures you can see not only that a pod is struggling, but also exactly what code or request is causing the trouble.

Unified observability streams

Tools like Fluentd or Fluent Bit run as DaemonSets to collect logs from every container on a host. These logs move to systems like Elasticsearch, where you can filter them by namespace or pod name. If you use structured JSON logging, you can quickly search for specific error codes across thousands of pods.

For microservices, tracing is essential. Jaeger or Zipkin can be deployed as sidecars to map distributed request flows. This reveals exactly which pod is causing latency in a complex chain of service calls.

Best practices and alerting strategies

A strong monitoring strategy follows a hierarchy: start at the pod level for granular alerts, move to the namespace for team management, and use the cluster level for capacity planning.

Proactive health management

• Resource Allocation: Set your resource requests and limits based on observed peaks (usually 50% to 70%). Use the Vertical Pod Autoscaler (VPA) to help find the right balance.
• Health Probes: Implement readiness and liveness probes to remove unhealthy pods quickly. Monitor the failure rates of these probes as early warning signs of app instability.
• Semantic Labeling: Use consistent labels like team or app name so you can filter your queries easily.
• Alert Quality: Focus on actionable alerts. Create runbooks that link specific alerts, like pod_cpu_high, to automated actions like Horizontal Pod Autoscaling (HPA).

Security and cost control

Monitor pod admission through Gatekeeper audits and use network policies to block suspicious data egress. From a financial perspective, comparing your resource limits to actual usage can save you 20% to 40% on your cloud bills by rightsizing your infrastructure.

Troubleshooting common pod issues

When a pod fails, the following checklist helps you find the root cause quickly:

• High Restarts: Run kubectl describe to check for OOMKilled (needs more memory) or CrashLoopBackOff (likely a code or image issue).
• Resource Starvation: Use kubectl top alongside scheduler logs to see if nodes are overcommitted.
• Noisy Neighbors: Use Pod Disruption Budgets (PDBs) and taints to protect critical pods from being evicted due to other busy workloads.
• Network Hangs: Use tools like Istio or Cilium Hubble to inspect the network proxy status.

Pod level resource usage monitoring

Granular monitoring of CPU, memory, and storage per container allows for precise quota enforcement. After you deploy the Metrics Server, you can query real-time data using:

kubectl top pod <pod-name> -n <namespace>

For historical trends, calculate the usage ratio (usage divided by limit). This helps you find "overcommitted" pods that risk crashing the node. If you use specialized hardware like GPUs, you can extend these metrics through Custom Resource Definitions (CRDs).

Practical commands and examples

Container resource check:

kubectl top pods --containers -n prod

Pod health report:

kubectl get pods -o custom-columns=NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,RESTARTS:.status.containerStatuses[0].restartCount

VPA configuration snippet: YAML

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
spec:
  targetRef:
    kind: Deployment
    name: myapp
  updatePolicy:
    updateMode: "Auto"

Advanced features with Applications Manager

ManageEngine Applications Manager stands out for its agentless Kubernetes monitoring. It can auto-discover entire clusters using only the credentials of the primary master node. This eliminates the need for per-node installations, drastically reducing setup time. Once connected, it maps your nodes, namespaces, deployments, and pods automatically.

Comprehensive pod KPI tracking

The platform provides a detailed view of pod-specific performance. It tracks several critical indicators, helping you stay ahead of potential failures:

• Resource Utilization: Visualizes CPU and memory requests versus actual limits in top-10 charts.
• Stability Metrics: Monitors restart counts and pod phase anomalies (such as pods stuck in "Pending").
• Infrastructure Context: Maps pod IP addresses, node assignments, and container images.
• Namespace Breakdowns: Uses pie charts to show running versus failed pods across different environments.

Unified dashboards and correlation

A major strength of Applications Manager is its ability to aggregate metrics from every layer of your stack. You can view pod health alongside your application layers, such as JVM, .NET, or databases.

This unified view makes it easy to correlate issues. If a pod shows slow response times, you can quickly check if an upstream database query or a memory leak in a Java application is the root cause. This "one-click deep-dive" allows you to move from a high-level cluster overview directly into specific pod events and log entries.

Comparison: Applications Manager vs. Prometheus

Alerting and capacity planning

Applications Manager uses proactive baselines to flag performance deviations before they become incidents. You can set alerting profiles for specific thresholds:

• Restarts: Get notified if a pod restarts more than 5 times in an hour.
• Exhaustion: Trigger alerts when resource use exceeds 90%.
• Remediation: Integrate these alerts with Slack, email, or webhooks to trigger automated fixes.

For long-term growth, the tool offers historical trending and "what-if" analysis. This allows you to simulate the impact of scaling your cluster before you commit to new hardware or cloud costs.

Prometheus integration with Applications Manager

Applications Manager integrates with Prometheus to combine the power of open-source metrics with enterprise-level monitoring. This integration is designed for teams that already use Prometheus but want a unified console for alerting, long-term storage, and correlation across their entire IT stack.

Key features of the integration

The integration allows Applications Manager to act as a management layer over your Prometheus servers. It supports several monitor types, including Kubernetes, Nginx, OpenShift, and Tomcat servers.

Benefits of combining both tools

Using Applications Manager alongside Prometheus helps resolve some of the common challenges found in purely open-source environments.

• Deeper insights: By ingesting Prometheus metrics, Applications Manager can correlate container-level data with your application layer (like Java or .NET). This helps you find if a CPU spike in a pod is tied to a specific background transaction.
• Enhanced security: The integration allows you to monitor environments within a managed security perimeter, minimizing exposure to external threats while still maintaining near-real-time updates.
• Reduced operational overhead: Unlike a standard Prometheus setup that requires manual YAML configuration for every alert, Applications Manager provides an out-of-the-box UI for setting thresholds and remediation workflows.
• Persistent historical data: While Prometheus is often configured for short-term retention, Applications Manager can store these metrics for up to a year, which is essential for capacity planning and "what-if" analysis.

How to get started

To enable this integration, navigate to Settings -> Add-on Settings in the Applications Manager console. From there, you can click the plus icon to add a Prometheus server. You will need to provide the server type, authentication details, and discovery specifications to begin importing your scrape jobs. Read more about how to integrate Prometheus server for Kubernetes here.