Kubernetes operations teams need more than dashboards displaying CPU and memory consumption and error rates. The complexity of cloud-native environments means that a single request passes through an ingress gateway, services, queues, storage, and background processes, while workloads move and versions change continuously. Therefore, a deployment may appear healthy at the Deployment level, while a later dependency, a retry loop, or pressure on a control-plane path causes latency to rise.
In an article published on the CNCF blog, Neel Shah of Stackgen explains that traditional monitoring answers specific predefined questions, such as: Has CPU usage exceeded a certain threshold? Are errors increasing? Observability, according to the article’s argument, aims to help the team investigate a problem it did not expect and move from observing the symptom to understanding the cause and scope.
Metrics Start the Investigation but Do Not End It
Metrics remain the natural starting point because they are numerical, efficient to store and query, and suitable for alerts and trend analysis. In Kubernetes, they can show node pressure, container restarts, rising request latency, a slowing API server, or accumulating queue items.
The article suggests using the RED method for services—request rate, errors, and duration—and the USE method for infrastructure—utilization, saturation, and errors. These indicators help create an initial picture of the incident: a rise in request rate with stable latency differs from a rise in duration and saturation with traffic remaining constant.
However, turning every detail into a metric label creates a cardinality problem, as the large number of unique combinations increases costs and slows queries. The article specifically warns against placing request or user identifiers, or near-unique values, in metrics; such details are better suited to logs or traces.
Each Signal Answers a Different Question
Logs add local context that graphs typically do not retain. When they are structured and use consistent fields such as time, severity level, service name, namespace, container identity, request path, and trace context, it becomes easier to link a specific event to the service or process that produced it. A metric may indicate that the payment service is affected, while a log reveals an expired timeout, an exception, or a dependency failure.
Distributed traces answer a different question: How did a single request move through the system, and where was the time spent? Their importance is particularly evident in Kubernetes because a failure may be distributed across multiple services, retries, queue limits, or database calls. Passing trace context makes it possible to link different spans within the context of a single request, while shared semantic conventions help standardize field and attribute names across metrics, logs, and traces.
The article also includes profiling in the picture. After metrics identify the slow service, tracing identifies the affected request path, and logs clarify the local event, profiling data can help identify the function or code path consuming CPU or memory.
What Changes Operationally During the Incident?
The article presents an example in which a checkout service begins exceeding its latency target after a new deployment, while CPU and memory indicators remain normal. Metrics reveal that there is a problem, then a trace of a slow request shows that most of the delay occurs in the payment authorization step. The logs subsequently show repeated timeout messages associated with the same request context.
This sequence moves the team from a general question—why has the payment service become slow?—to more specific operational options, such as rolling back a change in a dependency, reducing retry amplification, or temporarily rerouting traffic during the investigation. The article also emphasizes that the best alert reflects a risk to service quality or its reliability objective, not merely raw infrastructure-resource distress; it gives the example of an alert tied to the p99 response latency exceeding one second for ten minutes.
Applicable Design Principles
- Start with the available metrics and logs, then gradually expand coverage instead of collecting everything without a purpose.
- Prefer dimensions related to the service and workload over highly unique labels within metrics.
- Use a consistent metadata structure across metrics, logs, and traces.
- Add request or trace identifiers to logs to make it easier to move between signals.
- Build alerts around reliability risks and service quality, rather than resource pressure alone.
- Consider observability part of application and platform design, not a later addition after deployment.
Editorial reading from certi.news: The core value here is not a call to buy a tool or adopt a single implementation, but a redefinition of the goal of observability. What actually changes is how data is used: the metric captures the deviation, the trace identifies the path, the log explains the event, and profiling may identify the cause at the code level. Clear practical limitations remain; collecting more signals does not guarantee better understanding, and the absence of standardized names and fields can make correlation fragile, while unique details may raise metric costs and weaken query performance. Therefore, usefulness depends on the quality of the design and the connection between signals, not on the number of dashboards.