The problem begins with a clear operational paradox: GPU usage data in Adobe’s environment was collected every second inside a central Prometheus instance, but the teams bearing the cost of those GPUs could not view their metrics directly. According to engineers Bingi Narasimha Karthik and Ramkumar Nagaraj, this led to the discovery of a GPU that had remained at zero utilization for 11 days—allocated and running, but invisible to the team responsible for it.
The material published on the CNCF blog on September 9, 2026, does not introduce a new commercial product. Instead, it explains a practical pattern for building self-service, secure access to metrics in multi-tenant Kubernetes clusters. The core idea is to place a tenant-aware intermediary layer in front of the central Prometheus instance, then grant each team a selected slice of its data, with the option of copying that data to its own Prometheus instance.
Why Isn’t Opening the Central Prometheus Instance Enough?
The authors argue that granting teams read access to the central Prometheus query endpoint creates two problems. The first is security: the Prometheus query endpoint is not namespace-aware, and anyone able to execute a PromQL query could theoretically request data belonging to other teams, such as request rates or capacity plans.
The second problem is performance. The central store serves metrics for the entire fleet, and long or inefficient queries from hundreds of engineers could consume resources and increase latency for everyone. Opening the shared store therefore does not solve the visibility problem; it may instead add the risk of data leakage and a noisy-neighbor problem to the architecture itself.
An Intermediary Layer with Three Responsibilities
The design proposes a thin layer in front of Prometheus that performs three interconnected functions:
- Identification: Authenticate the requester and determine the tenant to which the requester belongs.
- Isolation: Restrict every query to the tenant’s namespace, enforcing the restriction before the query reaches Prometheus so it cannot be bypassed through PromQL.
- Delivery: Periodically copy a selected set of the tenant’s metrics to a tenant-owned Prometheus instance when needed.
The read path uses Nginx for load balancing, followed by kube-rbac-proxy for authentication and authorization. Requests then reach the proxy, which discovers Prometheus servers through the Kubernetes API and gathers results from healthy servers. The write path uses remote write to send selected metrics to the tenant’s Prometheus instance. In highly available environments, data is sent to all replicas through the Pods’ DNS names.
Isolation Starts with Identity and Ends with Data
kube-rbac-proxy relies on Kubernetes identity and RBAC to identify the requester, then passes the tenant identity as a namespace assertion. prom-label-proxy enforces isolation at query time by rewriting the request to add a namespace selector before sending it to Prometheus. In this way, isolation is not merely a policy recommended to users; it is a restriction applied to every query.
The source also points to hardening measures for the proxy itself, including running it as a non-root user with UID 65534, using a read-only root filesystem, removing all additional capabilities, preventing privilege escalation, and assigning it a service account with the fewest possible permissions.
What Changes in Practice for Cost and Performance?
Isolation is not limited to preventing visibility into other teams’ data. The metricIsolation setting applies a namespace filter during metric collection, so the tenant’s Prometheus instance stores only the series belonging to that tenant. According to the authors’ experience, the number of stored series for a typical tenant may decrease by approximately 97%, from more than 10,000 series to a few hundred.
This reduction means a smaller store, faster queries, and lower storage costs. It also reduces the likelihood of data leakage because unneeded metrics never reach the private store in the first place. The design also eases pressure on the central Prometheus instance, as dashboards and daily alerts move to tenant stores instead of continually relying on the shared store.
Self-Service Operations Require Clear Boundaries
The team defines a custom Kubernetes resource named MetricAccess, which specifies the namespace, requested metrics, remote write destination, and collection interval. Tenants can choose specific metric names, regular expressions, or PromQL selectors. The target Prometheus instance must enable remote write reception through web.enable-remote-write-receiver, while the rest of the architecture relies on standard Prometheus and Kubernetes components.
The material presents six useful query types, including average GPU utilization by namespace; counting units with utilization below 5% over an hour; measuring memory utilization; tracking power consumption; detecting busy units with no request activity; and identifying requests while GPUs remain idle. The example relies on DCGM-style metrics, with a warning that names must be aligned with the exporter actually in use.
The experience confirms that self-service does not mean removing guardrails. Selected metric groups and different collection intervals act as quotas that limit load. The remote write option should also be reserved for teams with actual dashboards and alerts, while restricted query-time access may be sufficient for smaller teams.
certi.news’s Take
The important change here is not the addition of another monitoring tool, but the transfer of visibility control from the platform team alone to the tenant while keeping isolation under infrastructure control. This addresses a security problem, a cost problem, and a performance problem at the same time. However, the solution is not automatic: metric selection, cardinality control, retry handling, writing to highly available replicas, and pinning exporter versions are all ongoing operational responsibilities.
The reported figures, including the approximately 97% reduction in series, come from the authors’ experience and a “typical tenant” and are not a general guarantee for every cluster. The pattern should therefore be tested against each environment’s metric names, series volume, and collection intervals before being adopted at scale. The project is available under Apache 2.0, and the material includes a link to the prometheus-multi-tenant-proxy repository on GitHub.