Operating production services requires knowing what is happening inside them, but adding metrics is not free. In a presentation published by InfoQ, Brian Martin, co-founder of IOP Systems, explains that the difference between implementations can turn a counter update from an operation costing about 5 nanoseconds into one exceeding a microsecond. With histograms, the difference can grow from about 7 nanoseconds to tens of microseconds under thread contention.
The central idea of the presentation is not choosing a particular Rust library, but treating measurement as part of performance design. A metric placed inside a path called millions of times amplifies any small cost, while the absence of measurement makes diagnosing slowness, production incidents, and performance improvements more difficult.
Start by Understanding the Data Type and the Cost of Updating It
Martin distinguishes three main types of metrics: a counter that usually does not decrease, such as the number of requests; a gauge that represents a current value, such as queue depth; and a histogram that describes a distribution of values, such as response times. This distinction matters because each type requires different operations, and histograms provide information that a single total counter does not.
In the simplest cases, atomic fetch_add is suitable for integer counters. Compare-and-swap loops, or CAS, usually need to retry when multiple threads compete for the same location. Much of the cost comes from synchronizing cache lines between cores. The presentation reports measurements on an AWS Graviton machine with 32 virtual CPUs: the theoretical processing ceiling reached about 119 million requests per second with a low-cost atomic update, compared with about 23 million when using a higher-cost implementation with Prometheus.
Reduce Contention with Per-CPU Sharding
When all threads share a single counter, the cache line continually moves between cores. Martin instead suggests creating a separate counter for each CPU, so writes are almost contention-free, and then summing the values when reading. This approach slightly increases read cost, but protects the hot write path, which is repeated for every request.
Attention should be paid here to false sharing. Having logically separate counters is not sufficient if they are placed on the same cache line; the line is 64 bytes, meaning that eight 64-bit counters may be adjacent within it. The presentation therefore recommends grouping and padding counters so that they occupy separate cache lines. According to the figures presented, theoretical performance can increase from about 119 million requests per second with the atomic counter to about 6.4 billion requests per second when appropriate sharding is used.
Design Histograms for the Update Path
The cost of a histogram begins with determining which bucket a value belongs to. Linear searching through a list of buckets is the simplest and slowest option, while binary search reduces the number of comparisons but remains dependent on the number of buckets. The faster alternative is direct indexing, which calculates the bucket number from the value instead of searching for it.
There are trade-offs in this indexing. Dividing values into linear ranges is fast, but it can produce relatively large errors for small values. Logarithmic indexing better preserves relative error, but calculating the logarithm itself is expensive. Martin presents the use of outer ranges based on Log2, with sub-buckets that adjust precision, as in HDR Histogram and H2Histogram. In a non-atomic test, determining the bucket and updating it took about 2.65 nanoseconds in HDR Histogram and about 2.15 nanoseconds in H2Histogram.
When Is Approximate Consistency Acceptable?
The cost of a histogram is not determined by its indexing method alone. Some applications perform multiple atomic operations per event, use CAS to aggregate values, or enforce a lock to obtain a consistent snapshot. The presentation notes that some implementations reached more than two microseconds, and even tens of microseconds at 32 cores, while an implementation based on direct indexing and a single atomic update was closer to the cost of a counter.
The alternative is approximate consistency: some buckets may change while the histogram is being read, but the difference between two successive readings remains useful when the metrics are inherently approximate. This is not a general rule; systems that require a perfectly consistent snapshot may prefer synchronization despite its cost.
Editorial Take from certi.news
What changes in practice is that the decision to add metrics should include the update architecture, not just metric names. An atomic counter, per-CPU sharding, and direct indexing can make measurement usable inside sensitive paths, while synchronized or dynamically resizable histograms may impose a significant cost under load. The presentation does not offer a single recipe valid for every library or service; it shows that flexibility, the ability to use a library within other projects, consistency, and performance are partially conflicting goals. Therefore, the actual implementation should be tested under the target levels of contention and processing volume, rather than relying on the library's name or the result from an uncontended case.
Martin also presents the use of eBPF through the Rezolus project to obtain precise metrics from the Linux kernel, including the scheduler, system-call paths, and the TCP stack, without modifying kernel code. The open question remains how much precision and consistency are required in each case, and whether the cost of reading or aggregating the parts will remain acceptable as the system scales.