Atlassian rebuilt its metrics platform around the OpenTelemetry Collector without initially asking service teams to change how they send metrics. Rather than removing the old pipeline and then reconfiguring thousands of services, the company kept the StatsD interface over UDP and gradually replaced the aggregation and processing layers behind it.
For most of the past decade, the previous platform had relied on gostatsd, an open-source StatsD application developed by Atlassian that operated as a sidecar agent on hosts and as an aggregation layer at the other end. According to its service-level objective, the platform handled metrics from approximately 100,000 hosts distributed across 14 regions, with a target of 99.95% and low latency.
Changing the Engine While Keeping the Contract Stable
Atlassian teams determined that reconfiguring every service to use the OpenTelemetry SDK before changing the infrastructure would be a multiyear process and could risk losing data on which alerts depend. The platform interface was therefore separated from its internal components: services continued communicating with the same address and in the StatsD format, while the internal layers were replaced with custom distributions of the OpenTelemetry Collector.
The design used four independent stages: collection, reception, aggregation, and routing. The collection layer was also made capable of receiving StatsD and OTLP simultaneously. This made it possible to begin the migration without requiring teams to switch client libraries, while opening a path for metrics originally created using OpenTelemetry.
What Changed in Practice?
In the collection stage, the gostatsd sidecar agent was replaced with an OpenTelemetry Collector distribution previously used by the tracing team, while applications retained their previous behavior. This made it possible to combine metrics and tracing in a single sidecar agent instead of running two agents on every host.
Atlassian says this consolidation reduced CPU consumption by an average of approximately 3.9% per service among its highest-cost Micros services, equivalent to an approximately 30% reduction in sidecar agent costs across the fleet. An OTLP receiver was also added so that the collection layer could receive OpenTelemetry metrics and route them directly.
In the reception stage, the main problem was related to time-series state. Every data point belonging to the same series had to be sent to a single aggregator. Through an internal agent called nomad, the old system used sharding based on the service-and-environment pair. However, differences in service sizes led to the emergence of hot aggregators and others with low utilization.
Atlassian addressed this by using the loadbalancingexporter component from the OpenTelemetry repository, with sharding based on streamID, the identity of an individual time series. This approach distributed data from large services across the aggregator pool while keeping each individual series on the same aggregator. As a result, CPU distribution among aggregators became more balanced, autoscaling capacity improved, and high-load alerts decreased.
Reducing Data and Cost in the Aggregation Layer
The aggregation layer handles approximately 4.8 billion data points per minute but stores only about 220 million, a reduction of approximately 96%. Because most metrics use delta temporality, and the previous components did not aggregate these differences in the way users expected, Atlassian developed a custom processor for aggregating metric deltas and released it as open source under the atlassian-labs scope.
After the migration, the aggregation layer required approximately half the previous CPU. The source attributes this improvement to several factors, including eliminating gostatsd format parsing, distributing load more effectively, and benefiting from improvements contributed by the OpenTelemetry community.
In the final stage, a custom internal router was replaced with a stateless Collector distribution that the company named metrics-gateway. This distribution relies on upstream exporters to support sending data to multiple destinations such as SignalFx and S3, with retry capabilities, queues, and backpressure control. According to the stated design, adding a new destination becomes a configuration change rather than a separate integration project.
Lessons from the Gradual Migration
- Start with the right teams: Atlassian selected development and testing environments, along with services that experienced the most problems, to obtain early feedback within a less sensitive scope.
- Continuously monitor production: Profiling under production loads revealed component behavior and costs in ways that small-scale tests or synthetic benchmarks did not.
- Maintain operational symmetry: Because the migration may continue for months or years while both systems operate in parallel, the company recommended retaining shared tools and operational procedures wherever possible.
- Expand the scope in stages: The rollout followed gradual percentages, starting at 1%, then 10% and 50%, and eventually reaching 100%, while testing problems in less sensitive services before critical paths.
Why Does This Approach Matter?
The experience shows that adopting a new standard in infrastructure does not necessarily require changing all consumers’ interfaces at the same time. Preserving the external contract transformed the migration from an organizational project involving every team into a project led by the infrastructure platform, while gradually opening the way to OTLP and OpenTelemetry components.
Atlassian notes that gostatsd and nomad together accounted for approximately 38% of CPU requests in the metrics clusters, while nomad alone accounted for approximately 13% of total resources. Therefore, the impact of the transition is not limited to tool consolidation; it is also tied to removing costly custom components and reducing the number of parts requiring internal maintenance.
However, this does not mean that the migration has been completed at the service-instrumentation level. The next step mentioned by the company is to move the instrumentation tools themselves to the OpenTelemetry SDK and gradually retire Datadog and DogStatsD clients and internal StatsD libraries that are still in use. The performance results and figures presented here also remain a description of Atlassian’s experience in its own environment, not a guarantee that the same values will recur in every monitoring architecture.