EVE Online has adopted an unconventional approach to distributed monitoring because of the nature of its legacy architecture and the scale of its time-based simulation. Instead of sending all spans and then deciding whether to retain them at the end of tracing, the platform samples from the start of tracing, using a mixture of deterministic and dynamically varying samples, with the aim of preserving an analyzable picture of the system without slowing the simulation or increasing the monitoring data bill.
Nicholas Herring of CCP Games presented this experience in material published through InfoQ, explaining that EVE Online operates in a single shared universe containing all players, rather than in separate regional servers. Its legacy system and the newer environment known as Quasar exchange approximately 30,000 messages per second. The platform also contains millions of lines of C, C++, Python, and SQL, making the introduction of a new centralized monitoring layer an operational challenge in its own right.
Why didn’t the platform use late sampling?
The decision was related to performance and the constraints of the existing system. Part of EVE Online runs using Stackless Python 2.7 on a single logical processor, so the team believed that extracting complete spans and then sending them to a central point for processing could harm the simulation itself. Late sampling also requires collecting spans in one place, which could conceal large parts of the system if the collection path or its connection fails.
By contrast, sampling from the start allows each component to make its decision locally. When certain parts degrade, the team can infer the location of the problem from differences in those parts’ ability to send their data, rather than losing visibility across the entire system.
A combination of determinism and EMA
The older systems use deterministic sampling to reduce cost and performance impact. The newer Quasar components, meanwhile, benefit from an exponential moving average algorithm, or EMA, which monitors keys such as protobuf message type and occurrence rate. The more frequently a message appears, the more its sampling rate is reduced; rare messages receive a greater chance of appearing.
This approach helps reduce the noise from repeated events, such as firing huge numbers of bullets or changing the appearance of ships. However, the team does not rely on it alone, because some important events come from legacy systems or need to be guaranteed to appear at least once. Distributing EMA across multiple instances creates what the presentation calls a “high-water mark”: a practical minimum data volume that cannot easily be reduced after the number of instances, keyspace, or message rate increases.
What changes in practice?
The experience shows that tuning monitoring does not begin with choosing a tracing tool, but with understanding the nature of the events the team wants to measure. CCP Games uses feature flags to temporarily intensify monitoring for a specific area, ship, or character, or to reduce monitoring of less important domains. It also relies on tools that display sampling rates and infer the actual volume of events, rather than reading raw figures as though they represented the entire universe.
The platform used approximately 60 million events per day in one Quasar configuration, while targeting utilization of about 90% of the available quota. This makes it possible to run intensive monitoring when needed while reducing the risk of exceeding the cost. However, this flexibility requires operational discipline; the presentation noted cases in which monitoring flags were left enabled, resulting in alerts about increased spending.
Limitations to keep in mind
This approach does not provide all the benefits of late sampling. The team cannot later control the ratio of successes to failures in the same way, so service-level indicators may become excessively sensitive when the event is extremely rare. The experience also concluded that making sampling decisions inside desktop clients could conceal important interactions; for this reason, the team is considering moving the decision to a trusted network within the backend infrastructure.
The practical conclusion is not that sampling from the start is always superior, but that real-world constraints may make a combination of deterministic sampling, EMA, and temporary activation more suitable than an expensive centralized architecture. The value of this approach remains tied to the team’s ability to define what is worth monitoring, understand the cost high-water mark, and preserve at least one sample of every important type of event.