Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

Abstract

📜 Abstract

Modern Internet services are often implemented as complex, large-scale distributed systems. These applications are constructed from collections of software modules that may be developed by different teams, perhaps in different programming languages, and could span many thousands of machines across multiple physical facilities. Tools that aid in understanding system behavior and reasoning about performance issues are invaluable in such an environment. Here we introduce the design of Dapper, Google’s production distributed systems tracing infrastructure, and describe how our design goals of low overhead, application-level transparency, and ubiquitous deployment on a very large scale system were met. Dapper shares conceptual similarities with other tracing systems, particularly Magpie [3] and X-Trace [12], but certain design choices were made that have been key to its success in our environment, such as the use of sampling and restricting the instrumentation to a rather small number of common libraries. The main goal of this paper is to report on our experience building, deploying and using the system for over two years, since Dapper’s foremost measure of success has been its usefulness to developer and operations teams. Dapper began as a self-contained tracing tool but evolved into a monitoring platform which has enabled the creation of many different tools, some of which were not anticipated by its designers. We describe a few of the analysis tools that have been built using Dapper, share statistics about its usage within Google, present some example use cases, and discuss lessons learned so far.

Description

✨ Summary

Dapper is Google’s production distributed-tracing infrastructure for following the work performed across services and machines on behalf of an individual request. Its central data model represents a trace as a tree of spans connected by parent-child causal relationships. Each span contains timestamps, RPC timing information, identifiers, and optional application annotations. Trace and span identifiers are propagated through common threading, asynchronous control-flow, and RPC libraries, allowing most applications to be traced without application-specific instrumentation.

The design emphasizes three operational requirements: low runtime overhead, application-level transparency, and scalability. Dapper addresses these requirements through instrumentation in a small set of ubiquitous libraries, probabilistic sampling, asynchronous local logging, and out-of-band collection into Bigtable. Sampling is applied both at runtime and, when necessary, during collection; the latter operates on complete traces by hashing the shared trace identifier. The paper also describes an adaptive-sampling approach intended to increase coverage for low-volume services while limiting overhead for high-volume services.

Dapper’s deployment experience showed that distributed tracing could be used not only for interactive latency debugging but also for development, testing, dependency discovery, resource accounting, network-usage analysis, shared-service analysis, and security-policy monitoring. The paper reports that Dapper was deployed across nearly all Google production processes, with most traces requiring no manual propagation support. Its analysis interfaces supported lookup by trace identifier, bulk MapReduce processing, indexed access, execution-pattern aggregation, latency histograms, and interactive trace visualization.

The reported use cases demonstrate several concrete benefits. Dapper helped identify unnecessary serialization on critical paths, incorrect use of database masters instead of replicas, expensive interactions among services, network degradation associated with long-tail latency, dynamically changing service dependencies, and application-level contributors to shared-storage consumption. The paper also documents limitations: coalesced or batched work can be attributed inaccurately, batch workloads require a different unit of tracing, tracing often identifies the slow component without proving the root cause, and kernel-level events are difficult to associate unobtrusively with user-level trace context.

The paper became a widely cited reference for production distributed tracing. Later industry discussions describe Dapper’s trace/span model, context propagation, sampling, and large-scale collection architecture as foundational concepts for systems such as Zipkin, Jaeger, and OpenTelemetry. (research.google) A survey of microservice tracing specifically identifies Dapper’s use of sampling and restrictive instrumentation as design choices that enabled tracing at very large scale, and notes the subsequent growth of industrial and open-source tracing systems. (pmc.ncbi.nlm.nih.gov) The paper’s influence is therefore both technical and industrial: it provided a production-validated design point that shaped the architecture and terminology of subsequent distributed-observability tools.