Building a Distributed Query Engine at Scale

Modern distributed systems generate massive volumes of telemetry data requiring fast query processing for real-time monitoring and analytics. As data volumes reach petabyte scale, traditional query architectures struggle to balance execution flexibility, resource utilization, modularity, cost efficiency, and performance. This thesis addresses these challenges through the design and implementation of Bolt, a query engine for large-scale event processing systems. The research demonstrates how to extend existing production infrastructure without disrupting operational stability, using a composable approach and standardized interfaces while maintaining compatibility with established distributed architectures. Chapter 1 establishes the problem statement: extending query capabilities for high-demand analytical workloads while addressing challenges in execution flexibility, resource utilization, modularity, and cost efficiency. Chapter 2 describes Datadog's Event Platform (EVP) architecture and its role in processing petabytes of telemetry data monthly, examining the multi-layered distributed query architecture that forms the integration environment of the new query engine. Chapter 3 explores Husky, Datadog's third-generation distributed event store, detailing its architecture that separates storage, compute, and metadata management, including its columnar storage format, coordination strategies, and multi-tenant isolation. Chapter 4 covers the theoretical foundation of distributed query engines, including query planning, execution engines, metadata management, partitioning strategies, scalability considerations, and fault tolerance mechanisms. Chapter 5 presents the design philosophy centered on composable data systems, introducing the MICE principles (Modular, Interoperable, Customizable, Extensible) and how standardized interfaces using Arrow, Substrait, and DataFusion enable incremental integration. Chapter 6 describes the implementation of the Bolt query engine, focusing on extending Husky's data access layer through Go and Rust runtime integration, documenting the technical challenges of bridging language boundaries, implementing data format translations, and preserving existing query semantics while enabling vectorized processing capabilities. Chapter 7 outlines the testing and validation approach, including a shadowing pipeline that enables safe production validation by executing queries on both engines simultaneously, alongside load testing methodologies and gradual rollout strategies. Chapter 8 presents evaluation results from synthetic benchmarks showing Bolt achieves 70% performance improvements for some common query patterns like filtering and aggregations on common fields. It also analyzes scalability characteristics and extensibility benefits while identifying areas for improvement for production readiness. Chapter 9 summarizes the system, lessons learned, and future directions. The resulting system shows how incremental integration can introduce new query processing capabilities into production environments while maintaining the reliability and operational characteristics required for critical infrastructure.