druid-multi-stage-query
developer notesThis document provides developer notes for the major packages of the druid-multi-stage-query
extension. It does not discuss future plans; these are discussed on the list or in GitHub issues.
Multi-stage queries are modeled as a directed acyclic graph (DAG) of stages. Each stage has some inputs (or, possibly, zero inputs, if the stage generates data out of thin air). Those inputs can be Druid tables, external data, or the outputs of other stages of the same query. There is one final stage that produces the query result. Stage outputs, whether they are inputs to other stages or results of the query itself, are optionally shuffled.
SQL-based ingestion jobs run as multi-stage query tasks. In this case, the result of the query is inserted into the target table.
Package org.apache.druid.msq.kernel
and org.apache.druid.msq.input
contain the model classes for multi-stage queries.
Main classes:
Package org.apache.druid.msq.indexing
contains code related to integrating with the indexing service. This allows multi-stage queries to run as indexing service tasks.
Main classes:
query_controller
task. Each query has one controller task. The controller task launches worker tasks to do the actual work.query_worker
task. These stick around for the lifetime of the query. Each task may do work for multiple stages. It has a specific worker number that is retained across all stages that the task may work on.Multi-stage queries, when run as SQL via query tasks, are planned in three phases:
query_controller
task generates a multi-stage QueryDefinition corresponding to the native query, using QueryKit.query_controller
task determines how many workers will run and generates WorkOrders for each worker.Once all three of these phases are complete, query_worker
tasks are launched, sent their WorkOrders, and query execution begins.
Packages org.apache.druid.msq.querykit
, org.apache.druid.msq.input
, and org.apache.druid.msq.kernel
contain code related to query planning.
Main classes:
Package org.apache.druid.msq.exec
and org.apache.druid.msq.kernel
contain code related to driving query execution.
Main classes:
Package org.apache.druid.msq.statistics
contains code related to determining partition boundaries as part of doing a range-based shuffle. During a stage that intends to do range-based shuffle, workers gather statistics using a ClusterByStatisticsCollector, which are then merged on the controller and used to generate partition boundaries.
Package org.apache.druid.msq.counters
contains code related to tracking and reporting query execution metrics.
Main classes:
Package org.apache.druid.msq.sql
contains code related to integration with Druid SQL APIs.
Main classes:
/druid/v2/sql/task
, where SQL queries are executed as multi-stage query tasks.