Doris reads data through the Thrift interface

Scenes

It is necessary to read a large amount of data, and spark/flink connectors use this method to read data.

process

1. Query the corresponding queryPlan according to SQL

Call api：/api///_query_plan Specifically, can see: https://doris.apache.org/zh-CN/docs/dev/admin-manual/http-actions/fe/table-query-plan-action This interface will return the tabletID hit by the corresponding SQL and the distributed backend information

2. Access backend and read tablet data directly

Interact with BE through Thrift interface, Thrift interface definition can be seen TDorisExternalService

2.1 doris will build a scan context for this session, context_id returned if success

TScanOpenResult open(1: TScanOpenParams params);

2.2 return the batch_size of data

TScanBatchResult getNext(1: TScanNextBatchParams params);

2.3 release the context resource associated with the context_id

TScanCloseResult close(1: TScanCloseParams params);

3. Convert Arrow data

Read data through Arrow and convert it to row data

For more design documents, please refer to: https://github.com/apache/doris/issues/1525