| --- |
| { |
| "title": "Trace Storage and Analysis", |
| "sidebar_label": "Trace", |
| "language": "en", |
| "description": "How do you store and analyze Trace data in Apache Doris? This article covers the full practice of table creation, OpenTelemetry collection integration, and Grafana queries.", |
| "keywords": [ |
| "Doris Trace", |
| "OpenTelemetry", |
| "distributed tracing", |
| "Trace storage", |
| "observability", |
| "Doris Exporter", |
| "OTLP", |
| "Grafana Trace" |
| ] |
| } |
| --- |
| |
| <!-- Knowledge type: Capability definition + Operation steps --> |
| <!-- Applicable scenario: Observability buildout / Distributed trace storage and analysis --> |
| |
| This article describes how to store and analyze **Trace data** in Apache Doris, covering the full path of table creation, collection, and query. |
| |
| - For the overall observability solution, see the [Overview](./overview.mdx). |
| - For resource estimation, cluster deployment, and tuning, see [Log](./log.md). |
| |
| ## Quick Navigation |
| |
| The full onboarding flow consists of three steps, which you can complete in order: |
| |
| | Step | Content | Goal | |
| | :--- | :--- | :--- | |
| | 1. [Create the table](#1-create-the-table) | Create a Trace storage table in Doris | Optimize performance for Trace write and query patterns | |
| | 2. [Collect](#2-trace-collection) | Write Trace data into Doris through OpenTelemetry | Connect the application to Collector to Doris pipeline | |
| | 3. [Query](#3-trace-query) | Visually analyze Trace data in Grafana | Search, view latency distribution, and inspect trace details | |
| |
| ## 1. Create the Table |
| |
| <!-- Knowledge type: Configuration parameters + Operation steps --> |
| |
| Trace data has clear characteristics in its write and query patterns. Targeted configuration during table creation yields better performance. |
| |
| ### 1.1 Key Configuration Items |
| |
| The following table summarizes the key configuration dimensions and recommended practices for table creation: |
| |
| | Configuration dimension | Recommended practice | Description | |
| | :--- | :--- | :--- | |
| | Partitioning | RANGE partition on the time field, with dynamic partitioning enabled to manage daily partitions automatically | Automatically rolls partitions, making hot/cold separation and expiration cleanup easier | |
| | Sort key | Use `service_name` and a `DATETIME`-typed time field as the key | Speeds up queries that scan a specific service over a time range by several times | |
| | Bucket count | Roughly 3 times the total number of disks in the cluster | Balances parallelism with control over small files | |
| | Bucketing strategy | Use `RANDOM` together with single tablet imports during writes | Improves write batching | |
| | Compaction | Use the `time_series` compaction policy | Reduces write amplification, which is critical for resource optimization under high-throughput Trace writes | |
| | Semi-structured fields | Use the VARIANT type for `span_attributes` and `resource_attributes` | Automatically splits JSON into sub-columns for storage, improving compression and sub-column filter and analysis performance | |
| | Indexes | Create inverted indexes on commonly queried fields | Speeds up equality filters and range queries | |
| | Full-text search | Specify the tokenizer through the `parser` parameter (typically `unicode` is sufficient), and enable `support_phrase` as needed | `support_phrase` enables phrase queries; turn it off when not needed to reduce storage space | |
| | Replicas | Cloud disks can use 1 replica; physical disks should use at least 2 replicas | Balances reliability with cost | |
| | Hot/cold separation | Configure the `log_s3` object storage and the `log_policy_3day` policy | Data older than 3 days is automatically moved to S3, reducing hot storage cost | |
| |
| ### 1.2 Table Creation SQL Example |
| |
| The following example covers the full creation process for the resource, storage policy, and table: |
| |
| ```sql |
| CREATE DATABASE log_db; |
| USE log_db; |
| |
| -- Not required in the storage-compute decoupled mode |
| CREATE RESOURCE "log_s3" |
| PROPERTIES |
| ( |
| "type" = "s3", |
| "s3.endpoint" = "your_endpoint_url", |
| "s3.region" = "your_region", |
| "s3.bucket" = "your_bucket", |
| "s3.root.path" = "your_path", |
| "s3.access_key" = "your_ak", |
| "s3.secret_key" = "your_sk" |
| ); |
| |
| -- Not required in the storage-compute decoupled mode |
| CREATE STORAGE POLICY log_policy_3day |
| PROPERTIES( |
| "storage_resource" = "log_s3", |
| "cooldown_ttl" = "259200" |
| ); |
| |
| CREATE TABLE trace_table |
| ( |
| service_name VARCHAR(200), |
| timestamp DATETIME(6), |
| service_instance_id VARCHAR(200), |
| trace_id VARCHAR(200), |
| span_id STRING, |
| trace_state STRING, |
| parent_span_id STRING, |
| span_name STRING, |
| span_kind STRING, |
| end_time DATETIME(6), |
| duration BIGINT, |
| span_attributes VARIANT, |
| events ARRAY<STRUCT<timestamp:DATETIME(6), name:STRING, attributes:MAP<STRING, STRING>>>, |
| links ARRAY<STRUCT<trace_id:STRING, span_id:STRING, trace_state:STRING, attributes:MAP<STRING, STRING>>>, |
| status_message STRING, |
| status_code STRING, |
| resource_attributes VARIANT, |
| scope_name STRING, |
| scope_version STRING, |
| INDEX idx_timestamp(timestamp) USING INVERTED, |
| INDEX idx_service_instance_id(service_instance_id) USING INVERTED, |
| INDEX idx_trace_id(trace_id) USING INVERTED, |
| INDEX idx_span_id(span_id) USING INVERTED, |
| INDEX idx_trace_state(trace_state) USING INVERTED, |
| INDEX idx_parent_span_id(parent_span_id) USING INVERTED, |
| INDEX idx_span_name(span_name) USING INVERTED, |
| INDEX idx_span_kind(span_kind) USING INVERTED, |
| INDEX idx_end_time(end_time) USING INVERTED, |
| INDEX idx_duration(duration) USING INVERTED, |
| INDEX idx_span_attributes(span_attributes) USING INVERTED, |
| INDEX idx_status_message(status_message) USING INVERTED, |
| INDEX idx_status_code(status_code) USING INVERTED, |
| INDEX idx_resource_attributes(resource_attributes) USING INVERTED, |
| INDEX idx_scope_name(scope_name) USING INVERTED, |
| INDEX idx_scope_version(scope_version) USING INVERTED |
| ) |
| ENGINE = OLAP |
| DUPLICATE KEY(service_name, timestamp) |
| PARTITION BY RANGE(timestamp) () |
| DISTRIBUTED BY RANDOM BUCKETS 250 |
| PROPERTIES ( |
| "compression" = "zstd", |
| "compaction_policy" = "time_series", |
| "inverted_index_storage_format" = "V2", |
| "dynamic_partition.enable" = "true", |
| "dynamic_partition.create_history_partition" = "true", |
| "dynamic_partition.time_unit" = "DAY", |
| "dynamic_partition.start" = "-30", |
| "dynamic_partition.end" = "1", |
| "dynamic_partition.prefix" = "p", |
| "dynamic_partition.buckets" = "250", |
| "dynamic_partition.replication_num" = "2", -- Not required in storage-compute decoupled mode |
| "replication_num" = "2", -- Not required in storage-compute decoupled mode |
| "storage_policy" = "log_policy_3day" -- Not required in storage-compute decoupled mode |
| ); |
| ``` |
| |
| ## 2. Trace Collection |
| |
| <!-- Knowledge type: Operation steps --> |
| <!-- Applicable scenario: Application onboarding / OpenTelemetry integration --> |
| |
| Doris provides an open and general Stream HTTP API that integrates with Trace collection systems such as OpenTelemetry. |
| |
| ### 2.1 Overall Pipeline |
| |
| Application to OpenTelemetry SDK/Agent to OpenTelemetry Collector (with Doris Exporter) to Doris table. |
| |
| ### 2.2 OpenTelemetry Integration Steps |
| |
| #### Step 1: Integrate the OpenTelemetry SDK on the Application Side |
| |
| This example uses the official Spring Boot [demo](https://docs.spring.io/spring-boot/tutorial/first-application/index.html) integrated with the OpenTelemetry Java SDK, returning a simple `Hello World!` string for the path `/`. |
| |
| Download the [OpenTelemetry Java Agent](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases). The advantage of using the Java Agent is that no modifications to the existing application are required. |
| |
| For other languages and integration methods, see: |
| |
| - [OpenTelemetry Language APIs & SDKs](https://opentelemetry.io/docs/languages/) |
| - [OpenTelemetry Zero-code Instrumentation](https://opentelemetry.io/docs/zero-code/) |
| |
| #### Step 2: Deploy and Configure the OpenTelemetry Collector |
| |
| Download the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector-releases/releases) and extract it. |
| |
| > Download the release package whose name is prefixed with `otelcol-contrib`. It includes the Doris Exporter component, which can import Trace data into Doris. |
| |
| Create the `otel_demo.yaml` configuration file as follows. For more configuration options, see the Doris Exporter [documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/dorisexporter): |
| |
| ```yaml |
| receivers: |
| otlp: # otlp protocol, receives data sent by the OpenTelemetry Java Agent |
| protocols: |
| grpc: |
| endpoint: 0.0.0.0:4317 |
| http: |
| endpoint: 0.0.0.0:4318 |
| |
| processors: |
| batch: |
| send_batch_size: 100000 # Number of records per batch; recommended batch size is between 100M and 1G |
| timeout: 10s |
| |
| exporters: |
| doris: |
| endpoint: http://localhost:8030 # FE HTTP address |
| database: doris_db_name |
| username: doris_username |
| password: doris_password |
| table: |
| traces: doris_table_name |
| create_schema: true # Whether to automatically create the schema; when set to false, you need to create the table manually |
| mysql_endpoint: localhost:9030 # FE MySQL address |
| history_days: 10 |
| create_history_days: 10 |
| timezone: Asia/Shanghai |
| timeout: 60s # http stream load client timeout |
| log_response: true |
| sending_queue: |
| enabled: true |
| num_consumers: 20 |
| queue_size: 1000 |
| retry_on_failure: |
| enabled: true |
| initial_interval: 5s |
| max_interval: 30s |
| headers: |
| load_to_single_tablet: "true" |
| ``` |
| |
| #### Step 3: Run the OpenTelemetry Collector |
| |
| ```bash |
| ./otelcol-contrib --config otel_demo.yaml |
| ``` |
| |
| #### Step 4: Start the Spring Boot Sample Application |
| |
| Before starting the application, only a few environment variables need to be set; no code changes are required: |
| |
| ```bash |
| export JAVA_TOOL_OPTIONS="${JAVA_TOOL_OPTIONS} -javaagent:/your/path/to/opentelemetry-javaagent.jar" # Path to the OpenTelemetry Java Agent |
| export OTEL_JAVAAGENT_LOGGING="none" # Disable otel logging to avoid interfering with the service's own logs |
| export OTEL_SERVICE_NAME="myproject" |
| export OTEL_TRACES_EXPORTER="otlp" # Use the otlp protocol to send trace data |
| export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" # Address of the OpenTelemetry Collector |
| |
| java -jar myproject-0.0.1-SNAPSHOT.jar |
| ``` |
| |
| #### Step 5: Access the Sample Application and Generate Trace Data |
| |
| Run `curl localhost:8080` to trigger the `hello` service call. The OpenTelemetry Java Agent automatically generates Trace data and sends it to the OpenTelemetry Collector. The Collector then writes the Trace data into a Doris table through the configured Doris Exporter (the default table name is `otel.otel_traces`). |
| |
| ## 3. Trace Query |
| |
| <!-- Knowledge type: Operation steps --> |
| <!-- Applicable scenario: Trace search / Troubleshooting --> |
| |
| Trace queries are typically performed through a visual interface such as Grafana. Common scenarios include: |
| |
| - Filter by time range and service name to display a Trace overview, including the latency distribution chart and the most recent traces. |
| |
|  |
| |
| - Click a link to view Trace details. |
| |
|  |
