Protobuf Extension

This extension provides support to ingest and understand the Protobuf data format. For more details, read the Protobuf extension docs.

Test Resources and Code Generation

The src/test/resources/ directory contains Protocol Buffer (.proto) files used in unit tests. For each .proto file, we make use of two generated files:

1. Java wrapper classes: Generated Java code for working with the protobuf messages, automatically generated during the build process.
2. Descriptor files (.desc): Binary representation of the protobuf schema with all dependencies, in source-control and manually generated using the process outlined below.

Automatic Code Generation

The project uses the io.github.ascopes.protobuf-maven-plugin to automatically generate Java wrapper classes from .proto files. This happens automatically when you run:

mvn generate-test-sources
# or any lifecycle phase that includes it, like:
mvn test

Descriptor File Generation

Unit tests may require manually generated descriptor files (.desc) which contain binary representations of the protobuf schema with all dependencies.

Prerequisites

• Docker installed and running
• The Dockerfile in src/test/resources/ creates a minimal Alpine Linux image with the correct protoc version

Build Docker Image

Important: Run all commands from the protobuf-extensions project root directory.

# Build the custom protoc image (only needed once or when Dockerfile changes)
docker build -t protoc-druid src/test/resources/

Generate Descriptor Files

docker run --rm -v $PWD:/workspace protoc-druid protoc \
  /workspace/src/test/resources/your_file.proto \
  --proto_path=/workspace/src/test/resources \
  --descriptor_set_out=/workspace/src/test/resources/your_file.desc \
  --include_imports