feat: add adaptive gzip compression for gRPC responses

Motivation:
The previous PR always used Identity (no compression) for native gRPC
responses to avoid CPU overhead on small messages. However, large
messages benefit significantly from gzip compression. An adaptive
approach that compresses only messages above a configurable threshold
provides the best of both worlds.

Modification:
- Add AdaptiveGzip codec that only compresses messages above a
  configurable threshold (default 1024 bytes). Messages below the
  threshold pass through uncompressed while the grpc-encoding header
  still advertises gzip.
- Add Codec.compressWithFlag() returning (ByteString, Boolean) to
  signal per-frame compression status accurately in the gRPC frame
  header (bit 0 of the 5-byte header). This fixes a correctness bug
  where streaming frames would incorrectly set the compression flag
  even for uncompressed data.
- Add pekko.grpc.server.compression-threshold config (default 1024).
  Generated Scala handlers read config once at handler creation and
  pass the threshold as Int to avoid per-request config overhead.
- Pre-compute NativeIdentityAdaptiveGzip negotiation result and cache
  the default AdaptiveGzip instance for zero per-request allocation
  on the common path.
- Add ProtobufFrameSerializer.serializedDataSize() for efficient
  size checking without double serialization.
- Add adaptive fast path in GrpcResponseHelpers for small messages
  using serializeDataFrame single-allocation path.

Result:
- Messages below threshold: zero compression CPU overhead, sent
  uncompressed with correct frame header flag.
- Messages above threshold: gzip compressed with correct frame header
  flag. Per gRPC spec, per-frame compression flag tells clients
  whether each frame is compressed.
- Default threshold of 1024 bytes aligns with industry consensus
  (Netty, Ktor, OneUptime gRPC guide) as the break-even point where
  gzip CPU cost is justified by bandwidth savings on protobuf data.
- JIT/GC optimized: zero per-request allocation on common path via
  pre-computed negotiation result and cached AdaptiveGzip singleton.

Tests:
- runtime / Test / test - 152 tests passed
- plugin-tester-scala / Test / testOnly ErrorReportingSpec - passed
- 30 new AdaptiveGzip tests covering codec behavior, boundary
  conditions, compressWithFlag, streaming frame header correctness,
  round-trip encode/decode, instance caching, and negotiate integration

References:
Refs #739
12 files changed
tree: 584b259989bc7827b0365ca26540204880932e95
  1. .github/
  2. benchmark-java/
  3. benchmarks/
  4. codegen/
  5. docs/
  6. gradle-plugin/
  7. interop-tests/
  8. legal/
  9. maven-plugin/
  10. plugin-tester-java/
  11. plugin-tester-scala/
  12. project/
  13. runtime/
  14. sbt-plugin/
  15. scalapb-protoc-plugin/
  16. scripts/
  17. .asf.yaml
  18. .git-blame-ignore-revs
  19. .gitattributes
  20. .gitignore
  21. .jvmopts-ci
  22. .scala-steward.conf
  23. .scalafmt.conf
  24. AGENTS.md
  25. build.sbt
  26. CHANGELOG.md
  27. CLAUDE.md
  28. CONTRIBUTING.md
  29. LICENSE
  30. NOTICE
  31. README.md
README.md

Apache Pekko gRPC

Support for building streaming gRPC servers and clients on top of Apache Pekko Streams.

This library is meant to be used as a building block in projects using the Pekko toolkit.

Documentation

Project Status

This library is ready to be used in production, but API's and build system plugins are still expected to be improved and may change.

The API on both sides (Client and Server) is a simple Pekko Streams-based one.

The client side is currently implemented on top of io.grpc:grpc-netty-shaded, we plan to replace this by just io.grpc:grpc-core and Pekko HTTP.

As for performance, we are currently relying on the JVM TLS implementation, which is sufficient for many use cases, but is planned to be replaced with conscrypt or netty-tcnative.

General overview

gRPC is a schema-first RPC framework, where your protocol is declared in a protobuf definition, and requests and responses will be streamed over an HTTP/2 connection.

Based on a protobuf service definition, pekko-grpc can generate:

  • Model classes (using plain protoc for Java or scalapb for Scala)
  • The API (as an interface for Java or a trait for Scala), expressed in Pekko Streams Sources
  • On the server side, code to create a Pekko HTTP route based on your implementation of the API
  • On the client side, a client for the API.

Project structure

The project is split up in a number of subprojects:

  • codegen: code generation shared among plugins
  • runtime: run-time utilities used by the generated code
  • sbt-plugin: the sbt plugin
  • scalapb-protoc-plugin: the scalapb Scala model code generation packaged as a protoc plugin, to be used from gradle
  • interop-tests

Additionally, ‘plugin-tester-java’ and ‘plugin-tester-scala’ contain an example project in Java and Scala respectively, with both sbt and Gradle configurations.

Building from Source

Prerequisites

  • Make sure you have installed a Java Development Kit (JDK) version 17 or later.
  • Make sure you have sbt installed.
  • Maven is needed for tasks related to building and testing Maven plugin support.
  • Gradle is needed for tasks related to building and testing Gradle plugin support. We have gradlew scripts that will install the right version of Gradle and run the gradle tasks using it.
  • Graphviz is needed for the scaladoc generation build task, which is part of the release.

Running the Build

  • Open a command window and change directory to your preferred base directory
  • Use git to clone the repo or download a source release from https://pekko.apache.org (and unzip or untar it, as appropriate)
  • Change directory to the directory where you installed the source (you should have a file called build.sbt in this directory)
  • sbt compile compiles the main source for project default version of Scala (2.13)
    • sbt +compile will compile for all supported versions of Scala
  • sbt test will compile the code and run the unit tests
  • sbt testQuick similar to test but when repeated in shell mode will only run failing tests
  • sbt package will build the jars
    • the jars will built into target dirs of the various modules
    • for the the ‘runtime’ module, the jar will be built to runtime/target/scala-2.13/
  • sbt publishLocal will push the jars to your local Apache Ivy repository
  • sbt publishM2 will push the jars to your local Apache Maven repository
  • sbt docs/paradox will build the docs (the ones describing the module features)
    • sbt docs/paradoxBrowse does the same but will open the docs in your browser when complete
    • the index.html file will appear in target/paradox/site/main/
  • sbt unidoc will build the Javadocs for all the modules and load them to one place (may require Graphviz, see Prerequisites above)
    • the index.html file will appear in target/scala-2.13/unidoc/
  • sbt sourceDistGenerate will generate source release to target/dist/
  • The version number that appears in filenames and docs is derived, by default. The derived version contains the most git commit id or the date/time (if the directory is not under git control).
    • You can set the version number explicitly when running sbt commands
      • eg sbt "set ThisBuild / version := \"1.0.0\"; sourceDistGenerate"
    • Or you can add a file called version.sbt to the same directory that has the build.sbt containing something like
      • ThisBuild / version := "1.0.0"

Maven plugin

  • the Maven plugin is built using sbt
  • Build the Maven plugin and note its version number because the tests need the version number
  • You can test the Maven plugin by changing directory into the plugin-tester-java dir
    • mvn -Dpekko.grpc.project.version=<version> pekko-grpc:generate compile
  • You can run the equivalent Scala tests by changing directory into the plugin-tester-scala dir
    • mvn -Dpekko.grpc.project.version=<version> pekko-grpc:generate scala:compile

Gradle plugin

  • the Gradle plugin is built using gradle
  • The gradle plugin will automatically derive the version of the artifact from sbt.
    • In other words sbt is the source of truth when it comes to deriving the version
  • You can test the Gradle plugin by change directory into the plugin-tester-java dir
    • ./gradlew clean test -Dpekko.grpc.project.version=<version>
  • You can run the equivalent Scala tests by changing directory into the plugin-tester-scala dir
    • ./gradlew clean test -Dpekko.grpc.project.version=<version>

License

Pekko gRPC is Open Source and available under the Apache 2 License.