A JVM SDK for Apache Airflow. You can use any JVM-compatible language to write workflow bundles, and have Airflow consume the result.
The SDK and execution-time logic is implemented in Kotlin. An example is bundled showing how the SDK can be used in Java.
See also Airflow documentation on Java SDK under Authoring and Scheduling for more details on using the Java SDK.
The SDK requires Java 11 or later at runtime. Optional components and development tools may have further requirements (see the toolchain in buildSrc/src/main/kotlin/airflow-jvm-conventions.gradle.kts).
./gradlew build
./gradlew dokkaGenerate
This uses Dokka to build documentation of the Java SDK. This generates both an HTML representation and Javadoc.
The SDK projects must first built and published:
./gradlew publishToMavenLocal -PskipSigning=true
After the build is successful, you should be able to see directories in ~/.m2/repository/org/apache/airflow/.
Now cd example into the example project, and
Package the example to ./example/build/bundle
# We're now in the 'example' directory, so gradlew is in parent. ../gradlew bundle
Put the DAG with stub tasks to somewhere Airflow can find.
Ensure the java command is available in the same environment the Airflow task worker is in.
Configure Airflow to route tasks in the java queue to be run with Java:
export AIRFLOW__SDK__COORDINATORS='{ "java": { "classpath": "airflow.sdk.coordinators.java.JavaCoordinator", "kwargs": {"jars_root": ["/opt/airflow/java-sdk/example/build/bundle"]} } }' export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java"}'
Ensure the Connection and Variable needed by the example DAG are available:
export AIRFLOW_CONN_TEST_HTTP='{ "conn_type": "http", "login": "user", "password": "pass", "host": "example.com", "port": 1234, "extra": {"param1": "val1", "param2": "val2"} }' export AIRFLOW_VAR_MY_VARIABLE=123
The SDK is published to Maven Central via the ASF Nexus staging repository. The full release process follows the ASF Maven publishing guide.
Any versions published to Maven Central (instead of Snapshots) are considered releases, including alpha, beta, etc (see the ASF Release Policy). Every release therefore requires a PMC vote before it is published; only -SNAPSHOT builds may be published without a vote.
Edit gradle.properties and set the version for this release:
projectVersion=<VERSION>
Commit the change and push it to the release branch.
Use a Maven-compatible version string, as defined by the Maven version order specification. For example, version 1 beta 1 is 1.0.0-beta1, and the eventual version 1 release is 1.0.0.
NOTE: Editing gradle.properties as above is the standard procedure. You can alternatively override the version for a single command without editing the file, by passing -PprojectVersion=<VERSION> to any Gradle invocation. This is handy for one-off or pre-release builds. Either way, main should stay on a -SNAPSHOT version between releases: a snapshot sorts after <VERSION> and before the GA, so no extra bump is needed after a beta.
Tag the release commit, keeping the RC number in the tag name so a failed vote simply bumps to the next RC (the artifact version itself does not carry the RC suffix). Push the tag before sending the vote so reviewers can check out the exact source being voted on.
git tag -s java-sdk/<VERSION>-rc<N> -m "Java SDK <VERSION> RC <N>" git push upstream java-sdk/<VERSION>-rc<N>
Before touching any remote repository, publish to your local Maven cache and inspect the generated POM:
rm -rf ~/.m2/repository/org/apache/airflow/ # Start clean. ./gradlew publishToMavenLocal -PskipSigning=true # The airflow-sdk runtime. less ~/.m2/repository/org/apache/airflow/airflow-sdk/*/airflow-sdk-*.pom # The bill of materials of airflow-sdk less ~/.m2/repository/org/apache/airflow/airflow-sdk-bom/*/*.pom # The annotation processor for the builder pattern. less ~/.m2/repository/org/apache/airflow/airflow-sdk-processor/*/airflow-sdk-*.pom # The Gradle plugin for bundling. less ~/.m2/repository/org/apache/airflow/airflow-sdk-gradle-plugin/*/airflow-sdk-*.pom # The Gradle plugin's registration. less ~/.m2/repository/org/apache/airflow/sdk/org.apache.airflow.sdk.gradle.plugin/*/*.pom
Check that the coordinates, description, license, SCM, and organization fields look correct.
To test the full publish flow without touching ASF infrastructure, override the repository URL to a local directory
rm -rf /tmp/local-maven-repo # Start clean. ./gradlew publish -PmavenUrl=file:///tmp/local-maven-repo -PskipSigning=true ls /tmp/local-maven-repo/org/apache/airflow/ # This should contain the same components in ~/.m2 as inspected in the previous step.
NOTE: Signing is not required since nothing goes to Maven Central. If you want to test signing, set the GPG private key and passphrase as described in the next section, and remove -PskipSigning=true from the above command.
Store the credentials in ~/.gradle/gradle.properties so they are not exposed in your shell history:
mavenUsername=your-asf-nexus-token-username mavenPassword=your-asf-nexus-token-password signing.password=your-gpg-key-passphrase
Then stage and close the release. The Gradle Nexus Publish Plugin (applied in the root build.gradle.kts) gathers every module into one staging repository, so there is no per-module fan-out to reconcile.
./gradlew publishToApache closeApacheStagingRepository \ --no-configuration-cache \ -P"signing.key=$(gpg --armor --export-secret-keys your-gpg-key-fingerprint)"
NOTE: The signing key is supplied through the command line since it contains newlines, which does not work well in a Gradle properties file.
NOTE: You can also use the following environment variables to provide the credentials instead: ASF_NEXUS_USERNAME, ASF_NEXUS_PASSWORD, SIGNING_KEY, and SIGNING_PASSWORD. This is especially useful on e.g. CI.
NOTE: We enable Gradle's configuration cache globally, but the staging tasks (publishToApache, closeApacheStagingRepository, releaseApacheStagingRepository) talk to the Nexus REST API and are not configuration-cache compatible. Hence the --no-configuration-cache flag on the release commands.
Under Staging Repositories on the ASF Nexus server, open the closed repository and verify it contains all modules, each with its jar, -sources.jar, -javadoc.jar (where applicable), .pom, and .asc signature.
Check Updated by (should be your ID), Uploaded Date, and Last Modified.
The closed staging repository from the previous step is the convenience-binary URL you link in the vote.
The signed source package is the artifact the vote is formally on; the Maven artifacts are convenience binaries. The sourceRelease task builds it from the committed java-sdk sources (LICENSE and NOTICE included) and produces its signature and checksum in one step:
# Signing uses your local gpg keyring, so have your key/passphrase ready. ./gradlew sourceRelease -PgitRef=java-sdk/<VERSION>-rc<N>
This writes three files to build/distributions/:
apache-airflow-java-sdk-<VERSION>-src.tar.gz apache-airflow-java-sdk-<VERSION>-src.tar.gz.asc apache-airflow-java-sdk-<VERSION>-src.tar.gz.sha512
NOTE: The source archive omits the Gradle wrapper scripts (gradlew, gradlew.bat) and gradle/wrapper/gradle-wrapper.jar since ASF source releases must not contain compiled code (see LEGAL-570), and the scripts are not useful without the jar. The gradle/wrapper/gradle-wrapper.properties file is kept so the archive pins the Gradle version and distribution checksum for verification when the wrapper is regenerated.
Copy the three files into your checkout of the ASF dist dev repo and commit them with Subversion. If you don‘t already have the repo checked out, replace <dist-dev-checkout> with wherever you want it and <path-to> with this project’s location:
# One-time: check out the Airflow dist dev area. svn checkout https://dist.apache.org/repos/dist/dev/airflow <dist-dev-checkout> cd <dist-dev-checkout> mkdir -p java-sdk/<VERSION>-rc<N> cp <path-to>/java-sdk/build/distributions/apache-airflow-java-sdk-<VERSION>-src.tar.gz* \ java-sdk/<VERSION>-rc<N>/ svn add --parents java-sdk/<VERSION>-rc<N> svn commit -m "Add Apache Airflow Java SDK <VERSION>-rc<N> source release candidate"
The commit publishes them under https://dist.apache.org/repos/dist/dev/airflow/java-sdk/<VERSION>-rc<N>/, which is the source-package URL you link in the vote.
Send a [VOTE] email to dev@airflow.apache.org linking the git tag and commit, the source package in dist/dev, the closed Nexus staging repository, and the KEYS file. See “Vote email template” below for the exact fields to fill in, and “Verifying a release” for what to ask reviewers to check.
Subject: [VOTE] Release Apache Airflow Java SDK <VERSION> based on <VERSION>-rc<N> Hi, I would like to call a vote to release Apache Airflow Java SDK <VERSION>, based on release candidate <VERSION>-rc<N>. Changes since <the last release>: <a one-line description> The release candidate contains the following Maven artifacts, all under group id org.apache.airflow: - Main SDK API - airflow-sdk - airflow-sdk-processor - Logger helpers - airflow-sdk-jpl - airflow-sdk-jul - airflow-sdk-log4j2 - airflow-sdk-slf4j - Gradle plugin and the marker artifact - airflow-sdk-gradle-plugin - org.apache.airflow.sdk.gradle.plugin - BOM - airflow-sdk-bom Git information: - Tag: java-sdk/<VERSION>-rc<N> - Commit: <full-commit-sha> - https://github.com/apache/airflow/releases/tag/java-sdk%2F<VERSION>-rc<N> Source release, signatures and checksums: https://dist.apache.org/repos/dist/dev/airflow/java-sdk/<VERSION>-rc<N>/ Convenience binaries (staged in the ASF Nexus repository): https://repository.apache.org/content/repositories/orgapacheairflow-<NNNN>/ KEYS file (public keys used to sign the release): https://downloads.apache.org/airflow/KEYS Please review and vote. The vote will remain open for at least 72 hours, until <YYYY-MM-DD HH:MM UTC>, or until the necessary number of binding votes is reached. [ ] +1 Release this package as Apache Airflow Java SDK <VERSION> [ ] +0 No opinion [ ] -1 Do not release, because ... Only votes from Airflow PMC members are binding, but everyone is welcome and encouraged to test the release and vote. The verification process can be found in the main repository: https://github.com/apache/airflow/tree/java-sdk/<VERSION>-rc<N>/java-sdk#verifying-a-release For more details on ASF release verification, see: https://www.apache.org/info/verification.html Best, <your name>
Pre-send checklist:
Changes since rc<N-1> and the vote deadline are filled in.grep '<' email.txt on the rendered email and confirm no output — any match means a template placeholder (<N>, <NNNN>, <YYYY-MM-DD ...>, etc.) was left unfilled.Anyone on dev@airflow.apache.org can (and should) independently verify a candidate before voting. Below is the checklist a reviewer — or the release manager, before sending the vote — should run against the source package in dist/dev.
Checksum. Confirm the published SHA-512 matches the downloaded tarball:
sha512sum -c apache-airflow-java-sdk-<VERSION>-src.tar.gz.sha512
Signature. Import the KEYS file and verify the GPG signature:
curl -O https://downloads.apache.org/airflow/KEYS gpg --import KEYS gpg --verify apache-airflow-java-sdk-<VERSION>-src.tar.gz.asc \ apache-airflow-java-sdk-<VERSION>-src.tar.gz
Diff against the git tag. Extract the tarball and compare it with a clean checkout of the tag it claims to be built from. They should be identical except for the files kept out of the source release via .gitattributes export-ignore. The extracted top-level directory should be apache-airflow-java-sdk-<version> without the -src suffix that only appears in the tarball's own filename:
tar xzf apache-airflow-java-sdk-<VERSION>-src.tar.gz git clone --branch java-sdk/<VERSION>-rc<N> \ https://github.com/apache/airflow.git tag-checkout diff -rq apache-airflow-java-sdk-<VERSION>/ tag-checkout/java-sdk/ \ | grep -vE ': (gradlew|gradlew\.bat|gradle-wrapper\.jar|scripts)$'
Any remaining diff output is unexpected and should block the vote.
No binary files. ASF source releases must not contain compiled code. Scan for anything that isn't text:
find apache-airflow-java-sdk-<VERSION>/ -type f \ -exec sh -c 'file -b "$1" | grep -qviE "text|json|xml|empty" && echo "$1"' _ {} \;
This should print nothing.
Build from source. Regenerate the Gradle wrapper from a locally installed Gradle (see the Upload the source package section above):
cd apache-airflow-java-sdk-<VERSION> gradle wrapper \ --gradle-distribution-url <GRADLE-DISTRIBUTION-URL> \ --gradle-distribution-sha256-sum <GRADLE-DISTRIBUTION-SHA> ./gradlew build
Use values from distributionUrl and distributionSha256Sum in the bundled gradle/wrapper/gradle-wrapper.properties to fill in <GRADLE-DISTRIBUTION-URL> and <GRADLE-DISTRIBUTION-SHA>.
Staged-binary smoke test. Resolve the staged Nexus artifacts from a throwaway project to confirm they're actually consumable, following the same pattern as the “Dry-run against a local repository” step: point a repositories {} block at the staging repository URL, declare a dependency on org.apache.airflow:airflow-sdk-bom:<VERSION>, and confirm the transitive artifacts (including airflow-sdk-jpl) resolve and the example bundle builds against them.
Reply with a [RESULT][VOTE] tally, then:
Release the staging repository so the artifacts sync to Maven Central (a few hours). Nothing is rebuilt or re-signed:
./gradlew releaseApacheStagingRepository --no-configuration-cache
(Or click Release on the repository in the Nexus UI.)
Move the source package from dist/dev to dist/release:
svn mv https://dist.apache.org/repos/dist/dev/airflow/java-sdk/<VERSION>-rc<N> \ https://dist.apache.org/repos/dist/release/airflow/java-sdk/<VERSION> \ -m "Release Apache Airflow Java SDK <VERSION>"
Tag the final version on the same commit that was voted:
git tag -s java-sdk/<VERSION> <voted-commit-hash> -m "Apache Airflow Java SDK <VERSION>" git push upstream java-sdk/<VERSION>
Keep the RC tag for traceability.
Publish a GitHub release. Attach the voted, signed source artifacts. From the directory holding the three signed files (e.g. dist/release):
gh release create java-sdk/<VERSION> \ --repo apache/airflow \ --title "Apache Airflow Java SDK <VERSION>" \ --notes "See the Java SDK README for the features in this release." \ --verify-tag \ --prerelease \ apache-airflow-java-sdk-<VERSION>-src.tar.gz \ apache-airflow-java-sdk-<VERSION>-src.tar.gz.asc \ apache-airflow-java-sdk-<VERSION>-src.tar.gz.sha512
Drop the --prerelease flag if this is not a prerelease.
Announce. Wait ~1 hour after promoting so Maven Central has synced, then send a plain-text [ANNOUNCE] email to users@airflow.apache.org (cc dev@airflow.apache.org) using the “Announce email template” below, and record the release (version + date) in the ASF Committee Report Helper: https://reporter.apache.org/addrelease.html?airflow.
Publish the API docs. Trigger the Publish Docs to S3 workflow in apache/airflow for the release tag:
gh workflow run "Publish Docs to S3" --repo apache/airflow --ref main \ -f ref=java-sdk/<VERSION> \ -f include-docs=java-sdk \ -f destination=live
Optionally use destination=staging first to check, then live. It may be possible to auto instead, which resolves to live for a release tag.
Confirm that https://airflow.apache.org/docs/java-sdk/stable/ resolves (allow time for cache invalidation) and /docs/java-sdk/ redirects to it.
Send to users@airflow.apache.org, cc dev@airflow.apache.org.
Subject: [ANNOUNCE] Apache Airflow Java SDK <VERSION> Released Dear Airflow community, I'm happy to announce that Apache Airflow Java SDK <VERSION> was just released. The signed source release can be downloaded from: https://downloads.apache.org/airflow/java-sdk/<VERSION>/ We also published the convenience binaries to Maven Central, under group id org.apache.airflow. Import airflow-sdk-bom:<VERSION> to keep the module versions aligned. The API documentation is available at: https://airflow.apache.org/docs/java-sdk/<VERSION>/ <Optional: one or two lines on notable changes; omit for the first release.> Thanks to everyone who contributed to and tested this release. Cheers, <your name>
Pre-send checklist:
grep '<' email.txt on the rendered email and confirm no output -- any match means a placeholder (<VERSION>, <your name>, ...) was left unfilled.Close the vote, drop the staging repository in Nexus, remove the dist/dev candidate, fix the issue, and cut the next RC (...-rc2). The released version stays the same (e.g. <VERSION>); only the RC counter in the tag increments.
The user implements a Java application containing task methods annotated (or registered) with the SDK. The application is packaged as a bundle and placed where Airflow can find it.
When the Airflow supervisor identifies that a task should run with Java, it launches the JVM application as a subprocess. The flow is:
JavaCoordinator.execute_task() (Python) scans jars_root, builds the classpath, and spawns java -cp <jars> <MainClass> --comm=<host>:<port> --logs=<host>:<port>.Server.kt connects to both sockets immediately on startup.StartupDetails MessagePack message; the JVM reads it, looks up the matching task by dag_id + task_id, and calls the user's task method.TaskState message and closes the socket. The JVM process then exits.Log messages produced by the SDK (not by user code) are forwarded over the --logs socket so the supervisor can append them to Airflow's log store.
The wire protocol is defined in task-sdk/src/airflow/sdk/execution_time/schema/schema.json. execution/Comm.kt implements the framing layer. Adding a new message type requires changes in both schema.json (Python side) and execution/Comm.kt + execution/Client.kt (JVM side).
See Architectural Design Records in the adr directory to learn more.
java-sdk/ ├── sdk/ # Core library: public API (org.apache.airflow.sdk) and internal │ # execution layer (org.apache.airflow.sdk.execution) ├── processor/ # Annotation processor that generates *Builder classes ├── plugin/ # Gradle plugin (org.apache.airflow.sdk) — bundle task, manifest │ # attribute injection, and verifyBundleMainClass ├── bom/ # Bill of Materials POM so consumers can import all SDK artifacts │ # at a consistent version ├── slf4j/ # SLF4J logging provider; routes SLF4J calls to the Airflow log store ├── jul/ # java.util.logging handler; routes JUL records to the Airflow log store ├── jpl/ # Java Platform Logging provider (System.Logger, JEP 264); routes JPL │ # calls to the Airflow log store ├── log4j2/ # Log4j 2 appender; routes Log4j 2 events to the Airflow log store ├── example/ # End-to-end example bundle (annotation + interface APIs, Java source) ├── adr/ # Architectural Decision Records for the Java SDK └── buildSrc/ # Shared Gradle convention plugins (Java version, lint, etc.)
The Python coordinator that launches the JVM subprocess lives outside this directory:
task-sdk/src/airflow/sdk/coordinators/java/ # JavaCoordinator (SubprocessCoordinator subclass) task-sdk/tests/coordinators/java/ # Python-side unit and integration tests
# Run all JVM tests ./gradlew test # Run a specific test class ./gradlew :sdk:test --tests "org.apache.airflow.sdk.execution.CommTest"
For the Python coordinator, use Breeze (never run pytest on the host directly):
breeze testing task-sdk-tests -- task_sdk/coordinators/java
End-to-end tests that exercise a real Airflow environment:
E2E_TEST_MODE=java_sdk uv run --project airflow-e2e-tests pytest \ tests/airflow_e2e_tests/java_sdk_tests/ -xvs
sdk/src/main/kotlin/ (the public API surface) free of internal implementation details; those belong in the execution/ sub-package.BuilderProcessor.kt) uses kapt. When adding a new annotation, define it in Builder.kt, handle it in BuilderProcessor.kt, and add a golden-output test in processor/src/test/kotlin/.SubprocessCoordinator. Do not reach into the JVM process from Python beyond what _build_execute_task_command provides../gradlew ktLintCheck spotlessCheck (or ktLintFormat spotlessApply) before submitting — the project enforces Kotlin and Java formatting.Adding a new Client method (e.g. a new Airflow API call):
schema.json if the message type is new.execution/Comm.kt or a new file.Client.kt which delegates to execution/Client.kt for the supervisor wire call.sdk/src/test/kotlin/.../ClientTest.kt mocking the socket layer.airflow-core/docs/authoring-and-scheduling/language-sdks/java.rst if the change is user-visible.Adding a new annotation:
Builder.kt.BuilderProcessor.kt — generate the appropriate code in the *Builder class.BuilderTest.kt with expected generated output.java.rst.Fixing a framing or protocol bug: focus on execution/Comm.kt and execution/Frame.kt. CommTest.kt covers encode/decode round-trips; add a regression test reproducing the bug before fixing it.
./gradlew build test (JVM) and the relevant pytest suite (Python coordinator).schema.json changed, verify both sides (JVM + Python) handle the new/changed fields.task-sdk/, add a newsfragment under airflow-core/newsfragments/.