This directory provides local infrastructure for developing and testing the Sourcelume Registry with PostgreSQL or Apache Atlas.
The default registry stores records directly in PostgreSQL and does not need Atlas. From the repository root, build with mvn clean install, then run docker compose up -d --build --wait.
For host or IDE development, from the repository root:
docker compose -f dev-support/docker-compose.postgres.yml up -d --wait mvn -pl sourcelume-registry-distribution-postgres quarkus:dev
The distribution uses the development database on port 55432 by default. See the root README for connection settings. This database is independent of the Atlas stack's atlas-db.
The remaining sections cover the Atlas distribution.
The local development environment consists of:
21000./q/health/{ready,live} on port 8082.sourcelume-network connecting the Atlas containers and the Sourcelume services.curl and jq (recommended for verifying JSON responses).The dev tooling is POSIX-shell based, but the full workflow runs on Windows in three ways:
Install Docker Desktop with the WSL2 backend and a WSL2 distro (wsl --install), then clone and run everything inside WSL2 exactly as the bash instructions below describe. Docker Desktop exposes the same engine to WSL2, so docker compose and the sourcelume-network bridge are shared with Windows. Two caveats:
~/...), not under /mnt/c — bind-mounts and Maven disk access are much faster on the ext4 filesystem.~/.m2 inside WSL2 is a different directory than your Windows Maven cache; the in-container Atlas build warms its own copy there.Git for Windows ships a POSIX shell; setup.sh, download-archives.sh, and docker compose all work from it unchanged — Docker Desktop's CLI and engine are already on the PATH. Use export VAR=value exactly as written in the bash snippets.
Use setup.ps1 instead of setup.sh:
.\dev-support\setup.ps1
Then note these equivalents:
export VAR=value becomes $env:VAR = "value" — e.g. $env:DOCKER_BUILDKIT = "1", $env:COMPOSE_DOCKER_CLI_BUILD = "1", $env:ATLAS_BACKEND = "postgres", and $env:SOURCELUME_ATLAS_URL = "http://localhost:21000".
mkdir -p "${HOME}/.m2" becomes New-Item -ItemType Directory -Force ~\.m2.
download-archives.sh still needs a POSIX shell — run it from Git Bash or WSL2 (bash download-archives.sh), or fetch the five archives listed in the vendored .env into downloads/ yourself. Only the Kafka archive is needed for the postgres backend.
curl in Windows PowerShell 5.1 aliases to Invoke-WebRequest; call curl.exe explicitly (ships with Windows 10+) or use Invoke-RestMethod for the same REST calls. jq is available via winget install jqlang.jq.
The Atlas server container is always named atlas (compose container_name), so attaching it to the shared network is simply:
docker network connect sourcelume-network atlas
Everything else — the docker compose files, mvn commands, and port numbers — is platform-independent.
Run setup.sh to create the shared Docker network and vendor Atlas's build tooling:
./dev-support/setup.sh
Atlas Kafka/Solr/ZooKeeper components require third-party archives before building:
cd dev-support/vendor/atlas-docker chmod +x download-archives.sh && ./download-archives.sh
export DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1 # 3a. Build the atlas-base image docker compose -f docker-compose.atlas-base.yml build # 3b. Build Atlas from source (takes ~15-45 mins on first run depending on ~/.m2 cache) mkdir -p "${HOME}/.m2" docker compose -f docker-compose.atlas-build.yml up
export ATLAS_BACKEND=postgres docker compose -f docker-compose.atlas.yml up -d --wait
Verify Atlas is running:
curl -u admin:atlasR0cks! http://localhost:21000/api/atlas/admin/version
sourcelume-networkAllow containerized Sourcelume services to reach Atlas by hostname (atlas). The Atlas server container is always named atlas (compose container_name):
docker network connect sourcelume-network atlas || true
There is no official prebuilt Atlas image — apache/atlas on Docker Hub publishes no tags, and the 2.5.0 release ships a source tarball only (no binary distribution). Building Atlas from source (docker-compose.atlas-build.yml, ~15–45 min the first time, faster with a warm ~/.m2) is therefore currently unavoidable if you want a real 2.5.0. The vendored stack then assembles these images:
| Image | Base | Why it exists |
|---|---|---|
atlas | atlas-base (Ubuntu + JDK) | The Atlas server, from the source-built dist tarball |
atlas-db | postgres:13 | Postgres backend + init script |
atlas-solr | solr:8 | Solr + Atlas configsets |
atlas-kafka | atlas-base + Kafka 2.8.2 | Broker with the Atlas Kafka hook preinstalled |
atlas-zk | zookeeper:3.9.2 | Stock image, re-tagged |
The dependency images are thin wrappers around stock images; the expensive parts are the Atlas source build itself and download-archives.sh (which fetches ~GBs of Hadoop/HBase/Hive/Kafka archives — only the Kafka archive is needed for the postgres backend).
Alternatives, in order of increasing trade-off:
docker save atlas:latest | gzip > atlas-2.5.0.tar.gz lets teammates docker load a real 2.5.0 instead of rebuilding.atlas_start.py can run embedded HBase+Solr+Kafka in one container (the old quickstart pattern). Fewer moving parts and still 2.5.0, but embedded-HBase semantics differ from the postgres stack we verify against.sburn/apache-atlas:2.3.0. The de facto community image — single container, fast start — but Atlas 2.3.0 (not 2.5.0), amd64-only, and unmaintained. Fine for a throwaway REST smoke test; not the supported configuration.# From the repo root # 1. Build project artifacts (also produces the Quarkus fast-jar for the image) mvn clean package # 2. Build and start the registry container docker compose -f docker-compose.atlas.yml build docker compose -f docker-compose.atlas.yml up -d # 3. View logs docker compose -f docker-compose.atlas.yml logs -f registry
# From the repo root, with Atlas reachable on localhost:21000 export SOURCELUME_ATLAS_URL=http://localhost:21000 export SOURCELUME_ATLAS_USER=admin export SOURCELUME_ATLAS_PASSWORD=atlasR0cks! # Run the Quarkus runtime in dev mode mvn -pl sourcelume-registry-distribution-atlas quarkus:dev
Or run/debug org.apache.sourcelume.registry.runtime.quarkus.SourcelumeRegistryApplication from your IDE using the Atlas distribution module's classpath.
The runtime exposes SmallRye Health probes at /q/health/{ready,live} on port 8082:
# Overall health (including the atlas readiness check) curl -s http://localhost:8082/q/health | jq . # Liveness and readiness probes (Kubernetes style) curl -s http://localhost:8082/q/health/live | jq . curl -s http://localhost:8082/q/health/ready | jq .
Expected readiness output when Atlas is reachable:
{ "status": "UP", "checks": [ { "name": "atlas", "status": "UP", "data": { "atlasUrl": "http://atlas:21000", "status": "CONNECTED" } } ] }
Prometheus metrics:
curl -s http://localhost:8082/q/metrics
On startup the runtime automatically registers the sourcelume-spec models in Atlas (retries via sourcelume.atlas.max-retries / retry-delay-ms).
# Check the registered type definitions curl -s -u admin:atlasR0cks! http://localhost:21000/api/atlas/v2/types/typedefs | jq . # Check the sourcelume_dataset entity definition curl -s -u admin:atlasR0cks! http://localhost:21000/api/atlas/v2/types/entitydef/name/sourcelume_dataset | jq .
# Create a test dataset entity directly in Atlas curl -u admin:atlasR0cks! -X POST http://localhost:21000/api/atlas/v2/entity/bulk \ -H "Content-Type: application/json" \ -d '{ "entities": [{ "typeName": "sourcelume_dataset", "attributes": { "name": "sample-dataset-1", "qualifiedName": "sourcelume://datasets/sample-1", "description": "First test Sourcelume dataset instance", "sourceUri": "https://github.com/example/repo", "licenseId": "Apache-2.0" } }] }' # Fetch the created entity by qualified name (the bulk endpoint only # resolves guids; the lookup goes through the unique-attribute endpoint) curl -s -u admin:atlasR0cks! \ "http://localhost:21000/api/atlas/v2/entity/uniqueAttribute/type/sourcelume_dataset?attr:qualifiedName=sourcelume://datasets/sample-1" | jq .
Sourcelume exposes high-level REST endpoints on port 8082. Records are written via POST /records; /api/v1/datasets offers read-only views:
# Publish a dataset record (the only way to write one) curl -s -X POST http://localhost:8082/records \ -H "Content-Type: application/ld+json" \ --data-binary @examples/simple-demo/sample-provenance-record.jsonld | jq . # Query by qualifiedName (the record's id; query parameter) curl -s "http://localhost:8082/api/v1/datasets?qualifiedName=https://sourcelume.apache.org/records/rec-glasgow-hackathon-001" | jq . # Fetch dataset by GUID curl -s "http://localhost:8082/api/v1/datasets/guid/<ASSIGNED_GUID>" | jq . # Inspect Sourcelume type definitions curl -s http://localhost:8082/api/v1/types/typedefs | jq . curl -s http://localhost:8082/api/v1/types/dataset | jq .
Run the full automated test suite (framework-free core tests plus the Quarkus wiring tests):
mvn clean verify
curl -u admin:atlasR0cks! -X DELETE http://localhost:21000/api/atlas/v2/types/typedefs \ -H "Content-Type: application/json" \ -d @sourcelume-registry-typedefs/src/main/resources/models/sourcelume/sourcelume_model.json
From the repository root:
docker compose -f docker-compose.atlas.yml down
cd dev-support/vendor/atlas-docker docker compose -f docker-compose.atlas.yml down
-Pintegration-tests)mvn verify -Pintegration-tests boots the vendored Atlas stack through Testcontainers and runs the end-to-end pipeline tests (*IT, see sourcelume-registry-distribution-atlas/src/.../it/). Prerequisites: the vendored tooling and the locally built atlas:latest image (the steps above), plus a reachable Docker daemon.
If your Docker daemon does not listen on the default socket path (/var/run/docker.sock — e.g. Rancher Desktop, colima), point the client at it via DOCKER_HOST. If the cleanup sidecar then cannot mount that socket, disable it with TESTCONTAINERS_RYUK_DISABLED=true (the compose stack is stopped explicitly by the test resource anyway):
export DOCKER_HOST=unix://<path-to-your-docker-socket> export TESTCONTAINERS_RYUK_DISABLED=true mvn verify -Pintegration-tests -pl sourcelume-registry-distribution-atlas
The API version pin for the docker client lives in sourcelume-registry-distribution-atlas/src/test/resources/docker-java.properties. If compose output is needed while debugging, add -Dquarkus.log.console.level=INFO.
To run the integration tests against an isolated Atlas instance that is already running, supply its URL. This bypasses the Compose lifecycle manager:
mvn verify -Pintegration-tests -Dsourcelume.it.atlas-url=http://localhost:21000
SOURCELUME_IT_ATLAS_USER and SOURCELUME_IT_ATLAS_PASSWORD override the local Atlas development defaults. The suite registers/updates Sourcelume type definitions and writes uniquely named test records; use a disposable instance. The ordinary HTTP tests explicitly select their mock profile, so the Atlas integration profile always uses the real storage provider.
perf/ seeds the issue-#24 workload (datasets plus model records sized 0/12/40/161 trainingData references) and times GET /records/{id} reference resolution through the public API, so the same run works against either distribution. See perf/README.md.