Sourcelume Dev Support: PostgreSQL & Apache Atlas

This directory provides local infrastructure for developing and testing the Sourcelume Registry with PostgreSQL or Apache Atlas.


PostgreSQL default distribution

The default registry stores records directly in PostgreSQL and does not need Atlas. From the repository root, build with mvn clean install, then run docker compose up -d --build --wait.

For host or IDE development, from the repository root:

docker compose -f dev-support/docker-compose.postgres.yml up -d --wait
mvn -pl sourcelume-registry-distribution-postgres quarkus:dev

The distribution uses the development database on port 55432 by default. See the root README for connection settings. This database is independent of the Atlas stack's atlas-db.

The remaining sections cover the Atlas distribution.

Overview

The local development environment consists of:

  1. Apache Atlas backend: an Apache Atlas 2.5.0 container stack with the PostgreSQL backend, listening on port 21000.
  2. Sourcelume Registry runtime: the Quarkus application that connects to Atlas, registers the Sourcelume typedefs on startup, and exposes SmallRye Health readiness/liveness probes at /q/health/{ready,live} on port 8082.
  3. Shared network: a Docker bridge network named sourcelume-network connecting the Atlas containers and the Sourcelume services.

Prerequisites

  • Docker Engine ≥ 20.10.5 and Docker Compose ≥ 1.28.5 (configured with at least 6 GB RAM).
  • Java 21 (e.g. Eclipse Temurin 21).
  • Maven 3.9+.
  • curl and jq (recommended for verifying JSON responses).

Running on Windows

The dev tooling is POSIX-shell based, but the full workflow runs on Windows in three ways:

WSL2 (recommended)

Install Docker Desktop with the WSL2 backend and a WSL2 distro (wsl --install), then clone and run everything inside WSL2 exactly as the bash instructions below describe. Docker Desktop exposes the same engine to WSL2, so docker compose and the sourcelume-network bridge are shared with Windows. Two caveats:

  • Clone the repo inside the WSL2 filesystem (~/...), not under /mnt/c — bind-mounts and Maven disk access are much faster on the ext4 filesystem.
  • ~/.m2 inside WSL2 is a different directory than your Windows Maven cache; the in-container Atlas build warms its own copy there.

Git Bash

Git for Windows ships a POSIX shell; setup.sh, download-archives.sh, and docker compose all work from it unchanged — Docker Desktop's CLI and engine are already on the PATH. Use export VAR=value exactly as written in the bash snippets.

Native PowerShell

Use setup.ps1 instead of setup.sh:

.\dev-support\setup.ps1

Then note these equivalents:

  • export VAR=value becomes $env:VAR = "value" — e.g. $env:DOCKER_BUILDKIT = "1", $env:COMPOSE_DOCKER_CLI_BUILD = "1", $env:ATLAS_BACKEND = "postgres", and $env:SOURCELUME_ATLAS_URL = "http://localhost:21000".

  • mkdir -p "${HOME}/.m2" becomes New-Item -ItemType Directory -Force ~\.m2.

  • download-archives.sh still needs a POSIX shell — run it from Git Bash or WSL2 (bash download-archives.sh), or fetch the five archives listed in the vendored .env into downloads/ yourself. Only the Kafka archive is needed for the postgres backend.

  • curl in Windows PowerShell 5.1 aliases to Invoke-WebRequest; call curl.exe explicitly (ships with Windows 10+) or use Invoke-RestMethod for the same REST calls. jq is available via winget install jqlang.jq.

  • The Atlas server container is always named atlas (compose container_name), so attaching it to the shared network is simply:

    docker network connect sourcelume-network atlas
    

Everything else — the docker compose files, mvn commands, and port numbers — is platform-independent.


Quick Start: Standing up the Atlas backend

1. Initialize dev support

Run setup.sh to create the shared Docker network and vendor Atlas's build tooling:

./dev-support/setup.sh

2. Download Atlas dependency archives

Atlas Kafka/Solr/ZooKeeper components require third-party archives before building:

cd dev-support/vendor/atlas-docker
chmod +x download-archives.sh && ./download-archives.sh

3. Build Atlas containers

export DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1

# 3a. Build the atlas-base image
docker compose -f docker-compose.atlas-base.yml build

# 3b. Build Atlas from source (takes ~15-45 mins on first run depending on ~/.m2 cache)
mkdir -p "${HOME}/.m2"
docker compose -f docker-compose.atlas-build.yml up

4. Start Atlas with the PostgreSQL backend

export ATLAS_BACKEND=postgres
docker compose -f docker-compose.atlas.yml up -d --wait

Verify Atlas is running:

curl -u admin:atlasR0cks! http://localhost:21000/api/atlas/admin/version

5. Connect Atlas to sourcelume-network

Allow containerized Sourcelume services to reach Atlas by hostname (atlas). The Atlas server container is always named atlas (compose container_name):

docker network connect sourcelume-network atlas || true

Atlas images: options & trade-offs

There is no official prebuilt Atlas image — apache/atlas on Docker Hub publishes no tags, and the 2.5.0 release ships a source tarball only (no binary distribution). Building Atlas from source (docker-compose.atlas-build.yml, ~15–45 min the first time, faster with a warm ~/.m2) is therefore currently unavoidable if you want a real 2.5.0. The vendored stack then assembles these images:

ImageBaseWhy it exists
atlasatlas-base (Ubuntu + JDK)The Atlas server, from the source-built dist tarball
atlas-dbpostgres:13Postgres backend + init script
atlas-solrsolr:8Solr + Atlas configsets
atlas-kafkaatlas-base + Kafka 2.8.2Broker with the Atlas Kafka hook preinstalled
atlas-zkzookeeper:3.9.2Stock image, re-tagged

The dependency images are thin wrappers around stock images; the expensive parts are the Atlas source build itself and download-archives.sh (which fetches ~GBs of Hadoop/HBase/Hive/Kafka archives — only the Kafka archive is needed for the postgres backend).

Alternatives, in order of increasing trade-off:

  • Build once, share the image. docker save atlas:latest | gzip > atlas-2.5.0.tar.gz lets teammates docker load a real 2.5.0 instead of rebuilding.
  • Single-container embedded Atlas. The distro's atlas_start.py can run embedded HBase+Solr+Kafka in one container (the old quickstart pattern). Fewer moving parts and still 2.5.0, but embedded-HBase semantics differ from the postgres stack we verify against.
  • sburn/apache-atlas:2.3.0. The de facto community image — single container, fast start — but Atlas 2.3.0 (not 2.5.0), amd64-only, and unmaintained. Fine for a throwaway REST smoke test; not the supported configuration.

Running & testing the Quarkus runtime

Option A: Docker Compose (container validation)

# From the repo root
# 1. Build project artifacts (also produces the Quarkus fast-jar for the image)
mvn clean package

# 2. Build and start the registry container
docker compose -f docker-compose.atlas.yml build
docker compose -f docker-compose.atlas.yml up -d

# 3. View logs
docker compose -f docker-compose.atlas.yml logs -f registry

Option B: Host / IDE (rapid development & debugging)

# From the repo root, with Atlas reachable on localhost:21000
export SOURCELUME_ATLAS_URL=http://localhost:21000
export SOURCELUME_ATLAS_USER=admin
export SOURCELUME_ATLAS_PASSWORD=atlasR0cks!

# Run the Quarkus runtime in dev mode
mvn -pl sourcelume-registry-distribution-atlas quarkus:dev

Or run/debug org.apache.sourcelume.registry.runtime.quarkus.SourcelumeRegistryApplication from your IDE using the Atlas distribution module's classpath.


Testing & verification workflows

1. SmallRye Health probes

The runtime exposes SmallRye Health probes at /q/health/{ready,live} on port 8082:

# Overall health (including the atlas readiness check)
curl -s http://localhost:8082/q/health | jq .

# Liveness and readiness probes (Kubernetes style)
curl -s http://localhost:8082/q/health/live | jq .
curl -s http://localhost:8082/q/health/ready | jq .

Expected readiness output when Atlas is reachable:

{
  "status": "UP",
  "checks": [
    {
      "name": "atlas",
      "status": "UP",
      "data": {
        "atlasUrl": "http://atlas:21000",
        "status": "CONNECTED"
      }
    }
  ]
}

Prometheus metrics:

curl -s http://localhost:8082/q/metrics

2. Typedef bootstrap verification

On startup the runtime automatically registers the sourcelume-spec models in Atlas (retries via sourcelume.atlas.max-retries / retry-delay-ms).

# Check the registered type definitions
curl -s -u admin:atlasR0cks! http://localhost:21000/api/atlas/v2/types/typedefs | jq .

# Check the sourcelume_dataset entity definition
curl -s -u admin:atlasR0cks! http://localhost:21000/api/atlas/v2/types/entitydef/name/sourcelume_dataset | jq .

3. Testing entity creation & querying in Atlas

# Create a test dataset entity directly in Atlas
curl -u admin:atlasR0cks! -X POST http://localhost:21000/api/atlas/v2/entity/bulk \
  -H "Content-Type: application/json" \
  -d '{
    "entities": [{
      "typeName": "sourcelume_dataset",
      "attributes": {
        "name": "sample-dataset-1",
        "qualifiedName": "sourcelume://datasets/sample-1",
        "description": "First test Sourcelume dataset instance",
        "sourceUri": "https://github.com/example/repo",
        "licenseId": "Apache-2.0"
      }
    }]
  }'

# Fetch the created entity by qualified name (the bulk endpoint only
# resolves guids; the lookup goes through the unique-attribute endpoint)
curl -s -u admin:atlasR0cks! \
  "http://localhost:21000/api/atlas/v2/entity/uniqueAttribute/type/sourcelume_dataset?attr:qualifiedName=sourcelume://datasets/sample-1" | jq .

4. Testing dataset operations directly in Sourcelume Registry

Sourcelume exposes high-level REST endpoints on port 8082. Records are written via POST /records; /api/v1/datasets offers read-only views:

# Publish a dataset record (the only way to write one)
curl -s -X POST http://localhost:8082/records \
  -H "Content-Type: application/ld+json" \
  --data-binary @examples/simple-demo/sample-provenance-record.jsonld | jq .

# Query by qualifiedName (the record's id; query parameter)
curl -s "http://localhost:8082/api/v1/datasets?qualifiedName=https://sourcelume.apache.org/records/rec-glasgow-hackathon-001" | jq .

# Fetch dataset by GUID
curl -s "http://localhost:8082/api/v1/datasets/guid/<ASSIGNED_GUID>" | jq .

# Inspect Sourcelume type definitions
curl -s http://localhost:8082/api/v1/types/typedefs | jq .
curl -s http://localhost:8082/api/v1/types/dataset | jq .

5. Automated tests

Run the full automated test suite (framework-free core tests plus the Quarkus wiring tests):

mvn clean verify

Teardown & reset

Clean up test typedefs from Atlas

curl -u admin:atlasR0cks! -X DELETE http://localhost:21000/api/atlas/v2/types/typedefs \
  -H "Content-Type: application/json" \
  -d @sourcelume-registry-typedefs/src/main/resources/models/sourcelume/sourcelume_model.json

Stop the Sourcelume container

From the repository root:

docker compose -f docker-compose.atlas.yml down

Stop the Apache Atlas stack

cd dev-support/vendor/atlas-docker
docker compose -f docker-compose.atlas.yml down

Running the integration tests (-Pintegration-tests)

mvn verify -Pintegration-tests boots the vendored Atlas stack through Testcontainers and runs the end-to-end pipeline tests (*IT, see sourcelume-registry-distribution-atlas/src/.../it/). Prerequisites: the vendored tooling and the locally built atlas:latest image (the steps above), plus a reachable Docker daemon.

If your Docker daemon does not listen on the default socket path (/var/run/docker.sock — e.g. Rancher Desktop, colima), point the client at it via DOCKER_HOST. If the cleanup sidecar then cannot mount that socket, disable it with TESTCONTAINERS_RYUK_DISABLED=true (the compose stack is stopped explicitly by the test resource anyway):

export DOCKER_HOST=unix://<path-to-your-docker-socket>
export TESTCONTAINERS_RYUK_DISABLED=true
mvn verify -Pintegration-tests -pl sourcelume-registry-distribution-atlas

The API version pin for the docker client lives in sourcelume-registry-distribution-atlas/src/test/resources/docker-java.properties. If compose output is needed while debugging, add -Dquarkus.log.console.level=INFO.

Testing against an existing Atlas instance

To run the integration tests against an isolated Atlas instance that is already running, supply its URL. This bypasses the Compose lifecycle manager:

mvn verify -Pintegration-tests -Dsourcelume.it.atlas-url=http://localhost:21000

SOURCELUME_IT_ATLAS_USER and SOURCELUME_IT_ATLAS_PASSWORD override the local Atlas development defaults. The suite registers/updates Sourcelume type definitions and writes uniquely named test records; use a disposable instance. The ordinary HTTP tests explicitly select their mock profile, so the Atlas integration profile always uses the real storage provider.

Performance benchmarking

perf/ seeds the issue-#24 workload (datasets plus model records sized 0/12/40/161 trainingData references) and times GET /records/{id} reference resolution through the public API, so the same run works against either distribution. See perf/README.md.