blob: c83d9e2619d699c539c443667efa98bd8b7c5aee [file] [view]
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# Sourcelume Dev Support: PostgreSQL & Apache Atlas
This directory provides local infrastructure for developing and testing the
Sourcelume Registry with PostgreSQL or Apache Atlas.
---
### PostgreSQL default distribution
The default registry stores records directly in PostgreSQL and does not need
Atlas. From the repository root, build with `mvn clean install`, then run
`docker compose up -d --build --wait`.
For host or IDE development, from the repository root:
```bash
docker compose -f dev-support/docker-compose.postgres.yml up -d --wait
mvn -pl sourcelume-registry-distribution-postgres quarkus:dev
```
The distribution uses the development database on port `55432` by default.
See the [root README](../README.md#running-the-quarkus-runtime) for connection
settings. This database is independent of the Atlas stack's `atlas-db`.
The remaining sections cover the Atlas distribution.
### Overview
The local development environment consists of:
1. **Apache Atlas backend**: an Apache Atlas 2.5.0 container stack with the
PostgreSQL backend, listening on port `21000`.
2. **Sourcelume Registry runtime**: the Quarkus application that connects to
Atlas, registers the Sourcelume typedefs on startup, and exposes SmallRye
Health readiness/liveness probes at `/q/health/{ready,live}` on port `8082`.
3. **Shared network**: a Docker bridge network named `sourcelume-network`
connecting the Atlas containers and the Sourcelume services.
---
### Prerequisites
- **Docker Engine** ≥ 20.10.5 and **Docker Compose** ≥ 1.28.5 (configured with
at least 6 GB RAM).
- **Java 21** (e.g. Eclipse Temurin 21).
- **Maven 3.9+**.
- `curl` and `jq` (recommended for verifying JSON responses).
---
### Running on Windows
The dev tooling is POSIX-shell based, but the full workflow runs on Windows
in three ways:
#### WSL2 (recommended)
Install Docker Desktop with the WSL2 backend and a WSL2 distro
(`wsl --install`), then clone and run everything inside WSL2 exactly as the
bash instructions below describe. Docker Desktop exposes the same engine to
WSL2, so `docker compose` and the `sourcelume-network` bridge are shared
with Windows. Two caveats:
- Clone the repo inside the WSL2 filesystem (`~/...`), not under `/mnt/c` —
bind-mounts and Maven disk access are much faster on the ext4 filesystem.
- `~/.m2` inside WSL2 is a different directory than your Windows Maven
cache; the in-container Atlas build warms its own copy there.
#### Git Bash
Git for Windows ships a POSIX shell; `setup.sh`, `download-archives.sh`,
and `docker compose` all work from it unchanged — Docker Desktop's CLI and
engine are already on the `PATH`. Use `export VAR=value` exactly as written
in the bash snippets.
#### Native PowerShell
Use `setup.ps1` instead of `setup.sh`:
```powershell
.\dev-support\setup.ps1
```
Then note these equivalents:
- `export VAR=value` becomes `$env:VAR = "value"` — e.g.
`$env:DOCKER_BUILDKIT = "1"`, `$env:COMPOSE_DOCKER_CLI_BUILD = "1"`,
`$env:ATLAS_BACKEND = "postgres"`, and
`$env:SOURCELUME_ATLAS_URL = "http://localhost:21000"`.
- `mkdir -p "${HOME}/.m2"` becomes `New-Item -ItemType Directory -Force ~\.m2`.
- `download-archives.sh` still needs a POSIX shell — run it from Git Bash
or WSL2 (`bash download-archives.sh`), or fetch the five archives listed
in the vendored `.env` into `downloads/` yourself. Only the Kafka archive
is needed for the postgres backend.
- `curl` in Windows PowerShell 5.1 aliases to `Invoke-WebRequest`; call
`curl.exe` explicitly (ships with Windows 10+) or use
`Invoke-RestMethod` for the same REST calls. `jq` is available via
`winget install jqlang.jq`.
- The Atlas server container is always named `atlas` (compose
`container_name`), so attaching it to the shared network is simply:
```powershell
docker network connect sourcelume-network atlas
```
Everything else — the `docker compose` files, `mvn` commands, and port
numbers — is platform-independent.
---
### Quick Start: Standing up the Atlas backend
#### 1. Initialize dev support
Run `setup.sh` to create the shared Docker network and vendor Atlas's build
tooling:
```bash
./dev-support/setup.sh
```
#### 2. Download Atlas dependency archives
Atlas Kafka/Solr/ZooKeeper components require third-party archives before
building:
```bash
cd dev-support/vendor/atlas-docker
chmod +x download-archives.sh && ./download-archives.sh
```
#### 3. Build Atlas containers
```bash
export DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1
# 3a. Build the atlas-base image
docker compose -f docker-compose.atlas-base.yml build
# 3b. Build Atlas from source (takes ~15-45 mins on first run depending on ~/.m2 cache)
mkdir -p "${HOME}/.m2"
docker compose -f docker-compose.atlas-build.yml up
```
#### 4. Start Atlas with the PostgreSQL backend
```bash
export ATLAS_BACKEND=postgres
docker compose -f docker-compose.atlas.yml up -d --wait
```
Verify Atlas is running:
```bash
curl -u admin:atlasR0cks! http://localhost:21000/api/atlas/admin/version
```
#### 5. Connect Atlas to `sourcelume-network`
Allow containerized Sourcelume services to reach Atlas by hostname (`atlas`).
The Atlas server container is always named `atlas` (compose
`container_name`):
```bash
docker network connect sourcelume-network atlas || true
```
---
### Atlas images: options & trade-offs
There is no official prebuilt Atlas image — `apache/atlas` on Docker Hub
publishes no tags, and the 2.5.0 release ships a source tarball only (no
binary distribution). Building Atlas from source
(`docker-compose.atlas-build.yml`, ~15–45 min the first time, faster with a
warm `~/.m2`) is therefore currently unavoidable if you want a real 2.5.0.
The vendored stack then assembles these images:
| Image | Base | Why it exists |
|---|---|---|
| `atlas` | `atlas-base` (Ubuntu + JDK) | The Atlas server, from the source-built dist tarball |
| `atlas-db` | `postgres:13` | Postgres backend + init script |
| `atlas-solr` | `solr:8` | Solr + Atlas configsets |
| `atlas-kafka` | `atlas-base` + Kafka 2.8.2 | Broker with the Atlas Kafka hook preinstalled |
| `atlas-zk` | `zookeeper:3.9.2` | Stock image, re-tagged |
The dependency images are thin wrappers around stock images; the expensive
parts are the Atlas source build itself and `download-archives.sh` (which
fetches ~GBs of Hadoop/HBase/Hive/Kafka archives — only the Kafka archive
is needed for the postgres backend).
Alternatives, in order of increasing trade-off:
- **Build once, share the image.** `docker save atlas:latest | gzip >
atlas-2.5.0.tar.gz` lets teammates `docker load` a real 2.5.0 instead of
rebuilding.
- **Single-container embedded Atlas.** The distro's `atlas_start.py` can
run embedded HBase+Solr+Kafka in one container (the old quickstart
pattern). Fewer moving parts and still 2.5.0, but embedded-HBase
semantics differ from the postgres stack we verify against.
- **`sburn/apache-atlas:2.3.0`.** The de facto community image — single
container, fast start — but Atlas 2.3.0 (not 2.5.0), amd64-only, and
unmaintained. Fine for a throwaway REST smoke test; not the supported
configuration.
---
### Running & testing the Quarkus runtime
#### Option A: Docker Compose (container validation)
```bash
# From the repo root
# 1. Build project artifacts (also produces the Quarkus fast-jar for the image)
mvn clean package
# 2. Build and start the registry container
docker compose -f docker-compose.atlas.yml build
docker compose -f docker-compose.atlas.yml up -d
# 3. View logs
docker compose -f docker-compose.atlas.yml logs -f registry
```
#### Option B: Host / IDE (rapid development & debugging)
```bash
# From the repo root, with Atlas reachable on localhost:21000
export SOURCELUME_ATLAS_URL=http://localhost:21000
export SOURCELUME_ATLAS_USER=admin
export SOURCELUME_ATLAS_PASSWORD=atlasR0cks!
# Run the Quarkus runtime in dev mode
mvn -pl sourcelume-registry-distribution-atlas quarkus:dev
```
Or run/debug `org.apache.sourcelume.registry.runtime.quarkus.SourcelumeRegistryApplication`
from your IDE using the Atlas distribution module's classpath.
---
### Testing & verification workflows
#### 1. SmallRye Health probes
The runtime exposes SmallRye Health probes at `/q/health/{ready,live}` on
port `8082`:
```bash
# Overall health (including the atlas readiness check)
curl -s http://localhost:8082/q/health | jq .
# Liveness and readiness probes (Kubernetes style)
curl -s http://localhost:8082/q/health/live | jq .
curl -s http://localhost:8082/q/health/ready | jq .
```
Expected readiness output when Atlas is reachable:
```json
{
"status": "UP",
"checks": [
{
"name": "atlas",
"status": "UP",
"data": {
"atlasUrl": "http://atlas:21000",
"status": "CONNECTED"
}
}
]
}
```
Prometheus metrics:
```bash
curl -s http://localhost:8082/q/metrics
```
#### 2. Typedef bootstrap verification
On startup the runtime automatically registers the `sourcelume-spec` models in
Atlas (retries via `sourcelume.atlas.max-retries` / `retry-delay-ms`).
```bash
# Check the registered type definitions
curl -s -u admin:atlasR0cks! http://localhost:21000/api/atlas/v2/types/typedefs | jq .
# Check the sourcelume_dataset entity definition
curl -s -u admin:atlasR0cks! http://localhost:21000/api/atlas/v2/types/entitydef/name/sourcelume_dataset | jq .
```
#### 3. Testing entity creation & querying in Atlas
```bash
# Create a test dataset entity directly in Atlas
curl -u admin:atlasR0cks! -X POST http://localhost:21000/api/atlas/v2/entity/bulk \
-H "Content-Type: application/json" \
-d '{
"entities": [{
"typeName": "sourcelume_dataset",
"attributes": {
"name": "sample-dataset-1",
"qualifiedName": "sourcelume://datasets/sample-1",
"description": "First test Sourcelume dataset instance",
"sourceUri": "https://github.com/example/repo",
"licenseId": "Apache-2.0"
}
}]
}'
# Fetch the created entity by qualified name (the bulk endpoint only
# resolves guids; the lookup goes through the unique-attribute endpoint)
curl -s -u admin:atlasR0cks! \
"http://localhost:21000/api/atlas/v2/entity/uniqueAttribute/type/sourcelume_dataset?attr:qualifiedName=sourcelume://datasets/sample-1" | jq .
```
#### 4. Testing dataset operations directly in Sourcelume Registry
Sourcelume exposes high-level REST endpoints on port `8082`. Records are written via `POST /records`; `/api/v1/datasets` offers read-only views:
```bash
# Publish a dataset record (the only way to write one)
curl -s -X POST http://localhost:8082/records \
-H "Content-Type: application/ld+json" \
--data-binary @examples/simple-demo/sample-provenance-record.jsonld | jq .
# Query by qualifiedName (the record's id; query parameter)
curl -s "http://localhost:8082/api/v1/datasets?qualifiedName=https://sourcelume.apache.org/records/rec-glasgow-hackathon-001" | jq .
# Fetch dataset by GUID
curl -s "http://localhost:8082/api/v1/datasets/guid/<ASSIGNED_GUID>" | jq .
# Inspect Sourcelume type definitions
curl -s http://localhost:8082/api/v1/types/typedefs | jq .
curl -s http://localhost:8082/api/v1/types/dataset | jq .
```
#### 5. Automated tests
Run the full automated test suite (framework-free core tests plus the Quarkus
wiring tests):
```bash
mvn clean verify
```
---
### Teardown & reset
#### Clean up test typedefs from Atlas
```bash
curl -u admin:atlasR0cks! -X DELETE http://localhost:21000/api/atlas/v2/types/typedefs \
-H "Content-Type: application/json" \
-d @sourcelume-registry-typedefs/src/main/resources/models/sourcelume/sourcelume_model.json
```
#### Stop the Sourcelume container
From the repository root:
```bash
docker compose -f docker-compose.atlas.yml down
```
#### Stop the Apache Atlas stack
```bash
cd dev-support/vendor/atlas-docker
docker compose -f docker-compose.atlas.yml down
```
---
### Running the integration tests (`-Pintegration-tests`)
`mvn verify -Pintegration-tests` boots the vendored Atlas stack through
Testcontainers and runs the end-to-end pipeline tests (`*IT`, see
`sourcelume-registry-distribution-atlas/src/.../it/`). Prerequisites: the
vendored tooling and the locally built `atlas:latest` image (the steps
above), plus a reachable Docker daemon.
If your Docker daemon does not listen on the default socket path
(`/var/run/docker.sock` — e.g. Rancher Desktop, colima), point the client
at it via `DOCKER_HOST`. If the cleanup sidecar then cannot mount that
socket, disable it with `TESTCONTAINERS_RYUK_DISABLED=true` (the compose
stack is stopped explicitly by the test resource anyway):
```bash
export DOCKER_HOST=unix://<path-to-your-docker-socket>
export TESTCONTAINERS_RYUK_DISABLED=true
mvn verify -Pintegration-tests -pl sourcelume-registry-distribution-atlas
```
The API version pin for the docker client lives in
`sourcelume-registry-distribution-atlas/src/test/resources/docker-java.properties`.
If compose output is needed while debugging, add
`-Dquarkus.log.console.level=INFO`.
### Testing against an existing Atlas instance
To run the integration tests against an isolated Atlas instance that is already
running, supply its URL. This bypasses the Compose lifecycle manager:
```bash
mvn verify -Pintegration-tests -Dsourcelume.it.atlas-url=http://localhost:21000
```
`SOURCELUME_IT_ATLAS_USER` and `SOURCELUME_IT_ATLAS_PASSWORD` override the local
Atlas development defaults. The suite registers/updates Sourcelume type
definitions and writes uniquely named test records; use a disposable instance.
The ordinary HTTP tests explicitly select their mock profile, so the Atlas
integration profile always uses the real storage provider.
### Performance benchmarking
`perf/` seeds the issue-#24 workload (datasets plus model records sized
0/12/40/161 `trainingData` references) and times `GET /records/{id}`
reference resolution through the public API, so the same run works
against either distribution. See [perf/README.md](perf/README.md).