Google GitGoogle Git
Sign in
apache/sourcelume-sample

Apache Sourcelume Sample: a tiny char-level model trained end-to-end with full training-data provenance

Mirrored from https://github.com/apache/sourcelume-sample.git
Clone this repo:

Branches

  1. b61de76 Fix CI startup failure: use only first-party actions by anierbeck · 15 hours ago main
  2. 836faa7 Initial import: end-to-end training sample with full provenance by anierbeck · 2 days ago

Apache Sourcelume Sample

Tiny model, full provenance. This repo trains a char-level language model (so small it trains in seconds on a laptop CPU) and walks the complete Sourcelume loop: dataset record → training → TrainingProvenanceRecord → registry → query.

The point of the demo: the training record is born from the training run with real token counts from model/training-run.json — not authored by hand. A provenance record that says “model X was trained on dataset Y” is only interesting if you can watch it happen.

The five steps

git clone https://github.com/apache/sourcelume-sample && cd sourcelume-sample
make demo
  1. Dataset record — data/provenance-record.json describes the bundled public-domain corpus (Alice's Adventures in Wonderland, 1865, Project Gutenberg ebook #11). The demo publishes this record itself: it bootstraps its own preconditions instead of assuming records it did not create.
  2. Train — src/train.py trains a tiny char-level MLP (numpy only, no torch, no GPU) on the corpus and writes model/training-run.json with the honest numbers of the run.
  3. Record — the trainer fills model/provenance-record.json: a TrainingProvenanceRecord referencing the dataset record by canonical record IRI, with tokens (characters per epoch) and share: 1.0 from the actual run.
  4. Validate — both records are validated twice: against the spec (JSON Schema + SHACL) and against the registry's /records/validate endpoint.
  5. Publish + query — with the Atlas stack up, records are published via the Python plugin; GET /records/{id} and the Atlas UI show the result.

Prerequisites

  • Python 3.11+ and uv
  • For the registry steps: a local Sourcelume Registry (see sourcelume-registry/dev-support, port 8082)
  • For publishing: the Atlas stack (make demo-atlas)

What works today, and what deliberately does not

StepState
Dataset record: spec + registry validation✅ works today
Dataset record: publish to Atlas✅ works with the Atlas stack up
Training record: spec validation✅ works — requires a spec checkout at the model-usage-record (PR #10) state or later; set SOURCELUME_SPEC accordingly
Training record: plugin validate/publish❌ expected to fail — the Python plugin itself only models ProvenanceRecord (its Pydantic model rejects the record client-side). The plugin needs the same type-dispatch the spec validator has (tooling follow-up; see make registry-validate).
Training record: registry validate/publish❌ expected to fail until the registry learns the TrainingProvenanceRecord type (registry follow-up to PR #10). Direct curl -X POST .../records/validate shows the registry‘s own rejection (HTTP 422, must be the constant value 'ProvenanceRecord') — the demo’s exhibit for that follow-up.

This demo builds on PR #10 as given: the training record type, the type-dispatching validator and the record examples come from that branch.

Honesty notes

  • The demo record is unsigned (Sourcelume Attest is still a discussion).
  • It is a producer's own claim — no certification, no adjudication, no “approved” flag anywhere.
  • disclosure: complete is only honest because the demo knows its data. Realistic partial/undisclosed cases are covered by the spec examples.
  • One token = one character (char-level model); tokens are per epoch, following the Mímir Table-10 convention.