Tiny model, full provenance. This repo trains a char-level language model (so small it trains in seconds on a laptop CPU) and walks the complete Sourcelume loop: dataset record → training → TrainingProvenanceRecord → registry → query.
The point of the demo: the training record is born from the training run with real token counts from model/training-run.json — not authored by hand. A provenance record that says “model X was trained on dataset Y” is only interesting if you can watch it happen.
git clone https://github.com/apache/sourcelume-sample && cd sourcelume-sample make demo
Dataset record — data/provenance-record.json describes the bundled public-domain corpus (Alice's Adventures in Wonderland, 1865, Project Gutenberg ebook #11). The demo publishes this record itself: it bootstraps its own preconditions instead of assuming records it did not create.
Train — src/train.py trains a tiny char-level MLP (numpy only, no torch, no GPU) on the corpus and writes model/training-run.json with the honest numbers of the run.
Record — the trainer fills model/provenance-record.json: a TrainingProvenanceRecord referencing the dataset record by canonical record IRI, with tokens (characters per epoch) and share: 1.0 from the actual run.
Validate — both records are validated twice: against the spec (JSON Schema + SHACL) and against the registry's /records/validate endpoint.
Publish + query — with the Atlas stack up, records are published via the Python plugin; make query asks GET /records/{id} for both records. The model record‘s answer lists each trainingData entry together with the referenced dataset record’s current status, looked up at read time (sourcelume-registry#22) — the registry answers “which datasets, in which state”, instead of leaving the reader to resolve the references by hand:
"trainingData": [ { "dataset": "https://sourcelume.apache.org/records/sample/gutenberg-alice-1865", "phase": "pretraining", "tokens": 20000, "share": 1.0, "recordStatus": "VALIDATED" } ]
Nothing about the referenced records is stored on the model entity: a dataset published or corrected later shows up on the next read.
sourcelume-registry/dev-support, port 8082)make demo-atlas)| Step | State |
|---|---|
| Dataset record: spec + registry validation | ✅ works today |
| Dataset record: publish to Atlas | ✅ works with the Atlas stack up |
| Training record: spec validation | ✅ works — validate against a sourcelume-spec checkout at main (the TrainingProvenanceRecord type was merged with PR #10); set SOURCELUME_SPEC accordingly |
| Training record: plugin validate/publish | ✅ works with the plugin from sourcelume-tooling main (the pass-through came with sourcelume-tooling#1 and has been on main since sourcelume-tooling#3): the plugin forwards a record that is not a ProvenanceRecord as it is, and the registry judges it. Earlier plugin versions reject the record client-side. |
| Training record: query resolves its datasets | ✅ works — GET /records/{id} lists each trainingData entry with the referenced dataset record's current status, looked up at read time (sourcelume-registry#22); nothing is stored on the model entity |
| Training record: registry validate/publish | ✅ works — the registry validates it (POST /records/validate → {"conforms": true}) and accepts it (201, VALIDATED, stored as sourcelume_model) since sourcelume-registry#15. Direct call: curl -X POST -H 'Content-Type: application/json' --data-binary @model/provenance-record.json .../records/validate. |
This sample builds on spec PR #10 as merged: the training record type, the type-dispatching validator and the record examples come from the spec's main.
disclosure: complete is only honest because the sample knows its data. Realistic partial/undisclosed cases are covered by the spec examples.tokens are per epoch, following the Mímir Table-10 convention.