Merge pull request #53 from apache/dependabot/github_actions/github-actions-5780e5ba6e ci: Bump the github-actions group across 1 directory with 3 updates
diff --git a/Cargo.lock b/Cargo.lock index 5f03a5b..2eeaf96 100644 --- a/Cargo.lock +++ b/Cargo.lock
@@ -1094,6 +1094,7 @@ "axum", "bytes", "dm-core", + "dm-index", "dm-ledger", "dm-server", "futures-util",
diff --git a/PLAN.md b/PLAN.md index 632953c..6f7ce4f 100644 --- a/PLAN.md +++ b/PLAN.md
@@ -30,6 +30,7 @@ | Proof of concept, and the production shape (M2) | Everything built so far proves the contracts on the simplest stores that hold them on one laptop: files, SQLite, JSONL, one process, one host. It stays that way as the reference, and production backends go in behind the same crates, one store at a time, each proven against the unchanged conformance suite and measured with the evaluation harness and `amem savings` before it is preferred. The targets are the reference stack's: the ledger in PostgreSQL, one schema per domain, a claim as one row-locked statement, first, since it is the contended path; telemetry as OpenTelemetry spans from the egress (model calls), the server (reads) and the ledger (task transitions) into a collector, beside the SQLite tables until the librarian's signals are computed from the stream, then instead of them; the index in Solr sharded by domain; the record in an object store with versioning through OpenDAL; the curation cycle event-driven off the log. A backend is selected by configuration in the root, never by a build flag, so one binary is the laptop and the sidecar | A proof of concept that is honest about being one can be judged on the contracts, and a production move that has to pass the same suite cannot quietly change them. The ledger first because ten thousand committers and a fleet claiming around the clock contend there and nowhere else on the hot path. OpenTelemetry because the librarian's signals, cost per task and the watch on the watchers are analytics, and the reference stack already puts them on a stream rather than in the record | | Policy and the perimeter (M12) | The checkable perimeter is Rego from the first day: the built-in policy ships in the policy crate and runs in process through regorus with the grants file as data, a root may replace it under `policy/*.rego`, and at scale the same files run in OPA as a sidecar with partial evaluation compiling the read filter into the index query. Its scope is the paper's: access, admission, spend and blast-radius gates, irreversibility; judgement is never encoded. Admission scales with what a task asks for: the system's own filings and a session's filings from a held task within that task's requirements are admitted where the domain admits its own, inflated requirements and filings on a session's own account stay proposed. Writes are scanned for what is not knowledge, instructions to the reader, broadcast to agents, tool syntax, hidden text, payloads, secrets, and from anyone but the owner such a body is quarantined for the owner, never served; a verified entry is replaced by no one but the owner; an agent session past the supersede count in `[gates]` is stopped. Every decision is logged with the policy version. Detection reads the logs, not the record: a write log beside the read log, `amem taint` for an entry's reach, and write-rate and relay alarms in the librarian's telemetry | Shared memory is an injection surface and a channel, and the paper's defence is structural: data is not authorization, the perimeter is outside the loop, and gates scale with blast radius. Rego from the start means the laptop and the fleet check the same rules, and a policy that is text in the record can be reviewed, versioned and cited by a decision log | | Tasks and their relations (M5d) | The model stays tasks, leases, handoffs and `depends_on`; relations are expressed in fields that exist, no coordinator process. A subagent task the board files carries `origin: task:<parent>`, so a child knows its parent and the tree is a query. When the board refuses a spawn it also files a continuation, "resume `<parent>` with what the children found", with `depends_on` naming the children and `origin: continues:<parent>`; the parent's run ends with its handoff as today, and the parent waits on the board, claimable by nobody while a continuation of it is open (decided 12 October, after act 4 of the demo showed it claimable and so done twice); the continuation's completion completes the parent with the same handoff, and its parent in turn, up the chain, unless a holder with a live lease has it; a continuation invalidated without a successor leaves the parent claimable again; and the continuation becomes claimable when the last child completes, with every child handoff and the parent's own in its prompt, which the prompt builder already does for `depends_on`. Fan-in is therefore the existing dependency rule; fan-out is filing; a chain is a sequence of continuations, each a task on the board, held by no process. Dedup at filing is `origin` plus a hash of the prompt, so the same spawn asked twice is one task. Admission is by rule: a domain grant admits tasks a session files from its own task, and the human stays in the loop for tasks from outside, a tap's say. No depth cap: a tree may be as deep as the work needs, and a step that only a person can take, a task whose requirements match a human principal, is a node like any other, so a human's answer sits in the chain and adds to its depth as a matter of course. Invalidation is recursive: deprecating a task deprecates its subtree, the children still open and the continuations that depend on them, each marked `superseded_by` the invalidation, and a dead end re-admits the root with the chain's handoffs attached as lessons, as M5d's register row says. Artifacts are entries, typed and labelled, written by the task that produced them and cited by its handoff; no new table. Memory written by a child is visible to the parent as it is to any session, through search, recall and T0; no push to a running agent, since no harness takes one and a run that needed a child's result before finishing should have been a continuation. The board is visible: the web terminal gets a page of tasks by status with their tree, where a human claims the tasks addressed to them | A task is a promise and a handoff is a report; a continuation is a new promise made from a report. Keeping them separate objects keeps intent and history legible, and keeping relations in `origin` and `depends_on` means nothing new has to be invented, only named. A depth cap would have been a guess at how work decomposes; the paper's answer is solvability, every leaf matches someone, and humans are claimants | +| Bundle records and `highest_label` | A bundle's `highest_label` counts its promotion records with its entries, and `manifest::FORMAT` stays `bundle/0` | There are no users yet, so the meaning of a field may change without a version bump. A reader from before the records refuses only a bundle whose records sit above all of its entries, which is the safe side: it never accepts more than before | ## M0: the developer weekend, 10 and 11 October 2026
diff --git a/crates/core/src/erasure.rs b/crates/core/src/erasure.rs index 92c7e1d..791a20a 100644 --- a/crates/core/src/erasure.rs +++ b/crates/core/src/erasure.rs
@@ -116,6 +116,18 @@ /// Copies this erasure could not reach, named so nobody believes it did. #[serde(default)] pub unresolved: Vec<String>, + /// How many records of the entry's promotions had their why removed from the + /// ledger. Who promoted which version to which tier, and when, is kept. + #[serde(default, skip_serializing_if = "is_zero")] + pub promotion_whys_removed: usize, + /// How many approval tasks of the entry's promotions now name it by its id alone: + /// their title and body were rewritten, and the open ones closed. + #[serde(default, skip_serializing_if = "is_zero")] + pub promotion_tasks_cleared: usize, +} + +fn is_zero(n: &usize) -> bool { + *n == 0 } /// The line a receipt carries while an erasure has not finished: the record step is done @@ -166,6 +178,8 @@ removed: vec![], flagged: vec![], unresolved: vec![], + promotion_whys_removed: 0, + promotion_tasks_cleared: 0, }; assert!(r.is_finished()); r.unresolved.push(UNFINISHED.into()); @@ -188,6 +202,8 @@ }], flagged: vec!["mem-me-00000002".into()], unresolved: vec!["exports issued before the erasure".into()], + promotion_whys_removed: 0, + promotion_tasks_cleared: 0, }; let text = r.to_yaml(); assert!(text.contains("reason: requested"));
diff --git a/crates/core/src/manifest.rs b/crates/core/src/manifest.rs index 79c0a21..7b51b62 100644 --- a/crates/core/src/manifest.rs +++ b/crates/core/src/manifest.rs
@@ -3,6 +3,11 @@ //! A bundle is a directory of entries plus `manifest.yaml`. The manifest names //! every entry with its digest and classification, and carries the highest //! classification inside the bundle, so a destination can be checked against it. +//! +//! A bundle may also carry why its entries were promoted: the records sit in a file of +//! their own, the manifest holds that file's digest and one line per record with the +//! entry it is about and its classification, and the highest classification counts them. +//! A manifest from before records were carried has none, and reads as before. use crate::Label; use serde::{Deserialize, Serialize}; @@ -21,6 +26,28 @@ pub classification: Label, } +/// The promotion records a bundle carries: why entries in it were promoted, in a file of +/// their own whose digest the manifest keeps, so a changed record is refused as a changed +/// entry is. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct ManifestPromotions { + /// Path of the file inside the bundle, relative to the manifest. + pub path: String, + /// Digest of the file's text, as [`crate::digest`] computes it. + pub digest: String, + /// One line per record, in the file's order. + pub records: Vec<ManifestPromotion>, +} + +/// One promotion record as the manifest lists it. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +pub struct ManifestPromotion { + /// The entry the record is about: one the manifest lists. + pub entry: String, + /// The record's classification, so a receiver can filter without opening the file. + pub classification: Label, +} + /// The manifest at the root of a bundle. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] pub struct Manifest { @@ -35,10 +62,15 @@ pub exported_by: String, /// The bound the export was asked for. pub max_label: Label, - /// The highest classification actually inside the bundle. + /// The highest classification actually inside the bundle, its promotion records + /// included. pub highest_label: Label, /// Every entry in the bundle. pub entries: Vec<ManifestEntry>, + /// The promotion records the bundle carries, if any. A bundle from before they were + /// carried has none. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub promotions: Option<ManifestPromotions>, } /// The current bundle format identifier. A format change bumps the trailing number. @@ -56,9 +88,41 @@ max_label, highest_label: Label::Public, entries: Vec::new(), + promotions: None, } } + /// Carry promotion records, after every entry is pushed; refuses a line above the + /// requested bound or about an entry the manifest does not list. + /// + /// # Errors + /// + /// [`crate::Error::Invalid`] naming the line. + pub fn set_promotions(&mut self, promotions: ManifestPromotions) -> Result<(), crate::Error> { + for r in &promotions.records { + if r.classification > self.max_label { + return Err(crate::Error::Invalid(format!( + "a promotion record of {} is {} which exceeds the export bound {}", + r.entry, r.classification, self.max_label + ))); + } + self.listed(&r.entry)?; + self.highest_label = self.highest_label.max(r.classification); + } + self.promotions = Some(promotions); + Ok(()) + } + + /// Whether `id` is an entry this manifest lists. + fn listed(&self, id: &str) -> Result<(), crate::Error> { + if self.entries.iter().any(|e| e.id == id) { + return Ok(()); + } + Err(crate::Error::Invalid(format!( + "a promotion record is about {id}, which the manifest does not list" + ))) + } + /// Add an entry; refuses anything above the requested bound. /// /// # Errors @@ -86,15 +150,20 @@ /// /// [`crate::Error::Invalid`] naming the field that disagrees. pub fn consistent(&self) -> Result<(), crate::Error> { + let records = self.promotions.iter().flat_map(|p| &p.records); + for r in records.clone() { + self.listed(&r.entry)?; + } let inside = self .entries .iter() .map(|e| e.classification) + .chain(records.map(|r| r.classification)) .max() .unwrap_or(Label::Public); if self.highest_label != inside { return Err(crate::Error::Invalid(format!( - "highest_label is {} but the entries listed reach {}", + "highest_label is {} but the entries and records listed reach {}", self.highest_label, inside ))); } @@ -151,6 +220,33 @@ self.consistent() } + /// Whether the promotion records a bundle carries are the ones its manifest lists: + /// `inside` is one pair per record in the file, in its order, each the entry it names + /// and its classification. The file's digest is the caller's to check first. + /// + /// # Errors + /// + /// [`crate::Error::Invalid`] naming the first record that disagrees. + pub fn check_promotions(&self, inside: &[(String, Label)]) -> Result<(), crate::Error> { + let lines = self.promotions.as_ref().map_or(&[][..], |p| &p.records[..]); + if lines.len() != inside.len() { + return Err(crate::Error::Invalid(format!( + "the manifest lists {} promotion records and the bundle carries {}", + lines.len(), + inside.len() + ))); + } + for (line, (entry, label)) in lines.iter().zip(inside) { + if *entry != line.entry || *label != line.classification { + return Err(crate::Error::Invalid(format!( + "a promotion record of {entry} at {label} is listed as one of {} at {}", + line.entry, line.classification + ))); + } + } + Ok(()) + } + /// Render as YAML. /// /// # Panics @@ -301,4 +397,64 @@ empty.highest_label = Label::Internal; assert!(empty.consistent().is_err()); } + + fn one_record(entry: &str, label: Label) -> ManifestPromotions { + ManifestPromotions { + path: "promotions.json".into(), + digest: "00".into(), + records: vec![ManifestPromotion { + entry: entry.into(), + classification: label, + }], + } + } + + #[test] + fn promotion_records_are_bounded_listed_and_counted_in_the_highest() { + let mut m = Manifest::new("dom-a", "memory:local:me", Label::Internal); + m.push(ManifestEntry { + id: "mem-a-1".into(), + path: "mem-a-1.md".into(), + digest: "00".into(), + classification: Label::Public, + }) + .unwrap(); + assert!(m + .clone() + .set_promotions(one_record("mem-a-1", Label::Restricted)) + .is_err()); + assert!(m + .clone() + .set_promotions(one_record("mem-a-9", Label::Public)) + .is_err()); + m.set_promotions(one_record("mem-a-1", Label::Internal)) + .unwrap(); + assert_eq!(m.highest_label, Label::Internal, "a record counts"); + m.consistent().unwrap(); + m.check_promotions(&[("mem-a-1".into(), Label::Internal)]) + .unwrap(); + assert!(m + .check_promotions(&[("mem-a-1".into(), Label::Public)]) + .is_err()); + assert!(m.check_promotions(&[]).is_err()); + // a highest that leaves the record out understates the bundle + let mut under = m.clone(); + under.highest_label = Label::Public; + assert!(under.consistent().is_err()); + let back = Manifest::from_yaml(&m.to_yaml()).unwrap(); + assert_eq!(back, m); + } + + #[test] + fn a_manifest_from_before_records_were_carried_reads_as_before() { + let old = "format: durable-memory/bundle/0\ndomain: dom-a\nexported_at: 2026-10-01T09:00:00Z\nexported_by: memory:local:me\nmax_label: internal\nhighest_label: public\nentries:\n- id: mem-a-1\n path: mem-a-1.md\n digest: '00'\n classification: public\n"; + let m = Manifest::from_yaml(old).unwrap(); + assert_eq!(m.promotions, None); + m.consistent().unwrap(); + m.check_promotions(&[]).unwrap(); + assert!( + !m.to_yaml().contains("promotions"), + "and writes as before when it carries none" + ); + } }
diff --git a/crates/ledger/src/lib.rs b/crates/ledger/src/lib.rs index 2fd172b..0e30851 100644 --- a/crates/ledger/src/lib.rs +++ b/crates/ledger/src/lib.rs
@@ -1,4 +1,5 @@ -//! The task ledger: tasks, leases, handoffs, origin dedup. +//! The task ledger: tasks, leases, handoffs, origin dedup, and the record of why each +//! entry was promoted. //! //! The ledger is event-sourced. Every operation appends one event to an append-only //! table and updates a projection of current task state in the same transaction. The @@ -22,6 +23,11 @@ //! ledger alike. An expired lease makes the task claimable again; the stale holder's //! heartbeat, release, and complete are refused. //! +//! A promotion is recorded on the same log, as an event of its own under the key +//! `promotion:<entry>`, and projected into a table of its own. It is no task: nothing +//! claims it, lists it, or reads it as delivered work. It is kept once per version of the +//! entry, and a rebuild replays it with the rest. +//! //! Time comes from a clock the ledger owns, so tests can move it. mod postgres; @@ -29,7 +35,7 @@ #[cfg(any(test, feature = "pg-test"))] pub mod testing; -use dm_core::Label; +use dm_core::{Label, TrustTier}; use serde::{Deserialize, Serialize}; use std::path::Path; use time::{format_description::well_known::Rfc3339, Duration, OffsetDateTime}; @@ -67,6 +73,10 @@ /// Who asked. holder: String, }, + /// Another connection holds the write-ahead log, so text just removed may still be + /// in it. Run the operation again. + #[error("ledger: another connection holds the log: {0}")] + Held(String), /// A row holds something the ledger did not write. Rebuild it. #[error("ledger row for {id}: {what}")] BadRow { @@ -191,6 +201,29 @@ pub posted_to: Option<String>, } +/// What an erasure did to an entry's promotion records and approval tasks. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct Scrubbed { + /// How many records of the entry hold no why now. + pub records: usize, + /// How many tasks filed from the entry's origin hold the redaction now. + pub tasks: usize, + /// Copies of the text the ledger could not remove, named so nobody believes it did. + pub beyond_reach: Vec<String>, +} + +/// What an erasure rewrites a task to when the task is about the erased entry: a title +/// and a body that name the entry by its id and nothing else of it. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Redaction { + /// The title every task gets. + pub title: String, + /// The body every task gets. + pub body: String, + /// Why an open task is invalidated. + pub reason: String, +} + /// The result of filing a task. #[derive(Clone, Debug, PartialEq, Eq)] pub struct Created { @@ -210,6 +243,49 @@ pub lease_until: OffsetDateTime, } +/// What a promotion records: which version of an entry was moved to which tier, by +/// whom, and why. A record of a change already judged, not work: it is no task, and no +/// query over tasks returns it. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct PromotionSpec { + /// The domain the entry lives in, whose ledger keeps the record. + pub domain: String, + /// The entry promoted. + pub entry: String, + /// The tier it was promoted to. + pub tier: TrustTier, + /// Who promoted it, system-qualified. + pub by: String, + /// Who wrote the entry. + pub author: String, + /// Why: what was checked, and against what. + pub why: String, + /// How sensitive the record is: at least the entry's classification. + pub classification: Label, + /// The digest of the version of the entry that was promoted. + pub digest: String, + /// The digest of the version the promotion writes, so a record whose version never + /// reached the record can be told from one whose version did. Empty in a record from + /// before it was kept. + #[serde(default)] + pub new_digest: String, +} + +/// A promotion as the ledger holds it. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct PromotionRecord { + /// What was recorded. + #[serde(flatten)] + pub spec: PromotionSpec, + /// The why was removed by an erasure of the entry: `why` is then empty, and the rest + /// is kept as the least an audit needs. + #[serde(default)] + pub why_erased: bool, + /// When. + #[serde(with = "time::serde::rfc3339")] + pub at: OffsetDateTime, +} + type Clock = Box<dyn Fn() -> OffsetDateTime + Send + Sync>; /// One appended event, as replay reads it. @@ -253,6 +329,14 @@ priority: i64, why: String, }, + /// An entry was promoted, and why. Appended under [`promotion_key`], never a task id. + /// An erasure of the entry rewrites it with the why removed and `why_erased` set. + Promoted { + #[serde(flatten)] + spec: PromotionSpec, + #[serde(default, skip_serializing_if = "is_false")] + why_erased: bool, + }, } impl Event { @@ -269,6 +353,7 @@ Event::Invalidated { .. } => "invalidated", Event::Posted { .. } => "posted", Event::Rescored { .. } => "rescored", + Event::Promoted { .. } => "promoted", } } @@ -288,6 +373,65 @@ } } +/// What a PostgreSQL ledger keeps of a why after the rows are rewritten: the server, not +/// the ledger, decides when old row versions and its log are gone. +const POSTGRES_RESIDUE: &str = "the ledger's PostgreSQL server: old row versions and its write-ahead log keep removed text until the server vacuums and recycles them"; + +/// A task event's payload with the redaction in place of its free text: a `created` +/// event's title and body, an `invalidated` event's reason, and the handoff of a +/// `handoff_left` or `completed` event, each text a person may have put the entry's text +/// in. `None` when it holds the redaction already, or carries no text. +fn redact_event(id: &str, payload: &str, redaction: &Redaction) -> Result<Option<String>, Error> { + let redacted = match Event::parse(id, payload)? { + Event::Created { + id: new_id, + mut spec, + } if spec.title != redaction.title || spec.body != redaction.body => { + spec.title.clone_from(&redaction.title); + spec.body.clone_from(&redaction.body); + Event::Created { id: new_id, spec } + } + Event::Invalidated { + reason, + superseded_by, + } if reason != redaction.reason => Event::Invalidated { + reason: redaction.reason.clone(), + superseded_by, + }, + Event::HandoffLeft { handoff } if handoff != redaction.reason => Event::HandoffLeft { + handoff: redaction.reason.clone(), + }, + Event::Completed { handoff } if handoff != redaction.reason => Event::Completed { + handoff: redaction.reason.clone(), + }, + _ => return Ok(None), + }; + Ok(Some(redacted.payload(id)?)) +} + +/// A `promoted` event's payload with its why removed, when the event is that domain's +/// and holds a why; `None` when there is nothing to rewrite. +fn scrub_payload(key: &str, domain: &str, payload: &str) -> Result<Option<String>, Error> { + match Event::parse(key, payload)? { + Event::Promoted { + mut spec, + why_erased, + } if spec.domain == domain && !(why_erased && spec.why.is_empty()) => { + spec.why.clear(); + let scrubbed = Event::Promoted { + spec, + why_erased: true, + }; + Ok(Some(scrubbed.payload(key)?)) + } + _ => Ok(None), + } +} + +fn is_false(b: &bool) -> bool { + !*b +} + fn fmt_time(t: OffsetDateTime) -> String { t.format(&Rfc3339) .unwrap_or_else(|_| "1970-01-01T00:00:00Z".to_string()) @@ -312,6 +456,56 @@ } } +/// The key a promotion's events are appended under. A task id starts `task-`, so the two +/// never meet. +fn promotion_key(entry: &str) -> String { + format!("promotion:{entry}") +} + +/// The columns of a promotion row, in the order both backends read them. +const PROMOTION_COLUMNS: &str = + "domain, entry, tier, promoter, author, why, label_rank, digest, new_digest, at, why_erased"; + +/// A promotion row as the database returns it, before the ledger has checked it. +struct RawPromotion { + domain: String, + entry: String, + tier: String, + promoter: String, + author: String, + why: String, + label_rank: i64, + digest: String, + new_digest: String, + at: String, + why_erased: i64, +} + +impl RawPromotion { + fn into_record(self) -> Result<PromotionRecord, Error> { + let key = promotion_key(&self.entry); + let tier = self.tier.parse::<TrustTier>().map_err(|_| Error::BadRow { + id: key.clone(), + what: format!("tier {:?}", self.tier), + })?; + Ok(PromotionRecord { + spec: PromotionSpec { + domain: self.domain, + entry: self.entry, + tier, + by: self.promoter, + author: self.author, + why: self.why, + classification: label_from_rank(&key, self.label_rank)?, + digest: self.digest, + new_digest: self.new_digest, + }, + at: parse_time(&key, &self.at)?, + why_erased: self.why_erased != 0, + }) + } +} + /// The columns of a task row, in the order both backends read them. const TASK_COLUMNS: &str = "id, domain, title, body, requirements, depends_on, origin, upstream_event, @@ -387,13 +581,16 @@ format!("continues:{parent}") } -/// Whether a manifest may claim `task` now: it is admitted, or claimed under a lease -/// that has run out, and the manifest carries every requirement. Whether its -/// dependencies are complete is the backend's question, asked inside the transaction. -fn claimable(task: &Task, now: OffsetDateTime, manifest: &[String]) -> bool { +/// Whether a session may claim `task` now: it is admitted, or claimed under a lease +/// that has run out; the session's manifest carries every requirement; and the +/// session's handling covers the task's classification. Whether its dependencies are +/// complete is the backend's question, asked inside the transaction. +fn claimable(task: &Task, now: OffsetDateTime, manifest: &[String], handling: Label) -> bool { let lease_live = task.lease_until.is_some_and(|t| t >= now); let state = task.status == Status::Admitted || (task.status == Status::Claimed && !lease_live); - state && task.spec.requirements.iter().all(|r| manifest.contains(r)) + state + && task.spec.classification <= handling + && task.spec.requirements.iter().all(|r| manifest.contains(r)) } fn check_holder(task: &Task, holder: &str, now: OffsetDateTime) -> Result<(), Error> { @@ -581,8 +778,10 @@ } /// Claim the highest-ranked claimable task in a domain whose requirements the - /// manifest satisfies and whose dependencies are complete. Claimable means admitted, - /// or claimed under a lease that has expired. One transaction, so one holder. + /// manifest satisfies, whose classification `handling` covers, and whose + /// dependencies are complete. Claimable means admitted, or claimed under a lease + /// that has expired. One transaction, so one holder; a task above `handling` is + /// passed over inside it, so it is never claimed and then given back. /// /// # Errors /// @@ -593,10 +792,11 @@ holder: &str, lease: Duration, manifest: &[String], + handling: Label, ) -> Result<Option<Claim>, Error> { let now = self.now(); let until = now + lease; - let claimed = on_backend!(&mut self.backend, db => db.claim_next(domain, now, holder, until, manifest))?; + let claimed = on_backend!(&mut self.backend, db => db.claim_next(domain, now, holder, until, manifest, handling))?; if let Some(id) = &claimed { self.trace("claimed", id).await; } @@ -614,17 +814,19 @@ /// # Errors /// /// [`Error::NotFound`]; [`Error::WrongState`] when it is not claimable, including when - /// another holder's lease is still live or a dependency is open. + /// another holder's lease is still live, a dependency is open, or the task is above + /// `handling`. pub async fn claim( &mut self, id: &str, holder: &str, lease: Duration, manifest: &[String], + handling: Label, ) -> Result<Claim, Error> { let now = self.now(); let until = now + lease; - on_backend!(&mut self.backend, db => db.claim(id, now, holder, until, manifest))?; + on_backend!(&mut self.backend, db => db.claim(id, now, holder, until, manifest, handling))?; self.trace("claimed", id).await; Ok(Claim { task: self.get(id).await?, @@ -936,6 +1138,121 @@ on_backend!(&self.backend, db => db.dependents(id)) } + /// Record a promotion: one event, projected into the promotions and nowhere else. + /// Once per promoter and version: a second record of the same entry, tier, digest + /// and promoter appends nothing and returns false. Another version, or another + /// promoter of the same version, is a record of its own, so a record whose promotion + /// failed never stands in for a later one that did not. + /// + /// # Errors + /// + /// A database error. + pub async fn record_promotion(&mut self, spec: &PromotionSpec) -> Result<bool, Error> { + let now = self.now(); + on_backend!(&mut self.backend, db => db.record_promotion(now, spec, false)) + } + + /// Record a promotion another ledger kept, as it was there: its time, and whether its + /// why was erased. An import of a bundle the same owner exported calls this, after + /// moving the record to the entry as it landed here. Once, as + /// [`Ledger::record_promotion`] is. + /// + /// # Errors + /// + /// A database error. + pub async fn restore_promotion(&mut self, record: &PromotionRecord) -> Result<bool, Error> { + on_backend!(&mut self.backend, db => db.record_promotion(record.at, &record.spec, record.why_erased)) + } + + /// The promotions of an entry in a domain, oldest first. + /// + /// # Errors + /// + /// A database error, or a row the ledger did not write. + pub async fn promotions( + &self, + domain: &str, + entry: &str, + ) -> Result<Vec<PromotionRecord>, Error> { + on_backend!(&self.backend, db => db.promotions(domain, entry)) + } + + /// Remove the why from every promotion record of an entry, for its erasure: the + /// events that hold it are rewritten and the projection with them, so no replay + /// brings it back. Who promoted which version to which tier, and when, is kept. + /// Running it again changes nothing. + /// + /// # Errors + /// + /// A database error, or [`Error::Held`] when another connection holds the log the + /// old text is in. + pub async fn scrub_promotions(&mut self, domain: &str, entry: &str) -> Result<Scrubbed, Error> { + let records = on_backend!(&mut self.backend, db => db.scrub_promotions(domain, entry))?; + Ok(Scrubbed { + records, + tasks: 0, + beyond_reach: self.beyond_reach(), + }) + } + + /// Rewrite the tasks filed from `origin` in `domain` for an erasure of what they are + /// about: each gets `redaction`'s title and body in its `created` event and in the + /// projection, its reason for being invalidated and the handoffs left on it or that + /// closed it become the redaction's reason, so no replay brings the old text back; + /// each still open is invalidated. The task's id, origin, who filed it and its history + /// stay. Returns how many tasks from that origin now hold the redaction. Running it + /// again changes nothing. + /// + /// # Errors + /// + /// A database error, or [`Error::Held`] when another connection holds the log the + /// old text is in. + pub async fn scrub_tasks( + &mut self, + domain: &str, + origin: &str, + redaction: &Redaction, + ) -> Result<usize, Error> { + let now = self.now(); + on_backend!(&mut self.backend, db => db.scrub_tasks(now, domain, origin, redaction)) + } + + /// Both scrubs of an erasure, [`Ledger::scrub_promotions`] of `entry` and + /// [`Ledger::scrub_tasks`] of `origin`, in one transaction, and on SQLite one rebuild + /// of the file after them, also when nothing was left to rewrite: an earlier run that + /// a reader held up left the old text in the write-ahead log. + /// + /// # Errors + /// + /// A database error, or [`Error::Held`] when another connection holds the log the + /// old text is in. + pub async fn scrub_entry( + &mut self, + domain: &str, + entry: &str, + origin: &str, + redaction: &Redaction, + ) -> Result<Scrubbed, Error> { + let now = self.now(); + let (records, tasks) = on_backend!( + &mut self.backend, + db => db.scrub_entry(now, domain, entry, origin, redaction) + )?; + Ok(Scrubbed { + records, + tasks, + beyond_reach: self.beyond_reach(), + }) + } + + /// What a scrub on this backend cannot remove. + fn beyond_reach(&self) -> Vec<String> { + match self.backend { + Backend::Sqlite(_) => Vec::new(), + Backend::Postgres(_) => vec![POSTGRES_RESIDUE.to_string()], + } + } + /// How many events the ledger holds. /// /// # Errors @@ -1029,6 +1346,37 @@ } } + /// Every event's payload, as the log holds it. + async fn payloads(&self) -> Vec<String> { + match self { + Site::Sqlite(dir) => { + let conn = rusqlite::Connection::open(dir.path().join("ledger.db")).unwrap(); + let mut stmt = conn.prepare("SELECT payload FROM events").unwrap(); + let rows = stmt.query_map([], |r| r.get(0)).unwrap(); + rows.map(Result::unwrap).collect() + } + Site::Postgres(server) => { + let c = pg_client(server).await; + let mut out = Vec::new(); + for row in c + .query("SELECT schema FROM public.ledger_domains", &[]) + .await + .unwrap() + { + let s: String = row.get(0); + for p in c + .query(&format!("SELECT payload FROM {s}.events"), &[]) + .await + .unwrap() + { + out.push(p.get(0)); + } + } + out + } + } + } + /// A ledger on a clock the test moves. async fn ledger(&self) -> (Ledger, Arc<Mutex<OffsetDateTime>>) { let now = Arc::new(Mutex::new(T0)); @@ -1071,6 +1419,7 @@ heartbeat_renews_only_for_the_holder_and_complete_closes, an_unrenewed_lease_is_lost_at_expiry_for_both_sides, requirements_match_the_manifest_and_priority_orders_the_queue, + a_task_goes_only_to_a_session_whose_handling_covers_it, dependencies_block_until_complete, release_returns_to_pool_and_invalidate_is_terminal, origin_and_upstream_event_dedup_a_second_filing, @@ -1084,6 +1433,14 @@ a_task_waits_on_its_continuation, a_task_closes_with_its_continuation, a_continuation_given_up_on_gives_its_task_back, + a_promotion_is_recorded_once_per_version, + a_promotion_record_is_no_task, + replay_keeps_the_promotions, + a_second_promoter_of_the_same_version_is_recorded, + an_erasure_takes_the_why_and_a_replay_keeps_it_out, + a_restored_record_keeps_its_time_and_its_erased_why, + an_erasure_rewrites_the_tasks_filed_from_an_origin, + an_erasure_takes_the_reason_and_the_handoff_of_the_tasks_too, } fn spec(title: &str) -> TaskSpec { @@ -1110,19 +1467,21 @@ Duration::minutes(n) } - /// `claim_next` in `dom-a` with an empty manifest. + /// `claim_next` in `dom-a` with an empty manifest and the highest handling. async fn next(l: &mut Ledger, holder: &str, lease: Duration) -> Option<Claim> { - l.claim_next("dom-a", holder, lease, &[]).await.unwrap() + l.claim_next("dom-a", holder, lease, &[], Label::Restricted) + .await + .unwrap() } - /// `claim` with an empty manifest. + /// `claim` with an empty manifest and the highest handling. async fn claim( l: &mut Ledger, id: &str, holder: &str, lease: Duration, ) -> Result<Claim, Error> { - l.claim(id, holder, lease, &[]).await + l.claim(id, holder, lease, &[], Label::Restricted).await } fn is_wrong_state<T>(r: &Result<T, Error>) -> bool { @@ -1229,7 +1588,13 @@ let c = next(l, "s1", mins(5)).await.unwrap(); assert_eq!(c.task.id, high.id, "highest priority the manifest can take"); let c = l - .claim_next("dom-a", "s1", mins(5), &["tool:github".into()]) + .claim_next( + "dom-a", + "s1", + mins(5), + &["tool:github".into()], + Label::Restricted, + ) .await .unwrap() .unwrap(); @@ -1241,6 +1606,41 @@ )); } + async fn a_task_goes_only_to_a_session_whose_handling_covers_it(b: &Site) { + let (mut l, _) = b.ledger().await; + let mut secret = spec("Rotate the signing key"); + secret.classification = Label::Restricted; + secret.priority = 10; + let secret = admitted(&mut l, secret).await; + let mut low = spec("Tidy the docs"); + low.priority = 1; + let low = admitted(&mut l, low).await; + handling_bounds_the_claim(&mut l, &secret, &low).await; + } + + /// `claim_next` in `dom-a` for a session whose handling is public. + async fn public(l: &mut Ledger) -> Option<Claim> { + l.claim_next("dom-a", "s1", mins(5), &[], Label::Public) + .await + .unwrap() + } + + async fn handling_bounds_the_claim(l: &mut Ledger, secret: &Task, low: &Task) { + let c = public(l).await.unwrap(); + assert_eq!( + c.task.id, low.id, + "the top task a public session can handle" + ); + assert!(public(l).await.is_none(), "nothing it may handle is left"); + assert!(is_wrong_state( + &l.claim(&secret.id, "s1", mins(5), &[], Label::Internal) + .await + )); + assert_eq!(l.get(&secret.id).await.unwrap().status, Status::Admitted); + let c = next(l, "s2", mins(5)).await.unwrap(); + assert_eq!(c.task.id, secret.id, "a restricted session takes it"); + } + async fn dependencies_block_until_complete(b: &Site) { let (mut l, _) = b.ledger().await; let parent = admitted(&mut l, spec("Parent")).await; @@ -1351,6 +1751,616 @@ claim(&mut l, &parent.id, "s1", mins(5)).await.unwrap(); } + fn promotion(entry: &str, digest: &str) -> PromotionSpec { + PromotionSpec { + domain: "dom-a".into(), + entry: entry.into(), + tier: TrustTier::Curated, + by: "agent:librarian".into(), + author: "human:test:me".into(), + why: "checked against the tree".into(), + classification: Label::Internal, + digest: digest.into(), + new_digest: format!("{digest}-curated"), + } + } + + async fn a_promotion_is_recorded_once_per_version(b: &Site) { + let (mut l, now) = b.ledger().await; + let first = promotion("mem-a-1", "d1"); + assert!(l.record_promotion(&first).await.unwrap(), "a new record"); + let events = l.event_count().await.unwrap(); + assert_eq!(events, 1, "one event for one record"); + assert!( + !l.record_promotion(&first).await.unwrap(), + "the same version again is not new" + ); + assert_eq!( + l.event_count().await.unwrap(), + events, + "and appends nothing" + ); + *now.lock().unwrap() = T0 + mins(1); + a_new_version_is_a_new_record(&mut l, first).await; + } + + async fn a_new_version_is_a_new_record(l: &mut Ledger, first: PromotionSpec) { + let again = PromotionSpec { + digest: "d2".into(), + why: "the id came back at raw and was checked again".into(), + ..first.clone() + }; + assert!( + l.record_promotion(&again).await.unwrap(), + "a new version is a new record" + ); + let got = l.promotions("dom-a", "mem-a-1").await.unwrap(); + assert_eq!( + got, + vec![ + PromotionRecord { + spec: first, + why_erased: false, + at: T0 + }, + PromotionRecord { + spec: again, + why_erased: false, + at: T0 + mins(1) + }, + ], + "every field kept, oldest first" + ); + assert!(l.promotions("dom-a", "mem-a-2").await.unwrap().is_empty()); + assert!(l.promotions("dom-b", "mem-a-1").await.unwrap().is_empty()); + } + + /// A promotion whose new version never landed leaves its record; a second promoter + /// of the same version is not that record, and is kept with its own why. + async fn a_second_promoter_of_the_same_version_is_recorded(b: &Site) { + let (mut l, _) = b.ledger().await; + let first = promotion("mem-a-1", "d1"); + assert!(l.record_promotion(&first).await.unwrap()); + let second = PromotionSpec { + by: "agent:checker".into(), + why: "read it against the release notes".into(), + ..first.clone() + }; + assert!( + l.record_promotion(&second).await.unwrap(), + "another promoter is another record" + ); + let got: Vec<PromotionSpec> = l + .promotions("dom-a", "mem-a-1") + .await + .unwrap() + .into_iter() + .map(|r| r.spec) + .collect(); + assert_eq!(got, vec![first, second.clone()]); + assert!(!l.record_promotion(&second).await.unwrap(), "once each"); + } + + const SECRET: &str = "passphrase-Qz7xx"; + + /// Two promotions of `mem-a-1` whose whys quote its secret, one of `mem-a-2`, and one + /// of an entry with the same id in `dom-b`, which is another entry. + async fn promotions_that_quote_a_secret(l: &mut Ledger) -> Vec<PromotionSpec> { + let first = PromotionSpec { + why: format!("the vault opens with {SECRET}, as it says"), + ..promotion("mem-a-1", "d1") + }; + let second = PromotionSpec { + by: "agent:checker".into(), + why: format!("tried {SECRET} on the vault"), + ..first.clone() + }; + let other = promotion("mem-a-2", "d9"); + let elsewhere = PromotionSpec { + domain: "dom-b".into(), + why: "dom-b's own reading of its own entry".into(), + ..promotion("mem-a-1", "d1") + }; + for p in [&first, &second, &other, &elsewhere] { + assert!(l.record_promotion(p).await.unwrap()); + } + vec![first, second, other, elsewhere] + } + + /// The records of `entry` in `domain` are exactly `spec`, why and all. + async fn kept_whole(l: &Ledger, domain: &str, entry: &str, spec: &PromotionSpec) { + let got = l.promotions(domain, entry).await.unwrap(); + assert_eq!(got.len(), 1, "{domain}/{entry}: {got:?}"); + assert_eq!(&got[0].spec, spec, "{domain}/{entry} keeps its why"); + assert!(!got[0].why_erased); + } + + async fn an_erasure_takes_the_why_and_a_replay_keeps_it_out(b: &Site) { + let (mut l, _) = b.ledger().await; + let specs = promotions_that_quote_a_secret(&mut l).await; + let scrubbed = l.scrub_promotions("dom-a", "mem-a-1").await.unwrap(); + assert_eq!(scrubbed.records, 2); + let kept: Vec<PromotionRecord> = specs[..2] + .iter() + .map(|s| PromotionRecord { + spec: PromotionSpec { + why: String::new(), + ..s.clone() + }, + why_erased: true, + at: T0, + }) + .collect(); + assert_eq!( + l.promotions("dom-a", "mem-a-1").await.unwrap(), + kept, + "the why goes; who, which version, which tier and when stay" + ); + another_entry_and_the_log_and_a_replay(b, &mut l, &specs, &kept).await; + } + + async fn another_entry_and_the_log_and_a_replay( + b: &Site, + l: &mut Ledger, + specs: &[PromotionSpec], + kept: &[PromotionRecord], + ) { + kept_whole(l, "dom-a", "mem-a-2", &specs[2]).await; + kept_whole(l, "dom-b", "mem-a-1", &specs[3]).await; + assert!( + !b.payloads().await.iter().any(|p| p.contains(SECRET)), + "the log no longer holds the why" + ); + l.rebuild().await.unwrap(); + assert_eq!( + l.promotions("dom-a", "mem-a-1").await.unwrap(), + kept, + "replay keeps it out" + ); + // and replay brings back every other why, in this domain and the other + kept_whole(l, "dom-a", "mem-a-2", &specs[2]).await; + kept_whole(l, "dom-b", "mem-a-1", &specs[3]).await; + let again = l.scrub_promotions("dom-a", "mem-a-1").await.unwrap(); + assert_eq!(again.records, 2, "running it again counts what is scrubbed"); + } + + /// The redaction an erasure of `mem-a-1` gives its tasks. + fn redaction() -> Redaction { + Redaction { + title: "Approve: verified for mem-a-1".into(), + body: "`mem-a-1` was erased.".into(), + reason: "the entry was erased".into(), + } + } + + /// A task filed from `origin` on `event` whose title and body quote the secret. + fn task_quoting_the_secret(origin: &str, event: &str) -> TaskSpec { + TaskSpec { + title: format!("Approve: verified for The vault opens with {SECRET}"), + body: format!("proposed because {SECRET} opens it"), + origin: Some(origin.into()), + upstream_event: Some(event.into()), + ..spec("") + } + } + + /// Three tasks: an open one and a completed one filed from `promote:mem-a-1`, and an + /// open one filed from `promote:mem-a-2`. + async fn tasks_quoting_the_secret(l: &mut Ledger) -> (Task, Task, Task) { + let open = admitted(l, task_quoting_the_secret("promote:mem-a-1", "verified")).await; + let done = admitted(l, task_quoting_the_secret("promote:mem-a-1", "again")).await; + claim(l, &done.id, "s1", mins(5)).await.unwrap(); + l.complete(&done.id, "s1", "mem-a-1").await.unwrap(); + let other = admitted(l, task_quoting_the_secret("promote:mem-a-2", "verified")).await; + (open, done, other) + } + + async fn an_erasure_rewrites_the_tasks_filed_from_an_origin(b: &Site) { + let (mut l, _) = b.ledger().await; + let (open, done, other) = tasks_quoting_the_secret(&mut l).await; + let scrubbed = l + .scrub_tasks("dom-a", "promote:mem-a-1", &redaction()) + .await; + assert_eq!(scrubbed.unwrap(), 2); + the_tasks_hold_the_redaction(&l, &open, &done, &other).await; + let holding = b + .payloads() + .await + .iter() + .filter(|p| p.contains(SECRET)) + .count(); + assert_eq!( + holding, 1, + "the log holds the text in the other entry's task alone" + ); + l.rebuild().await.unwrap(); + the_tasks_hold_the_redaction(&l, &open, &done, &other).await; + let again = l + .scrub_tasks("dom-a", "promote:mem-a-1", &redaction()) + .await; + assert_eq!( + again.unwrap(), + 2, + "running it again counts what holds the redaction" + ); + } + + async fn the_tasks_hold_the_redaction(l: &Ledger, open: &Task, done: &Task, other: &Task) { + let open_now = l.get(&open.id).await.unwrap(); + assert_eq!( + open_now.status, + Status::Invalidated, + "an open task is closed" + ); + assert_eq!(open_now.reason.as_deref(), Some("the entry was erased")); + redacted(&open_now); + let done_now = l.get(&done.id).await.unwrap(); + assert_eq!( + done_now.status, + Status::Completed, + "a closed task keeps its status" + ); + assert_eq!( + done_now.handoff.as_deref(), + Some("the entry was erased"), + "the handoff is free text, and goes with the rest" + ); + redacted(&done_now); + let other_now = l.get(&other.id).await.unwrap(); + assert_eq!( + other_now.spec, other.spec, + "another entry's task is untouched" + ); + assert_eq!(other_now.status, Status::Admitted); + } + + /// The task holds the redaction and nothing else of the entry, and the rest of it + /// is as it was filed. + fn redacted(t: &Task) { + assert_eq!(t.spec.title, "Approve: verified for mem-a-1"); + assert_eq!(t.spec.body, "`mem-a-1` was erased."); + assert_eq!( + t.spec.origin.as_deref(), + Some("promote:mem-a-1"), + "the origin stays" + ); + assert_eq!(t.spec.created_by, "human:test:me", "who filed it stays"); + assert_eq!(t.created, T0, "and when"); + } + + /// The reason a proposal was declined for, and the handoffs left on or closing a + /// task, are free text an approver can quote the entry in: the scrub takes them too, + /// in the events, in the rows and after a replay. + async fn an_erasure_takes_the_reason_and_the_handoff_of_the_tasks_too(b: &Site) { + let (mut l, _) = b.ledger().await; + let declined = admitted( + &mut l, + task_quoting_the_secret("promote:mem-a-1", "verified"), + ) + .await; + l.invalidate( + &declined.id, + &format!("declined: {SECRET} no longer opens it"), + None, + ) + .await + .unwrap(); + let left = admitted(&mut l, task_quoting_the_secret("promote:mem-a-1", "again")).await; + claim(&mut l, &left.id, "s1", mins(5)).await.unwrap(); + l.release(&left.id, "s1", Some(&format!("left: {SECRET}"))) + .await + .unwrap(); + claim(&mut l, &left.id, "s2", mins(5)).await.unwrap(); + l.complete(&left.id, "s2", &format!("done: {SECRET}")) + .await + .unwrap(); + assert_eq!( + l.scrub_tasks("dom-a", "promote:mem-a-1", &redaction()) + .await + .unwrap(), + 2 + ); + the_reason_and_the_handoff_hold_the_redaction(b, &l, &declined, &left).await; + l.rebuild().await.unwrap(); + the_reason_and_the_handoff_hold_the_redaction(b, &l, &declined, &left).await; + } + + async fn the_reason_and_the_handoff_hold_the_redaction( + b: &Site, + l: &Ledger, + declined: &Task, + left: &Task, + ) { + let declined_now = l.get(&declined.id).await.unwrap(); + assert_eq!(declined_now.status, Status::Invalidated); + assert_eq!( + declined_now.reason.as_deref(), + Some("the entry was erased"), + "the reason" + ); + let left_now = l.get(&left.id).await.unwrap(); + assert_eq!(left_now.status, Status::Completed); + assert_eq!( + left_now.handoff.as_deref(), + Some("the entry was erased"), + "the handoff" + ); + assert!( + !b.payloads().await.iter().any(|p| p.contains(SECRET)), + "no event holds the text" + ); + } + + /// The first ledger file under `dir` whose bytes hold `needle`. + fn file_holding(dir: &Path, needle: &str) -> Option<std::path::PathBuf> { + std::fs::read_dir(dir) + .unwrap() + .flatten() + .map(|f| f.path()) + .find(|p| { + std::fs::read(p) + .unwrap() + .windows(needle.len()) + .any(|w| w == needle.as_bytes()) + }) + } + + #[tokio::test] + async fn on_sqlite_an_erased_why_leaves_no_bytes_in_the_files() { + let dir = tempfile::tempdir().unwrap(); + let file = dir.path().join("ledger.db"); + let mut l = Ledger::open(&file).unwrap(); + // long enough to spill onto pages of its own, which an update frees and does not + // overwrite + let why = format!("{SECRET} {}", "and so on ".repeat(900)); + let long = PromotionSpec { + why, + ..promotion("mem-a-1", "d1") + }; + assert!(l.record_promotion(&long).await.unwrap()); + // and buried under later records, so the pages it sat on are deep in the file + for n in 0..40 { + let later = promotion(&format!("mem-a-{}", n + 2), "d1"); + assert!(l.record_promotion(&later).await.unwrap()); + } + // in the file itself, not only in the log + rusqlite::Connection::open(&file) + .unwrap() + .query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |r| { + r.get::<_, i64>(0) + }) + .unwrap(); + assert_eq!( + file_holding(dir.path(), SECRET), + Some(file.clone()), + "the file held it" + ); + let scrubbed = l.scrub_promotions("dom-a", "mem-a-1").await.unwrap(); + assert!(scrubbed.beyond_reach.is_empty()); + if let Some(f) = file_holding(dir.path(), SECRET) { + panic!("{} still holds the erased why", f.display()); + } + } + + #[test] + fn a_promoted_payload_from_before_the_new_digest_still_parses() { + let old = r#"{"kind":"promoted","domain":"dom-a","entry":"mem-a-1","tier":"curated","by":"agent:librarian","author":"human:test:me","why":"checked","classification":"internal","digest":"d1"}"#; + let Event::Promoted { spec, why_erased } = Event::parse("promotion:mem-a-1", old).unwrap() + else { + panic!("not a promotion"); + }; + assert_eq!(spec.new_digest, ""); + assert!(!why_erased); + } + + /// The shapes the promotions table had before this one, each a commit of the branch: + /// 41e0bc5 had neither `new_digest` nor `why_erased` and an index without the + /// promoter; 088f890 and 9db9e03 had `new_digest` and the promoter in the index, and + /// no `why_erased`. + const OLD_SHAPES: [&str; 2] = ["41e0bc5", "088f890"]; + + /// The `promotions` table and its index as `shape` made them, in SQLite's dialect + /// (`domain` leads the index) or PostgreSQL's (`{s}` the schema, no `domain`). + fn old_promotions_table(shape: &str, s: Option<&str>) -> String { + let (table, dot) = s.map_or(("promotions".to_string(), String::new()), |s| { + (format!("{s}.promotions"), format!("{s}.")) + }); + let (new_digest, key) = match (shape, s) { + ("41e0bc5", None) => ("", "domain, entry, tier, digest"), + ("41e0bc5", Some(_)) => ("", "entry, tier, digest"), + (_, None) => ( + "new_digest TEXT NOT NULL,", + "domain, entry, tier, digest, promoter", + ), + (_, Some(_)) => ("new_digest TEXT NOT NULL,", "entry, tier, digest, promoter"), + }; + let serial = if s.is_some() { + "BIGSERIAL PRIMARY KEY" + } else { + "INTEGER PRIMARY KEY AUTOINCREMENT" + }; + let new_digest_value = if new_digest.is_empty() { "" } else { "'', " }; + let new_digest_column = if new_digest.is_empty() { + "" + } else { + "new_digest, " + }; + format!( + "CREATE TABLE {table} (seq {serial}, + domain TEXT NOT NULL, entry TEXT NOT NULL, tier TEXT NOT NULL, + promoter TEXT NOT NULL, author TEXT NOT NULL, why TEXT NOT NULL, + label_rank BIGINT NOT NULL, digest TEXT NOT NULL, {new_digest} at TEXT NOT NULL); + CREATE UNIQUE INDEX promotions_version ON {table} ({key}); + INSERT INTO {dot}events (task_id, kind, at, payload) VALUES ('promotion:mem-a-1', + 'promoted', '2026-10-01T10:00:00Z', + '{{\"kind\":\"promoted\",\"domain\":\"dom-a\",\"entry\":\"mem-a-1\",\"tier\":\"curated\",\"by\":\"agent:librarian\",\"author\":\"human:test:me\",\"why\":\"checked\",\"classification\":\"internal\",\"digest\":\"d1\"}}'); + INSERT INTO {table} (domain, entry, tier, promoter, author, why, label_rank, + digest, {new_digest_column}at) VALUES ('dom-a', 'mem-a-1', 'curated', 'agent:librarian', + 'human:test:me', 'checked', 1, 'd1', {new_digest_value}'2026-10-01T10:00:00Z');" + ) + } + + /// A ledger file whose promotions table is as `shape` made it, holding one record. + fn ledger_of_shape(file: &Path, shape: &str) { + let c = rusqlite::Connection::open(file).unwrap(); + c.execute_batch(&format!( + "CREATE TABLE events (seq INTEGER PRIMARY KEY AUTOINCREMENT, task_id TEXT NOT NULL, + kind TEXT NOT NULL, at TEXT NOT NULL, payload TEXT NOT NULL); + {}", + old_promotions_table(shape, None) + )) + .unwrap(); + } + + #[tokio::test] + async fn on_sqlite_a_ledger_of_each_old_shape_is_migrated() { + for shape in OLD_SHAPES { + let dir = tempfile::tempdir().unwrap(); + let file = dir.path().join("ledger.db"); + ledger_of_shape(&file, shape); + let l = Ledger::open(&file).unwrap(); + the_migrated_ledger_works(l, shape).await; + } + } + + /// Reading, a second promoter of the old version, a rebuild and a scrub all work on + /// a ledger of an old shape once it is opened. + async fn the_migrated_ledger_works(mut l: Ledger, shape: &str) { + let old = l + .promotions("dom-a", "mem-a-1") + .await + .unwrap_or_else(|e| panic!("{shape}: {e}")); + assert_eq!(old.len(), 1, "{shape}"); + assert_eq!(old[0].spec.new_digest, "", "{shape}"); + assert!(!old[0].why_erased, "{shape}"); + let second = PromotionSpec { + by: "agent:checker".into(), + ..promotion("mem-a-1", "d1") + }; + assert!( + l.record_promotion(&second) + .await + .unwrap_or_else(|e| panic!("{shape}: {e}")), + "{shape}: the index has the promoter now" + ); + assert_eq!(l.rebuild().await.unwrap(), 2, "{shape}"); + assert_eq!( + l.promotions("dom-a", "mem-a-1").await.unwrap().len(), + 2, + "{shape}" + ); + let scrubbed = l.scrub_promotions("dom-a", "mem-a-1").await; + assert_eq!( + scrubbed.unwrap_or_else(|e| panic!("{shape}: {e}")).records, + 2, + "{shape}" + ); + } + + async fn a_restored_record_keeps_its_time_and_its_erased_why(b: &Site) { + let (mut l, _) = b.ledger().await; + let then = T0 - Duration::days(30); + let records = [ + PromotionRecord { + spec: promotion("mem-a-1", "d1"), + why_erased: false, + at: then, + }, + PromotionRecord { + spec: PromotionSpec { + why: String::new(), + ..promotion("mem-a-1", "d0") + }, + why_erased: true, + at: then - Duration::days(1), + }, + ]; + for r in &records { + assert!(l.restore_promotion(r).await.unwrap()); + assert!(!l.restore_promotion(r).await.unwrap(), "once"); + } + assert_eq!(l.promotions("dom-a", "mem-a-1").await.unwrap(), records); + l.rebuild().await.unwrap(); + assert_eq!( + l.promotions("dom-a", "mem-a-1").await.unwrap(), + records, + "a replay keeps both" + ); + } + + async fn a_promotion_record_is_no_task(b: &Site) { + let mut l = b.open().await; + let parent = admitted(&mut l, spec("A task beside it")).await; + assert!(l + .record_promotion(&promotion("mem-a-1", "d1")) + .await + .unwrap()); + only_the_task_is_listed(&l, &parent).await; + no_task_query_finds_the_record(&mut l, &parent).await; + } + + async fn only_the_task_is_listed(l: &Ledger, parent: &Task) { + for st in [ + Status::Proposed, + Status::Admitted, + Status::Claimed, + Status::Completed, + Status::Invalidated, + ] { + let ids: Vec<String> = l + .list("dom-a", st) + .await + .unwrap() + .into_iter() + .map(|t| t.id) + .collect(); + let expected = if st == Status::Admitted { + vec![parent.id.clone()] + } else { + Vec::new() + }; + assert_eq!(ids, expected, "{st:?}"); + } + } + + async fn no_task_query_finds_the_record(l: &mut Ledger, parent: &Task) { + assert!(l.unposted("dom-a").await.unwrap().is_empty()); + assert!(l.dependents("promotion:mem-a-1").await.unwrap().is_empty()); + assert!(l.filed_from("mem-a-1").await.unwrap().is_empty()); + assert!(matches!( + l.get("promotion:mem-a-1").await, + Err(Error::NotFound(_)) + )); + let c = next(l, "s1", mins(5)).await.unwrap(); + assert_eq!(c.task.id, parent.id); + assert!( + next(l, "s1", mins(5)).await.is_none(), + "nothing else to claim" + ); + } + + async fn replay_keeps_the_promotions(b: &Site) { + let mut l = b.open().await; + admitted(&mut l, spec("A task beside it")).await; + l.record_promotion(&promotion("mem-a-1", "d1")) + .await + .unwrap(); + l.record_promotion(&promotion("mem-a-1", "d2")) + .await + .unwrap(); + let before = l.promotions("dom-a", "mem-a-1").await.unwrap(); + assert_eq!(before.len(), 2); + assert_eq!(l.rebuild().await.unwrap(), 4); + assert_eq!(l.promotions("dom-a", "mem-a-1").await.unwrap(), before); + assert!( + !l.record_promotion(&promotion("mem-a-1", "d1")) + .await + .unwrap(), + "replay keeps what makes a record new" + ); + assert_eq!(l.list("dom-a", Status::Admitted).await.unwrap().len(), 1); + } + async fn invalidate_is_terminal(l: &mut Ledger, t: &Task) { let t = l .invalidate(&t.id, "duplicate", Some("task-a-other")) @@ -1581,8 +2591,8 @@ let first = admitted(&mut l1, spec("First")).await; let second = admitted(&mut l1, spec("Second")).await; let (c1, c2) = tokio::join!( - l1.claim_next("dom-a", "s1", mins(5), &[]), - l2.claim_next("dom-a", "s2", mins(5), &[]), + l1.claim_next("dom-a", "s1", mins(5), &[], Label::Restricted), + l2.claim_next("dom-a", "s2", mins(5), &[], Label::Restricted), ); let c1 = c1.unwrap().unwrap(); let c2 = c2.unwrap().unwrap(); @@ -1614,13 +2624,13 @@ vec![&other.id] ); let got = l - .claim_next("dom-b", "s", mins(5), &[]) + .claim_next("dom-b", "s", mins(5), &[], Label::Restricted) .await .unwrap() .unwrap(); assert_eq!(got.task.id, other.id); assert!(l - .claim_next("dom-b", "s", mins(5), &[]) + .claim_next("dom-b", "s", mins(5), &[], Label::Restricted) .await .unwrap() .is_none()); @@ -1635,7 +2645,7 @@ .unwrap() .is_empty()); assert!(l - .claim_next("dom-none", "s", mins(5), &[]) + .claim_next("dom-none", "s", mins(5), &[], Label::Restricted) .await .unwrap() .is_none()); @@ -1771,6 +2781,170 @@ assert_eq!(again.event_count().await.unwrap(), 3); } + /// A domain whose schema was made before promotions were recorded has no table for + /// them; connecting adds it, and a promotion there is kept. + #[tokio::test] + async fn on_postgres_a_domain_from_before_promotions_gets_their_table() { + let Some(server) = PgServer::start().await.unwrap() else { + return; + }; + let mut l = Ledger::connect(server.url()).await.unwrap(); + let t = admitted(&mut l, spec("Old")).await; + drop(l); + let s = crate::postgres::schema_for("dom-a"); + let c = pg_client(&server).await; + c.batch_execute(&format!("DROP TABLE IF EXISTS {s}.promotions")) + .await + .unwrap(); + let mut l = Ledger::connect(server.url()).await.unwrap(); + assert!(l + .record_promotion(&promotion("mem-a-1", "d1")) + .await + .unwrap()); + assert_eq!(l.promotions("dom-a", "mem-a-1").await.unwrap().len(), 1); + assert_eq!(l.get(&t.id).await.unwrap().status, Status::Admitted); + let routed = c + .query_one( + "SELECT COUNT(*) FROM public.ledger_tasks WHERE id LIKE 'promotion:%'", + &[], + ) + .await + .unwrap(); + assert_eq!( + routed.get::<_, i64>(0), + 0, + "a record is not routed as a task" + ); + } + + /// A connected ledger on a server whose `dom-a` schema has a promotions table of + /// `shape`, holding the one old record and no other event, and a client on the server. + async fn pg_domain_of_shape( + server: &PgServer, + shape: &str, + ) -> (Ledger, tokio_postgres::Client) { + let mut l = Ledger::connect(server.url()).await.unwrap(); + admitted(&mut l, spec("Makes the schema")).await; + drop(l); + let s = crate::postgres::schema_for("dom-a"); + let c = pg_client(server).await; + c.batch_execute(&format!( + "DROP TABLE {s}.promotions; DELETE FROM {s}.events; DELETE FROM {s}.tasks; + DELETE FROM public.ledger_tasks; {}", + old_promotions_table(shape, Some(&s)) + )) + .await + .unwrap(); + (Ledger::connect(server.url()).await.unwrap(), c) + } + + /// A domain schema of each old shape is migrated when the ledger connects. + #[tokio::test] + async fn on_postgres_a_domain_of_each_old_shape_is_migrated_at_connect() { + for shape in OLD_SHAPES { + let Some(server) = PgServer::start().await.unwrap() else { + return; + }; + let (l, _c) = pg_domain_of_shape(&server, shape).await; + the_migrated_ledger_works(l, shape).await; + } + } + + /// A domain schema of an old shape made after this ledger connected is migrated where + /// a record, a rebuild or a scrub reaches it. + #[tokio::test] + async fn on_postgres_a_domain_of_an_old_shape_made_after_connect_is_migrated() { + for shape in OLD_SHAPES { + let Some(server) = PgServer::start().await.unwrap() else { + return; + }; + let (mut l, c) = pg_domain_of_shape(&server, shape).await; + let s = crate::postgres::schema_for("dom-a"); + let back_to_old = format!( + "DROP TABLE {s}.promotions; DELETE FROM {s}.events; {}", + old_promotions_table(shape, Some(&s)) + ); + c.batch_execute(&back_to_old).await.unwrap(); + let second = PromotionSpec { + by: "agent:checker".into(), + ..promotion("mem-a-1", "d1") + }; + assert!( + l.record_promotion(&second).await.unwrap(), + "{shape}: record" + ); + assert_eq!( + l.promotions("dom-a", "mem-a-1").await.unwrap().len(), + 2, + "{shape}" + ); + c.batch_execute(&back_to_old).await.unwrap(); + assert_eq!(l.rebuild().await.unwrap(), 1, "{shape}: rebuild"); + assert_eq!( + l.promotions("dom-a", "mem-a-1").await.unwrap().len(), + 1, + "{shape}" + ); + c.batch_execute(&back_to_old).await.unwrap(); + let scrubbed = l.scrub_promotions("dom-a", "mem-a-1").await; + assert_eq!(scrubbed.unwrap().records, 1, "{shape}: scrub"); + } + } + + /// Connecting migrates only a schema that lacks the promotions table, so a domain + /// another process holds does not hold up the connect. + #[tokio::test] + async fn on_postgres_connect_waits_on_no_domain_that_has_its_table() { + let Some(server) = PgServer::start().await.unwrap() else { + return; + }; + let mut l = Ledger::connect(server.url()).await.unwrap(); + admitted(&mut l, spec("In a")).await; + drop(l); + let holder = pg_client(&server).await; + holder + .execute( + "SELECT pg_advisory_lock(hashtext('ledger_domain:' || $1))", + &[&"dom-a"], + ) + .await + .unwrap(); + let connected = tokio::time::timeout( + std::time::Duration::from_secs(5), + Ledger::connect(server.url()), + ) + .await; + assert!( + connected.is_ok(), + "connect waited on a domain that needed nothing" + ); + } + + /// A domain schema an older binary made after this ledger connected has no + /// promotions table; reading, recording and rebuilding there still work. + #[tokio::test] + async fn on_postgres_a_schema_without_the_table_made_after_connect_still_works() { + let Some(server) = PgServer::start().await.unwrap() else { + return; + }; + let mut l = Ledger::connect(server.url()).await.unwrap(); + admitted(&mut l, spec("In a")).await; + let c = pg_client(&server).await; + let drop_table = format!( + "DROP TABLE {}.promotions", + crate::postgres::schema_for("dom-a") + ); + c.batch_execute(&drop_table).await.unwrap(); + assert!(l.promotions("dom-a", "mem-a-1").await.unwrap().is_empty()); + assert!(l + .record_promotion(&promotion("mem-a-1", "d1")) + .await + .unwrap()); + c.batch_execute(&drop_table).await.unwrap(); + assert_eq!(l.rebuild().await.unwrap(), 3); + assert_eq!(l.promotions("dom-a", "mem-a-1").await.unwrap().len(), 1); + } + #[tokio::test] async fn every_transition_is_a_span_with_the_task_as_it_stands() { use opentelemetry::Value; @@ -1783,7 +2957,7 @@ let mut l = Ledger::in_memory().unwrap(); let t = admitted(&mut l, spec("Traced")).await; let c = l - .claim_next("dom-a", "agent:x", mins(5), &[]) + .claim_next("dom-a", "agent:x", mins(5), &[], Label::Restricted) .await .unwrap() .unwrap();
diff --git a/crates/ledger/src/postgres.rs b/crates/ledger/src/postgres.rs index 5b89f34..2c1eb24 100644 --- a/crates/ledger/src/postgres.rs +++ b/crates/ledger/src/postgres.rs
@@ -1,23 +1,30 @@ -//! The PostgreSQL backend: the same two tables and the same events, on a server several +//! The PostgreSQL backend: the same tables and the same events, on a server several //! processes can share. One connection, driven by a task of its own. //! //! One schema per domain, as the reference stack partitions everything by domain: a -//! domain's `events` and `tasks` live in a schema of their own, created when the domain -//! files its first task, so a domain's queue is a table no other domain's claims touch -//! and a domain can be moved, backed up or dropped as a unit. Two tables in `public` -//! route the rest: `ledger_domains`, each domain and its schema, and `ledger_tasks`, -//! each task id and its domain, written in the transaction that files the task, so an -//! operation by id finds its schema in one lookup. +//! domain's `events`, `tasks` and `promotions` live in a schema of their own, created +//! when the domain files its first task or records its first promotion, so a domain's +//! queue is a table no other domain's claims touch and a domain can be moved, backed up +//! or dropped as a unit. Two tables in `public` route the rest: `ledger_domains`, each +//! domain and its schema, and `ledger_tasks`, each task id and its domain, written in +//! the transaction that files the task, so an operation by id finds its schema in one +//! lookup. A promotion is no task and has no route there: it names its domain, and is +//! read by domain. //! //! Times are stored as the RFC 3339 text the SQLite backend stores, so a row reads back //! exactly as it was written and a replay produces the same projection on either side. //! A claim takes its candidates `FOR UPDATE SKIP LOCKED`, so two claimers on two //! connections cannot both win the same task. -use crate::{claimable, parse_time, Error, Event, RawTask, Status, Task, TaskSpec, TASK_COLUMNS}; +use crate::{ + claimable, parse_time, Error, Event, PromotionRecord, PromotionSpec, RawPromotion, RawTask, + Redaction, Status, Task, TaskSpec, PROMOTION_COLUMNS, TASK_COLUMNS, +}; +use dm_core::Label; use std::collections::HashMap; use std::sync::Mutex; use time::OffsetDateTime; +use tokio_postgres::error::SqlState; use tokio_postgres::types::ToSql; use tokio_postgres::{Client, GenericClient, NoTls, Row}; @@ -70,6 +77,65 @@ WHERE origin IS NOT NULL AND upstream_event IS NOT NULL; "; +/// One domain's promotions, created in its schema with the rest, and added at connect to +/// a schema made before promotions were recorded. A table an earlier shape made gets +/// each column it lacks, so the statements hold for every shape; the index is +/// [`PROMOTION_INDEX`], made again when the one there lacks the promoter. `{s}` is the +/// quoted schema name. +const PROMOTION_TABLE: &str = " +CREATE TABLE IF NOT EXISTS {s}.promotions ( + seq BIGSERIAL PRIMARY KEY, + domain TEXT NOT NULL, + entry TEXT NOT NULL, + tier TEXT NOT NULL, + promoter TEXT NOT NULL, + author TEXT NOT NULL, + why TEXT NOT NULL, + label_rank BIGINT NOT NULL, + digest TEXT NOT NULL, + new_digest TEXT NOT NULL DEFAULT '', + at TEXT NOT NULL, + why_erased BIGINT NOT NULL DEFAULT 0 +); +ALTER TABLE {s}.promotions ADD COLUMN IF NOT EXISTS new_digest TEXT NOT NULL DEFAULT ''; +ALTER TABLE {s}.promotions ADD COLUMN IF NOT EXISTS why_erased BIGINT NOT NULL DEFAULT 0; +"; + +/// The key a promotion is kept once under. `{s}` is the quoted schema name. +const PROMOTION_INDEX: &str = " +CREATE UNIQUE INDEX IF NOT EXISTS promotions_version + ON {s}.promotions (entry, tier, digest, promoter); +"; + +/// Whether the promotions table `$1` names (`<schema>.promotions`, quoted) has an index +/// `promotions_version` that the promoter is part of. +const INDEX_HAS_PROMOTER: &str = " +SELECT EXISTS ( + SELECT 1 FROM pg_index i + JOIN pg_class ic ON ic.oid = i.indexrelid + JOIN pg_attribute a ON a.attrelid = i.indrelid AND a.attnum = ANY (i.indkey) + WHERE i.indrelid = to_regclass($1) AND ic.relname = 'promotions_version' + AND a.attname = 'promoter') +"; + +/// The domains whose promotions table is missing, lacks a column, or is keyed without +/// the promoter: what a connect migrates, and nothing else. +const DOMAINS_TO_MIGRATE: &str = " +SELECT d.domain FROM public.ledger_domains d +LEFT JOIN pg_class c ON c.oid = to_regclass(d.schema || '.promotions') +WHERE c.oid IS NULL + OR NOT EXISTS (SELECT 1 FROM pg_attribute a WHERE a.attrelid = c.oid + AND a.attname = 'new_digest' AND NOT a.attisdropped) + OR NOT EXISTS (SELECT 1 FROM pg_attribute a WHERE a.attrelid = c.oid + AND a.attname = 'why_erased' AND NOT a.attisdropped) + OR NOT EXISTS (SELECT 1 FROM pg_index i + JOIN pg_class ic ON ic.oid = i.indexrelid + JOIN pg_attribute a ON a.attrelid = i.indrelid AND a.attnum = ANY (i.indkey) + WHERE i.indrelid = c.oid AND ic.relname = 'promotions_version' + AND a.attname = 'promoter') +ORDER BY d.domain +"; + /// The schema a domain's tables live in: `ledger_` and the domain with anything that is /// not a lowercase letter or digit made an underscore, then eight hex digits of the /// domain's digest, so two domains that read alike never share a schema. Quoted. @@ -137,9 +203,28 @@ known: Mutex::new(HashMap::new()), }; db.migrate_shared_tables().await?; + db.add_promotion_tables().await?; Ok(db) } + /// Give a promotions table to each domain schema made before promotions were + /// recorded, and its missing columns and key to one an earlier shape made: one + /// catalog query finds them, and only those are locked and migrated, so a connect + /// costs nothing per domain whose table is as it should be. + async fn add_promotion_tables(&mut self) -> Result<(), Error> { + let rows = self.client.query(DOMAINS_TO_MIGRATE, &[]).await?; + if rows.is_empty() { + return Ok(()); + } + let tx = self.client.transaction().await?; + for row in &rows { + let domain: String = row.try_get(0)?; + ensure_promotions(&tx, &domain).await?; + } + tx.commit().await?; + Ok(()) + } + fn cached(&self, domain: &str) -> Option<String> { self.known .lock() @@ -244,6 +329,151 @@ Ok(()) } + /// Record a promotion unless that promoter promoted that version of the entry to that + /// tier already. The domain's lock is held from the check to the commit, so two + /// processes recording the same promotion record it once. + pub(crate) async fn record_promotion( + &mut self, + at: OffsetDateTime, + spec: &PromotionSpec, + why_erased: bool, + ) -> Result<bool, Error> { + let tx = self.client.transaction().await?; + let s = ensure_domain(&tx, &spec.domain).await?; + // a schema an older binary made has no table, or one of an earlier shape + ensure_promotion_table(&tx, &s).await?; + let seen = tx + .query_opt( + &format!( + "SELECT 1 FROM {s}.promotions + WHERE entry = $1 AND tier = $2 AND digest = $3 AND promoter = $4" + ), + &[&spec.entry, &spec.tier.to_string(), &spec.digest, &spec.by], + ) + .await? + .is_some(); + if seen { + return Ok(false); + } + let key = crate::promotion_key(&spec.entry); + apply( + &tx, + &s, + &key, + at, + &Event::Promoted { + spec: spec.clone(), + why_erased, + }, + ) + .await?; + tx.commit().await?; + self.remember(&spec.domain, &s); + Ok(true) + } + + /// Remove the why from an entry's promotion events and rows, in one transaction under + /// the domain's lock. Returns how many records of the entry hold no why now. + pub(crate) async fn scrub_promotions( + &mut self, + domain: &str, + entry: &str, + ) -> Result<usize, Error> { + let Some(s) = self.schema(domain).await? else { + return Ok(0); + }; + let tx = self.client.transaction().await?; + ensure_promotions(&tx, domain).await?; + let n = scrub_promotions_in(&tx, &s, domain, entry).await?; + tx.commit().await?; + Ok(n) + } + + /// Rewrite the tasks filed from `origin` to the redaction, in their events and their + /// rows, and invalidate the open ones, in one transaction under the domain's lock. + /// Returns how many tasks from that origin there are. + pub(crate) async fn scrub_tasks( + &mut self, + at: OffsetDateTime, + domain: &str, + origin: &str, + redaction: &Redaction, + ) -> Result<usize, Error> { + let Some(s) = self.schema(domain).await? else { + return Ok(0); + }; + let tx = self.client.transaction().await?; + lock_domain(&tx, domain).await?; + let n = scrub_tasks_in(&tx, &s, at, origin, redaction).await?; + tx.commit().await?; + Ok(n) + } + + /// Both scrubs of an erasure in one transaction under the domain's lock. Returns the + /// records that hold no why now and the tasks from the origin. + pub(crate) async fn scrub_entry( + &mut self, + at: OffsetDateTime, + domain: &str, + entry: &str, + origin: &str, + redaction: &Redaction, + ) -> Result<(usize, usize), Error> { + let Some(s) = self.schema(domain).await? else { + return Ok((0, 0)); + }; + let tx = self.client.transaction().await?; + ensure_promotions(&tx, domain).await?; + let records = scrub_promotions_in(&tx, &s, domain, entry).await?; + let tasks = scrub_tasks_in(&tx, &s, at, origin, redaction).await?; + tx.commit().await?; + Ok((records, tasks)) + } + + pub(crate) async fn promotions( + &self, + domain: &str, + entry: &str, + ) -> Result<Vec<PromotionRecord>, Error> { + let Some(s) = self.schema(domain).await? else { + return Ok(Vec::new()); + }; + let rows = match self + .client + .query( + &format!( + "SELECT {PROMOTION_COLUMNS} FROM {s}.promotions + WHERE domain = $1 AND entry = $2 ORDER BY seq" + ), + &[&domain, &entry], + ) + .await + { + Ok(rows) => rows, + // a schema an older binary made has no table, and so no promotions + Err(e) if e.code() == Some(&SqlState::UNDEFINED_TABLE) => return Ok(Vec::new()), + Err(e) => return Err(e.into()), + }; + rows.iter() + .map(|r| { + RawPromotion { + domain: r.try_get(0)?, + entry: r.try_get(1)?, + tier: r.try_get(2)?, + promoter: r.try_get(3)?, + author: r.try_get(4)?, + why: r.try_get(5)?, + label_rank: r.try_get(6)?, + digest: r.try_get(7)?, + new_digest: r.try_get(8)?, + at: r.try_get(9)?, + why_erased: r.try_get(10)?, + } + .into_record() + }) + .collect() + } + /// The id of the task claimed, if any candidate qualified. The candidates are read /// in queue order; the first the manifest can take is then locked on its own, `FOR /// UPDATE SKIP LOCKED`, and read again as it now stands, so a row another claimer @@ -255,6 +485,7 @@ holder: &str, until: OffsetDateTime, manifest: &[String], + handling: Label, ) -> Result<Option<String>, Error> { let Some(s) = self.schema(domain).await? else { return Ok(None); @@ -273,7 +504,7 @@ .await?; for row in &rows { let seen = task_from_row(row)?; - if !claimable(&seen, now, manifest) { + if !claimable(&seen, now, manifest, handling) { continue; } let Some(locked) = tx @@ -289,7 +520,7 @@ continue; }; let task = task_from_row(&locked)?; - if !claimable(&task, now, manifest) + if !claimable(&task, now, manifest, handling) || !deps_complete(&tx, &task.spec.depends_on).await? || awaits_continuation(&tx, &s, &task.id).await? { @@ -313,6 +544,7 @@ holder: &str, until: OffsetDateTime, manifest: &[String], + handling: Label, ) -> Result<(), Error> { let s = self.schema_of_task(id).await?; let tx = self.client.transaction().await?; @@ -324,7 +556,7 @@ .await? .ok_or_else(|| Error::NotFound(id.to_string()))?; let task = task_from_row(&row)?; - if !claimable(&task, now, manifest) + if !claimable(&task, now, manifest, handling) || !deps_complete(&tx, &task.spec.depends_on).await? || awaits_continuation(&tx, &s, &task.id).await? { @@ -509,14 +741,56 @@ } } -/// Make a domain's schema and tables if they are not there, under a lock on the domain -/// so two processes filing a domain's first task do not race to create it. -async fn ensure_domain<C: GenericClient>(conn: &C, domain: &str) -> Result<String, Error> { +/// Take a domain's lock until the transaction ends. +async fn lock_domain<C: GenericClient>(conn: &C, domain: &str) -> Result<(), Error> { conn.execute( "SELECT pg_advisory_xact_lock(hashtext('ledger_domain:' || $1))", &[&domain], ) .await?; + Ok(()) +} + +/// Give a domain's schema its promotions table if it has none, under the domain's lock, +/// held until the caller's transaction ends. The schema's name comes from the routing +/// table, read under the lock. +async fn ensure_promotions<C: GenericClient>(conn: &C, domain: &str) -> Result<(), Error> { + lock_domain(conn, domain).await?; + let row = conn + .query_one( + "SELECT schema FROM public.ledger_domains WHERE domain = $1", + &[&domain], + ) + .await?; + let s: String = row.try_get(0)?; + ensure_promotion_table(conn, &s).await +} + +/// Make schema `s`'s promotions table if it is not there, give it each column it lacks, +/// and make its key again when the one there lacks the promoter. The caller holds the +/// domain's lock. +async fn ensure_promotion_table<C: GenericClient>(conn: &C, s: &str) -> Result<(), Error> { + conn.batch_execute(&PROMOTION_TABLE.replace("{s}", s)) + .await?; + let row = conn + .query_one(INDEX_HAS_PROMOTER, &[&format!("{s}.promotions")]) + .await?; + let keyed_by_promoter: bool = row.try_get(0)?; + if !keyed_by_promoter { + conn.batch_execute(&format!( + "DROP INDEX IF EXISTS {s}.promotions_version;{}", + PROMOTION_INDEX.replace("{s}", s) + )) + .await?; + } + Ok(()) +} + +/// Make a domain's schema and tables if they are not there, under a lock on the domain +/// so two processes filing a domain's first task do not race to create it. The lock is +/// held until the caller's transaction ends. +async fn ensure_domain<C: GenericClient>(conn: &C, domain: &str) -> Result<String, Error> { + lock_domain(conn, domain).await?; let known = conn .query_opt( "SELECT schema FROM public.ledger_domains WHERE domain = $1", @@ -528,8 +802,10 @@ } let s = schema_for(domain); conn.batch_execute(&format!( - "CREATE SCHEMA IF NOT EXISTS {s};{}", - DOMAIN_TABLES.replace("{s}", &s) + "CREATE SCHEMA IF NOT EXISTS {s};{}{}{}", + DOMAIN_TABLES.replace("{s}", &s), + PROMOTION_TABLE.replace("{s}", &s), + PROMOTION_INDEX.replace("{s}", &s) )) .await?; conn.execute( @@ -599,7 +875,11 @@ /// Drop one domain's projection and replay its events into it. async fn rebuild_domain<C: GenericClient>(conn: &C, domain: &str, s: &str) -> Result<u64, Error> { + // a schema an older binary made after this process connected has no promotions table + ensure_promotions(conn, domain).await?; conn.execute(&format!("DELETE FROM {s}.tasks"), &[]).await?; + conn.execute(&format!("DELETE FROM {s}.promotions"), &[]) + .await?; conn.execute( "DELETE FROM public.ledger_tasks WHERE domain = $1", &[&domain], @@ -624,6 +904,129 @@ Ok(n) } +/// The scrub of an entry's promotions in schema `s`, in the caller's transaction: the why +/// leaves its events and its rows. Returns how many records of the entry hold no why now. +async fn scrub_promotions_in<C: GenericClient>( + conn: &C, + s: &str, + domain: &str, + entry: &str, +) -> Result<usize, Error> { + let key = crate::promotion_key(entry); + let rows = conn + .query( + &format!( + "SELECT seq, payload FROM {s}.events + WHERE task_id = $1 AND kind = 'promoted' FOR UPDATE" + ), + &[&key], + ) + .await?; + for row in &rows { + let seq: i64 = row.try_get(0)?; + let payload: String = row.try_get(1)?; + if let Some(scrubbed) = crate::scrub_payload(&key, domain, &payload)? { + conn.execute( + &format!("UPDATE {s}.events SET payload = $2 WHERE seq = $1"), + &[&seq, &scrubbed], + ) + .await?; + } + } + conn.execute( + &format!( + "UPDATE {s}.promotions SET why = '', why_erased = 1 + WHERE domain = $1 AND entry = $2 AND (why_erased = 0 OR why != '')" + ), + &[&domain, &entry], + ) + .await?; + let row = conn + .query_one( + &format!( + "SELECT COUNT(*) FROM {s}.promotions + WHERE domain = $1 AND entry = $2 AND why_erased = 1" + ), + &[&domain, &entry], + ) + .await?; + let n: i64 = row.try_get(0)?; + Ok(usize::try_from(n).unwrap_or(0)) +} + +/// The scrub of the tasks filed from `origin` in schema `s`, in the caller's transaction: +/// each is rewritten to the redaction and the open ones are invalidated. Returns how many +/// tasks there are from that origin. +async fn scrub_tasks_in<C: GenericClient>( + conn: &C, + s: &str, + at: OffsetDateTime, + origin: &str, + redaction: &Redaction, +) -> Result<usize, Error> { + let rows = conn + .query( + &format!( + "SELECT id, status IN ('completed', 'invalidated') FROM {s}.tasks + WHERE origin = $1 ORDER BY created FOR UPDATE" + ), + &[&origin], + ) + .await?; + for row in &rows { + let id: String = row.try_get(0)?; + let terminal: bool = row.try_get(1)?; + if !terminal { + let closed = Event::Invalidated { + reason: redaction.reason.clone(), + superseded_by: None, + }; + apply(conn, s, &id, at, &closed).await?; + } + redact_task(conn, s, &id, redaction).await?; + } + Ok(rows.len()) +} + +/// One task's events and row, rewritten to the redaction: the title and body it was +/// filed with, the reason it was invalidated for, and the handoffs left on it or that +/// closed it, each free text an approver may have put the entry's text in. +async fn redact_task<C: GenericClient>( + conn: &C, + s: &str, + id: &str, + redaction: &Redaction, +) -> Result<(), Error> { + let rows = conn + .query( + &format!("SELECT seq, payload FROM {s}.events WHERE task_id = $1 FOR UPDATE"), + &[&id], + ) + .await?; + for row in &rows { + let seq: i64 = row.try_get(0)?; + let payload: String = row.try_get(1)?; + if let Some(redacted) = crate::redact_event(id, &payload, redaction)? { + conn.execute( + &format!("UPDATE {s}.events SET payload = $2 WHERE seq = $1"), + &[&seq, &redacted], + ) + .await?; + } + } + conn.execute( + &format!( + "UPDATE {s}.tasks SET title = $2, body = $3, + reason = CASE WHEN reason IS NULL THEN NULL ELSE $4 END, + handoff = CASE WHEN handoff IS NULL THEN NULL ELSE $4 END + WHERE id = $1" + ), + &[&id, &redaction.title, &redaction.body, &redaction.reason], + ) + .await?; + Ok(()) +} + /// Append the event and project it, in the caller's transaction. async fn apply<C: GenericClient>( conn: &C, @@ -685,6 +1088,38 @@ Ok(()) } +/// The row a `Promoted` event projects to. No route: a promotion is read by its domain. +async fn insert_promotion<C: GenericClient>( + conn: &C, + s: &str, + at: OffsetDateTime, + p: &PromotionSpec, + why_erased: bool, +) -> Result<(), Error> { + conn.execute( + &format!( + "INSERT INTO {s}.promotions (domain, entry, tier, promoter, author, why, + label_rank, digest, new_digest, at, why_erased) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)" + ), + &[ + &p.domain, + &p.entry, + &p.tier.to_string(), + &p.by, + &p.author, + &p.why, + &(p.classification as i64), + &p.digest, + &p.new_digest, + &crate::fmt_time(at), + &i64::from(why_erased), + ], + ) + .await?; + Ok(()) +} + /// Update the projection for one event. Replay calls this and nothing else. async fn project<C: GenericClient>( conn: &C, @@ -700,6 +1135,9 @@ }; let (sql, args): (&str, Vec<&(dyn ToSql + Sync)>) = match event { Event::Created { id: new_id, spec } => return insert_task(conn, s, new_id, at, spec).await, + Event::Promoted { spec, why_erased } => { + return insert_promotion(conn, s, at, spec, *why_erased).await + } Event::Admitted => ("SET status = 'admitted' WHERE id = $1", vec![&id]), Event::Claimed { holder, .. } => ( "SET status = 'claimed', holder = $2, lease_until = $3,
diff --git a/crates/ledger/src/sqlite.rs b/crates/ledger/src/sqlite.rs index 43fefa4..7aa2147 100644 --- a/crates/ledger/src/sqlite.rs +++ b/crates/ledger/src/sqlite.rs
@@ -3,7 +3,11 @@ //! The connection sits behind a mutex so the ledger is `Sync`; the server holds the whole //! ledger under an async lock, so the mutex is never contended. -use crate::{claimable, parse_time, Error, Event, RawTask, Status, Task, TaskSpec, TASK_COLUMNS}; +use crate::{ + claimable, parse_time, Error, Event, PromotionRecord, PromotionSpec, RawPromotion, RawTask, + Redaction, Status, Task, TaskSpec, PROMOTION_COLUMNS, TASK_COLUMNS, +}; +use dm_core::Label; use rusqlite::{params, Connection, OptionalExtension, Row}; use std::path::Path; use std::sync::{Mutex, MutexGuard, PoisonError}; @@ -44,6 +48,22 @@ DROP INDEX IF EXISTS tasks_origin; CREATE UNIQUE INDEX IF NOT EXISTS tasks_domain_origin ON tasks (domain, origin, upstream_event) WHERE origin IS NOT NULL AND upstream_event IS NOT NULL; +CREATE TABLE IF NOT EXISTS promotions ( + seq INTEGER PRIMARY KEY AUTOINCREMENT, + domain TEXT NOT NULL, + entry TEXT NOT NULL, + tier TEXT NOT NULL, + promoter TEXT NOT NULL, + author TEXT NOT NULL, + why TEXT NOT NULL, + label_rank INTEGER NOT NULL, + digest TEXT NOT NULL, + new_digest TEXT NOT NULL, + at TEXT NOT NULL, + why_erased INTEGER NOT NULL DEFAULT 0 +); +CREATE UNIQUE INDEX IF NOT EXISTS promotions_version + ON promotions (domain, entry, tier, digest, promoter); "; fn task_from_row(row: &Row<'_>) -> Result<Task, Error> { @@ -98,6 +118,7 @@ if has_posted == 0 { conn.execute_batch("ALTER TABLE tasks ADD COLUMN posted_to TEXT;")?; } + migrate_promotions(&conn)?; Ok(Db { conn: Mutex::new(conn), }) @@ -141,6 +162,137 @@ Ok(()) } + /// Record a promotion unless that promoter promoted that version of the entry to that + /// tier already. The check and the append are one transaction, taken for writing first. + pub(crate) fn record_promotion( + &mut self, + at: OffsetDateTime, + spec: &PromotionSpec, + why_erased: bool, + ) -> Result<bool, Error> { + let mut conn = self.conn(); + let tx = conn.transaction_with_behavior(rusqlite::TransactionBehavior::Immediate)?; + let seen = tx + .query_row( + "SELECT 1 FROM promotions + WHERE domain = ?1 AND entry = ?2 AND tier = ?3 AND digest = ?4 + AND promoter = ?5", + params![ + spec.domain, + spec.entry, + spec.tier.to_string(), + spec.digest, + spec.by + ], + |_| Ok(()), + ) + .optional()? + .is_some(); + if seen { + return Ok(false); + } + let key = crate::promotion_key(&spec.entry); + apply( + &tx, + &key, + at, + &Event::Promoted { + spec: spec.clone(), + why_erased, + }, + )?; + tx.commit()?; + Ok(true) + } + + /// Remove the why from an entry's promotion events and rows in one transaction, with + /// the space they leave overwritten, then rebuild the file without that space and + /// fold the log into it, so the old text is in neither. Returns how many records of + /// the entry hold no why now. + pub(crate) fn scrub_promotions(&mut self, domain: &str, entry: &str) -> Result<usize, Error> { + self.scrubbing(|tx| scrub_promotions_in(tx, domain, entry)) + } + + /// Rewrite the tasks filed from `origin` to the redaction, in their events and their + /// rows, and invalidate the open ones, in one transaction with the space the old text + /// leaves overwritten; then rebuild the file and fold the log into it. Returns how + /// many tasks from that origin there are. + pub(crate) fn scrub_tasks( + &mut self, + at: OffsetDateTime, + domain: &str, + origin: &str, + redaction: &Redaction, + ) -> Result<usize, Error> { + self.scrubbing(|tx| scrub_tasks_in(tx, at, domain, origin, redaction)) + } + + /// Both scrubs of an erasure in one transaction, and one rebuild of the file after + /// them. Returns the records that hold no why now and the tasks from the origin. + pub(crate) fn scrub_entry( + &mut self, + at: OffsetDateTime, + domain: &str, + entry: &str, + origin: &str, + redaction: &Redaction, + ) -> Result<(usize, usize), Error> { + self.scrubbing(|tx| { + let (records, changed_records) = scrub_promotions_in(tx, domain, entry)?; + let (tasks, changed_tasks) = scrub_tasks_in(tx, at, domain, origin, redaction)?; + Ok(((records, tasks), changed_records + changed_tasks)) + }) + } + + /// Run `scrub` in one transaction taken for writing, with what an update frees + /// overwritten with zeros, here and nowhere else: the rest of the ledger's writes need + /// not pay for it. Then rebuild the file and fold the log into it, whether or not + /// `scrub` rewrote anything: a scrub that rewrote rows and then found the log held + /// committed the rows and left the old text in the log, and the run after it finds + /// nothing to rewrite and still has to fold that log. Returns what `scrub` returned, + /// without the count of rows it rewrote. + fn scrubbing<T>( + &mut self, + scrub: impl FnOnce(&rusqlite::Transaction<'_>) -> Result<(T, usize), Error>, + ) -> Result<T, Error> { + let mut conn = self.conn(); + conn.pragma_update(None, "secure_delete", true)?; + let tx = conn.transaction_with_behavior(rusqlite::TransactionBehavior::Immediate)?; + let (out, _rewritten) = scrub(&tx)?; + tx.commit()?; + conn.pragma_update(None, "secure_delete", false)?; + compact(&conn)?; + Ok(out) + } + + pub(crate) fn promotions( + &self, + domain: &str, + entry: &str, + ) -> Result<Vec<PromotionRecord>, Error> { + let conn = self.conn(); + let mut stmt = conn.prepare(&format!( + "SELECT {PROMOTION_COLUMNS} FROM promotions WHERE domain = ?1 AND entry = ?2 + ORDER BY seq" + ))?; + let rows = stmt.query_map(params![domain, entry], |r| { + Ok(RawPromotion { + domain: r.get(0)?, + entry: r.get(1)?, + tier: r.get(2)?, + promoter: r.get(3)?, + author: r.get(4)?, + why: r.get(5)?, + label_rank: r.get(6)?, + digest: r.get(7)?, + new_digest: r.get(8)?, + at: r.get(9)?, + why_erased: r.get(10)?, + }) + })?; + rows.map(|r| r?.into_record()).collect() + } + /// The id of the task claimed, if any candidate qualified. pub(crate) fn claim_next( &mut self, @@ -149,6 +301,7 @@ holder: &str, until: OffsetDateTime, manifest: &[String], + handling: Label, ) -> Result<Option<String>, Error> { let mut conn = self.conn(); let tx = conn.transaction_with_behavior(rusqlite::TransactionBehavior::Immediate)?; @@ -164,7 +317,7 @@ rows.map(|r| r?).collect::<Result<_, _>>()? }; for task in candidates { - if !claimable(&task, now, manifest) + if !claimable(&task, now, manifest, handling) || !deps_complete(&tx, &task.spec.depends_on)? || awaits_continuation(&tx, &task)? { @@ -192,11 +345,12 @@ holder: &str, until: OffsetDateTime, manifest: &[String], + handling: Label, ) -> Result<(), Error> { let mut conn = self.conn(); let tx = conn.transaction_with_behavior(rusqlite::TransactionBehavior::Immediate)?; let task = get_in(&tx, id)?; - if !claimable(&task, now, manifest) + if !claimable(&task, now, manifest, handling) || !deps_complete(&tx, &task.spec.depends_on)? || awaits_continuation(&tx, &task)? { @@ -290,7 +444,7 @@ pub(crate) fn rebuild(&mut self) -> Result<u64, Error> { let mut conn = self.conn(); let tx = conn.transaction()?; - tx.execute_batch("DELETE FROM tasks;")?; + tx.execute_batch("DELETE FROM tasks; DELETE FROM promotions;")?; let events: Vec<(String, String, String)> = { let mut stmt = tx.prepare("SELECT task_id, at, payload FROM events ORDER BY seq")?; let rows = stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))?; @@ -308,6 +462,181 @@ } } +/// A ledger whose promotions table 41e0bc5 made has neither `new_digest` nor +/// `why_erased`, and its index lacks the promoter, so a second promoter of a version would +/// be refused; one 088f890 made has `new_digest` and the promoter, and no `why_erased`. +/// Each column that is missing is added, empty or false, and the index is made again +/// when it lacks the promoter. +fn migrate_promotions(conn: &Connection) -> Result<(), Error> { + let has = |column: &str| -> Result<bool, Error> { + let n: i64 = conn.query_row( + "SELECT COUNT(*) FROM pragma_table_info('promotions') WHERE name = ?1", + [column], + |r| r.get(0), + )?; + Ok(n > 0) + }; + // each column on its own: a ledger from between the two shapes has one and not the + // other + if !has("new_digest")? { + conn.execute_batch( + "ALTER TABLE promotions ADD COLUMN new_digest TEXT NOT NULL DEFAULT '';", + )?; + } + if !has("why_erased")? { + conn.execute_batch( + "ALTER TABLE promotions ADD COLUMN why_erased INTEGER NOT NULL DEFAULT 0;", + )?; + } + let keyed_by_promoter: i64 = conn.query_row( + "SELECT COUNT(*) FROM pragma_index_info('promotions_version') WHERE name = 'promoter'", + [], + |r| r.get(0), + )?; + if keyed_by_promoter == 0 { + conn.execute_batch( + "BEGIN; + DROP INDEX IF EXISTS promotions_version; + CREATE UNIQUE INDEX promotions_version + ON promotions (domain, entry, tier, digest, promoter); + COMMIT;", + )?; + } + Ok(()) +} + +/// The scrub of an entry's promotions, in the caller's transaction: the why leaves its +/// events and its rows. Returns how many records of the entry hold no why now, and how +/// many rows this call rewrote. +fn scrub_promotions_in( + tx: &Connection, + domain: &str, + entry: &str, +) -> Result<(usize, usize), Error> { + let key = crate::promotion_key(entry); + let events: Vec<(i64, String)> = { + let mut stmt = + tx.prepare("SELECT seq, payload FROM events WHERE task_id = ?1 AND kind = 'promoted'")?; + let rows = stmt.query_map(params![key], |r| Ok((r.get(0)?, r.get(1)?)))?; + rows.collect::<Result<_, _>>()? + }; + let mut changed = 0; + for (seq, payload) in events { + if let Some(scrubbed) = crate::scrub_payload(&key, domain, &payload)? { + changed += tx.execute( + "UPDATE events SET payload = ?2 WHERE seq = ?1", + params![seq, scrubbed], + )?; + } + } + changed += tx.execute( + "UPDATE promotions SET why = '', why_erased = 1 + WHERE domain = ?1 AND entry = ?2 AND (why_erased = 0 OR why != '')", + params![domain, entry], + )?; + let n: i64 = tx.query_row( + "SELECT COUNT(*) FROM promotions WHERE domain = ?1 AND entry = ?2 AND why_erased = 1", + params![domain, entry], + |r| r.get(0), + )?; + Ok((usize::try_from(n).unwrap_or(0), changed)) +} + +/// The scrub of the tasks filed from `origin`, in the caller's transaction: each is +/// rewritten to the redaction and the open ones are invalidated. Returns how many tasks +/// there are from that origin, and how many rows this call rewrote. +fn scrub_tasks_in( + tx: &Connection, + at: OffsetDateTime, + domain: &str, + origin: &str, + redaction: &Redaction, +) -> Result<(usize, usize), Error> { + let tasks: Vec<(String, bool)> = { + let mut stmt = tx.prepare( + "SELECT id, status IN ('completed', 'invalidated') FROM tasks + WHERE domain = ?1 AND origin = ?2 ORDER BY created", + )?; + let rows = stmt.query_map(params![domain, origin], |r| Ok((r.get(0)?, r.get(1)?)))?; + rows.collect::<Result<_, _>>()? + }; + let mut changed = 0; + for (id, terminal) in &tasks { + if !terminal { + let closed = Event::Invalidated { + reason: redaction.reason.clone(), + superseded_by: None, + }; + apply(tx, id, at, &closed)?; + changed += 1; + } + changed += redact_task(tx, id, redaction)?; + } + Ok((tasks.len(), changed)) +} + +/// One task's events and row, rewritten to the redaction: the title and body it was +/// filed with, the reason it was invalidated for, and the handoffs left on it or that +/// closed it, each free text an approver may have put the entry's text in. Returns how +/// many rows changed. +fn redact_task(conn: &Connection, id: &str, redaction: &Redaction) -> Result<usize, Error> { + let events: Vec<(i64, String)> = { + let mut stmt = conn.prepare("SELECT seq, payload FROM events WHERE task_id = ?1")?; + let rows = stmt.query_map(params![id], |r| Ok((r.get(0)?, r.get(1)?)))?; + rows.collect::<Result<_, _>>()? + }; + let mut changed = 0; + for (seq, payload) in events { + if let Some(redacted) = crate::redact_event(id, &payload, redaction)? { + changed += conn.execute( + "UPDATE events SET payload = ?2 WHERE seq = ?1", + params![seq, redacted], + )?; + } + } + changed += conn.execute( + "UPDATE tasks SET title = ?2, body = ?3, + reason = CASE WHEN reason IS NULL THEN NULL ELSE ?4 END, + handoff = CASE WHEN handoff IS NULL THEN NULL ELSE ?4 END + WHERE id = ?1 AND (title != ?2 OR body != ?3 + OR reason IS NOT ?4 AND reason IS NOT NULL + OR handoff IS NOT ?4 AND handoff IS NOT NULL)", + params![id, redaction.title, redaction.body, redaction.reason], + )?; + Ok(changed) +} + +/// Rebuild the file without the free space old text sat in, and fold the log into it: +/// the log first, so the rebuild reads the tables as they stand; then again, since the +/// rebuild itself went through the log. `secure_delete` alone is not enough: a why long +/// enough for overflow pages, with records written after it, left its text in the file. +fn compact(conn: &Connection) -> Result<(), Error> { + checkpoint(conn)?; + conn.execute_batch("VACUUM;")?; + checkpoint(conn) +} + +/// Fold the write-ahead log into the file and truncate it, waiting a little for a reader +/// mid-query; [`Error::Held`] when it still could not, since the log then still holds +/// what the tables no longer do. +fn checkpoint(conn: &Connection) -> Result<(), Error> { + conn.busy_timeout(std::time::Duration::from_millis(200))?; + let mut held = true; + for _ in 0..5 { + let busy: i64 = conn.query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |r| r.get(0))?; + if busy == 0 { + held = false; + break; + } + } + // rusqlite's own default, for every other statement on this connection + conn.busy_timeout(std::time::Duration::from_secs(5))?; + if held { + return Err(Error::Held("the write-ahead log".into())); + } + Ok(()) +} + fn get_in(conn: &Connection, id: &str) -> Result<Task, Error> { conn.query_row( &format!("SELECT {TASK_COLUMNS} FROM tasks WHERE id = ?1"), @@ -386,9 +715,38 @@ Ok(()) } +/// The row a `Promoted` event projects to. +fn insert_promotion( + conn: &Connection, + at: OffsetDateTime, + p: &PromotionSpec, + why_erased: bool, +) -> Result<(), Error> { + conn.execute( + "INSERT INTO promotions (domain, entry, tier, promoter, author, why, label_rank, + digest, new_digest, at, why_erased) + VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11)", + params![ + p.domain, + p.entry, + p.tier.to_string(), + p.by, + p.author, + p.why, + p.classification as i64, + p.digest, + p.new_digest, + crate::fmt_time(at), + i64::from(why_erased), + ], + )?; + Ok(()) +} + /// Update the projection for one event. Replay calls this and nothing else. fn project(conn: &Connection, id: &str, at: OffsetDateTime, event: &Event) -> Result<(), Error> { match event { + Event::Promoted { spec, why_erased } => insert_promotion(conn, at, spec, *why_erased)?, Event::Created { id: new_id, spec } => insert_task(conn, new_id, at, spec)?, Event::Admitted => { conn.execute(
diff --git a/crates/memory/src/hooks.rs b/crates/memory/src/hooks.rs index 93a95c9..3e258fc 100644 --- a/crates/memory/src/hooks.rs +++ b/crates/memory/src/hooks.rs
@@ -647,6 +647,12 @@ text.push_str(&earlier.join("; ")); text.push_str(". "); } + // held back and not named: the chain holds them in work this session may not read + if let Some(n) = answer["withheld"].as_u64().filter(|n| *n > 0) { + text.push_str(&format!( + "The board withheld {n}: asked for before in this chain, in work this session may not read; do not ask again. " + )); + } text.push_str("Finish what you can from what is already known, then leave a handoff."); Decision::Deny(text) }
diff --git a/crates/memory/src/main.rs b/crates/memory/src/main.rs index bd06e76..c9541c1 100644 --- a/crates/memory/src/main.rs +++ b/crates/memory/src/main.rs
@@ -602,13 +602,17 @@ /// curated or verified. #[arg(long)] to: String, - /// What was checked, and against what. - #[arg(long, default_value = "")] + /// What was checked, and against what. Required: the ledger keeps it. + #[arg(long)] why: String, }, /// Approve a proposal to promote an entry to verified. The approver did not /// propose it, and is a person. Approve { task: String }, + /// Why each promotion of an entry was made, as JSON, oldest first. A proposal to + /// verified is listed, not landed until a person approves it. An auditor reads it + /// after judging the change, not before. + Promotions { id: String }, /// The models a destination lists, as JSON: what the web form offers to pick from. /// Routed as a launch would be, with the destination's credential; a login mode has /// no list, since the login is the harness's own. @@ -4588,6 +4592,7 @@ Command::Taint { id } => json(&m.taint(&id)), Command::Promote { id, to, why } => json(&m.promote(&id, to.parse()?, &why).await?), Command::Approve { task } => json(&m.approve(&task).await?), + Command::Promotions { id } => json(&m.promotions(&id).await?), Command::Canary { op } => canary_cmd(m, op).await, Command::Erase { id, reason } => json(&m.erase(&id, reason.into()).await?), Command::Reindex => { @@ -4748,11 +4753,12 @@ let im = m.import(&src, domain.as_deref()).await?; println!( "imported {} entries with {} transcript excerpts, skipped {} already present, \ - refused {} erased here", + refused {} erased here, restored {} promotion records", im.written.len(), im.excerpts, im.skipped.len(), - im.erased.len() + im.erased.len(), + im.promotions.len() ); Ok(()) } @@ -4783,6 +4789,7 @@ | Command::Erase { .. } | Command::Promote { .. } | Command::Approve { .. } + | Command::Promotions { .. } | Command::Reindex | Command::Policy { .. } | Command::Quarantine { .. }
diff --git a/crates/memory/tests/cli.rs b/crates/memory/tests/cli.rs index 8572404..566dd6a 100644 --- a/crates/memory/tests/cli.rs +++ b/crates/memory/tests/cli.rs
@@ -317,6 +317,13 @@ .collect() }; assert!(ids(&ok(&d.root, &["search", "vault opens"], None)).contains(&secret)); + // a why that quotes the secret is the entry's too, and leaves with it + let why = format!("tried {marker} on the vault"); + ok( + &d.root, + &["promote", &secret, "--to", "curated", "--why", &why], + None, + ); let err = fails(&d.root, &["erase", &secret, "--reason", "because"]); assert!(err.contains("invalid value"), "{err}"); let receipt = json(&ok( @@ -328,6 +335,7 @@ assert_eq!(receipt["reason"], "sensitive"); assert_eq!(receipt["removed"][0]["version"], "current"); assert_eq!(receipt["flagged"], serde_json::json!([citing])); + assert_eq!(receipt["promotion_whys_removed"], 1); assert!(!ids(&ok(&d.root, &["search", "vault opens"], None)).contains(&secret)); let err = fails(&d.root, &["get", &secret]); assert!(err.contains("gone") && err.contains("erased"), "{err}"); @@ -348,6 +356,29 @@ } #[test] +fn promotions_says_why_an_entry_was_promoted() { + let mut d = step_init(); + step_write(&mut d); + let why = "read it against the pilot's notes"; + let promoted = json(&ok( + &d.root, + &["promote", &d.decision, "--to", "curated", "--why", why], + None, + )); + assert_eq!(promoted["tier"], "curated"); + let records = json(&ok(&d.root, &["promotions", &d.decision], None)); + let records = records.as_array().unwrap(); + assert_eq!(records.len(), 1, "{records:?}"); + assert_eq!(records[0]["why"], why); + assert_eq!(records[0]["tier"], "curated"); + assert_eq!(records[0]["by"], "human:test:me"); + assert_eq!(records[0]["landed"], true); + let none = json(&ok(&d.root, &["promotions", &d.handoff], None)); + assert_eq!(none, serde_json::json!([])); + assert!(fails(&d.root, &["promotions", "mem-none"]).contains("not found")); +} + +#[test] fn an_erasure_takes_the_proposal_to_verify_the_entry_off_the_board() { let d = step_init(); let id = id_of(&ok(
diff --git a/crates/server/src/bundle.rs b/crates/server/src/bundle.rs index bf38e13..ace1b3b 100644 --- a/crates/server/src/bundle.rs +++ b/crates/server/src/bundle.rs
@@ -6,14 +6,18 @@ //! //! Import is the mirror. Every entry lands at trust raw unless the same owner exported it, //! keeps its provenance and classification, and is keyed on origin so a second import -//! creates nothing twice. A manifest path that is not a plain file name is refused before +//! creates nothing twice. Why its entries were promoted travels with them: the records +//! about entries exported, at or below the bound, in a file the manifest holds the digest +//! of; an import keeps them where it keeps the tiers, from the same owner, and leaves them +//! behind where the entries land raw. A manifest path that is not a plain file name is refused before //! anything is read, and an entry whose digest does not match its manifest line is //! refused before anything is written. use dm_core::erasure::body_digest; -use dm_core::manifest::{Manifest, ManifestEntry}; -use dm_core::{digest, Entry, Label, ReadFilter, TrustTier}; +use dm_core::manifest::{Manifest, ManifestEntry, ManifestPromotion, ManifestPromotions}; +use dm_core::{digest, Entry, Label, ReadFilter, Status, TrustTier}; use dm_index::{Hit, Index}; +use dm_ledger::PromotionRecord; use dm_store::Store; use std::collections::{BTreeMap, HashSet}; use std::path::{Path, PathBuf}; @@ -112,6 +116,125 @@ pub excerpts: usize, /// Where the bundle came from: `bundle:<exporter>/<domain>`. pub from: String, + /// The promotion records to restore, each moved to its entry as it stands here: the + /// entries written, and those skipped because they are here already, whose records + /// an earlier import may not have reached. From the same owner's bundle only. Another + /// owner's entries land raw, and the reasons for the tier they lost are left behind. + pub promotions: Vec<PromotionRecord>, +} + +/// Where a bundle carries the promotion records of its entries. +pub const PROMOTIONS_FILE: &str = "promotions.json"; + +/// Refuse a manifest path that is not a plain file name in the bundle's directory. +fn plain_name(path: &str) -> Result<(), Error> { + if path.contains(['/', '\\']) || path.starts_with('.') || path.is_empty() { + return Err(Error::Invalid(format!( + "manifest path {path:?} is not a plain file name" + ))); + } + Ok(()) +} + +/// Write the records about entries the manifest lists, at or below its bound, and list +/// them in it. Nothing is written when there are none. +async fn carry_promotions( + manifest: &mut Manifest, + records: Vec<PromotionRecord>, + dest: &Path, +) -> Result<(), Error> { + let carried: Vec<PromotionRecord> = records + .into_iter() + .filter(|r| { + r.spec.classification <= manifest.max_label + && manifest.entries.iter().any(|e| e.id == r.spec.entry) + }) + .collect(); + if carried.is_empty() { + return Ok(()); + } + let text = serde_json::to_string_pretty(&carried) + .map_err(|e| Error::Invalid(format!("promotion records: {e}")))?; + let path = dest.join(PROMOTIONS_FILE); + tokio::fs::write(&path, &text).await.map_err(io(&path))?; + manifest.set_promotions(ManifestPromotions { + path: PROMOTIONS_FILE.into(), + digest: digest(&text), + records: carried + .iter() + .map(|r| ManifestPromotion { + entry: r.spec.entry.clone(), + classification: r.spec.classification, + }) + .collect(), + })?; + Ok(()) +} + +/// The promotion records a bundle carries, checked before anything is written: a plain +/// file name, its digest, and one record per manifest line, about the entry and at the +/// label the line says. A bundle with one bad record is refused whole. +fn carried_promotions(src: &Path, manifest: &Manifest) -> Result<Vec<PromotionRecord>, Error> { + let Some(listed) = &manifest.promotions else { + return Ok(Vec::new()); + }; + plain_name(&listed.path)?; + let path = src.join(&listed.path); + let text = std::fs::read_to_string(&path).map_err(io(&path))?; + if digest(&text) != listed.digest { + return Err(Error::Invalid(format!( + "{} does not match its manifest digest", + listed.path + ))); + } + let records: Vec<PromotionRecord> = + serde_json::from_str(&text).map_err(|e| Error::Invalid(format!("{}: {e}", listed.path)))?; + let inside: Vec<(String, Label)> = records + .iter() + .map(|r| (r.spec.entry.clone(), r.spec.classification)) + .collect(); + manifest.check_promotions(&inside)?; + Ok(records) +} + +/// An entry of the bundle as it stands here after the import: written now, or here +/// already from before. Its records are moved to it. +struct Placed { + /// The id the bundle names it by. + bundle_id: String, + /// The id it has here. + here: String, + /// The digest of its text in the bundle. + bundle_digest: String, + /// The digest of its text here: not the bundle's, since landing sets the origin and + /// may change the id, the domain and the classification. + digest: String, + /// Its classification here, which an import may have raised to the domain's default. + classification: Label, +} + +/// The records to restore: those about an entry that is here, moved to it as it stands. +/// A record of the version the bundle carries names the version as it landed, so it +/// reads as landed; a record is at least as classified as its entry, as the ledger keeps +/// it. A record about an entry that is not here, refused or never written, is left out. +fn landed_promotions( + carried: Vec<PromotionRecord>, + placed: &[Placed], + into_domain: &str, +) -> Vec<PromotionRecord> { + carried + .into_iter() + .filter_map(|mut r| { + let p = placed.iter().find(|p| p.bundle_id == r.spec.entry)?; + r.spec.entry.clone_from(&p.here); + r.spec.domain = into_domain.to_string(); + if r.spec.new_digest == p.bundle_digest { + r.spec.new_digest.clone_from(&p.digest); + } + r.spec.classification = r.spec.classification.max(p.classification); + Some(r) + }) + .collect() } /// Where the binary keeps the bundles it serves, under the root's state. @@ -149,8 +272,9 @@ Ok(scoped) } -/// Export `hits`, the entries [`export_scope`] listed for `domain`, into `dest`. The -/// index is consulted before this is called and never held across the record reads, +/// Export `hits`, the entries [`export_scope`] listed for `domain`, into `dest`, with +/// the `promotions` of those exported at or below the bound; those of an entry exported +/// as its erased stub go without the why. The index is consulted before this is called and never held across the record reads, /// so the export can run on any executor thread. The manifest's word on each entry's /// label is the record's, read from the text written, not the index's, which can lag /// it: every receiver checks the manifest against the text, so an exporter must say @@ -166,14 +290,20 @@ max_label: Label, exported_by: &str, dest: &Path, + promotions: Vec<PromotionRecord>, ) -> Result<Exported, Error> { tokio::fs::create_dir_all(dest).await.map_err(io(dest))?; let mut manifest = Manifest::new(domain, exported_by, max_label); + let mut stubs = HashSet::new(); for hit in hits { let Some(text) = store.get_text(domain, &hit.id).await? else { continue; }; - let classification = Entry::parse(&text)?.front.classification; + let front = Entry::parse(&text)?.front; + if front.status == Some(Status::Erased) { + stubs.insert(hit.id.clone()); + } + let classification = front.classification; let file = format!("{}.md", hit.id); let path = dest.join(&file); tokio::fs::write(&path, &text).await.map_err(io(&path))?; @@ -184,6 +314,19 @@ classification, })?; } + // an erasure writes the stub before it scrubs the ledger: the records of an entry + // exported as its stub go without the why, as the scrub leaves them + let promotions = promotions + .into_iter() + .map(|mut r| { + if stubs.contains(&r.spec.entry) { + r.spec.why.clear(); + r.why_erased = true; + } + r + }) + .collect(); + carry_promotions(&mut manifest, promotions, dest).await?; let mpath = dest.join("manifest.yaml"); tokio::fs::write(&mpath, manifest.to_yaml()) .await @@ -205,6 +348,108 @@ } } +/// The id a bundle entry with `origin` has here, when it is here: this domain's own, +/// come back in a bundle it left in (`home` is the prefix of such an origin), or one a +/// previous import landed, known by its origin. +async fn present_here( + store: &Store, + index: &Index, + into_domain: &str, + origin: &str, + home: &str, +) -> Result<Option<String>, Error> { + let own = origin.strip_prefix(home); + if is_own(store, into_domain, own).await? { + return Ok(own.map(str::to_string)); + } + Ok(index.by_origin(into_domain, origin)?) +} + +/// How a bundle entry that is here already would have landed: what the import sets on an +/// entry it writes, so the text here can be compared with the bundle's. +struct Landing<'a> { + origin: &'a str, + here: &'a str, + default_classification: Label, +} + +/// Where a bundle entry that is here already stands, for its records to be moved to: +/// only when what is here is this very text, landed as the import lands one, origin, +/// domain, id and classification set. A record is of a version, and the bundle's version +/// of the entry need not be the one here: another export of the entry, or a bundle +/// written by hand, carries a text of its own, and its records stay out. `None` too for +/// an erased one, whose records keep nothing of it. +async fn placed_here( + store: &Store, + into_domain: &str, + entry: &Entry, + landing: &Landing<'_>, + bundle_digest: String, +) -> Result<Option<Placed>, Error> { + let Some(e) = store.get(into_domain, landing.here).await? else { + return Ok(None); + }; + if e.front.status == Some(dm_core::Status::Erased) { + return Ok(None); + } + let mut as_landed = entry.clone(); + as_landed.front.origin = Some(landing.origin.to_string()); + as_landed.front.domain = into_domain.to_string(); + as_landed.front.id = landing.here.to_string(); + as_landed.front.classification = as_landed + .front + .classification + .max(landing.default_classification); + let digest_here = digest(&e.render()); + if digest(&as_landed.render()) != digest_here { + return Ok(None); + } + Ok(Some(Placed { + bundle_id: entry.front.id.clone(), + here: landing.here.to_string(), + bundle_digest, + digest: digest_here, + classification: e.front.classification, + })) +} + +/// Pass one of an import: every path and digest, before anything is written, so a bundle +/// with one bad entry is refused whole and a partial import never happens. Text this +/// domain erased is known by its digest and is not an error, the bundle being older than +/// the erasure: it is named on `out` and left out. The manifest's word on the labels must +/// be the entries' own, erased ones included: a line or a highest that understates is +/// refused whole. Returns each entry to land with the digest of its text in the bundle, +/// which its records name. +fn check_entries( + src: &Path, + manifest: &Manifest, + erased: &HashSet<String>, + out: &mut Imported, +) -> Result<Vec<(Entry, String)>, Error> { + let mut checked: Vec<(Entry, String)> = Vec::with_capacity(manifest.entries.len()); + let mut inside: Vec<(String, Label)> = Vec::with_capacity(manifest.entries.len()); + for m in &manifest.entries { + plain_name(&m.path)?; + let path = src.join(&m.path); + let text = std::fs::read_to_string(&path).map_err(io(&path))?; + if digest(&text) != m.digest { + return Err(Error::Invalid(format!( + "{} does not match its manifest digest", + m.path + ))); + } + let entry = Entry::parse(&text)?; + inside.push((entry.front.id.clone(), entry.front.classification)); + if erased.contains(&m.digest) || erased.contains(&body_digest(&entry.body)) { + out.erased.push(m.id.clone()); + continue; + } + checked.push((entry, m.digest.clone())); + } + manifest.check(&inside)?; + Ok(checked) +} + /// Import a bundle from `src` into `into_domain`. `owner` is this record's owner; a bundle /// the same owner exported keeps its trust tiers, every other bundle lands at raw. An /// entry whose text's digest, or whose trimmed body's, is in `erased`, what this @@ -234,41 +479,12 @@ format_read(&manifest)?; let same_owner = manifest.exported_by == owner; - // Pass one: every path and digest, before anything is written. A bundle with one bad - // entry is refused whole, so a partial import never happens. Text this domain erased - // is known by its digest and is not an error: the bundle is older than the erasure. let mut out = Imported { from: format!("bundle:{}/{}", manifest.exported_by, manifest.domain), ..Imported::default() }; - let mut checked: Vec<Entry> = Vec::with_capacity(manifest.entries.len()); - let mut inside: Vec<(String, Label)> = Vec::with_capacity(manifest.entries.len()); - for m in &manifest.entries { - if m.path.contains(['/', '\\']) || m.path.starts_with('.') || m.path.is_empty() { - return Err(Error::Invalid(format!( - "manifest path {:?} is not a plain file name", - m.path - ))); - } - let path = src.join(&m.path); - let text = std::fs::read_to_string(&path).map_err(io(&path))?; - if digest(&text) != m.digest { - return Err(Error::Invalid(format!( - "{} does not match its manifest digest", - m.path - ))); - } - let entry = Entry::parse(&text)?; - inside.push((entry.front.id.clone(), entry.front.classification)); - if erased.contains(&m.digest) || erased.contains(&body_digest(&entry.body)) { - out.erased.push(m.id.clone()); - continue; - } - checked.push(entry); - } - // the manifest's word on the labels must be the entries' own, erased ones included, - // before anything lands: a line or a highest that understates is refused whole - manifest.check(&inside)?; + let checked = check_entries(src, &manifest, erased, &mut out)?; + let carried = carried_promotions(src, &manifest)?; // Pass two: land what is new. An entry is already here when a previous import landed // it, known by its origin, or when it is this domain's own, come back in a bundle it @@ -277,16 +493,27 @@ // this domain holds is skipped, not landed as a version, since the record's versions // are written here and a bundle is a copy. let home = format!("bundle:{owner}/{into_domain}/"); - for mut entry in checked { + let mut placed: Vec<Placed> = Vec::with_capacity(checked.len()); + for (mut entry, bundle_digest) in checked { let origin = entry.front.origin.clone().unwrap_or_else(|| { format!( "bundle:{}/{}/{}", manifest.exported_by, manifest.domain, entry.front.id ) }); - if is_own(store, into_domain, origin.strip_prefix(&home)).await? - || index.by_origin(into_domain, &origin)?.is_some() - { + if let Some(here) = present_here(store, index, into_domain, &origin, &home).await? { + // its records may still be owed, when the import that landed it stopped + // before they were restored: only when what is here is this very text as + // the import would land it, since a record is of a version + if same_owner { + let landed = Landing { + origin: &origin, + here: &here, + default_classification, + }; + placed + .extend(placed_here(store, into_domain, &entry, &landed, bundle_digest).await?); + } out.skipped.push(entry.front.id.clone()); continue; } @@ -315,8 +542,18 @@ index.upsert(&old)?; } } - out.landed.insert(bundle_id, entry.front.id.clone()); + out.landed.insert(bundle_id.clone(), entry.front.id.clone()); out.written.push(entry.front.id.clone()); + placed.push(Placed { + bundle_id, + here: entry.front.id.clone(), + bundle_digest, + digest: digest(&entry.render()), + classification: entry.front.classification, + }); + } + if same_owner { + out.promotions = landed_promotions(carried, &placed, into_domain); } Ok(out) } @@ -355,6 +592,7 @@ Label::Restricted, "memory:test:me", dest.path(), + Vec::new(), ) .await .unwrap(); @@ -368,4 +606,64 @@ .check(&[(entry.front.id.clone(), Label::Restricted)]) .unwrap(); } + + /// A record about an entry that the export lists but does not write, since the + /// record no longer holds its text, stays home even within the bound: a bundle + /// carries records of its entries alone. + #[tokio::test] + async fn a_record_of_an_entry_left_home_is_not_carried_even_within_the_bound() { + let dir = tempfile::tempdir().unwrap(); + let store = Store::open_dir(dir.path()).unwrap(); + let entry = Entry::parse(HELD).unwrap(); + store.put(&entry).await.unwrap(); + let hit = |id: &str| Hit { + id: id.into(), + domain: "dom-a".into(), + entry_type: entry.front.entry_type, + trust_tier: entry.front.trust_tier, + classification: Label::Public, + title: "Held".into(), + summary: "Held.".into(), + score: 0.0, + review_due: false, + }; + let record = |id: &str| PromotionRecord { + spec: dm_ledger::PromotionSpec { + domain: "dom-a".into(), + entry: id.into(), + tier: TrustTier::Curated, + by: "agent:librarian".into(), + author: "human:test:me".into(), + why: format!("why {id} was promoted"), + classification: Label::Public, + digest: "d0".into(), + new_digest: "d1".into(), + }, + why_erased: false, + at: time::OffsetDateTime::UNIX_EPOCH, + }; + let dest = tempfile::tempdir().unwrap(); + let ex = export( + &store, + vec![hit(&entry.front.id), hit("mem-a-gone0001")], + "dom-a", + Label::Restricted, + "memory:test:me", + dest.path(), + vec![record(&entry.front.id), record("mem-a-gone0001")], + ) + .await + .unwrap(); + let carried = ex + .manifest + .promotions + .as_ref() + .expect("the held entry's record is carried"); + assert_eq!(carried.records.len(), 1, "{carried:?}"); + assert_eq!(carried.records[0].entry, entry.front.id); + let text = std::fs::read_to_string(dest.path().join(PROMOTIONS_FILE)).unwrap(); + assert!(!text.contains("mem-a-gone0001"), "{text}"); + let records = carried_promotions(dest.path(), &ex.manifest).unwrap(); + assert_eq!(records.len(), 1); + } }
diff --git a/crates/server/src/fabric.rs b/crates/server/src/fabric.rs index a2e5262..5912f17 100644 --- a/crates/server/src/fabric.rs +++ b/crates/server/src/fabric.rs
@@ -63,49 +63,75 @@ .to_string() } -type Ranks = HashMap<String, (i64, String)>; +/// A rank the rubric gave a task: the rank, why, and every task it came from, however +/// far back, so a report can leave out a rank lent by a task its reader may not see. +#[derive(Clone, Debug)] +struct Rank { + rank: i64, + why: String, + from: Vec<String>, +} + +type Ranks = HashMap<String, Rank>; + +/// A task's current rank and the tasks it came from: its rank this cycle if it has one, +/// else, for a claimed task, the priority it holds. +fn current(ranks: &Ranks, claimed: &[Task], id: &str) -> Option<(i64, Vec<String>)> { + ranks.get(id).map(|r| (r.rank, r.from.clone())).or_else(|| { + claimed + .iter() + .find(|c| c.id == id) + .map(|c| (c.spec.priority, Vec::new())) + }) +} + +/// The best rank `t` may take from what it blocks and from its parent, if better than +/// what it has: the rank, why, and where it came from. +fn lent(t: &Task, open: &[Task], claimed: &[Task], rubric: &Rubric, ranks: &Ranks) -> Option<Rank> { + let mut best = ranks.get(&t.id).map_or(0, |r| r.rank); + let mut lent = None; + for d in open + .iter() + .chain(claimed) + .filter(|d| d.spec.depends_on.contains(&t.id)) + { + if let Some((dr, from)) = current(ranks, claimed, &d.id) { + if dr + rubric.unblock_weight > best { + best = dr + rubric.unblock_weight; + lent = Some((format!("unblocks {}", d.id), d.id.clone(), from)); + } + } + } + let parent = t + .spec + .origin + .as_deref() + .and_then(|o| o.strip_prefix("task:")); + if let Some((pr, from)) = parent.and_then(|p| current(ranks, claimed, p)) { + if pr > best { + best = pr; + let p = parent.unwrap_or_default().to_string(); + lent = Some((format!("a child of {p}"), p, from)); + } + } + lent.map(|(why, by, mut from)| { + from.insert(0, by); + Rank { + rank: best, + why, + from, + } + }) +} /// Raise blockers above what they block and children to their parents, until nothing /// moves; bounded by the number of tasks, since each pass settles at least one edge. fn propagate(open: &[Task], claimed: &[Task], rubric: &Rubric, ranks: &mut Ranks) { - let current = |ranks: &Ranks, id: &str| -> Option<i64> { - ranks - .get(id) - .map(|(r, _)| *r) - .or_else(|| claimed.iter().find(|c| c.id == id).map(|c| c.spec.priority)) - }; for _ in 0..=open.len() { let mut moved = false; for t in open { - let mut best = ranks.get(&t.id).map_or(0, |(r, _)| *r); - let mut why = None; - for d in open - .iter() - .chain(claimed) - .filter(|d| d.spec.depends_on.contains(&t.id)) - { - if let Some(dr) = current(ranks, &d.id) { - if dr + rubric.unblock_weight > best { - best = dr + rubric.unblock_weight; - why = Some(format!("unblocks {}", d.id)); - } - } - } - if let Some(parent) = t - .spec - .origin - .as_deref() - .and_then(|o| o.strip_prefix("task:")) - { - if let Some(pr) = current(ranks, parent) { - if pr > best { - best = pr; - why = Some(format!("a child of {parent}")); - } - } - } - if let Some(why) = why { - ranks.insert(t.id.clone(), (best, why)); + if let Some(r) = lent(t, open, claimed, rubric, ranks) { + ranks.insert(t.id.clone(), r); moved = true; } } @@ -115,44 +141,58 @@ } } -/// Curation's floor: when its share of the completed work is under the floor, the best -/// curation task waiting goes to the head of the queue. Returns the share. -fn apply_floor(open: &[Task], completed: &[Task], rubric: &Rubric, ranks: &mut Ranks) -> f64 { - if completed.is_empty() { - return 1.0; - } +/// The share of completed work that is curation; with nothing completed, all of it is. +fn maintenance_share<'a>(completed: impl Iterator<Item = &'a Task>) -> f64 { + let (curation, all) = completed.fold((0_usize, 0_usize), |(c, n), t| { + (c + usize::from(maintenance(t)), n + 1) + }); #[allow(clippy::cast_precision_loss)] // counts of tasks - let share = completed.iter().filter(|t| maintenance(t)).count() as f64 / completed.len() as f64; - if share >= rubric.maintenance_floor { - return share; + if all == 0 { + 1.0 + } else { + curation as f64 / all as f64 } - let rank_of = |t: &Task, ranks: &Ranks| ranks.get(&t.id).map_or(t.spec.priority, |(r, _)| *r); +} + +/// Curation's floor: when its share of the completed work is under the floor, the best +/// curation task waiting goes to the head of the queue. +fn apply_floor(open: &[Task], completed: &[Task], rubric: &Rubric, ranks: &mut Ranks) { + let share = maintenance_share(completed.iter()); + if share >= rubric.maintenance_floor { + return; + } + let rank_of = |t: &Task, ranks: &Ranks| ranks.get(&t.id).map_or(t.spec.priority, |r| r.rank); let top_delivery = open .iter() .filter(|t| !maintenance(t)) - .map(|t| rank_of(t, ranks)) - .max(); + .max_by_key(|t| rank_of(t, ranks)); // the floor is for curation an agent can do; an alarm for the owner waits for a person let best_curation = open .iter() .filter(|t| maintenance(t) && !t.spec.requirements.iter().any(|r| r == "human")) .max_by_key(|t| rank_of(t, ranks)); if let (Some(top), Some(c)) = (top_delivery, best_curation) { - if rank_of(c, ranks) <= top { + let top_rank = rank_of(top, ranks); + if rank_of(c, ranks) <= top_rank { + let mut from = ranks + .get(&top.id) + .map(|r| r.from.clone()) + .unwrap_or_default(); + from.insert(0, top.id.clone()); ranks.insert( c.id.clone(), - ( - top + 1, - format!( + Rank { + rank: top_rank + 1, + why: format!( "curation's floor: {:.0}% of the completed work, under {:.0}%", share * 100.0, rubric.maintenance_floor * 100.0 ), - ), + from, + }, ); } } - share } /// Filings held back for requirements their filer does not hold, per creator, from the @@ -194,7 +234,11 @@ .unwrap_or_else(|| (Rubric::default(), "default".into())) } - /// Rank a domain's open tasks by the rubric, once: the paper's cycle. + /// Rank a domain's open tasks by the rubric, once: the paper's cycle. Every open task + /// is ranked, whatever the session's handling; the changes reported are those within + /// it, read, and only those whose rank came from nothing above it. The report hides + /// the reason, not the effect: the rank a hidden task lends is stored, and a listing + /// shows the priority it set, from which a hidden task's rank can be read. /// /// # Errors /// @@ -215,43 +259,74 @@ let age = (now - t.created).whole_days() * rubric.age_weight_per_day; ranks.insert( t.id.clone(), - ( - filed + weight + age, - format!("filed at {filed}, origin {weight:+}, age {age:+}"), - ), + Rank { + rank: filed + weight + age, + why: format!("filed at {filed}, origin {weight:+}, age {age:+}"), + from: Vec::new(), + }, ); } propagate(&open, &claimed, &rubric, &mut ranks); - let share = apply_floor(&open, &completed, &rubric, &mut ranks); + apply_floor(&open, &completed, &rubric, &mut ranks); let mut changed = Vec::new(); for t in &open { - let (to, why) = ranks - .get(&t.id) - .cloned() - .unwrap_or((t.spec.priority, String::new())); - if to != t.spec.priority { - l.rescore(&t.id, to, &why).await?; - changed.push(Change { + let Some(r) = ranks.get(&t.id).filter(|r| r.rank != t.spec.priority) else { + continue; + }; + l.rescore(&t.id, r.rank, &r.why).await?; + changed.push(( + Change { id: t.id.clone(), title: t.spec.title.clone(), from: t.spec.priority, - to, - why, - }); + to: r.rank, + why: r.why.clone(), + }, + r.from.clone(), + )); + } + // the queue is ranked whole, since rank lent by a task this session may not see + // is rank all the same; the session is told only of a change to a task it may + // see whose rank came from nothing it may not see, however far back, since the + // reason names the lender and the new rank gives its rank away. This hides the + // reason, not the effect: the stored priority is the same rank, and tasks() shows + // it, so a hidden task's rank is still readable from what it lifted + let handling = self.handling()?; + let visible: HashMap<&str, &Task> = open + .iter() + .chain(&claimed) + .filter(|t| self.sees(t, handling)) + .map(|t| (t.id.as_str(), t)) + .collect(); + let changed: Vec<Change> = changed + .into_iter() + .filter(|(c, from)| { + visible.contains_key(c.id.as_str()) + && from.iter().all(|f| visible.contains_key(f.as_str())) + }) + .map(|(c, _)| c) + .collect(); + for c in &changed { + if let Some(t) = visible.get(c.id.as_str()) { + self.observe_task(t); } } Ok(Rescore { domain, rubric: from, changed, - maintenance_share: share, + // the floor weighs all completed work; the report counts only what it may see + maintenance_share: maintenance_share( + completed.iter().filter(|t| self.sees(t, handling)), + ), floor: rubric.maintenance_floor, }) } /// How tasks flow in a domain, the paper's fabric telemetry: by family, how many /// are in each status; why tasks were invalidated; and the requirements of waiting - /// tasks that no completed task ever carried, a fleet capability gap. + /// tasks that no completed task ever carried, a fleet capability gap. Only tasks + /// within the session's handling are counted, and each counted is read. /// /// # Errors /// @@ -270,7 +345,7 @@ Status::Completed, Status::Invalidated, ] { - for t in l.list(&domain, status).await? { + for t in self.read_tasks(l.list(&domain, status).await?)? { *by_family .entry(family(&t)) .or_default()
diff --git a/crates/server/src/lib.rs b/crates/server/src/lib.rs index 9635646..77f1215 100644 --- a/crates/server/src/lib.rs +++ b/crates/server/src/lib.rs
@@ -30,7 +30,7 @@ use std::path::{Path, PathBuf}; use std::pin::Pin; use std::sync::atomic::{AtomicUsize, Ordering}; -use std::sync::{Mutex, MutexGuard, OnceLock, PoisonError, Weak}; +use std::sync::{Arc, Mutex, MutexGuard, OnceLock, PoisonError, Weak}; use time::{Date, Duration, OffsetDateTime}; use tokio::sync::Mutex as AsyncMutex; @@ -186,6 +186,20 @@ }, } +/// A promotion as [`Memory::promotions`] serves it: the ledger's record, and whether the +/// version it writes is in the record now. +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +pub struct PromotionView { + /// What the ledger keeps. + #[serde(flatten)] + pub record: dm_ledger::PromotionRecord, + /// Whether the version this promotion writes is the entry's current version or one + /// in its history. False for a promotion whose new version was never written. Two + /// promoters of one version write the same text, so this says the version landed, + /// not whose write it was. + pub landed: bool, +} + /// The record's policy entries, active, in every granted domain, at any tier: read /// through the index and the files the record keeps, so the policy can be compiled /// where no runtime is at hand. Policy is enforced whatever a session may read, so the @@ -236,10 +250,12 @@ manifest: Vec<String>, session_id: String, store: Store, - index: AsyncMutex<Index>, - ledger: AsyncMutex<Ledger>, + /// The handles a long-lived surface shares between its requests ([`Memory::for_request`]): + /// the index, the ledger, the read log and the policy in force. + index: Arc<AsyncMutex<Index>>, + ledger: Arc<AsyncMutex<Ledger>>, session: Mutex<Session>, - reads: reads::ReadLog, + reads: Arc<reads::ReadLog>, /// The recall deputy, held weakly: it lives as long as the launcher that built it. deputy: OnceLock<Weak<dyn recall::Deputy>>, /// Whether this session has issued two gets inside one turn. @@ -253,25 +269,27 @@ /// Entries this session wrote, and how many of them were handoffs. written: (AtomicUsize, AtomicUsize), /// The checkable perimeter: admission and the gates, as Rego over the grants. - policy: std::sync::RwLock<std::sync::Arc<Policy>>, + policy: Arc<std::sync::RwLock<std::sync::Arc<Policy>>>, /// The task this session holds, when the launcher runs it on one: what it files - /// from is judged against that task's requirements. - held: OnceLock<String>, + /// from is judged against that task's requirements. Shared by the views of a + /// session ([`Memory::for_request`]), as every count below that is kept per session. + held: Arc<OnceLock<String>>, /// How many entries this session has superseded: the blast-radius gate's count. - superseded: AtomicUsize, + superseded: Arc<AtomicUsize>, /// What this session may read, as the policy computed it once and the index applies /// inside every query. scope: OnceLock<ReadFilter>, /// How many decision contexts this session has kept. - contexts_kept: AtomicUsize, + contexts_kept: Arc<AtomicUsize>, /// The tools this session may call, when the launcher hosts them. tools: OnceLock<std::sync::Arc<dyn sources::Tools>>, /// The tape this session is re-enacting, when it is a re-run. playback: OnceLock<std::sync::Arc<tape::Playback>>, /// This session's own tape, as it is kept. - recording: Mutex<tape::Recording>, - /// When this service last stamped a write; see [`next_stamp`]. - last_stamp: Mutex<OffsetDateTime>, + recording: Arc<Mutex<tape::Recording>>, + /// When this service last stamped a write; see [`next_stamp`]. Shared with the + /// views [`Memory::for_request`] makes, which write for the same service. + last_stamp: Arc<Mutex<OffsetDateTime>>, } fn lock<T>(m: &Mutex<T>) -> MutexGuard<'_, T> { @@ -310,6 +328,11 @@ /// with a handoff, or still open. A resumed run asks for what it asked for before; /// the answer is the earlier task, not another. pub existing: Vec<Task>, + /// How many items were not filed because the chain holds each of them in a task + /// this session may not read: open work, or work of this parent that is done or + /// dropped and that the ledger would hand back as the same task. Nothing of those + /// tasks is named. The count is all the answer says of them. + pub withheld: usize, /// Resumes the parent when the new children are done; none when nothing was new. pub continuation: Option<Task>, } @@ -399,26 +422,74 @@ author: grants.owner.clone(), grants, manifest: Vec::new(), - reads, + reads: Arc::new(reads), session_id, store, - index: AsyncMutex::new(index), - ledger: AsyncMutex::new(ledger), + index: Arc::new(AsyncMutex::new(index)), + ledger: Arc::new(AsyncMutex::new(ledger)), session: Mutex::new(session), deputy: OnceLock::new(), fan_out: Mutex::new(recall::FanOut::default()), cache: Mutex::new(recall::Cache::default()), projection: None, written: (AtomicUsize::new(0), AtomicUsize::new(0)), - policy: std::sync::RwLock::new(std::sync::Arc::new(policy)), - held: OnceLock::new(), - superseded: AtomicUsize::new(0), + policy: Arc::new(std::sync::RwLock::new(std::sync::Arc::new(policy))), + held: Arc::new(OnceLock::new()), + superseded: Arc::new(AtomicUsize::new(0)), scope: OnceLock::new(), - contexts_kept: AtomicUsize::new(0), + contexts_kept: Arc::new(AtomicUsize::new(0)), tools: OnceLock::new(), playback: OnceLock::new(), - recording: Mutex::new(tape::Recording::default()), - last_stamp: Mutex::new(OffsetDateTime::UNIX_EPOCH), + recording: Arc::new(Mutex::new(tape::Recording::default())), + last_stamp: Arc::new(Mutex::new(OffsetDateTime::UNIX_EPOCH)), + }) + } + + /// This session, for one request of a surface that lives longer than a request + /// does, as the board does. The record, the index, the ledger, the policy in force, + /// the read log and the session id are this session's, shared: a read here and a + /// write in a later request carry the same session id, which is what joins them + /// for lineage. The label is the request's own: it starts at the bottom and rises + /// with what this request reads, so what one request lists does not classify what a + /// later one files. The read scope is this session's too, asked of the policy and + /// logged once, here, rather than once per request. The grants, the author, the + /// manifest, the tools and the projection carry over. What is kept per session and + /// keyed by the session id is shared too, so a view counts with its session: the + /// held task, the supersede count the blast-radius gate reads, the tape and the + /// number of kept contexts. What this view wrote starts at zero. + /// + /// # Errors + /// + /// When the policy fails to give the read scope. + pub fn for_request(&self) -> Result<Memory, Error> { + let scope = self.session_filter()?; + let mut session = self.grants.session(); + session.handling = session.handling.min(scope.handling); + Ok(Memory { + root: self.root.clone(), + grants: self.grants.clone(), + author: self.author.clone(), + manifest: self.manifest.clone(), + session_id: self.session_id.clone(), + store: self.store.clone(), + index: Arc::clone(&self.index), + ledger: Arc::clone(&self.ledger), + session: Mutex::new(session), + reads: Arc::clone(&self.reads), + deputy: self.deputy.clone(), + fan_out: Mutex::new(recall::FanOut::default()), + cache: Mutex::new(recall::Cache::default()), + projection: self.projection.clone(), + written: (AtomicUsize::new(0), AtomicUsize::new(0)), + policy: Arc::clone(&self.policy), + held: Arc::clone(&self.held), + superseded: Arc::clone(&self.superseded), + scope: OnceLock::from(scope), + contexts_kept: Arc::clone(&self.contexts_kept), + tools: self.tools.clone(), + playback: self.playback.clone(), + recording: Arc::clone(&self.recording), + last_stamp: Arc::clone(&self.last_stamp), }) } @@ -1020,21 +1091,42 @@ /// Promote an entry: to curated at once where the policy allows it, by anyone but /// its author; to verified only by proposal, which files a task for a person to /// approve with [`Memory::approve`]. The entry is never edited in place: a - /// promotion is a new version of it, the old one kept in its history. + /// promotion is a new version of it, the old one kept in its history. Every + /// promotion says why, and the why is kept on the ledger, never in the entry nor in + /// a task: a promotion is recorded there before its new version is written or its + /// proposal filed, and [`Memory::promotions`] reads it back. A proposal's task names + /// the entry, the proposer and the author, and points there. /// /// # Errors /// /// [`Error::NotFound`] for an entry this session cannot read; for one it could read /// that is no longer served, [`Error::Moved`] or [`Error::Gone`] as from - /// [`Memory::get`]; [`Error::Denied`] when the policy refuses, the entry is not live, - /// the entry is already at or above the tier, or a proposal of this version is open - /// by another principal or was closed. + /// [`Memory::get`]; [`Error::Denied`] for an empty why, when the policy refuses, the + /// entry is not live, the entry is already at or above the tier, or a proposal of + /// this version is open by another principal or was closed. pub async fn promote(&self, id: &str, to: TrustTier, why: &str) -> Result<Promotion, Error> { + let why = why.trim(); + if why.is_empty() { + return Err(Error::Denied( + "a promotion says why: what was checked, and against what".into(), + )); + } let f = self.session_filter()?; if self.index.lock().await.meta(id, &f)?.is_none() { return Err(self.not_served(id, None).await); } let (_, entry) = self.find(id).await?; + self.promote_entry(entry, to, why).await + } + + /// [`Memory::promote`] from the entry as the record held it when it was read. + async fn promote_entry( + &self, + entry: Entry, + to: TrustTier, + why: &str, + ) -> Result<Promotion, Error> { + let id = entry.front.id.as_str(); // the index can lag the record: the record says whether it is live if let Some(status) = entry.front.status { return Err(not_live(id, status)); @@ -1054,29 +1146,117 @@ if !d.allow { return Err(Error::Denied(d.reason)); } - if to != TrustTier::Verified { - return self.set_tier(entry, to).await; + if to == TrustTier::Verified { + return self.propose(entry, why).await; } - self.propose(entry, why).await + // the record first: a promotion the ledger cannot keep is not made, and one whose + // new version fails is made again with the record it already has + let record = self.promotion_spec(&entry, to, why); + { + let mut l = self.ledger.lock().await; + l.record_promotion(&record).await?; + self.scrub_if_erased(&mut l, &record).await?; + } + self.set_tier(entry, to).await + } + + /// The entry of `record`, read from the record again once `record` is on the ledger. + /// An erasure that ran whole since the entry was read scrubbed the ledger before the + /// record was there, and does not run again: the stub, which an erasure writes before + /// it scrubs, says so, and the scrub runs here instead, under the lock `l` the record + /// was written under. An erasure that writes its stub after this read scrubs the record + /// itself. + /// + /// # Errors + /// + /// [`Error::Denied`] for an erased entry, once its whys are out; a store or ledger + /// error. + async fn scrub_if_erased( + &self, + l: &mut Ledger, + record: &dm_ledger::PromotionSpec, + ) -> Result<(), Error> { + if self.scrub_erased(l, record).await? { + return Err(not_live(&record.entry, Status::Erased)); + } + Ok(()) + } + + /// [`Memory::scrub_if_erased`] that says whether the entry was erased, for a caller + /// that goes on after it. Returns true when it was, and its whys are out. + /// + /// # Errors + /// + /// A store or ledger error. When another connection holds the ledger's log, the + /// rows are rewritten and the log still holds the why: the erasure's receipt goes + /// back to unfinished, since running this operation again finds the entry erased and + /// does not scrub, and the [`dm_ledger::Error::Held`] says to run the erasure again. + async fn scrub_erased( + &self, + l: &mut Ledger, + record: &dm_ledger::PromotionSpec, + ) -> Result<bool, Error> { + let (domain, id) = (record.domain.as_str(), record.entry.as_str()); + let erased = self + .store + .get(domain, id) + .await? + .is_some_and(|e| e.front.status == Some(Status::Erased)); + if !erased { + return Ok(false); + } + match l + .scrub_entry(domain, id, &proposal_origin(id), &proposal_redaction(id)) + .await + { + Ok(_) => Ok(true), + Err(dm_ledger::Error::Held(what)) => { + self.reopen_erasure(domain, id).await?; + Err(dm_ledger::Error::Held(format!( + "{what}, after {id} was erased; run `erase {id}` again to finish its erasure" + )) + .into()) + } + Err(e) => Err(e.into()), + } + } + + /// Put a finished erasure's receipt back to unfinished, so the next + /// [`Memory::erase`] runs its steps again rather than refusing. + async fn reopen_erasure(&self, domain: &str, id: &str) -> Result<(), Error> { + if let Some(mut receipt) = self.store.receipt(domain, id).await? { + if receipt.is_finished() { + receipt.unresolved.push(erasure::UNFINISHED.into()); + self.store.put_receipt(&receipt).await?; + } + } + Ok(()) } /// File a proposal to verify the entry as it now stands. The proposal names the /// version, `verified:<digest>`, and the ledger keeps one task for each entry and /// version: a new version is a new proposal, and a proposal of an older one closes as - /// stale. The slot is checked before anything is admitted, under the ledger's lock. + /// stale. The slot is checked before anything is admitted, under the ledger's lock, + /// and the proposal is recorded as a promotion only when the slot is free: one + /// refused, or one handed its own open task back, records nothing. The record and + /// the task name one version, the record's digest, so what [`Memory::approve`] reads + /// from the task and what [`Memory::promotions`] shows are the same proposal. The + /// record is written before the task, as a direct promotion's is before its version: + /// a proposal the ledger cannot keep is not filed, and one whose task fails is filed + /// again with the record it already has. async fn propose(&self, entry: Entry, why: &str) -> Result<Promotion, Error> { let id = entry.front.id.clone(); - let origin = format!("promote:{id}"); - let event = verify_event(&entry); + let origin = proposal_origin(&id); + let record = self.promotion_spec(&entry, TrustTier::Verified, why); + let event = verify_event(&record.digest); let spec = self.stamp( TaskSpec { domain: entry.front.domain.clone(), - title: format!( - "Approve: verified for {}", - entry.front.title.as_deref().unwrap_or(&id) - ), + // by id, not title: an erasure of the entry takes its title with it, and + // the task then has nothing of the entry to lose but the id a stub keeps + title: proposal_title(&id), body: format!( - "`{}` proposes that `{id}`, written by `{}`, be promoted to verified.\n\n{why}\n\nRead it with `amem get {id}`. Approve with `amem approve <this task>` if it holds; a person who did not propose it approves, and an agent never does. Invalidate this task to decline.", + "`{}` proposes that `{id}`, written by `{}`, be promoted to verified.\n\nRead it with `amem get {id}` and judge it first. Then read why it was proposed, on the ledger: `amem promotions {id}`. Approve with `amem approve <this task>` if it holds; a person who did not propose it approves, and an agent never does. Invalidate this task to decline.", self.author, entry.front.author ), requirements: vec!["human".into()], @@ -1092,9 +1272,16 @@ )?; let domain = spec.domain.clone(); let mut l = self.ledger.lock().await; + if let Some(task) = l.by_origin(&domain, &origin, &event).await? { + return self.slot_taken(task); + } + l.record_promotion(&record).await?; + self.scrub_if_erased(&mut l, &record).await?; let created = match l.create(spec).await { Ok(c) => c, - // another process filed the same slot between the look and the insert + // another process filed the same slot between the look and the insert; its + // task is what this proposal gets, and the record just written stands for a + // proposal that was made and merged Err(e) => match l.by_origin(&domain, &origin, &event).await? { Some(task) => Created { task, @@ -1109,8 +1296,11 @@ let task = self .admit_per_policy(&mut l, created.task, Filing::Own, None) .await?; + // a proposal this session may not see is left for a session that may: closing it + // would act on work it cannot read + let handling = self.handling()?; for t in l.with_origin(&domain, &origin).await? { - if t.id != task.id && !t.status.is_terminal() { + if t.id != task.id && !t.status.is_terminal() && self.sees(&t, handling) { l.invalidate( &t.id, &format!( @@ -1128,11 +1318,20 @@ } /// What a proposal of a version gets when that version has a task already: its own - /// open task back. Another principal's open task is not joined, since + /// open task back, when this session may see it, and a refusal that names nothing of + /// it when not. Another principal's open task is not joined, since /// [`Memory::approve`] knows only who filed it and a proposer never approves its /// own; a closed task, declined or done, is not proposed again until the entry /// changes, and is never handed back as if it had been filed. fn slot_taken(&self, t: Task) -> Result<Promotion, Error> { + // a task this session may not see is not handed out, and the answer names nothing + // of it: not its id, its status, nor who filed it + if !self.sees(&t, self.handling()?) { + return Err(Error::Denied( + "this version of the entry is proposed already, in a task this session may not read" + .into(), + )); + } if t.status.is_terminal() { return Err(Error::Denied(format!( "task {} closed the proposal to verify this version of the entry ({:?}{}); it is not proposed again until the entry changes", @@ -1160,11 +1359,11 @@ /// # Errors /// /// [`Error::Denied`] for a task that is not a live promotion proposal, an entry - /// that is not live, or an approval the policy refuses; a ledger error for a task - /// another holds, or a store error. + /// that is not live or that changed since it was proposed, or an approval the policy + /// refuses; a ledger error for a task another holds, or a store error. pub async fn approve(&self, task_id: &str) -> Result<Promotion, Error> { let task = self.ledger.lock().await.get(task_id).await?; - self.task_in_grants(&task)?; + self.check_task(&task)?; let (id, entry) = self.proposal_entry(&task).await?; let d = self.decide_promotion( &entry, @@ -1197,6 +1396,7 @@ &self.author, Duration::minutes(5), &task.spec.requirements, + self.handling()?, ) .await?; Ok(()) @@ -1274,6 +1474,106 @@ Ok((id.to_string(), entry)) } + /// Why each promotion of an entry was made, oldest first: who moved which version to + /// which tier, the reason they gave, and whether that version reached the record. A + /// proposal to verified is here from when it is made, and lands when a person + /// approves it. Reading it is reading the entry: only for an entry this session + /// may read, replaced or demoted ones included and erased ones never; the session's + /// label rises to the entry's classification as a [`Memory::get`] would raise it, and + /// to each record's; the read log says the entry was read. A record above what the + /// session may handle is left out. + /// + /// # Errors + /// + /// [`Error::NotFound`] for an entry this session cannot read, or a store, index or + /// ledger error. + pub async fn promotions(&self, id: &str) -> Result<Vec<PromotionView>, Error> { + let mut f = self.session_filter()?; + // why an entry was promoted outlives the version: replaced ones are read too + f.include_inactive = true; + let Some(meta) = self.index.lock().await.meta(id, &f)? else { + return Err(Error::NotFound(id.to_string())); + }; + let domain = meta.hit.domain; + let entry = self + .store + .get(&domain, id) + .await? + .ok_or_else(|| Error::NotFound(id.to_string()))?; + if entry.front.status == Some(Status::Erased) + || !lock(&self.session).observe(entry.front.classification) + { + return Err(Error::NotFound(id.to_string())); + } + let records = self.ledger.lock().await.promotions(&domain, id).await?; + let versions: HashSet<String> = self + .store + .digests(&domain, id) + .await? + .into_iter() + .map(|v| v.digest) + .collect(); + let readable = self.readable_promotions(records, &versions); + let tokens = readable + .iter() + .map(|p| dm_core::approx_tokens(&p.record.spec.why)) + .sum(); + self.reads.served_ids( + "promotions", + None, + readable.len(), + tokens, + false, + None, + &[id.to_string()], + ); + Ok(readable) + } + + /// The records this session may handle, each raising its label, and whether the + /// version each writes is among `versions`. + fn readable_promotions( + &self, + records: Vec<dm_ledger::PromotionRecord>, + versions: &HashSet<String>, + ) -> Vec<PromotionView> { + let mut session = lock(&self.session); + let handling = session.handling; + let mut readable = Vec::new(); + for r in records { + if r.spec.classification <= handling { + session.observe(r.spec.classification); + let landed = versions.contains(&r.spec.new_digest); + readable.push(PromotionView { record: r, landed }); + } + } + readable + } + + /// The record of a promotion, on the ledger and not in the entry: the version + /// promoted, by whom, and why, so an auditor finds the reason beside the change. The + /// ledger keeps it once per promoter and version, so a retry adds nothing, and a + /// failed promotion's record never stands in for another promoter's. It carries the + /// digest of the version it writes as well as the one it promotes. It is classified as + /// the entry is, or as this session's label if that is higher, since the why is the + /// session's own text. + fn promotion_spec(&self, entry: &Entry, to: TrustTier, why: &str) -> dm_ledger::PromotionSpec { + // the version set_tier will write, so a reader can tell whether it was + let mut promoted = entry.clone(); + promoted.front.trust_tier = to; + dm_ledger::PromotionSpec { + domain: entry.front.domain.clone(), + entry: entry.front.id.clone(), + tier: to, + by: self.author.clone(), + author: entry.front.author.clone(), + why: why.to_string(), + classification: entry.front.classification.max(self.label()), + digest: dm_core::digest(&entry.render()), + new_digest: dm_core::digest(&promoted.render()), + } + } + fn decide_promotion( &self, entry: &Entry, @@ -1560,13 +1860,18 @@ .len(), ) }; + // a count, so nothing is read and the label stays; what it counts is only what + // the session may see + let handling = self.handling()?; let waiting = self .ledger .lock() .await .list(&domain, dm_ledger::Status::Admitted) .await? - .len(); + .iter() + .filter(|t| self.sees(t, handling)) + .count(); self.observe(&handoffs); self.observe(&decisions); self.observe(&conventions); @@ -1623,11 +1928,71 @@ // ---- tasks ---- - fn task_in_grants(&self, task: &Task) -> Result<(), Error> { - if self.grants.may_write(&task.spec.domain) { + /// The most this session may read: the grants, the launcher's cap on them, and the + /// policy's read scope, whichever is lowest. + fn handling(&self) -> Result<Label, Error> { + self.session_filter()?; + Ok(lock(&self.session).handling) + } + + /// Whether this session may see a task: one in a granted domain, classified within + /// `handling`. A task it may not see is left out of every result and every count, as + /// an entry above its handling is. + fn sees(&self, task: &Task, handling: Label) -> bool { + self.grants.may_write(&task.spec.domain) && task.spec.classification <= handling + } + + /// Whether this session may act on a named task. A task it may not see gets the + /// answer a missing task gets, so the answer says nothing that a listing would not. + /// Nothing is read yet: an operation that goes on to fail hands nothing out, and + /// [`Memory::observe_task`] is for the task it does hand out. + fn check_task(&self, task: &Task) -> Result<(), Error> { + if self.sees(task, self.handling()?) { Ok(()) } else { - Err(Error::NotFound(task.id.clone())) + Err(dm_ledger::Error::NotFound(task.id.clone()).into()) + } + } + + /// A task handed out is read: the session's label rises to its classification. + fn observe_task(&self, task: &Task) { + lock(&self.session).observe(task.spec.classification); + } + + /// Read one named task that is handed out as it is: checked, then observed. + fn read_task(&self, task: &Task) -> Result<(), Error> { + self.check_task(task)?; + self.observe_task(task); + Ok(()) + } + + /// The tasks of `tasks` this session may see, read: the label rises to the + /// classification of each, so to the highest handed out. + fn read_tasks(&self, tasks: Vec<Task>) -> Result<Vec<Task>, Error> { + let handling = self.handling()?; + let seen: Vec<Task> = tasks + .into_iter() + .filter(|t| self.sees(t, handling)) + .collect(); + let mut s = lock(&self.session); + for t in &seen { + s.observe(t.spec.classification); + } + Ok(seen) + } + + /// `tree` without the branches this session may not see, nothing read yet. What is + /// filed from a task is classified at least as high as the task, so a branch above + /// handling goes whole. The root is the caller's to have checked. + fn visible_tree(&self, tree: TaskTree, handling: Label) -> TaskTree { + TaskTree { + children: tree + .children + .into_iter() + .filter(|c| self.sees(&c.task, handling)) + .map(|c| self.visible_tree(c, handling)) + .collect(), + task: tree.task, } } @@ -1640,16 +2005,21 @@ /// /// # Errors /// - /// An ungranted domain, a proposal's origin, or a ledger or policy error. + /// An ungranted domain, or a ledger or policy error; [`Error::Denied`] for a + /// `promote:` origin, which only [`Memory::promote`] files, or when the origin and + /// upstream event name a task already filed that this session may not read. pub async fn create_task(&self, spec: TaskSpec) -> Result<Created, Error> { + // a task from this origin is an approval task: approve reads its event as the + // version proposed and its filer as the proposer, so one filed here would be a + // proposal nobody made if spec .origin .as_deref() - .is_some_and(|o| o.starts_with("promote:")) + .is_some_and(|o| o.starts_with(PROPOSAL_ORIGIN)) { - return Err(Error::Denied( - "a proposal to verify is filed by promote, not as a task".into(), - )); + return Err(Error::Denied(format!( + "a task with origin {PROPOSAL_ORIGIN}<id> is an approval task: only promote files one" + ))); } let mut l = self.ledger.lock().await; let held = match self.held.get() { @@ -1687,6 +2057,17 @@ ) -> Result<Created, Error> { let spec = self.stamp(spec, filing, parent)?; let created = l.create(spec).await?; + // a filing that names a task already there is handed that task, and admits it on + // its say, only when this session may see it; the answer names nothing of it. + // The label does not rise for it: the filer named the task by the upstream event + // it is filing itself, and a tap, which files at its domain's classification + // above its own ceiling, would otherwise stop on every replay + if created.deduplicated && !self.sees(&created.task, self.handling()?) { + return Err(Error::Denied( + "this origin and upstream event are filed already, in a task this session may not read" + .into(), + )); + } let task = self .admit_per_policy(l, created.task, filing, parent) .await?; @@ -1726,9 +2107,10 @@ /// A ledger error, or a task outside the grants. pub async fn admit(&self, id: &str) -> Result<Task, Error> { let mut l = self.ledger.lock().await; - self.task_in_grants(&l.get(id).await?)?; + self.check_task(&l.get(id).await?)?; let admitted = l.admit(id).await?; self.rank_edges(&mut l, &admitted).await?; + self.observe_task(&admitted); Ok(admitted) } @@ -1810,32 +2192,38 @@ /// hold every child's handoff and the parent's own, which is what resuming is. Where /// the domain admits a session's own filings, everything filed here is admitted. /// + /// An item that matches a task this session may not read is withheld, not filed and + /// not named: the answer counts it and says nothing else of it. An item whose match + /// is elsewhere in the chain and done or dropped is new work here. + /// /// # Errors /// - /// A parent outside the grants, or a ledger error. + /// A parent outside the grants or above the handling, or a ledger error; + /// [`Error::Denied`] when the parent has an open continuation this session may not + /// read. That check runs before the items are matched, so the refusal is the same + /// whatever the items are. pub async fn fan_out(&self, parent: &str, items: &[(String, String)]) -> Result<FanOut, Error> { let mut l = self.ledger.lock().await; let parent = l.get(parent).await?; - self.task_in_grants(&parent)?; - // the chain this parent belongs to: a continuation or a retry resumes a root, - // and what an earlier run of the chain already asked for is not asked again - let root = Memory::root_of(&l, &parent).await; - let mut done = Vec::new(); - Memory::tree_in(&l, root).await?.flatten(&mut done); + self.check_task(&parent)?; + let handling = self.handling()?; + // a harness that delegates twice from one run, as Claude Code does with one + // Agent call per subagent, gets one continuation waiting on all of them: an open + // continuation of this parent is folded into the new one and closed as superseded + let (mut ids, folded) = self.open_continuations(&l, &parent, handling).await?; + let (asked, mut existing, mut withheld) = self.spawns(&l, &parent, items, handling).await?; + if asked.is_empty() { + self.observe_task(&parent); + existing.iter().for_each(|t| self.observe_task(t)); + return Ok(FanOut { + children: Vec::new(), + existing, + withheld, + continuation: None, + }); + } let mut children = Vec::new(); - let mut existing = Vec::new(); - for (title, body) in items { - let hash = dm_core::digest(body)[..16].to_string(); - if let Some(earlier) = done.iter().find(|t| { - t.spec.upstream_event.as_deref() == Some(hash.as_str()) - && t.spec - .origin - .as_deref() - .is_some_and(|o| o.starts_with("task:")) - }) { - existing.push(earlier.clone()); - continue; - } + for (title, body, hash) in asked { let created = l .create(TaskSpec { domain: parent.spec.domain.clone(), @@ -1850,22 +2238,35 @@ priority: parent.spec.priority, }) .await?; - children.push( - self.admit_per_policy(&mut l, created.task, Filing::Held, Some(&parent)) - .await?, - ); + // the ledger dedups on the parent and the hash whatever the task's status: a + // task it hands back is not this call's, and one this session may not read + // is withheld, never handed out + if created.deduplicated { + if self.sees(&created.task, handling) { + existing.push(created.task); + } else { + withheld += 1; + } + continue; + } + let child = self + .admit_per_policy(&mut l, created.task, Filing::Held, Some(&parent)) + .await?; + if !ids.contains(&child.id) { + ids.push(child.id.clone()); + } + children.push(child); } if children.is_empty() { + self.observe_task(&parent); + existing.iter().for_each(|t| self.observe_task(t)); return Ok(FanOut { children, existing, + withheld, continuation: None, }); } - // a harness that delegates twice from one run, as Claude Code does with one - // Agent call per subagent, gets one continuation waiting on all of them: an open - // continuation of this parent is folded into the new one and closed as superseded - let (ids, folded) = Memory::fold_continuations(&l, &parent, &children).await?; let continuation = self.file_continuation(&mut l, &parent, ids).await?; for old in folded { l.invalidate( @@ -1878,13 +2279,61 @@ ) .await?; } + self.observe_task(&parent); + existing.iter().for_each(|t| self.observe_task(t)); Ok(FanOut { children, existing, + withheld, continuation: Some(continuation), }) } + /// The items of a fan-out to file, each with the hash that names it; the tasks the + /// chain already holds for the others, since what an earlier run of the chain asked + /// for is not asked again; and how many items are withheld. An item that matches a + /// task this session may not see is withheld when that task is open, or when it is + /// this parent's, since the ledger would hand that task back; filing it again would + /// break "the same spawn asked twice is one task", and the count names nothing of + /// it. A hidden match elsewhere in the chain that is done or dropped is new work + /// here: a retry asks again for what an earlier attempt gave up on. + async fn spawns( + &self, + l: &Ledger, + parent: &Task, + items: &[(String, String)], + handling: Label, + ) -> Result<(Vec<(String, String, String)>, Vec<Task>, usize), Error> { + // the chain this parent belongs to: a continuation or a retry resumes a root + let root = Memory::root_of(l, parent).await; + let mut done = Vec::new(); + Memory::tree_in(l, root).await?.flatten(&mut done); + let of_parent = format!("task:{}", parent.id); + let mut asked = Vec::new(); + let mut existing = Vec::new(); + let mut withheld = 0; + for (title, body) in items { + let hash = dm_core::digest(body)[..16].to_string(); + let earlier = done.iter().find(|t| { + t.spec.upstream_event.as_deref() == Some(hash.as_str()) + && t.spec + .origin + .as_deref() + .is_some_and(|o| o.starts_with("task:")) + }); + match earlier { + Some(t) if self.sees(t, handling) => existing.push(t.clone()), + Some(t) + if !t.status.is_terminal() || t.spec.origin.as_deref() == Some(&of_parent) => + { + withheld += 1; + } + _ => asked.push((title.clone(), body.clone(), hash)), + } + } + Ok((asked, existing, withheld)) + } + /// File the continuation that resumes `parent` once `ids` are done. async fn file_continuation( &self, @@ -1918,26 +2367,41 @@ Ok(continuation) } - /// The children a new continuation waits for: this fan-out's, and those of any - /// continuation of the same parent still open, which the new one replaces. - async fn fold_continuations( + /// The open continuations of `parent` a new one replaces, and the tasks they wait + /// for, which the new one waits for too. A continuation this session may not see, or + /// one waiting on a task it may not see, is not its to fold: the fan-out is refused, + /// so no hidden work is closed, or waited on unseen, by a session that cannot read + /// it. The refusal names nothing of it, and it comes before the items are looked at, + /// so it is the same refusal whatever they are. + async fn open_continuations( + &self, l: &Ledger, parent: &Task, - children: &[Task], + handling: Label, ) -> Result<(Vec<String>, Vec<String>), Error> { let mut ids: Vec<String> = Vec::new(); let mut folded = Vec::new(); let marker = dm_ledger::continues(&parent.id); for t in l.filed_from(&parent.id).await? { - if t.spec.origin.as_deref() == Some(marker.as_str()) && !t.status.is_terminal() { - ids.extend(t.spec.depends_on.iter().cloned()); - folded.push(t.id); + if t.spec.origin.as_deref() != Some(marker.as_str()) || t.status.is_terminal() { + continue; } - } - for c in children { - if !ids.contains(&c.id) { - ids.push(c.id.clone()); + let mut seen = self.sees(&t, handling); + for dep in &t.spec.depends_on { + seen &= match l.get(dep).await { + Ok(d) => self.sees(&d, handling), + Err(dm_ledger::Error::NotFound(_)) => true, + Err(e) => return Err(e.into()), + }; } + if !seen { + return Err(Error::Denied( + "this task has open work this session may not read; it is not fanned out again here" + .into(), + )); + } + ids.extend(t.spec.depends_on.iter().cloned()); + folded.push(t.id); } Ok((ids, folded)) } @@ -1976,8 +2440,13 @@ pub async fn task_tree(&self, id: &str) -> Result<TaskTree, Error> { let l = self.ledger.lock().await; let task = l.get(id).await?; - self.task_in_grants(&task)?; - Ok(Memory::tree_in(&l, task).await?) + self.check_task(&task)?; + let tree = Memory::tree_in(&l, task).await?; + let tree = self.visible_tree(tree, self.handling()?); + let mut all = Vec::new(); + tree.flatten(&mut all); + all.iter().for_each(|t| self.observe_task(t)); + Ok(tree) } /// Boxed, since the tree is as deep as the work and an async fn cannot recurse @@ -2006,10 +2475,20 @@ pub async fn retry_task(&self, id: &str) -> Result<Task, Error> { let mut l = self.ledger.lock().await; let root = l.get(id).await?; - self.task_in_grants(&root)?; - let tree = Memory::tree_in(&l, root.clone()).await?; + self.check_task(&root)?; + // the lessons are what this session may read of the chain; the retry carries the + // root and the tasks with a handoff, and is classified at the highest of those, + // not at what the walk passed over + let tree = self.visible_tree(Memory::tree_in(&l, root.clone()).await?, self.handling()?); let mut lessons = Vec::new(); tree.handoffs(&mut lessons); + let mut carried = Vec::new(); + tree.flatten(&mut carried); + carried.retain(|t| t.id == root.id || t.handoff.is_some()); + let classification = carried + .iter() + .map(|t| t.spec.classification) + .fold(self.label(), Label::max); let mut body = root.spec.body.trim_end().to_string(); if !lessons.is_empty() { body.push_str( @@ -2036,16 +2515,21 @@ origin: Some(format!("retries:{}", root.id)), upstream_event: None, created_by: self.author.clone(), - classification: root.spec.classification.max(self.label()), + classification, priority: root.spec.priority, }) .await?; - self.admit_per_policy(&mut l, created.task, Filing::Held, Some(&root)) - .await + let retry = self + .admit_per_policy(&mut l, created.task, Filing::Held, Some(&root)) + .await?; + carried.iter().for_each(|t| self.observe_task(t)); + Ok(retry) } /// Claim the highest-ranked claimable task in a domain that this session's manifest - /// can take. + /// can take and its handling covers. The ledger passes over a task above the + /// handling inside the claim's own transaction, so such a task is never claimed, + /// and never offered, to this session. /// /// # Errors /// @@ -2056,11 +2540,12 @@ lease: Duration, ) -> Result<Option<Claim>, Error> { let domain = self.resolve_domain(domain)?; + let handling = self.handling()?; let claim = self .ledger .lock() .await - .claim_next(&domain, &self.author, lease, &self.manifest) + .claim_next(&domain, &self.author, lease, &self.manifest, handling) .await?; if let Some(c) = &claim { lock(&self.session).observe(c.task.spec.classification); @@ -2068,15 +2553,19 @@ Ok(claim) } - /// Claim one named task. + /// Claim one named task. A task above the session's handling is not found, as it is + /// to every other read, and the ledger refuses it too. /// /// # Errors /// - /// A ledger error, or a task outside the grants. + /// A ledger error, including a task outside the grants or above the handling. pub async fn claim(&self, id: &str, lease: Duration) -> Result<Claim, Error> { let mut l = self.ledger.lock().await; - self.task_in_grants(&l.get(id).await?)?; - let c = l.claim(id, &self.author, lease, &self.manifest).await?; + self.check_task(&l.get(id).await?)?; + let handling = self.handling()?; + let c = l + .claim(id, &self.author, lease, &self.manifest, handling) + .await?; lock(&self.session).observe(c.task.spec.classification); Ok(c) } @@ -2088,8 +2577,10 @@ /// A ledger error, including a lease already lost, or a task outside the grants. pub async fn heartbeat(&self, id: &str, lease: Duration) -> Result<Task, Error> { let mut l = self.ledger.lock().await; - self.task_in_grants(&l.get(id).await?)?; - Ok(l.heartbeat(id, &self.author, lease).await?) + self.check_task(&l.get(id).await?)?; + let t = l.heartbeat(id, &self.author, lease).await?; + self.observe_task(&t); + Ok(t) } /// Complete a task with the handoff entry that records what happened. The handoff @@ -2101,7 +2592,7 @@ /// grants. pub async fn complete(&self, id: &str, handoff: &str) -> Result<Task, Error> { let task = self.ledger.lock().await.get(id).await?; - self.task_in_grants(&task)?; + self.check_task(&task)?; if self.store.get(&task.spec.domain, handoff).await?.is_none() { return Err(Error::NotFound(format!("handoff {handoff}"))); } @@ -2121,6 +2612,7 @@ }; by = closed; } + self.observe_task(&done); Ok(done) } @@ -2134,17 +2626,22 @@ self.get(h).await?; // the handoff must exist and be readable here } let mut l = self.ledger.lock().await; - self.task_in_grants(&l.get(id).await?)?; - Ok(l.release(id, &self.author, handoff).await?) + self.check_task(&l.get(id).await?)?; + let t = l.release(id, &self.author, handoff).await?; + self.observe_task(&t); + Ok(t) } /// Close a task without doing it, and everything still open that was filed from it: /// its subagent tasks, their continuations, and theirs, since a promise withdrawn - /// withdraws the promises made to keep it. + /// withdraws the promises made to keep it. Open work this session may not read is + /// not its to close: the call is refused before anything moves, and the refusal + /// names nothing of that work. /// /// # Errors /// - /// A ledger error, or a task outside the grants. + /// A ledger error, or a task outside the grants; [`Error::Denied`] when an open task + /// filed from it, however far down, is above the session's handling. pub async fn invalidate( &self, id: &str, @@ -2152,35 +2649,50 @@ superseded_by: Option<&str>, ) -> Result<Task, Error> { let mut l = self.ledger.lock().await; - self.task_in_grants(&l.get(id).await?)?; - let task = l.invalidate(id, reason, superseded_by).await?; + self.check_task(&l.get(id).await?)?; + let handling = self.handling()?; + let mut open = Vec::new(); let mut pending = l.filed_from(id).await?; while let Some(t) = pending.pop() { pending.extend(l.filed_from(&t.id).await?); - if !t.status.is_terminal() { - l.invalidate( - &t.id, - &format!("filed from {id}, which was invalidated: {reason}"), - superseded_by, - ) - .await?; + if t.status.is_terminal() { + continue; } + if !self.sees(&t, handling) { + return Err(Error::Denied( + "this task has open work this session may not read; it is not invalidated here" + .into(), + )); + } + open.push(t.id); } + let task = l.invalidate(id, reason, superseded_by).await?; + for t in open { + l.invalidate( + &t, + &format!("filed from {id}, which was invalidated: {reason}"), + superseded_by, + ) + .await?; + } + self.observe_task(&task); Ok(task) } - /// One task. + /// One task, read: the session's label rises to its classification. /// /// # Errors /// - /// [`Error::NotFound`], including for a task outside the grants. + /// The ledger's not-found, including for a task outside the grants or above the + /// session's handling. pub async fn get_task(&self, id: &str) -> Result<Task, Error> { let t = self.ledger.lock().await.get(id).await?; - self.task_in_grants(&t)?; + self.read_task(&t)?; Ok(t) } - /// Tasks in a domain with a status, ranked. + /// Tasks in a domain with a status, ranked: those within the session's handling, + /// each read, so the label rises to the highest classification handed out. /// /// # Errors /// @@ -2191,7 +2703,8 @@ status: dm_ledger::Status, ) -> Result<Vec<Task>, Error> { let domain = self.resolve_domain(domain)?; - Ok(self.ledger.lock().await.list(&domain, status).await?) + let listed = self.ledger.lock().await.list(&domain, status).await?; + self.read_tasks(listed) } // ---- leaving, arriving, rebuilding ---- @@ -2217,6 +2730,9 @@ let scope = bundle::export_scope(&self.session_filter()?, &domain, max_label)?; // the index is read and released before the record is, so the future is Send let hits = self.index.lock().await.list(&scope, None, usize::MAX / 2)?; + let promotions = self + .promotions_to_carry(&domain, &hits, scope.handling) + .await?; let ex = bundle::export( &self.store, hits, @@ -2224,6 +2740,7 @@ max_label, &self.grants.owner, dest, + promotions, ) .await?; self.export_excerpts(&domain, &ex.manifest.entries, max_label, dest) @@ -2238,15 +2755,19 @@ /// the index, the served bundles; then the receipt again without that line. A run /// that stops short leaves that receipt, and the next run picks up where it stopped, /// keeping the first run's reason. Entries that cite the erased one are flagged for - /// review today rather than re-derived here. An open proposal to verify the entry is - /// invalidated: nothing is left to approve. Exports already handed out, the - /// transcripts of sessions that read the entry, and the queries in the read log are - /// beyond reach and the receipt says so. + /// review today rather than re-derived here. The why of each promotion of the entry + /// leaves the ledger, its log and its projection alike, and the receipt counts them; + /// who promoted which version to which tier, and when, stays. Its approval tasks + /// keep nothing of it but its id, and an open one is invalidated: nothing is left to + /// approve. Run again once finished, it scrubs the ledger again, for a why recorded + /// after the first run's scrub, and is refused when nothing was left there. Exports + /// already handed out, the transcripts of sessions that read the entry, and the + /// queries in the read log are beyond reach and the receipt says so. /// /// # Errors /// /// [`Error::Denied`] for a session that is not the owner or an entry already - /// erased, [`Error::NotFound`] for an entry in no granted domain, or a store, index + /// erased with nothing left on the ledger to scrub, [`Error::NotFound`] for an entry in no granted domain, or a store, index /// or I/O error; an index another connection holds is [`dm_index::Error::Held`], /// and the erasure is finished by running it again. pub async fn erase(&self, id: &str, reason: Reason) -> Result<Receipt, Error> { @@ -2272,9 +2793,7 @@ let (domain, entry) = self.find(id).await?; let stubbed = entry.front.status == Some(Status::Erased); let mut receipt = match self.store.receipt(&domain, id).await? { - Some(r) if r.is_finished() => { - return Err(Error::Denied(format!("{id} is already erased"))); - } + Some(r) if r.is_finished() => return self.erase_again(&domain, id, r).await, Some(unfinished) => unfinished, None => { let removed = if stubbed { @@ -2292,6 +2811,8 @@ removed, flagged: Vec::new(), unresolved: vec![erasure::UNFINISHED.into()], + promotion_whys_removed: 0, + promotion_tasks_cleared: 0, }; self.store.put_receipt(&pending).await?; pending @@ -2310,8 +2831,36 @@ self.finish_erasure(&domain, id, receipt).await } + /// An erasure run again once it finished: the ledger's scrub alone. A promotion can + /// record a why after the first run's scrub, and take it out itself only when the + /// ledger lets it; this takes out what it left. The scrub counts every record without + /// a why and every task it holds redacted, so counts that grew mean it found more, and + /// the receipt takes them; counts that did not mean the entry was erased already. + async fn erase_again( + &self, + domain: &str, + id: &str, + mut receipt: Receipt, + ) -> Result<Receipt, Error> { + let scrubbed = self + .ledger + .lock() + .await + .scrub_entry(domain, id, &proposal_origin(id), &proposal_redaction(id)) + .await?; + if scrubbed.records == receipt.promotion_whys_removed + && scrubbed.tasks == receipt.promotion_tasks_cleared + { + return Err(Error::Denied(format!("{id} is already erased"))); + } + receipt.promotion_whys_removed = scrubbed.records; + receipt.promotion_tasks_cleared = scrubbed.tasks; + self.store.put_receipt(&receipt).await?; + Ok(receipt) + } + /// The steps of an erasure after the record's: the citing entries, the index, the - /// served bundles, every render and the open proposal to verify it; then the + /// served bundles, every render, and the ledger's whys and approval tasks; then the /// receipt, closed. async fn finish_erasure( &self, @@ -2336,15 +2885,22 @@ // every session's render, before the receipt says the erasure is done; what it // could not reach is said on the receipt let missed = self.erase_from_projections(domain, id); - // the ledger last: one another connection holds stops the erasure here, with the - // text already out of the index and the renders, and a rerun finishes it - self.close_proposals( - domain, - id, - &format!("{id} was erased; nothing is left to verify"), - ) - .await?; + // the whys of its promotions are the entry's text too, and so is what its + // approval tasks say of it; who promoted what stays, and so does each task's id. + // The scrub closes every open proposal to verify the entry, of any version, as + // it rewrites it. Last of the steps, since a ledger another connection holds + // stops here: the index and the renders are clean by then, and the receipt stays + // unfinished + let scrubbed = self + .ledger + .lock() + .await + .scrub_entry(domain, id, &proposal_origin(id), &proposal_redaction(id)) + .await?; + receipt.promotion_tasks_cleared = scrubbed.tasks; + receipt.promotion_whys_removed = scrubbed.records; close_receipt(&mut receipt, missed); + receipt.unresolved.extend(scrubbed.beyond_reach); self.store.put_receipt(&receipt).await?; self.refresh_projection_after_erasure().await; Ok(receipt) @@ -2364,8 +2920,10 @@ Ok(None) } - /// Invalidate every open proposal to verify an entry that is gone, of any version, - /// so the board stops offering them. A proposal already closed is left as it is. + /// Invalidate every open proposal to verify an entry that was replaced, of any + /// version, so the board stops offering them. A proposal already closed is left as it + /// is. An erasure closes its proposals through the scrub instead, which rewrites them + /// too. async fn close_proposals(&self, domain: &str, id: &str, why: &str) -> Result<(), Error> { let mut l = self.ledger.lock().await; for t in l.with_origin(domain, &format!("promote:{id}")).await? { @@ -2387,13 +2945,39 @@ } } +/// The prefix of the origin of approval tasks: `promote:` and the entry's id. +const PROPOSAL_ORIGIN: &str = "promote:"; + +/// The origin of the approval tasks of proposals to promote `id` to verified. +fn proposal_origin(id: &str) -> String { + format!("{PROPOSAL_ORIGIN}{id}") +} + +/// The title of an approval task: the entry by id, and nothing else of it. +fn proposal_title(id: &str) -> String { + format!("Approve: verified for {id}") +} + +/// What an erasure of `id` rewrites its approval tasks to: the id a stub keeps, and +/// nothing of the entry's title, nor of the why a task filed before the ledger kept the +/// why held in its body. +fn proposal_redaction(id: &str) -> dm_ledger::Redaction { + dm_ledger::Redaction { + title: proposal_title(id), + body: format!( + "`{id}` was erased, and this proposal is closed with it. Nothing of the entry, nor of why it was proposed, is kept here." + ), + reason: format!("{id} was erased"), + } +} + /// What the upstream event of a proposal to verify starts with; the digest of the /// version proposed follows. Filed with origin `promote:<id>`. const VERIFY_PREFIX: &str = "verified:"; -/// The upstream event of a proposal to verify the entry as it now stands. -fn verify_event(entry: &Entry) -> String { - format!("{VERIFY_PREFIX}{}", dm_core::digest(&entry.render())) +/// The upstream event of a proposal to verify the version of an entry with `digest`. +fn verify_event(digest: &str) -> String { + format!("{VERIFY_PREFIX}{digest}") } /// Why an entry that is not live is not promoted. @@ -2499,8 +3083,28 @@ Ok(()) } + /// The promotion records of `hits`, at or below `bound`, for an export to carry. + async fn promotions_to_carry( + &self, + domain: &str, + hits: &[Hit], + bound: Label, + ) -> Result<Vec<dm_ledger::PromotionRecord>, Error> { + let l = self.ledger.lock().await; + let mut out = Vec::new(); + for h in hits { + out.extend( + l.promotions(domain, &h.id) + .await? + .into_iter() + .filter(|r| r.spec.classification <= bound), + ); + } + Ok(out) + } + /// Import a bundle into a granted domain. Entries land raw unless this owner - /// exported them. + /// exported them; then why they were promoted lands with them, on the ledger. /// /// # Errors /// @@ -2548,10 +3152,26 @@ imported.excerpts = self .restore_excerpts(&domain, carried, &imported, default, &erased) .await?; + self.restore_promotions(&imported.promotions).await?; self.refresh_projection().await; Ok(imported) } + /// Restore the promotion records an import carried, about the entries as they landed. + /// An entry erased since it landed keeps its records without the why: each entry is + /// read again once its record is restored, as a promotion's is. + async fn restore_promotions( + &self, + records: &[dm_ledger::PromotionRecord], + ) -> Result<(), Error> { + let mut l = self.ledger.lock().await; + for r in records { + l.restore_promotion(r).await?; + self.scrub_erased(&mut l, &r.spec).await?; + } + Ok(()) + } + /// Rebuild the index from the record and the ledger's projection from its events. /// Returns entries indexed and events replayed. /// @@ -2610,6 +3230,26 @@ assert_eq!(e.front.created, ahead + Duration::nanoseconds(1)); } + #[tokio::test] + async fn a_request_view_dates_its_writes_after_its_session_s_last_one() { + let dir = tempfile::tempdir().unwrap(); + let m = mem(dir.path()); + let ahead = OffsetDateTime::now_utc() + Duration::seconds(30); + *lock(&m.last_stamp) = ahead; + let view = m.for_request().unwrap(); + let w = view + .write(req("Written through the board.\n")) + .await + .unwrap(); + let e = m.store.get("dom-me", &w.id).await.unwrap().unwrap(); + assert_eq!(e.front.created, ahead + Duration::nanoseconds(1)); + assert_eq!( + *lock(&m.last_stamp), + e.front.created, + "one clock per service" + ); + } + fn req(body: &str) -> WriteRequest { WriteRequest { body: body.into(), @@ -2988,6 +3628,1803 @@ assert!(versions >= 2, "the raw version is kept: {versions}"); } + const WHY: &str = "checked against the tree"; + + /// A session of the librarian, which did not write the owner's entries. + fn librarian(dir: &Path) -> Memory { + Memory::open(dir).unwrap().with_author("agent:librarian") + } + + /// Every task in the session's domain, whatever its status. + async fn all_tasks(m: &Memory) -> Vec<Task> { + let mut out = Vec::new(); + for st in [ + Status::Proposed, + Status::Admitted, + Status::Claimed, + Status::Completed, + Status::Invalidated, + ] { + out.extend(m.tasks(None, st).await.unwrap()); + } + out + } + + #[tokio::test] + async fn a_promotion_to_curated_is_recorded_on_the_ledger_and_is_no_task() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let theirs = owner.write(req("The owner's fact.\n")).await.unwrap().id; + let raw = owner.get(&theirs).await.unwrap(); + let agent = librarian(dir.path()); + let before = all_tasks(&agent).await; + agent + .promote(&theirs, TrustTier::Curated, WHY) + .await + .unwrap(); + let after = all_tasks(&agent).await; + assert_eq!(after, before, "a promotion files no task"); + let records = agent.promotions(&theirs).await.unwrap(); + assert_eq!(records.len(), 1, "one record: {records:?}"); + assert!(records[0].landed); + let r = &records[0].record.spec; + assert_eq!(r.entry, theirs); + assert_eq!(r.why, WHY); + assert_eq!(r.tier, TrustTier::Curated); + assert_eq!(r.by, "agent:librarian"); + assert_eq!(r.author, "human:test:me"); + assert_eq!( + r.digest, + dm_core::digest(&raw.render()), + "the version promoted" + ); + assert_eq!( + r.new_digest, + dm_core::digest(&agent.get(&theirs).await.unwrap().render()), + "the version written" + ); + } + + #[tokio::test] + async fn a_promotion_moves_only_the_tier_and_keeps_the_why_out_of_the_entry() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let theirs = owner + .write(WriteRequest { + title: Some("The owner's fact".into()), + summary: Some("What the owner found.".into()), + source: vec!["https://example.org/fact".into()], + ..req("The owner's fact.\n") + }) + .await + .unwrap() + .id; + let old = owner.get(&theirs).await.unwrap(); + librarian(dir.path()) + .promote(&theirs, TrustTier::Curated, WHY) + .await + .unwrap(); + let new = owner.get(&theirs).await.unwrap(); + assert_eq!(new.front.trust_tier, TrustTier::Curated); + let mut front = new.front.clone(); + front.trust_tier = old.front.trust_tier; + assert_eq!(front, old.front, "nothing but the tier moves"); + assert_eq!(new.body, old.body); + assert!( + !new.render().contains(WHY), + "the why is kept beside the entry" + ); + } + + /// The ledger's files made read-only, so a session opened on them cannot write + /// there; the permissions they had, to put back. + #[cfg(unix)] + fn ledger_read_only(root: &Path) -> Vec<(PathBuf, std::fs::Permissions)> { + use std::os::unix::fs::PermissionsExt; + let mut kept = Vec::new(); + for name in ["ledger.db", "ledger.db-wal", "ledger.db-shm"] { + let path = root.join("state").join(name); + let Ok(meta) = std::fs::metadata(&path) else { + continue; + }; + kept.push((path.clone(), meta.permissions())); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o444)).unwrap(); + } + assert!(!kept.is_empty(), "no ledger files under {}", root.display()); + kept + } + + #[cfg(unix)] + fn put_back(kept: Vec<(PathBuf, std::fs::Permissions)>) { + for (path, perms) in kept { + std::fs::set_permissions(path, perms).unwrap(); + } + } + + #[cfg(unix)] + #[tokio::test] + async fn a_ledger_that_fails_stops_the_promotion_before_the_tier_moves() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let theirs = owner.write(req("The owner's fact.\n")).await.unwrap().id; + let kept = ledger_read_only(dir.path()); + let refused = librarian(dir.path()) + .promote(&theirs, TrustTier::Curated, WHY) + .await; + put_back(kept); + assert!(refused.is_err(), "the record could not be written"); + assert_eq!( + owner.get(&theirs).await.unwrap().front.trust_tier, + TrustTier::Raw, + "no promotion without its record" + ); + let agent = librarian(dir.path()); + agent + .promote(&theirs, TrustTier::Curated, WHY) + .await + .unwrap(); + assert_eq!(agent.promotions(&theirs).await.unwrap().len(), 1); + } + + /// `record/dom-me` read-only while `f` runs, so no version can be written there. + #[cfg(unix)] + async fn with_record_read_only<T>(root: &Path, f: impl std::future::Future<Output = T>) -> T { + use std::os::unix::fs::PermissionsExt; + let record = root.join("record").join("dom-me"); + let perms = std::fs::metadata(&record).unwrap().permissions(); + std::fs::set_permissions(&record, std::fs::Permissions::from_mode(0o555)).unwrap(); + let out = f.await; + std::fs::set_permissions(&record, perms).unwrap(); + out + } + + #[cfg(unix)] + #[tokio::test] + async fn a_promotion_whose_new_version_fails_can_be_made_again_with_one_record() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let theirs = owner.write(req("The owner's fact.\n")).await.unwrap().id; + let agent = librarian(dir.path()); + let refused = + with_record_read_only(dir.path(), agent.promote(&theirs, TrustTier::Curated, WHY)) + .await; + assert!(refused.is_err(), "the new version could not be written"); + assert_eq!( + owner.get(&theirs).await.unwrap().front.trust_tier, + TrustTier::Raw + ); + let pending = agent.promotions(&theirs).await.unwrap(); + assert_eq!(pending.len(), 1); + assert!(!pending[0].landed, "its version was never written"); + agent + .promote(&theirs, TrustTier::Curated, WHY) + .await + .unwrap(); + assert_eq!( + owner.get(&theirs).await.unwrap().front.trust_tier, + TrustTier::Curated + ); + let done = agent.promotions(&theirs).await.unwrap(); + assert_eq!(done.len(), 1, "the retry records nothing twice"); + assert!(done[0].landed, "its version is the entry's now"); + } + + #[cfg(unix)] + #[tokio::test] + async fn a_second_promoter_is_recorded_after_a_first_one_failed() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let theirs = owner.write(req("The owner's fact.\n")).await.unwrap().id; + let first = librarian(dir.path()); + let refused = + with_record_read_only(dir.path(), first.promote(&theirs, TrustTier::Curated, WHY)) + .await; + assert!(refused.is_err()); + let checker = Memory::open(dir.path()) + .unwrap() + .with_author("agent:checker"); + checker + .promote( + &theirs, + TrustTier::Curated, + "read it against the release notes", + ) + .await + .unwrap(); + let said: Vec<(String, String)> = checker + .promotions(&theirs) + .await + .unwrap() + .into_iter() + .map(|p| (p.record.spec.by, p.record.spec.why)) + .collect(); + assert_eq!( + said, + vec![ + ("agent:librarian".to_string(), WHY.to_string()), + ( + "agent:checker".to_string(), + "read it against the release notes".to_string() + ), + ], + "the second promoter's why is kept" + ); + } + + #[tokio::test] + async fn reindex_replays_the_promotions() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let theirs = owner.write(req("The owner's fact.\n")).await.unwrap().id; + let agent = librarian(dir.path()); + agent + .promote(&theirs, TrustTier::Curated, WHY) + .await + .unwrap(); + let before = agent.promotions(&theirs).await.unwrap(); + assert_eq!(before.len(), 1); + // the projection's rows gone, the events kept: only a replay brings them back + rusqlite::Connection::open(dir.path().join(LEDGER_FILE)) + .unwrap() + .execute("DELETE FROM promotions", []) + .unwrap(); + assert!(agent.promotions(&theirs).await.unwrap().is_empty()); + agent.reindex().await.unwrap(); + assert_eq!(agent.promotions(&theirs).await.unwrap(), before); + assert_eq!( + librarian(dir.path()).promotions(&theirs).await.unwrap(), + before + ); + } + + #[tokio::test] + async fn why_a_replaced_entry_was_promoted_can_still_be_read() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let old = owner.write(req("The owner's fact.\n")).await.unwrap().id; + librarian(dir.path()) + .promote(&old, TrustTier::Curated, WHY) + .await + .unwrap(); + let new = owner + .write(WriteRequest { + supersedes: Some(old.clone()), + ..req("The owner's fact, corrected.\n") + }) + .await + .unwrap() + .id; + let reader = librarian(dir.path()); + let kept = reader.promotions(&old).await.unwrap(); + assert_eq!(kept.len(), 1, "the why outlives the version it promoted"); + assert_eq!(kept[0].record.spec.why, WHY); + assert!(kept[0].landed, "the promoted version is in the history"); + assert!(reader.promotions(&new).await.unwrap().is_empty()); + } + + #[tokio::test] + async fn reading_why_an_entry_was_promoted_is_logged_as_a_read_of_it() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let theirs = owner.write(req("The owner's fact.\n")).await.unwrap().id; + librarian(dir.path()) + .promote(&theirs, TrustTier::Curated, WHY) + .await + .unwrap(); + let reader = librarian(dir.path()); + reader.promotions(&theirs).await.unwrap(); + let after = reader + .write(req("Something learned from the why.\n")) + .await + .unwrap() + .id; + let t = owner.taint(&theirs); + assert!( + t.readers.contains(&reader.session_id().to_string()), + "{t:?}" + ); + assert!(t.written.contains(&after), "{t:?}"); + } + + #[tokio::test] + async fn a_why_written_above_what_a_reader_may_handle_is_left_out() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let secret = owner + .write(WriteRequest { + classification: Some(Label::Restricted), + ..req("The restricted fact.\n") + }) + .await + .unwrap() + .id; + let public = owner + .write(WriteRequest { + classification: Some(Label::Public), + ..req("The public fact.\n") + }) + .await + .unwrap() + .id; + // a librarian that has read restricted text writes its why at restricted + let lib = librarian(dir.path()); + lib.get(&secret).await.unwrap(); + assert_eq!(lib.label(), Label::Restricted); + lib.promote(&public, TrustTier::Curated, WHY).await.unwrap(); + let capped = librarian(dir.path()).with_handling_cap(Label::Public); + assert!( + capped.promotions(&public).await.unwrap().is_empty(), + "a public reader does not get a restricted why" + ); + assert_eq!(capped.label(), Label::Public, "and its label stays public"); + let full = librarian(dir.path()); + let got = full.promotions(&public).await.unwrap(); + assert_eq!(got.len(), 1); + assert_eq!(got[0].record.spec.classification, Label::Restricted); + assert_eq!( + full.label(), + Label::Restricted, + "reading the why raises the label to the why's" + ); + } + + #[tokio::test] + async fn reading_why_an_entry_was_promoted_is_reading_the_entry() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let secret = owner + .write(WriteRequest { + classification: Some(Label::Restricted), + ..req("The restricted fact.\n") + }) + .await + .unwrap() + .id; + librarian(dir.path()) + .promote(&secret, TrustTier::Curated, WHY) + .await + .unwrap(); + let capped = librarian(dir.path()).with_handling_cap(Label::Public); + assert!( + matches!(capped.promotions(&secret).await, Err(Error::NotFound(_))), + "a session that may not read the entry may not read why it was promoted" + ); + assert_eq!(capped.label(), Label::Public, "and its label does not rise"); + assert!(matches!( + capped.promotions("mem-none").await, + Err(Error::NotFound(_)) + )); + let full = librarian(dir.path()); + assert_eq!(full.label(), Label::Public); + assert_eq!(full.promotions(&secret).await.unwrap().len(), 1); + assert_eq!( + full.label(), + Label::Restricted, + "the label rises as a get's does" + ); + } + + /// The ledger under `root` keeps the record of `id`'s promotion, and no why: not in + /// the record, not in the log. + async fn the_ledger_keeps_no_why(root: &Path, id: &str, marker: &str) { + let l = Ledger::open(&root.join(LEDGER_FILE)).unwrap(); + let kept = l.promotions("dom-me", id).await.unwrap(); + assert_eq!(kept.len(), 1); + assert!( + kept[0].spec.why.is_empty() && kept[0].why_erased, + "{kept:?}" + ); + assert_eq!(kept[0].spec.by, "agent:librarian", "who stays"); + let conn = rusqlite::Connection::open(root.join(LEDGER_FILE)).unwrap(); + let mut stmt = conn.prepare("SELECT payload FROM events").unwrap(); + let payloads: Vec<String> = stmt + .query_map([], |r| r.get(0)) + .unwrap() + .map(Result::unwrap) + .collect(); + assert!( + !payloads.iter().any(|p| p.contains(marker)), + "the log keeps no why" + ); + } + + #[tokio::test] + async fn an_erasure_takes_the_why_from_the_ledger_and_says_so() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let secret = owner + .write(req(&format!("The vault opens with {marker}.\n"))) + .await + .unwrap() + .id; + librarian(dir.path()) + .promote( + &secret, + TrustTier::Curated, + &format!("tried {marker} on the vault"), + ) + .await + .unwrap(); + let receipt = owner.erase(&secret, Reason::Requested).await.unwrap(); + assert_eq!(receipt.promotion_whys_removed, 1, "the receipt says so"); + assert!(matches!( + owner.promotions(&secret).await, + Err(Error::NotFound(_)) + )); + the_ledger_keeps_no_why(dir.path(), &secret, marker).await; + owner.reindex().await.unwrap(); + the_ledger_keeps_no_why(dir.path(), &secret, marker).await; + } + + #[tokio::test] + async fn an_erasure_the_ledger_holds_up_still_takes_the_text_out_of_search() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let secret = owner + .write(req("The vault opens with passphrase-Qz7xx.\n")) + .await + .unwrap() + .id; + librarian(dir.path()) + .promote(&secret, TrustTier::Curated, "tried it on the vault") + .await + .unwrap(); + // a reader mid-query on the ledger: its log cannot be folded into the file + let reader = rusqlite::Connection::open(dir.path().join(LEDGER_FILE)).unwrap(); + reader + .execute_batch("BEGIN; SELECT COUNT(*) FROM events;") + .unwrap(); + let held = owner.erase(&secret, Reason::Requested).await; + assert!( + matches!(held, Err(Error::Ledger(dm_ledger::Error::Held(_)))), + "{held:?}" + ); + let found = owner.search("vault opens", None, 10).await.unwrap(); + assert!( + found.iter().all(|h| h.id != secret), + "the erased text is still searchable: {found:?}" + ); + assert!( + !owner.receipt(&secret).await.unwrap().unwrap().is_finished(), + "the receipt says the erasure is owed" + ); + reader.execute_batch("COMMIT;").unwrap(); + let receipt = owner.erase(&secret, Reason::Requested).await.unwrap(); + assert!(receipt.is_finished()); + assert_eq!(receipt.promotion_whys_removed, 1); + // the rows were rewritten on the first run; the rerun folds the log into the + // file, so no byte under the root holds the why or the text + for text in ["tried it on the vault", "passphrase-Qz7xx"] { + assert_eq!(bundle_holds(dir.path(), text), None, "{text}"); + } + } + + /// No task, whatever its status, holds `text` in its title or its body. + async fn no_task_holds(m: &Memory, text: &str) { + for t in all_tasks(m).await { + assert!( + !t.spec.title.contains(text) && !t.spec.body.contains(text), + "a task holds {text:?}: {t:?}" + ); + } + } + + /// A second proposer's proposal of the entry as it stands is a new task, approved in + /// place of the first proposer's, which named an older version and is closed as + /// stale: the ledger keeps one approval task per entry and version, and each + /// proposal is a record of its own. + #[tokio::test] + async fn any_proposer_s_standing_proposal_unblocks_the_approval() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let cited = owner.write(req("The cited fact.\n")).await.unwrap().id; + let x = owner + .write(req(&format!("A fact that rests on {cited}.\n"))) + .await + .unwrap() + .id; + let Promotion::Proposed { task } = librarian(dir.path()) + .promote(&x, TrustTier::Verified, WHY) + .await + .unwrap() + else { + panic!("verified is proposed") + }; + owner.erase(&cited, Reason::Requested).await.unwrap(); + assert!( + owner.approve(&task.id).await.is_err(), + "the first proposal is stale" + ); + let checker = Memory::open(dir.path()) + .unwrap() + .with_author("agent:checker"); + let Promotion::Proposed { task: again } = checker + .promote(&x, TrustTier::Verified, "read it as it stands") + .await + .unwrap() + else { + panic!("verified is proposed") + }; + assert_ne!(again.id, task.id, "a new version is a new proposal"); + let stale = owner.ledger.lock().await.get(&task.id).await.unwrap(); + assert_eq!(stale.status, Status::Invalidated, "the first is stale"); + owner.approve(&again.id).await.unwrap(); + assert_eq!( + owner.get(&x).await.unwrap().front.trust_tier, + TrustTier::Verified + ); + let by: Vec<(String, bool)> = owner + .promotions(&x) + .await + .unwrap() + .into_iter() + .map(|p| (p.record.spec.by, p.landed)) + .collect(); + assert_eq!( + by, + vec![ + ("agent:librarian".to_string(), false), + ("agent:checker".to_string(), true) + ] + ); + } + + /// An approval task with no proposal behind it, filed by a build from before the + /// event named the version or by anything but a proposal, approves nothing: the + /// entry is proposed first, and the proposal is a task of its own that closes the + /// old one. And nothing but `promote` files one. + #[tokio::test] + async fn an_approval_task_without_a_proposal_record_approves_nothing() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let x = owner.write(req("A fact.\n")).await.unwrap().id; + let squat = TaskSpec { + domain: "dom-me".into(), + title: "Approve: verified".into(), + body: "A task that looks like a proposal.".into(), + requirements: vec!["human".into()], + depends_on: Vec::new(), + origin: Some(format!("promote:{x}")), + upstream_event: Some("verified".into()), + created_by: String::new(), + classification: Label::Internal, + priority: 3, + }; + let refused = librarian(dir.path()).create_task(squat.clone()).await; + assert!( + matches!(&refused, Err(Error::Denied(why)) if why.contains("promote")), + "a session files no approval task: {refused:?}" + ); + // as a build from before records were kept filed it + let filed = librarian(dir.path()).file_own(squat).await.unwrap().task; + let refused = owner.approve(&filed.id).await; + assert!( + matches!(&refused, Err(Error::Denied(why)) if why.contains("not a proposal to verify a version")), + "{refused:?}" + ); + assert_eq!( + owner.get(&x).await.unwrap().front.trust_tier, + TrustTier::Raw + ); + assert!(owner.promotions(&x).await.unwrap().is_empty(), "no record"); + // a real proposal is a task of its own, closes the old one, and is approved + let Promotion::Proposed { task } = librarian(dir.path()) + .promote(&x, TrustTier::Verified, WHY) + .await + .unwrap() + else { + panic!("verified is proposed") + }; + assert_ne!(task.id, filed.id); + let old = owner.ledger.lock().await.get(&filed.id).await.unwrap(); + assert_eq!(old.status, Status::Invalidated); + owner.approve(&task.id).await.unwrap(); + assert_eq!( + owner.get(&x).await.unwrap().front.trust_tier, + TrustTier::Verified + ); + } + + /// An erasure between the approver's checks and its write: `approve` read the entry + /// and found the proposal standing, the owner erased it, and then the write of the + /// verified version runs. The store refuses it, so nothing of the text comes back, + /// and the approval fails as any failed write does. + #[tokio::test] + async fn an_erasure_between_the_check_and_the_write_leaves_the_stub() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let x = owner + .write(req(&format!("The vault opens with {marker}.\n"))) + .await + .unwrap() + .id; + let Promotion::Proposed { task } = librarian(dir.path()) + .promote(&x, TrustTier::Verified, "checked it on the vault") + .await + .unwrap() + else { + panic!("verified is proposed") + }; + // the steps of approve, by hand, with the erasure between them + let (_, entry) = owner.proposal_entry(&task).await.unwrap(); + owner.erase(&x, Reason::Requested).await.unwrap(); + let refused = owner.set_tier(entry, TrustTier::Verified).await; + assert!( + matches!(&refused, Err(Error::Store(dm_store::Error::Erased { .. }))), + "{refused:?}" + ); + let stub = owner.find(&x).await.unwrap().1; + assert_eq!( + stub.front.status, + Some(dm_core::Status::Erased), + "the stub stays" + ); + assert!(!stub.render().contains(marker)); + assert_eq!( + bundle_holds(dir.path(), marker), + None, + "nothing of the text came back" + ); + } + + /// A whole erasure between a promotion's read of the entry and its record: the + /// erasure's scrub ran before the record was written, and the erasure is finished, so + /// it does not run again by itself. The promotion reads the stub after it records, + /// takes the why out itself and fails: no record and no byte on disk holds the why, + /// and a proposal files no task for the erased entry. + #[tokio::test] + async fn an_erasure_between_the_read_and_the_record_leaves_no_why() { + let marker = "passphrase-Qz7xx"; + for to in [TrustTier::Curated, TrustTier::Verified] { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let x = owner + .write(req("The vault opens at nine.\n")) + .await + .unwrap() + .id; + let lib = librarian(dir.path()); + // the steps of promote, by hand, with the erasure between them + let (_, entry) = lib.find(&x).await.unwrap(); + owner.erase(&x, Reason::Requested).await.unwrap(); + let refused = lib + .promote_entry(entry, to, &format!("tried {marker} on the vault")) + .await; + assert!(refused.is_err(), "{to}: {refused:?}"); + let kept = Ledger::open(&dir.path().join(LEDGER_FILE)) + .unwrap() + .promotions("dom-me", &x) + .await + .unwrap(); + assert!( + kept.iter().all(|r| r.spec.why.is_empty() && r.why_erased), + "{to}: {kept:?}" + ); + assert_eq!(bundle_holds(dir.path(), marker), None, "{to}"); + let origin = proposal_origin(&x); + let filed: Vec<Task> = all_tasks(&owner) + .await + .into_iter() + .filter(|t| t.spec.origin.as_deref() == Some(origin.as_str())) + .collect(); + assert!(filed.is_empty(), "{to}: {filed:?}"); + } + } + + /// A why on the ledger after a finished erasure, as a promotion whose own scrub could + /// not run leaves it: the owner runs the erasure again, which takes the why out and + /// counts it on the receipt. A run that finds nothing more is refused as before. + #[tokio::test] + async fn an_erasure_run_again_takes_out_a_why_recorded_after_it() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let x = owner + .write(req("The vault opens at nine.\n")) + .await + .unwrap() + .id; + let lib = librarian(dir.path()); + let (_, entry) = lib.find(&x).await.unwrap(); + let first = owner.erase(&x, Reason::Requested).await.unwrap(); + assert_eq!(first.promotion_whys_removed, 0); + let late = lib.promotion_spec(&entry, TrustTier::Curated, &format!("tried {marker}")); + owner + .ledger + .lock() + .await + .record_promotion(&late) + .await + .unwrap(); + let again = owner.erase(&x, Reason::Requested).await.unwrap(); + assert_eq!(again.promotion_whys_removed, 1, "{again:?}"); + assert!(again.is_finished()); + assert_eq!( + owner + .receipt(&x) + .await + .unwrap() + .unwrap() + .promotion_whys_removed, + 1, + "the stored receipt counts it" + ); + assert_eq!(bundle_holds(dir.path(), marker), None); + let third = owner.erase(&x, Reason::Requested).await.unwrap_err(); + assert!(third.to_string().contains("already erased"), "{third}"); + } + + /// A proposal filed above what a session may handle: the proposer had read restricted + /// text, so its task is restricted though the entry is public. A session capped at + /// public that proposes the same version is refused with nothing of that task, and + /// one that proposes a newer version leaves that task open for a session that may + /// see it. + #[tokio::test] + async fn a_proposal_a_session_may_not_see_is_neither_handed_out_nor_closed_by_it() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let public = |body: &str| WriteRequest { + classification: Some(Label::Public), + ..req(body) + }; + let secret = owner + .write(WriteRequest { + classification: Some(Label::Restricted), + ..req("The restricted fact.\n") + }) + .await + .unwrap() + .id; + let cited = owner.write(public("The cited fact.\n")).await.unwrap().id; + let x = owner + .write(public(&format!("A fact that rests on {cited}.\n"))) + .await + .unwrap() + .id; + let lib = librarian(dir.path()); + lib.get(&secret).await.unwrap(); + let Promotion::Proposed { task: hidden } = + lib.promote(&x, TrustTier::Verified, WHY).await.unwrap() + else { + panic!("verified is proposed") + }; + assert_eq!(hidden.spec.classification, Label::Restricted); + let capped = librarian(dir.path()).with_handling_cap(Label::Public); + let refused = capped.promote(&x, TrustTier::Verified, WHY).await; + assert!( + matches!(&refused, Err(Error::Denied(why)) if !why.contains(&hidden.id) + && !why.contains("agent:librarian")), + "{refused:?}" + ); + assert_eq!(capped.label(), Label::Public); + // the erasure of what it cites flags it for review: a new version of it + owner.erase(&cited, Reason::Requested).await.unwrap(); + let Promotion::Proposed { task: newer } = + capped.promote(&x, TrustTier::Verified, WHY).await.unwrap() + else { + panic!("verified is proposed") + }; + assert_ne!(newer.id, hidden.id); + let left = owner.ledger.lock().await.get(&hidden.id).await.unwrap(); + assert!(!left.status.is_terminal(), "{left:?}"); + } + + /// An entry rewritten after it was proposed is not the entry the proposal named; the + /// approver is refused until it is proposed again, and then the new proposal lands. + #[tokio::test] + async fn an_approval_refuses_an_entry_that_changed_since_the_proposal() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let cited = owner.write(req("The cited fact.\n")).await.unwrap().id; + let x = owner + .write(req(&format!("A fact that rests on {cited}.\n"))) + .await + .unwrap() + .id; + let lib = librarian(dir.path()); + let Promotion::Proposed { task } = lib.promote(&x, TrustTier::Verified, WHY).await.unwrap() + else { + panic!("verified is proposed") + }; + let proposed = owner.get(&x).await.unwrap(); + // the erasure of what it cites flags it for review: a new version of it + owner.erase(&cited, Reason::Requested).await.unwrap(); + let now = owner.get(&x).await.unwrap(); + assert_ne!(now.render(), proposed.render(), "the entry changed"); + let refused = owner.approve(&task.id).await; + assert!( + matches!(&refused, Err(Error::Denied(why)) if why.contains("changed since")), + "{refused:?}" + ); + assert_eq!( + owner.get(&x).await.unwrap().front.trust_tier, + TrustTier::Raw, + "nothing moved" + ); + assert!(!owner.promotions(&x).await.unwrap()[0].landed); + // proposed again as it stands, then approved + let Promotion::Proposed { task: again } = lib + .promote(&x, TrustTier::Verified, "read it again after the flag") + .await + .unwrap() + else { + panic!("verified is proposed") + }; + owner.approve(&again.id).await.unwrap(); + assert_eq!( + owner.get(&x).await.unwrap().front.trust_tier, + TrustTier::Verified + ); + let landed: Vec<bool> = owner + .promotions(&x) + .await + .unwrap() + .iter() + .map(|p| p.landed) + .collect(); + assert_eq!(landed, vec![false, true], "the first proposal never landed"); + } + + #[tokio::test] + async fn a_proposal_to_verified_is_recorded_and_its_approval_lands_it() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let id = owner + .write(req("A fact worth verifying.\n")) + .await + .unwrap() + .id; + let why = "checked against the tree and the release notes"; + let Promotion::Proposed { task } = librarian(dir.path()) + .promote(&id, TrustTier::Verified, why) + .await + .unwrap() + else { + panic!("verified is proposed, not set") + }; + let proposed = owner.promotions(&id).await.unwrap(); + assert_eq!(proposed.len(), 1, "{proposed:?}"); + let r = &proposed[0].record.spec; + assert_eq!( + (r.tier, r.why.as_str(), r.by.as_str()), + (TrustTier::Verified, why, "agent:librarian") + ); + assert!(!proposed[0].landed, "not until a person approves it"); + assert!( + task.spec.body.contains(&format!("amem promotions {id}")), + "{}", + task.spec.body + ); + no_task_holds(&owner, why).await; + owner.approve(&task.id).await.unwrap(); + let approved = owner.promotions(&id).await.unwrap(); + assert_eq!(approved.len(), 1); + assert!(approved[0].landed, "the approval lands it"); + no_task_holds(&owner, why).await; + } + + /// A proposal refused, because another's open proposal of the version stands or a + /// closed one declined it, and a proposal handed its own open task back, record no + /// promotion: the record is written only when a task is filed. + #[tokio::test] + async fn a_refused_or_merged_proposal_records_no_promotion() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let x = owner.write(req("A fact.\n")).await.unwrap().id; + let Promotion::Proposed { task } = librarian(dir.path()) + .promote(&x, TrustTier::Verified, WHY) + .await + .unwrap() + else { + panic!("verified is proposed") + }; + assert_eq!(recorded(&owner, &x).await, 1); + let bob = Memory::open(dir.path()) + .unwrap() + .with_author("human:test:bob"); + let refused = bob.promote(&x, TrustTier::Verified, "me too").await; + assert!(matches!(refused, Err(Error::Denied(_))), "{refused:?}"); + assert_eq!( + recorded(&owner, &x).await, + 1, + "bob's refused proposal was recorded" + ); + // the librarian asking again gets its own task back, and no second record + let Promotion::Proposed { task: again } = librarian(dir.path()) + .promote(&x, TrustTier::Verified, "still holds") + .await + .unwrap() + else { + panic!("verified is proposed") + }; + assert_eq!(again.id, task.id); + assert_eq!(recorded(&owner, &x).await, 1); + // declined: a third proposer of the same version is refused, and not recorded + owner + .invalidate(&task.id, "does not hold", None) + .await + .unwrap(); + let checker = Memory::open(dir.path()) + .unwrap() + .with_author("agent:checker"); + let refused = checker + .promote(&x, TrustTier::Verified, "I say it holds") + .await; + assert!(matches!(refused, Err(Error::Denied(_))), "{refused:?}"); + assert_eq!( + recorded(&owner, &x).await, + 1, + "the checker's refused proposal was recorded" + ); + } + + /// How many promotions of `id` the ledger records. + async fn recorded(m: &Memory, id: &str) -> usize { + m.promotions(id).await.unwrap().len() + } + + /// The task of a proposal and its record name the same version: the event + /// `verified:<digest>` that `approve` reads carries the digest the record carries, so + /// what `approve` checks and what `amem promotions` shows are one proposal. + #[tokio::test] + async fn a_proposal_s_task_names_the_version_its_record_names() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let x = owner.write(req("A fact.\n")).await.unwrap().id; + let Promotion::Proposed { task } = librarian(dir.path()) + .promote(&x, TrustTier::Verified, WHY) + .await + .unwrap() + else { + panic!("verified is proposed") + }; + let records = owner.promotions(&x).await.unwrap(); + assert_eq!(records.len(), 1); + let record = &records[0].record.spec; + assert_eq!( + task.spec.upstream_event.as_deref(), + Some(format!("verified:{}", record.digest).as_str()), + "the task names the version the record names" + ); + assert_eq!( + record.digest, + dm_core::digest(&owner.get(&x).await.unwrap().render()), + "and that is the entry as it stands" + ); + owner.approve(&task.id).await.unwrap(); + let verified = owner.get(&x).await.unwrap(); + assert_eq!(verified.front.trust_tier, TrustTier::Verified); + let landed = owner.promotions(&x).await.unwrap(); + assert_eq!( + landed[0].record.spec.new_digest, + dm_core::digest(&verified.render()), + "the record names the version the approval wrote" + ); + assert!(landed[0].landed); + } + + /// The ledger's steps of an erasure, closing the proposal and taking its why and the + /// task's text, run after the index and before the receipt closes: a ledger another + /// connection holds leaves the receipt owed with the proposal still open, and the + /// next run closes both. + #[tokio::test] + async fn an_erasure_closes_the_proposal_and_takes_its_why_before_the_receipt_closes() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let secret = owner + .write(req(&format!("The vault opens with {marker}.\n"))) + .await + .unwrap() + .id; + let Promotion::Proposed { task } = librarian(dir.path()) + .promote( + &secret, + TrustTier::Verified, + &format!("tried {marker} on the vault"), + ) + .await + .unwrap() + else { + panic!("verified is proposed") + }; + let holder = rusqlite::Connection::open(dir.path().join(LEDGER_FILE)).unwrap(); + holder.execute_batch("BEGIN IMMEDIATE").unwrap(); + let held = owner.erase(&secret, Reason::Requested).await; + assert!(matches!(held, Err(Error::Ledger(_))), "{held:?}"); + holder.execute_batch("ROLLBACK").unwrap(); + drop(holder); + let open = owner.ledger.lock().await.get(&task.id).await.unwrap(); + assert!( + !open.status.is_terminal(), + "the held ledger took the proposal: {open:?}" + ); + assert!( + !owner.receipt(&secret).await.unwrap().unwrap().is_finished(), + "the receipt closed with the proposal open" + ); + assert!( + owner + .search("vault opens", None, 10) + .await + .unwrap() + .is_empty(), + "the text is still in search" + ); + let receipt = owner.erase(&secret, Reason::Requested).await.unwrap(); + assert!(receipt.is_finished()); + assert_eq!(receipt.promotion_whys_removed, 1); + assert_eq!(receipt.promotion_tasks_cleared, 1); + let closed = owner.ledger.lock().await.get(&task.id).await.unwrap(); + assert_eq!(closed.status, Status::Invalidated); + the_ledger_keeps_no_why(dir.path(), &secret, marker).await; + no_task_holds(&owner, marker).await; + } + + #[tokio::test] + async fn an_erasure_takes_the_why_of_a_proposal_to_verified() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let secret = owner + .write(req(&format!("The vault opens with {marker}.\n"))) + .await + .unwrap() + .id; + librarian(dir.path()) + .promote( + &secret, + TrustTier::Verified, + &format!("tried {marker} on the vault"), + ) + .await + .unwrap(); + let receipt = owner.erase(&secret, Reason::Requested).await.unwrap(); + assert_eq!(receipt.promotion_whys_removed, 1); + the_ledger_keeps_no_why(dir.path(), &secret, marker).await; + no_task_holds(&owner, marker).await; + } + + /// The `created` payloads of every task in the ledger under `root`. + fn created_payloads(root: &Path) -> Vec<String> { + let conn = rusqlite::Connection::open(root.join(LEDGER_FILE)).unwrap(); + let mut stmt = conn + .prepare("SELECT payload FROM events WHERE kind = 'created'") + .unwrap(); + let out: Vec<String> = stmt + .query_map([], |r| r.get(0)) + .unwrap() + .map(Result::unwrap) + .collect(); + out + } + + /// The approval tasks of `id`, whatever their status. + async fn approval_tasks(m: &Memory, id: &str) -> Vec<Task> { + all_tasks(m) + .await + .into_iter() + .filter(|t| t.spec.origin.as_deref() == Some(&proposal_origin(id))) + .collect() + } + + #[tokio::test] + async fn an_erasure_clears_the_approval_task_of_a_proposal() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let secret = owner + .write(WriteRequest { + title: Some(format!("The vault opens with {marker}")), + ..req(&format!("Try {marker} on the vault.\n")) + }) + .await + .unwrap() + .id; + let Promotion::Proposed { task } = librarian(dir.path()) + .promote(&secret, TrustTier::Verified, "checked it on the vault") + .await + .unwrap() + else { + panic!("verified is proposed") + }; + assert_eq!( + task.spec.title, + format!("Approve: verified for {secret}"), + "the task names the entry by id, not by title" + ); + let receipt = owner.erase(&secret, Reason::Requested).await.unwrap(); + assert!(receipt.is_finished()); + assert_eq!(receipt.promotion_tasks_cleared, 1); + no_task_holds(&owner, marker).await; + let kept = approval_tasks(&owner, &secret).await; + assert_eq!(kept.len(), 1, "{kept:?}"); + assert_eq!(kept[0].id, task.id, "the task keeps its id"); + assert_eq!(kept[0].status, Status::Invalidated, "and is closed"); + assert_eq!( + kept[0].spec.created_by, "agent:librarian", + "who filed it stays" + ); + assert!( + !created_payloads(dir.path()) + .iter() + .any(|p| p.contains(marker)), + "its created event holds nothing of the entry" + ); + owner.reindex().await.unwrap(); + no_task_holds(&owner, marker).await; + } + + /// A build from before the ledger kept the why filed the approval task with the + /// entry's title and the why in it, and recorded no promotion. An erasure after an + /// upgrade still takes both out, and the receipt says so. + #[tokio::test] + async fn an_erasure_clears_an_approval_task_filed_before_the_why_was_kept() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let secret = owner + .write(WriteRequest { + title: Some(format!("The vault opens with {marker}")), + ..req(&format!("Try {marker} on the vault.\n")) + }) + .await + .unwrap() + .id; + // as origin/main files it: no record, the why in the body, the title in the title + let filed = librarian(dir.path()) + .file_own(TaskSpec { + domain: "dom-me".into(), + title: format!("Approve: verified for The vault opens with {marker}"), + body: format!( + "`agent:librarian` proposes that `{secret}`, written by `human:test:me`, be promoted to verified.\n\ntried {marker} on the vault\n\nRead it with `amem get {secret}`." + ), + requirements: vec!["human".into()], + depends_on: Vec::new(), + origin: Some(format!("promote:{secret}")), + upstream_event: Some("verified".into()), + created_by: String::new(), + classification: Label::Internal, + priority: 3, + }) + .await + .unwrap() + .task; + let receipt = owner.erase(&secret, Reason::Requested).await.unwrap(); + assert!(receipt.is_finished()); + assert_eq!(receipt.promotion_whys_removed, 0, "no record to scrub"); + assert_eq!(receipt.promotion_tasks_cleared, 1, "but a task to clear"); + no_task_holds(&owner, marker).await; + let kept = approval_tasks(&owner, &secret).await; + assert_eq!(kept.len(), 1, "{kept:?}"); + assert_eq!(kept[0].id, filed.id); + assert_eq!(kept[0].status, Status::Invalidated); + assert!( + !created_payloads(dir.path()) + .iter() + .any(|p| p.contains(marker)), + "its created event holds neither the title nor the why" + ); + } + + /// The owner's entry, promoted by the librarian with `why`, exported at `max`. + async fn promoted_then_exported( + why: &str, + max: Label, + ) -> (tempfile::TempDir, Memory, String, tempfile::TempDir) { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let id = owner.write(req("The owner's fact.\n")).await.unwrap().id; + librarian(dir.path()) + .promote(&id, TrustTier::Curated, why) + .await + .unwrap(); + let bundle = tempfile::tempdir().unwrap(); + owner.export(None, max, bundle.path()).await.unwrap(); + (dir, owner, id, bundle) + } + + /// The first file in a bundle whose text holds `needle`. + fn bundle_holds(bundle: &Path, needle: &str) -> Option<PathBuf> { + walk(bundle).into_iter().find(|p| { + std::fs::read(p) + .unwrap() + .windows(needle.len()) + .any(|w| w == needle.as_bytes()) + }) + } + + /// The landing half of an import, by hand: a bundle of one entry promoted with a why + /// holding `marker`, landed in a new root, its promotion records not restored yet. + /// Returns the temporary directories to keep, the importing session, the landed id, + /// and what the import has left to restore. + async fn landed_by_hand( + marker: &str, + ) -> ( + [tempfile::TempDir; 3], + Memory, + String, + Vec<dm_ledger::PromotionRecord>, + ) { + let (dir, _owner, id, bundle) = + promoted_then_exported(&format!("tried {marker}"), Label::Restricted).await; + let home = tempfile::tempdir().unwrap(); + let there = mem(home.path()); + let default = there + .grants + .domains + .get("dom-me") + .map(|d| d.default_classification) + .unwrap_or_default(); + let imported = { + let mut ix = there.index.lock().await; + bundle::import( + &there.store, + &mut ix, + bundle::LogTo { + root: home.path(), + by: &there.author, + }, + &there.grants.owner, + bundle.path(), + "dom-me", + default, + &HashSet::new(), + ) + .await + .unwrap() + }; + let landed = imported.landed[&id].clone(); + assert_eq!(imported.promotions.len(), 1, "the why travels"); + ([dir, bundle, home], there, landed, imported.promotions) + } + + /// An erasure of an entry an import just landed, between the landing and the restore + /// of its promotion records: the erasure's scrub ran before the records were there. + /// The restore reads the stub after it restores and takes the why out itself. + #[tokio::test] + async fn an_erasure_between_the_landing_and_the_restore_leaves_no_why() { + let marker = "passphrase-Qz7xx"; + let ([_dir, _bundle, home], there, landed, promotions) = landed_by_hand(marker).await; + there.erase(&landed, Reason::Requested).await.unwrap(); + there.restore_promotions(&promotions).await.unwrap(); + let kept = there + .ledger + .lock() + .await + .promotions("dom-me", &landed) + .await + .unwrap(); + assert_eq!(kept.len(), promotions.len(), "the records are restored"); + assert!( + kept.iter().all(|r| r.spec.why.is_empty() && r.why_erased), + "{kept:?}" + ); + assert_eq!(bundle_holds(home.path(), marker), None); + } + + /// The same, with a reader holding the ledger when the restore scrubs: the scrub's + /// rows are written, and the log that still holds the why is not folded. The erasure's + /// receipt goes back to unfinished, so the owner knows to run it again, and that run + /// folds the log. + #[tokio::test] + async fn a_restore_the_ledger_holds_up_leaves_the_erasure_unfinished() { + let marker = "passphrase-Qz7xx"; + let ([_dir, _bundle, home], there, landed, promotions) = landed_by_hand(marker).await; + there.erase(&landed, Reason::Requested).await.unwrap(); + let reader = rusqlite::Connection::open(home.path().join(LEDGER_FILE)).unwrap(); + reader + .execute_batch("BEGIN; SELECT COUNT(*) FROM events;") + .unwrap(); + let held = there.restore_promotions(&promotions).await; + assert!( + matches!(&held, Err(Error::Ledger(dm_ledger::Error::Held(why))) if why.contains("erase")), + "{held:?}" + ); + reader.execute_batch("COMMIT;").unwrap(); + assert!( + !there.receipt(&landed).await.unwrap().unwrap().is_finished(), + "the receipt says the erasure is owed" + ); + let receipt = there.erase(&landed, Reason::Requested).await.unwrap(); + assert!(receipt.is_finished()); + assert_eq!(bundle_holds(home.path(), marker), None); + } + + #[tokio::test] + async fn a_same_owner_import_keeps_why_its_entries_were_promoted() { + let (_dir, owner, id, bundle) = promoted_then_exported(WHY, Label::Restricted).await; + let before = owner.promotions(&id).await.unwrap(); + let home = tempfile::tempdir().unwrap(); + let there = mem(home.path()); + let im = there.import(bundle.path(), None).await.unwrap(); + let landed = &im.landed[&id]; + assert_eq!( + there.get(landed).await.unwrap().front.trust_tier, + TrustTier::Curated, + "the same owner keeps the tier" + ); + let after = there.promotions(landed).await.unwrap(); + assert_eq!(after.len(), 1, "and why it was given: {after:?}"); + let (b, a) = (&before[0].record, &after[0].record); + assert_eq!( + (&a.spec.why, &a.spec.by, a.spec.tier, a.at), + (&b.spec.why, &b.spec.by, b.spec.tier, b.at) + ); + // landing sets the origin, so the text here is not the bundle's: the record + // names the version as it landed, or it reads as a promotion that never landed + assert!( + after[0].landed, + "the restored record names the landed version" + ); + assert_eq!( + a.spec.new_digest, + dm_core::digest(&there.get(landed).await.unwrap().render()) + ); + } + + #[tokio::test] + async fn a_restored_record_is_at_least_as_classified_as_its_entry_here() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let id = owner + .write(WriteRequest { + classification: Some(Label::Public), + ..req("The public fact.\n") + }) + .await + .unwrap() + .id; + librarian(dir.path()) + .promote(&id, TrustTier::Curated, WHY) + .await + .unwrap(); + let bundle = tempfile::tempdir().unwrap(); + owner + .export(None, Label::Restricted, bundle.path()) + .await + .unwrap(); + let home = tempfile::tempdir().unwrap(); + mem(home.path()); + // a domain whose default is internal raises a public entry to internal on import + grant_domain(home.path(), "dom-x"); + let there = Memory::open(home.path()).unwrap(); + let im = there.import(bundle.path(), Some("dom-x")).await.unwrap(); + let landed = &im.landed[&id]; + assert_eq!( + there.get(landed).await.unwrap().front.classification, + Label::Internal + ); + let after = there.promotions(landed).await.unwrap(); + assert_eq!(after.len(), 1); + assert_eq!( + after[0].record.spec.classification, + Label::Internal, + "the record is raised with its entry" + ); + } + + /// An import whose ledger fails after the entries landed owes their records; the next + /// import skips the entries, since they are here, and restores the records. + #[cfg(unix)] + #[tokio::test] + async fn a_second_import_restores_the_records_the_first_could_not() { + let (_dir, _owner, id, bundle) = promoted_then_exported(WHY, Label::Restricted).await; + let home = tempfile::tempdir().unwrap(); + mem(home.path()); + // a session opened on a ledger it cannot write + let kept = ledger_read_only(home.path()); + let failing = Memory::open(home.path()).unwrap(); + let failed = failing.import(bundle.path(), None).await; + drop(failing); + put_back(kept); + // the log and the shared memory the failing session made took the file's mode + for name in ["ledger.db-wal", "ledger.db-shm"] { + use std::os::unix::fs::PermissionsExt; + let path = home.path().join("state").join(name); + if path.exists() { + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).unwrap(); + } + } + assert!( + matches!(failed, Err(Error::Ledger(_))), + "the ledger could not take the records: {failed:?}" + ); + let there = Memory::open(home.path()).unwrap(); + assert_eq!( + there.get(&id).await.unwrap().front.trust_tier, + TrustTier::Curated, + "but the entry landed" + ); + assert!(there.promotions(&id).await.unwrap().is_empty()); + let again = there.import(bundle.path(), None).await.unwrap(); + assert_eq!(again.skipped, vec![id.clone()], "the entry is here already"); + assert_eq!(again.promotions.len(), 1, "{again:?}"); + let after = there.promotions(&id).await.unwrap(); + assert_eq!(after.len(), 1, "its record is here now: {after:?}"); + assert_eq!(after[0].record.spec.why, WHY); + assert!(after[0].landed); + let third = there.import(bundle.path(), None).await.unwrap(); + assert_eq!(there.promotions(&id).await.unwrap().len(), 1, "{third:?}"); + } + + /// A record is restored for an entry that is here, and for no other: one refused + /// because its text was erased here keeps its erased why. + #[tokio::test] + async fn no_record_is_restored_for_an_entry_that_is_not_here() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let kept = owner.write(req("The owner's fact.\n")).await.unwrap().id; + let gone = owner + .write(req(&format!("The vault opens with {marker}.\n"))) + .await + .unwrap() + .id; + let lib = librarian(dir.path()); + lib.promote(&kept, TrustTier::Curated, WHY).await.unwrap(); + lib.promote(&gone, TrustTier::Curated, &format!("tried {marker}")) + .await + .unwrap(); + let bundle = tempfile::tempdir().unwrap(); + owner + .export(None, Label::Restricted, bundle.path()) + .await + .unwrap(); + let home = tempfile::tempdir().unwrap(); + let there = mem(home.path()); + there.import(bundle.path(), None).await.unwrap(); + there.erase(&gone, Reason::Requested).await.unwrap(); + let again = there.import(bundle.path(), None).await.unwrap(); + assert_eq!(again.erased, vec![gone.clone()]); + assert!( + again.promotions.iter().all(|r| r.spec.entry == kept), + "a record about an entry that is not here is left out: {again:?}" + ); + the_ledger_keeps_no_why(home.path(), &gone, marker).await; + } + + /// A bundle of a later version of an entry that is here from an earlier bundle: the + /// entry is skipped, and its records stay out, since they are of a version that is + /// not here; a record of the version here is not shown as landed by mistake. + #[tokio::test] + async fn records_of_another_version_of_an_entry_here_stay_out() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let x = owner.write(req("The owner's fact.\n")).await.unwrap().id; + let lib = librarian(dir.path()); + lib.promote(&x, TrustTier::Curated, WHY).await.unwrap(); + let b1 = tempfile::tempdir().unwrap(); + owner + .export(None, Label::Restricted, b1.path()) + .await + .unwrap(); + let Promotion::Proposed { task } = lib + .promote(&x, TrustTier::Verified, "checked it twice") + .await + .unwrap() + else { + panic!("verified is proposed") + }; + owner.approve(&task.id).await.unwrap(); + let b2 = tempfile::tempdir().unwrap(); + owner + .export(None, Label::Restricted, b2.path()) + .await + .unwrap(); + let home = tempfile::tempdir().unwrap(); + let there = mem(home.path()); + there.import(b1.path(), None).await.unwrap(); + let second = there.import(b2.path(), None).await.unwrap(); + assert_eq!(second.skipped, vec![x.clone()]); + assert!(second.promotions.is_empty(), "{second:?}"); + assert_eq!( + there.get(&x).await.unwrap().front.trust_tier, + TrustTier::Curated, + "the version here is the first bundle's" + ); + let tiers: Vec<(TrustTier, bool)> = there + .promotions(&x) + .await + .unwrap() + .into_iter() + .map(|p| (p.record.spec.tier, p.landed)) + .collect(); + assert_eq!( + tiers, + vec![(TrustTier::Curated, true)], + "no record of a version that never landed here" + ); + } + + /// The bundle at `bundle` with `record` added to its promotion records, the manifest + /// made to agree: what a bundle written by hand can carry. + fn with_added_record(bundle: &Path, record: dm_ledger::PromotionRecord) { + let file = bundle.join(bundle::PROMOTIONS_FILE); + let mut records: Vec<dm_ledger::PromotionRecord> = + serde_json::from_str(&std::fs::read_to_string(&file).unwrap()).unwrap(); + records.push(record); + let text = serde_json::to_string_pretty(&records).unwrap(); + std::fs::write(&file, &text).unwrap(); + let mpath = bundle.join("manifest.yaml"); + let mut m = + dm_core::manifest::Manifest::from_yaml(&std::fs::read_to_string(&mpath).unwrap()) + .unwrap(); + m.promotions = None; + m.set_promotions(dm_core::manifest::ManifestPromotions { + path: bundle::PROMOTIONS_FILE.into(), + digest: dm_core::digest(&text), + records: records + .iter() + .map(|r| dm_core::manifest::ManifestPromotion { + entry: r.spec.entry.clone(), + classification: r.spec.classification, + }) + .collect(), + }) + .unwrap(); + std::fs::write(&mpath, m.to_yaml()).unwrap(); + } + + /// A bundle whose text of an entry is not the text here attaches no record to it: a + /// record is of a version, and a bundle written by hand says who it likes. + #[tokio::test] + async fn a_bundle_whose_text_differs_from_what_is_here_restores_nothing_for_it() { + let (_dir, owner, x, bundle) = promoted_then_exported(WHY, Label::Restricted).await; + let home = tempfile::tempdir().unwrap(); + let there = mem(home.path()); + there.import(bundle.path(), None).await.unwrap(); + // the entry moves on here: verified, so its text is no longer the bundle's + let Promotion::Proposed { task } = librarian(home.path()) + .promote(&x, TrustTier::Verified, "checked it here") + .await + .unwrap() + else { + panic!("verified is proposed") + }; + there.approve(&task.id).await.unwrap(); + let mut forged = owner.promotions(&x).await.unwrap().remove(0).record; + forged.spec.by = "agent:forger".into(); + forged.spec.why = "a why nobody here gave".into(); + forged.spec.tier = TrustTier::Verified; + with_added_record(bundle.path(), forged); + let again = there.import(bundle.path(), None).await.unwrap(); + assert_eq!(again.skipped, vec![x.clone()]); + assert!(again.promotions.is_empty(), "{again:?}"); + assert!( + there + .promotions(&x) + .await + .unwrap() + .iter() + .all(|p| p.record.spec.by != "agent:forger"), + "nothing of the bundle's attached to the entry here" + ); + } + + /// An erased stub that a bundle entry names, when the bundle's text is edited so the + /// receipt does not know it, gets no record: nothing of the entry comes back. + #[tokio::test] + async fn no_record_is_offered_for_an_erased_stub() { + let (dir, owner, x, bundle) = promoted_then_exported(WHY, Label::Restricted).await; + owner.erase(&x, Reason::Requested).await.unwrap(); + // the bundle's own entry comes home, with one word changed so its digest is not + // the receipt's + let file = bundle.path().join(format!("{x}.md")); + let text = std::fs::read_to_string(&file) + .unwrap() + .replace("The owner's fact.", "The owner's facts."); + std::fs::write(&file, &text).unwrap(); + let mpath = bundle.path().join("manifest.yaml"); + let mut m = + dm_core::manifest::Manifest::from_yaml(&std::fs::read_to_string(&mpath).unwrap()) + .unwrap(); + for e in &mut m.entries { + e.digest = dm_core::digest(&text); + } + std::fs::write(&mpath, m.to_yaml()).unwrap(); + let back = owner.import(bundle.path(), None).await.unwrap(); + assert_eq!(back.skipped, vec![x.clone()], "the stub is here: {back:?}"); + assert!(back.promotions.is_empty(), "{back:?}"); + the_ledger_keeps_no_why(dir.path(), &x, WHY).await; + } + + #[tokio::test] + async fn another_owner_s_import_lands_raw_without_the_whys() { + let (_dir, _owner, id, bundle) = promoted_then_exported(WHY, Label::Restricted).await; + let home = tempfile::tempdir().unwrap(); + let there = Memory::init(home.path(), "human:test:bob", "dom-me").unwrap(); + let im = there.import(bundle.path(), None).await.unwrap(); + let landed = &im.landed[&id]; + assert_eq!( + there.get(landed).await.unwrap().front.trust_tier, + TrustTier::Raw + ); + assert!( + there.promotions(landed).await.unwrap().is_empty(), + "the tier is not kept, and neither are the reasons for it" + ); + } + + #[tokio::test] + async fn a_changed_promotion_record_refuses_the_bundle() { + let (_dir, _owner, id, bundle) = promoted_then_exported(WHY, Label::Restricted).await; + let file = bundle.path().join("promotions.json"); + let text = std::fs::read_to_string(&file).unwrap(); + assert!(text.contains(WHY), "{text}"); + std::fs::write(&file, text.replace(WHY, "a why nobody gave")).unwrap(); + let home = tempfile::tempdir().unwrap(); + let there = mem(home.path()); + let refused = there.import(bundle.path(), None).await; + assert!(refused.is_err(), "{refused:?}"); + assert!(there.get(&id).await.is_err(), "nothing landed"); + } + + #[tokio::test] + async fn a_bundle_from_before_records_were_carried_still_imports() { + let (_dir, _owner, id, bundle) = promoted_then_exported(WHY, Label::Restricted).await; + // as a build from before records were carried wrote it: no file, no field + let mpath = bundle.path().join("manifest.yaml"); + let mut m = + dm_core::manifest::Manifest::from_yaml(&std::fs::read_to_string(&mpath).unwrap()) + .unwrap(); + m.promotions = None; + std::fs::write(&mpath, m.to_yaml()).unwrap(); + std::fs::remove_file(bundle.path().join(bundle::PROMOTIONS_FILE)).unwrap(); + assert!(!std::fs::read_to_string(&mpath) + .unwrap() + .contains("promotions")); + let home = tempfile::tempdir().unwrap(); + let there = mem(home.path()); + let im = there.import(bundle.path(), None).await.unwrap(); + let landed = &im.landed[&id]; + assert_eq!( + there.get(landed).await.unwrap().front.trust_tier, + TrustTier::Curated + ); + assert!(there.promotions(landed).await.unwrap().is_empty()); + } + + #[tokio::test] + async fn a_why_above_the_bound_or_for_an_entry_left_home_is_not_exported() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let secret = owner + .write(WriteRequest { + classification: Some(Label::Restricted), + ..req("The restricted fact.\n") + }) + .await + .unwrap() + .id; + let public = owner + .write(WriteRequest { + classification: Some(Label::Public), + ..req("The public fact.\n") + }) + .await + .unwrap() + .id; + let lib = librarian(dir.path()); + lib.promote(&secret, TrustTier::Curated, "the restricted entry's why") + .await + .unwrap(); + // read at restricted, so the why it gives next is restricted too + lib.get(&secret).await.unwrap(); + lib.promote(&public, TrustTier::Curated, "a why given at restricted") + .await + .unwrap(); + let bundle = tempfile::tempdir().unwrap(); + let ex = owner + .export(None, Label::Internal, bundle.path()) + .await + .unwrap(); + assert!(ex.manifest.entries.iter().any(|e| e.id == public)); + for why in ["the restricted entry's why", "a why given at restricted"] { + assert_eq!(bundle_holds(bundle.path(), why), None, "{why}"); + } + assert_eq!(ex.manifest.highest_label, Label::Public); + } + + #[tokio::test] + async fn an_erased_why_stays_erased_in_an_export() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let secret = owner + .write(req(&format!("The vault opens with {marker}.\n"))) + .await + .unwrap() + .id; + let kept = owner.write(req("The owner's fact.\n")).await.unwrap().id; + let lib = librarian(dir.path()); + lib.promote(&secret, TrustTier::Curated, &format!("tried {marker}")) + .await + .unwrap(); + lib.promote(&kept, TrustTier::Curated, WHY).await.unwrap(); + owner.erase(&secret, Reason::Requested).await.unwrap(); + let bundle = tempfile::tempdir().unwrap(); + owner + .export(None, Label::Restricted, bundle.path()) + .await + .unwrap(); + assert_eq!(bundle_holds(bundle.path(), marker), None); + let carried: serde_json::Value = serde_json::from_str( + &std::fs::read_to_string(bundle.path().join("promotions.json")).unwrap(), + ) + .unwrap(); + let of = |id: &str| { + carried + .as_array() + .unwrap() + .iter() + .find(|r| r["entry"] == id) + .cloned() + .unwrap_or_else(|| panic!("no record of {id}: {carried}")) + }; + assert_eq!(of(&secret)["why_erased"], true); + assert_eq!(of(&secret)["why"], ""); + assert_eq!(of(&kept)["why"], WHY); + } + + /// An export while an erasure is unfinished: the record holds the stub, and the index + /// and the ledger are not scrubbed yet. The bundle carries the stub, and its promotion + /// records without the why, as the scrub will leave them. + #[tokio::test] + async fn an_export_during_an_erasure_carries_no_why_of_the_stub() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let marker = "passphrase-Qz7xx"; + let secret = owner + .write(req("The vault opens at nine.\n")) + .await + .unwrap() + .id; + librarian(dir.path()) + .promote(&secret, TrustTier::Curated, &format!("tried {marker}")) + .await + .unwrap(); + // the first step of an erasure, and none after it + let (domain, entry) = owner.find(&secret).await.unwrap(); + let mut stub = erasure::stub(&entry, Reason::Requested); + stub.front.domain.clone_from(&domain); + stub.front.id.clone_from(&secret); + owner.store.erase(&domain, &secret, &stub).await.unwrap(); + let bundle = tempfile::tempdir().unwrap(); + owner + .export(None, Label::Restricted, bundle.path()) + .await + .unwrap(); + assert_eq!(bundle_holds(bundle.path(), marker), None); + let carried: serde_json::Value = serde_json::from_str( + &std::fs::read_to_string(bundle.path().join("promotions.json")).unwrap(), + ) + .unwrap(); + let record = &carried.as_array().unwrap()[0]; + assert_eq!(record["entry"], secret.as_str()); + assert_eq!(record["why_erased"], true); + } + + #[tokio::test] + async fn a_promotion_says_why() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let theirs = owner.write(req("The owner's fact.\n")).await.unwrap().id; + let agent = Memory::open(dir.path()) + .unwrap() + .with_author("agent:librarian"); + for to in [TrustTier::Curated, TrustTier::Verified] { + let err = agent.promote(&theirs, to, " ").await.unwrap_err(); + assert!(err.to_string().contains("why"), "{to}: {err}"); + } + assert_eq!( + agent.get(&theirs).await.unwrap().front.trust_tier, + TrustTier::Raw + ); + } + fn walk(dir: &Path) -> Vec<PathBuf> { let mut out = Vec::new(); for e in std::fs::read_dir(dir).unwrap().flatten() { @@ -3246,6 +5683,7 @@ "human:test:alice", Duration::minutes(5), &task.spec.requirements, + Label::Restricted, ) .await .unwrap(); @@ -3803,9 +6241,15 @@ async fn done(m: &Memory, s: TaskSpec) { let t = filed(m, s).await; let mut l = m.ledger.lock().await; - l.claim(&t.id, "w", Duration::minutes(5), &t.spec.requirements) - .await - .unwrap(); + l.claim( + &t.id, + "w", + Duration::minutes(5), + &t.spec.requirements, + Label::Restricted, + ) + .await + .unwrap(); l.complete(&t.id, "w", "mem-x").await.unwrap(); } @@ -3870,7 +6314,7 @@ .ledger .lock() .await - .claim_next("dom-me", "w2", Duration::minutes(5), &[]) + .claim_next("dom-me", "w2", Duration::minutes(5), &[], Label::Restricted) .await .unwrap() .unwrap(); @@ -4895,6 +7339,1153 @@ assert!(full.session_id().starts_with("sess-")); } + const EVERY_STATUS: [Status; 5] = [ + Status::Proposed, + Status::Admitted, + Status::Claimed, + Status::Completed, + Status::Invalidated, + ]; + + fn restricted(title: &str) -> TaskSpec { + TaskSpec { + classification: Label::Restricted, + ..tspec(title, 0, None, &[], &[]) + } + } + + /// Move a proposed task on the ledger to `st`. + async fn move_to(l: &mut Ledger, id: &str, st: Status) { + if st != Status::Proposed { + l.admit(id).await.unwrap(); + } + if matches!(st, Status::Claimed | Status::Completed) { + l.claim(id, "w", Duration::minutes(5), &[], Label::Restricted) + .await + .unwrap(); + } + if st == Status::Completed { + l.complete(id, "w", "mem-me-x").await.unwrap(); + } + if st == Status::Invalidated { + l.invalidate(id, "obsolete", None).await.unwrap(); + } + assert_eq!(l.get(id).await.unwrap().status, st); + } + + /// One restricted task in every status, filed straight on the ledger, and one public + /// task, admitted: the restricted ids, and the public one. + async fn restricted_in_every_status(m: &Memory) -> (Vec<String>, String) { + let mut l = m.ledger.lock().await; + let mut hidden = Vec::new(); + for (i, st) in EVERY_STATUS.iter().enumerate() { + let t = l + .create(restricted(&format!("Rotate signing key {i}"))) + .await + .unwrap() + .task; + move_to(&mut l, &t.id, *st).await; + hidden.push(t.id); + } + let shown = l + .create(tspec("Tidy the docs", 0, None, &[], &[])) + .await + .unwrap() + .task; + l.admit(&shown.id).await.unwrap(); + (hidden, shown.id) + } + + fn family_total(stats: &serde_json::Value) -> u64 { + stats["by_family"] + .as_object() + .unwrap() + .values() + .flat_map(|by_status| by_status.as_object().unwrap().values()) + .map(|n| n.as_u64().unwrap()) + .sum() + } + + #[tokio::test] + async fn a_task_above_the_session_s_handling_is_not_listed_counted_or_found() { + let dir = tempfile::tempdir().unwrap(); + let (hidden, shown) = restricted_in_every_status(&mem(dir.path())).await; + + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let mut listed = Vec::new(); + for st in EVERY_STATUS { + listed.extend(capped.tasks(None, st).await.unwrap()); + } + assert!(listed.iter().any(|t| t.id == shown), "{listed:?}"); + assert!( + listed.iter().all(|t| !hidden.contains(&t.id)), + "a restricted task is listed to a public session: {listed:?}" + ); + for id in &hidden { + // the answer a missing task gets, so it says nothing of what is there + assert!( + matches!( + capped.get_task(id).await, + Err(Error::Ledger(dm_ledger::Error::NotFound(_))) + ), + "{id} is found by a public session" + ); + } + let t0 = capped.t0(None).await.unwrap(); + assert!(t0.contains("- 1 tasks waiting"), "{t0}"); + assert_eq!( + family_total(&capped.task_stats(None).await.unwrap()), + 1, + "only the public task is counted" + ); + assert_eq!(capped.label(), Label::Public); + + let full = Memory::open(dir.path()).unwrap(); + assert_eq!(full.label(), Label::Public); + let admitted = full.tasks(None, Status::Admitted).await.unwrap(); + assert!(admitted.iter().any(|t| t.id == hidden[1]), "{admitted:?}"); + assert_eq!( + full.label(), + Label::Restricted, + "a listing raises the label to what it handed out" + ); + let one = Memory::open(dir.path()).unwrap(); + assert_eq!(one.get_task(&hidden[0]).await.unwrap().id, hidden[0]); + assert_eq!(one.label(), Label::Restricted, "a get_task is a read"); + } + + #[tokio::test] + async fn a_tree_a_retry_and_a_fan_out_leave_out_a_task_above_handling() { + let dir = tempfile::tempdir().unwrap(); + let (root, hidden) = { + let m = mem(dir.path()); + let root = m.create_task(spec("Root")).await.unwrap().task; + let mut l = m.ledger.lock().await; + let hidden = l + .create(TaskSpec { + origin: Some(format!("task:{}", root.id)), + upstream_event: Some(dm_core::digest("Rotate it.")[..16].to_string()), + ..restricted("Rotate the signing key") + }) + .await + .unwrap() + .task; + l.admit(&hidden.id).await.unwrap(); + l.claim( + &hidden.id, + "w", + Duration::minutes(5), + &[], + Label::Restricted, + ) + .await + .unwrap(); + l.complete(&hidden.id, "w", "mem-me-x").await.unwrap(); + (root, hidden) + }; + + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let tree = serde_json::to_string(&capped.task_tree(&root.id).await.unwrap()).unwrap(); + assert!(!tree.contains("signing key"), "{tree}"); + // the same spawn as a hidden child of this parent is withheld: the ledger holds + // it as that child, which is not handed out, and the answer names nothing of it + let f = capped + .fan_out(&root.id, &[("Again".into(), "Rotate it.".into())]) + .await + .unwrap(); + assert_eq!((f.children.len(), f.existing.len(), f.withheld), (0, 0, 1)); + assert!( + f.continuation.is_none(), + "nothing new, nothing to resume for" + ); + let answer = serde_json::to_string(&f).unwrap(); + assert!( + !answer.contains(&hidden.id) && !answer.contains("signing key"), + "{answer}" + ); + assert_eq!(capped.label(), Label::Public); + let retry = capped.retry_task(&root.id).await.unwrap(); + assert!( + !retry.spec.body.contains("signing key"), + "{}", + retry.spec.body + ); + assert_eq!(retry.spec.classification, Label::Public); + assert_eq!(capped.label(), Label::Public); + + let full = Memory::open(dir.path()).unwrap(); + let tree = full.task_tree(&root.id).await.unwrap(); + assert!(tree.children.iter().any(|c| c.task.id == hidden.id)); + assert_eq!(full.label(), Label::Restricted, "a tree is a read"); + let retry = full.retry_task(&root.id).await.unwrap(); + assert!( + retry.spec.body.contains("signing key"), + "{}", + retry.spec.body + ); + assert_eq!( + retry.spec.classification, + Label::Restricted, + "a retry that carries a restricted lesson is restricted" + ); + } + + /// A public root with an open restricted continuation, waiting on a restricted + /// child, all filed straight on the ledger: the root, the continuation, the child. + async fn hidden_continuation(m: &Memory) -> (Task, Task, Task) { + let root = m.create_task(spec("Root")).await.unwrap().task; + let mut l = m.ledger.lock().await; + let child = l + .create(TaskSpec { + origin: Some(format!("task:{}", root.id)), + upstream_event: Some("one".into()), + ..restricted("Rotate the signing key") + }) + .await + .unwrap() + .task; + let cont = l + .create(TaskSpec { + origin: Some(dm_ledger::continues(&root.id)), + upstream_event: Some("two".into()), + depends_on: vec![child.id.clone()], + ..restricted("Resume: Root") + }) + .await + .unwrap() + .task; + (root, cont, child) + } + + #[tokio::test] + async fn a_request_view_shares_the_session_id_and_starts_its_label_fresh() { + let dir = tempfile::tempdir().unwrap(); + let m = mem(dir.path()); + m.file_own(restricted("Rotate the signing key")) + .await + .unwrap(); + for st in [Status::Proposed, Status::Admitted] { + m.tasks(None, st).await.unwrap(); + } + assert_eq!(m.label(), Label::Restricted); + let first = m.for_request().unwrap(); + assert_eq!(first.session_id(), m.session_id()); + assert_eq!( + first.label(), + Label::Public, + "a view's label starts fresh, whatever the shared session read" + ); + for st in [Status::Proposed, Status::Admitted] { + first.tasks(None, st).await.unwrap(); + } + assert_eq!(first.label(), Label::Restricted); + let second = m.for_request().unwrap(); + assert_eq!(second.label(), Label::Public, "a view's label is its own"); + let t = second + .create_task(spec("Tidy the docs")) + .await + .unwrap() + .task; + assert_eq!(t.spec.classification, Label::Public); + assert!( + first.get_task(&t.id).await.is_ok(), + "one ledger behind every view" + ); + } + + /// With a hidden open continuation on the root, a spawn that matches its hidden + /// child and one that matches nothing get one and the same refusal: the answer + /// does not say which guess was right. + #[tokio::test] + async fn a_fan_out_refused_for_hidden_work_says_the_same_whatever_the_items() { + let dir = tempfile::tempdir().unwrap(); + let (root, cont, child) = { + let m = mem(dir.path()); + let root = m.create_task(spec("Root")).await.unwrap().task; + let mut l = m.ledger.lock().await; + let child = l + .create(TaskSpec { + origin: Some(format!("task:{}", root.id)), + upstream_event: Some(dm_core::digest("Rotate it.")[..16].to_string()), + ..restricted("Rotate the signing key") + }) + .await + .unwrap() + .task; + let cont = l + .create(TaskSpec { + origin: Some(dm_ledger::continues(&root.id)), + upstream_event: Some("two".into()), + depends_on: vec![child.id.clone()], + ..restricted("Resume: Root") + }) + .await + .unwrap() + .task; + (root, cont, child) + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let mut answers = Vec::new(); + for body in ["Rotate it.", "Something else."] { + match capped + .fan_out(&root.id, &[("Guess".into(), body.into())]) + .await + { + Err(Error::Denied(why)) => answers.push(why), + other => panic!("a fan-out over hidden open work: {other:?}"), + } + } + assert_eq!( + answers[0], answers[1], + "the refusal tells a right guess from a wrong one" + ); + for why in &answers { + assert!(!why.contains(&cont.id) && !why.contains(&child.id), "{why}"); + assert!(!why.contains("signing key"), "{why}"); + } + assert_eq!(capped.label(), Label::Public); + assert_eq!( + capped + .ledger + .lock() + .await + .filed_from(&root.id) + .await + .unwrap() + .len(), + 2, + "nothing is filed" + ); + } + + #[tokio::test] + async fn a_retry_fans_out_again_what_a_hidden_attempt_gave_up_on() { + let dir = tempfile::tempdir().unwrap(); + let (root, hidden) = { + let m = mem(dir.path()); + let root = m.create_task(spec("Root")).await.unwrap().task; + let mut l = m.ledger.lock().await; + let hidden = l + .create(TaskSpec { + origin: Some(format!("task:{}", root.id)), + upstream_event: Some(dm_core::digest("Rotate it.")[..16].to_string()), + ..restricted("Rotate the signing key") + }) + .await + .unwrap() + .task; + move_to(&mut l, &hidden.id, Status::Invalidated).await; + (root, hidden) + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let retry = capped.retry_task(&root.id).await.unwrap(); + let f = capped + .fan_out( + &retry.id, + &[ + ("Again".into(), "Rotate it.".into()), + ("New".into(), "Something new.".into()), + ], + ) + .await + .unwrap(); + assert_eq!( + (f.children.len(), f.existing.len(), f.withheld), + (2, 0, 0), + "a dropped hidden spawn is new work for the retry: {f:?}" + ); + let continuation = f.continuation.as_ref().expect("the retry is resumed"); + assert_eq!(continuation.spec.depends_on.len(), 2); + let answer = serde_json::to_string(&f).unwrap(); + assert!(!answer.contains(&hidden.id), "{answer}"); + assert_eq!(capped.label(), Label::Public); + } + + #[tokio::test] + async fn a_spawn_that_matches_hidden_open_work_is_withheld_and_the_rest_filed() { + let dir = tempfile::tempdir().unwrap(); + let (root, hidden) = { + let m = mem(dir.path()); + let root = m.create_task(spec("Root")).await.unwrap().task; + let mut l = m.ledger.lock().await; + let hidden = l + .create(TaskSpec { + origin: Some(format!("task:{}", root.id)), + upstream_event: Some(dm_core::digest("Rotate it.")[..16].to_string()), + ..restricted("Rotate the signing key") + }) + .await + .unwrap() + .task; + l.admit(&hidden.id).await.unwrap(); + (root, hidden) + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let f = capped + .fan_out( + &root.id, + &[ + ("Again".into(), "Rotate it.".into()), + ("New".into(), "Something new.".into()), + ], + ) + .await + .unwrap(); + assert_eq!( + (f.children.len(), f.existing.len(), f.withheld), + (1, 0, 1), + "{f:?}" + ); + assert_eq!(f.children[0].spec.title, "New"); + let continuation = f + .continuation + .as_ref() + .expect("the new child is waited for"); + assert_eq!( + continuation.spec.depends_on, + vec![f.children[0].id.clone()], + "the hidden task is not this session's to wait on" + ); + let answer = serde_json::to_string(&f).unwrap(); + assert!( + !answer.contains(&hidden.id) && !answer.contains("signing key"), + "{answer}" + ); + assert_eq!(capped.label(), Label::Public); + assert_eq!( + capped + .ledger + .lock() + .await + .get(&hidden.id) + .await + .unwrap() + .status, + Status::Admitted, + "the hidden task is left as it was" + ); + } + + #[tokio::test] + async fn a_fan_out_that_would_fold_a_hidden_continuation_is_refused() { + let dir = tempfile::tempdir().unwrap(); + let (root, cont, child) = hidden_continuation(&mem(dir.path())).await; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + match capped + .fan_out(&root.id, &[("More".into(), "Do more.".into())]) + .await + { + Err(Error::Denied(why)) => { + assert!(!why.contains(&cont.id) && !why.contains(&child.id), "{why}"); + } + other => panic!("a fan-out folded a hidden continuation: {other:?}"), + } + let l = capped.ledger.lock().await; + assert_eq!( + l.get(&cont.id).await.unwrap().status, + Status::Proposed, + "the hidden continuation is left as it was" + ); + assert_eq!( + l.filed_from(&root.id).await.unwrap().len(), + 2, + "nothing is filed" + ); + } + + #[tokio::test] + async fn the_rescore_report_counts_only_completed_work_the_session_may_see() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let survey = TaskSpec { + origin: Some("librarian:stale".into()), + upstream_event: Some("ev-1".into()), + ..spec("Survey") + }; + for s in [survey, restricted("Rotate the signing key")] { + let t = owner.file_own(s).await.unwrap().task; + if t.status == Status::Proposed { + owner.admit(&t.id).await.unwrap(); + } + owner.claim(&t.id, Duration::minutes(5)).await.unwrap(); + let h = owner + .write(WriteRequest { + entry_type: EntryType::Handoff, + body: "Done.\n".into(), + ..WriteRequest::default() + }) + .await + .unwrap(); + owner.complete(&t.id, &h.id).await.unwrap(); + } + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let r = capped.rescore(None).await.unwrap(); + // the one completed task it may see is curation; the restricted one is not counted + assert!( + (r.maintenance_share - 1.0).abs() < f64::EPSILON, + "{}", + r.maintenance_share + ); + } + + #[tokio::test] + async fn invalidating_a_task_raises_the_label_to_its_classification() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let t = owner + .file_own(TaskSpec { + classification: Label::Internal, + ..spec("Close me") + }) + .await + .unwrap() + .task; + let fresh = Memory::open(dir.path()).unwrap(); + assert_eq!(fresh.label(), Label::Public); + fresh.invalidate(&t.id, "obsolete", None).await.unwrap(); + assert_eq!(fresh.label(), Label::Internal); + } + + #[tokio::test] + async fn an_invalidation_that_would_close_hidden_work_is_refused() { + let dir = tempfile::tempdir().unwrap(); + let (root, cont, child) = hidden_continuation(&mem(dir.path())).await; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + match capped.invalidate(&root.id, "closed", None).await { + Err(Error::Denied(why)) => { + assert!(!why.contains(&cont.id) && !why.contains(&child.id), "{why}"); + assert!(!why.contains("signing key"), "{why}"); + } + other => panic!("an invalidation closed hidden work: {other:?}"), + } + { + let l = capped.ledger.lock().await; + for t in [&root, &cont, &child] { + assert_eq!( + l.get(&t.id).await.unwrap().status, + t.status, + "{} is left as it was", + t.id + ); + } + } + assert_eq!(capped.label(), Label::Public); + // hidden work already closed is not closed again, so it does not stand in the way + { + let mut l = capped.ledger.lock().await; + for t in [&cont, &child] { + l.invalidate(&t.id, "dropped", None).await.unwrap(); + } + } + let closed = capped.invalidate(&root.id, "closed", None).await.unwrap(); + assert_eq!(closed.status, Status::Invalidated); + } + + /// An internal task on the ledger, in `st`, held by `holder` when claimed. + async fn internal_in(m: &Memory, st: Status, holder: &str) -> String { + let mut l = m.ledger.lock().await; + let t = l + .create(TaskSpec { + classification: Label::Internal, + ..tspec("Tidy the internal docs", 0, None, &[], &[]) + }) + .await + .unwrap() + .task; + if st != Status::Proposed { + l.admit(&t.id).await.unwrap(); + } + if matches!(st, Status::Claimed | Status::Completed) { + l.claim(&t.id, holder, Duration::minutes(5), &[], Label::Restricted) + .await + .unwrap(); + } + if st == Status::Completed { + l.complete(&t.id, holder, "mem-me-x").await.unwrap(); + } + t.id + } + + #[tokio::test] + async fn a_refused_fan_out_does_not_raise_the_label_to_its_parent() { + let dir = tempfile::tempdir().unwrap(); + let root = { + let m = mem(dir.path()); + let mut l = m.ledger.lock().await; + let root = l + .create(TaskSpec { + classification: Label::Internal, + ..tspec("Root", 0, None, &[], &[]) + }) + .await + .unwrap() + .task; + let child = l + .create(TaskSpec { + origin: Some(format!("task:{}", root.id)), + upstream_event: Some("one".into()), + ..restricted("Rotate the signing key") + }) + .await + .unwrap() + .task; + l.create(TaskSpec { + origin: Some(dm_ledger::continues(&root.id)), + upstream_event: Some("two".into()), + depends_on: vec![child.id], + ..restricted("Resume: Root") + }) + .await + .unwrap(); + root.id + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Internal); + assert!(matches!( + capped + .fan_out(&root, &[("More".into(), "Do more.".into())]) + .await, + Err(Error::Denied(_)) + )); + assert_eq!( + capped.label(), + Label::Public, + "a refused fan-out read its parent" + ); + } + + #[tokio::test] + async fn a_refused_completion_does_not_raise_the_label() { + let dir = tempfile::tempdir().unwrap(); + let held = internal_in(&mem(dir.path()), Status::Claimed, "human:test:me").await; + let full = Memory::open(dir.path()).unwrap(); + assert!(matches!( + full.complete(&held, "mem-me-none").await, + Err(Error::NotFound(_)) + )); + assert_eq!( + full.label(), + Label::Public, + "a completion refused for its handoff read the task" + ); + } + + #[tokio::test] + async fn a_refused_invalidation_does_not_raise_the_label() { + let dir = tempfile::tempdir().unwrap(); + let done = internal_in(&mem(dir.path()), Status::Completed, "w").await; + let full = Memory::open(dir.path()).unwrap(); + assert!(matches!( + full.invalidate(&done, "late", None).await, + Err(Error::Ledger(dm_ledger::Error::WrongState { .. })) + )); + assert_eq!( + full.label(), + Label::Public, + "an invalidation refused by the ledger read the task" + ); + } + + #[tokio::test] + async fn a_refused_release_does_not_raise_the_label() { + let dir = tempfile::tempdir().unwrap(); + let theirs = internal_in(&mem(dir.path()), Status::Claimed, "w").await; + let full = Memory::open(dir.path()).unwrap(); + assert!(matches!( + full.release(&theirs, None).await, + Err(Error::Ledger(dm_ledger::Error::NotHolder { .. })) + )); + assert_eq!( + full.label(), + Label::Public, + "a release refused by the ledger read the task" + ); + } + + /// The floor lifts the best curation task over the top delivery task. When that + /// delivery task is hidden, the lift is not reported: its new rank is the hidden + /// task's rank plus one. + #[tokio::test] + async fn the_floor_s_report_leaves_out_a_lift_over_a_hidden_delivery_task() { + let dir = tempfile::tempdir().unwrap(); + let (hidden, curation) = { + let m = mem(dir.path()); + done(&m, tspec("Shipped", 0, None, &[], &[])).await; + let mut l = m.ledger.lock().await; + let hidden = l + .create(TaskSpec { + classification: Label::Restricted, + ..tspec("Rotate the signing key", 50, None, &[], &[]) + }) + .await + .unwrap() + .task; + let curation = l + .create(tspec( + "Resummarise", + -1, + Some("librarian:resummarise"), + &[], + &[], + )) + .await + .unwrap() + .task; + for t in [&hidden, &curation] { + l.admit(&t.id).await.unwrap(); + } + (hidden.id, curation.id) + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let r = capped.rescore(None).await.unwrap(); + assert!(r.maintenance_share < r.floor); + let text = serde_json::to_string(&r.changed).unwrap(); + assert!(!text.contains(&hidden), "{text}"); + assert!( + r.changed.iter().all(|c| c.id != curation), + "the floor's lift over a hidden task is reported: {text}" + ); + assert_eq!(capped.label(), Label::Public); + let full = Memory::open(dir.path()).unwrap(); + let lifted = full.get_task(&curation).await.unwrap(); + let top = full.get_task(&hidden).await.unwrap(); + assert_eq!( + lifted.spec.priority, + top.spec.priority + 1, + "the lift stands" + ); + } + + #[tokio::test] + async fn a_request_view_reads_within_the_policy_s_scope_not_the_grants() { + let dir = tempfile::tempdir().unwrap(); + let owner = mem(dir.path()); + let internal = { + let mut l = owner.ledger.lock().await; + l.create(TaskSpec { + classification: Label::Internal, + ..tspec("Tidy the internal docs", 0, None, &[], &[]) + }) + .await + .unwrap() + .task + .id + }; + // the root's own policy: the built-in one with agents narrowed to public + let text = dm_policy::decide::BUILTIN.replace( + "read_filter := base_filter\n", + "read_filter := object.union(base_filter, {\"handling\": \"public\"}) if class == \"agent\"\n\nread_filter := base_filter if class != \"agent\"\n", + ); + std::fs::create_dir_all(dir.path().join("policy")).unwrap(); + std::fs::write(dir.path().join("policy/local.rego"), text).unwrap(); + let agent = Memory::open(dir.path()) + .unwrap() + .with_author("agent:narrowed"); + assert!(agent.get_task(&internal).await.is_err()); + let view = agent.for_request().unwrap(); + assert!( + view.get_task(&internal).await.is_err(), + "a view read above the policy's scope, at the grants' handling" + ); + assert_eq!(view.label(), Label::Public); + } + + #[tokio::test] + async fn an_invalidation_is_refused_for_hidden_work_however_far_down() { + let dir = tempfile::tempdir().unwrap(); + let (root, child, grandchild) = { + let m = mem(dir.path()); + let mut l = m.ledger.lock().await; + let root = l + .create(tspec("Root", 0, None, &[], &[])) + .await + .unwrap() + .task; + let child = l + .create(tspec( + "Child", + 0, + Some(&format!("task:{}", root.id)), + &[], + &[], + )) + .await + .unwrap() + .task; + let grandchild = l + .create(TaskSpec { + origin: Some(format!("task:{}", child.id)), + upstream_event: Some("deep".into()), + ..restricted("Rotate the signing key") + }) + .await + .unwrap() + .task; + (root, child, grandchild) + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + match capped.invalidate(&root.id, "closed", None).await { + Err(Error::Denied(why)) => { + assert!( + !why.contains(&grandchild.id) && !why.contains("signing key"), + "{why}" + ); + } + other => panic!("an invalidation closed a hidden grandchild: {other:?}"), + } + let l = capped.ledger.lock().await; + for t in [&root, &child, &grandchild] { + assert_eq!(l.get(&t.id).await.unwrap().status, t.status, "{}", t.id); + } + } + + /// Root R has a hidden dropped child A with hash h; the retry Rt of R has a hidden + /// open child B with the same hash. A is elsewhere and dropped, so h is new work + /// under Rt by the chain's reading; the ledger then hands B back, and B is withheld. + #[tokio::test] + async fn a_spawn_the_ledger_dedups_to_a_hidden_task_is_withheld_not_handed_out() { + let dir = tempfile::tempdir().unwrap(); + let hash = dm_core::digest("Rotate it.")[..16].to_string(); + let (retry, b) = { + let m = mem(dir.path()); + let root = m.create_task(spec("Root")).await.unwrap().task; + let mut l = m.ledger.lock().await; + let a = l + .create(TaskSpec { + origin: Some(format!("task:{}", root.id)), + upstream_event: Some(hash.clone()), + ..restricted("Rotate the signing key") + }) + .await + .unwrap() + .task; + move_to(&mut l, &a.id, Status::Invalidated).await; + let retry = l + .create(TaskSpec { + domain: "dom-me".into(), + origin: Some(format!("retries:{}", root.id)), + ..spec("Root") + }) + .await + .unwrap() + .task; + let b = l + .create(TaskSpec { + origin: Some(format!("task:{}", retry.id)), + upstream_event: Some(hash.clone()), + ..restricted("Rotate the signing key again") + }) + .await + .unwrap() + .task; + (retry, b) + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let f = capped + .fan_out(&retry.id, &[("Again".into(), "Rotate it.".into())]) + .await + .unwrap(); + assert_eq!( + (f.children.len(), f.existing.len(), f.withheld), + (0, 0, 1), + "{f:?}" + ); + let answer = serde_json::to_string(&f).unwrap(); + assert!( + !answer.contains(&b.id) && !answer.contains("signing key"), + "{answer}" + ); + assert_eq!(capped.label(), Label::Public); + } + + #[tokio::test] + async fn the_views_of_a_session_count_supersedes_together() { + let dir = tempfile::tempdir().unwrap(); + mem(dir.path()); + let grants = dir.path().join("grants.toml"); + let text = std::fs::read_to_string(&grants).unwrap(); + std::fs::write( + &grants, + text.replace( + "max_supersedes_per_session = 20", + "max_supersedes_per_session = 2", + ), + ) + .unwrap(); + let agent = Memory::open(dir.path()).unwrap().with_author("agent:test"); + let (first, second) = (agent.for_request().unwrap(), agent.for_request().unwrap()); + let mut ids = Vec::new(); + for i in 0..3 { + ids.push(first.write(req(&format!("Fact {i}.\n"))).await.unwrap().id); + } + for id in &ids[..2] { + first + .write(WriteRequest { + supersedes: Some(id.clone()), + ..req("Rewritten.\n") + }) + .await + .unwrap(); + } + let err = second + .write(WriteRequest { + supersedes: Some(ids[2].clone()), + ..req("Rewritten.\n") + }) + .await + .expect_err("a view of the same session supersedes past the gate"); + assert!(err.to_string().contains("blast radius"), "{err}"); + agent.set_held_task("task-me-held"); + assert_eq!( + second.held.get().map(String::as_str), + Some("task-me-held"), + "the held task is the session's, shared with its views" + ); + } + + #[tokio::test] + async fn a_retry_is_classified_by_what_it_carries_not_by_what_it_passed() { + let dir = tempfile::tempdir().unwrap(); + let m = mem(dir.path()); + let (root, _, _) = hidden_continuation(&m).await; + let full = Memory::open(dir.path()).unwrap(); + let retry = full.retry_task(&root.id).await.unwrap(); + assert!(!retry.spec.body.contains("Lessons"), "{}", retry.spec.body); + assert_eq!( + retry.spec.classification, + Label::Public, + "open restricted tasks with no handoff are not in the retry" + ); + assert_eq!(full.label(), Label::Public); + } + + #[tokio::test] + async fn a_refused_operation_does_not_raise_the_label() { + let dir = tempfile::tempdir().unwrap(); + let held = { + let m = mem(dir.path()); + let mut l = m.ledger.lock().await; + let t = l.create(restricted("Held elsewhere")).await.unwrap().task; + move_to(&mut l, &t.id, Status::Claimed).await; + t.id + }; + let full = Memory::open(dir.path()).unwrap(); + assert!(matches!( + full.claim(&held, Duration::minutes(5)).await, + Err(Error::Ledger(dm_ledger::Error::WrongState { .. })) + )); + assert!(full.heartbeat(&held, Duration::minutes(5)).await.is_err()); + assert!(full.admit(&held).await.is_err()); + assert_eq!( + full.label(), + Label::Public, + "nothing was handed out, so nothing was read" + ); + } + + /// A visible blocker that a hidden task waits on, and a visible task that waits on a + /// visible blocker that the hidden one waits on too: what a public session's + /// rescore reports must not come from the hidden task. + #[tokio::test] + async fn a_rescore_reports_no_rank_lent_by_a_task_above_handling() { + let dir = tempfile::tempdir().unwrap(); + let (hidden, near, far) = { + let m = mem(dir.path()); + let mut l = m.ledger.lock().await; + let far = l + .create(tspec("Far blocker", 0, None, &[], &[])) + .await + .unwrap() + .task; + let near = l + .create(tspec("Near blocker", 0, None, &[&far.id], &[])) + .await + .unwrap() + .task; + let hidden = l + .create(TaskSpec { + classification: Label::Restricted, + ..tspec("Rotate the signing key", 50, None, &[&near.id], &[]) + }) + .await + .unwrap() + .task; + for t in [&far, &near, &hidden] { + l.admit(&t.id).await.unwrap(); + } + (hidden.id, near.id, far.id) + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let r = capped.rescore(None).await.unwrap(); + let text = serde_json::to_string(&r.changed).unwrap(); + assert!(!text.contains(&hidden), "{text}"); + assert!( + r.changed.iter().all(|c| c.id != near && c.id != far), + "a rank lent by a hidden task is reported: {text}" + ); + let full = Memory::open(dir.path()).unwrap(); + // the ranks moved all the same: the queue is ranked whole + let near = full.get_task(&near).await.unwrap(); + assert!(near.spec.priority > 50, "{near:?}"); + } + + #[tokio::test] + async fn a_rescore_reports_no_change_to_a_task_above_handling() { + let dir = tempfile::tempdir().unwrap(); + let hidden = { + let m = mem(dir.path()); + let mut l = m.ledger.lock().await; + let hidden = l + .create(restricted("Rotate the signing key")) + .await + .unwrap() + .task; + let shown = l + .create(tspec("Tidy the docs", 50, None, &[&hidden.id], &[])) + .await + .unwrap() + .task; + for t in [&hidden, &shown] { + l.admit(&t.id).await.unwrap(); + } + hidden.id + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let r = capped.rescore(None).await.unwrap(); + assert!(r.changed.iter().all(|c| c.id != hidden), "{:?}", r.changed); + assert_eq!(capped.label(), Label::Public); + } + + #[tokio::test] + async fn a_filing_that_dedups_against_a_task_above_handling_does_not_return_it() { + let dir = tempfile::tempdir().unwrap(); + let hidden = { + let m = mem(dir.path()); + let mut l = m.ledger.lock().await; + l.create(TaskSpec { + origin: Some("github:acme/x#7".into()), + upstream_event: Some("opened".into()), + ..restricted("Rotate the signing key") + }) + .await + .unwrap() + .task + }; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let filed = capped + .create_task(TaskSpec { + origin: Some("github:acme/x#7".into()), + upstream_event: Some("opened".into()), + ..tspec("Same event", 0, None, &[], &[]) + }) + .await; + match filed { + Err(Error::Denied(why)) => assert!(!why.contains("signing key"), "{why}"), + other => panic!("{other:?}"), + } + assert_eq!( + capped.get_task(&hidden.id).await.map(|t| t.status).ok(), + None + ); + assert_eq!(capped.label(), Label::Public); + } + + /// A restricted task at priority 10 and a public one at 1, both admitted. + async fn restricted_over_public(m: &Memory) -> (String, String) { + let mut l = m.ledger.lock().await; + let mut ids = Vec::new(); + for s in [ + TaskSpec { + priority: 10, + ..restricted("Rotate the signing key") + }, + tspec("Tidy the docs", 1, None, &[], &[]), + ] { + let t = l.create(s).await.unwrap().task; + l.admit(&t.id).await.unwrap(); + ids.push(t.id); + } + (ids.remove(0), ids.remove(0)) + } + + #[tokio::test] + async fn claim_next_offers_a_task_only_to_a_session_whose_handling_covers_it() { + let dir = tempfile::tempdir().unwrap(); + let (hidden, shown) = restricted_over_public(&mem(dir.path())).await; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + let c = capped + .claim_next(None, Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + assert_eq!(c.task.id, shown, "the top task it can handle"); + assert!(capped + .claim_next(None, Duration::minutes(5)) + .await + .unwrap() + .is_none()); + assert_eq!(capped.label(), Label::Public); + let full = Memory::open(dir.path()).unwrap(); + let c = full + .claim_next(None, Duration::minutes(5)) + .await + .unwrap() + .unwrap(); + assert_eq!(c.task.id, hidden); + assert_eq!(full.label(), Label::Restricted); + } + + #[tokio::test] + async fn a_named_claim_above_the_session_s_handling_is_not_found() { + let dir = tempfile::tempdir().unwrap(); + let (hidden, _) = restricted_over_public(&mem(dir.path())).await; + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(Label::Public); + assert!( + matches!( + capped.claim(&hidden, Duration::minutes(5)).await, + Err(Error::Ledger(dm_ledger::Error::NotFound(_))) + ), + "a public session claims a restricted task" + ); + assert_eq!(capped.label(), Label::Public); + let full = Memory::open(dir.path()).unwrap(); + full.claim(&hidden, Duration::minutes(5)).await.unwrap(); + assert_eq!(full.label(), Label::Restricted); + } + #[tokio::test] async fn reindex_rebuilds_from_the_record_alone() { let dir = tempfile::tempdir().unwrap();
diff --git a/crates/server/src/log.rs b/crates/server/src/log.rs index be711f9..475bc96 100644 --- a/crates/server/src/log.rs +++ b/crates/server/src/log.rs
@@ -185,7 +185,9 @@ impl Memory { /// Write a version of an entry through its domain's log: the change is appended - /// first, then the record takes it. Every record write of an entry comes here. + /// first, then the record takes it. Every record write of an entry comes here. A + /// version the record refuses over an erased stub is followed in the log by the + /// erasure again. pub(crate) async fn put_logged(&self, entry: &Entry) -> Result<(), Error> { append( &self.root, @@ -197,7 +199,14 @@ active: entry.is_active(), }, )?; - self.store.put(entry).await?; + if let Err(e) = self.store.put(entry).await { + // the record keeps the stub of an erased entry: the log says so again, after + // the version just appended, so it does not name a version the record refused + if matches!(e, dm_store::Error::Erased { .. }) { + self.log_erased(&entry.front.domain, &entry.front.id)?; + } + return Err(e.into()); + } // the record marks the entry this one replaces as superseded as it takes it: that // version goes in the log too, so the log says what the record holds. The digest is // of the stored text: the record leaves an inactive entry as it was written, which @@ -570,4 +579,28 @@ let done = m.materialize(None).await.unwrap(); assert!(done.behind.is_empty(), "{done:?}"); } + + /// A writer that read an entry before it was erased writes a version of it: the + /// record refuses it, and the log says what the record holds, that the entry is + /// erased, not that the version it refused is live. + #[tokio::test] + async fn a_write_the_record_refuses_over_a_stub_leaves_the_log_saying_erased() { + let dir = tempfile::tempdir().unwrap(); + let m = Memory::init(dir.path(), "human:test:me", "dom-me").unwrap(); + let id = m.write(req("# Key\n\nOn the wiki.\n")).await.unwrap().id; + let (_, mut read) = m.find(&id).await.unwrap(); + m.erase(&id, dm_core::erasure::Reason::Mistaken) + .await + .unwrap(); + read.front.trust_tier = dm_core::TrustTier::Curated; + let refused = m.put_logged(&read).await; + assert!( + matches!(&refused, Err(Error::Store(dm_store::Error::Erased { .. }))), + "{refused:?}" + ); + let events = m.log(None, 0).unwrap(); + assert_eq!(state(&events).get(&id), Some(&None), "{events:?}"); + let done = m.materialize(None).await.unwrap(); + assert!(done.behind.is_empty(), "{done:?}"); + } }
diff --git a/crates/server/src/mcp.rs b/crates/server/src/mcp.rs index cfa7822..829f087 100644 --- a/crates/server/src/mcp.rs +++ b/crates/server/src/mcp.rs
@@ -194,9 +194,9 @@ pub id: String, /// `curated` or `verified`. pub to: String, - /// One sentence of why: what was checked, and against what. - #[serde(default)] - pub why: Option<String>, + /// One sentence of why: what was checked, and against what. Required: the ledger + /// keeps it beside the entry. + pub why: String, } /// Parameters of `write`. @@ -426,6 +426,7 @@ "t0", "recall", "transcript_turn", + "promotions", "task_tree", "get_task", "list_tasks", @@ -665,13 +666,21 @@ .ok_or_else(|| McpError::invalid_params("to: curated or verified", None))?; let r = self .memory - .promote(&p.id, to, p.why.as_deref().unwrap_or("")) + .promote(&p.id, to, &p.why) .await .map_err(mcp_err)?; ok(serde_json::to_value(r).unwrap_or_default()) } #[tool( + description = "Why each promotion of an entry was made, oldest first: who promoted which version to which tier, the reason they gave, and whether that version reached the record (`landed`). A proposal to verified is listed from when it is made, with `landed` false until a person approves it. As an auditor, read it after you have judged the change yourself, not before, so the reasons do not lead your judgement. Only for an entry you may read, replaced ones included; reading it raises your label as a get does." + )] + async fn promotions(&self, p: Parameters<GetParams>) -> Result<CallToolResult, McpError> { + let r = self.memory.promotions(&p.0.id).await.map_err(mcp_err)?; + ok(json!({ "promotions": r })) + } + + #[tool( description = "Entries of one kind, newest first: the latest handoffs, the standing decisions. A keyed lookup, not a search." )] async fn list(&self, p: Parameters<ListParams>) -> Result<CallToolResult, McpError> { @@ -737,7 +746,7 @@ } #[tool( - description = "Fan a held task out: file one task per item as its subagents' work, and a continuation that resumes the parent when they are done, with their handoffs in its prompt. Filing the same prompt twice is one task." + description = "Fan a held task out: file one task per item as its subagents' work, and a continuation that resumes the parent when they are done, with their handoffs in its prompt. Filing the same prompt twice is one task: an item asked for before in this chain comes back in `existing`, and `withheld` counts the items held back because the chain holds them in work this session may not read." )] async fn fan_out(&self, p: Parameters<FanOutParams>) -> Result<CallToolResult, McpError> { let p = p.0; @@ -747,7 +756,10 @@ .fan_out(&p.parent, &items) .await .map_err(mcp_err)?; - ok(json!({ "children": f.children, "continuation": f.continuation })) + ok(json!({ + "children": f.children, "existing": f.existing, "withheld": f.withheld, + "continuation": f.continuation, + })) } #[tool( @@ -1071,6 +1083,7 @@ "fan_out", "task_tree", "retry_task", + "promotions", ] { assert!( names.contains(&expected), @@ -1435,4 +1448,139 @@ assert_ne!(invalid.code, internal.code); assert_eq!(t0_uri("dom-a"), "memory://t0/dom-a"); } + + /// A root with one restricted task and one public task, both admitted, and a server + /// over a session capped at public: the restricted task's id, and the server. + async fn restricted_and_capped() -> (tempfile::TempDir, String, McpServer) { + let (dir, _owner) = server().await; + let owner = Memory::open(dir.path()).unwrap(); + let mut ids = Vec::new(); + for (title, classification) in [ + ("Rotate the signing key", dm_core::Label::Restricted), + ("Tidy the docs", dm_core::Label::Public), + ] { + let t = owner + .file_own(dm_ledger::TaskSpec { + domain: "dom-me".into(), + title: title.into(), + body: "Do it.".into(), + requirements: vec![], + depends_on: vec![], + origin: None, + upstream_event: None, + created_by: String::new(), + classification, + priority: 0, + }) + .await + .unwrap() + .task; + if t.status == dm_ledger::Status::Proposed { + owner.admit(&t.id).await.unwrap(); + } + ids.push(t.id); + } + let capped = Memory::open(dir.path()) + .unwrap() + .with_handling_cap(dm_core::Label::Public); + (dir, ids.swap_remove(0), McpServer::new(Arc::new(capped))) + } + + #[tokio::test] + async fn a_capped_session_neither_lists_nor_gets_a_task_above_its_handling() { + let (_dir, hidden, s) = restricted_and_capped().await; + let listed = payload( + &s.list_tasks(Parameters(ListTasksParams::default())) + .await + .unwrap(), + ); + let text = listed.to_string(); + assert!(text.contains("Tidy the docs"), "{text}"); + assert!(!text.contains("signing key"), "{text}"); + let refused = s + .get_task(Parameters(TaskIdParams { id: hidden.clone() })) + .await + .map(|r| payload(&r)) + .unwrap_err(); + let absent = s + .get_task(Parameters(TaskIdParams { + id: "task-me-none".into(), + })) + .await + .unwrap_err(); + assert_eq!(refused.code, absent.code, "the answer a missing task gets"); + assert_eq!( + refused.message.replace(&hidden, "<id>"), + absent.message.replace("task-me-none", "<id>") + ); + assert_eq!(s.memory.label(), dm_core::Label::Public); + } + + #[tokio::test] + async fn a_spawn_asked_for_again_over_mcp_comes_back_in_existing() { + let (_dir, s) = server().await; + let parent = s + .memory + .create_task(dm_ledger::TaskSpec { + domain: "dom-me".into(), + title: "Root".into(), + body: "Do it.".into(), + requirements: vec![], + depends_on: vec![], + origin: None, + upstream_event: None, + created_by: String::new(), + classification: dm_core::Label::Public, + priority: 0, + }) + .await + .unwrap() + .task; + let ask = || { + Parameters(FanOutParams { + parent: parent.id.clone(), + items: vec![FanOutItem { + title: "Rotate".into(), + body: "Rotate it.".into(), + }], + }) + }; + let first = payload(&s.fan_out(ask()).await.unwrap()); + let child = first["children"][0]["id"].as_str().unwrap().to_string(); + assert_eq!( + first["existing"].as_array().map(Vec::len), + Some(0), + "{first}" + ); + assert_eq!(first["withheld"], 0, "{first}"); + let again = payload(&s.fan_out(ask()).await.unwrap()); + assert_eq!( + again["children"].as_array().map(Vec::len), + Some(0), + "{again}" + ); + assert_eq!( + again["existing"][0]["id"], child, + "a spawn asked for again comes back as the earlier task: {again}" + ); + assert!(again["continuation"].is_null(), "{again}"); + } + + #[tokio::test] + async fn claim_next_over_mcp_skips_a_task_above_the_session_s_handling() { + let (_dir, _hidden, s) = restricted_and_capped().await; + let claimed = payload( + &s.claim_next(Parameters(ClaimNextParams::default())) + .await + .unwrap(), + ); + assert_eq!(claimed["task"]["title"], "Tidy the docs", "{claimed}"); + let none = payload( + &s.claim_next(Parameters(ClaimNextParams::default())) + .await + .unwrap(), + ); + assert!(none["task"].is_null(), "{none}"); + assert_eq!(s.memory.label(), dm_core::Label::Public); + } }
diff --git a/crates/server/src/taps.rs b/crates/server/src/taps.rs index 3f94ff0..811c8de 100644 --- a/crates/server/src/taps.rs +++ b/crates/server/src/taps.rs
@@ -9,9 +9,10 @@ impl Memory { /// Completed tasks in a granted domain that came from upstream and whose handoff has - /// not been posted back, oldest first, each with its handoff. A task above `ceiling`, - /// or one whose handoff is, is left out rather than read; so is a task whose handoff - /// is gone. Each handoff handed out is a read, logged and counted like any other. + /// not been posted back, oldest first, each with its handoff. A task above `ceiling` + /// or the session's handling, or one whose handoff is, is left out rather than read; + /// so is a task whose handoff is gone. Each task and handoff handed out is a read, + /// and raises the label; each handoff is logged and counted like any other. /// /// # Errors /// @@ -24,10 +25,10 @@ let domain = self.resolve_domain(Some(domain))?; let tasks = self.ledger.lock().await.unposted(&domain).await?; let mut filter = self.session_filter()?; - filter.handling = filter.handling.min(ceiling); + filter.handling = self.handling()?.min(ceiling); let mut out = Vec::new(); for task in tasks { - if task.spec.classification > ceiling { + if task.spec.classification > filter.handling { continue; } let Some(handoff) = task.handoff.as_deref() else { @@ -50,6 +51,7 @@ } out.push((task, entry)); } + out.iter().for_each(|(task, _)| self.observe_task(task)); Ok(out) } @@ -61,7 +63,8 @@ /// /// [`Error::Denied`] for a task of another domain, above the ceiling, or a `to` /// that is not one line of at most 2048 characters; a ledger error, including a - /// task that is not completed or was posted already; or a task outside the grants. + /// task that is not completed or was posted already, and a task outside the grants + /// or above the session's handling, which is not found. pub async fn post( &self, id: &str, @@ -76,7 +79,7 @@ } let mut l = self.ledger.lock().await; let task = l.get(id).await?; - self.task_in_grants(&task)?; + self.check_task(&task)?; if task.spec.domain != domain { return Err(Error::Denied(format!("{id} is not a task of {domain}"))); } @@ -86,7 +89,9 @@ task.spec.classification ))); } - Ok(l.post(id, to).await?) + let posted = l.post(id, to).await?; + self.observe_task(&posted); + Ok(posted) } } @@ -294,4 +299,125 @@ .await .is_ok()); } + + #[tokio::test] + async fn a_tap_s_handling_bounds_the_tasks_it_is_handed_and_may_post() { + let dir = tempfile::tempdir().unwrap(); + let m = mem(dir.path()); + // a restricted task whose handoff is internal: only the task is above the tap + let h = m + .write(WriteRequest { + entry_type: EntryType::Handoff, + body: "Done.\n".into(), + classification: Some(Label::Internal), + ..WriteRequest::default() + }) + .await + .unwrap(); + let id = { + let mut l = m.ledger.lock().await; + let t = l + .create(TaskSpec { + domain: "dom-me".into(), + title: "Rotate the signing key".into(), + body: "Do it.\n".into(), + requirements: vec![], + depends_on: vec![], + origin: Some("github:acme/x#9".into()), + upstream_event: Some("open:2026-10-01T09:00:00Z".into()), + created_by: String::new(), + classification: Label::Restricted, + priority: 0, + }) + .await + .unwrap() + .task; + l.admit(&t.id).await.unwrap(); + l.claim( + &t.id, + "w", + time::Duration::minutes(5), + &[], + Label::Restricted, + ) + .await + .unwrap(); + l.complete(&t.id, "w", &h.id).await.unwrap(); + t.id + }; + let tap = Memory::open(dir.path()) + .unwrap() + .with_author("tap:comments") + .with_handling_cap(Label::Internal) + .without_read_log(); + let due = tap.unposted("dom-me", Label::Restricted).await.unwrap(); + assert!( + due.iter().all(|(t, _)| t.id != id), + "a task above the tap's handling is handed out: {due:?}" + ); + assert!(tap + .post(&id, "https://x/9", "dom-me", Label::Restricted) + .await + .is_err()); + assert_eq!(m.get_task(&id).await.unwrap().posted_to, None); + } + + #[tokio::test] + async fn a_task_handed_to_a_tap_raises_its_label_as_its_handoff_does() { + let dir = tempfile::tempdir().unwrap(); + let m = mem(dir.path()); + // an internal task whose handoff is public: only the task raises the label + let h = m + .write(WriteRequest { + entry_type: EntryType::Handoff, + body: "Done.\n".into(), + classification: Some(Label::Public), + ..WriteRequest::default() + }) + .await + .unwrap(); + let id = { + let mut l = m.ledger.lock().await; + let t = l + .create(TaskSpec { + domain: "dom-me".into(), + title: "Internal work".into(), + body: "Do it.\n".into(), + requirements: vec![], + depends_on: vec![], + origin: Some("github:acme/x#4".into()), + upstream_event: Some("open:2026-10-01T09:00:00Z".into()), + created_by: String::new(), + classification: Label::Internal, + priority: 0, + }) + .await + .unwrap() + .task; + l.admit(&t.id).await.unwrap(); + l.claim( + &t.id, + "w", + time::Duration::minutes(5), + &[], + Label::Restricted, + ) + .await + .unwrap(); + l.complete(&t.id, "w", &h.id).await.unwrap(); + t.id + }; + let tap = Memory::open(dir.path()) + .unwrap() + .with_author("tap:comments") + .without_read_log(); + let due = tap.unposted("dom-me", Label::Internal).await.unwrap(); + assert_eq!(due.len(), 1); + assert_eq!(due[0].0.id, id); + assert_eq!( + tap.label(), + Label::Internal, + "the task handed out is read, not only its handoff" + ); + } }
diff --git a/crates/server/src/transcripts.rs b/crates/server/src/transcripts.rs index c0dd847..24a81db 100644 --- a/crates/server/src/transcripts.rs +++ b/crates/server/src/transcripts.rs
@@ -937,6 +937,8 @@ removed, flagged: Vec::new(), unresolved: vec![erasure::UNFINISHED.into()], + promotion_whys_removed: 0, + promotion_tasks_cleared: 0, }; self.put_transcript_receipt(&pending).await?; Ok(pending)
diff --git a/crates/store/src/lib.rs b/crates/store/src/lib.rs index 2752b0e..3d94caf 100644 --- a/crates/store/src/lib.rs +++ b/crates/store/src/lib.rs
@@ -79,6 +79,23 @@ /// The id asked for. id: String, }, + /// Asked to write a version with text over an entry that was erased, when the put + /// reads the stub: a writer that read the entry before the erasure does not write + /// what it holds of it. A put whose read comes before the erasure's write is not + /// refused: see [`Store::put`]. + #[error("entry {id} in {domain} is erased: no version with text is written over its stub")] + Erased { + /// The domain of the stub. + domain: String, + /// Its id. + id: String, + }, +} + +/// Whether `text`, the current version of an entry, is an erased stub. Text that is not +/// an entry is not one: the write is then refused or accepted elsewhere. +fn is_erased(text: &str) -> bool { + Entry::parse(text).is_ok_and(|e| e.front.status == Some(Status::Erased)) } /// The outcome of a write. @@ -225,9 +242,15 @@ /// already current, nothing is written. If the entry supersedes another that is still /// active, that one is marked superseded, which is a write of its own. /// + /// A version with text is refused over an erased stub that this put reads. The read + /// and the writes after it are not one step: an [`Store::erase`] between them is not + /// seen. The version read before the erasure then goes into the history, and this + /// version, with its text, is written over the stub, so the entry is live again. + /// /// # Errors /// - /// [`Error::BadName`] for a domain or id that cannot be a path, or a backend error. + /// [`Error::BadName`] for a domain or id that cannot be a path, [`Error::Erased`] for + /// a version with text over an erased entry's stub, or a backend error. pub async fn put(&self, entry: &Entry) -> Result<Written, Error> { let domain = entry.front.domain.as_str(); let id = entry.front.id.as_str(); @@ -244,6 +267,19 @@ changed: false, }); } + // a writer that read the entry before the erasure, and now writes a version of + // what it read, would bring the text back over the stub. The check holds only when + // this read sees the stub: an erasure that writes its stub between this read and + // the writes below is not seen. The text read here then goes into the history, + // and this version is written over the stub, so the entry is live again with its + // text. Closing that needs a conditional put or a lock for each entry + if entry.front.status != Some(Status::Erased) && previous.as_deref().is_some_and(is_erased) + { + return Err(Error::Erased { + domain: domain.to_string(), + id: id.to_string(), + }); + } let mut archived = None; if let Some(old) = previous { @@ -673,6 +709,47 @@ assert_eq!(s.get("dom-a", "mem-a-9").await.unwrap(), None); } + /// A writer that read an entry before its erasure, and writes a version of what it + /// read after, is refused: the stub stays, and so does the history as the erasure + /// left it. A later erased version, the stub again under another reason, is written. + #[tokio::test] + async fn no_version_with_text_is_written_over_an_erased_stub() { + let s = Store::in_memory().unwrap(); + let e = entry( + "dom-a", + "mem-a-1", + "The vault opens with passphrase-Qz7xx.\n", + ); + s.put(&e).await.unwrap(); + let before = e.clone(); + let mut stub = entry("dom-a", "mem-a-1", ""); + stub.front.status = Some(Status::Erased); + s.put(&stub).await.unwrap(); + let mut promoted = before; + promoted.front.trust_tier = TrustTier::Verified; + let refused = s.put(&promoted).await; + assert!( + matches!(&refused, Err(Error::Erased { id, .. }) if id == "mem-a-1"), + "{refused:?}" + ); + assert_eq!( + s.get("dom-a", "mem-a-1").await.unwrap(), + Some(stub.clone()), + "the stub stays" + ); + assert_eq!( + s.digests("dom-a", "mem-a-1").await.unwrap().len(), + 2, + "and no version was added" + ); + let mut again = stub.clone(); + again.front.title = Some("erased".into()); + assert!( + s.put(&again).await.unwrap().changed, + "a stub over a stub is written" + ); + } + #[tokio::test] async fn every_write_is_a_new_version_and_the_old_text_survives() { let s = Store::in_memory().unwrap(); @@ -958,6 +1035,8 @@ removed, flagged: vec![], unresolved: vec![], + promotion_whys_removed: 0, + promotion_tasks_cleared: 0, }; s.put_receipt(&receipt).await.unwrap(); assert!(receipt.to_yaml().contains("body_digest:"));
diff --git a/crates/taps/tests/github_issues.rs b/crates/taps/tests/github_issues.rs index 21e26c7..af9fca5 100644 --- a/crates/taps/tests/github_issues.rs +++ b/crates/taps/tests/github_issues.rs
@@ -169,8 +169,12 @@ .await } +/// The proposed tasks, listed by a session of the owner's own: a listing is a read and +/// raises the label of the session that lists, so the tap's session does not list. async fn proposed(memory: &Memory) -> Vec<dm_ledger::Task> { - memory + Memory::open(memory.root()) + .unwrap() + .without_read_log() .tasks(Some("dom-widgets"), dm_ledger::Status::Proposed) .await .unwrap()
diff --git a/crates/web/Cargo.toml b/crates/web/Cargo.toml index 7fabc6a..b35bc5e 100644 --- a/crates/web/Cargo.toml +++ b/crates/web/Cargo.toml
@@ -31,6 +31,7 @@ wasm-bindgen-cli-support = "=0.2.129" [dev-dependencies] +dm-index = { version = "0.1.0", path = "../index" } tempfile = "3.27.0" tokio-tungstenite = "0.30" wasm-bindgen-cli-support = "=0.2.129"
diff --git a/crates/web/src/lib.rs b/crates/web/src/lib.rs index ea5e460..fc1564d 100644 --- a/crates/web/src/lib.rs +++ b/crates/web/src/lib.rs
@@ -209,9 +209,25 @@ } } +/// A request's own session ([`App::session`]), or the response that says why there is +/// none, returned from the handler. +macro_rules! session { + ($app:expr) => { + match $app.session() { + Ok(m) => m, + Err(e) => { + return (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(); + } + } + }; +} + #[derive(Clone)] struct App { config: Config, + /// The board's session for its life: the handles, the session id for the read and + /// write logs, and the read scope. Each request gets a view of it with a label of + /// its own ([`App::session`]). memory: Arc<Memory>, sessions: Arc<Mutex<BTreeMap<String, Arc<Session>>>>, /// Bundle exports run one at a time: each replaces a directory by renaming. @@ -480,6 +496,15 @@ } impl App { + /// The board's session for one request: the same handles and session id as every + /// other request, so what the board reads and then writes is joined in the logs, + /// with a label of its own. What one request reads raises the label of that request + /// only: a listing does not classify what a later request files or writes, as one + /// long-lived label would. + fn session(&self) -> Result<Memory, dm_server::Error> { + self.memory.for_request() + } + fn start(&self, req: &NewSession) -> Result<Arc<Session>, Error> { let argv = launch_argv(&self.config, req).map_err(Error::Request)?; let id = new_id()?; @@ -850,7 +875,8 @@ /// Recent entries, newest first: what the pane watches. async fn entries(State(app): State<App>) -> Response { - match app.memory.list(None, None, 30).await { + let m = session!(app); + match m.list(None, None, 30).await { Ok(hits) => Json( hits.iter() .map(|h| { @@ -868,7 +894,8 @@ /// One entry in full, for the pane. async fn entry(State(app): State<App>, Path(id): Path<String>) -> Response { - match app.memory.get(&id).await { + let m = session!(app); + match m.get(&id).await { Ok(e) => Json(json!({ "id": e.front.id, "title": e.title(), "type": e.front.entry_type.to_string(), "author": e.front.author, "classification": e.front.classification.to_string(), @@ -902,8 +929,9 @@ /// Every task of the domain, whatever its status, newest first within a status, with /// the domain named: what the board lists and nests by origin. async fn tasks(State(app): State<App>) -> Response { + let m = session!(app); use dm_ledger::Status; - let domain = match app.memory.resolve_domain(None) { + let domain = match m.resolve_domain(None) { Ok(d) => d, Err(e) => return task_error(&e), }; @@ -915,7 +943,7 @@ Status::Completed, Status::Invalidated, ] { - match app.memory.tasks(Some(&domain), status).await { + match m.tasks(Some(&domain), status).await { Ok(mut t) => all.append(&mut t), Err(e) => return task_error(&e), } @@ -936,6 +964,7 @@ } async fn file_task(State(app): State<App>, Json(req): Json<FileTask>) -> Response { + let m = session!(app); if req.title.trim().is_empty() { return (StatusCode::BAD_REQUEST, "a task needs a title").into_response(); } @@ -951,21 +980,23 @@ classification: dm_core::Label::Public, priority: req.priority.unwrap_or(0), }; - match app.memory.create_task(spec).await { + match m.create_task(spec).await { Ok(c) => Json(json!({ "task": c.task, "deduplicated": c.deduplicated })).into_response(), Err(e) => task_error(&e), } } async fn task_tree(State(app): State<App>, Path(id): Path<String>) -> Response { - match app.memory.task_tree(&id).await { + let m = session!(app); + match m.task_tree(&id).await { Ok(t) => Json(json!(t)).into_response(), Err(e) => task_error(&e), } } async fn admit_task(State(app): State<App>, Path(id): Path<String>) -> Response { - match app.memory.admit(&id).await { + let m = session!(app); + match m.admit(&id).await { Ok(t) => Json(json!({ "task": t })).into_response(), Err(e) => task_error(&e), } @@ -974,7 +1005,8 @@ /// The person claims a task, for half an hour at a time; the board's own heartbeat is /// the person coming back to complete it. async fn claim_task(State(app): State<App>, Path(id): Path<String>) -> Response { - match app.memory.claim(&id, time::Duration::minutes(30)).await { + let m = session!(app); + match m.claim(&id, time::Duration::minutes(30)).await { Ok(c) => Json(json!({ "task": c.task })).into_response(), Err(e) => task_error(&e), } @@ -993,10 +1025,11 @@ Path(id): Path<String>, Json(h): Json<Handoff>, ) -> Response { + let m = session!(app); if h.handoff.trim().is_empty() { return (StatusCode::BAD_REQUEST, "a handoff needs a body").into_response(); } - let task = match app.memory.get_task(&id).await { + let task = match m.get_task(&id).await { Ok(t) => t, Err(e) => return task_error(&e), }; @@ -1012,8 +1045,7 @@ ) .into_response(); } - let written = app - .memory + let written = m .write(dm_server::WriteRequest { domain: Some(task.spec.domain.clone()), entry_type: dm_core::EntryType::Handoff, @@ -1026,7 +1058,7 @@ Ok(w) => w, Err(e) => return task_error(&e), }; - match app.memory.complete(&id, &written.id).await { + match m.complete(&id, &written.id).await { Ok(t) => Json(json!({ "task": t, "handoff": written.id })).into_response(), Err(e) => task_error(&e), } @@ -1043,15 +1075,17 @@ Path(id): Path<String>, Json(r): Json<Reason>, ) -> Response { + let m = session!(app); let reason = r.reason.unwrap_or_else(|| "closed from the board".into()); - match app.memory.invalidate(&id, &reason, None).await { + match m.invalidate(&id, &reason, None).await { Ok(t) => Json(json!({ "task": t })).into_response(), Err(e) => task_error(&e), } } async fn retry_task(State(app): State<App>, Path(id): Path<String>) -> Response { - match app.memory.retry_task(&id).await { + let m = session!(app); + match m.retry_task(&id).await { Ok(t) => Json(json!({ "task": t })).into_response(), Err(e) => task_error(&e), } @@ -1115,13 +1149,11 @@ format!("domain {domain} is not granted"), )); } - if label > app.memory.grants().handling { + let handling = app.memory.grants().handling; + if label > handling { return Err(( StatusCode::FORBIDDEN, - format!( - "{label} is above the handling {}", - app.memory.grants().handling - ), + format!("{label} is above the handling {handling}"), )); } Ok(label) @@ -1156,7 +1188,8 @@ let (fresh, old) = (sibling(".new"), sibling(".old")); let _ = tokio::fs::remove_dir_all(&fresh).await; let _ = tokio::fs::remove_dir_all(&old).await; - app.memory + app.session() + .map_err(export_refusal)? .export(Some(domain), label, &fresh) .await .map_err(export_refusal)?; @@ -1934,6 +1967,233 @@ } #[tokio::test] + async fn the_board_leaves_out_a_task_above_the_handling() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path().join("memory"); + let owner = Memory::init(&root, "human:test:me", "dom-me").unwrap(); + let mut hidden = String::new(); + for (title, classification) in [ + ("Rotate the signing key", dm_core::Label::Restricted), + ("Tidy the docs", dm_core::Label::Public), + ] { + let t = owner + .file_own(dm_ledger::TaskSpec { + domain: "dom-me".into(), + title: title.into(), + body: "Do it.".into(), + requirements: vec!["human".into()], + depends_on: vec![], + origin: None, + upstream_event: None, + created_by: String::new(), + classification, + priority: 0, + }) + .await + .unwrap() + .task; + if t.status == dm_ledger::Status::Proposed { + owner.admit(&t.id).await.unwrap(); + } + if classification == dm_core::Label::Restricted { + hidden = t.id; + } + } + drop(owner); + // the board's handling drops to internal, under the restricted task + let grants = root.join("grants.toml"); + let lowered = std::fs::read_to_string(&grants) + .unwrap() + .replace("handling = \"restricted\"", "handling = \"internal\""); + assert!(lowered.contains("handling = \"internal\"")); + std::fs::write(&grants, lowered).unwrap(); + let app = router(config(dir.path(), &["true"])).unwrap(); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let addr = listener.local_addr().unwrap(); + let _h = tokio::spawn(async move { axum::serve(listener, app).await.unwrap() }); + + let (_, all) = get_json(addr, "/api/tasks").await; + let all = all.to_string(); + assert!(all.contains("Tidy the docs"), "{all}"); + assert!(!all.contains("signing key"), "{all}"); + for path in ["tree", "claim"] { + let method = if path == "tree" { "GET" } else { "POST" }; + let (status, body) = + http(addr, method, &format!("/api/tasks/{hidden}/{path}"), None).await; + let (missing, _) = http( + addr, + method, + &format!("/api/tasks/task-me-none/{path}"), + None, + ) + .await; + assert_eq!(status, missing, "{path}: {body}"); + assert!(!body.contains("signing key"), "{path}: {body}"); + } + } + + #[tokio::test] + async fn a_listing_on_the_board_does_not_classify_what_it_files_after() { + let dir = tempfile::tempdir().unwrap(); + let (addr, _h) = start(dir.path(), &["true"]).await; + let owner = Memory::open(&dir.path().join("memory")).unwrap(); + let secret = owner + .file_own(dm_ledger::TaskSpec { + domain: "dom-me".into(), + title: "Rotate the signing key".into(), + body: "Do it.".into(), + requirements: vec![], + depends_on: vec![], + origin: None, + upstream_event: None, + created_by: String::new(), + classification: dm_core::Label::Restricted, + priority: 0, + }) + .await + .unwrap() + .task; + drop(owner); + let file = |title: &'static str| async move { + let body = format!(r#"{{"title":"{title}"}}"#); + let (status, body) = http(addr, "POST", "/api/tasks", Some(&body)).await; + assert_eq!(status, 200, "{body}"); + serde_json::from_str::<Value>(&body).unwrap()["task"]["classification"].clone() + }; + assert_eq!(file("Before the listing").await, "public"); + let (_, all) = get_json(addr, "/api/tasks").await; + assert!(all.to_string().contains(&secret.id), "{all}"); + assert_eq!( + file("After the listing").await, + "public", + "a listing in one request classified a filing in a later one" + ); + } + + #[tokio::test] + async fn a_read_on_the_board_is_joined_to_what_a_later_board_request_writes() { + let dir = tempfile::tempdir().unwrap(); + let (addr, _h) = start(dir.path(), &["true"]).await; + let owner = Memory::open(&dir.path().join("memory")).unwrap(); + let poison = owner + .write(dm_server::WriteRequest { + body: "Something.\n".into(), + ..dm_server::WriteRequest::default() + }) + .await + .unwrap() + .id; + let (status, body) = http(addr, "GET", &format!("/api/entry/{poison}"), None).await; + assert_eq!(status, 200, "{body}"); + let (status, body) = + http(addr, "POST", "/api/tasks", Some(r#"{"title":"Sign off"}"#)).await; + assert_eq!(status, 200, "{body}"); + let id = serde_json::from_str::<Value>(&body).unwrap()["task"]["id"] + .as_str() + .unwrap() + .to_string(); + for step in ["admit", "claim"] { + let (status, body) = http(addr, "POST", &format!("/api/tasks/{id}/{step}"), None).await; + assert_eq!(status, 200, "{step}: {body}"); + } + let (status, body) = http( + addr, + "POST", + &format!("/api/tasks/{id}/complete"), + Some(r#"{"handoff":"Done, after reading."}"#), + ) + .await; + assert_eq!(status, 200, "{body}"); + let handoff = serde_json::from_str::<Value>(&body).unwrap()["handoff"] + .as_str() + .unwrap() + .to_string(); + let t = owner.taint(&poison); + assert_eq!( + t.readers.len(), + 1, + "the board reads as one session: {:?}", + t.readers + ); + assert_eq!( + t.written, + vec![handoff], + "the handoff the board wrote after reading the entry follows from the read" + ); + } + + #[tokio::test] + async fn the_board_answers_while_a_rebuild_holds_the_index() { + let dir = tempfile::tempdir().unwrap(); + let (addr, _h) = start(dir.path(), &["true"]).await; + let root = dir.path().join("memory"); + let owner = Memory::open(&root).unwrap(); + let id = owner + .write(dm_server::WriteRequest { + body: "Kept.\n".into(), + ..dm_server::WriteRequest::default() + }) + .await + .unwrap() + .id; + let entry = owner.get(&id).await.unwrap(); + drop(owner); + // a reindex in another process holds the index for writing until it has been + // given every entry; the board polls meanwhile + let (give, given) = std::sync::mpsc::channel::<()>(); + let (held, holding) = std::sync::mpsc::channel::<()>(); + let path = root.join(dm_server::INDEX_FILE); + let rebuild = std::thread::spawn(move || { + let mut ix = dm_index::Index::open(&path).unwrap(); + let mut pending = Some(entry); + ix.rebuild(std::iter::from_fn(move || { + let _ = held.send(()); + given.recv().ok()?; + pending.take() + })) + .unwrap() + }); + holding.recv().unwrap(); + let started = std::time::Instant::now(); + for path in ["/api/tasks", "/api/entries"] { + let (status, body) = http(addr, "GET", path, None).await; + assert_eq!(status, 200, "{path} while the index is held: {body}"); + } + assert!( + started.elapsed() < std::time::Duration::from_secs(4), + "the board waited on the index: {:?}", + started.elapsed() + ); + give.send(()).unwrap(); + drop(give); + assert_eq!(rebuild.join().unwrap(), 1); + } + + #[tokio::test] + async fn the_board_s_read_scope_is_decided_once_not_once_per_request() { + let dir = tempfile::tempdir().unwrap(); + let (addr, _h) = start(dir.path(), &["true"]).await; + let log = dir.path().join("memory/state/decisions.jsonl"); + let lines = || { + std::fs::read_to_string(&log) + .map(|t| t.lines().count()) + .unwrap_or(0) + }; + let (status, _) = http(addr, "GET", "/api/tasks", None).await; + assert_eq!(status, 200); + let after_one = lines(); + for _ in 0..5 { + let (status, _) = get_json(addr, "/api/tasks").await; + assert_eq!(status, 200); + } + assert_eq!( + lines(), + after_one, + "a board poll logged a decision; the read scope is decided once for the board" + ); + } + + #[tokio::test] async fn the_board_files_admits_claims_and_completes_a_person_s_task() { let dir = tempfile::tempdir().unwrap(); let (addr, _h) = start(dir.path(), &["true"]).await;
diff --git a/images/harness/Dockerfile b/images/harness/Dockerfile index 404db2d..310035d 100644 --- a/images/harness/Dockerfile +++ b/images/harness/Dockerfile
@@ -10,7 +10,7 @@ # The npm harnesses' versions are in package.json, and every package under them in # package-lock.json; the rest are here. ARG NODE_VERSION=24.22.2 -ARG PI=1.4.0 +ARG PI=1.5.0 ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update \
diff --git a/images/harness/package-lock.json b/images/harness/package-lock.json index a408fc4..6e2986b 100644 --- a/images/harness/package-lock.json +++ b/images/harness/package-lock.json
@@ -6,19 +6,19 @@ "": { "name": "durable-memory-harnesses", "dependencies": { - "@anthropic-ai/claude-code": "2.1.316", + "@anthropic-ai/claude-code": "2.1.317", "@github/copilot": "1.0.105", "@google/gemini-cli": "0.66.0", "@openai/codex": "0.166.0", - "@qwen-code/qwen-code": "0.26.5", + "@qwen-code/qwen-code": "0.26.6", "@xai-official/grok": "1.0.70", "opencode-ai": "1.19.5" } }, "node_modules/@anthropic-ai/claude-code": { - "version": "2.1.316", - "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code/-/claude-code-2.1.316.tgz", - "integrity": "sha512-i3BpzyDkcCPnsa0CSkWgxGPs7WMsyqDkxkqT0SDxyEC5XMIxEmcUmluWRHG9VSeBY8Vyv+l9X34JvJ/Bj54nBQ==", + "version": "2.1.317", + "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code/-/claude-code-2.1.317.tgz", + "integrity": "sha512-k467+KT/LcqJXr3Ye/C4lXnfRNmxOa/TZcIiWKoiyc/XbZGzAR0fxiI/IEOkpic+BFhYRP6BiXa7cvTx4fsOSg==", "hasInstallScript": true, "license": "SEE LICENSE IN README.md", "bin": { @@ -28,20 +28,20 @@ "node": ">=22.0.0" }, "optionalDependencies": { - "@anthropic-ai/claude-code-darwin-arm64": "2.1.316", - "@anthropic-ai/claude-code-darwin-x64": "2.1.316", - "@anthropic-ai/claude-code-linux-arm64": "2.1.316", - "@anthropic-ai/claude-code-linux-arm64-musl": "2.1.316", - "@anthropic-ai/claude-code-linux-x64": "2.1.316", - "@anthropic-ai/claude-code-linux-x64-musl": "2.1.316", - "@anthropic-ai/claude-code-win32-arm64": "2.1.316", - "@anthropic-ai/claude-code-win32-x64": "2.1.316" + "@anthropic-ai/claude-code-darwin-arm64": "2.1.317", + "@anthropic-ai/claude-code-darwin-x64": "2.1.317", + "@anthropic-ai/claude-code-linux-arm64": "2.1.317", + "@anthropic-ai/claude-code-linux-arm64-musl": "2.1.317", + "@anthropic-ai/claude-code-linux-x64": "2.1.317", + "@anthropic-ai/claude-code-linux-x64-musl": "2.1.317", + "@anthropic-ai/claude-code-win32-arm64": "2.1.317", + "@anthropic-ai/claude-code-win32-x64": "2.1.317" } }, "node_modules/@anthropic-ai/claude-code-darwin-arm64": { - "version": "2.1.316", - "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-darwin-arm64/-/claude-code-darwin-arm64-2.1.316.tgz", - "integrity": "sha512-wJiLVMzg2atF7C/a+0wt+yQShWIePsMNL86gIoIJanURY2fxrwHIxrYuxoTW8sWqfj7ai+otV/ik2r7m5aPy1g==", + "version": "2.1.317", + "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-darwin-arm64/-/claude-code-darwin-arm64-2.1.317.tgz", + "integrity": "sha512-Xnf9K4aGBuoYaODmEZ0o/lRvuYUCv2ZIDP+AXv15WVITryGgrsSA7CFL7KH+43kC6WhK8i6oysF6EbGdgBW4Aw==", "cpu": [ "arm64" ], @@ -52,9 +52,9 @@ ] }, "node_modules/@anthropic-ai/claude-code-darwin-x64": { - "version": "2.1.316", - "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-darwin-x64/-/claude-code-darwin-x64-2.1.316.tgz", - "integrity": "sha512-UVb9M3ViDyLwPE4D5TQ0G/gFC3kWU6x/m6DbDg5OuAZZGpdwi7P2ajLusJGaNk1FSxTDj0SnzA6VCEiNEOI46w==", + "version": "2.1.317", + "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-darwin-x64/-/claude-code-darwin-x64-2.1.317.tgz", + "integrity": "sha512-CyElNeIVipIXwn7oRY3a5PpeOm0RuwekKLNCZq/gFnrkmSMBnqSWlAKKc8Q1W2UBFlkejwbzI+uPfgD51aiJGw==", "cpu": [ "x64" ], @@ -65,9 +65,9 @@ ] }, "node_modules/@anthropic-ai/claude-code-linux-arm64": { - "version": "2.1.316", - "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-linux-arm64/-/claude-code-linux-arm64-2.1.316.tgz", - "integrity": "sha512-xqlAkhgSLCMdcaepQNIs1+ym5X9++apiEeOc0LHRVOHpfLlpYKAaAXBRccM6QVITN0T9sXLCIby7CO1Lczusgw==", + "version": "2.1.317", + "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-linux-arm64/-/claude-code-linux-arm64-2.1.317.tgz", + "integrity": "sha512-awlwqIvbePCG9zVrlBm5L/3eELJmGJThom+1QgHeBK5N9F1dFpyBQP9M5A8s0PAaWBbLwDIm9SHVmZ6j/TftbA==", "cpu": [ "arm64" ], @@ -81,9 +81,9 @@ ] }, "node_modules/@anthropic-ai/claude-code-linux-arm64-musl": { - "version": "2.1.316", - "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-linux-arm64-musl/-/claude-code-linux-arm64-musl-2.1.316.tgz", - "integrity": "sha512-MEirZ6NFW4GQhDwHZR3EuWEIQ73AiSmr+VkBbohiwZOXW6FaJ7TQJ4ypN3CddDBn8HKPXVgpjYSsUkfbKo1NSQ==", + "version": "2.1.317", + "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-linux-arm64-musl/-/claude-code-linux-arm64-musl-2.1.317.tgz", + "integrity": "sha512-pLoJ/spITubBkyP30LzejrAKskx18by23l/aVoFuIjs7dMFiI7jLLs7dpwO+g2ZUbBpxxxexJuANKUYJiT8GbA==", "cpu": [ "arm64" ], @@ -97,9 +97,9 @@ ] }, "node_modules/@anthropic-ai/claude-code-linux-x64": { - "version": "2.1.316", - "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-linux-x64/-/claude-code-linux-x64-2.1.316.tgz", - "integrity": "sha512-RQDM6N285IjNKcmlPRriayNzpTkEJreAw5/vBU+V0R3qljkQ5z6/pSbzcVUAXvasoOlvub0RIma3NWvSuZA8Cg==", + "version": "2.1.317", + "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-linux-x64/-/claude-code-linux-x64-2.1.317.tgz", + "integrity": "sha512-u19fYJ4llqi+dAGAj6P1BNu/CRSe0rv4O8Mv/5SQPU5oXCYmUssbJgrBI6rWbIFRAE3YcLCzKlwdHbzlSXBlSA==", "cpu": [ "x64" ], @@ -113,9 +113,9 @@ ] }, "node_modules/@anthropic-ai/claude-code-linux-x64-musl": { - "version": "2.1.316", - "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-linux-x64-musl/-/claude-code-linux-x64-musl-2.1.316.tgz", - "integrity": "sha512-fjJ+xM4cSW9m/q6Zh1W9tkq0Ia44GEUh1TGcJtcrPwFCRWeMCAluOf2st51Bcaet7MAjBn9SwvKts96whjhQXA==", + "version": "2.1.317", + "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-linux-x64-musl/-/claude-code-linux-x64-musl-2.1.317.tgz", + "integrity": "sha512-Zwp/FvlSVo/Nqmw8uxKEzv4hZhdEIXOb9+bkkrf/QgH/yjXWTWIjFCxYt5Gay6jqeX9/5fLrXv//XSUatdXReA==", "cpu": [ "x64" ], @@ -129,9 +129,9 @@ ] }, "node_modules/@anthropic-ai/claude-code-win32-arm64": { - "version": "2.1.316", - "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-win32-arm64/-/claude-code-win32-arm64-2.1.316.tgz", - "integrity": "sha512-B927x2sqr56hJZ29O1MfSSj5qPVmzqWfO7rOPiW+Osg0eA9xLFP2TdTNT5/UzyHYoN0Isa+TKwdGSk0gwb8NyA==", + "version": "2.1.317", + "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-win32-arm64/-/claude-code-win32-arm64-2.1.317.tgz", + "integrity": "sha512-fhCTwE3Vt1QHhxBcjhJqTQUJ3INAPWhtitvCVoQY10Mnigfbo+uO7Dxo0MTk5iv6J4t6kM2Qh/QYLuL3aF61Wg==", "cpu": [ "arm64" ], @@ -142,9 +142,9 @@ ] }, "node_modules/@anthropic-ai/claude-code-win32-x64": { - "version": "2.1.316", - "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-win32-x64/-/claude-code-win32-x64-2.1.316.tgz", - "integrity": "sha512-qs/GOpsQmgWECHVTpqUfTg33BaWoaH7CNOZQlOSqXbBOYst6YL1OqdutE12JsXg096NqGaE7vb5eFa6ZynOIKw==", + "version": "2.1.317", + "resolved": "https://registry.npmjs.org/@anthropic-ai/claude-code-win32-x64/-/claude-code-win32-x64-2.1.317.tgz", + "integrity": "sha512-kqVqDRmi5iTQMPpCg9JmYuJm2Cwv8CIh9AVvkLEq4tpb3uOCcUvybmJ5l+6gouDlFa76FbFupvmRmmowFYygfA==", "cpu": [ "x64" ], @@ -1188,9 +1188,9 @@ ] }, "node_modules/@qwen-code/audio-capture": { - "version": "0.26.5", - "resolved": "https://registry.npmjs.org/@qwen-code/audio-capture/-/audio-capture-0.26.5.tgz", - "integrity": "sha512-jH4mwF/UgzAme4xyhz5d0sLjNSLXz8u7kdy9mSw1uKecovkGt/IV3K6jK2meGuhZ5pNP2EpMD0+DxUCP+5NryA==", + "version": "0.26.6", + "resolved": "https://registry.npmjs.org/@qwen-code/audio-capture/-/audio-capture-0.26.6.tgz", + "integrity": "sha512-1tNX/Bu8qkrAtD2xX6+rogNgr/vzhZPavBk6RYnvAV3z8hSQzjqU+LlvXWpDiZYlSVrnHUdsdaeL5IYE6N0k+w==", "hasInstallScript": true, "optional": true, "dependencies": { @@ -1201,9 +1201,9 @@ } }, "node_modules/@qwen-code/qwen-code": { - "version": "0.26.5", - "resolved": "https://registry.npmjs.org/@qwen-code/qwen-code/-/qwen-code-0.26.5.tgz", - "integrity": "sha512-iqcBqLQDmKvOtRcVvF78l7bf0NpdymGFBo0qNY0ausPezht+PLXkh15vnIILaBKtM4BabtHSv60xWftMomJ7hw==", + "version": "0.26.6", + "resolved": "https://registry.npmjs.org/@qwen-code/qwen-code/-/qwen-code-0.26.6.tgz", + "integrity": "sha512-P+BAPNTTkgUAD7raYbYe14oFuwsAT3x/u+QgMLc0waNa94t6qLLh107Z9AC+ooyhQ5P+O3kjFYw9byqphEZy9Q==", "bin": { "qwen": "cli-entry.js" }, @@ -1222,7 +1222,7 @@ "@oven/bun-darwin-x64": "1.4.4", "@oven/bun-linux-aarch64": "1.4.4", "@oven/bun-linux-x64": "1.4.4", - "@qwen-code/audio-capture": "0.26.5", + "@qwen-code/audio-capture": "0.26.6", "@teddyzhu/clipboard": "0.0.5", "@teddyzhu/clipboard-darwin-arm64": "0.0.5", "@teddyzhu/clipboard-darwin-x64": "0.0.5",
diff --git a/images/harness/package.json b/images/harness/package.json index 3455d36..524a385 100644 --- a/images/harness/package.json +++ b/images/harness/package.json
@@ -3,11 +3,11 @@ "private": true, "description": "The npm harnesses in the harness image, at the versions the image carries.", "dependencies": { - "@anthropic-ai/claude-code": "2.1.316", + "@anthropic-ai/claude-code": "2.1.317", "@github/copilot": "1.0.105", "@google/gemini-cli": "0.66.0", "@openai/codex": "0.166.0", - "@qwen-code/qwen-code": "0.26.5", + "@qwen-code/qwen-code": "0.26.6", "@xai-official/grok": "1.0.70", "opencode-ai": "1.19.5" },
diff --git a/local/README.md b/local/README.md index 77f676f..2d2712b 100644 --- a/local/README.md +++ b/local/README.md
@@ -466,14 +466,47 @@ - A write that supersedes an entry lands curated when the session holds a librarian's task or the checker's task. - Anyone except the entry's author can mark the entry curated, with the `promote` memory - tool or with `amem promote <id> --to curated`. + tool or with `amem promote <id> --to curated --why "<what you checked>"`. Verified is the review gate. Nobody sets it directly. It has two steps: 1. One principal proposes it, with `promote` and `to: verified`. This files a task that requires a person. 2. A person who did not propose it approves it, with `amem approve <task>`. An agent never - approves. + approves. The approval is of the version the proposal named: if the entry changed + since, `approve` refuses it, and a proposer, any proposer, proposes the entry as it + stands. The ledger keeps one approval task per entry and version. The task names the + version it proposes, and the promotion record on the ledger names the same version. A + new proposal closes the open proposal of an older version. A second proposer is + refused while an open proposal of the same version stands, and a declined version is + not proposed again until the entry changes. Only `promote` files a task with a + `promote:` origin. + +Each promotion must say why. A promotion without a why is refused. The ledger keeps the +why beside the entry, not in the entry. `amem promotions <id>` and the `promotions` memory +tool show who promoted which version of the entry to which tier, and why. They also show +if that version reached the record (`landed`). A proposal to verified is in the list from +when it is made. It lands when a person approves it. Its approval task names the entry, +the proposer and the author, and points to `amem promotions <id>`, but does not hold the +why. An auditor reads the why after judging the change, not before. A session reads the +why only if it can read the entry. The why stays readable after the entry is replaced. When the owner erases the entry, the ledger removes each +why. It keeps who promoted which version, to which tier, and when. The erasure also +rewrites each approval task of the entry so that it names the entry by id alone, and it +closes the open ones; a task from a build that still put the why in the task body loses +it the same way. The erasure receipt gives the number of whys it removed +(`promotion_whys_removed`) and the number of tasks it rewrote +(`promotion_tasks_cleared`). An export carries the records of the +entries it exports, at or below its label bound, in `promotions.json`. The manifest holds +the digest of that file, so an import refuses a changed record. An import of the same +owner's bundle keeps the tiers and the records. An import of another owner's bundle +lands the entries raw and leaves the records out. + +A ledger made by a build of the `feat/promotion-why` branch before this change needs no +step. Commits 41e0bc5 and 088f890 made a `promotions` table with fewer columns, and the +first with a key without the promoter. A SQLite ledger gets each missing column and the +key the next time it opens. A PostgreSQL ledger gets them the next time a process +connects, in each domain schema that lacks them. Commit 9944964 kept each why in a +completed `promoted:` task. Those tasks stay as they are. When an entry holds, the librarian's verify task now ends in such a proposal. The rules are Rego, in the built-in policy. Each step is a logged decision. A promotion is a new
diff --git a/scripts/conformance-fake-server.py b/scripts/conformance-fake-server.py index 77acbb3..815f3d1 100644 --- a/scripts/conformance-fake-server.py +++ b/scripts/conformance-fake-server.py
@@ -23,6 +23,7 @@ no-holder a claim records no holder import-trusts import does not check digests highest-wrong the manifest's highest_label is below its lines + records-uncounted the manifest's highest_label leaves its promotion records out reindex-forgets reindex drops the ledger silent nothing is ever answered @@ -392,6 +393,20 @@ lines.append((e["id"], path, digest(text), e["classification"])) if LABELS.index(e["classification"]) > LABELS.index(highest): highest = e["classification"] + # the record of why the first entry was promoted travels with it; its why was + # given at internal, so the record is internal and the bound admits it, and the + # highest label counts it as the manifest does its entries + records = [] + if lines: + records.append({ + "domain": DOMAIN, "entry": lines[0][0], "tier": "curated", "by": "agent:fake:librarian", + "author": "human:fake:owner", "why": "checked against the tree", "classification": "internal", + "digest": lines[0][2], "new_digest": lines[0][2], "why_erased": False, + "at": "2026-10-08T00:00:00Z", + }) + for r in records: + if LABELS.index(r["classification"]) > LABELS.index(highest) and FLAW != "records-uncounted": + highest = r["classification"] if FLAW == "highest-wrong": highest = "public" if highest != "public" else "internal" out = [ @@ -399,18 +414,27 @@ f"domain: {DOMAIN}", "exported_at: 2026-10-08T00:00:00Z", "exported_by: human:fake:owner", - "max_label: public", + "max_label: internal", f"highest_label: {highest}", "entries:", ] for eid, path, dig, label in lines: out += [f"- id: {eid}", f" path: {path}", f" digest: {dig}", f" classification: {label}"] + if records: + text = json.dumps(records, indent=2) + with open(os.path.join(dest, "promotions.json"), "w", encoding="utf-8") as f: + f.write(text) + out += ["promotions:", " path: promotions.json", f" digest: {digest(text)}", " records:"] + for r in records: + out += [f" - entry: {r['entry']}", f" classification: {r['classification']}"] with open(os.path.join(dest, "manifest.yaml"), "w", encoding="utf-8") as f: f.write("\n".join(out) + "\n") def import_(self, src): with open(os.path.join(src, "manifest.yaml"), encoding="utf-8") as f: - lines = [l for l in f.read().splitlines() if l.startswith("- id:") or l.startswith(" ")] + # the entries block alone: the promotions block after it is not read here + entries = f.read().split("\npromotions:")[0] + lines = [l for l in entries.splitlines() if l.startswith("- id:") or l.startswith(" ")] items = [] for l in lines: if l.startswith("- id:"):
diff --git a/scripts/conformance.py b/scripts/conformance.py index c871a27..3d77186 100644 --- a/scripts/conformance.py +++ b/scripts/conformance.py
@@ -325,30 +325,46 @@ def parse_manifest(text): - """The subset of YAML a manifest is written in: scalars at the top, and under - `entries`, a list of maps. The format is the contract, so the shape is known.""" + """The subset of YAML a manifest is written in: scalars at the top; under `entries`, a + list of maps; and under `promotions`, when the bundle carries promotion records, a + map of `path`, `digest` and `records`, itself a list of maps. The format is the + contract, so the shape is known. Returns the top, the entries, and the promotions + block or None.""" top = {} entries = [] - in_entries = False + promotions = None + section = None for raw in text.splitlines(): if not raw.strip() or raw.lstrip().startswith("#"): continue if not raw.startswith(" ") and not raw.startswith("-"): key, _, value = raw.partition(":") - in_entries = key.strip() == "entries" - if not in_entries: + section = key.strip() if key.strip() in ("entries", "promotions") else None + if section == "promotions": + promotions = {"records": []} + elif section is None: top[key.strip()] = value.strip().strip("'\"") continue - if not in_entries: + if section is None: continue line = raw.lstrip() + if section == "entries": + items = entries + elif raw.startswith(" ") and not line.startswith("- ") and not raw.startswith(" "): + # a scalar of the promotions block itself, or the head of its records + key, _, value = line.partition(":") + if key.strip() != "records": + promotions[key.strip()] = value.strip().strip("'\"") + continue + else: + items = promotions["records"] if line.startswith("- "): - entries.append({}) + items.append({}) line = line[2:] key, _, value = line.partition(":") - check(entries, f"manifest: a field before the first entry: {raw!r}") - entries[-1][key.strip()] = value.strip().strip("'\"") - return top, entries + check(items, f"manifest: a field before the first item: {raw!r}") + items[-1][key.strip()] = value.strip().strip("'\"") + return top, entries, promotions def plain_name(path): @@ -701,6 +717,51 @@ finally: s.close() + def check_promotions(self, bundle, top, promotions, listed): + """The promotion records a bundle carries, as its manifest's `promotions` block + lists them: the file is a plain name with the digest the block gives, holds one + JSON record per line of the block, and each is about an entry the manifest lists, + at the classification its line says, within max_label. Returns the lines, or none + when the bundle carries no records.""" + if promotions is None: + return [] + for key in ("path", "digest"): + self.check(promotions.get(key), f"the manifest's promotions block has no {key}") + path = promotions["path"] + self.check(plain_name(path), f"promotions path {path!r} is not a plain file name inside the bundle") + full = os.path.join(bundle, path) + self.check(os.path.isfile(full), f"the manifest names {path}, which the bundle lacks") + with open(full, "rb") as f: + text = f.read() + self.check( + hashlib.sha256(text).hexdigest() == promotions["digest"], + f"{path}: the digest is not the file's sha256", + ) + try: + inside = json.loads(text.decode("utf-8")) + except ValueError as e: + self.check(False, f"{path} is not JSON: {e}") + self.check(isinstance(inside, list), f"{path} is not a list of records") + records = promotions["records"] + self.check( + len(inside) == len(records), + f"the manifest lists {len(records)} promotion records and {path} holds {len(inside)}", + ) + for line, record in zip(records, inside): + for key in ("entry", "classification"): + self.check(line.get(key), f"a promotion line without {key}: {line}") + self.check(line["entry"] in listed, f"a promotion record is about {line['entry']}, which the manifest does not list") + self.check(line["classification"] in LABELS, f"{line['entry']}: record classification {line['classification']!r}") + self.check( + LABELS.index(line["classification"]) <= LABELS.index(top["max_label"]), + f"a promotion record of {line['entry']} is {line['classification']}, above max_label {top['max_label']}", + ) + self.check( + record.get("entry") == line["entry"] and record.get("classification") == line["classification"], + f"a promotion record of {record.get('entry')} at {record.get('classification')} is listed as one of {line['entry']} at {line['classification']}", + ) + return records + # 6 def step_export_import(self): if not self.opts.operator: @@ -715,7 +776,7 @@ mpath = os.path.join(bundle, "manifest.yaml") self.check(os.path.isfile(mpath), "export wrote no manifest.yaml") with open(mpath, encoding="utf-8") as f: - top, lines = parse_manifest(f.read()) + top, lines, promotions = parse_manifest(f.read()) self.check(top.get("format") == BUNDLE_FORMAT, f"manifest format {top.get('format')!r}") self.check(top.get("domain") == d, f"manifest domain {top.get('domain')!r}, not {d}") for key in ("exported_at", "exported_by", "max_label", "highest_label"): @@ -751,12 +812,18 @@ ) if LABELS.index(m["classification"]) > LABELS.index(highest): highest = m["classification"] + # the promotion records a bundle carries are lines of the manifest too: about + # an entry it lists, within the bound, and counted in the highest label + records = self.check_promotions(bundle, top, promotions, seen) + for r in records: + if LABELS.index(r["classification"]) > LABELS.index(highest): + highest = r["classification"] self.check( top["highest_label"] == highest, f"manifest highest_label {top['highest_label']}, the lines say {highest}", ) self.check(h in seen and dec in seen, "the bundle lacks the entries this run wrote") - print(f" bundle of {len(lines)} entries, highest {highest}") + print(f" bundle of {len(lines)} entries and {len(records)} promotion records, highest {highest}") s = self.session() try:
diff --git a/scripts/conformance.test.sh b/scripts/conformance.test.sh index 79a3951..3513d02 100644 --- a/scripts/conformance.test.sh +++ b/scripts/conformance.test.sh
@@ -54,6 +54,7 @@ run no-holder no-holder run import-trusts import-trusts --operator "python3 $fake operator" run highest-wrong highest-wrong --operator "python3 $fake operator" +run records-uncounted records-uncounted --operator "python3 $fake operator" run reindex-forgets reindex-forgets --operator "python3 $fake operator" --drop-index "python3 $fake operator drop-index" run silent silent --timeout 5 wait @@ -98,6 +99,7 @@ expect no-holder 1 '4 of 7 steps passed; not run: 6, 7; FAILED: 5' 'the claimed task names no holder' expect import-trusts 1 '6 of 7 steps passed; FAILED: 6' 'import accepted an entry that does not match its digest' expect highest-wrong 1 '6 of 7 steps passed; FAILED: 6' 'the lines say' +expect records-uncounted 1 '6 of 7 steps passed; FAILED: 6' 'the lines say internal' expect reindex-forgets 1 '6 of 7 steps passed; FAILED: 7' 'after reindex the ledger lost the completed task' # a server that never answers is reported, not waited on expect silent 1 '0 of 7 steps passed; not run: 6, 7; could not run: 2, 3, 4, 5; FAILED: 1' 'no reply within'
diff --git a/scripts/e2e.sh b/scripts/e2e.sh index 074fdc1..1c9b82f 100755 --- a/scripts/e2e.sh +++ b/scripts/e2e.sh
@@ -8,6 +8,11 @@ set -eu here=$(CDPATH='' cd -- "$(dirname -- "$0")/.." && pwd) image=mcr.microsoft.com/playwright:v1.61.1-noble +# target/hosted is this script's own output, in the build directory. The CI cache keeps +# target/ but deletes every file in it before it saves, the marker too, and so leaves +# target/hosted as empty directories that hosted.sh refuses to replace. Each run starts +# without it. +rm -rf "$here/target/hosted" sh "$here/scripts/hosted.sh" "$here/target/hosted" if command -v podman >/dev/null 2>&1; then runtime=podman