docs(ui): correct what the diff tokenizer actually guarantees Review of #2300 caught a false claim in my own comment. Astryx's tokenizer is stateless between lines, which is what makes a diff's rows addressable — but "stateless" is not "line-bounded", and `tokenizeLine` bounds only where a match may start. `/\*[\s\S]*?\*\/`, `<!-- -->` and Python's triple quotes all run past their own line, so the interior of a multi-line comment colours as code. The renderer's clamp keeps every character exact; only the colour is wrong, and fixing it means carrying construct state across rows — a feature, not a clamp. Also names two consequences of the CSS that the rules had left for the reader to discover: context lines come up to full foreground on purpose (dimming the text between syntax-coloured keywords reads as damage, not de-emphasis), and the marker cell is rendered on markerless rows so `\ No newline at end of file` lines up with the code above them.
diff --git a/packages/ui/src/styles.css b/packages/ui/src/styles.css index 295b2a0..175ed15 100644 --- a/packages/ui/src/styles.css +++ b/packages/ui/src/styles.css
@@ -524,7 +524,9 @@ /* The marker column: one character wide, so an addition's code starts in the same place as the context line above it. It keeps the line's tint colour — the sign is what the colour is naming — while `.maka-tool-diff-code` hands - the code back to syntax colouring. */ + the code back to syntax colouring. Every row gets the cell, including the + markerless ones: a `\ No newline at end of file` that hugged column 0 while + the code beside it did not was the same misalignment on a different row. */ .maka-tool-diff-marker { display: inline-block; width: 1ch; @@ -533,7 +535,13 @@ /* Neutral base under the `astryx-token-*` spans. Without it an added line's uncoloured text would stay green and a deleted line's red, which is the flat single-tint reading the tokens exist to break up; the row's background - tint and its marker carry add/del on their own. */ + tint and its marker carry add/del on their own. + + This is also what brings context lines up from `--foreground-secondary` to + full foreground, which is deliberate rather than collateral: once a context + line's keywords and strings are syntax-coloured, dimming only the text + between them reads as damage, not as de-emphasis. The row's flat background + is what says "unchanged" now. */ .maka-tool-diff-code { color: var(--foreground); } .maka-tool-diff-line[data-line="meta"] .maka-tool-diff-code { color: inherit; } .maka-tool-diff-line[data-line="add"] { background: oklch(from var(--success) l c h / 0.1); color: var(--success-text); }
diff --git a/packages/ui/src/tool-activity/diff-syntax.ts b/packages/ui/src/tool-activity/diff-syntax.ts index 8a6b563..8c742e9 100644 --- a/packages/ui/src/tool-activity/diff-syntax.ts +++ b/packages/ui/src/tool-activity/diff-syntax.ts
@@ -8,12 +8,24 @@ * language, and a diff is neither (its marker column is not code, and its * lines are two files interleaved). * - * What Astryx does ship is the tokenizer underneath that component, and it is - * line-local by construction — `tokenize` runs each line against the language's - * anchored patterns independently, with line-relative offsets. That is exactly - * what a diff needs: the rows can be tokenized as one buffer of stripped code - * and the results still line up row for row, with no pretence that the old and - * new sides are each a compilable file. + * What Astryx does ship is the tokenizer underneath that component, and it + * carries no state between lines: `tokenize` returns one token array per line, + * each starting from that line's own offset with nothing inherited from the + * one above. That is what a diff can use — the rows tokenize as one buffer of + * stripped code and the results still line up row for row, with no pretence + * that the old and new sides are each a compilable file. + * + * Stateless is not the same as line-bounded, and the difference is visible. + * `tokenizeLine` bounds where a match may *start*, not how far it may run, and + * several patterns span newlines — the JS block comment `/\*[\s\S]*?\*\/`, the + * HTML `<!-- -->`, Python's triple-quoted strings. A row that opens one gets a + * token reaching into the rows below (clamped where it is rendered), and those + * rows, having inherited nothing, colour as ordinary code. So the interior of a + * multi-line comment reads as code rather than as comment. That is a colour + * being wrong, never a character: it is strictly better than the flat tint it + * replaced, and fixing it means carrying an "inside a construct" flag across + * rows here — a real feature, not a clamp, and not worth it until someone + * misreads a diff because of it. * * The `astryx-token-*` classes the spans carry are Astryx's own span-mode * fallback classes, injected by `ensureHighlightStyles()`; the product's @@ -69,7 +81,8 @@ * * Joining the rows into one buffer is a way to make a single `tokenize` call, * not a claim that the buffer is valid source — the tokenizer returns one - * entry per line and never consults its neighbours. + * entry per line and inherits no state from the line above (see the note on + * newline-spanning patterns at the top of this file). */ export function diffSyntaxTokens(paths: readonly string[], code: readonly string[]): TokenLine[] { if (code.length === 0) return [];