tree: 3ba8aeb2f08fbdc61281b9dadf3ec8183b20680f [search]
  1. fixtures/
  2. tools13/
  3. .gitignore
  4. calcval-cases.tsv
  5. calcval.py
  6. cmp7.py
  7. compare.py
  8. convert.py
  9. cov_corpus.py
  10. cov_report.py
  11. cov_schema.py
  12. cov_source.py
  13. crosscheck.py
  14. decrypt.py
  15. diff7.py
  16. doctypecheck.py
  17. drive.sh
  18. dump7.py
  19. env.sh
  20. fetch-odfvalidator.sh
  21. fetch-schemas.sh
  22. foreign.py
  23. formula-cases.tsv
  24. formulacheck.py
  25. gate-fixtures.tsv
  26. gate-volatile.tsv
  27. gate.py
  28. idcheck.py
  29. make-controls.py
  30. make-doctype-controls.py
  31. make-jobs.sh
  32. make-pkg-controls.py
  33. openformula-functions.txt
  34. pkgcheck.py
  35. README.md
  36. refresh-app.sh
  37. report.py
  38. roundtrip.py
  39. runpy.sh
  40. selftest.py
  41. setup-app.sh
  42. smoke13-known.tsv
  43. smoke13.py
  44. start.sh
  45. unohelp.py
  46. validate.py
test/odf-validate/README.md

ODF validation harness

This harness has OpenOffice save a corpus of documents as ODF, checks every saved package against the OASIS ODF 1.2, ODF 1.3 and ODF 1.4 RELAX NG schemas with jing, and reports the errors grouped by kind. Comparing those reports against a baseline shows whether an export change fixed errors or introduced new ones.

It runs on macOS only (zsh scripts, hdiutil, .app layout).

Requirements

  • An AOO macOS build (.dmg).
  • The Python that the office's pyuno links against. Check with otool -L <app>/Contents/program/libpyuno.dylib, then set ODFV_PYTHON.
  • jing.jar (set ODFV_JING) and java; ODFV_ODFVALIDATOR and ODFV_JAVA for crosscheck.py.
  • gtimeout (GNU coreutils), used by drive.sh.

Defaults for ODFV_APP, ODFV_PYTHON, ODFV_JING and ODFV_SOLVER are in env.sh.

Setup

./fetch-schemas.sh                 # OASIS schemas, checksum-verified, into schema/
./setup-app.sh <path-to-AOO.dmg>   # copies OpenOffice.app into app/
./fetch-odfvalidator.sh            # optional: odfvalidator for crosscheck.py
python3 -I selftest.py             # validator negative controls

The schemas are not committed because they carry OASIS copyright terms.

Running a corpus

./make-jobs.sh runs/base_strict/out > runs/base_strict/jobs.tsv
./drive.sh strict 2013 runs/base_strict/jobs.tsv runs/base_strict/done.tsv
python3 -I validate.py runs/base_strict/out runs/base_strict/val.json
python3 -I report.py runs/base_strict/val.json baseline/strict.kinds.json 12

The steps:

  • make-jobs.sh collects the test documents under main/ (doc, docx, rtf, xls, ppt, sx*, od*).
  • drive.sh <ext|strict> <port> <jobs> <done> saves them in ODF 1.2 Extended (DefaultVersion=3) or strict ODF 1.2 (DefaultVersion=4).
    • It runs a headless office on the given port, using a profile under runs/.
    • It resumes where it left off, kills a hung office and continues, and stops at the first crash.
  • validate.py checks content, styles, meta, settings and manifest streams against 1.2 as written, against 1.3 (13) and 1.4 (14) with office:version rewritten, and against 1.2 and 1.3 as extended documents (12x, 13x): foreign.py first removes foreign elements and attributes as Part 1 §2.2.2 B.1 and §3.17 describe (the manifest is left as written). Use report.py <val.json> <kinds.json> 12x to measure what is wrong apart from foreign content. Embedded objects whose content.xml is MathML are skipped (ODF 1.2 Part 1 §2.2.1 B.4).
  • report.py groups the errors by normalized message.
  • pkgcheck.py <outdir> <pkg.json> checks the ODF 1.2 Part 3 package rules (§2.2.1, 3.2, 3.3, 3.4.1): zip methods, mimetype first / stored / no extra field / equal to the / media type, one manifest entry per file, META-INF contents, manifest:size and STORED on encrypted entries, and signature files (well-formed, root, OpenDocument-v1.2-os-dsig-schema.rng). Findings are shall: or should:; only shall findings are non-conformances.
  • doctypecheck.py <outdir> <dt.json> checks Part 1 §2.2.1 A.2/E and §2.2.3-2.2.10: the mimetype file (or office:mimetype of a flat file), the root element and the office:body child must agree on one document type. Embedded sub documents are compared with their manifest media type (should: findings). Two deviations from the literal text of §2.2.5: it spells the drawing values application.vnd.oasis.opendocument.graphics and ...graphic-template; the registered ...graphics and ...graphics-template of Part 1 appendix C are accepted. ...text-web and ...formula-template are registered but no §2.2.x allows them, so they are reported.
  • formulacheck.py <outdir> <formulas.json> parses the style:condition, table:condition, table:expression, table:formula and text:formula values (Part 1 §2.2.1 D.3, §2.2.4 D/E) with the OpenFormula grammar (Part 2 chapter 5) and the condition syntaxes of Part 1 §19.468 and §19.596.
    • Syntax only; no semantics, function arity or type checking.
    • A prefix bound to another namespace is out of Part 2's reach: should: in general, shall: for table:formula inside office:spreadsheet (§2.2.4 D).
    • Non-ASCII name characters use Unicode categories instead of the XML 1.0 4th edition tables; an IRI is checked by character class, not the full RFC 3987 grammar; openformula-functions.txt holds the Part 2 function names.
    • The condition syntaxes follow the prose of Part 1, which has no grammar: , between arguments, lowercase and, != as the relational operator.
    • formula-cases.tsv is the table of good and bad values that selftest.py runs.
  • crosscheck.py <outdir> <val.json> <cross.json> runs the Apache ODF Toolkit odfvalidator 1.1.8 (fetch-odfvalidator.sh; MSV instead of jing, the same OASIS schemas) in conformance mode (-c) and extended mode (-e) and compares each stream with the 12 and 12x results of validate.py.
    • Error counts differ between engines; it compares whether a stream has any error and, in -c mode, where its first error is.
    • It exits if odfvalidator stops before finishing every package. Its failures are silent (an invalid schema prints “Validation aborted”; a crash on one package aborts the run): crashes are skipped and listed.
    • odfvalidator's own schema lookup fails on current JDKs, so the schemas are passed explicitly; its jar has no MathML 2 schema, so a schema that accepts anything stands in (as validate.py skips MathML).
    • Known leniencies: it truncates a draw:points over 2048 characters and adapts old OOo namespaces. It is stricter than Part 3 §3.2 in wanting META-INF/documentsignatures.xml listed in the manifest. It checks ID uniqueness, which jing -i does not.
  • fixtures/doctypes/ (run.sh, make.py, derive-chart.py), fixtures/encrypted/, fixtures/signed/, fixtures/base/: documents of the kinds the corpus lacks (templates, master, Writer/Web, Impress as Draw, Math, chart, encrypted, signed, Base), saved in strict and extended mode under runs/task12/gen-*. decrypt.py <in-dir> <out-dir> <password> writes plain copies of encrypted packages for the schema checkers; pkgcheck.py must still run on the real ones. Signing needs a visible (-invisible) office because -headless cancels the dialog: see the header of fixtures/signed/make.py.
  • fixtures/{writer,calc,draw,chart,styles-text,styles-gfx,leak}/make.py, run with fixtures/gen-run.sh <area> <strict|ext> <port> <abs-outdir>: documents built through UNO to exercise the elements, attributes and enumerated values that cov_report.py lists as gaps (leak classifies the names AOO emits that the schema does not define). Use an absolute <outdir>.
  • cov_schema.py, cov_source.py, cov_corpus.py, cov_report.py: schema-coverage report. The schema universe (ODF 1.2 elements, element/attribute pairs, enumerated values), the names the export code in main/ can emit (static C++ scan; a lower bound, table-driven names are missed) and what a corpus exercises are joined by namespace URI and local name into the gaps worth generating documents for. Outputs live under runs/.
  • idcheck.py <outdir> <ids.json> parses every stream namespace-aware and checks xml:id syntax and uniqueness per stream (ODF 1.2 Part 1 §2.2.1 D.1, D.2). jing -i does not check ID uniqueness.

The regression gate

python3 -I gate.py <tag> --prev runs/gate-<earlier tag>      # everything
python3 -I gate.py <tag> --build xmloff,sw --prev ...        # rebuild first
python3 -I gate.py smoke --limit 8 --prev ... --stages corpus,validate,dump,diff

gate.py runs the stages below in runs/gate-<tag>/, records results.json, skips stages already done (--force reruns) and exits 0 only if everything that ran passed.

stagedoespasses if
buildbuild.pl and deliver.pl for the --build modules (environment script: ODFV_BUILD_ENV)exit codes 0
refreshcopies --libs from the solver into app/ (refresh-app.sh)copies identical
corpussaves the job list in ext and strict modeno crash, every job accounted for, OK/FAIL counts equal --prev
validatevalidate.py, report.py and the four checkers per modekinds not worse than baseline/<mode>.kinds.json or than --prev (checked both ways); all four checkers ran; no shall finding from the id, package and document-type checkers; formula findings not above --prev; a baseline or --prev exists
dumpreloads every saved package with dump7.pyno office failure; hangs and crashes are listed
diffdiff7.py on the dumps and cmp7.py on the packages against --prevevery difference is a document in gate-volatile.tsv
fixturesgate-fixtures.tsv: saves in v00 v10 v11 ext strict, runs the reload checks, validatessaved, checked, and unchanged against --prev; the same checker rules as validate (formula findings are compared per mode with --prev)
  • Inputs are frozen: without --jobs the list of --prev is reused (outputs are rewritten), so the same documents are saved when files under main/ change. A later run's --prev may be any earlier gate directory or a directory with the same layout (<mode>/out, <mode>/dump).
  • The dump stage kills the office only when the dump makes no progress for --stall seconds (default 300), then writes a HANG marker for that document alone. A long batch that keeps progressing is never killed.
  • gate-volatile.tsv names the documents whose dumps differ between runs without an export change (volatile Calc formulas, redline timestamps, the 0238 loader order, the intermittent Basic library loss of 0048). A new entry needs a reason.
  • The first fixtures run has nothing to compare with: its mismatch pattern is only recorded. By design, strict, ODF 1.0 and ODF 1.1 mode fail the 14 reload rows that need gradient or transparency fills.

Checking a change

python3 -I compare.py baseline/strict.kinds.json runs/new/strict.kinds.json

compare.py exits 1 if any kind is new or any kind got worse.

To test a rebuilt module without repackaging, copy its libraries from the solver into the app copy:

./refresh-app.sh libxo.dylib

roundtrip.py <port> <file> <checks.json> loads a document and compares UNO expressions with expected values; --capture <out.json> records them instead.

Formula evaluation baseline (calcval)

./start.sh calcval 2792
./runpy.sh calcval.py 2792 calcval-cases.tsv runs/phase4/baseline.json
python3 -I calcval.py --compare runs/phase4/baseline.json runs/phase4/new.json

calcval.py opens a new Calc document, writes the data area described in the header of calcval-cases.tsv, and for each case enters the formula in a fresh scratch cell (cell, cell@N) or range (array:CxR), recalculates, records the result type, getError(), getString(), getValue() and the formula as read back, then clears the scratch area. expected is the value the ODF 1.3 Part 4 text requires (- records the current behaviour only); the status of each case is match, mismatch, record or exception. The run prints every mismatch, which are the current deviations from ODF 1.3. --compare lists each case whose recorded result changed (exit 1 if any). RECEIVED and BESSELK/BESSELY are Analysis add-ins and are entered by programmatic name.

Notes

  • AOO has no -convert-to; conversion goes over UNO (convert.py).
  • jing must run with -i, otherwise it rejects the ODF schemas themselves.
  • A jing fatal error can hide later errors. Such lines are reported as UNPARSED kinds, and compare.py fails on any new UNPARSED kind.
  • jing stops at a fatal error (a stream that is not well-formed); validate.py records it and carries on with the remaining files.
  • setup-app.sh renames an existing app/OpenOffice.app to OpenOffice.app.old.<timestamp> instead of deleting it.