release: final apache: true title: 3.1.0 date: 2021-05-18 summary: > Code generation backend, Schematron validation, variable improvements, improved SAX conformance and performance, miscellaneous bug fixes and improvements

source-dist: - “apache-daffodil-3.1.0-src.zip”

binary-dist: - “apache-daffodil-3.1.0-bin.tgz” - “apache-daffodil-3.1.0-bin.zip” - “apache-daffodil-3.1.0-bin.msi” - “apache-daffodil-3.1.0-1.noarch.rpm”

scala-version: 2.12

Code Generation Backend

A new code generation backend, called daffodil-runtime2, is added supporting generating C code with limited parse and unparse capabilities. See the Command Line Interface page for commands and options to generate code. Visit DaffodilC ToDos for a list of limitations and planned future work.

  • {% jira 2202 %} Code Gen Framework
  • {% jira 2500 %} Remove dependence on argp and gcc
  • {% jira 2505 %} README says Mini-XML 3.2 is required, but everything seems to work with 3.1
  • {% jira 2507 %} Unable to build/stage daffodil-cli with gcc version 4.8.2 on CentOS 7
  • {% jira 2508 %} Missing LICENSE and NOTICE files in daffodil-runtime2

Pluggable Validation

A new capability allowing pluggable infoset validation mechanisms is added. A Schematron validator is now available, which allows for validating the infoset against Schematron rules, either external or embedded in the DFDL schema. The CLI --validate option now allows for specifying a pluggable validator, in addition to the off/limited/full options. A new DataProcessor.withValidator() function is added to specify a new validator via the Java/Scala API.

  • {% jira 1685 %} Full validation should create and initialize the validator before parsing/unparsing begins
  • {% jira 1807 %} schematron validation
  • {% jira 2409 %} Custom Infoset validators with SPI
  • {% jira 2446 %} Embedded Schematron
  • {% jira 2482 %} Output SVRL from Schematron Validator

DFDL Variables

Multiple bugs have been fixed to improve conformance with the DFDL specification related to DFDL newVariableInstance and setVariable properties, primarily related to defaults and external values. A new dfdlx:direction extension property is also added to specify that certain variables should only be evaluated during parse, unparse, or both.

  • {% jira 1520 %} Unset variables referenced from dfdl:outputValueCalc expression should block/delay until they get a value.
  • {% jira 2352 %} dfdl:newVariableInstance with non-constant defaultValue fails
  • {% jira 2385 %} newVariableInstance has issues with suspensions during unparse
  • {% jira 2429 %} newVariableInstance and setVariable require ‘direction’ property
  • {% jira 2481 %} newVariableInstance does not use external values when defaultValue is not defined

SAX Updates

The SAX API has been updated, including improved conformance with SAX API namespace/prefix properties and improved unparse performance.

  • {% jira 2422 %} Implement allowed features for DaffodilXMLReader/DaffodilOutputContentHandler
  • {% jira 2427 %} Batch SAX Unparse Coroutine Events
  • {% jira 2457 %} xmlns="" incorrectly introduced due to prefixOrNull

Bug Fixes

  • {% jira 774 %} Improve detection of disallowed entities in property values
  • {% jira 900 %} dfdl:ref format reference not found - fails silently
  • {% jira 1007 %} Leftover Data Check not Working as Expected with No Trailing Newline
  • {% jira 1299 %} TDML Runner not checking uniqueness constraints on test case names
  • {% jira 1420 %} Error when unparsing an apostrophe (scala XMLEventReader bug)
  • {% jira 1580 %} SimpleTypes - Roundtrip Fail - Calendar Bugs
  • {% jira 1865 %} Should identify an invalid index (-1)
  • {% jira 1928 %} TDMLRunner removes apostrophes from expected when ' is used in infoset
  • {% jira 1945 %} Failures in Calendar tests
  • {% jira 2332 %} Javadoc fails to build with openjdk 13.0.2
  • {% jira 2348 %} Debugger breaks with prefixed lengths
  • {% jira 2399 %} Error diagnostics output even though there is an infoset
  • {% jira 2431 %} CSV data causes abort
  • {% jira 2432 %} Need validation of tunable values
  • {% jira 2455 %} Large CSV file causes “Attempting to backtrack too far” exception
  • {% jira 2461 %} Intermittent test failure in schematron CLI tests
  • {% jira 2468 %} Unparsing an infoset for an 800mb csv file runs out of memory
  • {% jira 2487 %} parser hangs on row with initial empty field in CSV-like data
  • {% jira 2492 %} Test fails on non-US locale
  • {% jira 2494 %} Fix user unfriendly issues with recent jline upgrade
  • {% jira 2498 %} anyEmpty with minOccurs=“0” and initial empty element - fails
  • {% jira 2517 %} IBM DFDL Crosstester always reports left over data

Miscellaneous Changes

  • {% jira 817 %} CLI: Remove --validation from save-parser options
  • {% jira 823 %} Need clear error message when using string with maxOccurs=“unbounded” occursCountKind=“implicit” w/o maxLength specified
  • {% jira 1271 %} OutputValueCalc - evaluation is allowed as early as possible
  • {% jira 1494 %} Remove Dynamic and CachedDynamic mechanism - redundant now
  • {% jira 1503 %} Warning needed for length=“0” being ignored.
  • {% jira 1605 %} Update Wiki based on CII best practices review
  • {% jira 2144 %} Standard schema project layout - suggested build.sbt is outdated
  • {% jira 2360 %} Committed files that match .gitignore patterns
  • {% jira 2430 %} ratCheck fails when run from the sbt console
  • {% jira 2434 %} Update sbt-rat plugin to 0.7.0
  • {% jira 2435 %} Prepare for 3.1.0
  • {% jira 2436 %} Ensure Daffodil works with newer versions of Java
  • {% jira 2440 %} TDML Runner must provide root namespace to processor
  • {% jira 2442 %} daffodil-site link to DFDL specification is to very obsolete old one
  • {% jira 2453 %} Debugger: support for displaying variable information
  • {% jira 2454 %} Eclipse - Not able to Compile or Daffodil/stage daffodil 3.0.0
  • {% jira 2463 %} Update Daffodil Incubator Status file/page
  • {% jira 2464 %} Document Apache Daffodil Maturity Model Assessment
  • {% jira 2465 %} Remove requirement of DAFFODIL_HOME for eclipse setup
  • {% jira 2467 %} Bump copyright years to 2021
  • {% jira 2469 %} Need to update unsupported features page
  • {% jira 2470 %} Remove instances of incubating
  • {% jira 2472 %} Add .asf.yml file to configure daffodil and daffodil-site github
  • {% jira 2474 %} how to deal with Control chars and newlines in pattern.
  • {% jira 2477 %} --validate option requires parameter
  • {% jira 2484 %} Upgrade CI to Java 16
  • {% jira 2491 %} Allow custom Debuggers through API
  • {% jira 2495 %} Need way to dry-run release candidates from repos/branches other than apache/daffodil@main
  • {% jira 2503 %} Add back missing unit tests for Coroutines library
  • {% jira 2506 %} Remove bintray.com
  • {% jira 2514 %} Prepare for 3.1.0 Release

Known Issues

A number of issues remain open that have been marked as critical, and are expected to be fixed in the next release. These issues are:

  • {% jira 1422 %} disallow doctype decls in all XML & XSD that we read in
  • {% jira 2400 %} New SAX API causes performance degradations
  • {% jira 2512 %} Unordered sequences with initiated content or discriminators does not parse correctly

Deprecation/Compatibility

The use of the ParseResult.isAtEnd() function has been deprecated, because it can cause unexpected blocking in some cases. Instead, the final bit position should be accessed via the ParseResult.bitPos1b() function, and the result compared with the expected data length. Alternatively, a new InputSourceDataInputStream.hasData() function is added, which blocks until at least 1 byte of data is available to parse, or end-of-file is reached.

  • {% jira 2502 %} Parse must behave properly for reading data from TCP sockets

Dependency Changes

The following dependencies have been added or updated

Core

  • FasterXML Jackson Core 2.12.3 (update)
  • FasterXML Woodstox Core 6.2.6 (update)
  • ICU4J 69.1 (update)
  • Xerces 2.12.1 (update)

Command Line Interface

  • JLine 3.20.0 (update)
  • Jansi 2.3.2 (update)
  • Scallop 4.0.2 (update)

Code Generator

  • OS-Lib 0.7.6 (new)

Schematron Validator

  • Lightbend Config 1.4.1 (new)
  • Saxon-HE 10.5 (new)

Test

  • JUnit 4.13.2 (update)
  • ScalaCheck 1.15.4 (update)