release: final apache: true title: 3.4.0 date: 2022-11-08 summary: > EXI binary XML support, pluggable character sets, embedded XML, C code generator updates

source-dist: - “apache-daffodil-3.4.0-src.zip”

binary-dist: - “apache-daffodil-3.4.0-bin.tgz” - “apache-daffodil-3.4.0-bin.zip” - “apache-daffodil-3.4.0-bin.msi” - “apache-daffodil-3.4.0-1.noarch.rpm”

scala-version: 2.12

EXI Support

The Daffodil CLI adds two new infoset types---I exi and -I exisa--to support infosets represented as EXI binary XML for non-schema aware and schema aware EXI, respectively. EXI infosets are significantly smaller in size than normal XML infosets, and are often even smaller than the original data format when made schema aware. The Daffodil CLI has added the Exificient library to support these infoset types.

API users can create EXI files by combining the existing SAXInfosetInputter and SAXInfosetOutputter classes with the Exificient SAX API or they can use the Agile Delta EXI SAX API which has been tested with Daffodil as well. Daffodil has added the new DaffodilXMLEntityResolver class to its Java and Scala public APIs to support creating schema aware EXI files too.

  • {% jira 1959 %} EXIficient Inputter and outputter for XML EXI representation
  • {% jira 2739 %} Expose DFDLCatalogResolver to the public API

Pluggable Character Sets

Custom character sets can now be added to Daffodil by implementing a custom BitsCharsetDefinition class and related functions/classes, listing it in a META-INF/services file, packaging it into a jar, and adding it to the Daffodil classpath.

  • {% jira 2663 %} charset encoders need to be pluggable (doc needed)

Embedded XML in the Infoset

When using the XMLTextInfosetInputter and XMLTextInfosetOutputter classes in the API, or -I xml in the CLI, simple string elements with the DFDL extension attribute dfdlx:runtimeProperties="stringAsXml=true" are treated as XML. This means that when parsing, instead of outputting the content as an XML escaped string, the parsed content is checked to be valid XML and output as if it were part of the XML infoset. When unparsing, the embedded XML part of the infoset is converted back to a string. Note that because there are multiple ways to read and write XML that are syntactically different but semantically the same, it is possible that parsed or unparsed data may differ from the original data.

  • {% jira 2708 %} XML String feature in XML Text Infoset Inputter/Outputter

C Code Generator Updates

The C code generator backend now supports reading and writing N-bit booleans and integers, where N is an explicit length from 1 to 64 bits. Additional miscellaneous changes include unit test support, fixes to nested choices, tweaks to how float numbers are output, and a command line option to choose the TDML implementation to run TDML tests.

  • {% jira 2696 %} Extend runtime2 to N-bit booleans and integers (1 <= N <= 64)
  • {% jira 2697 %} Choose TDML implementation to run TDML tests

Additional New Features/Improvements

  • {% jira 2357 %} Update recoverable error to not be an SDW
  • {% jira 2369 %} TDML language needs comment syntax in documentPart byte and bits
  • {% jira 2685 %} Upgrade Scala XML library to 2.1.0 version
  • {% jira 2702 %} Eliminate build warning on VSCode scala build of Daffodil
  • {% jira 2719 %} Add tests to illustrate some nillable and fixed-length scenarios
  • {% jira 2723 %} Change DataProcessor.copy() method to public
  • {% jira 2725 %} TDML runner needs to support pre-compiled DFDL schema

Miscellaneous Bug Fixes

  • {% jira 2237 %} XML comment not allowed inside dfdl:defineEscapeScheme
  • {% jira 2488 %} When executing a parse command with the debugger get incorrect result for display info unparser
  • {% jira 2616 %} End-user doc of how import/include schemaLocation resolver works
  • {% jira 2654 %} java.util.NoSuchElementException: head of empty list on commented out includes/choice branches
  • {% jira 2680 %} Add new transitive deps' NOTICE to Daffodil's bin.NOTICE
  • {% jira 2682 %} Update copyrights to 2022
  • {% jira 2693 %} Add Adam Rosien to https://daffodil.apache.org/people/
  • {% jira 2701 %} Facet checks throw exception for non-numeric float/double values
  • {% jira 2704 %} sbt tests are intermittenly hanging
  • {% jira 2710 %} build.md instructions fail for RedHat - no mxml-devel
  • {% jira 2713 %} Release candidate container has poor cacability and reproducability
  • {% jira 2717 %} unparser deadlocks in mil-std-2045 2-message file case
  • {% jira 2726 %} Index out of bounds -1 during unparsing
  • {% jira 2742 %} bin.LICENSE and bin.NOTICE need to be updated

Deprecation/Compatibility

  • dfdl:assert's with failureType="recoverableError" are now reported as validation errors instead of schema definition warnings {% jira 2357 %}

  • All InfosetInputter and InfosetOutputter functions now return Unit instead of Boolean. Errors are now expected to be thrown as exceptions {% jira 2721 %}

Dependency Changes

The following dependencies have been added or updated:

Core

  • FasterXML Jackson Core 2.13.4 (update)
  • FasterXML Woodstox Core 6.4.0 (update)
  • ICU4J 72.1 (update)
  • Log4j API 2.19.0 (update)
  • Scala Reflect & Standard Library 2.12.17 (update)
  • Scala XML 2.1.0 (update)

Command Line Interface

  • Exificient 1.0.4 (new)
  • Log4j Core 2.19.0 (update)

Schematron Validator

  • Saxon-HE 11.4 (update)

Test

  • Scalacheck 1.17.0 (update)

Changes to Transitive Dependencies

  • Apache Commons Codec 1.15 (update, used by httpclient5 & log4j)
  • Apache HttpClient 5.1.3 (new, used by XML Resolver)
  • Apache HttpComponents Core HTTP/1.1 5.1.3 (upgrade, used by XML Resolver)
  • Apache HttpComponents Core HTTP/2 5.1.3 (new, used by XML Resolver)
  • JAXB API 2.2.11 (new, used by Exificient)
  • JAXB Core Implementation 2.2.11 (new, used by Exificient)
  • JAXB Runtime 2.2.11 (new, used by Exificient)
  • JavaBeans Activation Framework Specification 1.1.1 (new, used by Exificient)
  • SLF4J API 1.7.25 (new, used by httpclient5)
  • XML Resolver & data 4.4.3 (update, used by Saxon-HE)
  • XmlPull 1.1.3.1 (new, used by Exificient)