release: final apache: true title: 3.2.0 date: 2021-12-06 summary: > WARNING: This release has been superceded. Use Release 3.2.1 instead.

source-dist: - “apache-daffodil-3.2.0-src.zip”

binary-dist: - “apache-daffodil-3.2.0-bin.tgz” - “apache-daffodil-3.2.0-bin.zip” - “apache-daffodil-3.2.0-bin.msi” - “apache-daffodil-3.2.0-1.noarch.rpm”

scala-version: 2.12

New DFDL Language Extension Features

Daffodil contains a number of extensions to the DFDL language. These extensions generally are identified by use of the DFDL extensions namespace and a namespace prefix for all related properties. These extensions are not yet officially part of the DFDL language and are considered experimental. They may evolve and schemas that use them may require update with future Daffodil releases.

The Daffodil “layering” extension to the DFDL language, has been enhanced to enable computation of checksums, CRCs, and parity fields. These extensions can be loaded from external Jar files.

  • {% jira 2221 %} Ability to compute CRC, checksums, parity fields. - related to layering
  • {% jira 1927 %} Ability to add new layering transformers in Jars external to Daffodil

New functions have been added to the Daffodil expression language. These include bit-wise AND/OR/NOT functions, and a conversion function that converts type xs:long to xs:double.

  • {% jira 2473 %} Need bit-wise AND, OR, NOT, and shift operations
  • {% jira 2353 %} long to double function - for preserving binary data floating point precision

An extension property, dfdlx:runtimeProperties can be used to annotate DFDL schemas where such annotations are available to the Daffodil runtime back end, such as for creating customized InfosetInputter and InfosetOutputter classes.

  • {% jira 2537 %} dfdlx:runtimeProperties property

Other New Features

The standard Log4J logging system is now used.

  • {% jira 2510 %} Thread Safety: Improve Logger/Logging

The Daffodil CLI now produces different exit codes for failures based on the type of failure.

  • {% jira 2335 %} Different exit codes based on failure type

Java 17 is now a supported JVM. Note that Use of Java 17 requires use of Scala libraries version 2.12.15.

  • {% jira 2579 %} java compile compatibility issue with Java 16/17
  • {% jira 2558 %} Update GitHub Actions Workflow to use Java 17

Code Generation Backend

The C code generation backend called daffodil-runtime2 has had these enhancements.

  • {% jira 2566 %} Extend C code generator to support hexBinary elements
  • {% jira 2578 %} Raise minimum C level to ISO C11 with GNU extensions
  • {% jira 2583 %} Rename runtime2 test files and make runtime2 tests round-trip

Bug Fixes

  • {% jira 2551 %} Get a StackOverflowError when using daffodil save-parser to compile/save ATO schema to parser.bin file
  • {% jira 2580 %} EDIFACT test hits Assert.impossible abort
  • {% jira 2512 %} Unordered sequences with initiated content or discriminators do not parse correctly
  • {% jira 2522 %} ULong library has bugs
  • {% jira 2532 %} NITF data causes abort in parse
  • {% jira 2552 %} Get Schema Definition Error when using save-parser to compile/save JSON schema to parser.bin file
  • {% jira 2568 %} SAX outputter adding redundant xmlns to elements, causing comparison to expected results to fail
  • {% jira 2565 %} PState.changedVariablesStack not being maintained by VRef in DPath expressions
  • {% jira 1556 %} Roundtrip: complex escape scenarios fail
  • {% jira 1654 %} CLI -v option doesn't turn on LogLevel.Info log messaging for parse or unparse, only compile
  • {% jira 2103 %} Strange behavior with nested defineFormat references
  • {% jira 2266 %} Daffodil does not validate compatability with pre-compiled parsers
  • {% jira 2460 %} Unordered Sequence misidentifies referenced elements Namespaces
  • {% jira 2480 %} Externally setting variables causes Abort/MatchError
  • {% jira 2527 %} DaffodilXMLLoader requires XML prolog. Should tolerate not having one
  • {% jira 2530 %} Embedded schemas broken for ibm-dfdl-crosstester
  • {% jira 2531 %} No details available when TMDLRunner AssumptionViolatedException can't be created
  • {% jira 2538 %} XML String with XML entity breaks in ScalaXMLInfosetInputter
  • {% jira 2553 %} dfdl:occursCountKind aborts with invalid path expression to array
  • {% jira 2555 %} dfdl:occursCountKind aborts with expression results in ParseError
  • {% jira 2585 %} Fix some code generation corner cases
  • {% jira 1512 %} dfdl prefix properties on dfdl annotation elements should generate useful diagnostic
  • {% jira 2445 %} Investigate use of VariableUtils.convert function
  • {% jira 2479 %} Using %#x25; character class causes invariant abort
  • {% jira 2533 %} TDMLRunner not detecting tdml:defineConfig's with same name
  • {% jira 2544 %} Running sbt in windows causes “Unrecognized option: -mem” failure
  • {% jira 2554 %} Debugger infoset output is confusing with hidden groups

Miscellaneous Changes

Performance, code cleanups or maintainability improvements

  • {% jira 1437 %} Clarity: DFA Rules - eliminate anonymous classes and functions
  • {% jira 2526 %} Remove overhead of unnecessary serialization of compiled schema
  • {% jira 2573 %} bad file name CheckDigitLayerTransformerFactory.scala
  • {% jira 1879 %} Compilation related data structures cannot be garbage collected due to RuntimeData strong references
  • {% jira 2520 %} TDML Runner SchemaCache leaks memory
  • {% jira 2521 %} Schema compiler should insure all lazy eval is over with preSerialization call
  • {% jira 2557 %} Fix Apache Conscious Language Checker findings
  • {% jira 2572 %} Regression in schema compilation memory requirements
  • {% jira 1816 %} Unit test rigs don't serialize parser/unparser - allows lazy evaluation during runtime.
  • {% jira 2326 %} Do not pass RuntimeData to DPathCompileInfo constructor

Project Infrastructure Improvements

  • {% jira 2584 %} Release candiate container fails to build MSI
  • {% jira 2252 %} Custom GitHub action to check pull requests with more than one commit
  • {% jira 2518 %} Minor fixes required to release candidate script
  • {% jira 2513 %} Add JFrog Artifactory releases for RPM convenience binaries
  • {% jira 2545 %} geny dependency typo in license file
  • {% jira 2528 %} Enable dependabot for github actions

Deprecation/Compatibility

The Daffodil layering extension formerly supported layering property names (E.G., layerTransform) not in the DFDL extensions namespace. The properties must now have a namespace prefix (usually “dfdlx:”)

  • {% jira 2564 %} Remove deprecated layerTransform properties not in dfdlx namespace.

The Daffodil enumerations and type-calc extension to the DFDL language has been simplified, and numerous extension features withdrawn in this release.

  • {% jira 2273 %} Simplify Enumerations/typeValueCalc feature
  • {% jira 2177 %} Update type-calc proposal

DFDL schemas, configuration files, and TDML files no longer allow any use of DocType declarations or XML Entity definitions. This improves security as these features are associated with a number of XML-related vulerabilities.

  • {% jira 1422 %} disallow doctype decls in all XML & XSD that we read in
  • {% jira 1659 %} Turn off XML general entities - creates vulnerabilities

Dependency Changes

The following dependencies have been added or updated

Core

  • Apache Commons I/O 2.11.0 (update)
  • FasterXML Jackson Core 2.13.0 (update)
  • FasterXML Woodstox Core 6.2.7 (update)
  • ICU4J 70.1 (update)
  • Log4j API Scala 12.0 (new)
  • Log4j core 2.14.1 (new)
  • Scala XML 1.3.0 (update)
  • Scala Parser Combinators 2.1.0 (update)
  • Typesafe Config 1.4.1 (update)

Command Line Interface

  • JLine 3.21.0 (update)
  • Jansi 2.4.0 (update)
  • Log4j core 2.14.1 (new)
  • Scallop 4.1.0 (update)

Code Generator

  • OS-Lib 0.7.8 (update)

Schematron Validator

  • Saxon-HE 10.6 (update)

Test

  • Junit-interface 0.13.2 (update)
  • ExpectIt 0.9.0 (update)