released: true apache: true title: 2.1.0 date: 2018-05-14 summary: > Apache incubation/relicensing, packed decimal support, ambiguous expression support, refactoring for future performance improvements

source-dist: - “apache-daffodil-2.1.0-incubating-src.zip”

binary-dist: - “apache-daffodil-2.1.0-incubating-bin.tgz” - “apache-daffodil-2.1.0-incubating-bin.zip” - “apache-daffodil-2.1.0.incubating-1.noarch.rpm”

scala-version: 2.11

Daffodil 2.1.0 is the first release of Daffodil as an Apache incubator project. This release includes relicensing the source to Apache v2, changing the namespace to org.apache.daffodil, and numerous bug fixes and DFDL feature additions to support more DFDL schemas.

This release contains changes which break compatibility with past releases. Please see the deprecation / compatibility notes section for more information.

Packed Decimal

Added support for binary packed decimal data types. In addition to standard, the dfdl:binaryNumberRep property now also accepts values of bcd, packed, and ibm4690Packed to support Binary Coded Decimal, IBM 390 Packed Decimal, and IBM 4690 Packed Decimal, respectively. The dfdl:binaryDecimalVirtualPoint property applies for the appropriate types. Also implemented are dfdl:binaryPackedSignCodes and dfdl:binaryNumberCheckPolicy to configure the valid nibble values when dfdl:binaryNumberRep is packed.

  • {% jira 1739 %} Implement ‘packed’ and ‘bcd’

Apache Incubation Support

Multiple changes were made to support the move to the Apache incubator. This includes changes to the source directory structure, relicensing files to Apache v2, changing the project package to org.apache.daffodil, updating the build infrastructure to SBT 1.x, and adding TravisCI continuous integration support.

  • {% jira 1612 %} sbt - branches without tags cause sbt compile to fail
  • {% jira 1841 %} daffodil root jar is published but shouldn't be, and its contents are wrong
  • {% jira 1856 %} Changes necessary for apache incubation
  • {% jira 1857 %} Update sbt to 1.x, simplifing build config and replacing deprecated syntax
  • {% jira 1858 %} Add support for TravisCI
  • {% jira 1859 %} Switch to apache.org namespace and Apache v2 license
  • {% jira 1870 %} invalid and nonfunctional URLs in source files
  • {% jira 1873 %} sbt warnings/errors
  • {% jira 1890 %} Update user related links from the wiki to daffodil.apache.org
  • {% jira 1896 %} Modify artifact names to include “apache” and “incubating”
  • {% jira 1897 %} Some Schemas fail to compile with latest updates
  • {% jira 1899 %} Fix NOTICE and LICENSE files
  • {% jira 1901 %} Apache Rat cannot detect licenses for Passera, Scala libs, and W3C files
  • {% jira 1906 %} Updates to LICENSE/NOTICE files

Ambiguous Path Expressions

Previously, ambiguous path expressions (i.e. path expressions that could potentially reference multiple non-array elements of the same name) were forbidden and caused a schema definition error at schema compilation time. As of 2.1.0, such ambiguities are detected at compile time but are now only considered schema definition warnings. However, if a path expression ever evaluates to more than one non-array element at runtime, it will then be considered an error and cause a runtime schema definition error. This change allows for schemas that appear to have ambiguous path expressions, but in only ever evaluate to a single element in practice.

  • {% jira 1773 %} Choice ambiguous element name results in failed expression
  • {% jira 1854 %} Remove concept of slots from InfosetImpl.scala
  • {% jira 1860 %} Minimize the number of HashMap allocations/insertions in the Infoset
  • {% jira 1869 %} Nato link16 doesn't work on 2.0.0 Daffodil
  • {% jira 1877 %} Slow schema compilation due to unambiguous checks

TDML Runner Enhancements

Changes were made to the TDML runner to support an embedded defaultConfig option and to verify expected schema definition warnings.

  • {% jira 1583 %} TDML Runner - expected warnings feature
  • {% jira 1846 %} TDML Runner - Infoset.contents is stripping off all attributes
  • {% jira 1868 %} TDML Runner defaultConfig cannot be embedded. Not checked properly.

BitOrder/ByteOrder/Encoding Refactoring

The way bitOrder, byteOrder, and encoding consistency was checked required fairly complicated logic at schema compilation time. Not only was this logic difficult to validate, which led to bugs, it also contributed to slow schema compilation times. This complex logic was removed and replaced with an efficient mechanism for runtime validation. This reduced code complexity, fixerd some bugs, and will facilitate future performance optimizations during schema compilation.

  • {% jira 1001 %} Invalid bitOrder and byteOrder combination should produce SDE
  • {% jira 1597 %} Too many ways that encoding, byteOrder, etc. are being setup
  • {% jira 1732 %} BitOrderChangeParser not inserted correctly for bitmap schema
  • {% jira 1767 %} byteOrderChangeParser optimization not working
  • {% jira 1843 %} Unparser bitOrder change and OVC (outputValueCalc) interaction
  • {% jira 1876 %} Improve performance regression due to FormatInfo changes
  • {% jira 1884 %} Regression in bitOrder changing

XML Output

The XML infoset output (via the XMLTextInfosetOutputter) now writes an XML declaration <?xml version="1.0" ... ?> at the beginning of output. It also ensures output is flushed at the end of writing, which should prevent issues users reported of receiving only partial data.

  • {% jira 1844 %} XMLTextInfosetOutputter should output an XML declaration
  • {% jira 1871 %} XMLTextInfosetOutputter endDocument should flush the writer

Miscellaneous Bug Fixes

  • {% jira 1473 %} DFDL Schema Validation not happening properly
  • {% jira 1838 %} Need more warning suppression options
  • {% jira 1851 %} Unparsing delimited data with escape characters could lead to truncation
  • {% jira 1855 %} Property Resolution - resolve onto ref objects
  • {% jira 1862 %} Better logging/reset mechanism for MarkPool
  • {% jira 1864 %} spurious SDE about all-text schema being non-scannable
  • {% jira 1866 %} Eclipse classpaths need fixing
  • {% jira 1867 %} Memory leak in Register
  • {% jira 1872 %} Eclipse classpaths for daffodil-io module need fixing
  • {% jira 1874 %} Incorrect warning message about default value and no empty representation
  • {% jira 1885 %} Initiators are not being inherited from parent choices
  • {% jira 1893 %} TDML Runner not caching schema compiles - only caches schemas if compileAllTopLevel is true
  • {% jira 1895 %} Eclipse classpath for daffodil-core needs fix
  • {% jira 1903 %} Eclipse classpath broken for daffodil-test and other tests - now must have daffodil-lib-unittest
  • {% jira 1907 %} Daffodil disallows attribute declarations for non-DFDL annotation elements.
  • {% jira 1910 %} Cannot unparse when array with minOccurs=“0” is last in a sequence
  • {% jira 1911 %} Move daffodil-cli tests to use sbt's “integration” test capabilities
  • {% jira 1913 %} Strange jar ends up in lib_managed of daffodil - trips up Eclipse IDE users
  • {% jira 1914 %} Eclipse classpath broken for daffodil-cli
  • {% jira 1920 %} toString representation of grammar prims and parser/unparsers need improvement
  • {% jira 1921 %} option to bypass Java 8 check

Deprecation/Compatability

As of 2.1.0, the following changes have been made which affect compatibility with past releases:

Package Organization Change

Now that Daffodil is an Apache incubator project, the package organization has been changed from edu.illinois.ncsa to org.apache.daffodil. Developers using the Daffodil Java and Scala API will need to update their source code and SBT/Maven configurations to use the new organization.

Daffodil Extension daf XML Namespace Change

The Daffodil extension daf XML namespace has been modified to represent the move to Apache, changing from urn:ogf:dfdl:2013:imp:opensource.ncsa.illinois.edu:2012:ext to urn:ogf:dfdl:2013:imp:daffodil.apache.org:2018:ext. Backwards compatibility is maintained for schemas using the old daf namespace, though it should be considered deprecated. All future Daffodil extensions will only be supported in the new namespace, so it is recommended that one update schemas to use the new namespace.

Deprecation of built-in-formats.xsd

The built-in-formats.xsd file is a schema that defines a set defaults for a wide array of DFDL properties. This file was intended for use only during testing, but was errantly made available for use in real schemas. This was an oversight, and so use of this schema will now result in a deprecation warning. Instead, one should import/include org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd and set the dfdl:format ref value to GeneralFormat to get a reasonable set of default property values or define all necessary properties within a schema.