site/_releases/2.3.0.md

released: true apache: true title: 2.3.0 date: 2019-02-26 summary: > New features, changes, and bug fixes to improve IBM DFDL compatability

source-dist: - “apache-daffodil-2.3.0-incubating-src.zip”

binary-dist: - “apache-daffodil-2.3.0-incubating-bin.tgz” - “apache-daffodil-2.3.0-incubating-bin.zip” - “apache-daffodil-2.3.0-incubating-bin.msi” - “apache-daffodil-2.3.0.incubating-1.noarch.rpm”

scala-version: 2.12

TDML Runner Cross Validation

It is now possible to use the TDML runner with tests with different DFDL implementations, including the IBM DFDL implementation. DFDL implementations must create a class named TDMLDFDLProcessorFactory and implement the abstract functions. A new implementation attribute is added to the tdml:parserTestCase and tdml:unparserTestCase elements to define which tests an implementation is known to work with.

{% jira 723 %} Compatibility: Cross Validation test rig for DFDL and IBM implementation
{% jira 2032 %} Improve Eclipse setup for cross testing

New in the TDML runner is type-aware infoset comparisons. One can now provide an xsi:type attribute in infoset elements, allowing the TDML runner to determine if two elements are logically the same even if there infoset values may differ.

{% jira 2026 %} TDML Runner - type-sensitive comparison

Prefixed Length Kind

Support is added for dfdl:lengthKind="prefixed" and the properties dfdl:prefixIncludesPrefixLength and dfdl:prefixLengthType.

{% jira 114 %} LengthKind Prefixed
{% jira 2035 %} prefixLengthType and element lengthUnits only needs to be the same when prefixIncludePrefixLength

Deprecation/Compatibility

The following changes have been made which affect compatibility with past releases:

Infoset elements with type xs:dateTime and xs:time no longer contain milliseconds precision if the milliseconds value is 0.

{% jira 2005 %} dateTime types without milliseconds precision should not output .000000

Previous versions of Daffodil automatically coerced the result of a DFDL expression to the expected type if they differed. This is incorrect behavior according to the DFDL specification, and now results in a warning when such coercion is required. Instead, expressions must explicitly cast the result type to the expected type. This auto-coercion + warning behavior is deprecated and will change to a schema definition error in a future release.

{% jira 2021 %} Daffodil expressions for properties should not auto-coerce to property type. Test BC000.
{% jira 1044 %} DPath not finding obvious type errors involving constants at compile time
{% jira 2054 %} Change coerecion errors to warnings and eventually deprecate

Elements with a type of xs:hexBinary now ignore the dfdl:byteOrder property and always parse data as if it were bigEndian. The result is that hexBinary bytes where dfdl:byteOrder="littleEndian" now appear reversed when compared to previous releases.

{% jira 2019 %} daf extension property to turn on/off “hexBinary with bits” lengthUnits behavior

Official support for Java 7 has been removed.

{% jira 2025 %} Remove OracleJDK7 from TravisCI

IBM DFDL Compatibility

Other changes and bug fixes specifically made to improve IBM DFDL compatibility:

{% jira 20 %} obtain and integrate/run IBM test cases (even if they all break)
{% jira 559 %} TDML runner needs feature to tolerate left-over data for IBM test compatibilitiy
{% jira 565 %} tolerate attributeFormDefault on xs:schema
{% jira 999 %} IBM's ISO8583 format not currently compatible with Daffodil
{% jira 1441 %} Implement examples for DFDLSchemas
{% jira 1616 %} textOutputMinLength required when it shouldn't be needed
{% jira 1965 %} dfdl:assert and dfdl:discriminator “message” attribute does not work for DFDL Expressions
{% jira 1986 %} Expression comparison of date/time w/o timezone defined causes abort
{% jira 1988 %} textOutputMinLength - incorrect code found by inspection
{% jira 1989 %} lengthKind ‘explicit’ with expression should not use minLength nor textOutputMinLength
{% jira 2006 %} DFDLGeneralFormat missing binaryCalendarEpoch.
{% jira 2015 %} Accept textStandardExponentCharacter (Old property name)
{% jira 2017 %} Non-portable date/time test_simple_type_properties_text_calendar_13_02
{% jira 2031 %} Type with xs:hexBinary requires the dfdl:binaryNumberRep property
{% jira 2033 %} dfdl:defineVariable defaultType does not properly convert value
{% jira 2063 %} DPath: long div long results in double. Should be decimal.

Improved Windows Support

Tests are updated to be compatible with Windows. A new MSI installer is now made available for releases.

{% jira 2047 %} Create Windows MSI Installer
{% jira 2051 %} Many tests are OS-dependent and fail on Windows platform
{% jira 2058 %} Some CLI tests do not work cleanly on the Windows platform.
{% jira 2066 %} Support building MSI installer in release script

Miscellaneous Changes and Bug Fixes

{% jira 711 %} Hefty Abort Message when using an expression (and previous elements) to set formatting properties
{% jira 809 %} assert with testKind pattern on simple element not executed sometimes.
{% jira 947 %} Warn when space separated fields start or end with a space
{% jira 1221 %} Abort: Invariant Broken when XPath expression moves past root
{% jira 1287 %} Vague “assertion failed” error when expression with whitespace used as separator
{% jira 1324 %} Unparse: Sequences need way to skip over optional branches
{% jira 1427 %} Use scala 2.11 options to clean up dead-code, unused imports, unused methods, unused local vals, etc.
{% jira 1440 %} JAPI/SAPI DataProcessor should be serializable
{% jira 1476 %} Line numbers in error messages off by size of comments above
{% jira 1840 %} dfdl:initiator=“%” does not cause a error or warning
{% jira 1946 %} Dates with out of range years will overflow to a valid value
{% jira 1953 %} CLI treats OutOfMemoryError as a bug
{% jira 1954 %} Java version greater than 1.8 cause failure
{% jira 1996 %} LICENSE has typo in “Open Grid Format”
{% jira 2002 %} javadoc for java api fails on JDK10+
{% jira 2010 %} binarySeconds/milliSeconds missing length check, leaking exception and mark
{% jira 2011 %} CLI should always use UTF-8 encoding when outputting the infoset
{% jira 2012 %} “charaters” typo in entity replacer error message
{% jira 2020 %} Infinite loop if the dfdl:hiddenGroupRef reference does not exist
{% jira 2022 %} infinite loops when using dfdl:property extra-long form for property values
{% jira 2034 %} Pool leaks if encodingErrorPolicy=“error” globally
{% jira 2036 %} poor diagnostic - choice with sequences as branches with all optional content.
{% jira 2038 %} computed elements inside choices - alignment-related abort
{% jira 2043 %} fn:round-half-to-even returns decimal string
{% jira 2044 %} Unexpected error on malformed config file
{% jira 2045 %} Performance degredation with hexBinary
{% jira 2046 %} Poor handling of missing source attribute on <xs:appinfo> element
{% jira 2050 %} Line numbers missing in schema definition errors
{% jira 2056 %} Implicit expression conversion from xs:string to xs:hexBinary causes exception
{% jira 2059 %} Computing the diff of large XML results in a stack overflow
{% jira 2061 %} Prepare for 2.3.0 release
{% jira 2062 %} NadaParser not optimized out
{% jira 2065 %} Incorrectly attempting to cast signed to unsigned type
{% jira 2067 %} Compile dependency on JUnit4

Infrastructure and Website

{% jira 1997 %} SHA1 hashes are no longer desired for releases
{% jira 1998 %} Document the need for more memory when running tests via sbt
{% jira 2008 %} Create DOAP file for Daffodil
{% jira 2009 %} Release script should create shasums with the binary flag
{% jira 2016 %} Remove Java 10 from the travis CI builds, causing failed builds
{% jira 2039 %} update unsupported features and errata page - too many green checkmarks
{% jira 2053 %} No link to https://daffodil.apache.org/infoset/ on any doc pages
{% jira 2055 %} TDML Page - typographical error