released: true apache: true title: 2.4.0 date: 2019-07-12 summary: > Improved DFDL schema compatibility, new extensions, explicit length choices, better DFDL schema validation, and many bug fixes.

source-dist: - “apache-daffodil-2.4.0-incubating-src.zip”

binary-dist: - “apache-daffodil-2.4.0-incubating-bin.tgz” - “apache-daffodil-2.4.0-incubating-bin.zip” - “apache-daffodil-2.4.0-incubating-bin.msi” - “apache-daffodil-2.4.0.incubating-1.noarch.rpm”

scala-version: 2.12

Explicit Length Choices

Support is added for the property dfdl:choiceLengthKind="explicit" and dfdl:choiceLength. Previously, Daffodil ignored the choiceLengthKind property, always acting as if it had the value of implicit. The property is now required to be defined in scope for all choices.

  • {% jira 640 %} choiceLengthKind=‘explicit’

Extension Namespace

A new dfdlx namespace (http://www.ogf.org/dfdl/dfdl-1.0/extensions) was created to support Daffodil extensions to the DFDL specification. The existing daf namespace is deprecated and all uses should be switched to dfdlx. Some properties, such as the layering extension properties are also moved to this namespace with backwards compatibility supported for now. Extensions should be considered experimental--support may change or be removed in a future release.

  • {% jira 2096 %} Add dfdlx extensions namespace prefix. Convert existing extensions to this namespace.

Extension: Empty Element Parse Policy [Proposal]

A new DFDL property named dfdlx:emptyElementParsePolicy is added to control the behavior of empty string and hexBinary elements. This property allows one to switch between different behaviors for empty elements which correspond to currently described behavior in the DFDL v1.0 specification document, and an alternate behavior planned for future inclusion in the DFDL v1.0 specification which also increase compatibility with IBM DFDL.

  • {% jira 110 %} Implement separator suppression policies
  • {% jira 1080 %} RepUnbounded Error: Sequence of Optional Comma-separated String items
  • {% jira 1400 %} broken test: test_text_entities_6_03b - separators with different encoding than terms they separate
  • {% jira 1919 %} Trailing Separator Suppression
  • {% jira 1975 %} Incorrect separator behavior - missing separator for empty sequence (non-backward compatible)
  • {% jira 1976 %} Incorrect creation of empty-string element
  • {% jira 2134 %} emptyElementParsePolicyError not accepted as warning ID in TDML define config.
  • {% jira 2135 %} Array index not maintained properly. Causes nillable failures among others.
  • {% jira 2143 %} dfdl:initiatedContent=“yes” on sequence should only affect optional children
  • {% jira 2154 %} nilled element of zero length inside optional complex element is deleted
  • {% jira 2162 %} direct dispatch choice that has an array branch causes abort during compilation

Extension: Enumeration Values [Proposal]

New DFDL properties and functions are added to support translation of enumeration values.

  • {% jira 2107 %} Implement Enum Support
  • {% jira 2146 %} Incorrectly recognize keyset-value transform as identity transfrom
  • {% jira 2148 %} No compile time error when using nonexistent TypeCalc function
  • {% jira 2165 %} Add *Double functions to the typeCalc family of functions

Extension: Ranged Direct Dispatch [Proposal]

A new property named dfdlx:choiceBranchKeyRanges is added to support direct dispatch based on a range of numeric values.

  • {% jira 2157 %} Implement choiceBranchKeyRanges

Extension: Lookahead Expression Function [Proposal]

A new DFDL expression function named dfdlx:lookAhead() is added to support a simple way to consider data that occurs at a future point in the bitstream.

  • {% jira 2140 %} Implement dfdl:lookAhead()

Improved DFDL Schema Validation

Daffodil now does a better job at validating attribute-, short-, and element-form DFDL properties, which will now result in an error if unknown properties are used, mostly commonly caused by typos. Additionally, if DFDL properties are ignored (e.g. dfdl:textPattern is provided when dfdl:lengthKind is not “pattern”), Daffodil will now provide a warning.

  • {% jira 598 %} No validation of short-form DFDL properties
  • {% jira 1619 %} Warn of when dfdl annotations are ignored
  • {% jira 2142 %} DFDL annotation on complex type is silently ignored
  • {% jira 2150 %} Excessive emptyElementParsePolicy warnings
  • {% jira 2163 %} False warning about unused dfdl:choiceBranchKey on direct dispatch array

Deprecation/Compatibility

The following changes have been made which affect compatibility with past releases:

  • The behavior of separators has been revised to better conform to the DFDL specification, which enables more DFDL schemas to work without Daffodil specific modifications. Schemas that relied on the previous non-comformant behavior may require changes.
  • The dfdl:choiceLengthKind property must now be defined in scope of all xs:choice's. Allowed values are either implicit or explicit. The previous behavior always treated the property as if it had the value of implicit.
  • The daf namespace has been deprecated and replaced with the dfdlx namespace. Some properties were also moved to the new dfdlx namespace. The following properties/functions should be updated to use the new namespace, with backwards compatibility supported for now:
    • dfdl:layerTransform
    • dfdl:layerEncoding
    • dfdl:layerLengthKind
    • dfdl:layerLength
    • dfdl:layerLengthUnits
    • dfdl:layerBoundaryMark
    • daf:parseUnparsePolicy
    • daf:trace()
    • daf:error()

DFDL Schema Compatibility

Many changes and bug fixes specifically made to improve compatibility with more DFDL schemas. With these fixes and features, the schemas HL7-v2.7, IBM4690-TLOG, and EDIFACT now work with Daffodil.

  • {% jira 784 %} Update default properties in examples to have missing required properties for portability.
  • {% jira 828 %} IBM Compatibility - Test out HL7 schemas
  • {% jira 1639 %} IBM Compatibility - EDIFACT
  • {% jira 1720 %} missing properties not reported
  • {% jira 1853 %} ipfix data format parse and unparse tests for “Schema Definition Error”
  • {% jira 1886 %} TLOG: Delimited parsing with occursCountKind=“implicit” with separators is not handling missing optional elements correctly
  • {% jira 1916 %} Properties leadingSkip alignment alignmentUnits trailingSkip should not be required by text formats.
  • {% jira 2121 %} occursStopValue=“-1” in test schemas is non-portable (currently)
  • {% jira 2156 %} daffodil-test-ibm1-crossTest now fails

Infrastructure and Website

  • {% jira 1380 %} Update genjavadoc plugin to 10 when released and enable doclint
  • {% jira 2003 %} Add Coverage Testing
  • {% jira 2070 %} KEYS file should not be included in source releases
  • {% jira 2073 %} Contact information in README.me is incorrect
  • {% jira 2074 %} Tests fail on non-US locales
  • {% jira 2080 %} Prepare for Daffodil 2.4.0 development
  • {% jira 2082 %} Links to hashes sigs and KEYS should use https
  • {% jira 2093 %} Cannot build and test daffodil in IntelliJ
  • {% jira 2095 %} Confirm all commented out tests either work or have an associated bug number
  • {% jira 2098 %} Update README to be more consistent and have extra information
  • {% jira 2105 %} TravisCI builds take too long
  • {% jira 2106 %} Update Line counter utility
  • {% jira 2113 %} Eclipse classpath fixup for removal of scala-debug directories
  • {% jira 2117 %} Generate WarnID.scala and DaffodilTunables.scala off of dafext.xsd
  • {% jira 2160 %} Ignore vim temp files
  • {% jira 2166 %} TravisCI open jdk 11 fails to download

Miscellaneous Changes and Bug Fixes

  • {% jira 1024 %} Property inputValueCalc should not be on element that is optional, global, or an array
  • {% jira 1106 %} Catch of Exception in DFDLSchemaFile.scala
  • {% jira 1109 %} Remove catch of Exception from OccursCountExpressionParser
  • {% jira 1387 %} Data dump no longer showing for left over data
  • {% jira 1407 %} xml comments not accepted in defineFormat, config files
  • {% jira 1475 %} Runtime terminator containing isolated %ES; provides poor diagnostic.
  • {% jira 1477 %} ES now allowed in terminator - causes a test to fail now.
  • {% jira 1478 %} Debugger uses the term arrayIndex. Really should be occursIndex to match the function of the same name.
  • {% jira 1493 %} %ES; delimiter doesn't work right
  • {% jira 1525 %} Determine if there is a better way to convert an XML String to a date/time
  • {% jira 1701 %} dfdl:outputValueCalc on a complex element causes a ClassCastException
  • {% jira 1931 %} Debug/trace - prints spurious error message about “..”
  • {% jira 1955 %} Missing closing dfdl:format tag causes FatalError exception
  • {% jira 1956 %} Terminator with leading space causes assertion failure
  • {% jira 2058 %} Some CLI tests do not work cleanly on the Windows platform.
  • {% jira 2062 %} NadaParser not optimized out
  • {% jira 2064 %} CLI XML text output incorrect with empty complex elements.
  • {% jira 2065 %} Incorrectly attempting to cast signed to unsigned type
  • {% jira 2069 %} Unparser interaction of outputValueCalc with hexBinary with following bitOrder LSBF element.
  • {% jira 2071 %} Performance of Schema Compiler - Need to eliminate redundant XML loading
  • {% jira 2075 %} dfdl:hexBinary does not correcty reduce xs:integer
  • {% jira 2076 %} fn:ceiling does not work on xs:decimal
  • {% jira 2078 %} Unexpected exception with dfdl:contentLength
  • {% jira 2083 %} abort in parser runtime - invariant failed with dynamic separator test
  • {% jira 2084 %} attributeFormDefault=“unqualified” appears in SchemaUtils used by TDML - generates spurious warning
  • {% jira 2089 %} Unparser escaping doesn't escape all the delimiters
  • {% jira 2092 %} Bad diagnostic when using dynamic separator the value of which is NUL (ascii 0)
  • {% jira 2097 %} Non-thread safe code related to EvCalendar
  • {% jira 2099 %} Original errors being surpressed in doParse loop
  • {% jira 2100 %} Error handling some parseErrors
  • {% jira 2108 %} Facet restrictions not allowed on xs:decimal types with bad errror
  • {% jira 2114 %} Delimiter expression should SDE when empty string returned
  • {% jira 2118 %} DFDLSchemas BMP does not work on Daffodil 2.3.0
  • {% jira 2119 %} Debugger RTL data dumps are nibbles-reversed
  • {% jira 2123 %} Exception on error when parsing hex data with trace on
  • {% jira 2125 %} Incorrect unsparse with suspensions and changing bitOrder
  • {% jira 2126 %} XML Catalog uri should use the same semantics as import/include schemaLocation
  • {% jira 2128 %} XML preamble encoding ignored when CLI unparsing with “xml” infoset type
  • {% jira 2130 %} Error on schema with large numbers of files
  • {% jira 2136 %} Fix whitespace spurious changes.
  • {% jira 2155 %} Add support for Link16 Charset DFI311:DUI002
  • {% jira 2164 %} Support Link16 charset DFI1661 DUI001
  • {% jira 2167 %} NITF schema fails to unparse with exception