title: 3.5.0 release: rc1 apache: true date: 2023-06-21 summary: > OSGI, Cobol, and improved EXI support, and miscellanous bug fixes

source-dist: - “apache-daffodil-3.5.0-src.zip”

binary-dist: - “apache-daffodil-3.5.0-bin.tgz” - “apache-daffodil-3.5.0-bin.zip” - “apache-daffodil-3.5.0-bin.msi” - “apache-daffodil-3.5.0-1.noarch.rpm”

scala-version: 2.12

This release focused primarily on bug fixes. A number of changes resulted in deprecation or backwards incompatible changes. See the Deprecation/Compatibility section for more details.

Additional New Features/Improvements

  • {% jira 847 %} NotYetImplemented Exception for list of textStandardDecimalSeparator
  • {% jira 853 %} COBOL: textNumberPattern: ‘V’ symbol
  • {% jira 2381 %} Refactor test util to eliminate redundant windows baggage from CLI Tests
  • {% jira 2400 %} New SAX API causes performance degradations
  • {% jira 2743 %} remove all deprecated methods/classes/objects
  • {% jira 2745 %} Prepare for 3.5.0 development
  • {% jira 2747 %} Add GitHub action to support macOS
  • {% jira 2749 %} Add Daffodil Developer Guide
  • {% jira 2750 %} Add option to encode/decode EXI files
  • {% jira 2756 %} Update CI workflows and build tools
  • {% jira 2762 %} Add more daffodilC support (arrays, alignment)
  • {% jira 2763 %} ‘P’ textNumberPattern feature
  • {% jira 2772 %} Eliminate CLI it:test target and src/it directory in favor of ordinary src/test
  • {% jira 2775 %} stringAsXML test needs to use mixed content
  • {% jira 2779 %} Add GitHub autolinking to the Jira issue tracker
  • {% jira 2786 %} Refactor how Main calls “generate c”
  • {% jira 2790 %} Merge runtime2's two redundant processors
  • {% jira 2796 %} Rename daffodil-runtime2
  • {% jira 2799 %} Separate “lint” checks from “OS/JDK” matrix checks
  • {% jira 2804 %} Make minor codegen-c improvements
  • {% jira 2807 %} Add left over data check to generated C parsers
  • {% jira 2819 %} Add codegen-c support for runtime length hexBinary

Miscellaneous Bug Fixes

  • {% jira 768 %} An element with both name and ref defined should cause an error
  • {% jira 908 %} Test should result in Runtime SDE, not abort exception
  • {% jira 961 %} Escape Scheme Scenario “scenario3_11_postfix” failing
  • {% jira 1026 %} Invalid configuration file causes exception
  • {% jira 1172 %} length units bits for xs:integer and xs:nonNegativeInteger, should be SDE
  • {% jira 1275 %} XCatalog and Classpath interaction with name resolution
  • {% jira 1454 %} dfdl:hexBinary vs xs:hexBinary constructors: string input
  • {% jira 1651 %} Unparse: dfdl:content/valueLength of an Array returns unexpected result
  • {% jira 1704 %} PE undetected - Binary number parsers are missing checks for bitLength
  • {% jira 2101 %} Issues with nested separators
  • {% jira 2133 %} Enable and configure sbt-scalariform/scalafmt plugin
  • {% jira 2147 %} DFDLGeneralFormat.dfdl.xsd wrong value for textStandardZeroRep. Should be ""
  • {% jira 2158 %} textNumberPattern not being used for xs:int ?
  • {% jira 2161 %} inputvalueCalc on direct dispatch choice branch is too verbose
  • {% jira 2212 %} Enumerations validate length on symbolic elements
  • {% jira 2219 %} Unparser with dfdl:separatorSuppressionPolicy=“never” bug
  • {% jira 2231 %} Separator unparsed even when dfdl:occursCount expression is zero
  • {% jira 2346 %} XML Output needs option to use <![CDATA[ ]]> around simple element values containing whitespace.
  • {% jira 2364 %} infinite loop with separators and suppression (NUL related)
  • {% jira 2390 %} Handle Exceptions from SAXInfosetOutputter's contentHandler Calls
  • {% jira 2393 %} relative path past root - incorrect error
  • {% jira 2441 %} Compiling a non DFDL schema with warnings turned off results in an Abort
  • {% jira 2496 %} implement emptyElementParsePolicy as a dfdl property. It is part of DFDL 1.0
  • {% jira 2499 %} separatorSuppressionPolicy ‘never’ incorrectly creates empty elements (needs diagnostic)
  • {% jira 2562 %} Optional element at end of choice branch triggers unparse error
  • {% jira 2586 %} Schema compilation hangs if incorrectly closed attribute
  • {% jira 2596 %} lengthKind property is not inherited for types with dfdlx:repType
  • {% jira 2607 %} Poor diagnostic - not an enum value but complains of keyset-value mapping
  • {% jira 2614 %} assert failure in schema-defined layer transform appears to be a Daffodil bug
  • {% jira 2615 %} group sharing causes obscure unparser runtime error
  • {% jira 2624 %} cannot use trace when doing streaming parse calls
  • {% jira 2628 %} abort: invariant broken when assert test expression does not return boolean
  • {% jira 2632 %} NumberFormatException for blank dfdlx:repValues
  • {% jira 2634 %} SDW message should include identifier of the warning for suppressing it
  • {% jira 2645 %} Parse Error: Insufficient bits. Needed 160 but found only 1192
  • {% jira 2647 %} Diagnostic says “The property ‘null’ ...”
  • {% jira 2668 %} Variable Error caused by ignored newVariableInstance in Group Annotation
  • {% jira 2676 %} number constructors from string do not tolerate leading whitespace
  • {% jira 2683 %} Don't split packages across jars - OGSI packaging requirements
  • {% jira 2728 %} Missing layer plugin causes abort “this is a bug”
  • {% jira 2744 %} stack_t and strnum conflict with macOS developer platform libraries
  • {% jira 2748 %} Update cli and runtime2-todos
  • {% jira 2751 %} Occasional network timeout exceptions can hang a CI job now
  • {% jira 2755 %} infoset nodes removed by InfosetWalker tha are still needed, null pointer exception
  • {% jira 2767 %} Mixed content with SAX unparsing hangs coroutine thread
  • {% jira 2770 %} Refactor and document SAX unparse implementation
  • {% jira 2774 %} ICU4J DecimalFormat is thread-safe as of ICU 59
  • {% jira 2776 %} GitHub action badge broken in README
  • {% jira 2777 %} Fix generated C code for offset between 2 array slots
  • {% jira 2778 %} Make the CLI thread-safe
  • {% jira 2782 %} NegativeArraySizeException when hexBinary has length > 1,073,741,823
  • {% jira 2785 %} Abort: Usage error: startBitAddress0b.>=(0) in from DataLocation.bytePos1b after parsing large GIF
  • {% jira 2787 %} log4j-core and log4j-api version mismatch can lead to very confusing errors
  • {% jira 2789 %} CI runs cancelled when merged in to main branch too quickly
  • {% jira 2791 %} dfdl:occursIndex() returns wrong value when emptyElementParsePolicy=“treatAsMissing”
  • {% jira 2800 %} appinfo annotations with distinct source are being misinterpreted as DFDL annotations.
  • {% jira 2802 %} Array with optional element followed by scalar optional element drops separator
  • {% jira 2803 %} Warnings included in serialized DataProcessor
  • {% jira 2805 %} Comment statement references incorrect test
  • {% jira 2808 %} Incorrect Validation Error on Optional Element with Children
  • {% jira 2811 %} sbt IntegrationTest being deprecated, need alternative for slow CLI tests
  • {% jira 2813 %} Schematron URIResolver uses Classpath rather than Daffodil Resolver
  • {% jira 2816 %} Saved parser version checks against application version, not Daffodil version
  • {% jira 2818 %} Failed assertion when parsing text with negative zero value
  • {% jira 2820 %} GeneralFormatPortable isn't actually portable due to emptyElementParsePolicy
  • {% jira 2821 %} Optional Time type element shows up in Infoset
  • {% jira 2822 %} Prepare 3.5.0 relesae

Deprecation/Compatibility

  • The GeneralFormatPortable define format has been removed. Instead, portable schemas should import a new portable file and reference the GeneralFormat format. For example, a schema that previously looked like this:

    <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
    
    <dfdl:format ref="GeneralFormatPortable" ... />
    

    Should now look like this:

    <xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormatPortable.dfdl.xsd" />
    
    <dfdl:format ref="GeneralFormat" ... />
    

    {% jira 2820 %}

  • The property textStandardZeroRep used to be defined to be “0” which is a mistake. However, existing schemas may be depending on this, or they may not be depending on this, but simply have tests which have expected data output containing the value 0.0 with “0” as the text representation. With this change to dfdl:textStandardZeroRep="" those tests may be broken.

    {% jira 2147 %}

  • Previous Daffodil releases let schemas define every type‘s length using “bits” as the length unit even though the specification allows bit length units only for a specific set of types’ binary representations and does not allow bit length units for any other type‘s binary representation or any type’s text representation. A deprecation warning is now issued when bit length units are incorrectly used.

    {% jira 1172 %}

  • Daffodil now uses SLF4J for logging instead of log4j. API users may need to add a new dependency to specify which logging backend to use, for example slf4j-log4j for the previous behavior. If no backend is found, a warning will be output to stderr and log messages will be dropped.

    DFDL schema projects that previously added log4j-core (or another log4j implementation) as a dependency to avoid warnings no longer need that dependency--a custom Daffodil specific logger is now used for TDML tests and is automatically pulled in as dependency.

    {% jira 2787 %}

  • The default and maximum value of the maxHexBinaryLengthInBytes tunable, and thus the maximum length of xs:hexBinary elements, is reduced in half from 2147483647 to 1073741823 bytes. Schemas requiring larger values should switch to the Binary Large Object extension or convert the single large hexBinary element to an array of multiple smaller hexBinary elements.

    {% jira 2782 %}

  • Daffodil now ensures that binary specified lengths are not larger than the max length of the associated type. For example, xs:int cannot be used if field length is larger than 32 bits. Instead, an xs:long or larger type should be used.

    {% jira 1704 %}

  • Some package names were modified to support OSGI. Classes in daffodil-tdml-processor, daffodil-japi, and daffodil-sapi jars were unchanged, so those using these APIs should not need changes. However, classes in other packages may need to adjust imports.

    {% jira 2683 %}

  • The behavior of textNumberCheckPolicy=“strict” was changed so that the pattern is used to determine if a decimal point must exist or not. For example, if text numbers are integers without a decimal, the pattern should not contain a decimal. Schema should be modified to either use a lax policy or use a pattern that correctly matches their data.

    {% jira 2158 %}

Dependency Changes

The following dependencies have been added or updated:

Core

  • Apache Commons IO 2.13.0 (update)
  • FasterXML Jackson Core 2.15.2 (update)
  • FasterXML Woodstox Core 6.5.1 (update)
  • ICU4J 73.1 (update)
  • Scala Parser Combinators 2.3.0 (update)
  • Scala Reflect & Standard Library 2.12.18 (update)

CLI

  • JLine 3.22 (update)

Code Generator

  • OS-Lib 0.9.1 (update)

Schematron Validator

  • Saxon-HE 12.2 (update)

Logging

  • SLF4J API 2.0.7 (update)
  • Scala Logging 3.9.5 (new)

Changes to Transitive Dependencies

  • XML Resolver & data 5.1.2 (update, used by Saxon-HE)
  • Geny 1.0.0 (update, used by OS-LIb)

Removed Dependencies

  • Log4j API 2.19.0 (removed)
  • Log4j API Scala 12.0 (removed)
  • Log4j core 2.14.1 (removed)