Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide Spark Expression Support for current support status.
StringType -> IntegerType; nullSafeEval returns codePointAt(0) of the first char, or 0 for the empty string. Wired via CometScalarFunction("ascii") and resolved to DataFusion ascii (chars().next() as i32); first-code-point semantics match for ASCII, BMP, and supplementary code points.inputTypes widened to StringTypeWithCollation(supportsTrimCollation = true); behaviour unchanged for UTF8_BINARY. Comet does not propagate collation, so non-default collations may diverge silently (#4496).(StringType|BinaryType) -> IntegerType; eval returns numBytes * 8 for strings and .length * 8 for binary.inputTypes widened to StringTypeWithCollation(supportsTrimCollation = true); semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).BinaryType input is reported Unsupported in getSupportLevel, so bit_length(<binary>) falls back to Spark cleanly (DataFusion's BitLengthFunc signature accepts string types only).StringTrimBoth is RuntimeReplaceable and rewritten to StringTrim(srcStr, trimStr) before serde runs. Support is provided by the trim entry; no dedicated serde registration.StringTrim (the rewrite target) routes through CollationSupport.StringTrim.exec and uses StringTypeNonCSAICollation(supportsTrimCollation = true); semantics unchanged for UTF8_BINARY. Non-default collations may diverge in Comet (#4496).Chr(LongType) -> StringType; lon < 0 returns "", else ((lon & 0xFF) as char).toString (so chr(256) and chr(0) both return \u0000).NullIntolerant trait replaced by override def nullIntolerant: Boolean = true. Resolves natively to datafusion_spark::function::string::char::CharFunc, which mirrors Spark's negative-input and & 0xFF semantics.Length. Same support as length.Length.Length. Same support as length.Length.Chr. Same support as char.Chr.SparkConcatWs for arrays and separator-only inputs, with native column coverage for multiple arrays, empty arrays, nulls, empty strings, Unicode, and varying separators. The existing all-foldable codegen fallback remains. A native adapter evaluates runtime scalars once and returns a scalar for broadcasting, including non-foldable scalar subqueries.UTF8String.concatWs without collation-dependent comparisons.UTF8String.contains on StringType; the parser routes (BinaryType, BinaryType) to BinaryPredicate, so Comet only ever sees the String form.CollationSupport.Contains.exec(..., collationId); behaviour identical for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringDecode(bin, charset) evaluated directly; invalid sequences silently substitute replacement characters via new String(bytes, charset).RuntimeReplaceable whose replacement is a StaticInvoke(StringDecode.decode, bin, charset, legacyCharsets, legacyErrorAction); the 4-arg form raises on malformed input unless legacy flags are set.decode runs through the codegen dispatcher on all versions (Spark 3.x via CometStringDecode, Spark 4.0 via the StaticInvoke replacement routed to CometStaticInvokeCodegenDispatch), so Spark's own evaluation runs inside the Comet pipeline. This honours the charset argument and the Spark 4.0 legacyCharsets / legacyErrorAction flags, and falls back to Spark when the dispatcher is disabled.CometExpressionSerde[StringDecode] registration, decode does not surface in the auto-generated compatibility docs (#4466).UTF8String.endsWith on StringType; binary form routed to BinaryPredicate before Comet.CollationSupport.EndsWith.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).string.toLowerCase.toTitleCase on UTF8String; word boundary is Java Character.isWhitespace. DataFusion initcap splits on !is_alphanumeric() (hyphens, apostrophes, and punctuation all split words), so CometInitCap reports Compatible with a NativeOptIn and dispatches to Spark's generated code by default. The native path is used only when spark.comet.expression.InitCap.allowIncompatible=true.CollationSupport.InitCap.exec (collation- and ICU-aware) and propagates child.dataType. Comet ignores collation; 3.x divergences persist plus collation/ICU mismatches (#4496).StringInstr(str, substr) -> IntegerType; returns string.indexOf(sub, 0) + 1 (1-based, 0 when not found, 1 on empty substring). Resolves to DataFusion strpos (alias instr) with matching semantics.CollationSupport.StringInstr.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Lower. Same support as lower.Lower.RuntimeReplaceable with replacement = Substring(str, Literal(1), len); accepts StringType or BinaryType plus IntegerType. Comet serde serialises expr.replacement, routing through SparkSubstring — so non-literal len is supported.inputTypes widened with StringTypeWithCollation; behaviour unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Length. Same support as length.Length.(StringType|BinaryType) -> IntegerType; eval returns numChars for strings and .length for binary. BinaryType input falls back via Unsupported (DataFusion's character_length accepts string types only).inputTypes widened to StringTypeWithCollation(supportsTrimCollation = true); semantics unchanged. Non-default collations not honoured by Comet (#4496).toLowerCase on UTF8String. Comet routes through the JVM codegen dispatcher by default, so results are byte-exact. The native path (DataFusion lower, Rust Unicode default case mapping with no locale awareness) is opt-in via spark.comet.caseConversion.enabled=true.CollationSupport.Lower.exec(v, collationId, useICU) with SQLConf.ICU_CASE_MAPPINGS_ENABLED; inputTypes widened to StringTypeWithCollation. Comet ignores collation and ICU mode, so non-default collations or ICU_CASE_MAPPINGS_ENABLED=true diverge even after opting in (#2190).StringLPad(str, len, pad) -> StringType; len <= 0 returns the empty string, empty pad returns str unchanged, NULL inputs propagate. Comet serde requires str to be a column and pad to be a literal; otherwise falls back.NullIntolerant trait replaced by override def nullIntolerant: Boolean = true; inputTypes widened to StringTypeWithCollation(supportsTrimCollation = true). Semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).lpad(<binary>, ...) is rewritten by Spark to BinaryPad / StaticInvoke(ByteArray.lpad) before serde runs. There is no native path, so CometStaticInvoke routes it through the JVM codegen dispatcher and the projection stays in the Comet pipeline; it falls back to Spark only when the dispatcher is disabled.StringTrimLeft extends String2TrimExpression; no-arg form strips ASCII space 0x20 only. The two-arg parser form ltrim(trimStr, srcStr) is swapped to (srcStr, Option(trimStr)) by Spark's secondary constructor, so children match DataFusion ltrim(str, chars).CollationSupport.StringTrimLeft.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).(StringType|BinaryType) -> IntegerType; eval returns numBytes for strings and .length for binary.inputTypes widened to StringTypeWithCollation; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).BinaryType input is reported Unsupported in getSupportLevel, so octet_length(<binary>) falls back to Spark cleanly (DataFusion's OctetLengthFunc signature accepts string types only).RegExpReplace(subject, regexp, rep, pos) with foldable pos > 0; uses Java Pattern. Comet supports only pos = 1 (other offsets fall back) and injects a 'g' flag because DataFusion's regexp_replace stops at the first match by default.nullIntolerant: Boolean = true; runtime semantics unchanged.regex crate vs Java Pattern). CometRegExpReplace therefore routes through the JVM codegen dispatcher by default; the native path is opt-in via spark.comet.expression.RegExpReplace.allowIncompatible=true and only applies when pos is the literal 1.StringRepeat(str, times) with nullSafeEval(s, n) = s.repeat(n); UTF8String.repeat returns the empty string for n <= 0. Comet casts times to LongType and delegates to DataFusion repeat, which mirrors Spark for negative counts.nullIntolerant: Boolean field; dataType becomes str.dataType (collation-tracking). Semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringReplace(src, search, replace); when search is empty, Spark returns src unchanged (short-circuit on search.numBytes == 0). DataFusion replace instead inserts replace between every character, so CometStringReplace reports Compatible with a NativeOptIn and runs Spark's own generated code inside the Comet pipeline by default. The native DataFusion replace is used only when spark.comet.expression.StringReplace.allowIncompatible=true.CollationSupport.StringReplace.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).RuntimeReplaceable with replacement = If(IsNull(str), null, If(len <= 0, "", Substring(str, -len, len))); accepts StringType plus IntegerType. Comet serde serialises expr.replacement, so NULL propagation for len <= 0 and non-literal len are handled by the replacement tree itself.inputTypes widened with collation; uses UnaryMinus(len, failOnError = false) to avoid integer-overflow exceptions on len = Int.MinValue. Semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringRPad(str, len, pad) -> StringType; same edge-case behaviour as lpad (negative len, empty pad, NULL propagation). Comet serde requires column str and literal pad.lpad; default-pad literal type tightened; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).BinaryPad / StaticInvoke rewrite as lpad, so rpad(<binary>, ...) also runs through the codegen dispatcher rather than falling back.StringTrimRight extends String2TrimExpression; semantically symmetric to ltrim.CollationSupport.StringTrimRight.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringSpace(IntegerType) -> StringType; negative input yields the empty string. Resolves natively to datafusion_spark::function::string::space::SparkSpace.NullIntolerant trait replaced by nullIntolerant: Boolean override.StringSplit(str, regex, limit); limit > 0 permits at most limit-1 splits, limit <= 0 is unlimited. Comet registers split as a custom UDF (native/spark-expr/src/string_funcs/split.rs) using the Rust regex crate, and is unconditionally Incompatible due to regex-engine differences.CollationSupport.collationAwareRegex and changes dataType to ArrayType(str.dataType, ...). Comet does not honour collation flags (#4496).UTF8String.startsWith on StringType; binary form routed to BinaryPredicate.CollationSupport.StartsWith.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Substring. Same support as substring.Substring.TernaryExpression; two-arg form defaults len = Integer.MAX_VALUE; supports StringType and BinaryType. Comet routes through the datafusion-spark SparkSubstring UDF (registered as substring), which accepts non-literal pos/len and Utf8/Utf8View/Binary/LargeBinary/BinaryView inputs.inputTypes widened with StringTypeWithCollation; semantics unchanged for UTF8_BINARY. SparkSubstring implements Spark's 1-indexed, negative-start, and negative-length clamping. Non-default collations not honoured by Comet (#4496).TernaryExpression(StringType, StringType, IntegerType) -> StringType. Comet casts count to LongType and delegates to DataFusion's substr_index UDF (alias substring_index).CollationSupport.SubstringIndex.exec and propagates strExpr.dataType; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringTranslate(src, from, to); UTF8String.translate(dict) is code-point based, and any character mapped explicitly to U+0000 in to is also deleted.CollationSupport.StringTranslate.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Incompatible: DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel. It falls back to Spark by default and runs natively only when incompatible expressions are explicitly allowed.StringTrim no-arg form strips ASCII space 0x20 only (matches DataFusion btrim‘s default); two-arg form’s children are (srcStr, trimStr) after Spark's secondary-constructor swap.CollationSupport.StringTrim.exec and uses StringTypeNonCSAICollation; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Upper. Same support as upper.Upper.doGenCode emits java.util.Base64.getMimeDecoder().decode(child.toString()). The MIME decoder skips every byte outside the base64 alphabet, so CRLF-wrapped output from Spark's own base64 round-trips cleanly; four terminal-shape errors surface as IllegalArgumentException. failOnError = true is set only when the node is constructed from to_binary('base64') / try_to_binary, which use a stricter RFC 4648 validator.NullIntolerant becomes override def nullIntolerant: Boolean = true; inputTypes widens to StringTypeWithCollation(supportsTrimCollation = true). Behaviour is byte-level and collation-independent, so no divergence for UTF8_BINARY and no shim is needed.contextIndependentFoldable; no behavioural change on the decode path.spark_unbase64, native/spark-expr/src/string_funcs/unbase64.rs) ports the JDK MIME decoder rules: 256-entry decode LUT, a reused per-batch scratch Vec<u8> copied into a preallocated BinaryBuilder, all four error messages reproduced verbatim. CometUnBase64 handles failOnError = false natively; failOnError = true (reachable from to_binary('base64') / try_to_binary) requires strict RFC 4648 validation and is not yet implemented natively, so those cases stay on the JVM codegen dispatcher via CodegenDispatchFallback.toUpperCase on UTF8String. Comet routes through the JVM codegen dispatcher by default, so results are byte-exact. The native path (DataFusion upper, Rust Unicode default case mapping with no locale awareness) is opt-in via spark.comet.caseConversion.enabled=true.CollationSupport.Upper.exec(v, collationId, useICU) with SQLConf.ICU_CASE_MAPPINGS_ENABLED. Comet does not propagate collation or ICU mode; non-default collations or ICU_CASE_MAPPINGS_ENABLED=true diverge even after opting in (#2190).