Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide Spark Expression Support for current support status.
StringType -> IntegerType; nullSafeEval returns codePointAt(0) of the first char, or 0 for the empty string. Wired via CometScalarFunction("ascii") and resolved to DataFusion ascii (chars().next() as i32); first-code-point semantics match for ASCII, BMP, and supplementary code points.inputTypes widened to StringTypeWithCollation(supportsTrimCollation = true); behaviour unchanged for UTF8_BINARY. Comet does not propagate collation, so non-default collations may diverge silently (#4496).(StringType|BinaryType) -> IntegerType; eval returns numBytes * 8 for strings and .length * 8 for binary.inputTypes widened to StringTypeWithCollation(supportsTrimCollation = true); semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).BinaryType input is reported Unsupported in getSupportLevel, so bit_length(<binary>) falls back to Spark cleanly (DataFusion's BitLengthFunc signature accepts string types only).StringTrimBoth is RuntimeReplaceable and rewritten to StringTrim(srcStr, trimStr) before serde runs. Support is provided by the trim entry; no dedicated serde registration.StringTrim (the rewrite target) routes through CollationSupport.StringTrim.exec and uses StringTypeNonCSAICollation(supportsTrimCollation = true); semantics unchanged for UTF8_BINARY. Non-default collations may diverge in Comet (#4496).Chr(LongType) -> StringType; lon < 0 returns "", else ((lon & 0xFF) as char).toString (so chr(256) and chr(0) both return \u0000).NullIntolerant trait replaced by override def nullIntolerant: Boolean = true. Resolves natively to datafusion_spark::function::string::char::CharFunc, which mirrors Spark's negative-input and & 0xFF semantics.Length. Same support as length.Length.Length. Same support as length.Length.Chr. Same support as char.Chr.Seq[Expression] -> StringType; NULL separator yields NULL, NULL element values are skipped, children can be StringType or ArrayType(StringType). Comet serde rewrites a NULL-literal separator to a NULL of the result type and bails out on all-foldable inputs so Spark's ConstantFolding handles them; otherwise delegates to DataFusion concat_ws.inputTypes widened to StringTypeWithCollation / AbstractArrayType; dataType becomes children.head.dataType (collation-derived). Semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).UTF8String.contains on StringType; the parser routes (BinaryType, BinaryType) to BinaryPredicate, so Comet only ever sees the String form.CollationSupport.Contains.exec(..., collationId); behaviour identical for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringDecode(bin, charset) evaluated directly; invalid sequences silently substitute replacement characters via new String(bytes, charset).RuntimeReplaceable whose replacement is a StaticInvoke(StringDecode.decode, bin, charset, legacyCharsets, legacyErrorAction); the 4-arg form raises on malformed input unless legacy flags are set.decode runs through the codegen dispatcher on all versions (Spark 3.x via CometStringDecode, Spark 4.0 via the StaticInvoke replacement routed to CometStaticInvokeCodegenDispatch), so Spark's own evaluation runs inside the Comet pipeline. This honours the charset argument and the Spark 4.0 legacyCharsets / legacyErrorAction flags, and falls back to Spark when the dispatcher is disabled.CometExpressionSerde[StringDecode] registration, decode does not surface in the auto-generated compatibility docs (#4466).UTF8String.endsWith on StringType; binary form routed to BinaryPredicate before Comet.CollationSupport.EndsWith.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).string.toLowerCase.toTitleCase on UTF8String; word boundary is Java Character.isWhitespace. DataFusion initcap splits on !is_alphanumeric() (hyphens, apostrophes, and punctuation all split words), so CometInitCap reports Compatible with a NativeOptIn and dispatches to Spark's generated code by default. The native path is used only when spark.comet.expression.InitCap.allowIncompatible=true.CollationSupport.InitCap.exec (collation- and ICU-aware) and propagates child.dataType. Comet ignores collation; 3.x divergences persist plus collation/ICU mismatches (#4496).StringInstr(str, substr) -> IntegerType; returns string.indexOf(sub, 0) + 1 (1-based, 0 when not found, 1 on empty substring). Resolves to DataFusion strpos (alias instr) with matching semantics.CollationSupport.StringInstr.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Lower. Same support as lower.Lower.RuntimeReplaceable with replacement = Substring(str, Literal(1), len); accepts StringType or BinaryType plus IntegerType. Comet serde serialises expr.replacement, routing through SparkSubstring — so non-literal len is supported.inputTypes widened with StringTypeWithCollation; behaviour unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Length. Same support as length.Length.(StringType|BinaryType) -> IntegerType; eval returns numChars for strings and .length for binary. BinaryType input falls back via Unsupported (DataFusion's character_length accepts string types only).inputTypes widened to StringTypeWithCollation(supportsTrimCollation = true); semantics unchanged. Non-default collations not honoured by Comet (#4496).toLowerCase on UTF8String. Comet routes through the JVM codegen dispatcher by default, so results are byte-exact. The native path (DataFusion lower, Rust Unicode default case mapping with no locale awareness) is opt-in via spark.comet.caseConversion.enabled=true.CollationSupport.Lower.exec(v, collationId, useICU) with SQLConf.ICU_CASE_MAPPINGS_ENABLED; inputTypes widened to StringTypeWithCollation. Comet ignores collation and ICU mode, so non-default collations or ICU_CASE_MAPPINGS_ENABLED=true diverge even after opting in (#2190).StringLPad(str, len, pad) -> StringType; len <= 0 returns the empty string, empty pad returns str unchanged, NULL inputs propagate. Comet serde requires str to be a column and pad to be a literal; otherwise falls back.NullIntolerant trait replaced by override def nullIntolerant: Boolean = true; inputTypes widened to StringTypeWithCollation(supportsTrimCollation = true). Semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).lpad(<binary>, ...) is rewritten by Spark to BinaryPad / StaticInvoke(ByteArray.lpad) before serde runs and always falls back to Spark.StringTrimLeft extends String2TrimExpression; no-arg form strips ASCII space 0x20 only. The two-arg parser form ltrim(trimStr, srcStr) is swapped to (srcStr, Option(trimStr)) by Spark's secondary constructor, so children match DataFusion ltrim(str, chars).CollationSupport.StringTrimLeft.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).(StringType|BinaryType) -> IntegerType; eval returns numBytes for strings and .length for binary.inputTypes widened to StringTypeWithCollation; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).BinaryType input is reported Unsupported in getSupportLevel, so octet_length(<binary>) falls back to Spark cleanly (DataFusion's OctetLengthFunc signature accepts string types only).RegExpReplace(subject, regexp, rep, pos) with foldable pos > 0; uses Java Pattern. Comet supports only pos = 1 (other offsets fall back) and injects a 'g' flag because DataFusion's regexp_replace stops at the first match by default.nullIntolerant: Boolean = true; runtime semantics unchanged.regex crate vs Java Pattern). CometRegExpReplace therefore routes through the JVM codegen dispatcher by default; the native path is opt-in via spark.comet.expression.RegExpReplace.allowIncompatible=true and only applies when pos is the literal 1.StringRepeat(str, times) with nullSafeEval(s, n) = s.repeat(n); UTF8String.repeat returns the empty string for n <= 0. Comet casts times to LongType and delegates to DataFusion repeat, which mirrors Spark for negative counts.nullIntolerant: Boolean field; dataType becomes str.dataType (collation-tracking). Semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringReplace(src, search, replace); when search is empty, Spark returns src unchanged (short-circuit on search.numBytes == 0). DataFusion replace instead inserts replace between every character, so CometStringReplace reports Compatible with a NativeOptIn and runs Spark's own generated code inside the Comet pipeline by default. The native DataFusion replace is used only when spark.comet.expression.StringReplace.allowIncompatible=true.CollationSupport.StringReplace.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).RuntimeReplaceable with replacement = If(IsNull(str), null, If(len <= 0, "", Substring(str, -len, len))); accepts StringType plus IntegerType. Comet serde serialises expr.replacement, so NULL propagation for len <= 0 and non-literal len are handled by the replacement tree itself.inputTypes widened with collation; uses UnaryMinus(len, failOnError = false) to avoid integer-overflow exceptions on len = Int.MinValue. Semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringRPad(str, len, pad) -> StringType; same edge-case behaviour as lpad (negative len, empty pad, NULL propagation). Comet serde requires column str and literal pad.lpad; default-pad literal type tightened; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).BinaryPad / StaticInvoke rewrite as lpad causes rpad(<binary>, ...) to fall back.StringTrimRight extends String2TrimExpression; semantically symmetric to ltrim.CollationSupport.StringTrimRight.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringSpace(IntegerType) -> StringType; negative input yields the empty string. Resolves natively to datafusion_spark::function::string::space::SparkSpace.NullIntolerant trait replaced by nullIntolerant: Boolean override.StringSplit(str, regex, limit); limit > 0 permits at most limit-1 splits, limit <= 0 is unlimited. Comet registers split as a custom UDF (native/spark-expr/src/string_funcs/split.rs) using the Rust regex crate, and is unconditionally Incompatible due to regex-engine differences.CollationSupport.collationAwareRegex and changes dataType to ArrayType(str.dataType, ...). Comet does not honour collation flags (#4496).UTF8String.startsWith on StringType; binary form routed to BinaryPredicate.CollationSupport.StartsWith.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Substring. Same support as substring.Substring.TernaryExpression; two-arg form defaults len = Integer.MAX_VALUE; supports StringType and BinaryType. Comet routes through the datafusion-spark SparkSubstring UDF (registered as substring), which accepts non-literal pos/len and Utf8/Utf8View/Binary/LargeBinary/BinaryView inputs.inputTypes widened with StringTypeWithCollation; semantics unchanged for UTF8_BINARY. SparkSubstring implements Spark's 1-indexed, negative-start, and negative-length clamping. Non-default collations not honoured by Comet (#4496).TernaryExpression(StringType, StringType, IntegerType) -> StringType. Comet casts count to LongType and delegates to DataFusion's substr_index UDF (alias substring_index).CollationSupport.SubstringIndex.exec and propagates strExpr.dataType; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).StringTranslate(src, from, to); UTF8String.translate(dict) is code-point based, and any character mapped explicitly to U+0000 in to is also deleted.CollationSupport.StringTranslate.exec; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Incompatible: DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel. It falls back to Spark by default and runs natively only when incompatible expressions are explicitly allowed.StringTrim no-arg form strips ASCII space 0x20 only (matches DataFusion btrim‘s default); two-arg form’s children are (srcStr, trimStr) after Spark's secondary-constructor swap.CollationSupport.StringTrim.exec and uses StringTypeNonCSAICollation; semantics unchanged for UTF8_BINARY. Non-default collations not honoured by Comet (#4496).Upper. Same support as upper.Upper.toUpperCase on UTF8String. Comet routes through the JVM codegen dispatcher by default, so results are byte-exact. The native path (DataFusion upper, Rust Unicode default case mapping with no locale awareness) is opt-in via spark.comet.caseConversion.enabled=true.CollationSupport.Upper.exec(v, collationId, useICU) with SQLConf.ICU_CASE_MAPPINGS_ENABLED. Comet does not propagate collation or ICU mode; non-default collations or ICU_CASE_MAPPINGS_ENABLED=true diverge even after opting in (#2190).