Audit notes for expressions in this category that have been audited. Absence of an entry means the expression has not been audited yet, not that it is unsupported. See the user guide Spark Expression Support for current support status.
%Remainder(left, right, evalMode) signature identical across versions. Native path uses Rust spark_modulo UDF; non-ANSI returns NULL on divide-by-zero, ANSI raises DIVIDE_BY_ZERO / REMAINDER_BY_ZERO. CometRemainder gates only on the left input's data type, so all three eval modes, including the EvalMode.TRY form used by try_mod (Spark 4.0+), are serialized natively.*Multiply(left, right, evalMode) signature identical. Decimal results exceeding DECIMAL128_MAX_PRECISION go through WideDecimalBinaryExpr (Decimal256 intermediate); smaller decimals and primitives use DataFusion BinaryExpr. ANSI integer overflow uses Rust checked_mul. Interval multiplication falls back.+Add(left, right, evalMode) with the same Decimal / ANSI plumbing as *. Date + Int8/16/32 dispatches to the Rust date_add UDF to work around DataFusion's Date32 + Interval-only kernel.-Subtract(left, right, evalMode) mirrors +. Date - Int8/16/32 uses the Rust date_sub UDF./Divide(left, right, evalMode). Non-ANSI mode wraps the divisor in If(EqualTo(right, 0), null, right) so DataFusion never throws. Decimal output is wrapped in CheckOverflow(failOnError = ANSI); ANSI surfaces NUMERIC_VALUE_OUT_OF_RANGE, non-ANSI returns NULL.Internal decimal wrapper emitted around every decimal + - * /, sum, and avg result to null out (non-ANSI) or raise on (ANSI) values that exceed the declared precision. Native impl: math_funcs/internal/checkoverflow.rs.
is_valid_decimal_precision, a small inlined bounds check scanned with all (short-circuits at the first overflow). When nothing overflows (the common shape) the input buffers are reused via to_data() (cheap Arc metadata clone) instead of allocating through null_if_overflow_precision (non-ANSI) or running the heavier per-value validate_decimal_precision (ANSI). The ANSI path only falls back to validate_decimal_precision when an overflow is present, to build the precise Spark error. ~10% faster on the no-overflow shape, ~17% with nulls, and ~69% faster for ANSI no-overflow (down to parity with non-ANSI); overflow shapes unchanged. Benchmark: benches/check_overflow.rs.Abs(child, failOnError) over NumericType plus the two interval types. failOnError (ANSI) is propagated to the native abs UDF, which throws ARITHMETIC_OVERFLOW on Int.MinValue / Long.MinValue / Decimal MIN. DayTimeIntervalType and YearMonthIntervalType fall back to Spark. Spark 4.0 / 4.1 do the NullIntolerant -> nullIntolerant: Boolean refactor; behaviour unchanged.UnaryMathExpression(math.acos, "ACOS") unchanged across versions; wired as CometScalarFunction("acos") to DataFusion's acos UDF. NaN for |x| > 1.StrictMath.log(x + sqrt(x*x - 1)) unchanged across versions. NaN for x < 1. Routes to DataFusion's acosh.UnaryMathExpression(math.asin, "ASIN") unchanged. NaN for |x| > 1.Double.NegativeInfinity to avoid log(NaN), otherwise StrictMath.log(x + sqrt(x*x + 1)). Identical across versions.UnaryMathExpression(math.atan, "ATAN") unchanged.BinaryMathExpression(math.atan2, "ATAN2") with both inputs adjusted by +0.0 to flip -0.0 to +0.0. CometAtan2 reproduces this by wrapping each child in Add(child, Literal.default(child.dataType)) before dispatching to DataFusion's atan2.0.5 * (log1p(x) - log1p(-x)) (SPARK-28519). NaN for |x| > 1, +/-Infinity for x = +/-1.Bin(child) over LongType -> StringType. Spark 4.x gains DefaultStringProducingExpression and the nullIntolerant: Boolean refactor; no behaviour change. Routes to datafusion-spark SparkBin.cbrt.ceil(expr) supported (LongType / DoubleType / DecimalType with scale >= 0). Decimal with negative scale falls back at convert time. The two-arg ceil(expr, scale) form (RoundCeil) is not wired and falls back to Spark.Ceil. Same support as ceil.UnaryMathExpression(math.cos, "COS") unchanged across versions.UnaryMathExpression(math.cosh, "COSH") unchanged.1 / math.tan(x). DataFusion's cot is also 1.0 / tan(x), so the result matches.1 / math.sin(x). Routed to datafusion-spark's SparkCsc (registered in jni_api.rs).UnaryMathExpression(math.toDegrees, "DEGREES") unchanged across versions.IntegralDivide(left, right, evalMode). Non-decimal operands are cast to DecimalType(19, 0); result is recomputed per IntegralDivide.resultDecimalType, wrapped in CheckOverflow, then cast to Long. ANSI overflow for Long.MinValue div -1 and decimal-overflow ANSI cases are covered by existing tests.Internal fused expression that rescales a Decimal128 value (changing scale) and checks output precision in one pass, replacing the CheckOverflow(Cast(expr, Decimal128(p, s))) pattern used by decimal-to-decimal casts. Native impl: math_funcs/internal/decimal_rescale_check.rs.
null_if_overflow_precision (a second full pass that allocates a new array) on every batch to turn overflow sentinels into nulls, even when nothing overflowed. Now that pass runs only when a sentinel is present (contains(&i128::MAX), short-circuiting), so the common no-overflow case skips the allocation. 8 to 26% faster on no-overflow shapes; overflow and ANSI shapes unchanged. Benchmark: benches/decimal_rescale.rs.pi).UnaryMathExpression(StrictMath.exp, "EXP") unchanged. ULP-level differences vs DataFusion exp are possible but unflagged.UnaryMathExpression(StrictMath.expm1, "EXPM1") unchanged.extends UnaryExpression with ImplicitCastInputTypes with NullIntolerant. Returns NULL for NULL input or values outside [0, 20].NullIntolerant trait replaced by nullIntolerant: Boolean method override; behavior unchanged.ceil. Two-arg floor(expr, scale) form (RoundFloor) falls back to Spark.CometScalarFunction("greatest") to DataFusion's GreatestFunc. Comet does not gate input types, so interval inputs and other Spark-only orderings rely on the native UDF accepting them; no explicit fallback path.LongType / BinaryType / StringType. Spark 4.x widens StringType to StringTypeWithCollation and preserves collation in dataType; CometHex passes expr.dataType to native SparkHex, which always returns Utf8 -- collation propagation may diverge on Spark 4.x.greatest; same caveats. Spark 4.1.1 adds contextIndependentFoldable (no Comet impact).Log. Comet wires through CometLog to DataFusion ln with a nullIfNegative rewrite to match Spark's NULL behaviour for x <= 0.log(x) -> CometLog (DataFusion ln); two-arg log(base, x) -> CometLogarithm (custom spark_log UDF, returns NULL when base <= 0 or x <= 0 to match Logarithm.nullSafeEval).UnaryLogExpression(StrictMath.log10, "LOG10"); returns NULL for x <= 0. Possible ULP differences from StrictMath.UnaryLogExpression(StrictMath.log(x) / StrictMath.log(2), "LOG2"); returns NULL for x <= 0.Remainder. Same support as %.UnaryMinus(child, failOnError) -> Rust NegativeExpr. ANSI overflow is detected for Int8/Int16/Int32/Int64 and IntervalYearMonth/IntervalDayTime. Float / Double / Decimal cannot overflow on negate. Spark 4.0 NullIntolerant -> nullIntolerant: Boolean refactor; no impact.LeafMathExpression(math.Pi, "PI"); foldable, so Spark ConstantFolding rewrites it to a Literal before Comet sees the plan. The CometScalarFunction("pi") registration is exercised only when ConstantFolding is excluded.UnaryPositive(child) is a regular expression. There is no Comet serde for UnaryPositive, so projections containing +col silently disable Comet for the projection on 3.4/3.5.UnaryPositive is RuntimeReplaceable with replacement = child; the optimizer removes it before Comet sees the plan, so the gap is transparent on 4.x.Pow(left, right) extends BinaryMathExpression(StrictMath.pow, "POWER"); routes to DataFusion pow. ULP-level differences possible.Pow. Same support as pow.UnaryMathExpression(math.toRadians, "RADIANS") unchanged across versions.misc_funcs / rand.misc_funcs / randn.UnaryMathExpression(math.rint, "ROUND") with funcName = "rint". Passthrough to DataFusion rint (round-half-to-even).BigDecimal-via-toString rounding cannot be precisely matched (documented inline in CometRound). ANSI failOnError is propagated for integer overflow. BRound (HALF_EVEN) is not wired.1 / math.cos(x). Routed to datafusion-spark's SparkSec.bitwise_funcs / << (audited in PR #4479). Same support as the operator alias added in 4.0.Signum. Same support as signum.Signum(child) over DoubleType. Spark also restricts to the two interval types via inputTypes; Comet handles only the Double case via DataFusion signum.UnaryMathExpression(math.sin, "SIN") unchanged.UnaryMathExpression(math.sinh, "SINH") unchanged.UnaryMathExpression(math.sqrt, "SQRT") unchanged.UnaryMathExpression(math.tan, "TAN") unchanged.UnaryMathExpression(math.tanh, "TANH") unchanged.TryAdd is RuntimeReplaceable and rewrites to Add(.., EvalMode.TRY) for numeric inputs (datetime / interval go through TryEval(Add(.., ANSI)) and fall back). Numeric path uses the Rust checked_add UDF, returning NULL on overflow. Decimal goes through WideDecimalBinaryExpr with EvalMode.Try.TryDivide rewrites to Divide(.., EvalMode.TRY). The nullIfWhenPrimitive wrapper swaps zero divisors to NULL; integer / float divide uses checked_div; decimal uses decimal_div + CheckOverflow(failOnError = false) returning NULL.Multiply(.., EvalMode.TRY). Integer path uses checked_mul; decimal uses WideDecimalBinaryExpr with EvalMode.Try, returning NULL on overflow.Subtract(.., EvalMode.TRY). Integer path uses checked_sub; decimal uses WideDecimalBinaryExpr as needed.Unhex(child, failOnError). Spark 4.x widens input to StringTypeWithCollation and wraps the inner call in try/catch; Comet CometUnhex forwards failOnError to native spark_unhex but does not gate on collation.BinaryBuilder, cutting per-byte branching and repeated buffer reallocations. Up to 31% faster on long strings. Benchmark: benches/unhex.rs.NullIntolerant -> nullIntolerant: Boolean refactor.CometExprShim rather than a CometExpressionSerde, so it bypasses the support-level framework and the auto-generated compatibility doc (#4485). Native path uses datafusion-spark SparkWidthBucket; interval input types are not exercised by Comet tests.