This document describes the limitations of velox backend by listing some known cases where exception will be thrown, gluten behaves incompatibly with spark, or certain plan's execution must fall back to vanilla spark, etc.
Gluten avoids to modify Spark‘s existing code and use Spark APIs if possible. However, some APIs aren’t exposed in Vanilla spark and we have to copy the Spark file and do the hardcode changes. The list of override classes can be found as ignoreClasses in package/pom.xml . If you use customized Spark, you may check if the files are modified in your spark, otherwise your changes will be overrided.
So you need to ensure preferentially load the Gluten jar to overwrite the jar of vanilla spark. Refer to How to prioritize loading Gluten jars in Spark.
If not officially supported spark3.2/3.3 version is used, NoSuchMethodError can be thrown at runtime. More details see issue-4514.
Except the unsupported operators, functions, file formats, data sources listed in , there are some known cases also fall back to Vanilla Spark.
Gluten currently doesn‘t support ANSI mode. If ANSI is enabled, Spark plan’s execution will always fall back to vanilla Spark.
Velox BloomFilter‘s serialization format is different from Spark’s. BloomFilter binary generated by Velox can't be deserialized by vanilla spark. So if might_contain falls back, we fall back bloom_filter_agg to vanilla spark also.
Gluten only supports spark default case-insensitive mode. If case-sensitive mode is enabled, user may get incorrect result.
In Velox, regexp functions (rlike, regexp_extract, etc.) are implemented based on RE2, while in Spark they are based on java.util.regex.
java.util.regex does.There are a few unknown incompatible cases. If user cannot tolerate the incompatibility risk, please enable the below configuration property.
spark.gluten.sql.fallbackRegexpExpressions
Currently, Gluten only fully supports parquet file format and partially support ORC. If other format is used, scan operator falls back to vanilla spark.
Gluten only support the partitioned table scan when the file path contain the partition info, otherwise will fall back to vanilla spark.
In certain cases, Gluten result may be different from Vanilla spark.
Velox only supports double quotes surrounded strings, not single quotes, in JSON data. If single quotes are used, gluten will produce incorrect result.
Velox doesn't support [*] in path when get_json_object function is called and returns null instead.
Gluten supports spark.files.ignoreCorruptFiles with default false, if true, the behavior is same as config false. Gluten ignores spark.sql.parquet.datetimeRebaseModeInRead, it only returns what write in parquet file. It does not consider the difference between legacy hybrid (Julian Gregorian) calendar and Proleptic Gregorian calendar. The result may be different with vanilla spark.
Spark has spark.sql.parquet.datetimeRebaseModeInWrite config to decide whether legacy hybrid (Julian + Gregorian) calendar or Proleptic Gregorian calendar should be used during parquet writing for dates/timestamps. If the parquet to read is written by Spark with this config as true, Velox's TableScan will output different result when reading it back.
Gluten only supports static partition writes and does not support dynamic partition writes.
spark.sql("CREATE TABLE t (c int, d long, e long) STORED AS PARQUET partitioned by (c, d)") spark.sql("INSERT OVERWRITE TABLE t partition(c=1, d=2) SELECT 3 as e")
Gluten does not support dynamic partition write and bucket write, Exception may be raised if you use. e.g.,
spark.range(100).selectExpr("id as c1", "id % 7 as p") .write .format("parquet") .partitionBy("p") .save(f.getCanonicalPath)
Gluten supports static partition writes and dynamic partition writes.
spark.sql("CREATE TABLE t (c int, d long, e long) STORED AS PARQUET partitioned by (c, d)") spark.sql("INSERT OVERWRITE TABLE t partition(c=1, d) SELECT 2 as d, 3 as e")
Gluten does not support bucket write, and will fall back to vanilla Spark.
spark.range(100).selectExpr("id as c1", "id % 7 as p") .write .format("parquet") .bucketBy(2, "c1") .save(f.getCanonicalPath)
Gluten does not create table as select. It may raise exception. e.g.,
spark.range(100).toDF("id") .write .format("parquet") .saveAsTable("velox_ctas")
Gluten supports create table as select with parquet file format.
spark.range(100).toDF("id") .write .format("parquet") .saveAsTable("velox_ctas")
Gluten supports writes of HiveFileFormat when the output file type is of type parquet only
Velox does NOT support NaN. So unexpected result can be obtained for a few cases, e.g., comparing a number with NaN.
Parquet write only support three configs, other will not take effect.
spark.sql.parquet.compression.codeccompression.codecspark.gluten.sql.columnar.parquet.write.blockSizeparquet.block.sizespark.gluten.sql.native.parquet.write.blockRowsparquet.block.rowsIf the user enables Spark‘s columnar reading, error can occur due to Spark’s columnar vector is not compatible with Gluten's.
OutOfMemoryException may still be triggered within current implementation of spill-to-disk feature, when shuffle partitions is set to a large number. When this case happens, please try to reduce the partition number to get rid of the OOM.
Byte type causes fallback to vanilla spark
Timestamp type
Only reading with INT96 and dictionary encoding is supported. When reading INT64 represented millisecond/microsecond timestamps, or INT96 represented timestamps of other encodings, exceptions can occur.
Complex types
The header option should be true. And now we only support DatasourceV1, i.e., user should set spark.sql.sources.useV1SourceList=csv. User defined read option is not supported, which will make CSV read fall back to vanilla Spark in most case. CSV read will also fall back to vanilla Spark and log warning when user specifies schema is different with file schema.
Spark uses the spark.sql.legacy.allowHashOnMapType configuration to support hash map key functions. Gluten enables this configuration during the creation of ColumnarShuffleExchange, as shown in the code link. This method bypasses Spark's unresolved checks and creates projects with the hash(mapType) operator before ColumnarShuffleExchange. However, if spark.sql.legacy.allowHashOnMapType is disabled in a test environment, projects using the hash(mapType) expression may throw an Invalid call to dataType on unresolved object exception during validation, causing them to fallback to vanilla Spark, as referenced in the code link. Enabling this configuration allows the project to be offloaded to Velox.