This repo contains the code that integrate Hudi with Spark. The repo is split into the following modules
hudi-spark
hudi-spark3.3.x
hudi-spark3.4.x
hudi-spark3.5.x
hudi-spark3-common
hudi-spark-common
HoodieSpark3_2ExtendedSqlAstBuilder
have comments in the spark3.2's code fork from org.apache.spark.sql.catalyst.parser.AstBuilder
, and additional withTimeTravel
method.SqlBase.g4
have comments in the code forked from spark3.2's parser, and add SparkSQL Syntax TIMESTAMP AS OF
and VERSION AS OF
.version | support |
---|---|
2.4.x | No |
3.0.x | No |
3.1.2 | No |
3.2.0 | Yes |
Spark3.3 support time travel syntax link SPARK-37219. Once Spark 3.3 released. The files in the following list will be removed:
HoodieSpark3_3ExtendedSqlAstBuilder.scala
, HoodieSpark3_3ExtendedSqlParser.scala
, TimeTravelRelation.scala
, SqlBase.g4
, HoodieSqlBase.g4
Tracking Jira: HUDI-4468Some other improvements undergoing: