DataFusion Comet 0.1.0 Changelog

This release consists of 343 commits from 41 contributors. See credits at the end of this changelog for more information.

Implemented enhancements:

  • feat: Add native shuffle and columnar shuffle #30 (viirya)
  • feat: Support Emit::First for SumDecimalGroupsAccumulator #47 (viirya)
  • feat: Nested map support for columnar shuffle #51 (viirya)
  • feat: Support Count(Distinct) and similar aggregation functions #42 (huaxingao)
  • feat: Upgrade to jni-rs 0.21 #50 (sunchao)
  • feat: Handle exception thrown from native side #61 (sunchao)
  • feat: Support InSet expression in Comet #59 (viirya)
  • feat: Add CometNativeException for exceptions thrown from the native side #62 (sunchao)
  • feat: Add cause to native exception #63 (viirya)
  • feat: Pull based native execution #69 (viirya)
  • feat: Add executeColumnarCollectIterator to CometExec to collect Comet operator result #71 (viirya)
  • feat: Add CometBroadcastExchangeExec to support broadcasting the result of Comet native operator #80 (viirya)
  • feat: Reduce memory consumption when writing sorted shuffle files #82 (sunchao)
  • feat: Add struct/map as unsupported map key/value for columnar shuffle #84 (viirya)
  • feat: Support multiple input sources for CometNativeExec #87 (viirya)
  • feat: Date and timestamp trunc with format array #94 (parthchandra)
  • feat: Support First/Last aggregate functions #97 (huaxingao)
  • feat: Add support of TakeOrderedAndProjectExec in Comet #88 (viirya)
  • feat: Support Binary in shuffle writer #106 (advancedxy)
  • feat: Add license header by spotless:apply automatically #110 (advancedxy)
  • feat: Add dictionary binary to shuffle writer #111 (viirya)
  • feat: Minimize number of connections used by parallel reader #126 (parthchandra)
  • feat: Support CollectLimit operator #100 (advancedxy)
  • feat: Enable min/max for boolean type #165 (huaxingao)
  • feat: Introduce CometTaskMemoryManager and native side memory pool #83 (sunchao)
  • feat: Fix old style names #201 (comphead)
  • feat: enable comet shuffle manager for comet shell #204 (zuston)
  • feat: Support bitwise aggregate functions #197 (huaxingao)
  • feat: Support BloomFilterMightContain expr #179 (advancedxy)
  • feat: Support sort merge join #178 (viirya)
  • feat: Support HashJoin operator #194 (viirya)
  • feat: Remove use of nightly int_roundings feature #228 (psvri)
  • feat: Support Broadcast HashJoin #211 (viirya)
  • feat: Enable Comet broadcast by default #213 (viirya)
  • feat: Add CometRowToColumnar operator #206 (advancedxy)
  • feat: Document the class path / classloader issue with the shuffle manager #256 (holdenk)
  • feat: Port Datafusion Covariance to Comet #234 (huaxingao)
  • feat: Add manual test to calculate spark builtin functions coverage #263 (comphead)
  • feat: Support ANSI mode in CAST from String to Bool #290 (andygrove)
  • feat: Add extended explain info to Comet plan #255 (parthchandra)
  • feat: Improve CometSortMergeJoin statistics #304 (planga82)
  • feat: Add compatibility guide #316 (andygrove)
  • feat: Improve CometHashJoin statistics #309 (planga82)
  • feat: Support Variance #297 (huaxingao)
  • feat: Support murmur3_hash and sha2 family hash functions #226 (advancedxy)
  • feat: Disable cast string to timestamp by default #337 (andygrove)
  • feat: Improve CometBroadcastHashJoin statistics #339 (planga82)
  • feat: Implement Spark-compatible CAST from string to integral types #307 (andygrove)
  • feat: Implement Spark-compatible CAST from string to timestamp types #335 (vaibhawvipul)
  • feat: Implement Spark-compatible CAST float/double to string #346 (mattharder91)
  • feat: Only allow incompatible cast expressions to run in comet if a config is enabled #362 (andygrove)
  • feat: Implement Spark-compatible CAST between integer types #340 (ganeshkumar269)
  • feat: Supports Stddev #348 (huaxingao)
  • feat: Improve cast compatibility tests and docs #379 (andygrove)
  • feat: Implement Spark-compatible CAST from non-integral numeric types to integral types #399 (rohitrastogi)
  • feat: Implement Spark unhex #342 (tshauck)
  • feat: Enable columnar shuffle by default #250 (viirya)
  • feat: Implement Spark-compatible CAST from floating-point/double to decimal #384 (vaibhawvipul)
  • feat: Add logging to explain reasons for Comet not being able to run a query stage natively #397 (andygrove)
  • feat: Add support for TryCast expression in Spark 3.2 and 3.3 #416 (vaibhawvipul)
  • feat: Supports UUID column #395 (huaxingao)
  • feat: correlation support #456 (huaxingao)
  • feat: Implement Spark-compatible CAST from String to Date #383 (vidyasankarv)
  • feat: Add COMET_SHUFFLE_MODE config to control Comet shuffle mode #460 (viirya)
  • feat: Add random row generator in data generator #451 (advancedxy)
  • feat: Add xxhash64 function support #424 (advancedxy)
  • feat: add hex scalar function #449 (tshauck)
  • feat: Add “Comet Fuzz” fuzz-testing utility #472 (andygrove)
  • feat: Use enum to represent CAST eval_mode in expr.proto #415 (prashantksharma)
  • feat: Implement ANSI support for UnaryMinus #471 (vaibhawvipul)
  • feat: Add specific fuzz tests for cast and try_cast and fix NPE found during fuzz testing #514 (andygrove)
  • feat: Add fuzz testing for arithmetic expressions #519 (andygrove)
  • feat: Add HashJoin support for BuildRight #437 (viirya)
  • feat: Fix Comet error message #544 (comphead)
  • feat: Support Ansi mode in abs function #500 (planga82)
  • feat: Enable xxhash64 by default #583 (andygrove)
  • feat: Add experimental support for Apache Spark 3.5.1 #587 (andygrove)
  • feat: add nullOnDivideByZero for Covariance #564 (huaxingao)
  • feat: Implement more efficient version of xxhash64 #575 (andygrove)
  • feat: Enable Spark SQL tests for Spark 3.5.1 #603 (andygrove)
  • feat: Initial support for Window function #599 (huaxingao)
  • feat: IsNaN expression in Comet #612 (eejbyfeldt)
  • feat: Add support for CreateNamedStruct #620 (eejbyfeldt)
  • feat: add cargo machete to remove udeps #641 (vaibhawvipul)
  • feat: Upgrade to DataFusion 40.0.0-rc1 #644 (andygrove)
  • feat: Use unified allocator for execution iterators #613 (viirya)
  • feat: Create new datafusion-comet-spark-expr crate containing Spark-compatible DataFusion expressions #638 (andygrove)
  • feat: Move IfExpr to spark-expr crate #653 (andygrove)
  • feat: Upgrade to DataFusion 40 #657 (andygrove)
  • feat: Show user a more intuitive message when queries fall back to Spark #656 (andygrove)
  • feat: Enable remaining Spark 3.5.1 tests #676 (andygrove)
  • feat: Spark-4.0 widening type support #604 (kazuyukitanimura)
  • feat: add scalar subquery pushdown to scan #678 (parthchandra)

Fixed bugs:

  • fix: Comet sink operator should not have children operators #26 (viirya)
  • fix: Fix the UnionExec match branches in CometExecRule #68 (wankunde)
  • fix: Appending null values to element array builders of StructBuilder for null row in a StructArray #78 (viirya)
  • fix: Fix compilation error for CometBroadcastExchangeExec #86 (viirya)
  • fix: Avoid exception caused by broadcasting empty result #92 (wForget)
  • fix: Add num_rows when building RecordBatch #103 (advancedxy)
  • fix: Cast string to boolean not compatible with Spark #107 (erenavsarogullari)
  • fix: Another attempt to fix libcrypto.dylib loading issue #112 (advancedxy)
  • fix: Fix compilation error for Spark 3.2 & 3.3 #117 (sunchao)
  • fix: Fix corrupted AggregateMode when transforming plan parameters #118 (viirya)
  • fix: bitwise shift with different left/right types #135 (viirya)
  • fix: Avoid null exception in removeSubquery #147 (viirya)
  • fix: rat check error in vscode ide #161 (thexiay)
  • fix: Final aggregation should not bind to the input of partial aggregation #155 (viirya)
  • fix: coalesce should return correct datatype #168 (viirya)
  • fix: attempt to divide by zero error on decimal division #172 (viirya)
  • fix: Aggregation without aggregation expressions should use correct result expressions #175 (viirya)
  • fix: Comet native operator can be executed after ReusedExchange #187 (viirya)
  • fix: Try to convert a static list into a set in Rust #184 (advancedxy)
  • fix: Include active spiller when computing peak shuffle memory #196 (sunchao)
  • fix: CometExecRule should handle ShuffleQueryStage and ReusedExchange #186 (viirya)
  • fix: Use makeCopy to change relation in FileSourceScanExec #207 (viirya)
  • fix: Remove duplicate byte array allocation for CometDictionary #224 (viirya)
  • fix: Remove redundant data copy in columnar shuffle #233 (viirya)
  • fix: Only maps FIXED_LEN_BYTE_ARRAY to String for uuid type #238 (huaxingao)
  • fix: Reduce RowPartition memory allocation #244 (viirya)
  • fix: Remove wrong calculation for Murmur3Hash for float with null input #245 (advancedxy)
  • fix: Deallocate row addresses and size arrays after exporting #246 (viirya)
  • fix: Fix wrong children expression order in IfExpr #249 (viirya)
  • fix: Average expression in Comet Final should handle all null inputs from partial Spark aggregation #261 (viirya)
  • fix: Only trigger Comet Final aggregation on Comet partial aggregation #264 (viirya)
  • fix: incorrect result on Comet multiple column distinct count #268 (viirya)
  • fix: Avoid using CometConf #266 (snmvaughan)
  • fix: Fix arrow error when sorting on empty batch #271 (viirya)
  • fix: Include license using # instead of using XML comment #274 (snmvaughan)
  • fix: Comet should not translate try_sum to native sum expression #277 (viirya)
  • fix: incorrect result with aggregate expression with filter #284 (viirya)
  • fix: Comet should not fail on negative limit parameter #288 (viirya)
  • fix: Comet columnar shuffle should not be on top of another Comet shuffle operator #296 (viirya)
  • fix: Iceberg scan transition should be in front of other data source v2 #302 (viirya)
  • fix: CometExec's outputPartitioning might not be same as Spark expects after AQE interferes #299 (viirya)
  • fix: CometShuffleExchangeExec logical link should be correct #324 (viirya)
  • fix: SortMergeJoin with unsupported key type should fall back to Spark #355 (viirya)
  • fix: limit with offset should return correct results #359 (viirya)
  • fix: Disable Comet shuffle with AQE coalesce partitions enabled #380 (viirya)
  • fix: Unknown operator id when explain with formatted mode #410 (leoluan2009)
  • fix: Reuse CometBroadcastExchangeExec with Spark ReuseExchangeAndSubquery rule #441 (viirya)
  • fix: newFileScanRDD should not take constructor from custom Spark versions #412 (ceppelli)
  • fix: fix CometNativeExec.doCanonicalize for ReusedExchangeExec #447 (viirya)
  • fix: Enable cast string to int tests and fix compatibility issue #453 (andygrove)
  • fix: Compute murmur3 hash with dictionary input correctly #433 (advancedxy)
  • fix: Only delegate to DataFusion cast when we know that it is compatible with Spark #461 (andygrove)
  • fix: ColumnReader.loadVector should initiate CometDictionary after re-import arrays #473 (viirya)
  • fix: substring with negative indices should produce correct result #470 (sonhmai)
  • fix: CometReader.loadVector should not overwrite dictionary ids #476 (viirya)
  • fix: Reuse previous CometDictionary Java arrays #489 (viirya)
  • fix: Fallback to Spark for LIKE with custom escape character #478 (sujithjay)
  • fix: Incorrect input schema when preparing result expressions for HashAggregation #501 (viirya)
  • fix: Input batch to ShuffleRepartitioner.insert_batch should not be larger than configured batch size #523 (viirya)
  • fix: Fix integer overflow in date_parser #529 (eejbyfeldt)
  • fix: null character not permitted in chr function #513 (vaibhawvipul)
  • fix: Overflow when reading Timestamp from parquet file #542 (eejbyfeldt)
  • fix: Re-implement some Parquet decode methods without copy_nonoverlapping #558 (andygrove)
  • fix: requested character too large for encoding in chr function #552 (vaibhawvipul)
  • fix: Running cargo build always triggers rebuild #579 (eejbyfeldt)
  • fix: Avoid recursive call to canonicalizePlans #582 (viirya)
  • fix: Return error in pre_timestamp_cast instead of panic #543 (eejbyfeldt)
  • perf: Add criterion benchmark for xxhash64 function #560 (andygrove)
  • fix: Fix range out of index error with a temporary workaround #584 (viirya)
  • fix: Improve error “BroadcastExchange is not supported” #577 (parthchandra)
  • fix: Avoid creating huge duplicate of canonicalized plans for CometNativeExec #639 (viirya)
  • fix: Tag ignored tests that require SubqueryBroadcastExec #647 (parthchandra)
  • fix: Optimize some functions to rewrite dictionary-encoded strings #627 (vaibhawvipul)
  • fix: Remove nightly flag in release-nogit target in Makefile #667 (andygrove)
  • fix: change the not exists base image apache/spark:3.4.3 to 3.4.2 #686 (haoxins)
  • fix: Spark 4.0 SparkArithmeticException test #688 (kazuyukitanimura)
  • fix: address failure caused by method signature change in SPARK-48791 #693 (parthchandra)

Documentation updates:

  • doc: Add Quickstart Comet doc section #125 (comphead)
  • doc: Minor fix Getting started reformatting #128 (comphead)
  • doc: Add initial doc how to expand Comet exceptions #170 (comphead)
  • doc: Update README.md with shuffle configs #208 (viirya)
  • doc: Update supported expressions #237 (viirya)
  • doc: Fix a small typo in README.md #272 (rz-vastdata)
  • doc: Update DataFusion project name and url #300 (viirya)
  • docs: Move existing documentation into new Contributor Guide and add Getting Started section #334 (andygrove)
  • docs: Add more content to the user guide #347 (andygrove)
  • docs: Generate configuration guide in mvn build #349 (andygrove)
  • docs: Add a plugin overview page to the contributors guide #345 (andygrove)
  • doc: Fix target typo in development.md #364 (jc4x4)
  • doc: Clean up supported JDKs in README #366 (edmondop)
  • doc: add contributing in README.md #382 (caicancai)
  • docs: fix the docs url of installation instructions #393 (haoxins)
  • docs: Running ScalaTest suites from the CLI #404 (edmondop)
  • docs: Remove spark.comet.exec.broadcast.enabled from config docs #421 (andygrove)
  • docs: fix various sphinx warnings #428 (tshauck)
  • doc: Add Plan Stability Testing to development guide #432 (viirya)
  • docs: Update Spark shell command to include setting additional class path #435 (andygrove)
  • doc: Add Tuning Guide with shuffle configs #443 (viirya)
  • docs: Add benchmarking guide #444 (andygrove)
  • docs: add guide to adding a new expression #422 (tshauck)
  • docs: changes in documentation #512 (SemyonSinchenko)
  • docs: Improve user documentation for supported operators and expressions #520 (andygrove)
  • docs: Proposal for source release process #556 (andygrove)
  • docs: Update benchmark results #687 (andygrove)
  • docs: Update percentage speedups in benchmarking guide #691 (andygrove)
  • doc: Add memory tuning section to user guide #684 (viirya)

Other:

  • Initial PR #1 (sunchao)
  • build: Add Maven wrapper to the project #13 (sunchao)
  • build: Add basic CI test pipelines #18 (sunchao)
  • Bump com.google.protobuf:protobuf-java from 3.17.3 to 3.19.6 #5 (dependabot[bot])
  • build: Add PR template #23 (sunchao)
  • build: Create ticket templates #24 (comphead)
  • build: Re-enable Scala style checker and spotless #21 (sunchao)
  • build: Remove license header from pull request template #28 (viirya)
  • build: Exclude .github from apache-rat-plugin check #32 (viirya)
  • build: Add CI for MacOS (x64 and aarch64) #35 (sunchao)
  • fix broken link in README.md #39 (nairbv)
  • test: Add some fuzz testing for cast operations #16 (andygrove)
  • test: Fix CI failure on libcrypto #41 (sunchao)
  • test: Reduce test time spent in CometShuffleSuite #40 (sunchao)
  • test: Add test for RoundRobinPartitioning #54 (viirya)
  • build: Fix potential libcrypto lib loading issue for X86 mac runners #55 (advancedxy)
  • refactor: Remove a few duplicated occurrences #53 (sunchao)
  • build: Fix mvn cache for containerized runners #48 (advancedxy)
  • test: Ensure traversed operators during finding first partial aggregaion are all native #58 (viirya)
  • build: Upgrade arrow-rs to 50.0.0 and DataFusion to 35.0.0 #65 (viirya)
  • build: Support built with java 1.8 #45 (advancedxy)
  • test: Add golden files for TPCDSPlanStabilitySuite #73 (sunchao)
  • test: Add TPC-DS test results #77 (sunchao)
  • build: Upgrade spotless version to 2.43.0 #85 (viirya)
  • test: Expose thrown exception when executing query in CometTPCHQuerySuite #96 (viirya)
  • test: Enable TPCDS q41 in CometTPCDSQuerySuite #98 (viirya)
  • build: Add CI for TPCDS queries #99 (viirya)
  • build: Add tpcds-sf-1 to license header excluded list #108 (viirya)
  • build: Show time duration for scala test #116 (advancedxy)
  • test: Move MacOS (x86) pipelines to post-commit #122 (sunchao)
  • build: Upgrade DF to 36.0.0 and arrow-rs 50.0.0 #66 (comphead)
  • test: Reduce end-to-end test time #109 (sunchao)
  • build: Separate and speedup TPC-DS benchmark #130 (advancedxy)
  • build: Re-enable TPCDS queries q34 and q64 in CometTPCDSQuerySuite #133 (viirya)
  • build: Refine names in benchmark.yml #132 (advancedxy)
  • build: Make the build system work out of box #136 (advancedxy)
  • minor: Update README.md with system diagram #148 (alamb)
  • test: Add golden files for test #150 (snmvaughan)
  • build: Add checker for PR title #151 (sunchao)
  • build: Support CI pipelines for Spark 3.2, 3.3 and 3.4 #153 (advancedxy)
  • minor: Only trigger PR title checker on pull requests #154 (sunchao)
  • chore: Fix warnings in both compiler and test environments #164 (advancedxy)
  • build: Upload test reports and coverage #163 (advancedxy)
  • minor: Remove unnecessary logic #169 (sunchao)
  • minor: Make QueryPlanSerde warning log less confusing #181 (viirya)
  • refactor: Skipping slicing on shuffle arrays in shuffle reader #189 (viirya)
  • build: Run Spark SQL tests for 3.4 #166 (sunchao)
  • build: Enforce scalafix check in CI #203 (advancedxy)
  • test: Follow up on Spark 3.4 diff #209 (sunchao)
  • build: Avoid confusion by using profile with clean #215 (snmvaughan)
  • test: Add TPC-H test results #218 (viirya)
  • build: Add CI for TPC-H queries #220 (viirya)
  • test: Enable Comet shuffle in Spark SQL tests #210 (sunchao)
  • test: Disable spark ui in unit test by default #235 (beryllw)
  • chore: Replace deprecated temporal methods #229 (snmvaughan)
  • build: Use specified branch of arrow-rs with workaround to invalid offset buffers from Java Arrow #239 (viirya)
  • test: Enable string-to-bool cast test #251 (andygrove)
  • test: Restore tests in CometTPCDSQuerySuite #252 (viirya)
  • test: Enable all remaining TPCDS queries #254 (viirya)
  • test: Enable all remaining TPCH queries #257 (viirya)
  • chore: Remove some calls to unwrap when calling create_expr in planner.rs #269 (andygrove)
  • chore: Fix typo in info message #279 (andygrove)
  • chore: Fix NPE when running CometTPCHQueriesList directly #285 (advancedxy)
  • chore: Update Comet repo description #291 (viirya)
  • Chore: Cleanup how datafusion session config is created #289 (psvri)
  • build: Update asf.yaml to use @datafusion.apache.org #294 (sunchao)
  • chore: Remove unused functions #301 (kazuyukitanimura)
  • chore: Ignore unused variables #306 (snmvaughan)
  • chore: Update documentation publishing domain and path #310 (andygrove)
  • chore: Add documentation publishing infrastructure #314 (andygrove)
  • build: Move shim directories #318 (kazuyukitanimura)
  • test: Suppress decimal random number tests for 3.2 and 3.3 #319 (kazuyukitanimura)
  • chore: Add allocation source to StreamReader #332 (viirya)
  • chore: Add more cast tests and improve test framework #351 (andygrove)
  • chore: Implement remaining CAST tests #356 (andygrove)
  • build: Add Spark SQL test pipeline with ANSI mode enabled #321 (parthchandra)
  • chore: Store EXTENSION_INFO as Set[String] instead of newline-delimited String #386 (andygrove)
  • build: Add scala-version to matrix #396 (snmvaughan)
  • chore: Add criterion benchmarks for casting between integer types #401 (andygrove)
  • chore: Make COMET_EXEC_BROADCAST_FORCE_ENABLED internal config #413 (viirya)
  • chore: Rename some columnar shuffle configs for code consistently #418 (leoluan2009)
  • chore: Remove an unused config #430 (andygrove)
  • tests: Move random data generation methods from CometCastSuite to new DataGenerator class #426 (andygrove)
  • test: Fix explain with exteded info comet test #436 (kazuyukitanimura)
  • chore: Add cargo bench for shuffle writer #438 (andygrove)
  • chore: improve fallback message when comet native shuffle is not enabled #445 (andygrove)
  • Coverage: Add a manual test to show what Spark built in expression the DF can support directly #331 (comphead)
  • build: Add spark-4.0 profile and shims #407 (kazuyukitanimura)
  • build: bump spark version to 3.4.3 #292 (huaxingao)
  • chore: Removing copying data from dictionary values into CometDictionary #490 (viirya)
  • chore: Update README to highlight Comet benefits #497 (andygrove)
  • test: fix ClassNotFoundException for Hive tests #499 (kazuyukitanimura)
  • build: Enable comet tests with spark-4.0 profile #493 (kazuyukitanimura)
  • chore: Switch to stable Rust #505 (andygrove)
  • Minor: Generate the supported Spark builtin expression list into MD file #455 (comphead)
  • chore: Simplify code in CometExecIterator and avoid some small overhead #522 (andygrove)
  • chore: Upgrade spark to 4.0.0-preview1 #526 (advancedxy)
  • chore: Add UnboundColumn to carry datatype for unbound reference #518 (viirya)
  • chore: Remove 3.4.2.diff #528 (kazuyukitanimura)
  • build: Switch back to official DataFusion repo and arrow-rs after Arrow Java 16 is released #403 (viirya)
  • chore: Add CometEvalMode enum to replace string literals #539 (andygrove)
  • chore: Create initial release process scripts for official ASF source release #429 (andygrove)
  • build: Use DataFusion 39.0.0 release #550 (viirya)
  • chore: disable xxhash64 by default #548 (andygrove)
  • chore: Remove unsafe use of from_raw_parts in Parquet decoder #549 (andygrove)
  • test: Add tests for Scalar and Inverval values for UnaryMinus #538 (vaibhawvipul)
  • chore: Add changelog generator #545 (andygrove)
  • chore: Remove unused hash_utils.rs #561 (andygrove)
  • chore: Use in_list func directly #559 (advancedxy)
  • chore: Fix most of the scala/java build warnings #562 (andygrove)
  • chore: Upgrade to Rust 1.78 and fix UB issues in unsafe code #546 (andygrove)
  • chore: Remove spark.comet.xxhash64.enabled from the config document #586 (viirya)
  • build: Drop Spark 3.2 support #581 (huaxingao)
  • test: Enable Spark 4.0 tests #537 (kazuyukitanimura)
  • refactor: Remove method get_global_jclass #580 (eejbyfeldt)
  • chore: Move some utility methods to submodules of scalar_funcs #590 (advancedxy)
  • chore: Upgrade to Rust 1.79 #570 (andygrove)
  • chore: Remove some calls to unwrap #598 (andygrove)
  • chore: Improve JNI safety #600 (andygrove)
  • chore: remove some unwraps from shuffle module #601 (andygrove)
  • chore: Use proper constructor of IndexShuffleBlockResolver #610 (viirya)
  • chore: Update benchmark results #614 (andygrove)
  • build: Upgrade to 2.13.14 for scala-2.13 profile #626 (viirya)
  • chore: Rename shuffle write metric #624 (andygrove)
  • minor: replace .downcast_ref::().is_some() with .is::() #635 (andygrove)
  • test: Add CometTPCDSQueryTestSuite #628 (viirya)
  • chore: Convert Rust project into a workspace #637 (andygrove)
  • chore: Add Miri workflow #636 (andygrove)
  • test: Run optimized version of q72 derived from TPC-DS #652 (viirya)
  • chore: Refactoring of CometError/SparkError #655 (andygrove)
  • chore: Move cast to spark-expr crate #654 (andygrove)
  • chore: Remove utils crate and move utils into spark-expr crate #658 (andygrove)
  • chore: Move temporal kernels and expressions to spark-expr crate #660 (andygrove)
  • chore: Move protobuf files to separate crate #661 (andygrove)
  • Use IfExpr to check when input to log2 is <=0 and return null #506 (PedroMDuarte)
  • chore: Change suffix on some expressions from Exec to Expr #673 (andygrove)
  • chore: Fix some regressions with Spark 3.5.1 #674 (andygrove)
  • chore: Improve fuzz testing coverage #668 (andygrove)
  • Create Comet docker file #675 (comphead)
  • chore: Add microbenchmarks #671 (andygrove)
  • build: Exclude protobug generated codes from apache-rat check #683 (viirya)
  • chore: Disable abs and signum because they return incorrect results #695 (andygrove)

Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

   100	Liang-Chi Hsieh
    82	Andy Grove
    28	advancedxy
    27	Chao Sun
    14	Huaxin Gao
    11	KAZUYUKI TANIMURA
     9	Vipul Vaibhaw
     8	Parth Chandra
     7	Emil Ejbyfeldt
     7	Steve Vaughan
     7	comphead
     4	Oleks V
     4	Pablo Langa
     4	Trent Hauck
     2	Edmondo Porcu
     2	Vrishabh
     2	Xin Hao
     2	Xuedong Luan
     1	Andrew Lamb
     1	Brian Vaughan
     1	Cancai Cai
     1	Eren Avsarogullari
     1	Holden Karau
     1	JC
     1	Junbo wang
     1	Junfan Zhang
     1	Pedro M Duarte
     1	Prashant K. Sharma
     1	RickestCode
     1	Rohit Rastogi
     1	Roman Zeyde
     1	Semyon
     1	Son
     1	Sujith Jay Nair
     1	Zhen Wang
     1	ceppelli
     1	dependabot[bot]
     1	thexia
     1	vidyasankarv
     1	wankun
     1	గణేష్

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.