18.0.0 (2023-02-10)

Full Changelog

Breaking changes:

  • Use DataFusionError instead of ArrowError in SendableRecordBatchStream #5101 (comphead)
  • Update to arrow 32 and Switch to RawDecoder for JSON #5056 [sql] (tustvold)

Implemented enhancements:

  • DiskManager to create a spill folder if doesn't exist #5186
  • cast expression may cause duplicate column name error #5174
  • Add type coercion from Dictionary to string for regular expressions #5154
  • Unnecessary Filter on Parquet datasources #5149
  • [sqllogictest] Support pg_typeof for Postgres compatibility tests #5147
  • Supporting Grafana global variables #5144
  • add example for standalone DataFusion server which supports Arrow Flight SQL JDBC driver #5139
  • Support for InList in datafusion-substrait #5134
  • Pipeline file opening in FileStream #5129
  • Make parse_physical_expr public #5107
  • Use DataFusionError in SendableRecordBatchStream #5039
  • Interval coercion:date_bin('1 hour',...) does not work but date_bin(interval '1 hour', ... does #4853
  • Explain <query> should not fail if meeting errors when optimizing the query #4766
  • Add option to determine whether to convert identifiers #4551
  • Replace &Option<T> with Option<&T>. #4424
  • Error type in RecordBatchStream #4172
  • Support non-equi join (e.g. ON clause) in Dataframe API #1254
  • Allow ParquetExec to parallelize work based on row groups #137

Fixed bugs:

  • Confusing schema errors when using window partition #5229
  • Propagating empty_relation generate an illegal plan #5218
  • The test in_list_types_struct_literal fails when setting skip_failed_rules as false #5217
  • Placeholder values are not replaced in ScalarSubqueries #5215
  • Querying against delta lake table does not seem to work #5202
  • Arithmetic operation doesn't work with DictionaryArray #5193
  • simplify_expr() invoke nullable() exist bug #5191
  • CI is currently broken on git diff: Not a git repository #5180
  • write_csv/json/parquet isn't cancel safe #5178
  • no hyperlink to blaze-rs [doc: README-“Use Cases”] #5175
  • Arithmetic scalar operation doesn't work with DictionaryArray #5150
  • Sort operator disappear in physical_plan #5100
  • Window function error: InvalidArgumentError(“number of columns(27) must match number of fields(35) in schema” #5090
  • INSERT statements without target column list are not working #5078
  • fix file stream time scanning metrics bug #5019
  • Date before 1678 causes panic #4875
  • Can not ORDER BY an aliased group column #4854
  • The filters expressions in TableScan may contain fields not included in schema. #4793
  • Comparing a Timestamp to a Date32 fails #4644
  • String --> TableReference parsing does not properly handle " and . #4532
  • can't compare NULL type with NULL type #4335
  • Add ambiguous check when generate selection plan #4196
  • Internal error in CAST from Timestamp[us] #3922
  • Run median expr on parquet file column got error #3805
  • aliasing a field renders it missing in the order by clause #669
  • Querying datetime data in DataFusion with an embedded timezone always fails #153

Documentation updates:

Closed issues:

  • Change coerced type for comparison between timestamp with date to timestamp #4761

Merged pull requests: