docs/source/user-guide/upgrade-guides.md

<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

# Upgrade Guides

## DataFusion 54.0.0

The `Config` class has been removed. It was a standalone wrapper around
`ConfigOptions` that could not be connected to a `SessionContext`, making it
effectively unusable. Use {py:class}`~datafusion.context.SessionConfig` instead,
which is passed directly to `SessionContext`.

Before:

```python
from datafusion import Config

config = Config()
config.set("datafusion.execution.batch_size", "4096")
# config could not be passed to SessionContext
```

After:

```python
from datafusion import SessionConfig, SessionContext

config = SessionConfig().set("datafusion.execution.batch_size", "4096")
ctx = SessionContext(config)
```

The aggregate functions {py:func}`~datafusion.functions.sum` and
{py:func}`~datafusion.functions.avg` now accept a `distinct` argument, matching
the other aggregate functions. `distinct` is inserted *before* `filter` in the
argument list, so any code that passed `filter` positionally must be updated to
pass it as a keyword argument. The types are distinct so a type checker should flag this.

Before:

```python
f.sum(column("a"), my_filter)
f.avg(column("a"), my_filter)
```

Now:

```python
f.sum(column("a"), filter=my_filter)
f.avg(column("a"), filter=my_filter)
```

## DataFusion 53.0.0

This version includes an upgraded version of `pyo3`, which changed the way to extract an FFI
object. Example:

Before:

```rust
let codec = unsafe { capsule.reference::<FFI_LogicalExtensionCodec>() };
```

Now:

```rust
let data: NonNull<FFI_LogicalExtensionCodec> = capsule
    .pointer_checked(Some(c_str!("datafusion_logical_extension_codec")))?
    .cast();
let codec = unsafe { data.as_ref() };
```

## DataFusion 52.0.0

This version includes a major update to the {ref}`ffi` due to upgrades
to the [Foreign Function Interface](https://doc.rust-lang.org/nomicon/ffi.html).
Users who contribute their own `CatalogProvider`, `SchemaProvider`,
`TableProvider` or `TableFunction` via FFI must now provide access to a
`LogicalExtensionCodec` and a `TaskContextProvider`. The function signatures
for the methods to get these `PyCapsule` objects now requires an additional
parameter, which is a Python object that can be used to extract the
`FFI_LogicalExtensionCodec` that is necessary.

A complete example can be found in the [FFI example](https://github.com/apache/datafusion-python/tree/main/examples/datafusion-ffi-example).
Your FFI hook methods — `__datafusion_catalog_provider__`,
`__datafusion_schema_provider__`, `__datafusion_table_provider__`, and
`__datafusion_table_function__` — need to be updated to accept an additional
`session: Bound<PyAny>` parameter, as shown in this example.

```rust
#[pymethods]
impl MyCatalogProvider {
    pub fn __datafusion_catalog_provider__<'py>(
        &self,
        py: Python<'py>,
        session: Bound<PyAny>,
    ) -> PyResult<Bound<'py, PyCapsule>> {
        let name = cr"datafusion_catalog_provider".into();

        let provider = Arc::clone(&self.inner) as Arc<dyn CatalogProvider + Send>;

        let codec = ffi_logical_codec_from_pycapsule(session)?;
        let provider = FFI_CatalogProvider::new_with_ffi_codec(provider, None, codec);

        PyCapsule::new(py, provider, Some(name))
    }
}
```

To extract the logical extension codec FFI object from the provided object you
can implement a helper method such as:

```rust
pub(crate) fn ffi_logical_codec_from_pycapsule(
    obj: Bound<PyAny>,
) -> PyResult<FFI_LogicalExtensionCodec> {
    let attr_name = "__datafusion_logical_extension_codec__";
    let capsule = if obj.hasattr(attr_name)? {
        obj.getattr(attr_name)?.call0()?
    } else {
        obj
    };

    let capsule = capsule.downcast::<PyCapsule>()?;
    validate_pycapsule(capsule, "datafusion_logical_extension_codec")?;

    let codec = unsafe { capsule.reference::<FFI_LogicalExtensionCodec>() };

    Ok(codec.clone())
}
```

The DataFusion FFI interface updates no longer depend directly on the
`datafusion` core crate. You can improve your build times and potentially
reduce your library binary size by removing this dependency and instead
using the specific datafusion project crates.

For example, instead of including expressions like:

```rust
use datafusion::catalog::MemTable;
```

Instead you can now write:

```rust
use datafusion_catalog::MemTable;
```