)]}' { "commit": "6f3b1cab75cfaa0cdf914f9b6fa023cb9afccd7d", "tree": "1ec0b32c45983ea935bf920d30fb47f413c51336", "parents": [ "af1c07b712d2aa9387e2ec5a55c9bd5198d5c97b" ], "author": { "name": "Tim Saucer", "email": "timsaucer@gmail.com", "time": "Sat Oct 11 10:28:32 2025 -0400" }, "committer": { "name": "GitHub", "email": "noreply@github.com", "time": "Sat Oct 11 10:28:32 2025 -0400" }, "message": "Unify Table representations (#1256)\n\n* Migrate Table → TableProvider; refactor registration and access, update\ndocs/tests, add DataFrame view support, and improve Send/concurrency\nsupport.\n\nmigrates the codebase from using `Table` to a\n`TableProvider`-based API, refactors registration and access paths to\nsimplify catalog/context interactions, and updates documentation and\nexamples. DataFrame view handling is improved (`into_view` is now\npublic), the test-suite is expanded to cover new registration and async\nSQL scenarios, and `TableProvider` now supports the `Send` trait across\nmodules for safer concurrency. Minor import cleanup and utility\nadjustments (including a refined `pyany_to_table_provider`) are\nincluded.\n\n* Refactors and bug fixes around TableProvider registration and\nDataFrame→TableProvider conversion, plus tests and FFI/pycapsule\nimprovements.\n\n-- Registration logic \u0026 API\n\n* Refactor of table provider registration logic for improved clarity and\n simpler call sites.\n* Remove PyTableProvider registration from an internal module (reduces\n surprising side effects).\n* Update table registration method to call `register_table` instead of\n `register_table_provider`.\n* Extend `register_table` to support `TableProviderExportable` so more\n provider types can be registered uniformly.\n* Improve error messages related to registration failures (missing\n PyCapsule name and DataFrame registration errors).\n\n-- DataFrame ↔ TableProvider conversions\n\n* Introduce utility functions to simplify table provider conversions and\n centralize conversion logic.\n* Rename `into_view_provider` → `to_view_provider` for clearer intent.\n* Fix `from_dataframe` to return the correct type and update\n `DataFrame.into_view` to import the correct `TableProvider`.\n* Remove an obsolete `dataframe_into_view` test case after the refactor.\n\n-- FFI / PyCapsule handling\n\n* Update `FFI_TableProvider` initialization to accept an optional\n parameter (improves FFI ergonomics).\n* Introduce `table_provider_from_pycapsule` utility to standardize\n pycapsule-based construction.\n* Improve the error message when a PyCapsule name is missing to help\n debugging.\n\n-- DeltaTable \u0026 specific integrations\n\n* Update TableProvider registration for `DeltaTable` to use the correct\n registration method (matches the new API surface).\n\n-- Tests, docs \u0026 minor fixes\n\n* Add tests for registering a `TableProvider` from a `DataFrame` and\n from a capsule to ensure conversion paths are covered.\n* Fix a typo in the `register_view` docstring and another typo in the\n error message for unsupported volatility type.\n* Simplify version retrieval by removing exception handling around\n `PackageNotFoundError` (streamlines code path).\n\n* TableProvider refactor \u0026 PyDataFrame integration\n\n* Removed unused helpers (`extract_table_provider`, `_wrap`) and dead code to simplify maintenance.\n* Consolidated and streamlined table-provider extraction and registration logic; improved error handling and replaced a hardcoded error message with `EXPECTED_PROVIDER_MSG`.\n* Marked `from_view` as deprecated; updated deprecation message formatting and adjusted the warning `stacklevel` so it points to caller code.\n* Removed the `Send` marker from TableProvider trait objects to increase type flexibility — review threading assumptions.\n* Added type hints to `register_schema` and `deregister_table` methods.\n* Adjusted tests and exceptions (e.g., changed one test to expect `RuntimeError`) and updated test coverage accordingly.\n* Introduced a refactored `TableProvider` class and enhanced Python integration by adding support for extracting `PyDataFrame` in `PySchema`.\n\nNotes:\n\n* Consumers should migrate away from `TableProvider::from_view` to the new TableProvider integration.\n* Audit any code relying on `Send` for trait objects passed across threads.\n* Update downstream tests and documentation to reflect the changed exception types and deprecation.\n\n* Normalize \u0026 simplify TableProvider/DataFrame registration; add\nutilities, docs, and robustness fixes\n\n* Normalized table-provider handling and simplified registration flow\n across the codebase; multiple commits centralize provider coercion and\nnormalization.\n* Introduced utility helpers (`coerce_table_provider`,\n `extract_table_provider`, `_normalize_table_provider`) to centralize\nextraction, error handling, and improve clarity.\n* Simplified `from_dataframe` / `into_view` behavior: clearer\n implementations, direct returns of DataFrame views where appropriate,\nand added internal tests for DataFrame flows.\n* Fixed DataFrame registration semantics: enforce `TypeError` for\n invalid registrations; added handling for `DataFrameWrapper` by\nconverting it to a view.\n* Added tests, including a schema registration test using a PyArrow\n dataset and internal DataFrame tests to cover new flows.\n* Documentation improvements: expanded `from_dataframe` docstrings with\n parameter details, added usage examples for `into_view`, and\ndocumented deprecations (e.g., `register_table_provider` →\n`register_table`).\n* Warning and UX fixes: synchronized deprecation `stacklevel` so\n warnings point to caller code; improved `__dir__` to return sorted,\nunique attributes.\n* Cleanup: removed unused imports (including an unused error import from\n `utils.rs`) and other dead code to reduce noise.\n\n* refactor: update documentation for DataFrame to Table Provider conversion\n\n* refactor: replace to_view_provider with inner_df for DataFrame access\n\n* refactor: streamline TableProvider creation from DataFrame by consolidating method calls\n\n* fix ruff errors\n\n* refactor: enhance autoapi_skip_member_fn to skip private variables and avoid documentation duplication\n\n* revert main 49.0.0 md\n\n* refactor: add comment in autoapi_skip_member_fn\n\n* refactor: remove isort and ruff comments to clean up import section\n\n* docs: enhance docstring for DataFrame.into_view method to clarify usage and advantages\n\n* docs: update example in DataFrame.into_view docstring for clarity\n\n* docs: update example for registering Delta Lake tables to simplify usage\n\n* docs: update table provider documentation for clarity and deprecate old methods\n\n* docs: update documentation to reflect removal of TableProvider and usage of Table instead\n\n* remove TableProvider in Python, update missing_exports function, doc\n\n* Fix Ruff errors\n\n* Refactor test_table_loading to use Table instead of TableProvider\n\n* Refactor aggregate tests to simplify result assertions and improve readability\n\n* Add comments to clarify table normalization in aggregate tests\n\n* Initial implementation of unified table suggestion\n\n* update unit tests\n\n* Change documentation to be more user oriented\n\n* Update ffi examples\n\n* Update documentation\n\n* More documentation\n\n* Make documentation more user facing\n\n* More documentation updates\n\n* remove cruft\n\n* fix ordering\n\n* give read_table the same treatment\n\n* Reuse Table constructor to idenfity non-ffi tables when using udtf\n\n---------\n\nCo-authored-by: Siew Kam Onn \u003ckosiew@gmail.com\u003e", "tree_diff": [ { "type": "modify", "old_id": "28db17d3585d7c7d7962898e8e56e8f3d132bd04", "old_mode": 33188, "old_path": "docs/source/conf.py", "new_id": "01813b0320e5a3b7c59dc71edc1507d93315f0df", "new_mode": 33188, "new_path": "docs/source/conf.py" }, { "type": "modify", "old_id": "e8a0398b8e7c1bec005ee5b524dfa31d4c2c06ba", "old_mode": 33188, "old_path": "docs/source/contributor-guide/ffi.rst", "new_id": "64413866f9ffb0e63cc3e8b3f9e2ddb0463ae833", "new_mode": 33188, "new_path": "docs/source/contributor-guide/ffi.rst" }, { "type": "modify", "old_id": "a9b119b93599c52af9c07e5242d27bf71f132595", "old_mode": 33188, "old_path": "docs/source/user-guide/data-sources.rst", "new_id": "26f1303c480010630ad83b08bc91a1f3785d4579", "new_mode": 33188, "new_path": "docs/source/user-guide/data-sources.rst" }, { "type": "modify", "old_id": "bd1d6b80f30a4207f3ce0f14dbd95884a12c5cbd", "old_mode": 33188, "old_path": "docs/source/user-guide/io/table_provider.rst", "new_id": "29e5d9880d8968e75e23344c9fc88809f9603064", "new_mode": 33188, "new_path": "docs/source/user-guide/io/table_provider.rst" }, { "type": "modify", "old_id": "72aadf64ca475e4d69a05996e3543f4b4cd6ff11", "old_mode": 33188, "old_path": "examples/datafusion-ffi-example/python/tests/_test_catalog_provider.py", "new_id": "1bf1bf136c6d63e3539b08b3df053cae39fb61cd", "new_mode": 33188, "new_path": "examples/datafusion-ffi-example/python/tests/_test_catalog_provider.py" }, { "type": "modify", "old_id": "f3c56a90a55dfe791e6947046d9d53d33bb2808e", "old_mode": 33188, "old_path": "examples/datafusion-ffi-example/python/tests/_test_table_function.py", "new_id": "4b8b21454126b12ba93987917330ab93ee27fe6d", "new_mode": 33188, "new_path": "examples/datafusion-ffi-example/python/tests/_test_table_function.py" }, { "type": "modify", "old_id": "6b24da06c948c69bb264f735e4305aca1394b8dc", "old_mode": 33188, "old_path": "examples/datafusion-ffi-example/python/tests/_test_table_provider.py", "new_id": "48feaff64074786007f541e4f133719faccabe88", "new_mode": 33188, "new_path": "examples/datafusion-ffi-example/python/tests/_test_table_provider.py" }, { "type": "modify", "old_id": "e9d2dba75f3b5c65335079c2f1c733ce53b9cc75", "old_mode": 33188, "old_path": "python/datafusion/__init__.py", "new_id": "9ebd58ea69cfba3c2c99aad11f0bebdb91a07728", "new_mode": 33188, "new_path": "python/datafusion/__init__.py" }, { "type": "modify", "old_id": "536b3a79067b11f2067f4abbf24a7a16c9330b36", "old_mode": 33188, "old_path": "python/datafusion/catalog.py", "new_id": "da54d233d974e149abd5134c676bf15d7f820dbd", "new_mode": 33188, "new_path": "python/datafusion/catalog.py" }, { "type": "modify", "old_id": "b6e728b51eb59aed1c017b5d649b3f7b12e37441", "old_mode": 33188, "old_path": "python/datafusion/context.py", "new_id": "0aa2f27c4beec0eaa53f34891a1d67ede53c66db", "new_mode": 33188, "new_path": "python/datafusion/context.py" }, { "type": "modify", "old_id": "c1b649e33d124a57dccf577049e7d86688ebcfa0", "old_mode": 33188, "old_path": "python/datafusion/dataframe.py", "new_id": "5a21d773b54730ab70adfc015ff6188d96851ef0", "new_mode": 33188, "new_path": "python/datafusion/dataframe.py" }, { "type": "modify", "old_id": "5d1180bd133f99118dc6f1176bfc288110027308", "old_mode": 33188, "old_path": "python/datafusion/expr.py", "new_id": "82e30a78c2c85eef5724b2a9ef8c55871c5e590c", "new_mode": 33188, "new_path": "python/datafusion/expr.py" }, { "type": "modify", "old_id": "551e20a6f56ab8ff169d4d73456bc79907e72f36", "old_mode": 33188, "old_path": "python/datafusion/io.py", "new_id": "67dbc730fa16a5798188db081fb04c130484755c", "new_mode": 33188, "new_path": "python/datafusion/io.py" }, { "type": "modify", "old_id": "1f9ecbfc398a8f341af7dd0d5bd8e64695ab7c90", "old_mode": 33188, "old_path": "python/tests/test_catalog.py", "new_id": "08f494dee04b3d863f6035812d6d1467aad73ba4", "new_mode": 33188, "new_path": "python/tests/test_catalog.py" }, { "type": "modify", "old_id": "6dbcc0d5efc6e659f03ba116972df6245a1c752b", "old_mode": 33188, "old_path": "python/tests/test_context.py", "new_id": "94d1e6a39a98c80d413d1e370014adec9693870f", "new_mode": 33188, "new_path": "python/tests/test_context.py" }, { "type": "modify", "old_id": "f484cb282422c339d93ef27e679a1c23224a7286", "old_mode": 33188, "old_path": "python/tests/test_wrapper_coverage.py", "new_id": "cf6719ecf2be5c33b0d8cc826e070586868a4743", "new_mode": 33188, "new_path": "python/tests/test_wrapper_coverage.py" }, { "type": "modify", "old_id": "b5fa3da72fc21d38112fd34483761ee379eddf3d", "old_mode": 33188, "old_path": "src/catalog.rs", "new_id": "398c5881fbf2df81fed15e8f5e22c6777669a782", "new_mode": 33188, "new_path": "src/catalog.rs" }, { "type": "modify", "old_id": "e3f978ee17595dd29c8cf5b8c4cf18ecd2a18305", "old_mode": 33188, "old_path": "src/context.rs", "new_id": "dc18a7676f834d066a5b3efbd73f281d6b1d7096", "new_mode": 33188, "new_path": "src/context.rs" }, { "type": "modify", "old_id": "555a8500d106e383fd4e4d9f900dc63243b05958", "old_mode": 33188, "old_path": "src/dataframe.rs", "new_id": "bfdc35e13728629110e134b2ae4c6c6b8a695e35", "new_mode": 33188, "new_path": "src/dataframe.rs" }, { "type": "modify", "old_id": "29d3f41da41b57e36d405f59a2e15329dd1a23de", "old_mode": 33188, "old_path": "src/lib.rs", "new_id": "0361c7315ea4c9bdd081b8926ed7682ce6722356", "new_mode": 33188, "new_path": "src/lib.rs" }, { "type": "add", "old_id": "0000000000000000000000000000000000000000", "old_mode": 0, "old_path": "/dev/null", "new_id": "b830f7764f906f5bc2da12d506e4b64aaee2d2b5", "new_mode": 33188, "new_path": "src/table.rs" }, { "type": "modify", "old_id": "55f306b1712f7ceb4e6a1861b468e51386f087af", "old_mode": 33188, "old_path": "src/udtf.rs", "new_id": "f6604e5bc8abb96ba55360d130f8172f62cb4ee3", "new_mode": 33188, "new_path": "src/udtf.rs" }, { "type": "modify", "old_id": "3b30de5de63f8b6cb204a24b7d04ef37c3e4a15c", "old_mode": 33188, "old_path": "src/utils.rs", "new_id": "0fcfadceaf1929c1762fbf59a4ae730e877f3b7b", "new_mode": 33188, "new_path": "src/utils.rs" } ] }