Make manifest cache size configurable (#2993)

<!--
Thanks for opening a pull request!
-->

<!-- In the case this PR will resolve an issue, please replace
${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${2952} -->

Closes #2952

# Rationale for this change
Through discussion in issue #2325, we realized that there was a memory
leak in the manifest cache. PR
https://github.com/apache/iceberg-python/pull/2951 fixed this memory
leak, but we decided that it would be best for developer experience if
users could configure the cache for their needs.

This includes the ability to configure the manifest cache size,
including setting it to `0` to disable caching.

## Are these changes tested?
Yes - unit tests are included to verify these changes

## Are there any user-facing changes?
Yes:
- `manifest-cache-size` can be used in `.pyiceberg.yaml` to configure
the size of the manifest cache
- `PYICEBERG_MANIFEST_CACHE_SIZE` can be used to configure the size of
the manifest cache

<!-- In the case of user-facing changes, please add the changelog label.
-->

---------

Co-authored-by: Kevin Liu <kevin.jq.liu@gmail.com>
4 files changed
tree: 43cbbb0d80f016faeb6220286dfafa83e5ce5767
  1. .github/
  2. dev/
  3. mkdocs/
  4. notebooks/
  5. pyiceberg/
  6. tests/
  7. vendor/
  8. .asf.yaml
  9. .codespellrc
  10. .gitignore
  11. .markdownlint.yaml
  12. .pre-commit-config.yaml
  13. AGENTS.md
  14. LICENSE
  15. Makefile
  16. MANIFEST.in
  17. NOTICE
  18. pyproject.toml
  19. README.md
  20. ruff.toml
  21. SECURITY-THREAT-MODEL.md
  22. setup.py
  23. uv.lock
README.md

Iceberg Python

PyIceberg is a Python library for programmatic access to Iceberg table metadata as well as to table data in Iceberg format. It is a Python implementation of the Iceberg table spec.

The documentation is available at https://py.iceberg.apache.org/.

Get in Touch