Status: pre-RFC binding-specific design proposal. The current Python binding remains a single native distribution.
This document applies the shared extension architecture to Python. It preserves Operator and AsyncOperator while moving service and layer implementation dependencies into independently installable distributions. The shared compatibility contract is canonical for native ABI, configuration, lifetime, and loader rules; this document defines Python deltas.
The current Python binding has several compile-time assumptions that cannot serve as a dynamic extension interface:
bindings/python builds one PyO3 cdylib, opendal._opendal.services-all feature, subject to its explicit exclusions and platform conditions.Scheme is a feature-gated Rust enum. It cannot represent a service installed after the base extension was compiled.opendal.services and opendal.layers are PyO3 submodules inserted into sys.modules, not filesystem packages that other distributions can extend.Layer stores Box<dyn PythonLayer> in the base native library. A PyO3 subclass marker does not make that Rust trait object transferable from an independently linked extension.opendal.config.ServiceConfig is one generated closed union of compiled services.The migration must change these internals without requiring every caller to adopt a new operator abstraction.
The proposed release family is:
opendal-runtime provides the shared native runtime opendal owns the `opendal` import package and Python adapter opendal-service-s3 contributes the S3 manifest, native code, and typing opendal-service-hdfs contributes libhdfs-backed HDFS lazily opendal-layer-timeout contributes Timeout opendal-layer-foyer contributes Foyer
The opendal distribution declares its required_runtime_protocol. The opendal-runtime distribution exposes its minimum and current protocol levels for the binding to check. Every native service/layer distribution requires an exact opendal-runtime release and embeds that OpenDAL version in its bootstrap metadata. Installing the base wheel resolves the runtime dependency:
python -m pip install opendal
An application installs the main binding and selected packages; the package manager resolves opendal-runtime:
python -m pip install \ opendal \ opendal-service-s3 \ opendal-layer-timeout \ opendal-layer-foyer
The extension package names are provisional.
The intended typed import layout is:
opendal regular package owned by the base distribution opendal.services namespace subpackage opendal.services.s3 supplied by opendal-service-s3 opendal.services.hdfs supplied by opendal-service-hdfs opendal.layers namespace subpackage opendal.layers.timeout supplied by opendal-layer-timeout opendal.layers.foyer supplied by opendal-layer-foyer
Python packaging supports splitting namespace subpackages across distributions, but every participant must follow one consistent layout. See the PyPA namespace package guide.
Before using this layout, the base binding must move the current native opendal.services and opendal.layers definitions under a private native module and expose real Python package directories. Existing flat names can be re-exported during migration.
If wheel-install and namespace ownership prototypes are not reliable across the supported installers, the first tracer packages may use unambiguous top-level imports such as opendal_service_s3. The runtime extension contract does not depend on the cosmetic import layout.
Python entry points advertise installed manifests:
[project.entry-points."opendal.services"] s3 = "opendal.services.s3:_register" [project.entry-points."opendal.layers"] foyer = "opendal.layers.foyer:_register"
Entry points allow the runtime to find an installed package without importing every package. The resolver follows these rules:
import opendal loads only the base adapter and runtime.Explicit imports remain useful for deterministic startup and access to typed configuration classes. Entry-point discovery preserves the current concise URI path for callers that only need strings.
Existing construction remains valid:
import opendal op = opendal.Operator("s3", bucket="photos", region="us-east-1") async_op = opendal.AsyncOperator.from_uri( "s3://photos/archive?region=us-east-1", endpoint="https://s3.example.com", )
Strings are the canonical dynamic scheme identifiers. The existing Scheme enum may remain as a frozen compatibility aid for previously bundled official services, but it is not an inventory of installed extensions.
Typed configuration moves into its service distribution:
from opendal import AsyncOperator from opendal.services.s3 import S3Config config = S3Config(scheme="s3", bucket="photos", region="us-east-1") op = AsyncOperator.from_config(config)
The base from_config runtime path accepts a generic mapping or service recipe. Package-local generated TypedDict or dataclass definitions provide field checking without extending one base ServiceConfig union. The service package owns structured serialization and validation for the matching OpenDAL release.
The adapter converts mappings to the shared ConfigValue grammar. It rejects unsupported Python objects, cyclic containers, oversized values, unknown fields, and numeric overflow before package construction. Package-local types can expose Python-native values, but no PyObject crosses the factory seam.
URI construction sends the original URI plus explicit string options to the service factory. It does not convert them through a central Python config schema. This preserves S3 and WebDAV configurator behavior.
Simple layer factories remain synchronous:
from opendal.layers.throttle import ThrottleLayer from opendal.layers.timeout import TimeoutLayer limit = ThrottleLayer(bandwidth=10 * 1024, burst=10 * 1024 * 1024) timeout = TimeoutLayer(timeout=60.0, io_timeout=10.0) layered = op.layer(limit).layer(timeout)
Resource-backed construction is asynchronous:
from opendal.layers.foyer import FoyerLayer cache = await FoyerLayer.create( memory_capacity=64 << 20, storage_path="/var/cache/opendal", ) cached = async_op.layer(cache)
A blocking helper may be provided for synchronous applications only if it uses the same runtime factory, releases the GIL while waiting, and has defined cancellation/cleanup behavior. It must not create a second Tokio runtime inside the Foyer package.
Every concrete Python layer wraps a base-owned opaque LayerHandle. It does not expose a package-local Rust trait object. Applying it returns a new operator and preserves the native layer's service and context hooks.
One layer object may carry shared state:
Throttle accepts only positive integer bandwidth and burst values in the supported u32 range. Python validation must reject invalid values before the native constructor can assert.
Timeout values remain finite, positive seconds at the Python interface. The adapter uses the current Duration::try_from_secs_f64 rule, which rounds to the nearest nanosecond with ties to even, and then emits SignedDuration. It rejects values outside SignedDuration's i64-seconds range instead of saturating them.
Later .layer() calls are outer layers. The binding preserves the canonical Timeout/Retry order and rejects the known unsafe composition when it can observe both layer IDs.
The shared runtime owns operation futures. AsyncOperator converts them into Python awaitables through the base adapter. A service or layer package does not capture Python event loops, PyObject references, or PyO3 runtime state in its native factory.
Cancellation of a Python awaitable must reach the runtime operation. The adapter must not detach a future merely because its Python wrapper was dropped.
Blocking Operator and AsyncOperator should retain the same constructed native operator graph when converted or cloned. Rebuilding from a scheme and options would lose stateful Foyer and Throttle identity.
The Python adapter should distinguish extension failures before mapping normal OpenDAL operation errors:
ExtensionNotInstalled ExtensionLoadError ExtensionIncompatible ExtensionConflict LayerInitializationError
ExtensionNotInstalled may include the canonical distribution name as an installation hint. It must not run pip, modify the environment, or infer a third-party package name from untrusted input.
Configuration and operation errors continue to use OpenDAL's Python exception hierarchy. Extension diagnostics include package and scheme/layer IDs but omit credentials and unredacted option maps.
A dynamic operator cannot rely on native pointer serialization. New pickle support must choose one of these explicit policies:
It must never silently discard layers. Reconstructing a Foyer layer creates or reopens a cache according to package policy; it does not preserve live in-memory entries. Reconstructing Throttle starts new token history.
Existing layered pickles did not record layer recipes, so migration code cannot recover that lost information retroactively.
The current release matrix includes CPython 3.10-specific wheels, CPython 3.11 abi3 wheels, and free-threaded CPython wheels. The dynamic design must prove which artifacts can actually be shared:
abi3 compatibility rules.abi3 does not imply free-threaded compatibility.The libhdfs-backed HDFS wheel may have a smaller platform allowlist or ship as a source distribution. Installing opendal, S3, WebDAV, or hdfs-native must not load HDFS code or require Java/Hadoop.
opendal.services and opendal.layers into filesystem/namespace packages and re-export existing names.abi3, and free-threaded artifact composition on every supported target.