Status: pre-RFC design exploration. OpenDAL has not accepted or implemented the extension interface described by these documents.
This design allows a language binding to install services and layers as independent packages. For example, an application can install only S3, Timeout, and Foyer without rebuilding or replacing the base binding.
The leading prototype candidate uses a shared native runtime package with a small common protocol. Python, Ruby, and Node.js use different language interfaces over the same runtime and native extension model. The proposed release process publishes the runtime with the language bindings as one coordinated release family. Selecting that candidate remains conditional on a successful cross-platform packaging prototype and the OpenDAL RFC process.
These documents are preliminary design input. They are not an accepted RFC or a commitment to package names, release dates, or compatibility guarantees. A subsequent proposal must follow the OpenDAL RFC process before implementation.
The design must satisfy all of the following requirements:
Layer behavior, including both service and operation-context composition.The proposal derives its constraints from these current implementations:
OperatorUri defines the initial URI and explicit-option merge.Layer trait composes both service and operation-context planes.Python package Ruby gem npm package | | | v v v Python adapter Ruby adapter Node-API adapter \ | / +-----------------+--------------------+ | v OpenDAL extension runtime registry, Tokio, core types, loader, errors, handles, library leases / \ v v service extension layer extension S3 / WebDAV / HDFS Timeout / Foyer / Throttle
The diagram shows a logical architecture. The selected distribution model puts the implementation in a shared runtime package instead of embedding a private copy in each main binding package. Ecosystem-specific packages may wrap target-specific artifacts, but they must resolve the same runtime release and runtime identity when loaded into one process.
Services and layers from independent packages must compose into the same language-level Operator. A native layer must be able to wrap an operator from another package while preserving both Layer::apply_service and Layer::apply_context. That requires shared OpenDAL types, Tokio resources, HTTP and executor context, registry ownership, and handle identity.
If each package owned a separate OpenDAL runtime graph, Rust Operator, Layer, and OperationContext values could not cross package boundaries through a supported stable ABI. Cross-package composition would then require either a second stable operation interface or a complete operation adapter in each language. The project does not plan to maintain those larger interfaces, and a language adapter cannot preserve arbitrary native layer semantics.
The common runtime graph therefore owns composition machinery and extensions register release-specific factories into it. Service packages still own their configuration, URI interpretation, credentials, redaction, and package-local dependencies. “Common” means that all bindings in one process resolve one runtime identity rather than loading binding-private runtime graphs.
The protocol can remain small even though the runtime owns substantial native state. Independently built packages need only a stable way to acquire a runtime API, declare their required protocol level, inspect the runtime's supported protocol range, and invoke runtime-owned factories. The API implementation can expose a small bootstrap surface, such as an API lookup and a construction or registration entry point, backed by an extensible function table. The function signatures, table layouts, value grammar, handle ownership, and error rules together form the protocol.
Each binding adapter translates language values, async behavior, exceptions, and garbage collection into runtime-owned handles. It does not implement service configuration or native layer composition.
The adapter preserves each language's normal interface:
Operator, AsyncOperator, await, type stubs, and Python package discovery.Operator operations, require, keyword arguments, and Ruby exceptions.The project should share the native extension contract, not force one public language API across all three bindings. The JSON manifest, bootstrap metadata, configuration value grammar, factory semantics, error categories, and conformance suite are common. Constructor names, typing, duration syntax, blocking behavior, and package discovery remain binding-specific.
The runtime module owns process-scoped native resources. Python and Ruby initially use one NativeRuntime per loaded runtime module and process. The Node.js runtime calls that owner ProcessRuntime and places one EnvironmentAdapter per napi_env over it.
The runtime module owns:
The current OperatorRegistry is not sufficient for this role. It stores plain function pointers and replaces an existing scheme during registration. The extension registry must record package ownership and perform atomic, conflict-detecting registration.
Each installable package contains:
Registering the JSON manifest must not load the native library. The runtime activates the library when a caller first constructs a declared service or layer. This isolates HDFS and similar native dependencies.
A native extension contains exactly one service or layer factory. The package exports one generated, package-unique bootstrap symbol. The final artifact must hide every other package-local symbol and pass the native symbol isolation contract. Language-runtime initializers remain explicit allowlist exceptions.
The bootstrap exposes the package identity, target, and exact OpenDAL version through a small C calling convention. The compatibility contract compares a bounded JSON document with an exact-release C-layout descriptor instead of removing either encoding before prototyping. Only after all checks pass may the runtime enter the release-specific internal interface. That internal interface is not a stable Rust ABI.
Node.js packages are an exception to the unique-initializer rule when the native artifact is itself a Node-API addon: Node defines the addon initializer. The Node adapter must use a language-appropriate initializer and validate the same package identity in the returned bootstrap metadata.
A service factory accepts one of two request forms:
UriRequest { raw_uri, explicit_string_options, } ConfigRequest { structured_values, }
structured_values uses the language-neutral ConfigValue grammar defined by the compatibility contract. Each binding validates and converts its public values before invoking a native factory; a native package never receives Python, Ruby, or JavaScript objects.
The URI request preserves the original URI and explicit options. The package constructs its local OperatorUri and calls its own Configurator::from_uri. The runtime must not replace service-specific behavior with a universal precedence rule.
A layer factory accepts structured configuration and returns an asynchronous result:
create_layer(layer_id, structured_values) -> future<LayerHandle>
LayerHandle retains state and can be applied to more than one operator. Applying a layer returns a new operator. The runtime never sorts layers: later applications are outer layers, matching OpenDAL core behavior.
The selected release family has three package roles:
shared runtime package native runtime, protocol, registries, and handles main binding package Python, Ruby, or Node.js public API and adapter service/layer package manifest, language API, and native implementation
The main binding package declares the required_runtime_protocol that its adapter needs. The runtime package exposes minimum_runtime_protocol and runtime_protocol so the adapter can verify that requirement before using the runtime API. A service or layer package also depends on the runtime package, while its native artifact follows the exact-release extension compatibility rules. A minimal installation uses the runtime package and one main binding package; applications then add selected extension packages.
The proposed release process publishes these packages together. Coordinated publication gives every binding and official extension a consistent runtime implementation, build contract, and compatibility matrix. Keeping the implementation in its own package also makes upgrades and dependency diagnostics easier to manage than embedding equivalent native code independently in every binding.
Independently installable does not imply independently ABI-versioned: every native extension package must be rebuilt for every OpenDAL release, even when the internal interface appears unchanged.
The 1,000-manifest requirement measures registry behavior, not a commitment to publish 1,000 official package families. Before a split release, each ecosystem needs reserved package names, trusted publishing, coordinated release tooling, and rollback rules. OpenDAL can publish only supported high-value extensions while leaving the same SDK available to third parties.
Native extensions are trusted in-process code. Compatibility validation prevents accidental mismatches; it does not sandbox, authenticate, or constrain a malicious package.
OpenDAL still owns these in-scope properties:
The loader must use an explicit package-provided artifact path. It must not scan arbitrary library search paths and execute every matching file.
The design is not ready for an RFC decision until prototypes demonstrate:
hdfs-native.If those gates pass, the next deliverable is a 0000_*.md RFC. These pre-RFC documents remain supporting analysis rather than evidence that the candidate has already been accepted.