Apache OpenDAL™ Python Binding

This package provides a native Python binding for Apache OpenDAL™, a data access layer that allows you to access various storage services in a unified way.

Useful Links

• Documentation
• Examples
• Upgrade Guide

Features

• Unified API: Access S3, GCS, Azure Blob, HDFS, FTP, and more with the same set of commands.
• Native Performance: Built in Rust for high performance and safety.
• Async Support: First-class async API for modern Python applications.
• Easy to Use: Simple and intuitive API design.

Installation

Install the package directly from PyPI:

pip install opendal

Usage

Here are a few examples of how to use OpenDAL with different storage backends.

Local Filesystem (fs)

import opendal

# Initialize the operator for the local filesystem
op = opendal.Operator("fs", root="/tmp")

# Write data to a file
op.write("test.txt", b"Hello World")

# Read data from the file
content = op.read("test.txt")
print(op.read("test.txt"))

# Get metadata
metadata = op.stat("test.txt")
print(f"Content length: {metadata.content_length}") # Output: 11

Amazon S3

The API remains the same—just change the scheme and credentials.

import opendal

# Initialize the operator for S3
op = opendal.Operator(
    "s3",
    bucket="your_bucket_name",
    region="your_region",
    root="/path/to/root"
)

op.write("test.txt", b"Hello World")
print(op.read("test.txt"))
print(op.stat("test.txt").content_length)

Async Usage (s3)

OpenDAL also provides a fully asynchronous API.

import asyncio
import opendal

async def main():
    # Use AsyncOperator for async operations
    op = opendal.AsyncOperator("s3", root="/tmp", bucket="your_bucket_name", region="your_region")

    await op.write("test.txt", b"Hello World")
    print(await op.read("test.txt"))

asyncio.run(main())

Development

This project uses just as a command runner to simplify the development workflow.

1. Clone the repository and set up the environment:

   # This will create a virtual environment and install all dependencies
   just setup
   

2. Run tests:

   # Example: Run tests for the 'fs' operator
   OPENDAL_TEST=fs OPENDAL_FS_ROOT=/tmp just test
   

For a complete guide on building, testing, and contributing, please see our CONTRIBUTING.md file.

Used By

Check out the users list for more details on who is using OpenDAL.

License and Trademarks

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation.