Apache OpenDAL™ Python binding

Installation

pip install opendal

Local Usage

Developer must set two required arguments to work with files locally:

• scheme: which should be specified as fs
• root: where OpenDAl considers the root of the directory for operations will be.

For example in the following operator: opendal.Operator("fs", root="/foo")

OpenDAL considers /foo to be the root of all paths, and that means that we can access paths inside of /foo without specifying anything else. If /foo happens to contain the file baz.txt, we can simply call .list("/")

We can see this in the following example:

from opendal import Operator
from pathlib import Path
import os


os.makedirs("foo", exist_ok=True)

file = Path("foo", "bar.txt")
file.touch()

op = Operator("fs", root=Path("foo").name)
for file in op.list("/"):
    print(file)

When running we get the following output:

/
bar.txt

If we want full access to our file system, we can specify a root of "/", but note that for any operations you will always need to specify the full path to a file or directory.

Reading

There are two ways to read data using OpenDAL. One way is to use the read method on an operator:

from opendal import Operator
from pathlib import Path
import os


os.makedirs("foo", exist_ok=True)

file = Path("foo", "bar.txt")
file.touch()
file.write_text("baz")


op = Operator("fs", root=Path("foo").name)

data = op.read(file.name)
print(data)

Yields the following:

b'baz'

Note that the output is bytes, but we can get it as a string by simply calling .decode() on the data we read. All reads with OpenDAL return bytes.

Now lets use the open method on an operator:

from opendal import Operator
from pathlib import Path
import os


os.makedirs("foo", exist_ok=True)

file = Path("foo", "bar.txt")
file.touch()
file.write_text("baz")


op = Operator("fs", root=Path("foo").name)

with op.open(file.name, "rb") as f:
    print(f.read())

This again yields

b'baz'

Again, note that all reads with OpenDAL return bytes, so specifying a mode of "r" will raise an exception.

Writing

Now that we know how to read data, let's replace the Pathlib code above with OpenDAL using the write method:

from opendal import Operator
from pathlib import Path


op = Operator("fs", root=Path("foo").name)

op.write("baz.txt", "my amazing data".encode())

print(op.read("baz.txt").decode())

This yields the following:

my amazing data

And again, but using the open method:

from opendal import Operator
from pathlib import Path


op = Operator("fs", root=Path("foo").name)

with op.open("baz.txt", "wb") as f:
    f.write("my amazing data".encode())

print(op.read("baz.txt").decode())

This again yields:

my amazing data

Again, note that all writing happens in bytes and a mode of "w" will raise an exception.

Async

OpenDAL supports async operation on all operator methods. One can simply replace the Operator with an AsyncOperator and await on method calls. The below example illustrates this behavior:

Standard API:

import opendal

op = opendal.Operator("fs", root="/tmp")
op.write("test.txt", b"Hello World")
print(op.read("test.txt"))
print(op.stat("test.txt").content_length)

Async API equivalent:

import asyncio

async def main():
    op = opendal.AsyncOperator("fs", root="/tmp")
    await op.write("test.txt", b"Hello World")
    print(await op.read("test.txt"))

asyncio.run(main())