tag	a69e09905fa16f11d0846584d4cd6004d8ecb6ae
tagger	Xuanwo <github@xuanwo.io>	Sat Feb 04 18:05:38 2023 +0800
object	50c2608f5e481820038e5f8ad4596ef7cac6ac6e

commit	50c2608f5e481820038e5f8ad4596ef7cac6ac6e	[log] [tgz]
author	Xuanwo <github@xuanwo.io>	Sat Feb 04 18:05:17 2023 +0800
committer	GitHub <noreply@github.com>	Sat Feb 04 18:05:17 2023 +0800
tree	41899abcca4f3ea94bc764b2278e5f55d63f8a45
parent	75c037801c0228fd02e7589d7a68779f1bf6f9fa [diff]

tree: 41899abcca4f3ea94bc764b2278e5f55d63f8a45

README.md

OpenDAL

Open Data Access Layer: Access data freely, painlessly, and efficiently

User Guide
API Documentation: stable | main
Release notes

Services

azblob: Azure Storage Blob services.
azdfs: Azure Data Lake Storage Gen2 services. (As known as abfs)
fs: POSIX alike file system.
ftp: FTP and FTPS support.
gcs: Google Cloud Storage Service.
ghac: Github Action Cache Service.
hdfs: Hadoop Distributed File System(HDFS).
http: HTTP read-only services.
ipfs: InterPlanetary File System HTTP Gateway support.
ipmfs: InterPlanetary File System MFS API support.
memcached: Memcached service support.
memory: In memory backend.
moka: moka backend support.
obs: Huawei Cloud Object Storage Service (OBS).
oss: Aliyun Object Storage Service (OSS).
redis: Redis services support.
rocksdb: RocksDB services support.
s3: AWS S3 alike services.
webdav: WebDAV Service Support.

Features

Access data freely

Access different storage services in the same way
Behavior tests for all services
Cross language/project bindings (working on)

Access data painlessly

100% documents covered
Powerful Layers
Automatic retry support
Full observability support: logging, tracing, metrics.
Native decompress support
Native service-side encryption support

Access data efficiently

Zero cost: mapping to underlying API calls directly
Best effort: auto pick the best read/seek/next implementations based on services
Auto metadata reuse: avoid extra metadata calls

Quickstart

use anyhow::Result;
use futures::StreamExt;
use futures::TryStreamExt;
use opendal::ObjectReader;
use opendal::Object;
use opendal::ObjectMetadata;
use opendal::ObjectMode;
use opendal::Operator;
use opendal::services;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator
    let op = Operator::create(services::Fs::default())?.finish();

    // Create object handler.
    let o = op.object("/tmp/test_file");

    // Write data info object;
    o.write("Hello, World!").await?;

    // Read data from object;
    let bs = o.read().await?;

    // Read range from object;
    let bs = o.range_read(1..=11).await?;

    // Get object's path
    let name = o.name();
    let path = o.path();

    // Fetch more meta about object.
    let meta = o.metadata().await?;
    let mode = meta.mode();
    let length = meta.content_length();
    let content_md5 = meta.content_md5();
    let etag = meta.etag();

    // Delete object.
    o.delete().await?;

    // List dir object.
    let o = op.object("test_dir/");
    let mut os = o.list().await?;
    while let Some(entry) = os.try_next().await? {
        let path = entry.path();
        let mode = entry.mode().await?;
    }

    Ok(())
}

More examples could be found at Documentation.

Projects

Databend: A modern Elasticity and Performance cloud data warehouse.
GreptimeDB: An open-source, cloud-native, distributed time-series database.
deepeth/mars: The powerful analysis platform to explore and visualize data from blockchain.
mozilla/sccache: sccache is ccache with cloud storage

Contributing

Check out the CONTRIBUTING.md guide for more details on getting started with contributing to this project.

Getting help

Submit issues for bug report or asking questions in discussion.