| "use strict";(self.webpackChunkopendal_website=self.webpackChunkopendal_website||[]).push([[8749],{1895:e=>{e.exports=JSON.parse('{"archive":{"blogPosts":[{"id":"/2025/03/01/2025-roadmap","metadata":{"permalink":"/blog/2025/03/01/2025-roadmap","editUrl":"https://github.com/apache/opendal/tree/main/website/blog/2025-03-01-2025-roadmap/index.md","source":"@site/blog/2025-03-01-2025-roadmap/index.md","title":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","description":"Hi everyone, long time no see. Although we have been actively developing OpenDAL and consistently releasing updates, we haven\u2019t provided a clear overview of where OpenDAL is heading. This post aims to address that. Here, we will discuss OpenDAL\'s current position and the future directions we are moving toward.","date":"2025-03-01T00:00:00.000Z","tags":[{"inline":true,"label":"announcement","permalink":"/blog/tags/announcement"}],"readingTime":4.095,"hasTruncateMarker":true,"authors":[{"name":"Xuanwo","url":"https://github.com/Xuanwo","key":"xuanwo","page":null}],"frontMatter":{"title":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","date":"2025-03-01T00:00:00.000Z","tags":["announcement"],"authors":"xuanwo"},"unlisted":false,"nextItem":{"title":"Apache OpenDAL\u2122 Beijing Meetup 1st","permalink":"/blog/2024/09/09/apache-opendal-meetup-beijing-1st"}},"content":"Hi everyone, long time no see. Although we have been actively developing OpenDAL and consistently releasing updates, we haven\u2019t provided a clear overview of where OpenDAL is heading. This post aims to address that. Here, we will discuss OpenDAL\'s current position and the future directions we are moving toward.\\n\\nI hope this post serves as a guide for our development, encourages more people to contribute, and ultimately helps achieve the vision of OpenDAL.\\n\\n\x3c!--truncate--\x3e\\n\\n## What\'s OpenDAL?\\n\\nApache OpenDAL (`/\u02c8o\u028a.p\u0259n.d\xe6l/`, pronounced \\"OH-puhn-dal\\") is an Open Data Access Layer that enables seamless interaction with diverse storage services. Our VISION is [**One Layer, All Storage**](https://opendal.apache.org/vision), and our core principles are Open Community, Solid Foundation, Fast Access, Object Storage First, and Extensible Architecture.\\n\\nWe are building:\\n\\n- [A core library built in Rust](https://crates.io/crates/opendal) that supports all services at zero cost and offers unified retry, concurrency, logging, tracing, metrics, timeout and more layers.\\n- Language bindings for [Python](https://pypi.org/project/opendal/), [Java](https://central.sonatype.com/artifact/org.apache.opendal/opendal), [Node.js](https://www.npmjs.com/package/opendal), C/C++, and more.\\n- Integrations with various frameworks, including [Parquet](https://crates.io/crates/parquet-opendal), [FUSE](https://crates.io/crates/fuse3_opendal), [DAV server](https://github.com/messense/dav-server-rs), and others.\\n- Binaries for different use cases, such as [CLI](https://crates.io/crates/oli) and [FUSE](https://crates.io/crates/ofs).\\n\\nIn short, through OpenDAL, users can access ALL storage services within ONE layer.\\n\\n## Where\'s OpenDAL?\\n\\nOpenDAL\'s rust core has already released 143 versions, has [67 reverse dependencies](https://crates.io/crates/opendal/reverse_dependencies) listed on [crates.io](http://crates.io/), and is used by [612 projects](https://github.com/apache/opendal/network/dependents) as recorded on GitHub.\\n\\nOpenDAL\'s production users include databases such as [Databend](https://github.com/databendlabs/databend), [GreptimeDB](https://github.com/GreptimeTeam/greptimedb), and [RisingWave](https://github.com/risingwavelabs/risingwave), as well as tools like [Loco](https://loco.rs/), [sccache](https://github.com/mozilla/sccache), and [Vector](https://vector.dev/).\\n\\n\\n\\nApart from OpenDAL\'s Rust core, its various language bindings have also seen significant growth over the past year. Take python binding as an example. [Dify](https://github.com/langgenius/dify/), an LLM app development platform, is using OpenDAL to access different storage services.\\n\\n\\n\\n## What\'s next for OpenDAL?\\n\\nThe following is the famous technology adoption lifecycle curve. If I were to indicate the position of OpenDAL, I would say it is at the end of the Innovators stage and moving toward the Early Adopters stage.\\n\\n\\n(picture from TechTarget [technology adoption lifecycle](https://www.techtarget.com/searchcio/definition/technology-adoption-lifecycle))\\n\\nInnovators are adopting OpenDAL. Projects like Databend, RisingWave, GreptimeDB, and sccache have been using OpenDAL in production for years. However, early adopters are still hesitant to use OpenDAL\u2014and they have valid reasons.\\n\\nFor examples:\\n\\n- OpenDAL hasn\'t reached version 1.0 yet and still introduces breaking changes from time to time, sometimes even requiring code modifications for an OpenDAL upgrade. This also creates a burden for libraries that depend on OpenDAL, as every breaking change affects them as well.\\n- OpenDAL lacks comprehensive documentation, particularly for its bindings in Python and Java. Users need to invest significant effort and love to integrate OpenDAL bindings into their projects.\\n- OpenDAL lacks some important features, such as checksum support, caching, metrics of underlying http requests and initialization from a URI.\\n\\nI believe we should perfect production adoption in 2025 to get OpenDAL ready for early adopters. Only in this way can we implement our vision: **One Layer, All Storage.**\\n\\n## What to do in 2025?\\n\\nOur plans for 2025 include the following:\\n\\n### Features Needed in Production\\n\\nIn 2025, we plan to implement the following features that are importmant for production adoption:\\n\\n- [Context](https://github.com/apache/opendal/issues/5480): Introduce context in OpenDAL so that services and layers can share the same context. This enables users to gain deeper insights into OpenDAL\'s internal operations by incorporating metrics, logging, and tracing into the underlying HTTP client we use.\\n- [Versioning](https://github.com/apache/opendal/issues/2611): Introduce full file versioning support in OpenDAL, enabling users to read, write, delete, list, and restore versioned files. This functionality will allow users to recover mistakenly deleted files and facilitate disaster recovery.\\n- [Checksum](https://github.com/apache/opendal/issues/5549): Introduce end-to-end checksum support in OpenDAL, enabling users to perform checksums during reading and writing without worrying about bit flips in memory or over the network.\\n- [Caching](https://github.com/apache/opendal/issues/5678): Provide high-quality built-in cache support in OpenDAL while ensuring users have the flexibility to implement their own caching logic.\\n- [Initialization From URI](https://github.com/apache/opendal/issues/3022): Allow users to initialize OpenDAL from a URI string, making it easier to configure and use OpenDAL in various environments.\\n\\n### Improvements Needed for Production\\n\\nIn 2025, we plan to improve the following aspects of OpenDAL to ensure users can confidently use it in production.\\n\\n- Documentation: Improve the documentation for OpenDAL, particularly for bindings like Python, Node.js, and Java. The first step is to generate well-structured documentation for the configuration values of each service.\\n- Communitation: Bring Back Our Community Meetings. OpenDAL used to hold regular tri-weekly meetings, but we have been unable to maintain them in the last year. We now plan to revive these meetings and encourage more face-to-face discussions in the coming years.\\n\\n## Conclusion\\n\\n2025 marks the third year of the OpenDAL community. A huge thanks to all OpenDAL contributors and users for helping us reach this milestone. There\'s still a long way to go to achieve our vision, and we invite you all to join us on this incredible journey!\\n\\n[Discuss about this post here](https://github.com/apache/opendal/discussions/5679)"},{"id":"/2024/09/09/apache-opendal-meetup-beijing-1st","metadata":{"permalink":"/blog/2024/09/09/apache-opendal-meetup-beijing-1st","editUrl":"https://github.com/apache/opendal/tree/main/website/blog/2024-09-09-apache-opendal-meetup-beijing-1st/index.md","source":"@site/blog/2024-09-09-apache-opendal-meetup-beijing-1st/index.md","title":"Apache OpenDAL\u2122 Beijing Meetup 1st","description":"Hello, everyone!","date":"2024-09-09T00:00:00.000Z","tags":[{"inline":true,"label":"announcement","permalink":"/blog/tags/announcement"}],"readingTime":1.72,"hasTruncateMarker":true,"authors":[{"name":"Xuanwo","url":"https://github.com/Xuanwo","key":"xuanwo","page":null}],"frontMatter":{"title":"Apache OpenDAL\u2122 Beijing Meetup 1st","date":"2024-09-09T00:00:00.000Z","tags":["announcement"],"authors":"xuanwo"},"unlisted":false,"prevItem":{"title":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","permalink":"/blog/2025/03/01/2025-roadmap"},"nextItem":{"title":"Apache OpenDAL\u2122 participates in Open Source Promotion Plan 2024","permalink":"/blog/apache-opendal-participates-in-ospp-2024"}},"content":"Hello, everyone!\\n\\nOn behalf of the OpenDAL community, I\'m pleased to announce that the first Apache OpenDAL\u2122 Beijing Meetup will be held on September 14th.\\n\\n\x3c!--truncate--\x3e\\n\\nYou are welcome to register for this event at: https://docs.qq.com/form/page/DYnpJWFFsSWVFeGRj\\n\\nThe event will take place at [\u5317\u4eac\u5e02\u671d\u9633\u533a\u5317\u8fb0\u897f\u8def 8 \u53f7\u5317\u8fb0\u4e16\u7eaa\u4e2d\u5fc3A\u5ea712\u5c421215\u623f\u95f4\uff08\u56fd\u5bb6\u4f1a\u8bae\u4e2d\u5fc3\u897f\u9762\u7684\u5199\u5b57\u697c\uff09](https://maps.app.goo.gl/4Tr7JCJ5HdCoVGdeA) on the afternoon of September 14th.\\n\\n## Agenda\\n\\n- 12:00 - 13:00: Lunch / Networking (food provided)\\n- 13:00 - 14:00: Welcome / Introductions\\n- 14:00 - 15:00: Talks\\n- 15:00 - 15:30: Coffee / Email Break\\n- 15:30 - 16:30: Breakouts / Discussions\\n- 16:30 - 17:00: Close / Conclusion\\n\\n## Welcome\\n\\nBrief introductory remarks, thank to the sponsor, outline of the day, participant introductions (name, github handle, optional affiliation, optional reason you came to the meetup)\\n\\n## Talks\\n\\nFive 10-15 minute talks about opendal.\\n\\n- [@tisonkun](https://github.com/tisonkun/) about OpenDAL Community Growth\\n- [@WenyXu](https://github.com/WenyXu/) for using OpenDAL in [GreptimeDB](https://github.com/GreptimeTeam/greptimedb)\\n- [@wcy-fdu](https://github.com/wcy-fdu/) for using OpenDAL in [RisingWave](https://github.com/risingwavelabs/risingwave)\\n- [@drmingdrmer](https://github.com/wcy-fdu/) for using OpenDAL in [OpenRaft](https://github.com/datafuselabs/openraft)\\n- [@Xuanwo](https://github.com/Xuanwo/) about OpenDAL Core 1.0\\n\\n## Breakouts / Discussions\\n\\nThe idea is that we will break into small groups for 3 x 20 minute sessions. I will ring a gong or something similar every 20 minutes and you should find a new group to discuss.\\n\\nPotential seed topics (but obviously feel free to discuss anything):\\n\\n- How do we make it easier for people to build systems on OpenDAL? What are the current barriers to community growth as you see them?\\n- How do we make it easier for people to contribute back to OpenDAL? What are the current barriers to contribution\\n- What features would you like to see / not see\\n\\nIdeally, in each session, one or more people would volunteer to write up any insights they got from the discussions, and post them to the overall ticket.\\n\\n---\\n\\nLooking forward to meet you in this meetup!"},{"id":"apache-opendal-participates-in-ospp-2024","metadata":{"permalink":"/blog/apache-opendal-participates-in-ospp-2024","editUrl":"https://github.com/apache/opendal/tree/main/website/blog/2024-04-30-apache-opendal-participates-in-ospp-2024/index.md","source":"@site/blog/2024-04-30-apache-opendal-participates-in-ospp-2024/index.md","title":"Apache OpenDAL\u2122 participates in Open Source Promotion Plan 2024","description":"Hello, everyone!","date":"2024-04-30T00:00:00.000Z","tags":[{"inline":true,"label":"announcement","permalink":"/blog/tags/announcement"}],"readingTime":1.815,"hasTruncateMarker":true,"authors":[{"name":"Xuanwo","url":"https://github.com/Xuanwo","key":"xuanwo","page":null}],"frontMatter":{"title":"Apache OpenDAL\u2122 participates in Open Source Promotion Plan 2024","date":"2024-04-30T00:00:00.000Z","slug":"apache-opendal-participates-in-ospp-2024","tags":["announcement"],"authors":"xuanwo"},"unlisted":false,"prevItem":{"title":"Apache OpenDAL\u2122 Beijing Meetup 1st","permalink":"/blog/2024/09/09/apache-opendal-meetup-beijing-1st"},"nextItem":{"title":"Apache OpenDAL\u2122 participates in Google Summer of Code 2024","permalink":"/blog/apache-opendal-participates-in-gsoc-2024"}},"content":"Hello, everyone!\\n\\nWe\'re writing this blog post to announce that the Apache OpenDAL\u2122 Project will be participating in [Open Source Promotion Plan (OSPP) 2024](https://summer-ospp.ac.cn/). If you\'re not eligible or interested in participating in OSPP, then most of this post likely isn\'t relevant to you; if you are, this should contain some useful information and links.\\n\\n\x3c!--truncate--\x3e\\n\\nOpen Source Promotion Plan is a summer program organized by the Institute of Software Chinese Academy of Sciences and long-term supported by the Open Source Software Supply Chain Promotion Plan. It aims to encourage college students to actively participate in the maintenance and development of open source software, promote the vigorous development of open source software communities, and build the open source software supply chain together.\\n\\nThe OSPP applicants now have several weeks to send project proposals to organizations that appeal to them. If their project proposal is accepted, they will embark on a 12-week journey during which they will try to complete their proposed project under the guidance of an assigned mentor.\\n\\nWe have prepared [a list of project ideas](https://github.com/apache/opendal/issues/4554) that can serve as inspiration for potential OSPP contributors that would like to send a project proposal to the OpenDAL project. You can try to find mentors on the [maillist](mailto:dev@opendal.apache.org) or [Discord](https://discord.gg/XQy8yGR2dg). We have also prepared [a proposal guide](https://opendal.apache.org/community/events/ospp-proposal-guide) that should help you with preparing your project proposals.\\n\\nYou can start discussing the project ideas with OpenDAL Project maintainers immediately. The project proposal application period starts on April 30, 2024, and ends on June 4, 2024. Take note of that deadline, as there will be no extensions!\\n\\nIf you are interested in contributing to the OpenDAL Project, we encourage you to check out our project idea list and send us a OSPP project proposal! Of course, you are also free to discuss these projects and/or try to move them forward even if you do not intend to (or cannot) participate in OSPP. We welcome all contributors to OpenDAL, as there is always enough work to do.\\n\\nWe are excited about this event. Hoping you all feel the same way!\\n\\n> This announcement is inspired a lot by the [Rust participates in Google Summer of Code 2024](https://blog.rust-lang.org/2024/02/21/Rust-participates-in-GSoC-2024.html)."},{"id":"apache-opendal-participates-in-gsoc-2024","metadata":{"permalink":"/blog/apache-opendal-participates-in-gsoc-2024","editUrl":"https://github.com/apache/opendal/tree/main/website/blog/2024-02-28-apache-opendal-participates-in-gsoc-2024/index.md","source":"@site/blog/2024-02-28-apache-opendal-participates-in-gsoc-2024/index.md","title":"Apache OpenDAL\u2122 participates in Google Summer of Code 2024","description":"Hello, everyone!","date":"2024-02-28T00:00:00.000Z","tags":[{"inline":true,"label":"announcement","permalink":"/blog/tags/announcement"}],"readingTime":1.94,"hasTruncateMarker":true,"authors":[{"name":"Xuanwo","url":"https://github.com/Xuanwo","key":"xuanwo","page":null}],"frontMatter":{"title":"Apache OpenDAL\u2122 participates in Google Summer of Code 2024","date":"2024-02-28T00:00:00.000Z","slug":"apache-opendal-participates-in-gsoc-2024","tags":["announcement"],"authors":"xuanwo"},"unlisted":false,"prevItem":{"title":"Apache OpenDAL\u2122 participates in Open Source Promotion Plan 2024","permalink":"/blog/apache-opendal-participates-in-ospp-2024"},"nextItem":{"title":"Apache OpenDAL\u2122 is now Graduated","permalink":"/blog/apache-opendal-graduated"}},"content":"Hello, everyone!\\n\\nWe\'re writing this blog post to announce that the Apache OpenDAL\u2122 Project will be participating in [Google Summer of Code (GSoC) 2024](https://summerofcode.withgoogle.com/). If you\'re not eligible or interested in participating in GSoC, then most of this post likely isn\'t relevant to you; if you are, this should contain some useful information and links.\\n\\n\x3c!--truncate--\x3e\\n\\nGoogle Summer of Code (GSoC) is an annual global program organized by Google that aims to bring new contributors to the world of open-source. The program pairs organizations (such as the OpenDAL Project) with contributors (usually students), with the goal of helping the participants make meaningful open-source contributions under the guidance of experienced mentors.\\n\\nGoogle is sponsoring the [2024 Summer of Code](https://summerofcode.withgoogle.com/) and [The Apache Software Foundation](https://www.apache.org/) (ASF) registered as a mentoring organization. The GSoC applicants now have several weeks to send project proposals to organizations that appeal to them. If their project proposal is accepted, they will embark on a 12-week journey during which they will try to complete their proposed project under the guidance of an assigned mentor.\\n\\nWe have prepared [a list of project ideas](https://github.com/apache/opendal/issues/4131) that can serve as inspiration for potential GSoC contributors that would like to send a project proposal to the OpenDAL project. However, applicants can also come up with their own project ideas. You can discuss project ideas or try to find mentors on the [maillist](mailto:dev@opendal.apache.org) or [Discord](https://discord.gg/XQy8yGR2dg). We have also prepared [a proposal guide](https://opendal.apache.org/community/events/gsoc-proposal-guide) that should help you with preparing your project proposals.\\n\\nYou can start discussing the project ideas with OpenDAL Project maintainers immediately. The project proposal application period starts on March 18, 2024, and ends on April 2, 2024 at 18:00 UTC. Take note of that deadline, as there will be no extensions!\\n\\nIf you are interested in contributing to the OpenDAL Project, we encourage you to check out our project idea list and send us a GSoC project proposal! Of course, you are also free to discuss these projects and/or try to move them forward even if you do not intend to (or cannot) participate in GSoC. We welcome all contributors to OpenDAL, as there is always enough work to do.\\n\\nWe are excited about this event. Hoping you all feel the same way!\\n\\n> This announcement is inspired a lot by the [Rust participates in Google Summer of Code 2024](https://blog.rust-lang.org/2024/02/21/Rust-participates-in-GSoC-2024.html)."},{"id":"apache-opendal-graduated","metadata":{"permalink":"/blog/apache-opendal-graduated","editUrl":"https://github.com/apache/opendal/tree/main/website/blog/2024-01-22-apache-opendal-graduated/index.md","source":"@site/blog/2024-01-22-apache-opendal-graduated/index.md","title":"Apache OpenDAL\u2122 is now Graduated","description":"Hello, everyone! I\'m happy to announce that Apache OpenDAL\u2122 has graduated from the Apache Incubator to become a Top-Level Project of the Apache Software Foundation.","date":"2024-01-22T00:00:00.000Z","tags":[{"inline":true,"label":"announcement","permalink":"/blog/tags/announcement"}],"readingTime":5.89,"hasTruncateMarker":true,"authors":[{"name":"Xuanwo","url":"https://github.com/Xuanwo","key":"xuanwo","page":null}],"frontMatter":{"title":"Apache OpenDAL\u2122 is now Graduated","date":"2024-01-22T00:00:00.000Z","slug":"apache-opendal-graduated","tags":["announcement"],"authors":"xuanwo"},"unlisted":false,"prevItem":{"title":"Apache OpenDAL\u2122 participates in Google Summer of Code 2024","permalink":"/blog/apache-opendal-participates-in-gsoc-2024"},"nextItem":{"title":"OwO #1: The v0.40 Release","permalink":"/blog/owo-1"}},"content":"Hello, everyone! I\'m happy to announce that [Apache OpenDAL\u2122](https://opendal.apache.org/) has graduated from the [Apache Incubator](https://incubator.apache.org/) to become a Top-Level Project of [the Apache Software Foundation](https://apache.org/).\\n\\n\x3c!--truncate--\x3e\\n\\n## What\'s Apache OpenDAL?\\n\\n**Apache OpenDAL** is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way. Our VISION is **access data freely**.\\n\\nOpenDAL could be used as a **better** SDK for your storage services: A SDK with native integration of [retry](https://opendal.apache.org/docs/rust/opendal/layers/struct.RetryLayer.html), [logging](https://opendal.apache.org/docs/rust/opendal/layers/struct.LoggingLayer.html), [metrics](https://opendal.apache.org/docs/rust/opendal/layers/struct.MetricsLayer.html), [tracing](https://opendal.apache.org/docs/rust/opendal/layers/struct.TracingLayer.html), [timeout](https://opendal.apache.org/docs/rust/opendal/layers/struct.TimeoutLayer.html), [throttle](https://opendal.apache.org/docs/rust/opendal/layers/struct.ThrottleLayer.html), and [more](https://opendal.apache.org/docs/rust/opendal/layers/index.html).\\n\\nOpenDAL could be used as a **super** connector for your storage services: A connector that supports all kinds of storage services from Object Storage (s3, gcs, azblob), File Storage (fs, azdls, hdfs), Consumer Cloud Storage (gdrive, onedrive), Key-Value Storage (rocksdb, sled) to Cache Storage (memcached, moka).\\n\\nOpenDAL could be used as an **elegant** client for your storage services: A client with well designed API and many language bindings: [Rust](https://crates.io/crates/opendal), C, Cpp, Dotnet, Go, Haskell, [Java](https://mvnrepository.com/artifact/org.apache.opendal/opendal-java), Lua, [Node.js](https://www.npmjs.com/package/opendal), OCaml, PHP, [Python](https://pypi.org/project/opendal/), Ruby, Swift and Zig.\\n\\nNeed to access data? Give OpenDAL a try!\\n\\n```rust\\nasync fn main() -> Result<()> {\\n // Init s3 service.\\n let mut builder = services::S3::default();\\n builder.bucket(\\"test\\");\\n\\n // Init an operator\\n let op = Operator::via_map(builder)?\\n // Add logging\\n .layer(LoggingLayer::default())\\n .finish();\\n\\n // Write data\\n op.write(\\"hello.txt\\", \\"Hello, World!\\").await?;\\n\\n // Read data\\n let bs = op.read(\\"hello.txt\\").await?;\\n\\n // Fetch metadata\\n let meta = op.stat(\\"hello.txt\\").await?;\\n let mode = meta.mode();\\n let length = meta.content_length();\\n\\n // Delete\\n op.delete(\\"hello.txt\\").await?;\\n\\n Ok(())\\n}\\n```\\n\\n## What\'s the ASF?\\n\\nThe Apache Software Foundation (ASF) is a nonprofit corporation to support a number of open-source software projects. The Apache Software Foundation exists to provide software for the public good. We believe in the power of community over code, known as The Apache Way. Thousands of people around the world contribute to ASF open source projects every day. \\n\\nThe OpenDAL Community believes [the apache way](https://www.apache.org/theapacheway/) that:\\n\\n- *Earned Authority*: all individuals are given the opportunity to participate, but their influence is based on publicly earned merit \u2013 what they contribute to the community.\\n- *Community of Peers*: individuals participate at the ASF, not organizations.\\n- *Open Communications*: as a virtual organization, the ASF requires all communications related to code and decision-making to be publicly accessible to ensure asynchronous collaboration, as necessitated by a globally-distributed community.\\n- *Consensus Decision Making*: Apache Projects are overseen by a self-selected team of active volunteers who are contributing to their respective projects.\\n- *Responsible Oversight*: The ASF governance model is based on trust and delegated oversight.\\n\\nThe original creators [Databend](https://github.com/datafuselabs/databend/) chosen to contribute OpenDAL to the ASF, embracing the Apache way through [joining the incubator program](https://opendal.apache.org/blog/opendal-entered-apache-incubator).\\n\\n## What\'s graduation?\\n\\nIn the [Apache Incubator](https://incubator.apache.org/), the OpenDAL community is learning the Apache Way through daily development activities, growing its community and producing Apache releases.\\n\\nDuring the incubation, we:\\n\\n- Consist of 19 committers, including mentors, with 12 serving as PPMC members.\\n- Boast 164 contributors.\\n- Made 9 releases\u2014averaging at least one per month.\\n- Had 7 different release managers to date.\\n- Used by 10 known entities and is a dependency for 263 GitHub projects and 18 crates.io packages.\\n- Opened 1,200+ issues with 1,100+ successfully resolved.\\n- Submitted a total of 2,400+ PRs, most of them have been merged or closed.\\n\\nThe graduation signifies that the OpenDAL Community is recognized as a [mature](https://opendal.apache.org/community/maturity) community, which entails:\\n\\n- CODE: OpenDAL is an [Apache 2.0 licensed](https://github.com/apache/opendal/blob/main/LICENSE) open-source project with [accessible, buildable code](https://github.com/apache/opendal) on GitHub, featuring [a traceable history and authenticated code provenance](https://github.com/apache/opendal/commits/main/). \\n- LICENSE: OpenDAL maintains [open-source compliance](https://github.com/apache/opendal/blob/main/DEPENDENCIES.md) for all code and dependencies, requires contributor agreements, and clearly documents copyright ownership.\\n- Releases: OpenDAL offers standardized, committee-approved [source code releases](https://downloads.apache.org/opendal/) with secure signatures, provides convenience binaries, and has [a well-documented, repeatable release process](https://opendal.apache.org/community/committers/release).\\n- Quality: OpenDAL is committed to code quality transparency, prioritizes security with quick issue responses, ensures backward compatibility with clear documentation, and actively addresses bug reports in a timely manner.\\n- Community: OpenDAL offers [a comprehensive homepage](https://opendal.apache.org/), welcomes diverse contributions, promotes a meritocratic approach for active contributors, operates on community consensus, and ensures timely responses to user queries through various channels. \\n- Consensus: OpenDAL has a [public list of key decision-makers](https://projects.apache.org/committee.html?opendal) and uses a consensus approach for decisions, documented on its [main communication channel](https://lists.apache.org/list.html?dev@opendal.apache.org). It follows standard voting rules and records all important discussions in writing.\\n- Independence: OpenDAL is independent, with contributors from various companies acting on their own, not as representatives of any organization.\\n\\n## What\'s next?\\n\\nAfter graduation, OpenDAL Community will continue to focus on the following aspects under the VISION: **access data freely**.\\n\\n### More Stable Services\\n\\nOpenDAL now supports 59 services, although only some of them are *stable*.\\n\\n*stable* for OpenDAL means that\\n\\n- Have integration tests covered.\\n- Have at least one production user.\\n\\nThe *stable* service established a feedback loop between the OpenDAL community and its users. Users can submit bug reports or feature requests to the OpenDAL community, which in turn can enhance the service using this feedback while ensuring existing features remain intact.\\n\\nAfter graduation, the OpenDAL community will focus on improving the stability of current services instead of just expanding our offerings.\\n\\nWe plan to:\\n\\n- Add features users wanted to services like [file version](https://github.com/apache/opendal/issues/2611), [concurrently list](https://github.com/apache/opendal/issues/3977) and [glob pattern](https://github.com/apache/opendal/issues/1251).\\n- Add integration tests for newly added services.\\n\\n### More Useful Documents\\n\\nOpenDAL have good docs for its rust core, but not for other language bindings.\\n\\nThe lack of comprehensive documentation makes OpenDAL challenging for users to operate in Java or Python. Without user feedback, the community is unable to enhance this documentation, leading to a detrimental cycle that must be broken.\\n\\nAfter graduation, the OpenDAL community will improve the documentation of other language bindings.\\n\\nWe plan to:\\n\\n- Introduce code generation to automatically create documentation for the service builder due to its numerous configurations.\\n- Add more API Docs and examples for other language bindings.\\n\\nOpenDAL have good docs for its public API, but not for its internal design.\\n\\nOpenDAL is proud of its elegant design, but it is not well documented. This makes it difficult for new contributors to understand the codebase and make contributions.\\n\\nAfter graduation, the OpenDAL community will improve the documentation of its internal design.\\n\\nWe plan to:\\n\\n- Optimize the codebase to make it easier to understand.\\n- Add more blog posts to explain the design of OpenDAL.\\n\\n### More Production Users\\n\\nOpenDAL requires more production users, as they are vital to the success of our project. Increased user production leads to more valuable feedback, a more engaged contributor base, and a stronger community. We\'ve started the initial loop; let\'s expand it!\\n\\nAfter graduation, the OpenDAL community will focus on attracting more production users.\\n\\nWe plan to:\\n\\n- Optimize the feature set for adoption like [uri initiation](https://github.com/apache/opendal/issues/3022) and [config](https://github.com/apache/opendal/issues/3240).\\n- Expand more ways to use OpenDAL via [fuse](https://github.com/apache/opendal/tree/main/bin/ofs), [cli](https://github.com/apache/opendal/tree/main/bin/oli), [S3/WebDAV API](https://github.com/apache/opendal/tree/main/bin/oli), [object_store binding](https://github.com/apache/opendal/tree/main/integrations/object_store).\\n\\n## Conclusion\\n\\nThe OpenDAL Community aims to create a world where users can freely access data across any storage service in any manner they choose. Graduation is just the beginning\u2014let\'s work together to make our VISION a reality!"},{"id":"owo-1","metadata":{"permalink":"/blog/owo-1","editUrl":"https://github.com/apache/opendal/tree/main/website/blog/2023-09-14-owo-1/index.md","source":"@site/blog/2023-09-14-owo-1/index.md","title":"OwO #1: The v0.40 Release","description":"OwO (Outcome, Working, Outlook) is an Apache OpenDAL\u2122 release blog series, where we share the current work status and future plans.","date":"2023-09-15T00:00:00.000Z","tags":[{"inline":true,"label":"owo","permalink":"/blog/tags/owo"}],"readingTime":3.92,"hasTruncateMarker":true,"authors":[{"name":"Xuanwo","url":"https://github.com/Xuanwo","key":"xuanwo","page":null}],"frontMatter":{"title":"OwO #1: The v0.40 Release","date":"2023-09-15T00:00:00.000Z","slug":"owo-1","tags":["owo"],"authors":"xuanwo"},"unlisted":false,"prevItem":{"title":"Apache OpenDAL\u2122 is now Graduated","permalink":"/blog/apache-opendal-graduated"},"nextItem":{"title":"Apache OpenDAL\u2122 Internal: Data Reading","permalink":"/blog/how-opendal-read-data"}},"content":"> OwO (Outcome, Working, Outlook) is an Apache OpenDAL\u2122 release blog series, where we share the current work status and future plans.\\n\\nHello! It\'s been a while since our last update. We\'ve been hard at work determining the optimal way to implement new features and improvements. We\'re thrilled to announce that we\'ll soon be releasing v0.40.\\n\\n\x3c!--truncate--\x3e\\n\\nThis post is structured into three main sections:\\n\\n- Outcome (1st `O` in `OwO`): Summarizes the key accomplishments in the v0.40 release.\\n- Working (the `w` in `OwO`): Provides an update on our current work.\\n- Outlook (2nd `O` in `OwO`): Discusses what lies ahead for OpenDAL.\\n\\n## Outcome\\n\\nOpenDAL now comprises four primary components:\\n\\n- Core: The core library written in Rust.\\n- Bindings: Language bindings powered by the OpenDAL Rust core.\\n- Applications: Applications built using the OpenDAL Rust core.\\n- Integrations: Collaborations with other projects.\\n\\n### Core\\n\\n#### Unifying Append and Write Functions\\n\\nOpenDAL has supported `append` operations since `v0.36`. We\'ve found, however, that this led to significant duplication between append and write. As a result, we\'ve streamlined the two functionalities into a single write function. Our users can now:\\n\\n```rust\\nlet mut w = op.writer_with(\\"test.txt\\").append(true).await?;\\nw.write(content_a).await?;\\nw.write(content_b).await?;\\nw.close().await?;\\n```\\n\\nThis way, users can reuse the `Writer` in their own logic without handling `append` separately.\\n\\n#### New Lister API\\n\\nTo improve API consistency, we\'ve made some adjustments to our listing functions. We\'ve added `list` and `list_with` methods that perform single operations and renamed the original `list` to `lister` and `lister_with`.\\n\\n```rust\\n// Old API\\nlet lister: Lister = op.list(\\"dir\\").await?;\\n\\n// New API\\nlet entries: Vec<Entry> = op.list(\\"dir\\").await?;\\nlet lister: Lister = op.lister(\\"dir\\").await?;\\n```\\n\\nThis brings uniformity to our API offerings.\\n\\n#### List With Metakey\\n\\nTo speed up list operations, OpenDAL can now fetch and store metadata during the listing process. This eliminates the need for separate metadata calls:\\n\\n```rust\\nlet entries: Vec<Entry> = op\\n .list_with(\\"dir/\\")\\n .metakey(Metakey::ContentLength | Metakey::ContentType).await?;\\n\\n// Use the metadata directly!\\nlet meta: &Metadata = entries[0].metadata();\\n```\\n\\nThis makes metadata retrieval more intuitive.\\n\\n#### Buffered Writer\\n\\nWe\'ve added general buffer support to optimize writing operations.\\n\\n```rust\\nlet w = op.writer_with(\\"path/to/file\\").buffer(8 * 1024 * 1024).await?\\n```\\n\\n#### Others\\n\\nOther improvements in the core library can be found in our [CHANGELOG](https://github.com/apache/opendal/blob/main/CHANGELOG.md).\\n\\n### Bindings\\n\\n#### C++\\n\\n[`opendal-cpp`](https://github.com/apache/opendal/tree/main/bindings/cpp) is ready for its first release! Welcome to check it out and give us some feedback.\\n\\n#### Haskell\\n\\n[`opendal-hs`](https://github.com/apache/opendal/tree/main/bindings/haskell) is ready for its first release! Welcome to check it out and give us some feedback.\\n\\n#### Java\\n\\n[`opendal-java`](https://github.com/apache/opendal/tree/main/bindings/java) enabled more available services in this release, allowing user to visit services like `redis` that not enabled by default in rust core. And `opendal-java` enabled blocking layer to allow users visit services like `s3` in blocking way.\\n\\nWelcome to integrate `opendal-java` into your project and give us some feedback.\\n\\n#### New bindings!\\n\\n- [`opendal-dotnet`](https://github.com/apache/opendal/tree/main/bindings/dotnet)\\n- [`opendal-php`](https://github.com/apache/opendal/tree/main/bindings/php)\\n\\n### Applications\\n\\n#### oay\\n\\n[oay](https://github.com/apache/opendal/tree/main/bin/oay) is OpenDAL Gateway that allows users to access OpenDAL services via existing protocols like `s3` and `webdav`. It works like a proxy that forwarding requests to OpenDAL services.\\n\\nIn this release, we implement basic `webdav` support. Users can convert any storage services to a webdav server!\\n\\n#### oli\\n\\n[oli](https://github.com/apache/opendal/tree/main/bin/oay) is OpenDAL CLI that allows users to access storage services via CLI like `s3cmd` and `gcloud` does.\\n\\nWe fixed some experience issues in this release and improved some docs. Welcome to try it out and give us some feedback.\\n\\n### Integrations\\n\\n#### object_store\\n\\n[object_store](https://github.com/apache/opendal/tree/main/integrations/object_store) instead to implement [`object_store`](https://github.com/apache/arrow-rs/tree/master/object_store)\'s trait over OpenDAL Operator so that users can use OpenDAL as a backend for `object_store`.\\n\\n`object_store` is mostly functional, but there are some edge use cases that OpenDAL has yet to support.\\n\\nSo far, this release hasn\'t seen progress in this area; we are awaiting the resolution of the issue [Allow list paths that do not end with `/`](https://github.com/apache/opendal/issues/2762).\\n\\n## Working\\n\\nWe are working on the following things:\\n\\n- `object_store` support: Make `object_store` integration works and find a user for it.\\n- Remove the `/` limitation for path, so we can list a path without ending with `/`.\\n- Expand the `start-after` support to more services (Address [#2786](https://github.com/apache/opendal/issues/2786)).\\n\\n## Outlook\\n\\nWe are exploring some innovative ideas:\\n\\n- [OpenDAL REST/gRPC API](https://github.com/apache/opendal/discussions/2951): A REST/gRPC Server for OpenDAL.\\n- [OpenDAL Cache](https://github.com/apache/opendal/discussions/2953): OpenDAL native cache libs that allowing users to access data more efficiently.\\n- [OpenDAL File System](https://github.com/apache/opendal/discussions/2952): A read-only file system that built upon OpenDAL in rust!\\n- [kio-opendal](https://github.com/apache/opendal/discussions/3042): A kio plugin powered by OpenDAL that allows users to visit different storage services in [KDE Dolphin](https://apps.kde.org/dolphin/).\\n- gvfs-opendal: A gvfs plugin powered by OpenDAL that allows users to visit different storage services in [GNOME Files](https://wiki.gnome.org/Apps/Files)\\n\\nFeel free to join in the discussion!\\n\\n## Summary\\n\\nThis marks our first OpenDAL `OwO` post. We welcome your feedback."},{"id":"how-opendal-read-data","metadata":{"permalink":"/blog/how-opendal-read-data","editUrl":"https://github.com/apache/opendal/tree/main/website/blog/2023-08-15-how-opendal-read-data/index.md","source":"@site/blog/2023-08-15-how-opendal-read-data/index.md","title":"Apache OpenDAL\u2122 Internal: Data Reading","description":"The first article will discuss OpenDAL\'s most commonly used data reading function. I will start from the outermost interface and then gradually unfold according to the calling sequence of OpenDAL.","date":"2023-08-15T00:00:00.000Z","tags":[{"inline":true,"label":"internal","permalink":"/blog/tags/internal"}],"readingTime":7.97,"hasTruncateMarker":true,"authors":[{"name":"Xuanwo","url":"https://github.com/Xuanwo","key":"xuanwo","page":null}],"frontMatter":{"title":"Apache OpenDAL\u2122 Internal: Data Reading","date":"2023-08-15T00:00:00.000Z","slug":"how-opendal-read-data","tags":["internal"],"description":"The first article will discuss OpenDAL\'s most commonly used data reading function. I will start from the outermost interface and then gradually unfold according to the calling sequence of OpenDAL.","authors":"xuanwo"},"unlisted":false,"prevItem":{"title":"OwO #1: The v0.40 Release","permalink":"/blog/owo-1"},"nextItem":{"title":"Apache OpenDAL\u2122: Access Data Freely","permalink":"/blog/opendal-access-data-freely"}},"content":"As the Apache OpenDAL\u2122 community continues to grow, new abstractions are constantly being added, which has brought some burdens to new contributors participating in development. Many maintainers hope to have a deeper understanding of OpenDAL\'s internal implementation. At the same time, OpenDAL\'s core design has not changed significantly for a long time, making it possible to write a series on internal implementation. I believe now is the time to write a series of articles on OpenDAL\'s internal implementation, to explain from the maintainer\'s perspective how OpenDAL is designed, implemented, and how it can be expanded. With the impending release of OpenDAL v0.40, I hope this series of articles will better help the community understand the past, master the present, and shape the future.\\n\\nThe first article will discuss OpenDAL\'s most commonly used data reading function. I will start from the outermost interface and then gradually unfold according to the calling sequence of OpenDAL. Let\'s get started!\\n\\n\x3c!--truncate--\x3e\\n\\n## Overall Framework\\n\\nBefore starting to introduce the specific OpenDAL interface, let\'s first get familiar with the OpenDAL project.\\n\\n[OpenDAL](https://github.com/apache/opendal) is an Apache Incubator project aimed at helping users access data from various storage services in a unified, convenient, and efficient way. Its project [vision](https://opendal.apache.org/vision) is \\"free access to data\\":\\n\\n- Free from services: Any service can be accessed freely through native interfaces\\n- Free from implementations: No matter how the underlying implementation is, it can be called in a unified way\\n- Free to integrate: Able to freely integrate with various services and languages\\n- Free to zero cost: Users don\'t have to pay for features they don\'t use\\n\\nOn this philosophical foundation, OpenDAL Rust Core can be mainly divided into the following components:\\n\\n- Operator: The outer interface exposed to users\\n- Layers: Specific implementation of different middleware\\n- Services: Specific implementation of different services\\n\\nFrom a macroscopic perspective, OpenDAL\'s data reading call stack would look like this:\\n\\n\\n\\nAll Layers and Services have implemented a unified Accessor interface, erasing all type information when building the Operator. For the Operator, regardless of what services are used or how many middleware are added, all call logic is consistent. This design splits OpenDAL\'s API into Public API and Raw API, where the Public API is directly exposed to users, providing convenient top-level interfaces, and Raw API is provided to OpenDAL internal developers, maintaining a unified internal interface and providing some convenient implementation.\\n\\n## Operator\\n\\nOpenDAL\'s Operator API will adhere to a consistent calling paradigm as much as possible, reducing users\' learning and usage costs. For example, OpenDAL offers the following APIs for `read`:\\n\\n- `op.read(path)`: Reads the entire content of the specified file\\n- `op.reader(path)`: Creates a Reader for streaming reading\\n- `op.read_with(path).range(1..1024)`: Reads file content using specified parameters, such as range\\n- `op.reader_with(path).range(1..1024)`: Creates a Reader for streaming reading with specified parameters\\n\\nIt\'s not hard to see that `read` is more like syntactic sugar, allowing users to quickly read files without considering various traits like `AsyncRead`. The `reader` provides more flexibility, implementing widely-used community traits like `AsyncSeek`, `AsyncRead`, allowing more flexible data reading. `read_with` and `reader_with` assist users in specifying various parameters in a more natural way through Future Builder functions.\\n\\nThe internal logic of the Operator would look like this:\\n\\n\\n\\nIts main job is to encapsulate the interface for the user:\\n\\n- Completing the construction of `OpRead`: the args for read operation.\\n- Calling the `read` function provided by `Accessor`\\n- Wrapping the returned value as `Reader` and implementing interfaces like `AsyncSeek`, `AsyncRead`, etc., based on `Reader`\\n\\n## Layers\\n\\nA little secret here is that OpenDAL will automatically apply some Layers to the Service to implement some internal logic. As of the completion of this article, OpenDAL\'s automatically added Layers include:\\n\\n- `ErrorContextLayer`: Injects context information, such as `scheme`, `path`, etc., into all returned errors of Operation\\n- `CompleteLayer`: Adds necessary capabilities to services, such as adding seek support to s3\\n- `TypeEraseLayer`: Implements type erasure, uniformly erasing associated types in `Accessor`, so users don\'t need to carry generic parameters when using it\\n\\nHere, `ErrorContextLayer` and `TypeEraseLayer` are relatively simple and won\'t be elaborated on. The focus is on `CompleteLayer`, aimed at adding `seek` or `next` support to OpenDAL\'s returned `Reader` in a zero-cost way, so users don\'t have to re-implement it. OpenDAL initially returned `Reader` and `SeekableReader` through different function calls in early versions, but the actual user feedback was not very good; almost all users were using `SeekableReader`. Therefore, OpenDAL subsequently added seek support as the first priority to the internal `Read` trait during the refactor:\\n\\n```rust\\npub trait Read: Unpin + Send + Sync {\\n /// Read bytes asynchronously.\\n fn poll_read(&mut self, cx: &mut Context<\'_>, buf: &mut [u8]) -> Poll<Result<usize>>;\\n\\n /// Seek asynchronously.\\n ///\\n /// Returns `Unsupported` error if underlying reader doesn\'t support seek.\\n fn poll_seek(&mut self, cx: &mut Context<\'_>, pos: io::SeekFrom) -> Poll<Result<u64>>;\\n\\n /// Stream [`Bytes`] from underlying reader.\\n ///\\n /// Returns `Unsupported` error if underlying reader doesn\'t support stream.\\n ///\\n /// This API exists for avoiding bytes copying inside async runtime.\\n /// Users can poll bytes from underlying reader and decide when to\\n /// read/consume them.\\n fn poll_next(&mut self, cx: &mut Context<\'_>) -> Poll<Option<Result<Bytes>>>;\\n}\\n```\\n\\nTo implement a service\'s reading capability in OpenDAL, one needs to implement this trait, which is an internal interface and will not be directly exposed to users. Among them:\\n\\n- `poll_read` is the most basic requirement; all services must implement this interface.\\n- When the service natively supports `seek`, `poll_seek` can be implemented, and OpenDAL will correctly dispatch, such as local fs;\\n- When the service natively supports `next`, meaning it returns streaming Bytes, `poll_next` can be implemented, like HTTP-based services, where the underlying layer is a TCP Stream, and hyper will encapsulate it as a bytes stream.\\n\\nThrough the `Read` trait, OpenDAL ensures that all services can expose their native support capabilities as much as possible, thereby achieving efficient reading for different services.\\n\\nBased on this trait, OpenDAL will complete according to the capabilities supported by each service:\\n\\n- Both seek/next are supported: Direct return\\n- No support for next: Encapsulate using `StreamableReader` to simulate next support\\n- No support for seek: Encapsulate using `ByRangeSeekableReader` to simulate seek support\\n- Neither seek/next supported: Encapsulate using both methods\\n\\n> `ByRangeSeekableReader` mainly utilizes the service\'s ability to support range read, dropping the current reader when the user seeks and initiating a new request at the specified location.\\n\\nOpenDAL exposes a unified Reader implementation through `CompleteLayer`, so users don\'t have to worry about whether the underlying service supports seek; OpenDAL will always choose the optimal way to initiate the request.\\n\\n## Services\\n\\nAfter the completion of the Layers, it\'s time to call the specific implementation of the Service. Here, the most common services `fs` and `s3` are used as examples to explain how data is read.\\n\\n### Service fs\\n\\n`tokio::fs::File` implements `tokio::AsyncRead` and `tokio::AsyncSeek`. Using `async_compat::Compat`, we have transformed it into `futures::AsyncRead` and `futures::AsyncSeek`. Based on this, we provide a built-in function `oio::into_read_from_file` to transform it into a type that implements `oio::Read`.\\n\\nThere\'s nothing particularly complex in the implementation of `oio::into_read_from_file`; read and seek are mostly calling the functions provided by the incoming File type. The tricky part is about the correct handling of seek and range: seeking to the right side of the range is allowed, and this will not cause an error, and reading will only return empty, but seeking to the left side of the range is illegal, and the Reader must return `InvalidInput` for proper upper-level handling.\\n\\n> Interesting history: there was [an issue](https://github.com/apache/opendal/issues/2717) in the initial implementation of this part, discovered during fuzz testing.\\n\\n### Services s3\\n\\nS3 is an HTTP-based service, and opendal provides a lot of HTTP-based wrappers to help developers reuse logic; they only need to build a request and return a well-constructed Body. OpenDAL Raw API encapsulates a set of reqwest-based interfaces, and the HTTP GET interface returns a `Response<IncomingAsyncBody>`:\\n\\n```rust\\n/// IncomingAsyncBody carries the content returned by remote servers.\\npub struct IncomingAsyncBody {\\n /// # TODO\\n ///\\n /// hyper returns `impl Stream<Item = crate::Result<Bytes>>` but we can\'t\\n /// write the types in stable. So we will box here.\\n ///\\n /// After [TAIT](https://rust-lang.github.io/rfcs/2515-type_alias_impl_trait.html)\\n /// has been stable, we can change `IncomingAsyncBody` into `IncomingAsyncBody<S>`.\\n inner: oio::Streamer,\\n size: Option<u64>,\\n consumed: u64,\\n chunk: Option<Bytes>,\\n}\\n```\\n\\nThe stream contained in this body is the bytes stream returned by reqwest, and opendal implements content length checks and read support on this basis.\\n\\nHere\'s an extra note about a small pitfall with reqwest/hyper: reqwest and hyper do not check the returned content length, so an illegal server may return a data volume that does not match the expected content length instead of an error, leading to unexpected data behavior. OpenDAL specifically added checks here, returning `ContentIncomplete` when data is insufficient and `ContentTruncated` when data exceeds expectations, avoiding users receiving illegal data.\\n\\n## Conclusion\\n\\nThis article introduces from top to bottom how OpenDAL implements data reading:\\n\\n- Operator is responsible for exposing user-friendly interfaces\\n- Layers are responsible for completing the capabilities of the services\\n- Services are responsible for the specific implementation of different services\\n\\nThroughout the entire chain, OpenDAL adheres as much as possible to the principle of zero cost, prioritizing the use of native service capabilities, then considering simulation through other methods, and finally returning unsupported errors. Through this three-tier design, users don\'t need to understand the details of the underlying service, nor do they need to integrate different service SDKs to easily call `op.read(path)` to access data in any storage service.\\n\\nThis is: How **OpenDAL** read data freely!"},{"id":"opendal-access-data-freely","metadata":{"permalink":"/blog/opendal-access-data-freely","editUrl":"https://github.com/apache/opendal/tree/main/website/blog/2023-07-07-apache-opendal-access-data-freely/index.md","source":"@site/blog/2023-07-07-apache-opendal-access-data-freely/index.md","title":"Apache OpenDAL\u2122: Access Data Freely","description":"If you\'re committed to building cloud-native, cross-cloud-first applications and services, or you want to support configurable storage backends to meet complex data access needs, or if you\'re tired of juggling various SDKs and hoping for a unified abstraction and development experience, Apache OpenDAL\u2122 will be your perfect partner.","date":"2023-07-07T00:00:00.000Z","tags":[{"inline":true,"label":"announcement","permalink":"/blog/tags/announcement"}],"readingTime":4.585,"hasTruncateMarker":true,"authors":[{"name":"PsiACE","url":"https://github.com/PsiACE","key":"psiace","page":null},{"name":"Xuanwo","url":"https://github.com/Xuanwo","key":"xuanwo","page":null}],"frontMatter":{"title":"Apache OpenDAL\u2122: Access Data Freely","date":"2023-07-07T00:00:00.000Z","slug":"opendal-access-data-freely","tags":["announcement"],"description":"If you\'re committed to building cloud-native, cross-cloud-first applications and services, or you want to support configurable storage backends to meet complex data access needs, or if you\'re tired of juggling various SDKs and hoping for a unified abstraction and development experience, Apache OpenDAL\u2122 will be your perfect partner.","authors":["psiace","xuanwo"]},"unlisted":false,"prevItem":{"title":"Apache OpenDAL\u2122 Internal: Data Reading","permalink":"/blog/how-opendal-read-data"},"nextItem":{"title":"Way to Go: OpenDAL successfully entered Apache Incubator","permalink":"/blog/opendal-entered-apache-incubator"}},"content":"If you\'re committed to building cloud-native, cross-cloud-first applications and services, or you want to support configurable storage backends to meet complex data access needs, or if you\'re tired of juggling various SDKs and hoping for a unified abstraction and development experience, Apache OpenDAL\u2122 will be your perfect partner.\\n\\n\\n\\n\x3c!--truncate--\x3e\\n\\n## What is OpenDAL?\\n\\n**OpenDAL** is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.\\n\\n**Data Access Layer** means: OpenDAL is in a critical position in the data read-write process. We shield the implementation details of different storage backends and provide a set of unified interface abstractions externally.\\n\\nNext, let\'s try to answer *\\"What OpenDAL is not\\"* and deconstruct OpenDAL from another perspective:\\n\\n### Opendal Is Not a Proxy Service\\n\\nOpenDAL is provided in the form of a library, not as a service or application that proxies various storage backends.\\n\\nIf you want to integrate OpenDAL into an existing project, you need to call OpenDAL\'s interface directly through the bindings supported by OpenDAL to access the storage services.\\n\\n### Opendal Is Not an SDK Aggregator\\n\\nAlthough OpenDAL replaces various SDKs in the application architecture, it is not implemented as an SDK aggregator.\\n\\nIn other words, OpenDAL does not simply call various storage service SDKs. We have developed our own docking with various storage services based on a unified Rust core to ensure that the differences between services are smoothed out.\\n\\nFor example, for S3, OpenDAL manually constructs HTTP requests and parses HTTP responses to ensure that all behaviors comply with API specifications and are fully under the control of OpenDAL. Due to OpenDAL\'s native takeover of the data access process, we can easily implement unified retry and logging mechanisms for various storage backends and ensure behavioral consistency.\\n\\nFor compatible services with S3, due to the limitations of native storage services and differences in API implementation, compatibility and behavioral details may differ from S3. For example, OSS needs to set an independent header to ensure consistent behavior for `Range`. In addition to docking with native storage services, OpenDAL will also perform targeted processing for compatible services to ensure users\' data access experience.\\n\\n## Advantages of OpenDAL\\n\\nOpenDAL is not the only project dedicated to providing data access abstraction, but compared to other similar projects, OpenDAL has the following advantages:\\n\\n### Rich Service Support\\n\\nOpenDAL supports dozens of storage services, covering a wide range of scenarios and supporting on-demand selection:\\n\\n- Standard Storage Protocols: FTP, HTTP, SFTP, WebDAV, etc.\\n- Object Storage Services: azblob, gcs, obs, oss, s3, etc.\\n- File Storage Services: fs, azdls, hdfs, webhdfs, ipfs, etc.\\n- Consumer Cloud Storage Service: Google Drive, OneDrive, Dropbox, etc.\\n- Key-Value Storage Service: Memory, Redis, Rocksdb, etc.\\n- Cache Storage Service: Ghac, Memcached, etc.\\n\\n### Complete Cross-Language Bindings\\n\\nWith Rust as the core, OpenDAL now provides binding support for multiple languages such as Python/Node.js/Java/C and is also actively developing bindings for other languages.\\n\\nCross-language bindings not only provide unified storage access abstractions for other languages but also follow naming conventions and development habits that are common in various languages as much as possible to pave the way for quick use.\\n\\n### Powerful Middleware Support\\n\\nOpenDAL offers native layer support, enabling users to implement middleware or intercept for all operations.\\n\\n- Error Retry: OpenDAL supports fine-grained error retry capabilities. In addition to common request retries, it supports breakpoint resumable transmission without having to re-read the entire file.\\n- Observability: OpenDAL implements logging,tracing,and metrics support for all operations. Turning on middleware can directly obtain observability capabilities for storage.\\n- Concurrency control, flow control, fuzz testing, and more.\\n\\n### Easy to Use\\n\\nOpenDAL\'s API has been well designed and polished in actual use. The documentation covers everything and is easy to get started with. Here\'s an example of using Python bindings to access HDFS:\\n\\n```python\\nimport opendal\\n \\nop = opendal.Operator(\\"hdfs\\", name_node=\\"hdfs://192.16.8.10.103\\")\\nop.read(\\"path/to/file\\")\\n```\\n\\n### Use Cases of OpenDAL\\n\\nCurrently, OpenDAL is widely used in various scenarios that require cloud-native capabilities, including but not limited to databases, data pipelines, and caches. The main user cases include:\\n\\n- [Databend](https://github.com/datafuselabs/databend/): A modern Elasticity and Performance cloud data warehouse. Using OpenDAL to read and write persistent data (s3, azblob, gcs, hdfs, etc.) and cache data (fs, redis, rocksdb, moka, etc.).\\n- [GreptimeDB](https://github.com/GreptimeTeam/greptimedb): An open-source, cloud-native, distributed time-series database. Using OpenDAL to read and write persistent data (s3, azblob, etc.).\\n- [mozilla/sccache](https://github.com/mozilla/sccache/): `sccache` is [`ccache`](https://github.com/ccache/ccache) with cloud storage. Using OpenDAL to read and write cache data (s3 and ghac, etc.).\\n- [RisingWave](https://github.com/risingwavelabs/risingwave): A Distributed SQL Database for Stream Processing. Using OpenDAL to read and write persistent data (s3, azblob, hdfs, etc.).\\n- [Vector](https://github.com/vectordotdev/vector): A high-performance observability data pipeline. Using OpenDAL to write persistent data (currently mainly using hdfs).\\n\\n## Future Plans of OpenDAL\\n\\nIn addition to further meeting the needs of cloud-native data access, OpenDAL will continue to expand user scenarios and actively explore its use in data science and mobile applications. At the same time, OpenDAL will continue to polish its existing implementations and bindings to provide users with a better integration experience.\\n\\nOpenDAL will also explore how to improve users\' workflows in data management and service integration:\\n\\n- Polish the `oli` command-line tool to help users manage data painlessly.\\n- Implement the `oay` proxy service to provide users with high-quality compatible APIs.\\n\\nIn addition, since OpenDAL is currently a cross-language project, we also plan to write a series of introductory tutorials to help everyone learn OpenDAL from scratch while learning the language."},{"id":"opendal-entered-apache-incubator","metadata":{"permalink":"/blog/opendal-entered-apache-incubator","editUrl":"https://github.com/apache/opendal/tree/main/website/blog/2023-03-16-opendal-entered-apache-incubator/index.md","source":"@site/blog/2023-03-16-opendal-entered-apache-incubator/index.md","title":"Way to Go: OpenDAL successfully entered Apache Incubator","description":"On February 27th, 2023, the OpenDAL project achieved a milestone by winning the approval to join the incubator of the Apache Software Foundation (ASF), the world\'s leading open source software organization. On March 15th, the OpenDAL project was officially transferred to the Apache Software Foundation.","date":"2023-03-16T00:00:00.000Z","tags":[{"inline":true,"label":"announcement","permalink":"/blog/tags/announcement"}],"readingTime":4.73,"hasTruncateMarker":true,"authors":[{"name":"PsiACE","url":"https://github.com/PsiACE","key":"psiace","page":null}],"frontMatter":{"title":"Way to Go: OpenDAL successfully entered Apache Incubator","date":"2023-03-16T00:00:00.000Z","slug":"opendal-entered-apache-incubator","tags":["announcement"],"description":"On February 27th, 2023, the OpenDAL project achieved a milestone by winning the approval to join the incubator of the Apache Software Foundation (ASF), the world\'s leading open source software organization. On March 15th, the OpenDAL project was officially transferred to the Apache Software Foundation.","authors":"psiace"},"unlisted":false,"prevItem":{"title":"Apache OpenDAL\u2122: Access Data Freely","permalink":"/blog/opendal-access-data-freely"}},"content":"\\n\\nOn February 27th, 2023, the [OpenDAL](https://github.com/apache/opendal) project achieved a milestone by winning the approval to join the incubator of the [Apache Software Foundation](https://www.apache.org/) (ASF), the world\'s leading open source software organization. On March 15th, the OpenDAL project was officially transferred to the Apache Software Foundation.\\n\\n\x3c!--truncate--\x3e\\n\\nThis is a significant moment for [Databend](https://github.com/datafuselabs/databend), as it means that OpenDAL\'s technology and vision have received wider recognition and support from the open source community.\\n\\n> The Apache Incubator was established in October 2002 to provide a path for projects and codebases that aspire to become part of the Apache Software Foundation. Incubating projects need to follow ASF\'s governance and operational practices, and use ASF\'s infrastructure and resources. Incubating projects need to go through a series of stages and evaluations before they can graduate to become top-level projects (TLPs) of ASF.\\n\\n\\n\\n_<https://incubator.apache.org/projects/opendal.html>_\\n\\n## What is OpenDAL?\\n\\nData is one of the most important assets in the future, and data access is the key for realizing data value.\\n\\nThere are various kinds of storage services in the market, each with its own unique interfaces and features, which bring a lot of complexity and inconvenience to data access.\\n\\nOpenDAL provides a unified, simple, efficient, reliable, and observable data access layer that allows developers to seamlessly use different storage services and enjoy the best user experience.\\n\\n\\n\\nOpenDAL simplifies the process of interfacing different storage services, and provides features such as automatic retry, request optimization, and observability. With OpenDAL, developers can directly access a bunch of storage services, without having to understand and master the details of specific SDKs.\\n\\nOpenDAL\'s features include but are not limited to:\\n\\n- Support for dozens of storage services, including local file system, HDFS, S3, OSS, etc.\\n- Provide a unified data access interface, without worrying about the underlying storage details.\\n- Support for various common data operations, including `read`, `write`, `list`, etc.\\n- Support for automatic retry, request optimization, and observability mechanisms.\\n- Zero cost, directly mapped to API calls.\\n- Cross-language bindings: Python, Node.js, C (being worked on), etc.\\n\\n## The Story about OpenDAL\\n\\n### Born for Universal Data Access\\n\\nOpenDAL originated from the vision of creating a universal, unified and user-friendly data access layer. It came into being in late 2021, initially as a component of the Databend project.\\n\\n- On December 21, 2021, [Xuanwo](http://github.com/Xuanwo) embarked on the design and re-implementation of Databend\'s storage access layer, [dal2: Add basic operations of `read`, `write`, `stat`, `delete`](https://github.com/datafuselabs/databend/pull/3575).\\n- On December 27, 2021, the [proposal: Vision of Databend DAL](https://github.com/datafuselabs/databend/discussions/3662) was put forward and discussed. On December 29th, dal2\'s implementation was integrated into Databend.\\n- On February 14th 2022 , dal2 officially separated from Databend\'s code repository and became a standalone top-level project. It was formally renamed OpenDAL.\\n\\n### From One to Multiple\\n\\nThanks to Xuanwo, [ClSlaid](https://github.com/ClSlaid) and many other contributors, OpenDAL quickly became a data access layer that supports mainstream storage services such as AWS S3 / Azure Blob / GCS / HDFS, and provided cross-cloud native storage and access support for Databend\'s `COPY INTO`, Stage, Storage.\\n\\n[GreptimeDB](https://github.com/GreptimeTeam/greptimedb) was the first large-scale Rust database project to actively use OpenDAL after Databend. Later, with Xuanwo\'s efforts, [sccache](https://github.com/mozilla/sccache) under Mozilla also tried to use OpenDAL to take over the storage layer. In order to provide more comprehensive support, OpenDAL soon added support for GitHub Action Cache.\\n\\nThen, [RisingWave](https://github.com/risingwavelabs/risingwave) and [Vector](https://github.com/vectordotdev/vector) were supported as well. The number of OpenDAL users started to grow. More and more users choose OpenDAL as their storage access layer.\\n\\n### Sky\'s the Limit\\n\\nOpenDAL has established a small community and formed a product matrix. In addition to the [Rust - opendal](https://crates.io/crates/opendal), it also provides [Python - opendal](https://pypi.org/project/opendal/) and [Node.js - opendal](https://www.npmjs.com/package/opendal) bindings.\\n\\nOpenDAL has released 99 versions since its open source, with 700+ GitHub stars, 349K downloads, and 48 developers. The project has been actively updated. We sincerely thank every contributor for their efforts and dedication.\\n\\nBeing a part of Apache incubator is an important milestone in OpenDAL\'s development history. We hope to leverage ASF\'s platform and resources to let OpenDAL focus on providing a neutral, vendor-free, painless, and efficient storage access layer, and better serve the developers.\\n\\nWe expect OpenDAL to be widely used in the following application scenarios:\\n\\n- Data analysis: OpenDAL can help data analysts quickly read or write data from different storage services, and perform various format conversions and operations.\\n- Data science: OpenDAL can help data scientists easily get or save data from different storage services, and perform various preprocessing and postprocessing.\\n- Data engineering: OpenDAL can help data engineers efficiently build and manage data pipelines between different storage services, and perform various monitoring and tuning.\\n\\n## Acknowledgements\\n\\n**_From Xuanwo_**\\n\\nHello everyone, I\'m Xuanwo, the Committer of Apache OpenDAL.\\n\\nThe OpenDAL project embodies my personal dream. Now it has entered the Apache incubator with the collaboration of the community. I feel very happy at this moment. Thank you all contributors for your contributions, thank Databend Labs for your support, thank Champion tison for your guidance, thank Mentors ningjiang, wusheng, tedliu and hexiaoqiao for your advice.\\n\\nLet us follow the guidance of Apache Way to build a community together and create value for users by providing free, painless and efficient data access experience!\\n\\n## Join Apache OpenDAL Community\\n\\nWe welcome developers and users who are interested in participating in OpenDAL project to join OpenDAL community and follow OpenDAL\'s latest news. You can get more information through the following ways:\\n\\n- Visit OpenDAL official website: <https://opendal.apache.org>\\n- Explore OpenDAL GitHub repository: <https://github.com/apache/opendal>\\n- Join OpenDAL Discord channel: <https://opendal.apache.org/discord>\\n- Subscribe to OpenDAL mailing list: <dev@opendal.apache.org>"}]}}')}}]); |