search-doc.json

{"searchDocs":[{"title":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","type":0,"sectionRef":"#","url":"/blog/2025/03/01/2025-roadmap/","content":"","keywords":"","version":null},{"title":"What's OpenDAL?​","type":1,"pageTitle":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","url":"/blog/2025/03/01/2025-roadmap/#whats-opendal","content":" Apache OpenDAL (/ˈoʊ.pən.dæl/, pronounced "OH-puhn-dal") is an Open Data Access Layer that enables seamless interaction with diverse storage services. Our VISION is One Layer, All Storage, and our core principles are Open Community, Solid Foundation, Fast Access, Object Storage First, and Extensible Architecture.  We are building:  A core library built in Rust that supports all services at zero cost and offers unified retry, concurrency, logging, tracing, metrics, timeout and more layers.Language bindings for Python, Java, Node.js, C/C++, and more.Integrations with various frameworks, including Parquet, FUSE, DAV server, and others.Binaries for different use cases, such as CLI and FUSE.  In short, through OpenDAL, users can access ALL storage services within ONE layer.  ","version":null,"tagName":"h2"},{"title":"Where's OpenDAL?​","type":1,"pageTitle":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","url":"/blog/2025/03/01/2025-roadmap/#wheres-opendal","content":" OpenDAL's rust core has already released 143 versions, has 67 reverse dependencies listed on crates.io, and is used by 612 projects as recorded on GitHub.  OpenDAL's production users include databases such as Databend, GreptimeDB, and RisingWave, as well as tools like Loco, sccache, and Vector.    Apart from OpenDAL's Rust core, its various language bindings have also seen significant growth over the past year. Take python binding as an example. Dify, an LLM app development platform, is using OpenDAL to access different storage services.    ","version":null,"tagName":"h2"},{"title":"What's next for OpenDAL?​","type":1,"pageTitle":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","url":"/blog/2025/03/01/2025-roadmap/#whats-next-for-opendal","content":" The following is the famous technology adoption lifecycle curve. If I were to indicate the position of OpenDAL, I would say it is at the end of the Innovators stage and moving toward the Early Adopters stage.  (picture from TechTarget technology adoption lifecycle)  Innovators are adopting OpenDAL. Projects like Databend, RisingWave, GreptimeDB, and sccache have been using OpenDAL in production for years. However, early adopters are still hesitant to use OpenDAL—and they have valid reasons.  For examples:  OpenDAL hasn't reached version 1.0 yet and still introduces breaking changes from time to time, sometimes even requiring code modifications for an OpenDAL upgrade. This also creates a burden for libraries that depend on OpenDAL, as every breaking change affects them as well.OpenDAL lacks comprehensive documentation, particularly for its bindings in Python and Java. Users need to invest significant effort and love to integrate OpenDAL bindings into their projects.OpenDAL lacks some important features, such as checksum support, caching, metrics of underlying http requests and initialization from a URI.  I believe we should perfect production adoption in 2025 to get OpenDAL ready for early adopters. Only in this way can we implement our vision: One Layer, All Storage.  ","version":null,"tagName":"h2"},{"title":"What to do in 2025?​","type":1,"pageTitle":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","url":"/blog/2025/03/01/2025-roadmap/#what-to-do-in-2025","content":" Our plans for 2025 include the following:  ","version":null,"tagName":"h2"},{"title":"Features Needed in Production​","type":1,"pageTitle":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","url":"/blog/2025/03/01/2025-roadmap/#features-needed-in-production","content":" In 2025, we plan to implement the following features that are importmant for production adoption:  Context: Introduce context in OpenDAL so that services and layers can share the same context. This enables users to gain deeper insights into OpenDAL's internal operations by incorporating metrics, logging, and tracing into the underlying HTTP client we use.Versioning: Introduce full file versioning support in OpenDAL, enabling users to read, write, delete, list, and restore versioned files. This functionality will allow users to recover mistakenly deleted files and facilitate disaster recovery.Checksum: Introduce end-to-end checksum support in OpenDAL, enabling users to perform checksums during reading and writing without worrying about bit flips in memory or over the network.Caching: Provide high-quality built-in cache support in OpenDAL while ensuring users have the flexibility to implement their own caching logic.Initialization From URI: Allow users to initialize OpenDAL from a URI string, making it easier to configure and use OpenDAL in various environments.  ","version":null,"tagName":"h3"},{"title":"Improvements Needed for Production​","type":1,"pageTitle":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","url":"/blog/2025/03/01/2025-roadmap/#improvements-needed-for-production","content":" In 2025, we plan to improve the following aspects of OpenDAL to ensure users can confidently use it in production.  Documentation: Improve the documentation for OpenDAL, particularly for bindings like Python, Node.js, and Java. The first step is to generate well-structured documentation for the configuration values of each service.Communitation: Bring Back Our Community Meetings. OpenDAL used to hold regular tri-weekly meetings, but we have been unable to maintain them in the last year. We now plan to revive these meetings and encourage more face-to-face discussions in the coming years.  ","version":null,"tagName":"h3"},{"title":"Conclusion​","type":1,"pageTitle":"Apache OpenDAL 2025 Roadmap: Perfecting Production Adoption","url":"/blog/2025/03/01/2025-roadmap/#conclusion","content":" 2025 marks the third year of the OpenDAL community. A huge thanks to all OpenDAL contributors and users for helping us reach this milestone. There's still a long way to go to achieve our vision, and we invite you all to join us on this incredible journey!  Discuss about this post here ","version":null,"tagName":"h2"},{"title":"Apache OpenDAL™ Beijing Meetup 1st","type":0,"sectionRef":"#","url":"/blog/2024/09/09/apache-opendal-meetup-beijing-1st/","content":"","keywords":"","version":null},{"title":"Agenda​","type":1,"pageTitle":"Apache OpenDAL™ Beijing Meetup 1st","url":"/blog/2024/09/09/apache-opendal-meetup-beijing-1st/#agenda","content":" 12:00 - 13:00: Lunch / Networking (food provided)13:00 - 14:00: Welcome / Introductions14:00 - 15:00: Talks15:00 - 15:30: Coffee / Email Break15:30 - 16:30: Breakouts / Discussions16:30 - 17:00: Close / Conclusion  ","version":null,"tagName":"h2"},{"title":"Welcome​","type":1,"pageTitle":"Apache OpenDAL™ Beijing Meetup 1st","url":"/blog/2024/09/09/apache-opendal-meetup-beijing-1st/#welcome","content":" Brief introductory remarks, thank to the sponsor, outline of the day, participant introductions (name, github handle, optional affiliation, optional reason you came to the meetup)  ","version":null,"tagName":"h2"},{"title":"Talks​","type":1,"pageTitle":"Apache OpenDAL™ Beijing Meetup 1st","url":"/blog/2024/09/09/apache-opendal-meetup-beijing-1st/#talks","content":" Five 10-15 minute talks about opendal.  @tisonkun about OpenDAL Community Growth@WenyXu for using OpenDAL in GreptimeDB@wcy-fdu for using OpenDAL in RisingWave@drmingdrmer for using OpenDAL in OpenRaft@Xuanwo about OpenDAL Core 1.0  ","version":null,"tagName":"h2"},{"title":"Breakouts / Discussions​","type":1,"pageTitle":"Apache OpenDAL™ Beijing Meetup 1st","url":"/blog/2024/09/09/apache-opendal-meetup-beijing-1st/#breakouts--discussions","content":" The idea is that we will break into small groups for 3 x 20 minute sessions. I will ring a gong or something similar every 20 minutes and you should find a new group to discuss.  Potential seed topics (but obviously feel free to discuss anything):  How do we make it easier for people to build systems on OpenDAL? What are the current barriers to community growth as you see them?How do we make it easier for people to contribute back to OpenDAL? What are the current barriers to contributionWhat features would you like to see / not see  Ideally, in each session, one or more people would volunteer to write up any insights they got from the discussions, and post them to the overall ticket.    Looking forward to meet you in this meetup! ","version":null,"tagName":"h2"},{"title":"Apache OpenDAL™ is now Graduated","type":0,"sectionRef":"#","url":"/blog/apache-opendal-graduated/","content":"","keywords":"","version":null},{"title":"What's Apache OpenDAL?​","type":1,"pageTitle":"Apache OpenDAL™ is now Graduated","url":"/blog/apache-opendal-graduated/#whats-apache-opendal","content":" Apache OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way. Our VISION is access data freely.  OpenDAL could be used as a better SDK for your storage services: A SDK with native integration of retry, logging, metrics, tracing, timeout, throttle, and more.  OpenDAL could be used as a super connector for your storage services: A connector that supports all kinds of storage services from Object Storage (s3, gcs, azblob), File Storage (fs, azdls, hdfs), Consumer Cloud Storage (gdrive, onedrive), Key-Value Storage (rocksdb, sled) to Cache Storage (memcached, moka).  OpenDAL could be used as an elegant client for your storage services: A client with well designed API and many language bindings: Rust, C, Cpp, Dotnet, Go, Haskell, Java, Lua, Node.js, OCaml, PHP, Python, Ruby, Swift and Zig.  Need to access data? Give OpenDAL a try!  async fn main() -> Result<()> { // Init s3 service. let mut builder = services::S3::default(); builder.bucket("test"); // Init an operator let op = Operator::via_map(builder)? // Add logging .layer(LoggingLayer::default()) .finish(); // Write data op.write("hello.txt", "Hello, World!").await?; // Read data let bs = op.read("hello.txt").await?; // Fetch metadata let meta = op.stat("hello.txt").await?; let mode = meta.mode(); let length = meta.content_length(); // Delete op.delete("hello.txt").await?; Ok(()) }   ","version":null,"tagName":"h2"},{"title":"What's the ASF?​","type":1,"pageTitle":"Apache OpenDAL™ is now Graduated","url":"/blog/apache-opendal-graduated/#whats-the-asf","content":" The Apache Software Foundation (ASF) is a nonprofit corporation to support a number of open-source software projects. The Apache Software Foundation exists to provide software for the public good. We believe in the power of community over code, known as The Apache Way. Thousands of people around the world contribute to ASF open source projects every day.  The OpenDAL Community believes the apache way that:  Earned Authority: all individuals are given the opportunity to participate, but their influence is based on publicly earned merit – what they contribute to the community.Community of Peers: individuals participate at the ASF, not organizations.Open Communications: as a virtual organization, the ASF requires all communications related to code and decision-making to be publicly accessible to ensure asynchronous collaboration, as necessitated by a globally-distributed community.Consensus Decision Making: Apache Projects are overseen by a self-selected team of active volunteers who are contributing to their respective projects.Responsible Oversight: The ASF governance model is based on trust and delegated oversight.  The original creators Databend chosen to contribute OpenDAL to the ASF, embracing the Apache way through joining the incubator program.  ","version":null,"tagName":"h2"},{"title":"What's graduation?​","type":1,"pageTitle":"Apache OpenDAL™ is now Graduated","url":"/blog/apache-opendal-graduated/#whats-graduation","content":" In the Apache Incubator, the OpenDAL community is learning the Apache Way through daily development activities, growing its community and producing Apache releases.  During the incubation, we:  Consist of 19 committers, including mentors, with 12 serving as PPMC members.Boast 164 contributors.Made 9 releases—averaging at least one per month.Had 7 different release managers to date.Used by 10 known entities and is a dependency for 263 GitHub projects and 18 crates.io packages.Opened 1,200+ issues with 1,100+ successfully resolved.Submitted a total of 2,400+ PRs, most of them have been merged or closed.  The graduation signifies that the OpenDAL Community is recognized as a mature community, which entails:  CODE: OpenDAL is an Apache 2.0 licensed open-source project with accessible, buildable code on GitHub, featuring a traceable history and authenticated code provenance.LICENSE: OpenDAL maintains open-source compliance for all code and dependencies, requires contributor agreements, and clearly documents copyright ownership.Releases: OpenDAL offers standardized, committee-approved source code releases with secure signatures, provides convenience binaries, and has a well-documented, repeatable release process.Quality: OpenDAL is committed to code quality transparency, prioritizes security with quick issue responses, ensures backward compatibility with clear documentation, and actively addresses bug reports in a timely manner.Community: OpenDAL offers a comprehensive homepage, welcomes diverse contributions, promotes a meritocratic approach for active contributors, operates on community consensus, and ensures timely responses to user queries through various channels.Consensus: OpenDAL has a public list of key decision-makers and uses a consensus approach for decisions, documented on its main communication channel. It follows standard voting rules and records all important discussions in writing.Independence: OpenDAL is independent, with contributors from various companies acting on their own, not as representatives of any organization.  ","version":null,"tagName":"h2"},{"title":"What's next?​","type":1,"pageTitle":"Apache OpenDAL™ is now Graduated","url":"/blog/apache-opendal-graduated/#whats-next","content":" After graduation, OpenDAL Community will continue to focus on the following aspects under the VISION: access data freely.  ","version":null,"tagName":"h2"},{"title":"More Stable Services​","type":1,"pageTitle":"Apache OpenDAL™ is now Graduated","url":"/blog/apache-opendal-graduated/#more-stable-services","content":" OpenDAL now supports 59 services, although only some of them are stable.  stable for OpenDAL means that  Have integration tests covered.Have at least one production user.  The stable service established a feedback loop between the OpenDAL community and its users. Users can submit bug reports or feature requests to the OpenDAL community, which in turn can enhance the service using this feedback while ensuring existing features remain intact.  After graduation, the OpenDAL community will focus on improving the stability of current services instead of just expanding our offerings.  We plan to:  Add features users wanted to services like file version, concurrently list and glob pattern.Add integration tests for newly added services.  ","version":null,"tagName":"h3"},{"title":"More Useful Documents​","type":1,"pageTitle":"Apache OpenDAL™ is now Graduated","url":"/blog/apache-opendal-graduated/#more-useful-documents","content":" OpenDAL have good docs for its rust core, but not for other language bindings.  The lack of comprehensive documentation makes OpenDAL challenging for users to operate in Java or Python. Without user feedback, the community is unable to enhance this documentation, leading to a detrimental cycle that must be broken.  After graduation, the OpenDAL community will improve the documentation of other language bindings.  We plan to:  Introduce code generation to automatically create documentation for the service builder due to its numerous configurations.Add more API Docs and examples for other language bindings.  OpenDAL have good docs for its public API, but not for its internal design.  OpenDAL is proud of its elegant design, but it is not well documented. This makes it difficult for new contributors to understand the codebase and make contributions.  After graduation, the OpenDAL community will improve the documentation of its internal design.  We plan to:  Optimize the codebase to make it easier to understand.Add more blog posts to explain the design of OpenDAL.  ","version":null,"tagName":"h3"},{"title":"More Production Users​","type":1,"pageTitle":"Apache OpenDAL™ is now Graduated","url":"/blog/apache-opendal-graduated/#more-production-users","content":" OpenDAL requires more production users, as they are vital to the success of our project. Increased user production leads to more valuable feedback, a more engaged contributor base, and a stronger community. We've started the initial loop; let's expand it!  After graduation, the OpenDAL community will focus on attracting more production users.  We plan to:  Optimize the feature set for adoption like uri initiation and config.Expand more ways to use OpenDAL via fuse, cli, S3/WebDAV API, object_store binding.  ","version":null,"tagName":"h3"},{"title":"Conclusion​","type":1,"pageTitle":"Apache OpenDAL™ is now Graduated","url":"/blog/apache-opendal-graduated/#conclusion","content":" The OpenDAL Community aims to create a world where users can freely access data across any storage service in any manner they choose. Graduation is just the beginning—let's work together to make our VISION a reality! ","version":null,"tagName":"h2"},{"title":"Apache OpenDAL™ participates in Google Summer of Code 2024","type":0,"sectionRef":"#","url":"/blog/apache-opendal-participates-in-gsoc-2024/","content":"Hello, everyone! We're writing this blog post to announce that the Apache OpenDAL™ Project will be participating in Google Summer of Code (GSoC) 2024. If you're not eligible or interested in participating in GSoC, then most of this post likely isn't relevant to you; if you are, this should contain some useful information and links. Google Summer of Code (GSoC) is an annual global program organized by Google that aims to bring new contributors to the world of open-source. The program pairs organizations (such as the OpenDAL Project) with contributors (usually students), with the goal of helping the participants make meaningful open-source contributions under the guidance of experienced mentors. Google is sponsoring the 2024 Summer of Code and The Apache Software Foundation (ASF) registered as a mentoring organization. The GSoC applicants now have several weeks to send project proposals to organizations that appeal to them. If their project proposal is accepted, they will embark on a 12-week journey during which they will try to complete their proposed project under the guidance of an assigned mentor. We have prepared a list of project ideas that can serve as inspiration for potential GSoC contributors that would like to send a project proposal to the OpenDAL project. However, applicants can also come up with their own project ideas. You can discuss project ideas or try to find mentors on the maillist or Discord. We have also prepared a proposal guide that should help you with preparing your project proposals. You can start discussing the project ideas with OpenDAL Project maintainers immediately. The project proposal application period starts on March 18, 2024, and ends on April 2, 2024 at 18:00 UTC. Take note of that deadline, as there will be no extensions! If you are interested in contributing to the OpenDAL Project, we encourage you to check out our project idea list and send us a GSoC project proposal! Of course, you are also free to discuss these projects and/or try to move them forward even if you do not intend to (or cannot) participate in GSoC. We welcome all contributors to OpenDAL, as there is always enough work to do. We are excited about this event. Hoping you all feel the same way! This announcement is inspired a lot by the Rust participates in Google Summer of Code 2024.","keywords":"","version":null},{"title":"Apache OpenDAL™ participates in Open Source Promotion Plan 2024","type":0,"sectionRef":"#","url":"/blog/apache-opendal-participates-in-ospp-2024/","content":"Hello, everyone! We're writing this blog post to announce that the Apache OpenDAL™ Project will be participating in Open Source Promotion Plan (OSPP) 2024. If you're not eligible or interested in participating in OSPP, then most of this post likely isn't relevant to you; if you are, this should contain some useful information and links. Open Source Promotion Plan is a summer program organized by the Institute of Software Chinese Academy of Sciences and long-term supported by the Open Source Software Supply Chain Promotion Plan. It aims to encourage college students to actively participate in the maintenance and development of open source software, promote the vigorous development of open source software communities, and build the open source software supply chain together. The OSPP applicants now have several weeks to send project proposals to organizations that appeal to them. If their project proposal is accepted, they will embark on a 12-week journey during which they will try to complete their proposed project under the guidance of an assigned mentor. We have prepared a list of project ideas that can serve as inspiration for potential OSPP contributors that would like to send a project proposal to the OpenDAL project. You can try to find mentors on the maillist or Discord. We have also prepared a proposal guide that should help you with preparing your project proposals. You can start discussing the project ideas with OpenDAL Project maintainers immediately. The project proposal application period starts on April 30, 2024, and ends on June 4, 2024. Take note of that deadline, as there will be no extensions! If you are interested in contributing to the OpenDAL Project, we encourage you to check out our project idea list and send us a OSPP project proposal! Of course, you are also free to discuss these projects and/or try to move them forward even if you do not intend to (or cannot) participate in OSPP. We welcome all contributors to OpenDAL, as there is always enough work to do. We are excited about this event. Hoping you all feel the same way! This announcement is inspired a lot by the Rust participates in Google Summer of Code 2024.","keywords":"","version":null},{"title":"Way to Go: OpenDAL successfully entered Apache Incubator","type":0,"sectionRef":"#","url":"/blog/opendal-entered-apache-incubator/","content":"","keywords":"","version":null},{"title":"What is OpenDAL?​","type":1,"pageTitle":"Way to Go: OpenDAL successfully entered Apache Incubator","url":"/blog/opendal-entered-apache-incubator/#what-is-opendal","content":" Data is one of the most important assets in the future, and data access is the key for realizing data value.  There are various kinds of storage services in the market, each with its own unique interfaces and features, which bring a lot of complexity and inconvenience to data access.  OpenDAL provides a unified, simple, efficient, reliable, and observable data access layer that allows developers to seamlessly use different storage services and enjoy the best user experience.    OpenDAL simplifies the process of interfacing different storage services, and provides features such as automatic retry, request optimization, and observability. With OpenDAL, developers can directly access a bunch of storage services, without having to understand and master the details of specific SDKs.  OpenDAL's features include but are not limited to:  Support for dozens of storage services, including local file system, HDFS, S3, OSS, etc.Provide a unified data access interface, without worrying about the underlying storage details.Support for various common data operations, including read, write, list, etc.Support for automatic retry, request optimization, and observability mechanisms.Zero cost, directly mapped to API calls.Cross-language bindings: Python, Node.js, C (being worked on), etc.  ","version":null,"tagName":"h2"},{"title":"The Story about OpenDAL​","type":1,"pageTitle":"Way to Go: OpenDAL successfully entered Apache Incubator","url":"/blog/opendal-entered-apache-incubator/#the-story-about-opendal","content":" ","version":null,"tagName":"h2"},{"title":"Born for Universal Data Access​","type":1,"pageTitle":"Way to Go: OpenDAL successfully entered Apache Incubator","url":"/blog/opendal-entered-apache-incubator/#born-for-universal-data-access","content":" OpenDAL originated from the vision of creating a universal, unified and user-friendly data access layer. It came into being in late 2021, initially as a component of the Databend project.  On December 21, 2021, Xuanwo embarked on the design and re-implementation of Databend's storage access layer, dal2: Add basic operations of read, write, stat, delete.On December 27, 2021, the proposal: Vision of Databend DAL was put forward and discussed. On December 29th, dal2's implementation was integrated into Databend.On February 14th 2022 , dal2 officially separated from Databend's code repository and became a standalone top-level project. It was formally renamed OpenDAL.  ","version":null,"tagName":"h3"},{"title":"From One to Multiple​","type":1,"pageTitle":"Way to Go: OpenDAL successfully entered Apache Incubator","url":"/blog/opendal-entered-apache-incubator/#from-one-to-multiple","content":" Thanks to Xuanwo, ClSlaid and many other contributors, OpenDAL quickly became a data access layer that supports mainstream storage services such as AWS S3 / Azure Blob / GCS / HDFS, and provided cross-cloud native storage and access support for Databend's COPY INTO, Stage, Storage.  GreptimeDB was the first large-scale Rust database project to actively use OpenDAL after Databend. Later, with Xuanwo's efforts, sccache under Mozilla also tried to use OpenDAL to take over the storage layer. In order to provide more comprehensive support, OpenDAL soon added support for GitHub Action Cache.  Then, RisingWave and Vector were supported as well. The number of OpenDAL users started to grow. More and more users choose OpenDAL as their storage access layer.  ","version":null,"tagName":"h3"},{"title":"Sky's the Limit​","type":1,"pageTitle":"Way to Go: OpenDAL successfully entered Apache Incubator","url":"/blog/opendal-entered-apache-incubator/#skys-the-limit","content":" OpenDAL has established a small community and formed a product matrix. In addition to the Rust - opendal, it also provides Python - opendal and Node.js - opendal bindings.  OpenDAL has released 99 versions since its open source, with 700+ GitHub stars, 349K downloads, and 48 developers. The project has been actively updated. We sincerely thank every contributor for their efforts and dedication.  Being a part of Apache incubator is an important milestone in OpenDAL's development history. We hope to leverage ASF's platform and resources to let OpenDAL focus on providing a neutral, vendor-free, painless, and efficient storage access layer, and better serve the developers.  We expect OpenDAL to be widely used in the following application scenarios:  Data analysis: OpenDAL can help data analysts quickly read or write data from different storage services, and perform various format conversions and operations.Data science: OpenDAL can help data scientists easily get or save data from different storage services, and perform various preprocessing and postprocessing.Data engineering: OpenDAL can help data engineers efficiently build and manage data pipelines between different storage services, and perform various monitoring and tuning.  ","version":null,"tagName":"h3"},{"title":"Acknowledgements​","type":1,"pageTitle":"Way to Go: OpenDAL successfully entered Apache Incubator","url":"/blog/opendal-entered-apache-incubator/#acknowledgements","content":" From Xuanwo  Hello everyone, I'm Xuanwo, the Committer of Apache OpenDAL.  The OpenDAL project embodies my personal dream. Now it has entered the Apache incubator with the collaboration of the community. I feel very happy at this moment. Thank you all contributors for your contributions, thank Databend Labs for your support, thank Champion tison for your guidance, thank Mentors ningjiang, wusheng, tedliu and hexiaoqiao for your advice.  Let us follow the guidance of Apache Way to build a community together and create value for users by providing free, painless and efficient data access experience!  ","version":null,"tagName":"h2"},{"title":"Join Apache OpenDAL Community​","type":1,"pageTitle":"Way to Go: OpenDAL successfully entered Apache Incubator","url":"/blog/opendal-entered-apache-incubator/#join-apache-opendal-community","content":" We welcome developers and users who are interested in participating in OpenDAL project to join OpenDAL community and follow OpenDAL's latest news. You can get more information through the following ways:  Visit OpenDAL official website: https://opendal.apache.orgExplore OpenDAL GitHub repository: https://github.com/apache/opendalJoin OpenDAL Discord channel: https://opendal.apache.org/discordSubscribe to OpenDAL mailing list: dev@opendal.apache.org ","version":null,"tagName":"h2"},{"title":"Apache OpenDAL™: Access Data Freely","type":0,"sectionRef":"#","url":"/blog/opendal-access-data-freely/","content":"","keywords":"","version":null},{"title":"What is OpenDAL?​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#what-is-opendal","content":" OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way.  Data Access Layer means: OpenDAL is in a critical position in the data read-write process. We shield the implementation details of different storage backends and provide a set of unified interface abstractions externally.  Next, let's try to answer "What OpenDAL is not" and deconstruct OpenDAL from another perspective:  ","version":null,"tagName":"h2"},{"title":"Opendal Is Not a Proxy Service​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#opendal-is-not-a-proxy-service","content":" OpenDAL is provided in the form of a library, not as a service or application that proxies various storage backends.  If you want to integrate OpenDAL into an existing project, you need to call OpenDAL's interface directly through the bindings supported by OpenDAL to access the storage services.  ","version":null,"tagName":"h3"},{"title":"Opendal Is Not an SDK Aggregator​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#opendal-is-not-an-sdk-aggregator","content":" Although OpenDAL replaces various SDKs in the application architecture, it is not implemented as an SDK aggregator.  In other words, OpenDAL does not simply call various storage service SDKs. We have developed our own docking with various storage services based on a unified Rust core to ensure that the differences between services are smoothed out.  For example, for S3, OpenDAL manually constructs HTTP requests and parses HTTP responses to ensure that all behaviors comply with API specifications and are fully under the control of OpenDAL. Due to OpenDAL's native takeover of the data access process, we can easily implement unified retry and logging mechanisms for various storage backends and ensure behavioral consistency.  For compatible services with S3, due to the limitations of native storage services and differences in API implementation, compatibility and behavioral details may differ from S3. For example, OSS needs to set an independent header to ensure consistent behavior for Range. In addition to docking with native storage services, OpenDAL will also perform targeted processing for compatible services to ensure users' data access experience.  ","version":null,"tagName":"h3"},{"title":"Advantages of OpenDAL​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#advantages-of-opendal","content":" OpenDAL is not the only project dedicated to providing data access abstraction, but compared to other similar projects, OpenDAL has the following advantages:  ","version":null,"tagName":"h2"},{"title":"Rich Service Support​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#rich-service-support","content":" OpenDAL supports dozens of storage services, covering a wide range of scenarios and supporting on-demand selection:  Standard Storage Protocols: FTP, HTTP, SFTP, WebDAV, etc.Object Storage Services: azblob, gcs, obs, oss, s3, etc.File Storage Services: fs, azdls, hdfs, webhdfs, ipfs, etc.Consumer Cloud Storage Service: Google Drive, OneDrive, Dropbox, etc.Key-Value Storage Service: Memory, Redis, Rocksdb, etc.Cache Storage Service: Ghac, Memcached, etc.  ","version":null,"tagName":"h3"},{"title":"Complete Cross-Language Bindings​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#complete-cross-language-bindings","content":" With Rust as the core, OpenDAL now provides binding support for multiple languages such as Python/Node.js/Java/C and is also actively developing bindings for other languages.  Cross-language bindings not only provide unified storage access abstractions for other languages but also follow naming conventions and development habits that are common in various languages as much as possible to pave the way for quick use.  ","version":null,"tagName":"h3"},{"title":"Powerful Middleware Support​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#powerful-middleware-support","content":" OpenDAL offers native layer support, enabling users to implement middleware or intercept for all operations.  Error Retry: OpenDAL supports fine-grained error retry capabilities. In addition to common request retries, it supports breakpoint resumable transmission without having to re-read the entire file.Observability: OpenDAL implements logging,tracing,and metrics support for all operations. Turning on middleware can directly obtain observability capabilities for storage.Concurrency control, flow control, fuzz testing, and more.  ","version":null,"tagName":"h3"},{"title":"Easy to Use​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#easy-to-use","content":" OpenDAL's API has been well designed and polished in actual use. The documentation covers everything and is easy to get started with. Here's an example of using Python bindings to access HDFS:  import opendal op = opendal.Operator("hdfs", name_node="hdfs://192.16.8.10.103") op.read("path/to/file")   ","version":null,"tagName":"h3"},{"title":"Use Cases of OpenDAL​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#use-cases-of-opendal","content":" Currently, OpenDAL is widely used in various scenarios that require cloud-native capabilities, including but not limited to databases, data pipelines, and caches. The main user cases include:  Databend: A modern Elasticity and Performance cloud data warehouse. Using OpenDAL to read and write persistent data (s3, azblob, gcs, hdfs, etc.) and cache data (fs, redis, rocksdb, moka, etc.).GreptimeDB: An open-source, cloud-native, distributed time-series database. Using OpenDAL to read and write persistent data (s3, azblob, etc.).mozilla/sccache: sccache is ccache with cloud storage. Using OpenDAL to read and write cache data (s3 and ghac, etc.).RisingWave: A Distributed SQL Database for Stream Processing. Using OpenDAL to read and write persistent data (s3, azblob, hdfs, etc.).Vector: A high-performance observability data pipeline. Using OpenDAL to write persistent data (currently mainly using hdfs).  ","version":null,"tagName":"h3"},{"title":"Future Plans of OpenDAL​","type":1,"pageTitle":"Apache OpenDAL™: Access Data Freely","url":"/blog/opendal-access-data-freely/#future-plans-of-opendal","content":" In addition to further meeting the needs of cloud-native data access, OpenDAL will continue to expand user scenarios and actively explore its use in data science and mobile applications. At the same time, OpenDAL will continue to polish its existing implementations and bindings to provide users with a better integration experience.  OpenDAL will also explore how to improve users' workflows in data management and service integration:  Polish the oli command-line tool to help users manage data painlessly.Implement the oay proxy service to provide users with high-quality compatible APIs.  In addition, since OpenDAL is currently a cross-language project, we also plan to write a series of introductory tutorials to help everyone learn OpenDAL from scratch while learning the language. ","version":null,"tagName":"h2"},{"title":"Apache OpenDAL™ Internal: Data Reading","type":0,"sectionRef":"#","url":"/blog/how-opendal-read-data/","content":"","keywords":"","version":null},{"title":"Overall Framework​","type":1,"pageTitle":"Apache OpenDAL™ Internal: Data Reading","url":"/blog/how-opendal-read-data/#overall-framework","content":" Before starting to introduce the specific OpenDAL interface, let's first get familiar with the OpenDAL project.  OpenDAL is an Apache Incubator project aimed at helping users access data from various storage services in a unified, convenient, and efficient way. Its project vision is "free access to data":  Free from services: Any service can be accessed freely through native interfacesFree from implementations: No matter how the underlying implementation is, it can be called in a unified wayFree to integrate: Able to freely integrate with various services and languagesFree to zero cost: Users don't have to pay for features they don't use  On this philosophical foundation, OpenDAL Rust Core can be mainly divided into the following components:  Operator: The outer interface exposed to usersLayers: Specific implementation of different middlewareServices: Specific implementation of different services  From a macroscopic perspective, OpenDAL's data reading call stack would look like this:    All Layers and Services have implemented a unified Accessor interface, erasing all type information when building the Operator. For the Operator, regardless of what services are used or how many middleware are added, all call logic is consistent. This design splits OpenDAL's API into Public API and Raw API, where the Public API is directly exposed to users, providing convenient top-level interfaces, and Raw API is provided to OpenDAL internal developers, maintaining a unified internal interface and providing some convenient implementation.  ","version":null,"tagName":"h2"},{"title":"Operator​","type":1,"pageTitle":"Apache OpenDAL™ Internal: Data Reading","url":"/blog/how-opendal-read-data/#operator","content":" OpenDAL's Operator API will adhere to a consistent calling paradigm as much as possible, reducing users' learning and usage costs. For example, OpenDAL offers the following APIs for read:  op.read(path): Reads the entire content of the specified fileop.reader(path): Creates a Reader for streaming readingop.read_with(path).range(1..1024): Reads file content using specified parameters, such as rangeop.reader_with(path).range(1..1024): Creates a Reader for streaming reading with specified parameters  It's not hard to see that read is more like syntactic sugar, allowing users to quickly read files without considering various traits like AsyncRead. The reader provides more flexibility, implementing widely-used community traits like AsyncSeek, AsyncRead, allowing more flexible data reading. read_with and reader_with assist users in specifying various parameters in a more natural way through Future Builder functions.  The internal logic of the Operator would look like this:    Its main job is to encapsulate the interface for the user:  Completing the construction of OpRead: the args for read operation.Calling the read function provided by AccessorWrapping the returned value as Reader and implementing interfaces like AsyncSeek, AsyncRead, etc., based on Reader  ","version":null,"tagName":"h2"},{"title":"Layers​","type":1,"pageTitle":"Apache OpenDAL™ Internal: Data Reading","url":"/blog/how-opendal-read-data/#layers","content":" A little secret here is that OpenDAL will automatically apply some Layers to the Service to implement some internal logic. As of the completion of this article, OpenDAL's automatically added Layers include:  ErrorContextLayer: Injects context information, such as scheme, path, etc., into all returned errors of OperationCompleteLayer: Adds necessary capabilities to services, such as adding seek support to s3TypeEraseLayer: Implements type erasure, uniformly erasing associated types in Accessor, so users don't need to carry generic parameters when using it  Here, ErrorContextLayer and TypeEraseLayer are relatively simple and won't be elaborated on. The focus is on CompleteLayer, aimed at adding seek or next support to OpenDAL's returned Reader in a zero-cost way, so users don't have to re-implement it. OpenDAL initially returned Reader and SeekableReader through different function calls in early versions, but the actual user feedback was not very good; almost all users were using SeekableReader. Therefore, OpenDAL subsequently added seek support as the first priority to the internal Read trait during the refactor:  pub trait Read: Unpin + Send + Sync { /// Read bytes asynchronously. fn poll_read(&mut self, cx: &mut Context<'_>, buf: &mut [u8]) -> Poll<Result<usize>>; /// Seek asynchronously. /// /// Returns `Unsupported` error if underlying reader doesn't support seek. fn poll_seek(&mut self, cx: &mut Context<'_>, pos: io::SeekFrom) -> Poll<Result<u64>>; /// Stream [`Bytes`] from underlying reader. /// /// Returns `Unsupported` error if underlying reader doesn't support stream. /// /// This API exists for avoiding bytes copying inside async runtime. /// Users can poll bytes from underlying reader and decide when to /// read/consume them. fn poll_next(&mut self, cx: &mut Context<'_>) -> Poll<Option<Result<Bytes>>>; }   To implement a service's reading capability in OpenDAL, one needs to implement this trait, which is an internal interface and will not be directly exposed to users. Among them:  poll_read is the most basic requirement; all services must implement this interface.When the service natively supports seek, poll_seek can be implemented, and OpenDAL will correctly dispatch, such as local fs;When the service natively supports next, meaning it returns streaming Bytes, poll_next can be implemented, like HTTP-based services, where the underlying layer is a TCP Stream, and hyper will encapsulate it as a bytes stream.  Through the Read trait, OpenDAL ensures that all services can expose their native support capabilities as much as possible, thereby achieving efficient reading for different services.  Based on this trait, OpenDAL will complete according to the capabilities supported by each service:  Both seek/next are supported: Direct returnNo support for next: Encapsulate using StreamableReader to simulate next supportNo support for seek: Encapsulate using ByRangeSeekableReader to simulate seek supportNeither seek/next supported: Encapsulate using both methods  ByRangeSeekableReader mainly utilizes the service's ability to support range read, dropping the current reader when the user seeks and initiating a new request at the specified location.  OpenDAL exposes a unified Reader implementation through CompleteLayer, so users don't have to worry about whether the underlying service supports seek; OpenDAL will always choose the optimal way to initiate the request.  ","version":null,"tagName":"h2"},{"title":"Services​","type":1,"pageTitle":"Apache OpenDAL™ Internal: Data Reading","url":"/blog/how-opendal-read-data/#services","content":" After the completion of the Layers, it's time to call the specific implementation of the Service. Here, the most common services fs and s3 are used as examples to explain how data is read.  ","version":null,"tagName":"h2"},{"title":"Service fs​","type":1,"pageTitle":"Apache OpenDAL™ Internal: Data Reading","url":"/blog/how-opendal-read-data/#service-fs","content":" tokio::fs::File implements tokio::AsyncRead and tokio::AsyncSeek. Using async_compat::Compat, we have transformed it into futures::AsyncRead and futures::AsyncSeek. Based on this, we provide a built-in function oio::into_read_from_file to transform it into a type that implements oio::Read.  There's nothing particularly complex in the implementation of oio::into_read_from_file; read and seek are mostly calling the functions provided by the incoming File type. The tricky part is about the correct handling of seek and range: seeking to the right side of the range is allowed, and this will not cause an error, and reading will only return empty, but seeking to the left side of the range is illegal, and the Reader must return InvalidInput for proper upper-level handling.  Interesting history: there was an issue in the initial implementation of this part, discovered during fuzz testing.  ","version":null,"tagName":"h3"},{"title":"Services s3​","type":1,"pageTitle":"Apache OpenDAL™ Internal: Data Reading","url":"/blog/how-opendal-read-data/#services-s3","content":" S3 is an HTTP-based service, and opendal provides a lot of HTTP-based wrappers to help developers reuse logic; they only need to build a request and return a well-constructed Body. OpenDAL Raw API encapsulates a set of reqwest-based interfaces, and the HTTP GET interface returns a Response<IncomingAsyncBody>:  /// IncomingAsyncBody carries the content returned by remote servers. pub struct IncomingAsyncBody { /// # TODO /// /// hyper returns `impl Stream<Item = crate::Result<Bytes>>` but we can't /// write the types in stable. So we will box here. /// /// After [TAIT](https://rust-lang.github.io/rfcs/2515-type_alias_impl_trait.html) /// has been stable, we can change `IncomingAsyncBody` into `IncomingAsyncBody<S>`. inner: oio::Streamer, size: Option<u64>, consumed: u64, chunk: Option<Bytes>, }   The stream contained in this body is the bytes stream returned by reqwest, and opendal implements content length checks and read support on this basis.  Here's an extra note about a small pitfall with reqwest/hyper: reqwest and hyper do not check the returned content length, so an illegal server may return a data volume that does not match the expected content length instead of an error, leading to unexpected data behavior. OpenDAL specifically added checks here, returning ContentIncomplete when data is insufficient and ContentTruncated when data exceeds expectations, avoiding users receiving illegal data.  ","version":null,"tagName":"h3"},{"title":"Conclusion​","type":1,"pageTitle":"Apache OpenDAL™ Internal: Data Reading","url":"/blog/how-opendal-read-data/#conclusion","content":" This article introduces from top to bottom how OpenDAL implements data reading:  Operator is responsible for exposing user-friendly interfacesLayers are responsible for completing the capabilities of the servicesServices are responsible for the specific implementation of different services  Throughout the entire chain, OpenDAL adheres as much as possible to the principle of zero cost, prioritizing the use of native service capabilities, then considering simulation through other methods, and finally returning unsupported errors. Through this three-tier design, users don't need to understand the details of the underlying service, nor do they need to integrate different service SDKs to easily call op.read(path) to access data in any storage service.  This is: How OpenDAL read data freely! ","version":null,"tagName":"h2"},{"title":"OwO #1: The v0.40 Release","type":0,"sectionRef":"#","url":"/blog/owo-1/","content":"","keywords":"","version":null},{"title":"Outcome​","type":1,"pageTitle":"OwO #1: The v0.40 Release","url":"/blog/owo-1/#outcome","content":" OpenDAL now comprises four primary components:  Core: The core library written in Rust.Bindings: Language bindings powered by the OpenDAL Rust core.Applications: Applications built using the OpenDAL Rust core.Integrations: Collaborations with other projects.  ","version":null,"tagName":"h2"},{"title":"Core​","type":1,"pageTitle":"OwO #1: The v0.40 Release","url":"/blog/owo-1/#core","content":" Unifying Append and Write Functions​  OpenDAL has supported append operations since v0.36. We've found, however, that this led to significant duplication between append and write. As a result, we've streamlined the two functionalities into a single write function. Our users can now:  let mut w = op.writer_with("test.txt").append(true).await?; w.write(content_a).await?; w.write(content_b).await?; w.close().await?;   This way, users can reuse the Writer in their own logic without handling append separately.  New Lister API​  To improve API consistency, we've made some adjustments to our listing functions. We've added list and list_with methods that perform single operations and renamed the original list to lister and lister_with.  // Old API let lister: Lister = op.list("dir").await?; // New API let entries: Vec<Entry> = op.list("dir").await?; let lister: Lister = op.lister("dir").await?;   This brings uniformity to our API offerings.  List With Metakey​  To speed up list operations, OpenDAL can now fetch and store metadata during the listing process. This eliminates the need for separate metadata calls:  let entries: Vec<Entry> = op .list_with("dir/") .metakey(Metakey::ContentLength | Metakey::ContentType).await?; // Use the metadata directly! let meta: &Metadata = entries[0].metadata();   This makes metadata retrieval more intuitive.  Buffered Writer​  We've added general buffer support to optimize writing operations.  let w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?   Others​  Other improvements in the core library can be found in our CHANGELOG.  ","version":null,"tagName":"h3"},{"title":"Bindings​","type":1,"pageTitle":"OwO #1: The v0.40 Release","url":"/blog/owo-1/#bindings","content":" C++​  opendal-cpp is ready for its first release! Welcome to check it out and give us some feedback.  Haskell​  opendal-hs is ready for its first release! Welcome to check it out and give us some feedback.  Java​  opendal-java enabled more available services in this release, allowing user to visit services like redis that not enabled by default in rust core. And opendal-java enabled blocking layer to allow users visit services like s3 in blocking way.  Welcome to integrate opendal-java into your project and give us some feedback.  New bindings!​  opendal-dotnetopendal-php  ","version":null,"tagName":"h3"},{"title":"Applications​","type":1,"pageTitle":"OwO #1: The v0.40 Release","url":"/blog/owo-1/#applications","content":" oay​  oay is OpenDAL Gateway that allows users to access OpenDAL services via existing protocols like s3 and webdav. It works like a proxy that forwarding requests to OpenDAL services.  In this release, we implement basic webdav support. Users can convert any storage services to a webdav server!  oli​  oli is OpenDAL CLI that allows users to access storage services via CLI like s3cmd and gcloud does.  We fixed some experience issues in this release and improved some docs. Welcome to try it out and give us some feedback.  ","version":null,"tagName":"h3"},{"title":"Integrations​","type":1,"pageTitle":"OwO #1: The v0.40 Release","url":"/blog/owo-1/#integrations","content":" object_store​  object_store instead to implement object_store's trait over OpenDAL Operator so that users can use OpenDAL as a backend for object_store.  object_store is mostly functional, but there are some edge use cases that OpenDAL has yet to support.  So far, this release hasn't seen progress in this area; we are awaiting the resolution of the issue Allow list paths that do not end with /.  ","version":null,"tagName":"h3"},{"title":"Working​","type":1,"pageTitle":"OwO #1: The v0.40 Release","url":"/blog/owo-1/#working","content":" We are working on the following things:  object_store support: Make object_store integration works and find a user for it.Remove the / limitation for path, so we can list a path without ending with /.Expand the start-after support to more services (Address #2786).  ","version":null,"tagName":"h2"},{"title":"Outlook​","type":1,"pageTitle":"OwO #1: The v0.40 Release","url":"/blog/owo-1/#outlook","content":" We are exploring some innovative ideas:  OpenDAL REST/gRPC API: A REST/gRPC Server for OpenDAL.OpenDAL Cache: OpenDAL native cache libs that allowing users to access data more efficiently.OpenDAL File System: A read-only file system that built upon OpenDAL in rust!kio-opendal: A kio plugin powered by OpenDAL that allows users to visit different storage services in KDE Dolphin.gvfs-opendal: A gvfs plugin powered by OpenDAL that allows users to visit different storage services in GNOME Files  Feel free to join in the discussion!  ","version":null,"tagName":"h2"},{"title":"Summary​","type":1,"pageTitle":"OwO #1: The v0.40 Release","url":"/blog/owo-1/#summary","content":" This marks our first OpenDAL OwO post. We welcome your feedback. ","version":null,"tagName":"h2"},{"title":"Community","type":0,"sectionRef":"#","url":"/community/","content":"","keywords":"","version":"Next"},{"title":"Mailing list​","type":1,"pageTitle":"Community","url":"/community/#mailing-list","content":" Name\tDesc\tSubscribe\tUnsubscribe\tPost\tArchivedev@opendal.apache.org\tDevelopment related discussions\tSubscribe\tUnsubscribe\tPost\tArchive commits@opendal.apache.org\tAll commits to our repositories\tSubscribe\tUnsubscribe\tRead only list\tArchive  Please make sure you are subscribed to the mailing list you are posting to!  If you are not subscribed to the mailing list, your message will either be rejected or you won't receive the response.  ","version":"Next","tagName":"h2"},{"title":"How to subscribe to a mailing list​","type":1,"pageTitle":"Community","url":"/community/#how-to-subscribe-to-a-mailing-list","content":" Before you can post a message to a mailing list, you need to subscribe to the list first.  Send an email without any contents or subject to listname-subscribe@opendal.apache.org. (replace listname with dev or user)Wait till you receive an email with the subject "confirm subscribe to listname@opendal.apache.org". Reply to that email, without editing the subject or including any contents.Wait till you receive an email with the subject "WELCOME to listname@opendal.apache.org".  If you email us with a code snippet, make sure that:  you do not link to files in external services as such files can change, get deleted or the link might break and thus make an archived email thread uselessyou paste text instead of screenshots of textyou keep formatting when pasting code to keep the code readablethere are enough import statements to avoid ambiguities  ","version":"Next","tagName":"h3"},{"title":"Issue tracker​","type":1,"pageTitle":"Community","url":"/community/#issue-tracker","content":" We use GitHub Issues to track all code related issues: https://github.com/apache/opendal/issues  You must have a GitHub account to log cases and issues.  ","version":"Next","tagName":"h2"},{"title":"Bug reports​","type":1,"pageTitle":"Community","url":"/community/#bug-reports","content":" Found bug? Enter an issue in the issue tracker.  Before submitting an issue, please:  Verify that the bug does in fact exist.Search the issue tracker to verify there is no existing issue reporting the bug you've found.Consider tracking down the bug yourself in the source code of OpenDAL and submitting a patch along with your bug report. This is a great time saver for the OpenDAL developers and helps ensure the bug will be fixed quickly.  ","version":"Next","tagName":"h3"},{"title":"Enhancement​","type":1,"pageTitle":"Community","url":"/community/#enhancement","content":" Enhancements or new feature proposals are also welcome. The more concrete and rationale the proposal is, the greater the chance it will be incorporated into future releases.  ","version":"Next","tagName":"h3"},{"title":"Source code​","type":1,"pageTitle":"Community","url":"/community/#source-code","content":" OpenDAL core repository: https://github.com/apache/opendal  ","version":"Next","tagName":"h2"},{"title":"Upcoming Community Events​","type":1,"pageTitle":"Community","url":"/community/#upcoming-community-events","content":" For now, we have a single community event known as the Community Sync Meeting. This meeting is open to everyone.  During the meeting, we will cover various topics including project status, roadmap updates, and other relevant discussions. We will also have a Q&A session at the end of the meeting.  Attending the meeting presents a valuable opportunity for you to meet fellow community members and gain deeper insights into the project. Moreover, it provides an excellent chance for active involvement in the project's activities.  We currently organize the community sync meeting every three weeks, with one scheduled at 09:00 UTC+8 and the other at 19:00 UTC+8. These two meeting times alternate between each three-week cycle.  The following is the calendar of the community events:  Click here to view the calendar in a new tab: Community Events Calendar.  If you prefer to add these events to your personal calendar, please use this iCal link:Add to your calendar.  ","version":"Next","tagName":"h2"},{"title":"People​","type":1,"pageTitle":"Community","url":"/community/#people","content":" Thank you to all the contributors for your selfless dedication and expertise in making this project more comprehensive and valuable. We sincerely appreciate your support and efforts!  ","version":"Next","tagName":"h2"},{"title":"PMC members and Committers​","type":1,"pageTitle":"Community","url":"/community/#pmc-members-and-committers","content":" The committers list could be found here.  ","version":"Next","tagName":"h3"},{"title":"Contributors​","type":1,"pageTitle":"Community","url":"/community/#contributors","content":" The contributor list could be found here. ","version":"Next","tagName":"h3"},{"title":"Onboarding","type":0,"sectionRef":"#","url":"/community/committers/onboarding/","content":"","keywords":"","version":"Next"},{"title":"Submit CLA​","type":1,"pageTitle":"Onboarding","url":"/community/committers/onboarding/#submit-cla","content":" Download the ICLA from https://www.apache.org/licenses/contributor-agreements.html#clas. If a corporation assigns employees to work on an Apache project, please download the CCLA.Complete the ICLA based on your particulars. Please note: The address field should be filled out accurately and in detail.You need to choose a unique ApacheID that hasn't been taken. Check https://people.apache.org/committer-index.html to see which IDs are still available. Sign the document by hand or by electronic signature Manually sign a printed copy, then scan it to produce a pdf.Digitally draw a signature on the document: Detail Instruction.Sign the document using PGP: Detail Instruction. Send your icla.pdf (and icla.pdf.asc if PGP-signed) to secretary@apache.org.  After waiting for some time, you will receive an email notifying you that your CLA has been successfully recorded.  ","version":"Next","tagName":"h2"},{"title":"Setup ASF Account​","type":1,"pageTitle":"Onboarding","url":"/community/committers/onboarding/#setup-asf-account","content":" When receiving an email with the subject "Welcome to the Apache Software Foundation" from root@apache.org, we can begin setting up an ASF account.  ","version":"Next","tagName":"h2"},{"title":"Setup LDAP Password​","type":1,"pageTitle":"Onboarding","url":"/community/committers/onboarding/#setup-ldap-password","content":" Go to https://id.apache.org/reset/enter and enter your ApacheID.Check your email and click the provided link to reset your password.  ","version":"Next","tagName":"h3"},{"title":"Link ASF Account to GitHub​","type":1,"pageTitle":"Onboarding","url":"/community/committers/onboarding/#link-asf-account-to-github","content":" Navigate to https://gitbox.apache.org/boxer/ and enter your ApacheID and password.Click "Authenticate with GitHub" and follow the given instructions to link your ASF account to GitHub.Check your email titled "[GitHub] @asfgit has invited you to join the @apache organization" and accept the invitation.Wait momentarily, and the website will refresh on its own.(If you do not enable 2FA on GitHub) Please follow the instruction.  Your ApacheID and GitHub ID will now both appear on https://gitbox.apache.org/boxer/. Congrats on successfully linking your ASF account to GitHub!  ","version":"Next","tagName":"h3"},{"title":"Email Settings​","type":1,"pageTitle":"Onboarding","url":"/community/committers/onboarding/#email-settings","content":" Note: Apache does not provide a mailbox directly.  ","version":"Next","tagName":"h2"},{"title":"Receive Email​","type":1,"pageTitle":"Onboarding","url":"/community/committers/onboarding/#receive-email","content":" You can change your forwarding email address at Apache Account Utility Platform  ","version":"Next","tagName":"h3"},{"title":"Send Email​","type":1,"pageTitle":"Onboarding","url":"/community/committers/onboarding/#send-email","content":" To send emails using your apache.org address, configure your email client to leverage the mail-relay service. For specifics, refer to this guide.  Here's an illustration for Gmail users:  Open Gmail settings and select "See all settings".Navigate to "Accounts and Import", then locate "Send mail as".Click "Add another email address" and enter your name and apache.org email address.Input the SMTP server information: SMTP Server: mail-relay.apache.orgPort: 587Username: your apacheIDPassword: your apacheID passwordSecured connection using TLS Click "Add account" and you will receive an email from Gmail that need to confirm.    ","version":"Next","tagName":"h3"},{"title":"Subscribe to Mailing List​","type":1,"pageTitle":"Onboarding","url":"/community/committers/onboarding/#subscribe-to-mailing-list","content":" Send email to dev-subscribe@opendal.apache.orgYou will receive an email with the subject "confirm subscribe to dev@opendal.apache.org"Reply to the email with "Confirm" in the body  If you receive an email with the subject "WELCOME to dev@opendal.apache.org", you have successfully subscribed to the mailing list.  ","version":"Next","tagName":"h3"},{"title":"Setup 1password​","type":1,"pageTitle":"Onboarding","url":"/community/committers/onboarding/#setup-1password","content":" OpenDAL offers a 1Password open-source team license for conducting local integration tests (Thanks to 1Password!). Once you have been added to OpenDAL's committer list, one of the PMC members will invite you to join the team.  Please download your preferred clients to begin using it. You can create your own vault that is accessible only by yourself. Neither the 1password team nor OpenDAL PMC members can access it unless you choose to share it. ","version":"Next","tagName":"h2"},{"title":"Request for adding secrets","type":0,"sectionRef":"#","url":"/community/committers/request_for_adding_secrets/","content":"Request for adding secrets OpenDAL's behavior tests depend on the secrets stored in 1password. If you want to sponsor any service, please email private@opendal.apache.org with the subject line Request to Add Secrets for <service>. Our PMC will review the request and reply to you. After the request is approved, you can use the secrets in the behavior tests when they are running on the GitHub Actions (You need to be a committer of OpenDAL). The email should also include the following information: The name of the serviceThe secrets in key-value pairs If the secrets have expiration dates, please also include the expiration dates. We may need you to update the secrets in the future. If the secrets are not under your control, please also include the contact information of the owner of the secrets. And finally, thank you for your support!","keywords":"","version":"Next"},{"title":"OSPP Project Proposal Guidance","type":0,"sectionRef":"#","url":"/community/events/ospp-proposal-guide/","content":"","keywords":"","version":"Next"},{"title":"Choosing a project​","type":1,"pageTitle":"OSPP Project Proposal Guidance","url":"/community/events/ospp-proposal-guide/#choosing-a-project","content":" You should start by deciding on which project do you want to work on. You can use our list of project ideas as an inspiration.  ","version":"Next","tagName":"h2"},{"title":"Interacting with the OpenDAL community​","type":1,"pageTitle":"OSPP Project Proposal Guidance","url":"/community/events/ospp-proposal-guide/#interacting-with-the-opendal-community","content":" If you want to discuss our suggested project ideas, you can do so on the maillist or Discord. Make sure to listen to the feedback of the mentors, and try to incorporate it in your project proposal.  When communicating on the OpenDAL mail list or discord (and when interacting with the OpenDAL community in general), please remember to be polite and uphold the ASF Code of Conduct. Keep in mind that most OpenDAL contributors (and OSPP OpenDAL project mentors) are volunteers, and work on OpenDAL in their free time, so please treat them with respect and avoid spamming.  ","version":"Next","tagName":"h2"},{"title":"Creating the project proposal​","type":1,"pageTitle":"OSPP Project Proposal Guidance","url":"/community/events/ospp-proposal-guide/#creating-the-project-proposal","content":" Ultimately, the project proposal is the main deciding factor on whether your project will be accepted or not, so make sure that you put energy into making it as good as possible.  Please carefully read the Project Application Suggestions.Prepare relevant materials based on the project requirements and refer to the Project Application Template.The quality of the project application is the primary criterion for the committee to evaluate whether the student's application is approved or not.This is the time to understand the project details and to become familiar with the community operation and etiquette.  ","version":"Next","tagName":"h2"},{"title":"Project information and timeline​","type":1,"pageTitle":"OSPP Project Proposal Guidance","url":"/community/events/ospp-proposal-guide/#project-information-and-timeline","content":" This is the most important part of your project proposal. You should include an abstract that explains your project in one or two paragraphs, and then a very detailed description that explains what exactly do you want to achieve in the proposed project. The proposal should also clearly state the designated mentor(s) for your project (you should get in touch with them before submitting the proposal).  Describe a brief outline of the work that you plan to do, and try to estimate how will the work be split in the individual weeks of the project.Define milestones that you intend to achieve in specific weeks (e.g. finish X in week 4, deliver Y in the middle of the project, have a final version prepared one week before the end of the project, etc.). You should focus specifically on the midterm point (week 6), because your mentor(s) will evaluate your progress at this time. You should be slightly more than half done at this moment, and have something reasonable to show.In week 11 (one week before the end of the project), you should consider doing a "code freeze", and spend the last week to polish tests and documentation.  Of course, it is quite difficult to predict this timeline exactly in advance, and it is not a problem to modify it while the project runs, but try to guesstimate it to the best of your ability.  Furthermore, let us know what is your intended time commitment for working on the project. How many hours per day can you work on it? Are there specific days of the week when you can work on it? Is there some period of time from May to August where you know in advance that you won't be able to work on it? Please include this information in the proposal.  There is a Community bonding period before the contributors start working on their projects. It is designed to help you learn about the community that you're going to contribute to, and to start familiarizing yourself with the code and/or technology of your project. Please include a short description of preparatory work that you intend to work on during this community bonding period (should your project be accepted).  ","version":"Next","tagName":"h2"},{"title":"How to increase your chance of being accepted?​","type":1,"pageTitle":"OSPP Project Proposal Guidance","url":"/community/events/ospp-proposal-guide/#how-to-increase-your-chance-of-being-accepted","content":" You can demonstrate your dedication (and ability) to work on the selected project proposal by contributing something related to it before your proposal is evaluated. This can encompass e.g. sending a pull request to the relevant repository, fixing a bug, writing documentation, etc. There is no specific template for these kinds of contributions, and it might not be possible to do for all types of projects. You can coordinate with the project mentors to find out if they can suggest some entry-level task for you.  You can also tell us more about your motivation in your proposal. Why did you choose OpenDAL for a OSPP project specifically? Do you like the OpenDAL language? Is the specific project that you want to work on sympathetic to you for some reason? We would like to know!  ","version":"Next","tagName":"h2"},{"title":"Don't forget to submit!​","type":1,"pageTitle":"OSPP Project Proposal Guidance","url":"/community/events/ospp-proposal-guide/#dont-forget-to-submit","content":" You will need to submit your project proposal through the Open Source Promotion Plan website. Please keep the deadline (4th Jun 2024) in mind, as there will be no extensions!  Good luck! :)  ","version":"Next","tagName":"h2"},{"title":"How to decrease your chance of being accepted?​","type":1,"pageTitle":"OSPP Project Proposal Guidance","url":"/community/events/ospp-proposal-guide/#how-to-decrease-your-chance-of-being-accepted","content":" There are some actions and behaviours that will make it much less likely that your application will be considered, so you should avoid these. For example:  Spamming or harassing mentors or other members of the OpenDAL community.Letting AI automatically generate your project proposal (you should put effort in it, don't be lazy!).Suggesting unreasonably grandiose project proposals, e.g. adding a garbage collector to OpenDAL.Suggesting unreasonably trivial project proposals, e.g. fixing a typo in the OpenDAL documentation.  This guide was inspired by https://github.com/rust-lang/google-summer-of-code/blob/main/proposal-guide.md. ","version":"Next","tagName":"h2"},{"title":"GSoC Project Proposal Guidance","type":0,"sectionRef":"#","url":"/community/events/gsoc-proposal-guide/","content":"","keywords":"","version":"Next"},{"title":"Choosing a project​","type":1,"pageTitle":"GSoC Project Proposal Guidance","url":"/community/events/gsoc-proposal-guide/#choosing-a-project","content":" You should start by deciding on which project do you want to work on. You can use our list of project ideasas an inspiration, or you can come up with your own project idea. However, you should keep in mind that each GSoC project needs at least one mentor available. Therefore, if you come up with a completely new project idea, you should also try to find someone from the OpenDAL community who could mentor you on the project.  If you decide to propose your own project idea, you're most likely to be able to find a mentor if you can describe clearly the utility of the project to OpenDAL community.  We encourage you to think of your own interesting project ideas! There are plenty of things that can be done within the OpenDAL project and contributors are generally happy to discuss and help you narrow down your thoughts into a concrete proposal. Don't be shy!  ","version":"Next","tagName":"h2"},{"title":"Interacting with the OpenDAL community​","type":1,"pageTitle":"GSoC Project Proposal Guidance","url":"/community/events/gsoc-proposal-guide/#interacting-with-the-opendal-community","content":" If you want to discuss our suggested project ideas or your own idea, you can do so on the maillist or Discord. Make sure to listen to the feedback of the mentors, and try to incorporate it in your project proposal.  When communicating on the OpenDAL mail list or discord (and when interacting with the OpenDAL community in general), please remember to be polite and uphold the ASF Code of Conduct. Keep in mind that most OpenDAL contributors (and GSoC OpenDAL project mentors) are volunteers, and work on OpenDAL in their free time, so please treat them with respect and avoid spamming.  ","version":"Next","tagName":"h2"},{"title":"Creating the project proposal​","type":1,"pageTitle":"GSoC Project Proposal Guidance","url":"/community/events/gsoc-proposal-guide/#creating-the-project-proposal","content":" Ultimately, the project proposal is the main deciding factor on whether your project will be accepted or not, so make sure that you put energy into making it as good as possible.  The proposal should contain (at least) the following things:  A descriptive title of the project that you want to work onInformation about yourself, including:  Description of your programming experience, attained education, university and study programme that you're currently studying, etc. (a short CV would be ideal)Link to a portfolio of projects that you have worked on (e.g. a GitHub profile or a personal website)Your knowledge of OpenDAL, since most projects will probably require at least some OpenDAL knowledgeYour existing open-source contribution experience. If you have already contributed to some open-source repositories, make sure to include a link to these contributions in your proposal!Your preferred time zone (for communicating with the mentor(s))Contact information  Information about your proposed project. This should be as detailed as possible, see more details below.Information about other commitments that might affect your ability to work on the project during the GSoC period. These can include vacations, exams, other jobs or internships etc. It's not necessarily an issue to have other commitments, but it would be great to know about them in advance, if possible.  ","version":"Next","tagName":"h2"},{"title":"Project information and timeline​","type":1,"pageTitle":"GSoC Project Proposal Guidance","url":"/community/events/gsoc-proposal-guide/#project-information-and-timeline","content":" This is the most important part of your project proposal. You should include an abstract that explains your project in one or two paragraphs, and then a very detailed description that explains what exactly do you want to achieve in the proposed project. The proposal should also clearly state the designated mentor(s) for your project (you should get in touch with them before submitting the proposal).  In addition to describing what do you intend to work on in the project, you should also specify the size of the project, according to the GSoC documentation:  Small: ~90 hoursMedium: ~175 hoursLarge: ~350 hours  You should also create an approximate weekly plan of work and a list of deliverables. Recall that the default project duration is 12 weeks, but it can be extended (for medium and large projects) by up to 22 weeks.  Describe a brief outline of the work that you plan to do, and try to estimate how will the work be split in the individual weeks of the project.Define milestones that you intend to achieve in specific weeks (e.g. finish X in week 4, deliver Y in the middle of the project, have a final version prepared one week before the end of the project, etc.). You should focus specifically on the midterm point (week 6), because your mentor(s) will evaluate your progress at this time. You should be slightly more than half done at this moment, and have something reasonable to show.In week 11 (one week before the end of the project), you should consider doing a "code freeze", and spend the last week to polish tests and documentation.  Of course, it is quite difficult to predict this timeline exactly in advance, and it is not a problem to modify it while the project runs, but try to guesstimate it to the best of your ability.  Furthermore, let us know what is your intended time commitment for working on the project. How many hours per day can you work on it? Are there specific days of the week when you can work on it? Is there some period of time from May to August where you know in advance that you won't be able to work on it? Please include this information in the proposal.  There is a Community bonding period before the contributors start working on their projects. It is designed to help you learn about the community that you're going to contribute to, and to start familiarizing yourself with the code and/or technology of your project. Please include a short description of preparatory work that you intend to work on during this community bonding period (should your project be accepted).  ","version":"Next","tagName":"h2"},{"title":"How to increase your chance of being accepted?​","type":1,"pageTitle":"GSoC Project Proposal Guidance","url":"/community/events/gsoc-proposal-guide/#how-to-increase-your-chance-of-being-accepted","content":" You can demonstrate your dedication (and ability) to work on the selected project proposal by contributing something related to it before your proposal is evaluated. This can encompass e.g. sending a pull request to the relevant repository, fixing a bug, writing documentation, etc. There is no specific template for these kinds of contributions, and it might not be possible to do for all types of projects. You can coordinate with the project mentors to find out if they can suggest some entry-level task for you.  You can also tell us more about your motivation in your proposal. Why did you choose OpenDAL for a GSoC project specifically? Do you like the OpenDAL language? Is the specific project that you want to work on sympathetic to you for some reason? We would like to know!  ","version":"Next","tagName":"h2"},{"title":"Don't forget to submit!​","type":1,"pageTitle":"GSoC Project Proposal Guidance","url":"/community/events/gsoc-proposal-guide/#dont-forget-to-submit","content":" You will need to submit your project proposal through the Google Summer of Code website. Please keep the deadline (2nd April 2024) in mind, as there will be no extensions!  Good luck! :)  ","version":"Next","tagName":"h2"},{"title":"How to decrease your chance of being accepted?​","type":1,"pageTitle":"GSoC Project Proposal Guidance","url":"/community/events/gsoc-proposal-guide/#how-to-decrease-your-chance-of-being-accepted","content":" There are some actions and behaviours that will make it much less likely that your application will be considered, so you should avoid these. For example:  Spamming or harassing mentors or other members of the OpenDAL community.Letting AI automatically generate your project proposal (you should put effort in it, don't be lazy!).Suggesting unreasonably grandiose project proposals, e.g. adding a garbage collector to OpenDAL.Suggesting unreasonably trivial project proposals, e.g. fixing a typo in the OpenDAL documentation. Remember that even the smallest project size should take about 90 hours!  This guide was inspired by https://github.com/rust-lang/google-summer-of-code/blob/main/proposal-guide.md. ","version":"Next","tagName":"h2"},{"title":"Board reporting","type":0,"sectionRef":"#","url":"/community/pmc_members/board-reporting/","content":"","keywords":"","version":"Next"},{"title":"Frequency of Reporting​","type":1,"pageTitle":"Board reporting","url":"/community/pmc_members/board-reporting/#frequency-of-reporting","content":" New graduates from the Apache Incubator are required to report monthly for the first three months, then quarterly.  The current reporting frequency for Apache OpenDAL™ is once per quarter.  ","version":"Next","tagName":"h3"},{"title":"Report Preparation​","type":1,"pageTitle":"Board reporting","url":"/community/pmc_members/board-reporting/#report-preparation","content":" It's recommended to discuss the report on dev@opendal.apache.org (subscribe), inviting all members to contribute. Here are the critical points your report should address:  Any concerns that require the attention of the ASF Board.Legal, infrastructure, cross-project, or personal issues that need addressing.Achievements (releases, milestones, etc.) since the last report.Any other significant points or thoughts you think should be included.  You can discuss the issues based on the template or refer to previous reports for preparation.  ","version":"Next","tagName":"h3"},{"title":"Report Writing and Submission​","type":1,"pageTitle":"Board reporting","url":"/community/pmc_members/board-reporting/#report-writing-and-submission","content":" The report should be written in Markdown format and submitted to the Apache Board of Directors via the Board Reporting Service. ","version":"Next","tagName":"h3"},{"title":"Nominate Committer","type":0,"sectionRef":"#","url":"/community/pmc_members/nominate-committer/","content":"","keywords":"","version":"Next"},{"title":"Start vote about the candidate​","type":1,"pageTitle":"Nominate Committer","url":"/community/pmc_members/nominate-committer/#start-vote-about-the-candidate","content":" Start a vote about the candidate via sending email to: private@opendal.apache.org:  candidate_name: The full name of the candidate.candidate_github_id: The GitHub id of the candidate.  Title:  [VOTE] Add candidate ${candidate_name} as a new committer   Content:  Hi, All OpenDAL PMC members. I'd like to take this chance to call the vote for inviting committed contributor ${candidate_name} (GitHub id: ${candidate_github_id}) as a new committer of Apache OpenDAL. ${candidate_contributions} ${candidate_name}'s great contributions can be found: - GitHub Account: https://github.com/${candidate_github_id} - GitHub Pull Requests: https://github.com/apache/opendal/pulls?q=is%3Apr+author%3A${candidate_github_id} - GitHub Issues: https://github.com/apache/opendal/issues?q=is%3Aissue+involves%3A${candidate_github_id} Please make your valuable evaluation on whether we could invite ${candidate_name} as a committer: [ +1 ] Agree to add ${candidate_name} as a committer of OpenDAL. [ 0 ] Have no sense. [ -1 ] Disagree to add ${candidate_name} as a committer of OpenDAL, because ..... This vote starts from the moment of sending and will be open for 3 days. Thanks and best regards, ${your_name}   Example: https://lists.apache.org/thread/j16lvkyrmvg8wyf3z4gqpjky5m594jhy (Private Link)  After at least 3 +1 binding vote and no veto, claim the vote result:  Title:  [RESULT][VOTE] Add candidate ${candidate_name} as a new committer   Content:  Hi, all: The vote for "Add candidate ${candidate_name} as a new committer" has PASSED and closed now. The result is as follows: 4 binding +1 Votes: - voter names Vote thread: https://lists.apache.org/thread/j16lvkyrmvg8wyf3z4gqpjky5m594jhy Then I'm going to invite ${candidate_name} to join us. Thanks for everyone's support! ${your_name}   ","version":"Next","tagName":"h2"},{"title":"Send invitation to the candidate​","type":1,"pageTitle":"Nominate Committer","url":"/community/pmc_members/nominate-committer/#send-invitation-to-the-candidate","content":" Send an invitation to the candidate and cc private@opendal.apache.org:  Title:  Invitation to become OpenDAL Committer: ${candidate_name}   Content:  Hello ${candidate_name}, The OpenDAL PMC hereby offers you committer privileges to the project. These privileges are offered on the understanding that you'll use them reasonably and with common sense. We like to work on trust rather than unnecessary constraints. Being a committer enables you to more easily make changes without needing to go through the patch submission process. Being a committer does not require you to participate any more than you already do. It does tend to make one even more committed. You will probably find that you spend more time here. Of course, you can decline and instead remain as a contributor, participating as you do now. A. This personal invitation is a chance for you to accept or decline in private. Either way, please let us know in reply to the [private@opendal.apache.org] address only. B. If you accept, the next step is to register an iCLA: 1. Details of the iCLA and the forms are found through this link: https://www.apache.org/licenses/#clas 2. Instructions for its completion and return to the Secretary of the ASF are found at https://www.apache.org/licenses/#submitting 3. When you transmit the completed iCLA, request to notify the Apache OpenDAL and choose a unique Apache ID. Look to see if your preferred ID is already taken at https://people.apache.org/committer-index.html This will allow the Secretary to notify the PMC when your iCLA has been recorded. When recording of your iCLA is noted, you will receive a follow-up message with the next steps for establishing you as a committer. With the expectation of your acceptance, welcome! ${your_name} (as represents of The Apache OpenDAL PMC)   ","version":"Next","tagName":"h2"},{"title":"Add the candidate to the committer list​","type":1,"pageTitle":"Nominate Committer","url":"/community/pmc_members/nominate-committer/#add-the-candidate-to-the-committer-list","content":" After the candidate accepts the invitation and the iCLA is recorded, add the candidate to the committer list by whimsy roster tools    ","version":"Next","tagName":"h2"},{"title":"Announcement​","type":1,"pageTitle":"Nominate Committer","url":"/community/pmc_members/nominate-committer/#announcement","content":" Once the nominee accepts the invitation and the commit bit is granted, it's encouraged to send an announcement email to dev@opendal.apache.org to welcome the new committers. Here is an email template:  Hello, everyone On behalf of the Apache OpenDAL PMC, I'm happy to announce that ${candidate_name} has accepted the invitation to become a committer of Apache OpenDAL. Welcome, and thank you for your contributions! ${your name}  ","version":"Next","tagName":"h2"},{"title":"Maturity Assessment for Apache OpenDAL™","type":0,"sectionRef":"#","url":"/community/maturity/","content":"","keywords":"","version":"Next"},{"title":"Status of this assessment​","type":1,"pageTitle":"Maturity Assessment for Apache OpenDAL™","url":"/community/maturity/#status-of-this-assessment","content":" This assessment is evaluated during OpenDAL's graduation, which is finished on 2024-01-18.  ","version":"Next","tagName":"h2"},{"title":"Maturity model assessment​","type":1,"pageTitle":"Maturity Assessment for Apache OpenDAL™","url":"/community/maturity/#maturity-model-assessment","content":" The following table is filled according to the Apache Maturity Model. Mentors and community members are welcome to comment and modify it.  ","version":"Next","tagName":"h2"},{"title":"CODE​","type":1,"pageTitle":"Maturity Assessment for Apache OpenDAL™","url":"/community/maturity/#code","content":" ID\tDescription\tStatusCD10\tThe project produces Open Source software for distribution to the public, at no charge.\tYES The project source code is licensed under the Apache License 2.0. CD20\tAnyone can easily discover and access the project's code..\tYES The official website includes GitHub link which can access the project's repository on GitHub directly. CD30\tAnyone using standard, widely-available tools, can build the code in a reproducible way.\tYES Apache OpenDAL provide how-to-build document for every component to tell user how to compile on bare metal, such as the core's. CD40\tThe full history of the project's code is available via a source code control system, in a way that allows anyone to recreate any released version.\tYES It depends on git, and anyone can view the full history of the project via commit logs. CD50\tThe source code control system establishes the provenance of each line of code in a reliable way, based on strong authentication of the committer. When third parties contribute code, commit messages provide reliable information about the code provenance.\tYES The project uses GitHub and managed by Apache Infra, it ensuring provenance of each line of code to a committer. And the third-party contributions are accepted in accordance with the contributing guides.  ","version":"Next","tagName":"h3"},{"title":"LICENSE​","type":1,"pageTitle":"Maturity Assessment for Apache OpenDAL™","url":"/community/maturity/#license","content":" ID\tDescription\tStatusLC10\tThe Apache License, version 2.0, covers the released code.\tYES The LICENSE is in GitHub repository. And all source files are with APLv2 header, checked by korandoru/hawkeye@v3.6.0. LC20\tLibraries that are mandatory dependencies of the project's code do not create more restrictions than the Apache License does.\tYES All dependencies are listed. LC30\tThe libraries mentioned in LC20 are available as Open Source software.\tYES All dependencies are listed are available as Open Source software LC40\tCommitters are bound by an Individual Contributor Agreement (the "Apache iCLA") that defines which code they may commit and how they need to identify code that is not their own.\tYES All committers have iCLAs. LC50\tThe project clearly defines and documents the copyright ownership of everything that the project produces.\tYES And all source files are with APLv2 header, checked by korandoru/hawkeye@v3.6.0.  ","version":"Next","tagName":"h3"},{"title":"Releases​","type":1,"pageTitle":"Maturity Assessment for Apache OpenDAL™","url":"/community/maturity/#releases","content":" ID\tDescription\tStatusRE10\tReleases consist of source code, distributed using standard and open archive formats that are expected to stay readable in the long term.\tYES Source release is distributed via dist.apache.org and linked from download page. RE20\tThe project's PPMC (Project Management Committee, see CS10) approves each software release in order to make the release an act of the Foundation.\tYES All releases have been voted at dev@opendal.apache.org and general@incubator.apache.org, and have at least 3 PPMC member's votes. RE30\tReleases are signed and/or distributed along with digests that anyone can reliably use to validate the downloaded archives.\tYES All releases are signed, and the KEYS are available. RE40\tThe project can distribute convenience binaries alongside source code, but they are not Apache Releases, they are provided with no guarantee.\tYES User can easily build binaries from source code, and we do not provide binaries as Apache Releases. RE50\tThe project documents a repeatable release process so that someone new to the project can independently generate the complete set of artifacts required for a release.\tYES We can follow the Release guide to make a new Apache OpenDAL release, and so far we had 6 different release managers.  ","version":"Next","tagName":"h3"},{"title":"Quality​","type":1,"pageTitle":"Maturity Assessment for Apache OpenDAL™","url":"/community/maturity/#quality","content":" ID\tDescription\tStatusQU10\tThe project is open and honest about the quality of its code. Various levels of quality and maturity for various modules are natural and acceptable as long as they are clearly communicated.\tYES We encourage user to report issues. QU20\tThe project puts a very high priority on producing secure software.\tYES All security issues will be addressed within 3 days. QU30\tThe project provides a well-documented, secure and private channel to report security issues, along with a documented way of responding to them.\tYes The official website provides a security page QU40\tThe project puts a high priority on backwards compatibility and aims to document any incompatible changes and provide tools and documentation to help users transition to new features.\tYes We follow semantic versions. As long as it's within one major version, it's backward compatible. And when any breaking changes added, we provide corresponding upgrade guides. QU50\tThe project strives to respond to documented bug reports in a timely manner.\tYES The project has resolved 1000+ issues and 2300+ pull requests so far, with very prompt response.  ","version":"Next","tagName":"h3"},{"title":"Community​","type":1,"pageTitle":"Maturity Assessment for Apache OpenDAL™","url":"/community/maturity/#community","content":" ID\tDescription\tStatusCO10\tThe project has a well-known homepage that points to all the information required to operate according to this maturity model.\tYES The official website includes all information user need to run Apache OpenDAL. CO20\tThe community welcomes contributions from anyone who acts in good faith and in a respectful manner, and who adds value to the project.\tYes We provide contributing guides for every component. And we also have a general contributing guide CO30\tContributions include source code, documentation, constructive bug reports, constructive discussions, marketing and generally anything that adds value to the project.\tYES All good contributions including code and non-code are welcomed. CO40\tThe community strives to be meritocratic and gives more rights and responsibilities to contributors who, over time, add value to the project.\tYES The community has elected 2 new PPMC members and 7 new committers so far. CO50\tThe project documents how contributors can earn more rights such as commit access or decision power, and applies these principles consistently.\tYES The community has clear docs on nominating committers and PPMC members CO60\tThe community operates based on consensus of its members (see CS10) who have decision power. Dictators, benevolent or not, are not welcome in Apache projects.\tYES All decisions are made after vote by community members. CO70\tThe project strives to answer user questions in a timely manner.\tYES We use dev@opendal.apache.org, GitHub issue and GitHub discussion to do this in a timely manner.  ","version":"Next","tagName":"h3"},{"title":"Consensus​","type":1,"pageTitle":"Maturity Assessment for Apache OpenDAL™","url":"/community/maturity/#consensus","content":" ID\tDescription\tStatusCS10\tThe project maintains a public list of its contributors who have decision power. The project's PPMC (Project Management Committee) consists of those contributors.\tYes See members with all PPMC members and committers. CS20\tDecisions require a consensus among PPMC members and are documented on the project's main communications channel. The PPMC takes community opinions into account, but the PPMC has the final word.\tYES All decisions are made by votes on dev@opendal.apache.org, and with at least 3 +1 votes from PPMC. CS30\tThe project uses documented voting rules to build consensus when discussion is not sufficient.\tYES The project uses the standard ASF voting rules. CS40\tIn Apache projects, vetoes are only valid for code commits. The person exercising the veto must justify it with a technical explanation, as per the Apache voting rules defined in CS30.\tYES Apache OpenDAL community has not used the veto power yet except for code commits. CS50\tAll "important" discussions happen asynchronously in written form on the project's main communications channel. Offline, face-to-face or private discussions that affect the project are also documented on that channel.\tYES All important discussions and conclusions are recorded in written form.  ","version":"Next","tagName":"h3"},{"title":"Independence​","type":1,"pageTitle":"Maturity Assessment for Apache OpenDAL™","url":"/community/maturity/#independence","content":" ID\tDescription\tStatusIN10\tThe project is independent from any corporate or organizational influence.\tYES The PPMC members and committer of Apache OpenDAL are from several different companies, and majority of them are NOT From the company that donated this project. IN20\tContributors act as themselves, not as representatives of a corporation or organization.\tYES The contributors act on their own initiative without representing a corporation or organization. ","version":"Next","tagName":"h3"},{"title":"Onboarding","type":0,"sectionRef":"#","url":"/community/pmc_members/onboarding/","content":"","keywords":"","version":"Next"},{"title":"Subscribe to Private Mailing List​","type":1,"pageTitle":"Onboarding","url":"/community/pmc_members/onboarding/#subscribe-to-private-mailing-list","content":" Send email to private-subscribe@opendal.apache.orgYou will receive an email with the subject "confirm subscribe to private@opendal.apache.org"Reply to the email with "Confirm" in the body  If you receive an email with the subject "WELCOME to private@opendal.apache.org", you have successfully subscribed to the private mailing list.  It's required for PMC members to subscribe the private mailing list. The private list is only for confidential discussions that should not be made public, such as the suitability of a particular individual to become a committer or a member of the PMC.  ","version":"Next","tagName":"h2"},{"title":"Setup 1Password Secrets Access​","type":1,"pageTitle":"Onboarding","url":"/community/pmc_members/onboarding/#setup-1password-secrets-access","content":" Once you have been added to OpenDAL's PMC member list, you will gain the access to all OpenDAL's secrets.  Please refrain from modifying secrets in the Services vault as this could disrupt our integration tests.Ensure that you keep all secrets secure and avoid sharing them with others or making them public. Do not commit them anywhere else.  ","version":"Next","tagName":"h2"},{"title":"Read PMC Member Guide​","type":1,"pageTitle":"Onboarding","url":"/community/pmc_members/onboarding/#read-pmc-member-guide","content":" Please read PMC GUIDE to know about the general responsibilities of Project Management Committee (PMC) members in managing our project and common how-to procedures for day to day maintenance. ","version":"Next","tagName":"h2"},{"title":"Nominate PMC Member","type":0,"sectionRef":"#","url":"/community/pmc_members/nominate-pmc-member/","content":"","keywords":"","version":"Next"},{"title":"Start vote about the candidate​","type":1,"pageTitle":"Nominate PMC Member","url":"/community/pmc_members/nominate-pmc-member/#start-vote-about-the-candidate","content":" Start a vote about the candidate via sending email to: private@opendal.apache.org:  candidate_name: The full name of the candidate.candidate_github_id: The GitHub id of the candidate.  Title:  [VOTE] Add candidate ${candidate_name} as a new PMC member   Content:  Hi, All OpenDAL PMC members. I would like to nominate ${candidate_name} (GitHub id: ${candidate_github_id}) as a candidate for the OpenDAL PMC member. Since becoming an OpenDAL committer, ${candidate_name} has made significant contributions to various modules of the project. ${candidate_name}'s great contributions can be found: - GitHub Account: https://github.com/${candidate_github_id} - GitHub Pull Requests: https://github.com/apache/opendal/pulls?q=is%3Apr+author%3A${candidate_github_id} - GitHub Issues: https://github.com/apache/opendal/issues?q=is%3Aissue+involves%3A${candidate_github_id} Please make your valuable evaluation on whether we could invite ${candidate_name} as a PMC member: [ +1 ] Agree to add ${candidate_name} as a PMC member of OpenDAL. [ 0 ] Have no sense. [ -1 ] Disagree to add ${candidate_name} as a PMC member of OpenDAL, because ..... This vote starts from the moment of sending and will be open for 3 days. Thanks and best regards, ${your_name}   Example: https://lists.apache.org/thread/yg2gz2tof3cvbrgp1wxzk6mf9o858h7t (Private Link)  After at least 3 +1 binding vote and no veto, claim the vote result:  Title:  [RESULT][VOTE] Add candidate ${candidate_name} as a new PMC member   Content:  Hi, all: The vote for "Add candidate ${candidate_name} as a new PMC member" has PASSED and closed now. The result is as follows: 4 binding +1 Votes: - voter names Vote thread: https://lists.apache.org/thread/yg2gz2tof3cvbrgp1wxzk6mf9o858h7t Then I'm going to invite ${candidate_name} to join us. Thanks for everyone's support! ${your_name}   ","version":"Next","tagName":"h2"},{"title":"Send NOTICE to Board after VOTE PASSED​","type":1,"pageTitle":"Nominate PMC Member","url":"/community/pmc_members/nominate-pmc-member/#send-notice-to-board-after-vote-passed","content":" The nominating PMC member should send a message to the Board board@apache.org with a reference to the vote result in the following form:  Title:  [NOTICE] ${candidate_name} for Apache OpenDAL PMC   Content:  ${candidate_name} has been voted as a new member of the Apache OpenDAL PMC. the vote thread is at: https://lists.apache.org/thread/yg2gz2tof3cvbrgp1wxzk6mf9o858h7t   ","version":"Next","tagName":"h2"},{"title":"Send invitation to the candidate​","type":1,"pageTitle":"Nominate PMC Member","url":"/community/pmc_members/nominate-pmc-member/#send-invitation-to-the-candidate","content":" Send an invitation to the candidate and cc private@opendal.apache.org:  Title:  Invitation to become Apache OpenDAL PMC Member: ${candidate_name}   Content:  Hello ${candidate_name}, In recognition of your contributions to Apache OpenDAL, the OpenDAL PMC has recently voted to add you as a PMC member. The role of a PMC member grants you access to the Project Management Committee (PMC) and enables you to take on greater responsibilities within the OpenDAL project. We hope that you accept this invitation and continue to help us make Apache OpenDAL better. Please reply to private@opendal.apache.org using the 'reply all' function for your responses. With the expectation of your acceptance, welcome! ${your_name} (as represents of The Apache OpenDAL PMC)   ","version":"Next","tagName":"h2"},{"title":"Add the candidate to the PMC member list​","type":1,"pageTitle":"Nominate PMC Member","url":"/community/pmc_members/nominate-pmc-member/#add-the-candidate-to-the-pmc-member-list","content":" After the candidate accepts the invitation, add the candidate to the PMC member list by whimsy roster tools   ","version":"Next","tagName":"h2"},{"title":"Generate release note","type":0,"sectionRef":"#","url":"/community/release/reference/generate_release_note/","content":"Generate release note This document describes how to generate release notes using GitHub: Go to https://github.com/apache/opendal/releases/new to start a new release.Fill the tag with draft.Click on Generate release notes to generate them.Copy the generated content and close the page. Please note that we only use this feature to generate release notes, so please do not click on Publish Release or Save draft. We will publish it after creating the tag.","keywords":"","version":"Next"},{"title":"Setup GPG key","type":0,"sectionRef":"#","url":"/community/release/reference/setup_gpg/","content":"","keywords":"","version":"Next"},{"title":"Install GPG​","type":1,"pageTitle":"Setup GPG key","url":"/community/release/reference/setup_gpg/#install-gpg","content":" For more details, please refer to GPG official website. Here shows one approach to install GPG with apt:  sudo apt install gnupg2   ","version":"Next","tagName":"h2"},{"title":"Generate GPG Key​","type":1,"pageTitle":"Setup GPG key","url":"/community/release/reference/setup_gpg/#generate-gpg-key","content":" Attentions:  Name is best to keep consistent with your full name of Apache ID;Email should be the Apache email;Name is best to only use English to avoid garbled.  Run gpg --full-gen-key and complete the generation interactively:  gpg (GnuPG) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) (14) Existing key from card Your selection? 1 # input 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 4096 # input 4096 Requested keysize is 4096 bits Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) 0 # input 0 Key does not expire at all Is this correct? (y/N) y # input y GnuPG needs to construct a user ID to identify your key. Real name: Hulk Lin # input your name Email address: hulk@apache.org # input your email Comment: # input some annotations, optional You selected this USER-ID: "Hulk <hulk@apache.org>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O # input O We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. # Input the security key ┌──────────────────────────────────────────────────────┐ │ Please enter this passphrase │ │ │ │ Passphrase: _______________________________ │ │ │ │ <OK> <Cancel> │ └──────────────────────────────────────────────────────┘ # key generation will be done after your inputting the key with the following output gpg: key E49B00F626B marked as ultimately trusted gpg: revocation certificate stored as '/Users/hulk/.gnupg/openpgp-revocs.d/F77B887A4F25A9468C513E9AA3008E49B00F626B.rev' public and secret key created and signed. pub rsa4096 2022-07-12 [SC] F77B887A4F25A9468C513E9AA3008E49B00F626B uid [ultimate] hulk <hulk@apache.org> sub rsa4096 2022-07-12 [E]   ","version":"Next","tagName":"h2"},{"title":"Upload your key to public GPG keyserver​","type":1,"pageTitle":"Setup GPG key","url":"/community/release/reference/setup_gpg/#upload-your-key-to-public-gpg-keyserver","content":" Firstly, list your key:  gpg --list-keys   The output is like:  ------------------------------- pub rsa4096 2022-07-12 [SC] F77B887A4F25A9468C513E9AA3008E49B00F626B uid [ultimate] hulk <hulk@apache.org> sub rsa4096 2022-07-12 [E]   Then, send your key id to key server:  gpg --keyserver keys.openpgp.org --send-key <key-id> # e.g., F77B887A4F25A9468C513E9AA3008E49B00F626B   Among them, keys.openpgp.org is a randomly selected keyserver, you can use keyserver.ubuntu.com or any other full-featured keyserver.  ","version":"Next","tagName":"h2"},{"title":"Check whether the key is created successfully​","type":1,"pageTitle":"Setup GPG key","url":"/community/release/reference/setup_gpg/#check-whether-the-key-is-created-successfully","content":" Uploading takes about one minute; after that, you can check by your email at the corresponding keyserver.  Uploading keys to the keyserver is mainly for joining a Web of Trust.  ","version":"Next","tagName":"h2"},{"title":"Add your GPG public key to the KEYS document​","type":1,"pageTitle":"Setup GPG key","url":"/community/release/reference/setup_gpg/#add-your-gpg-public-key-to-the-keys-document","content":" info SVN is required for this step.  The svn repository of the release branch is: https://dist.apache.org/repos/dist/release/opendal  Please always add the public key to KEYS in the release branch:  svn co https://dist.apache.org/repos/dist/release/opendal opendal-dist # As this step will copy all the versions, it will take some time. If the network is broken, please use svn cleanup to delete the lock before re-execute it. cd opendal-dist (gpg --list-sigs YOUR_NAME@apache.org && gpg --export --armor YOUR_NAME@apache.org) >> KEYS # Append your key to the KEYS file svn add . # It is not needed if the KEYS document exists before. svn ci -m "add gpg key for YOUR_NAME" # Later on, if you are asked to enter a username and password, just use your apache username and password.   ","version":"Next","tagName":"h2"},{"title":"Upload the GPG public key to your GitHub account​","type":1,"pageTitle":"Setup GPG key","url":"/community/release/reference/setup_gpg/#upload-the-gpg-public-key-to-your-github-account","content":" Enter https://github.com/settings/keys to add your GPG key.Please remember to bind the email address used in the GPG key to your GitHub account (https://github.com/settings/emails) if you find "unverified" after adding it. ","version":"Next","tagName":"h2"},{"title":"Security","type":0,"sectionRef":"#","url":"/community/security/","content":"Security The Apache Software Foundation takes a rigorous stance on eliminating security issues in its software projects. Likewise, Apache OpenDAL™ is also vigilant and takes security issues related to its features and functionality into the highest consideration. If you have any concerns regarding OpenDAL's security, or you discover a vulnerability or potential threat, please do not hesitate to get in touch with the Apache Security Team by dropping an email at private@opendal.apache.org. Please specify the project name as "OpenDAL" in the email, and provide a description of the relevant problem or potential threat. You are also urged to recommend how to reproduce and replicate the issue. The Apache Security Team and the OpenDAL community will get back to you after assessing and analyzing the findings. Please note that the security issue should be reported on the security email first, before disclosing it on any public domain.","keywords":"","version":"Next"},{"title":"Verify a release candidate","type":0,"sectionRef":"#","url":"/community/release/verify/","content":"","keywords":"","version":"Next"},{"title":"Download links are valid​","type":1,"pageTitle":"Verify a release candidate","url":"/community/release/verify/#download-links-are-valid","content":" To verify the release candidate, you need to download the release candidate from the dist directory.  Our current distribution contains many files, which we recommend downloading using svn.  Use the following command to download all artifacts, replace "${release_version}-${rc_version}" with the version ID of the version to be released:  svn co https://dist.apache.org/repos/dist/dev/opendal/${release_version}-${rc_version}/   ","version":"Next","tagName":"h2"},{"title":"Checksums and signatures​","type":1,"pageTitle":"Verify a release candidate","url":"/community/release/verify/#checksums-and-signatures","content":" Every file in a release candidate should have a checksum and signature file.  For example, if the release candidate is 0.46.0-rc1, the checksum and signature file should be like:  https://dist.apache.org/repos/dist/dev/opendal/0.46.0-rc1/apache-opendal-core-0.46.0-rc1-src.tar.gz.sha512 https://dist.apache.org/repos/dist/dev/opendal/0.46.0-rc1/apache-opendal-core-0.46.0-rc1-src.tar.gz.asc   ","version":"Next","tagName":"h2"},{"title":"Verify checksums and signatures​","type":1,"pageTitle":"Verify a release candidate","url":"/community/release/verify/#verify-checksums-and-signatures","content":" GnuPG is recommended here. It can be installed with the following command:  apt-get install gnupg # or yum install gnupg # or brew install gnupg   Firstly, import the OpenDAL release manager's public key:  curl https://downloads.apache.org/opendal/KEYS > KEYS # Download KEYS gpg --import KEYS # Import KEYS to local   Then, trust the public key:  gpg --edit-key <KEY-used-in-this-version> # Edit the key   It will enter the interactive mode, use the following command to trust the key:  gpg> trust   And then, select the level of trust, for example:  Please decide how far you trust this user to correctly verify other users' keys (by looking at passports, checking fingerprints from different sources, etc.) 1 = I don't know or won't say 2 = I do NOT trust 3 = I trust marginally 4 = I trust fully 5 = I trust ultimately m = back to the main menu   Select 5 to trust the key ultimately.  Now, we could start the verification.  We've provided a script to verify the checksum and signature of the release candidate.  The script is in the scripts directory of our repository. You can download it directly from here. Please put it in the same directory as the release candidate.  Run the script in a specific release candidate's folder:  python ./verify.py   You will see the following output if the verification is successful:  $ python ./verify.py > Checking apache-opendal-bin-oli-0.41.3-src.tar.gz gpg: Signature made 五 6/ 7 20:57:06 2024 CST gpg: using RSA key 8B374472FAD328E17F479863B379691FC6E298DD gpg: Good signature from "Zili Chen (CODE SIGNING KEY) <tison@apache.org>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 8B37 4472 FAD3 28E1 7F47 9863 B379 691F C6E2 98DD > Success to verify the gpg sign for apache-opendal-bin-oli-0.41.3-src.tar.gz apache-opendal-bin-oli-0.41.3-src.tar.gz: OK > Success to verify the checksum for apache-opendal-bin-oli-0.41.3-src.tar.gz ....... > Start checking LICENSE file in /Users/yan/Downloads/opendal-dev/apache-opendal-0.47.0-src > LICENSE file exists in /Users/yan/Downloads/opendal-dev/apache-opendal-0.47.0-src > Start checking NOTICE file in /Users/yan/Downloads/opendal-dev/apache-opendal-0.47.0-src > NOTICE file exists in /Users/yan/Downloads/opendal-dev/apache-opendal-0.47.0-src cargo 1.78.0 (54d8815d0 2024-03-26) Start building opendal core Success to build opendal core openjdk version "22.0.1" 2024-04-16 OpenJDK Runtime Environment Temurin-22.0.1+8 (build 22.0.1+8) OpenJDK 64-Bit Server VM Temurin-22.0.1+8 (build 22.0.1+8, mixed mode) Start building opendal java binding > Success to build opendal java binding   ","version":"Next","tagName":"h3"},{"title":"Check the file content of the source package​","type":1,"pageTitle":"Verify a release candidate","url":"/community/release/verify/#check-the-file-content-of-the-source-package","content":" Unzip apache-opendal-${release_version}-${rc_version}-src.tar.gz and check the follows:  LICENSE and NOTICE files are correct for the repository.All files have ASF license headers if necessary.Building is OK.  ","version":"Next","tagName":"h2"},{"title":"Check the Maven artifacts of opendal-java​","type":1,"pageTitle":"Verify a release candidate","url":"/community/release/verify/#check-the-maven-artifacts-of-opendal-java","content":" Download the artifacts from https://repository.apache.org/content/repositories/orgapacheopendal-${maven_artifact_number}/.  You can check the follows:  Checksum of JARs matches the bundled checksum file.Signature of JARs matches the bundled signature file.JARs are reproducible locally. This means you can build the JARs on your machine and verify the checksum is the same with the bundled one.  The reproducibility requires the same JDK distribution and the same Maven distribution. You should use Eclipse Temurin JDK 8 and the bundled Maven Wrapper to make the same artifacts. ","version":"Next","tagName":"h2"},{"title":"Create a release","type":0,"sectionRef":"#","url":"/community/release/","content":"","keywords":"","version":"Next"},{"title":"Introduction​","type":1,"pageTitle":"Create a release","url":"/community/release/#introduction","content":" Source Release is the key point which Apache values, and is also necessary for an ASF release.  Please remember that publishing software has legal consequences.  This guide complements the foundation-wide policies and guides:  Release PolicyRelease Distribution PolicyRelease Creation Process  ","version":"Next","tagName":"h2"},{"title":"Some Terminology of release​","type":1,"pageTitle":"Create a release","url":"/community/release/#some-terminology-of-release","content":" In the context of our release, we use several terms to describe different stages of the release process.  Here's an explanation of these terms:  opendal_version: the version of OpenDAL to be released, like 0.46.0.release_version: the version of release candidate, like 0.46.0-rc.1.rc_version: the minor version for voting round, like rc.1.maven_artifact_number: the number for Maven staging artifacts, like 1010. The number can be found by searching "opendal" on https://repository.apache.org/#stagingRepositories. And the Maven staging artifacts will be created automatically when we push a git tag to GitHub for now.  ","version":"Next","tagName":"h2"},{"title":"Preparation​","type":1,"pageTitle":"Create a release","url":"/community/release/#preparation","content":" caution This section is the requirements for individuals who are new to the role of release manager.  Refer to Setup GPG Key to make sure the GPG key has been set up.  ","version":"Next","tagName":"h2"},{"title":"Start discussion about the next release​","type":1,"pageTitle":"Create a release","url":"/community/release/#start-discussion-about-the-next-release","content":" Start a discussion at OpenDAL Discussion General:  Title:  [DISCUSS] Release Apache OpenDAL ${release_version}   Content:  Hello, Apache OpenDAL Community, This is a call for a discussion to release Apache OpenDAL version ${opendal_version}. The change lists about this release: https://github.com/apache/opendal/compare/v${opendal_last_version}...main Please leave your comments here about this release plan. We will bump the version in the repo and start the release process after the discussion. Thanks ${name}   ","version":"Next","tagName":"h2"},{"title":"Start a tracking issue about the next release​","type":1,"pageTitle":"Create a release","url":"/community/release/#start-a-tracking-issue-about-the-next-release","content":" Start a tracking issue on GitHub for the upcoming release to track all tasks that need to be completed.  ","version":"Next","tagName":"h2"},{"title":"Release List​","type":1,"pageTitle":"Create a release","url":"/community/release/#release-list","content":" Update the version list in the dev/src/release/package.rs file.  For example:  If there is any breaking change, please bump the minor version instead of the patch version.If this package is not ready for release, please skip.  ","version":"Next","tagName":"h2"},{"title":"GitHub Side​","type":1,"pageTitle":"Create a release","url":"/community/release/#github-side","content":" ","version":"Next","tagName":"h2"},{"title":"Bump version in project​","type":1,"pageTitle":"Create a release","url":"/community/release/#bump-version-in-project","content":" Run just update-version to bump the version in the project.  ","version":"Next","tagName":"h3"},{"title":"Update docs​","type":1,"pageTitle":"Create a release","url":"/community/release/#update-docs","content":" Update CHANGELOG.md, refer to Generate Release Note for more information.Update core/src/docs/upgrade.md if there are breaking changes in coreMake sure every released bindings' upgrade.md has been updated. java: bindings/java/upgrade.mdnode.js: bindings/nodejs/upgrade.mdpython: bindings/python/upgrade.md  ","version":"Next","tagName":"h3"},{"title":"Generate dependencies list​","type":1,"pageTitle":"Create a release","url":"/community/release/#generate-dependencies-list","content":" Download and setup cargo-deny. You can refer to cargo-deny.  Running python3 ./scripts/dependencies.py generate to update the dependency list of every package.  ","version":"Next","tagName":"h3"},{"title":"Push release candidate tag​","type":1,"pageTitle":"Create a release","url":"/community/release/#push-release-candidate-tag","content":" After bump version PR gets merged, we can create a GitHub release for the release candidate:  Create a tag at main branch on the Bump Version / Patch up version commit: git tag -s "v0.46.0-rc.1", please correctly check out the corresponding commit instead of directly tagging on the main branch.Push tags to GitHub: git push --tags.  note Pushing a Git tag to GitHub repo will trigger a GitHub Actions workflow that creates a staging Maven release on https://repository.apache.org which can be verified on voting.  ","version":"Next","tagName":"h3"},{"title":"Check the GitHub action status​","type":1,"pageTitle":"Create a release","url":"/community/release/#check-the-github-action-status","content":" After pushing the tag, we need to check the GitHub action status to make sure the release candidate is created successfully.  Python: Bindings Python CIJava: Bindings Java CI and Bindings Java ReleaseNode.js: Bindings Node.js CI  In the most cases, it would be great to rerun the failed workflow directly when you find some failures. But if a new code patch is needed to fix the failure, you should create a new release candidate tag, increase the rc number and push it to GitHub.  ","version":"Next","tagName":"h3"},{"title":"ASF Side​","type":1,"pageTitle":"Create a release","url":"/community/release/#asf-side","content":" If any step in the ASF Release process fails and requires code changes, we will abandon that version and prepare for the next one. Our release page will only display ASF releases instead of GitHub Releases.  Additionally, we should also drop the staging Maven artifacts on https://repository.apache.org.  ","version":"Next","tagName":"h2"},{"title":"Create an ASF Release​","type":1,"pageTitle":"Create a release","url":"/community/release/#create-an-asf-release","content":" After GitHub Release has been created, we can start to create ASF Release.  Checkout to released tag. (e.g. git checkout v0.46.0-rc.1, tag is created in the previous step)Use the release script to create a new release: just release This script will generate the release candidate artifacts under dist, including: apache-opendal-{package}-{version}-src.tar.gzapache-opendal-{package}-{version}-src.tar.gz.ascapache-opendal-{package}-{version}-src.tar.gz.sha512 Push the newly created branch to GitHub  This script will create a new release under dist.  For example:  dist ├── apache-opendal-bindings-c-0.44.2-src.tar.gz ├── apache-opendal-bindings-c-0.44.2-src.tar.gz.asc ├── apache-opendal-bindings-c-0.44.2-src.tar.gz.sha512 ... ├── apache-opendal-core-0.45.0-src.tar.gz ├── apache-opendal-core-0.45.0-src.tar.gz.asc ├── apache-opendal-core-0.45.0-src.tar.gz.sha512 ├── apache-opendal-integrations-dav-server-0.0.0-src.tar.gz ├── apache-opendal-integrations-dav-server-0.0.0-src.tar.gz.asc ├── apache-opendal-integrations-dav-server-0.0.0-src.tar.gz.sha512 ├── apache-opendal-integrations-object_store-0.42.0-src.tar.gz ├── apache-opendal-integrations-object_store-0.42.0-src.tar.gz.asc └── apache-opendal-integrations-object_store-0.42.0-src.tar.gz.sha512 1 directory, 60 files   ","version":"Next","tagName":"h3"},{"title":"Upload artifacts to the SVN dist repo​","type":1,"pageTitle":"Create a release","url":"/community/release/#upload-artifacts-to-the-svn-dist-repo","content":" info SVN is required for this step.  The svn repository of the dev branch is: https://dist.apache.org/repos/dist/dev/opendal  First, checkout OpenDAL to local directory:  # As this step will copy all the versions, it will take some time. If the network is broken, please use svn cleanup to delete the lock before re-execute it. svn co https://dist.apache.org/repos/dist/dev/opendal opendal-dist-dev   Then, upload the artifacts:  The ${release_version} here should be like 0.46.0-rc.1  cd opendal-dist-dev # create a directory named by version mkdir ${release_version} # copy source code and signature package to the versioned directory cp ${repo_dir}/dist/* ${release_version}/ # check svn status svn status # add to svn svn add ${release_version} # check svn status svn status # commit to SVN remote server svn commit -m "Prepare for ${release_version}"   Visit https://dist.apache.org/repos/dist/dev/opendal/ to make sure the artifacts are uploaded correctly.  ","version":"Next","tagName":"h3"},{"title":"Close the Nexus staging repo​","type":1,"pageTitle":"Create a release","url":"/community/release/#close-the-nexus-staging-repo","content":" To verify the Maven staging artifacts in the next step, close the Nexus staging repo as below.  Open https://repository.apache.org/#stagingRepositories with your Apache ID login.Find the artifact orgapacheopendal-${maven_artifact_number}, click the "Close" button.  The close operation means that the artifacts are ready for voting.  caution If the vote failed, click "Drop" to drop the staging Maven artifacts.  ","version":"Next","tagName":"h3"},{"title":"Rescue​","type":1,"pageTitle":"Create a release","url":"/community/release/#rescue","content":" If you accidentally published wrong or unexpected artifacts, like wrong signature files, wrong sha256 files, please cancel the release for the current release_version,increase th RC counting and re-initiate a release with the new release_version. And remember to delete the wrong artifacts from the SVN dist repo. Additionally, you should also drop the staging Maven artifacts on https://repository.apache.org.  ","version":"Next","tagName":"h3"},{"title":"Voting​","type":1,"pageTitle":"Create a release","url":"/community/release/#voting","content":" OpenDAL requires votes from both the OpenDAL Community.  Start a VOTE at OpenDAL Discussion General:  Title:  [VOTE] Release Apache OpenDAL ${release_version} - Vote Round 1   Content:  Hello, Apache OpenDAL Community, This is a call for a vote to release Apache OpenDAL version ${opendal_version}. The tag to be voted on is ${opendal_version}. The release candidate: https://dist.apache.org/repos/dist/dev/opendal/${release_version}/ Keys to verify the release candidate: https://downloads.apache.org/opendal/KEYS Git tag for the release: https://github.com/apache/opendal/releases/tag/v${release_version} Maven staging repo: https://repository.apache.org/content/repositories/orgapacheopendal-${maven_artifact_number}/ Pypi testing repo: https://test.pypi.org/project/opendal/ Website staged: https://opendal-v${release_version | replace('.', '-')}.staged.apache.org/ Please download, verify, and test. The VOTE will be open for at least 72 hours and until the necessary number of votes are reached. - [ ] +1 approve - [ ] +0 no opinion - [ ] -1 disapprove with the reason To learn more about apache opendal, please see https://opendal.apache.org/ Checklist for reference: - [ ] Download links are valid. - [ ] Checksums and signatures. - [ ] LICENSE/NOTICE files exist - [ ] No unexpected binary files - [ ] All source files have ASF headers - [ ] Can compile from source Use our verify.py to assist in the verify process: svn co https://dist.apache.org/repos/dist/dev/opendal/${release_version}/ opendal-dev cd opendal-dev curl -sSL https://github.com/apache/opendal/raw/v${release_version}/scripts/verify.py -o verify.py python verify.py Thanks ${name}   Example: https://github.com/apache/opendal/discussions/5211  The vote should be open for at least 72 hours except the following cases:  Security issuesThe wild user affected bug fixesAny other emergency cases  The Release manager should claim the emergency cases in the vote email if he wants to vote it rapidly.  Tips: The 72 hours is the minimum time for voting, so we can ensure that community members from various time zones can participate in the verification process.  After at least 3 +1 binding vote (from OpenDAL PMC member) and more +1 bindings than -1 bindings, claim the vote result:  Title:  [RESULT][VOTE] Release Apache OpenDAL ${release_version} - Vote Round 1   Content:  Hello, Apache OpenDAL Community, The vote to release Apache OpenDAL ${release_version} has passed. The vote PASSED with 3 +1 binding and 1 +1 non-binding votes, no +0 or -1 votes: Binding votes: - xxx - yyy - zzz Non-Binding votes: - aaa Vote thread: ${vote_thread_url} Thanks ${name}   It's better to use the real name or the public name which is displayed on the voters' profile page, or Apache ID of the voter, to show who voted in the vote result email, and avoid using nicknames, it will make the vote result hard for others to track the voter. We should make sure the binding votes are from the people who have the right to vote the binding one.  Example: https://lists.apache.org/thread/xk5myl10mztcfotn59oo59s4ckvojds6  ","version":"Next","tagName":"h2"},{"title":"Official Release​","type":1,"pageTitle":"Create a release","url":"/community/release/#official-release","content":" ","version":"Next","tagName":"h2"},{"title":"Push the release git tag​","type":1,"pageTitle":"Create a release","url":"/community/release/#push-the-release-git-tag","content":" # Checkout the tags that passed VOTE git checkout ${release_version} # Tag with the opendal version git tag -s ${opendal_version} # Push tags to GitHub to trigger releases git push origin ${opendal_version}   ","version":"Next","tagName":"h3"},{"title":"Publish artifacts to SVN RELEASE branch​","type":1,"pageTitle":"Create a release","url":"/community/release/#publish-artifacts-to-svn-release-branch","content":" svn mv https://dist.apache.org/repos/dist/dev/opendal/${release_version} https://dist.apache.org/repos/dist/release/opendal/${opendal_version} -m "Release ${opendal_version}"   ","version":"Next","tagName":"h3"},{"title":"Release Maven artifacts​","type":1,"pageTitle":"Create a release","url":"/community/release/#release-maven-artifacts","content":" Open https://repository.apache.org/#stagingRepositories.Find the artifact orgapacheopendal-${maven_artifact_number}, click the "Release" button.  It will take some time to sync the Maven artifacts to the Maven Central.  caution If the vote failed, click "Drop" to drop the staging Maven artifacts.  ","version":"Next","tagName":"h3"},{"title":"Check the language binding artifacts​","type":1,"pageTitle":"Create a release","url":"/community/release/#check-the-language-binding-artifacts","content":" We need to check the language binding artifacts in the language package repo to make sure they are released successfully.  Python: https://pypi.org/project/opendal/Java: https://repository.apache.org/#nexus-search;quick~opendalNode.js: https://www.npmjs.com/package/opendal  For Java binding, if we cannot find the latest version of artifacts in the repo, we need to check the orgapacheopendal-${maven_artifact_number} artifact status in staging repo.  For non-Java bindings, if we cannot find the latest version of artifacts in the repo, we need to check the GitHub action status.  ","version":"Next","tagName":"h3"},{"title":"Create a GitHub Release​","type":1,"pageTitle":"Create a release","url":"/community/release/#create-a-github-release","content":" Click here to create a new release.Pick the git tag of this release version from the dropdown menu.Make sure the branch target is main.Generate the release note by clicking the Generate release notes button.Add the release note from every component's upgrade.md if there are breaking changes before the content generated by GitHub. Check them carefully.Publish the release.  ","version":"Next","tagName":"h3"},{"title":"Send the announcement​","type":1,"pageTitle":"Create a release","url":"/community/release/#send-the-announcement","content":" Start an announcement to OpenDAL Discussion Announcements and send the same content to announce@apache.org.  Tips: Please follow the Committer Email guide to make sure you have already set up the email SMTP. Otherwise, your email cannot be sent to the announcement mailing list.  Instead of adding breaking changes, let's include the new features as "notable changes" in the announcement.  Title:  [ANNOUNCE] Release Apache OpenDAL ${opendal_version}   Content:  Hi all, The Apache OpenDAL community is pleased to announce that Apache OpenDAL ${opendal_version} has been released! OpenDAL is a data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way. The notable changes since ${opendal_version} include: 1. xxxxx 2. yyyyyy 3. zzzzzz Please refer to the change log for the complete list of changes: https://github.com/apache/opendal/releases/tag/v${opendal_version} Apache OpenDAL website: https://opendal.apache.org/ Download Links: https://opendal.apache.org/download OpenDAL Resources: - Issue: https://github.com/apache/opendal/issues - Mailing list: dev@opendal.apache.org Thanks On behalf of Apache OpenDAL community   Example: https://lists.apache.org/thread/oy77n55brvk72tnlb2bjzfs9nz3cfd0s  ","version":"Next","tagName":"h3"},{"title":"Post release​","type":1,"pageTitle":"Create a release","url":"/community/release/#post-release","content":" After the official release out, you may perform a few post-actions.  ","version":"Next","tagName":"h2"},{"title":"Remove the old releases​","type":1,"pageTitle":"Create a release","url":"/community/release/#remove-the-old-releases","content":" Remove the old releases if any. You only need the latest release there, and older releases are available through the Apache archive.  To clean up old releases, run:  # 1. Get the list of releases svn ls https://dist.apache.org/repos/dist/release/opendal # 2. Delete each release (except for the last one) svn del -m "Archiving OpenDAL release X.Y.Z" https://dist.apache.org/repos/dist/release/opendal/X.Y.Z  ","version":"Next","tagName":"h3"},{"title":"Ofs","type":0,"sectionRef":"#","url":"/apps/ofs/","content":"","keywords":"","version":"Next"},{"title":"Status​","type":1,"pageTitle":"Ofs","url":"/apps/ofs/#status","content":" ofs is a work in progress. we only support fs and s3 as backend on Linux currently.  ","version":"Next","tagName":"h2"},{"title":"How to use ofs​","type":1,"pageTitle":"Ofs","url":"/apps/ofs/#how-to-use-ofs","content":" ","version":"Next","tagName":"h2"},{"title":"Install FUSE on Linux​","type":1,"pageTitle":"Ofs","url":"/apps/ofs/#install-fuse-on-linux","content":" sudo pacman -S fuse3 --noconfirm # archlinux sudo apt-get -y install fuse # debian/ubuntu   ","version":"Next","tagName":"h3"},{"title":"Load FUSE kernel module on FreeBSD​","type":1,"pageTitle":"Ofs","url":"/apps/ofs/#load-fuse-kernel-module-on-freebsd","content":" kldload fuse   ","version":"Next","tagName":"h3"},{"title":"Install ofs​","type":1,"pageTitle":"Ofs","url":"/apps/ofs/#install-ofs","content":" ofs could be installed by cargo:  cargo install ofs   cargo is the Rust package manager. cargo could be installed by following the Installation from Rust official website.  ","version":"Next","tagName":"h3"},{"title":"Mount directory​","type":1,"pageTitle":"Ofs","url":"/apps/ofs/#mount-directory","content":" ofs <mount-point> 'fs://?root=<directory>'   ","version":"Next","tagName":"h3"},{"title":"Mount S3 bucket​","type":1,"pageTitle":"Ofs","url":"/apps/ofs/#mount-s3-bucket","content":" ofs <mount-point> 's3://?root=<path>&bucket=<bucket>&endpoint=<endpoint>&region=<region>&access_key_id=<access-key-id>&secret_access_key=<secret-access-key>'   ","version":"Next","tagName":"h3"},{"title":"Branding​","type":1,"pageTitle":"Ofs","url":"/apps/ofs/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Ofs","url":"/apps/ofs/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Oay","type":0,"sectionRef":"#","url":"/apps/oay/","content":"","keywords":"","version":"Next"},{"title":"Goal​","type":1,"pageTitle":"Oay","url":"/apps/oay/#goal","content":" Allow users to access different storage backend through their preferred APIs.    ","version":"Next","tagName":"h2"},{"title":"Status​","type":1,"pageTitle":"Oay","url":"/apps/oay/#status","content":" Our first milestone is to provide S3 APIs.  ","version":"Next","tagName":"h2"},{"title":"S3 API​","type":1,"pageTitle":"Oay","url":"/apps/oay/#s3-api","content":" Only list_object_v2 with start_after is supported.  ","version":"Next","tagName":"h3"},{"title":"Branding​","type":1,"pageTitle":"Oay","url":"/apps/oay/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Oay","url":"/apps/oay/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Oli","type":0,"sectionRef":"#","url":"/apps/oli/","content":"","keywords":"","version":"Next"},{"title":"How to use oli​","type":1,"pageTitle":"Oli","url":"/apps/oli/#how-to-use-oli","content":" oli provide basic sub-commands like oli ls, oli cat, oli stat, oli cp, oli rm, and oli bench just like what you use on your local filesystem.  ","version":"Next","tagName":"h2"},{"title":"Install oli​","type":1,"pageTitle":"Oli","url":"/apps/oli/#install-oli","content":" oli could be installed by cargo:  cargo install oli   cargo is the Rust package manager. cargo could be installed by following the Installation from Rust official website.  ","version":"Next","tagName":"h3"},{"title":"Configure oli Profile​","type":1,"pageTitle":"Oli","url":"/apps/oli/#configure-oli-profile","content":" oli requires a config file to work, it should be:  ~/.config/oli/config.toml on Linux~/Library/Application Support/oli/config.toml on macOSC:\\Users\\<UserName>\\AppData\\Roaming\\oli\\config.toml on Windows  The content of config.toml should follow these pattern:  [profiles.<profile_name>] configuration_key1 = "value1" configuration_key2 = "value2" [profiles.<another_profile_name>] configuration_key3 = "value3" configuration_key4 = "value4"   Here is an example of config.toml:  [profiles.s3] type = "s3" root = "/assets" bucket = "<bucket>" region = "<region>" endpoint = "https://s3.amazonaws.com" access_key_id = "<access_key_id>" secret_access_key = "<secret_access_key>" [profiles.r2] type = "s3" root = "/assets" bucket = "<bucket>" region = "auto" endpoint = "https://<account_id>.r2.cloudflarestorage.com" access_key_id = "<access_key_id>" secret_access_key = "<secret_access_key>"   For different services, you could find the configuration keys in the corresponding service document.  ","version":"Next","tagName":"h3"},{"title":"Example: use oli to upload file to AWS S3​","type":1,"pageTitle":"Oli","url":"/apps/oli/#example-use-oli-to-upload-file-to-aws-s3","content":" oli cp ./update-ecs-loadbalancer.json s3:/update-ecs-loadbalancer.json oli ls s3:/ # fleet.png # update-ecs-loadbalancer.json   ","version":"Next","tagName":"h3"},{"title":"Example: use oli copy file from S3 to R2​","type":1,"pageTitle":"Oli","url":"/apps/oli/#example-use-oli-copy-file-from-s3-to-r2","content":" oli cp s3:/fleet.png r2:/fleet.png oli ls r2:/ # fleet.png   ","version":"Next","tagName":"h3"},{"title":"Example: use oli bench command​","type":1,"pageTitle":"Oli","url":"/apps/oli/#example-use-oli-bench-command","content":" use the oli config file or create a new config file for bench command  Here is an example of config.toml using bench command:  [profiles.demo] type = "s3" access_key_id = "minioadmin" secret_access_key = "minioadmin" bucket = "test-bucket" endpoint = "http://127.0.0.1:9000" region = "us-east-1" root = "/benchmarks" enable_virtual_host_style = "false"   create a bench suite config file  Here is an example of suite.toml:  file_size = "64MiB" parallelism = 4 timeout = "60s" workload = "upload"   then can use bench command use config file and suite file  oli bench --config config.toml demo suite.toml   ","version":"Next","tagName":"h3"},{"title":"Contribute to oli​","type":1,"pageTitle":"Oli","url":"/apps/oli/#contribute-to-oli","content":" Contribution is not only about code, but also about documentation, examples, and so on! 🚀  If you have any questions or suggestions about oli, please feel free to open an issue on GitHub.  As oli is a part of Apache OpenDAL, you should follow the CONTRIBUTION documentation. There are still lots works to do with oli, you could track them on this GitHub Issue.  ","version":"Next","tagName":"h2"},{"title":"Branding​","type":1,"pageTitle":"Oli","url":"/apps/oli/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Oli","url":"/apps/oli/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"D 🚧","type":0,"sectionRef":"#","url":"/bindings/d/","content":"","keywords":"","version":"Next"},{"title":"Build​","type":1,"pageTitle":"D 🚧","url":"/bindings/d/#build","content":" To compile OpenDAL d binding from source code, you need:  dmd/ldc/gdc  # build libopendal_c (underneath call make -C ../c) dub build -b release # build and run unit tests dub test   ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"D 🚧","url":"/bindings/d/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"C 🚧","type":0,"sectionRef":"#","url":"/bindings/c/","content":"","keywords":"","version":"Next"},{"title":"Example​","type":1,"pageTitle":"C 🚧","url":"/bindings/c/#example","content":" A simple read and write example  #include "assert.h" #include "opendal.h" #include "stdio.h" int main() { /* Initialize a operator for "memory" backend, with no options */ opendal_result_operator_new result = opendal_operator_new("memory", 0); assert(result.operator_ptr != NULL); assert(result.error == NULL); /* Prepare some data to be written */ opendal_bytes data = { .data = (uint8_t*)"this_string_length_is_24", .len = 24, }; /* Write this into path "/testpath" */ opendal_error *error = opendal_operator_write(op, "/testpath", &data); assert(error == NULL); /* We can read it out, make sure the data is the same */ opendal_result_read r = opendal_operator_read(op, "/testpath"); opendal_bytes read_bytes = r.data; assert(r.error == NULL); assert(read_bytes.len == 24); /* Lets print it out */ for (int i = 0; i < 24; ++i) { printf("%c", read_bytes.data[i]); } printf("\\n"); /* the opendal_bytes read is heap allocated, please free it */ opendal_bytes_free(&read_bytes); /* the operator_ptr is also heap allocated */ opendal_operator_free(&op); }   For more examples, please refer to ./examples  ","version":"Next","tagName":"h2"},{"title":"Prerequisites​","type":1,"pageTitle":"C 🚧","url":"/bindings/c/#prerequisites","content":" To build OpenDAL C binding, the following is all you need:  A compiler that supports C11 and C++14, e.g. clang and gcc To format the code, you need to install clang-format The opendal.h is not formatted by hands when you contribute, please do not format the file. Use make format only.If your contribution is related to the files under ./tests, you may format it before submitting your pull request. But notice that different versions of clang-format may format the files differently. (optional) Doxygen need to be installed to generate documentations.  For Ubuntu and Debian:  # install C/C++ toolchain sudo apt install -y build-essential # install clang-format sudo apt install clang-format # install and build GTest library under /usr/lib and softlink to /usr/local/lib sudo apt-get install libgtest-dev # install CMake sudo apt-get install cmake # install Rust curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh   ","version":"Next","tagName":"h2"},{"title":"Makefile​","type":1,"pageTitle":"C 🚧","url":"/bindings/c/#makefile","content":" To build the library and header file. mkdir -p build && cd build cmake .. make The header file opendal.h is under ./include The library is under ../../target/debug after building. use FEATURES to enable services, like cmake .. -DFEATURES="opendal/services-memory,opendal/services-fs" To clean the build results. cargo clean cd build && make clean To build and run the tests. (Note that you need to install Valgrind and GTest) cd build make tests && ./tests To build the examples cd build make basic error_handle   ","version":"Next","tagName":"h2"},{"title":"Documentation​","type":1,"pageTitle":"C 🚧","url":"/bindings/c/#documentation","content":" The documentation index page source is under ./docs/doxygen/html/index.html. If you want to build the documentations yourself, you could use  # this requires you to install doxygen make doc   ","version":"Next","tagName":"h2"},{"title":"Used by​","type":1,"pageTitle":"C 🚧","url":"/bindings/c/#used-by","content":" Check out the users list for more details on who is using OpenDAL.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"C 🚧","url":"/bindings/c/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Cpp 🚧","type":0,"sectionRef":"#","url":"/bindings/cpp/","content":"","keywords":"","version":"Next"},{"title":"Example​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#example","content":" #include "opendal.hpp" #include <vector> int main() { auto op = opendal::Operator("memory"); std::vector<uint8_t> data = {1, 2, 3, 4, 5}; op.write("test", data); auto result = op.read("test"); // result == data }   More examples can be found here.  ","version":"Next","tagName":"h2"},{"title":"Using​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#using","content":" ","version":"Next","tagName":"h2"},{"title":"CMake​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#cmake","content":" You can use FetchContent to add OpenDAL to your project.  FetchContent_Declare( opendal-cpp GIT_REPOSITORY https://github.com/apache/opendal.git GIT_TAG v0.40.0 SOURCE_SUBDIR bindings/cpp ) FetchContent_MakeAvailable(opendal-cpp)   Or you can download the source code and add it to your project.  mkdir third_party cd third_party git clone https://github.com/apache/opendal.git git checkout v0.40.0   add_subdirectory(third_party/opendal/bindings/cpp)   Now you can use OpenDAL in your project.  target_link_libraries(your_target opendal_cpp)   ","version":"Next","tagName":"h3"},{"title":"Others​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#others","content":" Support for more package managers is coming soon!  ","version":"Next","tagName":"h3"},{"title":"Compiling​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#compiling","content":" ","version":"Next","tagName":"h2"},{"title":"Prerequisites​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#prerequisites","content":" CMake >= 3.22C++ compiler with C++17 support  ","version":"Next","tagName":"h3"},{"title":"Build​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#build","content":" mkdir build cd build # Add -DOPENDAL_DEV=ON to make development environment for OpenDAL cmake .. make   ","version":"Next","tagName":"h3"},{"title":"Test​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#test","content":" You should build the project with OPENDAL_ENABLE_TESTING option. Then run:  make test   ","version":"Next","tagName":"h3"},{"title":"Docs​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#docs","content":" You should build the project with OPENDAL_ENABLE_DOCUMENTATION option. Then run:  make docs   ","version":"Next","tagName":"h3"},{"title":"CMake Options​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#cmake-options","content":" OPENDAL_DEV: Enable development environment for OpenDAL. It will enable most development options. With this option, you don't need to set other options. Default: OFFOPENDAL_ENABLE_ADDRESS_SANITIZER: Enable address sanitizer. Default: OFFOPENDAL_ENABLE_DOCUMENTATION: Enable documentation. Default: OFFOPENDAL_DOCS_ONLY: Only build documentation. Default: OFFOPENDAL_ENABLE_TESTING: Enable testing. Default: OFF  ","version":"Next","tagName":"h3"},{"title":"License and Trademarks​","type":1,"pageTitle":"Cpp 🚧","url":"/bindings/cpp/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Dotnet 🚧","type":0,"sectionRef":"#","url":"/bindings/dotnet/","content":"","keywords":"","version":"Next"},{"title":"Build​","type":1,"pageTitle":"Dotnet 🚧","url":"/bindings/dotnet/#build","content":" To compile OpenDAL .NET binding from source code, you need:  .NET version 7.0  cargo build dotnet build dotnet test   ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Dotnet 🚧","url":"/bindings/dotnet/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Dart 🚧","type":0,"sectionRef":"#","url":"/bindings/dart/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Dart 🚧","url":"/bindings/dart/#useful-links","content":" Examples  ","version":"Next","tagName":"h2"},{"title":"Usage​","type":1,"pageTitle":"Dart 🚧","url":"/bindings/dart/#usage","content":" Api is designed to be like stdlib style.  This is stdlib  import 'dart:io'; void main() async { final file = File('file.txt'); var is_exists = await file.exists(); print(is_exists); }   This is opendal  import 'package:opendal/opendal.dart'; void main() async { final storage = await Storage.init(schemeStr: "fs", map: {"root": "/tmp"}); final File = storage.initFile(); // drop-in final file = File('file.txt'); var is_exists = await file.exists(); print(is_exists); }   ","version":"Next","tagName":"h2"},{"title":"Test​","type":1,"pageTitle":"Dart 🚧","url":"/bindings/dart/#test","content":" dart run tests/opendal_test.dart   ","version":"Next","tagName":"h2"},{"title":"Development​","type":1,"pageTitle":"Dart 🚧","url":"/bindings/dart/#development","content":" flutter pub get flutter_rust_bridge_codegen generate cd rust cargo build -r --target x86_64-unknown-linux-gnu # change to your arch, refer to https://doc.rust-lang.org/beta/rustc/platform-support.html   ","version":"Next","tagName":"h2"},{"title":"Update generated code​","type":1,"pageTitle":"Dart 🚧","url":"/bindings/dart/#update-generated-code","content":" This binding uses https://github.com/fzyzcjy/flutter_rust_bridge, when updating the codegen. First check FLUTTER_RUST_BRIDGE_CODEGEN_VERSION, then pin the version of flutter_rust_bridge in pubspec.yaml and rust/Cargo.toml. Make sure the runtime versions are matched.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Dart 🚧","url":"/bindings/dart/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Haskell 🚧","type":0,"sectionRef":"#","url":"/bindings/haskell/","content":"","keywords":"","version":"Next"},{"title":"Example​","type":1,"pageTitle":"Haskell 🚧","url":"/bindings/haskell/#example","content":" Basic usage  import OpenDAL main :: IO () main = do Right op <- newOperator "memory" runOp op operation where operation = do writeOp op "key1" "value1" writeOp op "key2" "value2" value1 <- readOp op "key1" value2 <- readOp op "key2"   Use logger  import OpenDAL import Colog (simpleMessageAction) main :: IO () main = do Right op <- newOperator "memory" {ocLogAction = Just simpleMessageAction} return ()   ","version":"Next","tagName":"h2"},{"title":"Build​","type":1,"pageTitle":"Haskell 🚧","url":"/bindings/haskell/#build","content":" cabal build   ","version":"Next","tagName":"h2"},{"title":"Test​","type":1,"pageTitle":"Haskell 🚧","url":"/bindings/haskell/#test","content":" cabal test   ","version":"Next","tagName":"h2"},{"title":"Doc​","type":1,"pageTitle":"Haskell 🚧","url":"/bindings/haskell/#doc","content":" To generate the documentation:  cabal haddock   If your cabal version is greater than 3.8, you can use cabal haddock --open to open the documentation in your browser. Otherwise, you can visit the documentation from dist-newstyle/build/$ARCH/ghc-$VERSION/opendal-$VERSION/doc/html/opendal/index.html.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Haskell 🚧","url":"/bindings/haskell/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Lua 🚧","type":0,"sectionRef":"#","url":"/bindings/lua/","content":"","keywords":"","version":"Next"},{"title":"Example​","type":1,"pageTitle":"Lua 🚧","url":"/bindings/lua/#example","content":" local opendal = require("opendal") local op, err = opendal.operator.new("fs",{root="/tmp"}) if err ~= nil then print(err) return end op:write("test.txt","hello world") print("read: ", op:read("test.txt"))   ","version":"Next","tagName":"h2"},{"title":"Lua version​","type":1,"pageTitle":"Lua 🚧","url":"/bindings/lua/#lua-version","content":" You have to enable one of the features: lua54, lua53, lua52, lua51, luajit(52) or luau in Cargo.toml, according to the chosen Lua version. Default Lua version is 5.2.  ","version":"Next","tagName":"h2"},{"title":"Build from source​","type":1,"pageTitle":"Lua 🚧","url":"/bindings/lua/#build-from-source","content":" Build OpenDAL Lua Interface  $ cd bindings/lua $ cargo build --package opendal-lua --release   Install opendal Lua library  # copy to lua share library directory # default lua5.2 share library directory is /usr/lib/lua/5.2 $ cp ../../target/release/libopendal_lua.so /usr/lib/lua/5.2/opendal.so   ","version":"Next","tagName":"h2"},{"title":"Install from LuaRocks​","type":1,"pageTitle":"Lua 🚧","url":"/bindings/lua/#install-from-luarocks","content":" $ luarocks make   ","version":"Next","tagName":"h2"},{"title":"Usage​","type":1,"pageTitle":"Lua 🚧","url":"/bindings/lua/#usage","content":" $ lua5.2 example/fs.lua read: hello world   ","version":"Next","tagName":"h2"},{"title":"Test​","type":1,"pageTitle":"Lua 🚧","url":"/bindings/lua/#test","content":" $ busted -o gtest test/opendal_test.lua [==========] Running tests from scanned files. [----------] Global test environment setup. [----------] Running tests from test/opendal_test.lua [ RUN ] test/opendal_test.lua @ 24: opendal unit test opendal fs schema operator function in fs schema [ OK ] test/opendal_test.lua @ 24: opendal unit test opendal fs schema operator function in fs schema (1.52 ms) [ RUN ] test/opendal_test.lua @ 36: opendal unit test opendal fs schema meta function in fs schema [ OK ] test/opendal_test.lua @ 36: opendal unit test opendal fs schema meta function in fs schema (0.24 ms) [----------] 2 tests from test/opendal_test.lua (3.47 ms total) [----------] Global test environment teardown. [==========] 2 tests from 1 test file ran. (3.54 ms total) [ PASSED ] 2 tests.   ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Lua 🚧","url":"/bindings/lua/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"OCaml 🚧","type":0,"sectionRef":"#","url":"/bindings/ocaml/","content":"","keywords":"","version":"Next"},{"title":"Requirements​","type":1,"pageTitle":"OCaml 🚧","url":"/bindings/ocaml/#requirements","content":" OCaml version > 4.03 and < 5.0.0  ","version":"Next","tagName":"h2"},{"title":"Contributing​","type":1,"pageTitle":"OCaml 🚧","url":"/bindings/ocaml/#contributing","content":" ","version":"Next","tagName":"h2"},{"title":"Setup​","type":1,"pageTitle":"OCaml 🚧","url":"/bindings/ocaml/#setup","content":" We recommend using OPAM the OCaml Package Manager to install and manage the OCaml environment.  Install OPAM​  The quickest way to get the latest opam up and working is to run this script:  bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)"   Similarly, you can also use your distribution's package manager to install  Arch​  pacman -S opam   Debian | Ubuntu​  apt-get install opam   macOS​  brew install opam   Init OPAM​  Do not put sudo in front of any opam commands. That would break your OCaml installation.  After Installing OPAM, we need to initialize it  For the general case, we can execute  opam init --bare -a -y   If you are using WSL1 on windows, run:  opam init --bare -a -y --disable-sandboxing   Create OPAM Switch​  Using opam, we can have multiple versions of ocaml at the same time; this is called switch.  Due to the upstream ocaml-rs, we currently do not support OCaml5, and recommend using the latest version of OCaml4 We can create use this command:  opam switch create opendal-ocaml4.14 ocaml-base-compiler.4.14.0 eval $(opam env)   Install OPAM Package​  OpenDAL does not depend on opam package except ounit2 for testing. However, to facilitate development in an IDE such as vscode, it is usually necessary to install the following content  opam install -y utop odoc ounit2 ocaml-lsp-server ocamlformat ocamlformat-rpc   ","version":"Next","tagName":"h3"},{"title":"Build​","type":1,"pageTitle":"OCaml 🚧","url":"/bindings/ocaml/#build","content":" cd bindings/ocaml dune build   ","version":"Next","tagName":"h3"},{"title":"Test​","type":1,"pageTitle":"OCaml 🚧","url":"/bindings/ocaml/#test","content":" To execute unit tests, we can simply use the following command:  cd bindings/ocaml dune test   ","version":"Next","tagName":"h3"},{"title":"License and Trademarks​","type":1,"pageTitle":"OCaml 🚧","url":"/bindings/ocaml/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Java","type":0,"sectionRef":"#","url":"/bindings/java/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Java","url":"/bindings/java/#useful-links","content":" DocumentationUpgrade Guide  ","version":"Next","tagName":"h2"},{"title":"Example​","type":1,"pageTitle":"Java","url":"/bindings/java/#example","content":" import java.util.HashMap; import java.util.Map; import org.apache.opendal.AsyncOperator; import org.apache.opendal.Operator; public class Main { public static void main(String[] args) { final Map<String, String> conf = new HashMap<>(); conf.put("root", "/tmp"); try (AsyncOperator op = AsyncOperator.of("fs", conf)) { op.write("/path/to/data", "Hello world").join(); System.out.println(new String(op.read("/path/to/data").join())); } } }   ","version":"Next","tagName":"h2"},{"title":"Getting Started​","type":1,"pageTitle":"Java","url":"/bindings/java/#getting-started","content":" This project is built upon the native OpenDAL lib. And it is released for multiple platforms that you can use a classifier to specify the platform you are building the application on.  ","version":"Next","tagName":"h2"},{"title":"Maven​","type":1,"pageTitle":"Java","url":"/bindings/java/#maven","content":" Generally, you can first add the os-maven-plugin for automatically detect the classifier based on your platform:  <build> <extensions> <extension> <groupId>kr.motd.maven</groupId> <artifactId>os-maven-plugin</artifactId> <version>1.7.0</version> </extension> </extensions> </build>   Then add the dependency to opendal as following:  <dependencies> <dependency> <groupId>org.apache.opendal</groupId> <artifactId>opendal</artifactId> <version>${opendal.version}</version> </dependency> <dependency> <groupId>org.apache.opendal</groupId> <artifactId>opendal</artifactId> <version>${opendal.version}</version> <classifier>${os.detected.classifier}</classifier> </dependency> </dependencies>   ","version":"Next","tagName":"h3"},{"title":"Gradle​","type":1,"pageTitle":"Java","url":"/bindings/java/#gradle","content":" For Gradle, you can first add the com.google.osdetector for automatically detect the classifier based on your platform:  plugins { id "com.google.osdetector" version "1.7.3" }   Then add the dependency to `opendal as following:  dependencies { implementation "org.apache.opendal:opendal:$opendalVersion" implementation "org.apache.opendal:opendal:$opendalVersion:$osdetector.classifier" }   ","version":"Next","tagName":"h3"},{"title":"Classified library​","type":1,"pageTitle":"Java","url":"/bindings/java/#classified-library","content":" Note that the dependency without classifier ships all classes and resources except the "opendal_java" shared library. And those with classifier bundle only the shared library.  For downstream usage, it's recommended:  Depend on the one without classifier to write code;Depend on the classified ones with "test" for testing.  To load the shared library correctly, you can choose one of the following approaches:  Append the classified JARs to the classpath at the runtime;Depend on the classified JARs and build a fat JAR (You may need to depend on all the provided classified JARs for running on multiple platforms);Build your own "opendal_java" shared library and specify "-Djava.library.path" to the folder containing that shared library.  ","version":"Next","tagName":"h3"},{"title":"Build​","type":1,"pageTitle":"Java","url":"/bindings/java/#build","content":" This project provides OpenDAL Java bindings with artifact name opendal. It depends on JDK 8 or later.  You can use Maven to build both Rust dynamic lib and JAR files with one command now:  ./mvnw clean package -DskipTests=true   ","version":"Next","tagName":"h2"},{"title":"Run tests​","type":1,"pageTitle":"Java","url":"/bindings/java/#run-tests","content":" Currently, all tests are written in Java.  You can run the base tests with the following command:  ./mvnw clean verify   ","version":"Next","tagName":"h2"},{"title":"Code style​","type":1,"pageTitle":"Java","url":"/bindings/java/#code-style","content":" This project uses spotless for code formatting so that all developers share a consistent code style without bikeshedding on it.  You can apply the code style with the following command::  ./mvnw spotless:apply   ","version":"Next","tagName":"h2"},{"title":"Run behavior tests​","type":1,"pageTitle":"Java","url":"/bindings/java/#run-behavior-tests","content":" Services behavior tests read necessary configs from env vars or the .env file.  You can copy .env.example to ${project.rootdir}/.env and change the values on need, or directly set env vars with export KEY=VALUE.  Take fs for example, we need to enable bench on fs on /tmp/:  OPENDAL_TEST=fs OPENDAL_FS_ROOT=/tmp/   You can run service behavior tests of enabled with the following command:  ./mvnw test -Dtest="behavior.*Test"   Remember to enable the necessary features via -Dcargo-build.features=services-xxx when running specific service test:  export OPENDAL_TEST=redis export OPENDAL_REDIS_ENDPOINT=tcp://127.0.0.1:6379 export OPENDAL_REDIS_ROOT=/ export OPENDAL_REDIS_DB=0 ./mvnw test -Dtest="behavior.*Test" -Dcargo-build.features=services-redis   ","version":"Next","tagName":"h2"},{"title":"Used by​","type":1,"pageTitle":"Java","url":"/bindings/java/#used-by","content":" Check out the users list for more details on who is using OpenDAL.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Java","url":"/bindings/java/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"PHP 🚧","type":0,"sectionRef":"#","url":"/bindings/php/","content":"","keywords":"","version":"Next"},{"title":"Example​","type":1,"pageTitle":"PHP 🚧","url":"/bindings/php/#example","content":" use OpenDAL\\Operator; $op = new Operator("fs", ["root" => "/tmp"]); $op->write("test.txt", "hello world"); echo $op->read("test.txt"); // hello world   ","version":"Next","tagName":"h2"},{"title":"Requirements​","type":1,"pageTitle":"PHP 🚧","url":"/bindings/php/#requirements","content":" PHP 8.1+Composer  ","version":"Next","tagName":"h2"},{"title":"Install Extension​","type":1,"pageTitle":"PHP 🚧","url":"/bindings/php/#install-extension","content":" We use ext-php-rs to build PHP extensions natively in Rust, it's different from the traditional PHP extension development and cannot be installed using pecl or phpize. Before installing the extension, it is necessary to install Rust and Cargo. For instructions on how to install them, please refer to Rust's website.  Clone the repository  git clone git@github.com:apache/opendal.git   Build the opendal-php extension  cd opendal/bindings/php cargo build   don't forget to add --release flag for production use.  Enable extension for PHP Manually  cd opendal # Linux cp target/debug/libopendal_php.so $(php -r "echo ini_get('extension_dir');")/libopendal_php.so echo "extension=libopendal_php.so" >> $(php -r "echo php_ini_loaded_file();") # macOS cp target/debug/libopendal_php.dylib $(php -r "echo ini_get('extension_dir');")/libopendal_php.dylib echo "extension=libopendal_php.dylib" >> $(php -r "echo php_ini_loaded_file();") # Windows cp target/debug/libopendal_php.dll $(php -r "echo ini_get('extension_dir');")/libopendal_php.dll echo "extension=libopendal_php.dll" >> $(php -r "echo php_ini_loaded_file();")   Enable extension for PHP using cargo-php  You can also use cargo-php directly to install the extension, see cargo-php for more details.  cargo install cargo-php cd opendal/bindings/php cargo php install   This command will automatically build the extension and copy it to the extension directory of the current PHP version.  Test  use php -m to check if the extension is installed successfully.  php -m | grep opendal-php   Composer test:  cd opendal/bindings/php composer install composer test   ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"PHP 🚧","url":"/bindings/php/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Go","type":0,"sectionRef":"#","url":"/bindings/go/","content":"","keywords":"","version":"Next"},{"title":"Basic Usage​","type":1,"pageTitle":"Go","url":"/bindings/go/#basic-usage","content":" package main import ( "fmt" "github.com/apache/opendal-go-services/memory" opendal "github.com/apache/opendal/bindings/go" ) func main() { // Initialize a new in-memory operator op, err := opendal.NewOperator(memory.Scheme, opendal.OperatorOptions{}) if err != nil { panic(err) } defer op.Close() // Write data to a file named "test" err = op.Write("test", []byte("Hello opendal go binding!")) if err != nil { panic(err) } // Read data from the file "test" data, err := op.Read("test") if err != nil { panic(err) } fmt.Printf("Read content: %s\\n", data) // List all entries under the root directory "/" lister, err := op.List("/") if err != nil { panic(err) } defer lister.Close() // Iterate through all entries for lister.Next() { entry := lister.Entry() // Get entry name (not used in this example) _ = entry.Name() // Get metadata for the current entry meta, _ := op.Stat(entry.Path()) // Print file size fmt.Printf("Size: %d bytes\\n", meta.ContentLength()) // Print last modified time fmt.Printf("Last modified: %s\\n", meta.LastModified()) // Check if the entry is a directory or a file fmt.Printf("Is directory: %v, Is file: %v\\n", meta.IsDir(), meta.IsFile()) } // Check for any errors that occurred during iteration if err := lister.Error(); err != nil { panic(err) } // Copy a file op.Copy("test", "test_copy") // Rename a file op.Rename("test", "test_rename") // Delete a file op.Delete("test_rename") }   ","version":"Next","tagName":"h2"},{"title":"Run Tests​","type":1,"pageTitle":"Go","url":"/bindings/go/#run-tests","content":" ","version":"Next","tagName":"h2"},{"title":"Behavior Tests​","type":1,"pageTitle":"Go","url":"/bindings/go/#behavior-tests","content":" cd tests/behavior_tests # Test a specific backend export OPENDAL_TEST=memory # Run all tests CGO_ENABLE=0 go test -v -run TestBehavior # Run specific test CGO_ENABLE=0 go test -v -run TestBehavior/Write # Run synchronously CGO_ENABLE=0 GOMAXPROCS=1 go test -v -run TestBehavior   ","version":"Next","tagName":"h3"},{"title":"Benchmark​","type":1,"pageTitle":"Go","url":"/bindings/go/#benchmark","content":" cd tests/behavior_tests # Benchmark a specific backend export OPENDAL_TEST=memory go test -bench .    A benchmark between purego+libffi vs CGO purego+libffi (as new.txt) goos: linux goarch: arm64 pkg: github.com/apache/opendal/bindings/go BenchmarkWrite4KiB-10 1000000 2844 ns/op BenchmarkWrite256KiB-10 163346 10092 ns/op BenchmarkWrite4MiB-10 12900 99161 ns/op BenchmarkWrite16MiB-10 1785 658210 ns/op BenchmarkRead4KiB-10 194529 6387 ns/op BenchmarkRead256KiB-10 14228 82704 ns/op BenchmarkRead4MiB-10 981 1227872 ns/op BenchmarkRead16MiB-10 328 3617185 ns/op PASS ok CGO (as old.txt) go test -bench=. -tags dynamic . goos: linux goarch: arm64 pkg: opendal.apache.org/go BenchmarkWrite4KiB-10 241981 4240 ns/op BenchmarkWrite256KiB-10 126464 10105 ns/op BenchmarkWrite4MiB-10 13443 89578 ns/op BenchmarkWrite16MiB-10 1737 646155 ns/op BenchmarkRead4KiB-10 53535 20939 ns/op BenchmarkRead256KiB-10 9008 132738 ns/op BenchmarkRead4MiB-10 576 1846683 ns/op BenchmarkRead16MiB-10 230 6305322 ns/op PASS ok Diff with benchstat benchstat old.txt new.txt goos: linux goarch: arm64 pkg: github.com/apache/opendal/bindings/go │ new.txt │ │ sec/op │ Write4KiB-10 2.844µ ± ∞ ¹ Write256KiB-10 10.09µ ± ∞ ¹ Write4MiB-10 99.16µ ± ∞ ¹ Write16MiB-10 658.2µ ± ∞ ¹ Read4KiB-10 6.387µ ± ∞ ¹ Read256KiB-10 82.70µ ± ∞ ¹ Read4MiB-10 1.228m ± ∞ ¹ Read16MiB-10 3.617m ± ∞ ¹ geomean 90.23µ ¹ need >= 6 samples for confidence interval at level 0.95 pkg: opendal.apache.org/go │ old.txt │ │ sec/op │ Write4KiB-10 4.240µ ± ∞ ¹ Write256KiB-10 10.11µ ± ∞ ¹ Write4MiB-10 89.58µ ± ∞ ¹ Write16MiB-10 646.2µ ± ∞ ¹ Read4KiB-10 20.94µ ± ∞ ¹ Read256KiB-10 132.7µ ± ∞ ¹ Read4MiB-10 1.847m ± ∞ ¹ Read16MiB-10 6.305m ± ∞ ¹ geomean 129.7µ ¹ need >= 6 samples for confidence interval at level 0.95   ","version":"Next","tagName":"h3"},{"title":"Capabilities​","type":1,"pageTitle":"Go","url":"/bindings/go/#capabilities","content":"  OperatorInfo Stat Metadata IsExist Read Read Reader -- implement as io.ReadCloser Write Write Writer -- Need support from the C binding Delete CreateDir Lister Entry Metadata -- Need support from the C binding Copy Rename  ","version":"Next","tagName":"h2"},{"title":"Development​","type":1,"pageTitle":"Go","url":"/bindings/go/#development","content":" The guide is based on Linux and Windows. For other platforms, please adjust the commands accordingly.  To develop the Go binding, you need to have the following dependencies installed:  zstdRust toolchainGo(Required for Windows) libffi-8.dll in the root of the workspace directory  We use go workspace to manage and build the dependencies. To set up the workspace, run the following commands:   For Linux mkdir opendal_workspace cd opendal_workspace git clone --depth 1 git@github.com:apache/opendal.git git clone --depth 1 git@github.com:apache/opendal-go-services.git go work init go work use ./opendal/bindings/go go work use ./opendal/bindings/go/tests/behavior_tests # use the backend you want to test, e.g., fs or memory go work use ./opendal-go-services/fs go work use ./opendal-go-services/memory cat <<EOF > ./make_test.sh #!/bin/bash # Check if OPENDAL_TEST is set if [ -z "\\$OPENDAL_TEST" ]; then echo "Error: OPENDAL_TEST environment variable is not set" echo "Please set OPENDAL_TEST to specify which backend to test (e.g., fs or memory)" exit 1 fi # Specify the backend to test export SERVICE="\\$OPENDAL_TEST" # Get architecture architecture=\\$(uname -m) if [ "\\$architecture" = "x86_64" ]; then ARCH="amd64" GOARCH="amd64" elif [ "\\$architecture" = "aarch64" ] || [ "\\$architecture" = "arm64" ]; then ARCH="arm64" GOARCH="arm64" else ARCH="unknown" fi # Build opendal cd opendal/bindings/c cargo build cd - # Set environment variables export GITHUB_WORKSPACE="\\$PWD/opendal-go-services" export VERSION="latest" export TARGET="linux" export DIR="\\$GITHUB_WORKSPACE/libopendal_c_\\${VERSION}_\\${SERVICE}_\\${TARGET}" # Create directory if not exists mkdir -p "\\$DIR" export OUTPUT="\\$DIR/libopendal_c.\\$TARGET.so.zst" # Compress with zstd zstd -19 opendal/bindings/c/target/debug/libopendal_c.so -o \\$OUTPUT # Set environment variables for test export MATRIX='{"build": [{"target":"linux", "goos":"linux", "goarch": "'\\$GOARCH'"}], "service": ["fs"]}' # Generate code cd opendal-go-services/internal/generate go run generate.go cd - # Delete unnecessary files rm -rf \\$DIR # Run tests go test ./opendal/bindings/go/tests/behavior_tests -v -run TestBehavior EOF chmod +x ./make_test.sh cd - To build and run tests, run the following commands: cd opendal_workspace # specify the backend to test export OPENDAL_TEST=fs export OPENDAL_FS_ROOT=/tmp/opendal # build the C binding and run the tests ./make_test.sh cd -    For Windows New-Item -ItemType Directory -Path opendal_workspace Set-Location -Path opendal_workspace git clone --depth 1 git@github.com:apache/opendal.git git clone --depth 1 git@github.com:apache/opendal-go-services.git go work init go work use ./opendal/bindings/go go work use ./opendal/bindings/go/tests/behavior_tests # use the backend you want to test, e.g., fs or memory go work use ./opendal-go-services/fs go work use ./opendal-go-services/memory @' # Check if OPENDAL_TEST is set\\;if (-not $env:OPENDAL_TEST) {\\; Write-Error "OPENDAL_TEST environment variable is not set"\\; Write-Host "Please set OPENDAL_TEST to specify which backend to test (e.g., fs or memory)"\\; exit 1\\;}\\;# Specify the backend to test\\;Set-Item -Path Env:SERVICE -Value "$env:OPENDAL_TEST"\\;# Get architecture\\;$architecture = (Get-WmiObject Win32_OperatingSystem).OSArchitecture\\;\\;if ($architecture -like "*64*") {\\; $ARCH = "x86_64"\\;} else {\\; $ARCH = "unknown" \\;}\\;\\;# Build opendal\\;Push-Location opendal/bindings/c\\;cargo build\\;Pop-Location\\;\\;# Rename dll file\\;Rename-Item opendal/bindings/c/target/debug/opendal_c.dll libopendal_c.dll\\;\\;# Set environment variables\\;Set-Item -Path Env:GITHUB_WORKSPACE -Value "$PWD/opendal-go-services"\\;Set-Item -Path Env:VERSION -Value "latest"\\;Set-Item -Path Env:TARGET -Value "windows"\\;Set-Item -Path Env:DIR -Value "$($env:GITHUB_WORKSPACE)/libopendal_c_$($env:VERSION)_$($env:SERVICE)_$($env:TARGET)"\\;\\;if (-not (Test-Path $env:DIR)) {\\; New-Item -ItemType Directory -Path $env:DIR\\;}\\;\\;# Compress with zstd\\;zstd -19 opendal/bindings/c/target/debug/libopendal_c.dll -o "$($env:DIR)/libopendal_c.windows.dll.zst"\\;\\;Push-Location opendal-go-services/internal/generate\\;go run generate.go\\;Pop-Location\\;# Remove Unnecessary files\\;Remove-Item -Path $env:DIR -Recurse -Force\\;# Set environment variables\\;Set-Item -Path Env:MATRIX -Value '{"build": [{"target":"windows", "goos":"windows", "goarch": "amd64"}], "service": ["fs"]}'\\;# Assume that libffi-8.dll is in the root of workspace directory\\;Set-Item -Path Env:PATH -Value "$($env:PATH);$PWD"\\;# Run tests\\;go test ./opendal/bindings/go/tests/behavior_tests -v -run TestBehavior\\; '@ -replace "\\\\;","`n" | Out-File -FilePath "MakeTest.ps1" -Encoding UTF8 Pop-Location To build and run tests, run the following commands: Set-Location -Path opendal_workspace # specify the backend to test $env:OPENDAL_TEST = "fs" $env:OPENDAL_FS_ROOT = $env:TEMP # build the C binding and run the tests .\\MakeTest.ps1 Pop-Location   ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Go","url":"/bindings/go/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Node.js","type":0,"sectionRef":"#","url":"/bindings/nodejs/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Node.js","url":"/bindings/nodejs/#useful-links","content":" DocumentationUpgrade Guide  ","version":"Next","tagName":"h2"},{"title":"Installation​","type":1,"pageTitle":"Node.js","url":"/bindings/nodejs/#installation","content":" npm install opendal   ","version":"Next","tagName":"h2"},{"title":"Docs​","type":1,"pageTitle":"Node.js","url":"/bindings/nodejs/#docs","content":" To build the docs locally, please run the following commands:  # Only need to run once unless you want to update the docs theme pnpm run build:theme # Build the docs pnpm run docs   ","version":"Next","tagName":"h2"},{"title":"Tests​","type":1,"pageTitle":"Node.js","url":"/bindings/nodejs/#tests","content":" Services behavior tests read necessary configs from env vars or the .env file.  You can copy .env.example to $(pwd)/.env and change the values on need, or directly set env vars with export KEY=VALUE.  Take fs for example, we need to enable bench on fs on /tmp:  OPENDAL_TEST=fs OPENDAL_FS_ROOT=/tmp   You can run service behavior tests of enabled with the following command:  pnpm build && pnpm test   ","version":"Next","tagName":"h2"},{"title":"Usage​","type":1,"pageTitle":"Node.js","url":"/bindings/nodejs/#usage","content":" import { Operator } from "opendal"; async function main() { const op = new Operator("fs", { root: "/tmp" }); await op.write("test", "Hello, World!"); const bs = await op.read("test"); console.log(new TextDecoder().decode(bs)); const meta = await op.stat("test"); console.log(`contentLength: ${meta.contentLength}`); } main();   ","version":"Next","tagName":"h2"},{"title":"Contributing​","type":1,"pageTitle":"Node.js","url":"/bindings/nodejs/#contributing","content":" Start with Contributing Guide.Submit Issues for bug report or feature requests.Asking questions in the Discussions. Talk to community at Discord.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Node.js","url":"/bindings/nodejs/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Ruby 🚧","type":0,"sectionRef":"#","url":"/bindings/ruby/","content":"","keywords":"","version":"Next"},{"title":"Development​","type":1,"pageTitle":"Ruby 🚧","url":"/bindings/ruby/#development","content":" Install gems:  bundle   Build bindings:  bundle exec rake compile   Run tests:  bundle exec rake test   ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Ruby 🚧","url":"/bindings/ruby/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Python","type":0,"sectionRef":"#","url":"/bindings/python/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Python","url":"/bindings/python/#useful-links","content":" DocumentationExamplesUpgrade Guide  ","version":"Next","tagName":"h2"},{"title":"Installation​","type":1,"pageTitle":"Python","url":"/bindings/python/#installation","content":" pip install opendal   ","version":"Next","tagName":"h2"},{"title":"Usage​","type":1,"pageTitle":"Python","url":"/bindings/python/#usage","content":" fs service example:  import opendal op = opendal.Operator("fs", root="/tmp") op.write("test.txt", b"Hello World") print(op.read("test.txt")) print(op.stat("test.txt").content_length)   Or using the async API:  import asyncio async def main(): op = opendal.AsyncOperator("fs", root="/tmp") await op.write("test.txt", b"Hello World") print(await op.read("test.txt")) asyncio.run(main())   s3 service example:  import opendal op = opendal.Operator("s3", root="/tmp", bucket="your_bucket_name", region="your_region") op.write("test.txt", b"Hello World") print(op.read("test.txt")) print(op.stat("test.txt").content_length)   Or using the async API:  import asyncio async def main(): op = opendal.AsyncOperator("s3", root="/tmp", bucket="your_bucket_name", region="your_region") await op.write("test.txt", b"Hello World") print(await op.read("test.txt")) asyncio.run(main())   ","version":"Next","tagName":"h2"},{"title":"Development​","type":1,"pageTitle":"Python","url":"/bindings/python/#development","content":" Setup virtualenv:  uv venv --python 3.11   Install all the dependencies:  uv sync --all-groups --all-extras   Run some tests:  # To run `test_write.py` and use `fs` operator OPENDAL_TEST=fs OPENDAL_FS_ROOT=/tmp uv run pytest -vk test_write   Build API docs:  uv run mkdocs build   ","version":"Next","tagName":"h2"},{"title":"Used by​","type":1,"pageTitle":"Python","url":"/bindings/python/#used-by","content":" Check out the users list for more details on who is using OpenDAL.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Python","url":"/bindings/python/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Zig 🚧","type":0,"sectionRef":"#","url":"/bindings/zig/","content":"","keywords":"","version":"Next"},{"title":"Build​","type":1,"pageTitle":"Zig 🚧","url":"/bindings/zig/#build","content":" To compile OpenDAL Zig binding from source code, you need:  Zig 0.14.0 or higher  # build libopendal_c (underneath call make -C ../c) zig build libopendal_c # build and run unit tests zig build test --summary all   ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Zig 🚧","url":"/bindings/zig/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Swift 🚧","type":0,"sectionRef":"#","url":"/bindings/swift/","content":"","keywords":"","version":"Next"},{"title":"Using the Package​","type":1,"pageTitle":"Swift 🚧","url":"/bindings/swift/#using-the-package","content":" ","version":"Next","tagName":"h2"},{"title":"Build C Dependencies​","type":1,"pageTitle":"Swift 🚧","url":"/bindings/swift/#build-c-dependencies","content":" The Swift binding depends on the C binding to OpenDAL. Before using this package, you need to build the C library first:  cd bindings/swift make build-c   To check whether the package is ready to use, simply run the test:  make test   ","version":"Next","tagName":"h3"},{"title":"Add Dependency to Your Project​","type":1,"pageTitle":"Swift 🚧","url":"/bindings/swift/#add-dependency-to-your-project","content":" The package manifest is not located at the root directory of its repository. To use it, add the path of this package to the Package.swift manifest of your project:  // swift-tools-version:5.7 import PackageDescription let package = Package( name: "MyTool", dependencies: [ .package(path: "/path/to/opendal/bindings/swift/OpenDAL"), ], targets: [ .target(name: "MyTool", dependencies: [ .product(name: "OpenDAL", package: "OpenDAL"), ]), ] )   ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Swift 🚧","url":"/bindings/swift/#example","content":" The demo below shows how to write a key to the memory storage, and read it back:  import OpenDAL // Create an operator with `memory` backend. let op = try Operator(scheme: "memory") // Write some data into path `/demo`. let someData = Data([1, 2, 3, 4]) try op.blockingWrite(someData, to: "/demo") // Read the data back. let readData = try op.blockingRead("/demo") // You can use the read data here. print(readData!)   ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Swift 🚧","url":"/bindings/swift/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Cloud Filter","type":0,"sectionRef":"#","url":"/integrations/cloud_filter/","content":"","keywords":"","version":"Next"},{"title":"Example​","type":1,"pageTitle":"Cloud Filter","url":"/integrations/cloud_filter/#example","content":" use anyhow::Result; use cloud_filter::root::PopulationType; use cloud_filter::root::SecurityId; use cloud_filter::root::Session; use cloud_filter::root::SyncRootIdBuilder; use cloud_filter::root::SyncRootInfo; use opendal::services; use opendal::Operator; use tokio::runtime::Handle; use tokio::signal; #[tokio::main] async fn main() -> Result<()> { // Create any service desired let op = Operator::from_iter::<services::S3>([ ("bucket".to_string(), "my_bucket".to_string()), ("access_key".to_string(), "my_access_key".to_string()), ("secret_key".to_string(), "my_secret_key".to_string()), ("endpoint".to_string(), "my_endpoint".to_string()), ("region".to_string(), "my_region".to_string()), ])? .finish(); let client_path = std::env::var("CLIENT_PATH").expect("$CLIENT_PATH is set"); // Create a sync root id let sync_root_id = SyncRootIdBuilder::new("cloud_filter_opendal") .user_security_id(SecurityId::current_user()?) .build(); // Register the sync root if not exists if !sync_root_id.is_registered()? { sync_root_id.register( SyncRootInfo::default() .with_display_name("OpenDAL Cloud Filter") .with_population_type(PopulationType::Full) .with_icon("shell32.dll,3") .with_version("1.0.0") .with_recycle_bin_uri("http://cloudmirror.example.com/recyclebin")? .with_path(&client_path)?, )?; } let handle = Handle::current(); let connection = Session::new().connect_async( &client_path, cloud_filter_opendal::CloudFilter::new(op, client_path.clone().into()), move |f| handle.block_on(f), )?; signal::ctrl_c().await?; // Drop the connection before unregister the sync root drop(connection); sync_root_id.unregister()?; Ok(()) }   ","version":"Next","tagName":"h2"},{"title":"Branding​","type":1,"pageTitle":"Cloud Filter","url":"/integrations/cloud_filter/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Cloud Filter","url":"/integrations/cloud_filter/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Dav Server","type":0,"sectionRef":"#","url":"/integrations/dav_server/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Dav Server","url":"/integrations/dav_server/#useful-links","content":" Documentation: release | dev  ","version":"Next","tagName":"h2"},{"title":"Examples​","type":1,"pageTitle":"Dav Server","url":"/integrations/dav_server/#examples","content":" use anyhow::Result; use dav_server::davpath::DavPath; use dav_server_opendalfs::OpendalFs; use opendal::services::Memory; use opendal::Operator; #[tokio::test] async fn test() -> Result<()> { let op = Operator::new(Memory::default())?.finish(); let webdavfs = OpendalFs::new(op); let metadata = webdavfs .metadata(&DavPath::new("/").unwrap()) .await .unwrap(); println!("{}", metadata.is_dir()); Ok(()) }   ","version":"Next","tagName":"h2"},{"title":"Branding​","type":1,"pageTitle":"Dav Server","url":"/integrations/dav_server/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Dav Server","url":"/integrations/dav_server/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Core","type":0,"sectionRef":"#","url":"/core/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Core","url":"/core/#useful-links","content":" Documentation: release | devExamplesRelease NotesUpgrade GuideRFC List  ","version":"Next","tagName":"h2"},{"title":"Services​","type":1,"pageTitle":"Core","url":"/core/#services","content":" OpenDAL supports the following storage services:  Type\tServicesStandard Storage Protocols\tftp http sftp webdav Object Storage Services\tazblob cos gcs obs oss s3 b2 openstack_swift upyun vercel_blob File Storage Services\tfs alluxio azdls azfile compfs dbfs gridfs hdfs hdfs_native ipfs webhdfs Consumer Cloud Storage Service\taliyun_drive gdrive onedrive dropbox icloud koofr pcloud seafile yandex_disk Key-Value Storage Services\tcacache cloudflare_kv dashmap memory etcd foundationdb persy redis rocksdb sled redb tikv atomicserver Database Storage Services\td1 mongodb mysql postgresql sqlite surrealdb Cache Storage Services\tghac memcached mini_moka moka vercel_artifacts Git Based Storage Services\thuggingface  ","version":"Next","tagName":"h2"},{"title":"Layers​","type":1,"pageTitle":"Core","url":"/core/#layers","content":" OpenDAL supports the following storage layers to extend the behavior:  Name\tDepends\tDescriptionAsyncBacktraceLayer\tasync-backtrace\tAdd Efficient, logical 'stack' traces of async functions for the underlying services. AwaitTreeLayer\tawait-tree\tAdd a Instrument await-tree for actor-based applications to the underlying services. BlockingLayer\ttokio\tAdd blocking API support for non-blocking services. ChaosLayer\trand\tInject chaos into underlying services for robustness test. ConcurrentLimitLayer\ttokio\tAdd concurrent request limit. DtraceLayer\tprobe\tSupport User Statically-Defined Tracing(aka USDT) on Linux LoggingLayer\tlog\tAdd log for every operations. MetricsLayer\tmetrics\tAdd metrics for every operations. MimeGuessLayer\tmime_guess\tAdd Content-Type automatically based on the file extension in the operation path. FastraceLayer\tfastrace\tAdd fastrace for every operations. OtelMetricsLayer\t[opentelemetry::metrics]\tAdd opentelemetry::metrics for every operations. OtelTraceLayer\topentelemetry::trace\tAdd opentelemetry::trace for every operations. PrometheusClientLayer\tprometheus_client\tAdd prometheus metrics for every operations. PrometheusLayer\tprometheus\tAdd prometheus metrics for every operations. RetryLayer\tbackon\tAdd retry for temporary failed operations. ThrottleLayer\tgovernor\tAdd a bandwidth rate limiter to the underlying services. TimeoutLayer\ttokio\tAdd timeout for every operations to avoid slow or unexpected hang operations. TracingLayer\ttracing\tAdd tracing for every operations.  ","version":"Next","tagName":"h2"},{"title":"Quickstart​","type":1,"pageTitle":"Core","url":"/core/#quickstart","content":" use opendal::Result; use opendal::layers::LoggingLayer; use opendal::services; use opendal::Operator; #[tokio::main] async fn main() -> Result<()> { // Pick a builder and configure it. let mut builder = services::S3::default(); builder.bucket("test"); // Init an operator let op = Operator::new(builder)? // Init with logging layer enabled. .layer(LoggingLayer::default()) .finish(); // Write data op.write("hello.txt", "Hello, World!").await?; // Read data let bs = op.read("hello.txt").await?; // Fetch metadata let meta = op.stat("hello.txt").await?; let mode = meta.mode(); let length = meta.content_length(); // Delete op.delete("hello.txt").await?; Ok(()) }   ","version":"Next","tagName":"h2"},{"title":"Examples​","type":1,"pageTitle":"Core","url":"/core/#examples","content":" Name\tDescriptionBasic\tShow how to use opendal to operate storage service. Concurrent Upload\tShow how to perform upload concurrently to a storage service. Multipart Upload\tShow how to perform a multipart upload to a storage service.  ","version":"Next","tagName":"h2"},{"title":"Contributing​","type":1,"pageTitle":"Core","url":"/core/#contributing","content":" Check out the CONTRIBUTING guide for more details on getting started with contributing to this project.  ","version":"Next","tagName":"h2"},{"title":"Used by​","type":1,"pageTitle":"Core","url":"/core/#used-by","content":" Check out the users list for more details on who is using OpenDAL.  ","version":"Next","tagName":"h2"},{"title":"Branding​","type":1,"pageTitle":"Core","url":"/core/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Core","url":"/core/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Fuse3","type":0,"sectionRef":"#","url":"/integrations/fuse3/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Fuse3","url":"/integrations/fuse3/#useful-links","content":" Documentation: release | dev  ","version":"Next","tagName":"h2"},{"title":"Examples​","type":1,"pageTitle":"Fuse3","url":"/integrations/fuse3/#examples","content":" use fuse3::path::Session; use fuse3::MountOptions; use fuse3::Result; use fuse3_opendal::Filesystem; use opendal::services::Memory; use opendal::Operator; #[tokio::test] async fn test() -> Result<()> { // Build opendal Operator. let op = Operator::new(Memory::default())?.finish(); // Build fuse3 file system. let fs = Filesystem::new(op, 1000, 1000); // Configure mount options. let mount_options = MountOptions::default(); // Start a fuse3 session and mount it. let mut mount_handle = Session::new(mount_options) .mount_with_unprivileged(fs, "/tmp/mount_test") .await?; let handle = &mut mount_handle; tokio::select! { res = handle => res?, _ = tokio::signal::ctrl_c() => { mount_handle.unmount().await? } } Ok(()) }   ","version":"Next","tagName":"h2"},{"title":"Branding​","type":1,"pageTitle":"Fuse3","url":"/integrations/fuse3/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Fuse3","url":"/integrations/fuse3/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Parquet","type":0,"sectionRef":"#","url":"/integrations/parquet/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Parquet","url":"/integrations/parquet/#useful-links","content":" Documentation: release | dev  ","version":"Next","tagName":"h2"},{"title":"Examples​","type":1,"pageTitle":"Parquet","url":"/integrations/parquet/#examples","content":" Add the following dependencies to your Cargo.toml with correct version:  [dependencies] parquet_opendal = "0.0.1" opendal = { version = "0.48.0", features = ["services-s3"] }   use std::sync::Arc; use arrow::array::{ArrayRef, Int64Array, RecordBatch}; use futures::StreamExt; use opendal::{services::S3Config, Operator}; use parquet::arrow::{AsyncArrowWriter, ParquetRecordBatchStreamBuilder}; use parquet_opendal::{AsyncReader, AsyncWriter}; #[tokio::main] async fn main() { let mut cfg = S3Config::default(); cfg.access_key_id = Some("my_access_key".to_string()); cfg.secret_access_key = Some("my_secret_key".to_string()); cfg.endpoint = Some("my_endpoint".to_string()); cfg.region = Some("my_region".to_string()); cfg.bucket = "my_bucket".to_string(); // Create a new operator let operator = Operator::from_config(cfg).unwrap().finish(); let path = "/path/to/file.parquet"; // Create an async writer let writer = AsyncWriter::new( operator .writer_with(path) .chunk(32 * 1024 * 1024) .concurrent(8) .await .unwrap(), ); let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef; let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap(); let mut writer = AsyncArrowWriter::try_new(writer, to_write.schema(), None).unwrap(); writer.write(&to_write).await.unwrap(); writer.close().await.unwrap(); /// `gap(512 * 1024)` - Sets the maximum gap size (in bytes) to merge small byte ranges /// to 512 KB. /// `chunk(16 * 1024 * 1024)` - Sets the chunk size (in bytes) for reading data to 16 MB. /// `concurrent(16)` - Sets the number of concurrent fetch operations to 16. let reader = operator .reader_with(path) .gap(512 * 1024) .chunk(16 * 1024 * 1024) .concurrent(16) .await .unwrap(); let content_len = operator.stat(path).await.unwrap().content_length(); // `with_prefetch_footer_size(512 * 1024)` - Sets the prefetch footer size to 512 KB. let reader = AsyncReader::new(reader, content_len).with_prefetch_footer_size(512 * 1024); let mut stream = ParquetRecordBatchStreamBuilder::new(reader) .await .unwrap() .build() .unwrap(); let read = stream.next().await.unwrap().unwrap(); assert_eq!(to_write, read); }   ","version":"Next","tagName":"h2"},{"title":"Branding​","type":1,"pageTitle":"Parquet","url":"/integrations/parquet/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Parquet","url":"/integrations/parquet/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Unftp Sbe","type":0,"sectionRef":"#","url":"/integrations/unftp_sbe/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Unftp Sbe","url":"/integrations/unftp_sbe/#useful-links","content":" Documentation: release | dev  ","version":"Next","tagName":"h2"},{"title":"Examples​","type":1,"pageTitle":"Unftp Sbe","url":"/integrations/unftp_sbe/#examples","content":" use anyhow::Result; use opendal::Operator; use opendal::Scheme; use opendal::services; use unftp_sbe_opendal::OpendalStorage; #[tokio::main] async fn main() -> Result<()> { // Create any service desired let op = opendal::Operator::from_map::<services::S3>( [ ("bucket".to_string(), "my_bucket".to_string()), ("access_key".to_string(), "my_access_key".to_string()), ("secret_key".to_string(), "my_secret_key".to_string()), ("endpoint".to_string(), "my_endpoint".to_string()), ("region".to_string(), "my_region".to_string()), ] .into_iter() .collect(), )?.finish(); // Wrap the operator with `OpendalStorage` let backend = OpendalStorage::new(op); // Build the actual unftp server let server = libunftp::ServerBuilder::new(Box::new(move || backend.clone())).build()?; // Start the server server.listen("0.0.0.0:0").await?; Ok(()) }   ","version":"Next","tagName":"h2"},{"title":"Branding​","type":1,"pageTitle":"Unftp Sbe","url":"/integrations/unftp_sbe/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Unftp Sbe","url":"/integrations/unftp_sbe/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Spring","type":0,"sectionRef":"#","url":"/integrations/spring/","content":"","keywords":"","version":"Next"},{"title":"Overview​","type":1,"pageTitle":"Spring","url":"/integrations/spring/#overview","content":" This project includes three primary modules:  opendal-spring: Core integration module for Spring applications.opendal-spring-boot-starter: Synchronous starter for Spring WebMVC.opendal-spring-boot-starter-reactive: Asynchronous starter for Spring WebFlux.  ","version":"Next","tagName":"h2"},{"title":"Features​","type":1,"pageTitle":"Spring","url":"/integrations/spring/#features","content":" SpringBoot autoconfiguration support for an OpenDALTemplate/ReactiveOpenDALTemplate instance.  ","version":"Next","tagName":"h2"},{"title":"Prerequisites​","type":1,"pageTitle":"Spring","url":"/integrations/spring/#prerequisites","content":" This project requires JDK 17 or later and supports Spring 6 and Spring Boot 3.  ","version":"Next","tagName":"h2"},{"title":"Getting Started With Spring Boot Starter​","type":1,"pageTitle":"Spring","url":"/integrations/spring/#getting-started-with-spring-boot-starter","content":" Below is a brief example demonstrating how to use the OpenGemini Spring Boot Starter in a Java application.  ","version":"Next","tagName":"h2"},{"title":"Maven Configuration​","type":1,"pageTitle":"Spring","url":"/integrations/spring/#maven-configuration","content":" Add the following dependency to your project's pom.xml:  <dependency> <groupId>org.apache.opendal</groupId> <artifactId>opendal-spring-boot-starter</artifactId> <version>${version}</version> </dependency>   ","version":"Next","tagName":"h3"},{"title":"SpringBoot Application Configuration​","type":1,"pageTitle":"Spring","url":"/integrations/spring/#springboot-application-configuration","content":" Following properties can be used in your application.yaml:  spring: opendal: schema: "fs" conf: root: "/tmp"   ","version":"Next","tagName":"h3"},{"title":"Getting Started With Spring Boot Reactive Starter​","type":1,"pageTitle":"Spring","url":"/integrations/spring/#getting-started-with-spring-boot-reactive-starter","content":" Below is a brief example demonstrating how to use the OpenGemini Spring Boot Starter in a Java application.  ","version":"Next","tagName":"h2"},{"title":"Maven Configuration​","type":1,"pageTitle":"Spring","url":"/integrations/spring/#maven-configuration-1","content":" Add the following dependency to your project's pom.xml:  <dependency> <groupId>org.apache.opendal</groupId> <artifactId>opendal-spring-boot-starter-reactive</artifactId> <version>${version}</version> </dependency>   ","version":"Next","tagName":"h3"},{"title":"SpringBoot Reactive Application Configuration​","type":1,"pageTitle":"Spring","url":"/integrations/spring/#springboot-reactive-application-configuration","content":" Following properties can be used in your application.yaml:  spring: opendal: schema: "fs" conf: root: "/tmp"  ","version":"Next","tagName":"h3"},{"title":"Virtiofs","type":0,"sectionRef":"#","url":"/integrations/virtiofs/","content":"","keywords":"","version":"Next"},{"title":"Branding​","type":1,"pageTitle":"Virtiofs","url":"/integrations/virtiofs/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Virtiofs","url":"/integrations/virtiofs/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Object Store","type":0,"sectionRef":"#","url":"/integrations/object_store/","content":"","keywords":"","version":"Next"},{"title":"Useful Links​","type":1,"pageTitle":"Object Store","url":"/integrations/object_store/#useful-links","content":" Documentation: release | dev  ","version":"Next","tagName":"h2"},{"title":"Examples​","type":1,"pageTitle":"Object Store","url":"/integrations/object_store/#examples","content":" opendal_store_opendal depends on the opendal crate. Please make sure to always use the latest versions of both.  latest object_store_opendal   latest opendal   ","version":"Next","tagName":"h2"},{"title":"1. using object_store API to access S3​","type":1,"pageTitle":"Object Store","url":"/integrations/object_store/#1-using-object_store-api-to-access-s3","content":" Add the following dependencies to your Cargo.toml with correct version:  [dependencies] object_store = "0.11.0" object_store_opendal = "xxx" # see the latest version above opendal = { version = "xxx", features = ["services-s3"] } # see the latest version above   Build OpendalStore via opendal::Operator:  use std::sync::Arc; use bytes::Bytes; use object_store::path::Path; use object_store::ObjectStore; use object_store_opendal::OpendalStore; use opendal::services::S3; use opendal::{Builder, Operator}; #[tokio::main] async fn main() { let builder = S3::from_map( vec![ ("access_key".to_string(), "my_access_key".to_string()), ("secret_key".to_string(), "my_secret_key".to_string()), ("endpoint".to_string(), "my_endpoint".to_string()), ("region".to_string(), "my_region".to_string()), ] .into_iter() .collect(), ).unwrap(); // Create a new operator let operator = Operator::new(builder).unwrap().finish(); // Create a new object store let object_store = Arc::new(OpendalStore::new(operator)); let path = Path::from("data/nested/test.txt"); let bytes = Bytes::from_static(b"hello, world! I am nested."); object_store.put(&path, bytes.clone().into()).await.unwrap(); let content = object_store .get(&path) .await .unwrap() .bytes() .await .unwrap(); assert_eq!(content, bytes); }   ","version":"Next","tagName":"h3"},{"title":"2. querying data in a S3 bucket using DataFusion​","type":1,"pageTitle":"Object Store","url":"/integrations/object_store/#2-querying-data-in-a-s3-bucket-using-datafusion","content":" Add the following dependencies to your Cargo.toml with correct version:  [dependencies] object_store = "0.11.0" object_store_opendal = "xxx" # see the latest version above opendal = { version = "xxx", features = ["services-s3"] } # see the latest version above datafusion = "44.0.0" url = "2.5.2"   Build OpendalStore via opendal::Operator and register it to DataFusion:  use datafusion::error::DataFusionError; use datafusion::error::Result; use datafusion::prelude::*; use opendal::services::S3; use opendal::Operator; use std::sync::Arc; use url::Url; #[tokio::main] async fn main() -> Result<()> { let ctx = SessionContext::new(); // Configure OpenDAL for S3 let region = "my_region"; let bucket_name = "my_bucket"; let builder = S3::default() .endpoint("my_endpoint") .bucket(bucket_name) .region(region) .access_key_id("my_access_key") .secret_access_key("my_secret_key"); let op = Operator::new(builder) .map_err(|err| DataFusionError::External(Box::new(err)))? .finish(); let store = object_store_opendal::OpendalStore::new(op); // Register the object store let path = format!("s3://{bucket_name}"); let s3_url = Url::parse(&path).unwrap(); ctx.register_object_store(&s3_url, Arc::new(store)); // Register CSV file as a table let path = format!("s3://{bucket_name}/csv/data.csv"); ctx.register_csv("trips", &path, CsvReadOptions::default()) .await?; // Execute the query let df = ctx.sql("SELECT * FROM trips LIMIT 10").await?; // Print the results df.show().await?; // Dynamic query using the file path directly let ctx = ctx.enable_url_table(); let df = ctx .sql(format!(r#"SELECT * FROM '{}' LIMIT 10"#, &path).as_str()) .await?; // Print the results df.show().await?; Ok(()) }   ","version":"Next","tagName":"h3"},{"title":"WASM support​","type":1,"pageTitle":"Object Store","url":"/integrations/object_store/#wasm-support","content":" To build with wasm32-unknown-unknown target, you need to enable the send_wrapper feature:  cargo build --target wasm32-unknown-unknown --features send_wrapper   ","version":"Next","tagName":"h2"},{"title":"Branding​","type":1,"pageTitle":"Object Store","url":"/integrations/object_store/#branding","content":" The first and most prominent mentions must use the full form: Apache OpenDAL™ of the name for any individual usage (webpage, handout, slides, etc.) Depending on the context and writing style, you should use the full form of the name sufficiently often to ensure that readers clearly understand the association of both the OpenDAL project and the OpenDAL software product to the ASF as the parent organization.  For more details, see the Apache Product Name Usage Guide.  ","version":"Next","tagName":"h2"},{"title":"License and Trademarks​","type":1,"pageTitle":"Object Store","url":"/integrations/object_store/#license-and-trademarks","content":" Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0  Apache OpenDAL, OpenDAL, and Apache are either registered trademarks or trademarks of the Apache Software Foundation. ","version":"Next","tagName":"h2"},{"title":"Vision","type":0,"sectionRef":"#","url":"/vision/","content":"","keywords":"","version":"Next"},{"title":"Charter​","type":1,"pageTitle":"Vision","url":"/vision/#charter","content":" One Layer, All Storage.  ","version":"Next","tagName":"h2"},{"title":"Principles​","type":1,"pageTitle":"Vision","url":"/vision/#principles","content":" Principles are guiding principles. They guide how decisions are made for the whole project. Ideally, we do all of them all the time. In some cases, though, we may be forced to decide between slightly penalizing one goal or another. In that case, we tend to support those goals that come earlier in the list over those that come later (but every case is different).  ","version":"Next","tagName":"h2"},{"title":"0. Open Community​","type":1,"pageTitle":"Vision","url":"/vision/#0-open-community","content":" OpenDAL SHOULD be an open storage library.  OpenDAL is an ASF project governed by the OpenDAL PMC. At ASF, we believe in "Community Over Code" and adhere to the Apache Way. We aim to develop OpenDAL to meet the needs of our community. We do not maintain private versions or include features that aren't useful to others.  For example, OpenDAL prefers to have clear and readable code, as this allows more people in the community to join the development.  ","version":"Next","tagName":"h3"},{"title":"1. Solid Foundation​","type":1,"pageTitle":"Vision","url":"/vision/#1-solid-foundation","content":" OpenDAL SHOULD be a solid storage library.  OpenDAL is a solid foundation of user projects that users can trust OpenDAL to perform operations on real-world storage services. OpenDAL SHOULD always focus on building a Solid Foundation.  For example, OpenDAL performs additional error checks for AWS S3 complete multipart operations, as S3 may return an error in response with a 200 status code, even though this may add extra costs that conflict with "Fast Access.”  ","version":"Next","tagName":"h3"},{"title":"2. Fast Access​","type":1,"pageTitle":"Vision","url":"/vision/#2-fast-access","content":" OpenDAL SHOULD be a fast storage library.  Its fast access ensures that OpenDAL implements storage support with zero overhead. Users can integrate with OpenDAL without concerns about additional costs. OpenDAL should be as fast as, or faster than, the SDK for any given storage service.  For example, OpenDAL uses Capability to describe the capabilities of different services and adopts native features of those services whenever possible.  ","version":"Next","tagName":"h3"},{"title":"3. Object Storage First​","type":1,"pageTitle":"Vision","url":"/vision/#3-object-storage-first","content":" OpenDAL SHOULD be an object storage first library.  OpenDAL does support all storage services, but it is usually optimized for modern storage services. At the time of writing, we can say OpenDAL is object storage first. When designing features, OpenDAL tends to prioritize optimization for object storage.  For example, OpenDAL's Buffer design is primarily optimized for HTTP-based services, helping to reduce extra allocation, in-memory copying, and memory usage.  ","version":"Next","tagName":"h3"},{"title":"4. Extensible Architecture​","type":1,"pageTitle":"Vision","url":"/vision/#4-extensible-architecture","content":" OpenDAL SHOULD be an extensible storage library.  OpenDAL can be extended to various languages, backends, and layers. Each is independent and does not depend on the others. Users can combine different layers, such as metrics, logging, tracing, and retry, and extend their own languages, backends, and layers.  For example, OpenDAL's core never relies on the behavior or dependency of a single layer. Users can stack as many layers as they want on a given operator.  ","version":"Next","tagName":"h3"},{"title":"Use Cases​","type":1,"pageTitle":"Vision","url":"/vision/#use-cases","content":" Who are typical OpenDAL users? How would they use OpenDAL?  ","version":"Next","tagName":"h2"},{"title":"Infrastructure Builders​","type":1,"pageTitle":"Vision","url":"/vision/#infrastructure-builders","content":" Examples:  DatabendRisingWaveGreptimeDBApache Iceberg Rust  Use Cases:  Building storage systems like databasesDeveloping data processing pipelinesCreating backup and archive solutions  Primary Concerns:  Solid Foundation: Need guaranteed consistency and predictability for storage operationsFast Access: Require minimal overhead and optimal performanceWhy: Infrastructure services demand both reliability and performance as foundational requirements  ","version":"Next","tagName":"h3"},{"title":"Application Developers​","type":1,"pageTitle":"Vision","url":"/vision/#application-developers","content":" Examples:  SccacheVectorRustic  Use Cases:  Building end-user applicationsDeveloping CLI toolsCreating web services  Primary Concerns:  Fast Access: Need efficient integration and optimal performanceObject Storage First: Benefit from optimizations for modern cloud storageWhy: Modern applications commonly use object storage and require responsive performance  ","version":"Next","tagName":"h3"},{"title":"Platform Developers​","type":1,"pageTitle":"Vision","url":"/vision/#platform-developers","content":" Examples:  PantsZinoShuttle  Use Cases:  Building AI/ML platformsDeveloping cloud servicesCreating developer tools  Primary Concerns:  Extensible Architecture: Need to customize and extend storage capabilitiesSolid Foundation: Require dependable storage operationsWhy: Platforms need flexibility to adapt to various use cases while maintaining reliability    This documentation is inpisred a lot by hyper’s VISION document. ","version":"Next","tagName":"h3"}],"options":{"id":"default"}}