Oxia is a scalable metadata store and coordination system that can be used as the core infrastructure to build large scale distributed systems.
Oxia was created with the primary goal of providing an alternative Pulsar to replace ZooKeeper as the long term preferred metadata store, overcoming all the current limitations in terms of metadata access throughput and data set size.
Add a Pulsar MetadataStore plugin that uses Oxia client SDK.
Users will be able to start a Pulsar cluster using just Oxia, without any ZooKeeper involved.
It's not in the scope of this proposal to change any default behavior or configuration of Pulsar.
Oxia semantics and client SDK were already designed with Pulsar and MetadataStore plugin API in mind, so there is not much integration work that needs to be done here.
Just few notes:
AbstractBatchedMetadataStore
/a/b/c
key without /a/b
and /a
existing. In the Pulsar integration for Oxia we’re forcing to create all parent keys when they are not there. This is due to several places in BookKeeper access where it does not create the parent keys, though it will later make getChildren()
operations on the parents.Unlike in the ZooKeeper implementation, the notification of events is guaranteed in Oxia, because the Oxia client SDK will use the transaction offset after server reconnections and session restarted events. This will ensure that brokers cache will always be properly invalidated. We will then be able to remove the current 5minutes automatic cache refresh which is in place to prevent the ZooKeeper missed watch issue.
Oxia: https://github.com/streamnative/oxia Oxia Java Client SDK: https://github.com/streamnative/oxia-java