Keynote

5:00 PM - 5:30 PM, CEST , May 23

Oxia: Scaling Pulsar’s Metadata to 100x

Pulsar has traditionally relied on ZooKeeper for storing metadata and coordinating brokers. However, the limitations of ZooKeeper have been a longstanding challenge in scaling Pulsar clusters. The maximum number of Pulsar topics in a cluster is determined by the capacity of ZooKeeper and its throughput limits for write and read requests. To overcome these limitations, we have introduced Oxia, a metadata store and coordination system designed explicitly to address the requirements of Pulsar. Oxia replaces ZooKeeper and similar systems like Etcd, providing a horizontally scalable system for storing metadata and coordinating large clusters of machines. In this talk, we will discuss the design goals of Oxia, its architecture, and the journey that led to its development. We will cover how Oxia design can take advantage of modern cloud-native environments and provide and highly flexible and dynamic operational environment. Join me to learn more about scaling Pulsar's metadata with Oxia and how it can improve the performance and scalability of your Pulsar clusters.

Speaker

Matteo Merli

Pulsar PMC Chair and CTO, StreamNative