Ecosystem

10:10 AM - 10:50 AM PDT , June 17

Kafka on Pulsar - bringing native Kafka protocol support to Pulsar

Apache Pulsar is a distributed and open-source pub-sub messaging system. It offers many advantages over Kafka, such as multi-tenant, geo-replication, decoupled storage or even SQL and FaaS directly integrated. The only thing missing for wide adoption is support for the de-facto standard for streaming: Kafka. And this is how our story begins.
In this talk, Sijie Guo from StreamNative and Pierre Zemb from OVHcloud will share the journey on building Kafka-on-Pulsar (KoP) to bring native Kafka protocol support to Pulsar. Before joining the force on building KoP, OVHcloud implemented a Kafka proxy in Rust capable of transforming the Kafka protocol to that Pulsar on the fly and encountered some challenges. After realizing that StreamNative was working on bringing the Kafka protocol natively to Pulsar broker via a pluggable protocol handler mechanism. OVHCloud joined forces with StreamNative to work on brining Kafka protocol support to Pulsar brokers.
At the end of this talk, you will know more about the inner workings of Kafka and Pulsar. You'll also get feedback from both companies from their initial proofs of concepts and the current implementation.

Speaker

Pierre Zemb

Tech Lead @ OVHcloud

Pierre Zemb is a Technical Leader working around distributed and storage systems at OVHcloud, the biggest European cloud provider. He is currently working on ioStream, a topic-as-a-service product built on top of Apache Pulsar. He started to become an Apache contributor through contributions to Flink, HBase and now Pulsar. He's also giving a hand to differents local events, such as the local Devoxx4Kids, JUG/GDG and DevFest.

Sijie Guo

Pulsar PMC Member and CEO @ StreamNative

Sijie Guo is the co-founder and CEO of StreamNative. StreamNative is a real-time data infrastructure startup offering a cloud-native event streaming platform powered by Apache Pulsar for the enterprises. Before StreamNative, he co-founded Streamlio. Before Streamlio, he worked for Twitter as the tech lead for the messaging infrastructure group, where he co-created DistributedLog and Twitter EventBus. Before Twitter, he worked on the push notification infrastructure at Yahoo!. He is also the VP of Apache BookKeeper and PMC member of Apache Pulsar.