Technology Deep Dive

10:50 AM - 11:20 AM, CEST , May 14th

Apache Kafka Done Right

Apache Kafka was first open-sourced in 2011 and quickly became the default infrastructure for building data streaming architectures. Some people hate it, some swear by it, but nearly every technology company uses it. The Kafka API itself isn’t the problem; rather, it was the implementation of Kafka that posed issues. Kafka is a well-designed piece of software that emerged during the Hadoop Era, a time when most infrastructure software was designed for on-premise deployment and optimized for slow networks. However, it's no longer 2011. A lot has changed in how we build modern cloud software, yet Kafka has remained more or less the same. Many organizations have managed to move Kafka into their cloud environments, but few are truly satisfied with the results. Meanwhile, numerous Kafka-like systems have emerged to tackle the challenges everyone is facing with Kafka. Apache Pulsar is clearly a standout. In this talk, we will explore the journey of developing Apache Pulsar, its evolution into the Ursa engine, unveil the design principles behind it, and discuss how we can reimagine Apache Kafka for today's complex hybrid environments.

Speaker

Sijie Guo

Co-Founder and CEO, StreamNative

Sijie Guo has worked on messaging and streaming data technologies for more than a decade.Sijie was the tech lead at Twitter for the messaging infrastructure group, where he co-created DistributedLog and Twitter EventBus. Prior to that, he worked on the push notification infrastructure at Yahoo!, where he was one of the original developers of BookKeeper and Pulsar. He is also the VP of Apache BookKeeper and PMC member of Apache Pulsar.