Use Case

11:45 AM - 12:15 PM PST

Pulsar's Journey in Yahoo!: On-prem, Cloud, and Hybrid

In today’s world, we are seeing a big shift toward the Cloud. With this shift comes a big shift in the expectations we have for a messaging system, especially when the messaging system is presented as managed service in a large-scale, multi-tenant environment. For any large-scale enterprise, it’s very important to evaluate messaging system and be confident before expanding complex distributed data systems like Apache Pulsar from on-premise to elastically scalable, fully managed services on cloud services. We must consider aspects such as: migration from and integration with large-scale on-premise clusters, security, cost efficiency, and the cloud friendliness of the architecture, modeling cost and capacity, tenant isolation, deployment robustness, availability, monitoring, etc. Not every messaging system is built to be cloud-native and run as a managed service with cost efficiency. We have been running large-scale Apache Pulsar at Yahoo for the last 8 years on various platforms and hardware configurations while meeting application SLAs and serving more than 1M topics in a cluster. In this talk, we will talk about Pulsar’s journey in Yahoo! from an on-premise platform to a hybrid cloud and on-premise system. We will talk about Pulsar’s architecture and features that make Pulsar a good cloud-native messaging-system choice for any enterprise.

Speaker

Ludwig Pummer

Principal Production Engineer, Yahoo!

Ludwig Pummer is a Principal Production Engineer at Yahoo working on messaging and distributed key-value storage systems. He is the lead production engineer for Pulsar in Yahoo and has supported it since prior to the inception of the current architecture. He has worked on many large scale platforms at Yahoo, including Login and Account Registration, and is experienced in automated deployment and monitoring of large scale globally distributed systems.

Rajan Dhabalia

Senior Principal Software Engineer, Yahoo!

Rajan Dhabalia is a Senior Principal Software Engineer at Yahoo! working on messaging and distributed key-value storage technologies. His interests lie in building reliable, scalable distributed data processing systems. He is a PMC member of Apache Pulsar, a committer of Apache BookKeeper, a lead developer of Pulsar in Yahoo!, and spent several years developing Pulsar in Yahoo before making it open source.