Technology Deep Dive

12:00AM-12:45AM CST, 20:00-20:45 PST , January 16

Designing Pulsar Clients to Run in Isolation

In this talk, we will go over the configurations knobs and all the virtual isolation features to run multiple clients in their assigned resource space without interfering with one another.
At nutanix we run a single physical pulsar cluster shared by multiple use cases and applications. The biggest challenge in such a setup is an unintended denial of service for all client just because one of the clients breached their quotas and exceeded load expectations.
In traditional distributed apps, one would do that with multiple physical clusters which is easier on day 1 but the operational complexity on day 2 or 3 can very quickly overwhelm a small team. Adding to its multi-tenant architecture, Pulsar provides a lot of features to restrict usage from different clients ranging from quotas, ttls, retention with simple configurations to more advanced features in namespace isolation, failure domains, anti affinity for namespaces, bookie groups and affinity between those etc.
In this talk we will first present the available options followed by our experience with them. As a bonus we will add a story where we migrated our whole cluster and the learnings in the journey.

Speaker

Shivji Kumar Jha

Senior Member of Technical Staff, Apache Pulsar Contributor, Nutanix

Shiv is a an architect at Nutanix and works for the beam team helping Nutanix customers minimise cloud costs and security risks for hybrid cloud usage. Shiv works on, among other things, all things pulsar at nutanix managing 4 clusters ((30 nodes)) of pulsar and the use cases around it. Shiv loves spending time on data stores (databases, streams, analytics etc) and has contributed to MySQL and pulsar codebases. Shiv is an avid reader (tech, fiction, economics etc) and is always looking at ways to simplify software architectures.

Sourabh Agrawal

Member of Technical Staff, Nutanix

Sourabh is an MTS-4 in Nutanix Data Platform team and manages 4 pulsar cluster and the use cases around it. Sourabh likes working on distributed systems and making the overall Software Development process optimised and efficient.