Technology Deep Dive

1:20 PM - 1:50 PM PST

Understanding Broker Rebalancing

Pulsar is a horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, in order to ensure full utilization of the broker layer. You can use multiple settings and tools to control the traffic distribution which requires a bit of context to understand how the traffic is managed in Pulsar. In this talk, we will walk you through the load balancing capabilities of Apache Pulsar, and highlight some of the control mechanisms available to control the distribution of load across the Pulsar brokers. Finally, we will discuss the various loading shedding strategies that are available. At the end of the talk, you will have a better understanding of how Pulsar's broker level auto-balancing works, and how to properly configure it to meet your workload demands.

Speaker

Heesung Sohn

Sr. Platform Engineer, StreamNative

Heesung Sohn joined StreamNative as Sr. platform engineer, helping Apache Pulsar development. He experienced big data technologies at Tapjoy and Amazon.com. At AWS, he built Aurora Mysql Serverless V2 and Amazon Forecast.