Technology Deep Dive

2:00 PM - 2:35 PM, PT , Thur Jun 17

Reduce Redundant Producers from Partitioned Producer

Yahoo Japan Corporation has been using Apache Pulsar as a centralized messaging platform.
Recently, we have found an efficiency issue caused by an unspecified number of producers. Morespecifically, in some situations partitioned producer doesn't need to connect to all partitions.
A couple of cases as below:
1. different rate producers
- we should set the number of partitions according to the highest-rate producer
- however, this number is excessively large for low-rate producers
2. SinglePartition routing mode
- each producer uses only one partition
Currently we are planning to make partitioned producer able to connect to not only all partitions, but also part of partitions. We expect that this feature allows us to use system resources more efficiently.
In this session we will explain more detail and some experimental results.

Speaker

Yuri Mizushima

Software Engineer, Yahoo Japan Corporation

Yuri Mizushima is working at Yahoo Japan Corp as a software engineer. He has been joined messaging platform team since 2019.

View VideosView Slides

Sign up for the Data Streaming Newsletter

FAQS

CODE OF CONDUCT

ORGANIZED BY

© Copyright 2020 - 2025 StreamNative. All rights reserved

Privacy Policy | Terms & Conditions

Apache, Apache Pulsar, Apache Kafka, Apache BookKeeper, Apache Flink, and associated open source project names are trademarks of the Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event.