Use Case

1:10 PM - 1:50 PM PDT , June 18

Large scale log pipeline using Apache Pulsar

Yahoo Japan Corporation has been using Apache Pulsar as a centralized pub-sub messaging platform for more than 3 years.
We adopted Pulsar because of its great performance, scalability and multi-tenancy capability.
It plays an important role to provide our 100+ services in various areas such as e-commerce media, advertising and more.

Recently, we addressed to solve our new use case: A large scale log pipeline.
In our production environment, we are starting to run a lot of our services on container environments.
Our goal is to send all logs and metrics from application containers to various monitoring or analyzing platforms.

We expect Pulsar to keep its performance even in tremendously high traffic volume situations (i.e. in tens of Gbps).
In this presentation, we will talk about our architecture design, producer/consumer side implementation and the result of performance test.
We will also share our experience and knowledge from our production environment operations for more than 3 years.

Takeaway:
- Practical use case of Apache Pulsar on production
- Knowledge of operating Apache Pulsar for large scale data stream

Speaker

Nozomi Kurihara

Yahoo Japan Corporation, Manager

Nozomi Kurihara is the Manager of the Messaging Platform team in Yahoo Japan Corporation responsible for creating a centralized pub-sub messaging platform which deals with a vast number of service/application traffics. The messaging platform is based on Apache Pulsar, a recently open-sourced pub-sub messaging platform. Nozomi is also a committer of the Apache Pulsar project.