Technology Deep Dive

1:30 PM CST, UTC+8 , November 19

Apache Pulsar in Volcano Engine E-MapReduce: Integration and Scenarios

Volcano Engine is an enterprise service brand of ByteDance. E-MapReduce on Volcano Engine is a stateless, cloud-native, and open-source big data platform, providing big data services on the cloud. In recent years, Apache Pulsar plays an increasingly important role in messaging and streaming. Its cloud-native nature brings tons of benefits in big data scenarios. Therefore, introducing Pulsar to the E-MapReduce ecosystem comes with great value.
In this session, we will first provide an overview of Volcano Engine E-MapReduce and its architecture, as well as the motivation for integrating Pulsar as a new cluster type into the Volcano Engine E-MapReduce ecosystem. Then, we will briefly illustrate how we integrate Pulsar into Volcano Engine E-MapReduce, providing features of open source Pulsar on the general control plane of E-MapReduce, and smooth interaction with other cluster types in E-MapReduce such as Hadoop and Flink. We will also introduce some scenarios of Pulsar in Volcano E-MapReduce, especially in the area of real-time data warehouse and merging batch and stream processing. We will discuss several typical real-time scenarios and respective problems users may encounter, and provide possible solutions powered by Pulsar with related services in Volcano Engine E-MapReduce. We will conclude our session with future plans for Pulsar in Volcano Engine E-MapReduce, including more integrated components and enhanced end-to-end user experience.

Speaker

Xin Liang

Senior Engineer at ByteDance Data Platform

Xin Liang works as a senior engineer at ByteDance Data Platform, focusing on the design and implementation of several modules in Volcano Engine E-MapReduce, including taking charge of introducing Apache Pulsar to E-MapReduce in Volcano Engine. In recent years, he is mainly interested in big data platforms and realtime data warehouses. Xin received his bachelor's and PhD degrees at Peking University. Before joining ByteDance, Xin worked at MicroStrategy, Meituan, 360, and Alibaba Cloud.