Ecosystem

3:20 PM - 3:50 PM PST

Simplify Pulsar Functions Development with SQL

Pulsar Functions is a succinct framework provided by Apache Pulsar to conduct real-time data processing. Its use cases include ETL pipeline, event-driven applications, and simple data analytics. While Pulsar Functions already provides an extremely simple programming interface, we want to further lower the barrier for users to access real-time data. Since SQL is one of the universal languages in the technology world and well accepted by the vast majority of data engineers, we decided to add a SQL expressing layer on top of Pulsar Functions runtime. In this talk, we will discuss the architecture and implementation of this new service. We will see how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment. We will also walk through use cases like filtering, routing, and projecting messages as well as integrating with the Pulsar IO Connectors framework.

Speaker

Neng Lu

Platform Engineering Lead, Compute, StreamNative

Neng Lu has rich experience in stream computing technologies. He is currently leading StreamNative's real-time computing product by integrating Apache Pulsar with big data ecosystems on top of Kubernetes. He is also an active Apache Pulsar contributor for Pulsar Function. Prior to StreamNative, Neng worked at Twitter focusing on the development of real-time computing engine -- Apache Incubator Heron. He is also an Apache Incubator Heron Committer and PPMC member. Neng lives in San Francisco, California.