Technology Deep Dive

1:30pm-2:10pm UTC+8 , November 28

Transactional Event Streaming with Apache Pulsar

The highest message delivery guarantee that Apache Pulsar provides is ‘exactly-once’, producing at a single partition via Idempotent Producer. Users are guaranteed that every message produced to a single partition via an Idempotent Producer will be persisted exactly once, without data loss. However, there is no ‘atomicity’ when a producer attempts to produce messages to multiple partitions. From the consumer side, acknowledgment is a best-effort operation, which results in message redelivery, hence consumer will receive duplicate messages. Pulsar only guarantees ‘at-least-once’ consumption for consumers. It creates inconvenience and brings in complexity when you use Pulsar to build mission-critical services (such as billing services).
Pulsar introduces transaction support in 2.7.0 version, to simplify the process of building reliable and fault resilient services using Apache Pulsar and Pulsar Functions. It only provides the capability to achieve end-to-end exactly-once for streaming jobs in other stream processing engines.
This presentation deep dives into the details of Pulsar transaction and how Pulsar transaction is applied to Pulsar Functions and other processing engines to achieve transactional event streaming. We will cover how Pulsar transaction works and how Pulsar Functions offers transaction support using Pulsar transaction.

Speaker

Bo Cong

Bo Cong is a software engineer at StreamNative. Prior to StreamNative, he worked at Zhaopin.com, responsible for the development of Zhaopin Event center base on Apache Pulsar.