Technology Deep Dive

11:30 AM - 12:15 PM, PT , Wed Jun 16

Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar

The highest message delivery guarantee that Apache Pulsar provides is 'exactly-once', producing at a single partition via Idempotent Producer. Users are guaranteed that every message produced to a single partition via an Idempotent Producer will be persisted exactly once, without data loss. However, there is no 'atomicity' when a producer attempts to produce messages to multiple partitions. From the consumer side, acknowledgment is a best-effort operation, which results in message redelivery, hence consumer will receive duplicate messages. Pulsar only guarantees 'at-least-once' consumption for consumers. It creates inconvenience and brings in complexity when you use Pulsar to build mission-critical services (such as billing services).We introduce Transaction support in Pulsar 2.8.0 release, to simplify the process of building reliable and fault resilient services using Apache Pulsar and Pulsar Functions. It provides the capability to achieve end-to-end exactly-once for streaming jobs in other stream processing engines.This presentation deep dives into the details of Pulsar transaction and how Pulsar transaction is applied to Pulsar Functions and other processing engines to achieve transactional event streaming. We will cover how Pulsar transaction works and how Pulsar Functions offers transaction support using Pulsar transaction.

Speaker

Sijie Guo

Co-Founder and CEO, StreamNative

Sijie Guo is the co-founder and CEO of StreamNative, which provides a cloud-native event streaming platform powered by Apache Pulsar. Sijie has worked on messaging and streaming data technologies for more than a decade. Prior to StreamNative, Sijie cofounded Streamlio, a company focused on real-time solutions. At Twitter, Sijie was the tech lead for the messaging infrastructure group, where he co-created DistributedLog and Twitter EventBus. Prior to that, he worked on the push notification infrastructure at Yahoo!, where he was one of the original developers of BookKeeper and Pulsar. He is also the VP of Apache BookKeeper and PMC member of Apache Pulsar.

Addison Higham

Chief Architect, StreamNative

Addison Higham is the Chief Architect at StreamNative. He has been focusing on micro-services, distributed systems, big data, and cloud architecture with extensive experience in various big data Apache projects, such as Pulsar, Flink, and Spark.