Keynote

4:30 PM - 5:00 PM PST , January 15

Building an Asynchronous Application Framework with Python and Pulsar

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Speaker

Zac Bentley

Lead Site Reliability Engineer, Klaviyo

I only have a problem with one word in that sentence, and that word is 'just'. Zac has been working on distributed systems, asynchronous messaging, and performance engineering for almost a decade. At work for Klaviyo, he works on infrastructure orchestration, message queueing, systems architecture, and a thousand other things. Outside of work, he is a poet, musician, and mycologist. He believes that Ratatouille was right about software as well as food, that “anyone can code”, and that no matter what you work on or how it’s used, you should always aim to make things easier for the next person who has to touch it.