Technology Deep Dive

12:20 PM - 1:00 PM PDT , June 18

Pulsar Architectural Patterns for CI/CD Automation and Self-Service

We examine real-world architectural patterns involving Apache Pulsar to automate the creation of function and pub/sub flows for improved operational scalability and ease of management. We’ll cover CI/CD automation patterns and reveal our innovative approach of leveraging streaming data to create a self-service platform that automates the provisioning of new users. We will also demonstrate the innovative approach of creating function flows through patterns and configuration, enabling non-developer users to create entire function flows simply by changing configurations. These patterns enable us to drive the automation of managing Pulsar to a whole new level. We also cover CI/CD for on-prem, GCP, and AWS users.

This is Part 2 of this presentation: https://www.youtube.com/watch?v=pmaCG1SHAW8&feature=youtu.be
In summary, we will cover:
CI/CD for on-prem, GCP, and AWS users
Automated creation of function flows by configuration
Automated provisioning of pub/sub users and topics
Architectural patterns and best practices that enable automation
Overstock has leveraged Pulsar as the backbone of a self-service data fabric, a unified data platform to enable users to publish and consume data across the company and integrate with other services. We utilized Pulsar to solve a data governance problem, and Pulsar has performed marvelously. To support our real-world production use cases, we have developed message flows, integrations, and architectural patterns to solve common use cases, maximize value, simplify ease-of-use, automate management, and unify company data and services around this new platform.

Speaker

Devin Bost

Senior Data Engineer @ Overstock.com

With over 10 years of experience in the software industry, Devin has developed software in over 15 different languages. Between his experience of performing data migrations, applying vector calculus for ML, and building enterprise applications, he learned the critical role of data in opening doors of insight into novel market opportunities. He also observed many companies architect their software with the mindset of “we’ll figure out the data later,” only to code themselves into life-threatening dead-ends. These observations fueled his interest in context-rich stream-based architectures like Kappa that thrive on live data capture and real-time analysis for instant value-creation.