Keynote

11:05 AM - 11:35 AM PST , January 15

Pulsar @ Scale. 200M RPM and Thousand of Instances

Our services team creates, builds, and maintains the as a service offering for base platform services within our organization. Several thousand applications use these custom services daily generating more than 700 million requests per minute. One of these services was our publish / subscriber offering, BQ with custom SDK and custom metrics based on Apache Pulsar. BQ is the core communication service within our organization, having more 200M RPM. All the core processes of the organization depend on this service for operation: the CDC of any of our RDBMS or NoSQL offering, all the eventing efforts of the organization, async communication between apps, notification systems, etc. The backend of the solution was Apache Pulsar running on EC2 on AWS and on top of that we built several components as wrappers of the actual backend, creating our own SDKs and abstractions and in many ways extending the features provided by Pulsar. We had a multi-cluster setup 100% on AWS, with custom Pulsar Docker images running on large ASG setups, along with our own wrapping and admin APIs and DBs. All of this in turn transformed the solution into a volatile solution.

Speaker

Ignacio Alvarez

Software Development Senior Expert, Mercado Libre; CTO, TuPase

Java EE Architect, and Teacher. Software Development Engineer. Open Source enthusiast. Enterprise Integration fan. Object Oriented Programming / Data Structures & Algorithms professor.