Streaming in the Delta Age
Our journey began with the ambitious task of moving microservices from Akka Streams to Spark Streaming. But the real discovery? Databricks Delta Lake’s power as an alternative to Kafka. Delta Lake shone in highly scalable processing, data storage, access, and querying, as well as handling Change Data Captures (CDCs) at a huge scale.
In this presentation, we will start from our migration journey and deep-dive into the performance challenges faced and why Delta Lake became our choice over Kafka.
We will cover:
- Migration Challenges: Practical hurdles and solutions in transitioning from Akka Streams to Spark Streaming.
- Spark Stateful Streaming: When and how to leverage it, and its limitations.
- Delta Lake’s Edge: Advantages of Delta Lake over Kafka (especially in CDC handling, data storage, access, and querying), and of using it in conjunction with Spark.
- Best Practices: Tips and lessons from our migration journey; for instance, how to handle multi-tenant data in a uniform manner.
- Streaming’s Future: Delta Lake’s potential role in the evolving streaming landscape.
Join us in this session to learn how Delta Lake could reshape the streaming landscape.
Book Now