If Streaming Is the Answer, Why Are We Still Doing Batch?
Streaming Audio: Apache Kafka® & Real-Time Data
Chapters
0:00
Intro
2:58
Is the Lambda Architecture here to stay?
6:27
What is preventing streaming adoption today?
10:00
Is streaming a semantic model?
20:53
Should we push for stream processing?
26:15
When should we use streaming vs. batch processing?
37:10
What is the future of stream processing?
41:48
It's a wrap!
Streaming Audio: Apache Kafka® & Real-Time Data
If Streaming Is the Answer, Why Are We Still Doing Batch?
Nov 09, 2022 Season 1 Episode 242
Confluent, founded by the original creators of Apache Kafka®

Is real-time data streaming the future, or will batch processing always be with us? Interest in streaming data architecture is booming, but just as many teams are still happily batching away. Batch processing is still simpler to implement than stream processing, and successfully moving from batch to streaming requires a significant change to a team’s habits and processes, as well as a meaningful upfront investment. Some are even running dbt in micro batches to simulate an effect similar to streaming, without having to make the full transition. Will streaming ever fully take over?

In this episode, Kris talks to a panel of industry experts with decades of experience building and implementing data systems. They discuss the state of streaming adoption today, if streaming will ever fully replace batch, and whether it even could (or should). Is micro batching the natural stepping stone between batch and streaming? Will there ever be a unified understanding on how data should be processed over time? Is the lack of agreement on best practices for data streaming an insurmountable obstacle to widespread adoption? What exactly is holding teams back from fully adopting a streaming model?

Recorded live at Current 2022: The Next Generation of Kafka Summit, the panel includes Adi Polak (Vice President of Developer Experience, Treeverse), Amy Chen (Partner Engineering Manager, dbt Labs), Eric Sammer (CEO, Decodable), and Tyler Akidau (Principal Software Engineer, Snowflake).

EPISODE LINKS