Monitoring Extreme-Scale Apache Kafka Using eBPF at New Relic
Streaming Audio: Apache Kafka® & Real-Time Data
Chapters
0:00
Who watches the watchers?
0:45
Intro
2:30
Monitoring 125 petabytes of data
3:59
Challenges
8:29
Customer data in real time
11:35
Monitoring data at scale
13:32
eBPF and Kafka
17:21
Pixie opens a new world of possibilities in Kafka monitoring
24:12
eBPF and Pixie
25:59
Use cases
27:31
What is Pixie?
29:41
Kafka Summit keynote
34:32
Tips to get started with eBPF
35:55
It's a wrap!
Streaming Audio: Apache Kafka® & Real-Time Data
Monitoring Extreme-Scale Apache Kafka Using eBPF at New Relic
Apr 13, 2022 Season 1 Episode 209
Confluent, original creators of Apache Kafka®

New Relic runs one of the larger Apache Kafka® installations in the world, ingesting circa 125 petabytes a month, or approximately three billion data points per minute. Anton Rodriguez is the architect of the system, responsible for hundreds of clusters and thousands of clients, some of them implemented in non-standard technologies. In addition to the large volume of servers, he works with many teams, which must all work together when issues arise.

Monitoring New Relic's large Kafka installation is critical and of course challenging, even for a company that itself specializes in monitoring. Specific obstacles include determining when rebalances are happening, identifying particularly old consumers, measuring consumer lag, and finding a way to observe all producing and consuming applications.

One way that New Relic has improved the monitoring of its architecture is by directly consuming metrics from the Linux kernel using its new eBPF technology, which lets programs run inside the kernel without changing source code or adding additional modules (the open-source tool Pixie enables access to eBPF in a Kafka context). eBPF is very low impact, so doesn’t affect services, and it allows New Relic to see what’s happening at the network level—and to take action as necessary.

EPISODE LINKS