Apache Kafka 3.2 - New Features & Improvements
Streaming Audio: Apache Kafka® & Real-Time Data
Chapters
0:00
Intro
0:35
KIP-704: Send a hint to the partition leader to recover the partition
1:00
KIP-764: Configurable backlog size for creating Acceptor
1:15
KIP-784: Add top-level error code field to DescribeLogDirsResponse
1:33
KIP-788: Allow configuring num.network.threads per listener
2:00
KIP-798: Add possibility to write kafka headers in Kafka Console Producer; KIP-810:Allow producing records with null values in Kafka Console Producer
2:25
KIP-800: Add reason to JoinGroupRequest and LeaveGroupRequest
2:48
KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata
3:14
KIP-814: Static membership protocol should let the leader skip assignment
3:42
KIP-708: Rack awareness for Kafka Streams
4:02
KIP-805: Add range and scan query over kv-store in IQv2
4:15
KIP-806: Add session and window query over kv-store in IQv2
4:37
KIP-791: Add Record Metadata to StateStoreContext
4:52
KIP-769: Connect APIs to list all connector plugins and retrieve their configuration definitions
5:18
KIP-808: Add support for different unix precisions in TimestampConverter SMT
5:34
KIP-779: Allow Source Tasks to Handle Producer Exceptions
5:59
That's a wrap
Streaming Audio: Apache Kafka® & Real-Time Data
Apache Kafka 3.2 - New Features & Improvements
May 17, 2022 Season 1 Episode 215
Confluent, original creators of Apache Kafka®

Apache Kafka® 3.2 delivers new  KIPs in three different areas of the Kafka ecosystem: Kafka Core, Kafka Streams, and Kafka Connect. On behalf of the Kafka community, Danica Fine (Senior Developer Advocate, Confluent), shares release highlights.

More than half of the KIPs in the new release concern Kafka Core. KIP-704 addresses unclean leader elections by allowing for further communication between the controller and the brokers. KIP-764 takes on the problem of a large number of client connections in a short period of time during preferred leader election by adding the configuration `socket.listen.backlog.size`. KIP-784 adds an error code field to the response of the `DescribeLogDirs` API, and KIP-788 improves network traffic by allowing you to set the pool size of network threads individually per listener on Kafka brokers. Finally, in accordance with the imminent KRaft protocol, KIP-801 introduces a built-in `StandardAuthorizer` that doesn't depend on ZooKeeper. 

There are five KIPs related to Kafka Streams in the AK 3.2 release. KIP-708 brings rack-aware standby assignment by tag, which improves fault tolerance. 

Then there are three projects related to Interactive Queries v2: KIP-796 specifies an improved interface for Interactive Queries; KIP-805 allows state to be queried over a specific range; and KIP-806 adds two implementations of the Query interface, `WindowKeyQuery` and `WindowRangeQuery`.

The final Kafka Streams project, KIP-791, enhances `StateStoreContext` with `recordMetadata`,which may be accessed from state stores.

Additionally, this Kafka release introduces Kafka Connect-related improvements, including KIP-769, which extends the `/connect-plugins` API, letting you list all available plugins, and not just connectors as before.  KIP-779 lets `SourceTasks` handle producer exceptions according to `error.tolerance`, rather than instantly killing the entire connector by default. Finally, KIP-808 lets you specify precisions with respect to TimestampConverter single message transforms. 

Tune in to learn more about the Apache Kafka 3.2 release!

EPISODE LINKS