Draining Apache Kafka topic

Deleting Kafka topic can be a painful task, as of now the only way to do it, is to ssh into one of the Kafka brokers and execute CLI command. Like this:

./kafka-topics --zookeeper localhost:2181 --delete --topic mykafkatopic

Even this approach might not work if topic deletion is not enabled on your cluster. Solution to that is to use topic configuration retention.ms. From documentation:

This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the “delete” retention policy. This represents an SLA on how soon consumers must read their data.

retention.ms is simply how long data will live in a topic before it gets discarded.
Using this config we can quickly drain the whole topic in a moment. Like so:

./kafka-topics --zookeeper localhost:2181 --alter --topic mykafkatopic --config retention.ms=1000

This sets retention.ms to 1000ms(keep data in a topic for 1 second), after a short wait, your topic will be completely empty.
WARNING!! Don’t forget to set back this configuration to it’s original value.

Also, the privilege of having access to the Kafka cluster is limited to certain people in a team and you may be not that person.
A lot of times during development you will need to delete or maybe drain topic so that you can pump there a new data.
In case you don’t have access to the Kafka cluster, but you have some UI tool, like Confluent Control Center or Landoop Lenses you can set retention.ms in configurations of topic using those UI tools.

Good luck!

Leave a Reply

Your email address will not be published. Required fields are marked *