After two and a half years of zero Kafka in my workflow, I’ve just gone and set up a new VM cluster on my Proxmox server.
After faffing for 90 minutes, went down the docker route instead – only so I can tear not down and rebuild faster as I go through and touch things either by accident or on purpose.
Anyway, in doing that I’ve come across dealing with the Kafka Listener part of the config. In Docker Compose these are exposed with KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS. There are a couple more I’ll try and explain as well.
Listeners in a nutshell
KAFKA_LISTENERS – internal to the Kafka Cluster, KAFKA_ADVERTISED_LIESTENERS – external to the Kafka Cluster.
When a client connects to any Kafka node (say for example a 3 node Kafka Cluster), it gets the info about the cluster, which which node it needs to use.
There are some settings I found which allow you to rename things, but I find distracting. KAFKA_LISTENER_SECURITY_PROTOCOL_MAP for instance, is a way for giving custom names to a protocol type, i.e. PLAINTEXT, SSL/TLS, SASL_PLAINTEXT and SASL_SSL. You can name these BOB, KEV, TOM and JON for example.
When it comes to using them as a KAFKA _LISTENER, you could set TOM as a synonym for PLAINTEXT (which is un-encrypted tcp), and use: PLAINTEXT://:9092 for example. Which still makes no sense typing that out.
Best thing I can do is link to https://rmoff.net/2018/08/02/kafka-listeners-explained/ that has a helpful page on listeners, though this bit in particular helped me out:
KAFKA_LISTENERS: LISTENER_BOB://kafka0:29092,LISTENER_FRED://localhost:9092
KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka0:29092,LISTENER_FRED://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_BOB