Single-Node Cluster Install - Docker

1. Install Docker

Install Docker for your appropriate Linux distribution. Installation guides for different platforms can found here.

Set Docker to start on system boot

# systemctl start docker
# systemctl enable docker

Create the Docker user group

sudo groupadd docker

[Optional] Remove sudo requirement

This prevents the need for sudo when invoking the docker command.

sudo usermod -aG docker $USER

2. Create docker-compose.yml

NB: If you are deploying to AWS, this file will require a configuration change!

Copy the following into a file named docker-compose.yml :

version: '3.2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092"
      - "9094:9094"
    environment:
      HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2"                    # Normal instances
#     HOSTNAME_COMMAND: "curl http://169.254.169.254/latest/meta-data/public-hostname"  # AWS Only
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_PROTOCOL_NAME: OUTSIDE
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_PROTOCOL_NAME: INSIDE
      KAFKA_PORT: 9092
      KAFKA_CREATE_TOPICS: jnpr.jti:1:1, 
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Alter CREATE_KAFKA_TOPICS in order to pre-allocate your topic names. Separate multiple topics with commas.
The general format is: <topic-name>:<number-of-partitions>:<number-of-replications>
If you are using AWS, comment the first HOSTNAME_COMMAND and uncomment the second one.

6. Deploy the Kafka stack

On the Master Node only, run the following command to bring the Kafka stack online.

docker stack deploy --compose-file docker-compose.yml kafka

This will deploy a Kafka broker to each node in the swarm (3 in total), and bring online ONE Zookeeper management container.
Additionally, any kafka topics specified in the docker-compose file will be initialized.

7. Verify status

Verify the status of the Kafka stack:

docker stack services kafka

The status for all containers should be shown. A successful launch should show 3/3 replicas for Kafka and 1/1 replicas for Zookeeper.

[Optional] Stop the Kafka stack

docker stack rm kafka

Sending Data to Kafka

The Kafka stack deployed above will initialize a single Kafka container on each node within the Swarm.
Hence the IP address of each node is the IP address of a Kafka broker within the Kafka cluster.
The Kafka brokers will listen for Consumer applications and Producers on port 9094.

Installing Kafka