[SOLVED] Create Kafka topic in Docker works from command line but not from Shell script file

Issue

I'm trying to create a topic in a Kafka container running in Docker using a shell script file.

docker-compose.yml

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    ports:
      - "22181:2181"

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    depends_on:
      - zookeeper

    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

init.sh

#!/bin/zsh
docker compose up -d
docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test

When I run docker-compose the containers start-up fine, but the topic creation fails.

[2021-11-04 17:35:07,957] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

However, when I run the same commands in the command line, without any changes, it works fine and creates the topic.

Created topic Test.

I'm at a loss and reading previous questions on similar issues here hasn't help: mostly they were due to typos or adding quotation marks where they are not needed. I have tried changing the #! to bash, sh, and usr/bin/env (amongst others) to no avail.

Any help appreciated.

Docker version: version 20.10.8 Running on a Mac (Intel)

Solution

As suggested by the error message, the broker is not available (in your case it hasn't started yet). You could wait for the broker to complete start up process (by looking at the logs with docker logs -f kafka or by using a sleep in your script) and then run your command to create the topic:

docker exec kafka kafka-topics --create --bootstrap-server localhost:29092 --partitions 1 --replication-factor 1 --topic Test

Alternatively, instead of having to figure out if your broker has completed the initialization process, you could use something like this official Confluent example. In this example, an utility is used to wait for a Kafka cluster to be ready.

Answered By - Luigi Cerone

Answer Checked By - David Marino (WPSolving Volunteer)