Kafka Topic Creation Fails With TopicExistsException After Docker‑compose Up

Q: Authorization side‑effects

If the broker is started with authorizer.class.name set (e.g., kafka.security.authorizer.AclAuthorizer) but the client lacks ACLs for the new topic, the AdminClient may receive an AuthorizationException instead of TopicExistsException. The evidence package includes a scenario where mixed KRaft/Zookeeper configurations caused this symptom. The presence of an AuthorizationException in logs should be treated as a separate problem, but it often appears together when stale metadata prevents the broker from evaluating ACLs correctly.

Q: Investigation and Debugging Steps 1. Inspect broker and Zookeeper logs

docker-compose logs kafka docker-compose logs zookeeper Typical relevant lines:

Q: 2. Verify Zookeeper znodes

docker exec -it zookeeper zkCli.sh -server localhost:2181 ls /brokers/topics Expected output when the topic truly exists:

Q: 3. Check broker log directory

docker exec -it kafka ls /tmp/kafka-logs Missing inference-logs-0 confirms the mismatch.

Q: 4. Confirm ACLs (if security is enabled)

docker exec -it kafka kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \ --list --topic inference-logs If the ACL list is empty and authorizer.class.name is set, the client will receive AuthorizationException on creation attempts.

Q: 5. Reproduce the race (optional)

Run two simultaneous AdminClient.createTopics calls from separate shells to see one succeed and the other fail with TopicExistsException. This helps differentiate a race from stale metadata.

Problem: Kafka topic creation fails with `TopicExistsException` after `docker‑compose up`

During integration testing of an AI model inference pipeline, the test harness attempts to create a Kafka topic named inference-logs via the AdminClient. The call consistently returns an error similar to:

org.apache.kafka.common.errors.TopicExistsException: Topic ‘inference-logs’ already exists.

Despite the exception, the topic is not visible in kafka-topics.sh --list, and subsequent producer/consumer attempts fail with UnknownTopicOrPartitionException. The failure blocks the end‑to‑end test run.

Root Cause Analysis

Stale Zookeeper metadata vs. broker log directories

In a Docker Compose sandbox the broker and Zookeeper containers share host‑mounted volumes for persistence (/var/lib/kafka/data and /var/lib/zookeeper). When docker‑compose down is executed without the --volumes flag, the Zookeeper data directory is retained. Upon the next docker‑compose up, the broker reads the existing Zookeeper znodes that claim the topic exists, but the broker’s own log directory (/tmp/kafka-logs in the official Confluent image) is empty because the previous log files were removed or never persisted.

This mismatch triggers the TopicExistsException path in the AdminClient’s CreateTopics operation, as documented in the AdminClient API. The broker reports the topic as existing (metadata in Zookeeper) while the filesystem does not contain the partition logs, leading to the “ghost” topic state observed in the real incident described in the evidence package.

Race conditions in multi‑broker compose setups

When multiple broker containers start simultaneously, each may issue a create‑topic request for the same name (e.g., via an application’s startup hook). The first request succeeds, the second receives TopicExistsException. This is a known race condition documented in GitHub issue confluentinc/cp-kafka#678. In a single‑broker sandbox the race is less common, but the stale‑metadata scenario can masquerade as a race.

Authorization side‑effects

If the broker is started with authorizer.class.name set (e.g., kafka.security.authorizer.AclAuthorizer) but the client lacks ACLs for the new topic, the AdminClient may receive an AuthorizationException instead of TopicExistsException. The evidence package includes a scenario where mixed KRaft/Zookeeper configurations caused this symptom. The presence of an AuthorizationException in logs should be treated as a separate problem, but it often appears together when stale metadata prevents the broker from evaluating ACLs correctly.

Investigation and Debugging Steps

1. Inspect broker and Zookeeper logs

docker-compose logs kafka
docker-compose logs zookeeper

Typical relevant lines:

[2024-09-14 10:12:03,456] INFO [AdminClient clientId=adminclient-1] Error while creating topic inference-logs: TopicExistsException

[2024-09-14 10:12:02,987] INFO [Controller 0] Created topic inference-logs with 1 partitions.

If the controller log shows a “Created topic” line but the broker later logs “TopicExistsException”, the metadata is out of sync.

2. Verify Zookeeper znodes

docker exec -it zookeeper zkCli.sh -server localhost:2181 ls /brokers/topics

Expected output when the topic truly exists:

[inference-logs]

If the znode is present but the broker’s log directory (/tmp/kafka-logs) lacks a folder for inference-logs-0, you have the stale‑metadata condition.

3. Check broker log directory

docker exec -it kafka ls /tmp/kafka-logs

Missing inference-logs-0 confirms the mismatch.

4. Confirm ACLs (if security is enabled)

docker exec -it kafka kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 \
  --list --topic inference-logs

If the ACL list is empty and authorizer.class.name is set, the client will receive AuthorizationException on creation attempts.

5. Reproduce the race (optional)

Run two simultaneous AdminClient.createTopics calls from separate shells to see one succeed and the other fail with TopicExistsException. This helps differentiate a race from stale metadata.

Resolution

Option A – Clean Zookeeper and broker state before restart

For a development sandbox, the safest approach is to wipe persisted volumes so that Zookeeper and the broker start from a clean slate.

# Stop the compose stack
docker-compose down -v   # -v removes named volumes

# Remove any bind‑mounted host directories (if used)
rm -rf /path/to/host/kafka-data/*
rm -rf /path/to/host/zookeeper-data/*

# Restart
docker-compose up -d

After a clean start, the AdminClient can create inference-logs without encountering TopicExistsException.

Option B – Manually delete the stale Zookeeper znode

If wiping volumes is undesirable (e.g., you need other topics), delete only the offending znode.

docker exec -it zookeeper zkCli.sh -server localhost:2181 deleteall /brokers/topics/inference-logs

Then restart the broker to force it to resync:

docker-compose restart kafka

Option C – Align broker log directory with Zookeeper

Configure the broker to store logs in a persistent volume that survives container restarts. In docker-compose.yml:

services:
  kafka:
    image: confluentinc/cp-kafka:7.5.0
    environment:
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka-data:/var/lib/kafka/data
volumes:
  kafka-data:

Now the log directory will be recreated automatically when the topic metadata reappears, preventing the mismatch.

Option D – Guard against race conditions

Wrap topic creation in idempotent code that tolerates TopicExistsException:

AdminClient admin = AdminClient.create(props);
NewTopic topic = new NewTopic("inference-logs", 1, (short)1);
CreateTopicsResult result = admin.createTopics(Collections.singleton(topic));
try {
    result.all().get();
    System.out.println("Topic created");
} catch (ExecutionException e) {
    if (e.getCause() instanceof TopicExistsException) {
        System.out.println("Topic already exists – proceeding");
    } else {
        throw e;
    }
}

This pattern follows the guidance in the AdminClient documentation.

Validation

Run the topic‑creation code again. It should log “Topic created” or “Topic already exists – proceeding”.
List topics from the broker:

docker exec -it kafka kafka-topics.sh --bootstrap-server localhost:9092 --list

Output must include inference-logs.

Produce a test message:

docker exec -it kafka kafka-console-producer.sh --broker-list localhost:9092 \
  --topic inference-logs < /dev/null

If the producer exits without error, the topic is functional.

Consume the message to confirm end‑to‑end flow:

docker exec -it kafka kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic inference-logs --from-beginning --max-messages 1

Operational Experience & Lessons Learned

Volume hygiene matters. In Docker Compose, docker-compose down does not delete named volumes. Stale Zookeeper data is a frequent source of “ghost” topics.
Bind mounts can hide the problem. A bind‑mounted /tmp/kafka-logs directory that is cleared between runs leaves Zookeeper thinking the topic exists, reproducing the exact failure described in the community thread “Topic creation fails with TopicExistsException after restart”.
Idempotent admin code reduces friction. Treating TopicExistsException as a non‑fatal outcome prevents test pipelines from failing when a previous run left the topic behind.
ACL misconfiguration masks the real error. When authorizer.class.name is set, an AuthorizationException may be thrown before the broker even checks Zookeeper metadata. Ensure the client principal has Write rights on the target topic.
Multi‑broker race conditions are real. In a compose file with replicas: 3, two brokers may race to create the same topic. Serializing creation (e.g., via a single “bootstrap” service) eliminates the race.

Best Practices & Prevention

Practice	Why it helps
Persist broker logs in a Docker volume	Ensures filesystem state matches Zookeeper metadata across restarts.
Include `docker-compose down -v` in CI cleanup	Guarantees a clean Zookeeper state for each test run.
Make AdminClient creation idempotent	Handles both genuine existence and race‑condition failures gracefully.
Enable `auto.create.topics.enable=false`	Prevents accidental topic auto‑creation that could hide permission problems.
Audit ACLs for test principals	Prevents unexpected `AuthorizationException` when security is enabled.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the topic appear in Zookeeper but not in the broker’s log directory?
Because the Zookeeper data volume was retained while the broker’s log directory was cleared. Zookeeper still holds the topic’s metadata, causing the broker to think the topic exists.
Can I keep the Zookeeper volume and still avoid the error?
Yes. Delete the specific znode (/brokers/topics/inference-logs) or run kafka-topics.sh --delete before recreating the topic.
What if I see AuthorizationException instead of TopicExistsException?
Verify that the client principal has Write ACLs on the target topic and that authorizer.class.name is correctly configured. Missing ACLs can surface as authorization errors even when the topic metadata is stale.
Is disabling auto.create.topics.enable safe in development?
Yes, it forces explicit topic creation via AdminClient, making failures easier to detect and preventing accidental topic creation that masks permission issues.
How do I avoid the race condition in a multi‑broker compose setup?
Centralize topic creation in a single “bootstrap” container that runs before other services, or use a coordination mechanism (e.g., a lock in Zookeeper) to ensure only one AdminClient request is issued.

Problem: Kafka topic creation fails with TopicExistsException after docker‑compose up