文章目录一、单节点Docker 运行 Kafka 4.0docker run 单机快速测试用docker‑compose 完整模板单机KRaft4.0.0进入容器测试命令SpringBoot application.yml 连接配置重点注意事项搭配kafka‑ui可视化扩展compose片段二、Kafka4.0.x KRaft 三节点高可用集群Docker Composedocker‑compose.yml启动步骤SpringBoot 连接配置集群测试命令关键特性说明高频踩坑停止清理一、单节点Docker 运行 Kafka 4.0Kafka 4.0彻底删除 Zookeeper 模式只支持 KRaft不再支持 zk。官方镜像apache/kafka没有 zk 相关任何配置。docker run 单机快速测试用dockerrun-d\--namekafka4\--restartalways\-p9092:9092\-p9093:9093\-v/data/kafka4/kraft-data:/tmp/kraft-combined-logs\-eKAFKA_NODE_ID1\-eKAFKA_PROCESS_ROLESbroker,controller\-eKAFKA_LISTENERSPLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093\-eKAFKA_ADVERTISED_LISTENERSPLAINTEXT://192.168.1.100:9092\-eKAFKA_CONTROLLER_LISTENER_NAMESCONTROLLER\-eKAFKA_CONTROLLER_QUORUM_VOTERS1kafka4:9093\-eKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR1\apache/kafka:4.0.0⚠️ 修改192.168.1.100为你的宿主机真实IPSpringBoot业务连接用这个IP:9092。docker‑compose 完整模板单机KRaft4.0.0version:3.8services:kafka:image:apache/kafka:4.0.0container_name:kafka4restart:alwaysports:-9092:9092-9093:9093volumes:-./kraft-data:/tmp/kraft-combined-logsenvironment:KAFKA_NODE_ID:1KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://192.168.1.100:9092KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka:9093KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:1KAFKA_LOG_RETENTION_HOURS:72KAFKA_LOG_DIRS:/tmp/kraft-combined-logs启动docker-composeup-d进入容器测试命令# 进入容器dockerexec-itkafka4bash# 创建topickafka-topics.sh--create--topicdemo --bootstrap-server127.0.0.1:9092--partitions2--replication-factor1# 列表kafka-topics.sh--list--bootstrap-server127.0.0.1:9092# 生产者kafka-console-producer.sh--topicdemo --bootstrap-server127.0.0.1:9092# 消费者kafka-console-consumer.sh--topicdemo --bootstrap-server127.0.0.1:9092 --from-beginning# 查看kafka版本kafka-topics.sh--versionSpringBoot application.yml 连接配置spring:kafka:bootstrap-servers:192.168.1.100:9092producer:key-serializer:org.apache.kafka.common.serialization.StringSerializervalue-serializer:org.apache.kafka.common.serialization.StringSerializerconsumer:key-deserializer:org.apache.kafka.common.serialization.StringDeserializervalue-deserializer:org.apache.kafka.common.serialization.StringDeserializer重点注意事项Kafka4.x 完全没有Zookeeper不要再写任何zk连接参数。镜像apache/kafka:4.0.0不要用confluentinc/cp‑kafkacp还没有4.0大版本。数据目录权限挂载宿主机目录报权限拒绝chmod-R777./kraft-data升级提醒旧的kafka3.x zookeeper集群不能直接升级到4.0。3.x zk集群 → 先升级到3.5/3.6执行迁移工具切换KRaft迁移完成后再升级到4.0。不能直接把zk的数据目录直接给kafka4使用元数据格式完全不一样。单机生产注意单机同时做brokercontrollerreplication‑factor1不能高可用仅测试/小规模使用真正生产集群至少3节点controller。搭配kafka‑ui可视化扩展compose片段kafka-ui:image:provectuslabs/kafka-ui:latestcontainer_name:kafka-uirestart:alwaysports:-8081:8080environment:KAFKA_CLUSTERS_0_NAME:local‑kraft‑40KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS:kafka:9092访问http://宿主机IP:8081二、Kafka4.0.x KRaft 三节点高可用集群Docker ComposeKafka4.0彻底移除Zookeeper仅KRaft模式3个节点全部同时充当broker controller控制器仲裁quorum3挂掉任意1台集群依然正常工作满足生产高可用基础条件。注意下面是单机docker模拟3节点集群同一台宿主机用于开发测试真正生产要部署在三台不同物理服务器。docker‑compose.ymlversion:3.8services:kafka‑1:image:apache/kafka:4.0.0container_name:kafka‑1restart:alwaysports:-9092:9092# broker端口对外-9093:9093# controller内部仲裁端口volumes:-./data‑1:/tmp/kraft‑combined‑logsenvironment:KAFKA_NODE_ID:1KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093# 修改为你的宿主机IP外部业务程序连接KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://192.168.1.100:9092KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka‑1:9093,2kafka‑2:9093,3kafka‑3:9093KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:3KAFKA_DEFAULT_REPLICATION_FACTOR:3KAFKA_LOG_DIRS:/tmp/kraft‑combined‑logskafka‑2:image:apache/kafka:4.0.0container_name:kafka‑2restart:alwaysports:-9094:9092-9095:9093volumes:-./data‑2:/tmp/kraft‑combined‑logsenvironment:KAFKA_NODE_ID:2KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://192.168.1.100:9094KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka‑1:9093,2kafka‑2:9093,3kafka‑3:9093KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:3KAFKA_DEFAULT_REPLICATION_FACTOR:3KAFKA_LOG_DIRS:/tmp/kraft‑combined‑logskafka‑3:image:apache/kafka:4.0.0container_name:kafka‑3restart:alwaysports:-9096:9092-9097:9093volumes:-./data‑3:/tmp/kraft‑combined‑logsenvironment:KAFKA_NODE_ID:3KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://192.168.1.100:9096KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka‑1:9093,2kafka‑2:9093,3kafka‑3:9093KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:3KAFKA_DEFAULT_REPLICATION_FACTOR:3KAFKA_LOG_DIRS:/tmp/kraft‑combined‑logskafka-ui:image:provectuslabs/kafka-ui:latestcontainer_name:kafka-uirestart:alwaysports:-8081:8080environment:KAFKA_CLUSTERS_0_NAME:kafka4‑3node‑kraftKAFKA_CLUSTERS_0_BOOTSTRAPSERVERS:kafka‑1:9092,kafka‑2:9092,kafka‑3:9092启动步骤# 创建数据目录mkdir-pdata‑1 data‑2 data‑3# 解决容器挂载目录权限问题chmod-R777data‑1 data‑2 data‑3# 启动集群docker-composeup-d# 查看容器状态docker-composeps访问UIhttp://宿主机IP:8081SpringBoot 连接配置spring:kafka:bootstrap-servers:192.168.1.100:9092,192.168.1.100:9094,192.168.1.100:9096producer:key-serializer:org.apache.kafka.common.serialization.StringSerializervalue-serializer:org.apache.kafka.common.serialization.StringSerializerconsumer:key-deserializer:org.apache.kafka.common.serialization.StringDeserializervalue-deserializer:org.apache.kafka.common.serialization.StringDeserializer集群测试命令# 进入kafka‑1容器dockerexec-itkafka‑1bash# 创建topic3分区3副本kafka-topics.sh--create--topictest3 --bootstrap-server127.0.0.1:9092--partitions3--replication-factor3# 查看topic详细信息kafka-topics.sh--describe--topictest3 --bootstrap-server127.0.0.1:9092# 查看集群版本kafka-topics.sh--version关键特性说明3个controller节点quorum3挂掉任意一台集群控制器依然正常工作挂2台控制器集群元数据不可用。default.replication.factor3新建topic默认3副本。同一台宿主机docker模拟仅用于开发测试真正生产三台物理服务器部署每台机器只跑一个kafka容器KAFKA_CONTROLLER_QUORUM_VOTERS填写三台机器ipcontroller端口KAFKA_ADVERTISED_LISTENERS填写各自机器外网/内网IP高频踩坑KAFKA_ADVERTISED_LISTENERS必须填写宿主机真实内网IP不能写127.0.0.1否则外部业务连接失败。容器内部之间通信使用容器名kafka‑1:9093不要用宿主机IP。数据目录权限报错chmod -R 777 data‑*。replication‑factor不能大于broker节点数。旧Zookeeper的kafka数据不能直接迁移到此集群需要重新搭建或使用官方迁移工具。停止清理# 停止集群docker-composedown# 彻底删除数据测试重置集群用rm-rfdata‑1 data‑2 data‑3mkdir-pdata‑1 data‑2 data‑3chmod-R777data‑1 data‑2 data‑3
文章目录一、单节点Docker 运行 Kafka 4.0docker run 单机快速测试用docker‑compose 完整模板单机KRaft4.0.0进入容器测试命令SpringBoot application.yml 连接配置重点注意事项搭配kafka‑ui可视化扩展compose片段二、Kafka4.0.x KRaft 三节点高可用集群Docker Composedocker‑compose.yml启动步骤SpringBoot 连接配置集群测试命令关键特性说明高频踩坑停止清理一、单节点Docker 运行 Kafka 4.0Kafka 4.0彻底删除 Zookeeper 模式只支持 KRaft不再支持 zk。官方镜像apache/kafka没有 zk 相关任何配置。docker run 单机快速测试用dockerrun-d\--namekafka4\--restartalways\-p9092:9092\-p9093:9093\-v/data/kafka4/kraft-data:/tmp/kraft-combined-logs\-eKAFKA_NODE_ID1\-eKAFKA_PROCESS_ROLESbroker,controller\-eKAFKA_LISTENERSPLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093\-eKAFKA_ADVERTISED_LISTENERSPLAINTEXT://192.168.1.100:9092\-eKAFKA_CONTROLLER_LISTENER_NAMESCONTROLLER\-eKAFKA_CONTROLLER_QUORUM_VOTERS1kafka4:9093\-eKAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR1\apache/kafka:4.0.0⚠️ 修改192.168.1.100为你的宿主机真实IPSpringBoot业务连接用这个IP:9092。docker‑compose 完整模板单机KRaft4.0.0version:3.8services:kafka:image:apache/kafka:4.0.0container_name:kafka4restart:alwaysports:-9092:9092-9093:9093volumes:-./kraft-data:/tmp/kraft-combined-logsenvironment:KAFKA_NODE_ID:1KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://192.168.1.100:9092KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka:9093KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:1KAFKA_LOG_RETENTION_HOURS:72KAFKA_LOG_DIRS:/tmp/kraft-combined-logs启动docker-composeup-d进入容器测试命令# 进入容器dockerexec-itkafka4bash# 创建topickafka-topics.sh--create--topicdemo --bootstrap-server127.0.0.1:9092--partitions2--replication-factor1# 列表kafka-topics.sh--list--bootstrap-server127.0.0.1:9092# 生产者kafka-console-producer.sh--topicdemo --bootstrap-server127.0.0.1:9092# 消费者kafka-console-consumer.sh--topicdemo --bootstrap-server127.0.0.1:9092 --from-beginning# 查看kafka版本kafka-topics.sh--versionSpringBoot application.yml 连接配置spring:kafka:bootstrap-servers:192.168.1.100:9092producer:key-serializer:org.apache.kafka.common.serialization.StringSerializervalue-serializer:org.apache.kafka.common.serialization.StringSerializerconsumer:key-deserializer:org.apache.kafka.common.serialization.StringDeserializervalue-deserializer:org.apache.kafka.common.serialization.StringDeserializer重点注意事项Kafka4.x 完全没有Zookeeper不要再写任何zk连接参数。镜像apache/kafka:4.0.0不要用confluentinc/cp‑kafkacp还没有4.0大版本。数据目录权限挂载宿主机目录报权限拒绝chmod-R777./kraft-data升级提醒旧的kafka3.x zookeeper集群不能直接升级到4.0。3.x zk集群 → 先升级到3.5/3.6执行迁移工具切换KRaft迁移完成后再升级到4.0。不能直接把zk的数据目录直接给kafka4使用元数据格式完全不一样。单机生产注意单机同时做brokercontrollerreplication‑factor1不能高可用仅测试/小规模使用真正生产集群至少3节点controller。搭配kafka‑ui可视化扩展compose片段kafka-ui:image:provectuslabs/kafka-ui:latestcontainer_name:kafka-uirestart:alwaysports:-8081:8080environment:KAFKA_CLUSTERS_0_NAME:local‑kraft‑40KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS:kafka:9092访问http://宿主机IP:8081二、Kafka4.0.x KRaft 三节点高可用集群Docker ComposeKafka4.0彻底移除Zookeeper仅KRaft模式3个节点全部同时充当broker controller控制器仲裁quorum3挂掉任意1台集群依然正常工作满足生产高可用基础条件。注意下面是单机docker模拟3节点集群同一台宿主机用于开发测试真正生产要部署在三台不同物理服务器。docker‑compose.ymlversion:3.8services:kafka‑1:image:apache/kafka:4.0.0container_name:kafka‑1restart:alwaysports:-9092:9092# broker端口对外-9093:9093# controller内部仲裁端口volumes:-./data‑1:/tmp/kraft‑combined‑logsenvironment:KAFKA_NODE_ID:1KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093# 修改为你的宿主机IP外部业务程序连接KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://192.168.1.100:9092KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka‑1:9093,2kafka‑2:9093,3kafka‑3:9093KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:3KAFKA_DEFAULT_REPLICATION_FACTOR:3KAFKA_LOG_DIRS:/tmp/kraft‑combined‑logskafka‑2:image:apache/kafka:4.0.0container_name:kafka‑2restart:alwaysports:-9094:9092-9095:9093volumes:-./data‑2:/tmp/kraft‑combined‑logsenvironment:KAFKA_NODE_ID:2KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://192.168.1.100:9094KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka‑1:9093,2kafka‑2:9093,3kafka‑3:9093KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:3KAFKA_DEFAULT_REPLICATION_FACTOR:3KAFKA_LOG_DIRS:/tmp/kraft‑combined‑logskafka‑3:image:apache/kafka:4.0.0container_name:kafka‑3restart:alwaysports:-9096:9092-9097:9093volumes:-./data‑3:/tmp/kraft‑combined‑logsenvironment:KAFKA_NODE_ID:3KAFKA_PROCESS_ROLES:broker,controllerKAFKA_LISTENERS:PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093KAFKA_ADVERTISED_LISTENERS:PLAINTEXT://192.168.1.100:9096KAFKA_CONTROLLER_LISTENER_NAMES:CONTROLLERKAFKA_CONTROLLER_QUORUM_VOTERS:1kafka‑1:9093,2kafka‑2:9093,3kafka‑3:9093KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:3KAFKA_DEFAULT_REPLICATION_FACTOR:3KAFKA_LOG_DIRS:/tmp/kraft‑combined‑logskafka-ui:image:provectuslabs/kafka-ui:latestcontainer_name:kafka-uirestart:alwaysports:-8081:8080environment:KAFKA_CLUSTERS_0_NAME:kafka4‑3node‑kraftKAFKA_CLUSTERS_0_BOOTSTRAPSERVERS:kafka‑1:9092,kafka‑2:9092,kafka‑3:9092启动步骤# 创建数据目录mkdir-pdata‑1 data‑2 data‑3# 解决容器挂载目录权限问题chmod-R777data‑1 data‑2 data‑3# 启动集群docker-composeup-d# 查看容器状态docker-composeps访问UIhttp://宿主机IP:8081SpringBoot 连接配置spring:kafka:bootstrap-servers:192.168.1.100:9092,192.168.1.100:9094,192.168.1.100:9096producer:key-serializer:org.apache.kafka.common.serialization.StringSerializervalue-serializer:org.apache.kafka.common.serialization.StringSerializerconsumer:key-deserializer:org.apache.kafka.common.serialization.StringDeserializervalue-deserializer:org.apache.kafka.common.serialization.StringDeserializer集群测试命令# 进入kafka‑1容器dockerexec-itkafka‑1bash# 创建topic3分区3副本kafka-topics.sh--create--topictest3 --bootstrap-server127.0.0.1:9092--partitions3--replication-factor3# 查看topic详细信息kafka-topics.sh--describe--topictest3 --bootstrap-server127.0.0.1:9092# 查看集群版本kafka-topics.sh--version关键特性说明3个controller节点quorum3挂掉任意一台集群控制器依然正常工作挂2台控制器集群元数据不可用。default.replication.factor3新建topic默认3副本。同一台宿主机docker模拟仅用于开发测试真正生产三台物理服务器部署每台机器只跑一个kafka容器KAFKA_CONTROLLER_QUORUM_VOTERS填写三台机器ipcontroller端口KAFKA_ADVERTISED_LISTENERS填写各自机器外网/内网IP高频踩坑KAFKA_ADVERTISED_LISTENERS必须填写宿主机真实内网IP不能写127.0.0.1否则外部业务连接失败。容器内部之间通信使用容器名kafka‑1:9093不要用宿主机IP。数据目录权限报错chmod -R 777 data‑*。replication‑factor不能大于broker节点数。旧Zookeeper的kafka数据不能直接迁移到此集群需要重新搭建或使用官方迁移工具。停止清理# 停止集群docker-composedown# 彻底删除数据测试重置集群用rm-rfdata‑1 data‑2 data‑3mkdir-pdata‑1 data‑2 data‑3chmod-R777data‑1 data‑2 data‑3
