【Kafka八】Zookeeper上关于Kafka的配置信息

编程技术  /  houtizong 发布于 3年前   99

问题：

1. Kafka的哪些信息记录在Zookeeper中

2. Consumer Group消费的每个Partition的Offset信息存放在什么位置

3. Topic的每个Partition存放在哪个Broker上的信息存放在哪里

4. Producer跟Zookeeper究竟有没有关系？没有关系！！！

 

//consumers、config、brokers、controller_epoch都是Kafka相关的znode[zk: localhost:2181(CONNECTED) 0] ls /[abc, admin, zookeeper, consumers, config, controller, storm, brokers, controller_epoch][zk: localhost:2181(CONNECTED) 1] ls///consumers目录下是所有的consumer group的定义[zk: localhost:2181(CONNECTED) 2] ls /consumers[test-consumer-group, topic-p6-r2-consumer-group, learn.topic.consumers.group, learn.topic.p8.r2.consumers.group][zk: localhost:2181(CONNECTED) 3] ls learn.topic.p8.r2.consumers.groupCommand failed: java.lang.IllegalArgumentException: Path must start with / character///consumer group下游offsets、owners和ids的定义[zk: localhost:2181(CONNECTED) 4] ls /consumers/learn.topic.p8.r2.consumers.group[offsets, owners, ids]///offsets下面是topic的定义[zk: localhost:2181(CONNECTED) 5] ls /consumers/learn.topic.p8.r2.consumers.group/offsets[learn.topic.p8.r2]///topic下面是partition的定义，learn.topic.p8.r2表示8个partition，2个副本///partition中的offset的设置在znode的data中，而不是子znode中[zk: localhost:2181(CONNECTED) 6] ls /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2[3, 2, 1, 0, 7, 6, 5, 4][zk: localhost:2181(CONNECTED) 7] ls /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2/2[][zk: localhost:2181(CONNECTED) 8] ls /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2/3[][zk: localhost:2181(CONNECTED) 9] ls /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2/4[]///consumer group属于哪个topic所有，代码中不需要显式指定///consumer group消费哪个topic，kafka自动将topic设置为这个consumer group的owner[zk: localhost:2181(CONNECTED) 19] ls /consumers/learn.topic.p8.r2.consumers.group/owners[learn.topic.p8.r2][zk: localhost:2181(CONNECTED) 20] ls /consumers/learn.topic.p8.r2.consumers.group/owners/learn.topic.p8.r2[3, 2, 1, 0, 7, 6, 5, 4][zk: localhost:2181(CONNECTED) 21] ls /consumers/learn.topic.p8.r2.consumers.group/owners/learn.topic.p8.r2/3[]///后面是broker的ID[zk: localhost:2181(CONNECTED) 23] ls /consumers/learn.topic.p8.r2.consumers.group/ids[learn.topic.p8.r2.consumers.group_TP-A383-D-1427853446455-efa1e47b][zk: localhost:2181(CONNECTED) 24] ls /consumers/learn.topic.p8.r2.consumers.group/ids/learn.topic.p8.r2.consumers.group_TP-A383-D-1427853446455-efa1e47b[][zk: localhost:2181(CONNECTED) 10] ls /brokers[topics, ids][zk: localhost:2181(CONNECTED) 11] ls /brokers/topics[topic_p10_r3, topic-p6-r2, test_many_partitions, test, topic_p10_r1, learn.topic.p8.r2, learn.topic, topic-3-replica-1-part][zk: localhost:2181(CONNECTED) 12] ls /brokers/topics/learn.topic.p8.r2[partitions][zk: localhost:2181(CONNECTED) 13] ls /brokers/topics/learn.topic.p8.r2/partitions[3, 2, 1, 0, 7, 6, 5, 4][zk: localhost:2181(CONNECTED) 14] ls /brokers/topics/learn.topic.p8.r2/partitions/3[state][zk: localhost:2181(CONNECTED) 15] ls /brokers/topics/learn.topic.p8.r2/partitions/3/state[][zk: localhost:2181(CONNECTED) 16] ls /brokers/ids[2, 1, 0][zk: localhost:2181(CONNECTED) 17] ls /brokers/ids/2[][zk: localhost:2181(CONNECTED) 18]

 

每个Consumer Group的每个Partition的Offset信息是作为值存放在Partition节点上的，而不是作为子节点，如下所示，4704是3号Partition已经消费的Offset数

[zk: localhost:2181(CONNECTED) 28] get /consumers/learn.topic.p8.r2.consumers.group/offsets/learn.topic.p8.r2/3 4704cZxid = 0x1bf14ctime = Tue Mar 31 06:25:39 EDT 2015mZxid = 0x1c658mtime = Tue Mar 31 22:03:48 EDT 2015pZxid = 0x1bf14cversion = 0dataVersion = 177aclVersion = 0ephemeralOwner = 0x0dataLength = 4numChildren = 0

 

Topic的每个Partition存放在哪个Broker上的信息存放在哪里？

[zk: localhost:2181(CONNECTED) 30] ls /brokers                                                                 [topics, ids][zk: localhost:2181(CONNECTED) 31] ls /brokers/topics[topic_p10_r3, topic-p6-r2, test_many_partitions, test, topic_p10_r1, learn.topic.p8.r2, learn.topic, topic-3-replica-1-part][zk: localhost:2181(CONNECTED) 32] ls /brokers/topics/learn.topic.p8.r2[partitions][zk: localhost:2181(CONNECTED) 33] ls /brokers/topics/learn.topic.p8.r2/partitions[3, 2, 1, 0, 7, 6, 5, 4][zk: localhost:2181(CONNECTED) 34] ls /brokers/topics/learn.topic.p8.r2/partitions/3[state][zk: localhost:2181(CONNECTED) 35] ls /brokers/topics/learn.topic.p8.r2/partitions/3/state[]///[zk: localhost:2181(CONNECTED) 36] get  /brokers/topics/learn.topic.p8.r2/partitions/3/state{"controller_epoch":42,"leader":0,"version":1,"leader_epoch":2,"isr":[0,1]}cZxid = 0x1bee8ctime = Tue Mar 31 05:34:16 EDT 2015mZxid = 0x1c247mtime = Tue Mar 31 21:48:01 EDT 2015pZxid = 0x1bee8cversion = 0dataVersion = 5aclVersion = 0ephemeralOwner = 0x0dataLength = 75numChildren = 0

 

每个Broker的信息：包括host和port

//三个broker[zk: localhost:2181(CONNECTED) 38] ls  /brokers/ids                                         [2, 1, 0][zk: localhost:2181(CONNECTED) 39] ls  /brokers/ids/2[]///host和port信息[zk: localhost:2181(CONNECTED) 40] get  /brokers/ids/2{"jmx_port":-1,"timestamp":"1427852973061","host":"192.168.26.140","version":1,"port":9094}cZxid = 0x1c257ctime = Tue Mar 31 21:49:33 EDT 2015mZxid = 0x1c257mtime = Tue Mar 31 21:49:33 EDT 2015pZxid = 0x1c257cversion = 0dataVersion = 0aclVersion = 0ephemeralOwner = 0x14c72a6b3610003dataLength = 91numChildren = 0

 

每个Topic的Partion存放在哪个或者哪些Broker上，信息存放在

///每个partition的leaderpartition和follower partition信息[zk: localhost:2181(CONNECTED) 53] get /brokers/topics/learn.topic.p8.r2{"version":1,"partitions":{"4":[2,1],"5":[0,2],"6":[1,2],"1":[2,0],"0":[1,2],"2":[0,1],"7":[2,0],"3":[1,0]}}

 

 

 

上一篇：【Spark八十四】Spark零碎知识点记录
下一篇：【Kafka九】Kafka High Level API vs. Low Level API

请勿发布不友善或者负能量的内容。与人为善，比聪明更重要！

留言需要登陆哦

技术博客集 - 网站简介：
前后端技术：
后端基于Hyperf2.1框架开发,前端使用Bootstrap可视化布局系统生成
网站主要作用：
1.编程技术分享及讨论交流，内置聊天系统;
2.测试交流框架问题，比如：Hyperf、Laravel、TP、beego;
3.本站数据是基于大数据采集等爬虫技术为基础助力分享知识，如有侵权请发邮件到站长邮箱，站长会尽快处理;
4.站长邮箱：[email protected];

      订阅博客周刊 去订阅

文章归档

文章标签

友情链接

首页
关于我们
Auther ·HouTiZong
侯体宗的博客
© 2020 zongscan.com
版权所有ICP证 : 粤ICP备20027696号
PHP交流群 也可以扫右边的二维码
侯体宗的博客