High Availability – Platform(Multi node Multi broker)
Introduction
The purpose of adding replication in Kafka is for stronger durability and higher availability. In this document we describe how to install, configure and run a muti node-multi-broker Apache Kafka cluster on AWS.
Environment
We use 3 AWS boxes, henceforth known as SM1, SM2 and SM3.
Note: Ensure that Java version 1.8 is used.
Installation steps
Download latest version of kafka .
1. Download and extract Kafka on SM1 .(1st Machine)
2. Download and extract Kafka on SM2. (2nd Machine)
3. Download and extract Kafka on SM3.(3rd Machine)
Configuring Zookeeper:-
Extract the zip/tar file .Goto config folder .Edit zookeeper.properties file .Copy and paste the below mentioned details on First machine .Similarly do for other machines .
Zookeeper-1 (on SM1)
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
server.3=ec2-<IP3>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2
Zookeeper-2 (on SM2)
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.2= ec2-<IP2>.amazonaws.com:2888:388
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.3=ec2-<IP3>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2
Zookeeper-3 (on SM3)
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.3= ec2-<IP3>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
server.1=ec2-<IP1>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2
3) After this Goto zookeeper data directory of every instance and create following things :
Goto zookeeper data directory of SM1 instance ,
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid #Zookeeper temp file
echo '1' >> myid
goto SM2 instance ,
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid #Zookeeper temp file
echo '2' >> myid yid
goto SM3 instance ,
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid #Zookeeper temp file
echo '3' >> myid
Configuring Kafka:-
Broker -1(on SM1)
1 Edit config/server.properties for kafka , add or replace the following configs.
broker.id=1
port=9092
host.name=ec2-<IP1>.amazonaws.com
log.dir=/tmp/kafka-logs-1
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M1>:9092
delete.topic.enable=true
zookeeper.connect=ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181
Broker -2(on SM2)
1. Edit config/server.properties for kafka , add or replace the following configs.
broker.id=2
port=9092
host.name=ec2-<IP2>.amazonaws.com
log.dir=/tmp/kafka-logs-2
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M2>:9092
delete.topic.enable=true
zookeeper.connect =ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181
Broker -3(on SM3)
1. Edit config/server.properties for kafka , add or replace the following configs.
broker.id=3
port=9092
host.name=ec2-<IP3>.amazonaws.com
log.dir=/tmp/kafka-logs-3
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M3>:9092
delete.topic.enable=true
zookeeper.connect=ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181
To start kafka ,
Ø nohup bin/kafka-server-start.sh config/server .properties &
why taking odd number of zookeeper ?
Zookeeper number should be odd(either 1,3,5 ..)For reliable ZooKeeper service, you should deploy ZooKeeper in a cluster known as an ensemble. As long as a majority of the ensemble are up, the service will be available. Because Zookeeper requires a majority, it is best to use an odd number of machines. For example, with four machines ZooKeeper can only handle the failure of a single machine; if two machines fail, the remaining two machines do not constitute a majority. However, with five machines ZooKeeper can handle the failure of two machines.
Handling Data loss:-
Be sure that ,while working on cluster ,your replication factor minimum 3,and it should be more than number of broker .
Handling Data loss:-
Be sure that ,while working on cluster ,your replication factor minimum 3,and it should be more than number of broker .
Nicely Explained. Great Post.. !!
ReplyDeleteusefull article. Thanks for sharing.
ReplyDeleteAWS Online Training
thanks..
ReplyDeleteReally very nice blog information for this one and more technical skills are improve,i like that kind of post.
ReplyDeleteAWS training in chennai
Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
ReplyDeleteAWS Training in Bangalore
AWS Online Training
Thanks for sharing this blog with us. I really enjoy reading your post. Keep Doing.
ReplyDelete"AWS Training in Hyderabad
"