High Availability – Platform(Multi node Multi broker)
Introduction
The purpose of adding replication in Kafka is for stronger durability and higher availability. In this document we describe how to install, configure and run a muti node-multi-broker Apache Kafka cluster on AWS.
Environment
We use 3 AWS boxes, henceforth known as SM1, SM2 and SM3.
Note: Ensure that Java version 1.8 is used.
Installation steps
Download latest version of kafka .
1. Download and extract Kafka on SM1 .(1st Machine)
2. Download and extract Kafka on SM2. (2nd Machine)
3. Download and extract Kafka on SM3.(3rd Machine)
Configuring Zookeeper:-
Extract the zip/tar file .Goto config folder .Edit zookeeper.properties file .Copy and paste the below mentioned details on First machine .Similarly do for other machines .
Zookeeper-1 (on SM1)
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
server.3=ec2-<IP3>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2
Zookeeper-2 (on SM2)
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.2= ec2-<IP2>.amazonaws.com:2888:388
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.3=ec2-<IP3>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2
Zookeeper-3 (on SM3)
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.3= ec2-<IP3>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
server.1=ec2-<IP1>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2
3) After this Goto zookeeper data directory of every instance and create following things :
Goto zookeeper data directory of SM1 instance ,
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid #Zookeeper temp file
echo '1' >> myid
goto SM2 instance ,
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid #Zookeeper temp file
echo '2' >> myid yid
goto SM3 instance ,
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid #Zookeeper temp file
echo '3' >> myid
Configuring Kafka:-
Broker -1(on SM1)
1 Edit config/server.properties for kafka , add or replace the following configs.
broker.id=1
port=9092
host.name=ec2-<IP1>.amazonaws.com
log.dir=/tmp/kafka-logs-1
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M1>:9092
delete.topic.enable=true
zookeeper.connect=ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181
Broker -2(on SM2)
1. Edit config/server.properties for kafka , add or replace the following configs.
broker.id=2
port=9092
host.name=ec2-<IP2>.amazonaws.com
log.dir=/tmp/kafka-logs-2
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M2>:9092
delete.topic.enable=true
zookeeper.connect =ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181
Broker -3(on SM3)
1. Edit config/server.properties for kafka , add or replace the following configs.
broker.id=3
port=9092
host.name=ec2-<IP3>.amazonaws.com
log.dir=/tmp/kafka-logs-3
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M3>:9092
delete.topic.enable=true
zookeeper.connect=ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181
To start kafka ,
Ø nohup bin/kafka-server-start.sh config/server .properties &
why taking odd number of zookeeper ?
Zookeeper number should be odd(either 1,3,5 ..)For reliable ZooKeeper service, you should deploy ZooKeeper in a cluster known as an ensemble. As long as a majority of the ensemble are up, the service will be available. Because Zookeeper requires a majority, it is best to use an odd number of machines. For example, with four machines ZooKeeper can only handle the failure of a single machine; if two machines fail, the remaining two machines do not constitute a majority. However, with five machines ZooKeeper can handle the failure of two machines.
Handling Data loss:-
Be sure that ,while working on cluster ,your replication factor minimum 3,and it should be more than number of broker .
Handling Data loss:-
Be sure that ,while working on cluster ,your replication factor minimum 3,and it should be more than number of broker .