Kafka clustering in Amazon Web Service (AWS)

High Availability – Platform(Multi node Multi broker)


Introduction

The purpose of adding replication in Kafka is for stronger durability and higher availability.  In this document we describe how to install, configure and run a muti node-multi-broker Apache Kafka cluster on AWS.

Environment

We use 3 AWS boxes, henceforth known as SM1, SM2 and SM3.
Note: Ensure that Java version 1.8 is used.

Installation steps


Download latest version of kafka .


1.       Download and extract Kafka on SM1 .(1st  Machine)
2.       Download and extract Kafka on SM2. (2nd Machine)
3.       Download and extract Kafka on SM3.(3rd Machine)

Configuring Zookeeper:-


Extract the zip/tar file .Goto config folder .Edit zookeeper.properties file  .Copy and paste the below mentioned details on First machine .Similarly do for other machines . 

Zookeeper-1 (on SM1)

dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
server.3=ec2-<IP3>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2

Zookeeper-2 (on SM2)

dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.2=ec2-<IP2>.amazonaws.com:2888:388
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.3=ec2-<IP3>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2

Zookeeper-3 (on SM3) 

dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.3=ec2-<IP3>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
server.1=ec2-<IP1>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2


3) After this Goto zookeeper data directory of every instance and create following things :
Goto zookeeper data directory of SM1 instance ,
 
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid  #Zookeeper temp file
echo '1' >> myid 

goto SM2 instance ,

goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid  #Zookeeper temp file
echo '2' >> myid yid

goto SM3 instance , 
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid  #Zookeeper temp file
echo '3' >> myid 

Configuring Kafka:-

 Broker -1(on SM1)

1         Edit config/server.properties  for kafka , add or replace the following configs.


broker.id=1
port=9092
host.name=ec2-<IP1>.amazonaws.com
log.dir=/tmp/kafka-logs-1
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M1>:9092
delete.topic.enable=true
zookeeper.connect=ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181

Broker -2(on SM2)

1.       Edit config/server.properties  for kafka , add or replace the following configs.
broker.id=2
port=9092
host.name=ec2-<IP2>.amazonaws.com
log.dir=/tmp/kafka-logs-2
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M2>:9092
delete.topic.enable=true
zookeeper.connect =ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181

Broker -3(on SM3)

1.       Edit config/server.properties  for kafka , add or replace the following configs.

broker.id=3
port=9092
host.name=ec2-<IP3>.amazonaws.com
log.dir=/tmp/kafka-logs-3
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M3>:9092
delete.topic.enable=true
zookeeper.connect=ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181

To start zookeeper ,

Ø  nohup bin/zookeeper-server-start.sh config/zookeeper.properties &

To start kafka  ,

Ø  nohup bin/kafka-server-start.sh config/server .properties  &

You can see zookeeper starts on port 2181 and kafka starts on 9092 port respectively .

IMP
As I am using 3 differnet machine I have taken same port number for 3 zookeepers and 3 kafkas .
If you are running single machine then change the port number .



why taking odd number of zookeeper ?



Zookeeper number should be odd(either 1,3,5 ..)For reliable ZooKeeper service, you should deploy ZooKeeper in a cluster known as an ensemble. As long as a majority of the ensemble are up, the service will be available. Because Zookeeper requires a majority, it is best to use an odd number of machines. For example, with four machines ZooKeeper can only handle the failure of a single machine; if two machines fail, the remaining two machines do not constitute a majority. However, with five machines ZooKeeper can handle the failure of two machines. 

Handling Data loss:-

Be sure that ,while working on cluster ,your replication factor minimum 3,and it should be more than number of broker .