Kafka clustering in Amazon Web Service (AWS)

High Availability – Platform(Multi node Multi broker)


Introduction

The purpose of adding replication in Kafka is for stronger durability and higher availability.  In this document we describe how to install, configure and run a muti node-multi-broker Apache Kafka cluster on AWS.

Environment

We use 3 AWS boxes, henceforth known as SM1, SM2 and SM3.
Note: Ensure that Java version 1.8 is used.

Installation steps


Download latest version of kafka .


1.       Download and extract Kafka on SM1 .(1st  Machine)
2.       Download and extract Kafka on SM2. (2nd Machine)
3.       Download and extract Kafka on SM3.(3rd Machine)

Configuring Zookeeper:-


Extract the zip/tar file .Goto config folder .Edit zookeeper.properties file  .Copy and paste the below mentioned details on First machine .Similarly do for other machines . 

Zookeeper-1 (on SM1)

dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
server.3=ec2-<IP3>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2

Zookeeper-2 (on SM2)

dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.2=ec2-<IP2>.amazonaws.com:2888:388
server.1=ec2-<IP1>.amazonaws.com:2888:3888
server.3=ec2-<IP3>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2

Zookeeper-3 (on SM3) 

dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=0
server.3=ec2-<IP3>.amazonaws.com:2888:3888
server.2=ec2-<IP2>.amazonaws.com:2888:3888
server.1=ec2-<IP1>.amazonaws.com:2888:3888
initLimit=5
syncLimit=2


3) After this Goto zookeeper data directory of every instance and create following things :
Goto zookeeper data directory of SM1 instance ,
 
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid  #Zookeeper temp file
echo '1' >> myid 

goto SM2 instance ,

goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid  #Zookeeper temp file
echo '2' >> myid yid

goto SM3 instance , 
goto /var/ (my case created in var)
cd zookeeper #Zookeeper temp dir
touch myid  #Zookeeper temp file
echo '3' >> myid 

Configuring Kafka:-

 Broker -1(on SM1)

1         Edit config/server.properties  for kafka , add or replace the following configs.


broker.id=1
port=9092
host.name=ec2-<IP1>.amazonaws.com
log.dir=/tmp/kafka-logs-1
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M1>:9092
delete.topic.enable=true
zookeeper.connect=ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181

Broker -2(on SM2)

1.       Edit config/server.properties  for kafka , add or replace the following configs.
broker.id=2
port=9092
host.name=ec2-<IP2>.amazonaws.com
log.dir=/tmp/kafka-logs-2
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M2>:9092
delete.topic.enable=true
zookeeper.connect =ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181

Broker -3(on SM3)

1.       Edit config/server.properties  for kafka , add or replace the following configs.

broker.id=3
port=9092
host.name=ec2-<IP3>.amazonaws.com
log.dir=/tmp/kafka-logs-3
advertised.listeners=PLAINTEXT://<PRIVATE-IP-OF-M3>:9092
delete.topic.enable=true
zookeeper.connect=ec2-<IP1>amazonaws.com:2181, ec2-<IP2>amazonaws.com 2181,ec2-<IP3>amazonaws.com:2181

To start zookeeper ,

Ø  nohup bin/zookeeper-server-start.sh config/zookeeper.properties &

To start kafka  ,

Ø  nohup bin/kafka-server-start.sh config/server .properties  &

You can see zookeeper starts on port 2181 and kafka starts on 9092 port respectively .

IMP
As I am using 3 differnet machine I have taken same port number for 3 zookeepers and 3 kafkas .
If you are running single machine then change the port number .



why taking odd number of zookeeper ?



Zookeeper number should be odd(either 1,3,5 ..)For reliable ZooKeeper service, you should deploy ZooKeeper in a cluster known as an ensemble. As long as a majority of the ensemble are up, the service will be available. Because Zookeeper requires a majority, it is best to use an odd number of machines. For example, with four machines ZooKeeper can only handle the failure of a single machine; if two machines fail, the remaining two machines do not constitute a majority. However, with five machines ZooKeeper can handle the failure of two machines. 

Handling Data loss:-

Be sure that ,while working on cluster ,your replication factor minimum 3,and it should be more than number of broker .

7 comments:

  1. Really very nice blog information for this one and more technical skills are improve,i like that kind of post.
    AWS training in chennai

    ReplyDelete
  2. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    AWS Training in Bangalore
    AWS Online Training

    ReplyDelete
  3. Thanks for sharing this blog with us. I really enjoy reading your post. Keep Doing.

    "AWS Training in Hyderabad
    "

    ReplyDelete

thank you