Apache Kafka is a distributed streaming platform providing you publish and subscribe features of an enterprise messaging system while also supporting data stream processing. In this blog we will setup a standalone Kafka topic on a local machine on Windows operating system. Please note, consider this setup as a Hello World application as it is not meant for production use.

Software versions used in this exercise:
Java: 1.8.0_181
Zookeeper: 3.4.13
Kafka: 2.11-2.0.0

As a pre-requisite to this exercise ensure you have Java 8 and above installed on your machine and the JAVA_HOME environment variable configured. Validate this by opening Command Prompt and checking Java version.
>java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
>echo %JAVA_HOME%
C:\Java\jdk1.8.0_92

Zookeeper Setup

Next we need to setup Zookeeper. Download Zookeeper and unzip the contents of the tar into a folder e.g. C:\Kafka
Setup environment variable for Zookeeper with key “ZOOKEEPER_HOME” and value “C:\Zookeeper” if this is the unzip folders path.
Add “%ZOOKEEPER_HOME%\bin” into the Path environment variable.
Go to “C:\Zookeeper\config” folder and rename “zoo_sample.cfg” to “zoo.cfg”.
Open “zoo.cg” file in an editor and set path for dataDir to “data/zoo”.
Default port can be changed from 2181 especially if the port is in use by some other application.
Open a new command prompt for running Zookeeper and type “zkserver”.
This should start the Zookeeper application.
>zkserver

Kafka Setup

Now we can setup Kafka for which we need to first download the binary file. Please note, don’t download the source version “src” as then you will be required to compile the downloaded files. Now unzip the tar into a folder e.g. “C:\Kafka”.
Setup environment variable “KAFKA_HOME” as “C:\Kafka”.
Add bin folder of Kafka to Path environment variable “%KAFKA_HOME%\bin\windows”. The files directly under bin folder are Linux specific so we need to configure path for windows compatible files which are in a sub-folder within bin folder.
Create a folder for logs under Kafka home folder, “C:\Kafka\logs”
Next open the “server.properties” file under the “C:\Kafka\config” folder and set the log.dir property value to this log folder, “log.dirs=E:\Tools\kafka\logs”.
Also check if your Zookeeper connection details are correct with respect to your Zookeeper instance, “zookeeper.connect=localhost:2182”. As you can see I am not using the default Zookeeper port 2181 so I had to change the port value.
Your Kafka will run on default port 9092 and will connect to Zookeeper on the custom port 2182.

Now start Kafka using the following commands. You can specify the path of server.properties as absolute value or relative value to the batch file.
>kafka-server-start.bat C:\Kafka\config\server.properties

Your Kafka should be up and running now. Go ahead and create a topic “test” using the following command in another Command terminal.
>kafka-topics.bat --create --zookeeper localhost:2182 --replication-factor 1 --partitions 1 --topic test
Created topic "test".

Test Kafka Setup

Now we can test the setup by creating a Producer and Consumer for the topic.

Start a producer in a command terminal using the following command:
>kafka-console-producer.bat --broker-list localhost:9092 --topic test
>

Start a consumer on another Command terminal by executing the following command. Note that the following command will connect to Kafka server to receive messages. In earlier versions of Kafka you used to connect to Zookeeper using the “–zookeeper” options which has now been deprecated.
>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test

Now if you type a message on the prompt of Kafka Producer, the same message will get displayed in the Kafka consumer’s window.

So in this blog we have seen how to setup a simple standalone Kafka instance and test it locally with a Producer and Consumer.

Note, you can also subscribe multiple consumers in a new Command terminal on the same topic to get the message from Producer delivered to all of them.