{"id":57660,"date":"2023-06-25T13:37:09","date_gmt":"2023-06-25T08:07:09","guid":{"rendered":"https:\/\/2thenew.xyz\/blog\/?p=57660"},"modified":"2023-07-03T13:45:47","modified_gmt":"2023-07-03T08:15:47","slug":"how-to-install-apache-kafka-kafdrop-on-windowsfor-and-integrate-with-drupal","status":"publish","type":"post","link":"https:\/\/2thenew.xyz\/blog\/how-to-install-apache-kafka-kafdrop-on-windowsfor-and-integrate-with-drupal\/","title":{"rendered":"How to install Apache Kafka | Kafdrop on WindowsFor and integrate with Drupal"},"content":{"rendered":"

Apache Kafka is an open-source distributed event streaming platform. That uses Publish & subscribe mechanism to stream the records.\u200b<\/p>\n

Download & Installing the Apache Kafka:<\/h2>\n

\u200bPrerequisite<\/strong>: You should have Java (JDK) installed on your Windows machine.<\/span><\/p>\n

Step -1:<\/strong> Download the binary version from the kafka official download page ( https:\/\/kafka.apache.org\/downloads )<\/p>\n

Step -2:<\/strong> Unzip it and and rename the folder like kafka_binary & paste the folder in c drive like C:\\dev\\kafka_binary<\/span><\/span><\/p>\n

Step -3: <\/strong> Change the log path setting in the below files<\/span><\/span><\/p>\n

filename : C:\\dev\\kafka_binary\\config\\server.properties<\/span><\/span>\u200b<\/span><\/pre>\n
Change :\u00a0 log.dirs<\/span><\/span>=\/<\/span><\/span>tmp<\/span><\/span>\/<\/span><\/span>kafka<\/span><\/span>-logs\u00a0 =>\u00a0\u00a0<\/span><\/span>log.dirs<\/span><\/span>=C:\/dev\/kafka_binary\/kafka-logs<\/span><\/span>\r\n\r\n\u200b\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0filename : C:\\dev\\kafka_binary\\config\\zookeeper.properties<\/span><\/span><\/span><\/pre>\n
Change : dataDir<\/span><\/span>=C:\/tmp\/zookeeper\u00a0 =>\u00a0<\/span><\/span>dataDir<\/span><\/span>=C:\/dev\/kafka_binary\/zookeeper<\/span><\/span><\/pre>\n
Now Kafka is installed in your window machine & we can start the Kafka server.\u00a0<\/span><\/p>\n
Kafka has a dependency on the zookeeper service, and this zookeeper is already inbuilt with our server; so first we need to start the zookeeper, then we can start the Kafka server.<\/p>\n
Kafka command uses 2 files as a key-value pair to execute the command, It means for every task, there is 1 file in .bin\\windows folder, which will work with the combination of .\\config folder on the right side bar.<\/p>\n
like<\/p>\n
 \r\nC:\\dev\\kafka_binary.\\bin\\windows\\\u00a0 => .\\config\\<\/span><\/span><\/span><\/span><\/pre>\n
Kafka Uses with the Help of CLI :<\/h4>\n
Open the CMD command prompt & perform the below commands,<\/p>\n
Step -1: <\/strong>Start the\u00a0<\/span><\/span>ZooKeeper<\/span><\/span> service by the below command, so it will start the Zookeeper server keep it running in this terminal; for performing any other cmd command, use another command prompt.<\/span><\/span>\u200b<\/span><\/p>\n
C:\\dev\\kafka_binary.\\bin\\windows\\zookeeper-server-start.bat .\\config\\zookeeper.properties<\/pre>\n
\"Start
Start zookeeper server<\/p><\/div>\n
Step -2 :<\/strong> Start the Kafka broker service by the below command. So it will start the Kafka server keep it running in this terminal; for performing any other cmd command use the another command prompt.<\/span><\/span><\/span><\/span><\/p>\n
C:\\dev\\kafka_binary\\.\\bin\\windows\\kafka-server-start.bat .\\config\\<\/span><\/span>server.properties<\/span><\/span>\u200b<\/span><\/pre>\n
\"Start
Start Kafka Server<\/p><\/div>\n
\u200bStep -3 : <\/strong>Now we can create a topic to store your event,<\/span><\/span><\/span><\/p>\n
\u200bC:\\dev\\kafka_binary\\.\\bin\\windows\\kafka-topics.bat --create --topic topic_part2 --bootstrap-server <\/span><\/span>localhost:9092<\/span><\/span><\/span><\/span><\/pre>\n
\"Create\u200b<\/span><\/span><\/span><\/p>\n
Step -4 :<\/strong> Write some events into the topics<\/span><\/span><\/p>\n
C:\\dev\\kafka_binary\\<\/span><\/span><\/span><\/span>.\\bin\\windows\\kafka-console-producer.bat --topic topic_part2 --bootstrap-server localhost:9092<\/span><\/span><\/pre>\n
\"Write<\/p>\n
\u200bStep -5 : <\/strong>Read the messages from the topic which we save in step -4,\u00a0<\/span><\/p>\n
C:\\dev\\kafka_binary\\<\/span><\/span><\/span><\/span><\/span><\/span>.\\bin\\windows\\kafka-console-consumer.bat --topic topic_part1 --from-beginning --bootstrap-server\u00a0<\/span><\/span>localhost:9092<\/span><\/span><\/pre>\n
\"Read<\/p>\n
 <\/p>\n
That is the whole process to create message in the topic and read the message from the topic.<\/p>\n
Download & Install PHP rd-Kafka Extension on Wamp :<\/h2>\n
I am using the Wamp & PHP version 8.0.26, if you have any other PHP version, update the command accordingly.<\/p>\n
To enable the rd-kafka extension on wamp we need to download the DLL files from the below URL,<\/p>\n
Reference : https:\/\/arnaud.le-blanc.net\/php-rdkafka-doc\/phpdoc\/rdkafka.installation.windows.html<\/span><\/span><\/a><\/p>\n
Download Link : http:\/\/pecl.php.net\/package\/rdkafka<\/p>\n
Before downloading the correct version please check your system information like below<\/p>\n
I am using the 5.0.2 version,<\/p>\n
http:\/\/pecl.php.net\/package\/rdkafka\/5.0.2\/windows<\/p>\n
I have Below configuration on my system please check all these config on your system if it differ then download the DLL based on that, otherwise it will not work properly.<\/p>\n
php -v = 8.0.26
\nrdkafka version = 5.0.2
\nOS Type = 64-bit
\nThread sefe = Enabled<\/p>\n
\"Dll-download-for<\/p>\n
 <\/p>\n
Steps to configure the <\/span><\/span>rd-kafka<\/span><\/span>\u00a0extension on Wamp<\/span><\/span>\u200b Server<\/span><\/h2>\n
Step -1 :<\/strong> Paste the DLL files which we download on below Paths<\/p>\n
C:\\wamp64\\bin\\php\\php8.0.26\\\u00a0\u00a0 Paste the\u00a0<\/span>librdkafka.dll<\/span><\/b>\u200b\r\n\r\nC:\\wamp64\\bin\\php\\php8.0.26\\ext\\ Paste the\u00a0<\/span>php_rdkafka.dll<\/span><\/b>\u200b<\/pre>\n
Step -2 : <\/strong>Enable the rd-kafka extension in below files<\/p>\n
C:\\wamp64\\bin\\php\\php8.0.26\\php.ini\u00a0 => Add\u00a0\u00a0<\/span>extension=php_rdkafka.dll<\/span><\/b>\u200b\r\n\r\nC:\\wamp64\\bin\\apache\\apache2.4.54.2\\bin\\php.ini\u00a0\u00a0 => Add\u00a0\u00a0<\/span>extension=php_rdkafka.dll<\/span><\/b>\u200b<\/pre>\n
Restart the server and verify with\u00a0<\/span><\/span>php<\/span><\/span><\/strong>\u00a0\u2013m<\/strong> , It should display the rdkafka extension in the list<\/span><\/span><\/p>\n
Install & configure the Kafka Web UI – Kafdrop\u200b<\/span><\/h2>\n
Requirement:\u00a0<\/span>\u200b<\/p>\n
\u00a0 \u00a0 Java 11 or newer<\/span>\u200b<\/p>\n
\u00a0 \u00a0Kafka (version 0.11.0 or newer)<\/span><\/p>\n
Reference to download the kafdrop : https:\/\/github.com\/obsidiandynamics\/kafdrop\/releases<\/span><\/span>\u200b<\/span><\/p>\n
Step -1:<\/strong> Download the jar files from\u00a0 below URL\u00a0<\/span><\/span><\/p>\n
curl -LO https:\/\/github.com\/obsidiandynamics\/kafdrop\/releases\/download\/3.30.0\/kafdrop-3.30.0.jar<\/span><\/span>\u200b<\/span><\/span><\/p>\n
Step -2:<\/strong> Rename the jar file<\/span><\/p>\n
mv kafdrop-3.30.0.jar kafdrop.jar<\/span><\/span>\u200b<\/span><\/span><\/p>\n
Step -3:<\/strong> Running\/Executing the jar file<\/span><\/span><\/span><\/p>\n
java –add-opens=<\/span>java.base<\/span>\/sun.nio.ch=ALL-UNNAMED \\<\/span>\u200b<\/p>\n
\u00a0\u00a0\u00a0 -jar \/c\/<\/span>kafdrop<\/span>\/kafdrop.jar \\<\/span>\u200b<\/p>\n
\u00a0\u00a0\u00a0 —<\/span>kafka.brokerConnect<\/span>=localhost:9092<\/span>\u200b<\/p>\n
Step -4: <\/strong>visit\u00a0http:\/\/localhost:9000\/<\/span><\/span>\u200b<\/span><\/strong><\/p>\n
It should display the UI like below,\u00a0 Make sure the Zookeeper & Kafka server is running.<\/p>\n
\"kafdrop\"<\/p>\n
Install & configure Drupal Module :<\/h2>\n
We can download the kafka drupal module from the https:\/\/www.drupal.org\/project\/kafka<\/span><\/span><\/a><\/p>\n
Use composer to download the module<\/p>\n
composer require ‘<\/span><\/span>drupal<\/span><\/span>\/<\/span><\/span>kafka<\/span><\/span>:^1.0@beta’<\/span><\/span>\u200b<\/span><\/p>\n
After download Enable the Module & update below setting in settings.php file to connect to the kafka server<\/p>\n
$settings['kafka'] = [\u200b\r\n'topic' => 'kafka-demo',\u200b\r\n'consumer' => [\u200b\r\n'brokers' => ['127.0.0.1:9092'],\u200b\r\n],\u200b\r\n'producer' => [\u200b\r\n'brokers' => ['127.0.0.1:9092'],\u200b\r\n],\u200b\r\n];<\/pre>\n
Now From the Drupal Menu you can view the Kafka Status Reports => Status report => Kafka<\/p>\n
It should be something like Below,<\/p>\n
\"drupal-kafka\"<\/p>\n
<\/h2>\n
That’s It,\u00a0 Congratulations !!<\/p>\n
<\/div>","protected":false},"excerpt":{"rendered":"
Apache Kafka is an open-source distributed event streaming platform. That uses Publish & subscribe mechanism to stream the records.\u200b Download & Installing the Apache Kafka: \u200bPrerequisite: You should have Java (JDK) installed on your Windows machine. Step -1: Download the binary version from the kafka official download page ( https:\/\/kafka.apache.org\/downloads ) Step -2: Unzip it […]<\/p>\n","protected":false},"author":1509,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":259,"footnotes":""},"categories":[3602],"tags":[5263,4862,5262,1604],"class_list":["post-57660","post","type-post","status-publish","format-standard","hentry","category-drupal","tag-apache-kafka","tag-drupal","tag-kafdrop","tag-kafka"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t