Running Kafka for a streaming collection service can feel somewhat opaque at times, this is why I was thrilled to find the Kafka Web Console project on Github yesterday. This Scala application can be easily downloaded and installed with a couple steps. An included web server can then be launched to serve it up quickly. Here’s how to do all that.
For a quick intro to what the web console does, see my video first. Instructions for getting started follow below, including the quick video.
Kafka Web Console Project – Download
The main repository for this project is available on Github (claudemamo/kafka-web-console). However, I wanted the ability to add and remove Kafka topics so I use a forked repository that has a specific branch with those capabilities. (These are planned to be added to the main project but have not been yet.)
Download the ZIP archive file and unzip.
Before doing anything further, we need another application to build and launch the console.
Download Play Framework
A program called Play with Activator is used to build and launch the web console. It’s a Java app for launching Scala apps.
Download it here, unzip it and add it to the system path so you can execute the activator command that it provides.
Build/Launch
Now back to the Kafka web console code. Enter the top level directory and execute the Play Activator start command (with one special option):
cd kafka-web-console-topic-add-remove activator start -DapplyEvolutions.default=true [info] Loading project definition from /home/demo/src/tmp/kafka-web-console-topic-add-remove/project [info] Set current project to kafka-web-console (in build file:/home/demo/src/tmp/kafka-web-console-topic-add-remove/) (Starting server. Type Ctrl+D to exit logs, the server will remain in background) Play server process ID is 8528 [info] play - database [default] connected at jdbc:h2:file:play [info] play - Starting application default Akka system. [info] play - Application started (Prod) [info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
The first time it runs will take some time to build the source and then launch the web server.
In the last line, above, you see it launches by default on port 9000.
Configuring the Console
Step 1 – Register your Zookeeper instance
Now you can launch the web console and start using the application. Step 1 figure shows the basic form for your Zookeeper configuration details. This is the only setup step required to get access to your Kafka brokers and topics.
The remained of the steps/figures are just showing the different screens. See the video for watching it in action.
Step 2 – Brokers tab shows list of Kafka Brokers
Step 3 – Topics tab lists the Kafka topics. Note that this forked version provides the red “delete” button for each topic, as well as the additional “create topic” tab above.
Step 4 – Selecting a topic allows you to see an active topic stream.
- Geography + Data - July 15, 2021
- DIY Battery – Weekend Project – Aluminum + Bleach? - January 17, 2021
- It’s all about the ecosystem – build and nurture yours - May 1, 2020
- Learnings from TigerGraph and Expero webinar - April 1, 2020
- 4 Webinars This Week – GPU, 5G, graph analytics, cloud - March 30, 2020
- Diving into #NoSQL from the SQL Empire … - February 28, 2017
- VID: Solving Performance Problems on Hadoop - July 5, 2016
- Storing Zeppelin Notebooks in AWS S3 Buckets - June 7, 2016
- VirtualBox extension pack update on OS X - April 11, 2016
- Zeppelin Notebook Quick Start on OSX v0.5.6 - April 4, 2016
Repository now seems to be archived 🙁
I’m looking for alternatives which are actively supported that let me view the Kafka cluster and browse topics. So far the only tool (that actually appears to be maintained) is Kafdrop 3 – https://github.com/obsidiandynamics/kafdrop. Is anyone aware of any others.
It’s been a while since I’ve tried but have you tried some of Confluent’s tooling via their control center, it might help you.
I am unable to run with chroot specified as /kafka for zookeeper. The console keeps saying connecting to the Zookeeper, but never connects. Does the chroot feature really work? Pls let me know.
thanks,
Venkat
Venkat – Sorry, I just saw this now for some reason. I haven’t touched Zeppelin in a while, but I’ll have to try it again and see if I can reproduce your problem.
I downloaded the version mentioned earlier on this page. Everything works great EXCEPT that I could never see a live feed under Topic feed tab. I tried to diagnose with both firebug on the client side and elcipse/typesafe activator on the server side. Nothing stood out for a possible cause. Is this related to timeout? Please kindly respond.
It was sporadic for me too!
After a restart I had to run with another parameter as well:
activator start -DapplyEvolutions.default=true -DapplyDownEvolutions.default=trueHi Tyler,
Thanks for getting back to me. I got it working in the end by putting the property before the start command:
activator -DapplyEvolutions.default=true start
So the ordering of the commands made a different to pick up the evolutions property. Hope this helps others.
Excellent, thanks for sharing. I assume it’s an operating system difference. Which one you using? I’m on OSX.
I got that error at first, but then ran it with the applyEvolutions option and it cleared up:
activator start -DapplyEvolutions.default=trueNot sure about all those exceptions though, maybe this will help. Let me know! Thanks for reading.
Hi,
I tried this but the server wouldn’t start. From the logs it has a warning:
2015-03-27 21:25:27,312 - [WARN] - from play in main
Your production database [default] needs evolutions!
It also outputs some exceptions:
CREATE TABLE groups (@6lip8i8m2: Database 'default' needs evolution!id LONG,
name VARCHAR,
at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1$$anonfun$apply$1.apply$mcV$sp(Evolutions.scala:484)
PRIMARY KEY (id)
at play.api.db.evolutions.EvolutionsPlugin.withLock(Evolutions.scala:507)
);
at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.scala:461)
at play.api.db.evolutions.EvolutionsPlugin$$anonfun$onStart$1.apply(Evolutions.scala:459)
CREATE TABLE status (
at scala.collection.immutable.List.foreach(List.scala:318)
id LONG,
at play.api.db.evolutions.EvolutionsPlugin.onStart(Evolutions.scala:459)
name VARCHAR,
at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88)
PRIMARY KEY (id)
at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88)
);
at scala.collection.immutable.List.foreach(List.scala:318)
at play.api.Play$$anonfun$start$1.apply$mcV$sp(Play.scala:88)
Any ideas?
Thanks