How to Set Up and Run Kafka on Kubernetes
Apache Kafka is a leading open-source distributed streaming platform first developed at LinkedIn. It consists of several APIs such as the Producer, the Consumer, the Connector and the Streams. Together, those systems act as high-throughput, low-latency platforms for handling real-time data. This is why Kafka is preferred among several of the top-tier tech companies such as Uber, Zalando and AirBnB.
Quite often, we would like to deploy a fully-fledged Kafka cluster in Kubernetes, just because we have a collection of microservices and we need a resilient message broker in the center. We also want to spread the Kafka instances across nodes, to minimize the impact of a failure.
In this tutorial, we are going to see an example Kafka deployment within Platform9 Free Tier Kubernetes platform, backed up by some DigitalOcean droplets. Let’s get started.
Setting Up the Platform9 Free Tier Cluster
Below are the brief instructions to get you up and running with a working Kubernetes Cluster from Platform9:
- Signup with Platform9.
- Click the Create Cluster button and inspect the instructions. We need a server to host the Cluster.
- Create a few Droplets with at least 3GB RAM and 2 vCPUs. Follow the instructions to install the
pf9clitool and prepping the nodes.
$ bash <(curl -sL http://pf9.io/get_cli) $ pf9ctl cluster prep-node -i
- Switch to the Platform9 UI and click the refresh button. You should see the new nodes in the list. Designate the first node as master and the rest as workers.