Quick Start for Confluent Platform

Use this quick start to get up and running locally with Confluent Platform and its main components using Docker containers.

Ready to build?

Sign up for Confluent Cloud with your cloud marketplace account and unlock $1000 in free credits: AWS Marketplace, Google Cloud Marketplace, or Microsoft Azure Marketplace.

In this quick start, you create Apache Kafka® topics, and use Kafka Connect to generate mock data to those topics. You then go to Confluent Control Center to monitor your data.

Prerequisites

To run this quick start, you will need Git, Docker and Docker Compose installed on a computer with a supported Operating System. Make sure Docker is running.

Tip

• These steps use Docker, but there are other ways to install Confluent Platform. For an example of how to download and configure a multi-broker cluster locally, without Docker, see Tutorial: Set Up a Multi-Broker Kafka Cluster. You can also use a TAR or ZIP Archive, or package managers like systemd, Kubernetes, and Ansible. For more information, see Install Confluent Platform On-Premises.

For full prerequisites, expand the Detailed prerequisites section that follows.

Detailed prerequisites

• A connection to the internet.

• Operating System currently supported by Confluent Platform.

  Note

  You can run the Confluent Platform Quick Start on Windows if you are running Docker Desktop for Windows on WSL 2. For more information, see How to Run Confluent on Windows in Minutes.

• Docker version 1.11 or later is installed and running.

• On Mac: Docker memory is allocated minimally at 6 GB (Mac). When using Docker Desktop for Mac, the default Docker memory allocation is 2 GB. Change the default allocation to 6 GB in the Docker Desktop app by navigating to Preferences > Resources > Advanced.

Step 1: Download and start Confluent Platform

In this step, you start by cloning a GitHub repository. This repository contains a Docker compose file and some required configuration files. The docker-compose.yml file sets ports and Docker environment variables such as the replication factor and listener properties for Confluent Platform and its components. To learn more about the settings in this file, see Docker Image Configuration Reference for Confluent Platform.

1. Clone the Control Center branch of the Confluent Platform all-in-one example repository, for example:

   git clone --branch control-center https://github.com/confluentinc/cp-all-in-one.git
   

2. Change to the cp-all-in-one directory:

   cd cp-all-in-one/cp-all-in-one
   

3. Check out the control-center branch:

   git checkout control-center
   

4. Start the Confluent Platform stack with the -d option to run in detached mode:

   docker compose up -d
   

   Note

   If you using an Docker Compose V1, you need to use a dash in the docker compose commands. For example:

   docker-compose up -d
   

   To learn more, see Migrate to Compose V2.

   Each Confluent Platform component starts in a separate container. Your output should resemble the following. Your output may vary slightly from these examples depending on your operating system.

   ✔ Container flink-jobmanager   Started         37.9s
   ✔ Container broker             Started         37.8s
   ✔ Container prometheus         Started         37.9s
   ✔ Container flink-taskmanager  Started         34.2s
   ✔ Container flink-sql-client   Started         34.3s
   ✔ Container alertmanager       Started         34.5s
   ✔ Container schema-registry    Started         34.3s
   ✔ Container rest-proxy         Started         34.7s
   ✔ Container connect            Started         34.8s
   ✔ Container ksqldb-server      Started         35.1s
   ✔ Container ksqldb-cli         Started         35.6s
   ✔ Container control-center     Started         35.9s
   ✔ Container ksql-datagen       Started         35.7s
   

5. Verify that the services are up and running:

   docker compose ps
   

   Your output should resemble:

   NAME                IMAGE                                                       COMMAND                  SERVICE             CREATED              STATUS              PORTS
   alertmanager        confluentinc/cp-enterprise-alertmanager:2.0.0               "alertmanager-start …"   alertmanager        2 minutes ago        Up About a minute   0.0.0.0:9093->9093/tcp
   broker              confluentinc/cp-server:7.9.0                                "/etc/confluent/dock…"   broker              2 minutes ago        Up About a minute   0.0.0.0:9092->9092/tcp, 0.0.0.0:9101->9101/tcp
   connect             cnfldemos/cp-server-connect-datagen:0.6.4-7.6.0             "/etc/confluent/dock…"   connect             2 minutes ago        Up About a minute   0.0.0.0:8083->8083/tcp, 9092/tcp
   control-center      confluentinc/cp-enterprise-control-center-next-gen:2.0.0    "bash -c 'dub templa…"   control-center      About a minute ago   Up About a minute   0.0.0.0:9021->9021/tcp
   flink-jobmanager    cnfldemos/flink-kafka:1.19.1-scala_2.12-java17              "/docker-entrypoint.…"   flink-jobmanager    2 minutes ago        Up About a minute   6123/tcp, 8081/tcp, 0.0.0.0:9081->9081/tcp
   flink-sql-client    cnfldemos/flink-sql-client-kafka:1.19.1-scala_2.12-java17   "/docker-entrypoint.…"   flink-sql-client    2 minutes ago        Up About a minute   6123/tcp, 8081/tcp
   flink-taskmanager   cnfldemos/flink-kafka:1.19.1-scala_2.12-java17              "/docker-entrypoint.…"   flink-taskmanager   2 minutes ago        Up About a minute   6123/tcp, 8081/tcp
   ksql-datagen        confluentinc/ksqldb-examples:7.9.0                          "bash -c 'echo Waiti…"   ksql-datagen        About a minute ago   Up About a minute
   ksqldb-cli          confluentinc/cp-ksqldb-cli:7.9.0                            "/bin/sh"                ksqldb-cli          About a minute ago   Up About a minute
   ksqldb-server       confluentinc/cp-ksqldb-server:7.9.0                         "/etc/confluent/dock…"   ksqldb-server       About a minute ago   Up About a minute   0.0.0.0:8088->8088/tcp
   prometheus          confluentinc/cp-enterprise-prometheus:2.0.0                 "prometheus-start --…"   prometheus          2 minutes ago        Up About a minute   0.0.0.0:9090->9090/tcp
   rest-proxy          confluentinc/cp-kafka-rest:7.9.0                            "/etc/confluent/dock…"   rest-proxy          2 minutes ago        Up About a minute   0.0.0.0:8082->8082/tcp
   schema-registry     confluentinc/cp-schema-registry:7.9.0                       "/etc/confluent/dock…"   schema-registry     2 minutes ago        Up About a minute   0.0.0.0:8081->8081/tcp
   

   After a few minutes, if the state of any component isn’t Up, run the docker-compose up -d command again, or try docker-compose restart <image-name>, for example:

   docker compose restart control-center
   

Step 2: Create Kafka topics for storing your data

In Confluent Platform, real-time streaming events are stored in a topic, which is an append-only log, and the fundamental unit of organization for Kafka. To learn more about Kafka basics, see Kafka Introduction.

In this step, you create two topics by using Control Center for Confluent Platform. Control Center provides the features for building and monitoring production data pipelines and event streaming applications.

The topics are named pageviews and users. In later steps, you create connectors that produce data to these topics.

Create the pageviews topic

Confluent Control Center enables creating topics in the UI with a few clicks.

1. Navigate to Control Center at http://localhost:9021. It takes a few minutes for Control Center to start and load. If needed, refresh your browser until it loads.

2. Click the controlcenter.cluster tile.

   

3. In the navigation menu, click Topics to open the topics list. Click + Add topic to start creating the pageviews topic.

   

4. In the Topic name field, enter pageviews and click Create with defaults. Topic names are case-sensitive.

   

Create the users topic

Repeat the previous steps to create the users topic.

1. In the navigation menu, click Topics to open the topics list. Click + Add topic to start creating the users topic.

2. In the Topic name field, enter users and click Create with defaults.

3. You can optionally inspect a topic. On the users page, click Configuration to see details about the users topic.

   

Step 3: Generate mock data

In Confluent Platform, you get events from an external source by using Kafka Connect. Connectors enable you to stream large volumes of data to and from your Kafka cluster. Confluent publishes many connectors for integrating with external systems, like MongoDB and Elasticsearch. For more information, see the Kafka Connect Overview page.

In this step, you run the Datagen Source Connector to generate mock data. The mock data is stored in the pageviews and users topics that you created previously. To learn more about installing connectors, see Install Self-Managed Connectors.

1. In the navigation menu, click Connect.

2. Click the connect-default cluster in the Connect clusters list.

3. Click Add connector to start creating a connector for pageviews data.

4. Select the DatagenConnector tile.

   Tip

   To see source connectors only, click Filter by category and select Sources.

5. In the Name field, enter datagen-pageviews as the name of the connector.

6. Enter the following configuration values in the following sections:

   Common section:

   • Key converter class: org.apache.kafka.connect.storage.StringConverter

   General section:

   • kafka.topic: Choose pageviews from the dropdown menu
   • max.interval: 100
   • quickstart: pageviews

7. Click Next to review the connector configuration. When you’re satisfied with the settings, click Launch.

   

Run a second instance of the Datagen Source connector connector to produce mock data to the users topic.

1. In the navigation menu, click Connect.

2. In the Connect clusters list, click connect-default.

3. Click Add connector.

4. Select the DatagenConnector tile.

5. In the Name field, enter datagen-users as the name of the connector.

6. Enter the following configuration values:

   Common section:

   • Key converter class: org.apache.kafka.connect.storage.StringConverter

   General section:

   • kafka.topic: Choose users from the dropdown menu
   • max.interval: 1000
   • quickstart: users

7. Click Next to review the connector configuration. When you’re satisfied with the settings, click Launch.

8. In the navigation menu, click Topics and in the list, click users.

9. Click Messages to confirm that the datagen-users connector is producing data to the users topic.

   

Inspect the schema of a topic

By default, the Datagen Source Connector produces data in Avro format, which defines the schemas of pageviews and users messages.

Schema Registry ensures that messages sent to your cluster have the correct schema. For more information, see Schema Registry Documentation.

1. In the navigation menu, click Topics, and in the topic list, click pageviews.

2. Click Schema to inspect the Avro schema that applies to pageviews message values.

   Your output should resemble:

   

Step 4: Uninstall and clean up

If you’re done working with Confluent Platform, you can stop and remove the Docker containers and images.

1. Run the following command to stop the Docker containers for Confluent:

   docker-compose stop
   

2. After stopping the Docker containers, run the following commands to prune the Docker system. Running these commands deletes containers, networks, volumes, and images, freeing up disk space:

   docker system prune -a --volumes --filter "label=io.confluent.docker"
   

   For more information, refer to the official Docker documentation.

Related content

• To download an automated version of this quick start, see the Quick Start on GitHub
• To configure and run a multi-broker cluster without Docker, see Tutorial: Set Up a Multi-Broker Kafka Cluster
• To learn how to develop with Confluent Platform, see Confluent Developer
• For training and certification guidance, including resources and access to hands-on training and certification exams, see Confluent Education
• To try out basic Kafka, Kafka Streams, and ksqlDB tutorials with step-by-step instructions, see Kafka Tutorials
• To learn how to build stream processing applications in Java or Scala, see Kafka Streams documentation
• To learn how to read and write data to and from Kafka using programming languages such as Go, Python, .NET, C/C++, see Kafka Clients documentation