Skip to content

Commit 95c3835

Browse files
author
Petter Graff
committed
Changed the order of steps for word count in Kafka
1 parent 99d7248 commit 95c3835

File tree

1 file changed

+43
-25
lines changed

1 file changed

+43
-25
lines changed

labs/06-Streaming/wordcount-kafka-streaming.md

Lines changed: 43 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -30,21 +30,9 @@ our `Topology` and hook it up to Kafka via our Kafka client properties, then `st
3030

3131
Lastly, for convenience, we wait for the user to hit enter to `close` the stream and exit.
3232

33-
### Build the streaming app
34-
35-
Now that we have coded our app, we need to build it. Fortunately, we can use `docker` for this so that we don't have to
36-
have `maven` and its prerequisites installed locally. Open a terminal in the lab's `wordcount-kafka-streaming` directory and issue
37-
the following command:
38-
39-
```shell
40-
$ docker run -it --rm -v "$(cd "$PWD/../.."; pwd)":/course-root -w "/course-root/$(basename $(cd "$PWD/.."; pwd))/$(basename "$PWD")" -v "$HOME/.m2/repository":/root/.m2/repository maven:3-jdk-11 ./mvnw clean package
41-
```
33+
## Start Kafka
4234

43-
On a windows machine, you have to replace the `$PWD` with the current directory and the `$HOME` with a directory where you have the `.m2` folder.
44-
45-
The command above will build and package our uber jar with the application and all of its dependencies.
46-
47-
## Run Kafka
35+
### Start containers
4836

4937
In order to run our app, we first need to run Kafka. First, ensure that you've shut down any prior docker containers.
5038

@@ -58,33 +46,56 @@ $ docker-compose -f kafka-streaming.yaml up
5846
You will see logs from all the containers that are launched as part of the solution. Once the terminal stops reflecting
5947
new output, the infrastructure is initialized.
6048

61-
## Start our streaming application
49+
### Create topics and start the consumer
6250

63-
Now, let's start our streaming application connecting to Kafka running in our Docker environment:
51+
In yet another terminal, change into the lab's root directory again and this time, start a bash session in the _kafka_
52+
container:
6453

6554
```shell
66-
$ docker run --network "$(cd .. && basename "$(pwd)" | tr '[:upper:]' '[:lower:]')_default" --rm -it -v "$PWD:/pwd" -w /pwd openjdk:11 java -jar target/wordcount-kafka-solution-*.jar
55+
$ docker-compose -f kafka-streaming.yaml exec kafka bash
6756
```
6857

69-
On a windows machine, you have to replace the `$PWD` with the current directory and the `$HOME` with a directory where you have the `.m2` folder.
58+
At the subsequent prompt, create the topics & start a Kafka console consumer:
59+
60+
```shell
61+
I have no name!@c07eea6aed61:/$ for it in input output; do kafka-topics.sh --create --bootstrap-server kafka:9092 --topic stream-$it; done
62+
I have no name!@c07eea6aed61:/$ kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic stream-output --from-beginning --property print.key=true
63+
```
7064

7165

72-
Now that our Kafka streaming application is running, it's time to feed it some input lines via the Kafka topic.
66+
## Streaming Application
7367

74-
In yet another terminal, change into the lab's root directory again and this time, start a bash session in the _kafka_
75-
container:
68+
### Build the streaming app
69+
70+
Now that we have coded our app, we need to build it. Fortunately, we can use `docker` for this so that we don't have to
71+
have `maven` and its prerequisites installed locally. Open a terminal in the lab's `wordcount-kafka-streaming` directory and issue
72+
the following command:
7673

7774
```shell
78-
$ docker-compose -f kafka-streaming.yaml exec kafka bash
75+
$ docker run -it --rm -v "$(cd "$PWD/../.."; pwd)":/course-root -w "/course-root/$(basename $(cd "$PWD/.."; pwd))/$(basename "$PWD")" -v "$HOME/.m2/repository":/root/.m2/repository maven:3-jdk-11 ./mvnw clean package
7976
```
8077

81-
At the subsequent prompt, create the topics & start a Kafka console consumer:
78+
On a windows machine, you have to replace the `$PWD` with the current directory and the `$HOME` with a directory where you have the `.m2` folder.
79+
80+
The command above will build and package our uber jar with the application and all of its dependencies.
81+
82+
### Start our streaming application
83+
84+
Now, let's start our streaming application connecting to Kafka running in our Docker environment:
8285

8386
```shell
84-
I have no name!@c07eea6aed61:/$ for it in input output; do kafka-topics.sh --create --bootstrap-server kafka:9092 --topic stream-$it; done
85-
I have no name!@c07eea6aed61:/$ kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic stream-output --from-beginning --property print.key=true
87+
$ docker run --network "$(cd .. && basename "$(pwd)" | tr '[:upper:]' '[:lower:]')_default" --rm -it -v "$PWD:/pwd" -w /pwd openjdk:11 java -jar target/wordcount-kafka-solution-*.jar
8688
```
8789

90+
On a windows machine, you have to replace the `$PWD` with the current directory and the `$HOME` with a directory where you have the `.m2` folder.
91+
92+
## Stream data
93+
94+
### Run a kafka-console-producer
95+
96+
Now that our Kafka streaming application is running, it's time to feed it some input lines via the Kafka topic.
97+
98+
8899
In another terminal, start a bash session and pump some lines into the kafka console producer:
89100

90101
```shell
@@ -107,12 +118,19 @@ me 20
107118
too 7
108119
```
109120

121+
### A larger data set
122+
110123
For fun, you can submit the full text of Leo Tolstoy's "War & Peace"!
111124

112125
```shell
113126
I have no name!@c07eea6aed61:/$ cat /lab-root/war-and-peace.txt | kafka-console-producer.sh --bootstrap-server kafka:9092 --topic stream-input
114127
```
115128

129+
130+
## Eplilogue
131+
132+
### Shutdown
133+
116134
In the lab's root directory, you can now bring down the cluster with the command
117135

118136
```shell

0 commit comments

Comments
 (0)