|
1 | 1 | # kafka-nodejs-tutorial
|
2 |
| -A sample application using Kafka, Node.js and Docker |
| 2 | +# A sample application using Kafka, Node.js and Docker |
| 3 | + |
| 4 | +## 1. About Kafka Streaming |
| 5 | + |
| 6 | +Below is the discription from the official Kafka web site. |
| 7 | + |
| 8 | +Apache Kafka® is a distributed streaming platform. |
| 9 | + |
| 10 | +It publishes and subscribes to streams of records, similar to a message queue or enterprise messaging syste |
| 11 | + |
| 12 | +Kafka is generally used for two broad classes of applications: |
| 13 | + |
| 14 | +Building real-time streaming data pipelines that reliably get data between systems or applications |
| 15 | +Building real-time streaming applications that transform or react to the streams of data |
| 16 | + |
| 17 | +To learn more about Kafka, please visit [Kafka official web site](https://kafka.apache.org/) |
| 18 | + |
| 19 | +Kafka is build in Java, it does provide APIs from many other lanhuages. |
| 20 | + |
| 21 | +Since I am a Javascript developer, I decided to use Node.js for this sample application. |
| 22 | + |
| 23 | +There are a few libraries for Node.js, some popular libraries include: |
| 24 | + |
| 25 | +* [node-rdkafka](https://github.com/Blizzard/node-rdkafka) |
| 26 | + |
| 27 | +* [Kafka-node](https://www.npmjs.com/package/kafka-node) |
| 28 | + |
| 29 | +* [KafkaJS](https://www.npmjs.com/package/kafkajs) |
| 30 | + |
| 31 | +I use KafkaJs in this example. The main reason is it has better documentation then the other two. |
| 32 | + |
| 33 | + |
| 34 | + |
| 35 | +## 2. Use case - track travellers who may infected by Coronarirus virus |
| 36 | + |
| 37 | +We are going to build an applicaiton, to track petential travellers who may have been infected by Coronavirus. |
| 38 | + |
| 39 | +1. We check travellers temperature at major airports. |
| 40 | +2. Temperature and traveller's personal infomration are sent to the Kafka queue (as producer). Note every airport is a prodcuer. |
| 41 | +3. We have another app check every message sent to the queue (near real time), and alert is triggered if we a passenger has higher temperature and has overseas travel history. |
| 42 | + |
| 43 | + |
| 44 | + |
| 45 | +## 3. Build local Kafka environemnt using Docker and Container |
| 46 | + |
| 47 | +In order to run the application locally, we need at least two docker imagers |
| 48 | + |
| 49 | +* zookeeper |
| 50 | +* Kafka Broker |
| 51 | + |
| 52 | +In the real envirnment, you will need multiple Kafka brokers, to form a cluster. |
| 53 | + |
| 54 | +In stead of build my own docker imagers, I cloned the code from https://github.com/wurstmeister/kafka-docker, which give me everything I need to run Kafka on my local container |
| 55 | + |
| 56 | + |
| 57 | + |
| 58 | + |
| 59 | + |
| 60 | +```yml |
| 61 | +version: "2" |
| 62 | +services: |
| 63 | + zookeeper: |
| 64 | + image: wurstmeister/zookeeper |
| 65 | + ports: |
| 66 | + - "2181:2181" |
| 67 | + kafka: |
| 68 | + build: . |
| 69 | + ports: |
| 70 | + - "9092:9092" |
| 71 | + environment: |
| 72 | + KAFKA_ADVERTISED_HOST_NAME: localhost |
| 73 | + KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 |
| 74 | + KAFKA_CREATE_TOPICS: "flight:1:1" |
| 75 | + volumes: |
| 76 | + - /var/run/docker.sock:/var/run/docker.sock |
| 77 | + |
| 78 | +``` |
| 79 | + |
| 80 | + |
| 81 | + |
| 82 | + |
| 83 | + |
| 84 | +Reference: https://github.com/wurstmeister/kafka-docker |
| 85 | + |
| 86 | + |
| 87 | +## 4. Run the application |
| 88 | + |
| 89 | + |
| 90 | + |
| 91 | +## References |
| 92 | + |
0 commit comments