Skip to content

Cea-Learning/data-tools

Repository files navigation

Deploying Kafka Using Strimzi

  1. Create Namespace if not already existing kubectl create namespace <namespace>

  2. Run the following command to create strimzi kafka cluster operator helm install strimzi-cluster-operator oci://quay.io/strimzi-helm/strimzi-kafka-operator -n <namespace>

  3. Run the kafka-values.yml ```kubectl apply -f kafka-values.yaml -n ``

  4. Run the kafka-connect-values.yml ```kubectl apply -f kafka-connect-values.yaml -n ``

  5. Create push secret kubectl create secret docker-registry <secret-name> \ --docker-username=<your-dockerhub-username> \ --docker-password=<your-dockerhub-password> \ --docker-email=<your-email>

  6. Get Artifact for kafka connect

    You can get them from maven repositories or other open source repositories

  7. Create and run custom KafkaConnectors resource in kafka-connectors-values.yaml kubectl apply -f kafka-connectors-values.yaml -n <namespace>

  • check if the custom connectors have been created kubectl get kctr --selector strimzi.io/cluster=<kafka-connect-cluster-name> This process is used for kafka connectors cluster that has strimzi.io/use-connector-resources: "true".

The other cluster would use the rest api to set the values of the connect

  1. Run Kubectl to confirm the data kubectl exec <my_kafka_cluster>-kafka-0 -i -t -- bin/kafka-console-consumer.sh --bootstrap-server <my_kafka_cluster>-kafka-bootstrap.NAMESPACE.svc:9092 --topic my-topic --from-beginning

Deploying Postgres with TLS

Generating Self Signed TLS for Development Purposes

openssl req -new -key tls.key -out request.csr
openssl x509 -req -days 365 -in request.csr -signkey tls.key -out tls.crt
rm request.csr
kubectl create secret tls my-tls-secret --cert=/path/to/cert.crt --key=/path/to/key.key

Processing Yaml Files

The project include a process-yaml-files.sh command that fills up secrets from .env file into the respective places. This should only be used in development purposes.

About

version control of data engineering tools

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published