Pieces and notes for local development

README.md

@@ -22,15 +22,53 @@ You are welcome to fork it and make it your own. OSS FTW 💚
 
 ## How is it going (a.k.a. Roadmap)
 
-- [x] Static backend, 1 day stale, stale on error, x-headers - `46` lines of VCL - [Initial commit](https://github.com/thechangelog/pipedream/commit/17d3899a52d9dc887efd7f49de92b24249431234)
-- [x] Dynamic backend, cache-status header - `60` lines of VCL - [PR #1](https://github.com/thechangelog/pipedream/pull/1)
-- [x] Add tests - `60` lines of VCL - [PR #3](https://github.com/thechangelog/pipedream/pull/3)
+- [x] Static backend, 1 day stale, stale on error, x-headers - [Initial commit](https://github.com/thechangelog/pipedream/commit/17d3899a52d9dc887efd7f49de92b24249431234)
+- [x] Dynamic backend, cache-status header - [PR #1](https://github.com/thechangelog/pipedream/pull/1)
+- [x] Add tests - [PR #3](https://github.com/thechangelog/pipedream/pull/3)
+- [x] Make it easy to develop locally - [PR #7](https://github.com/thechangelog/pipedream/pull/7)
 - [ ] Add feeds backend: /feed -> http://feeds.changelog.place/feed.xml
 - [ ] Send logs to Honeycomb.io (same structure as Fastly logs)
 - [ ] Send logs to S3 (for stats)
 - [ ] Implement purge across all app instances (Fly.io machines)
 - [ ] Add edge redirects from [Fastly service](https://manage.fastly.com/configure/services/7gKbcKSKGDyqU7IuDr43eG)
 
+## Local development and testing
+
+While it's fun watching other people experiment with digital resin (varnish
+😂), it's a whole lot more fun when you can repeat those experiments yourself,
+understand more how it works, and make your own modifications.
+
+You can find some instructions and notes for kicking the tires and [developing
+& testing this locally](docs/local_dev.md).
+
+A few other commands that you may be interested in:
+
+```bash
+# Requires https://github.com/casey/just
+just
+Available recipes:
+    report     # Open the test report
+    test *ARGS # Run the tests
+
+# Run the tests
+just test
+
+# There is a script which precedes `just`
+# Yes, we should combine them. Contributions welcome!
+./run
+First argument must be one of the following
+deploy          → deploys to Fly.io
+world-scale     → makes it World Scale™
+small-scale     → makes it Small Scale™
+http-detailed   → shows detailed http response
+http-measure    → measures http response times
+http-profile    → profiles http responses
+demo-2024-01-26 → runs through the first demo
+demo-2024-06-21 → runs through the second demo
+
+💡 All following arguments are passed to the command
+```
+
 ## How can you help
 
 If you have any ideas on how to improve this, please open an issue or go

dev-nginx.Dockerfile

@@ -0,0 +1,29 @@
+# https://hub.docker.com/_/nginx
+FROM nginx:latest
+
+ENV NGINX_LB_HOST="pipedream.changelog.com nginx"
+ENV NGINX_LB_PORT=80
+ENV NGINX_LB_SSL_PORT=443
+ENV NGINX_LB_PROXY_PASS="http://172.17.0.1:9000"
+
+ENV NGINX_BACK_PORT=4000
+ENV NGINX_BACK_HOST="changelog-2024-01-12.fly.dev"
+ENV NGINX_BACK_REV_PROXY_LOC_MATCH="/feed"
+ENV NGINX_BACK_REV_PROXY="http://feeds.changelog.place/feed.xml"
+
+COPY dev-nginx.conf.template /etc/nginx/templates/default.conf.template
+
+# Generate (weak/fast) self-signed cert for useless mocking
+# INSECURE: Bad idea to generate sensitive cryptographic keys and store in container image
+# If there is a better solution for development, please open an issue to discuss it.
+RUN apt update && apt -y install openssl
+ENV SSL_KEY="/etc/nginx/ssl/www.example.com.key"
+ENV SSL_CSR="/etc/nginx/ssl/www.example.com.csr"
+ENV SSL_REQ_COUNTRY="US"
+ENV SSL_REQ_STATE="Texas"
+ENV SSL_REQ_LOCALITY="Austin"
+ENV SSL_REQ_ORGANIZATION="Example Company"
+ENV SSL_REQ_ORGANIZATIONALUNIT="Example Department"
+ENV SSL_REQ_COMMONNAME="example.com"
+ENV SSL_REQ_EMAIL="[email protected]"
+RUN /usr/bin/openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/server.key -out /etc/nginx/server.crt -subj "/C=${SSL_REQ_COUNTRY}/ST=${SSL_REQ_STATE}/L=${SSL_REQ_LOCALITY}/O=${SSL_REQ_ORGANIZATION}/OU=${SSL_REQ_ORGANIZATIONALUNIT}/CN=${SSL_REQ_COMMONNAME}/emailAddress=${SSL_REQ_EMAIL}"

dev-nginx.conf.template

@@ -0,0 +1,65 @@
+# Mock HTTPS load balancer in front of Varnish
+server {
+    server_name ${NGINX_LB_HOST};
+
+    listen       ${NGINX_LB_PORT} default_server;
+    listen  [::]:${NGINX_LB_PORT} default_server;
+    listen       ${NGINX_LB_SSL_PORT} ssl;
+    listen  [::]:${NGINX_LB_SSL_PORT} ssl;
+
+    http2 on;
+
+    ssl_certificate      server.crt;
+    ssl_certificate_key  server.key;
+
+    # reverse proxy to Varnish
+    location  / {
+       proxy_pass  ${NGINX_LB_PROXY_PASS};
+    }
+}
+
+# Mock Backend
+server {
+    listen       ${NGINX_BACK_PORT};
+    listen  [::]:${NGINX_BACK_PORT};
+    server_name  ${NGINX_BACK_HOST};
+
+    root /usr/share/nginx/html;
+
+    # health check
+    location /health {
+        return 200 healthy;
+    }
+    location /varnish_health {
+        return 200 healthy;
+    }
+
+    # homepage
+    location / {
+        # index index.html index.htm;
+        return 200 homepage;
+    }
+
+    # feed
+    location /podcast/feed {
+        # rewrite ^(.*)$ /index.html last;
+        return 200 feed”;
+    }
+
+    # admin
+    location /admin {
+        absolute_redirect off;
+        return 302 / ;
+    }
+
+    # redirect server error pages to the static page /50x.html
+    error_page   500 502 503 504  /50x.html;
+    location = /50x.html {
+        root   /usr/share/nginx/html;
+    }
+
+    # reverse proxy somewhere else
+    location ${NGINX_BACK_REV_PROXY_LOC_MATCH} {
+       proxy_pass ${NGINX_BACK_REV_PROXY};
+    }
+}

dev-varnish.Dockerfile

@@ -0,0 +1,20 @@
+# https://hub.docker.com/_/varnish
+FROM varnish:7.4.3
+ENV VARNISH_HTTP_PORT=9000
+COPY default.vcl /etc/varnish/default.vcl
+
+# Hack up a local docker container to fake DNS to run locally
+# If there is a better solution for development, please open an issue to discuss it.
+USER root
+RUN apt update && apt -y install curl net-tools vim iproute2 dnsutils procps iputils-ping
+RUN apt -y install dnsmasq
+ENV VARNISH_BACKEND_DOMAIN="changelog-2024-01-12.internal"
+ENV VARNISH_BACKEND_IPV4="172.18.0.2"
+ENV VARNISH_BACKEND_IPV6="fd20:b007:398e::2"
+RUN echo 'listen-address=127.0.0.1' >> /etc/dnsmasq.conf
+RUN echo 'user=root' >> /etc/dnsmasq.conf
+RUN echo 'server=8.8.8.8' >> /etc/dnsmasq.conf
+RUN echo "address=/${VARNISH_BACKEND_DOMAIN}/${VARNISH_BACKEND_IPV4}" >> /etc/dnsmasq.conf
+RUN echo "address=/${VARNISH_BACKEND_DOMAIN}/${VARNISH_BACKEND_IPV6}" >> /etc/dnsmasq.conf
+RUN dnsmasq --test
+USER varnish

docs/README.md

@@ -0,0 +1,10 @@
+# Docs
+
+We aim to keep docs lean & simple. GitHub is the default front-end - search
+works well enough. Your code editor is the other option.
+
+Read next: [Developing & experimenting locally using Docker](local_dev.md)
+
+---
+
+If something is missing, feel free to open a GitHub issue. Pull requests always welcome 👍