503s - where are we at

[ranguard]

We are getting 503s multiple times a day on https://metacpan.org/

What has been done:

• Add uWSGI stats monitor

• New DataDog account https://app.datadoghq.eu/

  • Send Fastly CDN logs for Web ( source: fastly service_name: Web )
  • Send Fastly CDN logs for API ( source: fastly service_name: API )
  • Send uWSGI logs
  • Add K8s DataDog agent to cluster
  • Create Public Dash board for web
  • Create Private Dashboard to help identify IPs to block

• https://metacpan.org/robots.txt - updated to list most bots and specify do not crawl

• VCL snipped deployed in Fastly to block based on user agents (since rolled back)

• MANY IPs and IP ranges (A very large number belonging to Alibaba.com) blocked using Fastly's IP Block list feature

Reached out to Fastly and signed a contract with them (they are giving us these services for free)

• Enable DDoS protection, but the traffic we get is not triggering it
• Trying to enable their WAF but having problems, this is with their support team currently

Our own blocking attemps:

• Initial Attempts to add to add Anubis to our cluster failed and we had to disable it - not sure how this would actually have worked with Fastly as our CDN Caching proxy in anycase.

Other actions:

• RSS fixed in the containers for uWSGI which seemed to solve memory issues...
• our web containers have been given even more memory as of 4pm on Sun 1 Jun - but they were close to memory usage, but not often getting OOMKilled and when they were it didn't match with our 503 spikes
• Review https://grafana.do.metacpan.org/ to see if any other info visible

Areas to explore

• Is our k8s nginx ingress somehow causing this issue under load?

  • Grafana ingress nginx logs shows we are getting upstream errors from nginx-ingress

• Grafana logs

[ranguard]

Full stack diagram

graph TD
    A[User] --> B[Fastly CDN]
    B --> C[Digital Ocean K8s Load Balancer]
    C --> D[Nginx Ingress Container]
    D --> E[uWSGI]
    E --> F[Plack Plugin]
    F --> G[Perl Code]
    G --> H[HTTP API Calls to fastapi.metacpan.org hosted outside of k8s, but also behind Fastly]
    
    subgraph Container
        E
        F
        G
    end

Loading

[ranguard]

At DCL's suggestion we have migrated from 3 x S-8vcpu-16gb to 3 x m-4vcpu-32gb which is an increase in cost but gives room to run more pods (we were more memory limited than CPU limited) - we have doubled the number of web pods running now we have this capacity

Nginx-ingress dashboard from @ssoriche - https://grafana.do.metacpan.org/d/k8s-nginx-ingress-prometheus-ng/kubernetes-nginx-ingress-controller?orgId=1&from=now-3h&to=now&timezone=browser&var-controller_class=$__all&var-namespace=$__all&var-ingress=$__all&var-pod=$__all&var-datasource=default&refresh=1m

The Fastly block of bots has been removed for now as traffic levels do not seem to be the main cause - though does feel like a contributing factor

@haarg is adding further debugging to the workers so can see what is happening when they get stuck

[ranguard]

metacpan/metacpan-web#3374 should increase our CDN caching and take the edge off a bit as will serve stale content to users whilst revalidates in the backend - NOT deployed yet

[ranguard]

WAF has been setup and lightly configured - blocks have been added to the Dashboard https://p.datadoghq.eu/sb/c2941b7b-37bb-11f0-94e3-32bf19abf102-744ae84f98611bfd6781365a482e2155

The user gets a 406 status

[ranguard]

Added custom 503 error page to MetaCPAN to point to this ticket and the dashboard

[ranguard]

Rate limiting for web reduced from 30 requests/s to 20 - over a 10 second window

[ranguard]

Thoughts:

API and Web have each backend configured as:

Maximum connections 200,
Connections (ms) 1000,
First byte (ms) 15000,
Between bytes (ms) 10000 (edited)

Is either a slow API response or volume of traffic meaning we are hitting these values

https://app.datadoghq.eu/logs?query=source%3Afastly%20%40service_name%3AAPI%20%40http[…]ng=true&from_ts=1749290894915&to_ts=1749291155156&live=false - shows slow (but ok) API responses - and there was a spate of web errors of 503 at that same time https://app.datadoghq.eu/logs?query=source%3Afastly%20%40service_name%3AWeb%20%40http[…]ng=true&from_ts=1749290894915&to_ts=1749291155156&live=false