[Solved] Forward Auth in Traefik / Docker environment not working

[agrimpelhuber]

Describe your question

Trying to set up Forward Auth in front of a simple web application (Nginx+php-fpm in a container, for testing purposes) within a Docker / Traefik environment, description below. Whatever I try, no matter how much documentation or code I read, I end up with 2 different scenarios, depending on the setup.

Single Application setup: At best an infinite loop on Authentik's login form
Domain Application setup: HTTP code 400 and

{
	"Message": "no app for hostname",
	"Host": "auth.mydomain.com",
	"Detail": "Check the outpost settings and make sure 'auth.mydomain.com' is included."
}

EDIT: The most likely cause for this problem is the routing in Traefik. The above message does not come from the Forward Auth proxy outpost, but rather from Authentik itself, running under the same domain auth.mydomain.com as the proxy does . It's very likely you need to adjust route priorities in Traefic. Specifics in the comments.

---- Original post ----

Assuming that I'm doing something wrong, I give as much information as possible below for someone smart to spot the probably simple / stupid / obvious mistake I'm making. If it's not that obvious, I'll provide more info straight away.

Relevant infos
Setup I'm starting with:

• Web application (nginx + php-fpm in container, listening on port 80 in internal docker network)
• Traefik, also running in a container, routes HTTPS traffic to the application on subdomain app.mydomain.com
• The same applies to Authentik. It can be reached on auth.mydomain.com from the internet
• All 3 containers are connected to a bridged docker network called "common-bridge" (172.128.0.0/16)
• All components are running smoothly prior to my attempt to implement forward auth

Trying to set up Forward Auth with the following steps / configuration:

1. Application in Authentik:

Name: Forward Auth
Slug: forward-auth
Provider: Forward Auth
Policy: ANY [...]
UI settings / Launch URL: https://app.mydomain.com

2. Provider in Authentik:

Type: Proxy Provider
Name: Forward Auth
Authorization flow: default-provider-authorization-explicit-consent
[Type:] Forward Auth (domain level)
Authentication URL: https://auth.mydomain.com
Cookie domain: mydomain.com

3. Outpost in Authentik:

Name: Forward Auth
Type: Proxy
Integration: Local Docker connection
Applications: Forward Auth (https://auth.mydomain.com)
Configuration (leaving out the standard Kubernetes stuff):
    log_level: trace
    docker_labels:
      traefik.http.routers.ak-outpost-b650b531e5384d3ba8e8e605eac1938d-router.entrypoints: websecure
      traefik.http.routers.ak-outpost-b650b531e5384d3ba8e8e605eac1938d-router.tls.certresolver: myresolver
    authentik_host: https://auth.mydomain.com
    docker_network: common-bridge
    container_image: null
    docker_map_ports: false
    authentik_host_browser: ""
    object_naming_template: authentik-outpost-%(name)s
    authentik_host_insecure: false

4. Traefik middleware configuration in docker-compose of web app:

labels:
- "traefik.http.routers.nginx-2.middlewares=authentik-php@docker"
- "traefik.http.middlewares.authentik-php.forwardauth.address=http://authentik-outpost-forward-auth:9000/akprox/auth/traefik"
- "traefik.http.middlewares.authentik-php.forwardauth.trustForwardHeader=true"
- "traefik.http.middlewares.authentik-php.forwardauth.authResponseHeaders=X-authentik-username,X-authentik-groups,X-authentik-email,X-authentik-name,X-authentik-uid,X-authentik-jwt,X-authentik-meta-jwks,X-authentik-meta-outpost,X-authentik-meta-provider,X-authentik-meta-app,X-authentik-meta-version"

5. On startup, everything seems to be fine:

• Authentik brings up the "authentik-outpost-forward-auth" proxy container
• The container registers in Traefik with the rule Host(`auth.mydomain.com`) && PathPrefix(`/akprox`)
• No errors in the logs files, as far as I can see (log level "trace")
• All the (internal) network connections seem to be working

However, when I call https://app.mydomain.com, I'm automatically redirected (even when not authorized) to https://auth.mydomain.com/akprox/start with the "no app for hostname" output.

Screenshots
n/a

Logs
Output of container "authentik-outpost-forward-auth" (trace):

{"app":"Forward Auth","event":"Found app based on cookie domain","host":"app.mydomain.com","level":"debug","logger":"authentik.outpost.proxyv2","timestamp":"2022-01-28T15:22:38Z"}
{"event":"passing to application mux","host":"app.mydomain.com","level":"trace","logger":"authentik.outpost.proxyv2","timestamp":"2022-01-28T15:22:38Z"}
{"event":"tracing headers for debug","header":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["de-DE,de;q=0.9,en;q=0.8,en-US;q=0.7"],"Dnt":["1"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"97\", \"Chromium\";v=\"97\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"],"X-Forwarded-For":["xx.180.177.202"],"X-Forwarded-Host":["app.mydomain.com"],"X-Forwarded-Method":["GET"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Forwarded-Server":["b7eb2436cf7c"],"X-Forwarded-Uri":["/handler.php"],"X-Real-Ip":["xx.180.177.202"]},"level":"trace","logger":"authentik.outpost.proxyv2.application","name":"Forward Auth","timestamp":"2022-01-28T15:22:38Z"}
{"event":"traefik forwarded url","level":"trace","logger":"authentik.outpost.proxyv2.application","name":"Forward Auth","timestamp":"2022-01-28T15:22:38Z","url":"https://app.mydomain.com/handler.php"}
{"event":"Matching URL against allow list","level":"trace","logger":"authentik.outpost.proxyv2.application","name":"Forward Auth","regex":"https://app.mydomain.com/handler.php","timestamp":"2022-01-28T15:22:38Z","url":"https://app.mydomain.com/handler.php"}
{"event":"Redirecting to login","level":"debug","logger":"authentik.outpost.proxyv2.application","name":"Forward Auth","timestamp":"2022-01-28T15:22:38Z","url":"https://auth.mydomain.com/akprox/start"}
{"event":"/akprox/auth/traefik","host":"app.mydomain.com","level":"info","logger":"authentik.outpost.proxyv2.application","method":"GET","name":"Forward Auth","remote":"172.128.0.10:41948","request_protocol":"HTTP/1.1","request_useragent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36","runtime":"0.960","size":78,"status":307,"timestamp":"2022-01-28T15:22:38Z","upstream":""}

What is weird: The call to https://app.mydomain.com/handler.php does not even seem to reach the authentic core.

Version and Deployment (please complete the following information):

• authentik version: 2022.1.3
• traefik version: 2.6.0
• docker version: current
• Deployment: docker-compose (authentik, traefik in different containers) + portainer
• Single server, single domain and subdomains for different applications

Additional context
n/a

[Jafner]

+1 for this.
I would also like to use Authentik with Traefik ForwardAuth. And I am encountering the same issue.

[sungyoonc]

I'm having the same issue. It's seems like the outpost has problem with the connection through traefik. When I directly connect to the outpost it doesn't have any issue.

[agrimpelhuber]

Hi! Thanks for the feedback. Don't know when the intergration was last used / tested successfully, but there is a slight chance that a recent update of Traefik might have caused this. @winterolym and @Jafner , can you state the versions of Traefik and Authentik as well as the type of installation (assuming docker) you are running, to help someone more knowledgeable than us to find a clue?

In order to help debugging the Traefik side, I have set up another Nginx/PHP-fpm container as a fake forward auth proxy / traefik middleware. That container simply returns the 200 code (authenticated), but logs all the headers received from traefik. Traefik contacts that host on the internal docker network via http://172.128.0.x/handler.php.

Here are the relevant headers when I call /_phpinfo.php on the target host:

    [HTTP_X_REAL_IP] => 91.my.pcs.ip
    [HTTP_X_FORWARDED_URI] => /_phpinfo.php
    [HTTP_X_FORWARDED_SERVER] => b7eb2436cf7c
    [HTTP_X_FORWARDED_PROTO] => https
    [HTTP_X_FORWARDED_PORT] => 443
    [HTTP_X_FORWARDED_METHOD] => GET
    [HTTP_X_FORWARDED_HOST] => app.mydomain.com
    [HTTP_X_FORWARDED_FOR] => 91.my.pcs.ip
    [HTTP_UPGRADE_INSECURE_REQUESTS] => 1
    [HTTP_SEC_FETCH_USER] => ?1
    [HTTP_SEC_FETCH_SITE] => none
    [HTTP_SEC_FETCH_MODE] => navigate
    [HTTP_SEC_FETCH_DEST] => document
    [HTTP_SEC_CH_UA_PLATFORM] => "Windows"
    [HTTP_SEC_CH_UA_MOBILE] => ?0
    [HTTP_SEC_CH_UA] => " Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"
    [HTTP_DNT] => 1
    [HTTP_COOKIE] => SessionID=12345
    [HTTP_ACCEPT_LANGUAGE] => de-DE,de;q=0.9,en;q=0.8,en-US;q=0.7
    [HTTP_ACCEPT_ENCODING] => gzip, deflate, br
    [HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
    [HTTP_USER_AGENT] => Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36
    [HTTP_HOST] => nginx-php-2
    [SCRIPT_FILENAME] => /var/www/html/handler.php

Next step would be to debug how the Authentik proxy uses that data, and whether there is something wrong with this initial data it gets from Traefik in the first place. Hope that help.

I'd be willing to dig deeper into this, but without help, I haven't been able to figure out the exact workflow that follows in the Authentik Forward Auth proxy.

[sungyoonc]

@agrimpelhuber I'm running both traefik 2.6.0 and authentik 2022.1.3 with docker compose.

[BeryJu]

@agrimpelhuber @Jafner @winterolym could you try out the latest build, which should fix this issue

See https://goauthentik.io/docs/installation/beta, just replace gh-next with gh-master

[agrimpelhuber]

Sorry, no change, neither in the result nor in the log output. Here's the output from the forward proxy:

{"app":"Forward Auth","event":"Found app based on cookie domain","host":"app.mydomain.com","level":"debug","logger":"authentik.outpost.proxyv2","timestamp":"2022-02-01T09:54:43Z"}
{"event":"passing to application mux","host":"app.mydomain.com","level":"trace","logger":"authentik.outpost.proxyv2","timestamp":"2022-02-01T09:54:43Z"}
{"event":"tracing headers for debug","header":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["de-DE,de;q=0.9,en;q=0.8,en-US;q=0.7"],"Cookie":["SessionID=12345; authentik_proxy=MTY0MzcwODc3MHxOd3dBTkZKV01rUkhRemRMVEZjM1J6TXlTek0zVms1QldUTTBTRUpPVVRZMFRFTXpSMWhFUzBRek5FdzBUVFJWTmtsR1IxVkNSVkU9fLt-vqA3eVyrfpKq3rkd3SON8XsSBWkntvs801b41PS0"],"Dnt":["1"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"97\", \"Chromium\";v=\"97\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"],"X-Forwarded-For":["91.my.pcs.ip"],"X-Forwarded-Host":["app.mydomain.com"],"X-Forwarded-Method":["GET"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Forwarded-Server":["b7eb2436cf7c"],"X-Forwarded-Uri":["/handler.php"],"X-Real-Ip":["91.my.pcs.ip"]},"level":"trace","logger":"authentik.outpost.proxyv2.application","name":"Forward Auth","timestamp":"2022-02-01T09:54:43Z"}
{"event":"traefik forwarded url","level":"trace","logger":"authentik.outpost.proxyv2.application","name":"Forward Auth","timestamp":"2022-02-01T09:54:43Z","url":"https://app.mydomain.com/handler.php"}
{"event":"Matching URL against allow list","level":"trace","logger":"authentik.outpost.proxyv2.application","name":"Forward Auth","regex":"https://app.mydomain.com/handler.php","timestamp":"2022-02-01T09:54:43Z","url":"https://app.mydomain.com/handler.php"}
{"event":"Redirecting to login","level":"debug","logger":"authentik.outpost.proxyv2.application","name":"Forward Auth","timestamp":"2022-02-01T09:54:43Z","url":"https://auth.mydomain.com/akprox/start"}
{"event":"/akprox/auth/traefik","host":"app.mydomain.com","level":"info","logger":"authentik.outpost.proxyv2.application","method":"GET","name":"Forward Auth","remote":"172.128.0.10:43860","runtime":"1.045","scheme":"","size":78,"status":307,"timestamp":"2022-02-01T09:54:43Z","upstream":"","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36"}

What can I do?

[BeryJu]

Which error do you get with that? From the logs it seems all right, application is found based on cookie domain, it's detected the traefik params, and is redirecting you to login, which should carry forward to your browser

[agrimpelhuber]

Sorry, I implied that - the same as above:

{
	"Message": "no app for hostname",
	"Host": "auth.mydomain.com",
	"Detail": "Check the outpost settings and make sure 'auth.mydomain.com' is included."
}

[sungyoonc]

If I connect to the https://app.mydomain.com it will redirect me to https://auth.mydomain.com/akprox/start and it gives me this error.

{
	"Message": "no app for hostname",
	"Host": "auth.mydomain.com",
	"Detail": "Check the outpost settings and make sure 'auth.mydomain.com' is included."
}

However when I bypass traefik and go to http://auth.mydomain.com:9000/akprox/start it redirects me to this url
http://auth.mydomain.com:9000/application/o/authorize/?client_id=<some token>&redirect_uri=https://auth.mydomain.com/akprox/callback&response_type=code&scope=email+profile+openid+ak_proxy&state=<some token>

If I change the url to this
https://auth.mydomain.com/application/o/authorize/?client_id=&redirect_uri=https://auth.mydomain.com/akprox/callback&response_type=code&scope=email+profile+openid+ak_proxy&state=<some token>
and it shows me the login page.

After I login I get redirected to this url
https://auth.mydomain.com/akprox/callback?code=<some token>
and if I change this url to
http://auth.mydomain.com:9000/akprox/callback?code=<some token>
and visit it, it redirects me to the correct application.

[agrimpelhuber]

@BeryJu , I tried to follow your code in order to provide more information, but got lost between the redirect to auth.mydomain.com/akprox/start and the browser message, obviously created by func (ps *ProxyServer) Handle(rw http.ResponseWriter, r *http.Request) {. This is due to a lack of experience, but:

The more I look at it, the more I get the feeling that we need some more log output between those stages. For example, is the function handleRedirect still involved in the whole thing? What is written into which cookie, and is something lost, due to Traefik? The findings of @winterolym suggest this.

[BeryJu]

I'm pretty sure I actually know whats causing this; from the looks of it both of you are running authentik on auth.domain.tld, and then also using auth.domain.tld as external host for an outpost. This leads to requests to app.domain.tld going internally to the outpost, but since thats configured for auth.domain.tld, it redirects to that, however on that domain the embedded outpost seems to have a higher routing priority, and since the embedded outpost does not have the application selected, it errors. Can both of you try using the embedded outpost instead of a separate outpost?

[sungyoonc]

After giving the outpost the higher priority in traefik everything works now. Thank you so much.

[agrimpelhuber]

@BeryJu , you are a genius! Thanks ever so much!

@winterolym has beaten me by 2 hours. Maybe because I was running the integrated outpost, and had to figure out how to set the priority there, and to what values:

• I set the priority of the authentic docker container in docker-compose to 1, just to be on the safe side:

      - "traefik.enable=true"
      - "traefik.http.routers.authentik.priority=1"

• In the Authentic outpost configuration, I set the priority to 50, using the docker_labels option:

docker_labels:
  traefik.http.routers.ak-outpost-b650b531e5384d3ba8e8e605eac1938d-router.priority: "50"

What I wasn't aware of was not only the fact that the priority was part of the problem, but that traefik actually calculates the priority based on the actual number of characters in the router's rule. Thus, "50" didn't work for the proxy without setting "1" for Authentik itself, because the rule of he authentic docker was way longer.

Which again was another problem I had caused by accident: When you'd normally think that the rule with "/akprox" would be naturally longer than the one without, ...

Host(`auth.mydomain.com`) && PathPrefix(`/akprox`) #Higher Priority
Host(`auth.mydomain.com`) #Lower Priority

... it fell on my feet because in order to debug far more basic stuff, I had configured 2 subdomains for Authentic - doh

Host(`auth.mydomain.com`) && PathPrefix(`/akprox`) #Lower Priority
Host(`auth.mydomain.com`) || Host(`login.mydomain.com`) #Higher Priority

I hope this helps other people to avoid similar mistakes.

@BeryJu, I didn't succeed to use variables like %(name)s in the docker_labels configuration to avoid writing "b650b531e5384d3ba8e8e605eac1938d" time and time again. But that would be luxury - at least it's working now!