Modify apache config to record forwarded IPs

[XenoPhage]

If you run this container behind a proxy, the access logs show the IP of the proxy and not the client. This can be corrected by updating the /etc/apache2/sites-enabled/000-default.conf config as follows:

<VirtualHost *:80>
	# The ServerName directive sets the request scheme, hostname and port that
	# the server uses to identify itself. This is used when creating
	# redirection URLs. In the context of virtual hosts, the ServerName
	# specifies what hostname must appear in the request's Host: header to
	# match this virtual host. For the default virtual host (this file) this
	# value is not decisive as it is used as a last resort host regardless.
	# However, you must set it for any further virtual host explicitly.
	#ServerName www.example.com

	ServerAdmin webmaster@localhost
	DocumentRoot /var/www/html

	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog ${APACHE_LOG_DIR}/error.log

	LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
	LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy
	SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded
	CustomLog ${APACHE_LOG_DIR}/access.log combined env=!forwarded
	CustomLog ${APACHE_LOG_DIR}/access.log proxy env=forwarded

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf
</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Can this container be updated to do this by default instead of having to either rebuild it or add a volume?

[tianon]

Most of this configuration comes directly from Debian's default configuration -- we only make minor changes to it, if at all (and all of that comes from the php image, not something added in wordpress).

[XenoPhage]

Understood. This is simply a replacement config file to handle proxied communications with the container, which I would expect is a pretty common use case, and still provide the client IPs for logging purposes.

[hudsantos]

Also interested in a solution like this. But the /etc/apache2/apache2.conf says:

# Note that the use of %{X-Forwarded-For}i instead of %h is not recommended.
# Use mod_remoteip instead.

Also the snippet suggested by @XenoPhage and another form @brainlid on this issue didn't work for me.
So, I'm also still looking for another solution to tackle this.
As soon as I find something new I'll follow up here.

[gingerlime]

Also facing this problem using https://github.com/jwilder/nginx-proxy and wordpress in docker. The headers (X-Forwarded-For, X-Real-IP, X-forwaded-Proto etc) are being passed from nginx-proxy to the wordpress container correctly, but the logs still show the proxy's internal IP rather than the external client.

[chrisbloemker]

@gingerlime have you solved your issue yet? I'm running into a similar situation using Traefik. All the wordpress containers behind traffic are showing the internal IP of traefik rather than the external IP. This makes plugins useless that work by IP, aka whitelisting, security-based plugs etc. Still looking for a solution as I'm not convinced that the problem lies at the proxy level but rather the wordpress container level (apache).

[gingerlime]

@chrisbloemker It definitely looks like it's due to the way apache is set up for wordpress in docker. I haven't solved it yet (didn't find the time and it's not urgent for me, but I would definitely hope to find a simple solution).

[chrisbloemker]

@gingerlime I agree, I'm running a handful of wordpress containers and just realized this issue. I need to fix asap, but I'm not experienced with nginx and not so much apache :/ I'll keep looking. I'm surprised there's not much else talk on this problem. If I find a solution I'll keep you updated.

[gingerlime]

I think the changes that @XenoPhage originally suggested should work, but didn't get a chance to look at what it means to apply them in practice or test it.

[hudsantos]

I am also looking for a solution on it. Still facing this and didn't found a clever solution. Not a big problem here but should be better if the logged IP address was the external rather than the internal.

[XenoPhage]

FYI : This is pretty easy to test live without rebuilding a container if you want. Take the config file I provided above and docker cp it into your container :

docker cp 000-default.conf wordpresscontainer:/etc/apache2/sites-enabled/000-default.conf

And then just restart the container. Be careful not to remove (docker rm) the container first or it'll wipe out that change. Once it's restarted, check the logs and you can see whether the real IP is coming through or not.

If you want to make the change "permanent" you can use a docker volume to replace the file on start, or you can rebuild the container. The former solution works better if you want to continue getting updates from upstream, otherwise you have to monitor and rebuild on your own.

[gingerlime]

Thanks for the suggestion @XenoPhage. I managed to get the volume solution working for me.

In case it helps others, I'm using docker-compose and this is more or less how I overwrite the apache config in my docker-compose.yml

version: '3'
services:
  ...
  my_wordpress:
    image: wordpress:5
    environment:
      ...
    volumes:
      - /path/to/wordpress:/var/www/html
      - /path/to/000-default.conf:/etc/apache2/sites-enabled/000-default.conf

[XenoPhage]

For completeness, here's what I use for my wordpress containers. Note: This works for multisite as well if that's your thing.

version: '3.6'
services:
  wordpress:
    image: wordpress:latest
    container_name: wordpress
    restart: always
    volumes:
      - /srv/wordpress/wp-content:/var/www/html/wp-content:Z
      - /srv/wordpress/docker-php-upload-size.ini:/usr/local/etc/php/conf.d/docker-php-upload-size.ini:Z
      - /srv/wordpress/wp-config.php:/var/www/html/wp-config.php:Z
      - /srv/wordpress/.htaccess:/var/www/html/.htaccess:Z
      - /srv/wordpress/000-default.conf:/etc/apache2/sites-enabled/000-default.conf:Z
    labels:
      traefik.docker.network: "web"
      traefik.enable: true
      traefik.port: 80
      traefik.protocol: "http"
      traefik.frontend.rule: "Host:myblog.example.com"
      traefik.frontend.headers.customResponseHeaders: "X-Clacks-Overhead: GNU Terry Pratchett"
    networks:
      web:
        aliases:
          - wordpress
      database:
networks:
  web:
    external:
      name: web
  database:
    external:
      name: database

[chrisbloemker]

I will have to try that, @XenoPhage good suggestion. Slightly off topic, out of curiosity, what are the benefits to bind mounting the /srv/wordpress I'm assuming that would be a distributed FS like gluster or something if you're running multiple nodes?

[XenoPhage]

/srv is just where I decided docker volume mounts should go. I think when I first started playing with docker, the examples I saw did this and it just stuck. But yes, it could just as well be a shared filesystem.

As for the individual mounts I have listed, each one has a purpose. wp-content is the themes, plugins, and any uploaded files, the php.ini file just ups the max upload size allowed, the config file is self explanatory, .htaccess for security, and the apache config to fix the IP issue we've been discussing.

It was suggested to me that I should just do the entire wordpress directory, but I wanted to avoid putting the core wordpress files on a volume mount since that's what the container is for to begin with. Otherwise I could just as easily use an php apache container and point it at a wordpress volume mount.

I try to ensure only the state is stored in volumes and not the immutable bits.

[chrisbloemker]

@XenoPhage your solution seems to have worked! Thanks a bunch. The access logs show the correct external IP as well as the plugins are showing the same IP based on what apache is logging. Will soon test more plugins just to make sure.

107.45.92.65 - - [21/Jun/2019:18:44:06 +0000] "GET /?p=1 HTTP/1.1" 200 6653 "https://qa.domain.com/?p=1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15"
107.45.92.65 - - [21/Jun/2019:18:44:21 +0000] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 47 "https://qa.domain.com/wp-admin/options-general.php?page=limit-login-attempts" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15"

Where as before I was only getting the 10.255.0.3 (traefik's internal IP)

I have now made the 000-default.conf as such:

<VirtualHost *:80>
	ServerAdmin webmaster@localhost
	DocumentRoot /var/www/html
	ErrorLog ${APACHE_LOG_DIR}/error.log
	LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
	LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy
	SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded
	CustomLog ${APACHE_LOG_DIR}/access.log combined env=!forwarded
	CustomLog ${APACHE_LOG_DIR}/access.log proxy env=forwarded
</VirtualHost>

and then added this to my Dockerfile:

COPY ./000-default.conf /etc/apache2/sites-enabled/000-default.conf

All seems to be well with the world now.

To be clear, I built locally and pushed to docker hub and then just used Anisble to re-deploy my WordPress stack with the newer tagged image and the site was all intact, no issues :)

[chrisbloemker]

Well, after a few days, sadly the fix I posted above does not work anymore. I thought maybe the WordPress plugins used the apache logs. I implemented that edit into my WordPress image, and now after a few days, the internal IP's are back in the container logs and the plugins are not working now as it's reading the 10.X.X.X addresses instead of the real IP's :(

[tianon]

Looking at this again, I think this does make sense for us to do something with.

I think the best approach here is the one recommended by both Apache and Debian, namely mod_remoteip. FWIW, nextcloud has been including mod_remoteip for a while now: https://github.com/nextcloud/docker/blob/3f40b69c54460f565600da9dce0f04cf160ee54a/16.0/apache/Dockerfile#L104-L111

Their configuration is similar to what I was considering -- trust the non-public IPv4 ranges by default. I think the only real controversial points are whether to use RemoteIPTrustedProxy vs RemoteIPInternalProxy and which header to put in RemoteIPHeader (X-Real-IP, X-Client-IP, X-Forwarded-For, etc).

[chrisbloemker]

@tianon is that aen2mod mod_remoteip something I can incorporate into my Dockerfile or is this something that would have to be in the official image? I guess I'm asking where the appropriate place to include that mod. Here is my dockerfile:

# Set a health check
HEALTHCHECK --interval=30s \
            --timeout=30s \
            --start-period=5s \
            --retries=3 \
            CMD curl http://127.0.0.1:80 || exit 1

# Adding our custom PHP configuration
COPY ./php.ini /usr/local/etc/php/conf.d/php.ini

# Adding our custom Wordpress wp-config
COPY --chown=www-data:www-data wp-config.php /var/www/html/

# Overwriting the default entrypoint script
COPY docker-entrypoint.sh /usr/local/bin/

# Adding our custom themes
ADD theme.tar.gz /usr/src/wordpress/wp-content/themes

# Remove default themes
RUN rm -rf /usr/src/wordpress/wp-content/themes/twentyseventeen
RUN rm -rf /usr/src/wordpress/wp-content/themes/twentysixteen

# Makes sure all wordpress files are owned by www-data
RUN chown -R www-data:www-data /usr/src/wordpress/wp-content

# Modify Apache to extend log format to X-forwarded-for
COPY ./000-default.conf /etc/apache2/sites-enabled/000-default.conf

I'm also more confused as I've used the same WordPress image built from the dockerfile above for months and have not run into this issue until very recently.. I thought maybe originally it was an issue with traefik/nginx but it seems that's not the case.

[wglambert]

You can use Nextcloud's Dockerfile as an example for incorporating it into your own
https://github.com/nextcloud/docker/blob/8231878052899baae3202353be39df9bf056a399/16.0/apache/Dockerfile#L105-L112

[gingerlime]

@wglambert this looks much more elegant, and more secure (whitelisting the proxy IP range), but for some reason I can't get it to work. At least not on its own (i.e. without also changing the apache default config)...

Any tips on how to make it work?

[haozhou]

I submit PR #411 for this purpose.
If it gets merged, then you only need to mount your own virtual host config that uses %a instead of %h in the LogFormat to capture real client IP

[gingerlime]

@haozhou I think your PR is a bit less secure though than the nextcloud version, because it doesn't whitelist the trusted internal IP range (this means that anyone can add a X-Forwarded-For header and spoof the IP address in the log...).

Nevertheless, I still wonder about the changes you mention to the LogFormat, and how come Nextcloud doesn't seem to change it as far as I could find (yet, I assume, it still logs the right IP?)

@tianon you also mentioned Nextcloud, can you shed some light on this?

[haozhou]

@gingerlime The proxy internal IP/range varies between different system. I can make it an environment variable but a fixed value is nondeterministic. Nextcloud hardcoded the RemoteIPTrustedProxy which is not a good idea.

Per my understanding, after you enable mod_remoteip, %h is still the IP of the proxy, you need to use %a for the actual client IP. You can refer to https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html

[gingerlime]

The proxy internal IP/range varies between different system. I can make it an environment variable but a fixed value is nondeterministic.

Nextcloud's implementation includes an IP range (subnets) which are reserved for internal IPs only, and therefore usually safe. (10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16)

[haozhou]

ok. I added them in the remoteip.conf

[haozhou]

@gingerlime @tianon Can you please review the PR and raise your concern? If you have no concern, can you please approve it so that I can merge it?

[tianon]

My reading of https://httpd.apache.org/docs/2.4/mod/mod_log_config.html implies that mod_remoteip should affect %h just the same as it does %a (since %h falls back to %a if hostname lookups are disabled, which are disabled by default). Can someone please test and confirm?

[gingerlime]

I was also hoping it would work this way, but without also changing the LogFormat and CustomLog, it was only logging the internal IPs for me...

[haozhou]

@tianon Enabling mod_remoteip would not affect %h by default, the user has to specify LogFormat using %a instead of %h. I've verified that.

Luckily, both LogFormat and CustomLog can be overriden in virtual host configuration what can be mounted to the container.

If both of you are fine with the PR, can you please merge it?
Thanks.