LoadBalancer Services cannot discover the true ClientIP

[asauber]

X-Forwarded-For, X-Real-IP, ProxyProtocol and friends currently see only an IP in the range 192.168.255.0/24

[asauber]

Yes, thank you for your comment. 😄

ProxyProtocol support is coming soon to Linode NodeBalancers, which will allow all of these use cases either directly at the TCP level or beyond the Ingress controller. 🚀

[mortenbirkelund]

@asauber any news or time-line for when this will be supported? Thanks

[asauber]

Hi @mortenbirkelund! Proxy Protocol support was very recently added to NodeBalancers, so the next step is to add a Service annotation to this codebase that will allow enabling it. We don't have a specific timeline for this implementation at the moment. I will update this issue as soon as there is a PR or other progress on that.

[laszlocph]

🎉

[asauber]

All LKE clusters will be upgraded to use this version of the CCM within the next two weeks.

[mortenbirkelund]

@asauber great news. Thanks for the update

[carminalazu]

Is there anything else that we need to do to benefit from this other than thechangelog/changelog.com@a29bee7 & then just waiting for this to be rolled out across Node Balancers?

[asauber]

@carminalazu In the manifest that you linked, you're placing the annotation on an Ingress resource. However, this annotation is used by Service resources. In your case, it would be the Service that sits in front of ingress-nginx.

Once you've enabled Proxy Protocol for ingress-nginx, it looks like you would also need to set this setting in the ingress-nginx ConfigMap https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#use-proxy-protocol

[carminalazu]

I was assuming that this feature will make the X-Forwarded-For, X-Real-IP headers work as expected, which is what this captured:

https://github.com/thechangelog/changelog.com/blob/0e4e5831f494129d0697e34a54a5673e58c4491d/k8s/ingress-nginx/configmap.yml#L8-L12

The nginx-ingress docs explain that the Proxy protocol is a different approach:

Another option is to enable proxy protocol using use-proxy-protocol: "true".
In this mode NGINX does not use the content of the header to get the source IP address of the connection.

This is the ingress-nginx config update that we should end up with:

-use-forwarded-headers: "true"
-
-# TODO: https://github.com/linode/linode-cloud-controller-manager/issues/74
+use-proxy-protocol: "true"
enable-real-ip: "true"
proxy-real-ip-cidr: "192.168.255.0/24"

If I understand you correctly, the annotation needs to be put on this service. Is that right?

https://github.com/kubernetes/ingress-nginx/blob/274d09447e4735815846985c3133bbdc3f00a942/deploy/static/provider/cloud/deploy.yaml#L273-L301

One last question: what happens if the PROXY protocol is enabled on the nginx ingress, but the node balancers don't have this capability rolled out?

[asauber]

This feature adds an annotation for LoadBalancer Services which enables Proxy Protocol on the associated NodeBalancers. This means that an Ingress controller that supports Proxy Protocol, like nginx-ingress, can then provide X-Forwarded-For headers to its backends.

Yes, you would need to modify that Service resource to add the annotation service.beta.kubernetes.io/linode-loadbalancer-proxy-protocol: "v2" Also, your diff looks correct, which disables header forwarding and enables Proxy Protocol.

To answer your last question. I'm happy to report that as of October 2nd all Linode NodeBalancers and LKE clusters support these features!

[asauber]

A description for this Proxy Protocol annotation can be found in the top level Readme https://github.com/linode/linode-cloud-controller-manager#annotations