Skip to content

xDS with TrafficDirector and spiffe: CertificateException: no name matching foo.bar found #10810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
tmhdgsn opened this issue Jan 10, 2024 · 9 comments · Fixed by #11052
Closed
Labels
Milestone

Comments

@tmhdgsn
Copy link

tmhdgsn commented Jan 10, 2024

grpc_java version 1.60.0

Hello,

I'm trying to get traffic director to provide security configuration to a client but I am encountering a failure in checkServerTrusted of XdsX509TrustManager https://github.com/grpc/grpc-java/blob/v1.60.0/xds/src/main/java/io/grpc/xds/internal/security/trust/XdsX509TrustManager.java#L244
because the target hostname does not match the SAN (spiffe id) in the server cert. I am wondering if the problem lies with the config from TD or with the client.

here is a snippet of the java code, written using the xds example in this repo:

var credentials = XdsChannelCredentials.create(InsecureChannelCredentials.create());
var channel = Grpc.newChannelBuilder("xds:///foo.bar:8443", credentials);
var blockingStub = FooGrpc.newBlockingStub(channel);
blockingStub.sayHello("world")

stack trace:

Caused by java.security.cert.CertificateException: No name matching foo.bar found
  at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:229)
  at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
  at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:457)
  at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:417)
  at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:291)
  at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
  at io.grpc.xds.internal.security.trust.XdsX509TrustManager.checkServerTrusted(XdsX509TrustManager.java:244)
  at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296)

and this is xds config traffic director is giving the client:

{
  "config": [
    {
      "node": {
        "id": "projects/1337/networks/mesh:foo-mesh/nodes/1338",
        "metadata": {
          "INSTANCE_IP": "0.0.0.0",
          "TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "1337",
          "TRAFFICDIRECTOR_MESH_NAME": "foo-mesh",
          "TRAFFICDIRECTOR_NETWORK_NAME": "foo-network",
          "XDS_STREAM_TYPE": "ADS"
        },
        "locality": {
          "zone": "ib"
        },
        "userAgentName": "gRPC Java",
        "userAgentVersion": ""
      },
      "xdsConfig": [
        {
          "status": "SYNCED",
          "clusterConfig": {
            "dynamicActiveClusters": [
              {
                "cluster": {
                  "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
                  "name": "cloud-internal-istio:cloud_mp_7555213123_1231234123",
                  "altStatName:": "/projects/1339/global/backendServices/foo-bar-td-backendsvc",
                  "type": "EDS",
                  "edsClusterConfig": {
                    "edsConfig": {
                      "ads": {},
                      "initialFetchTimeout": "15s",
                      "resourceApiVersion": "V3"
                    }
                  },
                  "connectTimeout": "30s",
                  "http2ProtocolOptions": {
                    "maxConcurrentStreams": 100
                  },
                  "transportSocket": {
                    "name": "envoy.transport_sockets.tls",
                    "typedConfig": {
                      "@type": "",
                      "commonTlsContext": {
                        "tlsCertificateSdsSecretConfigs": [
                          {
                            "name": "tls_sds",
                            "sdsConfig": {
                              "path": "/etc/envoy/tls_certificate_context_sds_secret.yaml"
                            }
                          }
                        ],
                        "tlsCertificateCertificateProviderInstance": {
                          "instanceName": "google_cloud_private_spiffe",
                          "certificateName": "DEFAULT"
                        },
                        "combinedValidationContext": {
                          "defaultValidationContext": {
                            "matchSubjectAltNames": [
                              {
                                "exact": "spiffe://foo.svc.id.goog/ns/bar/sa/foo"
                              }
                            ]
                          },
                          "validationContextSdsSecretConfig": {
                            "name": "validation_context_sds",
                            "sdsConfig": {
                              "path": "/etc/envoy/tls_certificate_context_sds_secret.yaml"
                            }
                          },
                          "validationContextCertificateProviderInstance": {
                            "instanceName": "google_cloud_private_spiffe",
                            "certificateName": "ROOTCA"
                          }
                        },
                        "alpnProtocols": [
                          "h2"
                        ]
                      }
                    }
                  },
                  "metadata": {
                    "filterMetadata": {
                      "com.google.trafficdirector": {
                        "backend_service_name": "foo-bar-td-backendsvc",
                        "backend_service_project_number": 1339
                      }
                    }
                  }
                }
              }
            ]
          }
        },
        {
          "status": "SYNCED",
          "listenerConfig": {
            "dynamicListeners": [
              {
                "activeState": {
                  "listener": {
                    "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
                    "name": "foo.bar:8443",
                    "apiListener": {
                      "apiListener": {
                        "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                        "statPrefix": "trafficdirector",
                        "rds": {
                          "configSource": {
                            "ads": {},
                            "resourceApiVersion": "V3"
                          },
                          "routeConfigName": "projects/1339/locations/global/grpcRoutes/foo-bar-td-grpcroute_foo_bar:8443"
                        },
                        "httpFilters": [
                          {
                            "name": "envoy.filters.http.fault",
                            "typedConfig": {
                              "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
                            }
                          },
                          {
                            "name": "envoy.filters.http.router",
                            "typedConfig": {
                              "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
                              "suppressEnvoyHeaders": true
                            }
                          }
                        ]
                      }
                    }
                  }
                }
              }
            ]
          }
        },
        {
          "status": "SYNCED",
          "routeConfig": {
            "dynamicRouteConfigs": {
              "routeConfig": {
                "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
                "name": "projects/1339/locations/global/grpcRoutes/foo-bar-td-grpcroute_foo_bar:8443",
                "virtualHosts": [
                  {
                    "domains": [
                      "foo.bar:8443"
                    ],
                    "routes": [
                      {
                        "match": {
                          "prefix": ""
                        },
                        "route": {
                          "cluster": "cloud-internal-istio:cloud_mp_7555213123_1231234123"
                        }
                      }
                    ]
                  }
                ]
              }
            }
          }
        }
      ]
    }
  ]
}

I should mention also TD is being configured via a NetworkSecurityClientTLSPolicy attached to a ComputeBackendService as documented here: https://cloud.google.com/traffic-director/docs/security-overview#:~:text=A%20client%20TLS%20policy%20lets,a%20global%20backend%20service%20resource.

I've tried adding the spiffe id as an subjectAltName to the computebackend service which does appear in the xds config under defaultValidationContext but iiuc that isnt used until verifySubjectAltNameInChain in https://github.com/grpc/grpc-java/blob/v1.60.0/xds/src/main/java/io/grpc/xds/internal/security/trust/XdsX509TrustManager.java#L245

Thanks

@sergiitk
Copy link
Member

From a brief glance, spiffe id is incorrect. Normally it would be spiffe://{WORKLOAD_POOL}/ns/{NAMESPACE}/sa/{KUBERNETES_SERVICE_ACCOUNT}, where WORKLOAD_POOL is "{PROJECT_ID}.svc.id.goog". Not sure if the first foo in spiffe://foo.svc.id.goog/ns/bar/sa/foo is the same as the second one, but could you please double check it's a project id?

@tmhdgsn
Copy link
Author

tmhdgsn commented Jan 10, 2024

From a brief glance, spiffe id is incorrect. Normally it would be spiffe://{WORKLOAD_POOL}/ns/{NAMESPACE}/sa/{KUBERNETES_SERVICE_ACCOUNT}, where WORKLOAD_POOL is "{PROJECT_ID}.svc.id.goog". Not sure if the first foo in spiffe://foo.svc.id.goog/ns/bar/sa/foo is the same as the second one, but could you please double check it's a project id?

sorry thats part is on me removing some information before posting.. the actual spiffe id is correct but its that not being equal to foo.bar thats throwing the exception

@sergiitk
Copy link
Member

Also, that CSDS dump doesn't show any endpoints. It should be something like

 "endpoints": [{
      "locality": {
        "subZone": "ib:us-central1-a_2827676496526826176_neg"
      },
      "lbEndpoints": [{
        "endpoint": {
          "address": {
            "socketAddress": {
              "address": "10.12.0.189",
              "portValue": 8080
            }
          }
        },
        "healthStatus": "HEALTHY"
      }],
      "loadBalancingWeight": 1000000
    }]

I'm attaching a log file with xDS responses received by grpc-java 1.60.x xDS client setup for spiffe TLS: downloaded-logs-20240110-104310.csv. Hope this helps while we're looking into this.

Here's the TD setup we did for the attached client log: setup.log. However, It's not GCE, but k8s-to-k8s.

@ejona86
Copy link
Member

ejona86 commented Jan 11, 2024

Which Java version are you using? I'm suspicious that it may matter.

It seems weird to me that foo.bar is getting checked from the JDK's checkServerTrusted(). The gRPC code is at least trying to disable that check. The null check is useless, as getSSLParameters() is always non-null.

@tmhdgsn
Copy link
Author

tmhdgsn commented Jan 11, 2024

Which Java version are you using? I'm suspicious that it may matter.

It seems weird to me that foo.bar is getting checked from the JDK's checkServerTrusted(). The gRPC code is at least trying to disable that check. The null check is useless, as getSSLParameters() is always non-null.

zulu21.30.15-ca-jdk21.0.1-linux_x64

@ejona86
Copy link
Member

ejona86 commented Jan 12, 2024

I just ran our tests with azul/zulu-openjdk:21.

openjdk 21.0.1 2023-10-17 LTS
OpenJDK Runtime Environment Zulu21.30+15-CA (build 21.0.1+12-LTS)
OpenJDK 64-Bit Server VM Zulu21.30+15-CA (build 21.0.1+12-LTS, mixed mode, sharing)

The tests passed, so I commented out the setEndpointIdentificationAlgorithm(null) in XdsX509TrustManager. That caused the tests to fail:

io.grpc.xds.internal.security.trust.XdsX509TrustManagerTest > checkServerTrustedSslEngine FAILED
    java.security.cert.CertificateException: No subject alternative DNS name matching peer-host-from-mock found.
        at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:207)
        at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:457)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:431)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:291)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at io.grpc.xds.internal.security.trust.XdsX509TrustManager.checkServerTrusted(XdsX509TrustManager.java:244)
        at io.grpc.xds.internal.security.trust.XdsX509TrustManagerTest.checkServerTrustedSslEngine(XdsX509TrustManagerTest.java:535)

The failure path is close but not identical, at least partly because the test certs had a different SAN setup. But I think the important part is X509TrustManagerImpl.java:291. That line should only be executed if identityAlg != null. I'm at a loss, but it does make me wonder if something caused getSSLParameters() to return null.

@tmhdgsn
Copy link
Author

tmhdgsn commented Jan 12, 2024

stepping through I see that after disabling the check in sslParams, setSSLParameters in SSLConfiguration is ignoring it : https://github.com/openjdk/jdk/blob/8e12053e0352a26ecd7f2b9bc298ddb8fb4bb61b/src/java.base/share/classes/sun/security/ssl/SSLConfiguration.java#L312

@ejona86
Copy link
Member

ejona86 commented Mar 26, 2024

I thought I put a comment here, but now I wonder if I put it somewhere else... I noticed AdvancedTlsX509TrustManager sets the endpoint identity algorithm to "", not null.

@ejona86
Copy link
Member

ejona86 commented Mar 28, 2024

For posterity, both netty-tcnative and Conscrypt appear to observe null in their SSLEngines. So this appears to have only impacted SunJSSE.

XdsX509TrustManagerTest failing or passing doesn't tell us much, because it uses a fake SSLEngine.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants