Support GKE Workload Identity for Searchable Snapshots #82974
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Searchable snapshots perform naked calls of `GoogleCloudStorageBlobContainer#readBlob` without the Security Manager. The client fails to get Compute Engine credentials because of that. It works for normal snapshot/restore because they do a privileged call of `GoogleCloudStorageBlobStore.writeBlob` during the verification of the repo. The simplest fix is just to make sure `ServiceOptions.getDefaultProjectId` and `GoogleCredentials::getApplicationDefault` are get called under the SecurityManager (which they should because they perform network calls). Unfortunately, we can't write an integration test for the issue, because the test framework does the repo verification automatically, which works around the bug. Writing a unit test also seems not possible, because `ComputeEngineCredentials#getMetadataServerUrl` relies on the `GCE_METADATA_HOST` environment variable. See elastic/cloud-on-k8s#5230 Resolves elastic#82702
arteam
added
>bug
:Distributed Coordination/Snapshot/Restore
elasticmachine
added
the
Team:Distributed (Obsolete)
|
Pinging @elastic/es-distributed (Team:Distributed) |
I wonder if we can workaround this and add a QA test for searchable snapshots that would reuse the AbstractSearchableSnapshotsRestTestCase (without verifyng the repository) and the fixture used in I think it is worth to have such a test so that we don't reintroduce such bugs in the future. Note that this can be done in a follow up PR. What do you think? |
…nity-tokens-searchable-snapshots
|
Hi @arteam, I've created a changelog YAML for you. |
I think it's a good idea! Unfortunately, I've been unsuccessful in trying to come up with a test reproducing the failure. Even i I disable the repository verification, the client still gets called correctly under the SecurityManager via We probably need some ILM based test for recovery if want to reproduce the issue to trigger the following failing path |
tlrx
approved these changes
Jan 26, 2022
|
Thanks, Tanguy! |
arteam
added a commit
to arteam/elasticsearch
that referenced
this pull request
Jan 26, 2022
* Support GKE Workload Identity for Searchable Snapshots Searchable snapshots perform naked calls of `GoogleCloudStorageBlobContainer#readBlob` without the Security Manager. The client fails to get Compute Engine credentials because of that. It works for normal snapshot/restore because they do a privileged call of `GoogleCloudStorageBlobStore.writeBlob` during the verification of the repo. The simplest fix is just to make sure `ServiceOptions.getDefaultProjectId` and `GoogleCredentials::getApplicationDefault` are get called under the SecurityManager (which they should because they perform network calls). Unfortunately, we can't write an integration test for the issue, because the test framework does the repo verification automatically, which works around the bug. Writing a unit test also seems not possible, because `ComputeEngineCredentials#getMetadataServerUrl` relies on the `GCE_METADATA_HOST` environment variable. See elastic/cloud-on-k8s#5230 Resolves elastic#82702
arteam
added a commit
to arteam/elasticsearch
that referenced
this pull request
Jan 26, 2022
* Support GKE Workload Identity for Searchable Snapshots Searchable snapshots perform naked calls of `GoogleCloudStorageBlobContainer#readBlob` without the Security Manager. The client fails to get Compute Engine credentials because of that. It works for normal snapshot/restore because they do a privileged call of `GoogleCloudStorageBlobStore.writeBlob` during the verification of the repo. The simplest fix is just to make sure `ServiceOptions.getDefaultProjectId` and `GoogleCredentials::getApplicationDefault` are get called under the SecurityManager (which they should because they perform network calls). Unfortunately, we can't write an integration test for the issue, because the test framework does the repo verification automatically, which works around the bug. Writing a unit test also seems not possible, because `ComputeEngineCredentials#getMetadataServerUrl` relies on the `GCE_METADATA_HOST` environment variable. See elastic/cloud-on-k8s#5230 Resolves elastic#82702
arteam
deleted the
support-worload-idenity-tokens-searchable-snapshots
branch
arteam
added a commit
that referenced
this pull request
Jan 26, 2022
* Support GKE Workload Identity for Searchable Snapshots Searchable snapshots perform naked calls of `GoogleCloudStorageBlobContainer#readBlob` without the Security Manager. The client fails to get Compute Engine credentials because of that. It works for normal snapshot/restore because they do a privileged call of `GoogleCloudStorageBlobStore.writeBlob` during the verification of the repo. The simplest fix is just to make sure `ServiceOptions.getDefaultProjectId` and `GoogleCredentials::getApplicationDefault` are get called under the SecurityManager (which they should because they perform network calls). Unfortunately, we can't write an integration test for the issue, because the test framework does the repo verification automatically, which works around the bug. Writing a unit test also seems not possible, because `ComputeEngineCredentials#getMetadataServerUrl` relies on the `GCE_METADATA_HOST` environment variable. See elastic/cloud-on-k8s#5230 Resolves #82702
arteam
added a commit
to arteam/elasticsearch
that referenced
this pull request
Jan 26, 2022
… explicetly After we fixed getting application credentials in a GCE environment in elastic#82974, we can actually get credentials set automatically when creating a new GCS client Fixes elastic#83131
arteam
added a commit
to arteam/elasticsearch
that referenced
this pull request
Jan 26, 2022
… explicetly (elastic#83139) After we fixed getting application credentials in a GCE environment in elastic#82974, we can actually get credentials set automatically when creating a new GCS client Fixes elastic#83131
arteam
added a commit
that referenced
this pull request
Jan 26, 2022
arteam
added a commit
that referenced
this pull request
Jan 26, 2022
Searchable snapshots perform naked calls of `GoogleCloudStorageBlobContainer#readBlob` without the Security Manager. The client fails to get Compute Engine credentials because of that. It works for normal snapshot/restore because they do a privileged call of `GoogleCloudStorageBlobStore.writeBlob` during the verification of the repo. The simplest fix is just to make sure `ServiceOptions.getDefaultProjectId` and `GoogleCredentials::getApplicationDefault` are get called under the SecurityManager (which they should because they perform network calls). Unfortunately, we can't write an integration test for the issue, because the test framework does the repo verification automatically, which works around the bug. Writing a unit test also seems not possible, because `ComputeEngineCredentials#getMetadataServerUrl` relies on the `GCE_METADATA_HOST` environment variable. See elastic/cloud-on-k8s#5230 Resolves #82702
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Searchable snapshots perform naked calls of
GoogleCloudStorageBlobContainer#readBlobwithout the Security Manager. Theclient fails to get Compute Engine credentials because of that. It does work for for normal snapshot/restore we
do a privileged call of
GoogleCloudStorageBlobStore.writeBlobduring the verification of the repository.The simplest fix is just to make sure that both
ServiceOptions.getDefaultProjectIdandGoogleCredentials::getApplicationDefaultget called under the Security Manager (which they should because they perform network calls).Unfortunately, we can't write an integration test for the issue, because the test framework does the repo verification
automatically which kind works around the bug. Writing a unit test also seems not possible, because
ComputeEngineCredentials#getMetadataServerUrlrelies on theGCE_METADATA_HOSTenvironment variable.See elastic/cloud-on-k8s#5230
Resolves #82702