SearchableSnapshot: GKE Workload Identity and GCS repository plugin

[barkbay]

It's possible to use GKE Workload Identity to automatically setup the credentials for the GCS repository plugin (instead of creating a JSON service account file).
It works fine as long as snapshots are created or restored "manually" (through the _snapshot/ API).
It is however no more the case when the bucket must be read from the SearchableSnapshot code, for example in the context of a frozen or cold tier. In this case it fails with the following error:

{
	"type": "server",
	"timestamp": "2022-01-17T16:09:49,963Z",
	"level": "WARN",
	"component": "o.e.r.g.GoogleCloudStorageService",
	"cluster.name": "elasticsearch-sample",
	"node.name": "elasticsearch-sample-es-cold-1",
	"message": "failed to load default project id",
	"cluster.uuid": "lKo5VXDGRGeIg6aQAIEZ5g",
	"node.id": "lC0hhssQRfW9fdow0zYZJg",
	"stacktrace": ["java.security.AccessControlException: access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")",
		"at java.security.AccessControlContext.checkPermission(AccessControlContext.java:485) ~[?:?]",
		"...",
		"at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.readBlob(GoogleCloudStorageBlobContainer.java:63) [repository-gcs-7.16.1.jar:7.16.1]",
		"at org.elasticsearch.common.blobstore.support.FilterBlobContainer.readBlob(FilterBlobContainer.java:48) [elasticsearch-7.16.1.jar:7.16.1]",
		"at org.elasticsearch.xpack.searchablesnapshots.store.SearchableSnapshotDirectory$RateLimitingBlobContainer.readBlob(SearchableSnapshotDirectory.java:763) [searchable-snapshots-7.16.1.jar:7.16.1]"
	]
}

See the following ECK issue for more context: elastic/cloud-on-k8s#5230

I would be happy to help/contribute but I'm not familiar with how the security manager is used in this part of the code.

Thanks

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[tlrx]

@barkbay Thanks for reporting. Can you please post the complete stacktrace? It seems to be a different issue from the last stacktrace I can see in elastic/cloud-on-k8s#5230 (comment).

[barkbay]

@barkbay Thanks for reporting. Can you please post the complete stacktrace? It seems to be a different issue from the last stacktrace I can see in elastic/cloud-on-k8s#5230 (comment).

Both stacktraces are available here.

I think the stacktrace here is a side effect or a consequence of the one I mentioned in the first comment of this issue. Said differently my feeling is that the cold/frozen node is not able to initialize the GCS plugin ( ["java.security.AccessControlException: access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")") which leads to the 401 error later when there is an attempt to read the data from the repository.

[tlrx]

Thanks @barkbay, having the full stacktrace helps :) Yes it seems that we should run some code with privileges but are not.