Slow simple search across multiple indices with x-pack-security enabled

[lekrus]

Elasticsearch version (bin/elasticsearch --version):
7.5.2, 7.6.0

Plugins installed:
[x-pack-security]

JVM version (java -version):
JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.2/13.0.2+8

OS version (uname -a if on a Unix-like system):
Windows 10, Windows Server 2016

Description of the problem including expected versus actual behavior:
Very big overhead when running multi-indices searches when x-pack-security is enabled.

Steps to reproduce:
On a test cluster we have 197 indices (total size of all docs is around 100MB in all indices).
When we run a simple match_all query for all those indices without x-pack-security enabled, we receive response within 5-6ms.

{
  "size": 0,
  "query": {
    "constant_score": {
      "filter": {
        "match_all": {}
      }
    }
  }
}

After enabling x-pack-security and authenticating under a user with a simplest role:

sample_user_role:
  cluster: ["monitor","manage_ingest_pipelines"]
  indices:
    - names : ["*"]
      privileges : ['all']

the same query runs 300-330ms.
For a cluster with 1500 indices, the query without x-pack runs 100ms, with x-pack - 4500-5000ms.

We tried both file and native realms, both result in the same numbers.
The issue is similar to #42164, but we are getting bad performance even on reasonable number of indices.

[jasontedor]

For the cluster with 1500 indices, the query runs long enough that you should be able to capture hot threads output a few times while the query is executing. Can you do that, execute the query, grab the hot threads output, and share it with us?

[elasticmachine]

Pinging @elastic/es-security (:Security/Security)

[jasontedor]

Also pinging @elastic/es-perf for awareness.

[lekrus]

I've attached the output, if any params should be adjusted when running that endpoint, please let me know, thank you!
hot_threads.txt

[albertzaharovits]

Thanks for the report @lekrus !

I think this is a real performance problem for Security authorization with 1000s of indices/aliases. This is something we were vaguely aware of, but it never quite made the cut priority-wise.

Quite simply the authorization logic is complex for reasons of document and field level security (it uses a lot of maps and sets), there's no cache to it, and it's linearly worse with the number of indices queried.

I'm going to close this as duplicate of #67987 , but this is something we're currently looking into.

Thank you for raising it!