Having wrong password for kibana_system blocks cli based user queries with another authorized user

[tsarajar]

Elasticsearch Version

8.17.4

Installed Plugins

No response

Java Version

bundled

OS Version

Ubuntu 24.04.2

Problem Description

I had the wrong password set for kibana_system in Kibana. Ended up having elastic's log getting spammed with:

---

[2025-04-17T12:59:48,308][INFO ][o.e.x.s.a.RealmsAuthenticator] [elasticsearch.suffix.com] Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]

---

Meanwhile in another terminal and / or ansible run i got this:

---

root@elasticsearch:~# curl -u elastic:<password> -k https://localhost:9200/_security/user/kibana_system {"kibana_system":{"username":"kibana_system","roles":["kibana_system"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true}}

root@elasticsearch:~# curl -u elastic:<password> -k https://localhost:9200/_security/user/kibana_system {"kibana_system":{"username":"kibana_system","roles":["kibana_system"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true}}

root@elasticsearch:~# curl -u elastic:<password> -k https://localhost:9200/_security/user/kibana_system {}

root@elasticsearch:~# curl -u elastic:<password> -k https://localhost:9200/_security/user/kibana_system {"kibana_system":{"username":"kibana_system","roles":["kibana_system"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true}}

---

Note how the third query returned nothing as it most likely clashed with the kibana_system trying to log in with an incorrect password simultaneously. So if i keep running the curl, once in a while it just reports empty. If I stop Kibana's service, that never happens again. So it's a timing issue apparently.

Steps to Reproduce

Try to start Kibana with wrong password set and start querying users.

Logs (if relevant)

No response

[DaveCTurner]

Thanks very much for your interest in Elasticsearch.

This appears to be a user question, and we'd like to direct these kinds of things to the Elasticsearch forum. If you can stop by there, we'd appreciate it. This allows us to use GitHub for verified bug reports, feature requests, and pull requests.

Specifically, the behaviour you describe seems to be working as intended. If something talking to Elasticsearch does not have the correct credentials then Elasticsearch will not authenticate its requests. That's what credentials like passwords are for.

There's an active community in the forum that should be able to help get an answer to your question. As such, I hope you don't mind that I close this.

[tsarajar]

Ah seems you misunderstood me. So my admin user "elastic" with the correct password is querying users. It's trying to check whether kibana_system user exists or not. Sure that check cant fail and succeed randomly depending on Kibana trying to log in or not. It should always fail or always succeed.

Even if that was the case that i had the wrong password as you thought i had, the situation would be even worse .. now it would randomly succeeds when it never should.

[DaveCTurner]

Hmm ok I think I see but you have presented this in a way that is very confusing. The title "Having wrong password in Kibana blocks user queries" and the log spam you described seem unrelated to that? The rest of your text is basically unreadable, at least not without putting in more effort than would seem reasonable given the title and preceding paragraphs. Would you try again with a fresh issue, focussing on the actual problem, and check it carefully using the preview feature to make sure it makes sense before submitting it?