KVM | Cloudstack Agent Disconnected State Post Upgrade and Rollback

[midhunpjos]

Hello Community,

I have tried a minor upgrade for cloudstack from 4.19.1 to 4.19.2, but the upgrade faced some issues and had to rollback. Post this the agents got disconnected and in logs below are observed.

How I can reissue the certificates in agent.

Management Logs

2025-05-08 15:32:14,098 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-119:null) (logid:) SSL error caught during wrap data: no cipher suites in common, for local address=/172.16.3.221:8250, remote address=/172.16.3.106:46046.
2025-05-08 15:32:14,100 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Connection from /172.16.3.106 closed but no cleanup was done.

Agent Logs

2025-05-08 00:00:04,842 ERROR [utils.nio.Link] (main:null) (logid:) SSL error caught during unwrap data: Received fatal alert: handshake_failure, for local address=/172.16.3.106:49830, remote address=/172.16.3.221:8250. The client may have invalid ca-certificates.
2025-05-08 00:00:04,842 ERROR [utils.nio.NioClient] (main:null) (logid:) SSL Handshake failed while connecting to host: 172.16.3.221 port: 8250
2025-05-08 00:00:04,842 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 172.16.3.221 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
at com.cloud.agent.Agent.start(Agent.java:297)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:454)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415)
at com.cloud.agent.AgentShell.start(AgentShell.java:511)
at com.cloud.agent.AgentShell.main(AgentShell.java:541)
2025-05-08 00:00:04,843 INFO [utils.nio.NioClient] (main:null) (logid:) NioClient connection closed
2025-05-08 00:00:04,843 INFO [cloud.agent.Agent] (main:null) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again...
com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 172.16.3.221 port: 8250
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101)
at com.cloud.agent.Agent.start(Agent.java:297)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:454)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415)
at com.cloud.agent.AgentShell.start(AgentShell.java:511)
at com.cloud.agent.AgentShell.main(AgentShell.java:541)
Caused by: java.io.IOException: SSL Handshake failed while connecting to host: 172.16.3.221 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
... 6 more

[DaanHoogland]

@midhunpjos , sorry to see your query remains unanswerred for so long. Did you find what you are looking for?

On option is to force-reconnect to the host. You may need to remover the old certificates from the host first.

have a look at https://www.shapeblue.com/secure-kvm-vnc-connections/ for background.

[midhunpjos]

Yes, We did a rollback for management service using the DB dump and following reconnect after removing old certificates helped to restore.
Ref : #9565