VXLAN: Loss of Connectivity Between Two VMs After Idle Time on KVM Hosts – CloudStack 4.20

[lexmora]

Description:

Hello,

I hope you're doing well, I've recently returned to using CloudStack after a few years, and I'm currently facing a VXLAN connectivity issue between two virtual machines deployed on different KVM hosts.

Issue Summary:

Two VM's are deployed on separate KVM Hosts.
VXLAN bridges and interfaces are correctly created by CLoudStack.
Initial connectivity work fine -- VM's can ping each other and the gateway (Virtual Router).
After stopping traffic between them and waiting appoximately 5 minutes, the VMs can no loger communicate with each other.
The issue affects both directions (bi'directional connectivity loss).
The bridge fdb show command no longer shows the remote MAC addresses associated with the VXLAN interface.
This leads to traffic being dropped, and ping fails to resume.
Temporary Workaround:
The only way to restore traffic is by manually bringing down and up the physical interface on the bond carrying the VXLAN traffic.

Environment Details:

3 KVM Hosts (Kernel 6.2.0)

CloudStack version 4.20.0.0

Advaced Zone

Traffic types configured:

Management: VLAN
Storage: VLAN
Guest: VXLAN
public VLAN
VXLAN is configured using multicast
Native Linux bridge is in use (not OVS)

IGMP Snooping is disable on the physical switches

Concers and Question:

Why does VXLAN stop learning or maintaining remote VM MAC addresses after idle time?
is this a know issue with VXLAN multicast setup?
Are there kernel or bridge parameters that should be adjusted (e.g., aging time)?

Thanks you in advance for your help and support

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[joolli]

Can you provide the output of: ip -d link show type vxlan ?

[lexmora]

Hello, yes the output in the host is

24: vxlan7209: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue master brvx-7209 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 56:e4:52:49:15:52 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 65535
vxlan id 7209 group 239.0.28.41 dev cloudbr1.2434 srcport 0 0 dstport 8472 ttl 10 ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx
bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.56:e4:52:49:15:52 designated_root 8000.56:e4:52:49:15:52 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on bcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off locked off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

[DaanHoogland]

@lexmora , it seems you are the only one with this issue. If you have anymore information , please add it here. This could be specific to your environment. As a trouble shooting step, can you check if the path between the VMS hits any external deviced (e.g a switch between the hosts?)