Internal Network becomes partitioned when tring to run VMFleet

How is the network fabric set up?

Hyper-Converged over 10 gig arista switch.
Each Node a Dual port Mellanox X-5
Followed Standard S2D deployment guide 2 V nics for storage 1 vnic for mgmt everything works fine until I tried to run the VMFleet Below is the network errors when I try to validate cluster with the two internal vnics for VMFleet without them no issue cluster passes validation

 * Network interfaces MIS2DSVR01.MI.local - vEthernet (New Virtual Switch) and S2DSVR02 - vEthernet (New Virtual Switch) are on the same cluster network, yet address 169.16.0.1 is not reachable from 169.16.0.2 using UDP on port 3343.
 * Network interfaces MIS2DSVR02.MI.local - vEthernet (New Virtual Switch) and S2DSVR02 - vEthernet (New Virtual Switch) are on the same cluster network, yet address 169.16.0.2 is not reachable from 169.16.0.1 using UDP on port 3343.

What was the result for Test-RDMA ?

Those are APIPA addresses. Static IP addresses should be assigned to the ports with two subnets if two switches (best practice). We use dual port pNIC pairs in each node so port 0 = subnet 0 and port 1 - subnet 1 on each. Each port on the pNIC is connected to Switch 0 and Switch 1 respectively.

Make sure to limit Live Migration using QoS so that it does not choke storage traffic.

How many nodes?

I set all of the networking RDMA QOS etc up and the base networking of the cluster works fine.  I want to stress test it via VMFLEET.  When you run the PowerShell script to create the VMS for the test it first adds to internal virtual network cards to the cluster so the host can communicate with the 20 or so VMS created for the test.  It is the virtual internal NICs that get partitioned and no matter what I try I can’t get them to come up.  If I remove one of the internal vnics or reboot a node the other nodes internal vnic can communicate all of the sudden but as soon as I add it back or the other node comes back up the internal network fails again and says it partitioned.

Make sure the firewall exception is in place for all three firewall profiles. Make sure firewall logging is enabled. Check for dropped packets on UDP 3343.