Server 2012 R2 Connectivity problems.

We have a Windows 2012 R2 RDS server.  For the last 5 days, that box goes through periods of serveral hours where connectivity is very spotty.  RDP sessions won't connect or disconnect after just a few minutes.  Sometimes it won't even ping at all, or pings with very slow response times.  When it's not pinging or pinging slowly, we can't RDP or VNC into the box at all.  We have replaced the NIC, patch cable, switch port, and rebooted the switch it's plugged into.  Nothing helps.  It just gets better on it's own and then breaks on it's own.  Rebooting the server or rebooting the switch will sometimes get it going again, but RDP sessions will continue to disconnect after a few minutes until it just decides to get better.  We even hooked up a second NIC with a seperate IP address, and we get the same results with that NIC.  It affects both our local and VPN clients PCs, which are a mixture of Windows 7 and Windows 8.  I'm lookiing for any ideas here - I'm pulling my hair out.