Remote Desktop disconnections

I am seeing multiple users get a disconnect and subsequently reconnect to our 2012 R2 Terminal Server. I have not been able to find a pattern in the disconnects as of yet. There is a average of 16 users on the TS at a time. Sometimes a few of them will get a "random" disconnect, sometimes all of them will get a disconnect. Please see below for environment specific info:

--Terminal Server--
Virtualized Hyper-V VM
OS: Server 2012 R2 Standard
Ram: Dynamic (using 34576MB at time of writing)
vCPUS: 8

--Hyper V host--
OS: Server 2012 R2 Standard w/GUI
Ram: 96GB
Proc: 2x x5650's

The TS is has all the RDS roles installed (RDSH, Broker, License, etc). There doesn't seem to be a problem with the license role. The RD license Diagnoser doesn't find anything wrong. The SSL installed is from GoDaddy and I haven't seen any issues regarding that piece either. All client computers are connecting via local RDP client and not using the RDWeb access. I have gathered multiple instances of the following errors when I was notified that all the client workstations were disconnected:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Microsoft-Windows-TerminalServices-LocalSessionManager" Guid="{5D896912-022D-40AA-A3A8-4FA5515C76D7}" />
  <EventID>40</EventID>
  <Version>0</Version>
  <Level>4</Level>
  <Task>0</Task>
  <Opcode>0</Opcode>
  <Keywords>0x1000000000000000</Keywords>
  <TimeCreated SystemTime="2015-06-30T13:42:45.539110700Z" />
  <EventRecordID>5721</EventRecordID>
  <Correlation ActivityID="{F4206964-70F6-4FC5-8367-D1A0DA9A0000}" />
  <Execution ProcessID="1200" ThreadID="48720" />
  <Channel>Microsoft-Windows-TerminalServices-LocalSessionManager/Operational</Channel>
  <Computer>remoteTSserver.domain.local</Computer>
  <Security UserID="S-1-5-18" />
  </System>
- <UserData>
- <EventXML xmlns="Event_NS">
  <Session>89</Session>
  <Reason>0</Reason>
  </EventXML>
  </UserData>
  </Event>

AND

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Microsoft-Windows-TerminalServices-LocalSessionManager" Guid="{5D896912-022D-40AA-A3A8-4FA5515C76D7}" />
  <EventID>40</EventID>
  <Version>0</Version>
  <Level>4</Level>
  <Task>0</Task>
  <Opcode>0</Opcode>
  <Keywords>0x1000000000000000</Keywords>
  <TimeCreated SystemTime="2015-06-30T13:42:45.667112400Z" />
  <EventRecordID>5722</EventRecordID>
  <Correlation ActivityID="{F4208B7E-6769-48D9-B172-74A4D4790000}" />
  <Execution ProcessID="1200" ThreadID="48720" />
  <Channel>Microsoft-Windows-TerminalServices-LocalSessionManager/Operational</Channel>
  <Computer>remoteTSserver.domain.local</Computer>
  <Security UserID="S-1-5-18" />
  </System>
- <UserData>
- <EventXML xmlns="Event_NS">
  <Session>82</Session>
  <Reason>5</Reason>
  </EventXML>
  </UserData>
  </Event>

The short hand are "Session XX has been disconnected, reason code 5" and "Session XX has been disconnected, reason code 0"

I have already removed all "advanced" features form the NIC's on the host server (offloading, etc.) and I am still running into the issue.