RemoteFx With 2012 server to Windows 7 enterprise

We have been configuring a Windows 2012 server Datacenter edition server to host virtual machines and workstations for our IT department to do testing. We've hit an issue with enabling remoteFX on a Windows 7 enterprise VM. We've installed the RemoteFx adapter, and the Hyper-v console says that the VM is using remote graphics. This is all good, but when we try to connect to the VM through remote desktop, remote desktop launches and just closes itself. If we remove the RemoteFx adapter Remote desktop runs fine.

For comparison we deployed a Win 8.1 enterprise VM and enabled remote FX the same way on that VM and it seems to work flawlessly.

I've checked the event logs on the server, the VM, and the machines we've tried to remote desktop to the VM and I haven't seen any suspicious events that seem to involve Remote Desktop or terminal services.

I've seen that other people have run into issues with Server 2012 and 7 enterprise when trying remote FX. I'm curious if anyone has had success with this, or have an idea of what we can check on to get this running.

Server configuration:
Nvidia NVS 510
48 Gig of RAM
Hyper-V roll installed
Remote Desktop Services installed
  RD connection broker
  RD virtual host
  RD licensing (not activated 120 day grace period)
  RD Session host
  RD Web Access