I have a fresh-out-of-the-box workstation class system with dual quad-core i7 and 20gigs of memory.
I've seen this issue on vmware player 4.0.2 and 5.0.1.
I take a fresh copy of a VM that I have been running with no problems on other machines, and start it up on this system. The first time I start it, I have 20 minutes before the computer wedges after running out of memory. Task manager shows all memory is used, but the process display doesn't show any process that is using much memory. The vm is limited to 8gig, but appears to use less than three. If I shut down vmware player before the machine wedges, only about half of the memory is released.
If I shut the vm down, and reboot the base machine, it comes back fine, and grows until it is using 10.0gigs, then seems to be stable... the base system seems to use 1.85gig, and the vm seems to use just under 3 gigs. But 10.0gig is ok, this is the only application that I need running, and so I'm happy. When I shut the vm down, about 8 gigs of memory is released. so it seems ok now...
If I leave the vm running overnight, the machine is out of memory again. I kill the vm, and stop vmware, and only 5 gigs are released, but process manager shows nothing is using much memory, nothing over 250meg... even though 15 gigs is being used.
This vm is the clone of a vm running under esxi, that has many months of uptime, and no memory leak.
The base os of the machine is windows 7, and the os of the VM is windows server 2008 running sql and mediaroom personal server.