ESXi 5.5 pci passthrough for dedicated video
My situation is not like others before me. Here's the deal I have a project for a MetaCam application that I want to virtualize. Metacam requires Autocad and this vm during it's job nesting tasks will automatically open, edit and close cad files. They recommend some form of dedicated GPU.
Now next door I have a dl380 g9 running esxi 5.5 off sdcard for testing. I've tried a FirePro v5600 and a very old Quadro FX 1400. Both are passed through successfully but the client OS (server 2012 r2) only recognizes a Microsoft Basic Display Adapter.
I tried installing the Firepro drivers for Win 8.1 since AMD doesn t have drivers for 2012 and the weirdest thing happened. The hypervisor would go offline for 5 minutes and when I reconnected the vm would be powered off. I then tried the Quadro, it was also seen as a Microsoft Basic Display Driver but installing 331..65-quadro-tesla-grid-
Am I trying this with hardware that's too old? I have a pair of Quadro 4000's but this server does not provide 6-pin power.
Thanks!
Now next door I have a dl380 g9 running esxi 5.5 off sdcard for testing. I've tried a FirePro v5600 and a very old Quadro FX 1400. Both are passed through successfully but the client OS (server 2012 r2) only recognizes a Microsoft Basic Display Adapter.
I tried installing the Firepro drivers for Win 8.1 since AMD doesn t have drivers for 2012 and the weirdest thing happened. The hypervisor would go offline for 5 minutes and when I reconnected the vm would be powered off. I then tried the Quadro, it was also seen as a Microsoft Basic Display Driver but installing 331..65-quadro-tesla-grid-
Am I trying this with hardware that's too old? I have a pair of Quadro 4000's but this server does not provide 6-pin power.
Thanks!
ASKER
Thanks Andrew, But what about folks this these who end up having massive luck not using multi-thousand dollar cards? Or are you going by what's official?
Officially Grid K1, K2 and M60 (M6) are what you are supposed to use, it's supported.
VM Direct I/O (PCI Passthrough) is limited in what it can do, nVidia consumer cards do not work, except the Quadro Range (we've tested 4000, 5000), PCI passthrough was not really designed, to use consumer graphics card, so it's a workaround.
Your mileage will vary with unsupported consumer cards, if you can obtain them, and your biggest issue will be trying to get GPUs working in servers.
Because these blogs, and people that have done this, usually do it with workstations, and build your own whiteboxes! Not for production Services!
As I wrote some AMD cards do work, but will they work, in your server, or have correct power requirements!
So it comes down to your risk, your service....
Build a whitebox server in a desktop case, and out it in the datacenter, but is this a suitable solution?
It will put a tick in the box, and maybe work.
VM Direct I/O (PCI Passthrough) is limited in what it can do, nVidia consumer cards do not work, except the Quadro Range (we've tested 4000, 5000), PCI passthrough was not really designed, to use consumer graphics card, so it's a workaround.
Your mileage will vary with unsupported consumer cards, if you can obtain them, and your biggest issue will be trying to get GPUs working in servers.
Because these blogs, and people that have done this, usually do it with workstations, and build your own whiteboxes! Not for production Services!
As I wrote some AMD cards do work, but will they work, in your server, or have correct power requirements!
So it comes down to your risk, your service....
Build a whitebox server in a desktop case, and out it in the datacenter, but is this a suitable solution?
It will put a tick in the box, and maybe work.
ASKER
True about the cards actually working in the server itself. DL380 gen9's don't have 6-pin power required for 'decent' video cards. My issue being driver related has nothing to do with the server hardware but more with ESXi.
And sorry I didn't mean for it to sound like I want generic whiteboxes in my server room, cause I don't. But I could care less really what's official or supported seeing as how I don't pay for support.
I had hopes of people commenting here of what they've gotten to work as opposed to what VMware publishes.
And sorry I didn't mean for it to sound like I want generic whiteboxes in my server room, cause I don't. But I could care less really what's official or supported seeing as how I don't pay for support.
I had hopes of people commenting here of what they've gotten to work as opposed to what VMware publishes.
When I use the term support, it's more in terms of it does not work!
The biggest issue you will encounter, is you have a proper server, and do not have the correct cables.
Even when we purchase the Grid K1, K2 and M60, we need to purchase from Dell/HP the GPU conversion kits, and correct PSUs for the Servers, and we have to down grade the CPUs, because of the specific GPU and CPU combination because of power requirements,
Here you go, we come across this issue with clients, every week, and they ask us the same why can't we just stick a GTX 780 in the server! This is a nVidia issue, because GTX and Quadro use different firmwares. The GPU boards are mosly the same, but firmware changes the function of he card!
PCI Passthrough is troublesome at the best of times, don't assume because you have a PCI all will passthrough they dont!
As I've listed
nVidia 4000
nVidia 5000
Radeon HD 5750
Radeon HD 5450
Radeon HD 3450
Radeon HD 6850
Radeon HD 6800
These are cards we have tested and work with PCI Passthrough, if you have the correct PSU and connectors in your servers. There is an issue that we have spotted, that different vendors cards, and different motherboard in servers and workstations, can cause different scenarios, and BSODs or cards may not work.
It really is test and try it.
The biggest issue you will encounter, is you have a proper server, and do not have the correct cables.
Even when we purchase the Grid K1, K2 and M60, we need to purchase from Dell/HP the GPU conversion kits, and correct PSUs for the Servers, and we have to down grade the CPUs, because of the specific GPU and CPU combination because of power requirements,
I had hopes of people commenting here of what they've gotten to work as opposed to what VMware publishes.
Here you go, we come across this issue with clients, every week, and they ask us the same why can't we just stick a GTX 780 in the server! This is a nVidia issue, because GTX and Quadro use different firmwares. The GPU boards are mosly the same, but firmware changes the function of he card!
PCI Passthrough is troublesome at the best of times, don't assume because you have a PCI all will passthrough they dont!
As I've listed
nVidia 4000
nVidia 5000
Radeon HD 5750
Radeon HD 5450
Radeon HD 3450
Radeon HD 6850
Radeon HD 6800
These are cards we have tested and work with PCI Passthrough, if you have the correct PSU and connectors in your servers. There is an issue that we have spotted, that different vendors cards, and different motherboard in servers and workstations, can cause different scenarios, and BSODs or cards may not work.
It really is test and try it.
ASKER
I wish EE had some form of question parking or delaying. I will have hardware on order to try this by the end of the week so I don't really want to close the question until whether it will work or not is proven.
I'm getting a DL360 G9 with the GPU enablement kit and a Quadro M4000. HP did not indicate whether or not the CPU's I chose would need to be downgraded.
You sure do yell a lot there Andrew ;)
I'm getting a DL360 G9 with the GPU enablement kit and a Quadro M4000. HP did not indicate whether or not the CPU's I chose would need to be downgraded.
You sure do yell a lot there Andrew ;)
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
ROFL.
Yeah back in the day you could re-flash Quadros with Geforce FW and have yourself a nice up/side grade. I'll def report back my findings.
Yeah back in the day you could re-flash Quadros with Geforce FW and have yourself a nice up/side grade. I'll def report back my findings.
ASKER
This question fell off the radar because I quit that job recently. I am sorry for the trouble Hancock and yes your advice was spot on.
Sorry to hear about your job!
ASKER
Oh no.. it was for an upgrade ;)
oh, well that's even better. Good Luck!
Firstly very few AMD graphics card will work correctly in PCI Passthrough (VM Direct Path I/O), about 3 different devices do, in our testing and the BIOS, GPU need to be correct.
Standard nVidia cards are useless, e.g. GTX range, and you will need at least Quadro Cards, e.g. 4000's or later.
Yes, and no - you need to use Grid K1, K2 or M60, and then you need to purchase GPUs which are compatible and approved for use with your hardware, because of CPU and PSU requirements. (and you maybe the GPU conversion kits)