Solved

Infiniband Connection to ESXi 5.0 Server

Posted on 2013-02-01
4
1,301 Views
Last Modified: 2013-02-11
Hi, I have an ESXi 5.0 host with a mix of Server 2008R2 and 2012 Servers guests.  I am trying to connect a second physical Server 2012 machine to this host via Infiniband so that I'll have a fast link to the host for backup purposes.  This is a direct connection - I don't have an infiniband switch; just two cards and a cable.  OpenSM is running on the standalone Server 2012 machine.

Ideally, I would like to set the Mellanox as pci-pass-through to one of the VM's; especially a Server 2012 VM so that I can take advantage of the new SMB 3.0 functionality.  However, I have not been able to do this as the Server 2012 VM will crash whenever I add the card to its configuration.  Interesting however, that in this paper, it appears that pci passthrough is possible:  http://labs.vmware.com/publications/ib-researchnote-apr2012  But after some googling, others are having the same reboot problem I have.

Therefore, my second alternative is not to pass-through, but to setup the card as IPoIB at the ESXi host level.  I've done this and at this point, I can ping between a VM and the standalone Server 2012 Server.  On the guest VM, I have two vNIC's, a "regular" one with internet access which has an IP of 192.168.1.2/255.255.255.0.  The second vNIC is on the "infiniband" network which is set to 192.168.2.1/255.255.255.0 - there is no default gateway set on this second vNIC.

On the standalone server, the setup is similar - 192.168.1.3 for "regular" and 192.168.2.2 for the infiniband card.  On this server, I removed Windows File and Printer sharing on the regular NIC, but kept it checked on the Infiniband NIC with the hope that this will "channel" all the file sharing / copying packets via the fast infiniband connection (not sure if this is the best way to do it, but the only way I thought of).

My problem at this point is that while I can ping from each machine to the other via the Infiniband connection (eg ping 192.168.2.1 and ping 192.168.2.2), this connection is showing up as "public and unidentified" by Windows on both the VM guest and the standalone server.  As such, I can't see the shared folders I have set up and without that, I can't copy the files to be backed up to the standalone 2012 Server.

Any thoughts either to solve the public and unidentified network problem or is there a better way to accomplish what I an trying to do.  

If there is a way to pass through the Infiniband card, that would be best, but I can live with an alternative.  The cards I have are MHQH19B-XTR and both have been updated with the latest firmware from the Mellanox website.

Thank you!
0
Comment
Question by:Hammer8
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
4 Comments
 
LVL 30

Expert Comment

by:IanTh
ID: 38846451
see

http://social.technet.microsoft.com/Forums/en-US/winserver8setup/thread/ac1de5fe-335a-46bc-8fc2-45a96397f937

infiniband does your host see the infiniband in the pass-though config as a device
0
 

Author Comment

by:Hammer8
ID: 38846610
Yes, my hose sees the infiniband adapter as a passthrough device and I can add it as a PCI device to my Server 2012 guest VM, but when I go to start the VM, it crashes during start up.

I was able to set the network to private using mmc - basically change unidentified networks to be private instead of public.  But it still does not share properly - to share properly, I had to disable the non infiniband vNic (eg, the 192.168.1.2 adapter) and now, I can connect to 192.168.2.2 and see the shares on that server.  However, that defeats the purpose of this as I need the our regular clients on 192.168.1.x to see that server in order to access the files on it - the same files I want to backup via infiniband to 192.168.2.2!
0
 

Accepted Solution

by:
Hammer8 earned 0 total points
ID: 38847447
Hi, just to update readers.  I was able to solve my problem by creating a new VM Group under my existing vSwitch with just the Infiniband NIC and called it VM Network - IB.  Then in each VM that I would like access to the external server via Infiniband, I added a new vNIC and set that as belonging to the new VM Group.  This method avoids having to use a new subnet and I'm able to use 192.168.1.x on all NIC's including the Infiniband NIC on the standalone server.

The downside is I don't have internet access on the standalone server...

I was not able to figure a way to do hardware passthrough...
0
 

Author Closing Comment

by:Hammer8
ID: 38875253
Did not see an acceptable solution.
0

Featured Post

Efficient way to get backups off site to Azure

This user guide provides instructions on how to deploy and configure both a StoneFly Scale Out NAS Enterprise Cloud Drive virtual machine and Veeam Cloud Connect in the Microsoft Azure Cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
VM delta disk 2 50
VMotion Direct Attached Storage 9 95
No IP Address Assigned to VM 10 93
ESXI 5.5 image can't expand the hard disk 6 57
When rebooting a vCenters 6.0 and try to connect using vSphere Client we get this issue "Invalid URL: The hostname could not parsed." When we get this error we need to do some changes in the vCenter advanced settings to fix the issue.
It’s been over a month into 2017, and there is already a sophisticated Gmail phishing email making it rounds. New techniques and tactics, have given hackers a way to authentically impersonate your contacts.How it Works The attack works by targeti…
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…
Teach the user how to join ESXi hosts to Active Directory domains Open vSphere Client: Join ESXi host to AD domain: Verify ESXi computer account in AD: Configure permissions for domain user in ESXi: Test domain user login to ESXi host:

733 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question