Live Migrations are Slow on 10GB Network

When doing live migrations, transfers do not break 1GB even though we have a 10GB connection from host to host. The transfer mostly hang around the 600 - 900 Mbps range. A normal VM takes 10 minutes or so to move. In the very last second or two of a migration, task manger will show a transfer speed around 9GB.

What can I do to get that kind of speed the whole time? Here is my setup and what I have tried.

(2) Windows 2012 R2 servers running only the Hyper-V roll.
Intel S2600GZ board.
(2) 8 core 2.6 GHz Xeon E5-2650 processors.
96 GB of memory.
(2) Intel X540-T2 10GB NICs. (4 10GB ports per server)

I've enabled jumbo packets and can successfully ping with an 8000 byte unfragmented packet from host to host.
I've disabled all C states in BIOS
These servers have a four port 1GB NIC in them. I've disabled that in BIOS.
I've checked the CPU during transfer - no core maxes out at all (doesn't bring 10% usage).
I've tried a variety of live migrations setups (with compression and without). No difference that I can see.
I've tried a direct connection between these two host with no switch in the center. No difference.
I've teamed the NICs in Windows - seemed to add about 300 MBs to the transfers (that is why I have the 600-900 range)
I've configured the NICs to use the Hyper-V profile in the Intel management software.
I've updated the firmware on the servers to the latest available.

Finally, I let SCVMM manage the NICs (knowing that I shouldn't expect more than 3.5 Gbps due to VMQ). The live migration traffic then gets spilt between three vNICs (live migration, host management, client access). I found this behavior very odd because I have the IP range used on the Live Migration network as the only specified range to use under Migration Settings.

So where should I start looking now?
LVL 22
Joseph MoodyBlogger and wearer of all hats.Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Philip ElderTechnical Architect - HA/Compute/StorageCommented:
We work with a very similar setup.

In our case we set up each 10GbE port on its own subnet (192.168.100.0/24 - 192.168.+10.0/24). In this setting we should see four LM streams running across each 10GbE port. Real transfer rates would be ~800MB/Second or thereabouts per 10GbE port.

I have an EE Article on Hyper-V Hardware and Software Best Practices that should help with performance tuning.

The Performance Category on our blog (MPECS Inc.)

Make sure your Intel NIC drivers are up to date. Make sure your cables are CAT6 or CAT6a and above.
0
Joseph MoodyBlogger and wearer of all hats.Author Commented:
Thank you for the reply Philip. Great EE article and blog! I will be digging through both resources.

Three things to update this question with:

BIOS has high performance enabled for the CPU.
Cables are all cat6a - NICs all show 10gb connections.
Real time protection has the correct Hyper-V/SCVMM exclusions.
0
Philip ElderTechnical Architect - HA/Compute/StorageCommented:
A/V client should not be installed on the host. There is no need to put it there. It only saps resources.

Remove it and try again.
0
Introducing the "443 Security Simplified" Podcast

This new podcast puts you inside the minds of leading white-hat hackers and security researchers. Hosts Marc Laliberte and Corey Nachreiner turn complex security concepts into easily understood and actionable insights on the latest cyber security headlines and trends.

Michael OrtegaSales & Systems EngineerCommented:
You do realize that you can't transfer over the network faster than the max read/write speeds of your disk subsystem, right? For example, you can have a 10G network but if you're live migrating from a SATA 7.2k disk subsystem to another one you're not going to get anywhere near 10G transfer speeds. Not even 10k or 15k SAS drives will achieve 10Gbps (1.25GBps) throughput unless you have a ton of spindles.

Another consideration is the RAID Controller. It may only be 6Gbps which would also be less than the 10Gbps line speed at the network level.

If you want to measure your max throughput in MB/s, try this: http://timradney.com/2012/07/06/iops-verses-throughput/

MO
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Joseph MoodyBlogger and wearer of all hats.Author Commented:
Thank you both for your help! The network was not my bottle neck - it was with my disks.
0
Michael OrtegaSales & Systems EngineerCommented:
That is what I suspected. Sorry about that.

MO
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Hyper-V

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.