Virtual machines slow performance SoFS SMB 2012 R2 Hyper-V

I have a 3 cluster setup going with 3 physical nodes.

Cluster 1 - SoFS (Data)
Cluster 2 - VM Hosts Cluster (VS1, VS2)
Cluster 3 - Virtual SQL Cluster (SQL1, SQL2)

VS1/VS2 can send/receive files maxing out the 1Gbe connection at 110MB/sec perfectly.

However when communicating from Data to (virtual) SQL1, it starts out at 110MB/sec then drops to roughly ~20-40MB/sec after a second when transferring large files such as a SQL 2014 install ISO. During this transfer task manager shows 300Mbps~ usage.

What is causing this slow performance and how can I improve it?

More specs:

Using Intel Gigabit ET2 Quad port nics.
4 gigabit connections teamed via 2012 R2 teaming method for each virtual machine.
The data server has 8 gigabit connections teamed as well. All connect to the same physical switch.

Thanks!
Greg RogersAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Cliff GaliherCommented:
That sounds purely like disk I/O to me. It doesn't matter how fast the network is if data can only be fed from the disks at a slower speed. You'll start out at network speed as whatever is cached in memory goes through, but will slow as the cache empties and you start seeing direct disk performance.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Greg RogersAuthor Commented:
I would normally agree with you if it wasn't for there being 14 3TB drives of the best in that data server, and if the communication between VS1/VS2 and Data were the same.

Of course maybe it could be possible since perhaps the data is going from data -> vs1/vs2 -> back to data vhdx?

Not quite sure how data is routed when going from physical to virtual like that.


However I did just increase the VM's settings to all cpus, and from 1GB memory to 6GB memory, and it went thru 70% of a 4GB iso transfer before it dropped down to 20-40MB/sec, versus before that change it would go maybe 5% thru before it dropped.

Any ideas fellas? I prefer my vm's to not feed their transfers off memory.
Cliff GaliherCommented:
Upping memory and you get better transfers longer. What about that DOESN'T sound like disk I/O issues?

There are a lot of moving parts here. I don't know what disks you have (although 3TB disks do not, today, ever classify as "the best" in I/O, so I already disagree with your assessment there), but RAID config, cache, how SQL is configured for logs and temp storage; all these can explain what you are seeing, and comparing SQL to your SOFS and your VM cluster to your SOFS is comparing apples to spaceships. They have very different read and write patterns.

Basically I am more convinced from hour latest findings that you are hitting disk constraints now. But without a detailed topology with performance counters over time, I can't get more specific.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Virtual Server

From novice to tech pro — start learning today.