Solved

SQL 2005 High CPU on Server 2008 VM

Posted on 2012-12-28
9
1,282 Views
Last Modified: 2016-11-23
Aloha,

I have the current setup:

Dell R420 w/ PERC H310 controller
32 gb RAM
1.8tb local storage, RAID 10 (4x7.2K SAS drives)
VMWare Essentials 5

There are only two VMs on this server. Both are Server 2008 R2.

I am having a problem with one of the VMs. It is running SQL 2005 SP4. Whenever I launch the application (Microsoft Dynamics RMS 2.0) that accesses the database in SQL 2005, CPU usage hits 100% and the application often freezes and times out. In the times that the app does not time out, I can access the data through the application without any problems (with the exception of running reports, which are slow and tax the CPU).

The VM is configured for 16gb RAM and 2 socket x 2 core processors (4 total processors).

The machine I migrated this database from is six years old, with 4 gigs of RAM, RAID 1 (mirrored drives) and a single processor. Performance is great :)

Can someone point out where I have a configuration problem/issue? I have ran esxtop and  DAVG/rd/wr seldom exceed 15ms (same with GAVG/rd/wr). I initially thought it was a performance issue due to the 7.2K drives, but that doesn't seem to be the problem.

Any advice would be greatly appreciated. Apologies if I don't respond until Monday, it's almost the weekend (thankfully)

Many Thanks,

Tony
0
Comment
Question by:tlasoya808
  • 5
  • 2
  • 2
9 Comments
 
LVL 119

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE^2) earned 500 total points
ID: 38728572
Okay, some advice...

4 vCPUs could be TOO MANY, and if the VM cannot take advantage of the vCPUs, this can have the drastic effect of slowing the VM down, due to the vSMP scheduler within the VM.

So, drop the cpus in the VM to one. Check performance and add another.

Also use Sockets, and not cores (cores are used unless you need to specifically allocate cores for licensing reasons).

I was also going to suggest that 7200rpm drives could be too slow.

Check that you are not operating on a snapshot disk, see my EE Article, CPU utilisation increases when operating on a snapshot.

HOW TO:  Performance Monitor vSphere 4.x or 5.0

HOW TO:  VCP 5

Checkout Best Practice Guide
http://www.vmware.com/files/pdf/sql_server_best_practices_guide.pdf

SQL Best Practices
http://communities.vmware.com/docs/DOC-13249

Now finally, and this may not be want you want to hear, but this is not the first time or last time, I'll see this on EE or in real life, we've migrated many SQL servers Back to Physical Hardware because the Hypervisor (VMware or Microsoft), "sucks" too much performance away from the VM, causing High CPU when reports are run, causing the CPU to spike at 100%, until the reports have stopped running.

See my video here...

Video - Poor Microsoft SQL Performance caused by Virtualisation - Not all servers are equal!

I've had to obsure the data in the above video, because it had client data in the report, but this was a HP DL360G5 Server, with 64GB RAM, Dual Processor Quad Core Server, with a single VM, with 2 vCPUs, 16GB RAM, Windows 2008 R2 64 bit, and SQL 2008 running a concurrent database (500MB), the host OS was ESXi 5.0. 15k SAS disks.

When a single query was sent to the db, it would lock the VM and Host, at 100% for 11-13mins, when the query was finished CPU would return to normal, the VM and host operations were unusable.

The same server, reduced to 8GB RAM with No Hypervisor (ESXi), completed the query in seconds.

So we migrated back to the physical server.
0
 

Author Comment

by:tlasoya808
ID: 38733087
Thank you for the great information.

I scaled back the processor to a single socket. Horrible performance. Increased it to two sockets, same horrible performance. And this is just when the application is launched from the local server. CPU hits 100% and stays there until the application times out.

Once in the application, however, performance is quick (again, slowing down when a large report is ran).

I will read through the best practice guides. I have a hard time believing that anything in there will really improve performance. It could be this particular database (which was created before virtualization was mainstream). Or it can just be that this server just cant hack it. This is a 9gb database with no more than 10 concurrent connections at any given time. I am not a SQL expert by any stretch of the imagination, so there may be maintenance that can and should have been done to this database to increase performance, but as I stated before it is running great on the old server.

Just for fun I put this database on my laptop running Win 7 32 bit, 3gb ram Intel i5 processor. I installed SQL 2005. Performance was quick, both locally and accessing from a remote machine.

I am going to have to abandon the hypervisor on the new server, aren't I?
0
 
LVL 119
ID: 38733352
I'm sorry to say, that not all servers can be virtualised, as much as we would like to think so.
0
Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

 

Author Comment

by:tlasoya808
ID: 38733422
Just for fun, I am converting the current SQL server to the virtual environment, just to see how it performs.

I read the documents you sent me and the server is as optimized as it can be. Will keep you posted, but I suspect for this particular server virtualization may not be an option.
0
 
LVL 12

Expert Comment

by:Carlo-Giuliani
ID: 38733624
Hanccocka has stated many times (in his comments on this question, and others) that  SQL is prone to mysterious performance problems when virtualized.  Your seems like an example.   But I find it hard to accept such an extreme penalty for virtualization of a workload as light as the one you describe here.  I understand that you can't spend an unlimited amount of time trying to solve the mystery, but it is clear that there is an underlying issue here that none of us understand.   You should be able to virtualize this system.
0
 

Author Comment

by:tlasoya808
ID: 38733681
I agree with you, Carlo.

I finished with the P2V of the current production server. It performs flawlessly. No issues with SQL connections both locally and from client machines. This is a Server 2003 machine running SQL 2005.

What am I to make of this?
0
 
LVL 12

Expert Comment

by:Carlo-Giuliani
ID: 38733790
I would love to know what made the other instance perform so badly.  Generic statements like "don't virtualize SQL" don't really help anybody.  But I'm afraid I have no idea what caused the issue you ecountered.

Are you sure there was no contention for memory?  You mentioned you had given the SQL VM 16GB on a hypervisor with 32GB.  Were there other VMs totaling more than 12GB on the server?  Hypervisors are generally good at sharing CPUs, but bad at sharing overcommited memory.
0
 

Author Comment

by:tlasoya808
ID: 38733807
No, there is one other machine on the hypervisor, with 8gb of RAM assigned to it. This new P2V machine has 4gb assigned to it.

The P2V server continues to perform very well, the Server 2008 VM continues to struggle. Other than the OS, the only other difference is that the 2003 server is 32-bit and the 2008 server is 64-bit. Same with the versions of SQL.
0
 

Author Comment

by:tlasoya808
ID: 38742442
Just more info. I created a 64-bit Win 7 VM w/ 2 proc and 8gb RAM. Installed SQL2005 and the RMS application. The vm performed flawlessly. Clients connected without issue and performance was excellent. So, yeah, I either missed something (or botched something) in the Server 2008 install (can't imagine what) or there is an issue with this database/sql2005/server2008 running in ESX.
0

Featured Post

Active Directory Webinar

We all know we need to protect and secure our privileges, but where to start? Join Experts Exchange and ManageEngine on Tuesday, April 11, 2017 10:00 AM PDT to learn how to track and secure privileged users in Active Directory.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

HOW TO: Install and Configure VMware vSphere Hypervisor 6.5 (ESXi 6.5), Step by Step Tutorial with screenshots. From Download, Checking Media, to Completed Installation.
HOW TO: Upload an ISO image to a VMware datastore for use with VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere Host Client, and checking its MD5 checksum signature is correct.  It's a good idea to compare checksums, because many installat…
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This tutorial will show how to configure a new Backup Exec 2012 server and move an existing database to that server with the use of the BEUtility. Install Backup Exec 2012 on the new server and apply all of the latest hotfixes and service packs. The…

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question