Go Premium for a chance to win a PS4. Enter to Win


Recent Slow Website Performance in LAMP Setup When DB Job is Run in Background

Posted on 2014-09-06
Medium Priority
Last Modified: 2016-11-23
We are running a LAMP platform which has the following hardware:

-Two Dell R200's running Red Hat (one separate website on each) serving as web servers
-Cisco Smart switch
-One Dell 2950 w/32 gb RAM, 2Xquad core XEON 2.3 ghtz CPU's and 8 x 600gb SAS 10k RPB drives (two RAID groups 1-  RAID1 for Solaris OS and 2-  5+1 RAID5 w/hot spare

Things seem to be going well  until about a month ago, when the performance of the websites would take huge latency hits whenever we ran a database job in the background. Bear in mind that we have been doing this in production since early November, 2013.

Now, however, whenever we run one of these jobs (e.g., updating tables, backing up database, etc.) the response rate for the server tanks severely.

We've absolutely ruled out the issues being related to the network or the Web servers (after many hours of diagnostics).

We carefully checked out all aspects of the Dell 2950. We've run Dell diagnostics against the entire system (as well as Solaris CPU, memory test commands) and it reports that all of the CPU threads are healthy, the RAM is 100% operational in the disk drives in the raid groups are fine.

We decided that the volumes on the server are exceeding some performance threshold and we think that we're going to drop two gigabit network cards (Intel E1G42ET) into the 2950 and move everything into a Dell 3200 ISCSI San.

We could then re-create the luns across 12 drives (and probably continue to use the 2950's first raid one group for the OS).

My question is: will this alleviate these latency issues that have popped up over the last month?  What might be a better approach?
Question by:bhunger
LVL 11

Accepted Solution

Murfur earned 2000 total points
ID: 40312339
By your troubleshooting you seem to have decided that the problem must lie with the Dell 2950 offering the RAID but I'm not so sure.

My initial guess would be that the problem lies within MySQL

What is the slow MySQL job actually trying to do? Where is your MySQL installation - local or remote? Assumption is local given that you looked to the RAID for the solution but if not actually connecting to "localhost" then you might want to prevent MySQL from making DNS lookups by adding skip-name-resolve to your MySQL conf.

If you are running backups/dumps then you might want to consider taking them off that server as processes go, they are notoriously resource hungry. For example, you could set up MySQL replication and have the jobs run from the slave thereby leaving the sites to access the master with "normal" performance. FYI a master server will handle all inserts & updates but a slave will only accept selects and the backups running from that won't affect performance on the master.

Digging deeper, you need to look at all elements of the transaction as any one of them could be the cause of the bottleneck:
Web site -> web server -> MySQL service -> MySQL server and of course the connections in-between them all.

Activate the slow query log to see if you can spot a particular bottleneck. Look out for table locks or queries with too many joins etc.

That should get you going for a start but let me know how you get on...

Author Closing Comment

ID: 40313089
Thanks for the answer.  We'll follow your recommendations and see where it leads us.

Featured Post

Nothing ever in the clear!

This technical paper will help you implement VMware’s VM encryption as well as implement Veeam encryption which together will achieve the nothing ever in the clear goal. If a bad guy steals VMs, backups or traffic they get nothing.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The business world is becoming increasingly integrated with tech. It’s not just for a select few anymore — but what about if you have a small business? It may be easier than you think to integrate technology into your small business, and it’s likely…
Windows Server 2003 introduced persistent Volume Shadow Copies and made 2003 a must-do upgrade.  Since then, it's been a must-implement feature for all servers doing any kind of file sharing.
In this video, Percona Director of Solution Engineering Jon Tobin discusses the function and features of Percona Server for MongoDB. How Percona can help Percona can help you determine if Percona Server for MongoDB is the right solution for …
In this video, Percona Solutions Engineer Barrett Chambers discusses some of the basic syntax differences between MySQL and MongoDB. To learn more check out our webinar on MongoDB administration for MySQL DBA: https://www.percona.com/resources/we…
Suggested Courses

824 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question