Solved

Recent Slow Website Performance in LAMP Setup When DB Job is Run in Background

Posted on 2014-09-06
2
228 Views
Last Modified: 2016-11-23
We are running a LAMP platform which has the following hardware:

-Two Dell R200's running Red Hat (one separate website on each) serving as web servers
-Cisco Smart switch
-One Dell 2950 w/32 gb RAM, 2Xquad core XEON 2.3 ghtz CPU's and 8 x 600gb SAS 10k RPB drives (two RAID groups 1-  RAID1 for Solaris OS and 2-  5+1 RAID5 w/hot spare

Things seem to be going well  until about a month ago, when the performance of the websites would take huge latency hits whenever we ran a database job in the background. Bear in mind that we have been doing this in production since early November, 2013.

Now, however, whenever we run one of these jobs (e.g., updating tables, backing up database, etc.) the response rate for the server tanks severely.

We've absolutely ruled out the issues being related to the network or the Web servers (after many hours of diagnostics).

We carefully checked out all aspects of the Dell 2950. We've run Dell diagnostics against the entire system (as well as Solaris CPU, memory test commands) and it reports that all of the CPU threads are healthy, the RAM is 100% operational in the disk drives in the raid groups are fine.

We decided that the volumes on the server are exceeding some performance threshold and we think that we're going to drop two gigabit network cards (Intel E1G42ET) into the 2950 and move everything into a Dell 3200 ISCSI San.

We could then re-create the luns across 12 drives (and probably continue to use the 2950's first raid one group for the OS).

My question is: will this alleviate these latency issues that have popped up over the last month?  What might be a better approach?
0
Comment
Question by:bhunger
2 Comments
 
LVL 11

Accepted Solution

by:
Murfur earned 500 total points
ID: 40312339
By your troubleshooting you seem to have decided that the problem must lie with the Dell 2950 offering the RAID but I'm not so sure.

My initial guess would be that the problem lies within MySQL

What is the slow MySQL job actually trying to do? Where is your MySQL installation - local or remote? Assumption is local given that you looked to the RAID for the solution but if not actually connecting to "localhost" then you might want to prevent MySQL from making DNS lookups by adding skip-name-resolve to your MySQL conf.

If you are running backups/dumps then you might want to consider taking them off that server as processes go, they are notoriously resource hungry. For example, you could set up MySQL replication and have the jobs run from the slave thereby leaving the sites to access the master with "normal" performance. FYI a master server will handle all inserts & updates but a slave will only accept selects and the backups running from that won't affect performance on the master.

Digging deeper, you need to look at all elements of the transaction as any one of them could be the cause of the bottleneck:
Web site -> web server -> MySQL service -> MySQL server and of course the connections in-between them all.

Activate the slow query log to see if you can spot a particular bottleneck. Look out for table locks or queries with too many joins etc.

That should get you going for a start but let me know how you get on...
0
 

Author Closing Comment

by:bhunger
ID: 40313089
Thanks for the answer.  We'll follow your recommendations and see where it leads us.
0

Featured Post

NAS Cloud Backup Strategies

This article explains backup scenarios when using network storage. We review the so-called “3-2-1 strategy” and summarize the methods you can use to send NAS data to the cloud

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Ever notice how you can't use a new drive in Windows without having Windows assigning a Disk Signature?  Ever have a signature collision problem (especially with Virtual Machines?)  This article is intended to help you understand what's going on and…
Learn about cloud computing and its benefits for small business owners.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

920 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now