Solved

Recent Slow Website Performance in LAMP Setup When DB Job is Run in Background

Posted on 2014-09-06
2
236 Views
Last Modified: 2016-11-23
We are running a LAMP platform which has the following hardware:

-Two Dell R200's running Red Hat (one separate website on each) serving as web servers
-Cisco Smart switch
-One Dell 2950 w/32 gb RAM, 2Xquad core XEON 2.3 ghtz CPU's and 8 x 600gb SAS 10k RPB drives (two RAID groups 1-  RAID1 for Solaris OS and 2-  5+1 RAID5 w/hot spare

Things seem to be going well  until about a month ago, when the performance of the websites would take huge latency hits whenever we ran a database job in the background. Bear in mind that we have been doing this in production since early November, 2013.

Now, however, whenever we run one of these jobs (e.g., updating tables, backing up database, etc.) the response rate for the server tanks severely.

We've absolutely ruled out the issues being related to the network or the Web servers (after many hours of diagnostics).

We carefully checked out all aspects of the Dell 2950. We've run Dell diagnostics against the entire system (as well as Solaris CPU, memory test commands) and it reports that all of the CPU threads are healthy, the RAM is 100% operational in the disk drives in the raid groups are fine.

We decided that the volumes on the server are exceeding some performance threshold and we think that we're going to drop two gigabit network cards (Intel E1G42ET) into the 2950 and move everything into a Dell 3200 ISCSI San.

We could then re-create the luns across 12 drives (and probably continue to use the 2950's first raid one group for the OS).

My question is: will this alleviate these latency issues that have popped up over the last month?  What might be a better approach?
0
Comment
Question by:bhunger
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
2 Comments
 
LVL 11

Accepted Solution

by:
Murfur earned 500 total points
ID: 40312339
By your troubleshooting you seem to have decided that the problem must lie with the Dell 2950 offering the RAID but I'm not so sure.

My initial guess would be that the problem lies within MySQL

What is the slow MySQL job actually trying to do? Where is your MySQL installation - local or remote? Assumption is local given that you looked to the RAID for the solution but if not actually connecting to "localhost" then you might want to prevent MySQL from making DNS lookups by adding skip-name-resolve to your MySQL conf.

If you are running backups/dumps then you might want to consider taking them off that server as processes go, they are notoriously resource hungry. For example, you could set up MySQL replication and have the jobs run from the slave thereby leaving the sites to access the master with "normal" performance. FYI a master server will handle all inserts & updates but a slave will only accept selects and the backups running from that won't affect performance on the master.

Digging deeper, you need to look at all elements of the transaction as any one of them could be the cause of the bottleneck:
Web site -> web server -> MySQL service -> MySQL server and of course the connections in-between them all.

Activate the slow query log to see if you can spot a particular bottleneck. Look out for table locks or queries with too many joins etc.

That should get you going for a start but let me know how you get on...
0
 

Author Closing Comment

by:bhunger
ID: 40313089
Thanks for the answer.  We'll follow your recommendations and see where it leads us.
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Ready to improve network connectivity? Watch this webinar to learn how SD-WANs and a one-click instant connect tool can boost provisions, deployment, and management of your cloud connection.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Lets start to have a small explanation what is VAAI(vStorage API for Array Integration ) and what are the benefits using it. VAAI is an API framework in VMware that enable some Storage tasks. It first presented in ESXi 4.1, but only after 5.x sup…
Data center, now-a-days, is referred as the home of all the advanced technologies. In-fact, most of the businesses are now establishing their entire organizational structure around the IT capabilities.
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

632 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question