Solved

Recent Slow Website Performance in LAMP Setup When DB Job is Run in Background

Posted on 2014-09-06
2
227 Views
Last Modified: 2016-11-23
We are running a LAMP platform which has the following hardware:

-Two Dell R200's running Red Hat (one separate website on each) serving as web servers
-Cisco Smart switch
-One Dell 2950 w/32 gb RAM, 2Xquad core XEON 2.3 ghtz CPU's and 8 x 600gb SAS 10k RPB drives (two RAID groups 1-  RAID1 for Solaris OS and 2-  5+1 RAID5 w/hot spare

Things seem to be going well  until about a month ago, when the performance of the websites would take huge latency hits whenever we ran a database job in the background. Bear in mind that we have been doing this in production since early November, 2013.

Now, however, whenever we run one of these jobs (e.g., updating tables, backing up database, etc.) the response rate for the server tanks severely.

We've absolutely ruled out the issues being related to the network or the Web servers (after many hours of diagnostics).

We carefully checked out all aspects of the Dell 2950. We've run Dell diagnostics against the entire system (as well as Solaris CPU, memory test commands) and it reports that all of the CPU threads are healthy, the RAM is 100% operational in the disk drives in the raid groups are fine.

We decided that the volumes on the server are exceeding some performance threshold and we think that we're going to drop two gigabit network cards (Intel E1G42ET) into the 2950 and move everything into a Dell 3200 ISCSI San.

We could then re-create the luns across 12 drives (and probably continue to use the 2950's first raid one group for the OS).

My question is: will this alleviate these latency issues that have popped up over the last month?  What might be a better approach?
0
Comment
Question by:bhunger
2 Comments
 
LVL 11

Accepted Solution

by:
Murfur earned 500 total points
ID: 40312339
By your troubleshooting you seem to have decided that the problem must lie with the Dell 2950 offering the RAID but I'm not so sure.

My initial guess would be that the problem lies within MySQL

What is the slow MySQL job actually trying to do? Where is your MySQL installation - local or remote? Assumption is local given that you looked to the RAID for the solution but if not actually connecting to "localhost" then you might want to prevent MySQL from making DNS lookups by adding skip-name-resolve to your MySQL conf.

If you are running backups/dumps then you might want to consider taking them off that server as processes go, they are notoriously resource hungry. For example, you could set up MySQL replication and have the jobs run from the slave thereby leaving the sites to access the master with "normal" performance. FYI a master server will handle all inserts & updates but a slave will only accept selects and the backups running from that won't affect performance on the master.

Digging deeper, you need to look at all elements of the transaction as any one of them could be the cause of the bottleneck:
Web site -> web server -> MySQL service -> MySQL server and of course the connections in-between them all.

Activate the slow query log to see if you can spot a particular bottleneck. Look out for table locks or queries with too many joins etc.

That should get you going for a start but let me know how you get on...
0
 

Author Closing Comment

by:bhunger
ID: 40313089
Thanks for the answer.  We'll follow your recommendations and see where it leads us.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Suggested Solutions

AWS Glacier is Amazons cheapest storage option and is their answer to a ‘Cold’ storage service.  Customers primarily use this service for archival purposes and storage of infrastructure backups.  Its unlimited storage potential and low storage cost …
How to update Firmware and Bios in Dell Equalogic PS6000 Arrays and Hard Disks firmware update.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This tutorial will walk an individual through the process of installing the necessary services and then configuring a Windows Server 2012 system as an iSCSI target. To install the necessary roles, go to Server Manager, and select Add Roles and Featu…

760 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

24 Experts available now in Live!

Get 1:1 Help Now