Exchange 2003 SP2 High Disk Utilization

Yesterday morning, a number of users complained about Outlook/OWA being slow.  After taking a look around the server (which was not very responsive), I noticed the HD LED on the front was on solid -- and has been since then.  Nothing has been changed on the server.  A reboot didn't resolve the problem.  Exchange is the only application on this single, dedicated server (no Front End).

The Exchange Database and Log files are D: drive partition on a RAID 5 volume (3x150GB) in a Dell PowerEdge SC1420 with a Dell CERC SATA 1.5/6ch RAID Controller, 2GB of RAM (3069MB Recommended Initial and Maximum Page File Size on C: drive).

According to the RAID Storage Manager, all of the drives are reporting "Optimal" and none have SMART errors.  The Read Cache has been enabled but the Write Cache never has been (since I've been here over the past 2 years).  I'm considering turning that on, but not if I don't have to to help prevent file corruptions if there's a hard shutdown.

I ran Perfmon and can easily see the high Disk Write value but not sure why.  RPC requests are relatively low.  Screenshot attached.

Stopping all of the Exchange Services significantly reduces the disk utilization and the server is very responsive (via RDP).  The utilization skyrockets again when they're turned back on.

Before yesterday, Backup Exec was backing up Exchange at ~1,500MB/sec.  When it kicked in last night, it was running at about 64MB/sec before I stopped it.

Running an Analysis in Disk Defragmenter shows 43% Total Fragmentation and 87% File Fragmentation.  I understand that doesn't necessarily cause problems like this.

I managed to run ESEUTIL /D on the Public Folder store last night.  It took 7 hours for a 2.1GB EDB and STM file.  The PRIV1.EDB is about 44GB and the STM is about 22GB.  I also understand we're reaching the hard 75GB limitation so we're planning on a new server early next year.

Hopefully that's enough information for you.  If not, please let me know what else you might need.  Thanks!!
APTMAIL.JPG
Wade_ChestnutAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

E_T_BCommented:
Sounds like some pretty old hardware and an excuse to replace it.

Have you tried shutting down any antivirus and see if that speeds it up?
Wade_ChestnutAuthor Commented:
Actually, something like this happened a couple of months ago but seemed to be resolved with the removal of the antivirus software.  This solely looks like an Exchange problem.  

My guess is that the Exchange database files need to first be physically defragmented, then use the ESEUTIL /D on them to improve the performance.  To my knowledge, that's never been run on these files since the server was installed 5 years ago.  I haven't found any recommendations to do so at all on a regular basis or not.
Wade_ChestnutAuthor Commented:
Is there any way I can tell what exactly is being consistently written to the disk?  The PF data in the Process List isn't revealing much.  Is it possible Exchange has been running the maintenance routine the past 2 days?
Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

Gerald ConnollyCommented:
According to your stats

1.7MB/sec  -  nothing to write home about
165 IOPS   -  a bit high even for a 3 disk RAID-5

Do you really have your DB and Logs on the same drive?
Wade_ChestnutAuthor Commented:
The HD LED on the front of the server has been solid for the past 3 days now since this all started and the server's been very sluggish when the Exchange services are running.  When they're stopped, the server responds normally.  The server responds better after I re-start them, but not for long.  So, logically, I'm looking at the hard drive and/or file system as the cause of this performance issue.

That screenshot was just at one point in time.  There's always a consistent value in the Disk Write Bytes/sec ranging from 500Kbps up to about 2.5Mbps (that I've seen) with Write Caching turned off.

Yes, the EDB and STM are in the same drive and folder as the LOG files.  I inherited this server and there are no other drives available.

UPDATE: Last night, I stopped the Exchange services for a while and ran Disk Defragmenter.  I stopped it since it was 2am and I needed to go home and sleep and I had to turn the services back on before morning.  When I stopped it, it looked like about 1/3 of the files were contiguous.  I don't see much of an improvement in overall server performance today.

Tonight, I'm taking Exchange offline for the weekend and starting with the completion of Disk Defragmenter before I start the ESEUTIL /D PRIV1.EDB -- which I'm estimating will take the weekend into Monday to complete.  I already ran that on the 5GB (total) Public Folders which took 7 hours.

I'm just worried that both disk and file defrags aren't going to solve the problem.
Wade_ChestnutAuthor Commented:
I ended up fixing it myself by stopping all of the Exchange service, running Disk Defragmenter, then ESEUTIL /D on the priv1.* files and restarted the services.  The server's performance is now back to what it was with the HD LED back to flashing occasionally.

What I don't understand is why this happened all-of-the-sudden.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Wade_ChestnutAuthor Commented:
Since I didn't get very many suggestions, I pursued a solution myself and it worked -- thankfully.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.