Hello,
I'm administering a RAID5 Array running on a Windows Server 2008 system using a HighPoint RocketRAID 2320 controller and Seagate Barracuda 7200.11 1.5TB drives (w/ latest firmware).
The system uses a software based RAID1 Array on separate hard disks for the OS and allocats all of the RAID5 array to a single 'Data' partition.
We recently added another drive to RAID5 array, raising the number of drives from 3 (3 TB of usable space), to 4 (4.5 TB of usable space). I added the drive, expanded the array live via the Highpoint RAID Management Utility, and then added the new unformatted/unpartitioned space to the 'Data' partition using Disk Management "Extend Volume..." utility.
Everything worked fine, but now, a day after extending the partion, we seem to be experiencing a performance hit on the partition. We stream .wav files off of this partition over a local network, and we're not getting the throughput we were before, leading to periodic skipping in the audio.
Is this just a temporary artifact of Server 2008 completing it's partiton expansion onto the new space? When I extended the volume, it did it immediately, but I imagine there is still some background time it takes to have the volume completely up and back to full performance? Is this "immediate" expansion hiding the underlying process that actually expands and formats the new space, thus leading to a temporary performance hit immediately after expanding?
I am currently planning on waiting a few days and hoping the problem fixes itself, but any advise or insight would be appreciated.
Thanks,
Andy
This Question has been solved and asker verified All Experts Exchange premium technology solutions are available to subscription members.
Experts Exchange has been collecting answers to technology questions since 1996…3 million and counting! If you have a question, chances are we already have your answer.
If you can't find the exact answer you're looking for, ask our exclusive community of 50,000 experts. You’ll get a personalized answer from a trusted professional.
Thousands of free tech tips, tricks, how-to’s and tutorials are available in our peer reviewed articles section. See for yourself how smart our experts are, no login required.
Save solutions to your questions, answers you’ve discovered through searching plus helpful articles in your personal knowledgebase for easy future access.
Access the answers to your technology questions today.
30-day free trial. Register in 60 seconds.
Members of the expert community talk about why the experience at Experts Exchange is different than what you will find anywhere else.
Try it out and discover for yourself.
30-day free trial. Register in 60 seconds.
Join the community of experts here and help other tech pros by answering question in your area of expertise. You can earn FREE access to all Experts Exchange's premium features and resources.
The skipping disk read errors always seem to be accompanied by the "Highest Active Time" graph in the Reliability and Performance Monitor window going to 100%. This is followed by the disk throughput dropping to 0% and the streaming audio skipping.
I have attached an image of this occurrence. I am defragmenting the disk during the attached example to help force the throughput up and "encourage" the performance error. You can see the Defragmenter running, as well as the streaming .wav files, on the performance monitor. The error is highlight with the red box on the graph near the top.
Any suggestions would be appreciated!
Thanks,
Andy
Disk Error Screen Shot
There's a good chance you're correct in assuming the restructuring of the array isn't yet completed. A drive rebuild with a software-based array can easily take 2-3 days with 1.5TB component drives. Even with a hardware controller it's a 2-day process to add a drive with arrays of this size.
Wait 2-3 days and see if this "self corrects" ... I suspect it will :-)
It's been a week and we'res till seeing spikes and performance gaps when the array is under load. I've RMAed the newest drive and will hopefully have another to test soon.
Any other ideas?
I've attached another image showing the 100% Active ping as well as the corresponding 0% throughput drop.
-Andy
Perfromance Monitor Screenshot
"... I've RMAed the newest drive ..." ==> So are you running at risk? Or is the drive still installed until the replacement arrives?
What kind of array management interface do you have with the controller you're using? It should show the rebuild status (which should, after a week, have certainly finished -- it CAN take several days ... especially with very large drives).
Note that the Seagate 1.5TB drives -- even with the latest firmware (which corrects most of the well-known "stuttering" issues this drive had when first released) -- does NOT work well with many RAID controllers. That may be the root cause of your issue. I've seen many examples on the A/V forums of large storage arrays that simply wouldn't work with the Seagate drives -- Areca, Adaptec, etc. all have problems with these drives.
Our RAID5 Array Server is running in parallel with a mirrored redundant server that all primary operations have been shifted over to at the onset of this issue. We pulled the drive from the problematic array, and thus, that array in running at risk, but our mirrored server if fully up and still redundant, so this risk is fairly well mitigated.
The Array management Software (form Highpoint) showed the original expansion as complete several weeks ago. The questions as to whether or not the space expansion had completed lies in Windows' partition expansion onto the new space, which shows no status and seems to complete "instantly" when run from the Disk Management utility.
Our mirrored redundant server is a perfect twin to the server showing problems, with the same RAID controller, drives, etc. The mirrored server is showing no errors with these drives after completing an identical expansion process. It is now running under the same full load as the problem server (all operations have been switched over) without any of the audible streaming glitches or performance spikes evident on the main server.
-Andy
Very interesting. Given all that, I'd say the new drive was likely the culprit ... what is the performance like now (without it)??
Note that the Windows partition expansion after a RAID rebuild is indeed VERY quick -- once the array's been restructured (the rebuild) it just takes a second or so to modify the sizing info in Windows. In re-reading your question, you did in fact indicate the resize was already done ... I missed that the first time.
We replaced the newest drive in the array and it seems to have fixed our problem.
Disturbingly, however, is the fact that we briefly saw the same problem on our backup server (same config, see above) yesterday. A restart of the server seems to have fixed it, but there still seem to be some ghosts in the machine.
Any ideas are still appreciated.
Thanks,
Andy
"... ghosts in the machine ..." can be very difficult to isolate. I can't really think of anything that jumps out as a likely culprit with the possible exception of the Seagate 1.5TB drives. These are known to have issues with some RAID controllers ... and although they seem to be working well in your case, they still could be causing occasional "hiccups" if the recalibration timeouts exceed a particular threshold. There are several threads in the AVS forums about large arrays with various controllers -- and the Seagate drives are often the cause of problems with those arrays.
Error is still occurring, far less frequently than before, but now on both servers. I have attached a very good screenshot of the Performance Monitor showing the error occurring (this manifests itself as a skip in our streaming audio).
This screenshot is taken from our backup server (acting as the primary server while we are investigating the problem on the primary). Is shows the Active Time graph spiking at the moment the skip occurs.
What does Active Time represent? Any ideas what this could be indicating?
Active Time Graph Spike
Afraid that doesn't ring any bells. Is the spike always associated with Active Time? A quick Google shows many processes named "alsacti" -- you need to look through those results and see if any of them are from an application you know you're using. That may give you some insight into what's causing this.
Hi Again,
Some new info...
We're currently rebuilding one of our RAID5 arrays in sotware (Wundows Server 2008). We will then transition everything over to this array and see if it eliminates the problem. This should be able to rule in/out the culpability of the RAID controller.
Meanwhile, I have also attached another image from ProcMon showing the read operation that occurs on every skip. This image is form our server that is not having it's RAID array rebuilt and is thud still exhibiting the original issue. Notice the 2 second duration as opposed to the normal 10^-6 second duration.
Ideas?
Thanks,
Andy
Skip-Read Porcess Image
Are you still using the Seagate drives in the new array? I'd still suspect this is the "stuttering" issue those drives are known for. A really good test would be to build an array with a set of WD Caviar Blacks or RE drives ... but unless you happen to have the drives that's clearly not an inexpensive test. I do know that several RAID controllers don't work well with the Seagate 7200.11's; and I personally tried them in a Linux software RAID and found they generated too many "false failures" ... so I don't use them in that mode anymore. In the absense of any other apparent issues, trying a different drive (if possible) is still what I'd suggest.
It's strange that we never saw the issue when the array consisted of only 3 drives. If it is in fact a drive problem, we should have seen stuttering form the beginning, albeit getting worse with more drives. We ran these drives for 6 months with no issues before expanding the array from 3 to 4...
-Andy
Business Accounts
Answer for Membership
by: noxchoPosted on 2009-09-19 at 23:41:05ID: 25376121
I don't believe that your hope is correct. I mean that it will come to stabilized write\read speeds.
Did you check the speed of the drive before adding it? Is it the same speed and model as others?
Also, I've read somewhere that some RAID controllers are good while working with sizes up to 3.1TB
I am not sure if this could be yours problem but that is true.
Does RAID management utility indicate any problem with these drives?