Solved

Email system went down (Exchange 2007)

Posted on 2011-03-11
14
621 Views
Last Modified: 2012-05-11
hey guys

just around one hour ago, our entire emails system went down. Nobody could open Outlook (would hang) and the webmail service was the same. Throughout the day, it was constantly slow, very tempermental and evidently it took its toll and everything went down by the end of the day.

We;re running Exchange 2007. It does NOT have SP1. The disk space available for the database is 410GB and the free space on it is at 59GB! Now, around 6months ago we made changes to the registry so that it could accept a database size of 300GB. We're currently using 350GB from what it seems. I've gone into the registry and increased the limit to 400GB.

Nothing in the event viewer indicated anything about database size limit as far as I can see.

However, does this sound like it could have been the database size that caused the issue?

Also, there are so many deleted mailboxes and apparently only a defrag would get rid of them to free up space, but according to some experts this would take around 3 days?!
 
On the disk which holds the databases, we've created them so that it's not just one huge database file but we split into three database files. Is there a way of defraging each one separately so that it doesn't take 3days and going through this procedure without having to take down Exchange entirely?

Any solutions you propose would be much appreciated.
Thanks
Yashy
0
Comment
Question by:Yashy
  • 4
  • 4
  • 4
  • +2
14 Comments
 
LVL 8

Accepted Solution

by:
Acosta Technology Services earned 250 total points
Comment Utility
First thing I would do is check the 1211 event log (should be around 7AM) in the application log.  This will tell you how much whitespace is available within the database; that's how much you could free with an offline defrag.

If there is a lot of space available I would recommend an offline defrag for the DB with the most "freeable" space.

Also, how are your logs currently?  Has a full backup finished recently to clear the logs?
0
 
LVL 8

Expert Comment

by:Acosta Technology Services
Comment Utility
Sorry, that should be event 1221, NOT 1211.
0
 
LVL 10

Expert Comment

by:akhalighi
Comment Utility
- clear log files , restart exchange services , what do you see in event logs ? any specific errors ?
- how is resource usage on this box , memory , CPU , etc ...
0
 
LVL 41

Expert Comment

by:Amit
Comment Utility
I would first suggest you to upgrade to lastest Service pack. Which is SP3 for 2007.

For defrag part. Thumb rule is 3-4 GB per hour. I will take 4GB per hour. So 300/4 = 75 and 74/24 = 3.125 days. Which is very long downtime. Best option is to create new DB and move the mailboxes over to them. Try to split them in mulitple DB's rather keeping them in one.

For Server issue. Check the systems and applications logs, what events are generated during that window. Check for Virtual Memory setting also. It has to be Actual + 10MB. Make sure all AV exclusion are in place for Exchange and check for any memory leak issue.

0
 
LVL 8

Expert Comment

by:Acosta Technology Services
Comment Utility
The timing is based on the logical DB size, not the physical.  If DB1 is 100GB on disk, but has 40GB of whitespace you can calculate the total defrag time based on 60GB.  The total time will depend on the size of each DB in your information store and how much whitespace is available (event 1221).
0
 
LVL 7

Expert Comment

by:OctInv
Comment Utility
You say the entire system went down - do you mean the databases are dismounted? Can you give us a bit mroe info on your Exchange environment - are all roles on one server, are the log files on a seperate disk/volume (if so how are they doing for space?)....the more info you can give the easier it will be to suggest solutiuons.

Thanks
0
 
LVL 1

Author Comment

by:Yashy
Comment Utility
i'm looking all of your comments guys and giving feedback in the morning as I restarted the exchange server and things are working, but for how long I don't know.

Also, I have split our databases on our disk drive. So the total is 351gb, however they are split into three databases; around 150gb, 100gb and 151gb. Can they each be defragged separately?
0
What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 1

Author Comment

by:Yashy
Comment Utility
Guys in the applicatino log I'm getting a lot of warnings like this:

Reset to device, \Device\RaidPort1, was issued.

Also, I have checked the disk array utility and it shows the following, could this be why. Look at the picture I have attached.


disks.JPG
0
 
LVL 8

Expert Comment

by:Acosta Technology Services
Comment Utility
I would replace the predicted fail drive as soon as possible to avoid any other possible issues.

For Exchange:

-Are you seeing any Exchange related errors in the application event log?
-If you check event's 1221 from this morning how much freespace does it show on each DB?

For your questions:

-Yes, you can defrag each DB indivually
0
 
LVL 41

Expert Comment

by:Amit
Comment Utility
Open call with Hardware vendor for server hardware issue. Check the system logs...you might see issue related to disk
0
 
LVL 1

Author Comment

by:Yashy
Comment Utility
Spoke to HP guys and drive will be replaced on Monday. However our database file was 300gb...ouch. So we' re going to change disks and then im going to have to reduce that database size with your helpful comments. Then im going to have to look into doing some sort of disaster recovery so that i can prevent this from happening. Do any of you have a DR for Exchange? Otherwise how is your exchange environment resilient to an event such as one hard disk from failing? Thanks.
0
 
LVL 41

Assisted Solution

by:Amit
Amit earned 250 total points
Comment Utility
You can go with Exchange 2007 CCR cluster. This can give you the Server and disk resilency both. If you want to have site resilency also, then you can implement CCR over the WAN.

Storage planning is very important part of any Exchange env. Always use Storage calculator.
http://msexchangeteam.com/archive/2007/01/15/432207.aspx

Secondly, if you have CAS and HUB on different servers. You can use the NLB for load balancing

Finally, If you have any plans to move with Exchange 2010. Then you can use DAG, which is not dependent on Window clustering and easy to implement also.
0
 
LVL 1

Author Comment

by:Yashy
Comment Utility
Okay, I'll look into this.

By the way, if we have four disks at 146GB each with Raid5 configuration, then how much disk space should be left?

Reason is because i'm looking at the array which holds the four drives and it says:

410gb available, 59gb free to use.

I'm just wondering whether the free space is a correct indication of the current situation?
0
 
LVL 41

Expert Comment

by:Amit
Comment Utility
In Raid 5 one disk will be used for fault tolerance. You can check with your platform team, what RAID exactly configured currently. As you said it is HP server. Check from HP Array Console.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Exchange server is not supported in any cloud-hosted platform (other than Azure with Azure Premium Storage).
If you don't know how to downgrade, my instructions below should be helpful.
This Experts Exchange video Micro Tutorial shows how to tell Microsoft Office that a word is NOT spelled correctly. Microsoft Office has a built-in, main dictionary that is shared by Office apps, including Excel, Outlook, PowerPoint, and Word. When …
how to add IIS SMTP to handle application/Scanner relays into office 365.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now