Solved

Email system went down (Exchange 2007)

Posted on 2011-03-11
14
622 Views
Last Modified: 2012-05-11
hey guys

just around one hour ago, our entire emails system went down. Nobody could open Outlook (would hang) and the webmail service was the same. Throughout the day, it was constantly slow, very tempermental and evidently it took its toll and everything went down by the end of the day.

We;re running Exchange 2007. It does NOT have SP1. The disk space available for the database is 410GB and the free space on it is at 59GB! Now, around 6months ago we made changes to the registry so that it could accept a database size of 300GB. We're currently using 350GB from what it seems. I've gone into the registry and increased the limit to 400GB.

Nothing in the event viewer indicated anything about database size limit as far as I can see.

However, does this sound like it could have been the database size that caused the issue?

Also, there are so many deleted mailboxes and apparently only a defrag would get rid of them to free up space, but according to some experts this would take around 3 days?!
 
On the disk which holds the databases, we've created them so that it's not just one huge database file but we split into three database files. Is there a way of defraging each one separately so that it doesn't take 3days and going through this procedure without having to take down Exchange entirely?

Any solutions you propose would be much appreciated.
Thanks
Yashy
0
Comment
Question by:Yashy
  • 4
  • 4
  • 4
  • +2
14 Comments
 
LVL 8

Accepted Solution

by:
Acosta Technology Services earned 250 total points
ID: 35111362
First thing I would do is check the 1211 event log (should be around 7AM) in the application log.  This will tell you how much whitespace is available within the database; that's how much you could free with an offline defrag.

If there is a lot of space available I would recommend an offline defrag for the DB with the most "freeable" space.

Also, how are your logs currently?  Has a full backup finished recently to clear the logs?
0
 
LVL 8

Expert Comment

by:Acosta Technology Services
ID: 35111371
Sorry, that should be event 1221, NOT 1211.
0
 
LVL 10

Expert Comment

by:akhalighi
ID: 35111391
- clear log files , restart exchange services , what do you see in event logs ? any specific errors ?
- how is resource usage on this box , memory , CPU , etc ...
0
 
LVL 42

Expert Comment

by:Amit
ID: 35111621
I would first suggest you to upgrade to lastest Service pack. Which is SP3 for 2007.

For defrag part. Thumb rule is 3-4 GB per hour. I will take 4GB per hour. So 300/4 = 75 and 74/24 = 3.125 days. Which is very long downtime. Best option is to create new DB and move the mailboxes over to them. Try to split them in mulitple DB's rather keeping them in one.

For Server issue. Check the systems and applications logs, what events are generated during that window. Check for Virtual Memory setting also. It has to be Actual + 10MB. Make sure all AV exclusion are in place for Exchange and check for any memory leak issue.

0
 
LVL 8

Expert Comment

by:Acosta Technology Services
ID: 35111694
The timing is based on the logical DB size, not the physical.  If DB1 is 100GB on disk, but has 40GB of whitespace you can calculate the total defrag time based on 60GB.  The total time will depend on the size of each DB in your information store and how much whitespace is available (event 1221).
0
 
LVL 7

Expert Comment

by:OctInv
ID: 35112702
You say the entire system went down - do you mean the databases are dismounted? Can you give us a bit mroe info on your Exchange environment - are all roles on one server, are the log files on a seperate disk/volume (if so how are they doing for space?)....the more info you can give the easier it will be to suggest solutiuons.

Thanks
0
 
LVL 1

Author Comment

by:Yashy
ID: 35114032
i'm looking all of your comments guys and giving feedback in the morning as I restarted the exchange server and things are working, but for how long I don't know.

Also, I have split our databases on our disk drive. So the total is 351gb, however they are split into three databases; around 150gb, 100gb and 151gb. Can they each be defragged separately?
0
Integrate social media with email signatures

Is your company active on social media? Do you also use email signatures? Including social media icons in your email signature is a great way to get fans for free. Let all your email users know you’re on social media quickly and easily, in a single click.

 
LVL 1

Author Comment

by:Yashy
ID: 35115578
Guys in the applicatino log I'm getting a lot of warnings like this:

Reset to device, \Device\RaidPort1, was issued.

Also, I have checked the disk array utility and it shows the following, could this be why. Look at the picture I have attached.


disks.JPG
0
 
LVL 8

Expert Comment

by:Acosta Technology Services
ID: 35116106
I would replace the predicted fail drive as soon as possible to avoid any other possible issues.

For Exchange:

-Are you seeing any Exchange related errors in the application event log?
-If you check event's 1221 from this morning how much freespace does it show on each DB?

For your questions:

-Yes, you can defrag each DB indivually
0
 
LVL 42

Expert Comment

by:Amit
ID: 35117438
Open call with Hardware vendor for server hardware issue. Check the system logs...you might see issue related to disk
0
 
LVL 1

Author Comment

by:Yashy
ID: 35118403
Spoke to HP guys and drive will be replaced on Monday. However our database file was 300gb...ouch. So we' re going to change disks and then im going to have to reduce that database size with your helpful comments. Then im going to have to look into doing some sort of disaster recovery so that i can prevent this from happening. Do any of you have a DR for Exchange? Otherwise how is your exchange environment resilient to an event such as one hard disk from failing? Thanks.
0
 
LVL 42

Assisted Solution

by:Amit
Amit earned 250 total points
ID: 35121630
You can go with Exchange 2007 CCR cluster. This can give you the Server and disk resilency both. If you want to have site resilency also, then you can implement CCR over the WAN.

Storage planning is very important part of any Exchange env. Always use Storage calculator.
http://msexchangeteam.com/archive/2007/01/15/432207.aspx

Secondly, if you have CAS and HUB on different servers. You can use the NLB for load balancing

Finally, If you have any plans to move with Exchange 2010. Then you can use DAG, which is not dependent on Window clustering and easy to implement also.
0
 
LVL 1

Author Comment

by:Yashy
ID: 35122859
Okay, I'll look into this.

By the way, if we have four disks at 146GB each with Raid5 configuration, then how much disk space should be left?

Reason is because i'm looking at the array which holds the four drives and it says:

410gb available, 59gb free to use.

I'm just wondering whether the free space is a correct indication of the current situation?
0
 
LVL 42

Expert Comment

by:Amit
ID: 35123014
In Raid 5 one disk will be used for fault tolerance. You can check with your platform team, what RAID exactly configured currently. As you said it is HP server. Check from HP Array Console.
0

Featured Post

Does Powershell have you tied up in knots?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Possible fixes for Windows 7 and Windows Server 2008 updating problem. Solutions mentioned are from Microsoft themselves. I started a case with them from our Microsoft Silver Partner option to open a case and get direct support from Microsoft. If s…
Find out what you should include to make the best professional email signature for your organization.
To add imagery to an HTML email signature, you have two options available to you. You can either add a logo/image by embedding it directly into the signature or hosting it externally and linking to it. The vast majority of email clients display l…
Many of my clients call in with monstrous Gmail overloading issues with Outlook. A quick tip is to turn off the All Mail and Important folders from synching. Here is a quick video I made to show you how to turn off these and other folders in Gmail s…

895 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now