Solved

Email system went down (Exchange 2007)

Posted on 2011-03-11
14
625 Views
Last Modified: 2012-05-11
hey guys

just around one hour ago, our entire emails system went down. Nobody could open Outlook (would hang) and the webmail service was the same. Throughout the day, it was constantly slow, very tempermental and evidently it took its toll and everything went down by the end of the day.

We;re running Exchange 2007. It does NOT have SP1. The disk space available for the database is 410GB and the free space on it is at 59GB! Now, around 6months ago we made changes to the registry so that it could accept a database size of 300GB. We're currently using 350GB from what it seems. I've gone into the registry and increased the limit to 400GB.

Nothing in the event viewer indicated anything about database size limit as far as I can see.

However, does this sound like it could have been the database size that caused the issue?

Also, there are so many deleted mailboxes and apparently only a defrag would get rid of them to free up space, but according to some experts this would take around 3 days?!
 
On the disk which holds the databases, we've created them so that it's not just one huge database file but we split into three database files. Is there a way of defraging each one separately so that it doesn't take 3days and going through this procedure without having to take down Exchange entirely?

Any solutions you propose would be much appreciated.
Thanks
Yashy
0
Comment
Question by:Yashy
  • 4
  • 4
  • 4
  • +2
14 Comments
 
LVL 8

Accepted Solution

by:
Acosta Technology Services earned 250 total points
ID: 35111362
First thing I would do is check the 1211 event log (should be around 7AM) in the application log.  This will tell you how much whitespace is available within the database; that's how much you could free with an offline defrag.

If there is a lot of space available I would recommend an offline defrag for the DB with the most "freeable" space.

Also, how are your logs currently?  Has a full backup finished recently to clear the logs?
0
 
LVL 8

Expert Comment

by:Acosta Technology Services
ID: 35111371
Sorry, that should be event 1221, NOT 1211.
0
 
LVL 10

Expert Comment

by:akhalighi
ID: 35111391
- clear log files , restart exchange services , what do you see in event logs ? any specific errors ?
- how is resource usage on this box , memory , CPU , etc ...
0
What is SQL Server and how does it work?

The purpose of this paper is to provide you background on SQL Server. It’s your self-study guide for learning fundamentals. It includes both the history of SQL and its technical basics. Concepts and definitions will form the solid foundation of your future DBA expertise.

 
LVL 43

Expert Comment

by:Amit
ID: 35111621
I would first suggest you to upgrade to lastest Service pack. Which is SP3 for 2007.

For defrag part. Thumb rule is 3-4 GB per hour. I will take 4GB per hour. So 300/4 = 75 and 74/24 = 3.125 days. Which is very long downtime. Best option is to create new DB and move the mailboxes over to them. Try to split them in mulitple DB's rather keeping them in one.

For Server issue. Check the systems and applications logs, what events are generated during that window. Check for Virtual Memory setting also. It has to be Actual + 10MB. Make sure all AV exclusion are in place for Exchange and check for any memory leak issue.

0
 
LVL 8

Expert Comment

by:Acosta Technology Services
ID: 35111694
The timing is based on the logical DB size, not the physical.  If DB1 is 100GB on disk, but has 40GB of whitespace you can calculate the total defrag time based on 60GB.  The total time will depend on the size of each DB in your information store and how much whitespace is available (event 1221).
0
 
LVL 7

Expert Comment

by:OctInv
ID: 35112702
You say the entire system went down - do you mean the databases are dismounted? Can you give us a bit mroe info on your Exchange environment - are all roles on one server, are the log files on a seperate disk/volume (if so how are they doing for space?)....the more info you can give the easier it will be to suggest solutiuons.

Thanks
0
 
LVL 1

Author Comment

by:Yashy
ID: 35114032
i'm looking all of your comments guys and giving feedback in the morning as I restarted the exchange server and things are working, but for how long I don't know.

Also, I have split our databases on our disk drive. So the total is 351gb, however they are split into three databases; around 150gb, 100gb and 151gb. Can they each be defragged separately?
0
 
LVL 1

Author Comment

by:Yashy
ID: 35115578
Guys in the applicatino log I'm getting a lot of warnings like this:

Reset to device, \Device\RaidPort1, was issued.

Also, I have checked the disk array utility and it shows the following, could this be why. Look at the picture I have attached.


disks.JPG
0
 
LVL 8

Expert Comment

by:Acosta Technology Services
ID: 35116106
I would replace the predicted fail drive as soon as possible to avoid any other possible issues.

For Exchange:

-Are you seeing any Exchange related errors in the application event log?
-If you check event's 1221 from this morning how much freespace does it show on each DB?

For your questions:

-Yes, you can defrag each DB indivually
0
 
LVL 43

Expert Comment

by:Amit
ID: 35117438
Open call with Hardware vendor for server hardware issue. Check the system logs...you might see issue related to disk
0
 
LVL 1

Author Comment

by:Yashy
ID: 35118403
Spoke to HP guys and drive will be replaced on Monday. However our database file was 300gb...ouch. So we' re going to change disks and then im going to have to reduce that database size with your helpful comments. Then im going to have to look into doing some sort of disaster recovery so that i can prevent this from happening. Do any of you have a DR for Exchange? Otherwise how is your exchange environment resilient to an event such as one hard disk from failing? Thanks.
0
 
LVL 43

Assisted Solution

by:Amit
Amit earned 250 total points
ID: 35121630
You can go with Exchange 2007 CCR cluster. This can give you the Server and disk resilency both. If you want to have site resilency also, then you can implement CCR over the WAN.

Storage planning is very important part of any Exchange env. Always use Storage calculator.
http://msexchangeteam.com/archive/2007/01/15/432207.aspx

Secondly, if you have CAS and HUB on different servers. You can use the NLB for load balancing

Finally, If you have any plans to move with Exchange 2010. Then you can use DAG, which is not dependent on Window clustering and easy to implement also.
0
 
LVL 1

Author Comment

by:Yashy
ID: 35122859
Okay, I'll look into this.

By the way, if we have four disks at 146GB each with Raid5 configuration, then how much disk space should be left?

Reason is because i'm looking at the array which holds the four drives and it says:

410gb available, 59gb free to use.

I'm just wondering whether the free space is a correct indication of the current situation?
0
 
LVL 43

Expert Comment

by:Amit
ID: 35123014
In Raid 5 one disk will be used for fault tolerance. You can check with your platform team, what RAID exactly configured currently. As you said it is HP server. Check from HP Array Console.
0

Featured Post

Optimizing Cloud Backup for Low Bandwidth

With cloud storage prices going down a growing number of SMBs start to use it for backup storage. Unfortunately, business data volume rarely fits the average Internet speed. This article provides an overview of main Internet speed challenges and reveals backup best practices.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Large Outlook files lead to various unwanted errors and corruption issues. Furthermore, large outlook files can also make Outlook take longer to start-up, search, navigate, and shut-down. So, In this article, i will discuss a method to make your Out…
In-place Upgrading Dirsync to Azure AD Connect
This video demonstrates how to sync Microsoft Exchange Public Folders with smartphones using CodeTwo Exchange Sync and Exchange ActiveSync. To learn more about CodeTwo Exchange Sync and download the free trial, go to: http://www.codetwo.com/excha…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…

685 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question