Link to home
Start Free TrialLog in
Avatar of RimFire007
RimFire007

asked on

Starting information store freezes server

Hi

SBS 2003 R2, Prolian ML 350 2x72 GB System 2x 146GB Data (Exchange is there) HW RAID e200i.

The server where freezed and I was forced to do power cycle. The boot up hanged to Applying Computer Settings (waited 1,5 hrs to see if she would carry on) I was able to start this server in Safe Mode.I disabled all Exchange services and rebooted it. I come up nicely. Then I enabled all related exchange services and started the Information Store and the server freeze. I-ll tried to open taskmanager but nothing happened. I was forced to do cold reboot.

The server boots up only if Exchange services are disabled. CHKDSK doesn't find anything special as far as I know.

Any idea how to carry on? I saw from the logs that originally the freeze situation started last night when NT backup (information store) started (before that NT BU on drive c and e run succesfully). Same freeze problem has happened lastly one week ago and four months ago).

Any ideas how to carry on? Server Management says that all last 8 backups vere completed succesfully (two most old ones had problems).

Thanks in advantage,

Juha
Avatar of Hilal1924
Hilal1924
Flag of India image

ASKER CERTIFIED SOLUTION
Avatar of Alan Hardisty
Alan Hardisty
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
An Alternate Strategy for you would be to set all the Exchange Services in Manual Mode and restart them one by one and see which service is causing you grief ( I suspect it will be the Information Store Service) Do an offline Defrag (eseutil \d) and bring the store online. Take a full backup as soon as as offline defrag is over.  Best part is that the store will eb dismounted while the offline defarg is going on and server will not hang. Also please ensure that the server is patched up with latest Hotfixes and updates

Resources for Offline Defrag opf Exchange :
http://support.microsoft.com/kb/192185

Hilal
Alan, I don't think he needs to run "eseutil /p" since it is not really required if eseutil /d and ISinteg tests come out positive.

Hilal
Having had to switch the server off due to a freeze - I would expect the store to be unhappy.
eseutil /p will repair any damage, /d will defragment it and isinteg will clear up any integrity issues.
I am 99.9% certain this will clear up the issue and all will be well.  I have been there and done it myself.
Avatar of RimFire007
RimFire007

ASKER

Thaks guys

I presume that your solutions focus also to the fact that the server freezes to the state "applying competer settings."

Any idea how long all this might take? Non business hours are starting here and just wondering should I do this remotely during weekend. But surely I do first steps right now after reading completely you solutions.

Thanks again,

Juha

Also

The server is not completely win updated. Should I update it now when Exchange is disabled and the server is up?

Juha
How big is the main store?  Eseutil /p & /d run at about 4-6Gb per hour and isinteg is pretty similar.
So, a 20Gb store will take about 4 hours approximately for /p, then the same again for /d and a similar time for isinteg.
It would be best to do out of hours.
You can have the Public working without the Private and vice-versa.  Hopefully it is the smallest store.
You can update it - it won't hurt, but I would concentrate on getting the stores working first, then update.  You don't want to add to your problems (not that it should, but worth tackling one issue at a time IMHO).
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
priv1.edb 13 GB
priv1.stm 13 GB
pub1.edb 92MB

I think that I start following your solution. Will do first steps OnSite and carry on remotely via PPTP VPN.

Won't update anything before fix. Jus about to start your solution. Hopefully you can hang out there if anything special arises. eseutil is not too familiar to mee.



Juha
I disagree.
Fix your current problem with your store first - then when you have a working system, update.  Updates will not be causing your problems (or the lack of them), but could make matters worse (although unlikely).
Correct order to repair a damaged / corrupt database is:
eseutil /p
eseutil /d
isinteg -s servername -fix -test alltests
@RimFire007 - its your call as to whose advice you take : )
I think the eseutil tasks will finish sooner than you realize. The database size is not that big and it should be over in around 5 hours.

Hilal
Alan Bro, My assumption is that his store is merely acting up because of inconsistencies in file table. Why risk data loss when he runs eseutil /p if it can be fixed without running it. Hopefully it makes sense. Eseutil /p should always be the LAST resort.

Hilal
I'm around for another 12 hours (usually) - currently 2pm for me.
For eseutil bring up a command prompt.
Change directory to c:\program files\exchsrvr\bin
run eseutil /p "d:\program files\exchsrvr\mdbdata\priv1.edb"
(assuming Exchange is installed in c:\program files\exchsrvr and database is in d:\program files\exchsrvr\mdbdata - if not - change the patchs to match.  " - quote marks are needed to wrap any paths with spaces in them.)
same for eseutil /d
isinteg is also run from c:\program files\exchsrvr\bin directory exactly as shown apart from changing servername to match your servers name.  So if you server is called SBS2003, the command will be:
isinteg -s sbs2003 -fix -test alltests
Then pick your mailstore - 1 or 2 usually and let the tool run.
Drink lots of coffee!
Hilal - I have personally had this exact problem so I am speaking from 1st hand experience, not basing my suggestions on assumptions which you mention in your last comment.
To make sure you don't suffer any data loss caused by eseutil /p, take a copy of the database (.EDB and .STM files first) and keep those safe in case of emergency.
Eseutil /p can cause data loss, but it is going to be necessary in this case.
alanhardisty

I ment that I will start following your steps. Sorry that I didn't point precisly to whom I were answering.

Rgs Juha



Just rebooted while only the pri1 db were disabled and all related exchange servises were set to start automaticly. It booted up very fast without problems.

I assume that this indicates the priv1 is the problem one. But I will follow your original solution and setup both pri and pub disabled if needed. How do you feel?

Rgs, Juha
No - leave PUB1 running if it is fine.
If it starts happily without PRIV1 then that is where your problem lies - which is the usual place.
Do take a copy of the store first - just in case you need it later or the server dies mid eseutil (if you have a spare 26Gb of space on the server, otherwise copy the files to a PC with ample room).
Damn

Again I  were loose. I ment that I set up "store properties> Database Tab> Do Not Mount this store at start-up" - (Only) Mailbox Store" and all exchange services were set to start automaticly. It booted up nicely.
Ok

Thanks alanhardist

Just copying  priv1.edp and priv1.stm to the USB disk. Will carry on when at home via PPTP. So after one hour I will start eseuti routine.

Hawe a great weekend if you  are not going to be onLine! Appriciate a lot your help!

Rgs,

Juha
Good luck - Have a good one yourself.  I'll be around if you need me - laptop / iPhone always handy.  The day I'm not online will be the day I'm dead (or lose my fingers!!).
Alan
Update

Step 1.
Just started (at 19.16.01 Finnish time) eseutil /p "e:\program files\exchsrvr\mdbdata\priv1.edb"

I'm at home and will watch TW.  Will time to time check where I'm going. Have a pretty good feeling about this procedure.
Thanks for the update - still here for you.  Hope it won't be a long night.
Alan
Long night for you that is : )
Update 2

Step 1.
Just started (at 19.16.01 Finnish time) eseutil /p "e:\program files\exchsrvr\mdbdata\priv1.edb"
Done. Succesfully in 2665,62 seconds
Step 2.
Just started (at 20.14.46 Finnish time) eseutil /d "e:\program files\exchsrvr\mdbdata\priv1.edb"

and sitll wacthing TW...
Btw

Step 1 went throw unexpectedly fast. Also the db size is unexpectedly small. I haven't check the actua file size ever but i thought it would be larger since I have some 70 mailboxes there. Well let's see what happens.

A assume that I have 28 GB db since:
- priv1.edb 13,5 GB
- priv1.stm 14,5 GB
leads to 28 GB
Actually

I need to check if I have fixed the 25 GB default limit (I thought that I have done it to this server).  Wonder how to verify the situation?

Aha! Eseutil /d will take much more time in my situation
It should not, If Eseutil /p completed within 2665,62, Eseutil /d should take only that much time if not less.

Hilal
Exchange 2003 Standard Store limit is 16Gb by default
Exchange 2003 Standard with SP2 Store Limit is 18Gb by default
After tweaking the registry you can up the limit to 75Gb:
http://support.microsoft.com/kb/912375
Size is virtual - not physical as the addition of the .STM and .EDB files do not make up the store size.  Inside the store is "White Space" - space taken up by deleted items that has passed the store retention period and can be overwritten, so it depends on how much white space is in your store as to the actual size.
Sounds like you have increased the limit anyway, otherwise your store would be dismounting daily!
Also - run isinteg twice - once to check integrity - second time to make sure 0 errors and 0 fixes are reported on the last line.
If this is not the case, re-run until you see 0 errors and 0 fixes.
Yes

I have somewhere a server which dismounted daily - http://support.microsoft.com/kb/912375 fixed it if I remember correctly. I checked the registry of this server: it is a 75 GB server. Thank you for the hint.

Well, eseutil /d has nor run apx 1,5 hrs and apx 30 % is ready. I have high hopes that the procedure will fix it.

I seems that I will carry on in the morning, after some 9 hrs - but as i as I look the graphic of defragmetation it looks that the "speed" is now increasing - almost 40 % done now. Probably I can run Step 3 in this evening
I should imagine that /d will take about the same time as /p, so you should be able to kick off isinteg tonight and hopefully mount the store too.
Mail will queue up on the server, but delivery won't be until the store mounts again and people may think they have lost mail as there will be a gap in the time that mail last was received until the store is mounted.  Nothing should be lost - just delayed as the time the mail shows is the time the mail enters the store, not when it is received.
Aha, there is a internal queu too. Forgot that one. First time I ever see anything in the queue: there are 282 mail there right now - nice! Perhaps the queue wasn't working while the server were freezed.

I also have sec mx service from ISP which will hold mails 3 days if the server is down.

While having a sigarette i remembered that the NT BU might running and it were. Stopped it.

Update
Eseutil /d run succesfully in 7982,875 seconds.

Step 3.
I can start "isinteg -s servername -fix -test alltests" routine.
Excellent news. Should be done soon with any luck.
Well

The first test went through (take couple minutes only) - I run it only agains Mailbox Store which is my case. 0 errors. I run it again have a sigarette and try to mount the store.

Lets see..
Did you replace servername with your servers name in the isinteg command?

It should not be so quick!
Yes I did. My Computer / properties / full computer name / "myserver.mydomain.local". I used "myserver"
I chooce the OffLine  Mailbox store.

There were 0  errors 0 warnings 0 fixes 145934 rows and stuff like that.
Probably 21 lines

I would love to mount, when wait apx 10 mins and then reboot the server. How do you feel?





Go for it.  Seems too quick - but I won't complain.
Get mounting!!
Nice!

The webmail is working it sends and receives. The "Exchange Outlook" is working. Everything seems to be there - including mails queued.

I will now reboot the server.

Great news. Sounds very promising.
Jep

The boot up was fast. Everything seems to work. Nothing special in the event viewer. I even started the backup. I would say that case closed. Actually I have high hopes that we also fixed the problem related Information Store Backup.

So, it is time to thank you very much!

1. The solution you provided right beginning this case was just perfect.
2. Your solution were easy to follow and exact.
3. Your provied me extraordinary support and guidance to went through this case. Guick answers. Much, much more expeted.
4. It were so nice to work with you!

I will accept your solution in the morning so I can provide final update what happens during backup. It usually takes more than 8 hours.

Have a very special great weekend.

Rgs,

Juha
That's great news - appreciate the feedback and your comments.  Hope your backup runs smoothly too.  Very happy to have helped you and glad that your problem is resolved (until further notice).  Two birds with one stone - that's even better ; )
If you need any more help - just shout - close the question only when you are ready and 100% happy.
Have a good weekend too (and still 3½ before bed time for me)!
Best wishes
Alan
The solution was increble staright forward and easy to follow. Also valuable gardinal best practices vere offred. The overall proced lasted apx 8 hrs and the experts kindly guided me through from the beging to the end to Finnish the complete work around succesfully.

Both the sympton (freeze) and the original disease (bad information store)  were cured.

Best wishes to you both alanhardisty and Hilal1924!
Final Update

The backup has succesfully finshed. This case is closed: Thank's for the Experts.