Starting information store freezes server

Hi

SBS 2003 R2, Prolian ML 350 2x72 GB System 2x 146GB Data (Exchange is there) HW RAID e200i.

The server where freezed and I was forced to do power cycle. The boot up hanged to Applying Computer Settings (waited 1,5 hrs to see if she would carry on) I was able to start this server in Safe Mode.I disabled all Exchange services and rebooted it. I come up nicely. Then I enabled all related exchange services and started the Information Store and the server freeze. I-ll tried to open taskmanager but nothing happened. I was forced to do cold reboot.

The server boots up only if Exchange services are disabled. CHKDSK doesn't find anything special as far as I know.

Any idea how to carry on? I saw from the logs that originally the freeze situation started last night when NT backup (information store) started (before that NT BU on drive c and e run succesfully). Same freeze problem has happened lastly one week ago and four months ago).

Any ideas how to carry on? Server Management says that all last 8 backups vere completed succesfully (two most old ones had problems).

Thanks in advantage,

Juha
RimFire007Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Hilal1924Commented:
Please Check this one Out. It is definitely applicable in your case

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/SBS_Small_Business_Server/Q_23022492.html

Hilal
0
Alan HardistyCo-OwnerCommented:
Set the stores to Not Mount at startup (store properties> Database Tab> Do Not Mount this store at start-up) - Both Mailbox Store and Public Folders.
Then make sure all Exchange services are set to automatic and restart the server.
Once server is started - manually start one store at a time until you find the one that hangs.
Restart the server again and then repair the problem store as follows:
eseutil /p
eseutil /d
isinteg -s servername -fix -test alltests
Mount the store - problem should be solved.
Happened to me on my old 2003 server a while back!!
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Hilal1924Commented:
An Alternate Strategy for you would be to set all the Exchange Services in Manual Mode and restart them one by one and see which service is causing you grief ( I suspect it will be the Information Store Service) Do an offline Defrag (eseutil \d) and bring the store online. Take a full backup as soon as as offline defrag is over.  Best part is that the store will eb dismounted while the offline defarg is going on and server will not hang. Also please ensure that the server is patched up with latest Hotfixes and updates

Resources for Offline Defrag opf Exchange :
http://support.microsoft.com/kb/192185

Hilal
0
Making Bulk Changes to Active Directory

Watch this video to see how easy it is to make mass changes to Active Directory from an external text file without using complicated scripts.

Hilal1924Commented:
Alan, I don't think he needs to run "eseutil /p" since it is not really required if eseutil /d and ISinteg tests come out positive.

Hilal
0
Alan HardistyCo-OwnerCommented:
Having had to switch the server off due to a freeze - I would expect the store to be unhappy.
eseutil /p will repair any damage, /d will defragment it and isinteg will clear up any integrity issues.
I am 99.9% certain this will clear up the issue and all will be well.  I have been there and done it myself.
0
RimFire007Author Commented:
Thaks guys

I presume that your solutions focus also to the fact that the server freezes to the state "applying competer settings."

Any idea how long all this might take? Non business hours are starting here and just wondering should I do this remotely during weekend. But surely I do first steps right now after reading completely you solutions.

Thanks again,

Juha

0
RimFire007Author Commented:
Also

The server is not completely win updated. Should I update it now when Exchange is disabled and the server is up?

Juha
0
Alan HardistyCo-OwnerCommented:
How big is the main store?  Eseutil /p & /d run at about 4-6Gb per hour and isinteg is pretty similar.
So, a 20Gb store will take about 4 hours approximately for /p, then the same again for /d and a similar time for isinteg.
It would be best to do out of hours.
You can have the Public working without the Private and vice-versa.  Hopefully it is the smallest store.
You can update it - it won't hurt, but I would concentrate on getting the stores working first, then update.  You don't want to add to your problems (not that it should, but worth tackling one issue at a time IMHO).
0
Hilal1924Commented:
" The server is not completely win updated. Should I update it now when Exchange is disabled and the server is up?" This is a cardinal sin, You should always keep your crticial servers and applications patched up. For Example I am pretty certain that you might not have installed the VSS update and which would have prevented this situation from arising in first place.

Follow this Procedure:
1. Update your servers and applications.
2. Do isinteg -s servername -fix -test alltests
3. Use eseutil /d to defrag it and bring it to consistent state ( Do it in Off Hours)
4. If the above does not fix it ( I am fairly confident that it will) then run eseutil /p (Off Hours). This should be your last resort.
5. Run Exchange Best Practices Analyzer and follow the recommendations.

Hilal
0
RimFire007Author Commented:
priv1.edb 13 GB
priv1.stm 13 GB
pub1.edb 92MB

I think that I start following your solution. Will do first steps OnSite and carry on remotely via PPTP VPN.

Won't update anything before fix. Jus about to start your solution. Hopefully you can hang out there if anything special arises. eseutil is not too familiar to mee.



Juha
0
Alan HardistyCo-OwnerCommented:
I disagree.
Fix your current problem with your store first - then when you have a working system, update.  Updates will not be causing your problems (or the lack of them), but could make matters worse (although unlikely).
Correct order to repair a damaged / corrupt database is:
eseutil /p
eseutil /d
isinteg -s servername -fix -test alltests
@RimFire007 - its your call as to whose advice you take : )
0
Hilal1924Commented:
I think the eseutil tasks will finish sooner than you realize. The database size is not that big and it should be over in around 5 hours.

Hilal
0
Hilal1924Commented:
Alan Bro, My assumption is that his store is merely acting up because of inconsistencies in file table. Why risk data loss when he runs eseutil /p if it can be fixed without running it. Hopefully it makes sense. Eseutil /p should always be the LAST resort.

Hilal
0
Alan HardistyCo-OwnerCommented:
I'm around for another 12 hours (usually) - currently 2pm for me.
For eseutil bring up a command prompt.
Change directory to c:\program files\exchsrvr\bin
run eseutil /p "d:\program files\exchsrvr\mdbdata\priv1.edb"
(assuming Exchange is installed in c:\program files\exchsrvr and database is in d:\program files\exchsrvr\mdbdata - if not - change the patchs to match.  " - quote marks are needed to wrap any paths with spaces in them.)
same for eseutil /d
isinteg is also run from c:\program files\exchsrvr\bin directory exactly as shown apart from changing servername to match your servers name.  So if you server is called SBS2003, the command will be:
isinteg -s sbs2003 -fix -test alltests
Then pick your mailstore - 1 or 2 usually and let the tool run.
Drink lots of coffee!
0
Alan HardistyCo-OwnerCommented:
Hilal - I have personally had this exact problem so I am speaking from 1st hand experience, not basing my suggestions on assumptions which you mention in your last comment.
To make sure you don't suffer any data loss caused by eseutil /p, take a copy of the database (.EDB and .STM files first) and keep those safe in case of emergency.
Eseutil /p can cause data loss, but it is going to be necessary in this case.
0
RimFire007Author Commented:
alanhardisty

I ment that I will start following your steps. Sorry that I didn't point precisly to whom I were answering.

Rgs Juha



0
RimFire007Author Commented:
Just rebooted while only the pri1 db were disabled and all related exchange servises were set to start automaticly. It booted up very fast without problems.

I assume that this indicates the priv1 is the problem one. But I will follow your original solution and setup both pri and pub disabled if needed. How do you feel?

Rgs, Juha
0
Alan HardistyCo-OwnerCommented:
No - leave PUB1 running if it is fine.
If it starts happily without PRIV1 then that is where your problem lies - which is the usual place.
Do take a copy of the store first - just in case you need it later or the server dies mid eseutil (if you have a spare 26Gb of space on the server, otherwise copy the files to a PC with ample room).
0
RimFire007Author Commented:
Damn

Again I  were loose. I ment that I set up "store properties> Database Tab> Do Not Mount this store at start-up" - (Only) Mailbox Store" and all exchange services were set to start automaticly. It booted up nicely.
0
RimFire007Author Commented:
Ok

Thanks alanhardist

Just copying  priv1.edp and priv1.stm to the USB disk. Will carry on when at home via PPTP. So after one hour I will start eseuti routine.

Hawe a great weekend if you  are not going to be onLine! Appriciate a lot your help!

Rgs,

Juha
0
Alan HardistyCo-OwnerCommented:
Good luck - Have a good one yourself.  I'll be around if you need me - laptop / iPhone always handy.  The day I'm not online will be the day I'm dead (or lose my fingers!!).
Alan
0
RimFire007Author Commented:
Update

Step 1.
Just started (at 19.16.01 Finnish time) eseutil /p "e:\program files\exchsrvr\mdbdata\priv1.edb"

I'm at home and will watch TW.  Will time to time check where I'm going. Have a pretty good feeling about this procedure.
0
Alan HardistyCo-OwnerCommented:
Thanks for the update - still here for you.  Hope it won't be a long night.
Alan
0
Alan HardistyCo-OwnerCommented:
Long night for you that is : )
0
RimFire007Author Commented:
Update 2

Step 1.
Just started (at 19.16.01 Finnish time) eseutil /p "e:\program files\exchsrvr\mdbdata\priv1.edb"
Done. Succesfully in 2665,62 seconds
Step 2.
Just started (at 20.14.46 Finnish time) eseutil /d "e:\program files\exchsrvr\mdbdata\priv1.edb"

and sitll wacthing TW...
0
RimFire007Author Commented:
Btw

Step 1 went throw unexpectedly fast. Also the db size is unexpectedly small. I haven't check the actua file size ever but i thought it would be larger since I have some 70 mailboxes there. Well let's see what happens.

A assume that I have 28 GB db since:
- priv1.edb 13,5 GB
- priv1.stm 14,5 GB
leads to 28 GB
0
RimFire007Author Commented:
Actually

I need to check if I have fixed the 25 GB default limit (I thought that I have done it to this server).  Wonder how to verify the situation?

Aha! Eseutil /d will take much more time in my situation
0
Hilal1924Commented:
It should not, If Eseutil /p completed within 2665,62, Eseutil /d should take only that much time if not less.

Hilal
0
Alan HardistyCo-OwnerCommented:
Exchange 2003 Standard Store limit is 16Gb by default
Exchange 2003 Standard with SP2 Store Limit is 18Gb by default
After tweaking the registry you can up the limit to 75Gb:
http://support.microsoft.com/kb/912375
Size is virtual - not physical as the addition of the .STM and .EDB files do not make up the store size.  Inside the store is "White Space" - space taken up by deleted items that has passed the store retention period and can be overwritten, so it depends on how much white space is in your store as to the actual size.
Sounds like you have increased the limit anyway, otherwise your store would be dismounting daily!
0
Alan HardistyCo-OwnerCommented:
Also - run isinteg twice - once to check integrity - second time to make sure 0 errors and 0 fixes are reported on the last line.
If this is not the case, re-run until you see 0 errors and 0 fixes.
0
RimFire007Author Commented:
Yes

I have somewhere a server which dismounted daily - http://support.microsoft.com/kb/912375 fixed it if I remember correctly. I checked the registry of this server: it is a 75 GB server. Thank you for the hint.

Well, eseutil /d has nor run apx 1,5 hrs and apx 30 % is ready. I have high hopes that the procedure will fix it.

I seems that I will carry on in the morning, after some 9 hrs - but as i as I look the graphic of defragmetation it looks that the "speed" is now increasing - almost 40 % done now. Probably I can run Step 3 in this evening
0
Alan HardistyCo-OwnerCommented:
I should imagine that /d will take about the same time as /p, so you should be able to kick off isinteg tonight and hopefully mount the store too.
Mail will queue up on the server, but delivery won't be until the store mounts again and people may think they have lost mail as there will be a gap in the time that mail last was received until the store is mounted.  Nothing should be lost - just delayed as the time the mail shows is the time the mail enters the store, not when it is received.
0
RimFire007Author Commented:
Aha, there is a internal queu too. Forgot that one. First time I ever see anything in the queue: there are 282 mail there right now - nice! Perhaps the queue wasn't working while the server were freezed.

I also have sec mx service from ISP which will hold mails 3 days if the server is down.

While having a sigarette i remembered that the NT BU might running and it were. Stopped it.

Update
Eseutil /d run succesfully in 7982,875 seconds.

Step 3.
I can start "isinteg -s servername -fix -test alltests" routine.
0
Alan HardistyCo-OwnerCommented:
Excellent news. Should be done soon with any luck.
0
RimFire007Author Commented:
Well

The first test went through (take couple minutes only) - I run it only agains Mailbox Store which is my case. 0 errors. I run it again have a sigarette and try to mount the store.

Lets see..
0
Alan HardistyCo-OwnerCommented:
Did you replace servername with your servers name in the isinteg command?

It should not be so quick!
0
RimFire007Author Commented:
Yes I did. My Computer / properties / full computer name / "myserver.mydomain.local". I used "myserver"
I chooce the OffLine  Mailbox store.

There were 0  errors 0 warnings 0 fixes 145934 rows and stuff like that.
Probably 21 lines

I would love to mount, when wait apx 10 mins and then reboot the server. How do you feel?





0
Alan HardistyCo-OwnerCommented:
Go for it.  Seems too quick - but I won't complain.
Get mounting!!
0
RimFire007Author Commented:
Nice!

The webmail is working it sends and receives. The "Exchange Outlook" is working. Everything seems to be there - including mails queued.

I will now reboot the server.

0
Alan HardistyCo-OwnerCommented:
Great news. Sounds very promising.
0
RimFire007Author Commented:
Jep

The boot up was fast. Everything seems to work. Nothing special in the event viewer. I even started the backup. I would say that case closed. Actually I have high hopes that we also fixed the problem related Information Store Backup.

So, it is time to thank you very much!

1. The solution you provided right beginning this case was just perfect.
2. Your solution were easy to follow and exact.
3. Your provied me extraordinary support and guidance to went through this case. Guick answers. Much, much more expeted.
4. It were so nice to work with you!

I will accept your solution in the morning so I can provide final update what happens during backup. It usually takes more than 8 hours.

Have a very special great weekend.

Rgs,

Juha
0
Alan HardistyCo-OwnerCommented:
That's great news - appreciate the feedback and your comments.  Hope your backup runs smoothly too.  Very happy to have helped you and glad that your problem is resolved (until further notice).  Two birds with one stone - that's even better ; )
If you need any more help - just shout - close the question only when you are ready and 100% happy.
Have a good weekend too (and still 3½ before bed time for me)!
Best wishes
Alan
0
RimFire007Author Commented:
The solution was increble staright forward and easy to follow. Also valuable gardinal best practices vere offred. The overall proced lasted apx 8 hrs and the experts kindly guided me through from the beging to the end to Finnish the complete work around succesfully.

Both the sympton (freeze) and the original disease (bad information store)  were cured.

Best wishes to you both alanhardisty and Hilal1924!
0
RimFire007Author Commented:
Final Update

The backup has succesfully finshed. This case is closed: Thank's for the Experts.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
SBS

From novice to tech pro — start learning today.