background cleanup generates high cpu store on Exchange 5.5

Hi,
I have an Exchange 5.5 (Build 2653.23: Service Pack 4) server on NT4 (SP6) which takes about 95% of the server's cpu almost all the time.

The application event viewer logs lots of this first event (1101 - error):
Error 0xfffffbd3 occurred  on message <message_id>uring a background cleanup

then it logs a couple of 1104 (error) events :
Error 0xfffffbd3 occurred during a background cleanup on attachment <attachment_id>

Last, it logs a 7201 warning :
Background thread FDoPeriodic encountered a problem. Error code 0xfffffbd3

The only solution is to restart the server as the Store service doesn't answer to a stop request.

Thanks for help.

Fred
LVL 1
fhoAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Exchange_AdminCommented:
You may want to stop the Information Store service and make a copy of the PRIV.EDB.
Then run the following:
ISINTEG -FIX -PRI -TEST ALLTESTS
0
fhoAuthor Commented:
Thanks.
Do I need to save priv.edb before, or won't this test affect the file?
0
fhoAuthor Commented:
sorry, I didn't read your full answer. I'll save the file beofre running the command
0
Creating Active Directory Users from a Text File

If your organization has a need to mass-create AD user accounts, watch this video to see how its done without the need for scripting or other unnecessary complexities.

fhoAuthor Commented:
I ran it.
The summary reports no errors, 7 warning and 1 fix.
I could restart the service.
Was it supposed to solve my problem, or just to detect a potentiel corruption of the priv ?
Thanks
Fred
0
Exchange_AdminCommented:
Hopefully it may have resolved the problem.
0
fhoAuthor Commented:
Unfortunately, it didn't. The store is still  using 99 % of the CPU.
I couldn't stop it, and had to restart the server.

0
David WilhoitSenior Consultant, ExchangeCommented:
run the isinteg test until you get 0 warnings and 0 fixes. then you can effectively try the background cleanup again.

D
0
David WilhoitSenior Consultant, ExchangeCommented:
0
fhoAuthor Commented:
I've already applied kb 293836 with no result.
I'll run the isinteg test again
thanks
Fred
0
fhoAuthor Commented:
Kidego,
when you write : "you can effectively try the background cleanup again", does it mean that this isn't a "normal" maintenance task?
I don't remember having configured it, and I didn't see any piece of information about it in logs of other Exchange servers that are in the Org.

I ran isinteg once again and didn't have any warning nor errors.

I've also applied a post SP4 rollup (KB841765) which I had already used on another server to fix a high CPU use by the store (nothing to do with the background cleanup).

Fred
0
David WilhoitSenior Consultant, ExchangeCommented:
http://www.microsoft.com/technet/prodtechnol/exchange/55/maintain/perfopt.mspx

Have you run performance optimizer in a while? Definitely worth a shot, read this:

http://support.microsoft.com/default.aspx?scid=kb;en-us;234702

this gives you some ideas of how to use perfmon to track the issue, and this MTA tuning works really well in my environment. Just a question, but how much free space is available in this database, and how big is it overall? And have you run isinteg fix on the pub.edb as well?


D
0
fhoAuthor Commented:
I've already used the Perfmon after having installed new server in order to move stores on dedicated hard drives.
I'll read these threads.
The store is small (about 3GB) and there is at least 40 free GB on the hard disk where it is. No I haven't run isinteg on the pub.edb yet. I'll do it tomorrow morning.

The good piece of news is that we didn't have any issue today (the store used less than 2 min of CPU time since the last reboot this morning). But I'll wait until tomorrow morning to check, as cleanup tasks runs at night.

I was surprised to see so many records about Exchange (even info) in the application log and had a look to Diagnostic keys in the registry. Some of them have a value of 1 others a value of 3. I thought that the normal configuration was 0 for Diag keys.
Can you confirm?
Thanks
Fred
0
David WilhoitSenior Consultant, ExchangeCommented:
yea, diagnostics are turned off by default. Go into the server properties in the Admin program, and turn those off.

D
0
fhoAuthor Commented:
Thanks. I've done it. Lots of them had a higher level which produced lots of events in the Event Viewer.
Let's wait until tomorrow morning to see if the cpu use of the store came back to normal situation.
Fred
0
David WilhoitSenior Consultant, ExchangeCommented:
keep us posted :)

D
0
fhoAuthor Commented:
It's worse. There are lots of errors logged in the Application EV now during the night.

Database Page cache error 116 : MSExchangeIS (485) Synchronous read page checksum error -1018.
This first error is logged several times.
There are some errors 117 and 200 which also concern Database Page Cache.

I'm not sure that they weren't logged before .... Perhaps were they masked by lots of information messages produced by turned on diags (there were hundreds of info messages in the EV which size was limited to 512K filled in less than one hour).

What is very strange is that after having been restarted, Exchange works (or seems to). Users have access to their mailbox, and can send and receive emails.
I'm also suprised that isinteg command didn't return any error. Isn't is supposed to check database integrity?

Shoudl I launch it again tomorrow morning?
TIA
Fred
0
Exchange_AdminCommented:
-1018 errors=not good.
The ISINTEG utility only checks the relationship between the tables in the database and does not check at the page level of the database.
99% of the time these are caused by hardware problems such as SCSI controller cards or memory issues. If this is a DELL server with a PERC-2 controller card, make sure that you have the latest firmware patch.
Check your system lof to see if there are any entries concerning your SCSI controller cards.

You have 2 options:
Option 1:
Restore from backup. This would be a backup made prior to getting -1018 errors.

Option 2:
Run a hard repair of the database. WARNING:THIS CAN CAUSE LOSS OF DATA

But if you do not resolve the underlying cause of the -1018 errors, they will reoccur.

To run a hard repair:
Stop the Information Store service. Make a copy of the PRIV.EDB and PUB.EDB databases.
Run the following:
ESEUTIL /ISPRIV /P
ESEUTIL /ISPUB /P
ISINTEG -FIX -PRI -TEST ALLTESTS
ISINTEG -FIX -PUB -TEST ALLTESTS

Like I said earlier, you need to resolve the cause of the -1018 errors or they will just come back.
0
David WilhoitSenior Consultant, ExchangeCommented:
Most likely with the 116, you've got hard drive issues. Do you have a hardware contract? Any way you can check your hard drive status thru the monitoring?

D
0
fhoAuthor Commented:
Thanks.
It seems that there were some hardware issues that have been fixed now.
I launched an export of all mailboxes in pst files this morning. Then I'll try an hard repair of the store (/p).
Can you confirm that with pst (if the export runs without any error to its end), I should be able to recover all mailboxes if I need to reinstall a fresh store ?
On the other hand, is there a way to export public folders?
TIA
Fred
0
Exchange_AdminCommented:
After exporting to PST files, you can stop the Information Store service.
Rename the MDBDATA folders on all drives to MDBDATA.OLD. Create a new MDBDATA folder wherever you renamed one.
Start the Information Store service. This will create a new PRIV.EDB and PUB.EDB.
Send a test email to all users that have a mailbox in this database. You can then run EXMERGE to import the data back into the mailboxes.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
David WilhoitSenior Consultant, ExchangeCommented:
And yes, you can use outlook to export the public folders to a pst file, just remember the size limitation is 2 GB on a pst, and you're fine.

D
0
fhoAuthor Commented:
Ok, thanks to both of you.
I'll do it this week-end.
Shall I first try to hard repair (isinteg /p) the store, or isn't it worthwhile ?
0
fhoAuthor Commented:
Hi there,
I launched eseutil /p against priv and pub databases (after having exported mailboxes through Exmerge, and save database and log files).

Both finished without any error.
Then, I removed log files (as asked by eseutil) and launched isinteg /fix .... on both databases. I had to run it several times on the priv database before I got no warning in the summary.

I could restart all services.

The store is running, and I've just launched another export of mailboxes.

I will wait until tomorrow to ask users if they have everything.

Fred
0
David WilhoitSenior Consultant, ExchangeCommented:
/p is a guaranteed data loss situation. How much, depends on the database and how much corruption it has. /P is brute force fix. they may have lost nothing much, or they may have lost a lot. See a big difference in the size of the priv.edb, or is it about the same?

D
0
fhoAuthor Commented:
The size is almost the same as it was before I launched the repair command. As far as I could see, the size of  files generated by the export I launched just after the repair seems to be the same as well.
Now, I just have to wait until users open their mailbox today.
Fred
0
fhoAuthor Commented:
Well, at least the issue has been solved.
The repair finished without any issue, but users couldn't open or send emails with attachment.
That's why I finally created a new store (priv only) and restored pst.
IT's now working.
Thanks for your patience, and pugnacity
Fred
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Exchange

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.