[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

background cleanup generates high cpu store on Exchange 5.5

Posted on 2004-11-14
26
Medium Priority
?
1,129 Views
Last Modified: 2012-05-05
Hi,
I have an Exchange 5.5 (Build 2653.23: Service Pack 4) server on NT4 (SP6) which takes about 95% of the server's cpu almost all the time.

The application event viewer logs lots of this first event (1101 - error):
Error 0xfffffbd3 occurred  on message <message_id>uring a background cleanup

then it logs a couple of 1104 (error) events :
Error 0xfffffbd3 occurred during a background cleanup on attachment <attachment_id>

Last, it logs a 7201 warning :
Background thread FDoPeriodic encountered a problem. Error code 0xfffffbd3

The only solution is to restart the server as the Store service doesn't answer to a stop request.

Thanks for help.

Fred
0
Comment
Question by:fho
  • 14
  • 8
  • 4
26 Comments
 
LVL 27

Expert Comment

by:Exchange_Admin
ID: 12579019
You may want to stop the Information Store service and make a copy of the PRIV.EDB.
Then run the following:
ISINTEG -FIX -PRI -TEST ALLTESTS
0
 
LVL 1

Author Comment

by:fho
ID: 12579039
Thanks.
Do I need to save priv.edb before, or won't this test affect the file?
0
 
LVL 1

Author Comment

by:fho
ID: 12579062
sorry, I didn't read your full answer. I'll save the file beofre running the command
0
Veeam Disaster Recovery in Microsoft Azure

Veeam PN for Microsoft Azure is a FREE solution designed to simplify and automate the setup of a DR site in Microsoft Azure using lightweight software-defined networking. It reduces the complexity of VPN deployments and is designed for businesses of ALL sizes.

 
LVL 1

Author Comment

by:fho
ID: 12579167
I ran it.
The summary reports no errors, 7 warning and 1 fix.
I could restart the service.
Was it supposed to solve my problem, or just to detect a potentiel corruption of the priv ?
Thanks
Fred
0
 
LVL 27

Expert Comment

by:Exchange_Admin
ID: 12580478
Hopefully it may have resolved the problem.
0
 
LVL 1

Author Comment

by:fho
ID: 12582526
Unfortunately, it didn't. The store is still  using 99 % of the CPU.
I couldn't stop it, and had to restart the server.

0
 
LVL 24

Expert Comment

by:David Wilhoit
ID: 12586998
run the isinteg test until you get 0 warnings and 0 fixes. then you can effectively try the background cleanup again.

D
0
 
LVL 24

Expert Comment

by:David Wilhoit
ID: 12587037
0
 
LVL 1

Author Comment

by:fho
ID: 12590826
I've already applied kb 293836 with no result.
I'll run the isinteg test again
thanks
Fred
0
 
LVL 1

Author Comment

by:fho
ID: 12591112
Kidego,
when you write : "you can effectively try the background cleanup again", does it mean that this isn't a "normal" maintenance task?
I don't remember having configured it, and I didn't see any piece of information about it in logs of other Exchange servers that are in the Org.

I ran isinteg once again and didn't have any warning nor errors.

I've also applied a post SP4 rollup (KB841765) which I had already used on another server to fix a high CPU use by the store (nothing to do with the background cleanup).

Fred
0
 
LVL 24

Expert Comment

by:David Wilhoit
ID: 12595397
http://www.microsoft.com/technet/prodtechnol/exchange/55/maintain/perfopt.mspx

Have you run performance optimizer in a while? Definitely worth a shot, read this:

http://support.microsoft.com/default.aspx?scid=kb;en-us;234702

this gives you some ideas of how to use perfmon to track the issue, and this MTA tuning works really well in my environment. Just a question, but how much free space is available in this database, and how big is it overall? And have you run isinteg fix on the pub.edb as well?


D
0
 
LVL 1

Author Comment

by:fho
ID: 12596259
I've already used the Perfmon after having installed new server in order to move stores on dedicated hard drives.
I'll read these threads.
The store is small (about 3GB) and there is at least 40 free GB on the hard disk where it is. No I haven't run isinteg on the pub.edb yet. I'll do it tomorrow morning.

The good piece of news is that we didn't have any issue today (the store used less than 2 min of CPU time since the last reboot this morning). But I'll wait until tomorrow morning to check, as cleanup tasks runs at night.

I was surprised to see so many records about Exchange (even info) in the application log and had a look to Diagnostic keys in the registry. Some of them have a value of 1 others a value of 3. I thought that the normal configuration was 0 for Diag keys.
Can you confirm?
Thanks
Fred
0
 
LVL 24

Expert Comment

by:David Wilhoit
ID: 12596351
yea, diagnostics are turned off by default. Go into the server properties in the Admin program, and turn those off.

D
0
 
LVL 1

Author Comment

by:fho
ID: 12596502
Thanks. I've done it. Lots of them had a higher level which produced lots of events in the Event Viewer.
Let's wait until tomorrow morning to see if the cpu use of the store came back to normal situation.
Fred
0
 
LVL 24

Expert Comment

by:David Wilhoit
ID: 12596657
keep us posted :)

D
0
 
LVL 1

Author Comment

by:fho
ID: 12605728
It's worse. There are lots of errors logged in the Application EV now during the night.

Database Page cache error 116 : MSExchangeIS (485) Synchronous read page checksum error -1018.
This first error is logged several times.
There are some errors 117 and 200 which also concern Database Page Cache.

I'm not sure that they weren't logged before .... Perhaps were they masked by lots of information messages produced by turned on diags (there were hundreds of info messages in the EV which size was limited to 512K filled in less than one hour).

What is very strange is that after having been restarted, Exchange works (or seems to). Users have access to their mailbox, and can send and receive emails.
I'm also suprised that isinteg command didn't return any error. Isn't is supposed to check database integrity?

Shoudl I launch it again tomorrow morning?
TIA
Fred
0
 
LVL 27

Expert Comment

by:Exchange_Admin
ID: 12606621
-1018 errors=not good.
The ISINTEG utility only checks the relationship between the tables in the database and does not check at the page level of the database.
99% of the time these are caused by hardware problems such as SCSI controller cards or memory issues. If this is a DELL server with a PERC-2 controller card, make sure that you have the latest firmware patch.
Check your system lof to see if there are any entries concerning your SCSI controller cards.

You have 2 options:
Option 1:
Restore from backup. This would be a backup made prior to getting -1018 errors.

Option 2:
Run a hard repair of the database. WARNING:THIS CAN CAUSE LOSS OF DATA

But if you do not resolve the underlying cause of the -1018 errors, they will reoccur.

To run a hard repair:
Stop the Information Store service. Make a copy of the PRIV.EDB and PUB.EDB databases.
Run the following:
ESEUTIL /ISPRIV /P
ESEUTIL /ISPUB /P
ISINTEG -FIX -PRI -TEST ALLTESTS
ISINTEG -FIX -PUB -TEST ALLTESTS

Like I said earlier, you need to resolve the cause of the -1018 errors or they will just come back.
0
 
LVL 24

Expert Comment

by:David Wilhoit
ID: 12607086
Most likely with the 116, you've got hard drive issues. Do you have a hardware contract? Any way you can check your hard drive status thru the monitoring?

D
0
 
LVL 1

Author Comment

by:fho
ID: 12622495
Thanks.
It seems that there were some hardware issues that have been fixed now.
I launched an export of all mailboxes in pst files this morning. Then I'll try an hard repair of the store (/p).
Can you confirm that with pst (if the export runs without any error to its end), I should be able to recover all mailboxes if I need to reinstall a fresh store ?
On the other hand, is there a way to export public folders?
TIA
Fred
0
 
LVL 27

Accepted Solution

by:
Exchange_Admin earned 1000 total points
ID: 12627589
After exporting to PST files, you can stop the Information Store service.
Rename the MDBDATA folders on all drives to MDBDATA.OLD. Create a new MDBDATA folder wherever you renamed one.
Start the Information Store service. This will create a new PRIV.EDB and PUB.EDB.
Send a test email to all users that have a mailbox in this database. You can then run EXMERGE to import the data back into the mailboxes.
0
 
LVL 24

Expert Comment

by:David Wilhoit
ID: 12627998
And yes, you can use outlook to export the public folders to a pst file, just remember the size limitation is 2 GB on a pst, and you're fine.

D
0
 
LVL 1

Author Comment

by:fho
ID: 12632330
Ok, thanks to both of you.
I'll do it this week-end.
Shall I first try to hard repair (isinteg /p) the store, or isn't it worthwhile ?
0
 
LVL 1

Author Comment

by:fho
ID: 12638397
Hi there,
I launched eseutil /p against priv and pub databases (after having exported mailboxes through Exmerge, and save database and log files).

Both finished without any error.
Then, I removed log files (as asked by eseutil) and launched isinteg /fix .... on both databases. I had to run it several times on the priv database before I got no warning in the summary.

I could restart all services.

The store is running, and I've just launched another export of mailboxes.

I will wait until tomorrow to ask users if they have everything.

Fred
0
 
LVL 24

Assisted Solution

by:David Wilhoit
David Wilhoit earned 1000 total points
ID: 12641088
/p is a guaranteed data loss situation. How much, depends on the database and how much corruption it has. /P is brute force fix. they may have lost nothing much, or they may have lost a lot. See a big difference in the size of the priv.edb, or is it about the same?

D
0
 
LVL 1

Author Comment

by:fho
ID: 12641790
The size is almost the same as it was before I launched the repair command. As far as I could see, the size of  files generated by the export I launched just after the repair seems to be the same as well.
Now, I just have to wait until users open their mailbox today.
Fred
0
 
LVL 1

Author Comment

by:fho
ID: 12652129
Well, at least the issue has been solved.
The repair finished without any issue, but users couldn't open or send emails with attachment.
That's why I finally created a new store (priv only) and restored pst.
IT's now working.
Thanks for your patience, and pugnacity
Fred
0

Featured Post

Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Steps to fix error: “Couldn’t mount the database that you specified. Specified database: HU-DB; Error code: An Active Manager operation fail”
Exchange administrators are always vigilant about Exchange crashes and disasters that are possible any time. It is quite essential to identify the symptoms of a possible Exchange issue and be prepared with a proper recovery plan. There are multiple…
In this Micro Video tutorial you will learn the basics about Database Availability Groups and How to configure one using a live Exchange Server Environment. The video tutorial explains the basics of the Exchange server Database Availability grou…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…
Suggested Courses

873 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question