Cannot backup some Exchange 2003 Contacts with Backup Eexec 9.1 after server lockup
Hello Experts,
I am having some strange behavior with our Exchange 2003 Std SP2 server. We have a single Exchange server environment. We had a problem with this server completely locking up last week during a backup job that did not involve Exchange, but rather some file system data on the same server. Lots of I/O related to the backup at a time when the server was busy with some of its own I/O intensinve processes. There was no choice but to pull the plug and restart the server "hard", so to speak. We've since moved the backup to a different time window, with no more lock ups thus far. We are using Backup Exec 9.1 at the moment.
Anyway, since the server came back up, we have not been able to back up certain of our Contacts when doing a mailbox level backup. We will repeatedly see this in the backup log for many, but not all Contacts:
Access denied to file Jane Doe [jdoe]Top of Information StoreContactsÃ
The item \\FS01\Microsoft Exchange Mailboxes\Jane Doe [jdoe]Top of Information StoreContactsÃ
The contacts come from many mailboxes, and appear to be more or less the same list of Contacts each night (four nights so far). Those same contacts can be opened and used by users. When this occurs, the Exchange server logs a single 9646 error from MSExchangeIS. Based on the following KB article, I have increased the number of messages a MAPI client can open, but the first jump from 250 to 500 was not enough:
http://support.microsoft.com/kb/830836
Even were I to reach a number that was enough, the fact remains that pre lockup, this was not necessary. I wonder if there is store corruption that is not significant enough to impair the functioning of the server, but enough to impact the backup. I've reviewed this article:
http://msexchangeteam.com/archive/2004/06/18/159413.aspx
But it talks about these steps as a means to recover an unmountable store, etc. May be extreme in my circumstances. May be related or may not be, but we have one user who is having difficulty sending mail. I am not convinced yet that it isn't an issue with the MAPI profile on her machine, but we are still troubleshooting. In any event she had a few days of trouble free usage after the lock up event, so the timing is wrong for this to be related.
Any suggestions as to a reasonable approach to this issue are greatly appreciated.
Thanks!
I am having some strange behavior with our Exchange 2003 Std SP2 server. We have a single Exchange server environment. We had a problem with this server completely locking up last week during a backup job that did not involve Exchange, but rather some file system data on the same server. Lots of I/O related to the backup at a time when the server was busy with some of its own I/O intensinve processes. There was no choice but to pull the plug and restart the server "hard", so to speak. We've since moved the backup to a different time window, with no more lock ups thus far. We are using Backup Exec 9.1 at the moment.
Anyway, since the server came back up, we have not been able to back up certain of our Contacts when doing a mailbox level backup. We will repeatedly see this in the backup log for many, but not all Contacts:
Access denied to file Jane Doe [jdoe]Top of Information StoreContactsÃ
The item \\FS01\Microsoft Exchange Mailboxes\Jane Doe [jdoe]Top of Information StoreContactsÃ
The contacts come from many mailboxes, and appear to be more or less the same list of Contacts each night (four nights so far). Those same contacts can be opened and used by users. When this occurs, the Exchange server logs a single 9646 error from MSExchangeIS. Based on the following KB article, I have increased the number of messages a MAPI client can open, but the first jump from 250 to 500 was not enough:
http://support.microsoft.com/kb/830836
Even were I to reach a number that was enough, the fact remains that pre lockup, this was not necessary. I wonder if there is store corruption that is not significant enough to impair the functioning of the server, but enough to impact the backup. I've reviewed this article:
http://msexchangeteam.com/archive/2004/06/18/159413.aspx
But it talks about these steps as a means to recover an unmountable store, etc. May be extreme in my circumstances. May be related or may not be, but we have one user who is having difficulty sending mail. I am not convinced yet that it isn't an issue with the MAPI profile on her machine, but we are still troubleshooting. In any event she had a few days of trouble free usage after the lock up event, so the timing is wrong for this to be related.
Any suggestions as to a reasonable approach to this issue are greatly appreciated.
Thanks!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Thanks for the reply.
The path that I pasted from the log had different characters than what ended up in this post, for some reason. The path in the log is 173 characters long and includes square symbols, but none of the other garbage above. Not saying that's normal or anything, just different than above.
I will try exporting the Contacts from one of the problem mailboxes and re-importing them to see if that resolves the issue for those specific Contacts. I'm going to try to avoid increasing that limit if I can resolve this by another means, because I'm sure it is there for a reason.
If it turns out that export / re-import of the roughly 950 contacts that have trouble cleans up the backup issues, would you still run ISINTEG and ESEUTIL as described in the msexchange.org link referenced above, given the lockup, and even though the store appears to be otherwise functional?
Was not sure if this was a sledgehammer where a tack hammer would do...
The path that I pasted from the log had different characters than what ended up in this post, for some reason. The path in the log is 173 characters long and includes square symbols, but none of the other garbage above. Not saying that's normal or anything, just different than above.
I will try exporting the Contacts from one of the problem mailboxes and re-importing them to see if that resolves the issue for those specific Contacts. I'm going to try to avoid increasing that limit if I can resolve this by another means, because I'm sure it is there for a reason.
If it turns out that export / re-import of the roughly 950 contacts that have trouble cleans up the backup issues, would you still run ISINTEG and ESEUTIL as described in the msexchange.org link referenced above, given the lockup, and even though the store appears to be otherwise functional?
Was not sure if this was a sledgehammer where a tack hammer would do...
ASKER
I exported the contacts from a test mailbox to pst, shift-deleted all the contacts in the test mailbox, then re-imported all the contacts back to the test mailbox from the pst backup, but the error is exactly the same. It doesn't appear to be the items themselves, which is odd. Even within the same mailbox, there are only certain contacts that are problematic. This would have led me to believe that re-creating them would have solved the problem.
The sqaures that you see are probably some ASIAN CHARACHTERS for eg JAPANESE .You see squares and not the chars cause probably you do not have the language pack installed.
The JAP CHAR are DOUBLE BYTE CHARS and hence will be more than 173
The JAP CHAR are DOUBLE BYTE CHARS and hence will be more than 173
ASKER
Thanks. That makes sense.
It is starting to sound like there is some database corruption, especially given the lockup the server experienced last week. The user with the difficulty sending mail turns out to have non-machine specific issues. Cannot export many of her calendar and sent items with ExMerge. At this point I am looking for some guidance in terms of repair order and which steps I really need to do. I am guessing that because the app log says that the soft recovery completed successfully on server boot I don't need to do that. Not clear on whether ESEUTIL /P should be run on a store that can already be mounted (and is for the most part functional), based on what I read in your links.
Should it go like this?
1. backup private store
2. dismount private store
3. run integrity check (ESEUTIL /G)
4. if integrity check encounters trouble, repair with ESEUTIL /P
5. if I run ESEUTIL /P, defrag with ESEUTIL /D
6. run ISINTEG -FIX -TEST ALLTESTS until I get zero errors
Thanks
It is starting to sound like there is some database corruption, especially given the lockup the server experienced last week. The user with the difficulty sending mail turns out to have non-machine specific issues. Cannot export many of her calendar and sent items with ExMerge. At this point I am looking for some guidance in terms of repair order and which steps I really need to do. I am guessing that because the app log says that the soft recovery completed successfully on server boot I don't need to do that. Not clear on whether ESEUTIL /P should be run on a store that can already be mounted (and is for the most part functional), based on what I read in your links.
Should it go like this?
1. backup private store
2. dismount private store
3. run integrity check (ESEUTIL /G)
4. if integrity check encounters trouble, repair with ESEUTIL /P
5. if I run ESEUTIL /P, defrag with ESEUTIL /D
6. run ISINTEG -FIX -TEST ALLTESTS until I get zero errors
Thanks
ASKER
I had a support incident via my technet subscription that was about to expire, so I decided to use it for this. The tech spent some time poking around the system, and said that the absence of event ids 9518 and 9519 suggested that there was no physical level corruption in the database (no need to run eseutil). They did suggest running ISINTEG, so I am going to back up the private store and then give that a go tonight. I'll post back with my results.
Thanks
Thanks
ASKER
Following up on my efforts to resolve this issue. Here's what I did:
1. stopped any services like server based spam control / antivirus, and the smtp service on exchange
2. dismounted private store
3. copied db and steaming files out to another location for safekeeping
4. ran ISINTEG with the FIX and ALLTESTS options - no errors but several fixes applied
5. ran ISINTEG again with same options - no errors and no fixes, so I was done
6. mounted private store, started any services I had stopped
7. tested backup, tested export of items to PST that had been troublesome, tested sending email from user accounts that had experienced problems doing so. No more 9646 errors either.
Everything seems to be working fine. The one thing I got from Microsoft support that I hadn't gotten here was that because there were no 9518 or 9519 events in the event log, I knew I could skip anything to do with physical level corruption (eseutil). We also ran the exchange BPA and Troubleshooting Assistant to make sure there were no non-database related issues, which there were not.
Anyway, thanks for your help!
1. stopped any services like server based spam control / antivirus, and the smtp service on exchange
2. dismounted private store
3. copied db and steaming files out to another location for safekeeping
4. ran ISINTEG with the FIX and ALLTESTS options - no errors but several fixes applied
5. ran ISINTEG again with same options - no errors and no fixes, so I was done
6. mounted private store, started any services I had stopped
7. tested backup, tested export of items to PST that had been troublesome, tested sending email from user accounts that had experienced problems doing so. No more 9646 errors either.
Everything seems to be working fine. The one thing I got from Microsoft support that I hadn't gotten here was that because there were no 9518 or 9519 events in the event log, I knew I could skip anything to do with physical level corruption (eseutil). We also ran the exchange BPA and Troubleshooting Assistant to make sure there were no non-database related issues, which there were not.
Anyway, thanks for your help!
Also try increasing the mapi object limit from 500 to 1000 if you are still receiving error 9646.
You can also try recreating the contact.
If you have doubts about the integrity of yr exchange database you can use the ESEUTIL and ISINTEG to defrag and check/repair the d/b.