Link to home
Start Free TrialLog in
Avatar of Joe
JoeFlag for United States of America

asked on

Persistent VSS writer error 9607 when performing backups

I am having an issue with my Snap Manager for Exchange backups completing successfully consistently to our FAS270. When they fail i am getting this error.

"VSS_E_WRITERERROR_RETRYABLE: The writer failed due to an error that might not occur if another snapshot copy is created.

Backup SG [Average User Group] Error: SnapManager detected the following Exchange writer error. Please retry SnapManager operation.  "

Then this final event code after showing that all information stores failed.
Error code 0xC00413CA, please check SnapManager report and event logs for details.

This is the tell tale event that i found in the event log. Our problem is pretty much exactly the same as the microsoft link below.

Event ID 9607
Source: MSExchangeIS
Category: Exchange VSS Writer
Error code 0xfffff69f when preparing for backup.

I googled the event ID and Microsoft did have a solution for the issue.
http://support.microsoft.com/kb/910250
Which was to restart the information store. This DOES work. However when the backup goes to run the next night it fails again with the same error and the information store needs to be restarted again for the backup to complete. This is happening on both of our Exchange Servers. They are both scheduled to run at the same time.

I have also recreated backup jobs on both servers.

We have Exchange Server 2003, SnapManager version 6.

Please help :)

thanks
Avatar of Exchange_Geek
Exchange_Geek
Flag of India image

In case you're Exchange box is on VM - read the solution on the following link

https://communities.netapp.com/message/7148

As per NetApp support, there seems to be an inherit issue. However, if you're using Exchange on physical box - things are different.

Personally, I'd try to take backup using BETEST tool - this would prove if there is an issue with Exchange/Windows and if the tool succeeds, there is a must issue with NetApp.

Ref: http://msdn.microsoft.com/en-us/library/windows/desktop/bb530721(v=vs.85).aspx

If you're against VSS, you could take backup using native Windows Backup utility (my fav)

Regards,
Exchange_Geek
Avatar of Joe

ASKER

Thanks for the reply

We are using VMs so i will check out the link tomorrow.

I have personally never used this tool. Does it actually perform the backup or just test it?

I also noticed something else today when looking through the event-log. It seems that snap manager is starting 20 minutes earlier than scheduled on the days that the backup failed. I'm not sure if this is for the backup but if there is another snap-manager process running it will interfere with the backup. I looked in scheduled tasked just to verify that nothing was scheduled to run at this time and there was nothing. I rescheduled the backup for before the other SM task was running to see if this helps.

If not then i will try the BETEST tool.

Thanks again.
Avatar of Joe

ASKER

I used the BETEST tool on both exchange. I did not run a backup just a snapshot. The following are the results.


C:\Program Files\Microsoft\betest\obj\i386>betest.exe /snapshot J:

Error in c:\nt\drivers\storage\volsnap\vss\server\tests\betest\main.cpp(2313):
        - Call pvbc->AddToSnapshotSet(g_wszJustSnapshot, GUID_NULL, &idSnap) not succeeded.
          Error code = 0x80042308. Error description = VSS_E_OBJECT_NOT_FOUND
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

C:\Program Files\Microsoft\betest\obj\i386>betest.exe /snapshot M:
Error in c:\nt\drivers\storage\volsnap\vss\server\tests\betest\main.cpp(2312):
        - Call pvbc->StartSnapshotSet(&idSet) not succeeded.
          Error code = 0x80042316. Error description = VSS_E_SNAPSHOT_SET_IN_PROGRESS


I am getting different errors for the different drives. I am guessing this means that there is an issue with windows. Right?
Are you aware of NTBackup? If yes, can you take an Exchange backup using it?

However, before that restart Information store service.

Meanwhile I'm trying to surf around to ensure that you can take a backup using BETEST using the proper command lines.

Regards,
Exchange_Geek
Avatar of Joe

ASKER

It is saying that there is an error in "c:\nt\drivers\storage\volsnap\vss\server\tests\betest\" Which does not exist i tried to go there manually and it wasn't there. Hidden and protected folders are showing also.

i am aware of NTbackup, yes. I do have a question though. Will i still be able to backup to the FAS270 with this and not SME?

If i am going to run a backup it will have to be during our maintenance hours (2am- 4am)

I am currently running BETEST.exe /N which tests a new backup infrastructure. I am seeing similar errors to what we were getting with the SME backups.
Avatar of Joe

ASKER

Okay it is finished. It is shoeing that all the writers are good except the Exchange Writer

status After Prepare Backup (9 writers)

Status for writer System Writer: STABLE(0x00000000)
Status for writer MSDEWriter: STABLE(0x00000000)
Status for writer Microsoft Exchange Writer: FAILED_AT_PREPARE_BACKUP(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)
Status for writer BITS Writer: STABLE(0x00000000)
Status for writer WMI Writer: STABLE(0x00000000)
Status for writer Event Log Writer: STABLE(0x00000000)
Status for writer Registry Writer: STABLE(0x00000000)
Status for writer IIS Metabase Writer: STABLE(0x00000000)
Status for writer COM+ REGDB Writer: STABLE(0x00000000)

Im am looking into ways to troubleshoot the writer. I am curious why both of our exchanges are having the exact same issue... weird
Avatar of Joe

ASKER

Okay i possibly have a new lead. After looking in the system log i noticed a few event warnings and error that are logged when i run a backup or a test backup. Attached.

It is saying that the signature for disk 11 matches the signature of disk 8. It is also showing various other disks ie. 16 matches 3, 17 matches 5... The thing is we only have disk 0-10 as verified be the diskpart tool.  Any idea whast could be causing this.

Thanks
ERRORS.JPG
Odd, well have the Exchange writers in a good state first and then run the tool again.

NTBackup failed / succeeded?

Regards,
Exchange_Geek
Avatar of Joe

ASKER

I can't run the backup right now. Not until the maintenance hours. my guess is that it will fail with the same error regardless of the program that it is run with. I guess we'll see though.

That plugplaymanager error that i am getting is a similar error to when you improperly remove removable USB storage. I'm thinking that the LUN's were improperly shutdown at some point and are showing causing a lot of errors because they were never properly reconnected. Even when the backups are completing there are still exchange VSS writer errors. The backup still completes though.

My suggestion was to completely shut down the racks that the servers are on and the storage devices. Hopefully when i do it properly everything will reconnect and have no errors.
Avatar of Joe

ASKER

I am running an NTbackup on one of the drives. A small one that won't effect system performance. It might pass because i have restarted the information store a few time throughout the day.
Avatar of Joe

ASKER

I just made a very interesting discovery that i probably should have made a while ago that may be a part of the issue. I know that i am chasing a lot of different things but they really want this fixed. I went to the actual information stores and noticed .IFS database files and database temp files that i know for a fact are not supposed to be there. (Attached) It is like this for all of our information stores. I thought that you would know more about the file types.

Thanks
Information-store..JPG
Nothing to worry about those files and simple Information store service restart should get rid of them.

http://support.microsoft.com/kb/260532

I'll be interested to know about your server restart idea and about NT Backup success/failure.

Regards,
Exchange_Geek
Avatar of Joe

ASKER

The NTbackup passed . I tested it out on an information store. Good stuff. I am running a test SME backup now and there are absolutely no errors so far. It is only on one information store though. We'll see what happens when the actual backup runs tonight.
Superb

Regards,
Exchange_Geek
Avatar of Joe

ASKER

Well, both exchanges passed the backup with absolutely no errors. That being said, I want to wait a couple days to verify that they will continue to pass on their own. Without me babying it.
Not an issue at all.

Regards,
Exchange_Geek
Avatar of Joe

ASKER

Both backups passed and exchange svr #2 had no errors. However, Exchange server #1 had the following error.

Event ID 9617
Source: MSExchangeIS
Category: Exchange VSS Writer

Error code 0xfffff8ed when truncating the logs.

And this error in the backup log

[03:05:21.676]  Starting asynchronous BackupComplete. Please wait...

[03:05:22.691]  Operation pending, please wait...(1)
[03:05:27.690]  Asynchronous BackupComplete finished.

[03:05:27.894]  Status after BackupComplete (9 writers)
[03:05:27.894]  Status for writer System Writer: STABLE(0x00000000)
[03:05:27.894]  Status for writer Microsoft Exchange Writer: FAILED_AT_BACKUP_COMPLETE(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)

[03:05:27.894]  Exchange has reported error writer status.
[03:05:27.894]  Error code: 0x800423f3 VSS_E_WRITERERROR_RETRYABLE: The writer failed due to an error that might not occur if another snapshot is created.
[03:05:27.894]  Status for writer Event Log Writer: STABLE(0x00000000)
[03:05:27.894]  Status for writer BITS Writer: STABLE(0x00000000)
[03:05:27.894]  Status for writer Registry Writer: STABLE(0x00000000)
[03:05:27.894]  Status for writer COM+ REGDB Writer: STABLE(0x00000000)
[03:05:27.909]  Status for writer WMI Writer: STABLE(0x00000000)
[03:05:27.909]  Status for writer MSDEWriter: STABLE(0x00000000)
[03:05:27.909]  Status for writer IIS Metabase Writer: STABLE(0x00000000)


The writer only failed at backup complete though. Which is probably why the backup completed successfully.

Thanks
My next action plan is to ensure that your log files are cleared. However, let me know how many storage groups and database you have?

Regards,
Exchange_Geek
Avatar of Joe

ASKER

We have four storage groups on both exchange servers. The logs are being cleared on all of them except for two storage groups on exchange server #1. What could be causing this?
ASKER CERTIFIED SOLUTION
Avatar of Exchange_Geek
Exchange_Geek
Flag of India image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Joe

ASKER

Okay good stuff i did as instructed and there are only the last five left. Once we run the backup tonight i will let you know the outcome.

Thanks a lot!
Perfect, al the best.

Regards,
Exchange_Geek
Avatar of Joe

ASKER

This is kind of a side note if you don't mind answering another question. Can you run a eseutil defrag on an exchange server that is on a VM? I have heard that you cannot and that you can.
Absolutely you can run it, al though it makes sense to understand how much space you'd benefit from it.

Regards,
Exchange_Geek
Avatar of Joe

ASKER

They aren't letting me do. It and i understand how much space it will clear up because i have done it before on our other network which usually has no issues because there are less users.

If i can prove to them that it is a good idea to perform this operation i can get them to consider it. That is how the military works lol. You have you prove a reason to perform every troubleshooting step.

Thanks though I figured id ask a real expert
Look, offline defrag is a good idea - but as i said it makes sense to understand what do you gain from it.

If you have a 100 gig DB and you stand to gain 5 gig from it - it isn't worth the effort at all.

Instead, what you could do is it to for creating a new DB and move mailboxes to it. Once done, delete the old store.

Point to note is that it'll create a lot of transaction logs - hence circular logging is preferred.

Regards,
Exchange_Geek
Avatar of Joe

ASKER

Okay i see...  I have also heard of this method. Our databases are between 15 and 35 GB. Wouldn't moving all the mailboxes to a new store and then deleting the old store take just as long though? Or for the smaller databases should we do a defrag and for the bigger ones do the new info store method? What would you suggest?

Thanks
You're on E2003, so check for event id 1221 - that'll help you understand free spaces in your database. Let me know this info.

Regards,
Exchange_Geek
Avatar of Joe

ASKER

Gotchya i will let you know all the details tomorrow.

Thanks!
Avatar of Joe

ASKER

Okay awesome! The error went away but again i want to monitor for like two days or so to verify proper functionality of the backups.

Here is the info for event 1221-

The database "Average User Group\AverageMailboxStore1" has 1390 megabytes of free space after online defragmentation has terminated.
The database "Average User Group\AverageMailboxStore2" has 1344 megabytes of free space after online defragmentation has terminated.
The database "Medium User Group\MediumMailboxStore2" has 11730 megabytes of free space after online defragmentation has terminated.
The database "Medium User Group\MediumMailboxStore1" has 59 megabytes of free space after online defragmentation has terminated.
The database "Heavy User Group\HeavyMailboxStore2" has 772 megabytes of free space after online defragmentation has terminated.
The database "Heavy User Group\HeavyMailboxStore1" has 1690 megabytes of free space after online defragmentation has terminated.

Thanks a lot
Do you have database sizes of all of them?

If yes, value what percentage you gain if we work on offline defrag / move mailbox method.

Regards,
Exchange_Geek
Avatar of Joe

ASKER

Yes i do

AverageMailboxStore1- 32 GB
AverageMailboxStore2- 14 GB
MediumMailboxStore1- 15 GB
MediumMailboxStore2- 25 GB
HeavyMailboxStore1- 13 GB
HeavyMailboxStore2- 11 GB
The only ones that needs work/offlinedefrag/movemailbox are

AverageMailboxStore2- 14 GB where you gain 1.3 GB (i.e 10% of the DB Size)
MediumMailboxStore2- 25 GB where you gain 11.7 GB (i.e 47% of the DB Size)
HeavyMailboxStore1- 13 GB where you gain 1.6 GB ((i.e 10% of the DB Size)

Regards,
Exchange_Geek
Avatar of Joe

ASKER

I  appreciate it. I will make this suggestion. I will let the backups run for one more day and then call  this question complete.
Cool.

Regards,
Exchange_Geek
Avatar of Joe

ASKER

Thanks so much for the help! You're a life saver!