Avatar of Joe
Joe
Flag for United States of America asked on

Persistent VSS writer error 9607 when performing backups

I am having an issue with my Snap Manager for Exchange backups completing successfully consistently to our FAS270. When they fail i am getting this error.

"VSS_E_WRITERERROR_RETRYABLE: The writer failed due to an error that might not occur if another snapshot copy is created.

Backup SG [Average User Group] Error: SnapManager detected the following Exchange writer error. Please retry SnapManager operation.  "

Then this final event code after showing that all information stores failed.
Error code 0xC00413CA, please check SnapManager report and event logs for details.

This is the tell tale event that i found in the event log. Our problem is pretty much exactly the same as the microsoft link below.

Event ID 9607
Source: MSExchangeIS
Category: Exchange VSS Writer
Error code 0xfffff69f when preparing for backup.

I googled the event ID and Microsoft did have a solution for the issue.
http://support.microsoft.com/kb/910250
Which was to restart the information store. This DOES work. However when the backup goes to run the next night it fails again with the same error and the information store needs to be restarted again for the backup to complete. This is happening on both of our Exchange Servers. They are both scheduled to run at the same time.

I have also recreated backup jobs on both servers.

We have Exchange Server 2003, SnapManager version 6.

Please help :)

thanks
Storage SoftwareExchangeWindows Server 2003

Avatar of undefined
Last Comment
Joe

8/22/2022 - Mon
Exchange_Geek

In case you're Exchange box is on VM - read the solution on the following link

https://communities.netapp.com/message/7148

As per NetApp support, there seems to be an inherit issue. However, if you're using Exchange on physical box - things are different.

Personally, I'd try to take backup using BETEST tool - this would prove if there is an issue with Exchange/Windows and if the tool succeeds, there is a must issue with NetApp.

Ref: http://msdn.microsoft.com/en-us/library/windows/desktop/bb530721(v=vs.85).aspx

If you're against VSS, you could take backup using native Windows Backup utility (my fav)

Regards,
Exchange_Geek
Joe

ASKER
Thanks for the reply

We are using VMs so i will check out the link tomorrow.

I have personally never used this tool. Does it actually perform the backup or just test it?

I also noticed something else today when looking through the event-log. It seems that snap manager is starting 20 minutes earlier than scheduled on the days that the backup failed. I'm not sure if this is for the backup but if there is another snap-manager process running it will interfere with the backup. I looked in scheduled tasked just to verify that nothing was scheduled to run at this time and there was nothing. I rescheduled the backup for before the other SM task was running to see if this helps.

If not then i will try the BETEST tool.

Thanks again.
Joe

ASKER
I used the BETEST tool on both exchange. I did not run a backup just a snapshot. The following are the results.


C:\Program Files\Microsoft\betest\obj\i386>betest.exe /snapshot J:

Error in c:\nt\drivers\storage\volsnap\vss\server\tests\betest\main.cpp(2313):
        - Call pvbc->AddToSnapshotSet(g_wszJustSnapshot, GUID_NULL, &idSnap) not succeeded.
          Error code = 0x80042308. Error description = VSS_E_OBJECT_NOT_FOUND
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

C:\Program Files\Microsoft\betest\obj\i386>betest.exe /snapshot M:
Error in c:\nt\drivers\storage\volsnap\vss\server\tests\betest\main.cpp(2312):
        - Call pvbc->StartSnapshotSet(&idSet) not succeeded.
          Error code = 0x80042316. Error description = VSS_E_SNAPSHOT_SET_IN_PROGRESS


I am getting different errors for the different drives. I am guessing this means that there is an issue with windows. Right?
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
Exchange_Geek

Are you aware of NTBackup? If yes, can you take an Exchange backup using it?

However, before that restart Information store service.

Meanwhile I'm trying to surf around to ensure that you can take a backup using BETEST using the proper command lines.

Regards,
Exchange_Geek
Joe

ASKER
It is saying that there is an error in "c:\nt\drivers\storage\volsnap\vss\server\tests\betest\" Which does not exist i tried to go there manually and it wasn't there. Hidden and protected folders are showing also.

i am aware of NTbackup, yes. I do have a question though. Will i still be able to backup to the FAS270 with this and not SME?

If i am going to run a backup it will have to be during our maintenance hours (2am- 4am)

I am currently running BETEST.exe /N which tests a new backup infrastructure. I am seeing similar errors to what we were getting with the SME backups.
Joe

ASKER
Okay it is finished. It is shoeing that all the writers are good except the Exchange Writer

status After Prepare Backup (9 writers)

Status for writer System Writer: STABLE(0x00000000)
Status for writer MSDEWriter: STABLE(0x00000000)
Status for writer Microsoft Exchange Writer: FAILED_AT_PREPARE_BACKUP(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)
Status for writer BITS Writer: STABLE(0x00000000)
Status for writer WMI Writer: STABLE(0x00000000)
Status for writer Event Log Writer: STABLE(0x00000000)
Status for writer Registry Writer: STABLE(0x00000000)
Status for writer IIS Metabase Writer: STABLE(0x00000000)
Status for writer COM+ REGDB Writer: STABLE(0x00000000)

Im am looking into ways to troubleshoot the writer. I am curious why both of our exchanges are having the exact same issue... weird
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Joe

ASKER
Okay i possibly have a new lead. After looking in the system log i noticed a few event warnings and error that are logged when i run a backup or a test backup. Attached.

It is saying that the signature for disk 11 matches the signature of disk 8. It is also showing various other disks ie. 16 matches 3, 17 matches 5... The thing is we only have disk 0-10 as verified be the diskpart tool.  Any idea whast could be causing this.

Thanks
ERRORS.JPG
Exchange_Geek

Odd, well have the Exchange writers in a good state first and then run the tool again.

NTBackup failed / succeeded?

Regards,
Exchange_Geek
Joe

ASKER
I can't run the backup right now. Not until the maintenance hours. my guess is that it will fail with the same error regardless of the program that it is run with. I guess we'll see though.

That plugplaymanager error that i am getting is a similar error to when you improperly remove removable USB storage. I'm thinking that the LUN's were improperly shutdown at some point and are showing causing a lot of errors because they were never properly reconnected. Even when the backups are completing there are still exchange VSS writer errors. The backup still completes though.

My suggestion was to completely shut down the racks that the servers are on and the storage devices. Hopefully when i do it properly everything will reconnect and have no errors.
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
Joe

ASKER
I am running an NTbackup on one of the drives. A small one that won't effect system performance. It might pass because i have restarted the information store a few time throughout the day.
Joe

ASKER
I just made a very interesting discovery that i probably should have made a while ago that may be a part of the issue. I know that i am chasing a lot of different things but they really want this fixed. I went to the actual information stores and noticed .IFS database files and database temp files that i know for a fact are not supposed to be there. (Attached) It is like this for all of our information stores. I thought that you would know more about the file types.

Thanks
Information-store..JPG
Exchange_Geek

Nothing to worry about those files and simple Information store service restart should get rid of them.

http://support.microsoft.com/kb/260532

I'll be interested to know about your server restart idea and about NT Backup success/failure.

Regards,
Exchange_Geek
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Joe

ASKER
The NTbackup passed . I tested it out on an information store. Good stuff. I am running a test SME backup now and there are absolutely no errors so far. It is only on one information store though. We'll see what happens when the actual backup runs tonight.
Exchange_Geek

Superb

Regards,
Exchange_Geek
Joe

ASKER
Well, both exchanges passed the backup with absolutely no errors. That being said, I want to wait a couple days to verify that they will continue to pass on their own. Without me babying it.
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
Exchange_Geek

Not an issue at all.

Regards,
Exchange_Geek
Joe

ASKER
Both backups passed and exchange svr #2 had no errors. However, Exchange server #1 had the following error.

Event ID 9617
Source: MSExchangeIS
Category: Exchange VSS Writer

Error code 0xfffff8ed when truncating the logs.

And this error in the backup log

[03:05:21.676]  Starting asynchronous BackupComplete. Please wait...

[03:05:22.691]  Operation pending, please wait...(1)
[03:05:27.690]  Asynchronous BackupComplete finished.

[03:05:27.894]  Status after BackupComplete (9 writers)
[03:05:27.894]  Status for writer System Writer: STABLE(0x00000000)
[03:05:27.894]  Status for writer Microsoft Exchange Writer: FAILED_AT_BACKUP_COMPLETE(0x800423f3 - VSS_E_WRITERERROR_RETRYABLE)

[03:05:27.894]  Exchange has reported error writer status.
[03:05:27.894]  Error code: 0x800423f3 VSS_E_WRITERERROR_RETRYABLE: The writer failed due to an error that might not occur if another snapshot is created.
[03:05:27.894]  Status for writer Event Log Writer: STABLE(0x00000000)
[03:05:27.894]  Status for writer BITS Writer: STABLE(0x00000000)
[03:05:27.894]  Status for writer Registry Writer: STABLE(0x00000000)
[03:05:27.894]  Status for writer COM+ REGDB Writer: STABLE(0x00000000)
[03:05:27.909]  Status for writer WMI Writer: STABLE(0x00000000)
[03:05:27.909]  Status for writer MSDEWriter: STABLE(0x00000000)
[03:05:27.909]  Status for writer IIS Metabase Writer: STABLE(0x00000000)


The writer only failed at backup complete though. Which is probably why the backup completed successfully.

Thanks
Exchange_Geek

My next action plan is to ensure that your log files are cleared. However, let me know how many storage groups and database you have?

Regards,
Exchange_Geek
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Joe

ASKER
We have four storage groups on both exchange servers. The logs are being cleared on all of them except for two storage groups on exchange server #1. What could be causing this?
ASKER CERTIFIED SOLUTION
Exchange_Geek

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
Joe

ASKER
Okay good stuff i did as instructed and there are only the last five left. Once we run the backup tonight i will let you know the outcome.

Thanks a lot!
Exchange_Geek

Perfect, al the best.

Regards,
Exchange_Geek
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
Joe

ASKER
This is kind of a side note if you don't mind answering another question. Can you run a eseutil defrag on an exchange server that is on a VM? I have heard that you cannot and that you can.
Exchange_Geek

Absolutely you can run it, al though it makes sense to understand how much space you'd benefit from it.

Regards,
Exchange_Geek
Joe

ASKER
They aren't letting me do. It and i understand how much space it will clear up because i have done it before on our other network which usually has no issues because there are less users.

If i can prove to them that it is a good idea to perform this operation i can get them to consider it. That is how the military works lol. You have you prove a reason to perform every troubleshooting step.

Thanks though I figured id ask a real expert
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Exchange_Geek

Look, offline defrag is a good idea - but as i said it makes sense to understand what do you gain from it.

If you have a 100 gig DB and you stand to gain 5 gig from it - it isn't worth the effort at all.

Instead, what you could do is it to for creating a new DB and move mailboxes to it. Once done, delete the old store.

Point to note is that it'll create a lot of transaction logs - hence circular logging is preferred.

Regards,
Exchange_Geek
Joe

ASKER
Okay i see...  I have also heard of this method. Our databases are between 15 and 35 GB. Wouldn't moving all the mailboxes to a new store and then deleting the old store take just as long though? Or for the smaller databases should we do a defrag and for the bigger ones do the new info store method? What would you suggest?

Thanks
Exchange_Geek

You're on E2003, so check for event id 1221 - that'll help you understand free spaces in your database. Let me know this info.

Regards,
Exchange_Geek
Your help has saved me hundreds of hours of internet surfing.
fblack61
Joe

ASKER
Gotchya i will let you know all the details tomorrow.

Thanks!
Joe

ASKER
Okay awesome! The error went away but again i want to monitor for like two days or so to verify proper functionality of the backups.

Here is the info for event 1221-

The database "Average User Group\AverageMailboxStore1" has 1390 megabytes of free space after online defragmentation has terminated.
The database "Average User Group\AverageMailboxStore2" has 1344 megabytes of free space after online defragmentation has terminated.
The database "Medium User Group\MediumMailboxStore2" has 11730 megabytes of free space after online defragmentation has terminated.
The database "Medium User Group\MediumMailboxStore1" has 59 megabytes of free space after online defragmentation has terminated.
The database "Heavy User Group\HeavyMailboxStore2" has 772 megabytes of free space after online defragmentation has terminated.
The database "Heavy User Group\HeavyMailboxStore1" has 1690 megabytes of free space after online defragmentation has terminated.

Thanks a lot
Exchange_Geek

Do you have database sizes of all of them?

If yes, value what percentage you gain if we work on offline defrag / move mailbox method.

Regards,
Exchange_Geek
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Joe

ASKER
Yes i do

AverageMailboxStore1- 32 GB
AverageMailboxStore2- 14 GB
MediumMailboxStore1- 15 GB
MediumMailboxStore2- 25 GB
HeavyMailboxStore1- 13 GB
HeavyMailboxStore2- 11 GB
Exchange_Geek

The only ones that needs work/offlinedefrag/movemailbox are

AverageMailboxStore2- 14 GB where you gain 1.3 GB (i.e 10% of the DB Size)
MediumMailboxStore2- 25 GB where you gain 11.7 GB (i.e 47% of the DB Size)
HeavyMailboxStore1- 13 GB where you gain 1.6 GB ((i.e 10% of the DB Size)

Regards,
Exchange_Geek
Joe

ASKER
I  appreciate it. I will make this suggestion. I will let the backups run for one more day and then call  this question complete.
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
Exchange_Geek

Cool.

Regards,
Exchange_Geek
Joe

ASKER
Thanks so much for the help! You're a life saver!