Exchange Server Full Backup successfully executed on passive CCR node but not truncating Transaction Logs ?

Two questions..

1) Are you using the plugin for Exchange backups on the EMC Avamar product, and are you doing an Exchange aware backup? (VSS enabled)

2) If you run VSSADMIN LIST WRITERS do you see any errors on the passive node?

Last thing, what does the avamar backup show you in the logs?

It sounds like you are doing a flat backup (file level) and not an Exchange backup. If you do not run a backup utilizing an exchange aware, which utilizes the VSS snapshot utilities on the OS then it will not truncate your files. It could also be (and I have seen this a few times in the past) a database that is not flipping the correct switches.

1. Yes I am doing it using the Exchange VSS aware backup.
2. No error in all of the VSS writers except one, System Writer got Last error: Non-retryable error.

"It could also be (and I have seen this a few times in the past) a database that is not flipping the correct switches."

What sort of switch is this do you mean ?

So a thing I noticed in Exchange 2007 is that sometimes the attribute for the last backup date sometimes does not get updated correctly. Why, not sure.. but that occurred a few times in the past when I was working on various clients / accounts.

So you have a VSS Writer in a non-retryable error state, you can check the event viewer logs for this: http://technet.microsoft.com/en-us/library/bb218863%28EXCHG.80%29.aspx

I would try to reboot the server, and run the backup again. It sounds like something is not functioning either with the backup agent itself (you can try a reinstall if the reboot doesnt help) or the VSS writing agent itself (which I somewhat doubt).

Thanks for replying to my case,

however I just saw that there are many events listed in the event viewer such as the following Event ID:

Log Name:      Application
Source:        VSS
Date:          4/12/2013 5:39:23 AM
Event ID:      8194
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CCR-MBX-DR1.domain.com
Description:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {40815a03-d488-4477-b2aa-08ce2eb3b26c}

Operation:
   Gathering Writer Data

I have just granted the full access permission on the DCOM configuration as follows:

- Start | Run and type dcomcnfg and hit Enter key
- Expand Component Services
- Expand Computers | My Computer
- Right Click on My Computer and choose Properties
- Select COM Security
- Select Edit Default on Access Permission
- Add the Cluster Service Domain account with Local Access and Remote Access allowed

and also I have confirmed that from the Avamar console and the Windows Event viewer below the backup has finished successfully:

Log Name:      Application
Source:        MSExchangeRepl
Date:          4/12/2013 5:39:25 AM
Event ID:      2046
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      CCR-MBX-DR1.domain.com
Description:
The Microsoft Exchange Replication Service VSS writer (instance e2453332-2AF11-22cc-968d-aba64024b0dc) has successfully completed the backup of storage group 'SG-Office2'.  

Database log truncation has been requested for this storage group. Log truncation will occur on the active copy after the next log generation is created. Log truncation will occur automatically on the passive copy after that log file is copied.

But somehow the transaction log is not truncated in the active node.

Man.. this is pretty weird.

Can you tell me which OS you are on? It sounds like something is wrong with the VSS writer(s) and the information does tell me that.

What OS, and what security patches / hotfixes are installed?

I was skimming through some of my notes and also found this. I would check it out and make sure (if you are running Server 2003, which is a great possibility with Exchange 2007) that this patch is in fact installed (on both nodes): http://support.microsoft.com/kb/940349

ok, I've just found the root cause from the KB article that it is confirmed bug:

http://support.microsoft.com/kb/2297394/en-us

So shall I perform the backup from the active node instead ? Before I plan the testing and implementation for Update Roll Up 11 (which is the latest). http://support.microsoft.com/kb/2873746

The server is Windows Server 2008 SP2 enterprise. all of the Server roles are on SP3 with no additional Release update.

You can, but if my memory served me correctly that may not resolve the issue.

I would recommend updating the Exchange 2007 machines to at least SP3UR1 as there are several backup related issues fixed in that version.

http://support.microsoft.com/kb/2279665

Also, from a MSFT supportability standpoint you are in an unsupportable state: http://support.microsoft.com/lifecycle/default.aspx?LN=en-us&x=12&y=13&c2=730

EOL for Exchange 2007 SP3 was up in July of 2011. I would recommend updating to at least Exchange 2007 SP3UR11

Download link: http://support.microsoft.com/kb/2873746
Exchange Roll Up History: http://social.technet.microsoft.com/wiki/contents/articles/240.exchange-server-and-update-rollups-build-numbers.aspx

Sorry I didn't catch that sooner, but I assumed that you were somewhat patched. If the case is you are at 2007 SP3 RTM, I would just download UR11 for SP3 (you dont need to download each update roll-up) and install that, starting with the passive node and then the active node. Fail back and forth as needed.

Ok but in this case I cannot just apply the UR11 to the mailbox server only ?

In the mean time what can I do to stop the log file growth ?

OK, when patching the mailbox server. Do I have to suspend the CCR replication before rebooting the server from the passive node ?

I am worried about losing the email flow when I failover the mailbox server especially when patching the primary production CCR mailbox.