Solved

Exchange Server Full Backup successfully executed on passive CCR node but not truncating Transaction Logs ?

Posted on 2013-12-03
12
1,310 Views
Last Modified: 2013-12-05
Hi People,

I got Exchange Server 2007 CCR mailbox node as follows:

CCR-MBX-Prod1 and CCR-MBX-DR1 nodes running as normal.

My current problem is that even after successful backup using EMC Avamar from the passive CCR node, the Exchange Transaction logs is not prunned or truncated successfully ?

By using this script:

Get-StorageGroupCopyStatus -Server ExMbxCluster1| Select Identity, CCRTargetNode, SummaryCopyStatus, CopyQueueLength, ReplayQueueLength, Suspend, Seeding, LatestFullBackupTime, ServiceDown, FailedMessage | ft -AutoSize

Open in new window


I have checked that the CCR replication is working without any issue so far with zero CopyQueueLength and ReplayQueueLength.

The only thing that I can see not right is that the LatestFullBackupTime still shown as 4 months ago instead of last night.

Can anyone please assist me with what should I do here ?

Thanks.
0
Comment
  • 6
  • 6
12 Comments
 
LVL 19

Expert Comment

by:Adam Farage
Comment Utility
Two questions..

1) Are you using the plugin for Exchange backups on the EMC Avamar product, and are you doing an Exchange aware backup? (VSS enabled)

2) If you run VSSADMIN LIST WRITERS do you see any errors on the passive node?

Last thing, what does the avamar backup show you in the logs?

It sounds like you are doing a flat backup (file level) and not an Exchange backup. If you do not run a backup utilizing an exchange aware, which utilizes the VSS snapshot utilities on the OS then it will not truncate your files. It could also be (and I have seen this a few times in the past) a database that is not flipping the correct switches.
0
 
LVL 7

Author Comment

by:Senior IT System Engineer
Comment Utility
1. Yes I am doing it using the Exchange VSS aware backup.
2. No error in all of the VSS writers except one, System Writer got Last error: Non-retryable error.

"It could also be (and I have seen this a few times in the past) a database that is not flipping the correct switches."

What sort of switch is this do you mean ?
0
 
LVL 19

Expert Comment

by:Adam Farage
Comment Utility
So a thing I noticed in Exchange 2007 is that sometimes the attribute for the last backup date sometimes does not get updated correctly. Why, not sure.. but that occurred a few times in the past when I was working on various clients / accounts.

So you have a VSS Writer in a non-retryable error state, you can check the event viewer logs for this: http://technet.microsoft.com/en-us/library/bb218863%28EXCHG.80%29.aspx

I would try to reboot the server, and run the backup again. It sounds like something is not functioning either with the backup agent itself (you can try a reinstall if the reboot doesnt help) or the VSS writing agent itself (which I somewhat doubt).
0
 
LVL 7

Author Comment

by:Senior IT System Engineer
Comment Utility
Thanks for replying to my case,

however I just saw that there are many events listed in the event viewer such as the following Event ID:


Log Name:      Application
Source:        VSS
Date:          4/12/2013 5:39:23 AM
Event ID:      8194
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CCR-MBX-DR1.domain.com
Description:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr = 0x80070005. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Name: System Writer
   Writer Instance ID: {40815a03-d488-4477-b2aa-08ce2eb3b26c}

Operation:
   Gathering Writer Data

I have just granted the full access permission on the DCOM configuration as follows:

- Start | Run and type dcomcnfg and hit Enter key
- Expand Component Services
- Expand Computers | My Computer
- Right Click on My Computer and choose Properties
- Select COM Security
- Select Edit Default on Access Permission
- Add the Cluster Service Domain account with Local Access and Remote Access allowed
0
 
LVL 7

Author Comment

by:Senior IT System Engineer
Comment Utility
and also I have confirmed that from the Avamar console and the Windows Event viewer below the backup has finished successfully:


Log Name:      Application
Source:        MSExchangeRepl
Date:          4/12/2013 5:39:25 AM
Event ID:      2046
Task Category: Exchange VSS Writer
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      CCR-MBX-DR1.domain.com
Description:
The Microsoft Exchange Replication Service VSS writer (instance e2453332-2AF11-22cc-968d-aba64024b0dc) has successfully completed the backup of storage group 'SG-Office2'.  

Database log truncation has been requested for this storage group. Log truncation will occur on the active copy after the next log generation is created. Log truncation will occur automatically on the passive copy after that log file is copied.

But somehow the transaction log is not truncated in the active node.
0
 
LVL 19

Expert Comment

by:Adam Farage
Comment Utility
Man.. this is pretty weird.

Can you tell me which OS you are on? It sounds like something is wrong with the VSS writer(s) and the information does tell me that.

What OS, and what security patches / hotfixes are installed?

I was skimming through some of my notes and also found this. I would check it out and make sure (if you are running Server 2003, which is a great possibility with Exchange 2007) that this patch is in fact installed (on both nodes): http://support.microsoft.com/kb/940349
0
How does your email signature look on mobiles?

Do your employees use mobile devices to reply to emails? With mobile becoming increasingly important to the business world, it is in your best interest to make sure that your email signature looks great across all types of devices.

 
LVL 7

Author Comment

by:Senior IT System Engineer
Comment Utility
ok, I've just found the root cause from the KB article that it is confirmed bug:

http://support.microsoft.com/kb/2297394/en-us

So shall I perform the backup from the active node instead ? Before I plan the testing and implementation for Update Roll Up 11 (which is the latest). http://support.microsoft.com/kb/2873746

The server is Windows Server 2008 SP2 enterprise. all of the Server roles are on SP3 with no additional Release update.
0
 
LVL 19

Expert Comment

by:Adam Farage
Comment Utility
You can, but if my memory served me correctly that may not resolve the issue.

I would recommend updating the Exchange 2007 machines to at least SP3UR1 as there are several backup related issues fixed in that version.

http://support.microsoft.com/kb/2279665

Also, from a MSFT supportability standpoint you are in an unsupportable state: http://support.microsoft.com/lifecycle/default.aspx?LN=en-us&x=12&y=13&c2=730

EOL for Exchange 2007 SP3 was up in July of 2011. I would recommend updating to at least Exchange 2007 SP3UR11

Download link: http://support.microsoft.com/kb/2873746
Exchange Roll Up History: http://social.technet.microsoft.com/wiki/contents/articles/240.exchange-server-and-update-rollups-build-numbers.aspx

Sorry I didn't catch that sooner, but I assumed that you were somewhat patched. If the case is you are at 2007 SP3 RTM, I would just download UR11 for SP3 (you dont need to download each update roll-up) and install that, starting with the passive node and then the active node. Fail back and forth as needed.
0
 
LVL 7

Author Comment

by:Senior IT System Engineer
Comment Utility
Ok but in this case I cannot just apply the UR11 to the mailbox server only ?

In the mean time what can I do to stop the log file growth ?
0
 
LVL 19

Accepted Solution

by:
Adam Farage earned 500 total points
Comment Utility
Ok but in this case I cannot just apply the UR11 to the mailbox server only ?

Apply it to all Exchange 2007 servers in the organization. Always keep the OS patch level AND the Exchange patch level the same across all Exchange servers, and the OS patch level (if possible) the same across the DC / GCS in the organization. The DC / GCS is recommended, but the patch level for the Exchange servers is required.

As for your log growth, I would highly recommend updating ASAP to SP3UR11 and then running a full backup. Your other solution is to grab a chunk of the older log files and move them elsewhere. Be aware that a differential backup will not function as intended unless all the log files are present (as the checkpoint file during the backup check will freak out and not truncate anything) but you can run a full backup AFTER the SP3UR11 install.

Update it, that is all I can really say.. its a bug in SP3 RTM.
0
 
LVL 7

Author Comment

by:Senior IT System Engineer
Comment Utility
OK, when patching the mailbox server. Do I have to suspend the CCR replication before rebooting the server from the passive node ?

I am worried about losing the email flow when I failover the mailbox server especially when patching the primary production CCR mailbox.
0
 
LVL 19

Assisted Solution

by:Adam Farage
Adam Farage earned 500 total points
Comment Utility
Installing a UR in Exchange 2007 CCR is a bit, interesting. I would rather point you to this document as I mainly do 2003-2013 migration, and 2010 / 2013 support these days (with some 2007 and O365)

http://technet.microsoft.com/en-us/library/bb885047.aspx
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

Disabling the Directory Sync Service Account in Office 365 will stop directory synchronization from working.
Follow this checklist to learn more about the 15 things you should never include in an email signature from personal quotes, animated gifs and out-of-date marketing content.
This tutorial will walk an individual through the process of configuring their Windows Server 2012 domain controller to synchronize its time with a trusted, external resource. Use Google, Bing, or other preferred search engine to locate trusted NTP …
how to add IIS SMTP to handle application/Scanner relays into office 365.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now