Solved

Exchange 2007 - Transaction Logs not truncated on one storage group CCR Cluster

Posted on 2010-08-17
19
3,807 Views
Last Modified: 2013-12-01
We have an issue where logs are not being flushed on one storage group.  The other three are fine.
BEXEC 12.5 appears to back this up as a corrupt file but fails with;

Backup- \\ICM-EXCHANGE\Microsoft Information Store\Fourth Storage Group V-79-57344-65247 - A failure occurred reading an object.

and

Backup- \\ICM-EXCHANGE\Microsoft Information Store\Fourth Storage Group WARNING: "\\ICM-EXCHANGE\Microsoft Information Store\Fourth Storage Group\Logs" is a corrupt file.
This file cannot verify.

Event Viewer gives the following;

Log Name:      Application
Source:        MSExchangeIS
Date:          16/08/2010 20:25:28
Event ID:      9782
Task Category: Exchange VSS Writer
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ICM-MBX-NODE1.ICM.local
Description:
Exchange VSS Writer (instance e08f6985-a0a3-4721-9fca-a660b74b5316:4) has unsuccessfully completed the backup of storage group 'Fourth Storage Group'. No log files have been truncated for this storage group.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchangeIS" />
    <EventID Qualifiers="49193">9782</EventID>
    <Level>2</Level>
    <Task>41</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2010-08-16T19:25:28.000Z" />
    <EventRecordID>159278</EventRecordID>
    <Channel>Application</Channel>
    <Computer>ICM-MBX-NODE1.ICM.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>e08f6985-a0a3-4721-9fca-a660b74b5316:4</Data>
    <Data>Fourth Storage Group</Data>
    <Binary>46756C6C00</Binary>
  </EventData>
</Event>

and

Log Name:      Application
Source:        ESE
Date:          16/08/2010 20:25:28
Event ID:      2007
Task Category: ShadowCopy
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ICM-MBX-NODE1.ICM.local
Description:
Information Store (3716) Shadow copy instance 4 aborted.

For more information, click http://www.microsoft.com/contentredirect.asp.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ESE" />
    <EventID Qualifiers="0">2007</EventID>
    <Level>2</Level>
    <Task>16</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2010-08-16T19:25:28.000Z" />
    <EventRecordID>159279</EventRecordID>
    <Channel>Application</Channel>
    <Computer>ICM-MBX-NODE1.ICM.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Information Store</Data>
    <Data>3716</Data>
    <Data>
    </Data>
    <Data>4</Data>
  </EventData>
</Event>

The servers are running Windows 2008 x64 and Exchange was recently updated to SP2 following the correct method for CCR clusters.

I have sctratched around and found some referances to issue with the Exchange backup extension not being registered correctly (MS 979227) but when I follow the script I get errors relating to the snapin not being found.

My question, apart from how to fix this is will SP3 for Exchange fix this?  Will the issue affect only one storage group or am I going the wrong direction in thinking it maybe an SP issue?  If the script WILL sort the issue, any ideas on how to get it to work?

I notice that the versions of WSBExhange differ between the servers but the backup fails on both nodes.  BEXEC is set to only use the active node for backup.

Here is the red output of the script for referance.




[PS] C:\Program Files\Microsoft\Exchange Server>Register-WSBExchange.ps1 -regist
er
Add-PSSnapin : Windows PowerShell snap-in Microsoft.Exchange.Management.PowerSh
ell.Setup is not installed on the machine.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
171 char:15
+   Add-PSSnapin  <<<< "Microsoft.Exchange.Management.PowerShell.Setup"
Now installing WSBExchange...
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
29 char:30
+               Start-SetupProcess  <<<< -Name:"$servicecmd" -Args:"delete wsbe
xchange";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
32 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKCR\CLSID\{D8A2E3
12-3B17-4293-B71E-CD72A7C04BF3}`" /t REG_SZ /d  `"CExchangeHelper Class`" /f";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
33 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKCR\CLSID\{D8A2E3
12-3B17-4293-B71E-CD72A7C04BF3}`" /v AppId /t REG_SZ /d `"{D8A2E312-3B17-4293-B
71E-CD72A7C04BF3}`" /f";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
34 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKCR\CLSID\{D8A2E3
12-3B17-4293-B71E-CD72A7C04BF3}\LocalServer32`" /t REG_SZ /d `"$WsbBinPath`" /f
";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
35 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKCR\APPID\{D8A2E3
12-3B17-4293-B71E-CD72A7C04BF3}`" /t REG_SZ /d `"CExchangeHelper Class`" /f";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
36 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKCR\APPID\{D8A2E3
12-3B17-4293-B71E-CD72A7C04BF3}`" /v LocalService /t REG_SZ /d `"wsbexchange`"
/f";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
37 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKCR\APPID\{D8A2E3
12-3B17-4293-B71E-CD72A7C04BF3}`" /v LaunchPermission /t REG_BINARY /d `"010004
806000000070000000000000001400000002004c0003000000000014001f0000000101000000000
00512000000000018001f0000000102000000000005200000002002000000001800030000000102
0000000000052000000027020000010200000000000520000000200200000102000000000005200
0000020020000`" /f";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
38 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKCR\APPID\wsbexch
ange.exe`" /v AppId /t REG_SZ /d `"{D8A2E312-3B17-4293-B71E-CD72A7C04BF3}`" /f"
;
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
39 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKLM\Software\Micr
osoft\windows nt\currentversion\WindowsServerBackup\Application Support\{76fe1a
c4-15f7-4bcd-987e-8e1acb462fb7}`"  /v `"Application Identifier`" /t REG_SZ /d E
xchange /f";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
40 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKLM\Software\Micr
osoft\windows nt\currentversion\WindowsServerBackup\Application Support\{76fe1a
c4-15f7-4bcd-987e-8e1acb462fb7}`" /v CLSID /t REG_SZ /d `"{D8A2E312-3B17-4293-B
71E-CD72A7C04BF3}`" /f";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
41 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKLM\Software\Micr
osoft\windows nt\currentversion\WSBAppExchangeHelper`" /v AutoMarkDbRecoverable
 /t REG_DWORD /d 1 /f";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
42 char:24
+         Start-SetupProcess  <<<< -Name:"$reg" -Args:"add `"HKLM\Software\Micr
osoft\windows nt\currentversion\WSBAppExchangeHelper`" /v AutoMountOnPITRecover
y /t REG_DWORD /d 1 /f";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
44 char:24
+         Start-SetupProcess  <<<< -Name:"$servicecmd" -Args:"create wsbexchang
e binpath= `"$WsbBinPath`" type= own start= demand error= ignore obj= LocalSyst
em DisplayName= `"Microsoft Exchange Server Extension for Windows Server Backup
`"";
The term 'Start-SetupProcess' is not recognized as a cmdlet, function, operable
 program, or script file. Verify the term and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
45 char:24
+         Start-SetupProcess  <<<< -Name:"$servicecmd" -Args:"description wsbex
change `"Enables Windows Server Backup users to back up and recover application
 data for Microsoft Exchange Server.`"";
Remove-PSSnapin : Cannot remove the Windows PowerShell snap-in Microsoft.Exchan
ge.Management.PowerShell.Setup because it is not loaded. Verify the name of the
 snap-in being removed and try again.
At C:\Program Files\Microsoft\Exchange Server\Scripts\Register-WSBExchange.ps1:
185 char:18
+   Remove-PSSnapin  <<<< "Microsoft.Exchange.Management.PowerShell.Setup"
0
Comment
Question by:abssupport
  • 9
  • 5
  • 4
  • +1
19 Comments
 
LVL 10

Expert Comment

by:dhruvarajp
ID: 33453368
check the log truncation part of the deep dive below
http://technet.microsoft.com/en-us/library/cc535020(EXCHG.80).aspx

and also ensure the redundant machine(passive ) for ccr is online


Thank you
Dhruv
0
 
LVL 7

Expert Comment

by:Illusionist
ID: 33453570
It seems that a log file is corrupt and your replication of  the Fourth Storage Group has broken.
you need to check the replication status of the Fourth Storage Group.
If Replication is broken, VSS backup will not purge the Log Files.
SP will not break replication.
Now the workaround this will be reseed the Database from the Fourth Storage Group.
How Big is the database. If it is large enough then you can schedule a downtime. Dismount the database and copy it to the Passive Node . Delete all the Log Files from the Active and Passive node, and then Mount the database from the active Node and Replication will start again.
0
 

Author Comment

by:abssupport
ID: 33454001
Thanks for your quick response guys.

Illusionist, I would be inclined to agree over the corrupt log file.  However, replication is showing as healthy and I have rolled over from the previous active to the passive node with no errors and this still shows as healthy.

Any other ideas in light of this info?
0
 
LVL 7

Expert Comment

by:Illusionist
ID: 33454175
The possible cause for the Log Files not being truncated while doing a VSS backup is a Corrupt Log File.
Did you take a backup after you rolled over the active to passive and then check the Log Files?
0
 
LVL 10

Expert Comment

by:dhruvarajp
ID: 33454435
if all log files at active node have been shipped and replayed into passive i would say, log file is not currupt. instead vss writer issue
0
 
LVL 10

Expert Comment

by:dhruvarajp
ID: 33454442
EXBPA might point us towards possible issues
0
 

Author Comment

by:abssupport
ID: 33454461
Backups were taken by BEXEC on the previous active now passive node.
I also ran a Windows Server Backup on that volume on that node, no logs flushed.
I rolled over and ran a backup (BEXEC) on the now active (previously passive) node.

Log files remain.

Same result from all backups.
0
 

Author Comment

by:abssupport
ID: 33454631
dhruvarajp, I ran EXBPA and it reveals no issues about replication or database problems.  The only relevant altert is a backup warning on the problem database saying that it hasn't been backed up since 8/8/10.

The VSS writer issue is always a possible in these situations but it looks good.


Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {9a767e21-1394-4b59-a788-7da43b44fd4d}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {d14876d1-a7a1-477b-8a1a-752550600852}
   State: [1] Stable
   Last error: No error

Writer name: 'Cluster Database'
   Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
   Writer Instance Id: {2dd1b009-b76a-4d12-a62e-b6a07f242ef3}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {b590f22d-9048-479f-9d64-6cdb12f9626b}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {4ca0bfca-1cdd-47ec-8aea-dd6c98b3e5d1}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {22dcb5b5-8b07-455f-84f5-4bc33a7a5443}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {c890b7a5-67ee-4eb0-9d29-fed3dab2fc78}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {74f05bec-e0ab-4ac4-9c66-ac3fc2c6b31b}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {9f79bd66-2efc-427c-b7f8-eb3b4711a3c3}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {236d01e7-737c-434d-ba83-abf534566bce}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {175e36db-2fd1-45bc-805c-b660ca254990}
   State: [1] Stable
   Last error: No error
0
 
LVL 10

Expert Comment

by:dhruvarajp
ID: 33455185
try best tool discussed at the end of this article
http://technet.microsoft.com/en-us/library/dd233256(EXCHG.80).aspx

dhruv
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 7

Expert Comment

by:Illusionist
ID: 33458342
From the Exchange Management Shell run the Command
Test-Replicationhealth and check if there is any error reported in it.
If listed paste it in here
0
 

Author Comment

by:abssupport
ID: 33462855
Illusionist, here you go, results of Test-Replicationhealth

Server          Check                      Result     Error
------          -----                      ------     -----
ICM-MBX-NODE2   PassiveNodeUp              Passed
ICM-MBX-NODE2   ClusterNetwork             Passed
ICM-MBX-NODE2   QuorumGroup                Passed
ICM-MBX-NODE2   FileShareQuorum            Passed
ICM-MBX-NODE2   CmsGroup                   Passed
ICM-MBX-NODE2   NodePaused                 Passed
ICM-MBX-NODE2   DnsRegistrationStatus      Passed
ICM-MBX-NODE2   ReplayService              Passed
ICM-MBX-NODE2   DBMountedFailover          Passed
ICM-MBX-NODE2   SGCopySuspended            Passed
ICM-MBX-NODE2   SGCopyFailed               Passed
ICM-MBX-NODE2   SGInitializing             Passed
ICM-MBX-NODE2   SGCopyQueueLength          Passed
ICM-MBX-NODE2   SGReplayQueueLength        Passed

Looks rosy.
0
 

Author Comment

by:abssupport
ID: 33463203
I have tried to run BETEST but it wont output to a text file.  When running BETEST without the txt the output is too great to take from the command window.  But from what I CAN see there are not failiures\errors.
0
 

Author Comment

by:abssupport
ID: 33463255
Have since found out that BETEST only runs on 2003 server (or at least the version in the VSSDK); doesn't seem to be one for 2008 server.
0
 
LVL 7

Accepted Solution

by:
Illusionist earned 125 total points
ID: 33463583
Betest Tool is used to check if the VSS writer is working correctly and is run to isolate the issue, if it is Database or if its the writer which is causing the issue.
As you are able to backup the other storage groups successfully, its not the VSS writer issue.

From the Replication log it looks clean. Now as the logs are not truncating for a particular storage group, then as i said earlier, the workaround will be dismount the database, delete the log files from both active and passive and then reseed the database, if the database is big enough then do a flatfile copy to passive node and then resume the replication and after a few files are generated and replicated, take a backup and let us know.
0
 
LVL 10

Assisted Solution

by:dhruvarajp
dhruvarajp earned 125 total points
ID: 33463675
well. i would say you did find relevant mcirosfot article and that script did not work, i did look at the script that should have worked,
you can look at the eseuti /mk on checkpoint and move all logs older than the WAYPOINT. to another directory..rather thatn dismounting and inviting
downtime
eseuti /mk against .chk file would not need dismount

however.. i would encourage you to open a tickcket with PSS and check why that script is not running for you

Thank you
Dhruv
0
 

Author Comment

by:abssupport
ID: 33463779
Thanks, will do some more and get back to you tomorrow.  Thanks so far!
0
 
LVL 3

Expert Comment

by:tussharawaghan
ID: 33480988
Update Exchange to SP3. And use eseutil to get the database in consistent state (Clean Shutdown).
Then delete all the log files and mount the database again. Also do a offline defrag of the database if possible.
0
 

Author Comment

by:abssupport
ID: 33519509
Hello, Back again appologies for the delay I have been away.

I checked the the database with eseutil /MH to check the waypoint.  According to this all logs had been commited.  I then moved the logs as suggested out of the log file area except E04 and the most recent one.
The database mounted ok, but after reseeding the replication failed sugesting that it had a log file missing.

I ran a backup and YES!, log files are clearing.  After the backup I attempted another reseeding and now all is fine so thanks.  I will divvy the points between Dhruv and Illusionist.
0
 

Author Closing Comment

by:abssupport
ID: 33519521
The lack of errors made this difficult to trouble shoot.
0

Featured Post

Want to promote your upcoming event?

Attending an event? Speaking at a conference? Or exhibiting at a tradeshow? Easily inform your contacts by using a promotional banner in your email signature. This will ensure your organization’s most important contacts are in the know.

Join & Write a Comment

Resolve Outlook connectivity issues after moving mailbox to new Exchange 2016 server
Scam emails are a huge burden for many businesses. Spotting one is not always easy. Follow our tips to identify if an email you receive is a scam.
To show how to generate a certificate request in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.:  First we need to log into the Exchange Admin Center. Navigate to the Servers >> Certificates…
how to add IIS SMTP to handle application/Scanner relays into office 365.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now