Link to home
Start Free TrialLog in
Avatar of arkr12
arkr12Flag for Australia

asked on

Integrity check on Exchange databases freezes the database itself and users can't connect to Exchange

Hi,

We use ARCserve backup 16 sp1 and I tried taking full backup of Exchange server 2007 databases, when the backup job started at 9.30pm it was doing integrity check, at the same time exchange databases went freezing, Outlook, OWA users unable to connect to Exchange server. When I checked with ARCserve support they say it's normal.

Is there any solution to this problem? I should be able to avoid downtime while taking backup
Avatar of Manpreet SIngh Khatra
Manpreet SIngh Khatra
Flag of India image

As its Exchange 2007 why dont you enable LCR or SCR and try to take a backup of those database files ........ is the Backup software upgraded with all recent updates ? Was it working earlier and when did the issue start happening ?

- Rancy
Exchange intergrity check by arcserve is fine but it should not freez the database.

Stop the backup for a day and observe if it is happening because of backup.

check for the event.
Avatar of arkr12

ASKER

I agree that I should configure clustering, but I need to have atleast a full backup immediately before I try out other things. I installed recently and trying to take a full backup after that.
But it tool long time for integrity check and also database went to freezing state.

Following are the errors logged on the Exchange server,

Log Name:      Application
Source:        VSS
Date:          26/03/2013 9:32:50 PM
Event ID:      8194
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr =

0x80070005. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer

Name: System Writer
   Writer Instance ID: {e8c10508-ef95-44ab-935d-052f34d01239}

Log Name:      Application
Source:        Microsoft-Windows-IIS-W3SVC-WP
Date:          26/03/2013 9:49:16 PM
Event ID:      2262
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
ISAPI 'C:\Windows\Microsoft.NET\Framework64\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the

following reason: 'Deadlock detected'.

Log Name:      Application
Source:        ESE
Date:          26/03/2013 9:49:54 PM
Event ID:      508
Task Category: Performance
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
MSExchangeIS (3452) Second Storage Group: A request to write to the file "D:\Exchange Server\Mailbox\Second

Storage Group\Logs\E01.log" at offset 4096 (0x0000000000001000) for 512 (0x00000200) bytes succeeded, but took

an abnormally long time (75 seconds) to be serviced by the OS. This problem is likely due to faulty hardware.

Please contact your hardware vendor for further assistance diagnosing the problem.

Log Name:      Application
Source:        MSExchangeMailSubmission
Date:          26/03/2013 10:11:46 PM
Event ID:      1009
Task Category: MSExchangeMailSubmission
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
The Microsoft Exchange Mail Submission Service is currently unable to contact any Hub Transport servers in the

local Active Directory site. The servers may be too busy to accept new connections at this time.

Log Name:      Application
Source:        MSExchangeSA
Date:          26/03/2013 10:15:39 PM
Event ID:      9176
Task Category: NSPI Proxy
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
NSPI Proxy can contact Global Catalog aiadc02.aiatsis.local but it does not support the NSPI service. After a

Domain Controller is promoted to a Global Catalog, the Global Catalog must be rebooted to support MAPI Clients.

 Reboot aiadc02.aiatsis.local as soon as possible.

Log Name:      Application
Source:        MSExchangeAL
Date:          26/03/2013 10:15:46 PM
Event ID:      8365
Task Category: Service Control
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Could not read the Security Descriptor from the Exchange Server object with

guid=E0AC4DD83D59344AAB76428FF1B48B41. As a result the Proxy Address Calculation RPC interface will not be

available on the local Exchange Server.  


Log Name:      Application
Source:        ASP.NET 2.0.50727.0
Date:          26/03/2013 10:16:02 PM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Event code: 3001
Event message: The request has been aborted.
Event time: 26/03/2013 10:16:02 PM
Event time

(UTC): 26/03/2013 11:16:02 AM
Event ID: 387a9cd4b22a4b218547e40f85d0088b
Event sequence: 2078236
Event

occurrence: 1
Event detail code: 0
 
Application information:
    Application domain: /LM/W3SVC/1/ROOT/EWS-1-

130074504441996592
    Trust level: Full
    Application Virtual Path: /EWS
    Application Path: C:\Program

Files\Microsoft\Exchange Server\ClientAccess\exchweb\EWS\
    Machine name: AIAMSG01
 
Process information:
   

Process ID: 3608
    Process name: w3wp.exe
    Account name: NT AUTHORITY\SYSTEM
 
Exception information:
   

Exception type: HttpException
    Exception message: Request timed out.
 
Request information:
    Request URL:

https://aiamsg01.aiatsis.local:443/EWS/Exchange.asmx 
    Request path: /EWS/Exchange.asmx
    User host

address: 10.0.0.241
    User: AIATSIS\arunk
    Is authenticated: True
    Authentication Type: Negotiate
   

Thread account name: NT AUTHORITY\SYSTEM
 
Thread information:
    Thread ID: 24
    Thread account name: NT

AUTHORITY\SYSTEM
    Is impersonating: False
    Stack trace:


Log Name:      Application
Source:        ESE
Date:          26/03/2013 10:16:06 PM
Event ID:      519
Task Category: Logging/Recovery
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
caagstart (1412) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Exchange

Server\Mailbox\First Storage Group\Logs\E00): The range of log files \\?\GLOBALROOT\Device

\HarddiskVolumeShadowCopy2\Exchange Server\Mailbox\First Storage Group\Logs\E000000BE29.log to \\?\GLOBALROOT

\Device\HarddiskVolumeShadowCopy2\Exchange Server\Mailbox\First Storage Group\Logs\E0000012CDF.log is missing

(error -528) and cannot be used. If these log files are required for recovery, a good copy of these log files

will be needed for recovery to complete successfully.

Log Name:      Application
Source:        Storage Group Consistency Check
Date:          26/03/2013 11:27:05 PM
Event ID:      402
Task Category: Termination
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Instance 1: Termination of the physical consistency check was prevented because the operation has either

already been performed or was not yet ready to be performed. The operation failed with error code -1059

(0xfffffbdd).

Log Name:      Application
Source:        VSS
Date:          26/03/2013 11:27:05 PM
Event ID:      8194
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr =

0x80070005. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer

Name: System Writer
   Writer Instance ID: {e8c10508-ef95-44ab-935d-052f34d01239}


Log Name:      Application
Source:        MSExchangeIS
Date:          26/03/2013 11:27:05 PM
Event ID:      9782
Task Category: Exchange VSS Writer
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Exchange VSS Writer (instance cd717955-57d1-4375-8fe6-50ecb5cdcad1:1) has unsuccessfully completed the backup

of storage group 'First Storage Group'. No log files have been truncated for this storage group.
Log Name:      Application
Source:        ESE
Date:          26/03/2013 11:27:05 PM
Event ID:      2007
Task Category: ShadowCopy
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Information Store (3452) Shadow copy instance 1 aborted.



Log Name:      Application
Source:        Microsoft-Windows-CertificateServicesClient-AutoEnrollment
Date:          26/03/2013 11:39:32 PM
Event ID:      64
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Certificate for local system with Thumbprint 75 f4 4e 94 11 42 a6 73 d7 05 d4 72 f2 9d c4 e3 b5 2e f2 e1 is

about to expire or already expired.
Is there some Spam filter and AV ? If AV does it has Exclusions set not to Scan any Exchange related Files\Folders ??

What all other 3rd parties do we have .....

- Rancy
Avatar of arkr12

ASKER

Hi Rancy,

Yes the spam filter 'TrendMicro' but its present external to our network and there is no antivirus installed on exchange server. No other 3rd parties that is installed other than the ARCserve backup agent
What is the size of the Database ? How many DB's do you have ?

- Rancy
Avatar of arkr12

ASKER

There are 3 storage groups,

1st SG - 1 database
2ndSG - 1 database
3rd SG - 3 databases

Total of 4 databases, the size of all databases in total is 312 gb.
I would have broken the Jobs for each database or SG just to see if some specific SG or Database is causing the issue due to some Corruption :)

- Rancy
Avatar of arkr12

ASKER

@ Rancy,

thats not a bad idea. If the backup job will fail, I may dismount the databases, check the status of them to make sure they are clean shutdown and will move all the log files and then try the backup.

Log files for couple of databases listed above are of 60 GB each.
Please do check and share your feedback with us

- Rancy
Avatar of arkr12

ASKER

Hi,

I have tried taking backup of databases one at a time, but only successful with public folder database. When I started the backup job on the single database in 1st storage group, exchange server crashed and I had to restart the virtual machine through VSphere console.  I am not sure the reason for the server crash.

I have also moved most of the log files for that particular database but still its the same result. Now I am trying Windows Server backup to take backup of D: drive which holds the exchange databases to a remote folder in an external hard disk. Windows Server Backup started at around 1.30 pm today and so far 61% completed.
What is the DB sizes ?
You have 4 mailbox databases right ?
Are all on different drives and did you try individual backup of other databases ?

- Rancy
Avatar of arkr12

ASKER

1st db - 45 GB
2nd db - 15 GB
3rd db - 70 GB
4th db - 100 GB
5th db - 85 GB

there are a total of 5 dbs and I have tried taking backup of individual databases, only pf database (2nd db) backed up successfully and other backups failed
Are they all on the same Harddrive or some SAN ? What is the error ? what is the status of the Vss writers before and after the backup ?

- Rancy
Avatar of arkr12

ASKER

They are all in the same LUN, same hard drive which is part of Dell poweredge 710 server. You can see the errors in earlier post. VSS writer status is stable before the backup but after the backup fails, vss service goes to a stopped state
So all Database including PF are on the same LUN\SAN ..... PF backup is fine but Mailbox Backup fails .... hope we have enough space and our smallest DB is 15GB maybe with some white space

Can we create a new DB under that same SG and move Mailboxes from "2nd DB" and then take backup of the new DB

- Rancy
Avatar of arkr12

ASKER

Rancy,

The smallest DB is 45 GB and the Exchange databases are not on the SAN but on the local storage. Virtual hard disk files will be moved to SAN soon. Available free space on the data store that has exchange virtual hard disk has 70 GB free space.  There are few snapshot files for exchange virtual machine and exchange in running on one of those snapshot files. We will be consolidating all the snapshot files and will take backup of those databases.

I will update you.
Sure please do share your feedback

- Rancy
Avatar of arkr12

ASKER

Hi all,

Consolidation of Exchange server 2007 snapshot files has been completed and full backup (database level backup) of exchange databases was taken and it completed successfully. However the integrity check took a long time, the database dint freeze but the mail flow was slow.
Yes if its taking time it could affect the DB ... i would suggest check for Updates to master and ClientAgents as that might help

- Rancy
Avatar of arkr12

ASKER

@ Rancy,

I will check and get back to you.
ASKER CERTIFIED SOLUTION
Avatar of arkr12
arkr12
Flag of Australia image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of arkr12

ASKER

Exchange VM Moved to SAN