Avatar of arkr12
arkr12
Flag for Australia asked on

Integrity check on Exchange databases freezes the database itself and users can't connect to Exchange

Hi,

We use ARCserve backup 16 sp1 and I tried taking full backup of Exchange server 2007 databases, when the backup job started at 9.30pm it was doing integrity check, at the same time exchange databases went freezing, Outlook, OWA users unable to connect to Exchange server. When I checked with ARCserve support they say it's normal.

Is there any solution to this problem? I should be able to avoid downtime while taking backup
Microsoft Server AppsExchangeServer Software

Avatar of undefined
Last Comment
arkr12

8/22/2022 - Mon
Manpreet SIngh Khatra

As its Exchange 2007 why dont you enable LCR or SCR and try to take a backup of those database files ........ is the Backup software upgraded with all recent updates ? Was it working earlier and when did the issue start happening ?

- Rancy
R--R

Exchange intergrity check by arcserve is fine but it should not freez the database.

Stop the backup for a day and observe if it is happening because of backup.

check for the event.
arkr12

ASKER
I agree that I should configure clustering, but I need to have atleast a full backup immediately before I try out other things. I installed recently and trying to take a full backup after that.
But it tool long time for integrity check and also database went to freezing state.

Following are the errors logged on the Exchange server,

Log Name:      Application
Source:        VSS
Date:          26/03/2013 9:32:50 PM
Event ID:      8194
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr =

0x80070005. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer

Name: System Writer
   Writer Instance ID: {e8c10508-ef95-44ab-935d-052f34d01239}

Log Name:      Application
Source:        Microsoft-Windows-IIS-W3SVC-WP
Date:          26/03/2013 9:49:16 PM
Event ID:      2262
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
ISAPI 'C:\Windows\Microsoft.NET\Framework64\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the

following reason: 'Deadlock detected'.

Log Name:      Application
Source:        ESE
Date:          26/03/2013 9:49:54 PM
Event ID:      508
Task Category: Performance
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
MSExchangeIS (3452) Second Storage Group: A request to write to the file "D:\Exchange Server\Mailbox\Second

Storage Group\Logs\E01.log" at offset 4096 (0x0000000000001000) for 512 (0x00000200) bytes succeeded, but took

an abnormally long time (75 seconds) to be serviced by the OS. This problem is likely due to faulty hardware.

Please contact your hardware vendor for further assistance diagnosing the problem.

Log Name:      Application
Source:        MSExchangeMailSubmission
Date:          26/03/2013 10:11:46 PM
Event ID:      1009
Task Category: MSExchangeMailSubmission
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
The Microsoft Exchange Mail Submission Service is currently unable to contact any Hub Transport servers in the

local Active Directory site. The servers may be too busy to accept new connections at this time.

Log Name:      Application
Source:        MSExchangeSA
Date:          26/03/2013 10:15:39 PM
Event ID:      9176
Task Category: NSPI Proxy
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
NSPI Proxy can contact Global Catalog aiadc02.aiatsis.local but it does not support the NSPI service. After a

Domain Controller is promoted to a Global Catalog, the Global Catalog must be rebooted to support MAPI Clients.

 Reboot aiadc02.aiatsis.local as soon as possible.

Log Name:      Application
Source:        MSExchangeAL
Date:          26/03/2013 10:15:46 PM
Event ID:      8365
Task Category: Service Control
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Could not read the Security Descriptor from the Exchange Server object with

guid=E0AC4DD83D59344AAB76428FF1B48B41. As a result the Proxy Address Calculation RPC interface will not be

available on the local Exchange Server.  


Log Name:      Application
Source:        ASP.NET 2.0.50727.0
Date:          26/03/2013 10:16:02 PM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Event code: 3001
Event message: The request has been aborted.
Event time: 26/03/2013 10:16:02 PM
Event time

(UTC): 26/03/2013 11:16:02 AM
Event ID: 387a9cd4b22a4b218547e40f85d0088b
Event sequence: 2078236
Event

occurrence: 1
Event detail code: 0
 
Application information:
    Application domain: /LM/W3SVC/1/ROOT/EWS-1-

130074504441996592
    Trust level: Full
    Application Virtual Path: /EWS
    Application Path: C:\Program

Files\Microsoft\Exchange Server\ClientAccess\exchweb\EWS\
    Machine name: AIAMSG01
 
Process information:
   

Process ID: 3608
    Process name: w3wp.exe
    Account name: NT AUTHORITY\SYSTEM
 
Exception information:
   

Exception type: HttpException
    Exception message: Request timed out.
 
Request information:
    Request URL:

https://aiamsg01.aiatsis.local:443/EWS/Exchange.asmx 
    Request path: /EWS/Exchange.asmx
    User host

address: 10.0.0.241
    User: AIATSIS\arunk
    Is authenticated: True
    Authentication Type: Negotiate
   

Thread account name: NT AUTHORITY\SYSTEM
 
Thread information:
    Thread ID: 24
    Thread account name: NT

AUTHORITY\SYSTEM
    Is impersonating: False
    Stack trace:


Log Name:      Application
Source:        ESE
Date:          26/03/2013 10:16:06 PM
Event ID:      519
Task Category: Logging/Recovery
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
caagstart (1412) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Exchange

Server\Mailbox\First Storage Group\Logs\E00): The range of log files \\?\GLOBALROOT\Device

\HarddiskVolumeShadowCopy2\Exchange Server\Mailbox\First Storage Group\Logs\E000000BE29.log to \\?\GLOBALROOT

\Device\HarddiskVolumeShadowCopy2\Exchange Server\Mailbox\First Storage Group\Logs\E0000012CDF.log is missing

(error -528) and cannot be used. If these log files are required for recovery, a good copy of these log files

will be needed for recovery to complete successfully.

Log Name:      Application
Source:        Storage Group Consistency Check
Date:          26/03/2013 11:27:05 PM
Event ID:      402
Task Category: Termination
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Instance 1: Termination of the physical consistency check was prevented because the operation has either

already been performed or was not yet ready to be performed. The operation failed with error code -1059

(0xfffffbdd).

Log Name:      Application
Source:        VSS
Date:          26/03/2013 11:27:05 PM
Event ID:      8194
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface.  hr =

0x80070005. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:
   Gathering Writer Data

Context:
   Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer

Name: System Writer
   Writer Instance ID: {e8c10508-ef95-44ab-935d-052f34d01239}


Log Name:      Application
Source:        MSExchangeIS
Date:          26/03/2013 11:27:05 PM
Event ID:      9782
Task Category: Exchange VSS Writer
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Exchange VSS Writer (instance cd717955-57d1-4375-8fe6-50ecb5cdcad1:1) has unsuccessfully completed the backup

of storage group 'First Storage Group'. No log files have been truncated for this storage group.
Log Name:      Application
Source:        ESE
Date:          26/03/2013 11:27:05 PM
Event ID:      2007
Task Category: ShadowCopy
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Information Store (3452) Shadow copy instance 1 aborted.



Log Name:      Application
Source:        Microsoft-Windows-CertificateServicesClient-AutoEnrollment
Date:          26/03/2013 11:39:32 PM
Event ID:      64
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      AIAMSG01.aiatsis.local
Description:
Certificate for local system with Thumbprint 75 f4 4e 94 11 42 a6 73 d7 05 d4 72 f2 9d c4 e3 b5 2e f2 e1 is

about to expire or already expired.
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
Manpreet SIngh Khatra

Is there some Spam filter and AV ? If AV does it has Exclusions set not to Scan any Exchange related Files\Folders ??

What all other 3rd parties do we have .....

- Rancy
arkr12

ASKER
Hi Rancy,

Yes the spam filter 'TrendMicro' but its present external to our network and there is no antivirus installed on exchange server. No other 3rd parties that is installed other than the ARCserve backup agent
Manpreet SIngh Khatra

What is the size of the Database ? How many DB's do you have ?

- Rancy
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
arkr12

ASKER
There are 3 storage groups,

1st SG - 1 database
2ndSG - 1 database
3rd SG - 3 databases

Total of 4 databases, the size of all databases in total is 312 gb.
Manpreet SIngh Khatra

I would have broken the Jobs for each database or SG just to see if some specific SG or Database is causing the issue due to some Corruption :)

- Rancy
arkr12

ASKER
@ Rancy,

thats not a bad idea. If the backup job will fail, I may dismount the databases, check the status of them to make sure they are clean shutdown and will move all the log files and then try the backup.

Log files for couple of databases listed above are of 60 GB each.
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
Manpreet SIngh Khatra

Please do check and share your feedback with us

- Rancy
arkr12

ASKER
Hi,

I have tried taking backup of databases one at a time, but only successful with public folder database. When I started the backup job on the single database in 1st storage group, exchange server crashed and I had to restart the virtual machine through VSphere console.  I am not sure the reason for the server crash.

I have also moved most of the log files for that particular database but still its the same result. Now I am trying Windows Server backup to take backup of D: drive which holds the exchange databases to a remote folder in an external hard disk. Windows Server Backup started at around 1.30 pm today and so far 61% completed.
Manpreet SIngh Khatra

What is the DB sizes ?
You have 4 mailbox databases right ?
Are all on different drives and did you try individual backup of other databases ?

- Rancy
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
arkr12

ASKER
1st db - 45 GB
2nd db - 15 GB
3rd db - 70 GB
4th db - 100 GB
5th db - 85 GB

there are a total of 5 dbs and I have tried taking backup of individual databases, only pf database (2nd db) backed up successfully and other backups failed
Manpreet SIngh Khatra

Are they all on the same Harddrive or some SAN ? What is the error ? what is the status of the Vss writers before and after the backup ?

- Rancy
arkr12

ASKER
They are all in the same LUN, same hard drive which is part of Dell poweredge 710 server. You can see the errors in earlier post. VSS writer status is stable before the backup but after the backup fails, vss service goes to a stopped state
Your help has saved me hundreds of hours of internet surfing.
fblack61
Manpreet SIngh Khatra

So all Database including PF are on the same LUN\SAN ..... PF backup is fine but Mailbox Backup fails .... hope we have enough space and our smallest DB is 15GB maybe with some white space

Can we create a new DB under that same SG and move Mailboxes from "2nd DB" and then take backup of the new DB

- Rancy
arkr12

ASKER
Rancy,

The smallest DB is 45 GB and the Exchange databases are not on the SAN but on the local storage. Virtual hard disk files will be moved to SAN soon. Available free space on the data store that has exchange virtual hard disk has 70 GB free space.  There are few snapshot files for exchange virtual machine and exchange in running on one of those snapshot files. We will be consolidating all the snapshot files and will take backup of those databases.

I will update you.
Manpreet SIngh Khatra

Sure please do share your feedback

- Rancy
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
arkr12

ASKER
Hi all,

Consolidation of Exchange server 2007 snapshot files has been completed and full backup (database level backup) of exchange databases was taken and it completed successfully. However the integrity check took a long time, the database dint freeze but the mail flow was slow.
Manpreet SIngh Khatra

Yes if its taking time it could affect the DB ... i would suggest check for Updates to master and ClientAgents as that might help

- Rancy
arkr12

ASKER
@ Rancy,

I will check and get back to you.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
ASKER CERTIFIED SOLUTION
arkr12

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
arkr12

ASKER
Exchange VM Moved to SAN