Commvault backup of Exchange 2010 mailbox database hangs with no error

Infrastructure:
VMwAre 5.5
6 host VMware cluster
Backup Server: Windows server 2012 r2 on VMware with Commvault Simpana 9.0.0 R2 plus all updates.
Exchange 2010 Mailbox Store on Windows Server 2008 r2 Enterprise on VMware.
There are a lot of datastores and 6 allocated to the Exchange server however none of these show any alerts.
Backup is to Quantum i40.

All other servers backup fine however the Exchange incremental backup with granular restore enabled passes all initial start tasks but when the actual backup copy starts it hangs until cancelled. We would check with Commvault support however the support contract has lapsed. ID we cancel all servers backup and just run this one backup, it just hangs.

The last full backup took 4 days. Mailbox Stores are 1.25TB.
It manages a full backup occasionally.

The backup jobs does not show any errors or reasons for delay in the backup monitor on the console.

On the Exchange server under windows event log:
System: The on-access driver was unable to create an impersonation token for file ........
Exchange: Cmdlet failed. Cmdlet Get-OrganizationConfig, parameters {}.
Exchange: (PID 2444, Thread 28) Task Get-Message throwing unhandled exception: System.Management.Automation.PipelineStoppedException: The pipeline has been stopped.
Exchange: Cmdlet failed. Cmdlet Add-ADPermission, parameters {User=domain\exbackup, ExtendedRights={Receive-As}, Identity=Mailbox Store 1}.
Exchange: Cmdlet failed. Cmdlet Move-DatabasePath, parameters {Identity=Archive Mailbox, EdbFilePath=D:\Program Files\Microsoft\Exchange Server\V14\Mailbox\Archive Mailbox\Archive Mailbox.edb, LogFolderPath=D:\Program Files\Microsoft\Exchange Server\V14\Mailbox\Archive Mailbox}.

Commvault log on Exchange Server:
* Module     : exTiDbBackup  

7808 13b8 04/21 23:46:55 55803 [ResetForceFull]: Working on 'Microsoft Information Store\Mailbox Store 2' ...
7808 13b8 04/21 23:46:55 55803 [ResetForceFull]: Pushing 'Microsoft Information Store\Public Folders+0' into backup vector...
7808 13b8 04/21 23:46:55 55803 [ResetForceFull]: Working on 'Microsoft Information Store\Public Folders' ...
7808 13b8 04/21 23:46:55 55803 ::Backup() - Closing pipeline.
7808 13b8 04/21 23:46:55 55803 CVArchive::ClosePipeline() - Closing Pipeline
7808 13b8 04/21 23:46:55 55803 CVArchive::ClosePipeline() - Reseting allocated stream
7808 13b8 04/21 23:46:56 55803 CCVAPipelayer::ClosePipeline() - About to destroy Data Mover
7808 13b8 04/21 23:46:56 55803 CCVAPipelayer::SendCommandToDSBackup() - Sent Command 133 to MediaAgent, Waiting for Response...
7808 13b8 04/21 23:46:56 ##### [PIPELAYER  ] Tail response = DataMover destroyed Status = 0 dwCode = 0 MsgType = 16 MsgSubType = 133 EventError = 0
7808 13b8 04/21 23:46:56 55803 CCVAPipelayer::SendCommandToDSBackup() - Sent Command 2 to MediaAgent, Waiting for Response...
7808 13b8 04/21 23:46:56 ##### [PIPELAYER  ] Tail response = Tail Generated STOP_PL and ready to Terminate Status = 0 dwCode = 0 MsgType = 4 MsgSubType = 2 EventError = 0
7808 13b8 04/21 23:46:56 55803 SdtBase::releaseRef: Going to delete the SdtBase object because the reference count has gone down to 0. RCId [61667]
7808 13b8 04/21 23:46:56 55803 SdtBase::() - SdtBase is being destroyed...
7808 13b8 04/21 23:46:56 55803 SdtNetLink::shutdown() - Performing graceful connection termination
7808 13b8 04/21 23:46:56 55803 SdtNetLink::closeSocket() - Closing socket connection 1600
7808 13b8 04/21 23:46:56 55803 ::Backup() - Sent PL_LAST to tail.
7808 13b8 04/21 23:46:56 55803 ::Backup() - Has completed in time [18421.000000].
7808 13b8 04/21 23:46:56 55803 [::cleanUp]: Notifying Job Manager of backup phase success.
7808 13b8 04/21 23:46:56 55803 [::cleanUp]: Phase successful! Updating backup reference time 1429605595 with Job Manager
7808 13b8 04/21 23:46:56 55803 BKP CALLED COMPLETE (PHASE Status::SUCCESS), 55803. Token [55803:4:4]
7808 13b8 04/21 23:46:57 55803 [::cleanUp]: back from notifying Job Manager of complete backup phase success.
7808 9ec  04/21 23:46:57 55803 CVJobCtrlLog::unregisterProcess(): successfuly removed file [C:\Program Files\CommVault\Simpana\Base\JobControl\7.808]
7808 13b8 04/21 23:46:57 55803 [::Backup]: Path = 'Microsoft Information Store\Public Folders'. Size = '925760 KB'.
7808 13b8 04/21 23:46:57 55803 [::Backup]: Path = 'Microsoft Information Store\Mailbox Store 2'. Size = '39460928 KB'.
7808 13b8 04/21 23:46:57 55803 [::Backup]: Path = 'Microsoft Information Store\Mailbox Store 1'. Size = '1257585216 KB'.
7808 13b8 04/21 23:46:57 55803 [::Backup]: Path = 'Microsoft Information Store\Archive Mailbox'. Size = '8266304 KB'.
7808 f4   04/21 23:46:57 55803 ** CVSession::sendBytes(...) iSocketMember=2620 send failed, err=[1][10093][Either the application has not called WSAStartup, or WSAStartup failed.].

7808 f4   04/21 23:46:57 55803 ** CVSession::sendMessageInt(...):
        - RemoteHost=clienthostnamexx.
        - RemoteProcess=AppMgrSvc.exe.
        - sendBytes() failed. Error=9000012.

7808 f4   04/21 23:46:57 55803 ** CVSession::disconnect (void)
- RemoteHost=clienthostnamexx.
        - RemoteProcess=AppMgrSvc.exe.
        - sendMessageInt CVS_PM_DISCONNECT_REQ failed. Error=9000012.


This was run:
Get-MailboxDatabase | Add-Adpermission -user "domain\exbackup" -extendedrights Receive-As
Two of four DBs threw a warning as they had these permissions already. Other two appeared to go through.


MAPI connections and backup profile have been redone and now we are at a loss as to the issue.
It could possible be service acc permissions.

No errors in the Commvault event viewer for this backup job except that follow on backup fail as a backup is still running being the backup that is hung.

I cannot figure out why this is not working.
Any ideas?
webdude2000Asked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Roshan MohammedCloud Engineering OfficerCommented:
can you create separate backup jobs for each database and run 1 by 1, and you will come to 1 database that's  not playing well.

also, what time does your backup run and what time does your exchange maintenance runs?
Roshan MohammedCloud Engineering OfficerCommented:
in addition, you should be on the latest SP, what level is it?
webdude2000Author Commented:
Apologies, I need to still come back to this issue.
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

SteveCommented:
important question: has it ever worked?

If this is the first attempt I'd suspect permissions on the user account Commvault is using. Try it with another user (possibly an enterprise/domain or exchange organisation admin) to see what happens.

If it worked before and has stopped, permissions seem unlikely unless you know of changes that have been made.
it may be worth a restart of the exchange AD service (which will restart most of the others too unfortunately) to rule out a simple token issue.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
webdude2000Author Commented:
Yes, it used to work.
We suspect the issues is on the exchange server and/or MAPI connectivity related.
Roshan MohammedCloud Engineering OfficerCommented:
can you create separate backup jobs for each database and run 1 by 1, and you will come to 1 database that's  not playing well.

also, what time does your backup run and what time does your exchange maintenance runs?
webdude2000Author Commented:
Someone tinkered with the issue and all databases back up now however some mailboxes don't backup at a mailbox level even though they are split into groups. Since all dbs are backing up, we are not spending more time on the issue.
Thanks to all the comments. I clearly need some Commvault training.
SteveCommented:
no problem. shame you didn't get it fixed but at least it's mostly backing up.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.