Restore of individual email items to Exchange 2003 using Backup Exec 2010 consistently fails

We are having problems restoring individual email items.  Our scenario is as follows :

Windows 2008 SP2 Server running Backup Exec 2010 rev 2896 in a workgroup, with the following hotfix's, 348284, 348518, 348315, 351870, 355131, 354911, 354451, 355938, 355922 and 351400.  All services, other than Remote Agent For Windows, are running under a domain administrator account.
Windows 2003 SP2 Server running Exchange 2003 (6.5.7638.1) in a VM environment, with 13 other servers, all in the same domain, using Remote Agent for Windows (we redeployed the agent again today).
Scheduled job which runs a backup-to-disk each day using the domain administrator account and GRT
The backup-to-disk folder, among other things, are written off to tape daily

We only have room for a single days data, but each days backup-to-disk job successfully deletes the previous days data.

Our problem comes when we attempt to restore an individual email item, the whole reason for purchasing the software.

We can browse the backup-to-disk media, choose the an item from any mailbox, but when the job runs, we get the following error consistently

V-79-57344-65298 The exchange store service is not responding.  Backup set cancelled.
V-79-57344-65072 Connection to the restore target has been lost.  Operation cancelled.

The job returns an error code of 0x000ff12 in Job Monitor.

Exchange is running, the restore path is not on C:, the mailbox that we are redirecting the restore to is not open, it isn't hidden from GAL and the account is a domain admin.

I have looked at a number of articles including  There is no mention of a backup server that is in its own workgroup causing a problem.  Does anyone have any ideas?

Thanks in advance.
JulieSenior Analyst/ProgrammerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

The question I have first is - are you backing up brick level or the whole store?  The other question is why are you not backing up one full per week and the incremental daily?  How much data are we talking about?
JulieSenior Analyst/ProgrammerAuthor Commented:
Firstly, we are backing up using GRT, which to my understanding is a brick-level backup. Secondly, it is company policy to perform full backups every night.  We are also backing up entire servers via Veeam for DR purposes.

Our data is approximately in total 275GB, which is private and public.

Rodney BarnhardtServer AdministratorCommented:
I found a case with the exact same problem. This looks like it may be a conflict of some type and as of about 2 months ago, no comment on whether this is a fix for it. There is a suggested work around. I know it is Exchange 2007, but the errors are the same.
10 Tips to Protect Your Business from Ransomware

Did you know that ransomware is the most widespread, destructive malware in the world today? It accounts for 39% of all security breaches, with ransomware gangsters projected to make $11.5B in profits from online extortion by 2019.

Rodney BarnhardtServer AdministratorCommented:
Sorry, similar, not the same.
Here are some articles: - You might have done this one

One backup everyday is not the MS recommended way of exchange backups.  If the backup you are using is corrupted and you don't have any other backups available then you are fully unprotected for a disaster.

I would recommend buy a 1 tb drive and having 3 backups min on that drive or buy two 1 tb drives and have multiples to each drive for at least a week.  that way you care covered and the likely hood of all backups being corrupt is slim.

I've been and exchange Admin for many years and have had the experience of not having sufficient backups.  The real recommended way would be do a full local and have a full offsite (mozy pro, crashplan or any online backup solution that has an exchange plugin) then do incrementals everyday.  Mozy Pro's exchange plug in is great and keeps track of all revisions of the store.

That is just my 2 cents and just try to watch out for my fellow brother of exchange.
This could be a permissions issue.
If the restore is being manually performed by an administrator that does not have full permissions in Exchange, it will fail. You need to make sure that the account that runs the restore job(which can be set manually) has full permissions to the Exchange server, Information Store, and User's Mailbox.
If the Send As and Recieve As are expressly denied (per default from quite awhile ago), the restore will fail.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
JulieSenior Analyst/ProgrammerAuthor Commented:

ssparks827: I really appreciate your insights regarding backup strategy and its something I am more than willing to address, however, unless I can restore an email from either a public folder or private mailbox, it is rather a mute point.  I am going to look further at the first article.  The second one refers to re-creating the backup job.  Does this mean the job itself, or to actually run it.  The process takes almost three hours and has a fair impact on the users during office hours.  Incidentally, when you use Exchange Recover database, does that require the entire public and private store to be restored first and then an email can be restored?  Apologies if I am asking the obvious, Exchange is not my forte and the network administrator is off for 2 weeks!

Modru: The account used for the restore job is exactly the same as the one that runs the backup, which is a domain administrator account.  However, the machine running Backup Exec is in its own workgroup, whose services are running under an administrator account.  I'm not sure how I would go about adding this account to the permissions on Exchange in relation to the Information Store, user Mailboxes or Send As/Received As.

Apparently the Exchange Server was restarted over the weekend, but I still can not restore.  Does the account under which the Backup Exec Media Server is running have any baring on the restore?  I change the resource credentials from local system account to the domain administrator account.
The resource credentials for the backup and restore should be the same. It needs to be a domain admin and Exchange admin without any express deny priviledges, as mentioned before. The user account you run this as also has to have an active mailbox, not hidden from the global address book, and must have had a piece of mail sent to it and in some extreme cases a message sent from it.

I have run into issues with GRT before where all sorts of odd errors until every one of the concerns listed above were addressed.
JulieSenior Analyst/ProgrammerAuthor Commented:
Dunedan79: The resource credentials are the same for the backup job and the restore job.  The account used is a domain administrator account.  This is the account under which the Exchange services are running.  I'm not sure what you mean when you say user account?  Do you mean the user account of the Backup Exec Media Server?  This machine is in a workgroup and definately wouldn't have a mailbox of its own.  The backup job successfully attaches to each mailbox and backs it up.  I can also successfully browse the backup-to-disk folder and see the emails and folder of all mailboxes when attempting to restore.  I have tried restoring to the original mailbox, which definitely exists, and tried redirecting to an alternate mailbox which also exists (mine in fact) and both are visible in the global address list.
When I say user account I mean the account credentials used to attach to the Exchange server. For GRT to restore the account you are connecting to Exchange with has to have an email account, the mailbox cannot be hidden, and must have Exchange admin rights.

Basically if you cannot see the account you are using for connection in the global address list you are going to have failures. You will also want to have the Exchange tools loaded on the media server so Backup Exec has the necessary resources to interact with Exchange.
JulieSenior Analyst/ProgrammerAuthor Commented:

Dunedan79: Ah, OK, the credentials used is a member of Domain Admins.  It is implicitly added to Exchange Services and Exchange Domain Servers, which in itself is a member of Exchange Enterprise, although I have now added it directly into that one as well.  I can see the account in the global address list.  

This is the tool installed on the exchange server :

Remote Agent For Windows

These are the tools installed on the backup server :

Symantec Backup Exec 2010
Remote Agent for windows and Netware Servers
Agent for Microsoft Exchange Servers

Is there something we have missed and need?

When this Exchange Server wasn't in the VM environment, backups were performed directly on the box to a tape device directly attached, using the account credentials I am attempting to use here.  Admittedly, the version was very old (i think ver 9.1, but I can't be sure) and didn't use GRT or backup-to-disk technology, just straight to tape and that's how we restored it.  Is it simply I have sufficient permission to read the data from Exchange, but insufficient to write?
As stated earlier by another poster, Exchange 2003 adds inherited deny entries for send-as/receive-as to mailboxes. You are going to need this right back on the target mailbox. That means editing the security on the mailbox to add these back before you can restore. Out of the box you could restore an entire database but an individual email is a struggle.
As I stated earlier, you need to have an account with full Exchange permissions. This account can not have expressly denied permissions. A couple years ago Microsoft made a security change that set Domain Admins to expressly denied by default. I suggest you check the security of the account your using, in Exchange itself, as well as whatever groups it has membership.
If the account is a member of a group that is denied, the account itself will be denied. This is the single most common problem with brick level backups on Exchange.
If you are unable to see all of the Security tabs within Exchange, you will need to do a reg hack to show them;
Start Registry Editor (Regedt32.exe).
Locate the following key in the registry:
On the Edit menu, click Add Value, and then add the following registry value:
Value Name: ShowSecurityPage
Data Type: REG_DWORD
Radix: Binary
Value: 1
Quit Registry Editor
JulieSenior Analyst/ProgrammerAuthor Commented:
modru: Ok, I'll take a step back.  I've now popped the backup server into the domain.  I'll create a new AD account and mailbox from scratch and make it a member of Domain Admins and Exchange Services (would this be the equivalent of Exchange Admins?).  Would there be any other groups?  I will implicitly add this user to the Information Store(s) permissions by selecting the Security tab of the Mailbox Store properties and give it Full Control (which should include Send As and Receive As).  Do the same for Public Folder Store.  It will have access to all users mailboxes by membership of Domain Admins.  Then, I should change the resource credentials of the backup job to the new user account and use the same for the restore job.  Would that sound like a fair plan?
Due to recent recent hotfixes, windows updates, and security updates, your Domain Admins group may be expressly denied access to mailboxes. The easiest way to confirm this is to logon to a domain computer with a domain admin account, configure outlook to open a different user's mailbox, open outlook. If you are unable to see the user's email, the Domain Admins are denied acces, and you must not add the new account to this group.
All of the rest sounds good, as well as delegating or manually adding the account to the top level of the Exchange Site and Exchange Server if you are able to, from the Exchange Management Console.
Make the account a member of the local administrators group on both the backup and Exchange servers.

The Account must have a unique mailbox on the Exchange Server.

Here is a link to NON-GRT brick level backups for Backup Exec 11.D and later;

As taken directly from the Backup Exec Admin guide (Agent for Exchange server section) available on the product CD:-

Using Backup Exec Logon Accounts with Exchange

In order to back up and restore Exchange Server resources, you must use a Backup Exec service account that has domain and local administrator rights on every Exchange server that you want to back up.
In order to back up and restore individual Exchange mailboxes and public folders when
using Exchange 2000 or 2003, use Microsoft?s System Manager utility to grant the Exchange Administrator role to the user account at the Administrative Group level (for all Administrative Groups if there is more than one). In order to back up and restore individual Exchange mailboxes and public folders when using Exchange 5.5, use
Microsoft?s Exchange Administrator utility to grant the Service Account Admin role to the user account at the Site level (for all Sites if there is more than one).
Additionally, Backup Exec must have access to a uniquely named mailbox within the Exchange organization. When selecting mailboxes or public folders for backup, Backup
Exec will attempt to find a mailbox with the same name as the username stored in the Backup Exec logon account used to connect to the Exchange server. If you use a Backup
Exec logon account that stores the credentials of a user account that is unique and has a corresponding mailbox of the same name, then you are not prompted for an additional
logon account when selecting mailboxes or public folders. Otherwise, you must choose or create a Backup Exec logon account that stores the name of a unique mailbox within the
Exchange organization. A unique name is one that does not share the first five characters in another mailbox name. For example, if EXCH1 has been entered as the mailbox name, and there are other mailbox names such as EXCH1BACKUP, or BACKUPEXCH1, then Backup Exec will not
accept the name and you are prompted to choose another mailbox name. If you cannot create a unique mailbox name, you must create a Backup Exec logon account and enter the
fully qualified name in the username field of the logon account.
For example: /O=Company/OU=Orlando/CN=Test/CN=EXCH1
JulieSenior Analyst/ProgrammerAuthor Commented:
Looking at this another way.  Each time the restore process runs, its takes down the RAWS service on the exchange server after about 20 seconds.  The service is set to restart of failure, so wasn't immediately noticeable.  Am I correct in thinking it could be the service failure that causes the communications error with the exchange stores rather than possible restricted permissions causes the service to fail?
If the remote agent fails you will definitely get communication failures. Have you pushed the agent to the Exchange server recently? When you run liveupdate on your Backup Exec server it gets patches but never pushes them to the client unless you do it.

I'd suggest running liveupdate and then pushing the client down so it updates the remote server.
JulieSenior Analyst/ProgrammerAuthor Commented:
I ran liveupdate and found there were two hotfixes and a service pack.  I installed all three, restarted the backup server and redeployed the agent.  I'm afraid the RAWS service still terminates.
Rodney BarnhardtServer AdministratorCommented:
Have you looked at the event viewer for any event ID's or causes for the service crash? That may be more helpful.
JulieSenior Analyst/ProgrammerAuthor Commented:
In the application log I have :

Event ID 257 Source BEDBG: A memory dump has been captured ...
Event ID 1000 Source Application Error: Faulting application beremote.exe version 13.0.2896.120, faulting module uknown, version, fault address 0x00000000

The system log just has :

Event ID 7034 Source Source Service Control Manager: The Backup Exec Remote Agent For Windows Systems service terminated unexpectedly.  It has done this 1 time.
How about Anti-Virus or other applications ? Have you updated or installed anything recently ? Even something as simple as a network card driver could be a cause for this. I suggest you check the installed applications list, recent updates, and driver versions. See if you can pinpoint any recent changes or additions.
JulieSenior Analyst/ProgrammerAuthor Commented:
Tried running it after disabling the Anti-Virus, but still no luck.  Redeployed the agent, manually this time, no change.  Been trying for two days to get some assistance from Symantec.

Now looking at alternative products, one which we are trialing now and seems to do exactly what we want, if not more.
JulieSenior Analyst/ProgrammerAuthor Commented:
I've got an ongoing technical support call for three days now.  I'll update with the solution when, hopefully, we get it resolved.
JulieSenior Analyst/ProgrammerAuthor Commented:

Following two staff holidays, two weeks of technical support calls by multiple members of staff and several debug sessions, the following seems to have resolved the issue :

Created a new account and added it to the delegate list of Exchange System Manager and assigned full control
Removed Outlook 2002 client software, which relates somewhat to the MAPI settings mentioned in the last comment posted; I can't even remember why this was installed, but it never posed a problem backing the server up directly on the same machine with BE9.1

Thanks for everybody's input.

Regards and thanks.
JulieSenior Analyst/ProgrammerAuthor Commented:
Although the suggestions didn't resolve the problem, they was related to the solution provided by Symantec technical support.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.