Restore of individual email items to Exchange 2003 using Backup Exec 2010 consistently fails

The question I have first is - are you backing up brick level or the whole store?  The other question is why are you not backing up one full per week and the incremental daily?  How much data are we talking about?

Firstly, we are backing up using GRT, which to my understanding is a brick-level backup. Secondly, it is company policy to perform full backups every night.  We are also backing up entire servers via Veeam for DR purposes.

Our data is approximately in total 275GB, which is private and public.

I found a case with the exact same problem. This looks like it may be a conflict of some type and as of about 2 months ago, no comment on whether this is a fix for it. There is a suggested work around. I know it is Exchange 2007, but the errors are the same.
http://www.symantec.com/connect/de/forums/got-open-ticket-w-symantec-will-post-here-too-exchange-2007-w-backupexec-2010
 

Sorry, similar, not the same.

Here are some articles:

http://seer.entsupport.symantec.com/docs/351373.htm - You might have done this one

http://www.symantec.com/connect/forums/v-79-57344-65247-v-79-57344-65247-exchange-store-service-not-responding

Advice:
One backup everyday is not the MS recommended way of exchange backups.  If the backup you are using is corrupted and you don't have any other backups available then you are fully unprotected for a disaster.

I would recommend buy a 1 tb drive and having 3 backups min on that drive or buy two 1 tb drives and have multiples to each drive for at least a week.  that way you care covered and the likely hood of all backups being corrupt is slim.

I've been and exchange Admin for many years and have had the experience of not having sufficient backups.  The real recommended way would be do a full local and have a full offsite (mozy pro, crashplan or any online backup solution that has an exchange plugin) then do incrementals everyday.  Mozy Pro's exchange plug in is great and keeps track of all revisions of the store.

That is just my 2 cents and just try to watch out for my fellow brother of exchange.

Hi

ssparks827: I really appreciate your insights regarding backup strategy and its something I am more than willing to address, however, unless I can restore an email from either a public folder or private mailbox, it is rather a mute point.  I am going to look further at the first article.  The second one refers to re-creating the backup job.  Does this mean the job itself, or to actually run it.  The process takes almost three hours and has a fair impact on the users during office hours.  Incidentally, when you use Exchange Recover database, does that require the entire public and private store to be restored first and then an email can be restored?  Apologies if I am asking the obvious, Exchange is not my forte and the network administrator is off for 2 weeks!

Modru: The account used for the restore job is exactly the same as the one that runs the backup, which is a domain administrator account.  However, the machine running Backup Exec is in its own workgroup, whose services are running under an administrator account.  I'm not sure how I would go about adding this account to the permissions on Exchange in relation to the Information Store, user Mailboxes or Send As/Received As.

Apparently the Exchange Server was restarted over the weekend, but I still can not restore.  Does the account under which the Backup Exec Media Server is running have any baring on the restore?  I change the resource credentials from local system account to the domain administrator account.

The resource credentials for the backup and restore should be the same. It needs to be a domain admin and Exchange admin without any express deny priviledges, as mentioned before. The user account you run this as also has to have an active mailbox, not hidden from the global address book, and must have had a piece of mail sent to it and in some extreme cases a message sent from it.

I have run into issues with GRT before where all sorts of odd errors until every one of the concerns listed above were addressed.

Dunedan79: The resource credentials are the same for the backup job and the restore job.  The account used is a domain administrator account.  This is the account under which the Exchange services are running.  I'm not sure what you mean when you say user account?  Do you mean the user account of the Backup Exec Media Server?  This machine is in a workgroup and definately wouldn't have a mailbox of its own.  The backup job successfully attaches to each mailbox and backs it up.  I can also successfully browse the backup-to-disk folder and see the emails and folder of all mailboxes when attempting to restore.  I have tried restoring to the original mailbox, which definitely exists, and tried redirecting to an alternate mailbox which also exists (mine in fact) and both are visible in the global address list.

Hi

Dunedan79: Ah, OK, the credentials used is a member of Domain Admins.  It is implicitly added to Exchange Services and Exchange Domain Servers, which in itself is a member of Exchange Enterprise, although I have now added it directly into that one as well.  I can see the account in the global address list.  

This is the tool installed on the exchange server :

Remote Agent For Windows

These are the tools installed on the backup server :

Symantec Backup Exec 2010
Remote Agent for windows and Netware Servers
Agent for Microsoft Exchange Servers

Is there something we have missed and need?

When this Exchange Server wasn't in the VM environment, backups were performed directly on the box to a tape device directly attached, using the account credentials I am attempting to use here.  Admittedly, the version was very old (i think ver 9.1, but I can't be sure) and didn't use GRT or backup-to-disk technology, just straight to tape and that's how we restored it.  Is it simply I have sufficient permission to read the data from Exchange, but insufficient to write?

As stated earlier by another poster, Exchange 2003 adds inherited deny entries for send-as/receive-as to mailboxes. You are going to need this right back on the target mailbox. That means editing the security on the mailbox to add these back before you can restore. Out of the box you could restore an entire database but an individual email is a struggle.

As I stated earlier, you need to have an account with full Exchange permissions. This account can not have expressly denied permissions. A couple years ago Microsoft made a security change that set Domain Admins to expressly denied by default. I suggest you check the security of the account your using, in Exchange itself, as well as whatever groups it has membership.
If the account is a member of a group that is denied, the account itself will be denied. This is the single most common problem with brick level backups on Exchange.
If you are unable to see all of the Security tabs within Exchange, you will need to do a reg hack to show them;
Start Registry Editor (Regedt32.exe).
Locate the following key in the registry:
HKEY_CURRENT_USER\Software\Microsoft\Exchange\EXAdmin
On the Edit menu, click Add Value, and then add the following registry value:
Value Name: ShowSecurityPage
Data Type: REG_DWORD
Radix: Binary
Value: 1
Quit Registry Editor

modru: Ok, I'll take a step back.  I've now popped the backup server into the domain.  I'll create a new AD account and mailbox from scratch and make it a member of Domain Admins and Exchange Services (would this be the equivalent of Exchange Admins?).  Would there be any other groups?  I will implicitly add this user to the Information Store(s) permissions by selecting the Security tab of the Mailbox Store properties and give it Full Control (which should include Send As and Receive As).  Do the same for Public Folder Store.  It will have access to all users mailboxes by membership of Domain Admins.  Then, I should change the resource credentials of the backup job to the new user account and use the same for the restore job.  Would that sound like a fair plan?

Due to recent recent hotfixes, windows updates, and security updates, your Domain Admins group may be expressly denied access to mailboxes. The easiest way to confirm this is to logon to a domain computer with a domain admin account, configure outlook to open a different user's mailbox, open outlook. If you are unable to see the user's email, the Domain Admins are denied acces, and you must not add the new account to this group.
All of the rest sounds good, as well as delegating or manually adding the account to the top level of the Exchange Site and Exchange Server if you are able to, from the Exchange Management Console.
Make the account a member of the local administrators group on both the backup and Exchange servers.

The Account must have a unique mailbox on the Exchange Server.

Here is a link to NON-GRT brick level backups for Backup Exec 11.D and later;
http://seer.entsupport.symantec.com/docs/285797.htm

As taken directly from the Backup Exec Admin guide (Agent for Exchange server section) available on the product CD:-

Using Backup Exec Logon Accounts with Exchange
Resources

In order to back up and restore Exchange Server resources, you must use a Backup Exec service account that has domain and local administrator rights on every Exchange server that you want to back up.
In order to back up and restore individual Exchange mailboxes and public folders when
using Exchange 2000 or 2003, use Microsoft?s System Manager utility to grant the Exchange Administrator role to the user account at the Administrative Group level (for all Administrative Groups if there is more than one). In order to back up and restore individual Exchange mailboxes and public folders when using Exchange 5.5, use
Microsoft?s Exchange Administrator utility to grant the Service Account Admin role to the user account at the Site level (for all Sites if there is more than one).
Additionally, Backup Exec must have access to a uniquely named mailbox within the Exchange organization. When selecting mailboxes or public folders for backup, Backup
Exec will attempt to find a mailbox with the same name as the username stored in the Backup Exec logon account used to connect to the Exchange server. If you use a Backup
Exec logon account that stores the credentials of a user account that is unique and has a corresponding mailbox of the same name, then you are not prompted for an additional
logon account when selecting mailboxes or public folders. Otherwise, you must choose or create a Backup Exec logon account that stores the name of a unique mailbox within the
Exchange organization. A unique name is one that does not share the first five characters in another mailbox name. For example, if EXCH1 has been entered as the mailbox name, and there are other mailbox names such as EXCH1BACKUP, or BACKUPEXCH1, then Backup Exec will not
accept the name and you are prompted to choose another mailbox name. If you cannot create a unique mailbox name, you must create a Backup Exec logon account and enter the
fully qualified name in the username field of the logon account.
For example: /O=Company/OU=Orlando/CN=Test/CN=EXCH1

Looking at this another way.  Each time the restore process runs, its takes down the RAWS service on the exchange server after about 20 seconds.  The service is set to restart of failure, so wasn't immediately noticeable.  Am I correct in thinking it could be the service failure that causes the communications error with the exchange stores rather than possible restricted permissions causes the service to fail?

If the remote agent fails you will definitely get communication failures. Have you pushed the agent to the Exchange server recently? When you run liveupdate on your Backup Exec server it gets patches but never pushes them to the client unless you do it.

I'd suggest running liveupdate and then pushing the client down so it updates the remote server.

I ran liveupdate and found there were two hotfixes and a service pack.  I installed all three, restarted the backup server and redeployed the agent.  I'm afraid the RAWS service still terminates.

Have you looked at the event viewer for any event ID's or causes for the service crash? That may be more helpful.

In the application log I have :

Event ID 257 Source BEDBG: A memory dump has been captured ...
Event ID 1000 Source Application Error: Faulting application beremote.exe version 13.0.2896.120, faulting module uknown, version 0.0.0.0, fault address 0x00000000

The system log just has :

Event ID 7034 Source Source Service Control Manager: The Backup Exec Remote Agent For Windows Systems service terminated unexpectedly.  It has done this 1 time.

How about Anti-Virus or other applications ? Have you updated or installed anything recently ? Even something as simple as a network card driver could be a cause for this. I suggest you check the installed applications list, recent updates, and driver versions. See if you can pinpoint any recent changes or additions.

Tried running it after disabling the Anti-Virus, but still no luck.  Redeployed the agent, manually this time, no change.  Been trying for two days to get some assistance from Symantec.

Now looking at alternative products, one which we are trialing now and seems to do exactly what we want, if not more.

I've got an ongoing technical support call for three days now.  I'll update with the solution when, hopefully, we get it resolved.

Hi

Following two staff holidays, two weeks of technical support calls by multiple members of staff and several debug sessions, the following seems to have resolved the issue :

Created a new account and added it to the delegate list of Exchange System Manager and assigned full control
Removed Outlook 2002 client software, which relates somewhat to the MAPI settings mentioned in the last comment posted; I can't even remember why this was installed, but it never posed a problem backing the server up directly on the same machine with BE9.1

Thanks for everybody's input.

Regards and thanks.

Although the suggestions didn't resolve the problem, they was related to the solution provided by Symantec technical support.