Backup Exec 2012 End Marker Unreadable VIRTUAL Tapes

Hello, I am running BE 2012(which is the biggest piece of crap) on a Windows 2003 Standard server.  This media server is connected via iSCSI to a REO 4500.  I am getting "end marker unreadable" on all my VIRTUAL tapes.  I opened a support ticket with Symantec, they told me to long erase my tapes and that would resolve the issue.  This of course did not work.  Symantec can't seem to wrap their heads around a virtual tape.  I keep telling them it isn't a physical tape, but they are lost.  Now Symantec is pointing the finger and the VTL manufactuer, Overland Storage.  The VTL library worked perfectly with version 12, so I doubt it is the VTL.
Anyone out there who can lend a hand?
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Rodney BarnhardtServer AdministratorCommented:
mrbodjaglesAuthor Commented:
I previously installed the hotfixes, service packs, live updates, etc.
Rodney BarnhardtServer AdministratorCommented:
Check the logs and see if the job engine stops during the time when the backup is running. Also, did this problem start right after you upgraded?
The 7 Worst Nightmares of a Sysadmin

Fear not! To defend your business’ IT systems we’re going to shine a light on the seven most sinister terrors that haunt sysadmins. That way you can be sure there’s nothing in your stack waiting to go bump in the night.

Also make sure that the iSCSI path is optimal. Disabling RSS and TOE on Windows 2003 may help if not done already. The "end marker unreadable" can happen when communication between Backup Exec and the library is suddenly interrupted. Job engine or Device and Media service crashes can be a problem as rbarnhardt mentioned, but also network communication could be a problem.
I know it worked in BE12, but you're not saying if this BE2012 is running on the same server with the same OS instance as BE12.
mrbodjaglesAuthor Commented:
How do I disable RSS and TOE?  BE2012 is running on the same server with the same OS and the same iSCSI connection to my VTL.
denbosseCommented: describes how to prevent problems with RSS and TOE. Method 3 indicates how to disable RSS and TOE from the registry.
mrbodjaglesAuthor Commented:
I disabled both RSS and TOE.  That did not solve the issue.  I am still getting "end marker unreadable" indicated on my virtual tape slots.  
Any other suggestions?
I'm willing to try anything at this point.
Thanks again.
Have you checked the event logs for "Application Crash" events as rbarnhardt suggested?
Do the Backup Exec jobs that use these virtual tapes fail? Any indication of something going wrong in the Job Logs?
mrbodjaglesAuthor Commented:
Come on this is Symantec, of course there are errors!  
Here are the three most common:

Backup Exec Alert: Job Failed
(Server: "Servername") (Job: "servername.domain.comBackup -Full") Backup -Full -- The job failed with the following error: Unable to attach to a resource.  Make sure that all selected resources exist and are online, and then try again.  If the server or resource no longer exists, remove it from the selection list. Edit the selection list properties, click the View Selection Details tab, and then remove the resource.
Backup Exec Alert: Job Failed
(Server: "Servername") (Job: " Backup -Full") Backup -Full -- The job failed with the following error: The Backup Exec job engine system service is not responding.
This error is the most perplexing.

Update to catalog index (Catalog index database) failed.
Reason:  C:\Program Files\Symantec\Backup Exec\Catalogs\server\ PerformBulkInsert()
Not sure about r:\honshu\1798r.i386\becat\catindex\dll\..\catdbupdatethread.cpp(499)

There are no drive mappings to r:\ on the media server.
The second one - the Job engine problem - is probably causing the end marker problem, as rbarnhardt pointed out earlier.
No "application error" event 1000 in the event logs at the same time. They may give a clue.

Symantec have some open issues with bengine.exe 

Make sure to run liveupdate to get all the latest updates.

The root cause could be loads of things. Backup Exec, the network between server and VTL or the library.
If really nothing changed apart from the version of Backup Exec, I would tell the Symantec 1st-line support guys to escalate to their 2nd level and then their backline engineers. Ask for a duty manager if you're not getting anywhere.

In the meantime make sure your network drivers are fully up to date, make sure the Overland firmware is fully up to date, check that there's no speed/duplex mismatches between NICs and network switch ports, etc.

I would also exclude bengine.exe from antivirus scanning.

You'll probably also have to enable Backup Exec debug logs - this will show you exactly what the last thing is the Job Engine service does before becoming unresponsive.

The r: drive thing is not that perplexing. The r: drive refers to the drive on a symantec system, where they probably keep their cpp (c++) source code.

And yes - I have seen Backup Exec systems that are completely free of errors :-)
mrbodjaglesAuthor Commented:
Network drivers on the media server are the latest.  Firmware on the Overland is the latest.  The connection between the overland and the media server is direct via a cat5 cable.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
mrbodjaglesAuthor Commented:
None of the answers provided solved my issue.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.