Link to home
Start Free TrialLog in
Avatar of lrbarrios
lrbarrios

asked on

Backup Exec 9 keeps alternating between Active jobs

I'm using Backup Exec 9 on this particular server.  It used to backup to a robotic library that had two drives.  The drives have both failed.  I've attached an external drive to another server and shared it out.  Created a removable Backup-to-Disk device to the external drive.  I used the default parameters for BTD folder (1GB size, etc).  Modified the backup jobs to backup to the new BTD device instead of 'All Drives' just to make sure they would go there.  Everything seems to be working... except...  Back in the day when the tape library was working, BE saw that the library had two drives and would simultaneously run two jobs at once.  All jobs (about 6) were set to Medium priority and scheduled for 12:01am.  It didn't matter which jobs ran first or last as long as they all ran during the night.  Now, what's happening is this... (and it actually may have something to do with the fact that I force-started all the jobs by right-clicking them as choosing 'Run Now').  BE changed the 4 jobs that I'm dealing with from On Hold to Queued.  I think it was at this point that the Elapsed Time counter started for all the jobs.  Then it started one of the jobs.  Several minutes later, that job changed its status to Loading and then back to Queued.  Another job started.  Then that second job did the same thing.  BE keeps doing this with all of the jobs.  Finally, one of the jobs finished and now there are only 3 jobs bouncing around.  They look like they're backing up normally except for this weird behavior.  What I've noticed, when looking at the Backup-to-Disk folder (in Windows Explorer) on the external drive is that last file in directory (for example B2D000150.lck) keeps changing it's name after the current job changes its status to Loading.  The filename keeps incrementing through all of the previous versions of the .bkf files (for example B2D000023.lck, B2D000024.lck, B2D000025.lck, etc.) until it reaches the highest numbered .bkf file.  At this point, the job changes from Loading to Queued and another job will resume where it left off.  Another problem that this creates (and again it could just because of the 'Run Now' in this instance) is that the Elapsed Time for all of the jobs are basically the same.  This won't help me if I need to actually alter the jobs' scheduled start times so they don't overlap and start doing this again.  What is going on here and how can I stop this behavior?  Although everything is backing up, it's taking the backup to run much longer than it should.  Any suggestions?  Thanks.
ASKER CERTIFIED SOLUTION
Avatar of Barthax
Barthax
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of Member_2_231077
Member_2_231077

Just an aside - the default settings for B2D folders lead to bad fragmentation, there's an option to assign the maximum size to the B2D file when the file is created, that stops the fragmentation since BE will just overwrite the contents and won't have to stretch it next time it's used.
Avatar of lrbarrios

ASKER

Barthax - Thanks for the response.  Sorry it took me so long to get back to you.  Okay, here's the dealio... I think you mostly understood what I was saying and I mostly understand your response.  There are a couple of things that I want to clarify just to be certain you understand what I'm saying.  

Firstly, because the backup 'device' is a Removable Backup-to-Disk folder, it can only run one concurrent backup job at a time to it.  The option to change (as you referred me to in your response) is not even available.  That's okay because I think running more than one job at a time across the network to a USB drive would probably be counter-productive.

Secondly, I did a Run Now on all four jobs.  This caused all four jobs to change their status from On Hold to Queued.  I thought the jobs would just run in succession, one after the other.  Instead, what it looks like they are doing is running until the 'media' (the .bkf file) fills up.  I used the default value of 1GB (I think in BE 12, which I also have on another server, the default is 4GB).  Once the .bkf file reaches 1GB, the job that was running goes back to Queued and one of the other four jobs resumes (starting by digging through the previous .bkf files looking for available media).  When it can't find any overwritable media (because I'm just starting out with an empty BTD folder) BE creates a new .bkf file.  Once that one fills, the process starts again with yet another of the four jobs.  Why doesn't BE just concentrate on the job that it's working on until it finishes?  Another result of this bouncing around is that the .bkf files that hold the entire backup for a particular resource are not contiguous (BD000023, BD000027, BD000031, etc.)  Again, all this might have something to do with the fact that I said Run Now.  Since there was a break in the action when a .bkf filled up, maybe BE went looking for something else to do with one of the other jobs that were patiently waiting.  The longer the backup runs, the more .bkf files I have.  As a result, it takes longer and longer to do that scan for overwritable media.  I'm sure some of the 'slowness' probably comes from the fact that the BTD folder actually sits on an external USB drive across the network on another server (my BE v12 server).  Originally that server was going to do the backups, but then I learned that 12 doesn't support NT and that's what I'm needing to backup.  Next week I'll be moving the USB drive to the W2K server that's running this BE v9 to eliminate an unnecessary network hop.  (On a side note: I'll need to figure out how to do that.  It's the same server backing up to the same drive, it's just that the drive moved -- ie it's not hanging off of a share on server x, it'll be directly attached to the BE v9 server's USB).

A couple of thoughts I had on the matter that might help the situation out:
1) Move the USB drive to the server that's actually doing the backup.
2) Change the maximum size for the .bkf files from 1GB to 4GB.
3) Also considering making a separate BTD folder for each job or server.
4) Changing the priority of the jobs so that they will have some addition 'guidance' in addition to a scheduled start date/time.

andyalder - Thanks also for your response.  I'll look into the fragmentation issue.  I don't recall seeing such a setting in v9 though.  Thanks.

Any other suggestions/thoughts?  Thanks.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
I don't think backing up a UNC will work for my purposes.  I'm trying to get a full backup of the entire server and there will probably be several files open.  Backing up c$ probably won't work.

The concurrency limitation seems to only be with Removable Backup-to-Disk folders.

I think the original behavior (the 'bouncing around') was a direct result of me choosing Run Now on all four jobs.  Last night, everything worked fine.  I increased filesize for the BTD files from 1GB to 4GB.  Using software compression, I managed to get 2:1 compression on my differential backup and only needed to use 2 BTD files.  The backups were done in no time.  The problem was that those 2 files were in the same backup-to-disk 'device' as the previous full backup.  There are 250 .bkf files in there.  I need to figure out how I want to separate everything out.  Do I create a different BTD device for each server or job.  Some servers have multiple jobs because otherwise the AOFO will fail because of diskspace issues.  I've re-read your last paragraph above and I'm embarrassed to say, I don't follow.

Also, what is the 'folder' in BE under the Devices\Back-to-Disk device in the tree view?  It's named 'FLDR000002' and contains all of the .bkf files.  The folders structure on the actual USB drive doesn't contain this folder.  In BE, can I rename that to one of my servers and have that server's BTD files put there and then duplicate this for each server.  OR do I need to create a separate BTD device for each server (repeating myself now)?

On a related note...  I cannot get this USB drive to work on the Windows 2000 Server BE v9 server so the USB drive is back on a remote server being shared out.  Backups are being sent across the network.  This drive is USB 3.0 capable and I've seen other people fix the USB issue by adding a third-party USB adapter.  Unfortunately, I can't find a USB 3.0 adapter that's NOT PCI express.  The server that I would like to attach the USB drive to is a Dell PowerEdge 2650 that only has PCI/PCI-X (PCI Extended, not Express).  I guess I could look for a USB 2.0 card.  I'm not even sure if the PE has USB 2.0.  (Just looked.  Nope PE 2650 is USB 1.1)  Any suggestions there?  Thanks so much for your help.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Sorry for the delay in getting back to you.  Everything seems to be backing up now, but there are still some weird issues.  Even though I've set the maximum size for the individual backup files (.bkf files) to be 4GB, they continue to grow and grow.  I have one "Full" file that is 90GB.  When I look at the "Full" media in BE, instead of showing 4GB as its capacity, it's showing the remaining space on the drive as its capacity.  Another issue that I have is that I have two jobs (both for the same server) where I can't choose a destination media set other than "Media Set 1" -- that's all that's available in the drop-down.  Once those backup jobs are complete, I can move the resulting media to their proper media set (i.e., Full or Differential).  None of the other jobs have this problem.  I have tapes that show that they're online (or offline, as the case may be) and they're not.  The SMTP notifications just stopped working one day (about a year ago).  I'll probably never use the tapes again because the robotic tape library is shot.  I think I want to blow the whole Backup Exec library away and start from scratch.  While that may fix many of the weird media-related problems that I have, I don't think it'll fix the SMTP problem.  A reinstall may be required for that.  If I reinstall, will I have to recreate all of my jobs?  I realize that I've drifted from the original subject/question.  You've been very helpful and if you don't mind, I'd like to continue working with you.  Would increasing the point value help to persuade you?  :)  Thanks.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Well, I'm embarrassed to say, but it looks like I have some updates available.  Probably the most important one being 9.0 to 9.1.  Didn't even know that was available.  I'll try to install that tonight.  After I install that, I'd like to start with a clean slate.  Is there a way to dump the catalog information and start over?

I don't use job templates, so that's not what is limiting my media sets in that one job.

I can send test messages to my e-mail and my cell phone from within BE.  I've verified that notifications have been turned on for the alerts I want to know about (e.g. Job Failed) and that I'm marked as the recipient.  At one time (a long time ago) it worked.  Then, one day, it stopped sending me notifications.  I've Googled the problem in the past and other people have experienced the same thing.  I never could find a solution.  Maybe tonight's update will fix that as well.
In Tools \ Options, select the Catalogue option down the left-hand side.  You should find a Truncate option within there (it could be named differently back in v9).  Set that to truncate at the shortest possible time span & allow BE to do the work (but remember to remove the setting once they're gone!).

Good luck with the update - I've done many from v8 onwards & I don't recall any glaring issues. :)
Thanks, I'll give that Truncate option a try.

I've applied the 9.1 upgrade and it looks like everything went ok.  As a bonus, my e-mail notifications are mysteriously working again.  The 'Media Set 1' issue still exists.  I think it's time to just wipe BE memory and start over.  I've already removed all of my tapes from the library (Retired them).  Now BE only sees my BTD folders/files.  I'll give an update once I've rebuilt it.  Thanks.
The media set issue is really being a nuisance.  If I delete Media Set 1 and restart BE, BE puts it back.  Fine.  If I delete Media Set 1 and then look at the two jobs that keep getting sent to Media Set 1, there is nothing in the media set drop-down.  After restarting, it's back.  If I create a new media set, it'll show up in the drop-down with Media Set 1, however any new media sets aren't available to all of the other jobs that are already exist.  I don't use job templates so I don't understand why this is happening.  Looking at Tools/Option/General, the default media set drop-down only includes Media Set 1 and the other two newly created media sets.  It doesn't contain the other media sets that I'm actually using.  Also, it looks like BE started two jobs at the same time again last night.  Ugh!
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
I was leaning towards some kind of corruption/'crossed wires' too.  I'll give the links a shot.  Thanks.

On a related note... (and I'm just mentioning it because of the timing of your mentioning the SQL instance -- I may actually open a new question on EE today).  Yesterday, on my Backup Exec 12d, I was on the phone for 5 hours with Sophos (our anti-virus vendor) tech support.  I was trying to upgrade to the new version of their Enterprise Console and endpoint software (because the current version will retire at the end of the month).  It too uses an instance of SQL.  The upgrade said that I had to manually upgrade the SQL on the server (Windows 2003 R2) to either 2005 or 2008.  It mentioned that I could download and use the Express Editions for free.  (To be honest the whole SQL/MSDE thing is somewhat a mystery to me.  How one machine could have 3 different flavors for different products and still work always baffled me.)  Anyway... I didn't know what SQL it was currently running, but I wanted to go ahead and upgrade to SQL 2008 Express.  While going through the setup for that, I didn't see the SOPHOS instance in the drop-down.  I decided to call tech support.  He remoted to my machine.  Long story less long, we couldn't update to 2008 because of the existing 2005 that I had on there.  He suggested that I remove all the SQL stuff in Add/Remove Programs and start with a fresh install of Sophos (I would lose all of my Sophos configurations/policies/database etc.).  I agreed.  While he was blowing away SQL-related things in Add/Remove Programs and dumping SQL-related folders, I kept reminding him that that machine is also running Backup Exec and that it uses an instance of SQL also.  He assured me that everything woul d be put back in place after we successfully re-installed SQL 2005 Express.  We re-installed and got the new version of Sophos working.  Yeah!!  But now BE doesn't work anymore.  I can't even start the services.  The Event Viewer is talking about a missing service: MSSQL$BKUPEXEC.  That WAS my instance for BE that I was talking about.  Here's where I'm at... After all of the installing, removing, deleting folders and re-installing of SQL stuff, it looks like the first instance of the new SQL 2005 Express decided to use the same MSSQL.1\MSSQL\Data folder that contained my now-orphaned BE database.  I've created a new instance (which creates the MSSQL.2 folder tree).  The MSSQL.1 has, what looks like backup files of the database files (master, mastlog, model, modellog, msdbdata and msdblog$4idr).  I may be able to move them, rename them, and attach them to the new instance.  I'm not sure that will work though because the Date Modified of those files looks to be the date when I installed 12d on that server.  They may actually be empty-ish files and don't actually contain anything.  Which brings me to my next question... What is even in those files?  Jobs, Selection Lists, Templates, etc.?  Honestly, I only had 4 jobs (for 2 different servers) on that install of BE.  Worst case, I guess I could just start over.  Without re-installing BE, how can I get the database going so that I can at least start the BE services so that I can get into the program and then re-create everything.  Please don't feel obligated to answer these last questions.  I was just throwing them out there.  I'll be working on that today because I need to get it fixed before I go on a week-long vacation starting this weekend.
I recognise that MSSQL$ style services are MSDE instances and not SQL Server 2000 - the old equivalent of Express version vs. full-blown in 2005/2008 (so, yeah, six different versions of the same product on the same machine?!).  As (based on my knowledge of SQL 2005) the DB is directory-based, I'd suggest trying the following:

- Stop the new instance of MSSQL.
- Move (keep the new in case it goes wrong) the new SQL files out of that directory.
- Copy (to preserve the originals) the old SQL files into that directory.
- Start the new instance of MSSQL.

In theory, that should give you the old data in the new instance, but I bet they're will be a sticky point I don't know about. :(
I think I'm going to be able to put this 'question' to rest.  You've been very helpful, but I think I've wandered from the original scope of the question and should close this one.  I've increased the point value from what I had originally set it at -- mostly due to your ongoing assistance with whatever else I threw into the mix.  I think I'm good now.  Thanks so much for the help!
No problem.  Thanks for the increase in points.  Glad to have helped. :)