Solved

Backup Exec 9 keeps alternating between Active jobs

Posted on 2011-02-10
18
787 Views
Last Modified: 2012-08-14
I'm using Backup Exec 9 on this particular server.  It used to backup to a robotic library that had two drives.  The drives have both failed.  I've attached an external drive to another server and shared it out.  Created a removable Backup-to-Disk device to the external drive.  I used the default parameters for BTD folder (1GB size, etc).  Modified the backup jobs to backup to the new BTD device instead of 'All Drives' just to make sure they would go there.  Everything seems to be working... except...  Back in the day when the tape library was working, BE saw that the library had two drives and would simultaneously run two jobs at once.  All jobs (about 6) were set to Medium priority and scheduled for 12:01am.  It didn't matter which jobs ran first or last as long as they all ran during the night.  Now, what's happening is this... (and it actually may have something to do with the fact that I force-started all the jobs by right-clicking them as choosing 'Run Now').  BE changed the 4 jobs that I'm dealing with from On Hold to Queued.  I think it was at this point that the Elapsed Time counter started for all the jobs.  Then it started one of the jobs.  Several minutes later, that job changed its status to Loading and then back to Queued.  Another job started.  Then that second job did the same thing.  BE keeps doing this with all of the jobs.  Finally, one of the jobs finished and now there are only 3 jobs bouncing around.  They look like they're backing up normally except for this weird behavior.  What I've noticed, when looking at the Backup-to-Disk folder (in Windows Explorer) on the external drive is that last file in directory (for example B2D000150.lck) keeps changing it's name after the current job changes its status to Loading.  The filename keeps incrementing through all of the previous versions of the .bkf files (for example B2D000023.lck, B2D000024.lck, B2D000025.lck, etc.) until it reaches the highest numbered .bkf file.  At this point, the job changes from Loading to Queued and another job will resume where it left off.  Another problem that this creates (and again it could just because of the 'Run Now' in this instance) is that the Elapsed Time for all of the jobs are basically the same.  This won't help me if I need to actually alter the jobs' scheduled start times so they don't overlap and start doing this again.  What is going on here and how can I stop this behavior?  Although everything is backing up, it's taking the backup to run much longer than it should.  Any suggestions?  Thanks.
0
Comment
Question by:lrbarrios
  • 9
  • 8
18 Comments
 
LVL 19

Accepted Solution

by:
Barthax earned 500 total points
Comment Utility
First, you are seeing normal behaviour. :)  The fact the jobs queue up after each other suggests you have the limit set to 1 concurrent job.  Check on the Device section, right-click the Backup-to-Disc folder and choose Properties: the "Allow ... concurrent" setting (wording has changed over the versions - I believe it was there in v9).  Two concurrent jobs seems what you are after.

When a running job is using a media file in a Backup-to-disc folder, it places a .lck file in the directory to signal that the file is in use.  Other Backup-to-disc jobs will honour that lock and simply ignore the media file.  When a a job starts, the need to overwrite or append media is taken on board.  It then checks each media in the "device" - which means every media file in the Backup-too-Disc directory - as to whether it can append to or overwrite that particular media. This is why you are seeing the .lck files appearing & disappearing.  Once it has found a relevant media file (remember depending on the overwrite & append settings vs. the used media settings & factoring in the Backup-to-disc settings for size + number of files), it continues to use the file, keeping the .lck file in place.
0
 
LVL 19

Assisted Solution

by:Barthax
Barthax earned 500 total points
Comment Utility
Missed a bit!  Elapsed time: two different overall start times are registered in each job: the first start time is when the job becomes scheduled.  When the job actually begins to attempt to backup (by first finding the necessary media) is the second start time.  This second start time is when the Elapsed Time is calculated from.  You can find in the log additional times - when backing up actually begins and ends.
0
 
LVL 55

Expert Comment

by:andyalder
Comment Utility
Just an aside - the default settings for B2D folders lead to bad fragmentation, there's an option to assign the maximum size to the B2D file when the file is created, that stops the fragmentation since BE will just overwrite the contents and won't have to stretch it next time it's used.
0
 

Author Comment

by:lrbarrios
Comment Utility
Barthax - Thanks for the response.  Sorry it took me so long to get back to you.  Okay, here's the dealio... I think you mostly understood what I was saying and I mostly understand your response.  There are a couple of things that I want to clarify just to be certain you understand what I'm saying.  

Firstly, because the backup 'device' is a Removable Backup-to-Disk folder, it can only run one concurrent backup job at a time to it.  The option to change (as you referred me to in your response) is not even available.  That's okay because I think running more than one job at a time across the network to a USB drive would probably be counter-productive.

Secondly, I did a Run Now on all four jobs.  This caused all four jobs to change their status from On Hold to Queued.  I thought the jobs would just run in succession, one after the other.  Instead, what it looks like they are doing is running until the 'media' (the .bkf file) fills up.  I used the default value of 1GB (I think in BE 12, which I also have on another server, the default is 4GB).  Once the .bkf file reaches 1GB, the job that was running goes back to Queued and one of the other four jobs resumes (starting by digging through the previous .bkf files looking for available media).  When it can't find any overwritable media (because I'm just starting out with an empty BTD folder) BE creates a new .bkf file.  Once that one fills, the process starts again with yet another of the four jobs.  Why doesn't BE just concentrate on the job that it's working on until it finishes?  Another result of this bouncing around is that the .bkf files that hold the entire backup for a particular resource are not contiguous (BD000023, BD000027, BD000031, etc.)  Again, all this might have something to do with the fact that I said Run Now.  Since there was a break in the action when a .bkf filled up, maybe BE went looking for something else to do with one of the other jobs that were patiently waiting.  The longer the backup runs, the more .bkf files I have.  As a result, it takes longer and longer to do that scan for overwritable media.  I'm sure some of the 'slowness' probably comes from the fact that the BTD folder actually sits on an external USB drive across the network on another server (my BE v12 server).  Originally that server was going to do the backups, but then I learned that 12 doesn't support NT and that's what I'm needing to backup.  Next week I'll be moving the USB drive to the W2K server that's running this BE v9 to eliminate an unnecessary network hop.  (On a side note: I'll need to figure out how to do that.  It's the same server backing up to the same drive, it's just that the drive moved -- ie it's not hanging off of a share on server x, it'll be directly attached to the BE v9 server's USB).

A couple of thoughts I had on the matter that might help the situation out:
1) Move the USB drive to the server that's actually doing the backup.
2) Change the maximum size for the .bkf files from 1GB to 4GB.
3) Also considering making a separate BTD folder for each job or server.
4) Changing the priority of the jobs so that they will have some addition 'guidance' in addition to a scheduled start date/time.

andyalder - Thanks also for your response.  I'll look into the fragmentation issue.  I don't recall seeing such a setting in v9 though.  Thanks.

Any other suggestions/thoughts?  Thanks.
0
 
LVL 19

Assisted Solution

by:Barthax
Barthax earned 500 total points
Comment Utility
> Originally that server was going to do the backups, but then I learned that 12 doesn't support NT and that's what I'm needing to backup.

Just going to pick up on this bit first as that may negate the rest: BE v12 can perform backups of UNC paths without the Remote Agent being required on the destination machine.  You could, therefore, just amend the text-version of the selection list & append \\<ntserver>\<share>\<path> format for the files you need.  We have a couple of customers using this method for backing up key Win XP machines as separate jobs.

> Firstly, [...], it can only run one concurrent backup job at a time to it.  The option to change (as you referred me to in your response) is not even available.

The concurrency must be a feature of the newer versions, in that case - you should see the option in the BE v12 if you have a Backup-to-Disc folder there, for example.

I'm actually a little surprised by the behaviour of the backup jobs - I am used to them forming an orderly queue for the devices and not relinquishing the device just because of a "media change" occurring!  Backups across the network are never the fastest, so moving the USB drive closer will aid in this matter significantly - though you're only going to be able to use the USB speed of the hardware involved and for an W2K server, my initial reaction is: USB 1.0 support only?  My memory doesn't serve me well there, sorry.

Anyway, certainly increasing the maximum size of the backup files to 4GB will be worthwhile - less "media" to contend with & therefore less opportunity for those symptoms to occur.

Separate BTD folders per job: it is a reasonably idea to reduce the media checking per job.  What you would experience is each BTD folder steadily increasing until there's sufficient media in that folder for the backup job which utilises it.  When calculating a maximum for the BTD folder, don't forget to factor in how often the job runs vs. the media cycle (overwrite/append settings) - for example, a backup job of 10GB which runs every day with overwrite protection for 1 week on the media pool will require 70GB disc space plus room to grow as your data grows.
0
 

Author Comment

by:lrbarrios
Comment Utility
I don't think backing up a UNC will work for my purposes.  I'm trying to get a full backup of the entire server and there will probably be several files open.  Backing up c$ probably won't work.

The concurrency limitation seems to only be with Removable Backup-to-Disk folders.

I think the original behavior (the 'bouncing around') was a direct result of me choosing Run Now on all four jobs.  Last night, everything worked fine.  I increased filesize for the BTD files from 1GB to 4GB.  Using software compression, I managed to get 2:1 compression on my differential backup and only needed to use 2 BTD files.  The backups were done in no time.  The problem was that those 2 files were in the same backup-to-disk 'device' as the previous full backup.  There are 250 .bkf files in there.  I need to figure out how I want to separate everything out.  Do I create a different BTD device for each server or job.  Some servers have multiple jobs because otherwise the AOFO will fail because of diskspace issues.  I've re-read your last paragraph above and I'm embarrassed to say, I don't follow.

Also, what is the 'folder' in BE under the Devices\Back-to-Disk device in the tree view?  It's named 'FLDR000002' and contains all of the .bkf files.  The folders structure on the actual USB drive doesn't contain this folder.  In BE, can I rename that to one of my servers and have that server's BTD files put there and then duplicate this for each server.  OR do I need to create a separate BTD device for each server (repeating myself now)?

On a related note...  I cannot get this USB drive to work on the Windows 2000 Server BE v9 server so the USB drive is back on a remote server being shared out.  Backups are being sent across the network.  This drive is USB 3.0 capable and I've seen other people fix the USB issue by adding a third-party USB adapter.  Unfortunately, I can't find a USB 3.0 adapter that's NOT PCI express.  The server that I would like to attach the USB drive to is a Dell PowerEdge 2650 that only has PCI/PCI-X (PCI Extended, not Express).  I guess I could look for a USB 2.0 card.  I'm not even sure if the PE has USB 2.0.  (Just looked.  Nope PE 2650 is USB 1.1)  Any suggestions there?  Thanks so much for your help.
0
 
LVL 19

Assisted Solution

by:Barthax
Barthax earned 500 total points
Comment Utility
It never rains but it pours!  Sorry for the delay in getting back to your question.

Don't be embarrassed - I find I write in a manner which isn't always obvious!  I'll see if I can re-phrase and we'll meet in the middle.

Increasing your files sizes from 1GB to 4GB will certainly prevent regular file swapping and will also aid in each job holding the BTD folder for itself (I'm still not sure why you experienced the jobs swapping in & out, mind you... still you're passed that issue it seems).

Splitting the backup jobs to different BTD folders:
In terms of what BE is doing, splitting the backup jobs to different BTD folders should not make much difference.  From a viewpoint of management of the files and disc space, it does seem a logical step.  You would be able to "move" the individual BTD folders to different locations if your disc space becomes full, for example (plugging in two USB devices instead, for example).

What you should cater for from each of those new BTD folders is the potential maximum disc use & ensure you have ample - this is what I was trying to cover in my previous post's last paragraph.  You already have a BTD folder which has catered for the size of multiple backups - that media is going to remain on the disc.  If you then create a new BTD folder for, say a 10GB backup set each day and your media pool is set to no overwrite for 7 days, you need to be certain there's 70GB free (note: you mention differential, so I'm guessing you're more a 40GB first day plus 6 days of 2GB = 52GB total).  When you create the new BTD folder and move that job to using that folder, you can expect 70GB/52GB/whatever to get used in additional to the capacity already used by the existing BTD folder because the existing one will not decrease. (Note: you can manually decrease it by "formatting" the files via BE in much the same way you would a tape.)

Folder in BE Devices\BtD:
It has been a while since I've seen BE9 but from memory the 'FLDR000002' can be considered the "name" of the "device" much like you would see a 'Seagate xxx' or 'HP xxx' underneath the Tape device pool.  In newer versions this behaviour is dropped in favour of simply naming the BTD folder - which is probably what you're used to.

USB/LAN performance:
I'm not surprised the USB drive didn't work on the W2K server - USB 1.0 only came in at the tail end of Win98's life span as I recall, so I guess it was an emerging technology at WinME/W2K's release.  Good luck in finding a suitable W2K USB card+driver combination - I doubt there will be any off-the-shelf these days.  Perhaps adding a new NIC to the W2K server and form a private network with a simple WinXP box - house the USB device on the WinXP box.  This would cut down on both the LAN network traffic and the headache sorting out the W2K USB drivers.
0
 

Author Comment

by:lrbarrios
Comment Utility
Sorry for the delay in getting back to you.  Everything seems to be backing up now, but there are still some weird issues.  Even though I've set the maximum size for the individual backup files (.bkf files) to be 4GB, they continue to grow and grow.  I have one "Full" file that is 90GB.  When I look at the "Full" media in BE, instead of showing 4GB as its capacity, it's showing the remaining space on the drive as its capacity.  Another issue that I have is that I have two jobs (both for the same server) where I can't choose a destination media set other than "Media Set 1" -- that's all that's available in the drop-down.  Once those backup jobs are complete, I can move the resulting media to their proper media set (i.e., Full or Differential).  None of the other jobs have this problem.  I have tapes that show that they're online (or offline, as the case may be) and they're not.  The SMTP notifications just stopped working one day (about a year ago).  I'll probably never use the tapes again because the robotic tape library is shot.  I think I want to blow the whole Backup Exec library away and start from scratch.  While that may fix many of the weird media-related problems that I have, I don't think it'll fix the SMTP problem.  A reinstall may be required for that.  If I reinstall, will I have to recreate all of my jobs?  I realize that I've drifted from the original subject/question.  You've been very helpful and if you don't mind, I'd like to continue working with you.  Would increasing the point value help to persuade you?  :)  Thanks.
0
 
LVL 19

Assisted Solution

by:Barthax
Barthax earned 500 total points
Comment Utility
Backup files being in excess of the given maximum sounds like a bug.  Have you updated the installation (Tools menu -> Updates)?

Limited destination media sets are typically governed by the job templates.  On the Job Setup, the middle pane is the template & can limit which media is available to the job.  Check the associated template there for any limits.

Tapes being online or offline: very typical of a standard tape drive.  However, you mention the robotic library is in use & broken...  If it is broken, then the information about which tape is online cannot be reliable.

SMTP problems are a different matter.  Ignoring Backup Exec, can you take the details and test that set up with an e-mail client or telnet (if you know the protocol)?  In my experience, the SMTP is merely a configuration detail - server domain, port number (should be TCP 25) and possible authentication (should not be needed if the server receiving the e-mail is also the destination - no relay issues).
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 

Author Comment

by:lrbarrios
Comment Utility
Well, I'm embarrassed to say, but it looks like I have some updates available.  Probably the most important one being 9.0 to 9.1.  Didn't even know that was available.  I'll try to install that tonight.  After I install that, I'd like to start with a clean slate.  Is there a way to dump the catalog information and start over?

I don't use job templates, so that's not what is limiting my media sets in that one job.

I can send test messages to my e-mail and my cell phone from within BE.  I've verified that notifications have been turned on for the alerts I want to know about (e.g. Job Failed) and that I'm marked as the recipient.  At one time (a long time ago) it worked.  Then, one day, it stopped sending me notifications.  I've Googled the problem in the past and other people have experienced the same thing.  I never could find a solution.  Maybe tonight's update will fix that as well.
0
 
LVL 19

Expert Comment

by:Barthax
Comment Utility
In Tools \ Options, select the Catalogue option down the left-hand side.  You should find a Truncate option within there (it could be named differently back in v9).  Set that to truncate at the shortest possible time span & allow BE to do the work (but remember to remove the setting once they're gone!).

Good luck with the update - I've done many from v8 onwards & I don't recall any glaring issues. :)
0
 

Author Comment

by:lrbarrios
Comment Utility
Thanks, I'll give that Truncate option a try.

I've applied the 9.1 upgrade and it looks like everything went ok.  As a bonus, my e-mail notifications are mysteriously working again.  The 'Media Set 1' issue still exists.  I think it's time to just wipe BE memory and start over.  I've already removed all of my tapes from the library (Retired them).  Now BE only sees my BTD folders/files.  I'll give an update once I've rebuilt it.  Thanks.
0
 

Author Comment

by:lrbarrios
Comment Utility
The media set issue is really being a nuisance.  If I delete Media Set 1 and restart BE, BE puts it back.  Fine.  If I delete Media Set 1 and then look at the two jobs that keep getting sent to Media Set 1, there is nothing in the media set drop-down.  After restarting, it's back.  If I create a new media set, it'll show up in the drop-down with Media Set 1, however any new media sets aren't available to all of the other jobs that are already exist.  I don't use job templates so I don't understand why this is happening.  Looking at Tools/Option/General, the default media set drop-down only includes Media Set 1 and the other two newly created media sets.  It doesn't contain the other media sets that I'm actually using.  Also, it looks like BE started two jobs at the same time again last night.  Ugh!
0
 
LVL 19

Assisted Solution

by:Barthax
Barthax earned 500 total points
Comment Utility
I keep coming back to this looking for something that I've overlooked that might explain something but I'm not finding it. :(  OK, so thinking this through... The Job Templates are not in use, so it is not a template issue.  The Media Sets can be altered and worked upon by the Media section but the jobs do not necessarily see the Media Sets available.  The only common thing I'm finding is the SQL server which holds this information.  There should be an SQL instance called BACKEXEC or BCKPEXEC or something similar.  In itself, the SQL server must be working otherwise you will be getting a lot of issues & various errors.

OK, so presuming the SQL service is running and has no generic issue that only leaves data held within it.  Putting my database hat on, your symptoms are a classic sign of a corrupt index, however.  Unfortunately, reindexing an SQL server has changed over the years and isn't my strong suit.  I expect the Backup Exec is going to be an MSDE instance, so http://web2.minasi.com/forum/topic.asp?TOPIC_ID=29650 should probably be the best starting point for learning reindexing on that.

SQL Server 2005/2008 (just in case): http://9montree.blogspot.com/2010/10/reindex-sql-server.html
0
 

Author Comment

by:lrbarrios
Comment Utility
I was leaning towards some kind of corruption/'crossed wires' too.  I'll give the links a shot.  Thanks.

On a related note... (and I'm just mentioning it because of the timing of your mentioning the SQL instance -- I may actually open a new question on EE today).  Yesterday, on my Backup Exec 12d, I was on the phone for 5 hours with Sophos (our anti-virus vendor) tech support.  I was trying to upgrade to the new version of their Enterprise Console and endpoint software (because the current version will retire at the end of the month).  It too uses an instance of SQL.  The upgrade said that I had to manually upgrade the SQL on the server (Windows 2003 R2) to either 2005 or 2008.  It mentioned that I could download and use the Express Editions for free.  (To be honest the whole SQL/MSDE thing is somewhat a mystery to me.  How one machine could have 3 different flavors for different products and still work always baffled me.)  Anyway... I didn't know what SQL it was currently running, but I wanted to go ahead and upgrade to SQL 2008 Express.  While going through the setup for that, I didn't see the SOPHOS instance in the drop-down.  I decided to call tech support.  He remoted to my machine.  Long story less long, we couldn't update to 2008 because of the existing 2005 that I had on there.  He suggested that I remove all the SQL stuff in Add/Remove Programs and start with a fresh install of Sophos (I would lose all of my Sophos configurations/policies/database etc.).  I agreed.  While he was blowing away SQL-related things in Add/Remove Programs and dumping SQL-related folders, I kept reminding him that that machine is also running Backup Exec and that it uses an instance of SQL also.  He assured me that everything woul d be put back in place after we successfully re-installed SQL 2005 Express.  We re-installed and got the new version of Sophos working.  Yeah!!  But now BE doesn't work anymore.  I can't even start the services.  The Event Viewer is talking about a missing service: MSSQL$BKUPEXEC.  That WAS my instance for BE that I was talking about.  Here's where I'm at... After all of the installing, removing, deleting folders and re-installing of SQL stuff, it looks like the first instance of the new SQL 2005 Express decided to use the same MSSQL.1\MSSQL\Data folder that contained my now-orphaned BE database.  I've created a new instance (which creates the MSSQL.2 folder tree).  The MSSQL.1 has, what looks like backup files of the database files (master, mastlog, model, modellog, msdbdata and msdblog$4idr).  I may be able to move them, rename them, and attach them to the new instance.  I'm not sure that will work though because the Date Modified of those files looks to be the date when I installed 12d on that server.  They may actually be empty-ish files and don't actually contain anything.  Which brings me to my next question... What is even in those files?  Jobs, Selection Lists, Templates, etc.?  Honestly, I only had 4 jobs (for 2 different servers) on that install of BE.  Worst case, I guess I could just start over.  Without re-installing BE, how can I get the database going so that I can at least start the BE services so that I can get into the program and then re-create everything.  Please don't feel obligated to answer these last questions.  I was just throwing them out there.  I'll be working on that today because I need to get it fixed before I go on a week-long vacation starting this weekend.
0
 
LVL 19

Expert Comment

by:Barthax
Comment Utility
I recognise that MSSQL$ style services are MSDE instances and not SQL Server 2000 - the old equivalent of Express version vs. full-blown in 2005/2008 (so, yeah, six different versions of the same product on the same machine?!).  As (based on my knowledge of SQL 2005) the DB is directory-based, I'd suggest trying the following:

- Stop the new instance of MSSQL.
- Move (keep the new in case it goes wrong) the new SQL files out of that directory.
- Copy (to preserve the originals) the old SQL files into that directory.
- Start the new instance of MSSQL.

In theory, that should give you the old data in the new instance, but I bet they're will be a sticky point I don't know about. :(
0
 

Author Comment

by:lrbarrios
Comment Utility
I think I'm going to be able to put this 'question' to rest.  You've been very helpful, but I think I've wandered from the original scope of the question and should close this one.  I've increased the point value from what I had originally set it at -- mostly due to your ongoing assistance with whatever else I threw into the mix.  I think I'm good now.  Thanks so much for the help!
0
 
LVL 19

Expert Comment

by:Barthax
Comment Utility
No problem.  Thanks for the increase in points.  Glad to have helped. :)
0

Featured Post

Backup Your Microsoft Windows Server®

Backup all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

Join & Write a Comment

Suggested Solutions

I previously wrote an article addressing the use of UBCD4WIN and SARDU. All are great, but I have always been an advocate of SARDU. Recently it was suggested that I go back and take a look at Easy2Boot in comparison.
Create your own, high-performance VM backup appliance by installing NAKIVO Backup & Replication directly onto a Synology NAS!
To efficiently enable the rotation of USB drives for backups, storage pools need to be created. This way no matter which USB drive is installed, the backups will successfully write without any administrative intervention. Multiple USB devices need t…
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now