Link to home
Start Free TrialLog in
Avatar of edform
edform

asked on

How can I test disk space and delete the oldest files in a set to make room for new backups?

I've recently had a nasty fright when NTBackup refused to read its own bkf files and at the same time I discovered that one of my clients needs the ability to restore files created on a particular day. We were able to get into the bkf files using a third party bkf recovery product, but the backup rotation was too coarse to allow a particular folder to be restored to the condition it was in one day in the middle of the previous week.

To resolve this problem I have moved over to using Disk Snapshot as the backup program; beginning by writing a full system backup to all 3 of the USB drives in the backup set and then automating the writing of a differential backup each night with the date of the write in the name of the file - eg C-20100407.sna. The capacity of the USB drives is such that I will be able to write upwards of twenty differentials on each of the 3 drives which theoretically allows me to store the system state for about the last 60 working days.

Recovering files from a given day in the past will be trivially straightforward - doubleclicking the correct differential file will open the Disk Snapshot program, combine the differential with the original full backup, and then map the resulting image as a virtual drive identical to the disk in question as it stood at the time the differential file was written - Disk Snapshot is a VSS aware program so the state of exchange files and SQL databases is also exactly replicated.

Now for the bit that has me foxed: I'd like to be able to test the free space on the USB drive before the backup procedure starts and, if necessary, delete the earliest differential backups to make room for the backup about to be written. Choosing which files to delete is complicated by the fact that each backup set may contain a dozen or more files - a typical differential fileset would start with c-20100407.sna and contain one or more other files with names in the form c-20100407.sn1, c-20100407.sn2 etc. The attached image file shows a directory containing a full backup of a C-drive and a single differential backup with a dated filename. After 20 or 30 iterations there will be hundreds of files in the backup directory, from which a choice will need to be made to delete everything older than files with names containing YYYYMMDD.

To complicate the matter, the approach can't really use a BAT/CMD file because they do not run with the machine logged off, but I've seen utilitites to turn bat/cmd files into exe files, so that approach might work.

Has anyone got any advice or pointers for how this can be done?
backup-directory.JPG
Avatar of Cris Hanna
Cris Hanna
Flag of United States of America image

When you say you're running NT Backup...were you running regular NT Backup or were you using the SBS 2003 Backup Wizard?
Avatar of edform
edform

ASKER

Hi CrisHanna
I was running the SBS2003 backup Wizard - it ran without problems for months and months with the backups gradually getting bigger - this client is expanding quickly - then last week we needd to get a folder back and the SBS2003 backup program couldn't read it's files. I tried all the usual stuff, like moving the bkf file onto an internal drive etc, but no joy. I eventually got into the file using Stellar Phoenix bkf Recovery but we didn't have a file available at the correct date so it didn't help - hence the desire to go to a version backup approach.
What is the max storage space on the server and what is the size of the USB drives, how many do you have?...when you ran the SBS wizard, how many previous backups are you telling it to store?    How far back is the customer wanting to keep backups?
Additionally are you familiar with "Previous Versions" which doesn't even require getting into the backup?
 
Avatar of edform

ASKER

Hi CrisHanna,
You said...
> What is the max storage space on the server and what is the size
> of the USB drives, how many do you have?
70Gig plus 220Gig in the server and 250 Gig on each of three backup drives. The current data and program load on the server is about 170Gig and it was producing 120Gig bkf files from the backup wizard. The full Snapshot backup is about 65Gig.
> ...when you ran the SBS wizard, how many previous backups are you
> telling it to store?    How far back is the customer wanting to keep backups?
Disk capacity in the three USB drives effectively limited us to 6 full backups - we used to have room for 12. We have now seen a data loss situation in which we explicitly needed the state of the system 10 days earlier and we had 1, 2, 3, 4, 11 and 19 days.
> Additionally are you familiar with "Previous Versions" which doesn't
> even require getting into the backup?
I was horrified to find that the shadow copy files system was not turned on for the 220Gig drive or there would have been no issue - it is on now.

So I want to be clear...you have 3 USB drives, each with a capacity of 250 GB
You programs and data on the server totaling 170GB but your bkf file is only 120GB?  Something's not adding up there
My first suggestion is buy new drives...storage is cheap...don't know how long you've had those drives, but drives can go bad as well.  And typical rule of thumb, your backup destination drive needs to be as big as your total storage.
You can get 2TB external Drives for 139.00 http://www.buy.com/prod/caxm-series-caxm3702t0-hard-drive-2tb-7200rpm-usb-usb-serial-ata-300/q/loc/101/212446688.html
I've used the 1TB version of these drives for several customers...they just work flawlessly and we get successful backups every nice and have never had a restore issue.
Avatar of edform

ASKER

You are correct - 3 USB drives of 250Gig each.
I just logged on to the system and checked the dataload and its actually only 120Gigs - so the last bkf file at 122Gigs was bigger than the data it contained?
The disks are perfect with no read or write issues of any kind and using a version backup approach I can store something like 60days worth of exact backups.
I'm intrigued by your comment about no restore issues because the web is filled with comments on the dreaded unreadable bkf file - I did not know about this problem and had, in the past, restored vital data several times, but bkf files which the SBS backup wizard cannot read are apparently commonplace.
When I used Stellar Phoenix bkf Recovery it read the files quickly and found no issues with them, so the problem seems to be in SBS2003 Backup. I also wonder what the point is in producing a gigantic single file the same size as or bigger than the data it contains when a simple copy, one file at a time would occupie the same space and *always* be readable.
The change I have made to Disk Snapshot is set in stone: I've used it for years as a disk cloning system and as primary backup and I have never had even a hint of problems - the virtual drives are usually quicker than the original physical drive they represent - it just works. My problem is how to implement the FIFO automatic deletion of the earliest backup sets.
I can certainly ask the moderators to move this to a difference zone as your need is not really about an SBS issue.  You're looking to automate a process. Not fix an SBS issue  Have you looked at the Disk Snap Shot forums, perhaps someone there has done something similar
What you need is a script (command or bat) which could be scheduled to run without human intervention
I do something like that in Unix to clean up file exports
###############################################
# this job will delete files in /app01/workbrain/payroll_export/wbp/archive
# that are 360 days old or older.
################################################
cd /app01/workbrain/payroll_export/wbp/archive
find /app01/workbrain/payroll_export/wbp/archive -mtime +360 -exec rm {} \;
 
ASKER CERTIFIED SOLUTION
Avatar of gemarti
gemarti
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
... and to call your backup program.
You may want to look at the Forfiles command to delete old backup files. This can be run directly from a scheduled task or a batch file. You can tell scheduled tasks to run if not logged in. You just have to have a userid and password on the task and make sure that the password is updated or doesn't expire.

Or you can use the TweakUI to re-log in after a reboot.


http://technet.microsoft.com/en-us/library/cc755872%28WS.10%29.aspx
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Just a comment on:
> I just logged on to the system and checked the dataload and its actually only 120Gigs -
> so the last bkf file at 122Gigs was bigger than the data it contained?

Unless you're using some sort of compression (which tape drives do in HW at no cost, or which
in a disk drive would have to be done in software, probably slowing down your backup) this is
expected.

Think about it this way -- for each file, you not only have to store the contents of the file, you have to
store the file's metadata: its name, its complete path, all of its access information (create, last
modified, last accessed) and all of the owner and permissions information.   In your case, this
is adding up to about 2% of the data itself.  With the preponderance of tiny files on most Windows
servers, this sounds pretty normal to me.
SelfGovern has a point. We had an app the would store files in CD/DVD sized folders. In some of the super-directories (many (50+) subdirecties would show 15GB actual size but 16GB Size on disk.
Size-On-Disk.jpg
Avatar of edform

ASKER

Comments for all contributors.
Thanks to all for the cocnsideration given to my situation: what would I do without Experts Exchange?
It looks as though I will be accepting both Gemarti's and markdmac's suggestions and sharing the points - Gemarti's because he has shown me how to run bat/cmd files while logged off and markdmac's because the code is close to what I want.
I need to test the two suggestions first which I will do on Monday evening.
Question for markdmac: what language is that code fragment written in? It's not MS CMD language because that has no DIM statement - Visual basic?
On the general discussion of MS Backup: my point in questioning the way it makes backups wasn't really the increase in size over the raw files but the fact that the program breaks down when its own backup files: grow too large - are on external media - etc etc. If the backup was a string of individual filecopies plus a set of files to define the partition and the directory structure there might not be such a problem.
I should also add that Disk Snapshot produces a backup only just over half the size in less than half the time, and, because its a sector copy, it can be loaded as a disk drive to allow drag and drop restore.
That looks to be a vbscript file. Save it as filenam.vbs and then use the Cshell command to run it.
So you would enter:

cmd.exe /c "cscript markdmac.vbs"

Hope all works out. I'm going to look into disk snapshot for around the home; never heard of it but it looks useful.

Thanks.
Yes, it is a vbscript.  No need to run it from the command line.  You can just double click it.  Change the drive letter and the number of days (20 is set in it now) of how old you want to delete the files.

Also note that I am checking for less than 10% of drive space here:
 'Check if 10% disk space is free
                If objItem.FreeSpace /1024 /1024 \1024 < ((objItem.Capacity /1024 /1024 \1024)*.1) Then


You can set that to 20 percent by changing .1 to .2, or 15% with .15 etc.

If you want to do a dry run and just see what files will be deleted then do run it from a command prompt with CSCRIPT and you can change this line:
    oFile.Delete True
to this:
    Wscript.Echo "Will delete :" & oFile.Name

Happy scripting.
markdmac:
edform will need to run from a cmd line since the script will be running at times when the user is logged off; this will be a scheduled task.

Good script.

Avatar of edform

ASKER

Hi markdmac,
I tried the script with 4 modifications...
Replaced both J: with Z: because I have a permanetly online Z: drive.
Replaced your 10% [ .1] with 80% [.8] because the disk is actually 74% empty.
Replaced  oFile.Delete True with Wscript.Echo "Will delete :" & oFile.Name
I saved the modified script as dryrun.vbs, CD'd to the correct directory and then opened a command prompt and typed 'cscript dryrun.vbs'. This gave me the prompt back immediately, so I changed the command line to 'cscript dryrun.vbs //x'.
This time I got an error message external to the command window which I've attached as an image.
This left me at a basic prompt and I had to use CTRL-C to get back to the command prompt.
Any ideas?
 

error.JPG
Please post the full modified code you are using so I can make sure there are not any problems due to copy/paste.

Avatar of edform

ASKER

The attached image shows the code.

dryrunvbs.JPG
You added the word PAUSE, that is not a VBScript command.  Please run the code as provided and tell me if you encounter any errors.
Avatar of edform

ASKER

I didn't notice that I had left the word 'pause' in; the error refers to the code without the 'pause' command. I put 'pause' in there to try to run the script a bit at a time and identify where the actual problem is.
I just edited the script again to remove 'pause' and got precisely the same error.
I also just tried to run it in an Elevated Command Window and again the same result.
It would appear you have set a debugger since your screen shot shows it is using Visual Studio.  Normally you would get a line number telling you where there was a problem.  Can you remove the link to the debugger?

Also can you please post the code you are using.  The screen shot is difficult to troubleshoot as I cannot identify extra spaces etc.
Avatar of edform

ASKER

Here is the code - hope it shows correctly, I've never tried to the use the attach code button before.
The debugger is engaged by using the //x switch to try to see what is happening.
If I simply run the thing with the command 'cscript dryrun.vbs' it terminates immediately and returns me to the prompt.

On Error Resume Next 
Dim fso ,oFolder, oFile, strComputer, objWMIService, colItems, objItem 
 
Set fso = createobject("Scripting.FileSystemObject") 
 
strComputer = "." 
Path1 = "Z:\" 
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2") 
Set colItems = objWMIService.ExecQuery("Select * from Win32_Volume",,48) 
For Each objItem in colItems 
        If objItem.DriveLetter = "Z:" Then 
                'Check if 10% disk space is free 
                If objItem.FreeSpace /1024 /1024 \1024 < ((objItem.Capacity /1024 /1024 \1024)*.8) Then 
                        Set oFolder = fso.GetFolder(Path1) 
                        For Each oFile In oFolder.Files 
                                'Delete files older than 20 days 
                        If DateDiff("D", oFile.DateCreated, Now) > 20 Then 
                                Wscript.Echo "Will delete :" & oFile.Name 
                        End If 
                        Next 
        End If 
    End If 
Next

Open in new window

Umm...I've been watching this thread and was waiting for markdmac to fix his code.


Here is some code to use  (Note: you would be wise to test before putting into production service; I cannot be responsible for the outcome of this script):
+++++++++++++++++++++++++++++++++++++++++++++++++++++
Dim fso, oFolder, oFile, strComputer, objWMIService, colItems, objItem

Set fso = CreateObject("Scripting.FileSystemObject")

strComputer = "."
Set objWMIService = GetObject("winmgmts:" _
    & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set colDisks = objWMIService.ExecQuery _
    ("Select * from Win32_LogicalDisk Where DriveType = 1")
For Each objDisk in colDisks
    intFreeSpace = objDisk.FreeSpace
    intTotalSpace = objDisk.Size
    pctFreeSpace = intFreeSpace / intTotalSpace
    Wscript.Echo objDisk.DeviceID, FormatPercent(pctFreeSpace)

    If intFreeSpace < (intTotalSpace * . 2) then
        Set oFolder = fso.GetFolder("Z:\")
        For Each oFile in oFolder.Files
        If DateDiff("d", oFile.DateCreated,Now) > 20 then
              Wscript.Echo "Delete : " & oFile.Name & " " & oFile.DateCreated
              oFile.DeleteFile oFile.Name
        End If
        Next
    end if
   
Next
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In reference to DriveType see here: http://ns7.webmasters.com/caspdoc/html/vbscript_drivetype_constants.h

Note too that in the line where it says fso.GetFolder("Z:\") you should put the appropriate folder location as I don't know the drive letters or folders where you are storing your backups.

Germarti, the code you posted will compare the percentage on each drive of type 1 and take action on the Z drive.

In my code I first verify the drive letter, then do the comparisons.

It would appear you have recycled most of my logic but I think this is a crucial part to be missing.

Having another disk with less space should not trigger a deletion on the backup drive if it is not needed.

I've reviewed my code and don't see anything wrong with it.  If you do then please comment on what you see since you mentioned you have been waiting for me to fix my code.
objItem.FreeSpace /1024 /1024 \1024 < ((objItem.Capacity /1024 /1024 \1024)

Should be:

objItem.FreeSpace /1024 /1024 /1024 < ((objItem.Capacity /1024 /1024 /1024)
DateDiff("D", oFile.DateCreated, Now)

Should be:

DateDiff("d", oFile.DateCreated, Now)

Ref: http://www.w3schools.com/vbScript/func_datediff.asp
Thanks you are correct on the DateDiff.  Incorrect on the math, that is intentional to provide rounding of the GB.
Avatar of edform

ASKER

OK!
I've made the 2 alterations gemarti suggested and the code still falls over. If I run it with the debugger switch set it gives the same error I already posted.
Then I set to commenting out all of the logic and then removing the comments on each loop in turn. I've now reached the stage shown in the attached code and the thing still runs. I know the logic is doing something because I get 9 echoes of 'will delete:' after removing the comments from one of the loops and only 2 echoes with the code as attached.
If I remove the comment from the outer loop...
'If objItem.DriveLetter = "Z:" Then
'end if
The code falls over and switching in the debugger shows the same error I first posted.
Something seems to be wrong with that loop.

On Error Resume Next 
Dim fso ,oFolder, oFile, strComputer, objWMIService, colItems, objItem 
 
Set fso = createobject("Scripting.FileSystemObject") 
 
strComputer = "." 
Path1 = "Z:\" 
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\cimv2") 
Set colItems = objWMIService.ExecQuery("Select * from Win32_Volume",,48) 
For Each objItem in colItems 
        'If objItem.DriveLetter = "Z:" Then 
                'Check if 10% disk space is free 
                If objItem.FreeSpace /1024 /1024 /1024 < ((objItem.Capacity /1024 /1024 /1024)*.8) Then 
                        Set oFolder = fso.GetFolder(Path1) 
                        For Each oFile In oFolder.Files 
                                'Delete files older than 20 days 
                        If DateDiff("d", oFile.DateCreated, Now) > 20 Then 
                                Wscript.Echo "Will delete :" '& oFile.Name 
                        End If 
                        Next 
        End If 
    'End If 
Next

Open in new window

Avatar of edform

ASKER

I've just seen markdmac's comment on his mathematics and changing that line back to his version [/1024 /1024 \1024] worked correctly. I also took the comment off the & ofile.name bit and the result was a single echo telling me that the system would delete snapshot.exe, the only file on the disk older than 20 days. With gemarti's code that echo was repeated twice.
Interestingly, the code has actually returned the exact result needed *without* the loop that is commented out even though both of the disks int he system are less than 80% empty. Why is that loop there?
Avatar of edform

ASKER

I've split the points between two excellent experts. The discussion between them gave me enough detail to see how my task can be done. I haven't ever touched Visual Basic Scripting before, so the learning curve was steep, but I'm very happy that I can automate a FIFO disk space test and clear solution from the information supplied.
Thank you all!
Thanks ....glad to help

Why is that loop there?
The script searches the folder for every file greater than 20 days old.

oops hit submit too soon
...so if there are more than one file in the folder you should see all of them.

Avatar of edform

ASKER

Refering to the code listing in my message timed at 12:18, I have commented out lines 11 and 22 which are an 'if/end if' pair. The code runs perfectly without that pair of lines.
I just altered the path in line 7 to Z:\My Music and ran the script again. It listed 121 files it would delete if I let it. So the code as it stands operates on the folder the path points it to and deletes multiple files quite happily based on their age.
I cannot for the life of me see what lines 11 and 22 are for. What I do know is, if I uncomment those two lines the code falls over with the exception I attached to a message yesterday.
That If/End If pair are to ensure that the drive being evaluated is the target drive.  Without it you could have multiple drives detected and if the drive being evaluated is low on disk space (even if it is not the backup drive you want to keep clean) it will tell the script to delete files on the backup drive.

In other words:

Drive Y Size 50GB with free space of 1GB
Drive Z Size 95GB with free space of 30GB

When the script enumerated drives it sees the free space on Y is less than 10% so it says "need to free up space" so it binds to the Z drive and deletes files even though there is more than sufficient space.  With the conditional code in place it enumerates the drives and doesn't check for free space UNLESS it is the target drive.
Avatar of edform

ASKER

Ok! I've gone over the code I've been given and come up with a script that does exactly what I want.
Apart from the fact that the test for Drive Z causes the code to fall over, for which I am totally unable to determine a cause, the biggest problem was the use of the property 'objItem.Capacity' in gemarti's original file. There is no property called 'Capacity' in VB Scripting; the correct property is 'TotalSize'
In fact I do not need to do any disk choosing. The disk I need to interrogate for free space is always the same drive letter, so I have been able to eliminate a lot of the code and simply set the drive and folder with explicit commands.
In the script the listed below, the variables that can be set are disk letter - called 'odisk', folder path - called 'ofolder',  the percentage of free space to test for, which, in my test case is set as '.8' for 80% in the expression '((odrive.TotalSize /1024 /1024 \1024)*.8)' in line 12, and the age of the files to be deleted in days since their creation date which is set in the expression '>20' in line 15.

On Error Resume Next

Dim fso, odrive, ofolder, ofile

Set fso = createobject("Scripting.FileSystemObject")

Set odrive = fso.GetDrive("z:")

Set ofolder = fso.getFolder("Z:\My Music")

'Check if 80% disk space is free 
If odrive.FreeSpace /1024 /1024 \1024 < ((odrive.TotalSize /1024 /1024 \1024)*.8) Then
	For Each oFile In oFolder.Files 
		'List files older than 20 days 
			If DateDiff("d", oFile.DateCreated, Now) > 20 Then 
				oFile.Delete True
			End If 
	Next 
End If

Open in new window

"...the biggest problem was the use of the property 'objItem.Capacity' in gemarti's original file..."

That's actually markdmac's code :-) No problem though. I think I was using objItem.Size

Anyway its all good...glad you got this going.

Just for the record, objItem.Capacity is valid.  It is new with Windows 7 and I am running Windows 7 x64 to develop on.  Sorry I did not take that into account.  Supporting documentation for Win32_Volume class from Microsoft is here: http://msdn.microsoft.com/en-us/library/aa394515%28VS.85%29.aspx.
Avatar of edform

ASKER

I ran all of my tests on Windows 7 Pro 32 bit and the property 'Capacity' does not work.
Anyway, the end result was just what the doctor ordered and I hope lots of folks find this thread and benefit from the great help all of you guys gave.
From the link:

Capacity

    Data type: uint64
    Access type: Read-only

    Size of the volume in bytes.

It is an x64 implementation.