asked on

Script to calculate backup size?

Hi All,

I have a large directory with 2000 folders in it.

I back this directory up to 3 usb harddrives. The only problem is that I have to manually calculate how many folders from the directory will fit on each usb disk. I simply do this by highlighting folders Alpha - Foxtrot then right clicking and clicking properties. I can then see the size and if it will fit I can copy those folders to the 1st disk. This is a pain as a lot of it is trial and error.

Is there some way of scripting this so that I am prompted for:
a)The drive to be backed up
b)The size of the backup disk

Then having an output that states 'for <backup disk mentioned above> you can backup folders Alpha - Golf from <drive mentioned above>'

oBdA

Try the following Powershell script; you can pass it up to three parameters:
-Path: The root path to be backed up; default: current directory.
-Destination: The drive to backup to (free space will be retrieved automatically)
-SkipUntil: Skip folders (inside the root path) up until and including this name; this is so that you can continue from a previous backup.
Example:
Plug in the first drive, run
.\Whatever.ps1 -Path D: -Destination X:
Assuming this would return "Alpha, Bravo Charlie", you can backup, then plug in the next drive and run
.\Whatever.ps1 -Path D: -Destination X: -SkipUntil Charlie
If all backup drives can be plugged in at the same time, the script could of course be changed to return a list of which folders to copy to which drives.

Param (
	[Parameter(Mandatory=$False)][ValidateNotNull()]
	[string]$Path = (Get-Location -PSProvider FileSystem).Path,
	[Parameter(Mandatory=$True)][ValidateNotNull()]
	[string]$Destination,
	[Parameter(Mandatory=$False)]
	[string]$SkipUntil = ""
)

Function  Get-Subdirectories {
Param (
	[string]$Path = (Get-Location -PSProvider FileSystem).Path,
	[string]$SkipUntil = $Script:SkipUntil,
	[uint64]$Limit = 0
)
	$Results = @()
	$ChildItems = Get-ChildItem -Path $Path | Where-Object {$_.PSIsContainer}
	If ([string]::IsNullOrEmpty($Script:SkipUntil)) {
		$CollectData = $True
	} Else {
		If (($ChildItems | Select-Object -ExpandProperty Name) -NotContains $SkipUntil) {
			"'Skip until' folder '$Script:SkipUntil' not found in folder list:" | Write-Host -Fore Red
			($ChildItems | Select-Object -ExpandProperty Name) | Write-Host -Fore Red
			Return $Null
		}
		$CollectData = $False
	}
	$Index = 0
	$TotalSize = 0
	Write-Progress -Activity "Collecting size information" -PercentComplete 0 -SecondsRemaining -1
	ForEach ($Item In $ChildItems) {
		$Index += 1
		Write-Progress -Activity "Collecting size information" -CurrentOperation $Item.FullName -PercentComplete ((100 * $Index) / $ChildItems.Count) -SecondsRemaining -1
		If ($CollectData) {
			[uint64]$DirectorySize = (Get-ChildItem -Path $Item.FullName -Recurse | Measure-Object -Property Length -Sum).Sum
			$TotalSize += $DirectorySize
			If (($Limit -eq 0) -Or ($TotalSize -lt $Limit)) {

			$Results += $Item | Select-Object -Property *,
				@{Name="TotalSize"; Expression={[uint64]$DirectorySize}},
				@{Name="TotalSizeKB"; Expression={[uint64]($DirectorySize / 1KB)}},
				@{Name="TotalSizeMB"; Expression={[uint64]($DirectorySize / 1MB)}},
				@{Name="TotalSizeGB"; Expression={[uint64]($DirectorySize / 1GB)}},
				@{Name="TotalSizeTB"; Expression={[uint64]($DirectorySize / 1TB)}}
			} Else {
				Break
			}
		} Else {
			If ($Item.Name -eq $Script:SkipUntil) {
				$CollectData = $True
			}
		}
	}
	Write-Progress -Activity "Collecting size information" -Completed
	Return $Results
}

$Drive = (Split-Path -Path $Destination -Qualifier)
If (-Not (Get-PSDrive -Name $Drive.Trim(":"))) {
	"Destination drive not found: '$Drive'." | Write-Error
	Exit 1
}
$DestinationWmi = Get-WmiObject -Query "SELECT * FROM WIN32_LogicalDisk WHERE DeviceID='$Drive'"
$BackupSpace = $DestinationWmi.FreeSpace
"Destination drive information:" | Write-Host
$DestinationWmi | Select DeviceID, VolumeName, @{Name="FreeSpace GB"; Expression={[uint64]($_.FreeSpace / 1GB)}}, @{Name="Size GB"; Expression={[uint64]($_.Size / 1GB)}} | Format-Table -AutoSize | Out-String | Write-Host

$Results = Get-Subdirectories -Path $Path -Limit $BackupSpace | Sort-Object -Property Name
$BackupTotalSize = ($Results | Measure-Object -Property TotalSize -Sum).Sum

"Backup source:      '$($Path)'" | Write-Host -Fore White
"Backup total size:  {0,9:N2} GB" -f ($BackupTotalSize / 1GB) | Write-Host -Fore White
"Backup destination: '$($Drive)'" | Write-Host -Fore Yellow
"Backup free space:  {0,9:N2} GB" -f ($BackupSpace / 1GB) | Write-Host -Fore Yellow
If ($Results.Count -gt 0) {
	"Folders fitting:" | Write-Host -Fore Green
	$Results | Select-Object FullName, TotalSizeGB | Format-Table -AutoSize | Out-String | Write-Host -Fore Green
	$Results
} Else {
	"Not enough free space on the backup drive, or argument error!" | Write-Error
}

Open in new window

Edit: changed error output.

robclarke41

ASKER

Thank you very much for your reply. After I run the script I get the following:

DeviceID VolumeName FreeSpace GB Size GB
-------- ---------- ------------ -------
J:       NAS1a                50    2795




cmdlet Write-Progress at command pipeline position 1
Supply values for the following parameters:
(Type !? for Help.)
Status:

Open in new window

SOLUTION

oBdA

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

robclarke41

ASKER

Yes I'm on version 2.0, guess this is the problem?

oBdA

Yes; the script above should fix the "Write-Progress" issue. Below is an updated version that fixes a potential issue in PS2.0 when there are no subfolders in the root folder specified.
PS2.0 has a really serious bug in that it will enter a "ForEach" loop even if the expression is empty; try this in PS2.0:

ForEach ($foo In $Null) {"Bar"}

Open in new window

Param (
	[Parameter(Mandatory=$False)][ValidateNotNull()]
	[string]$Path = (Get-Location -PSProvider FileSystem).Path,
	[Parameter(Mandatory=$True)][ValidateNotNull()]
	[string]$Destination,
	[Parameter(Mandatory=$False)]
	[string]$SkipUntil = ""
)

Function  Get-Subdirectories {
Param (
	[string]$Path = (Get-Location -PSProvider FileSystem).Path,
	[string]$SkipUntil = $Script:SkipUntil,
	[uint64]$Limit = 0
)
	$Results = @()
	$ChildItems = Get-ChildItem -Path $Path | Where-Object {$_.PSIsContainer}
	If ([string]::IsNullOrEmpty($Script:SkipUntil)) {
		$CollectData = $True
	} Else {
		If (($ChildItems | Select-Object -ExpandProperty Name) -NotContains $SkipUntil) {
			"'Skip until' folder '$Script:SkipUntil' not found in folder list:" | Write-Host -Fore Red
			($ChildItems | Select-Object -ExpandProperty Name) | Write-Host -Fore Red
			Return $Null
		}
		$CollectData = $False
	}
	$Index = 0
	$TotalSize = 0
	Write-Progress -Status "Scanning directory tree" -Activity "Collecting size information" -PercentComplete 0 -SecondsRemaining -1
	If ($ChildItems -ne $Null) {
		ForEach ($Item In $ChildItems) {
			$Index += 1
			Write-Progress -Status "Scanning directory tree" -Activity "Collecting size information" -CurrentOperation $Item.FullName -PercentComplete ((100 * $Index) / $ChildItems.Count) -SecondsRemaining -1
			If ($CollectData) {
				[uint64]$DirectorySize = (Get-ChildItem -Path $Item.FullName -Recurse | Measure-Object -Property Length -Sum).Sum
				$TotalSize += $DirectorySize
				If (($Limit -eq 0) -Or ($TotalSize -lt $Limit)) {

				$Results += $Item | Select-Object -Property *,
					@{Name="TotalSize"; Expression={[uint64]$DirectorySize}},
					@{Name="TotalSizeKB"; Expression={[uint64]($DirectorySize / 1KB)}},
					@{Name="TotalSizeMB"; Expression={[uint64]($DirectorySize / 1MB)}},
					@{Name="TotalSizeGB"; Expression={[uint64]($DirectorySize / 1GB)}},
					@{Name="TotalSizeTB"; Expression={[uint64]($DirectorySize / 1TB)}}
				} Else {
					Break
				}
			} Else {
				If ($Item.Name -eq $Script:SkipUntil) {
					$CollectData = $True
				}
			}
		}
	}
	Write-Progress -Status "Scanning directory tree" -Activity "Collecting size information" -Completed
	Return $Results
}

$Drive = (Split-Path -Path $Destination -Qualifier)
If (-Not (Get-PSDrive -Name $Drive.Trim(":"))) {
	"Destination drive not found: '$Drive'." | Write-Error
	Exit 1
}
$DestinationWmi = Get-WmiObject -Query "SELECT * FROM WIN32_LogicalDisk WHERE DeviceID='$Drive'"
$BackupSpace = $DestinationWmi.FreeSpace
"Destination drive information:" | Write-Host
$DestinationWmi | Select DeviceID, VolumeName, @{Name="FreeSpace GB"; Expression={[uint64]($_.FreeSpace / 1GB)}}, @{Name="Size GB"; Expression={[uint64]($_.Size / 1GB)}} | Format-Table -AutoSize | Out-String | Write-Host

$Results = Get-Subdirectories -Path $Path -Limit $BackupSpace | Sort-Object -Property Name
$BackupTotalSize = ($Results | Measure-Object -Property TotalSize -Sum).Sum

"Backup source:      '$($Path)'" | Write-Host -Fore White
"Backup total size:  {0,9:N2} GB" -f ($BackupTotalSize / 1GB) | Write-Host -Fore White
"Backup destination: '$($Drive)'" | Write-Host -Fore Yellow
"Backup free space:  {0,9:N2} GB" -f ($BackupSpace / 1GB) | Write-Host -Fore Yellow
If ($Results.Count -gt 0) {
	"Folders fitting:" | Write-Host -Fore Green
	$Results | Select-Object FullName, TotalSizeGB | Format-Table -AutoSize | Out-String | Write-Host -Fore Green
	$Results
} Else {
	"Not enough free space on the backup drive, or argument error!" | Write-Error
}

Open in new window

oBdA

Oh, and I forgot to mention: the script will not only output to the screen, but return an array of objects with the folders in the list in case you want to pass that along to another script.
If you don't need this, either assign a variable, pipe the output through Out-Null, or comment line 80.
$Folders = .\Whatever.ps1 -Path D: -Destination X:
.\Whatever.ps1 -Path D: -Destination X: | Out-Null

robclarke41

ASKER

The script looks great so far. Instead of scanning the free disk space though I just wanted a report of what would fit based on the entire disk as if it were empty (In order of folder name). Is this possible?

oBdA

In line 66, "$BackupSpace = $DestinationWmi.FreeSpace", replace the ".FreeSpace" with ".Size":

$BackupSpace = $DestinationWmi.Size

Open in new window

This will use the full disk size as reported by WMI.

Gerald Connolly

so is the 2.8TB the total size you are trying to backup?

robclarke41

ASKER

Looks like we're still a little off the mark, I now get:

DeviceID VolumeName FreeSpace GB Size GB
-------- ---------- ------------ -------
J:       NAS2a                31    2795



Backup source:      'X:'
Backup total size:      29.67 GB
Backup destination: 'J:'
Backup free space:      30.88 GB
Folders fitting:

Open in new window

The backup total size for X: including subfolders should be 5.40TB

I was then expecting to see a list of all the folders (alphabetically) that I could fit on drive J: from drive X:

oBdA

Sorry, always tested this with a trailing backslash. This takes care of that and will now by default use the backup drive size; you can use the -UseFreeSpace switch to use only the free space:

Param (
	[Parameter(Mandatory=$False)][ValidateNotNull()]
	[string]$Path = (Get-Location -PSProvider FileSystem).Path,
	[Parameter(Mandatory=$True)][ValidateNotNull()]
	[string]$Destination,
	[Parameter(Mandatory=$False)]
	[string]$SkipUntil = "",
	[switch]$UseFreeSpace
)

Function  Get-Subdirectories {
Param (
	[string]$Path = (Get-Location -PSProvider FileSystem).Path,
	[string]$SkipUntil = $Script:SkipUntil,
	[uint64]$Limit = 0
)
	$Results = @()
	$Path = Join-Path $Path "\"
	$ChildItems = Get-ChildItem -Path $Path | Where-Object {$_.PSIsContainer}
	If ([string]::IsNullOrEmpty($Script:SkipUntil)) {
		$CollectData = $True
	} Else {
		If (($ChildItems | Select-Object -ExpandProperty Name) -NotContains $SkipUntil) {
			"'Skip until' folder '$Script:SkipUntil' not found in folder list:" | Write-Host -Fore Red
			($ChildItems | Select-Object -ExpandProperty Name) | Write-Host -Fore Red
			Return $Null
		}
		$CollectData = $False
	}
	$Index = 0
	$TotalSize = 0
	Write-Progress -Status "Scanning directory tree" -Activity "Collecting size information" -PercentComplete 0 -SecondsRemaining -1
	If ($ChildItems -ne $Null) {
		ForEach ($Item In $ChildItems) {
			$Index += 1
			Write-Progress -Status "Scanning directory tree" -Activity "Collecting size information" -CurrentOperation $Item.FullName -PercentComplete ((100 * $Index) / $ChildItems.Count) -SecondsRemaining -1
			If ($CollectData) {
				[uint64]$DirectorySize = (Get-ChildItem -Path $Item.FullName -Recurse | Measure-Object -Property Length -Sum).Sum
				$TotalSize += $DirectorySize
				If (($Limit -eq 0) -Or ($TotalSize -lt $Limit)) {

				$Results += $Item | Select-Object -Property *,
					@{Name="TotalSize"; Expression={[uint64]$DirectorySize}},
					@{Name="TotalSizeKB"; Expression={[uint64]($DirectorySize / 1KB)}},
					@{Name="TotalSizeMB"; Expression={[uint64]($DirectorySize / 1MB)}},
					@{Name="TotalSizeGB"; Expression={[uint64]($DirectorySize / 1GB)}},
					@{Name="TotalSizeTB"; Expression={[uint64]($DirectorySize / 1TB)}}
				} Else {
					Break
				}
			} Else {
				If ($Item.Name -eq $Script:SkipUntil) {
					$CollectData = $True
				}
			}
		}
	}
	Write-Progress -Status "Scanning directory tree" -Activity "Collecting size information" -Completed
	Return $Results
}

$Drive = (Split-Path -Path $Destination -Qualifier)
If (-Not (Get-PSDrive -Name $Drive.Trim(":"))) {
	"Destination drive not found: '$Drive'." | Write-Error
	Exit 1
}
$DestinationWmi = Get-WmiObject -Query "SELECT * FROM WIN32_LogicalDisk WHERE DeviceID='$Drive'"
If ($UseFreeSpace) {
	$BackupSpace = $DestinationWmi.FreeSpace # - 32000000000
} Else {
	$BackupSpace = $DestinationWmi.Size
}
"Destination drive information:" | Write-Host
$DestinationWmi | Select DeviceID, VolumeName, @{Name="FreeSpace GB"; Expression={[uint64]($_.FreeSpace / 1GB)}}, @{Name="Size GB"; Expression={[uint64]($_.Size / 1GB)}} | Format-Table -AutoSize | Out-String | Write-Host

$Results = Get-Subdirectories -Path $Path -Limit $BackupSpace | Sort-Object -Property Name
$BackupTotalSize = ($Results | Measure-Object -Property TotalSize -Sum).Sum

"Backup source:      '$($Path)'" | Write-Host -Fore White
"Backup total size:  {0,9:N2} GB" -f ($BackupTotalSize / 1GB) | Write-Host -Fore White
"Backup destination: '$($Drive)'" | Write-Host -Fore Yellow
"Backup space:       {0,9:N2} GB" -f ($BackupSpace / 1GB) | Write-Host -Fore Yellow
If ($Results.Count -gt 0) {
	"Folders fitting:" | Write-Host -Fore Green
	$Results | Select-Object FullName, TotalSizeGB | Format-Table -AutoSize | Out-String | Write-Host -Fore Green
	$Results
} Else {
	"Not enough free space on the backup drive, or argument error!" | Write-Error
}

Open in new window

robclarke41

ASKER

Nearly there I think, only problem now is that I get so much info like this:

Extension         :
CreationTime      : 24/08/2012 07:09:23
CreationTimeUtc   : 24/08/2012 06:09:23
LastAccessTime    : 24/08/2012 07:09:23
LastAccessTimeUtc : 24/08/2012 06:09:23
LastWriteTime     : 24/08/2012 07:22:01
LastWriteTimeUtc  : 24/08/2012 06:22:01
Attributes        : Directory
TotalSize         : 5778158890
TotalSizeKB       : 5642733
TotalSizeMB       : 5510
TotalSizeGB       : 5
TotalSizeTB       : 0

PSPath            : Microsoft.PowerShell.Core\FileSystem::W:\
PSParentPath      : Microsoft.PowerShell.Core\FileSystem::W:\

Open in new window

That I can't even see the green output data I need as it passes the buffer size! Is there anyway to hide all this extra stuff?

oBdA

Yes, see http:#a40208549 (line 80 wandered down to 86).
Or you can use this, which now only returns the directory items if "-PassThru" is specified:

Param (
	[Parameter(Mandatory=$False)][ValidateNotNull()]
	[string]$Path = (Get-Location -PSProvider FileSystem).Path,
	[Parameter(Mandatory=$True)][ValidateNotNull()]
	[string]$Destination,
	[Parameter(Mandatory=$False)]
	[string]$SkipUntil = "",
	[switch]$UseFreeSpace,
	[switch]$PassThru
)

Function  Get-Subdirectories {
Param (
	[string]$Path = (Get-Location -PSProvider FileSystem).Path,
	[string]$SkipUntil = $Script:SkipUntil,
	[uint64]$Limit = 0
)
	$Results = @()
	$Path = Join-Path $Path "\"
	$ChildItems = Get-ChildItem -Path $Path | Where-Object {$_.PSIsContainer}
	If ([string]::IsNullOrEmpty($Script:SkipUntil)) {
		$CollectData = $True
	} Else {
		If (($ChildItems | Select-Object -ExpandProperty Name) -NotContains $SkipUntil) {
			"'Skip until' folder '$Script:SkipUntil' not found in folder list:" | Write-Host -Fore Red
			($ChildItems | Select-Object -ExpandProperty Name) | Write-Host -Fore Red
			Return $Null
		}
		$CollectData = $False
	}
	$Index = 0
	$TotalSize = 0
	Write-Progress -Status "Scanning directory tree" -Activity "Collecting size information" -PercentComplete 0 -SecondsRemaining -1
	If ($ChildItems -ne $Null) {
		ForEach ($Item In $ChildItems) {
			$Index += 1
			Write-Progress -Status "Scanning directory tree" -Activity "Collecting size information" -CurrentOperation $Item.FullName -PercentComplete ((100 * $Index) / $ChildItems.Count) -SecondsRemaining -1
			If ($CollectData) {
				[uint64]$DirectorySize = (Get-ChildItem -Path $Item.FullName -Recurse | Measure-Object -Property Length -Sum).Sum
				$TotalSize += $DirectorySize
				If (($Limit -eq 0) -Or ($TotalSize -lt $Limit)) {

				$Results += $Item | Select-Object -Property *,
					@{Name="TotalSize"; Expression={[uint64]$DirectorySize}},
					@{Name="TotalSizeKB"; Expression={[uint64]($DirectorySize / 1KB)}},
					@{Name="TotalSizeMB"; Expression={[uint64]($DirectorySize / 1MB)}},
					@{Name="TotalSizeGB"; Expression={[uint64]($DirectorySize / 1GB)}},
					@{Name="TotalSizeTB"; Expression={[uint64]($DirectorySize / 1TB)}}
				} Else {
					Break
				}
			} Else {
				If ($Item.Name -eq $Script:SkipUntil) {
					$CollectData = $True
				}
			}
		}
	}
	Write-Progress -Status "Scanning directory tree" -Activity "Collecting size information" -Completed
	Return $Results
}

$Drive = (Split-Path -Path $Destination -Qualifier)
If (-Not (Get-PSDrive -Name $Drive.Trim(":"))) {
	"Destination drive not found: '$Drive'." | Write-Error
	Exit 1
}
$DestinationWmi = Get-WmiObject -Query "SELECT * FROM WIN32_LogicalDisk WHERE DeviceID='$Drive'"
If ($UseFreeSpace) {
	$BackupSpace = $DestinationWmi.FreeSpace
} Else {
	$BackupSpace = $DestinationWmi.Size
}
"Destination drive information:" | Write-Host
$DestinationWmi | Select DeviceID, VolumeName, @{Name="FreeSpace GB"; Expression={[uint64]($_.FreeSpace / 1GB)}}, @{Name="Size GB"; Expression={[uint64]($_.Size / 1GB)}} | Format-Table -AutoSize | Out-String | Write-Host

$Results = Get-Subdirectories -Path $Path -Limit $BackupSpace | Sort-Object -Property Name
$BackupTotalSize = ($Results | Measure-Object -Property TotalSize -Sum).Sum

"Backup source:      '$($Path)'" | Write-Host -Fore White
"Backup total size:  {0,9:N2} GB" -f ($BackupTotalSize / 1GB) | Write-Host -Fore White
"Backup destination: '$($Drive)'" | Write-Host -Fore Yellow
"Backup space:       {0,9:N2} GB" -f ($BackupSpace / 1GB) | Write-Host -Fore Yellow
If ($Results.Count -gt 0) {
	"Folders fitting:" | Write-Host -Fore Green
	$Results | Select-Object FullName, TotalSizeGB | Format-Table -AutoSize | Out-String | Write-Host -Fore Green
	If ($PassThru) {$Results}
} Else {
	"Not enough free space on the backup drive, or argument error!" | Write-Error
}

Open in new window

Gerald Connolly

Nice script, but seems to give an incorrect total for one of my directories.

The "From G72" directory is showing as 2GB (1869MB), but Windirstat shows this directory to be 8.8GB, with the difference being in a subfolder called AppData, this tree is s copy from the HDD of a previous (now dead) laptop.

ASKER CERTIFIED SOLUTION

oBdA

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Gerald Connolly

Well that seems better, although it comes up with a bigger number for the G72 directory (approx 200MB) and its doesnt seem to take into account files in the top level directory (approx 550MB) (i have modified script to show MB's)