Powershell 1 to many file copy

Hello,

I have a need to copy a large file to several hundred hosts to prep for the installation of said file.  The file is a couple of GB.  I was running the following:

ForEach($strServer in (GC .\Servers.txt)){
   Copy-Item "\\SERVER1\SHARE\File.exe" -Destination "\\$strServer\C$\Temp"
}

Which works fine, but is obviously doing the copies one at a time, which is taking ~4 minutes per file times several hundred servers.

Is there a quick way to get 5 to 10 copies happening at the same time, with verification that the file copied successfully?

Thank you for any information or direction.
omnipower321Asked:
Who is Participating?
 
Bryan ButlerConnect With a Mentor Commented:
You can check how many processes are running and limit it that way:

$numrun = (ps | findstr "start-job")
if ($numrun.Count -lt < 5) {
   # start another job
}
0
 
rlandquistConnect With a Mentor Commented:
I came across this code a while back, but I have not used it yet.  It looks very promising.
Credit to the original author is in the code.

Take a look at it, I think it will do what you are looking for.
#requires -version 1.0
################################################################################
## Run commands in multiple concurrent pipelines
##   by Arnoud Jansveld - www.jansveld.net/powershell
## Version History
## 0.93   Improve error handling: errors originating in the Scriptblock now
##        have more meaningful output
##        Show additional info in the progress bar (thanks Stephen Mills)
##        Add SnapIn parameter: imports (registered) PowerShell snapins
##        Add Function parameter: imports functions
##        Add SplitJobRunSpace variable; allows scripts to test if they are
##        running in a runspace
##        Add seconds remaining to progress bar (experimental)
## 0.92   Add UseProfile switch: imports the PS profile
##        Add Variable parameter: imports variables
##        Add Alias parameter: imports aliases
##        Restart pipeline if it stops due to an error
##        Set the current path in each runspace to that of the calling process
## 0.91   Revert to v 0.8 input syntax for the script block
##        Add error handling for empty input queue
## 0.9    Add logic to distinguish between scriptblocks and cmdlets or scripts:
##        if a ScriptBlock is specified, a foreach {} wrapper is added
## 0.8    Adds a progress bar
## 0.7    Stop adding runspaces if the queue is already empty
## 0.6    First version. Inspired by Gaurhoth's New-TaskPool script
################################################################################

function Split-Job
{
	param(
		$Scriptblock = $(throw 'You must specify a command or script block!'),
		[int]$MaxPipelines = 10,
		[switch]$UseProfile,
		[string[]]$Variable,
		[string[]]$Function = @( ),
		[string[]]$Alias = @( ),
		[string[]]$SnapIn
	)
	
	function Init($InputQueue)
	{
		# Create the shared thread-safe queue and fill it with the input objects
		$Queue = [Collections.Queue]::Synchronized([Collections.Queue] @($InputQueue))
		$QueueLength = $Queue.Count
		# Do not create more runspaces than input objects
		if ($MaxPipelines -gt $QueueLength) { $MaxPipelines = $QueueLength }
		# Create the script to be run by each runspace
		$Script = "Set-Location '$PWD'; "
		$Script += {
			$SplitJobQueue = $($Input)
			& {
				trap { continue }
				while ($SplitJobQueue.Count) { $SplitJobQueue.Dequeue() }
			} |
		}.ToString() + $Scriptblock
		
		# Create an array to keep track of the set of pipelines
		$Pipelines = New-Object System.Collections.ArrayList
		
		# Collect the functions and aliases to import
		$ImportItems = ($Function -replace '^', 'function:') +
		($Alias -replace '^', 'Alias:') |
		Get-Item | select PSPath, Definition
		$stopwatch = New-Object System.Diagnostics.Stopwatch
		$stopwatch.Start()
	}
	
	function Add-Pipeline
	{
		# This creates a new runspace and starts an asynchronous pipeline with our script.
		# It will automatically start processing objects from the shared queue.
		$Runspace = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspace($Host)
		$Runspace.Open()
		$Runspace.SessionStateProxy.SetVariable('SplitJobRunSpace', $True)
		
		function CreatePipeline
		{
			param($Data, $Scriptblock)
			$Pipeline = $Runspace.CreatePipeline($Scriptblock)
			if ($Data)
			{
				$Null = $Pipeline.Input.Write($Data, $True)
				$Pipeline.Input.Close()
			}
			$Null = $Pipeline.Invoke()
			$Pipeline.Dispose()
		}
		
		# Optionally import profile, variables, functions and aliases from the main runspace
		if ($UseProfile)
		{
			CreatePipeline -Script "`$PROFILE = '$PROFILE'; . `$PROFILE"
		}
		if ($Variable)
		{
			foreach ($var in(Get-Variable $Variable -Scope 2))
			{
				trap { continue }
				$Runspace.SessionStateProxy.SetVariable($var.Name, $var.Value)
			}
		}
		if ($ImportItems)
		{
			CreatePipeline $ImportItems {
				foreach ($item in $Input) { New-Item -Path $item.PSPath -Value $item.Definition }
			}
		}
		if ($SnapIn)
		{
			CreatePipeline(Get-PSSnapin $Snapin -Registered) { $Input | Add-PSSnapin }
		}
		$Pipeline = $Runspace.CreatePipeline($Script)
		$Null = $Pipeline.Input.Write($Queue)
		$Pipeline.Input.Close()
		$Pipeline.InvokeAsync()
		$Null = $Pipelines.Add($Pipeline)
	}
	
	function Remove-Pipeline($Pipeline)
	{
		# Remove a pipeline and runspace when it is done
		$Pipeline.RunSpace.Close()
		$Pipeline.Dispose()
		$Pipelines.Remove($Pipeline)
	}
	
	# Main
	# Initialize the queue from the pipeline
	. Init $Input
	# Start the pipelines
	while ($Pipelines.Count -lt $MaxPipelines -and $Queue.Count) { Add-Pipeline }
	
	# Loop through the runspaces and pass their output to the main pipeline
	while ($Pipelines.Count)
	{
		# Only update the progress bar once a second
		if (($stopwatch.ElapsedMilliseconds - $LastUpdate) -gt 1000)
		{
			$Completed = $QueueLength - $Queue.Count - $Pipelines.count
			$LastUpdate = $stopwatch.ElapsedMilliseconds
			$SecondsRemaining = $(if ($Completed)
				{
					(($Queue.Count + $Pipelines.Count)*$LastUpdate / 1000 / $Completed)
				}
				else
			{-1 } )
			Write-Progress 'Split-Job' ( "Queues: $($Pipelines.Count)  Total: $($QueueLength)  " +
			"Completed: $Completed  Pending: $($Queue.Count)" ) `
			-PercentComplete([Math]::Max((100-[Int] ( $Queue.Count + $Pipelines.Count ) / $QueueLength*100), 0)) `
			-CurrentOperation "Next item: $(trap {continue}; if ($Queue.Count) {$Queue.Peek()})" `
			-SecondsRemaining $SecondsRemaining
		}
		foreach ($Pipeline in @($Pipelines))
		{
			if (-not $Pipeline.Output.EndOfPipeline -or -not $Pipeline.Error.EndOfPipeline )
			{
				$Pipeline.Output.NonBlockingRead()
				$Pipeline.Error.NonBlockingRead() | Out-Default
			}
			else
			{
				# Pipeline has stopped; if there was an error show info and restart it
				if ($Pipeline.PipelineStateInfo.State -eq 'Failed')
				{
					$Pipeline.PipelineStateInfo.Reason.ErrorRecord |
					Add-Member NoteProperty writeErrorStream $True -PassThru |
					Out-Default
					# Restart the runspace
					if ($Queue.Count -lt $QueueLength) { Add-Pipeline }
				}
				Remove-Pipeline $Pipeline
			}
		}
		Start-Sleep -Milliseconds 100
	}
}

Get-Content "C:\Scripts\computers.txt"|Split-Job  {%{$result = test-connection -computername $_ -quiet;Write-Host "$_ `t $result"}}

Open in new window

0
 
Bryan ButlerCommented:
You can run them in parrallel, with start-job:

http://technet.microsoft.com/en-us/library/dd347692.aspx

Or maybe you could user remote execution to have the machines copy it themselves:

http://social.technet.microsoft.com/Forums/en/winserverpowershell/thread/55c7c141-56e7-4bc3-914d-72c7550fa815
0
 
omnipower321Author Commented:
Thank you for the responses.  I have been trying with Start-Job, but am having some trouble figuring out a way to control how many active threads are copying.  
0
 
omnipower321Author Commented:
This is what I ended up with.  Seems to be working well so far.  Thank you both for the input!

$intThreadCount = 5
Get-Job | Stop-Job
Get-Job | Remove-Job
Function Get-ActiveThreads{
	$strHost = $Args[0]
	If((@(Get-Job | ?{$_.State -eq "Running"}).Count) -lt $intThreadCount){
		Start-Job -Name $strHost -Scriptblock {Copy-Item "\\SERVER1\SHARE\File.exe" -Destination "\\$($args[0])\C`$\Temp"} -ArgumentList $strHost
	}
	Else{
		Start-Sleep 1
		Get-ActiveThreads $strHost
	}
}
ForEach($strHost in (GC .\Hosts.txt)){
	Get-ActiveThreads $strHost
}

Open in new window

0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.