?
Solved

Powershell 1 to many file copy

Posted on 2011-05-02
5
Medium Priority
?
925 Views
Last Modified: 2012-06-27
Hello,

I have a need to copy a large file to several hundred hosts to prep for the installation of said file.  The file is a couple of GB.  I was running the following:

ForEach($strServer in (GC .\Servers.txt)){
   Copy-Item "\\SERVER1\SHARE\File.exe" -Destination "\\$strServer\C$\Temp"
}

Which works fine, but is obviously doing the copies one at a time, which is taking ~4 minutes per file times several hundred servers.

Is there a quick way to get 5 to 10 copies happening at the same time, with verification that the file copied successfully?

Thank you for any information or direction.
0
Comment
Question by:omnipower321
  • 2
  • 2
5 Comments
 
LVL 12

Assisted Solution

by:rlandquist
rlandquist earned 400 total points
ID: 35504600
I came across this code a while back, but I have not used it yet.  It looks very promising.
Credit to the original author is in the code.

Take a look at it, I think it will do what you are looking for.
#requires -version 1.0
################################################################################
## Run commands in multiple concurrent pipelines
##   by Arnoud Jansveld - www.jansveld.net/powershell
## Version History
## 0.93   Improve error handling: errors originating in the Scriptblock now
##        have more meaningful output
##        Show additional info in the progress bar (thanks Stephen Mills)
##        Add SnapIn parameter: imports (registered) PowerShell snapins
##        Add Function parameter: imports functions
##        Add SplitJobRunSpace variable; allows scripts to test if they are
##        running in a runspace
##        Add seconds remaining to progress bar (experimental)
## 0.92   Add UseProfile switch: imports the PS profile
##        Add Variable parameter: imports variables
##        Add Alias parameter: imports aliases
##        Restart pipeline if it stops due to an error
##        Set the current path in each runspace to that of the calling process
## 0.91   Revert to v 0.8 input syntax for the script block
##        Add error handling for empty input queue
## 0.9    Add logic to distinguish between scriptblocks and cmdlets or scripts:
##        if a ScriptBlock is specified, a foreach {} wrapper is added
## 0.8    Adds a progress bar
## 0.7    Stop adding runspaces if the queue is already empty
## 0.6    First version. Inspired by Gaurhoth's New-TaskPool script
################################################################################

function Split-Job
{
	param(
		$Scriptblock = $(throw 'You must specify a command or script block!'),
		[int]$MaxPipelines = 10,
		[switch]$UseProfile,
		[string[]]$Variable,
		[string[]]$Function = @( ),
		[string[]]$Alias = @( ),
		[string[]]$SnapIn
	)
	
	function Init($InputQueue)
	{
		# Create the shared thread-safe queue and fill it with the input objects
		$Queue = [Collections.Queue]::Synchronized([Collections.Queue] @($InputQueue))
		$QueueLength = $Queue.Count
		# Do not create more runspaces than input objects
		if ($MaxPipelines -gt $QueueLength) { $MaxPipelines = $QueueLength }
		# Create the script to be run by each runspace
		$Script = "Set-Location '$PWD'; "
		$Script += {
			$SplitJobQueue = $($Input)
			& {
				trap { continue }
				while ($SplitJobQueue.Count) { $SplitJobQueue.Dequeue() }
			} |
		}.ToString() + $Scriptblock
		
		# Create an array to keep track of the set of pipelines
		$Pipelines = New-Object System.Collections.ArrayList
		
		# Collect the functions and aliases to import
		$ImportItems = ($Function -replace '^', 'function:') +
		($Alias -replace '^', 'Alias:') |
		Get-Item | select PSPath, Definition
		$stopwatch = New-Object System.Diagnostics.Stopwatch
		$stopwatch.Start()
	}
	
	function Add-Pipeline
	{
		# This creates a new runspace and starts an asynchronous pipeline with our script.
		# It will automatically start processing objects from the shared queue.
		$Runspace = [System.Management.Automation.Runspaces.RunspaceFactory]::CreateRunspace($Host)
		$Runspace.Open()
		$Runspace.SessionStateProxy.SetVariable('SplitJobRunSpace', $True)
		
		function CreatePipeline
		{
			param($Data, $Scriptblock)
			$Pipeline = $Runspace.CreatePipeline($Scriptblock)
			if ($Data)
			{
				$Null = $Pipeline.Input.Write($Data, $True)
				$Pipeline.Input.Close()
			}
			$Null = $Pipeline.Invoke()
			$Pipeline.Dispose()
		}
		
		# Optionally import profile, variables, functions and aliases from the main runspace
		if ($UseProfile)
		{
			CreatePipeline -Script "`$PROFILE = '$PROFILE'; . `$PROFILE"
		}
		if ($Variable)
		{
			foreach ($var in(Get-Variable $Variable -Scope 2))
			{
				trap { continue }
				$Runspace.SessionStateProxy.SetVariable($var.Name, $var.Value)
			}
		}
		if ($ImportItems)
		{
			CreatePipeline $ImportItems {
				foreach ($item in $Input) { New-Item -Path $item.PSPath -Value $item.Definition }
			}
		}
		if ($SnapIn)
		{
			CreatePipeline(Get-PSSnapin $Snapin -Registered) { $Input | Add-PSSnapin }
		}
		$Pipeline = $Runspace.CreatePipeline($Script)
		$Null = $Pipeline.Input.Write($Queue)
		$Pipeline.Input.Close()
		$Pipeline.InvokeAsync()
		$Null = $Pipelines.Add($Pipeline)
	}
	
	function Remove-Pipeline($Pipeline)
	{
		# Remove a pipeline and runspace when it is done
		$Pipeline.RunSpace.Close()
		$Pipeline.Dispose()
		$Pipelines.Remove($Pipeline)
	}
	
	# Main
	# Initialize the queue from the pipeline
	. Init $Input
	# Start the pipelines
	while ($Pipelines.Count -lt $MaxPipelines -and $Queue.Count) { Add-Pipeline }
	
	# Loop through the runspaces and pass their output to the main pipeline
	while ($Pipelines.Count)
	{
		# Only update the progress bar once a second
		if (($stopwatch.ElapsedMilliseconds - $LastUpdate) -gt 1000)
		{
			$Completed = $QueueLength - $Queue.Count - $Pipelines.count
			$LastUpdate = $stopwatch.ElapsedMilliseconds
			$SecondsRemaining = $(if ($Completed)
				{
					(($Queue.Count + $Pipelines.Count)*$LastUpdate / 1000 / $Completed)
				}
				else
			{-1 } )
			Write-Progress 'Split-Job' ( "Queues: $($Pipelines.Count)  Total: $($QueueLength)  " +
			"Completed: $Completed  Pending: $($Queue.Count)" ) `
			-PercentComplete([Math]::Max((100-[Int] ( $Queue.Count + $Pipelines.Count ) / $QueueLength*100), 0)) `
			-CurrentOperation "Next item: $(trap {continue}; if ($Queue.Count) {$Queue.Peek()})" `
			-SecondsRemaining $SecondsRemaining
		}
		foreach ($Pipeline in @($Pipelines))
		{
			if (-not $Pipeline.Output.EndOfPipeline -or -not $Pipeline.Error.EndOfPipeline )
			{
				$Pipeline.Output.NonBlockingRead()
				$Pipeline.Error.NonBlockingRead() | Out-Default
			}
			else
			{
				# Pipeline has stopped; if there was an error show info and restart it
				if ($Pipeline.PipelineStateInfo.State -eq 'Failed')
				{
					$Pipeline.PipelineStateInfo.Reason.ErrorRecord |
					Add-Member NoteProperty writeErrorStream $True -PassThru |
					Out-Default
					# Restart the runspace
					if ($Queue.Count -lt $QueueLength) { Add-Pipeline }
				}
				Remove-Pipeline $Pipeline
			}
		}
		Start-Sleep -Milliseconds 100
	}
}

Get-Content "C:\Scripts\computers.txt"|Split-Job  {%{$result = test-connection -computername $_ -quiet;Write-Host "$_ `t $result"}}

Open in new window

0
 
LVL 16

Expert Comment

by:Bryan Butler
ID: 35507643
You can run them in parrallel, with start-job:

http://technet.microsoft.com/en-us/library/dd347692.aspx

Or maybe you could user remote execution to have the machines copy it themselves:

http://social.technet.microsoft.com/Forums/en/winserverpowershell/thread/55c7c141-56e7-4bc3-914d-72c7550fa815
0
 

Author Comment

by:omnipower321
ID: 35507689
Thank you for the responses.  I have been trying with Start-Job, but am having some trouble figuring out a way to control how many active threads are copying.  
0
 
LVL 16

Accepted Solution

by:
Bryan Butler earned 1600 total points
ID: 35507751
You can check how many processes are running and limit it that way:

$numrun = (ps | findstr "start-job")
if ($numrun.Count -lt < 5) {
   # start another job
}
0
 

Author Comment

by:omnipower321
ID: 35690089
This is what I ended up with.  Seems to be working well so far.  Thank you both for the input!

$intThreadCount = 5
Get-Job | Stop-Job
Get-Job | Remove-Job
Function Get-ActiveThreads{
	$strHost = $Args[0]
	If((@(Get-Job | ?{$_.State -eq "Running"}).Count) -lt $intThreadCount){
		Start-Job -Name $strHost -Scriptblock {Copy-Item "\\SERVER1\SHARE\File.exe" -Destination "\\$($args[0])\C`$\Temp"} -ArgumentList $strHost
	}
	Else{
		Start-Sleep 1
		Get-ActiveThreads $strHost
	}
}
ForEach($strHost in (GC .\Hosts.txt)){
	Get-ActiveThreads $strHost
}

Open in new window

0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This script can help you clean up your user profile database by comparing profiles to Active Directory users in a particular OU, and removing the profiles that don't match.
Auditing domain password hashes is a commonly overlooked but critical requirement to ensuring secure passwords practices are followed. Methods exist to extract hashes directly for a live domain however this article describes a process to extract u…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an antispam), the admini…
Exchange organizations may use the Journaling Agent of the Transport Service to archive messages going through Exchange. However, if the Transport Service is integrated with some email content management application (such as an anti-spam), the admin…

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question