We help IT Professionals succeed at work.
Get Started

Running parallel robocopy jobs

cdummy
cdummy asked
on
1,597 Views
Last Modified: 2015-04-09
Hello, I'm a novice PowerShell user and I've put together a script that's an amalgam of many that I've been able to find that have parts and pieces of what I need it to do. I know it's not elegant :)  

So far it:
1) imports a csv file that contains the name of a share, it's source, and the destination
2) calls robocopy to copy the file and create a backup of the files plus creates a timestamped log directory and log files for each line in the csv
3) evaluates the log and if anything was copied or errored then it runs one more time to ensure that everything was copied after the second run
4) evaluates the results and, if any files weren't copied the second time, will send an email alert to our help desk software

I have a working copy of that script right now but I need it to run faster since it'll take too long to have all the jobs run in series. I have been struggling to find a way to run the jobs in parallel, though, and that's where I need some help.

The CSV is a file that just has headers named Share, Source, and Backup with values like this:
ShareName,Source,Backup
Corporate_Maintenance_Secure,\\fws.blah.com\corporate\Maintenance Secure,\\FWG-Commvault\Share Backups\Corporate Maintenance Secure
Corporate_Safety,\\fws.blah.com\corporate\Safety,\\fwg-commvault\Share Backups\Corporate Safety

The working script (that needs to be run in parallel rather than in serial) is:

Clear-Host
$ErrorActionPreference = "SilentlyContinue"
$DebugPreference = "Continue"
$VerbosePreference = "Continue"



## robocopy_shares.ps1 ########################################################
## Purpose:      Run robocopy to sync network shares to CommVault server
## History:      2015-04-01 - Created
##                         Robocopies, alerts if any files are moved.
##                         Double-checks if files are copied. Not parallel
###############################################################################



## User Supplied Variables
$Shares = import-csv C:\temp\sharemonitor\shares.csv
$BaseLogLocation = "C:\Temp\ShareMonitor"
$CurrentLogLocation = New-Item "$BaseLogLocation\$(get-date -f yyyy-MM-dd-hh)" -ItemType Directory -Force
$PSEmailServer = "mailhost.fws.farweststeel.com"

ForEach ($Share in $Shares) {

$Source = ( '"' + $Share.Source + '"')
$Dest = ( '"' + $Share.Backup + '"')
$ShareName = $Share.ShareName
$LogFile = "$CurrentLogLocation\$($ShareName)_$(get-date -f yyyy-MM-dd-hh-mm).txt"
$RoboSwitches = "/MIR /R:1 /W:1 /Log+:$LogFile"


## Robocopy Run

$robo_Test = Start-Process -ArgumentList "$Source $Dest $RoboSwitches" robocopy.exe -WAIT


## Use Regular Expression to grab the following Table
#               Total    Copied   Skipped  Mismatch    FAILED    Extras
#    Dirs :         1         0         1         0         0         0
#   Files :         1         0         1         0         0         0
$robo_results = Get-Content $LogFile
$robo_fin = $robo_results -match '^(?= *?\b(Total|Dirs|Files)\b)((?!    Files).)*$'

## Convert Table above into an array
$robo_arr = @()
foreach ($line in $robo_fin){
     $robo_arr += $line
}

## Create Powershell object to tally Robocopy results
$row = "" |select COPIED, MISMATCH, FAILED, EXTRAS
$row.COPIED = [int](($robo_arr[1] -split "\s+")[4]) + [int](($robo_arr[2] -split "\s+")[4])
$row.MISMATCH = [int](($robo_arr[1] -split "\s+")[6]) + [int](($robo_arr[2] -split "\s+")[6])
$row.FAILED = [int](($robo_arr[1] -split "\s+")[7]) + [int](($robo_arr[2] -split "\s+")[7])
$row.EXTRAS = [int](($robo_arr[1] -split "\s+")[8]) + [int](($robo_arr[2] -split "\s+")[8])

$Copied = $row.COPIED
$Mismatch = $row.MISMATCH
$Failed = $row.FAILED
$Extras = $row.EXTRAS

## If there are differences, lets run Robocopy one more time
if ( ($COPIED + $MISMATCH + $FAILED + $EXTRAS) -gt 0 ){

    ## Robocopy Run

    $robo_Test = Start-Process -ArgumentList "$Source $Dest $RoboSwitches" robocopy.exe -WAIT

    ## Use Regular Expression to grab the following Table
    #               Total    Copied   Skipped  Mismatch    FAILED    Extras
    #    Dirs :         1         0         1         0         0         0
    #   Files :         1         0         1         0         0         0
    ## $robo_results = $robo_test -match '^(?= *?\b(Total|Dirs|Files)\b)((?!    Files).)*$'
    $robo_results = Get-Content $LogFile
    $pattern = "-match '^(?= *?\b(Total|Dirs|Files)\b)((?!    Files).)*$'"
    $robo_fin = $robo_results -match '^(?= *?\b(Total|Dirs|Files)\b)((?!    Files).)*$'

    ## Convert Table above into an array
    $robo_arr = @()
    foreach ($line in $robo_fin){
         $robo_arr += $line
    }

    ## Create Powershell object to tally Robocopy results
    $row = "" |select COPIED, MISMATCH, FAILED, EXTRAS
    $row.COPIED = [int](($robo_arr[5] -split "\s+")[4]) + [int](($robo_arr[6] -split "\s+")[4])
    $row.MISMATCH = [int](($robo_arr[5] -split "\s+")[6]) + [int](($robo_arr[6] -split "\s+")[6])
    $row.FAILED = [int](($robo_arr[5] -split "\s+")[7]) + [int](($robo_arr[6] -split "\s+")[7])
    $row.EXTRAS = [int](($robo_arr[5] -split "\s+")[8]) + [int](($robo_arr[6] -split "\s+")[8])

    $Copied = $row.COPIED
    $Mismatch = $row.MISMATCH
    $Failed = $row.FAILED
    $Extras = $row.EXTRAS
   
    if ( ($COPIED + $MISMATCH + $FAILED + $EXTRAS) -gt 0 ){
        "Not all data is replicated." >> $Logfile
          $anonUser = "anonymous"
        $anonPass = ConvertTo-SecureString "anonymous" -AsPlainText -Force
        $anonCred = New-Object System.Management.Automation.PSCredential($anonUser, $anonPass)
        $MessageSubject = ("Verify backups for "+$Share.Sharename+"")
          $MessageBody = ("Backups for "+$Share.Sharename+" are not the same size as the data on the share. Please reference the robocopy logs and correct the situation.")
          Send-MailMessage -to "blah@blah.com" -from "Backup Verification <do-not-reply@blah.com>" -subject $MessageSubject -body $MessageBody -Credential $anonCred
   
    }

else {
    "All data is replicated." >> $Logfile
      $anonUser = "anonymous"
    $anonPass = ConvertTo-SecureString "anonymous" -AsPlainText -Force
    $anonCred = New-Object System.Management.Automation.PSCredential($anonUser, $anonPass)
    $MessageSubject = ("Verify backups for "+$Share.Sharename+"")
      $MessageBody = ("Backups for "+$Share.Sharename+" are the same size as the data on the share. No actions are needed.")
      Send-MailMessage -to "blah@blah.com" -from "Backup Verification <do-not-reply@blah.com>" -subject $MessageSubject -body $MessageBody -Credential $anonCred
      }

}

}

Please lend a hand with some code that would allow me to run this thing in parallel. Thanks!
Comment
Watch Question
Owner
Commented:
This problem has been solved!
Unlock 1 Answer and 6 Comments.
See Answer
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE