Need some verification on Powershell Multi-threading

I don't like massive amounts of code in a single file.  I know it can't always be avoided, but in my case, I actually need to avoid putting some blocks of code in the same file.  I decided to break-out my code in four different files:
File 1:  Automate.ps1  (serves as initiator)
File 2:  au_CreateADUser.ps1
File 3:  au_CreateEXContact.ps1 (creates an Exchange 2010 contact card)
File 4:  au_CreateMSOLUser.ps1 (creates a student's email account and assigns licensing)

I want to execute files 2 - 4 in parallel.  Each file has to iterate through a DataSet that I will pass.  I have the paremterization setup fine, but I need confirmation that my syntax for executing each script in parallel using RunSpaces is correct.  Following is the code I have generated for doing this.  By the way, I have borrowed heavily from forums for what limited understanding I have about it.
#region Create Active Directory User
#######################################################################################
    $AD = 
    {
        Invoke-Expression ".\automation\au_CreateADUser.ps1 -DataSet $ds"
    }
           
    $thread1 = [System.Management.Automation.PowerShell]::Create()
    $thread1.RunspacePool = $Pool
    $thread1.AddScript($AD)
    $thread1.BeginInvoke()
#endregion


#region Create MSOL Services Account
#######################################################################################
    $MSOL = 
    {
        Invoke-Expression ".\automation\au_CreateMSOLUser.ps1 -DataSet $ds"
    }
           
    $thread2 = [System.Management.Automation.PowerShell]::Create()
    $thread2.RunspacePool = $Pool
    $thread2.AddScript($MSOL)
    $thread2.BeginInvoke()
#endregion


#region CreateExchange Server Contact
#######################################################################################
    $ExContact = 
    {
        Invoke-Expression ".\automation\au_CreateEXContact.ps1 -DataSet $ds"
    }
           
    $thread3 = [System.Management.Automation.PowerShell]::Create()
    $thread3.RunspacePool = $Pool
    $thread3.AddScript($ExContact)
    $thread3.BeginInvoke()
#endregion

Open in new window

LVL 5
Eric GreeneDirector of TechnologyAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

QlemoBatchelor, Developer and EE Topic AdvisorCommented:
No. The dataset will not get provided correctly. But before I go into details: any reason you do not make use of PowerShell jobs?
0
Eric GreeneDirector of TechnologyAuthor Commented:
Probably lack of understanding.

Most of what I read seemed to indicate that if I passed my dataset to a job, it would try to process multiple rows of the dataset simultaneously.  My need is to process the same dataset in three simultaneous scripts.

I can't create an Exchange Contact and an MSOL user in the same script if I import both sessions and leave them imported.  If Id o, I have ambiguous commands.  It is possible that the script may have to iterate through 20 records at a time that will all requre an AD user, an MSOL user, and an Exchange Contact.  It would take forever for the script to complete if I tried to establish those sessions individually, close them, then repeat for each of 20 records for three different processes.

I figured my best bet was to make a call to separate scripts, pass the dataset to each one, and hopefully process them in seperate threads so they can process simultaneously.  I'm open to suggestions.

I have attached a zip file with the respective scripts (called by my posted code section) in txt format.
scripts.zip
0
Eric GreeneDirector of TechnologyAuthor Commented:
Any other thoughts, or guidance?
0
Problems using Powershell and Active Directory?

Managing Active Directory does not always have to be complicated.  If you are spending more time trying instead of doing, then it's time to look at something else. For nearly 20 years, AD admins around the world have used one tool for day-to-day AD management: Hyena. Discover why

David Johnson, CD, MVPOwnerCommented:
with conflicting commands use the modulename\command syntax i.e.
function onlineuser {
        #region Connect to MSOL Services
        ###################### Connect to MSOL Services ##########################
        try
        {
            New-MsolUser -DisplayName $FullName -UserPrincipalName $email -UsageLocation "US" -FirstName $NickAsFirst -LastName $LastName -LicenseAssignment "studentsccbbc:STANDARDWOFFPACK_IW_STUDENT" -StrongPasswordRequired $false -Password $emailPass -ForceChangePassword 0 -ErrorAction Stop| Out-File $log -Append  
            $tryblock = $true
        }
        catch
        {
            Send-MailMessage -From erroradmin@ourdomain.edu -To it@ourdomain.edu -Subject "Failed to create AD account for $NickAsFirst $LastName" -Body $_.Exception.Message + "`n`r`n`r" + $_.Exception.ItemName -SmtpServer mail.ourdomain.edu
            $logdate + ": Failed to create new MSOL user ($Fullname):`n$_.Exception.Message" | Out-File $log -Append
            $tryblock = $false
        }
        finally
        {
            if ($tryblock -eq $true) { $logdate + ": Created new MSOL user ($Fullname)" | Out-File $log -Append }
            $tryblock = $false
        }
        Start-Sleep -Seconds 30
        Try
        {
            msonline\Set-Mailbox -identity $email -CustomAttribute1 $SecEmail -CustomAttribute2 "Student" -CustomAttribute3 $Affiliation -ErrorAction Stop
            $tryblock = $true
        }
        Catch
        {
            Send-MailMessage -From erroradmin@ourdomain.edu -To it@ourdomain.edu -Subject "Failed to create MSOL account for $NickAsFirst $LastName" -Body $_.Exception.Message + "`n`r`n`r" + $_.Exception.ItemName -SmtpServer mail.ourdomain.edu
            $logdate + ": Failed to update custom attributes for $FullName" +":`n" + $_.Exception.Message | Out-File $log -Append
            $tryblock = $false
        }
        Finally
        {
            if($tryblock -eq $true) 
            { 
                $logdate + ": Finished New-Mailbox command for " + $FullName | Out-File $log -Append 
                $logdate + ": New MSOL user mailbox custom attributes set" | Out-File $log -Append
            }
            $tryblock = $false
        }
    }
function aduser{
$path = "ou=Students,dc=OURAD,dc=LAN"
        $chgpass = $false

        if($Affiliation -like "*Online*") 
        {
            $path = "ou=Online,ou=Students,dc=OURAD,dc=LAN"
            $chgpass = $false
        }
        elseif($Affiliation -like "*On Campus*")  { $path = "ou=OnCampus,ou=Students,dc=OURAD,dc=LAN" }
        elseif($Affiliation -like "*Commuter*") { $path = "ou=OnCampus,ou=Students,dc=OURAD,dc=LAN" }
        elseif($Affiliation -like "*Hybrid*")  { $path = "ou=Hybrid,ou=Students,dc=OURAD,dc=LAN" }
        else {$path = "ou=Students,dc=OURAD,dc=LAN"}
        try
        {
            $student = activedirectory\Get-ADUser -LDAPFilter "(sAMAccountName=$alis)"
            if($student -eq $null) {
                "Creating New AD User: $First $LastName" | Out-File -Append
                New-ADUser $Alias -GivenName $First -Surname $LastName -UserPrincipalName $email -Server CCDC01.CCBBC.LAN -Description $StudentID -AccountPassword (ConvertTo-SecureString -String $emailPass -AsPlainText -Force) -ChangePasswordAtLogon $chgpass -Credential $ADCred -Path $path -Displayname $Display -PassThru | Enable-ADAccount -ErrorAction Stop
                $tryblock = $true
            }
            else { $tryblock = $true }
        }
        catch
        {
            [System.Exception]
            $logdate + ": Failed to create new AD user:`n$_.Exception.Message" | Out-File $log -Append
            Send-MailMessage -From erroradmin@ourdomain.edu -To it@ourdomain.edu -Subject "Failed to create AD account for $NickAsFirst $LastName" -Body $_.Exception.Message + "`n`r`n`r" + $_.Exception.ItemName -SmtpServer mail.ourdomain.edu
            $tryblock = $false
        }
        Finally
        {
            if ($tryblock -eq $true) { $logdate + ": Successfully created new Active Directory user." | Out-File $log -Append }
            $tryblock = $false
        }
    }
function local_exchange_user {
       #region ###################### Create Student Contact at On-Premise Exchange Server ##########################
            Try
            {
                $student = Exchange\Get-MailContact -Identity $alias
                If ($student -eq $null) {
                    Exchange\New-MailContact -Name $display -ExternalEmailAddress $email -OrganizationalUnit $path -FirstName $first -LastName $last -alias ($alias)
                }
                Exchange\Set-MailContact -identity $display -Alias $alias -DisplayName $display -CustomAttribute1 $secEmail -CustomAttribute2 "Student" -CustomAttribute3 $ca3
                $tryblock = $true
            }
            Catch
            {
                [system.Exception]
                Send-MailMessage -From erroradmin@ourdomain.edu -To it@ourdomain.edu -Subject "Failed to create Contact for $first $last" -Body $_.Exception.Message + "`n`r`n`r" + $_.Exception.ItemName -SmtpServer mail.ourdomain.edu
                $logdate + ": Failed to create new mail contact for " + $email + ":`n$_.Exception.Message" | Out-File $log -Append
                $tryblock = $false
            }
            Finally
            {
                if($tryblock -eq $true) { $logdate + ": New mail contact ($DisplayName) created" | Out-File $log -Append }
                $tryblock = $false
                $student = $null
            }
        #endregion
        }


param(
    [Parameter(Position=0,Mandatory=$true)][System.Data.DataSet]$DataSet
)
Import-Module activedirectory
import-module msonline
import-module exchange

$fpath = "C:\Automation\"
$ldate = Get-Date
$logdate = $ldate.ToString("MM/dd/yyyy HH:mm:ss")
$log = $fpath + "NewStudentLog" + $ldate.ToString("MMddyyyyHHmm") + ".log"
$logdate + ": ============= Begin New Student Addtion =============" | Out-File $log
$ADUser = "ourAdmin"
$File = "C:\Temp 2\Password.txt"
$ADPASS = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $ADUser, (Get-Content $File | ConvertTo-SecureString)
$ACred =  New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $ADUser, $ADPass
$tryblock = $false
#region ###################### Connect to On-Premise Exchange Server ##########################
    Try
    {
        $Exchange = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://ccmail01.ourad.lan/PowerShell/ -Authentication Kerberos -ErrorAction Stop
        Import-PSSession $Exchange -ErrorAction Stop
        $tryblock = $true
    }
    Catch
    {
        [System.Exception]
        Exchange\Send-MailMessage -From erroradmin@ourdomain.edu -To it@ourdomain.edu -Subject "Exchange: Unable to connect or load" -Body $_.Exception.Message + "`n`r`n`r" + $_.Exception.ItemName -SmtpServer mail.ourdomain.edu
        $logdate + ": Failed to connect to on-premise Microsoft Exchange service: `n" + $_.Exception.Message | Out-File $log -Append
        $tryblock = $false
    }
    Finally
    {
        if($tryblock -eq $true) { $logdate + ": Connected to on-premise Microsoft Exchange service" | Out-file $log -Append }
        $tryblock = $false
    }
#endregion

#region ################### Connect to Microsoft Online Exchange Server
        Try
        {
            Import-Module MSOnline

            $MSOL = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $LiveCred -Authentication Basic -AllowRedirection -ErrorAction Stop
            Import-PSSession $MSOL
            Connect-MsolService -Credential $LiveCred -ErrorAction Stop
            $tryblock = $true
        }
        Catch
        {
            exchange\Send-MailMessage -From erroradmin@ourdomain.edu -To it@ourdomain.edu -Subject "MSOL: Unable to conenct or load" -Body $_.Exception.Message + "`n`r`n`r" + $_.Exception.ItemName -SmtpServer mail.ourdomain.edu
            $logdate + ": Failed to connect to MSOL service: `n" + $_.Exception.Message | Out-File $log -Append
            $tryblock = $false
        }
        Finally
        {
            if ($tryblock -eq $true) { $logdate + ": Successfully connected to MSOL service" | Out-File $log -Append }
            $tryblock = $false
        }
        #endregion
#region ############ process user ###############
foreach ($row in $DataSet.Tables[0].Rows)
{
    $greet = ""
    $NickAsFirst = ""
    if([string]::IsNullOrEmpty($row[1])) { $FullName = "" } else { $FullName = $row[1].ToString().Trim() }
    if([string]::IsNullOrEmpty($row[3])) { $Nickname = "" } else { $Nickname = $row[3].ToString().Trim() }
	if([string]::IsNullOrEmpty($row[5])) { $SSN = "EMPTY4321" } else { $SSN = $row[5].ToString().Trim() }
	if([string]::IsNullOrEmpty($row[17])) { $SecEmail = "" } else { $SecEmail = $row[17].ToString().Trim() }
	if([string]::IsNullOrEmpty($row[18])) { $Affiliation = "" } else { $Affiliation = $row[18].ToString().Trim() }
	if([string]::IsNullOrEmpty($row[21])) { $LastName = "" } else { $LastName = $row[21].ToString().Trim() }
	if([string]::IsNullOrEmpty($row[22])) { $FirstName = "" } else { $FirstName = $row[22].ToString().Trim() }
	if([string]::IsNullOrEmpty($row[23])) { $Processed = "" } else { $Processed = $row[23].ToString().Trim() }
    if([string]::IsNullOrEmpty($row[25])) { $StudentID = "" } else { $StudentID = $row[25].ToString().Trim() }
    if([string]::IsNullOrEmpty($row[30])) { $pin = "" } else { $pin = $row[30].ToString() }
    if([string]::IsNullOrEmpty($row[31])) { $email = "" } else { $email = $row[31].ToString() }
    
    if($Prefix.Length -gt 0) { $greet = $Prefix.Trim() + " " }

    if($Nickname.Trim().Length -gt 0) { 
        $First = $Nickname.Trim()
    } 
    else {
        $First = $FirstName.Trim()
    }

    $Display = $LastName.Trim() + ", " + $First.Trim()
    $Alias = $First + "." + $LastName.Trim()
    $LastFour = $SSN.Trim().Substring(5,4)
    if($email.Length -eq 0) { $email = $First.Trim() + "." + $LastName.Trim() + "@students.ourdomain.edu" }
    if($SSN.Length -eq 0) { $SSN = "EMPTY4321" }
    if($pin.Length -eq 0) { $emailPass = ($First.SubString(0,2) + $LastName.Trim().Substring(0,2) + "@" + $LastFour) } else { $emailPass = $pin }
        if ($Affiliation -like "*Online*") { $CA3 = "Online" } else { $CA3 = "On-Campus" }
            if ($Processed -eq "" -or $Processed -eq "False")
                {
                aduser
                exchange
                onlineuser
                }
}
#endregion

Open in new window

0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Eric GreeneDirector of TechnologyAuthor Commented:
David,

Thank you for that advice.  This all started because I could not find a way around the ambiguous commands.  Would you believe that searching for "problem with ambiguous commands in Powershell" did not return anything about "modulename\command" syntax?

I will try this today and see if it resolves my problem.  Though I am still interested in the multi-threading approach (whether using jobs or RunSpace).
0
Eric GreeneDirector of TechnologyAuthor Commented:
David,

Thank you for that advice.  This all started because I could not find a way around the ambiguous commands.  Would you believe that searching for "problem with ambiguous commands in Powershell" did not return anything about "modulename\command" syntax?

I will try this today and see if it resolves my problem.  Though I am still interested in the multi-threading approach (whether using jobs or RunSpace).
0
Jian An LimSolutions ArchitectCommented:
OR, when you import your session you can put prefix


Import-PSSession $Session -Prefix oo

then you can run get-mailbox  and get-oomailbox to differentiate who is who.
0
Eric GreeneDirector of TechnologyAuthor Commented:
Since both solutions work, I'll give you both credit.  I'm going to use the prefix method though --it suits me better.
0
QlemoBatchelor, Developer and EE Topic AdvisorCommented:
Additional note: You can also provide a prefix with Import-Module, which should be more straight-forward than setting the prefix on importing the session, and is available in all cases.
0
QlemoBatchelor, Developer and EE Topic AdvisorCommented:
Thinking more about it, the Runspace idea is not that bad, because you can choose to have isolated environments. But you also have no access to common variables. It is similar with jobs, but those at least can share a common (parent) runspace.
Having a dataset $ds, jobs would be used with
start-job -scriptBlock { .\automation\au_CreateADUser.ps1 -DataSet $args[0] } -ArgumentList $ds
start-job -scriptBlock { .\automation\au_CreateMSOLUser.ps1 -DataSet $args[0] } -ArgumentList $ds
start-job -scriptBlock { .\automation\au_CreateEXContact.ps1 -DataSet $args[0] } -ArgumentList $ds
get-job | wait-job   # wait until all jobs are done
get-job | receive-job   # get all output
get-job | remove-job   # cleanup

Open in new window

1
Eric GreeneDirector of TechnologyAuthor Commented:
It looks like I was right about the results of the start-job method.  Unfortunately, Powershell is serializing the datatable.  See the error below:

Cannot process argument transformation on parameter 'DataSet'. Cannot convert value "System.Data.DataSet" to type "System.Data.DataSet". Error: "Cannot 
convert the "System.Data.DataSet" value of type "Deserialized.System.Data.DataSet" to type "System.Data.DataSet"."

Open in new window

0
QlemoBatchelor, Developer and EE Topic AdvisorCommented:
I didn't expect it to be a real dataset ;-). No, you cannot use a serialized dataset. You would have to collect data first, then stream them into the jobs.
0
Eric GreeneDirector of TechnologyAuthor Commented:
Oh, yeah.  More research :)

Thank you for the quick response.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Powershell

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.