Avatar of Eric Greene
Eric Greene
Flag for United States of America asked on

Need some verification on Powershell Multi-threading

I don't like massive amounts of code in a single file.  I know it can't always be avoided, but in my case, I actually need to avoid putting some blocks of code in the same file.  I decided to break-out my code in four different files:
File 1:  Automate.ps1  (serves as initiator)
File 2:  au_CreateADUser.ps1
File 3:  au_CreateEXContact.ps1 (creates an Exchange 2010 contact card)
File 4:  au_CreateMSOLUser.ps1 (creates a student's email account and assigns licensing)

I want to execute files 2 - 4 in parallel.  Each file has to iterate through a DataSet that I will pass.  I have the paremterization setup fine, but I need confirmation that my syntax for executing each script in parallel using RunSpaces is correct.  Following is the code I have generated for doing this.  By the way, I have borrowed heavily from forums for what limited understanding I have about it.
#region Create Active Directory User
#######################################################################################
    $AD = 
    {
        Invoke-Expression ".\automation\au_CreateADUser.ps1 -DataSet $ds"
    }
           
    $thread1 = [System.Management.Automation.PowerShell]::Create()
    $thread1.RunspacePool = $Pool
    $thread1.AddScript($AD)
    $thread1.BeginInvoke()
#endregion


#region Create MSOL Services Account
#######################################################################################
    $MSOL = 
    {
        Invoke-Expression ".\automation\au_CreateMSOLUser.ps1 -DataSet $ds"
    }
           
    $thread2 = [System.Management.Automation.PowerShell]::Create()
    $thread2.RunspacePool = $Pool
    $thread2.AddScript($MSOL)
    $thread2.BeginInvoke()
#endregion


#region CreateExchange Server Contact
#######################################################################################
    $ExContact = 
    {
        Invoke-Expression ".\automation\au_CreateEXContact.ps1 -DataSet $ds"
    }
           
    $thread3 = [System.Management.Automation.PowerShell]::Create()
    $thread3.RunspacePool = $Pool
    $thread3.AddScript($ExContact)
    $thread3.BeginInvoke()
#endregion

Open in new window

PowershellMicrosoft Legacy OSWindows OSScripting LanguagesMicrosoft Development

Avatar of undefined
Last Comment
Eric Greene

8/22/2022 - Mon
Qlemo

No. The dataset will not get provided correctly. But before I go into details: any reason you do not make use of PowerShell jobs?
Eric Greene

ASKER
Probably lack of understanding.

Most of what I read seemed to indicate that if I passed my dataset to a job, it would try to process multiple rows of the dataset simultaneously.  My need is to process the same dataset in three simultaneous scripts.

I can't create an Exchange Contact and an MSOL user in the same script if I import both sessions and leave them imported.  If Id o, I have ambiguous commands.  It is possible that the script may have to iterate through 20 records at a time that will all requre an AD user, an MSOL user, and an Exchange Contact.  It would take forever for the script to complete if I tried to establish those sessions individually, close them, then repeat for each of 20 records for three different processes.

I figured my best bet was to make a call to separate scripts, pass the dataset to each one, and hopefully process them in seperate threads so they can process simultaneously.  I'm open to suggestions.

I have attached a zip file with the respective scripts (called by my posted code section) in txt format.
scripts.zip
Eric Greene

ASKER
Any other thoughts, or guidance?
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
ASKER CERTIFIED SOLUTION
David Johnson, CD

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Eric Greene

ASKER
David,

Thank you for that advice.  This all started because I could not find a way around the ambiguous commands.  Would you believe that searching for "problem with ambiguous commands in Powershell" did not return anything about "modulename\command" syntax?

I will try this today and see if it resolves my problem.  Though I am still interested in the multi-threading approach (whether using jobs or RunSpace).
Eric Greene

ASKER
David,

Thank you for that advice.  This all started because I could not find a way around the ambiguous commands.  Would you believe that searching for "problem with ambiguous commands in Powershell" did not return anything about "modulename\command" syntax?

I will try this today and see if it resolves my problem.  Though I am still interested in the multi-threading approach (whether using jobs or RunSpace).
SOLUTION
Jian An Lim

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Eric Greene

ASKER
Since both solutions work, I'll give you both credit.  I'm going to use the prefix method though --it suits me better.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Qlemo

Additional note: You can also provide a prefix with Import-Module, which should be more straight-forward than setting the prefix on importing the session, and is available in all cases.
Qlemo

Thinking more about it, the Runspace idea is not that bad, because you can choose to have isolated environments. But you also have no access to common variables. It is similar with jobs, but those at least can share a common (parent) runspace.
Having a dataset $ds, jobs would be used with
start-job -scriptBlock { .\automation\au_CreateADUser.ps1 -DataSet $args[0] } -ArgumentList $ds
start-job -scriptBlock { .\automation\au_CreateMSOLUser.ps1 -DataSet $args[0] } -ArgumentList $ds
start-job -scriptBlock { .\automation\au_CreateEXContact.ps1 -DataSet $args[0] } -ArgumentList $ds
get-job | wait-job   # wait until all jobs are done
get-job | receive-job   # get all output
get-job | remove-job   # cleanup

Open in new window

Eric Greene

ASKER
It looks like I was right about the results of the start-job method.  Unfortunately, Powershell is serializing the datatable.  See the error below:

Cannot process argument transformation on parameter 'DataSet'. Cannot convert value "System.Data.DataSet" to type "System.Data.DataSet". Error: "Cannot 
convert the "System.Data.DataSet" value of type "Deserialized.System.Data.DataSet" to type "System.Data.DataSet"."

Open in new window

Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
Qlemo

I didn't expect it to be a real dataset ;-). No, you cannot use a serialized dataset. You would have to collect data first, then stream them into the jobs.
Eric Greene

ASKER
Oh, yeah.  More research :)

Thank you for the quick response.