Working for a Tier 1 CSP (Cloud Solution Provider), a lot of the code I produce has to “do stuff” in multiple Azure tenants/subscription in one go. This can be anything from deploying resources to getting tokens and calling APIs against multiple environments, all from the same script.
To reduce the execution runtime of my code, I often try to run things in parallel where possible.
One of the issues I’ve often found with this is that PowerShell does not like it when you try to authenticate against different Azure tenants in parallel. The tenants/subscriptions/tokens get all mixed up and you’ll often find that successful auth will only work for one tenant in the iteration.
It makes sense really. As the parallel iterations are running in the same PowerShell session, and Azure authentication is session based, the two don’t mix well. I’ll show you an example of this shortly if it’s not clear. See The Problem section.
I’ve known about this issue for a couple of years but I’ve never revisited it up until now. I was inspired to give it another go having stumbled upon a great article by Paul Higinbotham on the ForEach-Object Parallel feature in Powershell 7.
Let’s dig into some code so I can demonstrate the issue and the solution I devised.
PowerShell 7
The first thing to do is get set-up with PowerShell 7.
FYI - there is no longer any PowerShell vs PowerShell Core. PowerShell 7 is the latest major update and is fully cross platform. Check out the announcement here.
You can follow this guide to install PowerShell 7 and then this this guide to install the Azure PowerShell modules.
Or, you could just install Docker and run the below command. Easy!
docker run -it mcr.microsoft.com/azure-powershell pwsh
The Problem
Let’s say I want to write a PowerShell script that:
connects to three customer Azure tenants using a Service Principal.
returns a list of resource groups from one specific subscription in each customer tenant.
Let’s presume I either have a Service Principal created in each tenant or have one multi-tenant application that each customer has consented to. I will need to refer to the Service Principal ID and secret in order to authenticate.
For the sake of this example, let’s make one hashtable per customer with the details I’ll need to authenticate, then add each hashtable to a list:
$Customer01 = @{
"tenantId" = "539e78b6-4354-459c-8b50-9191abbf1e73"
"subscriptionId" = "2375914c-2ea4-4d2e-84f1-62773d434f72"
"clientId" = "***"
"secret" = "***"
}
$Customer02 = @{
"tenantId" = "99e3ba62-81ec-4b0e-885c-15d78c5572ae"
"subscriptionId" = "0666886f-d172-4833-8c7e-46ebc2d73b7a"
"clientId" = "***"
"secret" = "***"
}
$Customer03 = @{
"tenantId" = "f3bc3cfd-1c1a-4db7-aafb-20aa78bef7d4"
"subscriptionId" = "86c486e8-4d6a-4a9d-a5f5-b8913b87d178"
"clientId" = "***"
"secret" = "***"
}
$Customers = $Customer01, $Customer02, $Customer03
Now, let’s write a function to connect to each tenant sequentially:
function CustomerAuthSeq {
$Timer = (Measure-Command {
$Customers | ForEach-Object {
Write-Host "Working on customer: $($_.tenantId) with subscription: $($_.subscriptionId)" -ForegroundColor Green
$SecurePassword = ConvertTo-SecureString $_.secret -AsPlainText -Force
$Credential = New-Object System.Management.Automation.PSCredential($_.clientId , $SecurePassword)
Connect-AzAccount -Tenant $_.tenantId -Subscription $_.subscriptionId -Credential $Credential -ServicePrincipal -WarningAction "SilentlyContinue" | Out-Null
$currentAzureContext = Get-AzContext
Write-Host "Connected to Azure tenant $($currentAzureContext.Tenant.Id)" -ForegroundColor Green
$ResourceGroups = Get-AzResourceGroup
ForEach ($ResourceGroup in $ResourceGroups) {
Write-Host "I found Resource Group: $($ResourceGroup.ResourceGroupName)"
}
}
}
).Seconds
Write-Host "Script completed in $Timer" -ForegroundColor Yellow
}
CustomerAuthSeq
As you can see from the below - looping through my list of customers in-turn has worked as expected.
Focus on the green lines in the output. You can see which tenant ID the script attempted to connect to, and which tenant ID it was able to connect to within the loop.
Also, note how long the script took to complete:
Now, let’s convert this function to use ForEach-Object -Parallel (with a Throttle Limit of 3) to see what happens:
function CustomerParallel01 {
$Timer = (Measure-Command {
$Customers | ForEach-Object -Parallel {
Write-Host "Working on customer: $($_.tenantId) with subscription: $($_.subscriptionId)" -ForegroundColor Green
$SecurePassword = ConvertTo-SecureString $_.secret -AsPlainText -Force
$Credential = New-Object System.Management.Automation.PSCredential($_.clientId , $SecurePassword)
Connect-AzAccount -Tenant $_.tenantId -Subscription $_.subscriptionId -Credential $Credential -ServicePrincipal -WarningAction "SilentlyContinue" | Out-Null
$currentAzureContext = Get-AzContext
Write-Host "Connected to Azure tenant $($currentAzureContext.Tenant.Id)" -ForegroundColor Green
$ResourceGroups = Get-AzResourceGroup
ForEach ($ResourceGroup in $ResourceGroups) {
Write-Host "I found Resource Group: $($ResourceGroup.ResourceGroupName)"
}
} -ThrottleLimit 3
}
).Seconds
Write-Host "Script completed in $Timer" -ForegroundColor Yellow
}
CustomerParallel01
Look at these crazy results! Yes, the script was quicker to complete, but it connected to the same tenant each time.
You need to bear in mind that the output will be out-of place as the loop is run in parallel. Since the script blocks within the function are run in parallel, the order of execution is not guaranteed.
Still, it clearly ran into issues when trying to connect to each tenant individually.
These results will vary each time you run it i.e. it may not always be the same tenant that it connects to.
The Solution
I racked my brains for a while on this one! How can I connect to different Azure tenants/subscriptions in parallel without hitting the above issue as well as reducing the execution time.
I had an Inception moment and it occurred to me that I could spawn a new PowerShell (pwsh) session within each parallel iteration. So, in parallel, I could spawn a new pwsh session, authenticate against a specific tenant, do my “stuff” against the tenant/subscription, then exit and return. Let’s check it out:
function CustomerParallel02 {
$Timer = (Measure-Command {
$Customers | ForEach-Object -Parallel {
Write-Host "Working on customer: $($_.tenantId) with subscription: $($_.subscriptionId)" -ForegroundColor Green
pwsh -c {
$_ = $Args[0];
$SecurePassword = ConvertTo-SecureString $_.secret -AsPlainText -Force
$Credential = New-Object System.Management.Automation.PSCredential($_.clientId , $SecurePassword)
Connect-AzAccount -Tenant $_.tenantId -Subscription $_.subscriptionId -Credential $Credential -ServicePrincipal -WarningAction "SilentlyContinue" | Out-Null
$currentAzureContext = Get-AzContext
Write-Host "Connected to Azure tenant $($currentAzureContext.Tenant.Id)" -ForegroundColor Green
$ResourceGroups = Get-AzResourceGroup
ForEach ($ResourceGroup in $ResourceGroups) {
Write-Host "I found Resource Group: $($ResourceGroup.ResourceGroupName)"
}
} -args $_
} -ThrottleLimit 3
}
).Seconds
Write-Host "Script completed in $Timer" -ForegroundColor Yellow
}
CustomerParallel02
To get this working, I use the pwsh cmdlet to spawn a new session.
As you can see from the below (particularly when comparing the results to that of the first test when I ran the script sequentially), the output is more along the lines of what I was expecting.
The correct tenants were connected to, and the correct resource groups were returned.
The execution time was also reduced when compared to that of the original. Not by much, but when scales up to span many more tenants, the reduction in time should be more worthwhile.
Summary
The above isn’t an example of my finest PowerShell code - it was a quick knock-up to demonstrate the issue I faced and how I resolved it.
In summary, authenticating against Azure tenants in parallel is still problematic. The work-around I found involved spawning new PowerShell sessions within each parallel iteration.
Note - you could always call powershell.exe and install/run PSParallel if you don’t want to use PowerShell 7.
Also note - the Azure tenant IDs, subscription IDs, application IDs and secrets have all been deleted before this article was published.
Please feel free to get in touch, particularly if you have any feedback and/or find this article useful.