diff --git a/documentation/documents/psframework.md b/documentation/documents/psframework.md index 5b61939..a3e9c1b 100644 --- a/documentation/documents/psframework.md +++ b/documentation/documents/psframework.md @@ -10,22 +10,22 @@ Whether it is message handling, logging, configuration, parameter classes, custo The logging system provides powerful logging for little effort: -- Log to file, eventlog, SQL, Graylog, Splunk or Azure Log Analytics out-of-the-box -- Freely extensible for custom logging needs -- Runspace-Safe -- Asynchronous -- Log to multiple locations at the same time -- Logging can be defined in script, at the process level, by configuration per user or per computer. All without conflict. ++ Log to file, eventlog, SQL, Graylog, Splunk or Azure Log Analytics out-of-the-box ++ Freely extensible for custom logging needs ++ Runspace-Safe ++ Asynchronous ++ Log to multiple locations at the same time ++ Logging can be defined in script, at the process level, by configuration per user or per computer. All without conflict. For more details on the logging system, see the [dedicated component page](psframework/logging.html) ## Configuration The configuration system offers the means to ... - - Implement options for modules, similar to an application's options menu - - Implement configuration of CI/CD pipelines - - Control modules and scripts by group policy or DSC - - utilize a cache that persists across powershell sessions ++ Implement options for modules, similar to an application's options menu ++ Implement configuration of CI/CD pipelines ++ Control modules and scripts by group policy or DSC ++ utilize a cache that persists across powershell sessions ```powershell Get-PSFConfig @@ -38,12 +38,22 @@ For more details on the configuration system, see the [dedicated component page] Custom Tab Completion allows both developers and users to easily deploy custom tab completion. Put an end to Mr. Typo and improve your everyday console experience! - - Implement fast, reusable tab completion. - - Add Tab Completion to own or existing commands. - - Significantly improve tab completion. ++ Implement fast, reusable tab completion. ++ Add Tab Completion to own or existing commands. ++ Significantly improve tab completion. For more details on the tab completion system, see the [dedicated component page](psframework/tab-completion.html). +## Runspace Workflows +Runspace workflows allow architecting parallelized jobs that exchange data among each other. +This allows you to focus on your business logic while fully leveraging runspaces to accelerate the overall result. + ++ Easily design workflow ++ Exchange results between stages ++ None of the complex custom code - just bring the business logic + +For more details on the Runspace Workflow system, see the [dedicated component page](psframework/runspace-workflows.html). + ## Flow-Control The Flow-Control component focuses on help govern how your code "flows". This includes support for error handling, enhancing readability of your flow constructs (such as how you implement -WhatIf & -Confirm) and other utilities for governing your own code's layout. diff --git a/documentation/documents/psframework/runspace-workflows.md b/documentation/documents/psframework/runspace-workflows.md new file mode 100644 index 0000000..5e65c5d --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows.md @@ -0,0 +1,46 @@ +--- +title: PSFramework: Runspace Workflows +--- + +# Runspace Workflows + +[Back to PSFramework](https://psframework.org/documentation/documents/psframework.html) + +## Synopsis + +Runspace Workflows allow easily architecting a set of steps that will be execute in parallel: + ++ For Each step of a workflow, define how many times it should be executed in parallel ++ The results of one step become the input for the next step ++ No need to deal with the details - just provide the business logic for each step + +## Description + +The fundamental system of Runspace Workflow consists of three main components: +The Workflow Object, the Worker and the Queue: + ++ Workflow Object: Represents the entire workflow and hosts all the interna, including the Workers and the Queues. ++ Worker: An individual step in the workflow, the worker contains the business logic of that step, the configuration and the runspaces actually executing it. ++ Queue: A workflow can have as many queues as desired. They ensure data gets exchanged between steps, usually the output of one step becoming the input of the next step. + +## Core Concepts + ++ [The Workflow Object](runspace-workflows/workflow.html) ++ [The Worker](runspace-workflows/worker.html) ++ [The Queue](runspace-workflows/queue.html) + +## Examples + +While the details on the components may provide value, this system benefits most from some examples: + ++ [A simple workflow](runspace-workflows/examples-simple.html) ++ [Begin & End](runspace-workflows/examples-begin-end.html) ++ [Including Variables, Functions and Modules](runspace-workflows/examples-resources.html) ++ [Different variable value per runspace](runspace-workflows/examples-perrunspacevariables.html) ++ [Automatically ending Workflows](runspace-workflows/examples-auto-close.html) ++ [When the first step generates the input](runspace-workflows/examples-first-step-data.html) ++ [Branching Flows - One step feeding multiple others](runspace-workflows/examples-multi-pronged-flows.html) ++ [Data outside of the Queues](runspace-workflows/examples-the-data-field.html) ++ [Rate limiting - throttling and workflows](runspace-workflows/examples-throttling.html) + +[Back to PSFramework](https://psframework.org/documentation/documents/psframework.html) diff --git a/documentation/documents/psframework/runspace-workflows/examples-auto-close.md b/documentation/documents/psframework/runspace-workflows/examples-auto-close.md new file mode 100644 index 0000000..0cc3935 --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/examples-auto-close.md @@ -0,0 +1,100 @@ +# Automatically Closing Queues and Workers + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Basics + +A documented example of the basic workflow setup [can be found on this page](examples-simple.html). + +## Description + +One of the issues with this parallel executing workflow is "How do I know when I'm done?". +A worker and its runspaces have no idea, where in the processing sequence they are and whether new input is going to arrive. + +As a matter of fact, the entire workflow is unaware of this, the original input always comes from outside of it and there is nothing preventing the script using it from adding more input at a later time. +Thus the concept of "closing queues" was introduced: +A queue can, at any time, be closed - a flag that prevents any further data from being added to it. + +A worker using that queue as input will then know, not to expect any further input and can close itself. +Once every runspace of a worker is done, it can then - if so configured - close its own out queue, as it knows it will never add another item to it. +Which then signals the next worker, using _that_ queue as input that no more data will be forthcoming. + +In the end, the entire chain of workers can use this mechanism to end itself, halting the entire workflow. + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-Closing' + +$workflow | Add-PSFRunspaceWorker -Name Processing -InQueue Input -OutQueue Processed -Count 3 -ScriptBlock { + param ($Value) + + [PSCustomObject]@{ + Input = $Value + Processed = $Value * 2 + Result = $null + } +} -CloseOutQueue +$workflow | Add-PSFRunspaceWorker -Name Result -InQueue Processed -OutQueue Done -Count 2 -ScriptBlock { + param ($Value) + + $Value.Result = $Value.Processed * 3 + $Value +} -CloseOutQueue + +$workflow | Write-PSFRunspaceQueue -Name Input -BulkValues (1..1000) -Close +$workflow | Start-PSFRunspaceWorkflow + +$workflow | Wait-PSFRunspaceWorkflow -WorkerName Result -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Done -All +$workflow | Remove-PSFRunspaceWorkflow +``` + ++ The `-CloseOutQueue` switch parameter on `Add-PSFRunspaceWorker` tells the worker to close its out-going queue, once it knows it cannot receive more input. This happens when its input queue has been closed _and all remaining items processed_ or its `MaxItems` count has been reached (and processed). ++ The `-Close` parameter on `Write-PSFRunspaceQueue` has the effect of immediately closing the queue after adding values to it. Usually used with the original input queue after having added all input needed. ++ The `Wait-PSFRunspaceWorkflow` command can wait for a specific worker or queue is closed, avoiding the need to specify just how many items to wait for. + +> MaxItems + +Another way for a worker to know when to be done, is by telling it, just how many items it is supposed to process. +This is done via the `-MaxItems` parameter on the `Add-PSFRunspaceWorker` command: + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-MaxItems' + +$workflow | Add-PSFRunspaceWorker -Name Processing -InQueue Input -OutQueue Processed -Count 3 -ScriptBlock { + param ($Value) + + [PSCustomObject]@{ + Input = $Value + Processed = $Value * 2 + Result = $null + } +} -CloseOutQueue -MaxItems 10 # Stop after the first 10 items +$workflow | Add-PSFRunspaceWorker -Name Result -InQueue Processed -OutQueue Done -Count 2 -ScriptBlock { + param ($Value) + + $Value.Result = $Value.Processed * 3 + $Value +} -CloseOutQueue + +$workflow | Write-PSFRunspaceQueue -Name Input -BulkValues (1..1000) +$workflow | Start-PSFRunspaceWorkflow + +$workflow | Wait-PSFRunspaceWorkflow -WorkerName Result -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Done -All +$workflow | Remove-PSFRunspaceWorkflow + +$results.Count # 10 +``` + +> QueuesToClose + +The `Add-PSFRunspaceWorker` function also has a `-QueuesToClose` parameter, which allows a worker to close more than one queue when it is done. +While not necessarily needed in most scenarios, it might come in handy when implementing [non-linear workflows](examples-multi-pronged-flows.html). + +## Next Steps + ++ [How do I avoid spamming a service with too many requests?](examples-throttling.html) ++ [Are all workflows a linear processing chain?](examples-multi-pronged-flows.html) ++ [Are queues all we have to exchange data?](examples-the-data-field.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/examples-begin-end.md b/documentation/documents/psframework/runspace-workflows/examples-begin-end.md new file mode 100644 index 0000000..6809b24 --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/examples-begin-end.md @@ -0,0 +1,52 @@ +# Begin and End + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Basics + +A documented example of the basic workflow setup [can be found on this page](examples-simple.html). + +## Description + +As seen in the [basic example](examples-simple.html), runspace workflows allow defining steps ([Workers](worker.html)), where a scriptblock is executed once per input item. +That is all great and similar to the `process` block of the PowerShell pipeline. + +But if we have a `process`, do we also have a `begin` and `end`? + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-BeginEnd' + +$begin = { + $global:sqlInstance = Connect-DbaInstance -SqlInstance sql01.contoso.com\userdb +} +$process = { + $_ | Write-DbaDataTable -SqlInstance $global:sqlInstance -Database userDB -Table Users +} +$end = { + Disconnect-DbaInstance $global:sqlInstance +} + +$workflow | Add-PSFRunspaceWorker -Name Users -InQueue UserList -OutQueue Users -Count 5 -ScriptBlock { + Get-ADUser -Identity $_ +} -CloseOutQueue +$workflow | Add-PSFRunspaceWorker -Name WriteToDB -InQueue Users -OutQueue Done -Count 1 -Begin $begin -ScriptBlock $process -End $end -CloseOutQueue + +# Add values and execute +$workflow | Write-PSFRunspaceQueue -Name UserList -BulkValues $users -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName WriteToDB -Closed -PassThru | Remove-PSFRunspaceWorkflow +``` + +We do. + +That said, this example also includes one thing not yet covered in the [basic example](examples-simple.html): Closing queues. +More on that in another, [dedicated example scenario](examples-auto-close.html). + +> Note: Another difference to the basic example is, that the final step will produce no output. While we have to specify an `OutQueue`, it will never receive items. This also means we never have to read from the queue and can skip over `Stop-*` and move straight to `Remove-PSFRunspaceWorkflow`. + +## Next Steps + ++ [Automatically close queues and workers when done](examples-auto-close.html) ++ [Providing variables, functions or modules to the worker code](examples-resources.html) ++ [How do I avoid spamming a service with too many requests?](examples-throttling.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/examples-first-step-data.md b/documentation/documents/psframework/runspace-workflows/examples-first-step-data.md new file mode 100644 index 0000000..7148fe8 --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/examples-first-step-data.md @@ -0,0 +1,93 @@ +# Zero Input: When the first worker produces the information + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Basics + +A documented example of the basic workflow setup [can be found on this page](examples-simple.html). + +## Description + +Sometimes, we do not actually already have input at hand - instead the first step in the workflow is supposed to generate it. +A common example are classic, long-running commands, that generate data as it comes, but can't really be interrupted without massive performance penalties. +`Get-Mailbox` in a large Exchange Online environment can run for many hours. + +Not only will it not require any input, but we would also like to start working on the first mailboxes, without having to wait for it all to complete ... + +Well, fret not, where there is a will, there is a script. + +> Note: This example leans heavily on the [begin/end setup](examples-begin-end.html). + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-FirstStepData' + +# First Worker: Get Mailboxes +$variables = @{ + exAppID = $exAppID + exOrganization = $exOrganization + exCert = $exCert +} +$begin = { + Connect-ExchangeOnline -AppID $exAppID -Organization $exOrganization -Certificate $exCert +} +$process = { + Get-EXOMailbox | Write-PSFRunspaceQueue -Name Mailboxes -Name '' -InputObject $null +} +$end = { + Disconnect-ExchangeOnline +} +$workflow | Add-PSFRunspaceWorker -Name Mailboxes -InQueue Input -OutQueue Mailboxes -Begin $begin -ScriptBlock $process -End $end -Count 1 -Variables $variables -Modules ExchangeOnlineManagement -KillToStop -CloseOutQueue + +# Second Worker: Match Information from Active Directory +$process2 = { + param ($Value) + $adUser = Get-ADUser -LdapFilter "(mail=$($Value.PrimarySmtpAddress))" + + [PSCustomObject]@{ + SamAccountName = $adUser.SamAccountName + SID = $adUser.ObjectSID + DistinguishedName = $adUser.DistinguishedName + Mail = $Value.PrimarySmtpAddress + ProxyAddresses = $value.ProxyAddresses -join ',' + } +} +$workflow | Add-PSFRunspaceWorker -Name ADUser -InQueue Mailboxes -OutQueue ADUser -ScriptBlock $process2 -Count 10 -Modules ActiveDirectory -CloseOutQueue + +# Third Worker: Write Results to CSV +$variables3 = @{ Path = 'C:\temp\users.csv' } +$begin3 = { + $global:command = { Export-Csv -Path $Path }.GetSteppablePipeline() + $global:command.Begin($true) +} +$process3 = { + $global:command.Process($_) +} +$end3 = { + $global:command.End() +} +$workflow | Add-PSFRunspaceWorker -Name CSV -InQueue ADUser -OutQueue Nothing -Begin $begin3 -ScriptBlock $process3 -End $end3 -Count 1 -Variables $variables3 -CloseOutQueue + +# Add one piece of input because we need to run Mailbox exactly once +$workflow | Write-PSFRunspaceQueue -Name Input -Value 1 -Close + +$workflow | Start-PSFRunspaceWorkflow +$workflow | Wait-PSFRunspaceWorkflow -WorkerName CSV -Closed -PassThru | Remove-PSFRunspaceWorkflow +``` + +Admittedly, not the shortest of snippets, but it's an end-to-end example of how that works. + +It performs three steps: + +1. Retrieve all mailboxes from Exchange Online (one runspace only) +2. Matches the mailboxes against Active Directory (10 runspaces in parallel) +3. Exports the results to CSV (one runspace only, avoiding write conflicts). The Steppable pipeline allows us to keep the CSV file open and avoid the cost of opening, parsing and closing for _each_ item. + +> Note: This example used the `-CloseOutQueue`, `-Close` and `-Closed` parameters - those are designed to tell the workflow when it is done. There is a [dedicated documented example explaining just how that works](examples-auto-close.html) + +## Next Steps + ++ [Throttling: The Art of not Spamming the Server](examples-throttling.html) ++ [Providing different values to different Runspaces](examples-perrunspacevariables.html) ++ [The Beginning and The End](examples-begin-end.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/examples-multi-pronged-flows.md b/documentation/documents/psframework/runspace-workflows/examples-multi-pronged-flows.md new file mode 100644 index 0000000..1a0f7ea --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/examples-multi-pronged-flows.md @@ -0,0 +1,74 @@ +# Non-Linear Workflows + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Basics + +A documented example of the basic workflow setup [can be found on this page](examples-simple.html). + +## Description + +In many simple workflows, there is a strict sequence of actions taken: Step 1, then Step 2 and finally Step 3. +A linear sequence of actions, similar to commands on a single PowerShell pipeline. + +Well, nothing says Runspace Workflows must be linear. + +Any given worker only looks to its own `InQueue` for work orders. +Using the `Write-PSFRunspaceQueue` command it is quite possible to provide values to more than just one out queue: + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-MultiPronged' + +# Define Workers +$workflow | Add-PSFRunspaceWorker -Name ADUser -InQueue Input -OutQueue Users -Count 5 -ScriptBlock { + param ($Value) + Get-ADUser -Identity $Value +} +$workflow | Add-PSFRunspaceWorker -Name Filter -InQueue Users -OutQueue goodUsers -Count 1 -ScriptBlock { + param ($Value) + if ($Value.SamAccountName -notmatch '^[a-z]{1}\d{6}[abc]$') { + Write-PSFRunspaceQueue -Name badUsers -Value $Value + return + } + $Value +} +$workflow | Add-PSFRunspaceWorker -Name BadUser -InQueue badUsers -OutQueue Completed -Count 3 -ScriptBlock { + param ($Value) + Send-BadUserReport -User $Value + $Value +} +$workflow | Add-PSFRunspaceWorker -Name GoodUsers -InQueue goodUsers -OutQueue Completed -Count 3 -ScriptBlock { + param ($Value) + Send-GoodUserReport -User $Value + $Value +} +$workflow | Add-PSFRunspaceWorker -Name PostProcessing -InQueue Completed -OutQueue AllUsers -Count 1 -ScriptBlock { + param ($Value) + $Value.SamAccountName +} + +# Execute +$workflow | Write-PSFRunspaceQueue -Name Input -BulkValues (Get-Content .\users.txt) +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName PostProcessing -ReferenceQueue Users -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name AllUsers -All +$workflow | Remove-PSFRunspaceWorkflow +``` + +In this workflow we ... + ++ 1: Retrieve the AD User object for each user provided to input ++ 2: Match the account against a specified pattern and either send it on to the queue for 3a (badUsers) or 3b (goodUsers) ++ 3a: Reports on bad users and sends it on to 4 ++ 3b: Reports on good users and sends it on to 4 ++ 4: Provides the SamAccountName of all users, irrespective of whether they come from 3a or 3b + +In other words, it is perfectly fine for data flows to split between workers and queues, so long as we keep in mind how we are going to track successful conclusion. +`Wait-PSFRunspaceWorkflow` can compare the number of items that went through one worker with the total number another queue held to help make that determination, when [closing out queues](examples-auto-close.html) becomes difficult. + +## Next Steps + ++ [Throttling: The Art of not Spamming the Server](examples-throttling.html) ++ [Providing different values to different Runspaces](examples-perrunspacevariables.html) ++ [Data outside of the queue system](examples-the-data-field.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/examples-perrunspacevariables.md b/documentation/documents/psframework/runspace-workflows/examples-perrunspacevariables.md new file mode 100644 index 0000000..c1a05d1 --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/examples-perrunspacevariables.md @@ -0,0 +1,50 @@ +# Per Runspace Variables + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Basics + +A documented example of the basic workflow setup [can be found on this page](examples-simple.html). + +## Description + +In some situations, it might be useful to have the same variable in all runspaces of a worker, but have each instance contain a different value. +Imagine a worker executing an API where each runspace should authenticate with different credentials to avoid throttling. + +Not a problem with the `-VarPerRunspace` parameter of `Add-PSFRunspaceWorker`: + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-PerRunspaceVars' + +$variables = @{ + ID = 1,2,3,4,5 +} +$workflow | Add-PSFRunspaceWorker -Name Multiply -InQueue Numbers -OutQueue Results -Count 5 -ScriptBlock { + param ($Value) + Start-Sleep -Milliseconds 200 + [PSCustomObject]@{ + Input = $Value + Index = $ID + Result = $Value * $ID + } +} -VarPerRunspace $variables -CloseOutQueue + +$workflow | Write-PSFRunspaceQueue -Name Numbers -BulkValues (1..20) -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName Multiply -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Results -All +$workflow | Remove-PSFRunspaceWorkflow + +$results +``` + +The `-VarPerRunspace` parameter expects a hashtable, where the key is the variable name and the value is an array of the values. +When later the runspaces for the worker get initialized, the first runspace launched will take the first of the provided values, the second runspace the next and so on, until each runspace has exactly one value from the list of values provided. +This means, the number of values provided should match the number of runspaces - providing too few values will cause some runspaces to leave the variable empty, while providing too many just means some of them do not get used at all. + +## Next Steps + ++ [Providing variables, functions or modules to the worker code](examples-resources.html) ++ [Throttling: The Art of not Spamming the Server](examples-throttling.html) ++ [The Beginning and The End](examples-begin-end.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/examples-resources.md b/documentation/documents/psframework/runspace-workflows/examples-resources.md new file mode 100644 index 0000000..1d1c079 --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/examples-resources.md @@ -0,0 +1,155 @@ +# Resources: Providing Variables, functions and Modules + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Basics + +A documented example of the basic workflow setup [can be found on this page](examples-simple.html). + +## Description + +One of the pain points with multithreading is that while it may look similar to the rest of the script - it's still PowerShell code, right? - but does not really have access to all of that context - variables do not exist, functions are unknown - even modules might be missing, if you do not have them installed and instead load them by path! + +Fortunately, providing those to a Runspace Workflow worker is quite simple: + +### Variables + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-Variables' + +$variables = @{ + Multiplier = 3 +} +$workflow | Add-PSFRunspaceWorker -Name Multiply -InQueue Numbers -OutQueue Results -Count 5 -ScriptBlock { + param ($Value) + Start-Sleep -Milliseconds 200 + [PSCustomObject]@{ + Input = $Value + Multiplier = $Multiplier + Result = $Value * $Multiplier + } +} -Variables $variables -CloseOutQueue + +$workflow | Write-PSFRunspaceQueue -Name Numbers -BulkValues (1..20) -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName Multiply -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Results -All +$workflow | Remove-PSFRunspaceWorkflow + +$results +``` + +### Functions + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-Functions' + +function Get-RandomNumber { + [CmdletBinding()] + param() + Get-Random -Minimum 10 -Maximum 99 +} +$functions = @{ + 'Get-RandomNumber' = (Get-Command Get-RandomNumber).Definition +} + +$workflow | Add-PSFRunspaceWorker -Name Multiply -InQueue Numbers -OutQueue Results -Count 5 -ScriptBlock { + param ($Value) + Start-Sleep -Milliseconds 200 + [PSCustomObject]@{ + Input = $Value + Random = Get-RandomNumber + } +} -Functions $functions -CloseOutQueue + +$workflow | Write-PSFRunspaceQueue -Name Numbers -BulkValues (1..20) -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName Multiply -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Results -All +$workflow | Remove-PSFRunspaceWorkflow + +$results +``` + +> Functions and Security: Under the Aegis of Constrained Language Mode + +In a secured environment, one of the main concern is what code is trusted and what code is not. +The background runspaces are trusted (even if the worker code itself is not). However, functions are injected into the runspace itself, and thus must be trusted. + +When deploying code in such an environment, PSFramework needs to make sure functions should be trusted ... but it cannot determine this base on the string value of the function definition as shown above. +Simply put, if this were allowed, attackers could use this to run their code as trusted code: + +````powershell +$functions = @{ + 'Get-RandomNumber' = (Get-Command Get-RandomNumber).Definition +} +```` + +Instead, under CLM, even if the caller is a trusted source, function-code must be provided as scriptblock, rather than a string: + +````powershell +$functions = @{ + 'Get-RandomNumber' = [ScriptBlock]::Create((Get-Command Get-RandomNumber).Definition) +} +```` + +That way, PSFramework can verify the language mode (and thus trust level) of the scriptblock to ensure the function can safely be loaded into a trusted context. + +### Modules + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-Modules' + +$workflow | Add-PSFRunspaceWorker -Name Multiply -InQueue Numbers -OutQueue Results -Count 5 -ScriptBlock { + param ($Value) + Start-Sleep -Milliseconds 200 + [PSCustomObject]@{ + Input = $Value + Random = Get-RandomNumber + } +} -Modules C:\scripts\modules\MyModule\MyModule.psd1 -CloseOutQueue + +$workflow | Write-PSFRunspaceQueue -Name Numbers -BulkValues (1..20) -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName Multiply -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Results -All +$workflow | Remove-PSFRunspaceWorkflow + +$results +``` + +This will pre-load the module from source to ensure the exact correct version is loaded. +It is also possible to provide just the name (e.g. `ActiveDirectory`) to pre-load the module at the beginning. + +> Note: The PSFramework module is automatically added to this list. + +### Initial SessionState + +If the previous options are not enough, you can also add an entire pre-prepared initial session state object (The PowerShell way of defining what's initially available in a runspace): + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-SessionState' + +$initialSessionState = [initialsessionstate]::CreateDefault() +$workflow | Add-PSFRunspaceWorker -Name Multiply -InQueue Numbers -OutQueue Results -Count 5 -ScriptBlock { + param ($Value) + Start-Sleep -Milliseconds 200 + [PSCustomObject]@{ + Input = $Value + Random = Get-RandomNumber + } +} -SessionState $initialSessionState -CloseOutQueue + +$workflow | Write-PSFRunspaceQueue -Name Numbers -BulkValues (1..20) -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName Multiply -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Results -All +$workflow | Remove-PSFRunspaceWorkflow + +$results +``` + +> Please note: If the provided session state is not enough to run the basic module code, the workflow will fail. Use very much at your own risk! + +## Next Steps + ++ [Providing different values to different Runspaces](examples-perrunspacevariables.html) ++ [Data outside of the queue system](examples-the-data-field.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/examples-simple.md b/documentation/documents/psframework/runspace-workflows/examples-simple.md new file mode 100644 index 0000000..d25d70d --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/examples-simple.md @@ -0,0 +1,171 @@ +# Example: A simple workflow + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Description + +On a fundamental level, to get this system to work, we need to take the following steps: + +1. Create a Runspace Workflow +2. Add a worker for each step +3. Add the original input +4. Start the Workflow +5. Wait until it is completed +6. Receive results and end + +### 1: Creating a Runspace Workflow + +First we need to create a new [workflow object](workflow.html): + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow' +``` + +This is where all the other pieces of the puzzle get added to. +It is also registered within the PSFramework module, so you can later retrieve it using `Get-PSFRunspaceWorkflow`. + +We store it in a variable, as the later step require the object anyway. + +### 2: Adding the workers + +The workers perform the actual logic of the workflow. +For the sake of this example, we are performing a simple mathematical operation (something everybody can run on their own machine without any further dependencies). + +> 2a: The First Worker + +We take the input and generate an object that will end up showing the original input, the result after the first calculation (Processed) and later the final result after the second worker (Result). + +```powershell +$workflow | Add-PSFRunspaceWorker -Name Processing -InQueue Input -OutQueue Processed -Count 3 -ScriptBlock { + [PSCustomObject]@{ + Input = $_ + Processed = $_ * 2 + Result = $null + } +} +``` + ++ The `name` - "Processing" - is arbitrary, so long as it is unique. Pick something that explains what this step is about. ++ `InQueue` is the name of the queue providing the data being processed. The scriptblock gets executed once per item in that queue. ++ `OutQueue` is the name of the queue that output of the scriptblock gets written to. ++ `Count` is the number of parallel runspaces are used - in this case, three items from the `InQueue` are processed in parallel. ++ `ScriptBlock` is the logic doing the actual processing. The scriptblock receives a single input - the current item from the `InQueue` - and can be as complex as needed. +It is executed in the respective background runspace and does not see any of the outside variables, modules or commands. + +> 2b: The Second Worker + +In the second step, we will take the object produced by the first worker and calculate the `.Result` property, before passing it on + +```powershell +$workflow | Add-PSFRunspaceWorker -Name Result -InQueue Processed -OutQueue Done -Count 2 -ScriptBlock { + $_.Result = $_.Processed * 3 + $_ +} +``` + +### 3: Add the original input + +Workers get their work orders from their `InQueue`. +But how does the first worker receive its input? + +Well, we need to provide that information using `Write-PSFRunspaceQueue`: + +```powershell +$workflow | Write-PSFRunspaceQueue -Name Input -BulkValues (1..1000) +``` + +### 4: Start the workflow + +Once we are ready, we can launch the workflow, which will cause each worker to start its runspaces and will make things happen: + +```powershell +$workflow | Start-PSFRunspaceWorkflow +``` + +This will take a moment, but then things are happening in the background and we can either wait for it to complete or move on and do other things in the meantime. +We can also add more input after it is already running, repeating step 3 as needed. + +### 5: Waiting for the Workflow to complete + +If we want to wait until everything is done, we can use the `Wait-PSFRunspaceWorkflow` command. +There are different criteria based on which it can know to workflow is done, but in our current example, we assume once 1000 results have been gathered in the `Done` queue (the output of the second worker), our workflow is complete. + +```powershell +$workflow | Wait-PSFRunspaceWorkflow -Queue Done -Count 1000 +``` + +We should also stop the workflow at that point, as otherwise the background runspaces will remain open: + +```powershell +$workflow | Stop-PSFRunspaceWorkflow +``` + +This can be condensed to: + +```powershell +$workflow | Wait-PSFRunspaceWorkflow -Queue Done -Count 1000 -PassThru | Stop-PSFRunspaceWorkflow +``` + +Other examples show different ways for `Wait-PSFRunspaceWorkflow` to know when the workflow is done. +The [workers might know when they are done](examples-auto-close.html), for example. +But that is a story for documentation page other than the introductory example. + +### 6: Results and Cleanup + +Once everything has completed, it is time to receive the results (if any) and clean up the workflow: + +```powershell +# Retrieve results +$results = $workflow | Read-PSFRunspaceQueue -Name Done -All + +# Final Cleanup +$workflow | Remove-PSFRunspaceWorkflow +``` + +With that we have the result of the workflow in `$results` and everything is over. +There are more complex examples, that better show the benefit provided by this system (e.g. [Gathering data from Exchange Online, adding Active Directory information to it, then finally export everything to csv](examples-first-step-data.html)), but the flow shown above are the simple, basic implementation answering the "how?" question. + +## End-To-End Example + +Summarizing the explanation above, here is the full example script in one package: + +```powershell +# Create Workflow +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow' + +# Add Workers +$workflow | Add-PSFRunspaceWorker -Name Processing -InQueue Input -OutQueue Processed -Count 3 -ScriptBlock { + [PSCustomObject]@{ + Input = $_ + Processed = $_ * 2 + Result = $null + } +} +$workflow | Add-PSFRunspaceWorker -Name Result -InQueue Processed -OutQueue Done -Count 2 -ScriptBlock { + $_.Result = $_.Processed * 3 + $_ +} + +# Add input +$workflow | Write-PSFRunspaceQueue -Name Input -BulkValues (1..1000) + +# Start Workflow +$workflow | Start-PSFRunspaceWorkflow + +# Wait for Workflow to complete and stop it +$workflow | Wait-PSFRunspaceWorkflow -Queue Done -Count 1000 -PassThru | Stop-PSFRunspaceWorkflow + +# Retrieve results +$results = $workflow | Read-PSFRunspaceQueue -Name Done -All + +# Final Cleanup +$workflow | Remove-PSFRunspaceWorkflow +``` + +## Next Steps + ++ [What if there is no real input and the first step is supposed to generate it?](examples-first-step-data.html) ++ [Providing variables, functions or modules to the worker code](examples-resources.html) ++ [So we have Process, but where are my Begin and End blocks?](examples-begin-end.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/examples-the-data-field.md b/documentation/documents/psframework/runspace-workflows/examples-the-data-field.md new file mode 100644 index 0000000..584b13c --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/examples-the-data-field.md @@ -0,0 +1,54 @@ +# The Data Field: Exchanging Information without Queues + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Basics + +A documented example of the basic workflow setup [can be found on this page](examples-simple.html). + +## Description + +As shown in the basics and throughout all the other examples, the runspace workflow system depends heavily on queues to exchange data. +This has its ups - it's a central location for data and with queues, passing the data on also removes it from the queue, reducing the risk of memory leaks. + +On the other hand, that does not always work all that well after all, for example when caching information in a persistent manner that workers should have access to. +That is where the `Data` dictionary of the workflow itself comes in: + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-Data' + +$workflow | Add-PSFRunspaceWorker -Name Multiply -InQueue Numbers -OutQueue Double -Count 2 -ScriptBlock { + param ($Value) + Start-Sleep -Milliseconds 200 + if (-not $__PSF_Workflow.Data["Name"]) { $__PSF_Workflow.Data["Name"] = 'Fred' } + $Value * 2 +} -CloseOutQueue +$workflow | Add-PSFRunspaceWorker -Name Gather -InQueue Double -OutQueue Results -Count 1 -ScriptBlock { + param ($Value) + Start-Sleep -Milliseconds 200 + [PSCustomObject]@{ + Name = $__PSF_Workflow.Data["Name"] + Value = $Value + } +} -CloseOutQueue + +$workflow | Write-PSFRunspaceQueue -Name Numbers -BulkValues (1..20) -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName Gather -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Results -All +$workflow | Remove-PSFRunspaceWorkflow + +$results +``` + +The `$__PSF_Workflow` variable is provided to all worker runspaces and provides access to workflow information, including the thread-safe `.Data` dictionary/hashtable. +It is the exact same object as stored in the `$workflow` variable in the example above, making it absolutely possible to provide data from outside of the workflow, as an alternative to [providing variables](examples-resources.html), for example if the data should be updated and shared across all workers. + +> Note: Modifying the properties on the workflow object other than the `Data` property is not recommended and might affect workflow execution. + +## Next Steps + ++ [Providing variables, functions or modules to the worker code](examples-resources.html) ++ [Providing different values to different Runspaces](examples-perrunspacevariables.html) ++ [Throttling: The Art of not Spamming the Server](examples-throttling.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/examples-throttling.md b/documentation/documents/psframework/runspace-workflows/examples-throttling.md new file mode 100644 index 0000000..18b263c --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/examples-throttling.md @@ -0,0 +1,123 @@ +# Throttling: Don't hate your servers + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Basics + +A documented example of the basic workflow setup [can be found on this page](examples-simple.html). + +## Description + +Parallelization is a good way to swiftly hammer a server into submission under a flood of requests. +Sometimes they retaliate. + +In order to avoid problems with that kind of concern, it is possible to include throttling when creating a worker. + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-Throttling' + +$throttle = New-PSFThrottle -Interval '3s' -Limit 5 +$workflow | Add-PSFRunspaceWorker -Name S1 -InQueue Q1 -OutQueue Q2 -Count 10 -ScriptBlock { + param ($Value) + Start-Sleep -Milliseconds 200 + [PSCustomObject]@{ + Value = $Value + Stage1 = Get-Date + Stage2 = $null + } +} -CloseOutQueue +$workflow | Add-PSFRunspaceWorker -Name S2 -InQueue Q2 -OutQueue Q3 -Count 10 -ScriptBlock { + param ($Value) + $Value.Stage2 = Get-Date + $Value +} -CloseOutQueue -Throttle $throttle + +$workflow | Write-PSFRunspaceQueue -Name Q1 -BulkValues (1..20) -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName S2 -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Q3 -All +$workflow | Remove-PSFRunspaceWorkflow + +$results +``` + +> Wait until + +Sometimes, web services tell us how long we should be waiting for the next request. +This can be done with some slight adjustments: + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-Throttling2' + +$throttle = New-PSFThrottle -Interval '1m' -Limit 100 +$workflow | Add-PSFRunspaceWorker -Name S1 -InQueue Q1 -OutQueue Q2 -Count 1 -ScriptBlock { + param ($Value) + Start-Sleep -Milliseconds 200 + [PSCustomObject]@{ + Value = $Value + Stage1 = Get-Date + Stage2 = $null + } +} -CloseOutQueue +$workflow | Add-PSFRunspaceWorker -Name S2 -InQueue Q2 -OutQueue Q3 -Count 2 -ScriptBlock { + param ($Value) + if (10 -eq $Value.Value) { + $__PSF_Worker.Throttle.NotBefore = (Get-Date).AddSeconds(10) + } + $Value.Stage2 = Get-Date + $Value +} -CloseOutQueue -Throttle $throttle + +$workflow | Write-PSFRunspaceQueue -Name Q1 -BulkValues (1..20) -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName S2 -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Q3 -All +$workflow | Remove-PSFRunspaceWorkflow + +$results +``` + +> Note: The variable `$__PSF_Worker` represents the worker of the current runspace and is available in [all three phases](examples-begin-end.html) of the worker. + +## Queue Capacity Limits + +Sometimes we may want to limit, just how large a backlog a worker can generate. +In this case, we can assign a queue capacity limit - any attempt to add to it beyond that limit is going to block the caller until another runspace removes an item from that queue. + +Be aware that this carries the risk of deadlocks or hanging workers if misused. + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name 'ExampleWorkflow-Throttling3' + +# Limit Q2 to no more than 5 items +$workflow.Queues.Q2.MaxItemCount = 5 + +$workflow | Add-PSFRunspaceWorker -Name S1 -InQueue Q1 -OutQueue Q2 -Count 1 -ScriptBlock { + param ($Value) + [PSCustomObject]@{ + Value = $Value + Stage1 = Get-Date + Stage2 = $null + } +} -CloseOutQueue +$workflow | Add-PSFRunspaceWorker -Name S2 -InQueue Q2 -OutQueue Q3 -Count 2 -ScriptBlock { + param ($Value) + Start-Sleep -Second 1 + $Value.Stage2 = Get-Date + $Value +} -CloseOutQueue + +$workflow | Write-PSFRunspaceQueue -Name Q1 -BulkValues (1..20) -Close +$workflow | Start-PSFRunspaceWorkflow -PassThru | Wait-PSFRunspaceWorkflow -WorkerName S2 -Closed -PassThru | Stop-PSFRunspaceWorkflow +$results = $workflow | Read-PSFRunspaceQueue -Name Q3 -All +$workflow | Remove-PSFRunspaceWorkflow + +$results +``` + +Note how in this log the first ~10 values are on the same second in Stage1, thereafter moving to 2/second to match the processing speed of S2. + +## Next Steps + ++ [Automatically close queues and workers when done](examples-auto-close.html) ++ [Are all workflows a linear processing chain?](examples-multi-pronged-flows.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/queue.md b/documentation/documents/psframework/runspace-workflows/queue.md new file mode 100644 index 0000000..e6eda52 --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/queue.md @@ -0,0 +1,187 @@ +# The Queue + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Synopsis + +The queues are the way individual workers interact with each other - one worker's output-queue becomes another worker's input queue. +The queues are hosted on the [Workflow Object](workflow.html). + +Before executing the workflow, the original input of the workflow must be provided from outside of it - whether that is actual data or just a dummy value because the first worker is supposed to do the actual data collecting. + +## Defining Queues + +At no point will you need to "define" a queue - simply by using a queue name, it gets created as applicable: + ++ Defining a worker with queue names ++ Using `Write-PSFRunspaceQueue` ++ Using `Read-PSFRunspaceQueue` ++ Accessing the queues on the [Workflow Object](workflow.html) + +## Writing to the Queue + +The tool of choice to write to a queue is the `Write-PSFRunspaceQueue` command: + +> A Single value + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name QueueExample + +$workflow | Write-PSFRunspaceQueue -Name Input -Value 42 +``` + +> Multiple Values + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name QueueExample + +$workflow | Write-PSFRunspaceQueue -Name Input -BulkValues 42,23,1,5 +``` + +There are different parameters for this, as otherwise it would be impossible for a collection of values (such as an array) to be provided as a single value. + +> Close Call + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name QueueExample + +$workflow | Write-PSFRunspaceQueue -Name Input -BulkValues 42,23,1,5 -Close +``` + +The state of a queue is one of the ways we can determine, whether a workflow has concluded. +By [closing it](examples-auto-close.html) - as shown above - we prevent adding any more values to that queue. + +A [worker](worker.html) reading from this can know, that after it finished processing the last item from such a queue, there can be no further input and it is done. + +> From within a worker (Implicit) + +```powershell +$result +``` + +All output of a worker's runspace scriptblock is automatically queued to the specified output queue. + +> From within a worker (Explicit) + +This only works from within the runspace of a worker: + +```powershell +Write-PSFRunspaceQueue -Name user -Value $user +``` + +As each runspace has access to the workflow object, they can know what workflow they are a part of and where to look for a queue. +This can be useful when designing a [non-linear workflow](examples-multi-pronged-flows.html) + +> Using the [Workflow Object](workflow.html) + +```powershell +$workflow = New-PSFRunspaceWorkflow -Name QueueExample + +# As mentioned above: No need to explicitly create the queue first +$workflow.Queues.Input.Enqueue(42) +``` + +## Reading from a Queue + +By the nature of queues, reading from it and removing the item from it are usually synonymous. +Due to that, it is generally only useful to read from the output queue of the final worker. + +That said, the usual command to read from a queue is `Read-PSFRunspaceQueue`. + +> Read one Item + +```powershell +$workflow = Get-PSFRunspaceWorkflow -Name QueueExample + +$workflow | Read-PSFRunspaceQueue -Name Result +``` + +> Read all that remains in the queue + +```powershell +$workflow = Get-PSFRunspaceWorkflow -Name QueueExample + +$workflow | Read-PSFRunspaceQueue -Name Result -All +``` + +> From within a worker (Implicit) + +All worker runspaces automatically retrieve input from the queue configured for their input and provide that as an argument to the worker scriptblock. +Inside that code, simply define a param block to bind it to a proper variable: + +```powershell +param ($Value) +``` + +For a simple example. [check out the introductory example](examples-simple.html) + +> From within a worker (Explicit) + +As the worker is aware of what workflow it is part of, reading from a specific queue of your choice is simpler: + +```powershell +Read-PSFRunspaceQueue -Name input +``` + +As this is usually not needed - a worker runspace receives its input as argument (see above) - this is a less likely scenario, but the precursor of a more interesting option: + +> Continual Output + +This is intended within a worker that needs to pipe values from a queue to another command: + +```powershell +Read-PSFRunspaceQueue -Name mail -Continual | Get-Mailbox +``` + +With that a steady stream of input values will be piped to the subsequent command, `Read-PSFRunspaceQueue` blocking, waiting for the next item, until its queue [gets closed](examples-auto-close.html). + +> Using the [Workflow Object](workflow.html) + +```powershell +$workflow = Get-PSFRunspaceWorkflow -Name QueueExample + +# As mentioned above: No need to explicitly create the queue first +$workflow.Queues.Output.Dequeue() +``` + +## Configuring Queues + +Queues actually have a few settings that can be applied. +These usually are only relevant in rarer scenarios ... but then they might come in handy: + +> Closing a Queue + +Queues that are closed no longer accept input (they will silently ignore it). +This signals the workers that no more values can be expected from that queue and is part of signaling the end of a workflow. + +This is covered extensively in the [example for closing queues](examples-auto-close.html). + +To close a queue, there are two ways: + ++ As part of a `Write-PSFRunspaceQueue` call (usually to finalize the original input queue; see above) ++ On the [Workflow Object](workflow.html) + +```powershell +$workflow = Get-PSFRunspaceWorkflow -Name QueueExample +$workflow.CloseQueue("processing") +``` + +> Maximum number of Items in the queue + +In really rare cases, it might be useful to prevent a queue from gathering too much data. +Imagine one worker providing data faster than the next worker can process - in the long run, this is going to bloat the memory until we run out. +Not good. + +Using the `MaxItemCount` property on a queue, it becomes possible to limit the number of items that can be in a queue at any given time. +Trying to add more to it will then block the call until another thread removes an item from the queue. + +Please note that this can lead to deadlocks or failed tasks (e.g. paging timeouts on the data source). + +```powershell +$workflow = Get-PSFRunspaceWorkflow -Name QueueExample +$workflow.Queues.Processing.MaxItemCount = 2000 +``` + +For more details on that and a full example implementation, see [the throttling example](examples-throttling.html) + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/worker.md b/documentation/documents/psframework/runspace-workflows/worker.md new file mode 100644 index 0000000..607a95e --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/worker.md @@ -0,0 +1,10 @@ +# The Worker + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Description + +Nothing to see here (yet). +Check out the [Introductory Example](examples-simple.html) and take it from there. + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file diff --git a/documentation/documents/psframework/runspace-workflows/workflow.md b/documentation/documents/psframework/runspace-workflows/workflow.md new file mode 100644 index 0000000..ad07f8c --- /dev/null +++ b/documentation/documents/psframework/runspace-workflows/workflow.md @@ -0,0 +1,10 @@ +# The Workflow Object + +[Back to Runspace Workflows](../runspace-workflows.html) + +## Description + +Nothing to see here (yet). +Check out the [Introductory Example](examples-simple.html) and take it from there. + +[Back to Runspace Workflows](../runspace-workflows.html) \ No newline at end of file