Task Flow
An entity rarely leaves after completing just one task. An order is checked, perhaps wrapped as a gift, packed, sometimes packed again, and shipped. This page shows how a finished task chooses the next one, what the entity carries from task to task, and which values a task draws and reads on the way.
What It Is
A flow is the sequence of tasks one entity passes through. Each task names what follows it: always the same task, the first of several rules whose condition holds, or a task drawn at random by weight. A rule can send the entity back to a task it has passed, for rework. A task that names nothing ends the flow. The entity keeps its key and its attributes through the whole flow, and every task can read them.
Each task also has a key for its records, can draw values for its own use, and can read the records that earlier tasks published.
A Flow of Tasks builds a pick, pack and ship flow step by step.
How to Use It
Choose the Next Task
next takes one of these forms:
- A task name,
next: pack_order: always that task. - Rules,
whenwith an optionaldefault: each rule is a condition and a task. Rules are tried in order. The first matching rule wins, anddefaultapplies if none matches. With no default and no matching rule, the flow ends. - Weights, below.
- In the builder, a function that takes the
Upstreamand returns a task name orNone.
| docs/snippets/yaml/task_flow.yaml:condition | |
|---|---|
A Router takes the same rules as next. When a task finishes, its process builds the Upstream, asks route.choose(upstream, env, levels=...) for the next task's name, and starts that task with the same entity. env lets a condition read registry[...] and t. levels, a mapping from store and container names to the objects, lets it read level.<name>.
The condition is evaluated when the task finishes, after its set. payload. reads the payload of the task that just finished, attr. the entity's attributes, level. a store's or container's level, and registry["..."] a live parameter. How It Works lists everything a condition can use.
Weights
weights maps task names to numbers or expressions, and the next task is drawn in proportion to them. The draw comes from a random stream of the routing task's own, route.<task>, so a seeded run repeats it and it changes no other draw:
| docs/snippets/pages/task_flow.py:builder-weights | |
|---|---|
| docs/snippets/pages/task_flow.py:low-level-weights | |
|---|---|
A weight written as an expression is read when the task finishes. It may use attr.<name>, registry["..."], level.<name>, t and the task's vars, as weights: {ship_order: 9, pack_order: "attr.rework"} does. A weight of infinity or true is refused when the blueprint loads. A weight below 0, or text that is not a number, fails the run with the weight's name. Weights that add up to 0 fail the run.
Attributes
An arrival gives each new entity its starting attributes, and a task's set changes them when the task finishes. Each entity gets its own copy. Attributes are not published: a condition reads them as attr.<name>, and a payload includes them only when it asks.
| docs/snippets/yaml/task_flow.yaml:set | |
|---|---|
A task's attributes list adds the named attributes to a mapping payload, as ship_order does in the full blueprint, task_flow.yaml.
An Entity(key, arrival, attributes) carries them. The process updates entity.attributes itself after a task, as the low-level tab of A Flow of Tasks shows.
Record Key and Fields
Each arrival starts a new entity, keyed by the arrival's name and a counter from 1: orders-1, orders-2. Every event of every task in its flow carries that key as its record key, so a sink can follow one order through the whole flow. Kafka sends records with the same key to one partition, so it keeps one order's events in order. The builder can replace the key with a function of the counter:
A blueprint arrival uses the default key.
The process that creates each Entity gives it its key.
Below are the finished events of gifts-5 in task_flow.yaml, the blueprint whose parts this page shows. The gift order is checked while the shelf has stock, so it is not backordered and its shipping record shows backordered: None. It is wrapped, packed and shipped, all under one key:
A task can also set the key of its own records. A blueprint task has these keys, and the builder's @app.task takes the same names:
key_fieldnames a payload field. Its value becomes the key of the task'sfinishedrecord, and so the Kafka message key. Thequeuedandstartedrecords keep the entity's key. A payload without the field fails the run.keyis a value place for the key of all the task's records: text, a whole number or a generator. It is drawn once for each instance, at its first record, and written as text. Writekeyorkey_field, not both.id_fieldadds the task's instance id to a mapping payload under that name,item_fieldadds the item the task's storegettook, andattributesadds the named entity attributes. They need a mapping payload, and a null payload fails the task.payload: nullpublishes nofinishedrecord, and the task runs its other steps. It is refused together with a storeput,key_field,id_field,item_field,attributesand apayload.read.
A process gives the key to env.publish_event(key, value, ...) itself, as the low-level tab of Tasks shows, so it can publish any record under any key.
Upstream
A task started by next receives what the task before it handed on, the Upstream: the finished task's instance id, the entity's key and attributes, and the finished task's payload. A builder function gets it by declaring a parameter called upstream, and gets the entity by declaring entity. A function that declares neither is called without them. The first task of a flow gets upstream=None.
A task that on_fail, on_renege or on_balk reaches reads the record that sent it there as upstream.status and, after a failure, upstream.reason.
A blueprint task has no function to declare upstream. Its payload names the attributes it carries, its generators read the finished task's payload as {gen: upstream, path: <field>}, and an expression reads it as upstream.<field>. Its next conditions read the finished task's payload as payload.<field> and the entity's attributes as attr.<name>.
A low-level process holds the order and its attributes itself. It can build an Upstream and call Router.choose, as in Choose the Next Task, or decide the next step in plain Python.
Values a Task Draws
A task can draw values for its own use. They come in three groups:
varsare drawn in order, once for each instance of the task, and each can read the vars above it.vars_onceare drawn once for each entity and task, at the first use. A task that an entity runs again, as a rework does, keeps itsvars_once.finish_varsare drawn after the service and can read the vars.
on_start values are published as one record when the task starts. A task that has on_start draws its vars when it starts. Without on_start, the task draws them after the service, just before it builds its payload, and a service_time that reads a var draws it first.
A payload reads a var as {gen: var, name: <name>}. The expr and format generators and the put, fail, next.when and next.weights expressions of the same task read it by its bare name. A wait timeout value may read vars_once names but no vars. A var name is an identifier that is not a name of the expression language (all, any, len, sum, cause, waiting and the others) and not the name of a var that is visible there. It is in one group only. A task that can end early (patience, balk, wait, on_interrupt: abandon or queue_limit) refuses an on: value that reads a var.
A process draws its values with its own Sampler generators and keeps them in Python variables, so it has no groups to order. A generator it names, such as sampler.generator("pack.weight"), draws from its own stream of the run's seed.
Every generator node draws from its own stream of the run's seed, so adding a value does not change the draws of the others.
Read Earlier Records
A task can read the records that tasks finished before it. A generator reads them, in place of a function that keeps its own list:
lookuptakes a field of a row of another task's finished payloads, chosen at random unlessstrategyisfirstorlast.histogramsplits the rows, in the order they were written, into bins with a share each,latest_bykeeps the newest row of each value of a field, anddistinctdraws without replacement inside onerepeat.previoustakes a field of the last finished payload of the same task on the same stream. The stream is the arrival, or the entity for an arrival withcountorkeys. It gives null when there is none and nodefaultis written.upstreamtakes a field of the payload of the task that routed here.aggregategives themean,sum,min,maxorcountof a field over the last rows of a task.where, an expression over the row's fields, keeps some of them. It does not wait for a row.
A task keeps its last history finished payloads, one million by default, when a lookup or aggregate in the file names it or when history is written. Look Up Other Records and Build on the Last Value work through them.
A generator that reads records can also set how long a task takes. These tasks run one after another. deliver takes the mean weight of the last 5 packings over 1 kg, restock the newest weight of one customer, and inspect two past weights drawn without replacement, mostly from the older part of the rows. settle takes as long as the last settling, and 2 seconds for the first, and finish as long as the settling before it:
A process keeps the earlier values itself, in a list or a variable, and reads them where it needs them.
A task whose generators look up another task does not start before that task has a row. An arrival, a spawn or a next that would start it is dropped, with one INFO line that gives the wait in simulation time, and a task that never started gets one more line at the end of the run. A dropped start runs none of the task's on: actions. A lookup in an arrival place fails when there is no row. In where, a field of the row wins over t.
The blueprint is refused when it loads if no task starts the target of a lookup, if the path is not in the target's payload, if lookups form a cycle, if previous is outside a task, if upstream is in a task that no next reaches or that on_fail or on_interrupt_next also reaches, or if distinct is outside the value of a repeat.
Carry Values and Switch Arrivals
A task can write names of its own process with set_args. This task and the tasks that follow in the same flow read them as arg.<name>. A child flow has its own, and a parent passes names to it with args on a task branch of its wait or on a spawn. A spawn evaluates args once for each child. A read of a name that was neither passed nor written is a run failure. set_args is read after set, so a payload reads the value from before the write.
activate and deactivate turn the sources of the named arrivals on and off when the task finishes, after set_args. Each must name an arrival that has a spawn, and the blueprint and the builder refuse a name that is not an arrival. A source with count or keys obeys them too. Under off_mode: drop an arrival that falls while its source is off is lost and takes no key number. Under restart the source starts a whole new gap when it goes on. Arrivals and Services describes the arrival keys.
A process passes values to the next one as plain arguments, and decides in its own code which arrival processes keep running.
Below are the records of task_values.yaml that these sections show: the first sign-up, keyed by its customer id; the first order, which looks up customer c-0 and starts its running total; the packing of that order, with the weight published at its start and the finished record with the box, the weight, the fee and the order's total; and the day report that the shop's closing starts at 100 seconds.
How It Works
Conditions. A condition is a Python expression, parsed with Python's ast module and checked against a list of what is allowed before it is ever evaluated. Nothing goes through eval. Each name carries its source as a prefix, so adding a payload field cannot change what an attribute of the same name means. Names an Expression Reads lists every name by the place an expression sits in:
| Name | Value |
|---|---|
payload.<field>, attr.<name>, attr["<name>"] |
a field of the payload of the task that just finished, an attribute of the entity, and an attribute whose name Python reads as a word |
level.<name> |
the level of a store or container |
users.<name>, queue.<name> |
how many units of a resource are held, and how many requests wait for it (Units and Queues) |
putters.<name>, getters.<name> |
how many puts and gets wait at a store or container |
count(<store>), items(<store>) |
how many items a store holds, and the list of them (Stores and Containers) |
item, item.<field>, item_priority |
the item the task's store get took, a field of it at any depth, and the priority a priority store gave it (Stores and Containers) |
arg.<name>, cause.<name>, upstream.<field> |
a name set_args wrote or a parent passed, why the task was interrupted, and a field of the payload of the task that routed here |
held(<resource>), waiting(<resource>) |
the holders in grant order and the waiting requests in queue order, as mappings with entity, rank, time, preempt and key, and since for a holder |
putting(<store>), getting(<store>) |
the puts and gets waiting at a store or container, with entity and, for a put, item or amount |
running(<task>), running(<task>, <condition>) |
how many instances of a task run, or meet the condition |
wait.<branch>.done, .ok, .value, signal.<name>.count, .ok, .value, .processed, handle.<name>.triggered, .processed, .ok, .value, .alive, done_count |
the task's wait, the signals, the names the task opened, and in the when of a wait group how many branches are done (Signals and Waiting) |
process.name, .instance, .target, .key, .flow_name |
the running process (Processes) |
registry["<path>"] |
a live parameter, written whole because its path contains dots |
t, peek() |
the simulation time in seconds, and the time of the next scheduled event |
| a var name | a var of the task, read bare in put, fail, next.when, next.weights, expr and format |
It may also use numbers, text, True, False and None; the arithmetic +, -, *, /, //, % and ** (a power above 10,000, or one with a result of more than 1,000,000 bits, is refused); the comparisons <, <=, >, >=, ==, !=, is None, in and not in; and, or, not and a if c else b; list and mapping literals, subscripts, slices and one-clause list comprehensions; and the functions abs, min, max, round, floor, ceil, sqrt, exp, log, log10, sin, cos, tan, len, sum, sorted, range, any, all, str, int, float, bool, has(attr.<name>), get_key, set_key and drop_key. Text may be compared, indexed, added with + and passed to the key functions, and % on text is refused. One reading of an expression that makes more than 1,000,000 items in all is a named run failure. Anything else is refused with a message that names it:
A blueprint condition that uses anything else is refused when the blueprint is built, with the same message.
A low-level Router takes these conditions, as in Choose the Next Task, and a process can also decide in plain Python.
A blueprint is checked when it loads, and each error names its YAML line. Every condition must parse, and every task a rule names must exist. Every level.<name> must name a store or container, every users.<name> and queue.<name> a resource, every putters.<name> and getters.<name> a store or container, and every count(...) and items(...) a store. Every registry["..."] must be a path of the model. An item.<field> in a place that no get reaches is refused. The builder makes the same checks when run() starts. A name that is missing when a condition is evaluated, such as a payload field, raises an error naming the expression.
Weights. Weights that are numbers are turned into cumulative shares once. Weights that are expressions are read each time the task finishes. Each choice draws one uniform number from the routing task's stream and picks the task whose share it falls in.
Starting the Next Task. When a task finishes, it applies set, builds the Upstream and asks its rules for the next task. It runs that task next, in the same SimPy process and the same step, with the same entity and the next instance id from the task's counter. An arrival that spawns the task uses the same counter, so no two runs of a task share an id. In SimPy terms, the next task is the next statement of one process. SimPy and desflow describes processes.
Stability and Limits
- Flow balance. The rate into each task follows from the rates into the tasks before it and the routing shares. With weights of 9 to ship and 1 back to packing, one packing in ten sends the order back to be packed again. Every order is packed once, one in ten a second time, one in a hundred a third time, and so on. An order is therefore packed 1 + 0.1 + 0.01 + ... = 1 รท 0.9 = 1.11 times on average, so the packers see 11% more work than there are orders. Weights that are expressions give
desflow explainno shares, so it reports no flow balance for them. - Rework raises utilisation. Work the load out with the rework included, as Queues and Capacity does, or a rework loop can push a resource past full use. Queueing Basics explains utilisation.
- Refused expressions. A call other than the listed functions, a name without a source,
payload["total"](writepayload.total), a comprehension with more than one clause or over a mapping,lambda,~and the shift operators are refused.**with a negative base and a fractional exponent, andround(x, -n)with a very largen, fail the run when the expression is evaluated, naming the expression. An expression nested more than 200 levels is refused when it is read. - A loop needs time. A loop of tasks that take no service time, resource, wait or get never lets the clock move. The run stops with an error that names the tasks after the flow has run 10,000 instances at one instant, as a resource breakdown does. Give one task of the loop a
service_timeabove 0, or a way out in itsnextrules. - One entity, one flow. A task leads to at most one next task. Splitting one order into several, or joining several into one, comes after 1.0.
decideis kept free. The task keydecideis kept for decisions sent from outside for one entity, planned after 1.0. Today it is refused like any unknown key.
Example: A Flow of Tasks
In the flow example, orders are picked, gift orders are wrapped, one parcel in ten is packed again, and every order is shipped. Wrapping and shipping are timed steps. It needs no Docker service and ends on its own.
The tutorial page A Flow of Tasks teaches these features step by step.
Quick Start
desflow examples get downloads the example at the release you install, into a folder of its own. Pick uv or pip:
The whole file is examples/yaml/flow.yaml.
Quick Start
desflow examples get downloads the example at the release you install, into a folder of its own. Pick uv or pip:
The whole file is examples/builder/flow_example.py.
Quick Start
desflow examples get downloads the example at the release you install, into a folder of its own. Pick uv or pip:
| examples/low-level/flow_example.py:routes | |
|---|---|
The whole file is examples/low-level/flow_example.py.