API Reference
Simulation Context
desflow.SimulationContext(sim_id, factor=1.0, random_seed=None, logical_start_time=None, go_live_at=None, warmup=None, initial_time=0.0, lag_interval=None, timezone=None, strict=None, environment=None)
Builder and Orchestrator for Event-Driven Digital Twins.
The SimulationContext implements the Builder Pattern to construct the
foundational architecture of a Digital Twin. It acts as the central control
plane, aggregating discrete machine capacities, continuous physical states
(wear and tear), and statistical distributions into a unified SimParameter
object.
Its public methods fall into four groups:
- Declaring parts:
from_yaml,add_resource,add_container,add_store,add_breakdown,add_variable,add_service,add_arrival,add_initial_arrival,add_arrival_replay,add_signal,add_scenario,add_step,add_readings,add_process,add_ingress,add_egress,with_stats,with_batching,task,note_values,arrival_loopandtelemetry_loop. - Checks made before a run:
family_read_problem,compile_parameters,registry_pathsandanalyze, which read the declarations;analyze(verify=True)also builds the model again and runs it.start()andanalyze()check the declarations before they build anything, and raiseValueErrorfor the first that is wrong;run()checks them throughstart(). - Running:
run,start,step,peek,advance,finish,closeand theenvproperty. - Calls a task makes during a run:
publish,spawn,subscribe,call,signal,interrupt,get_resource,get_store,get_container,next_instance,wait_for_arrivalandstop_arrivals.
Architectural Guarantees:
1. Immutability During Configuration: The underlying DynamicRealtimeEnvironment
and its background asynchronous I/O threads are completely deferred until
.run() is explicitly invoked. This prevents partial state leaks or
unintended clock starts.
2. One registry: All registered configurations are flattened
and managed by the internal SimulationRegistry, allowing external
Kafka Admin commands to mutate these states safely during runtime.
3. Deterministic Stochasticity: The Sampler gives each distribution a
random sequence of its own, made from random_seed, so a seeded run repeats
exactly, and adding a distribution leaves the others' draws unchanged. A run
without random_seed draws a seed and logs it, so it can be repeated.
Attributes:
| Name | Type | Description |
|---|---|---|
sim_id |
str
|
The unique namespace for this simulation instance (e.g., 'HotRolling'). |
factor |
float
|
The real-time synchronization multiplier. A factor of 1.0 syncs strictly with wall-clock time, while 0.0 executes instantly in a fast-forward loop (ideal for historical data processing). |
go_live_at |
Optional[datetime]
|
Logical instant at which |
warmup |
Optional[float]
|
The end of the warm-up, in simulation seconds. |
random_seed |
Optional[int]
|
The run's seed. When None, |
sampler |
Optional[Sampler]
|
The centralized Random Number Generator engine.
Accessible to user-defined raw generators only after |
on_change |
List[Callable[[ParamChange], None]]
|
Callbacks that receive a
|
Initializes the context builder with a designated namespace and temporal factor.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sim_id
|
str
|
The unique prefix for this group of parameters. All generated telemetry and event paths will be prefixed with this ID. |
required |
factor
|
float
|
The real-time synchronization factor. Defaults to 1.0. |
1.0
|
random_seed
|
Optional[int]
|
Optional seed for deterministic execution. |
None
|
logical_start_time
|
Optional[datetime]
|
Optional override for the environment's base clock,
forwarded to |
None
|
go_live_at
|
Optional[datetime]
|
Optional logical instant at which the run stops using |
None
|
warmup
|
Union[float, str, None]
|
The end of the warm-up, in simulation seconds or as a duration
such as |
None
|
initial_time
|
Union[float, str]
|
The simulation clock's starting value, in seconds or as a
duration such as |
0.0
|
lag_interval
|
Union[float, str, None]
|
Simulation seconds, or a duration such as |
None
|
timezone
|
Optional[str]
|
An IANA time zone name, such as |
None
|
strict
|
Optional[bool]
|
When True a step raises |
None
|
environment
|
Optional[Type[DynamicRealtimeEnvironment]]
|
A subclass of |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 | |
Attributes
env
property
Read-only handle to the active DynamicRealtimeEnvironment.
Grants raw generators access to environment primitives such as
publish_event, timeout, now, and start_datetime without
reaching into private builder state.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If accessed during the Builder Phase (before |
Methods:
add_arrival(name, dist=None, *, rate=None, mean=None, std=None, low=None, value=None, high=None, mode=None, shape=None, scale=None, values=None, weights=None, edges=None, counts=None, p=None, gap=None, attributes=None, key=None, spawn=None, first_at=None, group=1, active=True, off_mode='drop', initial=0, max_events=None, vars=None)
Registers a statistical distribution defining the inter-arrival times of entities.
Each arrival starts a new entity, which wait_for_arrival returns. Its key,
the record key of every event it causes, is the arrival name and a counter
(orders-1, orders-2) unless key gives another.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The internal identifier (e.g., 'orders'). |
required |
dist
|
Optional[str]
|
The family, as for |
None
|
rate
|
Optional[float]
|
Events per second, for exponential. |
None
|
mean
|
Optional[Union[float, str]]
|
The mean, for normal, lognormal, truncated_normal and poisson. |
None
|
std
|
Optional[Union[float, str]]
|
The standard deviation, for the first three (0 when not given). |
None
|
low
|
Optional[Union[float, str]]
|
The lowest value, for truncated_normal, uniform and triangular. |
None
|
value
|
Optional[Union[float, str]]
|
The time a constant distribution always gives. |
None
|
high
|
Optional[Union[float, str]]
|
The highest value, for uniform and triangular. |
None
|
mode
|
Optional[Union[float, str]]
|
The most likely value, for triangular. |
None
|
shape
|
Optional[float]
|
The shape, for gamma and weibull. |
None
|
scale
|
Optional[Union[float, str]]
|
The scale, for gamma and weibull. |
None
|
values
|
Optional[Sequence[Union[float, str]]]
|
The values an empirical draws from, taken at load. |
None
|
weights
|
Optional[List[float]]
|
One weight per empirical value. |
None
|
edges
|
Optional[Sequence[Union[float, str]]]
|
The bin edges of an empirical histogram. |
None
|
counts
|
Optional[List[float]]
|
One count per empirical histogram bin. |
None
|
p
|
Optional[float]
|
The chance of success on each try, for geometric. |
None
|
gap
|
Any
|
The time to the next arrival in place of |
None
|
attributes
|
Optional[Mapping[str, Any]]
|
The attributes each entity starts with, read in conditions
as |
None
|
key
|
Any
|
The entity's key. A function of the entity's counter, from 1, returning
its key; or a value as a blueprint's |
None
|
spawn
|
Optional[str]
|
The task each arrival starts, with the arrival's entity and the
task's next instance id, as YAML's |
None
|
first_at
|
Union[float, str, None]
|
The simulation time of the first arrival, in seconds or as a
duration such as |
None
|
group
|
Any
|
How many entities one arrival starts, with |
1
|
active
|
bool
|
False starts the source off, with |
True
|
off_mode
|
str
|
With |
'drop'
|
initial
|
int
|
With |
0
|
max_events
|
Any
|
With |
None
|
vars
|
Optional[Mapping[str, Any]]
|
Named values read once for each entity, before its attributes and
key, as a blueprint's arrival |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the family is unknown, or a parameter is missing, out of
range or not one the family takes, or both or neither of |
Source code in src/desflow/builder/context.py
1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 | |
add_arrival_replay(name, source, *, time='time', unit='s', fields='all', on_invalid='error', attributes=None, key=None, spawn=None, active=True, off_mode='drop', vars=None)
Registers an arrival taken from recorded rows instead of a distribution.
Each row becomes one arrival at its recorded time, in the order given, and
its fields become the entity's attributes. wait_for_arrival(name) returns
the wait for the next row, so an arrival loop is written as for any other
arrival. Rows at equal times arrive at that time in their order; rows at or
after until never arrive; after the last row the arrival stops, with one
INFO line. The rows are read and checked when the run starts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The arrival's name. |
required |
source
|
Union[str, Path, Iterable[Mapping[str, Any]]]
|
A CSV or Parquet file (Parquet needs the |
required |
time
|
str
|
The column of each row's time. |
'time'
|
unit
|
str
|
|
's'
|
fields
|
Union[str, List[str]]
|
The columns each entity carries, or |
'all'
|
on_invalid
|
str
|
|
'error'
|
attributes
|
Optional[Mapping[str, Any]]
|
Attributes every entity starts with; a row's fields of the same name replace them. |
None
|
key
|
Any
|
The entity's key, in every form |
None
|
spawn
|
Optional[str]
|
The task each row starts, as for |
None
|
active
|
bool
|
False starts the source off, as for |
True
|
off_mode
|
str
|
|
'drop'
|
vars
|
Optional[Mapping[str, Any]]
|
Named values read once for each entity, before its attributes and
key, as for |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 2226 2227 2228 2229 2230 2231 2232 2233 2234 2235 2236 2237 2238 2239 2240 2241 2242 2243 2244 2245 2246 2247 2248 2249 2250 2251 2252 2253 2254 2255 2256 2257 2258 2259 2260 2261 2262 2263 2264 2265 | |
add_breakdown(resource, time_to_failure, repair, work='interrupt', clock='calendar')
Makes a resource break down at random and be repaired.
After each repair, or the start, the time to the next failure is drawn. The
calendar clock counts it from then, busy or idle. The busy clock counts
it only while the resource holds a unit, so an idle resource does not wear out.
While the resource is down its capacity is 0, its current_cap keeps the
planned value, and a change to it during the repair sets the capacity the
resource returns to.
Each failure and repair publishes an event keyed by the resource's name,
{"status": "down", "reason": "failure"} and {"status": "repaired", ...}.
The two distributions are registry paths,
<sim_id>.breakdown.<resource>.time_to_failure and .repair.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
resource
|
str
|
The resource's name. |
required |
time_to_failure
|
Mapping[str, Any]
|
A distribution, such as
|
required |
repair
|
Mapping[str, Any]
|
A distribution, such as |
required |
work
|
str
|
|
'interrupt'
|
clock
|
str
|
|
'calendar'
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a distribution, |
Source code in src/desflow/builder/context.py
add_container(name, current_cap=None, max_cap=None, level=0.0, type=None, get_queue=None, put_queue=None, family=False, members=None)
Registers a container: an amount that tasks take from and add to, such as stock.
current_cap is the capacity, the largest level, as for a resource. The
level starts at level and has a live registry path of its own,
<sim_id>.containers.<name>.level, so a scenario or an ingress message can
set it during the run, such as a restock. Models that used current_cap as a
level, such as wear, use level instead.
A container with no current_cap has no limit, and max_cap defaults to
current_cap. Use math.inf or float("inf") for no limit.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The internal identifier (e.g., 'stock'). |
required |
current_cap
|
Union[float, str, None]
|
The capacity at simulation start. None, the default, is no
limit, or |
None
|
max_cap
|
Union[float, str, None]
|
The largest capacity a live update can set. None, the default,
is |
None
|
level
|
Union[float, str]
|
The amount held at simulation start. Defaults to 0. |
0.0
|
type
|
Optional[Type[ContainerBase]]
|
A subclass of |
None
|
get_queue
|
Optional[Mapping[str, Any]]
|
A mapping |
None
|
put_queue
|
Optional[Mapping[str, Any]]
|
As |
None
|
family
|
bool
|
When True, the entry is a family of containers, as for
|
False
|
members
|
Any
|
With |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 | |
add_egress(provider, when=None, batch_size=None, flush_interval=None, envelope=True)
Registers an asynchronous egress provider for data exfiltration.
Every registered provider receives every record, so attaching a stream sink and
a lake sink writes the same dataset to both in one pass. Pass when to send
only some records to a provider, which reads the same way as the path_router
that ParquetStorageEgress already accepts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
provider
|
Any
|
An initialized connector subclassing |
required |
when
|
Union[Callable[[dict], bool], str, None]
|
Optional predicate taking one record and returning True to send it to
this provider. None sends every record, which is the usual case.
|
None
|
batch_size
|
Optional[int]
|
Records this provider buffers before flushing. None uses the
context default from |
None
|
flush_interval
|
Optional[float]
|
Simulation seconds this provider waits before forcing a
flush. None uses the context default. Ignored when |
None
|
envelope
|
bool
|
False makes this a plain-output provider. It receives no
telemetry record and no |
True
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Each provider has its own buffer and its own queue of batches. A record that several providers take is stored once, and stays in memory until the last provider that holds it has written it.
Example
Source code in src/desflow/builder/context.py
561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 | |
add_ingress(provider)
Registers an asynchronous ingress provider to listen for external mutations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
provider
|
Any
|
An initialized connector subclassing |
required |
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Source code in src/desflow/builder/context.py
add_initial_arrival(name, initial, *, spawn, attributes=None, key=None, gap=None, vars=None)
Registers an arrival that has only initial entities and no source: initial
entities exist when the run starts, in creation order, before any step runs,
each keyed <name>-n unless key gives another, and each starts spawn. A
blueprint's arrival with initial alone.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The arrival's name. |
required |
initial
|
int
|
How many entities, 1 or more. |
required |
spawn
|
str
|
The task each entity starts. |
required |
attributes
|
Optional[Mapping[str, Any]]
|
The attributes each entity starts with. |
None
|
key
|
Any
|
The entity's key, in every form |
None
|
gap
|
Any
|
With a gap the arrival is not sourceless: it also starts an entity
after each gap once the first |
None
|
vars
|
Optional[Mapping[str, Any]]
|
Named values read once for each entity, before its attributes and
key, as for |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
add_process(func, **kwargs)
Registers a generator function to start when the run starts.
This is the builder form of spawn, for a process that is not an arrival or
telemetry loop, such as a drift engine or a scripted scenario. It is how a
YAML blueprint starts the processes it references with !python.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
func
|
Callable
|
A generator function. It is called as |
required |
**kwargs
|
Any
|
Extra keyword arguments passed to |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Example
Source code in src/desflow/builder/context.py
add_readings(interval, readings)
Publishes expressions of the run as telemetry every interval seconds.
It is the builder form of a YAML telemetry entry with readings, and
telemetry_loop with a function that publishes values, written as
expressions. Each reading is evaluated over the run (level., users.,
queue., signal., registry[...], t and the functions of the expression
language, and no name of an entity or a task), at the start and then every
interval. Each interval publishes one telemetry record for each reading, in
the order given, named <sim_id>.<reading> as publish names one. The
readings of one interval are all evaluated before the first is published.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
interval
|
Union[float, str]
|
Simulation seconds, or a duration such as |
required |
readings
|
Mapping[str, str]
|
A reading name mapped to an expression, such as
|
required |
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 2111 2112 2113 2114 2115 2116 | |
add_resource(name, current_cap=None, max_cap=None, queue=DEFAULT_QUEUE_ORDER, preempt=False, queue_limit=None, type=None, family=False, members=None)
Registers a discrete logical resource capable of processing tasks.
Resources act as the primary gating mechanism in the simulation, blocking concurrent processes to prevent physical collisions (e.g., restricting the number of slabs inside a mill stand).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The internal identifier (e.g., 'mill_structural'). |
required |
current_cap
|
Optional[Union[int, float]]
|
The active physical token capacity at simulation start, a
whole number or |
None
|
max_cap
|
Optional[Union[int, float]]
|
The absolute physical maximum bound of tokens allowed. Defaults
to |
None
|
queue
|
Union[str, Mapping[str, Any]]
|
The order waiting tasks are served in: |
DEFAULT_QUEUE_ORDER
|
preempt
|
Union[bool, Mapping[str, Any]]
|
When True, a task of a lower priority value takes a unit from
a task of a higher one when every unit is busy, as SimPy's
|
False
|
queue_limit
|
Optional[int]
|
The most requests that may wait, or None for no limit. A
task that finds that many waiting balks; a request made in the
low-level API raises |
None
|
type
|
Optional[Type[ResourceBase]]
|
A subclass of |
None
|
family
|
bool
|
When True, the entry is a family: a set of resources, one for each
key, made from this entry. A task names a member with |
False
|
members
|
Any
|
With |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 | |
add_scenario(steps)
Adds scenario steps: changes and records at set simulation times.
Each step is a mapping: at, seconds or a duration from the start of the
run, and a path and value to set in the registry, a record to publish
as a ground-truth label, an interrupt ({"task": ..., "where": ...,
"cause": ...}), or more than one. A step applies before every other event
at its time, so an arrival gap drawn at that moment uses a new rate; a step
at or after until never applies. Steps at equal times apply in the order
given.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
steps
|
List[Mapping[str, Any]]
|
The steps. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a step is malformed. A path the model does not have is
refused when the run starts, a path on a member of a family too when
the family can never have it: no task names the family with a |
Source code in src/desflow/builder/context.py
add_service(name, dist, *, rate=None, mean=None, std=None, low=None, value=None, high=None, mode=None, shape=None, scale=None, values=None, weights=None, edges=None, counts=None, p=None)
Registers a statistical distribution defining a process service duration.
The parameters are keyword-only, so their order never changes the meaning.
Give only those dist takes; desflow.models.distributions.DISTRIBUTIONS
lists each family's parameters and rules. A time-valued parameter (mean,
std, low, mode, high, scale, value, and the values and edges
of an empirical) takes seconds or a duration such as "4 min", as a
blueprint's does.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The internal identifier (e.g., 'packing'). |
required |
dist
|
str
|
The family: exponential, normal, lognormal, truncated_normal, poisson, geometric, constant, uniform, triangular, gamma, weibull or empirical. |
required |
rate
|
Optional[float]
|
Events per second, for exponential. |
None
|
mean
|
Optional[Union[float, str]]
|
The mean, for normal, lognormal, truncated_normal and poisson. |
None
|
std
|
Optional[Union[float, str]]
|
The standard deviation, for the first three (0 when not given). |
None
|
low
|
Optional[Union[float, str]]
|
The lowest value, for truncated_normal (0 when not given), uniform and triangular. |
None
|
value
|
Optional[Union[float, str]]
|
The time a constant distribution always gives. |
None
|
high
|
Optional[Union[float, str]]
|
The highest value, for uniform and triangular. |
None
|
mode
|
Optional[Union[float, str]]
|
The most likely value, for triangular. |
None
|
shape
|
Optional[float]
|
The shape, for gamma and weibull. |
None
|
scale
|
Optional[Union[float, str]]
|
The scale, for gamma and weibull. |
None
|
values
|
Optional[Sequence[Union[float, str]]]
|
The values an empirical draws from, taken at load. |
None
|
weights
|
Optional[List[float]]
|
One weight per empirical value. |
None
|
edges
|
Optional[Sequence[Union[float, str]]]
|
The bin edges of an empirical histogram. |
None
|
counts
|
Optional[List[float]]
|
One count per empirical histogram bin. |
None
|
p
|
Optional[float]
|
The chance of success on each try, for geometric. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the family is unknown, or a parameter is missing, out of range or not one the family takes. |
Source code in src/desflow/builder/context.py
add_signal(name, once=True, on=None, family=False, members=None, defused=False)
Declares a signal that tasks fire and wait for.
A task fires it with signal=name when it finishes, and fails it when the
task fails; a task waits for it with wait={"signal": name}; a process
reaches it with context.signal(name).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The signal's name. |
required |
once
|
bool
|
When True, the default, the signal fires once, as SimPy's |
True
|
on
|
Optional[List[Mapping[str, Any]]]
|
Actions run when the signal fires, each a mapping with one of |
None
|
family
|
bool
|
When True, the entry is a family of signals, one for each key,
made from this entry. A task names a member with |
False
|
members
|
Any
|
With |
None
|
defused
|
bool
|
When True, a failure of the signal that no waiter handles does
not stop the run, as a callback's |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
add_step(at, path=None, value=None, record=None, interrupt=None)
Adds one scenario step; add_scenario describes the steps.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
at
|
Union[float, str]
|
Seconds, or a duration such as |
required |
path
|
Optional[str]
|
A registry path to set, with |
None
|
value
|
Any
|
The value to set. |
None
|
record
|
Optional[Mapping[str, Any]]
|
A ground-truth record to publish. |
None
|
interrupt
|
Optional[Mapping[str, Any]]
|
|
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Source code in src/desflow/builder/context.py
add_store(name, current_cap=None, max_cap=None, priority=False, items=None, type=None, get_queue=None, put_queue=None, family=False, members=None)
Registers a store: a place that holds items, which tasks put and get.
A task puts its finished payload as an item and gets the next item,
waiting when the store is full or empty. A fifo store, the default,
returns the oldest item; a priority store returns the item whose task
gave the lowest put priority first, and the oldest among equal
priorities.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The internal identifier (e.g., 'parcels'). |
required |
current_cap
|
Optional[float]
|
The number of items it can hold at simulation start, a
whole number or infinity. None, the default, is no limit, or
|
None
|
max_cap
|
Optional[float]
|
The largest capacity a live update can set. None, the default,
is |
None
|
priority
|
bool
|
When True, items are returned by the |
False
|
items
|
Optional[List[Any]]
|
The items it holds when the run starts, in the order a get
returns them. A priority store takes |
None
|
type
|
Optional[Type[StoreBase]]
|
A subclass of |
None
|
get_queue
|
Optional[Mapping[str, Any]]
|
A mapping |
None
|
put_queue
|
Optional[Mapping[str, Any]]
|
As |
None
|
family
|
bool
|
When True, the entry is a family of stores, as for
|
False
|
members
|
Any
|
With |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 | |
add_variable(name, value)
Registers a dynamic, untyped variable accessible via the central Registry.
Variables act as arbitrary state payloads that can be targeted by the Control Plane. For example, storing a JSON dictionary defining the velocity and frequency of a gradual wear injection loop.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The internal identifier (e.g., 'velocity_structural'). |
required |
value
|
Any
|
The initial payload (supports primitives, lists, and dicts). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Source code in src/desflow/builder/context.py
advance(until)
Runs a started run up to until, then returns with the run still open.
Call it again to run the next segment, or finish() to run to the end.
Segments give the same records as one run, because nothing is flushed at a
segment boundary. Between segments the model can be read, changed with
context.env, and stepped with step(). advance() never moves the
wall-clock anchor, so a paced run that waits between segments is behind
when it goes on; call context.env.resume_pacing() first to pace again from
now.
The run stops at the earlier of until and its own end: start()'s until,
the stop or the end of the drain. An until that is an event, such as
context.signal("gate").wait(), runs to that event and returns its value;
one that has been processed already returns at once.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
until
|
Union[float, str, Event]
|
A time in simulation seconds or as a duration such as |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
None when the run stops at a time, the event's value for an event,
and the value |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the run has not started or is closed, or it has ended
( |
ValueError
|
If |
SignalFailed
|
If the signal that |
Source code in src/desflow/builder/context.py
2843 2844 2845 2846 2847 2848 2849 2850 2851 2852 2853 2854 2855 2856 2857 2858 2859 2860 2861 2862 2863 2864 2865 2866 2867 2868 2869 2870 2871 2872 2873 2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 | |
analyze(verify=False, *, until=None, seed=None, replications=None, target=0.85)
Explains the model: its graph, and the queueing estimates that apply to it.
The graph and the analysis come from what the builder declared, before
start(), and leave the context as it was: it is not started, and a run
after it gives the same records as a run without it. desflow explain
prints the same Explanation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
verify
|
bool
|
Also run the model unpaced and compare its statistics with the analysis. The run uses a new copy of the model, built again from the blueprint or the Python file it came from. |
False
|
until
|
Union[float, str, None]
|
The end of the verifying run, in seconds or as a duration; the
blueprint's |
None
|
seed
|
Optional[int]
|
The verifying run's seed; the model's |
None
|
replications
|
Optional[int]
|
Verify with this many runs, 2 or more, and an interval for each measure, instead of one run. |
None
|
target
|
float
|
The utilisation the tuning hints aim for, above 0 and at most 10; at 1 or above the hints are for deliberate overload. |
0.85
|
Returns:
| Name | Type | Description |
|---|---|---|
Explanation |
Explanation
|
The graph, the estimates per arrival, task, resource, store
and container, the bottleneck, the hints, and with |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the model names a task or another part that is not
added, as |
Source code in src/desflow/builder/context.py
arrival_loop(arrival_id)
Registers an infinite generator loop to automatically start on boot.
The wrapped function will be injected with the SimulationContext
instance at runtime, granting it access to context.wait_for_arrival().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
arrival_id
|
str
|
The ID of the configured arrival rate to dynamically sample. |
required |
Source code in src/desflow/builder/context.py
call(task, *args, entity=None, upstream=None, **kwargs)
Starts a task as a child flow and returns its process.
The task runs with its own instance id, and so do the tasks its next
leads to. The process ends when that flow ends: its value is the last
task's finished payload, and it fails with TaskFailed when a task of the
flow fails and no on_fail handles it. A flow started this way is not
counted as the entity's flow finishing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The task's name. |
required |
*args
|
Any
|
Positional arguments its function takes after the task id. |
()
|
entity
|
Optional[Entity]
|
The entity, such as the caller's own; a new one when None. |
None
|
upstream
|
Optional[Upstream]
|
What the task receives as |
None
|
**kwargs
|
Any
|
Keyword arguments its function takes. |
{}
|
Returns:
| Type | Description |
|---|---|
Process
|
simpy.Process: The child flow. A failure nobody waits for does not stop the run. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If called before the run starts. |
ValueError
|
If no task has that name. |
Source code in src/desflow/builder/context.py
close()
Ends a started run: publishes the last statistics, flushes and stops the egress connectors, then stops the ingress connectors. Closing twice does nothing.
Source code in src/desflow/builder/context.py
compile_parameters()
Compiles the builder state into the SimParameter that run() registers.
Useful before the run, to see which registry paths a configuration creates or to compare two configurations.
Returns:
| Name | Type | Description |
|---|---|---|
SimParameter |
SimParameter
|
The parameters, sharing their config objects with the
builder; |
Source code in src/desflow/builder/context.py
family_read_problem(expression)
What is wrong with the way an expression reads families, or None: a family read without an index, or an index on a name that is no family.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
expression
|
Expression
|
The parsed expression. |
required |
Returns:
| Type | Description |
|---|---|
Optional[str]
|
Optional[str]: The reason, naming the read. |
Source code in src/desflow/builder/context.py
finish()
Runs a started run to its end, then closes it, as the rest of run() does.
The end is until, the stop or the end of the drain that start() set
up. After one or more advance() or step() calls it runs on from where
they stopped. When advance() already reached the end, it returns at once
with the end's value: None for until or a drain, the stop's value for a
stop, and a failed stop's exception raised again. When advance() or step()
has brought the clock to until or past it and there is no stop or drain,
it returns None at once, as a run that ended at until does. When step()
brought the clock to exactly until, the records hold one run's records
(which end with the final statistics) plus those of the events at until
that step() processed, as in SimPy; a statistics reading due at until
is not published again, and the close adds only the values that changed
after it. After steps past until, the run closes where the steps stopped,
with the final statistics at that time.
close() runs even when the run stops with an error or Ctrl+C.
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value of the event that stopped the run, as for |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the run has not started. |
SignalFailed
|
If the signal that |
Source code in src/desflow/builder/context.py
from_yaml(path, preview=None, variables=None)
classmethod
Builds a context from a YAML blueprint file.
The blueprint is validated before anything is built, and every error names the file and line. The context is built through the same builder methods a Python script calls, so the result is the same simulation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Union[str, Path]
|
The blueprint file. |
required |
preview
|
Optional[Preview]
|
The preview |
None
|
variables
|
Optional[Mapping[str, str]]
|
Values for |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The built context. Its |
Raises:
| Type | Description |
|---|---|
BlueprintError
|
If the blueprint is invalid. |
Source code in src/desflow/builder/context.py
get_container(container_id)
Retrieves an active DynamicContainer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
container_id
|
str
|
The container's name. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DynamicContainer |
DynamicContainer
|
The container. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If called during the Builder Phase. |
KeyError
|
If the model has no such holder, or the last start failed before it made the run's holders. |
Source code in src/desflow/builder/context.py
get_resource(resource_id)
Returns the resource a running model built: a DynamicResource, or the
ResourceBase subclass its type names.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
resource_id
|
str
|
The resource's name, as given to |
required |
Returns:
| Name | Type | Description |
|---|---|---|
ResourceBase |
ResourceBase
|
The resource, for raw requests and readings. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If called before the run has started. |
KeyError
|
If the model has no such holder, or the last start failed before it made the run's holders. |
Source code in src/desflow/builder/context.py
get_store(store_id)
Retrieves an active DynamicStore.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
store_id
|
str
|
The store's name. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DynamicStore |
DynamicStore
|
The store. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If called during the Builder Phase. |
KeyError
|
If the model has no such holder, or the last start failed before it made the run's holders. |
Source code in src/desflow/builder/context.py
interrupt(task, where=None, cause=None)
Interrupts every running instance of a task whose condition holds.
Each instance receives SimPy's Interrupt with cause, publishes
interrupted and follows its task's on_interrupt. The caller's own
instance is never interrupted, and an instance that has finished is not
running.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The task's name. |
required |
where
|
Union[str, Expression, None]
|
A condition read for each instance, such as |
None
|
cause
|
Any
|
What the instances receive as the interrupt's cause, published
as the record's |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
How many instances were interrupted. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If called before the run starts. |
ValueError
|
If no task has that name. |
Source code in src/desflow/builder/context.py
next_instance(task)
Returns the next instance id of a task, counting from 0.
Each task has one counter, which next uses when it starts the task. An
arrival loop that spawns a task by hand takes the id from here too, so a
task started both by an arrival and by next never repeats an id.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The task's name. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
The task's next instance id. |
Source code in src/desflow/builder/context.py
note_values(task, *places)
Notes what the values of a place read from other records, so the run keeps
what they need: the rows of the tasks a lookup or aggregate names, and the
last payload of a task whose values hold a previous. A task with a lookup
does not start before each task it names has a row.
A blueprint calls this for the places that task() cannot see, such as a
payload and an arrival's attributes. A builder script needs it only to
search a place of its own.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
Optional[str]
|
The task whose values the places are, or None for a place that belongs to no task. |
required |
places
|
Any
|
Values: generators, mappings, lists or constants, or a |
()
|
Raises:
| Type | Description |
|---|---|
TypeError
|
If a place is a set or an iterator, which are not searched. |
Source code in src/desflow/builder/context.py
peek()
Returns the time of the next model event of a started run, or infinity if none.
The events of desflow's own, such as a statistics reading or a flush timer,
are not counted. It is infinity once a row with on_end: stop or the last
entry of run.segments has ended the run, because step() runs no more. After
close() it still gives the time of the next event, which step() no longer
runs.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the run has not started. |
Source code in src/desflow/builder/context.py
publish(metric_name, value)
Publishes a telemetry value, named <sim_id>.<metric_name>, to the sinks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
metric_name
|
str
|
The value's name, after the sim id, such as |
required |
value
|
Any
|
The value, which must be a JSON value. |
required |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If called before the run has started. |
Source code in src/desflow/builder/context.py
registry_paths()
Returns the registry paths a run will have, with their starting values.
It builds the registry the way a run does, runs no model code and starts no connector or thread. A server's inspector uses it to list the paths a live change can set.
A capacity with no limit has the value math.inf, which JSON cannot hold,
so a caller that writes JSON converts it.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: Each path, such as |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the registry refuses the model's parameters, as a run would at its start. |
Source code in src/desflow/builder/context.py
run(until=None, stop=None, drain=None, on_unhandled_failure=None, segments=None, steps=None, sync=None)
Compiles the defined infrastructure architecture and triggers the simulation loop.
run() is start(), then SimPy's env.run until the end, then close(),
which runs even when the run stops with an error or Ctrl+C.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
until
|
Any
|
The absolute simulation termination time. Can be a numeric float
(representing base seconds) or a human-readable string parsed by
the utility module (e.g., "1 week", "8 hours"). If None, a context
built by |
None
|
stop
|
Any
|
Ends the run before |
None
|
drain
|
Any
|
At |
None
|
on_unhandled_failure
|
Optional[str]
|
|
None
|
segments
|
Any
|
Runs the model in parts and publishes |
None
|
steps
|
Optional[bool]
|
True publishes |
None
|
sync
|
Optional[bool]
|
True calls |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value of the event that stopped the run, as SimPy's |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a task's |
SignalFailed
|
If the signal that |
Source code in src/desflow/builder/context.py
2731 2732 2733 2734 2735 2736 2737 2738 2739 2740 2741 2742 2743 2744 2745 2746 2747 2748 2749 2750 2751 2752 2753 2754 2755 2756 2757 2758 2759 2760 2761 2762 2763 2764 2765 2766 2767 2768 2769 2770 2771 2772 2773 2774 2775 2776 2777 2778 2779 2780 2781 2782 2783 2784 2785 2786 2787 2788 2789 2790 2791 2792 2793 2794 2795 | |
signal(name)
Returns a declared signal, to fire, fail or wait for from a process.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
A name |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Signal |
Signal
|
The signal: |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If called before the run starts. |
Source code in src/desflow/builder/context.py
spawn(process, delay=None)
Starts a generator as a SimPy process, such as a task instance.
This acts as the "escape hatch" for complex logic (like physical slab
tracking) that outgrows the strict abstraction of the @task decorator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
process
|
Any
|
A generator, such as |
required |
delay
|
Optional[float]
|
Seconds to wait before starting it, as SimPy's |
None
|
Returns:
| Type | Description |
|---|---|
Process
|
simpy.Process: The process. Yield it to wait until it ends; its value is
what the generator returns, and |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
start(until=None, stop=None, drain=None, *, seed=None, overrides=None, stats=None, egress=True, ingress=True, run_settings=True, on_unhandled_failure=None, source=None, actor=None, reason=None, segments=None, steps=None, sync=None)
Builds and starts everything run() does before the clock moves.
It checks the tasks, calls the blueprint's run.before functions, builds the
environment, the sampler, the statistics, the parameters, the connectors,
the resources, stores and containers, and starts every loop and process.
Then step() processes one event at a time and peek() reads the time of
the next, as SimPy's env.step() and env.peek() do; close() ends the
run. A context built by from_yaml takes its run section's values.
Replications and experiments start each run here too, with the arguments after
drain, and finish it with finish(), so a replication builds and ends
exactly as run() does, run.stop and run.drain included.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
until
|
Any
|
As for |
None
|
stop
|
Any
|
As for |
None
|
drain
|
Any
|
As for |
None
|
seed
|
Optional[int]
|
The run's seed, in place of |
None
|
overrides
|
Optional[Mapping[str, Any]]
|
Registry paths and the values they take before the clock starts, applied as a live update is, except that a value the registry would refuse or clamp raises. |
None
|
stats
|
Optional[str]
|
The lowest statistics level the run keeps; a higher level set
with |
None
|
egress
|
bool
|
False builds the run without its sinks and without the statistics timer, as a replication runs: nothing receives its records, and its statistics are read at the end. |
True
|
ingress
|
bool
|
False builds the run without its ingress, so no connector listens for parameter changes. |
True
|
run_settings
|
bool
|
False leaves out a blueprint's |
True
|
source
|
Optional[str]
|
How the |
None
|
actor
|
Optional[str]
|
How the |
None
|
reason
|
Optional[str]
|
How the |
None
|
segments
|
Any
|
As for |
None
|
steps
|
Optional[bool]
|
As for |
None
|
sync
|
Optional[bool]
|
As for |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
DynamicRealtimeEnvironment |
DynamicRealtimeEnvironment
|
The environment, also |
Raises:
| Type | Description |
|---|---|
ValueError
|
As for |
Source code in src/desflow/builder/context.py
3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 3189 3190 3191 3192 3193 3194 3195 3196 3197 3198 3199 3200 3201 3202 3203 3204 3205 3206 3207 3208 3209 3210 3211 3212 3213 3214 3215 3216 3217 3218 3219 3220 3221 3222 3223 3224 3225 3226 3227 3228 3229 3230 3231 3232 3233 3234 3235 3236 3237 3238 3239 3240 3241 3242 3243 3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 | |
step()
Processes the next model event of a started run, as SimPy's env.step() does.
The events of desflow's own that are due before it, such as a statistics reading or a flush timer, run in the same call and are not counted.
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the run has not started or is closed, or it was ended by a
|
EmptySchedule
|
If no model event is left. |
Source code in src/desflow/builder/context.py
stop_arrivals()
Stops every arrival loop, so no new entity arrives; the entities already in
the model carry on. A drain calls it at until.
Source code in src/desflow/builder/context.py
subscribe(event)
Asks to be interrupted when event happens, as SimPy's subscribe_at.
The calling process keeps running; when event fires it is interrupted one
urgent step later with the cause {"event": name, "value": value}, where
name is the process's name for a process and None for any other event.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
event
|
Any
|
Any SimPy event, a process included. |
required |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If called outside a process, or |
Source code in src/desflow/builder/context.py
task(service_id=None, resource_id=None, *, name=None, priority=None, get=None, put=None, next=None, set=None, patience=None, balk=None, wait=None, open=None, listen=None, cancel=None, seize=None, release=None, signal=None, fail=None, fail_reason=None, on_fail=None, on_renege=None, on_balk=None, service_time=None, key_field=None, key=None, keep=None, preempt=True, on_interrupt='resume', on_interrupt_next=None, on_interrupt_run=None, interrupt=None, resume=None, batch=None, spawn=None, subscribe=None, on=None, vars=None, vars_once=None, on_start=None, finish_vars=None, set_args=None, activate=None, deactivate=None, history=None)
Transforms a standard Python function into an event-driven SimPy task.
This decorator completely abstracts the boilerplate of resource acquisition, clock synchronization, and standardized telemetry lineage.
Lifecycle Execution:
1. Checks balk; when it holds, emits balked and stops.
2. Emits a queued event, when the task waits for anything: its wait, a
resource or a get.
3. Waits for its wait, then for a unit of each resource in resource_id,
in the order listed and each in its queue order, then for its get;
patience counts the time spent waiting for all three, and pauses
while the task is served.
4. Emits a started event.
5. Executes the wrapped user function to extract custom payload data.
6. Samples the live service_id distribution, or reads service_time, and
yields the timeout.
7. Adds its put, waiting for room, and checks fail.
8. Emits its finished event with the user's payload, or failed.
9. Fires its signal (or fails it), sends its interrupt action, applies
set to the entity's attributes and starts the next task, or
on_fail.
Every event carries the entity's key as its record key, and the task's name
and instance id as the record's task and instance fields.
A task with a service and no resource is a timed step: no queued event, so
its wait is 0. A task with no service, resource or get takes no time: when
spawned it emits the payload at once (after its put, if it has one), with
no queued or started event.
Call the task as task(task_id, ...). Pass entity= with the entity
wait_for_arrival returned; without one the task starts an entity of its
own, keyed by its name and a counter. The function receives entity,
upstream (what the previous task handed on, or None) and item (what a
store get took) by keyword, each only when it declares that parameter, so
existing functions keep working. A task started by next is called with its
own instance id and the same other arguments as the task before it. A
function that declares wait receives the wait's branches, each a mapping
of done, ok and value.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
service_id
|
Optional[str]
|
The ID of the configured distribution dictating the execution time. |
None
|
service_time
|
Union[float, str, Callable[[Entity], float], Any]
|
The execution time of this task, in place of |
None
|
resource_id
|
Union[str, Mapping[str, Any], List[Any], None]
|
The resource the task holds while it works, or a list of
them, acquired in the order listed while the earlier ones are held.
A member of a family is |
None
|
name
|
Optional[str]
|
The task's name in records, statistics and |
None
|
priority
|
Union[float, str, Callable[[Any, Dict[str, Any]], float], None]
|
For a resource whose queue order is |
None
|
get
|
Union[Transfer, Mapping[str, Any], None]
|
What the task takes before it starts: |
None
|
put
|
Union[Transfer, Mapping[str, Any], None]
|
What the task adds when it finishes, in the same form; a store receives the finished payload. |
None
|
next
|
NextSpec
|
The task that follows: a task name, |
None
|
set
|
Optional[Mapping[str, Any]]
|
Attribute values applied to the entity when the task finishes. |
None
|
patience
|
Union[float, str, Callable[[Any, Dict[str, Any]], float], Any]
|
How long the task may wait, for its |
None
|
balk
|
Optional[str]
|
A condition, such as |
None
|
wait
|
Optional[Mapping[str, Any]]
|
What the task waits for before it queues for its resource: one
branch, |
None
|
open
|
Optional[Mapping[str, Any]]
|
Events the task makes at its step, before its |
None
|
listen
|
Optional[Mapping[str, Any]]
|
Callbacks added to an event when the task starts, after its
|
None
|
cancel
|
Union[str, Mapping[str, Any], None]
|
The name of an opened request, get or put, taken out of its line
( |
None
|
seize
|
Union[str, Mapping[str, Any], None]
|
A unit of a resource the flow keeps across later tasks until a
|
None
|
release
|
Union[str, Mapping[str, Any], None]
|
Gives back units when the task finishes: a resource name (the
oldest unit of it the flow holds, and nothing if it holds none),
|
None
|
signal
|
Union[str, Mapping[str, Any], None]
|
A signal, added with |
None
|
fail
|
Union[str, Callable[[Any, Any], bool], None]
|
A condition checked after the service, such as
|
None
|
fail_reason
|
Any
|
The |
None
|
on_fail
|
Optional[str]
|
The task a failed instance goes to. Without it, a failure ends
the entity's flow. A function that raises fails the task the same
way, with the error as the reason, and goes to |
None
|
on_renege
|
Optional[str]
|
The task an instance goes to after it reneges, in place of the
end of its flow. It still publishes |
None
|
on_balk
|
Optional[str]
|
The task an instance goes to after it balks, in place of the end
of its flow, whether its |
None
|
key_field
|
Optional[str]
|
A field of the task's payload whose value becomes the key of
the finished event, and so the Kafka message key, in place of the
entity's key. The |
None
|
key
|
Any
|
One record key for every record of an instance, in place of the
entity's key: a constant, or a typed generator from
|
None
|
keep
|
Optional[List[str]]
|
Resources in |
None
|
preempt
|
Union[bool, str]
|
On a resource added with |
True
|
on_interrupt
|
str
|
What an interrupt or a preemption does: |
'resume'
|
on_interrupt_next
|
Optional[str]
|
The task an abandoned instance goes to. |
None
|
on_interrupt_run
|
Optional[str]
|
The task that runs as a child flow of the same entity
after each interrupt or preemption, under |
None
|
resume
|
Optional[Mapping[str, Any]]
|
An action applied when the task finishes:
|
None
|
batch
|
Optional[Mapping[str, Any]]
|
Members that wait together: |
None
|
interrupt
|
Optional[Mapping[str, Any]]
|
An action applied when the task finishes:
|
None
|
vars
|
Optional[Mapping[str, Any]]
|
Named values, each a generator mapping such as
|
None
|
vars_once
|
Optional[Mapping[str, Any]]
|
Named values made once for each entity, at its first event, and the same in every later instance of this task for the entity. A wait timeout value can read them. |
None
|
on_start
|
Optional[Mapping[str, Any]]
|
A mapping of values published as a record when the task starts,
after |
None
|
finish_vars
|
Optional[Mapping[str, Any]]
|
Named values evaluated after |
None
|
activate
|
Union[str, List[str], None]
|
The arrival, or arrivals, whose source the task turns on when it
finishes, after |
None
|
deactivate
|
Union[str, List[str], None]
|
The same, turning the source off. A source already in that state is left as it is. |
None
|
set_args
|
Optional[Mapping[str, Any]]
|
Names written in this process's own locals after |
None
|
history
|
Optional[int]
|
How many finished payloads the task keeps for |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If a resource is given without a service, a priority without
a resource, the name is taken, |
Returns:
| Name | Type | Description |
|---|---|---|
Callable |
Callable
|
A decorator that wraps the target function into a SimPy generator. |
Example
Source code in src/desflow/builder/context.py
1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 1839 1840 1841 1842 1843 1844 1845 1846 1847 1848 1849 1850 1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 | |
telemetry_loop(interval)
Registers an infinite background daemon loop for periodic data publishing.
This is primarily used for observing and extracting continuous state (like wear containers) at strict intervals.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
interval
|
Union[float, str]
|
Simulation seconds, or a duration such as |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/context.py
wait_for_arrival(arrival_id)
Returns the wait for the next arrival: a timeout from the live distribution, or for a replayed arrival the wait for its next row.
The timeout's value is the new Entity, so an arrival loop writes
entity = yield context.wait_for_arrival("orders") and passes
entity=entity to the task it spawns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
arrival_id
|
str
|
The arrival's name. |
required |
Returns:
| Type | Description |
|---|---|
Event
|
simpy.Event: A timeout for the gap to the next arrival, or the wait for
the next replayed row, which succeeds with the new |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If called during the Builder Phase (before |
Source code in src/desflow/builder/context.py
with_batching(batch_size, flush_interval, max_queued_batches=2000, drain_stall_seconds=30.0)
Configures the internal egress buffering strategy to mitigate I/O lock contention.
batch_size and flush_interval are defaults. A provider that names either
on add_egress uses its own instead, which is how a stream sink and a lake
sink share one run.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
batch_size
|
int
|
The maximum number of standard events to hold in memory before forcing an asynchronous flush to the egress thread. |
required |
flush_interval
|
float
|
The maximum simulation time (in seconds) to wait before forcefully flushing the buffer, regardless of capacity. |
required |
max_queued_batches
|
int
|
Upper bound on batches waiting for the egress threads. A full queue blocks the producer, so a sink that cannot keep up slows the simulation instead of building a backlog that teardown would then have to discard. |
2000
|
drain_stall_seconds
|
float
|
How long teardown keeps waiting once the queue has stopped shrinking. While it keeps draining the wait is unbounded, so a slow sink finishes rather than losing its tail. |
30.0
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Source code in src/desflow/builder/context.py
with_stats(level='basic', interval=DEFAULT_STATS_INTERVAL, on_change=False, trace=False)
Switches on the built-in statistics, published as telemetry.
Each level publishes what the levels before it publish, and more: minimal
the readings of every resource, store and container right now; basic
finished counts and time-weighted averages; typical the mean and maximum
wait and service time of each task; all their 50th, 95th and 99th
percentiles. Names look like <sim_id>.stats.tasks.<task>.wait_mean.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
level
|
str
|
|
'basic'
|
interval
|
Union[float, str]
|
Simulation seconds, or a duration such as |
DEFAULT_STATS_INTERVAL
|
on_change
|
bool
|
True also publishes a resource's readings, the names it
publishes at each interval, right after each |
False
|
trace
|
bool
|
True publishes one record of the kind |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The current instance for method chaining. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the level is unknown, the interval is not positive, or
|
Source code in src/desflow/builder/context.py
Task Flow
A task's flow is built from the classes below. The signals and failures here are shared between tasks.
desflow.Signal(env, name='signal', once=True, defused=False)
A named event that fires once, or many times with once=False, each firing waking
every waiter.
Attributes:
| Name | Type | Description |
|---|---|---|
env |
The environment. |
|
name |
str
|
The signal's name. |
count |
int
|
How many times it has fired or failed. |
ok |
Optional[bool]
|
Whether the last firing succeeded; None before the first. |
value |
Any
|
The value of the last firing, or the failure reason. |
defused |
bool
|
Whether every failure of the signal is defused, as a callback's
|
Creates a signal that has not fired yet.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
env
|
Environment
|
The environment. |
required |
name
|
str
|
Its name, as a blueprint's |
'signal'
|
once
|
bool
|
When True, the default, the signal fires once, as SimPy's |
True
|
defused
|
bool
|
When True, a failure with no waiter does not stop the run, as
a callback that sets |
False
|
Source code in src/desflow/builder/signals.py
Attributes
processed
property
bool: Whether the latest firing's event has been processed, as SimPy's
event.processed; False before any firing. It is signal.<name>.processed in
an expression.
Methods:
attach(callbacks)
Keeps callbacks as the signal's on: callbacks and adds them to the
current pending event, so they run when the signal next fires, before any
waiter, as SimPy runs a callback added before the waiters.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
callbacks
|
List[Callable[[Any], None]]
|
Functions called with the fired event, copied onto each new pending event so they run on every firing. |
required |
Source code in src/desflow/builder/signals.py
fail(reason, urgent=False, after=0.0, priority=None, defused=False)
Fails the signal: every waiter receives SignalFailed with the reason.
A failure of a signal that re-arms is always defused, and a failure that nobody
waits for does not stop the run then. A one-shot signal's failure with no waiter
is left to stop the run under on_unhandled_failure: stop, as SimPy's failed
event does, unless defused is given or the signal is defused.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
reason
|
Any
|
Why it failed, such as a task's failure reason. |
required |
urgent
|
bool
|
As for |
False
|
after
|
float
|
As for |
0.0
|
priority
|
Optional[int]
|
As for |
None
|
defused
|
bool
|
When True, this failure does not stop the run if nobody waits for
it, as |
False
|
Raises:
| Type | Description |
|---|---|
ValueError
|
As for |
TypeError
|
As for |
Source code in src/desflow/builder/signals.py
fire(value=None, urgent=False, after=0.0, priority=None)
Fires the signal: every waiter resumes with value.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
Any
|
What the waiters receive, such as a task's finished payload. |
None
|
urgent
|
bool
|
When True the waiters resume before the other events at their
time, as SimPy's |
False
|
after
|
float
|
Simulation seconds from now at which the waiters resume. The signal counts as fired now, so a second firing is refused. Needs a one-shot signal. |
0.0
|
priority
|
Optional[int]
|
A whole number, 1 when left out, the priority the firing is
scheduled with: an event at one time runs by priority, lowest first,
and then in the order it was scheduled, as
|
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
TypeError
|
If |
Source code in src/desflow/builder/signals.py
wait()
Returns the event of the next firing, for a process to yield.
Returns:
| Type | Description |
|---|---|
Event
|
simpy.Event: Succeeds with the firing's value, or fails with
|
Source code in src/desflow/builder/signals.py
desflow.SignalFailed(signal, reason)
Bases: Exception
The exception a waiter of a failed signal receives.
Attributes:
| Name | Type | Description |
|---|---|---|
signal |
str
|
The signal's name. |
reason |
Any
|
What |
Source code in src/desflow/builder/signals.py
desflow.TaskFailed(task, reason)
Bases: Exception
A task instance failed, and no on_fail handled it.
A child flow, started by a wait branch or context.call, fails with it, so
the task waiting for the flow fails too.
Attributes:
| Name | Type | Description |
|---|---|---|
task |
str
|
The task that failed. |
reason |
Any
|
Why: its |
Source code in src/desflow/builder/task.py
desflow.NO_RECORD = _NoRecord()
module-attribute
What a task function returns to finish with no record: the Builder form of a
blueprint's payload: null.
Return desflow.NO_RECORD from the function a task wraps. The task still runs its
service, set, signal and next, and publishes no finished event.
desflow.Entity(key, arrival, attributes=dict(), held=dict(), entered=False, instances=None, spawned_by=None, ends_pending=0, ending=False, last_ended=None, loop=None)
dataclass
One thing moving through the model, such as an order, a slab or a session.
Each arrival starts a new entity. Its key is the record key of every event it causes, in every task it passes through.
Attributes:
| Name | Type | Description |
|---|---|---|
key |
str
|
The record key: the arrival name and a counter, such as
|
arrival |
str
|
The arrival that started it, which names its flow in the statistics. |
attributes |
Dict[str, Any]
|
State kept across tasks: set by the arrival's
|
held |
Dict[str, Any]
|
Resource units a task kept for the next task with
|
Attributes
left
property
True once RunState.entity_left has run: the flow has ended and nothing
is pending.
desflow.Upstream(task_id, key, attributes, payload=None)
dataclass
What a finished task hands the task routed after it.
Attributes:
| Name | Type | Description |
|---|---|---|
task_id |
Any
|
The instance id of the finished task. |
key |
str
|
The entity's key. |
attributes |
Dict[str, Any]
|
The entity's attributes, after the finished
task's |
payload |
Any
|
The finished task's payload, read in conditions as
|
Attributes
fields
property
The payload when it is a mapping, and an empty mapping otherwise.
desflow.Router(rules, rng=None)
Chooses the task that follows a finished one.
The rules take one of these forms, in every API:
- a task name: always that task;
{"when": [{"condition": "...", "task": "..."}, ...], "default": "..."}: the first rule whose condition holds wins, anddefault, if given, catches the rest; with no default and no rule holding, the flow ends;{"weights": {"ship": 9, "rework": 1}}: a task drawn with these weights from the routing task's own random sequence, so seeded runs repeat;- a function taking the
Upstreamand returning a task name or None; - None: the flow ends.
A route back to an earlier task (rework) is allowed.
Attributes:
| Name | Type | Description |
|---|---|---|
tasks |
Optional[Set[str]]
|
Every task the rules can name, so a loader can check they exist; None for a function. |
rng |
Optional[Generator]
|
The generator |
Checks the rules and parses their conditions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
rules
|
NextSpec
|
The rules, in one of the forms above. |
required |
rng
|
Optional[Generator]
|
The generator weights draw from, such as
|
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If the rules are malformed or a condition is refused. |
Source code in src/desflow/builder/flow.py
Attributes
computed
property
True when a weight is an expression, so the shares are known only when a task finishes.
conditions
property
The parsed conditions, in rule order.
probabilities
property
The share of finished instances each next task receives, where the rules fix it: 1.0 for a task name, the normalised weights for weights, and an empty mapping when the flow ends. None for when rules or a function, whose split depends on the run.
weighted
property
True when the rules are weights, which draw from rng.
Methods:
choose(upstream, env=None, levels=None, item=None, scope=None)
Returns the next task's name, or None when the flow ends.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
upstream
|
Upstream
|
What the finished task hands on. |
required |
env
|
Any
|
The environment, for |
None
|
levels
|
Optional[Mapping[str, Any]]
|
Stores and containers by name, for |
None
|
item
|
Any
|
The item the finished task's store get took, for |
None
|
scope
|
Optional[Scope]
|
What the conditions read, when the caller has built it; it replaces the one built from the arguments above. |
None
|
Returns:
| Type | Description |
|---|---|
Optional[str]
|
Optional[str]: The next task, or None. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a condition reads a name that is missing. |
Source code in src/desflow/builder/flow.py
desflow.Transfer(container=None, store=None, amount=1.0, priority=0.0, filter=None, wait=True, item=None)
What a task takes from, or adds to, a store or a container.
A container transfer moves an amount: a number, an expression such as
payload.quantity, or in Python a function of the task id and the upstream
payload. A store transfer moves one item: a get takes the next item, and a
put adds the task's finished payload.
A put with wait=False is made and not waited for: the task goes on at once, as
a SimPy process that calls store.put(item) without yield. The put stays in the
queue, completes in first in, first out order whenever there is room, and is
never cancelled by an interrupt, a patience or an abandon. It publishes no
record, a drain does not wait for it, and one still queued when the run ends is
dropped.
Attributes:
| Name | Type | Description |
|---|---|---|
container |
Optional[str]
|
The container's name, or a |
store |
Optional[str]
|
The store's name, or a |
amount |
The amount, for a container. |
|
wait |
bool
|
False for a put the task does not wait for. |
Checks the transfer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
container
|
Optional[str]
|
The container's name, or a reference |
None
|
store
|
Optional[str]
|
The store's name, instead of a container, or a reference. |
None
|
amount
|
Amount
|
For a container, a number, an expression or a function. |
1.0
|
priority
|
Any
|
For a put into a priority store, any value SimPy can order: a
number, a string, a tuple, or an expression or function that
returns one, lowest served first. A string is an expression. It is
kept as given, so |
0.0
|
filter
|
Union[str, Callable[[Any], bool], None]
|
For a store get, which item it takes: a condition such as
|
None
|
wait
|
bool
|
False for a put the task does not wait for (see above). Defaults to True. |
True
|
item
|
Any
|
For a store put, the item to put: a constant, text, or a |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If not exactly one of container and store is given, an
amount is given for a store, a priority is given for a
container, the amount is malformed, the priority is an expression
that is malformed, or |
Source code in src/desflow/builder/flow.py
514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 | |
Attributes
expressions
property
Every expression the transfer holds, so a loader can check its names.
reference_expressions
property
The parsed key or lowest of a reference the transfer names.
target
property
The name of the store or container.
Methods:
amount_for(task_id, upstream, scope)
Returns the amount for one task instance.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id
|
Any
|
The instance id. |
required |
upstream
|
Optional[Upstream]
|
What the previous task handed on. |
required |
scope
|
Scope
|
What an expression's names read. |
required |
Returns:
| Type | Description |
|---|---|
Union[int, float]
|
Union[int, float]: The amount, as written when it is a number. |
Source code in src/desflow/builder/flow.py
filter_for(scope, entity=None)
Returns the function a store get offers each item to, or None for any item.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scope
|
Scope
|
What the condition's other names read, taken when the get is made;
|
required |
entity
|
Any
|
The entity that makes the get, passed by name to a function that
names a parameter |
None
|
Returns:
| Type | Description |
|---|---|
Optional[Callable[[Any], bool]]
|
Optional[Callable[[Any], bool]]: True for an item the get may take. |
Source code in src/desflow/builder/flow.py
of(spec)
classmethod
Returns a transfer from a mapping such as {"container": "stock", "amount": 2}.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
Union[Transfer, Mapping[str, Any]]
|
A mapping with |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Transfer |
Transfer
|
The transfer. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the mapping has other keys or is malformed. |
Source code in src/desflow/builder/flow.py
priority_for(task_id, upstream, scope)
Returns the priority for one put into a priority store, lowest first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task_id
|
Any
|
The instance id. |
required |
upstream
|
Optional[Upstream]
|
What the previous task handed on. |
required |
scope
|
Scope
|
What an expression's names read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The priority as it was given or computed, not changed to a number. |
Source code in src/desflow/builder/flow.py
desflow.Expression(text, records=False, names=None, rule=False, batch=False, member=False)
A parsed and checked expression, ready to evaluate against a Scope.
Attributes:
| Name | Type | Description |
|---|---|---|
text |
str
|
The expression as written. |
references |
Dict[str, Set[str]]
|
The names it reads, by source: |
where |
str
|
The place of the expression in a blueprint, such as
|
Parses and checks an expression.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
text
|
str
|
The expression, such as |
required |
records
|
bool
|
True for an egress |
False
|
names
|
Optional[Container[str]]
|
The bare names it may read, such as a task's vars. None refuses
every bare name but |
None
|
rule
|
bool
|
True for the |
False
|
batch
|
bool
|
True for a batch's |
False
|
member
|
bool
|
True for the |
False
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If it is not valid Python or uses anything outside the whitelist. The message names what was refused. |
Source code in src/desflow/core/expressions.py
720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 | |
Attributes
reads_queues
property
bool: Whether the expression reads a queue length or what a store holds, so a wait on it is checked again after a change of a queue length.
Methods:
evaluate(scope)
Evaluates the expression.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scope
|
Scope
|
What its names read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value, such as a bool for a condition or a number for an amount. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a name it reads is missing, or the arithmetic fails, such as a division by zero. The message names the expression, after its place when it has one. |
Source code in src/desflow/core/expressions.py
outside_run(names=True)
Names the sources the expression reads that a place reading the run only
cannot: the entity's attr and has, payload, item, a wait branch,
process and, unless names is False, a bare name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
names
|
bool
|
Whether a bare name counts, as it does where no var is readable. |
True
|
Returns:
| Type | Description |
|---|---|
Optional[str]
|
Optional[str]: The sources, such as |
Optional[str]
|
none of them. |
Source code in src/desflow/core/expressions.py
desflow.Scope(payload=dict(), attributes=dict(), levels=dict(), resources=dict(), registry=None, now=0.0, item=None, waits=dict(), signals=dict(), process=dict(), record=None, names=dict(), entity=None, args=None, cause=None, upstream=None, item_priority=None, done_count=None, loops=dict(), peek=None, instances=None, has_item=False, stream=None, handles=None, request=None, holder=None, batch=None, each=None, member=None, families=dict(), family_keys=dict())
dataclass
What the names in an expression read.
Attributes:
| Name | Type | Description |
|---|---|---|
payload |
Optional[Mapping[str, Any]]
|
The upstream payload, read as |
attributes |
Mapping[str, Any]
|
The entity's attributes, read as |
levels |
Mapping[str, Any]
|
Stores and containers by name, read as |
resources |
Mapping[str, Any]
|
Resources by name, read as |
registry |
Optional[Any]
|
The simulation registry, read as |
now |
float
|
The simulation time, read as |
item |
Optional[Mapping[str, Any]]
|
The item a store get took, or a filter's candidate, read as
|
waits |
Mapping[str, Mapping[str, Any]]
|
The task's |
signals |
Mapping[str, Any]
|
Signals by name, read as |
process |
Mapping[str, Any]
|
The running task's |
record |
Optional[Mapping[str, Any]]
|
The record an egress |
names |
Mapping[str, Any]
|
The bare names: a task's vars, and the names of an |
entity |
Optional[str]
|
The key of the entity the task runs for, which |
args |
Optional[Mapping[str, Any]]
|
The process's local names, read as |
cause |
Optional[Mapping[str, Any]]
|
The cause of the interrupt the task was reached by, read as
|
upstream |
Optional[Mapping[str, Any]]
|
The payload of the task that routed here, read as |
item_priority |
Any
|
The priority of the item a priority store gave, or None. |
done_count |
Optional[int]
|
How many branches of a |
loops |
Mapping[str, Any]
|
The names of the comprehensions being evaluated. |
peek |
Optional[Callable[[], Any]]
|
Gives the time of the next scheduled event. |
instances |
Optional[Callable[[str], List[Scope]]]
|
Gives the scopes of the running instances of a task. |
has_item |
bool
|
Whether a get has given the task an item (it may be None). |
stream |
Any
|
The stream |
handles |
Optional[Callable[[str], Any]]
|
Gives the event opened under a name, read as |
request |
Optional[Mapping[str, Any]]
|
The request a preempt rule tests, read as |
holder |
Optional[Mapping[str, Any]]
|
The user whose unit it would take, read as |
batch |
Optional[Mapping[str, Any]]
|
The open batch a |
each |
Optional[Mapping[str, Any]]
|
The element an |
member |
Optional[Mapping[str, Any]]
|
The member a |
families |
Mapping[str, Any]
|
The declared families by name, which |
family_keys |
Mapping[str, List[Any]]
|
The keys of the members the run has made, by family name,
which |
Loading a Model
SimulationContext.from_yaml is the usual entry point for a blueprint. from_yaml calls build, which returns the context and the run section. load_context loads a blueprint or a builder Python file from a folder, local or in object storage, in the way desflow run loads a blueprint. desflow run --sample and --stdout pass a Preview to these functions. preview_builder changes a context that a builder file defines, so that it runs as such a preview. check_model checks a blueprint or a builder file as desflow check does and returns a ModelCheck.
desflow.load_context(folder, entry, preview=None, variables=None)
Builds the simulation that entry in folder defines.
An entry ending in .py is a builder file: it is imported under a new module
name each call, with folder first on sys.path during the import, so its
imports resolve from its folder, not from a file of another folder loaded
earlier, and its if __name__ == "__main__": block does not run. Its
DESFLOW_APP names the context to use; without it, the file must define exactly
one SimulationContext at module level. Any other entry is a YAML blueprint,
built by SimulationContext.from_yaml.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
folder
|
Union[str, Path]
|
A local folder, or a URI that |
required |
entry
|
str
|
The entry file's name inside |
required |
preview
|
Optional[Preview]
|
The preview |
None
|
variables
|
Optional[Mapping[str, str]]
|
Values for |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
SimulationContext |
SimulationContext
|
The built context, not yet run. |
Raises:
| Type | Description |
|---|---|
NoContextError
|
If a Python entry defines no context. It is a
|
EntryNotFoundError
|
If |
ValueError
|
If a Python entry defines several contexts with no |
ImportError
|
If |
BlueprintError
|
If a blueprint is invalid. |
Source code in src/desflow/loading.py
desflow.blueprint.build
Reads, validates and builds a blueprint file.
The structure, keys and names of the whole file are validated with or without
a preview. A preview builds none of the connectors it replaces or skips, so it
checks only their type; their config is checked only by a run that builds
them, such as desflow run FILE without --sample or --stdout.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Union[str, Path]
|
The YAML file. |
required |
preview
|
Optional[Preview]
|
Replace the file's egress with a JSON console, and for a sample leave out its ingress; None builds the file as written. |
None
|
variables
|
Optional[Mapping[str, str]]
|
Values for |
None
|
Returns:
| Type | Description |
|---|---|
Tuple[SimulationContext, Run]
|
The built context, and the |
Raises:
| Type | Description |
|---|---|
BlueprintError
|
If the file is invalid. The message names the file and line. |
Source code in src/desflow/blueprint/build.py
desflow.blueprint.Preview(sample=None)
dataclass
A preview run of a blueprint, as desflow run --sample and --stdout make.
Every egress entry is replaced by a JSON console on stdout, with one more console
for labels when the file has a when: labels entry and one for the trace when it
has a when: trace entry, so no sink module the file names by a short type is
imported, and no sink is written to.
Attributes:
| Name | Type | Description |
|---|---|---|
sample |
Optional[int]
|
Print this many events, unpaced and with no ingress, then stop.
None, for |
desflow.blueprint.preview_builder(context, preview)
Makes a context that a builder file defined run as a preview, as a blueprint built with the same preview runs.
Every egress sink the file added is replaced by a JSON console on stdout, with
one more console for labels when the file added a labels sink and one for the
trace when it added a trace sink. A sample also leaves out the ingress, and runs
unpaced from logical_start_time, with go_live_at cleared, until it has
printed that many events. The file's connectors are already built, because
importing the file built them, but none of them is set up or written to.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
SimulationContext
|
The context the builder file defines, not yet run. |
required |
preview
|
Preview
|
The preview |
required |
Source code in src/desflow/blueprint/build.py
desflow.blueprint.check_model(file, variables=None)
Checks one blueprint or builder file as desflow check does, without running it.
For a YAML blueprint that is accepted, it also reads the run section and the
start time, with the loader that desflow run uses.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file
|
Path
|
A YAML blueprint, or a Python file with a module-level context. |
required |
variables
|
Optional[Mapping[str, str]]
|
Values for |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
ModelCheck |
ModelCheck
|
The built context, or why the file was refused, and for an
accepted YAML blueprint its |
Raises:
| Type | Description |
|---|---|
Exception
|
An error of a YAML file other than a |
Source code in src/desflow/blueprint/model_check.py
desflow.blueprint.ModelCheck
Bases: NamedTuple
What check_model found in one file.
Attributes:
| Name | Type | Description |
|---|---|---|
context |
Optional[SimulationContext]
|
The built context, or None when the file was refused. |
problem |
Optional[str]
|
Why the file was refused, as one message, or None. |
no_context |
bool
|
True when the file is a Python file refused only because it
defines no module-level |
run |
Optional[Run]
|
The |
start |
Optional[str]
|
The |
desflow.blueprint.Run
Bases: StrictModel
The run section.
before lists !python functions called with no arguments before the run
starts, for setup such as creating topics or tables. stop ends the run before
until at the next firing of a signal, {signal: name}, or when a condition
first holds, {condition: ...}. drain, true or {limit: 1 h}, stops the
arrivals at until and runs on until no entity is left in the model.
segments runs the model in parts and publishes a reading after each, as a
SimPy program that calls env.run(until=...) in a loop does: a list of
RunSegment, or {every: <time>} for equal parts up to until. steps: true
publishes system.simulation.step, {step, now, peek}, after each model event, as
a loop of env.step() and env.peek() reads them. sync: true calls env.sync()
as the run starts. A run with no until ends when no model event is left, except
that a list of segments with no until whose last entry has an until or a
stop ends there, as a SimPy program that stops calling env.run does; a last
entry {} runs to the end.
Methods:
drain_setting()
drain in the form SimulationContext.run takes.
desflow.blueprint.RunSegment
Bases: StrictModel
A run.segments entry: the run goes on to until or to stop, then waits for
the next entry.
An entry has at most one of until and stop; {} runs to the end. After the
segment stops, look names a task whose new entity reads the model before any
event at the stop time runs, and sync re-anchors the wall clock at the start of
the segment. A record at system.simulation.segment reports index, now and
peek (null when no model event is left) as the segment ends.
desflow.blueprint.SegmentEvery
Bases: StrictModel
run.segments: {every: 10 s}: a segment every every seconds up to run.until.
desflow.blueprint.Drain
Bases: StrictModel
A run.drain mapping: the longest a drain may run after until.
desflow.blueprint.BlueprintError
Bases: ValueError
A blueprint that cannot be built. The message names the file and line.
Replications
desflow.replicate(model, n, *, seed, until=None, warmup=None, measures=None, level=0.95, workers=1, overrides=None, stop=None, drain=None)
Runs a model n times with spawned seeds and summarises its statistics.
Each replication runs unpaced, with no ingress and no egress, and reads the
built-in statistics at its end. When the model has sinks (a blueprint's
egress, or a builder's add_egress), they receive the summary as telemetry
once, at the end: each replication's value and the mean, half-width and count of
each measure.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Model
|
A YAML blueprint path, a builder factory (a function with no
arguments returning a new |
required |
n
|
int
|
How many replications, 2 or more. |
required |
seed
|
Optional[int]
|
The seed the replications' seeds are spawned from. None draws one and logs it. |
required |
until
|
Union[float, str, None]
|
Where each replication's arrivals stop, in seconds or as a duration
such as |
None
|
warmup
|
Union[float, str, None]
|
The end of each replication's warm-up; None keeps the model's own. |
None
|
measures
|
Optional[Sequence[str]]
|
Statistic names without the sim id, such as
|
None
|
level
|
float
|
The confidence level of the half-widths. Defaults to 0.95. |
0.95
|
workers
|
int
|
Replications run at once, each in its own process. Above 1 the model must be a path or a function defined at the top of a module. |
1
|
overrides
|
Optional[Mapping[str, Any]]
|
Registry paths and the values they take in every replication. |
None
|
stop
|
Any
|
Ends each replication before |
None
|
drain
|
Any
|
At |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Replications |
Replications
|
Every value with its seed, and the summary. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/analysis/runner.py
175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | |
desflow.Replications(sim_id, seeds, measures, values, level=0.95, streams=list())
dataclass
The result of replicate: every value with its seed, and a summary per measure.
Attributes:
| Name | Type | Description |
|---|---|---|
sim_id |
str
|
The model's sim id. |
seeds |
List[int]
|
The seed of each replication, in order. Replication |
measures |
List[str]
|
The statistics summarised, such as
|
values |
Dict[str, List[float]]
|
Each measure's value in each replication. |
level |
float
|
The confidence level of the half-widths. |
streams |
List[List[str]]
|
The random streams each replication drew from. |
Attributes
summary
property
One row per measure: measure, n, mean, half_width and level.
table
property
One row per replication and measure: replication (from 1), seed,
measure and value.
Methods:
summary_text()
The summary as a table: each measure's n, mean and half-width.
Source code in src/desflow/analysis/runner.py
telemetry()
The telemetry records the sinks receive, as (path_id, value) pairs.
One per measure and replication, <sim_id>.stats.<measure>.replication.<i>,
then the mean, half-width and count across replications,
<sim_id>.stats.<measure>.mean, .half_width and .n.
Source code in src/desflow/analysis/runner.py
desflow.LowLevelModel(function, sim_id)
dataclass
A model written with the low-level API, for replicate and run_experiment.
Attributes:
| Name | Type | Description |
|---|---|---|
function |
Callable
|
Called as |
sim_id |
str
|
The sim id the parameters are registered under, which the statistics' names start with. |
desflow.analysis.interval(values, level=0.95)
Returns the mean of values and the Student t half-width of its interval.
The half-width is t × s / √n, where s is the sample standard deviation and
t the quantile at (1 + level) / 2 with n - 1 degrees of freedom.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
Sequence[float]
|
Independent values, such as one statistic from each replication. |
required |
level
|
float
|
The confidence level, between 0 and 1. Defaults to 0.95. |
0.95
|
Returns:
| Name | Type | Description |
|---|---|---|
Interval |
Interval
|
The mean and its half-width. If a value is infinite or NaN, the mean is infinite or NaN and the half-width is NaN. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If there are fewer than 2 values or the level is not between 0 and 1. |
Source code in src/desflow/analysis/intervals.py
desflow.analysis.Interval(n, mean, std, half_width, level)
dataclass
A mean and the half-width of its confidence interval.
Attributes:
| Name | Type | Description |
|---|---|---|
n |
int
|
How many values the mean is of. |
mean |
float
|
Their mean. |
std |
float
|
Their sample standard deviation. |
half_width |
float
|
The Student t half-width at |
level |
float
|
The confidence level, such as 0.95. |
Attributes
high
property
The upper bound, mean + half_width.
low
property
The lower bound, mean - half_width.
desflow.analysis.t_quantile(p, df)
Returns the value a Student t variable is at most with probability p.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p
|
float
|
The probability, between 0 and 1. |
required |
df
|
float
|
The degrees of freedom, above 0. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
float |
float
|
The quantile, such as 2.262 for |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/analysis/intervals.py
Experiments
desflow.run_experiment(spec, *, raise_on_failure=True, output=None)
Runs an experiment: its variants and sweep, replications, comparisons and checks.
Every artifact is written to the experiment's output folder before anything is
raised: experiment.json (settings and evidence), results.csv, summary.csv,
comparisons.csv, checks.csv and report.md.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
Union[str, Path, Mapping[str, Any], Any]
|
An experiment file, a mapping of the same settings, or an
|
required |
raise_on_failure
|
bool
|
When True, a failed check raises |
True
|
output
|
Union[str, Path, None]
|
The folder to write the artifacts to, in place of the experiment's
|
None
|
Returns:
| Name | Type | Description |
|---|---|---|
ExperimentResult |
Any
|
What the experiment found. |
Raises:
| Type | Description |
|---|---|
BlueprintError
|
If the experiment file is invalid, or the model or observed file it names does not exist, naming the file and line. |
ValueError
|
If the settings are invalid, a variant sets a path the model does not have, or the observed values do not match the measures; for settings given as a mapping, also a missing model or observed file. |
CheckFailed
|
If a check failed and |
Source code in src/desflow/analysis/runner.py
desflow.ExperimentSpec
Bases: _Model
A whole experiment file, or the same settings given in Python.
model is a blueprint path, a builder factory (a function with no arguments
returning a new SimulationContext), a LowLevelModel, or in a file a mapping
of a low-level function and its sim_id. Relative paths are read from the
experiment file's folder. until is where each replication's arrivals stop;
stop and drain end each replication as SimulationContext.run does, and
when they are not set a blueprint's own run.stop and run.drain apply.
desflow.analysis.Check
Bases: _Model
A checks entry: a bound on one measure of one variant, or of every variant.
on: interval, the default, checks the far end of the confidence interval (the
upper end against max, the lower against min), so noise cannot pass a check;
on: mean checks the mean only.
desflow.analysis.Compare
Bases: _Model
A compare entry: two variants compared run by run with the same seeds.
desflow.analysis.Observed
Bases: _Model
The observed section: a CSV of values measured on the real system.
Its columns are measure, value, unit, population and horizon. Each row
is compared with the simulated measure of variant, the first variant when it
is not given.
desflow.analysis.load_experiment(path)
Reads and validates an experiment file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Union[str, Path]
|
The experiment file. |
required |
Returns:
| Type | Description |
|---|---|
tuple[ExperimentSpec, SourceMap]
|
The settings, and the map from each key to its line. |
Raises:
| Type | Description |
|---|---|
BlueprintError
|
If the file is invalid; the message names the file and line. |
Source code in src/desflow/analysis/spec.py
desflow.analysis.SourceMap(path, lines)
Maps a location in the document, such as ("tasks", "mill", "service"), to a line.
Validation errors and cross-reference errors carry a location rather than a line,
so this is what turns them into file:line messages a reader can act on.
Source code in src/desflow/blueprint/loader.py
Methods:
anchor(location)
Returns location with a part under an alias moved to its anchor.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
location
|
Iterable[Union[str, int]]
|
The keys and list indexes from the document's top. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Location |
Location
|
The place of the same value in the anchor's collection, or
|
Source code in src/desflow/blueprint/loader.py
error(location, message)
Builds a BlueprintError pointing at location.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
location
|
Iterable[Union[str, int]]
|
The keys and list positions from the document's root. |
required |
message
|
str
|
What is wrong, and how to fix it. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
BlueprintError |
BlueprintError
|
An error whose text starts with the file and line. |
Source code in src/desflow/blueprint/loader.py
has(location)
Whether the document has a key or list item at exactly location.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
location
|
Iterable[Union[str, int]]
|
The keys and list indexes from the document's top. |
required |
Source code in src/desflow/blueprint/loader.py
keys(location)
The keys the document writes directly under location, as written.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
location
|
Iterable[Union[str, int]]
|
The keys and list indexes from the document's top. |
required |
Source code in src/desflow/blueprint/loader.py
line(location)
Returns the line of the deepest part of location the document has.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
location
|
Iterable[Union[str, int]]
|
The keys and list positions from the document's root, such as
|
required |
Returns:
| Name | Type | Description |
|---|---|---|
int |
int
|
The line, counting from 1; 1 when no part of the location is found. |
Source code in src/desflow/blueprint/loader.py
desflow.ExperimentResult(spec, output, designs, results, summary, comparisons, checks, observed, report)
dataclass
What an experiment found, as run_experiment returns it and its artifacts hold it.
Attributes:
| Name | Type | Description |
|---|---|---|
spec |
ExperimentSpec
|
The experiment's settings. |
output |
Path
|
The folder the artifacts were written to. |
designs |
List[Design]
|
Each variant at each sweep point, with its seeds. |
results |
List[Dict]
|
One row per design, replication and statistic. |
summary |
List[Dict]
|
One row per design and statistic. |
comparisons |
List[Dict]
|
One row per compared pair, point and measure. |
checks |
List[Dict]
|
One row per check, design and bound. |
observed |
List[Dict]
|
Each observed value beside its simulated interval. |
report |
str
|
The Markdown report. |
Attributes
failed
property
The check rows that failed.
desflow.analysis.Design(name, variant, point, overrides, seeds, runs=list())
dataclass
One variant at one point of the sweep, with the seeds its replications use.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
The variant's name, followed by the swept values, such as
|
variant |
str
|
The variant's name. |
point |
Dict[str, Any]
|
The swept path and value, by path; empty without a sweep. |
overrides |
Dict[str, Any]
|
Every registry path the design sets. |
seeds |
List[int]
|
The seed of each replication. |
desflow.analysis.RunResult(readings, streams, sim_id, parameters=dict())
dataclass
What one unpaced run of a model measured.
Attributes:
| Name | Type | Description |
|---|---|---|
readings |
Dict[str, Any]
|
Every statistic, by its published name. |
streams |
List[str]
|
The names of the random streams the run drew from. |
sim_id |
str
|
The model's sim id. |
parameters |
Dict[str, Any]
|
Every distribution and capacity by registry path, as the run started, after its overrides. |
desflow.CheckFailed(result)
Bases: Exception
An experiment's check failed. The artifacts are written before it is raised.
Attributes:
| Name | Type | Description |
|---|---|---|
result |
ExperimentResult
|
The whole result, failed checks included. |
Source code in src/desflow/analysis/results.py
Input Fitting
desflow.fit_column(values, *, unit='s', candidates=None, lower=None, upper=None, bins=None, min_count=20, tolerance=0.25, drop_invalid=False)
Fits a column of measured times to desflow's distribution families.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
values
|
Sequence[Any]
|
The data, as numbers or text; an empty cell or None is missing. |
required |
unit
|
str
|
The data's unit: |
's'
|
candidates
|
Optional[Sequence[str]]
|
The families to try; None tries every family of times (not |
None
|
lower
|
Optional[float]
|
The lowest plausible value, in the data's unit; a value below it is invalid, and a family that can draw below it is flagged. |
None
|
upper
|
Optional[float]
|
The highest plausible value, in the same way. |
None
|
bins
|
Optional[int]
|
Offer the empirical distribution as a histogram with this many bins, rather than as the values themselves. |
None
|
min_count
|
int
|
The fewest values a family is fitted to. Defaults to 20. |
20
|
tolerance
|
float
|
How far a fitted family's mean and standard deviation may be from the data's, as a share of the data's. Defaults to 0.25. |
0.25
|
drop_invalid
|
bool
|
Leave invalid values out rather than raising. |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
FitResult |
FitResult
|
Every candidate's outcome, and the best. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the unit or a candidate is unknown, or a value is invalid
and |
ImportError
|
If the fit extra is not installed. |
Source code in src/desflow/analysis/fit.py
462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 | |
desflow.FitResult(candidates, values, unit='s', rows=0, missing=0, invalid=0, lower=None, upper=None)
dataclass
The outcome of fitting one column: every candidate, the best and the data.
Attributes:
| Name | Type | Description |
|---|---|---|
candidates |
List[Candidate]
|
One per family tried, empirical last. |
values |
List[float]
|
The values fitted, in seconds. |
unit |
str
|
The unit the data was written in. |
rows |
int
|
Rows read. |
missing |
int
|
Empty cells, left out. |
invalid |
int
|
Values that are not numbers, not finite, or outside the
bounds; left out with |
lower |
Optional[float]
|
The lowest plausible value, in seconds, or None. |
upper |
Optional[float]
|
The highest plausible value, in seconds, or None. |
Attributes
best
property
The fitted family with the lowest AIC that cannot draw outside the bounds;
constant when every value is the same. When no family within the bounds
fitted, the empirical distribution of the data, which stays within them;
then the fitted family with the lowest AIC; None when nothing is usable.
table
property
One row per candidate, with its status, reason, parameters and diagnostics, and whether it is the best.
Methods:
plot(path, kind=None)
Draws the data against values generated from a candidate, as a PNG.
The left panel puts the histogram of the data over the histogram of
generated values; the right is a Q-Q plot, each sorted data value against
the generated value at the same quantile. The generated values come from
desflow's Sampler, as a run draws them, with the same smallest delay.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Union[str, Path]
|
The image to write. |
required |
kind
|
Optional[str]
|
The family; None takes the best. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Path |
Path
|
The image written. |
Source code in src/desflow/analysis/fit.py
settings(kind=None)
Returns a usable candidate's settings, for add_service(name, **settings).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kind
|
Optional[str]
|
The family; None takes the best. |
None
|
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: |
Raises:
| Type | Description |
|---|---|
ValueError
|
If no candidate is usable, or |
Source code in src/desflow/analysis/fit.py
text()
The table as text: one line per candidate, the best marked.
Source code in src/desflow/analysis/fit.py
yaml(kind=None, source='')
The settings as a YAML mapping a blueprint loads with !include.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
kind
|
Optional[str]
|
The family; None takes the best. |
None
|
source
|
str
|
Where the data came from, for the comment at the top. |
''
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
A comment, then |
Source code in src/desflow/analysis/fit.py
desflow.analysis.Candidate(family, status, reason='', parameters=dict(), log_likelihood=None, aic=None, ks_statistic=None, ks_p_value=None, bounds='')
dataclass
One family's outcome: its parameters, diagnostics and status.
Attributes:
| Name | Type | Description |
|---|---|---|
family |
str
|
The family, as |
status |
str
|
|
reason |
str
|
Why it was rejected, did not fit or was skipped; empty when fitted. |
parameters |
Dict[str, Any]
|
desflow's parameters, in seconds. |
log_likelihood |
Optional[float]
|
The log-likelihood of the data. |
aic |
Optional[float]
|
Akaike's information criterion, 2k minus twice the log-likelihood, where k counts the fitted parameters. |
ks_statistic |
Optional[float]
|
The Kolmogorov-Smirnov distance between the data and the fitted family. |
ks_p_value |
Optional[float]
|
Its p-value. It is approximate, because the parameters were estimated from the same data. |
bounds |
str
|
Where the family can draw outside |
Attributes
usable
property
True when the family was fitted and its result passed every check.
desflow.analysis.read_column(path, column)
Reads one column of a CSV file, as text, in row order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Union[str, Path]
|
A CSV file with a header row. |
required |
column
|
str
|
The column's name in the header. |
required |
Returns:
| Type | Description |
|---|---|
List[str]
|
List[str]: Each row's cell, empty where the cell is empty, for |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the file has no such column. |
Source code in src/desflow/analysis/fit.py
Explaining a Model
SimulationContext.analyze is the usual entry point, and desflow explain prints the same Explanation. build_graph, analyze and verify are the steps it takes, and render_terminal and to_json print the result. The JSON form is experimental through 1.x.
desflow.explain.Explanation
Bases: _Model
A model's graph and its analysis.
Attributes:
| Name | Type | Description |
|---|---|---|
schema_version |
Literal['0']
|
The version of this shape, |
sim_id |
str
|
The model's sim id. |
target |
float
|
The utilisation the hints aim for. |
graph |
Graph
|
The graph. |
arrivals |
List[ArrivalResult]
|
Each arrival. |
tasks |
List[TaskResult]
|
Each task. |
resources |
List[ResourceResult]
|
Each resource. |
holders |
List[HolderResult]
|
Each store and container. |
bottleneck |
Optional[str]
|
The resource with the highest utilisation the formulas cover. |
bottleneck_reason |
Optional[str]
|
Why no bottleneck can be named, when none is. |
hints |
List[Hint]
|
Changes that bring each resource to |
verification |
Optional[Verification]
|
What |
desflow.explain.build_graph(context)
Builds the graph of a context from what it declares, without starting it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
SimulationContext
|
A builder or YAML context, before or after a run. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Graph |
Graph
|
Its arrivals, tasks, resources, stores, containers, opaque code and the exit, with the edges between them. |
Source code in src/desflow/explain/graph.py
28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | |
desflow.explain.Graph
Bases: _Model
A model as nodes and edges, read from its declared linkage.
Attributes:
| Name | Type | Description |
|---|---|---|
sim_id |
str
|
The model's sim id. |
nodes |
List[Node]
|
Every node. |
edges |
List[Edge]
|
Every edge. |
features |
List[str]
|
What changes demand or capacity during the run, for the whole
model, such as |
desflow.explain.Node
Bases: _Model
One part of the model.
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
Unique in the graph: the kind and the name, such as |
kind |
NodeKind
|
|
label |
str
|
The name in the model, such as |
opaque |
bool
|
True for code |
distribution |
Optional[Dict[str, Any]]
|
An arrival's gaps or a task's service, as its parameters
such as |
capacity |
Optional[int]
|
A resource's |
max_capacity |
Optional[int]
|
A resource's |
features |
List[str]
|
What the queueing formulas leave out, such as |
queue |
Optional[str]
|
A resource's queue order: |
priority |
Optional[str]
|
A task's priority as it was given: a number, an expression's
text, or |
desflow.explain.Edge
Bases: _Model
A link between two nodes.
Attributes:
| Name | Type | Description |
|---|---|---|
source |
str
|
The id of the node it leaves. |
target |
str
|
The id of the node it reaches. |
kind |
EdgeKind
|
|
probability |
Optional[float]
|
The share of the source's entities that take a |
desflow.explain.ArrivalResult
Bases: _Model
An arrival's rate and the variability of its gaps.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
The arrival's name. |
rate |
Optional[float]
|
λ, entities a second; None when unknown. |
ca2 |
Optional[float]
|
The squared coefficient of variation of the gaps. |
reason |
Optional[str]
|
Why the rate is unknown. |
desflow.explain.CapacityRow
Bases: _Model
Utilisation at one number of units.
Attributes:
| Name | Type | Description |
|---|---|---|
capacity |
int
|
The number of units. |
utilization |
float
|
ρ at that number. |
status |
Status
|
|
desflow.explain.Comparison
Bases: _Model
One analytical value beside the value a run measured.
Attributes:
| Name | Type | Description |
|---|---|---|
node |
str
|
The resource, task or arrival. |
measure |
str
|
What is compared: |
analytical |
Optional[float]
|
The value of the analysis; None where it has none. |
measured |
Optional[float]
|
The run's value, or the mean of the replications; None when a run has no reading for it. |
half_width |
Optional[float]
|
The half-width of the replications' interval. |
differs |
Optional[bool]
|
True when they disagree: by more than 10 percent for one run; for replications, by more than the interval's half-width and more than 1 percent; None without an analytical value or a reading. |
reason |
Optional[str]
|
Why the measure is not compared: where |
desflow.explain.Hint
Bases: _Model
A change that brings a resource to the target utilisation.
Attributes:
| Name | Type | Description |
|---|---|---|
resource |
str
|
The resource's name. |
kind |
Literal['servers', 'rate']
|
|
value |
float
|
The number of units, or the rate. |
text |
str
|
The hint as a sentence. |
desflow.explain.HolderResult
Bases: _Model
A store's or container's put and get rates.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Its name. |
kind |
Literal['store', 'container']
|
|
put_rate |
Optional[float]
|
Puts a second from the tasks that add to it, where known. |
get_rate |
Optional[float]
|
Gets a second from the tasks that take from it, where known. |
reason |
str
|
Why there is no queueing analysis. |
desflow.explain.ResourceResult
Bases: _Model
A resource's load, utilisation, status and wait estimate.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
The resource's name. |
rate |
Optional[float]
|
λ, requests a second, summed over its tasks. |
service_mean |
Optional[float]
|
τ, the mean service time weighted by each task's rate. |
service_rate |
Optional[float]
|
μ = 1 / τ. |
servers |
Optional[int]
|
c, its |
utilization |
Optional[float]
|
ρ = λ τ / c. |
status |
Optional[Status]
|
|
wait |
Optional[float]
|
Wq, the mean wait in the queue, where a formula applies. |
queue_length |
Optional[float]
|
Lq = λ Wq. |
in_system |
Optional[float]
|
L = λ (Wq + τ). |
method |
Optional[str]
|
The formula behind the wait: |
reason |
Optional[str]
|
Why the utilisation is unknown. |
wait_reason |
Optional[str]
|
Why the wait is not estimated. |
capacities |
List[CapacityRow]
|
ρ at each number of units the resource allows. |
desflow.explain.TaskResult
Bases: _Model
A task's rate from flow balance, and its service.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
The task's name. |
rate |
Optional[float]
|
λ, instances started a second; None when unknown. |
service_mean |
float
|
The effective mean service time; 0 without a service. |
cs2 |
float
|
The squared coefficient of variation of the service time. |
reason |
Optional[str]
|
Why the rate is unknown. |
desflow.explain.Verification
Bases: _Model
What --verify ran and found.
Attributes:
| Name | Type | Description |
|---|---|---|
until |
float
|
The end of each run, or where the arrivals stop when |
stopped |
bool
|
True when the model's |
warmup |
Optional[float]
|
The model's warm-up; the measures leave it out. |
seed |
int
|
The seed, or the seed the replications' seeds come from. |
replications |
Optional[int]
|
The number of runs, or None for one. |
comparisons |
List[Comparison]
|
Every comparison. |
note |
str
|
What the check does and does not show. |
desflow.explain.analyze(graph, target=0.85)
Analyses a graph: rates by flow balance, then each resource's load and wait.
A value is given only where the formulas cover it (support.py); everywhere
else it is None, with the reason.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
graph
|
Graph
|
The graph |
required |
target
|
float
|
The utilisation the hints aim for, above 0 and at most 10. |
0.85
|
Returns:
| Name | Type | Description |
|---|---|---|
Explanation |
Explanation
|
The graph and its analysis. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/explain/analysis.py
desflow.explain.verify
Runs the model unpaced and adds the comparison to the explanation.
Each run is a replication's run (run_once): no ingress, no egress, statistics
at typical. It compares, per resource, utilization_avg / 100 with ρ and
queue_length_avg with Lq; per task with one resource, wait_mean with Wq; per
arrival, flows.<arrival>.finished over the time after the warm-up with λ, except
in a model whose run.stop can end a run early, which has no throughput comparison.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Any
|
A blueprint path, or a function that builds the context again. |
required |
explanation
|
Explanation
|
The analysis to compare with. |
required |
until
|
Union[float, str, None]
|
The end of each run; the blueprint's |
None
|
seed
|
Optional[int]
|
The run's seed, or the seed the replications' seeds come from; the
model's |
None
|
replications
|
Optional[int]
|
The number of runs, 2 or more; None for one run. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Explanation |
Explanation
|
A copy of |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the model is a |
Source code in src/desflow/explain/verify.py
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 | |
desflow.explain.render_terminal(explanation, width=None)
Returns the explanation as plain text: the graph as a tree from each arrival,
the resources, the units each allows, the bottleneck, what is not estimated and
why, the hints, and with --verify the comparison.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
explanation
|
Explanation
|
What |
required |
width
|
Optional[int]
|
The width in characters; the terminal's, or 80, when None. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The text, ending in a newline. |
Source code in src/desflow/explain/render.py
desflow.explain.to_json(explanation)
Returns the explanation as JSON, with schema_version "0".
The shape is experimental through 1.x; explain.schema.json, beside this
module, is its schema.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
explanation
|
Explanation
|
What |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The JSON text. |
Source code in src/desflow/explain/render.py
Environment
desflow.DynamicRealtimeEnvironment(initial_time=0, factor=1.0, strict=None, logical_start_time=None, go_live_at=None, timezone=None, on_unhandled_failure='stop')
Bases: RealtimeEnvironment, RegistryMixIn, IngressMixIn, EgressMixIn, StatisticsMixIn
The core simulation engine for desflow.
This environment extends SimPy's RealtimeEnvironment by incorporating
a centralized SimulationRegistry for dynamic state updates, and MixIns
for managing high-throughput asynchronous I/O with external systems.
Attributes:
| Name | Type | Description |
|---|---|---|
start_datetime |
datetime
|
The logical start time: |
zone |
tzinfo
|
The time zone timestamps are written in. |
Initializes the real-time simulation environment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
initial_time
|
float
|
The initial simulation time. Defaults to 0. |
0
|
factor
|
float
|
The real-time factor (e.g., 1.0 = 1 sim second per real second). Defaults to 1.0. |
1.0
|
strict
|
bool
|
If True, a step raises |
None
|
logical_start_time
|
datetime
|
Overrides the environment's base clock. Crucial for historical backfilling (e.g., generating data from last week). |
None
|
go_live_at
|
datetime
|
Logical instant at which pacing switches to
real time. Until then the run uses |
None
|
timezone
|
str
|
An IANA time zone name, such as
|
None
|
on_unhandled_failure
|
str
|
|
'stop'
|
The clock runs in UTC, so a daylight-saving change in the zone never adds or drops simulation time. An instant at or before the start of the run makes the whole run real time, which is the degenerate case of the same rule.
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/core/environment.py
1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 | |
Methods:
setup_ingress(providers)
Initializes the ingress queues and starts the background listener threads.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
providers
|
List[BaseIngress]
|
A list of initialized ingress connector instances. |
required |
Source code in src/desflow/core/environment.py
setup_egress(providers, batch_size=500, flush_interval=1.0, lag_monitor_interval='auto', max_queued_batches=2000, drain_stall_seconds=30.0, predicates=None, batch_sizes=None, flush_intervals=None, labels=None, warmup=None, envelopes=None, traces=None)
Initializes the egress buffers and starts the background publisher threads.
It first calls each provider's _load_libraries() in the calling thread. In a
strict environment it then moves real_start forward by the time that takes,
so the load is not counted as lag, and a paced run's records trail the wall
clock by the load time. A run with strict False counts the load as lag and
catches up, so its records stay on the wall clock.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
providers
|
List[BaseEgress]
|
A list of initialized egress connector instances. |
required |
batch_size
|
int
|
The maximum number of events to buffer before flushing. Defaults to 500. |
500
|
flush_interval
|
float
|
Maximum simulation seconds to wait before
forcing a flush. Defaults to 1.0. Ignored when |
1.0
|
lag_monitor_interval
|
float
|
Simulation seconds between
|
'auto'
|
max_queued_batches
|
int
|
Upper bound on batches waiting for the egress threads. A full queue blocks the producer, so a sink that cannot keep up slows the simulation rather than accumulating a backlog that teardown would have to discard. Defaults to 2000. |
2000
|
drain_stall_seconds
|
float
|
At teardown every queue is drained until empty. The wait is abandoned only if a queue stops shrinking for this long, so a slow sink finishes rather than losing its tail. Defaults to 30.0. |
30.0
|
predicates
|
List
|
One entry per provider, aligned by position. A callable takes a record and returns True to send it to that provider. None means that provider receives every record. Defaults to None, which is every provider receiving everything. |
None
|
batch_sizes
|
List
|
One entry per provider, aligned by position.
None falls back to |
None
|
flush_intervals
|
List
|
One entry per provider, aligned by
position. None falls back to |
None
|
labels
|
List
|
One entry per provider, aligned by position.
True makes it a labels sink, which receives the ground-truth records
from |
None
|
warmup
|
float
|
The length of the warm-up, in simulation seconds from now, the start of the run when the egress is set up with the model. No provider receives a record stamped before it ends, and the built-in statistics are reset then. Defaults to None, no warm-up. |
None
|
envelopes
|
List
|
One entry per provider, aligned by position.
False makes it a plain-output provider: it receives no telemetry and
no |
None
|
traces
|
List
|
One entry per provider, aligned by position.
True makes it a trace sink, which receives the event trace
( |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If a per-provider list has the wrong length, or |
Each provider gets its own queue. They previously shared one, so several providers competed for batches instead of each receiving them, and a run with two sinks wrote part of the data to each with no error and exit code 0.
Source code in src/desflow/core/environment.py
356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 | |
publish_telemetry(path_id, value)
Publishes a low-volume telemetry metric (e.g., utilization, queue length).
Telemetry shares the main event buffer to ensure efficient batching for high-throughput file storage (like Parquet or S3). Nothing happens when no egress is set up.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path_id
|
str
|
The value's name, written as the record's |
required |
value
|
Any
|
The value, made a JSON value on the way out, as an event's is. |
required |
Source code in src/desflow/core/environment.py
publish_event(event_key, value, task=None, instance=None, key=None, lifecycle=False)
Buffers a high-volume discrete event (e.g., a task starting or finishing).
Events are buffered and flushed either when batch_size is reached
or when flush_interval occurs to optimize external I/O throughput.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
event_key
|
str
|
The record key: the entity's key, such as |
required |
value
|
Any
|
A dictionary or a Pydantic model containing the event
payload, or None, which a Kafka sink with |
required |
task
|
str
|
The name of the task the event belongs to, written
as the record's |
None
|
instance
|
Any
|
The task's instance id, written as the record's
|
None
|
key
|
Any
|
The record's key, when it is not |
None
|
lifecycle
|
bool
|
True for a task's |
False
|
Source code in src/desflow/core/environment.py
watch_changes(callback, queues=False)
Calls callback after every change a shared condition can read.
A change is a store or container level, a resource's capacity or use, a
registry update or a signal firing. A wait on a condition, such as
{condition: "level.stock >= 5"}, watches this way and checks its condition
again after each one, as a SimPy loop over a fresh "changed" event would.
A change of the length of a store's or container's put_queue or
get_queue is not one of these: it calls only the functions added with
queues=True, so a condition that does not read a queue is not checked
again for it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
callback
|
Callable[[], None]
|
A function of no arguments. |
required |
queues
|
bool
|
True to call it also after a change of the length of a
store's or container's put or get queue. A condition that reads
|
False
|
Source code in src/desflow/core/environment.py
unwatch_changes(callback)
Stops calling a function that watch_changes added; does nothing if it is gone.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
callback
|
Callable[[], None]
|
The function |
required |
Source code in src/desflow/core/environment.py
notify_queue_change()
Calls the functions watch_changes added with queues=True. A store or
container calls it after the length of its put_queue or get_queue
changes; it does not change the holder's statistics.
Source code in src/desflow/core/environment.py
resume_pacing()
Restarts pacing from the current simulation time and the wall clock now.
After a pause, such as a debugger stop or a wait between advance() calls,
the wall clock is ahead of the simulation. The next paced event would run at
once, or raise under strict. Calling this makes the next event wait factor
real seconds for each simulation second after now. SimPy's sync() moves
only the wall-clock anchor, so it is not enough on its own once now has
moved. At factor 0 the call changes nothing a run reads.
Source code in src/desflow/core/environment.py
teardown()
Gracefully terminates the environment.
Ensures that any remaining data in event buffers is flushed to the
egress connectors, and that background asyncio threads for both
ingress and egress are cleanly stopped. Should be called in a finally block.
Source code in src/desflow/core/environment.py
Registry and Parameters
desflow.SimulationRegistry(env)
A central registry that maps dot-notation paths to dynamic simulation parameters.
The Registry acts as the single source of truth for the simulation's state. It allows external streams (like Kafka or Redis) to update parameters while the simulation runs, and passes each change to the SimPy processes that use it.
Attributes:
| Name | Type | Description |
|---|---|---|
env |
Environment
|
The active SimPy environment. |
on_change |
List[Callable[[ParamChange], None]]
|
Callbacks that receive a
|
Source code in src/desflow/core/registry.py
Methods:
change(path, value, *, source=None, actor=None, reason=None)
Changes a value as update does, and reports what happened.
The checks are the ones update runs. Where update logs a refusal and goes
on, change raises ParamRejected, for a caller that must know. A value that
is cast or clamped is accepted, and ParamChange.new shows the value stored.
Every on_change callback receives the change before change returns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
The dot-notation path to change. |
required |
value
|
Any
|
The new value. |
required |
source
|
Optional[str]
|
Where the change came from, such as |
None
|
actor
|
Optional[str]
|
Who made the change. |
None
|
reason
|
Optional[str]
|
Why the change was made. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
ParamChange |
ParamChange
|
The old value, the value stored and the time of the change. |
Raises:
| Type | Description |
|---|---|
ParamRejected
|
If the path does not exist, the value cannot be converted to the stored type, or a distribution parameter is out of its range. |
Source code in src/desflow/core/registry.py
check_value(path, new_value)
The value a path accepts for new_value, cast to the path's type.
The one check a change, a scenario step and an on: action share, so each
refuses the same values with the same text. A capacity, max_cap or queue
limit is a whole number of 0 or more (a container's capacity any number of 0
or more), read from text only when the text is such a number; a container
level is a finite number; a true-or-false path takes true, false or the
text "true" or "false"; a variable takes true as 1, and a
distribution parameter refuses it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
The registered path. |
required |
new_value
|
Any
|
The value to write. |
required |
Raises:
| Type | Description |
|---|---|
ParamRejected
|
For an unknown path, a value the path refuses, or a distribution parameter the distribution refuses. |
Source code in src/desflow/core/registry.py
configs()
Returns every configuration object by its path, such as Shop.service.packing.
An experiment records them as the model's compiled parameters. The objects are the live ones, so read them rather than change them.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: Path to |
Source code in src/desflow/core/registry.py
get(path)
Retrieve the DynamicValue object at a specific path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
The dot-notation path (e.g., 'Shop.arrival.orders.rate'). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DynamicValue |
DynamicValue
|
The wrapper object for the requested parameter. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If the path does not exist in the registry. |
Source code in src/desflow/core/registry.py
get_config(path)
Retrieve the live configuration object (e.g., DistributionConfig).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
The dot-notation path (e.g., 'Shop.service.packing'). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The synchronized configuration object. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If the config path does not exist in the registry. |
Source code in src/desflow/core/registry.py
is_queue_limit(path)
Whether path is a resource's queue_limit, as registered.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
A registry path. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True for a registered |
Source code in src/desflow/core/registry.py
paths()
Returns every value path, such as Shop.resources.packers.current_cap.
Returns:
| Type | Description |
|---|---|
List[str]
|
List[str]: The paths |
Source code in src/desflow/core/registry.py
register_breakdown(sim_id, name, breakdown)
Registers a resource's time to failure and repair time.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sim_id
|
str
|
The simulation ID. |
required |
name
|
str
|
The resource's name, which for a member of a family is its
bracketed name such as |
required |
breakdown
|
Any
|
The breakdown, with |
required |
Source code in src/desflow/core/registry.py
register_container(sim_id, name, cap_config)
Registers a container's capacity paths and its .level path.
A current_cap left as None is the max_cap if that is given and no limit
(infinity) if not, and a max_cap left as None is the current_cap. The
config is updated to the values in effect.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sim_id
|
str
|
The simulation ID. |
required |
name
|
str
|
The container's name. |
required |
cap_config
|
CapacityConfig
|
Its capacity and starting level. |
required |
Source code in src/desflow/core/registry.py
register_resource(sim_id, name, cap_config)
Registers a resource's capacity paths, filling in what the config leaves out.
A current_cap left as None is 1, and a max_cap left as None is the
current_cap. The config is updated to the values in effect.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sim_id
|
str
|
The simulation ID. |
required |
name
|
str
|
The resource's name. |
required |
cap_config
|
CapacityConfig
|
Its capacity, which may leave values as None. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If only |
Source code in src/desflow/core/registry.py
register_sim_parameter(param)
Takes a SimParameter instance and flattens it into the registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
param
|
SimParameter
|
The initial state schema to register. |
required |
Source code in src/desflow/core/registry.py
register_store(sim_id, name, cap_config)
Registers a store's capacity paths, filling in what the config leaves out.
A current_cap left as None is the max_cap if that is given and no limit
(infinity) if not, and a max_cap left as None is the current_cap. The
config is updated to the values in effect.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sim_id
|
str
|
The simulation ID. |
required |
name
|
str
|
The store's name. |
required |
cap_config
|
CapacityConfig
|
Its capacity, which may leave values as None. |
required |
Source code in src/desflow/core/registry.py
set_exactly(path, value, *, source=None, actor=None, reason=None)
Sets a path to exactly value, raising where update would log and go on.
Replications and experiments set their variants' values this way before the clock starts, so a misspelt path or a refused value stops the experiment rather than leaving the model's own value in place.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
The dot-notation path. |
required |
value
|
Any
|
The value it must hold. |
required |
source
|
Optional[str]
|
Where the change came from; see |
None
|
actor
|
Optional[str]
|
Who made the change. |
None
|
reason
|
Optional[str]
|
Why the change was made. |
None
|
Raises:
| Type | Description |
|---|---|
ParamRejected
|
A |
Source code in src/desflow/core/registry.py
814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 | |
update(path, new_value, *, source=None, actor=None, reason=None)
Update a value safely and synchronize its parent configuration object.
Includes dynamic type casting to protect the simulation from crashing if external systems send improperly typed data (e.g., sending the string "5" instead of the integer 5). A number written over a number is stored as given on a distribution parameter, a container value and a variable, so 0.5 over 2 stays 0.5 and 9.5 over a variable of 10 stays 9.5.
A capacity path (current_cap or max_cap) takes inf and the text "inf"
as no limit, whatever type it held.
A distribution parameter outside its range (for example a negative std) is
refused with an error log line, as a value of the wrong type is, and the old
value is kept. A current_cap outside [0, max_cap] is clamped to that range
with a warning log line, also when a lower max_cap arrives, and so is a
container level outside [0, current_cap]. A path the model does not have,
such as a parameter the distribution's family does not take, is refused with a
warning log line. update never raises for a refused value; change does, and
returns what was applied.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
The dot-notation path to update. |
required |
new_value
|
Any
|
The new value to apply. |
required |
source
|
Optional[str]
|
Where the change came from, such as |
None
|
actor
|
Optional[str]
|
Who made the change. |
None
|
reason
|
Optional[str]
|
Why the change was made. |
None
|
Source code in src/desflow/core/registry.py
desflow.DynamicValue(env, name, initial_value)
The atom of the registry. Holds a value and a SimPy event that triggers whenever the value changes.
This class is used internally by the SimulationRegistry to wrap configuration
values so that SimPy processes can yield and wait for them to change.
Attributes:
| Name | Type | Description |
|---|---|---|
env |
Environment
|
The active SimPy environment. |
name |
str
|
The dot-notation path/name of this value (e.g., 'Shop.resources.packers.current_cap'). |
Source code in src/desflow/core/registry.py
Attributes
value
property
The current underlying value.
Methods:
track(new_value)
Records a value the model changed itself, without signalling a change.
A container's level changes whenever a task puts or takes an amount. The registry keeps that level so a reader sees it, but the container that made the change must not be told to apply it again.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
new_value
|
Any
|
The value now in effect. |
required |
Source code in src/desflow/core/registry.py
update(new_value)
Updates the value and signals any waiting SimPy processes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
new_value
|
Any
|
The new value to store. |
required |
Source code in src/desflow/core/registry.py
wait_for_change()
Returns a SimPy event that will be triggered when the value updates.
Returns:
| Type | Description |
|---|---|
StoreGet
|
simpy.events.StoreGet: A get-request for the internal signal. |
desflow.ParamChange(*, path, old, new, requested, at, source=None, actor=None, reason=None)
dataclass
One accepted change to a registry path.
SimulationRegistry.change returns it, and each on_change callback receives it.
It is frozen, so a callback cannot alter what the next callback receives.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
str
|
The dot-notation path that changed, such as
|
old |
Any
|
The value before the change. |
new |
Any
|
The value stored, after any cast or clamp. |
requested |
Any
|
The value the caller asked for. |
at |
float
|
The simulation time at which the change was applied. |
source |
Optional[str]
|
Where the change came from, such as |
actor |
Optional[str]
|
Who made the change, such as the ingress connector's class name. |
reason |
Optional[str]
|
Why the caller made the change. |
Attributes
changed
property
Whether the stored value differs from the value before the change.
desflow.ParamRejected(path, value, reason)
Bases: ValueError
Raised by SimulationRegistry.change when the registry refuses a value.
It is a ValueError, so a caller that catches ValueError also catches it.
str(exc) is the reason.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
str
|
The path the caller asked to change. |
value |
Any
|
The value the registry refused. |
reason |
str
|
Why the registry refused it. |
Source code in src/desflow/core/registry.py
desflow.SimParameter(sim_id, arrival=dict(), service=dict(), resources=dict(), containers=dict(), stores=dict(), breakdowns=dict(), variables=dict())
dataclass
The master schema representing the state of a specific simulation entity (like a production line).
This object is registered with the SimulationRegistry, which flattens
the nested dictionaries into dot-notation paths (e.g., 'Shop.arrival.orders.rate').
Attributes:
| Name | Type | Description |
|---|---|---|
sim_id |
str
|
The unique prefix for this group of parameters (e.g., 'Shop'). |
arrival |
Dict[str, DistributionConfig]
|
Configurations for arrival generation rates. |
service |
Dict[str, DistributionConfig]
|
Configurations for process task durations. |
resources |
Dict[str, CapacityConfig]
|
Configurations for standard SimPy Resources. |
containers |
Dict[str, CapacityConfig]
|
Configurations for continuous SimPy Containers.
A |
stores |
Dict[str, CapacityConfig]
|
Configurations for discrete SimPy Stores. |
breakdowns |
Dict[str, BreakdownConfig]
|
How each named resource breaks
down, registered at |
variables |
Dict[str, Any]
|
Flexible user-defined state variables (e.g., int, float, bool, str) used to drive custom logic, track string-based states, or act as external control dials. |
desflow.CapacityConfig(current_cap=None, max_cap=None)
dataclass
Configuration for the capacity of a simulated resource, container, or store.
A value left as None is filled in when the holder registers. A resource's
current_cap is 1, and max_cap is current_cap. A store's or container's
current_cap is max_cap when that is given and no limit if not, and its
max_cap is current_cap. float("inf") also means no limit.
Attributes:
| Name | Type | Description |
|---|---|---|
current_cap |
Optional[Union[int, float]]
|
The currently active capacity. |
max_cap |
Optional[Union[int, float]]
|
The absolute physical maximum capacity. |
desflow.DistributionConfig(dist, rate=None, mean=None, std=None, low=None, value=None, high=None, mode=None, shape=None, scale=None, values=None, weights=None, edges=None, counts=None, p=None)
dataclass
Configuration for a statistical distribution used in the simulation.
This config is dynamically updatable. For example, you can change the rate
of an 'exponential' distribution via Kafka, and the Sampler will instantly
use the new rate for the next event. The family (dist) is fixed at load, and
only its parameters change live.
Numbers are stored as given: an integer stays an integer, as in SimPy, and a live
update to 0.5 is not cast to an integer.
The family and its parameters are checked when the config is built, against the
registry of distributions in desflow.models.distributions.
Attributes:
| Name | Type | Description |
|---|---|---|
dist |
str
|
The family: exponential, normal, lognormal, truncated_normal, poisson, geometric, constant, uniform, triangular, gamma, weibull or empirical. |
rate |
Optional[float]
|
Events per second, for exponential. Greater than 0. |
mean |
Optional[float]
|
The mean, for normal, lognormal, truncated_normal and poisson. Greater than 0 for lognormal; 0 or more for poisson. |
p |
Optional[float]
|
The chance of success on each try, for geometric. Above 0 and at most 1. |
std |
Optional[float]
|
The standard deviation, 0 or more; 0 (or no value) gives the mean every time. |
low |
Optional[float]
|
The lowest value: of a truncated normal (0 by default, and a draw below it is drawn again), a uniform or a triangular. |
value |
Optional[float]
|
The time a constant distribution always gives. 0 or more. |
high |
Optional[float]
|
The highest value of a uniform or a triangular. |
mode |
Optional[float]
|
The most likely value of a triangular. |
shape |
Optional[float]
|
The shape of a gamma or a weibull. Greater than 0. |
scale |
Optional[float]
|
The scale of a gamma or a weibull. Greater than 0. |
values |
Optional[List[float]]
|
The values an empirical draws from. |
weights |
Optional[List[float]]
|
One weight per empirical value. |
edges |
Optional[List[float]]
|
The bin edges of an empirical histogram. |
counts |
Optional[List[float]]
|
One count per empirical histogram bin. |
stream |
Optional[str]
|
The name of the random sequence it draws from: its
registry path without the sim id, such as |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the family is unknown, a parameter it does not take is given, or a parameter it takes is missing or out of range. |
desflow.ContainerConfig(current_cap=None, max_cap=None, level=0)
dataclass
Bases: CapacityConfig
Configuration for a container: its capacity and the level it starts at.
current_cap is the container's capacity, its largest level, as it is a
resource's capacity. The level has a registry path of its own, .level, so a
scenario or an ingress message can set it during a run, such as a restock.
Attributes:
| Name | Type | Description |
|---|---|---|
current_cap |
Union[int, float]
|
The capacity: the largest level. |
max_cap |
Union[int, float]
|
The largest capacity a live update can set. |
level |
Union[int, float]
|
The amount held when the run starts, kept as the
type given, as |
desflow.BreakdownConfig(time_to_failure, repair, work='interrupt', clock='calendar')
dataclass
How a resource breaks down: the time to each failure, the repair time, and what happens to the work in progress.
The next failure is drawn when a repair ends, or at the start. With the calendar
clock it counts that time busy or idle, and with the busy clock only while the
resource holds a unit. While down, the resource's capacity is 0; its current_cap
keeps the planned value, which a change during the repair sets for its return.
Attributes:
| Name | Type | Description |
|---|---|---|
time_to_failure |
DistributionConfig
|
The time from a repair, or the start, to the next failure. |
repair |
DistributionConfig
|
How long a repair takes. |
work |
str
|
|
clock |
str
|
|
Data Payloads
desflow.TelemetryPayload
Bases: BaseStreamPayload
Schema for low-volume, single-metric telemetry updates.
This payload is used for publishing continuous system state variables, such as resource utilization, queue lengths, or system lag, typically used for real-time dashboards.
Attributes:
| Name | Type | Description |
|---|---|---|
stream_type |
Literal['telemetry']
|
Hardcoded to "telemetry" for downstream routing. |
path_id |
str
|
The dot-notation path of the metric (e.g., 'Shop.packers.utilization'). |
value |
Any
|
The scalar value of the metric at the given simulation time. |
desflow.EventPayload
Bases: BaseStreamPayload
Schema for high-volume discrete simulation events.
This payload tracks state transitions of specific entities (like tasks or orders) throughout
the simulation lifecycle. The key attribute allows message brokers like Kafka to maintain
strict chronological ordering for specific tasks.
Attributes:
| Name | Type | Description |
|---|---|---|
stream_type |
Literal['event']
|
Hardcoded to "event" for downstream routing. |
key |
str
|
The entity key, the record and partition key (e.g., 'orders-1'). |
task |
Optional[str]
|
The name of the task the event belongs to, or None for an event published outside a task. |
instance |
Optional[Any]
|
The task's instance id, or None. |
value |
Optional[Dict[str, Any]]
|
The event's payload, a mapping, or None when the payload is null. |
desflow.LabelPayload
Bases: BaseStreamPayload
Schema for ground-truth records: what a scenario step changed, and why.
They are sent only to sinks with when: labels, which receive nothing else, so
data and labels always reach separate sinks.
Attributes:
| Name | Type | Description |
|---|---|---|
stream_type |
Literal['label']
|
Hardcoded to "label" for downstream routing. |
path |
Optional[str]
|
The registry path the step changed, or None for a step
with only a |
old |
Any
|
The value before the step, or None. |
new |
Any
|
The value after the step, or None. |
label |
Dict[str, Any]
|
The step's |
desflow.models.BaseStreamPayload
Bases: BaseModel
Base schema for all egress data emitted by the simulation engine.
This schema guarantees that all outgoing data streams share a common temporal context, allowing downstream systems to read each record at its place on the simulated calendar.
Attributes:
| Name | Type | Description |
|---|---|---|
stream_type |
Literal['telemetry', 'event', 'label']
|
An identifier indicating the nature of the payload. |
sim_ts |
float
|
The simulation clock time (in seconds) when the payload was generated. |
timestamp |
str
|
The ISO 8601 logical time of the payload: the run's logical start time plus |
Utilities
desflow.Sampler(rng=None, *, seed=None)
Generates random numbers based on live DistributionConfig objects.
The Sampler evaluates the configuration at the exact moment it is called. This allows the simulation to dynamically respond to external parameter changes.
Each distribution draws from a random sequence of its own, made from the run's
seed and the distribution's registry path without the sim id (arrival.orders,
service.packing): SeedSequence(seed, spawn_key=stream_key(path)). Adding
an arrival therefore leaves every other distribution's draws unchanged, so two
versions of a model run with one seed draw the same numbers for every
distribution they share (common random numbers).
Attributes:
| Name | Type | Description |
|---|---|---|
rng |
Optional[Generator]
|
The run's main generator, for draws in your own code. Distributions do not draw from it, so adding one does not change these draws either. None when nothing is drawn at random. |
seed |
Optional[int]
|
The seed the sequences come from, or None. |
Initializes the Sampler.
With neither rng nor seed, nothing is drawn at random: each sample is
the distribution's mean (for truncated_normal, its mean or low, whichever
is higher).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
rng
|
Optional[Generator]
|
A seeded NumPy generator, such as |
None
|
seed
|
Optional[int]
|
The run's seed, instead of |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If both |
Source code in src/desflow/core/sampler.py
Methods:
sample(config, min_delay=MIN_DELAY)
Draws a random sample using the current parameters in the provided config.
A config the registry has registered draws from its own sequence; one built
by hand and never registered draws from rng.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
DistributionConfig
|
The live configuration object retrieved from the Registry. |
required |
min_delay
|
float
|
The smallest value returned. |
MIN_DELAY
|
Returns:
| Type | Description |
|---|---|
Union[int, float]
|
Union[int, float]: A sampled time duration (e.g., arrival gap or service time),
an integer when the config holds integers and its draw is one.
Returns |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the |
Source code in src/desflow/core/sampler.py
desflow.time_to_seconds(time_str)
Converts a human-readable duration into seconds.
Examples:
"1 week" -> 604800.0 "2.5 hours" -> 9000.0 "30 mins" -> 1800.0
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
time_str
|
str
|
A number and a unit, such as |
required |
Returns:
| Name | Type | Description |
|---|---|---|
float |
float
|
The seconds. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the text is not a number and a known unit. |
Source code in src/desflow/utils.py
Resources
desflow.ResourceBase(env, sim_id, resource_id, queue=DEFAULT_QUEUE_ORDER, preempt=False, queue_limit=None, ties='fifo')
Bases: BaseDynamicResource
A resource of units that tasks request, with a capacity that follows the registry.
It is DynamicResource built on SimPy's hooks, so a subclass changes it as a
subclass of SimPy's Resource does: set PutQueue or GetQueue, set request
or release to a BoundClass of a request or release class, or override
_do_put(request) and _do_get(release). Use a subclass with type= in
add_resource or type: !python module.Class in a blueprint.
The constructor is DynamicResource's, because a resource reads its capacity
from the registry. A subclass that defines __init__ takes *args, **kwargs and
passes them on.
Grants follow SimPy's Resource: a request is granted when it is made if a unit
is free and nothing waits ahead of it, and a release grants the head of the
queue one event later, when its release event is processed; a release that
gives back nothing tries the head too. Equal-time grants therefore happen in
the order SimPy gives them.
Waiting requests are served in the resource's queue order: fifo (the
default) serves the oldest first, lifo the newest first, and priority by
the request's key, (priority, time, not preempt), as SimPy's
PriorityResource does: the lowest priority value first, then the earliest,
then one that may preempt. A sequence number breaks the remaining ties, so
such requests are served first in, first out.
Attributes:
| Name | Type | Description |
|---|---|---|
env |
DynamicRealtimeEnvironment
|
The active simulation environment. |
sim_id |
str
|
The prefix ID (e.g., 'Shop'). |
resource_id |
str
|
The specific resource name (e.g., 'packers'). |
queue_order |
str
|
|
PutQueue |
Any
|
The type of |
GetQueue |
Any
|
The type of |
put_queue |
The waiting requests in service order. |
|
queue |
An alias of |
|
get_queue |
The releases being served, empty between events. |
|
users |
List[DynamicResourceRequest]
|
The granted requests, oldest first. |
count |
int
|
How many users there are; the same as |
request |
Makes a |
|
release |
Makes a |
Initializes the resource and binds it to the registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
env
|
Environment
|
The SimPy environment (must include the RegistryMixIn). |
required |
sim_id
|
str
|
The parent simulation ID. |
required |
resource_id
|
str
|
The name of the resource. |
required |
queue
|
str
|
The queue order: |
DEFAULT_QUEUE_ORDER
|
preempt
|
bool
|
When True, a request that finds every unit busy takes
one from the user SimPy's |
False
|
queue_limit
|
Optional[int]
|
The most requests that may wait, or None for
no limit. A request made while that many wait raises |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/resources/resource.py
275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 | |
Attributes
request = BoundClass(DynamicResourceRequest)
class-attribute
instance-attribute
Requests one unit: request(priority=0, preempt=True) returns a
DynamicResourceRequest, an event that succeeds when a unit is granted. Lower
priority values are served first under queue="priority"; preempt lets the
request take a unit from a user of a higher value on a resource built with
preempt=True. Under queue="priority" every priority is a number, or every
one is a list, because a number cannot be compared with a list; a request of the
other kind raises ValueError. Set it to BoundClass(MyRequest) to use a subclass.
release = BoundClass(DynamicRelease)
class-attribute
instance-attribute
Gives a granted unit back: release(request) returns a DynamicRelease,
triggered at once. Releasing a request twice, or one never granted, gives back
nothing; the queue is still tried when the release is processed. With no request,
the oldest unit held is given back. While units are busy above a lowered
capacity, a release brings the busy count down and grants nothing until it is
within the capacity. Set it to BoundClass(MyRelease) to use a subclass.
in_use
property
Units currently held by tasks, including units above a lowered capacity.
count
property
Number of users currently using the resource, as SimPy's Resource.count;
the same as in_use.
capacity
property
The current active capacity of the resource.
This value reflects the live state from the Registry and may change during the simulation.
desflow.DynamicResource(env, sim_id, resource_id, queue=DEFAULT_QUEUE_ORDER, preempt=False, queue_limit=None, ties='fifo')
Bases: ResourceBase
A SimPy Resource with a dynamically adjustable capacity.
Unlike a standard simpy.Resource where capacity is fixed at creation,
the DynamicResource listens to a SimulationRegistry path (e.g., 'Shop.resources.packers.current_cap').
When the registry updates the capacity, the resource adjusts at once, granting
waiting requests if the capacity rises. It is ResourceBase as built, and the
class to subclass for a resource of your own is ResourceBase.
Source code in src/desflow/resources/resource.py
275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 | |
desflow.ContainerBase
Bases: _Notifier, Container
A SimPy Container that tells DynamicContainer about every change, the class
to subclass for a container of your own.
Subclass it as you subclass simpy.Container: set PutQueue or GetQueue, set
put or get to a BoundClass of an event class, or override _do_put(event)
and _do_get(event). Use the subclass with type= in add_container or type:
!python module.Class in a blueprint. The constructor is SimPy's,
(env, capacity=inf, init=0); a subclass that defines __init__ passes its
arguments on.
A change of the level made by _do_put or _do_get is reported once to the
statistics and to the conditions that read the container, whether the override
calls super() or not, and whatever it returns. A change of the length of
put_queue or get_queue is reported to the conditions that read putters. or
getters.; a queue class must support len() and remove().
Attributes:
| Name | Type | Description |
|---|---|---|
level |
The amount held. |
|
put_queue |
The waiting puts, SimPy's queue. |
|
get_queue |
The waiting gets, SimPy's queue. |
|
put |
Makes a |
|
get |
Makes a |
Attributes
put = BoundClass(DynamicContainerPut)
class-attribute
instance-attribute
get = BoundClass(DynamicContainerGet)
class-attribute
instance-attribute
desflow.DynamicContainer(env, sim_id, container_id, init=None, type=None, get_ties=None, put_ties=None)
Bases: _WrappedHolder, Container
A wrapper for SimPy Container with dynamic capacity updates.
It is also a SimPy Container, so isinstance holds. It runs none of SimPy's
constructor: put_queue, get_queue, level and capacity read the wrapped
container, and put and get call it.
Useful for modeling continuous bulk materials (fluids, gases, battery charge) and
stock. Unlike a standard simpy.Container, the capacity of this container listens
to a SimulationRegistry path and can be mutated at runtime.
current_cap is the capacity, the largest level, as it is for a resource. The
level has a registry path of its own, <sim_id>.containers.<name>.level: the
registry follows every put and get, and a live update to the path sets the level,
such as a restock, waking the tasks it lets through.
Capacity Shrinkage Behavior:
If the capacity is dynamically shrunk below the current level of the container,
the material is NOT destroyed. The container will temporarily exist in an
"overflow" state. All pending and future put requests will be strictly
blocked until downstream processes get enough material to drain the level
back below the new, smaller capacity.
Attributes:
| Name | Type | Description |
|---|---|---|
env |
Environment
|
The active SimPy simulation environment. |
sim_id |
str
|
The parent simulation ID prefix. |
obj_id |
str
|
The specific container name/identifier. |
put_queue |
list
|
The waiting puts, SimPy's queue. |
get_queue |
list
|
The waiting gets, SimPy's queue. |
PutQueue |
list
|
The type of |
GetQueue |
list
|
The type of |
Initializes the DynamicContainer and binds its capacity to the registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
env
|
Environment
|
The SimPy environment (must include Registry interface). |
required |
sim_id
|
str
|
The parent simulation ID. |
required |
container_id
|
str
|
The unique name of this container. |
required |
init
|
float
|
The amount in the container at simulation start.
None, the default, takes the registry's |
None
|
type
|
Type[ContainerBase]
|
A subclass of |
None
|
get_ties
|
str
|
None, the default, keeps SimPy's queue of
waiting gets, first in, first out. |
None
|
put_ties
|
str
|
As |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/resources/container.py
122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 | |
Attributes
level
property
float: The current amount of bulk material stored inside the container.
capacity
property
float: The current active maximum capacity of the container. This value reflects the live state from the Registry.
putters_waiting
property
int: Puts waiting for room. A put granted at once is not counted,
although SimPy removes it from put_queue only after the grant.
getters_waiting
property
int: Gets waiting for an amount. A get granted at once is not counted,
although SimPy removes it from get_queue only after the grant.
Methods:
put(*args, **kwargs)
Request to put a specific amount of material into the container.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*args
|
Any
|
The amount of material to add. A custom |
()
|
**kwargs
|
Any
|
Keyword arguments, passed to the container's |
{}
|
Returns:
| Type | Description |
|---|---|
Event
|
simpy.events.Event: A SimPy event that triggers when capacity is available. |
Source code in src/desflow/resources/container.py
get(*args, **kwargs)
Request to get a specific amount of material from the container.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*args
|
Any
|
The amount of material to withdraw. A custom |
()
|
**kwargs
|
Any
|
Keyword arguments, passed to the container's |
{}
|
Returns:
| Type | Description |
|---|---|
ContainerGet
|
simpy.resources.container.ContainerGet: A SimPy event that triggers when enough material is available. |
Source code in src/desflow/resources/container.py
desflow.StoreBase
Bases: _Notifier, Store
A SimPy Store that tells DynamicStore about every change, the class to
subclass for a store of your own.
Subclass it as you subclass simpy.Store: set PutQueue or GetQueue, set
put or get to a BoundClass of an event class, or override _do_put(event)
and _do_get(event). Use the subclass with type= in add_store or type:
!python module.Class in a blueprint. The constructor is SimPy's,
(env, capacity=inf); a subclass that defines __init__ passes its arguments
on.
A change of the number of items made by _do_put or _do_get is reported
once to the statistics and to the conditions that read the store, whether the
override calls super() or not. A change of the length of put_queue or
get_queue is reported to the conditions that read putters. or getters.
(and count, items); a queue class must support len() and remove().
Attributes:
| Name | Type | Description |
|---|---|---|
items |
The items held, in the order a get returns them. |
|
put_queue |
The waiting puts, SimPy's queue. |
|
get_queue |
The waiting gets, SimPy's queue. |
|
put |
Makes a |
|
get |
Makes a |
Attributes
put = BoundClass(DynamicStorePut)
class-attribute
instance-attribute
get = BoundClass(DynamicStoreGet)
class-attribute
instance-attribute
desflow.DynamicStore(env, sim_id, store_id, priority=False, items=None, type=None, get_ties=None, put_ties=None)
Bases: _WrappedHolder, Store
A wrapper for SimPy FilterStore and PriorityStore with dynamic capacity updates.
It is also a SimPy Store, so isinstance holds. It runs none of SimPy's
constructor: put_queue, get_queue, items and capacity read the wrapped
store, and put and get call it.
Useful for modeling buffers, queues, or staging areas that hold discrete,
heterogeneous items. By setting priority=True, the store will automatically
sort incoming items (which must be sortable or wrapped in simpy.PriorityItem)
so high-priority items are retrieved first.
Capacity Shrinkage Behavior:
If the capacity is dynamically shrunk below the current number of items
in the store, the existing items are NOT destroyed. The store will temporarily
exist in an "over-capacity" state. All pending and future put requests
will be blocked until downstream processes get enough items to bring the
total item count below the new capacity.
Attributes:
| Name | Type | Description |
|---|---|---|
env |
Environment
|
The active SimPy simulation environment. |
sim_id |
str
|
The parent simulation ID prefix. |
obj_id |
str
|
The specific store name/identifier. |
put_queue |
list
|
The waiting puts, SimPy's queue. |
get_queue |
list
|
The waiting gets, SimPy's queue. |
PutQueue |
list
|
The type of |
GetQueue |
list
|
The type of |
Initializes the DynamicStore and binds its capacity to the registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
env
|
Environment
|
The SimPy environment (must include Registry interface). |
required |
sim_id
|
str
|
The parent simulation ID. |
required |
store_id
|
str
|
The unique name of this store. |
required |
priority
|
bool
|
If True, uses |
False
|
items
|
Iterable
|
The items it holds when the run starts, in
the order a get returns them. A priority store takes |
None
|
type
|
Type[StoreBase]
|
A subclass of |
None
|
get_ties
|
str
|
None, the default, keeps SimPy's queue of
waiting gets, first in, first out. |
None
|
put_ties
|
str
|
As |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If there are more items than the capacity, a priority
store's item is not a |
Source code in src/desflow/resources/store.py
167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 | |
Attributes
items
property
writable
list: The actual list of distinct item objects currently residing
in the store. Assigning a list replaces the items, which FilterStore code does
with store.items = [...]. As in SimPy, a get that is waiting is served at the
next put or get, not at the assignment, and the store keeps the list given.
A priority store takes a list that is already a heap.
capacity
property
int: The current active maximum slot capacity of the store. This value reflects the live state from the Registry.
putters_waiting
property
int: Puts waiting for room. A put granted at once is not counted,
although SimPy removes it from put_queue only after the grant.
getters_waiting
property
int: Gets waiting for an item. A get granted at once is not counted,
although SimPy removes it from get_queue only after the grant.
Methods:
put(*args, **kwargs)
Request to put a specific distinct item into the store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*args
|
Any
|
The item to place into the store. A custom |
()
|
**kwargs
|
Any
|
Keyword arguments, passed to the store's |
{}
|
Returns:
| Type | Description |
|---|---|
Event
|
simpy.events.Event: A SimPy event that triggers when a slot is available. |
Source code in src/desflow/resources/store.py
get(*args, **kwargs)
Request to get the next available item from the store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
*args
|
Any
|
The filter, if any. A custom |
()
|
**kwargs
|
Any
|
Keyword arguments, passed to the store's |
{}
|
Returns:
| Type | Description |
|---|---|
StoreGet
|
simpy.resources.store.StoreGet: A SimPy event that yields the
requested item object when available. Its |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a priority store is given a filter: SimPy has no store that both ranks and filters. |
Source code in src/desflow/resources/store.py
desflow.resources.DynamicResourceRequest(resource, priority=0, preempt=True)
Bases: PriorityRequest
A request for one unit of a DynamicResource, and the event of its grant.
It succeeds when a unit is granted. Used in a with block, leaving the block
cancels it while it waits and releases the unit once it is granted, as a SimPy
Request does. It is a SimPy PriorityRequest, so isinstance holds, and it
does not run SimPy's constructor: the resource ranks and queues the request.
Attributes:
| Name | Type | Description |
|---|---|---|
resource |
ResourceBase
|
The resource it asks. |
priority |
float
|
Its rank under the |
preempt |
bool
|
Whether it may take a unit from a user of a higher priority
value, on a resource built with |
key |
tuple
|
|
time |
float
|
The simulation time the request was made. |
usage_since |
Optional[float]
|
The simulation time of the grant, or None. |
proc |
The process that made the request, the one an interrupt would reach. |
|
preempted |
bool
|
Whether a request of a higher rank took the unit. |
freed |
bool
|
Whether a |
attributes |
Any
|
The attributes of the entity that asked, which a |
Source code in src/desflow/resources/resource.py
Methods:
cancel()
Leaves the queue at once while waiting; does nothing once granted.
A granted unit is given back with DynamicResource.release(request), as in
SimPy, where cancel never releases a unit either.
Source code in src/desflow/resources/resource.py
desflow.resources.DynamicRelease(resource, request=None)
Bases: Event
The release of a unit of a ResourceBase, as SimPy's Release event.
It is triggered when it is made, because a release is served at once. The
resource then tries the head of its queue one event later, when this event is
processed. A subclass of this class, set as release = BoundClass(MyRelease) on
a ResourceBase subclass, replaces it.
Attributes:
| Name | Type | Description |
|---|---|---|
resource |
ResourceBase
|
The resource it gives a unit back to. |
request |
Optional[DynamicResourceRequest]
|
The request it releases. A release made with no request holds the oldest user, or None when nobody holds a unit. |
proc |
The process that made the release. |
Source code in src/desflow/resources/resource.py
desflow.resources.PreemptRule(when=None, victim=None)
How a resource chooses the user a request preempts, and when it may.
Each argument is a function of the new request and a holder, both
DynamicResourceRequests. The resource gives the functions the request that
found no free unit, and the holders that use a unit.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
when
|
Optional[Callable[[Any, Any], Any]]
|
Returns whether |
None
|
victim
|
Optional[Callable[[Any, Any], Any]]
|
Returns a value for each holder, and the holder with the highest value
is the one |
None
|
Source code in src/desflow/resources/resource.py
desflow.resources.QueueFull
Bases: RuntimeError
A request was refused because its resource's waiting line is full.
It is a RuntimeError with SimPy's message, as SortedQueue(maxlen) raises.
desflow.resources.DynamicContainerPut(resource, *args, rank=None, **kwargs)
Bases: _Ranked, _CancelReports, ContainerPut
The put event of ContainerBase: SimPy's ContainerPut, whose cancel() also
reports the change of the queue length. Its rank keyword ranks it in a queue
that DynamicContainer was built to rank. Subclass it and set
put = BoundClass(MyPut).
Source code in src/desflow/resources/base.py
desflow.resources.DynamicContainerGet(resource, *args, rank=None, **kwargs)
Bases: _Ranked, _CancelReports, ContainerGet
The get event of ContainerBase: SimPy's ContainerGet, whose cancel() also
reports the change of the queue length. Its rank keyword ranks it in a queue
that DynamicContainer was built to rank. Subclass it and set
get = BoundClass(MyGet).
Source code in src/desflow/resources/base.py
desflow.resources.DynamicStorePut(resource, *args, rank=None, **kwargs)
Bases: _Ranked, _CancelReports, StorePut
The put event of StoreBase: SimPy's StorePut, whose cancel() also reports
the change of the queue length. Its rank keyword ranks it in a queue that
DynamicStore was built to rank. Subclass it and set put = BoundClass(MyPut).
Source code in src/desflow/resources/base.py
desflow.resources.DynamicStoreGet(resource, *args, rank=None, **kwargs)
Bases: _Ranked, _CancelReports, StoreGet
The get event of StoreBase: SimPy's StoreGet, whose cancel() also reports
the change of the queue length. Its rank keyword ranks it in a queue that
DynamicStore was built to rank. Subclass it and set get = BoundClass(MyGet).
Source code in src/desflow/resources/base.py
desflow.resources.DynamicFilterStoreGet(resource, *args, rank=None, **kwargs)
Bases: _Ranked, _CancelReports, FilterStoreGet
The get event of a filtered store: SimPy's FilterStoreGet, whose cancel()
also reports the change of the queue length. It takes a rank keyword, as
DynamicStoreGet does.
Source code in src/desflow/resources/base.py
Breakdowns, Schedules and Replay
The builder and YAML start these SimPy processes and readers for you. Low-level code starts them itself.
desflow.run_breakdowns(env, resource, sampler, work='interrupt', clock='calendar')
Breaks a resource down and repairs it, for as long as the run lasts.
The time to failure and the repair time are read from the registry, at
<sim_id>.breakdown.<resource>.time_to_failure and .repair, which a
SimParameter's breakdowns registers, so they can change during the run. Each
failure publishes an event with the resource's name as its key and
{"status": "down", "reason": "failure"}, and each repair one with
{"status": "repaired", "reason": "failure"}; neither has a task.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
env
|
Any
|
The environment. |
required |
resource
|
ResourceBase
|
The resource that breaks down. |
required |
sampler
|
Sampler
|
The sampler the times are drawn with. |
required |
work
|
str
|
|
'interrupt'
|
clock
|
str
|
|
'calendar'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/builder/breakdowns.py
desflow.run_scenario(env, steps, until=None, interrupt=None, evaluate=None)
Applies each step at its time, in time order and, at equal times, in the order
given. A step's at counts from when this process starts, the start of the run
when it is started with the model, so it does not depend on initial_time.
A step with a path and value changes the registry, and each on_change
callback receives the change with the source "scenario"; one with a record
publishes a ground-truth record of the change (env.publish_label); one with an
interrupt, {task, where, cause}, calls interrupt(task, where=..., cause=...),
which the builder and YAML pass as context.interrupt.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
env
|
Any
|
The environment. |
required |
steps
|
Iterable[Mapping[str, Any]]
|
The steps, each a mapping as |
required |
until
|
Optional[float]
|
The run's end, a reading of the clock; steps due at or after it are
dropped, as an event due at exactly |
None
|
interrupt
|
Optional[Callable[..., Any]]
|
What an |
None
|
evaluate
|
Optional[Callable[[Any], Any]]
|
Evaluates a value that holds a generator when its step is due;
the builder and YAML pass the run's value runtime. A |
None
|
Raises:
| Type | Description |
|---|---|
ParamRejected
|
When the run reaches a step whose value the registry refuses,
such as an exponential rate of 0. It stops the run, as any exception in
a process does. The builder and a blueprint check the values with the
run's overrides before the clock starts, so their runs reach it only
when an ingress message, an |
Source code in src/desflow/builder/scenario.py
desflow.read_trace(source, time='time', unit='s', fields='all', on_invalid='error', start=None, initial_time=0.0)
Reads and checks the rows of a trace, in their order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
Union[str, Path, Iterable[Mapping[str, Any]]]
|
A CSV file, a Parquet file (which needs the |
required |
time
|
str
|
The column that holds each row's time. |
'time'
|
unit
|
str
|
|
's'
|
fields
|
Union[str, Sequence[str]]
|
The columns each arrival carries as attributes, or |
'all'
|
on_invalid
|
str
|
|
'error'
|
start
|
Optional[datetime]
|
The date and time of simulation time 0, as for every record's
|
None
|
initial_time
|
float
|
The simulation clock's start. |
0.0
|
Returns:
| Type | Description |
|---|---|
List[Row]
|
List[Row]: The rows, each with its simulation time. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the unit or |
ImportError
|
If a Parquet file is read without pyarrow. |
Source code in src/desflow/core/replay.py
desflow.Row(time, number, fields)
dataclass
One arrival of a trace, as read_trace gives it (the row generator is
desflow.values.Row).
Attributes:
| Name | Type | Description |
|---|---|---|
time |
float
|
Its simulation time. |
number |
int
|
Its row number in the source, counting the first row as 1. |
fields |
Dict[str, Any]
|
The fields it carries, which become the entity's attributes. |
Admin and Infrastructure
desflow.connectors.admin.kafka.KafkaAdminConnector(bootstrap_servers, max_tasks=100, **kwargs)
Unified Kafka Admin and Monitoring Connector.
This connector acts as a management layer for the simulation's Kafka
infrastructure. It creates topics, sends config and collects telemetry and
events, all with aiokafka; create_topics is a synchronous method.
It maintains an internal state of simulation vitals and task lifecycles by consuming from the simulation's egress topics.
Attributes:
| Name | Type | Description |
|---|---|---|
bootstrap_servers |
str
|
Kafka broker addresses. |
max_tasks |
int
|
The maximum number of task records to keep per service in memory to prevent unbounded growth. |
kwargs |
dict
|
Additional arguments passed to Kafka clients. |
Initialize the connector with broker settings and state limits.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bootstrap_servers
|
str
|
Kafka broker addresses. |
required |
max_tasks
|
int
|
The maximum number of task records to keep per service in memory (rolling window). |
100
|
**kwargs
|
Any
|
Additional arguments passed to Kafka clients (e.g., security settings). |
{}
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/connectors/admin/kafka.py
Methods:
create_topics(topics_config)
Creates Kafka topics required for the simulation.
This is a synchronous call to ensure that all necessary infrastructure (config, telemetry, and event topics) is ready before the simulation environment starts. It runs the asynchronous admin client to completion. When the calling thread already runs an event loop, it does so in a worker thread. A topic that already exists is ignored. A topic error that says the broker is not the controller is retried once with a new client. Any other error that the broker reports for a topic is raised.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topics_config
|
List[Dict[str, Any]]
|
A list of topic configurations.
Each dict should contain 'name', and optionally 'partitions',
'replication' and 'topic_configs', a mapping of topic setting to
value such as |
required |
Raises:
| Type | Description |
|---|---|
TypeError
|
If a setting passed to the connector is not one of
|
KafkaError
|
If the broker reports an error for a topic, such as an
invalid replication factor. The error is the |
Source code in src/desflow/connectors/admin/kafka.py
send_config(topic, path, value)
async
Sends a surgical parameter update to a simulation config topic.
This method acts as an external controller, allowing users to
dynamically update registry paths (e.g., arrival rates or resource
capacities) while the simulation is running. The message is the JSON
object {"path": path, "value": value}, the format KafkaIngress reads.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic the simulation is listening to for config. |
required |
path
|
str
|
The registry dot-notation path (e.g., 'Shop.resources.packers.max_cap'). |
required |
value
|
Any
|
The new value to be applied to the path. |
required |
Source code in src/desflow/connectors/admin/kafka.py
collect_data(topics, auto_offset_reset='latest')
async
Async loop to consume from telemetry and event topics.
This loop continuously listens to the simulation's egress and updates
the connector's internal _vitals and _state attributes. A message whose
value is null, JSON null or not JSON changes neither and is passed over,
because KafkaEgress writes a null value for compacted topics.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topics
|
List[str]
|
List of topics to consume from (telemetry and events). |
required |
auto_offset_reset
|
str
|
Where to start consuming if no offset is committed. |
'latest'
|
Source code in src/desflow/connectors/admin/kafka.py
get_state()
Returns the aggregated event state for task lifecycles.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
A nested dictionary: sim_id -> service -> task_id -> {status: timestamp}. |
get_vitals()
Returns the latest system telemetry metrics.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
A dictionary of path_id to latest value (e.g., utilization, queue length). |
Connectors (Ingress)
desflow.connectors.ingress.base.BaseIngress
Base class for all ingress providers in the simulation.
Ingress providers act as asynchronous listeners that bridge external data sources (such as Kafka topics, Redis channels, or local schedules) with the simulation's internal state. They are responsible for fetching updates and placing them into a thread-safe queue for the registry to process.
Methods:
run(ingress_queue)
async
Listens to an external source and pushes updates to the ingress queue.
This method should be implemented as an asynchronous loop that waits for external signals or data and converts them into (path, value) tuples suitable for the SimulationRegistry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ingress_queue
|
Queue
|
A thread-safe queue used to transmit updates to the main simulation environment. |
required |
Raises:
| Type | Description |
|---|---|
NotImplementedError
|
If the subclass does not override this method. |
Source code in src/desflow/connectors/ingress/base.py
desflow.connectors.ingress.file.FileIngress(path, poll=1.0)
Bases: BaseIngress
Ingress provider that applies the entries of a local YAML file when you edit it.
The file is a mapping of registry paths to values, one path: value line each:
1 2 | |
The connector needs no service and no extra. It checks the file's modification
time every poll wall-clock seconds and reads the file only when the time has
changed. The file as it stands when the run starts counts as a first read.
An entry is applied when it first appears or when its value differs from the last valid read. An entry that is removed changes nothing. A file that does not exist yet, or is empty, is waited for without a log line. A file that is not valid YAML, or is not a mapping, is logged as one WARNING line and skipped, and the next valid read is compared with the last valid one. An entry with an empty path or a null value is logged as one WARNING line and skipped, as the other ingress connectors do.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
Path
|
The file to watch. |
poll |
float
|
Wall-clock seconds between checks of the file's modification time. |
Initializes the FileIngress.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Union[str, Path]
|
The YAML file to watch. A relative path is relative to the working directory of the run. |
required |
poll
|
float
|
Wall-clock seconds between checks of the file's modification time. The default is 1.0 second, so a saved edit lands within about one second; pass a smaller value for a faster reaction. |
1.0
|
Source code in src/desflow/connectors/ingress/file.py
Methods:
run(ingress_queue)
async
Watches the file and puts each new or changed entry on the queue.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ingress_queue
|
Queue
|
A thread-safe queue used to transmit the (path, value) updates to the SimulationRegistry. |
required |
Source code in src/desflow/connectors/ingress/file.py
desflow.connectors.ingress.kafka.KafkaIngress(topic, bootstrap_servers, topic_deserializers=None, default_deserializer=None, topic_policy=None, ssl=None, sasl_oauth_token_provider=None, **kwargs)
Bases: BaseIngress
Resilient Kafka consumer for dynamic simulation configuration updates.
This connector subscribes to a specified Kafka topic and decodes incoming
messages into state updates for the SimulationRegistry. It reads JSON by default.
default_deserializer or topic_deserializers sets another deserializer, such as
Avro, JSON Schema or Protobuf.
The expected resulting dictionary format must contain a 'path' string and a non-null 'value' of any serializable type. A message that cannot be decoded or does not have both is logged as a warning and skipped. A schema registry that refuses the deserializer's credentials or access ends the task.
Attributes:
| Name | Type | Description |
|---|---|---|
topic |
str
|
The Kafka topic name used for configuration signals. |
bootstrap_servers |
str
|
Comma-separated string of Kafka broker addresses. |
topic_deserializers |
Optional[Dict[str, MessageDeserializer]]
|
Mapping of topics to specific deserializers. |
default_deserializer |
Optional[MessageDeserializer]
|
Fallback deserializer. Defaults to |
kwargs |
dict
|
Additional configuration parameters for the AIOKafkaConsumer. |
Initializes the KafkaIngress with connection and subscription details.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic to subscribe to. |
required |
bootstrap_servers
|
str
|
Kafka broker address or list of addresses. |
required |
topic_deserializers
|
Optional[Dict[str, MessageDeserializer]]
|
Mapping of topics to their specific deserializers. |
None
|
default_deserializer
|
Optional[MessageDeserializer]
|
Fallback deserializer if the topic isn't explicitly mapped. |
None
|
topic_policy
|
Any
|
How the control topic is treated before the run: a mode word
( |
None
|
ssl
|
Optional[Dict[str, Any]]
|
The keys |
None
|
sasl_oauth_token_provider
|
Any
|
|
None
|
**kwargs
|
Any
|
Arbitrary keyword arguments passed to the underlying AIOKafkaConsumer. |
{}
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ImportError
|
If |
Source code in src/desflow/connectors/ingress/kafka.py
Methods:
run(ingress_queue)
async
Consumes parameter changes from Kafka and puts each on the ingress queue.
Each message is read as JSON with path and value; a message that cannot be
read is logged and skipped. A deserializer runs in a worker thread, because it
may call a schema registry, which waits and retries, and must not hold the loop
that every ingress connector of the run shares. A registry that fails for a time (a
5xx status, a throttled call, a timeout or a lost connection) keeps the message,
which is decoded again after a longer wait. A registry that refuses the
deserializer's credentials or access (a 401 or 403 status, or the Glue error
AccessDeniedException) would refuse every message, so it ends the task.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ingress_queue
|
Queue
|
The queue the environment applies changes from, as
|
required |
Raises:
| Type | Description |
|---|---|
SchemaRegistryError
|
If the registry refuses the deserializer's credentials or access. |
Source code in src/desflow/connectors/ingress/kafka.py
desflow.connectors.ingress.local.LocalIngress(schedule)
Bases: BaseIngress
Ingress provider for deterministic, time-scheduled parameter updates.
This connector is primarily used for testing, benchmarking, or local simulation runs where parameter changes need to occur at specific wall-clock intervals without requiring an external message broker like Kafka. It feeds a pre-defined sequence of updates into the simulation registry based on relative delays.
Attributes:
| Name | Type | Description |
|---|---|---|
schedule |
List[Tuple[float, str, Any]]
|
A chronologically sorted list of updates, where each tuple contains (delay_seconds, path, value). |
Initializes the LocalIngress and sorts the provided schedule.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
schedule
|
List[Tuple[float, str, Any]]
|
A list of tuples representing scheduled updates. Format: [(delay_from_start, "registry_path", new_value), ...]. The list is automatically sorted by the delay time. |
required |
Source code in src/desflow/connectors/ingress/local.py
Methods:
run(ingress_queue)
async
Processes the schedule and pushes updates to the queue at the correct wall-clock times.
This method calculates the relative sleep intervals between
scheduled events to ensure that updates are placed in the
ingress queue precisely when requested. Because it uses
asyncio.sleep, it does not block the main simulation execution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ingress_queue
|
Queue
|
A thread-safe queue used to transmit the (path, value) updates to the SimulationRegistry. |
required |
Source code in src/desflow/connectors/ingress/local.py
desflow.connectors.ingress.postgres.PostgresIngress(connection_dsn, table_name='simulation_params', poll_interval=2.0, **kwargs)
Bases: BaseIngress
Asynchronous ingress provider for polling updates from a PostgreSQL database.
This connector polls a specified configuration table for rows where 'is_applied' is FALSE. It parses the parameter path and value, forwards them to the simulation registry, and then updates the row to mark it as applied. A row with a null path or value is logged as a warning, skipped and marked as applied, so it is not read again.
Initializes the PostgresIngress.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
connection_dsn
|
str
|
PostgreSQL connection string (DSN). |
required |
table_name
|
str
|
Table to poll for parameter updates. Must have columns: id (serial), path (text), value (text/json), is_applied (boolean). |
'simulation_params'
|
poll_interval
|
float
|
Seconds to wait between database polls. |
2.0
|
**kwargs
|
Any
|
Additional connection pool arguments for asyncpg. |
{}
|
Source code in src/desflow/connectors/ingress/postgres.py
Methods:
run(ingress_queue)
async
Main execution loop fetching dynamic parameter updates from PostgreSQL.
If PostgreSQL does not answer at start, the ingress retries with a back-off
that grows to one minute, as the Kafka ingress does. An error that no wait
fixes, such as a table_name the server refuses, or a role that may read the table
but not update it, ends the task at once. The rows of a poll are marked applied
before their changes are sent, so an update that fails sends nothing. When the
connection fails during that update, the next poll asks which rows the server
marked and sends their changes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ingress_queue
|
Queue
|
A thread-safe queue used to transmit (path, value) tuples to the SimulationRegistry. |
required |
Raises:
| Type | Description |
|---|---|
PostgresError
|
If the server refuses a setting, such as the
password, |
Source code in src/desflow/connectors/ingress/postgres.py
desflow.connectors.ingress.redis.RedisIngress(url, channel_name='params', poll_interval=0.5, **kwargs)
Bases: BaseIngress
Asynchronous ingress provider for dynamic parameter updates via Redis Pub/Sub.
This connector utilizes redis.asyncio to listen on a specified Redis channel
for parameter updates, parsing them from JSON, and pushing them to the simulation's
internal update queue. Each message is a JSON object with path and value,
such as {"path": "Shop.arrival.orders.rate", "value": 10.0}; a message that
cannot be parsed or does not have both is logged as a warning and skipped.
Attributes:
| Name | Type | Description |
|---|---|---|
url |
str
|
The connection URL for the Redis server. |
channel_name |
str
|
The Redis Pub/Sub channel to subscribe to. |
poll_interval |
float
|
Sleep interval when no messages are found. |
client |
Redis | None
|
The underlying redis-py asynchronous client instance. |
pubsub |
PubSub | None
|
The active Pub/Sub subscription instance. |
**kwargs |
PubSub | None
|
Additional configuration dictionary passed to |
Initializes the RedisIngress with connection and channel settings.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url
|
str
|
Redis connection URL (e.g., redis://127.0.0.1:6379/0). |
required |
channel_name
|
str
|
Redis Pub/Sub channel to subscribe to. |
'params'
|
poll_interval
|
float
|
Sleep interval when no messages are found. |
0.5
|
**kwargs
|
Any
|
Additional configuration dictionary passed to |
{}
|
Source code in src/desflow/connectors/ingress/redis.py
Methods:
run(update_queue)
async
The main execution loop for subscribing to Redis and receiving updates.
Continuously polls the Pub/Sub channel for incoming JSON messages containing
path and value, placing successfully parsed updates onto the
internal update queue. When Redis is down, refuses the subscription or the
connection fails or stops answering, it logs the error and connects again
with a back-off that grows to one minute, as the Kafka ingress does. The
back-off starts again from the first wait once Redis confirms a subscription.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
update_queue
|
Queue
|
A thread-safe queue where incoming parameters are pushed so they can be applied within the SimPy context. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
TypeError
|
If redis-py refuses a keyword setting, such as a misspelt
|
Source code in src/desflow/connectors/ingress/redis.py
Deserializers
desflow.connectors.ingress.kafka.MessageDeserializer
Bases: Protocol
Protocol defining the interface for all Kafka ingress deserializers.
Any custom deserializer passed to KafkaIngress must implement the
deserialize method according to this signature.
Methods:
deserialize(topic, payload)
Deserializes a raw byte payload from Kafka into a Python object.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic from which the message was consumed. |
required |
payload
|
bytes
|
The raw binary payload of the Kafka message. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The deserialized data, typically a dictionary containing a 'path' and 'value'. |
Source code in src/desflow/connectors/ingress/kafka.py
desflow.connectors.ingress.kafka.JsonDeserializer
Default fallback deserializer providing backward compatibility via standard JSON.
This deserializer assumes the incoming payload is a UTF-8 encoded JSON string.
Methods:
deserialize(topic, payload)
Deserializes a JSON byte string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic from which the message was consumed (unused). |
required |
payload
|
bytes
|
The UTF-8 encoded JSON byte string. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The parsed JSON data as a Python dictionary. |
Source code in src/desflow/connectors/ingress/kafka.py
desflow.connectors.ingress.kafka.ConfluentAvroDeserializer(registry_url, *, auth=UNSET, verify=UNSET, cert=UNSET, timeout=UNSET, headers=UNSET, proxy=UNSET, max_retries=UNSET)
Lazy-loaded deserializer for Confluent Schema Registry using fastavro and httpx.
This class converts Avro-encoded byte strings back into Python dictionaries.
It fetches the writer schema from the registry using the schema id (or GUID)
in the message's prefix, once for each id, so no schema_str is required
during initialization.
Initializes the Confluent Avro deserializer. It makes no request to the registry.
UNSET as a default, in the signature or the Default column of the API page,
means the argument was not given; each argument's description says its real
default.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_url
|
str
|
The base URL of the Confluent Schema Registry. Several URLs separated by commas are tried in order. |
required |
auth
|
Optional[Tuple[str, str]]
|
A username and password for basic
authentication, for example |
UNSET
|
verify
|
bool | str | SSLContext
|
How to check the registry's certificate: True (the default), False, the path of a CA file or directory, or an SSL context. |
UNSET
|
cert
|
str | tuple | None
|
A client certificate file, or a tuple of the certificate file, the key file and the key password. The default is None. |
UNSET
|
timeout
|
Optional[float]
|
Seconds each step of a request to the registry (connecting, sending, reading) may take. The default is 10.0. None waits without a limit. |
UNSET
|
headers
|
Optional[Mapping[str, str]]
|
Headers sent with every request,
for example |
UNSET
|
proxy
|
Optional[str]
|
The URL of a proxy. The default is None. |
UNSET
|
max_retries
|
int
|
How many times a request is repeated after a transport error or a status of 408, 429, 500, 502, 503 or 504. The default is 3. |
UNSET
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'fastavro' or 'httpx' package is not installed. |
Source code in src/desflow/connectors/ingress/kafka.py
91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 | |
Methods:
deserialize(topic, payload)
Deserializes an Avro-encoded byte string.
The first message with a schema id makes a request to the registry for the writer schema.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic from which the message was consumed. |
required |
payload
|
bytes
|
The Avro-encoded binary payload containing the schema ID. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The deserialized data as a Python dictionary. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the payload has no valid prefix, the writer schema is not Avro, or the schema has Data Contract read rules. |
SchemaRegistryError
|
If the registry has no schema for the id. |
TransportError
|
If no registry URL could be reached. |
Source code in src/desflow/connectors/ingress/kafka.py
desflow.connectors.ingress.kafka.GlueAvroDeserializer(registry_name, **boto3_kwargs)
Lazy-loaded deserializer for AWS Glue Schema Registry.
This class converts Avro-encoded byte strings back into Python dictionaries, integrating directly with AWS Glue Schema Registry. It resolves schemas dynamically based on the metadata embedded in the AWS Glue message payload. A schema version of the JSON data format is read as JSON and validated against its schema.
Initializes the AWS Glue Avro deserializer.
The constructor makes no call to Glue. The first message with a schema version id fetches that version.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_name
|
str
|
The name of the AWS Glue Schema Registry. |
required |
**boto3_kwargs
|
Any
|
Additional arguments passed directly to the |
{}
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'boto3', 'fastavro' or 'fastjsonschema' package is not installed. |
Source code in src/desflow/connectors/ingress/kafka.py
Methods:
deserialize(topic, payload)
Deserializes an Avro or JSON byte string using AWS Glue.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic from which the message was consumed. |
required |
payload
|
bytes
|
The binary payload with the Glue prefix. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The record, a dict for a record or object schema. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the payload has no valid Glue prefix, the schema version is of a data format other than AVRO or JSON, the JSON Schema cannot be compiled, or the data is not JSON, does not match its schema or cannot be checked against it. |
SchemaRegistryError
|
If Glue has no available schema version for the id. |
Source code in src/desflow/connectors/ingress/kafka.py
desflow.connectors.ingress.kafka.ConfluentJsonSchemaDeserializer(registry_url, *, auth=UNSET, verify=UNSET, cert=UNSET, timeout=UNSET, headers=UNSET, proxy=UNSET, max_retries=UNSET)
Lazy-loaded deserializer for Confluent Schema Registry using jsonschema and httpx.
This class converts JSON in the Confluent wire format back into Python dictionaries. It fetches the writer schema from the registry using the schema id (or GUID) in the message's prefix, once for each id, and checks each record against it.
Initializes the Confluent JSON Schema deserializer. It makes no request to the registry.
UNSET as a default, in the signature or the Default column of the API page,
means the argument was not given; each argument's description says its real
default.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_url
|
str
|
The base URL of the Confluent Schema Registry. Several URLs separated by commas are tried in order. |
required |
auth
|
Optional[Tuple[str, str]]
|
A username and password for basic
authentication, for example |
UNSET
|
verify
|
bool | str | SSLContext
|
How to check the registry's certificate: True (the default), False, the path of a CA file or directory, or an SSL context. |
UNSET
|
cert
|
str | tuple | None
|
A client certificate file, or a tuple of the certificate file, the key file and the key password. The default is None. |
UNSET
|
timeout
|
Optional[float]
|
Seconds each step of a request to the registry (connecting, sending, reading) may take. The default is 10.0. None waits without a limit. |
UNSET
|
headers
|
Optional[Mapping[str, str]]
|
Headers sent with every request,
for example |
UNSET
|
proxy
|
Optional[str]
|
The URL of a proxy. The default is None. |
UNSET
|
max_retries
|
int
|
How many times a request is repeated after a transport error or a status of 408, 429, 500, 502, 503 or 504. The default is 3. |
UNSET
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'jsonschema' or 'httpx' package is not installed. |
Source code in src/desflow/connectors/ingress/kafka.py
330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 | |
Methods:
deserialize(topic, payload)
Deserializes a JSON byte string with the Confluent prefix.
The first message with a schema id makes a request to the registry for the writer schema.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic from which the message was consumed. |
required |
payload
|
bytes
|
The JSON payload with the Confluent prefix. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The record, a dict for an object schema. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the payload has no valid prefix, the writer schema is not JSON Schema or has Data Contract read rules, or the data is not JSON or does not match the schema. |
SchemaRegistryError
|
If the registry has no schema for the id. |
TransportError
|
If no registry URL could be reached. |
Source code in src/desflow/connectors/ingress/kafka.py
desflow.connectors.ingress.kafka.ConfluentProtobufDeserializer(registry_url, message_type, *, auth=UNSET, verify=UNSET, cert=UNSET, timeout=UNSET, headers=UNSET, proxy=UNSET, max_retries=UNSET)
Lazy-loaded deserializer for Confluent Schema Registry using protobuf and httpx.
This class converts Protobuf messages in the Confluent wire format back into Python dictionaries. It reads each message as the class it is given, and skips the message index. It fetches the writer schema only to check its type and rules, once for each id.
Initializes the Confluent Protobuf deserializer. It makes no request to the registry.
UNSET as a default, in the signature or the Default column of the API page,
means the argument was not given; each argument's description says its real
default.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_url
|
str
|
The base URL of the Confluent Schema Registry. Several URLs separated by commas are tried in order. |
required |
message_type
|
type[Message]
|
The generated Protobuf message class that every message is read as. |
required |
auth
|
Optional[Tuple[str, str]]
|
A username and password for basic
authentication, for example |
UNSET
|
verify
|
bool | str | SSLContext
|
How to check the registry's certificate: True (the default), False, the path of a CA file or directory, or an SSL context. |
UNSET
|
cert
|
str | tuple | None
|
A client certificate file, or a tuple of the certificate file, the key file and the key password. The default is None. |
UNSET
|
timeout
|
Optional[float]
|
Seconds each step of a request to the registry (connecting, sending, reading) may take. The default is 10.0. None waits without a limit. |
UNSET
|
headers
|
Optional[Mapping[str, str]]
|
Headers sent with every request,
for example |
UNSET
|
proxy
|
Optional[str]
|
The URL of a proxy. The default is None. |
UNSET
|
max_retries
|
int
|
How many times a request is repeated after a transport error or a status of 408, 429, 500, 502, 503 or 504. The default is 3. |
UNSET
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'protobuf' or 'httpx' package is not installed. |
Source code in src/desflow/connectors/ingress/kafka.py
471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 | |
Methods:
deserialize(topic, payload)
Deserializes a Protobuf byte string with the Confluent prefix.
The first message with a schema id makes a request to the registry for the writer schema.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic from which the message was consumed. |
required |
payload
|
bytes
|
The Protobuf payload with the Confluent prefix and the message index. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The record as a dict, with the field names of the schema. A 64 bit integer field is a string. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the payload has no valid prefix or index, the writer schema is not Protobuf or has Data Contract read rules, or the data is not a message of the class. |
SchemaRegistryError
|
If the registry has no schema for the id. |
TransportError
|
If no registry URL could be reached. |
Source code in src/desflow/connectors/ingress/kafka.py
desflow.connectors.ingress.kafka.GlueJsonSchemaDeserializer(registry_name, **boto3_kwargs)
Lazy-loaded deserializer for AWS Glue Schema Registry using boto3 and fastjsonschema.
This class converts JSON with the AWS Glue prefix back into Python dictionaries.
It fetches the schema version named in the prefix, once for each id, and checks
each record against it as aws-glue-schema-registry 1.1.3 did, which fills in a
missing property that has a default.
Initializes the AWS Glue JSON Schema deserializer.
The constructor makes no call to Glue. The first message with a schema version id fetches that version.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_name
|
str
|
The name of the AWS Glue Schema Registry. |
required |
**boto3_kwargs
|
Any
|
Additional arguments passed directly to the |
{}
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'boto3' or 'fastjsonschema' package is not installed. |
Source code in src/desflow/connectors/ingress/kafka.py
Methods:
deserialize(topic, payload)
Deserializes a JSON byte string using AWS Glue.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic from which the message was consumed. |
required |
payload
|
bytes
|
The JSON payload with the Glue prefix. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The record, a dict for an object schema. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the payload has no valid Glue prefix, the schema version is of a data format other than JSON, the JSON Schema cannot be compiled, or the data is not JSON, does not match the schema or cannot be checked against it. |
SchemaRegistryError
|
If Glue has no available schema version for the id. |
Source code in src/desflow/connectors/ingress/kafka.py
desflow.connectors.ingress.kafka.GlueProtobufDeserializer(registry_name, message_type, **boto3_kwargs)
Lazy-loaded deserializer for AWS Glue Schema Registry using boto3 and protobuf.
This class converts Protobuf messages with the AWS Glue prefix back into Python dictionaries. It reads each message as the class it is given, and skips the message index. It fetches the schema version only to check its data format, once for each id.
Initializes the AWS Glue Protobuf deserializer.
The constructor makes no call to Glue. The first message with a schema version id fetches that version.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_name
|
str
|
The name of the AWS Glue Schema Registry. |
required |
message_type
|
type[Message]
|
The generated Protobuf message class that every message is read as. |
required |
**boto3_kwargs
|
Any
|
Additional arguments passed directly to the |
{}
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'boto3' or 'protobuf' package is not installed. |
Source code in src/desflow/connectors/ingress/kafka.py
Methods:
deserialize(topic, payload)
Deserializes a Protobuf byte string using AWS Glue.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The Kafka topic from which the message was consumed. |
required |
payload
|
bytes
|
The Protobuf payload with the Glue prefix and the message index. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The record as a dict, with the field names of the schema. A 64 bit integer field is a string. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the payload has no valid Glue prefix or index, the schema version is of a data format other than PROTOBUF, or the data is not a message of the class. |
SchemaRegistryError
|
If Glue has no available schema version for the id. |
Source code in src/desflow/connectors/ingress/kafka.py
Connectors (Egress)
desflow.connectors.egress.base.BaseEgress
Base class for all egress providers in the simulation.
Egress providers act as asynchronous bridges that consume processed simulation data (telemetry and events) from a thread-safe internal queue and transmit it to external destinations such as Kafka, databases, or the console.
Attributes:
| Name | Type | Description |
|---|---|---|
envelope |
bool
|
False when the provider was added with |
Methods:
run(egress_queue)
async
Listens to the internal queue and pushes data to an external sink.
This method should contain an asynchronous loop that polls the provided queue and handles the networking/I/O logic specific to the destination system.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
egress_queue
|
Queue
|
A thread-safe queue containing batches of dictionaries to be exported. |
required |
Raises:
| Type | Description |
|---|---|
NotImplementedError
|
If the subclass does not override this method. |
Source code in src/desflow/connectors/egress/base.py
desflow.connectors.egress.kafka.KafkaEgress(bootstrap_servers, telemetry_topic='sim-telemetry', event_topic='sim-events', topic_router=None, topic_serializers=None, default_serializer=None, include_stream_type=False, topic_policy=None, topics=None, serializer=None, ssl=None, sasl_oauth_token_provider=None, **kwargs)
Bases: BaseEgress
High-throughput Kafka producer for simulation telemetry and events.
This connector utilizes aiokafka for asynchronous I/O and orjson for fast
serialization. It implements a resilient connection loop with exponential
backoff.
By default, data is routed to the telemetry_topic or event_topic based on
its stream_type. If a topic_router callable is provided, topic selection
is delegated to that function instead, allowing for advanced multiplexing
(e.g., splitting ML vs. UI events).
Without a topic_router, the event and telemetry topics are created at start
when they do not exist, with one partition and a replication factor of one.
With a router the topics are left to the caller, because only the router knows
which ones it uses.
Note
By default stream_type is treated as routing metadata and is left out of
the published message, because the topic already identifies the stream.
ParquetStorageEgress keeps stream_type as a column, so a downstream
filter written against Parquet files matches nothing when it is pointed at
a Kafka topic. Set include_stream_type=True to keep the field in the
message and make both paths carry it.
An event whose value holds __topic__ is sent to the topic it names, and the key
is removed from the message. This replaces a topic_router for the common case.
A topic_router wins over __topic__, and telemetry always goes to
telemetry_topic. topic_policy and topics create the topics, serializer
and topics.<topic>.serializer set the serializers from a mapping, ssl and
sasl_oauth_token_provider build the connection settings from a mapping, and
an entry added with envelope=False sends the event's value alone as the
message. An event whose value is null is then sent as a tombstone: a message
with the record's key and a null value, which a compacted topic reads as a
delete of that key.
Attributes:
| Name | Type | Description |
|---|---|---|
bootstrap_servers |
str
|
Comma-separated list of Kafka brokers. |
telemetry_topic |
str
|
The default topic name for telemetry data. |
event_topic |
str
|
The default topic name for lifecycle events. |
topic_router |
Callable
|
Optional external logic to determine the topic. |
topic_serializers |
Optional[Dict[str, MessageSerializer]]
|
Optional mapping of topics to specific serializers. |
default_serializer |
Optional[MessageSerializer]
|
Fallback serializer if a topic is not in |
include_stream_type |
bool
|
Whether the published message keeps the |
producer_config |
dict
|
Configuration dictionary passed to AIOKafkaProducer. |
Examples:
Defining a custom topic router to split machine learning events from standard telemetry:
Initializes the KafkaEgress with topic and connection settings.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bootstrap_servers
|
str
|
Comma-separated list of Kafka brokers. |
required |
telemetry_topic
|
str
|
Destination topic for telemetry stream. |
'sim-telemetry'
|
event_topic
|
str
|
Destination topic for event stream. |
'sim-events'
|
topic_router
|
Optional[Callable[[dict], str]]
|
Optional callable to dynamically route payloads to specific topics. |
None
|
topic_serializers
|
Optional[Dict[str, MessageSerializer]]
|
Mapping of target topics to their specific |
None
|
default_serializer
|
Optional[MessageSerializer]
|
The fallback serializer to use if a topic lacks a specific mapping. Defaults to |
None
|
include_stream_type
|
bool
|
Keep |
False
|
topic_policy
|
Any
|
How the sink treats its topics when the run starts: a mode word
( |
None
|
topics
|
Optional[Mapping[str, Mapping[str, Any]]]
|
Settings per topic, a mapping of topic name to a mapping with the
keys of |
None
|
serializer
|
Any
|
The serializer for every topic that |
None
|
ssl
|
Optional[Mapping[str, Any]]
|
The keys |
None
|
sasl_oauth_token_provider
|
Any
|
|
None
|
**kwargs
|
Any
|
Additional overrides for the AIOKafkaProducer configuration. |
{}
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If a policy, serializer, |
ImportError
|
If |
Source code in src/desflow/connectors/egress/kafka.py
946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 | |
Methods:
run(egress_queue)
async
The main execution loop that consumes the egress queue and publishes to Kafka.
This method maintains a persistent connection to Kafka. If the connection
is lost, it implements an exponential backoff retry strategy. It polls
the internal egress_queue for batches of data, determines the target
topic based on the 'stream_type', and performs asynchronous sends using
the configured serializers, which run in a worker thread. A record that its
serializer refuses, with a SerializationError, a value error or a
SchemaRegistryError with a 4xx status other than a throttled call, is
skipped with a warning that names its key and the reason, and the rest of
the batch is sent. A record that Kafka refuses (larger than
max_request_size, or for a topic Kafka does not have or refuses) is
skipped the same way, and a topic_router that returns None drops its
record. Any other error of a send or a delivery, such as a broker that
restarts, restarts the producer, and the records of the batch the broker has
not confirmed are sent again first; a record that was on its way at the error
can arrive twice. A registry that fails for a time (a 5xx status, a
throttled call, a timeout or a lost connection) keeps the batch, which is
tried again after a longer wait. A registry that refuses the sink's
credentials or access (a 401 or 403 status, or the Glue error
AccessDeniedException) ends the task with that error, and nothing of that
batch is sent.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
egress_queue
|
Queue
|
A thread-safe queue containing lists of dictionaries or Pydantic models generated by the EgressMixIn. |
required |
Raises:
| Type | Description |
|---|---|
SchemaRegistryError
|
If the registry refuses the sink's credentials or access. |
Note
The loop exits gracefully upon receiving an asyncio.CancelledError,
ensuring the Kafka producer is stopped correctly.
Source code in src/desflow/connectors/egress/kafka.py
1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 | |
desflow.connectors.egress.local.ConsoleEgress(format='log')
Bases: BaseEgress
Local egress provider that outputs simulation data to the console.
This connector serves as a debugging and local development tool. The log
format logs each batch, distinguishing telemetry metrics from discrete events
by prefixes. The json format writes each record to stdout as one JSON line,
with its stream_type, for a pipe such as jq.
Chooses how records are written.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
format
|
Literal['log', 'json']
|
|
'log'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/connectors/egress/local.py
Methods:
run(egress_queue)
async
Continuously polls the egress queue and writes each batch.
This method implements an asynchronous polling loop. It extracts batches
from the internal queue. With the json format it writes each batch to
stdout in one write, one JSON line per record as published. With the
log format it writes each batch with one log call at INFO
level, one line per record, each line starting with a prefix:
- [TEL]: Telemetry data (e.g., resource utilization, queue lengths).
- [EVT]: Event data (e.g., task lifecycle transitions).
- [LBL]: Ground-truth records, on a sink added with when: labels.
One call per batch rather than per record costs far less (issue #24), so the logging format's prefix, such as the time and level, appears on the first line of each batch only.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
egress_queue
|
Queue
|
A thread-safe queue containing lists of dictionaries (batches) generated by the EgressMixIn. |
required |
Note
If the queue is empty, the method yields control to the asyncio event loop for a short duration to prevent CPU pinning.
Source code in src/desflow/connectors/egress/local.py
desflow.connectors.egress.postgres.PostgresEgress(connection_dsn, table_name=None, upsert_keys=None, tables=None, table_policy='create', **kwargs)
Bases: BaseEgress
Asynchronous egress provider for persisting simulation data to PostgreSQL.
This connector handles long-term storage of simulation results or continuous generation of relational CDC data, performing bulk inserts of batched data into a specified PostgreSQL table using asyncpg.
By default a record whose key already exists is skipped (ON CONFLICT DO
NOTHING). Name the key columns in upsert_keys to update the existing row
instead, which a simulation needs when it changes rows it wrote earlier, such
as an order moving from processing to shipped.
A payload field __table__ names the table of that event, and is removed from the
row. An entry writes the events of its own table_name and ignores the others, so
one entry per table, all fed by the same events, splits a stream into tables. Names
are compared as PostgreSQL reads them: Orders and public.orders are the table
orders. A sink warns once about a __table__ that no PostgresEgress of the
process writes, which is usually a misspelt name. An
entry with upsert_keys is keyed: it skips an event with a null or missing key column,
and logs one warning at the end of each batch with the count. Before 1.0.0 such an
event stopped the batch with an error.
An ISO time string bound for a timestamp or date column is converted to a
datetime or date first, because asyncpg rejects the string itself.
Tables named in tables are created at start when they do not exist, so a run
needs no separate schema step.
An event's row also carries the record's key as event_id, and its task and
instance, unless the payload has fields of those names. A ground-truth record,
sent to a sink added with when: labels, is written as sim_ts, timestamp,
path, old, new and label, the last three as JSON text.
Examples:
Updating orders as their status changes:
Creating the table at start, in the shape a YAML file gives:
Initializes the PostgresEgress with connection details.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
connection_dsn
|
str
|
PostgreSQL connection string (DSN). |
required |
table_name
|
str | None
|
Target table for simulation records. Defaults to the one
table in |
None
|
upsert_keys
|
list[str] | str | None
|
Optional key columns, a list or one column name. When given, a record whose key already exists updates that row's other columns, and the columns need a unique index or constraint. When None, such a record is skipped. |
None
|
tables
|
dict[str, Any] | None
|
Optional tables to create at start when they do not exist. Each
table name maps to |
None
|
table_policy
|
str
|
What the sink does about the tables in |
'create'
|
**kwargs
|
Any
|
Additional connection pool arguments for asyncpg. |
{}
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If a |
Source code in src/desflow/connectors/egress/postgres.py
Methods:
run(egress_queue)
async
Main execution loop for PostgreSQL data persistence.
If PostgreSQL does not answer at start, the sink retries with a back-off that grows to one minute, as the Kafka sinks do. An error that no wait fixes, such as a table definition the server refuses, ends the task at once.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
egress_queue
|
Queue
|
A thread-safe queue containing batches of simulation data. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a column in |
PostgresError
|
If the server refuses a setting, such as the password or a table definition. |
Source code in src/desflow/connectors/egress/postgres.py
desflow.connectors.egress.redis.RedisEgress(url, stream_name='simulation_data', **kwargs)
Bases: BaseEgress
High-throughput asynchronous egress provider for streaming simulation data to Redis.
This connector utilizes redis.asyncio for asynchronous I/O and publishes batched
simulation results (events and telemetry) to Redis Streams (XADD) for real-time consumers.
By default, it uses a unified stream, but supports dynamic stream multiplexing
by reading the __stream__ key from the data payloads.
Attributes:
| Name | Type | Description |
|---|---|---|
url |
str
|
The connection URL for the Redis server (e.g., redis://127.0.0.1:6379/0). |
stream_name |
str
|
The default target Redis Stream name. |
client |
Redis | None
|
The underlying redis-py asynchronous client instance. |
**kwargs |
Redis | None
|
Additional configuration dictionary passed to |
Initializes the RedisEgress with topic and connection settings.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
url
|
str
|
Redis connection URL (e.g., redis://127.0.0.1:6379/0). |
required |
stream_name
|
str
|
Target Redis Stream name. |
'simulation_data'
|
**kwargs
|
Any
|
Additional arguments for |
{}
|
Source code in src/desflow/connectors/egress/redis.py
Methods:
run(egress_queue)
async
The main execution loop that consumes the egress queue and publishes to Redis.
This method maintains a connection to Redis and uses pipeline (XADD)
commands to insert batches of data efficiently. A __stream__ that is not
text is warned about and ignored. A TypeError from redis-py, which refuses
a keyword setting such as a misspelt socket_timeout when it makes the
connection, ends the task, because no wait fixes it. Any other error is
logged and the sink goes on with the next batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
egress_queue
|
Queue
|
A thread-safe queue containing lists of dictionaries or Pydantic models generated by the EgressMixIn. |
required |
Source code in src/desflow/connectors/egress/redis.py
desflow.connectors.egress.storage.JsonlStorageEgress(default_path=None, filesystem=None, path_router=None)
Bases: BaseEgress
High-throughput JSONL batch writer for local file systems or object storage.
This connector uses PyArrow's Virtual File System (VFS) to write data to local disks, S3-compatible storage such as AWS S3 or SeaweedFS, Google Cloud Storage or Azure Blob Storage. It writes a uniquely named file for every batch, so no two writes go to the same file and a file, once written, is never changed.
By default, events are appended to the default_path as flat rows: each
event's value mapping is unpacked into top-level fields, and telemetry is left
out. If a path_router callable is provided, destination logic is delegated to
that function, allowing for advanced multiplexing (e.g., splitting logs vs.
errors) or dynamically dropping specific records by returning None. Records
are then written as the router leaves them.
An event whose value holds __path__, a file path such as packed/orders.jsonl,
is written to that path, and the key is removed from the row. This replaces a
path_router for the common case. A path_router wins over __path__, and a
record with no __path__ goes to default_path. The folder of each path is
created. With envelope=False on the entry, the row is the event's value
alone.
Attributes:
| Name | Type | Description |
|---|---|---|
default_path |
Optional[str]
|
The fallback destination path if no router is provided. |
filesystem |
Optional[Any]
|
A PyArrow |
path_router |
Optional[Callable]
|
Optional logic to dynamically route or drop payloads. |
Examples:
Routing errors to a specific file and dropping meaningless metrics:
Initializes the JsonlStorageEgress with routing and VFS settings.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
default_path
|
Optional[str]
|
The target file path prefix (e.g., "data/logs.jsonl"),
used when no router is given. Only events are written there, with
their |
None
|
filesystem
|
Optional[Any]
|
An instantiated PyArrow FileSystem (e.g., |
None
|
path_router
|
Optional[Callable[[dict], Optional[str]]]
|
A function taking a dict payload and returning a string path, or None to drop the record. |
None
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If pyarrow is not installed (the |
Source code in src/desflow/connectors/egress/storage.py
Methods:
run(egress_queue)
async
The main execution loop that consumes the egress queue and writes JSONL chunks.
This loop continuously polls the internal egress_queue for data batches.
When a batch is received, it offloads the synchronous PyArrow file I/O operations
to a background thread to prevent blocking the asyncio event loop.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
egress_queue
|
Queue
|
A thread-safe queue containing batches of dictionaries generated by the environment's EgressMixIn. |
required |
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'pyarrow' package is not installed. |
Note
The loop exits gracefully upon receiving an asyncio.CancelledError,
ensuring background file writers have completed before shutting down.
Source code in src/desflow/connectors/egress/storage.py
desflow.connectors.egress.storage.ParquetStorageEgress(default_path=None, filesystem=None, path_router=None)
Bases: BaseEgress
Columnar Parquet writer for local disks or object storage.
This connector buffers simulation records and converts them into heavily
compressed Parquet tables using pyarrow. It natively supports writing to
local disks, S3-compatible object stores (AWS S3, SeaweedFS), Google Cloud Storage
or Azure Blob Storage via PyArrow's VFS.
To support massive parallel processing (e.g., Athena, Databricks), it implements file rotation (chunking), creating a uniquely named Parquet part-file for every flush of the buffer. A file's columns come from every row of its batch; a column a later batch adds widens the path's schema, and each batch is cast to it, so a column keeps its type across chunks.
Without a path_router, events are written to default_path as flat rows:
each event's value mapping becomes columns, and telemetry is left out. With a
router, records are written as the router leaves them.
An event whose value holds __path__, a file path such as orders/orders.parquet,
is written to that path, and the key is removed from the row. This replaces a
path_router for the common case. A path_router wins over __path__, and a
record with no __path__ goes to default_path. The folder of each path is
created. With envelope=False on the entry, the row is the event's value
alone.
Attributes:
| Name | Type | Description |
|---|---|---|
default_path |
Optional[str]
|
The fallback destination path. |
filesystem |
Optional[Any]
|
A PyArrow |
path_router |
Optional[Callable]
|
Optional logic to dynamically route or drop payloads. |
schemas |
Dict[str, Any]
|
Internal registry of inferred PyArrow schemas per file path. |
Examples:
Splitting simulation events and telemetry into distinct Parquet datasets directly to S3:
Initializes the ParquetStorageEgress with routing and VFS settings.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
default_path
|
Optional[str]
|
The target file path prefix (e.g., "data/events.parquet"),
used when no router is given. Only events are written there, with
their |
None
|
filesystem
|
Optional[Any]
|
An instantiated PyArrow FileSystem (e.g., |
None
|
path_router
|
Optional[Callable[[dict], Optional[str]]]
|
A function taking a dict payload and returning a string path, or None to drop the record. |
None
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If pyarrow is not installed (the |
Source code in src/desflow/connectors/egress/storage.py
Methods:
run(egress_queue)
async
The main execution loop that consumes the egress queue and writes Parquet chunks.
This loop continuously polls the internal egress_queue for data batches.
When a batch is received, it offloads the synchronous PyArrow table conversion
and file I/O operations to a background thread to prevent blocking the asyncio loop.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
egress_queue
|
Queue
|
A thread-safe queue containing batches of dictionaries generated by the environment's EgressMixIn. |
required |
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'pyarrow' package is not installed. |
Note
The loop exits gracefully upon receiving an asyncio.CancelledError,
ensuring the final Parquet table has been fully flushed to disk.
Source code in src/desflow/connectors/egress/storage.py
desflow.connectors.egress.iceberg.IcebergStorageEgress(catalog, default_table=None, table_router=None, schemas=None, locations=None, upsert_keys=None, table_policy='create')
Bases: BaseEgress
Apache Iceberg table writer for lakehouse ingestion patterns.
ParquetStorageEgress writes files and leaves registration to the caller. This
connector appends straight into an Iceberg table instead, so one run can feed a
stream sink and a queryable table in the same pass. Each flush of a provider's
buffer becomes exactly one Iceberg commit, which is why the buffer size matters:
every commit writes a manifest, a manifest list and a new metadata.json, and
query planning degrades as the snapshot count grows. Give this provider a large
batch_size on add_egress so the run produces a handful of snapshots rather
than hundreds.
Without a table_router, events are written to default_table as flat rows:
each event's value mapping becomes columns, and telemetry is left out. With a
router, records are written as the router leaves them.
By default every record is appended. Name a table's key columns in
upsert_keys to write it with PyIceberg's upsert instead: a record whose key
exists updates that row, and a new key is inserted, so rerunning a seeded
simulation over the same window does not duplicate rows. An upsert reads the
matching rows before writing, so it is slower than an append, and it never
deletes rows. It replaces the whole row, so every record must carry every column: a
missing column is written as null. Within one flush, the last record for a key wins. An upsert
flush is still one commit, but it can add up to three snapshots: an overwrite
and two appends.
An event whose value holds __table__, a namespace.table identifier, is written
to that table, and the key is removed from the row. This replaces a
table_router for the common case. A table_router wins over __table__, and a
record with no __table__ goes to default_table. An entry with upsert_keys is
keyed. It skips a record that has no destination, and a record with a null or
missing key column of its table, and it logs one warning at the end of each flush
with the counts skipped. Before 1.0.0 such a record stopped the batch with an
error.
The catalog is supplied by the caller, either as a built Catalog or as a
mapping of PyIceberg catalog properties that is passed to load_catalog. The
supported catalogs, which the iceberg extra installs, each with the test that
writes through it, are:
- Iceberg REST, in the integration tests against the odctl
catalogprofile; - SQL on SQLite, in the unit tests;
- SQL on PostgreSQL, in the integration tests against the odctl
postgresprofile; - Glue, in the integration tests against Floci, an AWS emulator.
A table without a schema in schemas is created with the columns of every row of
its first batch, and a column a later batch adds is added to the table, as the
Parquet writer widens its schema, except on a table with upsert_keys. Inference reads a whole number as a long, so a
later fraction in that column is refused, and an ISO timestamp as a string. The
fields that as_doubles reads as doubles, such as a telemetry value, are the
exception. A table whose column takes fractions, or whose consumer expects a real
timestamp, needs an explicit schema. A table with upsert_keys cannot gain a
column, because PyIceberg's upsert compares the batch with the matching rows of
the old data files, which lack it. When the table is not named in schemas, a
batch that brings a column the table lacks is refused. A table named in schemas
writes only the columns listed there and leaves any other field out, so name every
column of an upsert table in schemas. An
ISO string bound for a timestamp or date column of the table is converted to a
datetime or date before the write, because PyArrow rejects the string itself.
Attributes:
| Name | Type | Description |
|---|---|---|
catalog |
Any
|
An instantiated pyiceberg |
catalog_properties |
Optional[Dict[str, Any]]
|
The PyIceberg catalog
properties given instead of a |
default_table |
Optional[str]
|
Fallback |
table_router |
Optional[Callable]
|
Logic returning |
schemas |
Dict[str, Any]
|
PyArrow schema per table identifier. Seeded with any explicit schemas, then filled in from the tables that get created. |
locations |
Dict[str, str]
|
Explicit storage location per table identifier. |
upsert_keys |
Dict[str, list]
|
Key columns per table identifier, for the
tables written with |
Examples:
Splitting events and telemetry into two tables on a REST catalog:
Initializes the IcebergStorageEgress with a catalog and routing settings.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalog
|
Any
|
An instantiated pyiceberg |
required |
default_table
|
Optional[str]
|
The target |
None
|
table_router
|
Optional[Callable[[dict], Optional[str]]]
|
A function taking a dict payload and returning a
|
None
|
schemas
|
Optional[Dict[str, Any]]
|
Optional PyArrow schema per table identifier. A table named here is created with that schema and every batch is cast to it, instead of the schema being inferred from the first batch. A schema can also be a mapping of column name to an Iceberg type name: string, int, long, float, double, boolean, timestamp, timestamptz, date or binary. |
None
|
locations
|
Optional[Dict[str, str]]
|
Optional storage location per table identifier. Pin this when a
consumer reads the table by path, because a catalog with
|
None
|
upsert_keys
|
Optional[Dict[str, Union[str, list]]]
|
Optional key columns per table identifier, a list or one column name. A table named
here is written with |
None
|
table_policy
|
str
|
What the sink does about a table at its first write.
|
'create'
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ImportError
|
If pyiceberg or pyarrow is not installed (the |
Source code in src/desflow/connectors/egress/iceberg.py
334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 | |
Methods:
run(egress_queue)
async
The main execution loop that consumes the egress queue and commits to Iceberg.
This loop continuously polls the internal egress_queue for data batches.
When a batch is received, it offloads the synchronous PyArrow conversion and
the Iceberg commit to a background thread, so the asyncio loop is not blocked
while the catalog is written.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
egress_queue
|
Queue
|
A thread-safe queue containing batches of dictionaries generated by the environment's EgressMixIn. |
required |
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'pyarrow' package is not installed. |
ValueError
|
If PyIceberg refuses a catalog setting, such as an unknown
catalog |
Note
The loop exits gracefully upon receiving an asyncio.CancelledError, so
the final commit has completed before shutdown returns.
Source code in src/desflow/connectors/egress/iceberg.py
Serializers
desflow.connectors.egress.kafka.MessageSerializer
Bases: Protocol
Protocol defining the interface for all Kafka egress serializers.
Any custom serializer passed to KafkaEgress must implement the
serialize method according to this signature.
Methods:
serialize(topic, data)
Serializes the given payload into a byte string for Kafka transmission.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The destination Kafka topic. This is often required by schema registries (like Avro) to resolve the correct schema subject, even if simpler serializers (like JSON) choose to ignore it. |
required |
data
|
Any
|
The payload to serialize. This is typically a raw dictionary or a Pydantic model instance. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bytes |
bytes
|
The fully encoded byte string ready to be published to Kafka. |
Source code in src/desflow/connectors/egress/kafka.py
desflow.connectors.egress.kafka.JsonSerializer
Default fallback serializer providing backward compatibility via orjson.
This serializer converts dictionaries or Pydantic models into standard JSON byte strings.
Methods:
serialize(topic, data)
Serializes the given data to a JSON byte string.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The target Kafka topic (unused by this serializer but required by protocol). |
required |
data
|
Any
|
The payload to serialize (dictionary or Pydantic model). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bytes |
bytes
|
The JSON-encoded byte string. |
Source code in src/desflow/connectors/egress/kafka.py
desflow.connectors.egress.kafka.ConfluentAvroSerializer(registry_url, schema_str, *, auth=UNSET, verify=UNSET, cert=UNSET, timeout=UNSET, headers=UNSET, proxy=UNSET, max_retries=UNSET)
Lazy-loaded serializer for Confluent Schema Registry using fastavro and httpx.
This class converts simulation data into Avro-encoded byte strings in the
Confluent wire format. It registers the schema under the subject
<topic>-value on the first record of each topic, and reuses the schema id
after that.
Initializes the Confluent Avro serializer. It makes no request to the registry.
UNSET as a default, in the signature or the Default column of the API page,
means the argument was not given; each argument's description says its real
default.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_url
|
str
|
The base URL of the Confluent Schema Registry. Several URLs separated by commas are tried in order. |
required |
schema_str
|
str
|
The Avro schema defined as a JSON string (typically
generated from a dataclasses-avroschema |
required |
auth
|
Optional[Tuple[str, str]]
|
A username and password for basic
authentication, for example |
UNSET
|
verify
|
bool | str | SSLContext
|
How to check the registry's certificate: True (the default), False, the path of a CA file or directory, or an SSL context. |
UNSET
|
cert
|
str | tuple | None
|
A client certificate file, or a tuple of the certificate file, the key file and the key password. The default is None. |
UNSET
|
timeout
|
Optional[float]
|
Seconds each step of a request to the registry (connecting, sending, reading) may take. The default is 10.0. None waits without a limit. |
UNSET
|
headers
|
Optional[Mapping[str, str]]
|
Headers sent with every request,
for example |
UNSET
|
proxy
|
Optional[str]
|
The URL of a proxy. The default is None. |
UNSET
|
max_retries
|
int
|
How many times a request is repeated after a transport error or a status of 408, 429, 500, 502, 503 or 504. The default is 3. |
UNSET
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'fastavro' or 'httpx' package is not installed. |
Source code in src/desflow/connectors/egress/kafka.py
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 | |
Methods:
serialize(topic, data)
Serializes the given data into an Avro byte string.
The first call for a topic registers the schema under <topic>-value, which
makes a request to the registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The target Kafka topic, used for schema subject naming. |
required |
data
|
Any
|
The payload to serialize (dictionary or Pydantic model). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bytes |
bytes
|
The Avro-encoded byte string, with the Confluent prefix. |
Raises:
| Type | Description |
|---|---|
SchemaRegistryError
|
If the registry refuses the schema. |
TransportError
|
If no registry URL could be reached. |
Source code in src/desflow/connectors/egress/kafka.py
desflow.connectors.egress.kafka.GlueAvroSerializer(registry_name, schema_str, **boto3_kwargs)
Lazy-loaded serializer for AWS Glue Schema Registry using boto3.
This class converts simulation data into Avro-encoded byte strings, integrating directly with AWS Glue Schema Registry for schema validation.
Initializes the AWS Glue Avro serializer.
The constructor makes no call to Glue. The first serialize for a topic finds
or registers the schema.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_name
|
str
|
The name of the AWS Glue Schema Registry. |
required |
schema_str
|
str
|
The Avro schema defined as a JSON string (typically
generated from a dataclasses-avroschema |
required |
**boto3_kwargs
|
Any
|
Additional arguments passed directly to the |
{}
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'boto3' or 'fastavro' package is not installed. |
Source code in src/desflow/connectors/egress/kafka.py
Methods:
serialize(topic, data)
Serializes the given data into an Avro byte string using AWS Glue.
The first call for a topic finds or registers the schema under
<topic>-value, which calls Glue.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The target Kafka topic. |
required |
data
|
Any
|
The payload to serialize (dictionary or Pydantic model). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bytes |
bytes
|
The Avro-encoded byte string, with the Glue prefix. |
Raises:
| Type | Description |
|---|---|
SchemaRegistryError
|
If a Glue call fails or the new version does not become available. |
Source code in src/desflow/connectors/egress/kafka.py
desflow.connectors.egress.kafka.ConfluentJsonSchemaSerializer(registry_url, schema_str, *, auth=UNSET, verify=UNSET, cert=UNSET, timeout=UNSET, headers=UNSET, proxy=UNSET, max_retries=UNSET)
Lazy-loaded serializer for Confluent Schema Registry using jsonschema and httpx.
This class converts simulation data into JSON in the Confluent wire format. It
checks each record against the schema, registers the schema under the subject
<topic>-value on the first record of each topic, and reuses the schema id after
that.
Initializes the Confluent JSON Schema serializer. It makes no request to the registry.
UNSET as a default, in the signature or the Default column of the API page,
means the argument was not given; each argument's description says its real
default.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_url
|
str
|
The base URL of the Confluent Schema Registry. Several URLs separated by commas are tried in order. |
required |
schema_str
|
str
|
The JSON Schema as a JSON string. A schema with no
|
required |
auth
|
Optional[Tuple[str, str]]
|
A username and password for basic
authentication, for example |
UNSET
|
verify
|
bool | str | SSLContext
|
How to check the registry's certificate: True (the default), False, the path of a CA file or directory, or an SSL context. |
UNSET
|
cert
|
str | tuple | None
|
A client certificate file, or a tuple of the certificate file, the key file and the key password. The default is None. |
UNSET
|
timeout
|
Optional[float]
|
Seconds each step of a request to the registry (connecting, sending, reading) may take. The default is 10.0. None waits without a limit. |
UNSET
|
headers
|
Optional[Mapping[str, str]]
|
Headers sent with every request,
for example |
UNSET
|
proxy
|
Optional[str]
|
The URL of a proxy. The default is None. |
UNSET
|
max_retries
|
int
|
How many times a request is repeated after a transport error or a status of 408, 429, 500, 502, 503 or 504. The default is 3. |
UNSET
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'jsonschema' or 'httpx' package is not installed. |
SerializationError
|
If the schema is not valid JSON or not a valid schema. |
Source code in src/desflow/connectors/egress/kafka.py
328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 | |
Methods:
serialize(topic, data)
Serializes the given data into a JSON byte string.
The first call for a topic registers the schema under <topic>-value, which
makes a request to the registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The target Kafka topic, used for schema subject naming. |
required |
data
|
Any
|
The payload to serialize (dictionary or Pydantic model). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bytes |
bytes
|
The JSON bytes, with the Confluent prefix. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the record does not match the schema. |
SchemaRegistryError
|
If the registry refuses the schema. |
TransportError
|
If no registry URL could be reached. |
Source code in src/desflow/connectors/egress/kafka.py
desflow.connectors.egress.kafka.ConfluentProtobufSerializer(registry_url, message_type, *, auth=UNSET, verify=UNSET, cert=UNSET, timeout=UNSET, headers=UNSET, proxy=UNSET, max_retries=UNSET)
Lazy-loaded serializer for Confluent Schema Registry using protobuf and httpx.
This class converts simulation data into Protobuf messages in the Confluent wire
format. The schema is the file of the message class. On the first record of each
topic it registers the files that the schema imports, then the schema under the
subject <topic>-value, and it reuses the schema id after that.
Initializes the Confluent Protobuf serializer. It makes no request to the registry.
UNSET as a default, in the signature or the Default column of the API page,
means the argument was not given; each argument's description says its real
default.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_url
|
str
|
The base URL of the Confluent Schema Registry. Several URLs separated by commas are tried in order. |
required |
message_type
|
type[Message]
|
The generated Protobuf message class. Its file is registered as the schema. |
required |
auth
|
Optional[Tuple[str, str]]
|
A username and password for basic
authentication, for example |
UNSET
|
verify
|
bool | str | SSLContext
|
How to check the registry's certificate: True (the default), False, the path of a CA file or directory, or an SSL context. |
UNSET
|
cert
|
str | tuple | None
|
A client certificate file, or a tuple of the certificate file, the key file and the key password. The default is None. |
UNSET
|
timeout
|
Optional[float]
|
Seconds each step of a request to the registry (connecting, sending, reading) may take. The default is 10.0. None waits without a limit. |
UNSET
|
headers
|
Optional[Mapping[str, str]]
|
Headers sent with every request,
for example |
UNSET
|
proxy
|
Optional[str]
|
The URL of a proxy. The default is None. |
UNSET
|
max_retries
|
int
|
How many times a request is repeated after a transport error or a status of 408, 429, 500, 502, 503 or 504. The default is 3. |
UNSET
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'protobuf' or 'httpx' package is not installed. |
Source code in src/desflow/connectors/egress/kafka.py
447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 | |
Methods:
serialize(topic, data)
Serializes the given data into a Protobuf byte string.
The first call registers the imported files and, for a topic, the schema under
<topic>-value, which makes requests to the registry.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The target Kafka topic, used for schema subject naming. |
required |
data
|
Any
|
An instance of the message class, or a dictionary or Pydantic model. A key the schema does not have is dropped. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bytes |
bytes
|
The Protobuf bytes, with the Confluent prefix and message index. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the record does not fit the message class. |
SchemaRegistryError
|
If the registry refuses a schema. |
TransportError
|
If no registry URL could be reached. |
Source code in src/desflow/connectors/egress/kafka.py
desflow.connectors.egress.kafka.GlueJsonSchemaSerializer(registry_name, schema_str, **boto3_kwargs)
Lazy-loaded serializer for AWS Glue Schema Registry using boto3 and fastjsonschema.
This class converts simulation data into JSON with the AWS Glue prefix. It checks
each record against the schema as aws-glue-schema-registry 1.1.3 did, so a
missing property that has a default is filled in.
Initializes the AWS Glue JSON Schema serializer.
The constructor makes no call to Glue. The first serialize for a topic finds
or registers the schema.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_name
|
str
|
The name of the AWS Glue Schema Registry. |
required |
schema_str
|
str
|
The JSON Schema as a JSON string. |
required |
**boto3_kwargs
|
Any
|
Additional arguments passed directly to the |
{}
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'boto3' or 'fastjsonschema' package is not installed. |
SerializationError
|
If the schema is not valid JSON or cannot be compiled. |
Source code in src/desflow/connectors/egress/kafka.py
Methods:
serialize(topic, data)
Serializes the given data into a JSON byte string using AWS Glue.
The first call for a topic finds or registers the schema under
<topic>-value, which calls Glue.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The target Kafka topic. |
required |
data
|
Any
|
The payload to serialize (dictionary or Pydantic model). |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bytes |
bytes
|
The JSON bytes, with the Glue prefix. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the record does not match the schema. |
SchemaRegistryError
|
If a Glue call fails or the new version does not become available. |
Source code in src/desflow/connectors/egress/kafka.py
desflow.connectors.egress.kafka.GlueProtobufSerializer(registry_name, message_type, schema_str, **boto3_kwargs)
Lazy-loaded serializer for AWS Glue Schema Registry using boto3 and protobuf.
This class converts simulation data into Protobuf messages with the AWS Glue
prefix and message index. Glue stores the schema as .proto text, which the
caller passes in, because protobuf cannot print it from the message class.
Initializes the AWS Glue Protobuf serializer.
The constructor makes no call to Glue. The first serialize for a topic finds
or registers the schema.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
registry_name
|
str
|
The name of the AWS Glue Schema Registry. |
required |
message_type
|
type[Message]
|
The generated Protobuf message class. |
required |
schema_str
|
str
|
The text of the |
required |
**boto3_kwargs
|
Any
|
Additional arguments passed directly to the |
{}
|
Raises:
| Type | Description |
|---|---|
ImportError
|
If the 'boto3' or 'protobuf' package is not installed. |
Source code in src/desflow/connectors/egress/kafka.py
Methods:
serialize(topic, data)
Serializes the given data into a Protobuf byte string using AWS Glue.
The first call for a topic finds or registers the schema under
<topic>-value, which calls Glue.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
topic
|
str
|
The target Kafka topic. |
required |
data
|
Any
|
An instance of the message class, or a dictionary or Pydantic model. A key the schema does not have is dropped. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bytes |
bytes
|
The Protobuf bytes, with the Glue prefix and message index. |
Raises:
| Type | Description |
|---|---|
SerializationError
|
If the record does not fit the message class. |
SchemaRegistryError
|
If a Glue call fails or the new version does not become available. |
Source code in src/desflow/connectors/egress/kafka.py
desflow.connectors.egress.kafka.SchemaRegistryError(http_status, error_code, message)
Bases: RuntimeError
A schema registry call failed, or its answer was an error or an unusable version.
Attributes:
| Name | Type | Description |
|---|---|---|
http_status |
int
|
The HTTP status of the response, or -1 when the call failed before the registry answered (credentials, endpoint) or the answer was a schema version that cannot be used. |
error_code |
int | str
|
The registry's own error code: an int from a
Confluent registry, or the name of the error from AWS Glue, such as
|
message |
str
|
The registry's message, the raw body when it was not JSON, or desflow's description of the failure. |
Source code in src/desflow/connectors/_schema_registry/errors.py
desflow.connectors.egress.kafka.SerializationError
Bases: ValueError
A payload or a schema cannot be written or read.
Raised for data too short to hold the registry's framing, a magic byte that is not known, a schema of a format the class does not handle, and a schema with rules that desflow does not run.
Generated Values
desflow.values.Template(payload, name=None, vars=None, vars_once=None, check=True)
A payload of generated values, for a builder or low-level script.
Return it from a task function and the task renders it when the finished event is
published, or call render in your own code. It is the same mapping a blueprint's
payload takes: a mapping whose values may be generators ({"gen": "uniform",
"low": 2, "high": 200}), or one generator that gives a mapping or null. A null
gives an event whose value is null, which a Kafka sink with envelope off writes
as a tombstone.
Example
Attributes:
| Name | Type | Description |
|---|---|---|
name |
Optional[str]
|
What the template is called; it names the random
sequences of its generators, so give two templates different names. A name
with dots is a path: |
payload |
Any
|
The validated mapping. |
vars |
Dict[str, Any]
|
The named values a |
vars_once |
Dict[str, Any]
|
The same, evaluated once for each entity. |
uses_previous |
bool
|
Whether a generator in it is a |
looks_up |
Set[str]
|
The tasks a |
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
payload
|
Any
|
A mapping, or one generator that gives a mapping or null. |
required |
name
|
Optional[str]
|
What the template is called. |
None
|
vars
|
Optional[Mapping[str, Any]]
|
Named values, each seeing the ones before it by name. |
None
|
vars_once
|
Optional[Mapping[str, Any]]
|
Named values made once for each entity. |
None
|
check
|
bool
|
Whether to check the generators now, such as that a |
True
|
Raises:
| Type | Description |
|---|---|
ValidationError
|
If a generator or a mapping is not valid. |
ValueError
|
If a check finds a problem, such as a |
Source code in src/desflow/values/template.py
Methods:
bind(entity=None, **fields)
The template with an entity and fixed fields, rendered by the task that
publishes it. fields are written over the generated ones.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
entity
|
Any
|
The entity whose attributes are read, or None. |
None
|
**fields
|
Any
|
Fields written over the generated ones. |
{}
|
Source code in src/desflow/values/template.py
render(context, entity=None)
Evaluates the template at the simulation time now.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
Any
|
The |
required |
entity
|
Any
|
The entity whose attributes |
None
|
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, Any]]
|
Optional[Dict[str, Any]]: The mapping, as plain Python data, or None when |
Optional[Dict[str, Any]]
|
the payload is one generator that gave null. |
Source code in src/desflow/values/template.py
desflow.values.BoundPayload(template, entity=None, fields=None, attributes=None)
A Template bound to an entity and to fields that overwrite generated ones,
such as the id_field of a task. A task renders it where it would publish a
plain payload.
Attributes:
| Name | Type | Description |
|---|---|---|
template |
Template
|
The template. |
entity |
The entity whose attributes |
|
fields |
Dict[str, Any]
|
Fields written over the generated mapping. |
attributes |
List[str]
|
Names of entity attributes written over the mapping, each under its own name, read when the payload is built. |
Source code in src/desflow/values/template.py
Methods:
render(context)
Evaluates the template, then writes fields over it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
context
|
Any
|
The simulation context that holds the value runtime. |
required |
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, Any]]
|
Optional[Dict[str, Any]]: The mapping, or None when the template gave |
Optional[Dict[str, Any]]
|
null and there are no fields or attributes to write over it. |
Raises:
| Type | Description |
|---|---|
ValueGenerationError
|
If the template gives something other than a mapping or null, or gives null while there are fields or attributes to write over it. |
Source code in src/desflow/values/template.py
desflow.values.generator(spec)
Validates one generator written as a mapping, such as {"gen": "uniform",
"low": 2, "high": 200}, and gives the typed object.
Pass the result where a builder method takes a value, such as attributes of
add_arrival, items of add_store, set of a scenario step, or the value
of an on action. The place gives it its random sequence when it is added.
A row is read from the working directory when the place is added, and a missing
file raises ValueError there.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
Any
|
The generator as a mapping. |
required |
Raises:
| Type | Description |
|---|---|
ValidationError
|
If the generator is not valid. |
Source code in src/desflow/values/template.py
desflow.values.ValueGenerationError
Bases: ValueError
A generator could not give its value: a bad argument, a missing var, a draw from an empty choice. The message names the place in the file.
desflow.values.ValueRuntime(seed, env, sampler=None, context=None, live_scope=None, stop=None)
The state the generators of one run share, and the means to evaluate them.
Attributes:
| Name | Type | Description |
|---|---|---|
seed |
Optional[int]
|
The seed every generator's random sequence comes from; None gives a different run each time. |
context |
The |
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
seed
|
Optional[int]
|
The run's seed, or None. |
required |
env
|
Any
|
The environment, for the simulation clock, its start and its zone. |
required |
sampler
|
Any
|
The run's |
None
|
context
|
Any
|
The |
None
|
live_scope
|
Optional[LiveScope]
|
The run's |
None
|
stop
|
Optional[Callable[[], None]]
|
Stops the run, for a replay whose rows end with |
None
|
Source code in src/desflow/values/runtime.py
Attributes
zone
property
tzinfo: The run's time zone.
Methods:
aggregate(node, scope)
The value of an aggregate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Aggregate
|
The generator. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The mean, sum, minimum, maximum or count of the rows kept. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/values/runtime.py
condition(node, text, scope)
Whether an expression written in a generator holds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Gen
|
The generator, or the value, to act on. |
required |
text
|
str
|
The expression. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Source code in src/desflow/values/runtime.py
copy(value)
A mapping or list copied, so a payload never shares a var or an attribute with the entity.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
Any
|
The value. |
required |
Source code in src/desflow/values/runtime.py
count(node)
How many times node has been asked for a count; the call counts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Gen
|
The generator, or the value, to act on. |
required |
Source code in src/desflow/values/runtime.py
distinct_frame()
Opens what a repeat is drawing without replacement: a lookup with
distinct inside it does not give the same row twice until the repeat ends.
Source code in src/desflow/values/runtime.py
draw_service(node, service)
One draw from a service's live distribution.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Gen
|
The generator, or the value, to act on. |
required |
service
|
str
|
The name of the service. |
required |
Source code in src/desflow/values/runtime.py
evaluate(node, scope)
The value of a node: a generator is evaluated, a mapping, a list and a tuple are walked (a tuple gives a list), and anything else is a constant.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Any
|
The generator, or the value, to act on. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Raises:
| Type | Description |
|---|---|
ValueGenerationError
|
If a generator cannot give a value. The message names the file, the line and the generator. |
Source code in src/desflow/values/runtime.py
evaluate_vars(specs, scope, known=None)
Evaluates named values in order, each seeing the ones before it as a bare name; returns the new values.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
specs
|
Dict[str, Any]
|
The vars to evaluate, in order, by name. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
known
|
Optional[Dict[str, Any]]
|
The names that already have values. |
None
|
Source code in src/desflow/values/runtime.py
faker(node, locale)
The Faker of a fake generator, seeded from the generator's own sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Fake
|
The generator, or the value, to act on. |
required |
locale
|
str
|
The Faker locale. |
required |
Source code in src/desflow/values/runtime.py
forget(key)
Drops the vars_once kept for an entity whose flow has ended, so a long
run does not keep one entry for each entity it has seen.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
Any
|
The key of the entity. |
required |
Source code in src/desflow/values/runtime.py
iso_time(offset)
The logical time now plus offset seconds, as the text of a timestamp.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
offset
|
float
|
Seconds to add to the time now. |
required |
Source code in src/desflow/values/runtime.py
keep_previous(task)
Has task keep its last finished payload for each stream, for previous.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The name of the task. |
required |
keep_rows(task, size)
Has task keep the last size of its finished payloads, which lookup and
aggregate read.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The name of the task. |
required |
size
|
int
|
How many rows to keep. |
required |
Source code in src/desflow/values/runtime.py
logical_time(seconds)
The logical time seconds of simulation time after the start.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
seconds
|
float
|
Seconds of simulation time. |
required |
Source code in src/desflow/values/runtime.py
lookup(node, scope)
The value of a lookup.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Lookup
|
The generator. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value, as a copy. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the task has no row, |
Source code in src/desflow/values/runtime.py
note_started(task, waiting_for)
Writes the one INFO line about a task whose starts were refused, once it starts.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The task that starts. |
required |
waiting_for
|
Any
|
The tasks it looks up. |
required |
Source code in src/desflow/values/runtime.py
now_time()
omitted(node, scope)
Whether a mapping field is left out: its generator's omit_rate or
omit_when says so. The field is not evaluated then.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Any
|
The generator, or the value, to act on. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Source code in src/desflow/values/runtime.py
once(task, key, make)
The vars_once of a task for an entity: made on the first call and the
same afterwards.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The name of the task. |
required |
key
|
Any
|
The key of the entity. |
required |
make
|
Callable[[], Dict[str, Any]]
|
Makes the values when none were kept. |
required |
Source code in src/desflow/values/runtime.py
pick_row(node, rows)
The next row of a row generator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Row
|
The generator, or the value, to act on. |
required |
rows
|
List[Dict[str, Any]]
|
The rows of the file. |
required |
Source code in src/desflow/values/runtime.py
previous(node, scope)
The value of a previous.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Previous
|
The generator. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value of the last record on the stream, or the default (null when none is written) when the stream has none yet. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If it is read outside a task or the path is not in the record. |
Source code in src/desflow/values/runtime.py
ready(tasks)
Whether every task in tasks has a row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
tasks
|
Any
|
Task names. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True when each task that keeps rows has at least one. |
Source code in src/desflow/values/runtime.py
record(task, payload, entity)
Takes in the finished payload of a task, in the step its record is published. Only a mapping is kept, by reference.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The name of the task. |
required |
payload
|
Any
|
The payload that was published. |
required |
entity
|
Any
|
The entity the task ran for, whose stream |
required |
Source code in src/desflow/values/runtime.py
refuse_start(task, waiting_for, consumed_item=False)
Notes that a start of task was refused for want of a row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
task
|
str
|
The task that was not started. |
required |
waiting_for
|
Any
|
The tasks it looks up. |
required |
consumed_item
|
bool
|
Whether the dropped start had already taken its |
False
|
Source code in src/desflow/values/runtime.py
render(template, entity=None, scope=None, extras=None, attributes=None)
Evaluates a Template: its vars_once once for the entity, its vars,
then its payload, which must give a mapping or null.
A null gives an event whose value is null, which a Kafka sink with
envelope off writes as a tombstone.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
template
|
Any
|
The |
required |
entity
|
Any
|
The entity whose attributes |
None
|
scope
|
Optional[Scope]
|
The scope of a running task, whose bare names the template's vars follow; without it a scope outside a task is made. |
None
|
extras
|
Optional[Dict[str, Any]]
|
Fields written over the result. |
None
|
attributes
|
Optional[List[str]]
|
Names of entity attributes written over the result, read now. |
None
|
Returns:
| Type | Description |
|---|---|
Optional[Dict[str, Any]]
|
Optional[Dict[str, Any]]: The mapping, or None when the payload gave null. |
Raises:
| Type | Description |
|---|---|
ValueGenerationError
|
If the payload gives something other than a mapping
or null, or gives null while |
Source code in src/desflow/values/runtime.py
493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 | |
report_refused()
Writes the INFO line for each task whose starts were all refused to the end of the run, so an empty task has its reason in the log.
Source code in src/desflow/values/runtime.py
rng(node, suffix='')
The random sequence of a generator, made from the seed and the place of
the generator in the file: SeedSequence(seed, spawn_key=stream_key(place)),
the same key a Sampler uses for a distribution of that name.
Adding or removing another field of a mapping leaves it unchanged; inserting a
list item moves the streams of the items after it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
node
|
Gen
|
The generator, or the value, to act on. |
required |
suffix
|
str
|
Text added to the place, to make a second stream for the same generator. |
''
|
Source code in src/desflow/values/runtime.py
scope_of(entity=None, names=None, keyed=False)
The scope generators read outside a task: the registry, the clock, and the
attributes of entity when given.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
entity
|
Any
|
The entity whose attributes are read, or None. |
None
|
names
|
Optional[Dict[str, Any]]
|
Bare names to add, by name. |
None
|
keyed
|
bool
|
Whether |
False
|
Source code in src/desflow/values/runtime.py
time_text(moment)
A time as the text a record's timestamp has, in the run's zone.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
moment
|
datetime
|
The time. |
required |
value(spec, entity=None)
Evaluates a generator now, for a low-level script: a typed generator from
desflow.values.generator, or any value that holds one.
The generator is checked as a Template checks its payload, so a row file is
read from the working folder and a generator that cannot work is refused. It
takes its random stream from the first call, so call value with the same
object each time: a sequence counts and a unique choice refuses on it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
Any
|
The generator, a mapping or list that holds one, or a constant. |
required |
entity
|
Any
|
The entity whose attributes |
None
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in src/desflow/values/runtime.py
desflow.values.Aggregate
Bases: Gen
aggregate: mean, sum, min, max or count over the last last rows of
the task task that where keeps. where is an expression over the row's own
fields, such as lead_days == 3 and precipitation > 0; without it every row counts.
count counts rows and takes no path; the others read the number at path. A
reading looks back through the rows the task keeps (its history) until last rows
are kept, and takes the rows there are when where keeps fewer.
Methods:
evaluate(run, scope)
The value of this generator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run
|
ValueRuntime
|
The value runtime, which holds the generator's random stream. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value. |
Source code in src/desflow/values/models.py
desflow.values.Fake
Bases: Gen
fake: a value from the Faker library. method names any method of Faker's
providers, such as name, email, isbn13 or credit_card_number, and args
and kwargs are passed to it. locale picks the language and country, en_US
by default. It needs the fake extra.
Each fake generator has a Faker of its own, seeded from the run's seed and its
place in the file, so the same seed gives the same values and adding a field
changes no other. With no seed, the values change on every run. A date, a time
or a decimal result is written as text or a float.
Methods:
evaluate(run, scope)
The value of this generator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run
|
ValueRuntime
|
The value runtime, which holds the generator's random stream. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value. |
Source code in src/desflow/values/models.py
desflow.values.HistogramBin
Bases: BaseModel
One bin of a lookup histogram: bin is the share of the rows it covers, in
the order they were written, and frequency is the chance that a draw falls in it.
Both are scaled to add up to 1, so {bin: 2, frequency: 8}, {bin: 8, frequency: 2}
means the same as 0.2 with 0.8, and 0.8 with 0.2.
desflow.values.Lookup
Bases: Gen
lookup: a row of the finished payloads of the task task, or the part of it
at path (dotted; an integer part indexes a list).
A row is readable from the instant the task's finished record is published. The
task keeps its last history rows (default 1,000,000).
strategy is random (the default), first (the oldest row kept) or last (the
newest). histogram splits the rows kept, in the order they were written, into
bins of the stated shares; a draw picks a bin by its frequency and then a row of the
bin at random, so it is for random only. A bin that holds no row yet is left out
of the draw, and while no bin with a frequency above 0 holds a row (a few rows and
a last bin of frequency 0), the draw is over all the rows. latest_by keeps only
the newest row for each value of that field of the row, so a lookup sees a
customer's latest address; strategy and histogram then choose among those rows,
oldest first by the time each was written. distinct draws without replacement
within one repeat, so the items of one order differ.
Methods:
evaluate(run, scope)
The value of this generator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run
|
ValueRuntime
|
The value runtime, which holds the generator's random stream. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value. |
Source code in src/desflow/values/models.py
desflow.values.Previous
Bases: Gen
previous: the last finished payload of the same task for the same stream, or
the part of it at path. The stream is the arrival, so all the entities it starts
share one, except for an arrival with count or keys, where each entity has its
own. default is evaluated only when the stream has no finished payload yet, and
without one the first reading is null.
Methods:
evaluate(run, scope)
The value of this generator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run
|
ValueRuntime
|
The value runtime, which holds the generator's random stream. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value. |
Source code in src/desflow/values/models.py
desflow.values.Row
Bases: Gen
row: one row of a CSV or Parquet file as a mapping, read once at load. A cell
of digits is read as a number, so a postal code with leading zeros loses them;
write them with {gen: format, template: "{n:05d}"}. where is an expression over
the row's columns that keeps some rows; weight names a column of weights; order
is random (the default), sequential, shuffle (every row once, in a random
order) or circular (in file order, again from the first); on_end says what a
sequential or shuffle order does when the rows run out: error (the default),
loop or stop, which ends the run.
Methods:
evaluate(run, scope)
The value of this generator.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run
|
ValueRuntime
|
The value runtime, which holds the generator's random stream. |
required |
scope
|
Scope
|
What the expressions here read. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Any |
Any
|
The value. |