A pipeline is easier to test when its input behaves like production: orders arrive at random, queues grow, a machine slows down. A simulation can produce that data, but when the simulation is a Python program, every change is a code change. Someone who only wants to double an arrival rate has to read and edit code, and a reviewer has to check that nothing else moved. A file that holds only settings is easier to read, to compare in a pull request and to run in CI.

dynamic-des 0.16.0 adds that file. A YAML blueprint describes the whole simulation, and the ddes command runs it. Each of the seven examples in the repository now has a YAML version in plain YAML. The documentation was rebuilt around the three ways to write a simulation.

The project started as a way to change a running SimPy model from Kafka (Building an Event-Driven Hybrid Digital Twin with dynamic-des). It then learned to write Parquet for model training (One Simulation, Two Pipelines) and gained a declarative Python API with PostgreSQL and Redis connectors (A Declarative API with Postgres and Redis Connectors). This release removes the need for Python in most simulations.

A Complete Blueprint

This file is examples/yaml/local.yaml. It needs no containers and prints events and telemetry to the terminal:

 1simulation:
 2  sim_id: Factory_A
 3  factor: 1.0
 4
 5egress:
 6  - type: Console
 7
 8resources:
 9  lathe: {current_cap: 2, max_cap: 5}
10
11services:
12  milling: {dist: normal, mean: 3.0, std: 0.5}
13
14arrivals:
15  # Each arrival spawns one process_part task.
16  standard: {dist: exponential, rate: 1.0, spawn: process_part}
17
18tasks:
19  process_part:
20    service: milling
21    resource: lathe
22    # The value of the task's finished event. id_field adds the task id as part_id.
23    payload: {event_type: part_produced, quality: A}
24    id_field: part_id
25
26telemetry:
27  # Samples the lathe every 2 simulation seconds.
28  - interval: 2.0
29    publish:
30      utilization: lathe.utilization
31      queue_length: lathe.queue_length
32
33run:
34  until: 60

Install the package from PyPI, download the file and run it:

1pip install dynamic-des
2curl -O https://raw.githubusercontent.com/jaehyeon-kim/dynamic-des/main/examples/yaml/local.yaml
3ddes run local.yaml
4
5# Override run.until, in seconds or as a duration such as "2 min"
6ddes run local.yaml --until 30

The documentation has the full reference. The ddes command comes with the core package. Typer and PyYAML are now core dependencies, so no extra is needed to run a blueprint. From Python, SimulationContext.from_yaml("local.yaml") returns the same simulation as a builder you can extend.

The file is checked before the run starts. An unknown key, a task that names a missing service, or a scenario path that does not exist is reported with the file name and the line number.

What a Blueprint Can Express

Each section of a blueprint maps to one call of the declarative API. Beyond the sections above, a blueprint supports:

  • Environment variables. ${VAR} and ${VAR:-default} are replaced in string values, so one file serves a laptop and CI.
  • Relative times. logical_start_time and go_live_at take now or a signed duration such as -10m.
  • Backfill then live in one run. Each egress entry takes when: history or when: live. Records stamped before go_live_at go to one sink, and later records go to the other.
  • Scenarios on simulation time. A scenario list sets parameters at given simulation times, such as a capacity change at 30 seconds. The steps wait on the simulation clock, so they repeat exactly and also work at factor: 0.
  • Tasks without a service or a resource. Such a task emits its payload as soon as it is spawned, which suits order or click events that do not queue.
  • Connectors that prepare their target. Kafka creates its event and telemetry topics at start. PostgreSQL creates the tables listed under tables. Parquet and JSONL create the destination folder and take a filesystem mapping for local disk or S3. Iceberg takes its catalog properties and table schemas as plain mappings.

This excerpt from examples/yaml/backfill_live.yaml uses three of them. It writes ten minutes of history to Parquet as fast as the machine allows, then sends the same events to Kafka in real time:

 1simulation:
 2  sim_id: Line_A
 3  factor: 0.0
 4  random_seed: 42
 5  logical_start_time: -10m
 6  go_live_at: now
 7
 8egress:
 9  - type: Parquet
10    config:
11      default_path: data/backfill/events.parquet
12    when: history
13  - type: Kafka
14    config:
15      event_topic: sim-events
16      telemetry_topic: sim-telemetry
17      bootstrap_servers: ${KAFKA_BOOTSTRAP_SERVERS:-localhost:9092}
18    when: live

Python for Logic YAML Cannot Express

Some logic is not configuration, for example an order with a random number of line items. A blueprint references such code with the !python tag, and the module sits beside the file. The repository has one example of this, examples/yaml/advanced/postgres_orders.yaml. Everything else in that file is still plain YAML:

1processes:
2  # Called as order_generator(context, arrival="customer_order", max_items=5).
3  - function: !python postgres_orders_logic.order_generator
4    kwargs: {arrival: customer_order, max_items: 5}

!python also works for a task payload, a telemetry function, an egress when, a connector class and any value under config, simulation or run.until. A file without it is read with PyYAML’s safe loader and imports only the connector modules it names.

Three Ways to Write the Same Simulation

A simulation can now be written in three ways, and all three build the same parameters and run on the same environment:

  • Low-level API: DynamicRealtimeEnvironment used directly, with SimPy processes you write yourself.
  • Declarative API: the SimulationContext builder and its decorators.
  • YAML blueprint: the file above, run with ddes.

The registry paths, the records and the connectors are the same whichever you choose, so a team can start in YAML and move one part to Python when it needs to.

Documentation Rebuilt

The documentation was reorganised around those three ways:

Other Changes in This Release

  • Fractional container capacity. A capacity change on a DynamicContainer was rounded down to a whole number. A tank set to 62.5 now holds 62.5, also when it started at a whole number.
  • DynamicContainer and DynamicStore exported. Both now import from dynamic_des, like DynamicResource.
  • Live max_cap changes. A change to max_cap used to have no effect on a running resource. It now applies at once, and a lower limit shrinks the capacity.
  • Flat rows for files and tables. Parquet, JSONL and Iceberg sinks with no router now write each event’s value as columns and leave telemetry out. A router keeps the earlier behaviour.
  • Time strings in time columns. Iceberg and PostgreSQL convert ISO time strings for timestamp and date columns, using the column types of the table.
  • Stricter blueprint checks. A zero interval, batch size or until is rejected, because an interval of zero made a run hang. A go_live_at and a start time where only one has a time zone are rejected with the line. A connector that rejects its settings is reported with the line instead of a Python error.
  • add_arrival takes std, so normal and lognormal arrivals can set a standard deviation.
  • CI checks. The docs are built in strict mode on every pull request, and the core package alone, with no extra, must run a YAML blueprint.

The simulations in two recent posts feed real pipelines: Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect and Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL. Why Digital Twins Are Rewiring Industry 4.0 explains where a simulation like this fits in a digital twin. Building an Agentic Analytics System over an Iceberg Lakehouse uses dynamic-des to fill an Iceberg lakehouse with the data an agent queries.

Try It Out

1# Core library and the ddes command
2pip install dynamic-des
3
4# Every connector: Kafka, Redis, PostgreSQL, Avro, Parquet and Iceberg
5pip install "dynamic-des[all]"