desflow
Realistic event streams from a simulation you can steer while it runs.
Most test data generators write random rows. desflow writes the events of a working process: orders queue for packers, waits grow at busy times, and a backlog builds when orders arrive faster than the packers can pack them. The data comes from a SimPy discrete-event simulation, so each change in it has a cause in the model, as in a real system. Why desflow explains why it was built and what it is for.
The three ways to write a model build one real-time run. Ingress connectors change its live parameters while it runs, and egress connectors write its events and telemetry:
The control dashboard of the Kafka Digital Twin example changes simulation parameters during a run, and the telemetry shows the effect without a restart. The example's Quick Start runs the dashboard beside the simulation:
Key Features
| Group | What it covers |
|---|---|
| Ways to write a model | a YAML blueprint with no Python, a Python script with the builder API, or SimPy code with the low-level API. All three build the same model and publish the same records. |
| Model parts | arrivals drawn from a distribution, changed on a schedule or replayed from a file; tasks that run in a flow, with routing, rework and priorities; resources with queues, preemption, breakdowns and queue limits; stores and containers; signals, waits, patience and failures |
| Time | a run in real time or at full speed, history written at full speed and then carried on live, and a run that ends at a time, on an event or once the last work has left |
| Live changes | arrival rates, service times and capacities changed during a run by name, such as Shop.arrival.orders.rate, from a timed scenario, a local schedule, a file you edit, Kafka, Redis or PostgreSQL |
| Records | an event for each step of each entity, telemetry with the statistics, and label records that hold the ground truth of each planned change. Records are JSON, Avro, JSON Schema or Protobuf through the Confluent or AWS Glue Schema Registry, or Pydantic models. |
| Connectors | several sinks in one run, each with its own filter: Kafka, including SASL, mTLS, OAuth and AWS IAM clusters, Redis, PostgreSQL, Parquet or JSONL files on local disk, S3-compatible storage, Google Cloud Storage or Azure Blob Storage, and Iceberg tables. A connector of your own is one class. |
| Studies | built-in statistics with a warm-up, replications with a 95% interval for each statistic, experiments that compare designs with the same random numbers and write a report, a check against the queueing formulas before a long run, and distributions fitted to measured times |
| Tools | the desflow command, a JSON Schema that gives an editor completion for blueprints, and a skill with which an AI coding assistant writes and checks blueprints |
Commands
The package comes with the desflow command. It runs a blueprint, and checks, compares and fits models, with no Python to write. Command Line uses every command on the shop of the experiment example.
| Command | What you use it for | How to use it | Reference |
|---|---|---|---|
desflow examples |
list the examples with list, and download one with get |
Get an Example | desflow examples, desflow examples list, desflow examples get |
desflow explain |
check a model for bottlenecks before a long run | Check It Before a Long Run | desflow explain |
desflow run |
run a blueprint, or its replications with a 95% interval for each statistic | Run It | desflow run |
desflow experiment |
compare designs of a model and check them against bounds | Compare Designs | desflow experiment |
desflow report |
build an experiment's report again from its files | Compare Designs | desflow report |
desflow fit |
fit every distribution family to measured times | Fit Measured Times | desflow fit |
desflow dist |
turn a mean and a standard deviation into a family's parameters, and back | Convert Parameters | desflow dist |
desflow schema |
print the blueprint JSON Schema, for an editor | Set Up an Editor | desflow schema |
Where to Start
| To | Start with |
|---|---|
| Learn why desflow exists and what it is for | Why desflow |
| Learn what a discrete-event simulation is | What Is DES? |
| Build a first model and run it | First Simulation |
| Find how many packers keep orders from waiting long | Queues and Capacity |
| Stream events to Kafka | Kafka Digital Twin |
| Write days of history to Parquet or Iceberg at full speed | Fast-Forward to Parquet or Fast-Forward to Iceberg |
| Write history, then keep streaming in real time | Parquet History, Kafka Live |
| Change parameters at set times during a run | Changing Demand |
| Change parameters from Kafka, Redis or PostgreSQL during a run | Live Parameters |
| Compare two designs and say which is better | Comparing Designs |
| Fit service times to measured data | Fitting the Inputs |
| Model customers who give up waiting | Patience |
| Model machines that break down | Model Breakdowns |
| Find every option of a part of desflow | User Guide |
| Do one task, step by step | How-To |
| Move a SimPy model to desflow | Coming from SimPy |
| Look up a YAML key | YAML Keys |
| Look up a class or function | API Reference |
| See public projects that generate their data with desflow | Projects Using desflow |