Input Fitting
A model's distributions should come from data where data exists. Input fitting takes a column of measured times, fits every distribution family desflow has, checks each result, offers the data itself as an empirical distribution, and writes the chosen settings in a form a model loads. This page describes the command and the Python call, every option, each outcome and how to deal with it, and how the fitting is done.
What It Is
desflow fit and fit_column fit the families of Arrivals and Services to service durations or to the gaps between arrivals, by maximum likelihood with scipy. Maximum likelihood chooses the parameters under which the measured values are most probable. Each family ends as one row of a table, with its parameters in seconds and its diagnostics, and the best is marked. Fit the Inputs fits the shop's 40 packing times and shows why the best by the numbers still needs checking against the model's answer. Fit a Distribution fits the gaps of an order log.
Fitting needs the fit extra, which installs scipy and matplotlib: pip install "desflow[fit]". A base install runs without it, and imports them only when a fit runs.
How to Use It
Fit a Column
The command prints the table, writes the best candidate's settings to --out and the plot to --plot. It exits with status 1 when no candidate is usable, and with status 2 when the input is invalid.
read_column(path, column), imported with from desflow.analysis.fit import read_column, reads one CSV column as text; fit_column takes any list of numbers or text, where an empty cell or None is a missing value.
Options
| Command option | fit_column argument |
What it sets | Default |
|---|---|---|---|
--column |
the CSV column to fit | required | |
--unit |
unit |
the data's unit: ms, s, min or h; the settings are always in seconds |
s |
--candidates |
candidates |
the families to try, such as gamma,lognormal; an unknown name is refused |
every family |
--lower, --upper |
lower, upper |
the plausible range, in the data's unit: a value outside it is invalid, and a family that can draw outside it is flagged | none |
--bins |
bins |
offer the empirical distribution as a histogram with this many bins | the values themselves |
--min-count |
min_count |
the fewest values a family is fitted to | 20 |
--tolerance |
tolerance |
how far a fitted mean and standard deviation may be from the data's, as a share of the data's | 0.25 |
--drop-invalid |
drop_invalid |
leave invalid values out, counted, rather than stopping | off |
--out |
result.yaml() |
write the best candidate's settings as YAML | |
--plot |
result.plot(path) |
draw the data against the best candidate |
A value is invalid when it is not a number, is not finite, or lies outside --lower and --upper. The table's first line counts the rows, and the values used, missing and invalid.
Read the Result
fit_column returns a FitResult:
table: one row per candidate, withfamily,status,reason,parameters,log_likelihood,aic,ks_statistic,ks_p_value,boundsandbest.aicis Akaike's information criterion, which scores how well a family fits with a penalty for each parameter; lower is better.best: the best candidate's family.settings(kind=None): the settings of the best, or of the familykind, foradd_service(name, **settings); a family that is not usable is refused, naming those that are.text(): the table as the command prints it.yaml(kind=None): the settings as the YAML file--outwrites.plot(path, kind=None): the PNG--plotdraws: the data's histogram over values generated from the candidate, and a Q-Q plot, which puts each sorted data value against the generated value at the same quantile, so points on the diagonal mean the two agree.
The settings have six significant digits, the same numbers the YAML file holds, so a script and a blueprint that load one fit draw the same values.
How the Best Is Chosen
The best is the fitted family with the lowest AIC among those that cannot draw outside --lower and --upper. When every value is the same, it is constant. When no family within the bounds fitted, it is empirical, which never draws outside the data's range. Only when there is no empirical candidate is a fitted family that can draw outside the bounds chosen. A family that is rejected, did not fit or was skipped is never chosen.
An AIC difference below about 2 does not tell two families apart. Check what the choice changes in the model's answer, as Fit the Inputs does.
Outcomes
Each candidate ends with one status. Every one but fitted has a reason, and none stops the fit:
| Status | Cause | What to do |
|---|---|---|
skipped |
fewer values than --min-count |
collect more data, or use the empirical candidate, which is still offered; lower --min-count only if the values are known to be typical |
skipped |
a value of 0 or below, for gamma, weibull or lognormal, which are defined above 0 only |
find out what the zeros are: rounding (record finer times), a different kind of event (remove it from the data), or real zero durations (use a family with no positive-only support, such as exponential or truncated_normal) |
skipped |
every value is the same, so only constant and empirical are fitted |
use constant, or check that the column is the one meant |
skipped |
a value below low, for truncated_normal, whose low is --lower, or 0 |
set --lower to the true lower bound |
did not fit |
scipy's fit raised an error or its search did not settle; the reason is scipy's message | the data is far from that family's shape: choose another family; more data can help |
rejected |
a fitted parameter is not finite or is out of the family's range | as for did not fit |
rejected |
the fitted mean or standard deviation is more than --tolerance from the data's |
look for outliers: correct or remove a value that is a recording error, and keep one that is real. --upper leaves out values above it only with --drop-invalid; without it, such a value stops the fit with status 2. With --upper, only the families bounded above, uniform and triangular, can be chosen as best, then empirical. Or use --bins for an empirical histogram |
rejected |
a value lies where the family cannot draw | as for the zeros above |
fitted with a bounds note |
the family can draw below --lower or above --upper, such as normal below 0 |
it is chosen only when nothing within the bounds fits; prefer a family with the right support |
The command fails only when no candidate is usable, empirical included, which happens only when there are no values or --candidates leaves empirical out.
Load the Settings
!include reads a mapping from another file, relative to the blueprint, so the file --out wrote is loaded and checked as if it were written in place:
| examples/yaml/fitting.yaml:include | |
|---|---|
An arrival also needs its spawn, so the include names the file and adds keys to it: orders: !include {file: fitted/orders.yaml, spawn: pack}. A YAML syntax error in the included file, or a file that is not a mapping, names that file and its line. A wrong key or value in it is reported at the blueprint's !include line, and so is a missing file. A file that includes itself, directly or through another, is refused at the !include that closes the loop.
DistributionConfig(**result.settings()), as the fit section above shows.
Example: Fitted Inputs
The fitting example fits the 40 packing times of the tutorials and runs one hour of a two-packer shop with the fitted packing time. It needs no Docker service and ends on its own.
Quick Start
desflow examples get downloads the example at the release you install, into a folder of its own. Pick uv or pip:
The file the fit writes, which the download already holds:
| examples/yaml/packing_fit.yaml | |
|---|---|
The whole blueprint is examples/yaml/fitting.yaml.
Quick Start
desflow examples get downloads the example at the release you install, into a folder of its own. Pick uv or pip:
The whole file is examples/builder/fitting_example.py.
Quick Start
desflow examples get downloads the example at the release you install, into a folder of its own. Pick uv or pip:
The whole file is examples/low-level/fitting_example.py.
How It Works
Data first. The values are read, missing ones counted and left out, and invalid ones refused or, with --drop-invalid, counted and left out. They are converted to seconds. Then each family is checked against the data before it is fitted: enough values, positive values where the family needs them, and not a constant column.
Maximum likelihood, location at 0. desflow's families have no location parameter: a gamma or a Weibull starts at 0. scipy can fit a location too, which moves the start of the distribution to wherever the data suggests: for the 40 packing times plus one outlier of 120 seconds, a free location puts a gamma's start at 0.70 seconds, the smallest value, with a shape of 0.21 and a scale of 81, which desflow cannot use. So the location of exponential, gamma, weibull and lognormal is fixed at 0, and each fit is converted to desflow's parameters:
| Family | Fitted with | desflow's parameters |
|---|---|---|
exponential |
scipy.stats.expon.fit(x, floc=0) |
rate = 1 ÷ scale |
normal |
scipy.stats.norm.fit(x) |
mean, std |
lognormal |
scipy.stats.lognorm.fit(x, floc=0) |
the variable's mean = scale × e^(s²/2) and std = mean × √(e^(s²) − 1) |
truncated_normal |
the likelihood of scipy.stats.truncnorm with low fixed, maximised by scipy.optimize.minimize |
mean, std, low |
uniform |
scipy.stats.uniform.fit(x) |
low, high |
triangular |
scipy.stats.triang.fit(x) |
low, mode, high |
gamma |
scipy.stats.gamma.fit(x, floc=0) |
shape, scale |
weibull |
scipy.stats.weibull_min.fit(x, floc=0) |
shape, scale |
constant |
every value is the same | value |
empirical |
the values, or numpy.histogram with --bins |
values, or edges and counts |
Each fit runs on its own, so one that fails is a row, not a crash.
Checks. Every fitted result is checked before it can be chosen: each parameter must be finite and within the family's rules, the same rules a blueprint's distribution is checked against, and the family's mean and standard deviation, from desflow's own formulas, must be within --tolerance of the data's. A result that fails is rejected with the reason. Some bad fits pass these checks: the three values 1.0, 1.0000001 and 1.0000002 fitted to a gamma give a shape of 1.4 × 10¹⁴ and a scale of 7.1 × 10⁻¹⁵, with the right mean and spread, which is why --min-count exists.
Diagnostics. For each fitted family: the log-likelihood of the data, AIC computed as 2k − 2 × log-likelihood, where k counts the fitted parameters (a fixed location does not count), and the Kolmogorov-Smirnov statistic and p-value from scipy.stats.kstest. The p-value assumes the parameters were known in advance; they were estimated from the same data, so it is approximate and tends to be too high. empirical has no AIC.
Plot. The generated values are drawn through desflow's Sampler, seeded, with the same smallest delay a run applies, so the plot shows what a run draws and repeats exactly.
Stability and Limits
- Size of the extra. Measured in new Python 3.12 environments on macOS, scipy 1.18.1 takes about 70 MB installed with uv and about 100 MB with pip, which also compiles bytecode, and 20 to 37 MB to download, depending on the platform. matplotlib and its dependencies, beyond the NumPy desflow already needs, add about 55 MB with uv and about 70 MB with pip. That is why fitting is not in the base install.
- Local files only. The command reads a CSV file. Reading from a database or an object store, and automated selection beyond the lowest AIC, come later.
- A fit is not a validation. A family that fits the data well can still change the model's answer, and a model whose inputs fit can still answer the question badly. Compare the model's output with the real system's, as Experiments does.
- One column at a time. Each column is fitted on its own, so a dependence between two inputs, such as larger orders taking longer to pack, is not captured.