Measuring a Run read five numbers from one run of the busy hour. The run started with an empty shop: no orders waiting and every packer free. A real shop at that hour is never empty. The first orders of the run find free packers, so they wait less than orders in the real shop would. This page asks: is one run enough?
Decision: how much of the start to leave out, and how many runs give an answer precise enough to act on.
Measures: the mean wait and the packers' utilisation, and how much each varies from run to run.
Runs: the busy hour with 3 packers, run once with the seed 42, then 40 times with seeds spawned from 42, then 5 times after each of three warm-ups.
Assumptions: those of Measuring a Run: orders at random at 1.8 a second, and the 40 packing times measured at the bench.
What You Will Learn
This page shows how to:
leave a warm-up out of a run's statistics;
run replications with spawned seeds and read the mean with its interval;
choose a warm-up and a number of replications.
What You Add
This page adds a warmup to the run, then replications: the same run repeated with other seeds. The model simulates the same busy hour as Measuring a Run.
Plot the Start
Some runs have a natural start and end, such as a shop that opens empty at 9:00 and closes at 17:00. These are terminating runs, and the empty start is part of what they measure. Other runs stand for a system that keeps going, such as the shop in the middle of its busy hour. These are steady-state runs, and their empty start is a distortion called initialisation bias. The busy hour is a steady-state run.
To see how long the distortion lasts, plot an output over time and look for where it stops trending. Save the files from Full Files in one folder and run the commands there. Publish the statistics every 10 seconds: wait_mean is then the mean wait of every order so far, and queue_length is the queue at that moment.
# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_tsenv.setup_stats("Shop","typical",interval=10)# The warm-up is set with the sinks: no sink receives a record stamped before it# ends, and the statistics start again from zero when it endsenv.setup_egress([ConsoleEgress()],warmup=warmup)
Plot the two against sim_ts for the first 15 minutes:
The queue reaches its usual range within the first minute. The mean wait so far climbs for about a minute and a half, because the first orders found free packers. After that it moves up and down with no trend. The plot shows one seed only, and another seed could climb for longer. So the plot alone cannot settle the warm-up. It suggests three candidates:
1 minute, when the queue reaches its usual range;
2 minutes, which covers the climb of the mean wait;
4 minutes, twice that, to see whether a longer warm-up still moves the mean.
The next section sets 2 minutes. Compare Warm-ups, after the replications, checks the three candidates with five seeds.
Leave Out the Start
A warmup time leaves the start out of everything the run reports:
# Records and statistics start after the first 2 minutes; None keeps everythingWARMUP=120.0# seconds, passed to setup_egress in run below
In the low-level API the warm-up is an argument of env.setup_egress, as the plot step above shows, because the sinks are what leave the records out.
A run has one warm-up, in simulation time. Every sink leaves out the records and telemetry stamped before the warm-up ends. The built-in statistics start again from zero when it ends, so they describe only the rest of the run. An order that queued before the warm-up ended and started after it counts with its whole wait. The warm-up differs from go_live_at, which sends early records to one sink and later ones to another, and drops nothing. Time and Pacing compares the two.
Run the hour without and with the warm-up. Each tab prints the mean wait without it, then with it:
# The mean wait without the warm-up, then with itWARMUP=0desflowruntrusting_results.yaml--stdout|grep-E'pack_order\.wait_mean"'|tail-1
desflowruntrusting_results.yaml--stdout|grep-E'pack_order\.wait_mean"'|tail-1
without a warm-up: mean wait 1.382 s
with a 2 min warm-up: mean wait 1.381 s
Run It With Other Seeds
The run in Leave Out the Start used one seed, 42. Another seed draws other arrival gaps and packing times, and gives another mean wait. One run cannot show how far apart those answers are. Within one run, an order's wait depends on the orders queued ahead of it, so the waits are not independent samples. Replications show it: the same run repeated with other seeds, each giving one independent value of every statistic.
Run the busy hour 40 times. The 40 seeds are spawned from the seed 42, so every run of the command uses the same 40 seeds. Each replication leaves out the same 2-minute warm-up:
# 40 runs: the mean wait and the packers' utilisation, each with its intervaldesflowruntrusting_results.yaml--replications40--measuretasks.pack_order.wait_mean--measureresources.packers.utilization_avg
MEASURES=["tasks.pack_order.wait_mean","resources.packers.utilization_avg"]defreplications(n=40):"""Runs the busy hour n times, each after the warm-up, with seeds spawned from 42."""returnreplicate(shop,n,seed=42,until="1 h",measures=MEASURES)
MEASURES=["tasks.pack_order.wait_mean","resources.packers.utilization_avg"]defmodel(env,sampler):"""The busy hour in a given environment: replicate sets the statistics."""env.registry.register_sim_parameter(PARAMS)packers=DynamicResource(env,"Shop","packers")env.process(order_generator(env,sampler,packers))defreplications(n=40):"""Runs the busy hour n times, each after the warm-up, with seeds spawned from 42."""shop=LowLevelModel(model,sim_id="Shop")returnreplicate(shop,n,seed=42,until=3600,warmup=WARMUP,measures=MEASURES)
LowLevelModel names the function that builds the model in an environment it is given, and the warm-up is an argument of replicate, because a replication has no sinks.
Each replication runs unpaced with no sinks and reads the built-in statistics at its end. Replications describes every option.
The mean of the 40 values is the estimate. Its 95% interval says how far off it may be: with 95% confidence, the true mean lies within the estimate plus or minus the half-width. The seeds are spawned in order, so the first 10 of the 40 replications are the 10 a run with --replications 10 makes. The table reads the estimate after the first 5, 10 and 20 replications and after all 40:
Replications
Mean wait (s)
Half-width (s)
Wait half-width share
Utilisation (%)
Half-width (%)
Utilisation half-width share
5
2.21
0.47
21.2%
89.72
1.37
1.5%
10
2.25
0.25
11.0%
90.34
0.91
1.0%
20
2.48
0.32
13.0%
90.33
0.63
0.7%
40
2.32
0.20
8.4%
90.03
0.45
0.5%
The mean waits of the 40 replications range from 1.45 to 4.39 seconds. The single run with the seed 42 measured 1.38 seconds, lower than any of them.
Choose the Precision
How many replications are enough depends on the decision, not on a fixed number such as 30. Choose the precision each output needs. Here it is a half-width within 10% of the mean. Utilisation meets it with 5 replications. Of the replication counts in the table of Run It With Other Seeds, the mean wait first meets it at 40, because waits vary far more from run to run than how busy the packers are. Each output therefore needs its own number of replications, and the study needs enough replications for the least precise output the decision uses.
The half-width shrinks in proportion to 1 ÷ √n, so a half-width k times smaller needs about k² times as many replications. 10 replications gave 11.0%, so 10% needs about 10 × (11.0 ÷ 10)² ≈ 12. That figure is a guide, not a promise: the half-width is itself an estimate, and 20 replications gave 13.0%. Add replications, then read the half-width again.
The precision also depends on the confidence level, which is 95% unless you choose another. --level 0.9 gives 90% intervals from the same 40 runs (level=0.9 in replicate, for a builder or a low-level model):
measure n mean half-width (90%)
tasks.pack_order.wait_mean 40 2.319 0.1625
The half-width falls from 0.20 to 0.16 seconds, 7.0% of the mean, with no extra run. A 90% interval misses the true mean more often, 1 study in 10 instead of 1 in 20, so choose the level with the precision, before the runs.
Compare Warm-ups
One seed gives one path through the hour. Run five replications after each candidate warm-up, with the same five seeds spawned from 42, and compare the mean waits. Five replications are enough here, although the mean wait needs 40 for 10% precision. The comparison does not need a precise mean: it asks only whether a longer warm-up moves the mean by more than its half-width. Each candidate runs the same five seeds, so a difference between candidates comes from the warm-up and not from the seeds:
defwarmups(n=5)->dict:"""The mean wait of n runs after each candidate warm-up, with seeds spawned from 42."""wait=["tasks.pack_order.wait_mean"]return{warmup:replicate(shop,n,seed=42,until="1 h",warmup=warmup,measures=wait)forwarmupin("1 min","2 min","4 min")}
defwarmups(n=5)->dict:"""The mean wait of n runs after each candidate warm-up, with seeds spawned from 42."""shop=LowLevelModel(model,sim_id="Shop")wait=["tasks.pack_order.wait_mean"]return{warmup:replicate(shop,n,seed=42,until=3600,warmup=warmup,measures=wait)forwarmupin(60,WARMUP,240)}
The mean wait over the five replications, in seconds, with its 95% half-width:
Warm-up
Mean wait (s)
1 minute
2.196 ± 0.47
2 minutes
2.207 ± 0.47
4 minutes
2.232 ± 0.47
Output Analysis keeps the shortest candidate that passes this comparison and covers the climb in the plot. After 1 minute, the longer warm-ups move the mean by 0.011 and 0.036 seconds. Both are far less than the half-width of 0.47 seconds, so every candidate passes. The plot climbs until about a minute and a half, so the warm-up is 2 minutes.
What You Get and Why
Is one run enough? No, because one run's answer depends on its seed. The run with the seed 42 measured 1.38 seconds, lower than every one of 40 replications. Over 40 replications the mean wait is 2.32 ± 0.20 seconds with 95% confidence, within the chosen 10% precision, and the packers are busy 90.03 ± 0.45% of the time.
The empty start is not the problem. The warm-up is 2 minutes, the shortest candidate that passes the comparison over five seeds and covers the climb in the plot. In the run with the seed 42, a 2-minute warm-up changes the mean wait only from 1.382 to 1.381 seconds. The shop fills within a minute or two, and those minutes are a small part of one hour. A shorter run, or a start further from the usual state, such as a shop that opens with a backlog of orders, needs the warm-up much more.
Initialisation bias (Output Analysis explains it) fades once the state of the shop no longer depends on how the run started. Here the queue, empty for the first orders, reaches its usual range within a minute. The warmed-up numbers still pass the Little's law check from Measuring a Run. In the 3,480 seconds after the warm-up, 6,024 orders were packed, 1.73 a second. 1.73 orders a second × 1.381 seconds of mean wait = 2.39 orders waiting on average, which matches the queue_length_avg of 2.40 that the same statistics report. Statistics describes how the reset works.
Replications (Output Analysis explains them) are independent of each other, because each has its own seed. So the spread of their values measures how much the answer depends on chance. With few replications the spread itself is uncertain. The Student t quantile allows for it: the half-width is the standard error times 2.776 for 5 replications, and times 2.023 for 40.
Key Takeaways
One run's answer depends on its seed. Replications measure how much.
Report the mean of the replications with its 95% interval, not the value of one run.
A warm-up removes the start of a run, which begins empty. Choose the shortest warm-up that gives the same answer as longer ones.
# The Trusting Results tutorial: the busy hour, measured after a warm-up.# Run it with: desflow run trusting_results.yaml# Without the warm-up: WARMUP=0 desflow run trusting_results.yamlsimulation:sim_id:Shopfactor:0random_seed:42# Records and statistics start after the first 2 minutes; 0 keeps everythingwarmup:${WARMUP:-2 min}resources:packers:{current_cap:3, max_cap:10}arrivals:orders:{dist:exponential, rate:1.8, spawn:pack_order}services:# The 40 packing times measured at the bench, in secondspacking:dist:empiricalvalues:[1.5,2.5,2.4,1.2,1.3,1.2,1.8,1.4,2.0,0.7,2.7,1.4,1.9,1.4,1.3,1.8,2.0,1.4,1.4,1.9,1.1,0.8,1.7,1.1,0.7,1.1,1.2,0.9,0.8,1.5,2.1,1.3,1.1,1.7,1.9,1.3,1.8,2.2,1.4,1.1]tasks:pack_order:service:packingresource:packerspayload:{event_type:order_packed}# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_tsstats:{level:typical, interval:10}egress:-type:Consolerun:until:1 h
"""The Trusting Results tutorial: the busy hour, measured after a warm-up, builder.Three packers pack orders that arrive at 1.8 a second on average, with the packingtimes measured at the bench. Every record is printed to the terminal. At the end thescript prints the mean wait without and with the warm-up, the mean wait of 5 runsafter each of three candidate warm-ups, then the mean and half-width of two outputsover 5, 10, 20 and 40 replications. To see only those lines, run:python trusting_results.py 2>/dev/null"""importloggingfromdesflowimportConsoleEgress,SimulationContext,replicatefromdesflow.analysisimportintervallogging.basicConfig(level=logging.INFO,format="%(levelname)s [%(asctime)s] %(message)s")# The 40 packing times measured at the bench, in secondsMEASURED=[1.5,2.5,2.4,1.2,1.3,1.2,1.8,1.4,2.0,0.7,2.7,1.4,1.9,1.4,1.3,1.8,2.0,1.4,1.4,1.9,1.1,0.8,1.7,1.1,0.7,1.1,1.2,0.9,0.8,1.5,2.1,1.3,1.1,1.7,1.9,1.3,1.8,2.2,1.4,1.1,]# fmt: skipdefshop(warmup="2 min")->SimulationContext:"""The busy hour; records and statistics start after `warmup`."""app=(# Records and statistics start after the first 2 minutes; None keeps everythingSimulationContext(sim_id="Shop",factor=0,random_seed=42,warmup=warmup).add_resource("packers",current_cap=3,max_cap=10).add_arrival("orders",dist="exponential",rate=1.8).add_service("packing",dist="empirical",values=MEASURED)# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_ts.with_stats("typical",interval=10).add_egress(ConsoleEgress()))@app.task(service_id="packing",resource_id="packers")defpack_order(task_id:int):return{"event_type":"order_packed"}@app.arrival_loop("orders")deforder_generator(context:SimulationContext):task_id=0whileTrue:order=yieldcontext.wait_for_arrival("orders")context.spawn(pack_order(task_id,entity=order))task_id+=1returnappdefmean_wait(warmup)->float:"""Runs the busy hour and returns the mean wait it measured."""app=shop(warmup)app.run(until="1 h")returnapp.env.stats.readings()["Shop.stats.tasks.pack_order.wait_mean"]MEASURES=["tasks.pack_order.wait_mean","resources.packers.utilization_avg"]defreplications(n=40):"""Runs the busy hour n times, each after the warm-up, with seeds spawned from 42."""returnreplicate(shop,n,seed=42,until="1 h",measures=MEASURES)defwarmups(n=5)->dict:"""The mean wait of n runs after each candidate warm-up, with seeds spawned from 42."""wait=["tasks.pack_order.wait_mean"]return{warmup:replicate(shop,n,seed=42,until="1 h",warmup=warmup,measures=wait)forwarmupin("1 min","2 min","4 min")}defprecision(result)->dict:"""Each output's mean and half-width over the first 5, 10, 20 and 40 replications."""return{(measure,n):interval(result.values[measure][:n])formeasureinMEASURESfornin(5,10,20,40)}if__name__=="__main__":print(f"without a warm-up: mean wait {mean_wait(None):.3f} s")print(f"with a 2 min warm-up: mean wait {mean_wait('2 min'):.3f} s")forwarmup,resultinwarmups().items():row=result.summary[0]print(f"warm-up {warmup}: mean wait {row['mean']:.3f} ± {row['half_width']:.2f} s")for(measure,n),resultinprecision(replications()).items():share=result.half_width/result.meanprint(f"{measure}, {n} runs: {result.mean:.2f} ± {result.half_width:.2f} ({share:.1%})")
"""The Trusting Results tutorial: the busy hour, measured after a warm-up, low-level.Three packers pack orders that arrive at 1.8 a second on average, with the packingtimes measured at the bench. Every record is printed to the terminal. At the end thescript prints the mean wait without and with the warm-up, the mean wait of 5 runsafter each of three candidate warm-ups, then the mean and half-width of two outputsover 5, 10, 20 and 40 replications. To see only those lines, run:python trusting_results_low_level.py 2>/dev/null"""importloggingfromdesflowimport(CapacityConfig,ConsoleEgress,DistributionConfig,DynamicRealtimeEnvironment,DynamicResource,LowLevelModel,Sampler,SimParameter,replicate,)fromdesflow.analysisimportintervallogging.basicConfig(level=logging.INFO,format="%(levelname)s [%(asctime)s] %(message)s")# The 40 packing times measured at the bench, in secondsMEASURED=[1.5,2.5,2.4,1.2,1.3,1.2,1.8,1.4,2.0,0.7,2.7,1.4,1.9,1.4,1.3,1.8,2.0,1.4,1.4,1.9,1.1,0.8,1.7,1.1,0.7,1.1,1.2,0.9,0.8,1.5,2.1,1.3,1.1,1.7,1.9,1.3,1.8,2.2,1.4,1.1,]# fmt: skipPARAMS=SimParameter(sim_id="Shop",arrival={"orders":DistributionConfig(dist="exponential",rate=1.8)},service={"packing":DistributionConfig(dist="empirical",values=MEASURED)},resources={"packers":CapacityConfig(current_cap=3,max_cap=10)},)defpack_order(env,sampler,packers,key,task_id):"""One order, which reports each step to the statistics."""path_id="Shop.service.packing"defpublish(value):env.publish_event(key,value,task="pack_order",instance=task_id)publish({"path_id":path_id,"status":"queued"})env.stats.flow_started("orders")# the order enters the shopenv.stats.task_queued("pack_order",task_id)withpackers.request()asrequest:yieldrequestpublish({"path_id":path_id,"status":"started"})env.stats.task_started("pack_order",task_id)yieldenv.timeout(sampler.sample(env.registry.get_config(path_id)))publish({"event_type":"order_packed"})env.stats.task_finished("pack_order",task_id)env.stats.flow_finished("orders")# the order leaves the shopdeforder_generator(env,sampler,packers):arrival=env.registry.get_config("Shop.arrival.orders")task_id=0whileTrue:yieldenv.timeout(sampler.sample(arrival))env.process(pack_order(env,sampler,packers,f"orders-{task_id+1}",task_id))task_id+=1# Records and statistics start after the first 2 minutes; None keeps everythingWARMUP=120.0# seconds, passed to setup_egress in run belowdefrun(warmup=WARMUP,until=3600,factor=0.0):env=DynamicRealtimeEnvironment(factor=factor)env.registry.register_sim_parameter(PARAMS)# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_tsenv.setup_stats("Shop","typical",interval=10)# The warm-up is set with the sinks: no sink receives a record stamped before it# ends, and the statistics start again from zero when it endsenv.setup_egress([ConsoleEgress()],warmup=warmup)packers=DynamicResource(env,"Shop","packers")env.process(order_generator(env,Sampler(seed=42),packers))try:env.run(until=until)finally:env.teardown()returnenvdefmean_wait(warmup)->float:"""Runs the busy hour and returns the mean wait it measured."""returnrun(warmup).stats.readings()["Shop.stats.tasks.pack_order.wait_mean"]MEASURES=["tasks.pack_order.wait_mean","resources.packers.utilization_avg"]defmodel(env,sampler):"""The busy hour in a given environment: replicate sets the statistics."""env.registry.register_sim_parameter(PARAMS)packers=DynamicResource(env,"Shop","packers")env.process(order_generator(env,sampler,packers))defreplications(n=40):"""Runs the busy hour n times, each after the warm-up, with seeds spawned from 42."""shop=LowLevelModel(model,sim_id="Shop")returnreplicate(shop,n,seed=42,until=3600,warmup=WARMUP,measures=MEASURES)defwarmups(n=5)->dict:"""The mean wait of n runs after each candidate warm-up, with seeds spawned from 42."""shop=LowLevelModel(model,sim_id="Shop")wait=["tasks.pack_order.wait_mean"]return{warmup:replicate(shop,n,seed=42,until=3600,warmup=warmup,measures=wait)forwarmupin(60,WARMUP,240)}defprecision(result)->dict:"""Each output's mean and half-width over the first 5, 10, 20 and 40 replications."""return{(measure,n):interval(result.values[measure][:n])formeasureinMEASURESfornin(5,10,20,40)}if__name__=="__main__":print(f"without a warm-up: mean wait {mean_wait(None):.3f} s")print(f"with a 2 min warm-up: mean wait {mean_wait(WARMUP):.3f} s")forwarmup,resultinwarmups().items():row=result.summary[0]print(f"warm-up {warmup:g} s: mean wait {row['mean']:.3f} ± {row['half_width']:.2f} s")for(measure,n),resultinprecision(replications()).items():share=result.half_width/result.meanprint(f"{measure}, {n} runs: {result.mean:.2f} ± {result.half_width:.2f} ({share:.1%})")
