Measuring a Run read five numbers from one run of the busy hour. The run started with an empty shop: no orders waiting and every packer free. A real shop at that hour is never empty. The first orders of the run find free packers, so they wait less than orders in the real shop would. This page asks: is one run enough?
Decision: how much of the start to leave out, and how many runs give an answer precise enough to act on.
Measures: the mean wait and the packers' utilisation, and how much each varies from run to run.
Runs: the busy hour with 3 packers, run once with the seed 42, then 40 times with seeds spawned from 42, then 5 times after each of three warm-ups.
Assumptions: those of Measuring a Run: orders at random at 1.8 a second, and the 40 packing times measured at the bench.
What You Will Learn
This page shows how to:
leave a warm-up out of a run's statistics;
run replications with spawned seeds and read the mean with its interval;
choose a warm-up and a number of replications.
What You Add
This page adds a warmup to the run, then replications: the same run repeated with other seeds. The model simulates the same busy hour as Measuring a Run.
Plot the Start
Some runs have a natural start and end, such as a shop that opens empty at 9:00 and closes at 17:00. These are terminating runs, and the empty start is part of what they measure. Other runs stand for a system that keeps going, such as the shop in the middle of its busy hour. These are steady-state runs, and their empty start is a distortion called initialisation bias. The busy hour is a steady-state run.
To see how long the distortion lasts, plot an output over time and look for where it stops trending. Save the files from Full Files in one folder and run the commands there. Publish the statistics every 10 seconds: wait_mean is then the mean wait of every order so far, and queue_length is the queue at that moment.
# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_tsstats:{level:typical, interval:10}egress:-{type:Console}
WARMUP=0 keeps the first minutes, and --until stops the run at 15 minutes. head -4 keeps the first four lines, the queue and the mean wait at 10 and 20 seconds:
# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_ts.with_stats("typical",interval=10).add_egress(ConsoleEgress())
The script runs the first 15 minutes without a warm-up when it is given the argument start. head -4 keeps the first four lines, the queue and the mean wait at 10 and 20 seconds:
# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_tsenv.setup_stats("Shop","typical",interval=10)# The warm-up is set with the sinks: no sink receives a record stamped before it# ends, and the statistics start again from zero when it endsenv.setup_egress([ConsoleEgress()],warmup=warmup)
The script runs the first 15 minutes without a warm-up when it is given the argument start. head -4 keeps the first four lines, the queue and the mean wait at 10 and 20 seconds:
Without head -4, each tab prints the two values every 10 seconds of the first 15 minutes, 90 lines for each path. Plot the value of the two paths against sim_ts. Some of the values:
sim_ts (s)
Queue
Mean wait so far (s)
10
0
0.06
30
0
0.21
60
2
0.65
90
1
0.60
120
0
0.59
300
0
1.31
600
10
2.60
900
0
2.35
The plot of all 90 readings:
The queue is empty for the first 40 seconds and then moves between 0 and 17 orders. The mean wait so far climbs for about a minute, because the first orders found free packers, and stays near 0.6 seconds until two minutes. After that it climbs in steps, each after a burst of orders, to about 2.5 seconds at eight minutes, and then moves up and down with no trend. The plot shows one seed only, and another seed could climb for longer. So the plot alone cannot settle the warm-up. It suggests three candidates:
1 minute, when the first orders have queued and the first climb of the mean wait has ended;
2 minutes, twice that, to cover the first climb with a margin;
4 minutes, twice that again, to see whether a longer warm-up still moves the mean.
The next section sets 2 minutes. Compare Warm-ups, after the replications, checks the three candidates with five seeds.
Leave Out the Start
A warmup time leaves the start out of everything the run reports:
# Records and statistics start after the first 2 minutes; None keeps everythingWARMUP=120.0# seconds, passed to setup_egress in run below
In the low-level API the warm-up is an argument of env.setup_egress, as the plot step above shows, because the sinks are what leave the records out.
A run has one warm-up, in simulation time. Every sink leaves out the records and telemetry stamped before the warm-up ends. The built-in statistics start again from zero when it ends, so they describe only the rest of the run. An order that queued before the warm-up ended and started after it counts with its whole wait. The warm-up differs from go_live_at, a setting of simulation that names the time when records stop counting as history and count as live, so that one sink can take the earlier records and another the later ones. go_live_at drops nothing. Time and Pacing compares the two.
Run the hour without and with the warm-up. Each tab prints the mean wait without it, then with it:
# The mean wait without the warm-up, then with itWARMUP=0desflowruntrusting_results.yaml--stdout|grep-E'pack_order\.wait_mean"'|tail-1
desflowruntrusting_results.yaml--stdout|grep-E'pack_order\.wait_mean"'|tail-1
without a warm-up: mean wait 2.007 s
with a 2 min warm-up: mean wait 2.050 s
Run It With Other Seeds
The run in Leave Out the Start used one seed, 42. Another seed draws other arrival gaps and packing times, and gives another mean wait. One run cannot show how far apart those answers are. Within one run, an order's wait depends on the orders queued ahead of it, so the waits are not independent samples. Replications show it: the same run repeated with other seeds, each giving one independent value of every statistic.
Run the busy hour 40 times. The 40 seeds are spawned from the seed 42, so every run of the command uses the same 40 seeds. Each replication leaves out the same 2-minute warm-up:
# 5, 10, 20 and 40 runs: the mean wait and the packers' utilisation, each with its intervalfornin5102040;dodesflowruntrusting_results.yaml--replications$n--measuretasks.pack_order.wait_mean--measureresources.packers.utilization_avg2>/dev/null|grep-E'^(tasks|resources)'done
MEASURES=["tasks.pack_order.wait_mean","resources.packers.utilization_avg"]defreplications(n=40):"""Runs the busy hour n times, each after the warm-up, with seeds spawned from 42."""returnreplicate(shop,n,seed=42,until="1 h",measures=MEASURES)
MEASURES=["tasks.pack_order.wait_mean","resources.packers.utilization_avg"]defmodel(env,sampler):"""The busy hour in a given environment: replicate sets the statistics."""env.registry.register_sim_parameter(PARAMS)packers=DynamicResource(env,"Shop","packers")env.process(order_generator(env,sampler,packers))defreplications(n=40):"""Runs the busy hour n times, each after the warm-up, with seeds spawned from 42."""shop=LowLevelModel(model,sim_id="Shop")returnreplicate(shop,n,seed=42,until=3600,warmup=WARMUP,measures=MEASURES)
It prints the same eight lines as the Builder script.
LowLevelModel names the function that builds the model in an environment it is given, and the warm-up is an argument of replicate, because a replication has no sinks.
Each replication runs unpaced with no sinks and reads the built-in statistics at its end. Replications describes every option.
The mean of the 40 values is the estimate. Its 95% interval says how far off it may be: with 95% confidence, the true mean lies within the estimate plus or minus the half-width. The seeds are spawned in order, so the first 10 of the 40 replications are the 10 a run with --replications 10 makes. Each tab prints the estimate after the first 5, 10 and 20 replications and after all 40, which the table gives with the half-width as a share of the mean:
Replications
Mean wait (s)
Half-width (s)
Wait half-width share
Utilisation (%)
Half-width (%)
Utilisation half-width share
5
2.01
0.43
21.4%
89.60
1.78
2.0%
10
2.15
0.30
14.0%
90.00
1.08
1.2%
20
2.15
0.15
7.1%
89.99
0.60
0.7%
40
2.10
0.10
4.8%
89.82
0.39
0.4%
The mean waits of the 40 replications range from 1.52 to 3.05 seconds. The single run with the seed 42 measured 2.01 seconds, inside that range and close to the mean of the 40.
Choose the Precision
How many replications are enough depends on the decision, not on a fixed number such as 30. Choose the precision each output needs. Here it is a half-width within 10% of the mean. Utilisation meets it with 5 replications. Of the replication counts in the table of Run It With Other Seeds, the mean wait first meets it at 20, because waits vary far more from run to run than how busy the packers are. Each output therefore needs its own number of replications, and the study needs enough replications for the least precise output the decision uses.
The half-width shrinks in proportion to 1 ÷ √n, so a half-width k times smaller needs about k² times as many replications. 10 replications gave 14.0%, so 10% needs about 10 × (14.0 ÷ 10)² ≈ 20. That figure is a guide, not a promise: the half-width is itself an estimate, and 20 replications gave 7.1%. Add replications, then read the half-width again.
The precision also depends on the confidence level, which is 95% unless you choose another. --level 0.9 gives 90% intervals from the same 40 runs (level=0.9 in replicate, for a builder or a low-level model):
measure n mean half-width (90%)
tasks.pack_order.wait_mean 40 2.1 0.08363
The run also prints its seeds and how to repeat one replication. The half-width falls from 0.10 to 0.084 seconds, 4.0% of the mean, with no extra run. A 90% interval misses the true mean more often, 1 study in 10 instead of 1 in 20, so choose the level with the precision, before the runs.
Compare Warm-ups
One seed gives one path through the hour. Run five replications after each candidate warm-up, with the same five seeds spawned from 42, and compare the mean waits. Five replications are few, and the mean wait needs 20 for 10% precision. The comparison does not need a precise mean: it asks only whether a longer warm-up moves the mean by a visible amount. Each candidate runs the same five seeds, so a difference between candidates comes from the warm-up and not from the seeds:
# 5 runs with each candidate warm-upforwin"1 min""2 min""4 min";doWARMUP="$w"desflowruntrusting_results.yaml--replications5--measuretasks.pack_order.wait_mean2>/dev/null|grep'^tasks'done
defwarmups(n=5)->dict:"""The mean wait of n runs after each candidate warm-up, with seeds spawned from 42."""wait=["tasks.pack_order.wait_mean"]return{warmup:replicate(shop,n,seed=42,until="1 h",warmup=warmup,measures=wait)forwarmupin("1 min","2 min","4 min")}
defwarmups(n=5)->dict:"""The mean wait of n runs after each candidate warm-up, with seeds spawned from 42."""shop=LowLevelModel(model,sim_id="Shop")wait=["tasks.pack_order.wait_mean"]return{warmup:replicate(shop,n,seed=42,until=3600,warmup=warmup,measures=wait)forwarmupin(60,WARMUP,240)}
The mean wait over the five replications, in seconds, with its 95% half-width:
Warm-up
Mean wait (s)
1 minute
2.010 ± 0.41
2 minutes
2.006 ± 0.43
4 minutes
2.005 ± 0.49
Output Analysis keeps the shortest candidate that passes this comparison and covers the first climb in the plot with a margin. After 1 minute, the longer warm-ups move the mean by 0.004 and 0.005 seconds, under 0.3% of the mean wait, so no candidate shows a visible change in the mean. The first climb in the plot ends at about a minute, which 1 minute only just covers, so the warm-up is 2 minutes, which covers the climb with a minute to spare. The plotted wait goes on rising in steps after 2 minutes, to about 2.5 seconds at 8 minutes. A 4-minute warm-up, which leaves out half of that rise, moves the mean by only 0.001 seconds from the 2-minute one, so the rise changes the estimate little. This comparison does not show whether the rise is part of the start-up or ordinary variation, because both warm-ups keep the part after 4 minutes.
What You Get and Why
Is one run enough? No, because one run's answer depends on its seed. The 40 replications range from 1.52 to 3.05 seconds, so a run with another seed could have given a very different answer, even though the run with the seed 42 measured 2.01 seconds, close to their mean. Over 40 replications the mean wait is 2.10 ± 0.10 seconds with 95% confidence, within the chosen 10% precision, and the packers are busy 89.82 ± 0.39% of the time.
The empty start is not the problem. The warm-up is 2 minutes: 1 minute passes the comparison over five seeds but only just covers the climb in the plot, and 2 minutes covers it with a margin. In the run with the seed 42, a 2-minute warm-up changes the mean wait only from 2.007 to 2.050 seconds. The shop fills within a minute or two, and those minutes are a small part of one hour. A shorter run, or a start further from the usual state, such as a shop that opens with a backlog of orders, needs the warm-up much more.
Initialisation bias (Output Analysis explains it) fades once the state of the shop no longer depends on how the run started. Here the queue, empty for the first orders, reaches its usual range within about two minutes. The warmed-up numbers still pass the Little's law check from Measuring a Run. In the 3,480 seconds after the warm-up, 6,294 orders were packed, 1.81 a second. 1.81 orders a second × 2.050 seconds of mean wait = 3.71 orders waiting on average, which matches the queue_length_avg of 3.71 that the same statistics report. Each tab prints the two counts:
Replications (Output Analysis explains them) are independent of each other, because each has its own seed. So the spread of their values measures how much the answer depends on chance. With few replications the spread itself is uncertain. The Student t quantile allows for it: the half-width is the standard error times 2.776 for 5 replications, and times 2.023 for 40.
Key Takeaways
One run's answer depends on its seed. Replications measure how much.
Report the mean of the replications with its 95% interval, not the value of one run.
A warm-up removes the start of a run, which begins empty. Choose the shortest warm-up that gives the same answer as longer ones.
# The Trusting Results tutorial: the busy hour, measured after a warm-up.# Run it with: desflow run trusting_results.yaml# Without the warm-up: WARMUP=0 desflow run trusting_results.yamlsimulation:sim_id:Shopfactor:0random_seed:42# Records and statistics start after the first 2 minutes; 0 keeps everythingwarmup:${WARMUP:-2 min}resources:packers:{current_cap:3, max_cap:10}arrivals:orders:{dist:exponential, rate:1.8, spawn:pack_order}services:# The 40 packing times measured at the bench, in secondspacking:dist:empiricalvalues:[1.5,2.5,2.4,1.2,1.3,1.2,1.8,1.4,2.0,0.7,2.7,1.4,1.9,1.4,1.3,1.8,2.0,1.4,1.4,1.9,1.1,0.8,1.7,1.1,0.7,1.1,1.2,0.9,0.8,1.5,2.1,1.3,1.1,1.7,1.9,1.3,1.8,2.2,1.4,1.1]tasks:pack_order:service:packingresource:packerspayload:{event_type:order_packed}# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_tsstats:{level:typical, interval:10}egress:-{type:Console}run:{until:1 h}
"""The Trusting Results tutorial: the busy hour, measured after a warm-up, builder.Three packers pack orders that arrive at 1.8 a second on average, with the packingtimes measured at the bench. Every record is printed to the terminal. At the end thescript prints the mean wait without and with the warm-up, the mean wait of 5 runsafter each of three candidate warm-ups, then the mean and half-width of two outputsover 5, 10, 20 and 40 replications. To see only those lines, run:python trusting_results.py 2>/dev/nullWith the argument start, it runs the first 15 minutes without a warm-up, to plot:python trusting_results.py start 2>&1"""importloggingimportsysfromdesflowimportConsoleEgress,SimulationContext,replicatefromdesflow.analysisimportintervallogging.basicConfig(level=logging.INFO,format="%(levelname)s [%(asctime)s] %(message)s")# The 40 packing times measured at the bench, in secondsMEASURED=[1.5,2.5,2.4,1.2,1.3,1.2,1.8,1.4,2.0,0.7,2.7,1.4,1.9,1.4,1.3,1.8,2.0,1.4,1.4,1.9,1.1,0.8,1.7,1.1,0.7,1.1,1.2,0.9,0.8,1.5,2.1,1.3,1.1,1.7,1.9,1.3,1.8,2.2,1.4,1.1,]# fmt: skipdefshop(warmup="2 min")->SimulationContext:"""The busy hour; records and statistics start after `warmup`."""app=(# Records and statistics start after the first 2 minutes; None keeps everythingSimulationContext(sim_id="Shop",factor=0,random_seed=42,warmup=warmup).add_resource("packers",current_cap=3,max_cap=10).add_arrival("orders",dist="exponential",rate=1.8).add_service("packing",dist="empirical",values=MEASURED)# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_ts.with_stats("typical",interval=10).add_egress(ConsoleEgress()))@app.task(service_id="packing",resource_id="packers")defpack_order(task_id:int):return{"event_type":"order_packed"}@app.arrival_loop("orders")deforder_generator(context:SimulationContext):task_id=0whileTrue:order=yieldcontext.wait_for_arrival("orders")context.spawn(pack_order(task_id,entity=order))task_id+=1returnappdefbusy_hour(warmup)->dict:"""Runs the busy hour and returns the statistics it measured."""app=shop(warmup)app.run(until="1 h")returnapp.env.stats.readings()defmean_wait(warmup)->float:"""Runs the busy hour and returns the mean wait it measured."""returnbusy_hour(warmup)["Shop.stats.tasks.pack_order.wait_mean"]MEASURES=["tasks.pack_order.wait_mean","resources.packers.utilization_avg"]defreplications(n=40):"""Runs the busy hour n times, each after the warm-up, with seeds spawned from 42."""returnreplicate(shop,n,seed=42,until="1 h",measures=MEASURES)defwarmups(n=5)->dict:"""The mean wait of n runs after each candidate warm-up, with seeds spawned from 42."""wait=["tasks.pack_order.wait_mean"]return{warmup:replicate(shop,n,seed=42,until="1 h",warmup=warmup,measures=wait)forwarmupin("1 min","2 min","4 min")}defprecision(result)->dict:"""Each output's mean and half-width over the first 5, 10, 20 and 40 replications."""return{(measure,n):interval(result.values[measure][:n])formeasureinMEASURESfornin(5,10,20,40)}WAIT="Shop.stats.tasks.pack_order.wait_mean"PACKED="Shop.stats.tasks.pack_order.finished"QUEUE="Shop.stats.resources.packers.queue_length_avg"defreport()->None:"""Prints the mean waits, the Little's law figures and the precision."""print(f"without a warm-up: mean wait {mean_wait(None):.3f} s")warmed=busy_hour("2 min")print(f"with a 2 min warm-up: mean wait {warmed[WAIT]:.3f} s")print(f"after the warm-up: {warmed[PACKED]} orders packed, "f"queue_length_avg {warmed[QUEUE]:.2f}")forwarmup,resultinwarmups().items():row=result.summary[0]print(f"warm-up {warmup}: mean wait {row['mean']:.3f} ± {row['half_width']:.2f} s")for(measure,n),resultinprecision(replications()).items():share=result.half_width/result.meanprint(f"{measure}, {n} runs: {result.mean:.2f} ± {result.half_width:.2f} ({share:.1%})")if__name__=="__main__":ifsys.argv[1:]==["start"]:# The statistics every 10 seconds of the first 15 minutes, without a warm-upshop(None).run(until="15 min")else:report()
"""The Trusting Results tutorial: the busy hour, measured after a warm-up, low-level.Three packers pack orders that arrive at 1.8 a second on average, with the packingtimes measured at the bench. Every record is printed to the terminal. At the end thescript prints the mean wait without and with the warm-up, the mean wait of 5 runsafter each of three candidate warm-ups, then the mean and half-width of two outputsover 5, 10, 20 and 40 replications. To see only those lines, run:python trusting_results_low_level.py 2>/dev/nullWith the argument start, it runs the first 15 minutes without a warm-up, to plot:python trusting_results_low_level.py start 2>&1"""importloggingimportsysfromdesflowimport(CapacityConfig,ConsoleEgress,DistributionConfig,DynamicRealtimeEnvironment,DynamicResource,LowLevelModel,Sampler,SimParameter,replicate,)fromdesflow.analysisimportintervallogging.basicConfig(level=logging.INFO,format="%(levelname)s [%(asctime)s] %(message)s")# The 40 packing times measured at the bench, in secondsMEASURED=[1.5,2.5,2.4,1.2,1.3,1.2,1.8,1.4,2.0,0.7,2.7,1.4,1.9,1.4,1.3,1.8,2.0,1.4,1.4,1.9,1.1,0.8,1.7,1.1,0.7,1.1,1.2,0.9,0.8,1.5,2.1,1.3,1.1,1.7,1.9,1.3,1.8,2.2,1.4,1.1,]# fmt: skipPARAMS=SimParameter(sim_id="Shop",arrival={"orders":DistributionConfig(dist="exponential",rate=1.8)},service={"packing":DistributionConfig(dist="empirical",values=MEASURED)},resources={"packers":CapacityConfig(current_cap=3,max_cap=10)},)defpack_order(env,sampler,packers,key,task_id):"""One order, which reports each step to the statistics."""path_id="Shop.service.packing"defpublish(value):env.publish_event(key,value,task="pack_order",instance=task_id)publish({"path_id":path_id,"sim_id":"Shop","service":"packing","status":"queued"})env.stats.flow_started("orders")# the order enters the shopenv.stats.task_queued("pack_order",task_id)withpackers.request()asrequest:yieldrequestpublish({"path_id":path_id,"sim_id":"Shop","service":"packing","status":"started",})env.stats.task_started("pack_order",task_id)yieldenv.timeout(sampler.sample(env.registry.get_config(path_id)))publish({"event_type":"order_packed"})env.stats.task_finished("pack_order",task_id)env.stats.flow_finished("orders")# the order leaves the shopdeforder_generator(env,sampler,packers):arrival=env.registry.get_config("Shop.arrival.orders")task_id=0whileTrue:yieldenv.timeout(sampler.sample(arrival))env.process(pack_order(env,sampler,packers,f"orders-{task_id+1}",task_id))task_id+=1# Records and statistics start after the first 2 minutes; None keeps everythingWARMUP=120.0# seconds, passed to setup_egress in run belowdefrun(warmup=WARMUP,until=3600,factor=0.0):env=DynamicRealtimeEnvironment(factor=factor)env.registry.register_sim_parameter(PARAMS)# Every 10 seconds, wait_mean is the mean wait so far: plot it against sim_tsenv.setup_stats("Shop","typical",interval=10)# The warm-up is set with the sinks: no sink receives a record stamped before it# ends, and the statistics start again from zero when it endsenv.setup_egress([ConsoleEgress()],warmup=warmup)packers=DynamicResource(env,"Shop","packers")env.process(order_generator(env,Sampler(seed=42),packers))try:env.run(until=until)finally:env.teardown()returnenvdefmean_wait(warmup)->float:"""Runs the busy hour and returns the mean wait it measured."""returnrun(warmup).stats.readings()["Shop.stats.tasks.pack_order.wait_mean"]MEASURES=["tasks.pack_order.wait_mean","resources.packers.utilization_avg"]defmodel(env,sampler):"""The busy hour in a given environment: replicate sets the statistics."""env.registry.register_sim_parameter(PARAMS)packers=DynamicResource(env,"Shop","packers")env.process(order_generator(env,sampler,packers))defreplications(n=40):"""Runs the busy hour n times, each after the warm-up, with seeds spawned from 42."""shop=LowLevelModel(model,sim_id="Shop")returnreplicate(shop,n,seed=42,until=3600,warmup=WARMUP,measures=MEASURES)defwarmups(n=5)->dict:"""The mean wait of n runs after each candidate warm-up, with seeds spawned from 42."""shop=LowLevelModel(model,sim_id="Shop")wait=["tasks.pack_order.wait_mean"]return{warmup:replicate(shop,n,seed=42,until=3600,warmup=warmup,measures=wait)forwarmupin(60,WARMUP,240)}defprecision(result)->dict:"""Each output's mean and half-width over the first 5, 10, 20 and 40 replications."""return{(measure,n):interval(result.values[measure][:n])formeasureinMEASURESfornin(5,10,20,40)}WAIT="Shop.stats.tasks.pack_order.wait_mean"PACKED="Shop.stats.tasks.pack_order.finished"QUEUE="Shop.stats.resources.packers.queue_length_avg"defreport()->None:"""Prints the mean waits, the Little's law figures and the precision."""print(f"without a warm-up: mean wait {mean_wait(None):.3f} s")warmed=run(WARMUP).stats.readings()print(f"with a 2 min warm-up: mean wait {warmed[WAIT]:.3f} s")print(f"after the warm-up: {warmed[PACKED]} orders packed, "f"queue_length_avg {warmed[QUEUE]:.2f}")forwarmup,resultinwarmups().items():row=result.summary[0]print(f"warm-up {warmup:g} s: mean wait {row['mean']:.3f} ± {row['half_width']:.2f} s")for(measure,n),resultinprecision(replications()).items():share=result.half_width/result.meanprint(f"{measure}, {n} runs: {result.mean:.2f} ± {result.half_width:.2f} ({share:.1%})")if__name__=="__main__":ifsys.argv[1:]==["start"]:# The statistics every 10 seconds of the first 15 minutes, without a warm-uprun(None,until=900)else:report()