Prefect
Run a desflow study as a task of a Prefect flow, and return its figures from the flow. The flow has two tasks: one runs the shop, and one reads the figures from the file the run writes.
Before You Start
- Integrations shows the shop, the command and the figures the result file holds.
-
You need Prefect 3.9 and desflow 1.1 or later in one environment:
docs/snippets/orchestrators/commands.sh:prefect-install -
Run the flow with that environment active, because the
run_shoptask starts thedesflowcommand found onPATH. - Put
shop.yamlandprefect_flow.pyin one folder.
Steps
Write the Flow
The run_shop task runs the command with subprocess.run in the flow's folder, so it finds shop.yaml and writes result.json beside it. check=True raises an error on a non-zero exit, which fails the task and the flow. The task returns the path of the file, and the read_result task opens it and returns the three figures. Two flow runs at the same time would share one result.json, so a deployment that runs them side by side gives each its own file, such as one named by the flow run's id. A task called inside a flow returns its value, so the flow returns the figures:
The figures pass through the file because desflow logs to stderr and prints no result in a single run.
Run the Flow Once
Run this from the folder that holds the two files:
| docs/snippets/orchestrators/commands.sh:prefect-run | |
|---|---|
With no Prefect server set in PREFECT_API_URL, Prefect's default profile starts a temporary server for the run and stops it at the end. It keeps the run in a database under ~/.prefect, or under PREFECT_HOME when that is set.
Check the Result
Prefect logs the flow run and each task run, and the script prints the value the flow returns:
These are the figures of the command run in a terminal. The prefect-shell package can also run a command as a task, but subprocess.run needs no package beyond Prefect.
Sources
The default profile, ephemeral, is in prefect/settings/profiles.toml of Prefect 3.9.3, and the default home folder in prefect/settings/models/_defaults.py. Flows and tasks are described in Prefect's Write and run workflows.