Airflow
Run a desflow study as a task of an Airflow DAG, and pass its figures to the next task. The DAG has two tasks: a BashOperator that runs the shop, and a Python task that reads the figures from the file the run writes.
Before You Start
- Integrations shows the shop, the command and the figures the result file holds.
-
You need Airflow 3.4 and desflow 1.1 or later in one environment. Airflow asks to be installed with its constraints file, which names one tested version of each package for your Python:
-
Put
shop.yamlandairflow_dag.pyin the DAGs folder, which is~/airflow/dagsunlessAIRFLOW_HOMEnames another folder for Airflow's files.
Steps
Write the DAG
The run_shop task runs the command with cwd set to the DAG's folder, so it finds shop.yaml and writes result.json beside it. Without cwd, Airflow runs the command in a temporary folder. A non-zero exit fails the task.
The read_result task opens result.json and returns the three figures, which Airflow keeps as the task's XCom value for later tasks. The result passes through the file because BashOperator pushes only the last line of the command's output to XCom, and desflow's last line is a log line:
When Airflow stops a running task, it sends SIGTERM to the command. desflow then flushes its sinks and exits without writing result.json, so read_result does not read a stale file.
Run the DAG Once
dag.test() runs every task of one DAG run in the current process, with no scheduler or API server. It needs Airflow's metadata database, which airflow db migrate creates as a SQLite file in Airflow's home folder. Run these from the DAGs folder:
| docs/snippets/orchestrators/commands.sh:airflow-run | |
|---|---|
Check the Result
Airflow logs each task as it runs, with desflow's log lines inside run_shop's output. The last line the script prints is the value of read_result:
Three packers packed 1775 orders in the hour, and an order waited 0.26 seconds for a packer on average. These are the figures of the command run in a terminal.
In a deployed Airflow, the scheduler runs the DAG instead of python airflow_dag.py, and each worker that runs run_shop needs desflow installed and the two files in its DAGs folder. Passing result.json between the two tasks needs both on one machine, such as the LocalExecutor or a single worker, and a DAGs folder that Airflow can write to. With several workers, run_shop writes its summary to storage that both tasks read, or the two tasks become one task. Two runs of the DAG at the same time share one file, so each run writes its own, such as result-{{ run_id }}.json.
Sources
The operator's options and its XCom value are those of BashOperator in apache-airflow-providers-standard 1.21.0. dag.test() and the metadata database it needs are described in Airflow's Debugging Airflow DAGs, and the constraints file in Installation from PyPI.