Dagster
Run a desflow study as a Dagster asset, and keep its figures as the asset's metadata. The asset runs the shop through PipesSubprocessClient, then reads the figures from the file the run writes.
Before You Start
- Integrations shows the shop, the command and the figures the result file holds.
-
You need Dagster 1.13 and desflow 1.1 or later in one environment:
docs/snippets/orchestrators/commands.sh:dagster-install -
Put
shop.yamlanddagster_asset.pyin one folder.
Steps
Write the Asset
PipesSubprocessClient runs the command with cwd set to the asset's folder, so it finds shop.yaml and writes result.json beside it. It passes desflow's output to Dagster's logs, and a non-zero exit fails the asset with DagsterPipesExecutionError. When a run is cancelled, it sends SIGINT to desflow and waits up to 20 seconds; desflow then flushes its sinks and writes no result.json.
The asset then opens result.json and returns the three figures as MaterializeResult metadata, one number each, which Dagster shows on the asset's page. Two materializations at the same time would share one result.json, so a deployment that runs them side by side gives each its own file, such as one named by context.run_id. The figures pass through the file because desflow does not report to Dagster itself:
defs holds the asset and its resource for Dagster's own tools: dagster dev -f dagster_asset.py starts Dagster's web server and daemon on your machine with the asset in them, once pip install dagster-webserver has added the web server, which dagster does not install. The __main__ block materializes the asset once with materialize(), which runs in the current process and needs no DAGSTER_HOME, web server or daemon.
Materialize It Once
Run this from the folder that holds the two files:
| docs/snippets/orchestrators/commands.sh:dagster-run | |
|---|---|
Check the Result
Dagster logs each step of the run, with desflow's log lines among them. The last line the script prints is the asset's metadata:
These are the figures of the command run in a terminal. Dagster also logs the warning "[pipes] did not receive any messages from external process", because desflow does not use the dagster-pipes package to report back. The asset is still materialized, with the metadata above.
Sources
The client's options, its failure and its cancel handling are those of PipesSubprocessClient in Dagster 1.13.30 (dagster/_core/pipes/subprocess.py), and materialize() is described in its docstring in dagster/_core/definitions/materialize.py. Dagster Pipes describes the client and the dagster-pipes package.