# Open Data Stack (odctl) > Run an open data stack on your machine, with its services already connected, from one command. # Home # Open Data Stack (odctl) **Run an open data stack on your machine, with its services already connected, from one command.** odctl is a command-line tool that runs an open data stack on your machine with Docker Compose. You name the [profiles](https://jaehyeon.me/odctl/latest/how-it-works/#profiles) you want, such as `kafka-lite` or `airflow`, and odctl starts them together with the services they depend on. A Compose file of your own is enough for one small project. [Why odctl](https://jaehyeon.me/odctl/latest/about/why-odctl/index.md) explains what odctl adds when several projects need different parts of the stack. ## Commands Each command has its own section in the CLI reference, with its options and examples. The Example column links to a page that runs the command and shows its output, and [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md) runs them in order on one profile. | Command | Use it to | Example | | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | [`odctl list`](https://jaehyeon.me/odctl/latest/cli/#odctl-list) | See every profile and what it runs, as a table or, with `--json`, as JSON | [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#start-a-profile) | | [`odctl explain`](https://jaehyeon.me/odctl/latest/cli/#odctl-explain) | See a profile's services, images, ports and the profiles it depends on, as a panel or, with `--json`, as JSON | [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#start-a-profile) | | [`odctl ps`](https://jaehyeon.me/odctl/latest/cli/#odctl-ps) | See the running containers, their health and their ports, as a table or, with `--json`, as JSON | [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#see-what-is-running) | | [`odctl info`](https://jaehyeon.me/odctl/latest/cli/#odctl-info) | Check the installed odctl version and whether Docker is reachable; with `--json`, also the image tag, the workspace and odctl's own images | [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#see-what-is-running) | | [`odctl init`](https://jaehyeon.me/odctl/latest/cli/#odctl-init) | Copy the compose files and settings into `.odctl`, so you can edit them | [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#create-a-workspace) | | [`odctl pull`](https://jaehyeon.me/odctl/latest/cli/#odctl-pull) | Download a profile's images before you start it | [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#which-images-run) | | [`odctl up`](https://jaehyeon.me/odctl/latest/cli/#odctl-up) | Start profiles and the profiles they depend on | [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#start-a-profile) | | [`odctl down`](https://jaehyeon.me/odctl/latest/cli/#odctl-down) | Stop profiles and remove their containers | [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#stop-a-profile) | | [`odctl logs`](https://jaehyeon.me/odctl/latest/cli/#odctl-logs) | Read the logs of a profile's containers | [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#see-what-is-running) | | [`odctl restart`](https://jaehyeon.me/odctl/latest/cli/#odctl-restart) | Restart a profile's containers and keep them | [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#restart-a-profile) | | [`odctl recreate`](https://jaehyeon.me/odctl/latest/cli/#odctl-recreate) | Replace a profile's containers, to apply an edit to a compose file or `.env` | [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) | | [`odctl mcp`](https://jaehyeon.me/odctl/latest/cli/#odctl-mcp) | Give an AI agent odctl's commands as MCP tools; needs the `mcp` extra | [MCP Server](https://jaehyeon.me/odctl/latest/guides/mcp-server/index.md) | ## What It Runs | Area | Technologies | | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | [Messaging](https://jaehyeon.me/odctl/latest/profiles/messaging/index.md) | Kafka (KRaft), Karapace schema registry, Kafka Connect, Kafka UI (kafbat) | | [Stream and Batch Processing](https://jaehyeon.me/odctl/latest/profiles/stream-and-batch-processing/index.md) | Apache Flink, Apache Spark | | [Analytics](https://jaehyeon.me/odctl/latest/profiles/analytics/index.md) | ClickHouse, Trino, Metabase | | [Orchestration](https://jaehyeon.me/odctl/latest/profiles/orchestration/index.md) | Apache Airflow, Temporal | | [MLOps](https://jaehyeon.me/odctl/latest/profiles/mlops/index.md) | MLflow, Feast, Evidently, Label Studio | | [Metadata and Lineage](https://jaehyeon.me/odctl/latest/profiles/metadata-and-lineage/index.md) | OpenMetadata, Marquez | | [Observability](https://jaehyeon.me/odctl/latest/profiles/observability/index.md) | grafana/otel-lgtm (OpenTelemetry Collector, Prometheus, Tempo, Loki, Grafana) | | [Storage and Catalog](https://jaehyeon.me/odctl/latest/profiles/storage-and-catalog/index.md) | PostgreSQL 18 with pgvector, pg_textsearch and PostGIS, SeaweedFS (S3), Iceberg REST catalog, Valkey Bundle, Apache Fluss | | [AI Engineering](https://jaehyeon.me/odctl/latest/profiles/ai-engineering/index.md) | Langfuse | Kafka Connect comes with a set of connectors, such as the Iceberg sink and the Debezium PostgreSQL source. [Kafka Connect](https://jaehyeon.me/odctl/latest/guides/kafka-connect/index.md) lists them all. ## Where to Go Next - [Why odctl](https://jaehyeon.me/odctl/latest/about/why-odctl/index.md) says why odctl exists and what it is for. - [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md) installs odctl and starts a first profile. - [Profiles](https://jaehyeon.me/odctl/latest/profiles/index.md) lists every profile with its ports, images and memory limits. - [Guides](https://jaehyeon.me/odctl/latest/guides/kafka/index.md) show how to use each service, starting with Kafka. ## Projects That Use odctl - [benchtop](https://github.com/jaehyeon-kim/benchtop): data engineering and machine learning projects, each run from a cold clone on odctl. - [desflow](https://github.com/jaehyeon-kim/desflow) ([docs](https://jaehyeon.me/desflow/)): a SimPy library for simulations that stream to Kafka, PostgreSQL, Redis and Iceberg. - [agentic-analytics-system](https://github.com/jaehyeon-kim/agentic-analytics-system): conversational analytics over an Iceberg lakehouse. Blog posts about odctl and these projects are tagged [odctl](https://jaehyeon.me/tags/odctl/) on jaehyeon.me. # Getting Started # Getting Started This page installs odctl, starts a first profile, runs each command on it and stops it. It needs no earlier page. The recording shows the main steps: install odctl, preview a profile, start it, list the containers and stop them. ## Requirements - Docker Engine or Docker Desktop, running. Give Docker 8 to 16 GB of memory. - Python 3.10 or later. ## Install odctl is a command-line tool, so install it into its own environment with uv or pipx: ```bash uv tool install odctl # or pipx install odctl ``` `pip install odctl` also works. `odctl --help` then lists the commands. ## Create a Workspace Run this in the folder of your project: ```bash odctl init ``` It prints `Workspace initialized!` with the path of the new folder. The command copies the compose files and their settings into `.odctl` in the current directory, and writes `.odctl/.env` with `TAG` set to the CLI version. odctl reads from `.odctl` whenever it exists, so you can edit any file there. [Workspace](https://jaehyeon.me/odctl/latest/how-it-works/#workspace) explains what changes with and without it, and how to customise it. ## Start a Profile A [profile](https://jaehyeon.me/odctl/latest/how-it-works/#profiles) is a named group of services that start together. This lists every profile, with its parent stack and a description: ```bash odctl list ``` Add `-d` to list each profile's services and host ports as well: ```bash odctl list -d ``` Add `--json` to print the profiles as JSON, which a script or an AI assistant reads more easily than a table. odctl prints the JSON on standard output and every warning on standard error, so a pipe receives only the JSON. The warning for a stale workspace is one example. The full output is an array with one object for each profile, and this keeps only `kafka-lite`: ```bash # Print only kafka-lite; the full output lists every profile. odctl list --json | python3 -c 'import json, sys; profiles = {p["profile"]: p for p in json.load(sys.stdin)}; print(json.dumps(profiles["kafka-lite"], indent=4))' ``` ```text { "profile": "kafka-lite", "stack": "kafka", "description": "Distributed event streaming and schema registry. Lite uses 1 Broker; Full uses 3 Brokers.", "compose_file": "compose-kafka.yml", "depends_on": [ "deps" ], "starts": [ "deps", "kafka-lite" ], "services": [ { "name": "connect", "image": "apache/kafka:4.3.2", "ports": [ { "host": 8083, "container": 8083, "protocol": "tcp" } ] }, { "name": "kafka", "image": "apache/kafka:4.3.2", "ports": [ { "host": 9092, "container": 9092, "protocol": "tcp" } ] }, { "name": "kafka-ui", "image": "ghcr.io/kafbat/kafka-ui:v1.5.0", "ports": [ { "host": 8086, "container": 8080, "protocol": "tcp" } ] }, { "name": "karapace", "image": "ghcr.io/aiven-open/karapace:6.2.4", "ports": [ { "host": 8081, "container": 8081, "protocol": "tcp" } ] } ] } ``` Each profile is one object, and `starts` lists the profiles that `odctl up kafka-lite` starts. [JSON Output](https://jaehyeon.me/odctl/latest/json-output/index.md) lists the keys of every command that takes `--json`. This shows what one profile runs: ```bash odctl explain kafka-lite ``` It prints a panel for `kafka-lite`: the parent stack `kafka`, the compose file, the profile it requires (`deps`), the containers `connect`, `kafka`, `kafka-ui` and `karapace` with their images, the host ports and how to reach each service. `--json` prints the same details as one object, with `usage`, `role` and `volumes` added to the keys of `list`. `usage` is text for people to read, not a value to parse: ```bash odctl explain kafka-lite --json | python3 -m json.tool ``` ```text { "profile": "kafka-lite", "stack": "kafka", "description": "Distributed event streaming and schema registry. Lite uses 1 Broker; Full uses 3 Brokers.", "compose_file": "compose-kafka.yml", "depends_on": [ "deps" ], "starts": [ "deps", "kafka-lite" ], "services": [ { "name": "connect", "image": "apache/kafka:4.3.2", "ports": [ { "host": 8083, "container": 8083, "protocol": "tcp" } ] }, { "name": "kafka", "image": "apache/kafka:4.3.2", "ports": [ { "host": 9092, "container": 9092, "protocol": "tcp" } ] }, { "name": "kafka-ui", "image": "ghcr.io/kafbat/kafka-ui:v1.5.0", "ports": [ { "host": 8086, "container": 8080, "protocol": "tcp" } ] }, { "name": "karapace", "image": "ghcr.io/aiven-open/karapace:6.2.4", "ports": [ { "host": 8081, "container": 8081, "protocol": "tcp" } ] } ], "usage": "\u2022 Kafka UI:\n - External (host): http://127.0.0.1:8086\n - Internal (Docker): http://kafka-ui:8080\n\u2022 Schema Registry & Connect:\n - External (host): http://127.0.0.1:8081 / http://127.0.0.1:8083\n\u2022 Bootstrap Servers (1-Node Lite):\n - External (host): 127.0.0.1:9092\n - Internal (Docker): broker-1:19092\n", "role": "KRaft broker(s) with Aiven Karapace for Avro/JSON schema validation and Connect for data ingestion.", "volumes": [] } ``` This shows the profiles `kafka-lite` would start, in order, without starting anything: ```bash odctl up kafka-lite --dry-run ``` The dry run prints which compose files and profiles it would start: Start it: ```bash odctl up kafka-lite ``` odctl starts the profiles `kafka-lite` depends on first, then Kafka. It waits until the services are running, and healthy where they define a health check, then prints `All requested profiles successfully started!`. The [Messaging page](https://jaehyeon.me/odctl/latest/profiles/messaging/#kafka-lite) lists the same addresses as `odctl explain`. ## See What Is Running This prints the installed CLI version and whether Docker is reachable: ```bash odctl info ``` The output has a title line, `Open Data Stack (ODCTL)`, then `CLI Version:` with the version and `Docker Daemon: โœ… Reachable`. With `--json`, it also prints the image tag, the workspace and odctl's own images, and whether each image is already on your machine. `workspace.path` is the `.odctl` folder in the folder where you run odctl; the output below shortens it to `/home/you/project/.odctl`: ```bash odctl info --json | python3 -m json.tool ``` ```text { "cli_version": "", "tag": "", "workspace": { "path": "/home/you/project/.odctl", "exists": true, "stale_tag": null }, "docker": { "reachable": true }, "images": [ { "reference": "ghcr.io/jaehyeon-kim/odctl/airflow:", "local": false }, { "reference": "ghcr.io/jaehyeon-kim/odctl/deps:", "local": true }, { "reference": "ghcr.io/jaehyeon-kim/odctl/mlflow:", "local": false }, { "reference": "ghcr.io/jaehyeon-kim/odctl/postgres:", "local": false }, { "reference": "ghcr.io/jaehyeon-kim/odctl/spark:", "local": false } ] } ``` `stale_tag` is `null` unless the workspace's `TAG` differs from the CLI version and no `TAG` is set in the shell; then it shows the workspace's `TAG`. This lists the containers of a profile, with their state, health and ports: ```bash odctl ps kafka-lite ``` It prints the active profiles, then a table with one row per container: ```text ๐ŸŒŸ Active Profiles: kafka-lite โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”ƒ Container โ”ƒ Service โ”ƒ State โ”ƒ Health โ”ƒ Ports โ”ƒ โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ โ”‚ connect โ”‚ connect โ”‚ running โ”‚ - โ”‚ 8083 โ†’ 8083/tcp โ”‚ โ”‚ kafka โ”‚ kafka โ”‚ running โ”‚ - โ”‚ 9092 โ†’ 9092/tcp โ”‚ โ”‚ kafka-ui โ”‚ kafka-ui โ”‚ running โ”‚ - โ”‚ 8086 โ†’ 8080/tcp โ”‚ โ”‚ karapace โ”‚ karapace โ”‚ running โ”‚ healthy โ”‚ 8081 โ†’ 8081/tcp โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` A dash under Health means the service has no health check. `--json` prints the same containers as one object. Like the table, it needs profile names or `--all`. Each published port gives its host and container port as numbers, and a missing health check is `null`. `active_profiles` lists the profiles you named whose services are all running, with their prerequisites from the same compose file. It never holds `deps`, and it leaves out a profile whose services all belong to another listed profile, such as `spark-lite` beside `spark-full`: ```bash odctl ps kafka-lite --json | python3 -m json.tool ``` ```text { "active_profiles": [ "kafka-lite" ], "containers": [ { "name": "connect", "service": "connect", "state": "running", "health": null, "ports": [ { "host": 8083, "container": 8083, "protocol": "tcp" } ] }, { "name": "kafka", "service": "kafka", "state": "running", "health": null, "ports": [ { "host": 9092, "container": 9092, "protocol": "tcp" } ] }, { "name": "kafka-ui", "service": "kafka-ui", "state": "running", "health": null, "ports": [ { "host": 8086, "container": 8080, "protocol": "tcp" } ] }, { "name": "karapace", "service": "karapace", "state": "running", "health": "healthy", "ports": [ { "host": 8081, "container": 8081, "protocol": "tcp" } ] } ] } ``` This lists every container odctl manages, from every profile, including one-shot containers that have finished, such as `odctl-init-deps`: ```bash odctl ps --all ``` This prints the last three log lines of one service, `connect`: ```bash odctl logs kafka-lite -s connect -n 3 ``` Each line starts with the service name, such as `connect | [2026-10-24 05:14:03,065] INFO [Worker clientId=connect-connect:8083, ...`. Leave out `-s connect` to read every service of the profile, and add `-f` to follow the logs until you press Ctrl+C. This prints the last two lines written between ten minutes and one second ago, each with a timestamp before it: ```bash odctl logs kafka-lite -s connect --since 10m --until 1s -t -n 2 ``` `--since` and `--until` each take a time such as `2026-10-24T05:00:00` or a duration before now such as `10m`. ## Restart a Profile This restarts the profile's containers and keeps them, with their data: ```bash odctl restart kafka-lite ``` It prints `Restarting containers...` and `Restart complete.` A restart does not apply an edit to a compose file or `.env`. `odctl recreate` replaces the containers to apply one, as [Apply a Change](https://jaehyeon.me/odctl/latest/how-it-works/#apply-a-change) explains with [examples](https://jaehyeon.me/odctl/latest/how-it-works/#examples). `odctl pull` downloads a profile's images before you start it, as [Which Images Run](https://jaehyeon.me/odctl/latest/how-it-works/#which-images-run) shows. ## Stop a Profile This prints what `odctl down` would stop, without stopping anything: ```bash odctl down kafka-lite --dry-run ``` With `--volumes`, the dry run also lists the named volumes that `odctl down --volumes` would remove, and adds a line for each container it stops that has unnamed volumes, with their number: ```bash odctl down kafka-lite --volumes --dry-run ``` ```text โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ Dry Run Teardown Plan โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ๐Ÿ›‘ compose-kafka.yml โ””โ”€ ๐Ÿ’ฅ Stop Profile: kafka-lite โ””โ”€ ๐Ÿ—‘ Remove Volume: 3 unnamed volume(s) of connect โ””โ”€ ๐Ÿ—‘ Remove Volume: 3 unnamed volume(s) of kafka ๐Ÿ›‘ compose-deps.yml โ””โ”€ ๐Ÿ’ฅ Stop Profile: deps โ””โ”€ ๐Ÿ—‘ Remove Volume: odctl-shared-deps ``` `kafka-lite` has no named volume of its own, and no other profile has a container, so the only named volume listed is the shared jar volume `odctl-shared-deps`. The dry run reads each container's volumes from Docker, so `connect` and `kafka` show their three unnamed volumes each, and a container with none, such as `karapace`, gets no line. Without Docker the dry run cannot count them, and says so. `odctl down --volumes` removes the shared jar volume with the last profile, and the unnamed volumes with their containers, as [Where Data Lives](https://jaehyeon.me/odctl/latest/how-it-works/#where-data-lives) explains. This stops the profile and removes its containers: ```bash odctl down kafka-lite ``` It prints `Teardown complete.` The data inside the containers goes with them, as [Where Data Lives](https://jaehyeon.me/odctl/latest/how-it-works/#where-data-lives) explains. It leaves the profiles that `kafka-lite` depends on in place, as [Which Containers a Command Touches](https://jaehyeon.me/odctl/latest/how-it-works/#which-containers-a-command-touches) explains. This stops every profile and also removes the named volumes, such as the shared jar volume `odctl-shared-deps`. It asks `Are you sure?` first, and stops only after you answer `y`: ```bash odctl down --all --volumes ``` In a script, `--yes` answers the prompt: ```bash odctl down --all --volumes --yes ``` ## Code in the Guides Every command and file in these pages is kept in the repository under [`docs/snippets`](https://github.com/jaehyeon-kim/odctl/tree/v1.10.0/docs/snippets), one folder per page, and the end-to-end tests run it on every release. A code block whose title names a file is a copy of that file, or of the section after the colon. The links on the published site point at the release that the docs describe. ## Use It with an AI Assistant An AI coding assistant can start and use odctl for you. The [AI Assistants](https://jaehyeon.me/odctl/latest/ai/index.md) section explains the odctl [plugin](https://jaehyeon.me/odctl/latest/ai/plugin/index.md), which installs the [agent skill](https://jaehyeon.me/odctl/latest/ai/agent-skill/index.md) and the [MCP server](https://jaehyeon.me/odctl/latest/guides/mcp-server/index.md) in Claude Code, Codex and Antigravity. The skill tells an assistant which profile to start and how to reach each service, and the server gives it odctl's commands as tools. The site also publishes [`llms.txt`](https://jaehyeon.me/odctl/llms.txt), an index of these pages for assistants. # How It Works # How It Works This page explains how odctl chooses, starts and reaches its services, where their data lives and how to change them. It builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). ## Profiles A profile is a named group of services that start together, such as `kafka-lite` or `airflow`. odctl uses Docker Compose profiles: each service in a compose file lists the profiles it belongs to. `registry.yml` lists every profile, the compose file it lives in and the profiles it depends on. A tool with a small and a clustered form has two profiles, `-lite` and `-full`, such as `kafka-lite` with one broker and `kafka-full` with three. `odctl list` prints every profile, and [Profiles](https://jaehyeon.me/odctl/latest/profiles/index.md) has a page for each one. ## What `odctl up` Does Take `odctl up flink-lite` as an example. Flink keeps its tables in Iceberg, so it needs the Iceberg REST catalog. The catalog keeps its metadata in PostgreSQL and its files in SeaweedFS, an object store with an S3 API. All of them need the shared jars in `deps`. odctl reads these links from `registry.yml` and starts the profiles from left to right: 1. `deps` copies connectors and jars into a shared Docker volume, creates the `odctl` network, and exits. 1. `postgres` and `storage` start, and odctl waits until they are healthy. 1. `catalog` starts once both are healthy. 1. `flink-lite` starts last. This command prints the plan without starting anything: ```bash odctl up flink-lite --dry-run ``` It lists each compose file with its profiles, from `compose-deps.yml` with `deps` to `compose-flink.yml` with `flink-lite`. Each [profile page](https://jaehyeon.me/odctl/latest/profiles/index.md) lists what its profile also starts. ### Dependencies and the Start Order Each profile lists the profiles it needs under `depends_on` in `registry.yml`. odctl follows those lists from the profiles you name, one level at a time, until no new profile appears. It then sorts the result: `deps` first, `postgres` and `storage` next, `catalog` and `valkey` after them, and every other profile last, in alphabetical order. The profiles are then grouped by compose file, in that order, except that a file starts after the files that hold its profiles' dependencies. When the dependencies of two files point at each other, that order cannot hold, and the order above decides. Each file is started with one `docker compose up --wait`. That call returns once every service it started is running, and healthy where it has a health check, so a file starts only after the files before it are ready. `deps` is the exception: it starts attached, because its init container copies the jars and exits, and `--wait` treats an exiting container as a failure. ### Before Anything Starts `odctl up` checks four things before it starts a container. The diagram shows them in order, and the list below explains each one: 1. **Docker is reachable.** If not, it stops. 1. **Every profile you name is in `registry.yml`.** An unknown name stops it, with the name in the error. 1. **Every service can be started.** A service that declares a profile owned by another compose file would never start, so odctl warns about it and carries on. 1. **odctl's own images exist for the tag.** It resolves `TAG` as compose does and asks the registry for each odctl image that is not already on your machine. If the registry says one is missing, odctl stops and names the tag, rather than leaving Docker to fail on one service. If the registry cannot be reached, it starts anyway, so a machine with no network can run images it pulled before. ### Which Containers a Command Touches `down`, `ps`, `logs`, `restart` and `recreate` act only on the profiles you name, not on their dependencies. `odctl down flink-lite` stops Flink and leaves PostgreSQL, SeaweedFS and the catalog running for other profiles. odctl finds the containers by their compose labels: the project is the one the compose file names in its top-level `name:`, such as `odctl-flink`, and the service belongs to one of the named profiles. odctl runs each file as that project, so a `COMPOSE_PROJECT_NAME` set in the shell or in `.env` does not change it. There is one exception. When a profile depends on another profile in the same compose file, both are included, because compose refuses a command on a service whose dependency is left out. `odctl down --volumes` removes the shared `deps` volume and the `odctl` network only when no other profile has a container left. ## How You Reach a Service Every service publishes its ports on the host at `127.0.0.1` only, so a browser or client on your machine uses `http://127.0.0.1:`, and other machines on your network cannot reach them. This works the same on macOS, Linux and WSL2: Docker Desktop forwards the port to the host's `127.0.0.1`, and WSL2 forwards it to Windows' `localhost`. To open a port to your network, remove `127.0.0.1:` from its line in `.odctl/compose-*.yml` and run `odctl recreate `. Each profile page lists the ports. Inside Docker, services reach each other by name on the `odctl` network, for example `http://catalog:8181` or `broker-1:19092` for Kafka. A container of your own reaches them the same way when it joins that network: ```bash docker run --rm --network odctl curlimages/curl:8.17.0 -s http://catalog:8181/v1/config ``` With the `catalog` profile running, it prints the catalog's configuration as JSON, which starts with `{"defaults":`. ## Where Data Lives Most services keep their data inside their containers. PostgreSQL, SeaweedFS, ClickHouse with its Keeper, and Fluss's ZooKeeper are the exception: their images declare data volumes, so Docker gives each of these containers unnamed volumes, which a recreate passes on to the new container. The Kafka image declares three volumes too, so `kafka`, `kafka-1` to `kafka-3` and `connect` get unnamed volumes. Kafka keeps its topics in `/tmp/kafka-logs` inside the container, not in those volumes, so a recreate loses a topic's data. | Command | Containers | Your data | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `odctl restart` | kept | kept | | `odctl recreate` | replaced | lost, except in PostgreSQL, SeaweedFS, ClickHouse with its Keeper, and Fluss's ZooKeeper, which keep their volumes | | `odctl down` | removed | lost; the unnamed volumes stay on disk, unused | | `odctl down --volumes` | removed, with the unnamed volumes and the profile's named volumes such as `spark-events-data`. The `odctl-shared-deps` volume is removed too once no other profile has a container | lost | A database, a bucket, a topic or an Iceberg table therefore lasts until its profile stops, and every `odctl up` is a clean start. `odctl down --volumes` also frees the disk the unnamed volumes hold. The `odctl-shared-deps` volume holds only jars, so keeping it makes the next start faster. The Spark containers (`spark-master`, the workers and `spark-history`) share the `spark-events-data` volume for Spark's event logs. Docker keeps that volume in memory, so it holds nothing once the stack is down. Three profiles read their code from a SeaweedFS bucket, so uploading a file deploys it, and the upload lasts as long as the bucket does. The `storage` profile creates the buckets. | Profile | Prefix | What it holds | How a change is picked up | | ---------- | ------------------------------ | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `airflow` | `s3://airflow/dags` | DAG files | Airflow's S3 DAG bundle checks the prefix every 15 seconds (`refresh_interval` in `compose-orch.yml`), and the next parse of the file picks the change up | | `feast` | `s3://feast/repo` | the feature repository | `feast-ui` and `feast-serve` list the prefix every 15 seconds (`x-feast-repo-sync` in `compose-mlops.yml`), copy it again when it changed, and restart Feast. An empty prefix gives the repository bundled with odctl | | `metadata` | `s3://openmetadata/connectors` | connector modules | `openmetadata-ingestion` lists the prefix every 15 seconds (`openmetadata/connector_sync.py`) and copies it again when it changed, with no restart, because each agent run imports the modules from disk | ## Which Images Run Most services run their project's official image, pinned to an exact version. odctl builds five images of its own and publishes them to `ghcr.io/jaehyeon-kim/odctl/`: | Image | Contents | | ---------- | -------------------------------------------------------------------------------------------------------------------- | | `deps` | Connectors, jars and the Prometheus JMX agent, including the Iceberg REST catalog and the Kafka Connect Iceberg sink | | `postgres` | PostgreSQL 18 with pgvector, pg_textsearch and PostGIS | | `airflow` | Airflow with the MLflow and Feast clients and the model runtimes | | `mlflow` | The MLflow server and model server with the model runtimes | | `spark` | Spark with the Python clients jobs use | These five are tagged with the CLI version through `TAG`. [Workspace](#workspace) explains how `TAG` is set with and without `odctl init`. `odctl up` downloads the images it does not have. To download them first, for example before you go offline, pull a profile and the profiles it depends on: ```bash odctl pull catalog ``` It prints `Fetching images for` each compose file, a `Pulled` line for each image and `All images pulled successfully!`. `odctl pull --all` pulls the images of every profile, which is many GB. To fetch newer images of the same tags as you start a profile, add `--pull`: ```bash odctl up catalog --pull ``` ## Workspace odctl runs from one of two places: the files bundled with the CLI, or a workspace that `odctl init` copies them into. ### With and Without `odctl init` | | Without a workspace | With a workspace | | ------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | Files odctl reads | the compose files and `registry.yml` inside the installed package | the copies in `.odctl` in the current directory | | Image tag of odctl's own images | the CLI version | the CLI version, written to `.odctl/.env` as `TAG` | | Your changes | none; the package is replaced on every upgrade | any file in `.odctl` | | After a CLI upgrade | new files and images at once | the old files and images, with a warning on every command until `odctl init --force` | A `TAG` set in the shell overrides both. This shows the image the `postgres` profile would run with another tag, without changing the workspace: ```bash TAG=1.1.0 odctl explain postgres | grep odctl/postgres ``` It prints the panel's line for the `postgres` container, with the image `ghcr.io/jaehyeon-kim/odctl/postgres:1.1.0`. odctl looks for `.odctl` in the current directory. `--workspace PATH` uses the workspace folder at `PATH` instead, such as `../shared/.odctl`, so one workspace can serve several projects. It goes before the command: ```bash odctl --workspace ./.odctl ps catalog ``` ### Customise It `odctl init` copies everything odctl runs into `.odctl`. It copies each top-level file or folder that is missing. A folder that exists gets no new files, and `.env` is written only when it is missing or with `--force`. Running it again therefore keeps your edits. | File | What to change | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `compose-*.yml` | Host ports, memory limits and environment variables of each service | | `registry.yml` | The profiles, their compose files and their dependencies | | `.env` | `TAG`, the time zone of the services that read it (`TZ`, `UTC` by default), extra Python packages for Airflow (`_AIRFLOW_PIP_DEPS`), the MLflow services (`_MLOPS_PIP_DEPS`), the OpenMetadata ingestion container (`_METADATA_PIP_DEPS`), where Airflow sends its OpenLineage events (`AIRFLOW_OPENLINEAGE_TRANSPORT`, empty by default, so Airflow sends none; [Lineage from Airflow](https://jaehyeon.me/odctl/latest/guides/lineage/#lineage-from-airflow) sets it), whether OpenMetadata's AutoPilot creates default agents for a service added in its UI (`METADATA_AUTOPILOT`, off by default), and the model the MLflow model server serves (`MODEL_URI`) | | `grafana/dashboards/` | The Grafana dashboard for each service | | `grafana/alerting/` | Grafana alert rules, contact points and notification policies, loaded at start | | `trino/rules.json` | Trino's file-based access control | | `evidently/config.yaml` | The Evidently server's settings | | `openmetadata/connector_sync.py` | How the ingestion container copies custom connectors from S3 | | `openmetadata/autopilot_init.py` | How the `openmetadata-init` service sets AutoPilot from `METADATA_AUTOPILOT` | ### Apply a Change A restart keeps the container, so it does not apply an edited compose file or `.env`. Run `odctl recreate ` to replace the profile's containers from their current definition. Add `--pull` to fetch the images again first: ```bash odctl recreate catalog --pull ``` Recreating discards the data inside those containers, with the exceptions that [Where Data Lives](#where-data-lives) lists. `odctl restart ` is enough for files a service reads when it starts, such as `trino/rules.json`, the Grafana dashboards and the Grafana alert rules. Removing an alerting file needs `odctl down telemetry` and then `odctl up telemetry`, as [Add an Alert Rule](https://jaehyeon.me/odctl/latest/guides/telemetry/#add-an-alert-rule) explains. ### Examples Each example edits the workspace, then applies the change. `sed -i.bak` works with both GNU and macOS `sed`, and keeps the original as a `.bak` file. Give SeaweedFS 2 GB instead of 1 GB, for example when several projects share it. SeaweedFS is a Go program, and `GOMEMLIMIT` is a soft memory limit: as the memory the Go runtime manages nears it, the garbage collector runs more often and returns memory to the system sooner. It is three quarters of `mem_limit`, so change both: ```bash sed -i.bak -e '/^ seaweed:/,/^ [a-z]/ s/mem_limit: 1024m/mem_limit: 2g/' \ -e '/^ seaweed:/,/^ [a-z]/ s/GOMEMLIMIT: 768MiB/GOMEMLIMIT: 1536MiB/' .odctl/compose-infra.yml odctl recreate storage docker inspect seaweed --format '{{.HostConfig.Memory}}' ``` The last command prints `2147483648`, which is 2 GB in bytes. Publish Trino on port 8180 instead of 8080, when another program already uses 8080: ```bash sed -i.bak 's/"127.0.0.1:8080:8080" # Trino/"127.0.0.1:8180:8080" # Trino/' .odctl/compose-analytics.yml odctl recreate trino docker port trino ``` The last command prints `8080/tcp -> 127.0.0.1:8180`. The shipped `trino/rules.json` grants every identity full table privileges and denies `analyst` schema ownership. Add your own table rules there for row filtering and column masking, then restart Trino. Install extra Python packages in the Airflow container, for DAGs that import them: ```bash sed -i.bak 's/^_AIRFLOW_PIP_DEPS=.*/_AIRFLOW_PIP_DEPS="dateparser==1.4.3"/' .odctl/.env odctl recreate airflow docker exec airflow python -c "import dateparser; print(dateparser.__version__)" ``` The last command prints `1.4.3`. Install an extra Python package in the OpenMetadata ingestion container, for a custom connector that imports it: ```bash sed -i.bak 's/^_METADATA_PIP_DEPS=.*/_METADATA_PIP_DEPS="dateparser==1.4.3"/' .odctl/.env odctl recreate metadata docker exec openmetadata-ingestion python -c "import dateparser; print(dateparser.__version__)" ``` The last command prints `1.4.3`. Turn on OpenMetadata's AutoPilot, which creates and runs default agents for a service added in the OpenMetadata UI and is off by default. The `openmetadata-init` service applies the key each time `metadata` starts, and prints the result: ```bash sed -i.bak 's/^METADATA_AUTOPILOT=.*/METADATA_AUTOPILOT="true"/' .odctl/.env odctl recreate metadata docker logs openmetadata-init 2>&1 | tail -1 ``` The last command prints `AutoPilot active: true`. Install an extra Python package in the MLflow containers, for a model that imports it: ```bash sed -i.bak 's/^_MLOPS_PIP_DEPS=.*/_MLOPS_PIP_DEPS="tabulate==0.9.0"/' .odctl/.env odctl recreate mlflow docker exec mlflow-serve python -c "import tabulate; print(tabulate.__version__)" ``` The last command prints `0.9.0`. Each key installs its packages for its own containers: - `_AIRFLOW_PIP_DEPS`: the Airflow containers. - `_MLOPS_PIP_DEPS`: the `mlflow` and `mlflow-serve` containers, for example a library a served model needs to load. - `_METADATA_PIP_DEPS`: the `openmetadata-ingestion` container, for example a library a custom connector imports. All three install their packages from PyPI again in every new container: at the first `odctl up`, after `odctl recreate`, and after `odctl down` and `odctl up`. `odctl restart` keeps the container, so the packages stay. Each new container needs PyPI, and pinned versions (`transformers==5.20.2`) keep the installed packages the same. Set the time zone of the services that read `TZ`, here the MLflow containers: ```bash sed -i.bak 's/^TZ=.*/TZ=Australia\/Sydney/' .odctl/.env odctl recreate mlflow docker exec mlflow date +%Z ``` The last command prints `AEDT` from October to April and `AEST` for the rest of the year. PostgreSQL reads `TZ` only when it creates its data folder, and a recreate keeps that folder, so its time zone changes after `odctl down postgres` and `odctl up postgres`. `MODEL_URI` in `.env` sets the model `mlflow-serve` serves, as [MLflow Tracking and Model Serving](https://jaehyeon.me/odctl/latest/guides/mlflow/#serve-the-model) shows. Create a bucket before its profile exists, for example for a project that has no profile. `seaweed-init` creates a fixed list of buckets, and `.odctl/seaweed/s3.json` gives the login `user` the `Admin` action, which creating a bucket needs. This runs boto3 in a container on the `odctl` network, where `seaweed:8333` resolves, and prints the buckets, which now include `desflow`: ```bash docker run --rm -i --network odctl python:3.12-slim \ sh -c 'pip install -q --root-user-action=ignore boto3==1.43.118 && python -' <<'PY' import boto3 s3 = boto3.client("s3", endpoint_url="http://seaweed:8333", region_name="us-east-1", aws_access_key_id="user", aws_secret_access_key="password") s3.create_bucket(Bucket="desflow") print(sorted(b["Name"] for b in s3.list_buckets()["Buckets"])) PY ``` Two other ways exist. `weed shell` inside the `seaweed` container runs `s3.bucket.create -name desflow`, the command `seaweed-init` runs. Adding the name to `BUCKETS` in `.odctl/compose-infra.yml` creates it at every start of the `storage` profile. An upgrade needs `odctl init --force`, which removes that edit. ### Start Again `odctl init --force` deletes `.odctl` and copies the bundled files again, with `TAG` set to the current CLI version. Your edits are lost, so it asks for confirmation first. In a script, `--yes` answers the prompt: ```bash odctl init --force --yes ``` ## What Stays Stable in 1.x A project that pins `odctl>=1.0,<2` keeps working, because every 1.x release keeps what a script or a project reads from odctl. These seven things stay the same: - The profile names, such as `kafka-lite` ([Profiles](#profiles)). - The host ports that the services publish ([How You Reach a Service](#how-you-reach-a-service)). - The keys of `.odctl/.env` that `odctl init` writes ([Customise It](#customise-it)). - The names of the named volumes, `odctl-shared-deps` and `spark-events-data`, and which services keep their data in a volume ([Where Data Lives](#where-data-lives)). - The files `odctl init` copies into `.odctl` ([Workspace](#workspace)). - The keys of the `--json` output of `list`, `explain`, `ps` and `info` ([JSON Output](https://jaehyeon.me/odctl/latest/json-output/index.md)). - The names of the MCP tools and of their parameters ([MCP Server](https://jaehyeon.me/odctl/latest/guides/mcp-server/index.md)). Removing or renaming a profile, an `.env` key, a named volume, a workspace file, a JSON key, an MCP tool or an MCP tool's parameter, changing a published port's number, or changing the type of a JSON value, waits for 2.0. Adding a profile, a port, an `.env` key, a JSON key, an MCP tool or an optional MCP parameter breaks nothing. Any other change of behaviour gets a line under Upgrading in the [Changelog](https://jaehyeon.me/odctl/latest/about/changelog/index.md). # Profiles # Profiles A profile is what `odctl up` starts. These pages are generated from `registry.yml` and the compose files when the site is built. ## [Messaging](https://jaehyeon.me/odctl/latest/profiles/messaging/index.md) | Profile | Technology | Description | | ------------------------------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------- | | [`kafka-lite`](https://jaehyeon.me/odctl/latest/profiles/messaging/#kafka-lite) | Kafka | Distributed event streaming and schema registry. Lite uses 1 Broker; Full uses 3 Brokers. | | [`kafka-full`](https://jaehyeon.me/odctl/latest/profiles/messaging/#kafka-full) | Kafka | Distributed event streaming and schema registry. Lite uses 1 Broker; Full uses 3 Brokers. | ## [Stream and Batch Processing](https://jaehyeon.me/odctl/latest/profiles/stream-and-batch-processing/index.md) | Profile | Technology | Description | | ------------------------------------------------------------------------------------------------- | ------------ | ------------------------------------------------------------------------------------------------ | | [`flink-lite`](https://jaehyeon.me/odctl/latest/profiles/stream-and-batch-processing/#flink-lite) | Apache Flink | Stateful, real-time stream processing engine. Lite uses 1 TaskManager; Full uses 3 TaskManagers. | | [`flink-full`](https://jaehyeon.me/odctl/latest/profiles/stream-and-batch-processing/#flink-full) | Apache Flink | Stateful, real-time stream processing engine. Lite uses 1 TaskManager; Full uses 3 TaskManagers. | | [`spark-lite`](https://jaehyeon.me/odctl/latest/profiles/stream-and-batch-processing/#spark-lite) | Apache Spark | Distributed batch and micro-batch data processing. Lite uses 1 Worker; Full uses 3 Workers. | | [`spark-full`](https://jaehyeon.me/odctl/latest/profiles/stream-and-batch-processing/#spark-full) | Apache Spark | Distributed batch and micro-batch data processing. Lite uses 1 Worker; Full uses 3 Workers. | ## [Analytics](https://jaehyeon.me/odctl/latest/profiles/analytics/index.md) | Profile | Technology | Description | | --------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------- | | [`ch-lite`](https://jaehyeon.me/odctl/latest/profiles/analytics/#ch-lite) | ClickHouse | Real-time OLAP database (ClickHouse), Federated SQL engine (Trino), and BI dashboards (Metabase). | | [`ch-full`](https://jaehyeon.me/odctl/latest/profiles/analytics/#ch-full) | ClickHouse | Real-time OLAP database (ClickHouse), Federated SQL engine (Trino), and BI dashboards (Metabase). | | [`trino`](https://jaehyeon.me/odctl/latest/profiles/analytics/#trino) | Trino | Real-time OLAP database (ClickHouse), Federated SQL engine (Trino), and BI dashboards (Metabase). | | [`metabase`](https://jaehyeon.me/odctl/latest/profiles/analytics/#metabase) | Metabase | Real-time OLAP database (ClickHouse), Federated SQL engine (Trino), and BI dashboards (Metabase). | ## [Orchestration](https://jaehyeon.me/odctl/latest/profiles/orchestration/index.md) | Profile | Technology | Description | | ------------------------------------------------------------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------- | | [`airflow`](https://jaehyeon.me/odctl/latest/profiles/orchestration/#airflow) | Apache Airflow | Workflow orchestration and pipeline scheduling (Airflow), and durable workflows written in code (Temporal). | | [`temporal`](https://jaehyeon.me/odctl/latest/profiles/orchestration/#temporal) | Temporal | Workflow orchestration and pipeline scheduling (Airflow), and durable workflows written in code (Temporal). | ## [MLOps](https://jaehyeon.me/odctl/latest/profiles/mlops/index.md) | Profile | Technology | Description | | ------------------------------------------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`mlflow`](https://jaehyeon.me/odctl/latest/profiles/mlops/#mlflow) | MLflow | Experiment tracking, model registry and model serving (MLflow), the feature store (Feast), ML monitoring (Evidently), and data labelling (Label Studio). | | [`feast`](https://jaehyeon.me/odctl/latest/profiles/mlops/#feast) | Feast | Experiment tracking, model registry and model serving (MLflow), the feature store (Feast), ML monitoring (Evidently), and data labelling (Label Studio). | | [`evidently`](https://jaehyeon.me/odctl/latest/profiles/mlops/#evidently) | Evidently | Experiment tracking, model registry and model serving (MLflow), the feature store (Feast), ML monitoring (Evidently), and data labelling (Label Studio). | | [`label-studio`](https://jaehyeon.me/odctl/latest/profiles/mlops/#label-studio) | Label Studio | Experiment tracking, model registry and model serving (MLflow), the feature store (Feast), ML monitoring (Evidently), and data labelling (Label Studio). | ## [Metadata and Lineage](https://jaehyeon.me/odctl/latest/profiles/metadata-and-lineage/index.md) | Profile | Technology | Description | | -------------------------------------------------------------------------------------- | ------------ | ------------------------------------------------- | | [`metadata`](https://jaehyeon.me/odctl/latest/profiles/metadata-and-lineage/#metadata) | OpenMetadata | Data catalog, discovery, and governance platform. | | [`lineage`](https://jaehyeon.me/odctl/latest/profiles/metadata-and-lineage/#lineage) | Marquez | System telemetry, monitoring, and data lineage. | ## [Observability](https://jaehyeon.me/odctl/latest/profiles/observability/index.md) | Profile | Technology | Description | | --------------------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------- | | [`telemetry`](https://jaehyeon.me/odctl/latest/profiles/observability/#telemetry) | Telemetry (grafana/otel-lgtm) | System telemetry, monitoring, and data lineage. | ## [Storage and Catalog](https://jaehyeon.me/odctl/latest/profiles/storage-and-catalog/index.md) | Profile | Technology | Description | | ------------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | [`postgres`](https://jaehyeon.me/odctl/latest/profiles/storage-and-catalog/#postgres) | PostgreSQL | PostgreSQL metadata backend. | | [`storage`](https://jaehyeon.me/odctl/latest/profiles/storage-and-catalog/#storage) | SeaweedFS | SeaweedFS S3-compatible object storage. | | [`catalog`](https://jaehyeon.me/odctl/latest/profiles/storage-and-catalog/#catalog) | Iceberg REST Catalog | Iceberg REST Catalog. | | [`valkey`](https://jaehyeon.me/odctl/latest/profiles/storage-and-catalog/#valkey) | Valkey | Stream storage (Fluss) and multi-model data store (Valkey Bundle). Valkey includes advanced JSON, Bloom, and Vector Search capabilities. | | [`fluss`](https://jaehyeon.me/odctl/latest/profiles/storage-and-catalog/#fluss) | Apache Fluss | Stream storage (Fluss) and multi-model data store (Valkey Bundle). Valkey includes advanced JSON, Bloom, and Vector Search capabilities. | | [`deps`](https://jaehyeon.me/odctl/latest/profiles/storage-and-catalog/#deps) | Shared Dependencies | Shared dependencies, JARs, connectors, and Docker network | ## [AI Engineering](https://jaehyeon.me/odctl/latest/profiles/ai-engineering/index.md) | Profile | Technology | Description | | -------------------------------------------------------------------------------- | ---------- | ----------------------------------------------------- | | [`langfuse`](https://jaehyeon.me/odctl/latest/profiles/ai-engineering/#langfuse) | Langfuse | Tracing and evaluation of LLM and agent applications. | # AI Engineering ## Langfuse Profiles: `langfuse`. Stack `aieng`, defined in `compose-aieng.yml`. ### langfuse Depends on: `postgres`, `storage`, `ch-lite`, `valkey`. `odctl up langfuse` also starts: `ch-lite`, `deps`, `postgres`, `storage`, `valkey`. ```text โ€ข Langfuse UI (Credentials: user@odctl.io / password): - External (host): http://127.0.0.1:3005 - Internal (Docker): http://langfuse-web:3000 โ€ข OTLP endpoint for traces, over HTTP: - External (host): http://127.0.0.1:3005/api/public/otel - Internal (Docker): http://langfuse-web:3000/api/public/otel โ€ข Project keys: public key pk-lf-odctl, secret key sk-lf-odctl, sent as HTTP Basic auth. Send the header x-langfuse-ingestion-version: 4 so Langfuse shows a trace at once, not up to 10 minutes later. โ€ข Starts ch-lite, so it cannot run beside ch-full. ``` | Service | Image | Host:container ports | Memory limit | | ----------------- | --------------------------------- | -------------------- | ------------ | | `langfuse-web` | `langfuse/langfuse:4.72.0` | `3005:3000` | `2g` | | `langfuse-worker` | `langfuse/langfuse-worker:4.72.0` | none | `2g` | # Analytics ## ClickHouse Profiles: `ch-lite`, `ch-full`. Stack `analytics`, defined in `compose-analytics.yml`. ### ch-lite Depends on: `deps`. `odctl up ch-lite` also starts: `deps`. ```text โ€ข ClickHouse HTTP (Shard 1 / Replica 1 & 2): * External (host): http://127.0.0.1:8123, http://127.0.0.1:8124 * Internal (Docker): http://ch-11:8123, http://ch-12:8123 ``` | Service | Image | Host:container ports | Memory limit | | ---------------- | ------------------------------------------- | ------------------------ | ------------ | | `ch-keeper` | `clickhouse/clickhouse-keeper:25.12-alpine` | `9181:9181` | `512m` | | `ch-shard2-stub` | `alpine:3.24` | none | `64m` | | `ch-11` | `clickhouse/clickhouse-server:25.12-alpine` | `8123:8123`, `9000:9000` | `4g` | | `ch-12` | `clickhouse/clickhouse-server:25.12-alpine` | `8124:8123`, `9001:9000` | `4g` | ### ch-full Depends on: `deps`. `odctl up ch-full` also starts: `deps`. ```text โ€ข ClickHouse HTTP (Shards 1 & 2 / Replicas 1 & 2): * External (host): http://127.0.0.1:8123, http://127.0.0.1:8124, http://127.0.0.1:8125, http://127.0.0.1:8126 * Internal (Docker): http://ch-11:8123, http://ch-12:8123, http://ch-21:8123, http://ch-22:8123 ``` | Service | Image | Host:container ports | Memory limit | | ------------ | ------------------------------------------- | ------------------------ | ------------ | | `ch-keeper` | `clickhouse/clickhouse-keeper:25.12-alpine` | `9181:9181` | `512m` | | `ch-11-full` | `clickhouse/clickhouse-server:25.12-alpine` | `8123:8123`, `9000:9000` | `4g` | | `ch-12-full` | `clickhouse/clickhouse-server:25.12-alpine` | `8124:8123`, `9001:9000` | `4g` | | `ch-21` | `clickhouse/clickhouse-server:25.12-alpine` | `8125:8123`, `9002:9000` | `4g` | | `ch-22` | `clickhouse/clickhouse-server:25.12-alpine` | `8126:8123`, `9003:9000` | `4g` | ## Trino Profiles: `trino`. Stack `analytics`, defined in `compose-analytics.yml`. ### trino Depends on: `postgres`. `odctl up trino` also starts: `deps`, `postgres`. ```text โ€ข Trino UI & JDBC connection (no password: no authenticator is configured, so any username is accepted): * External (host): http://127.0.0.1:8080 * Internal (Docker): http://trino:8080 โ€ข Identities: `docker exec trino trino --execute ''` runs the query as `trino`, the CLI default. Pass `--user user` for the stack-wide convention, or `--user analyst`, which trino/rules.json denies schema ownership while leaving every table privilege in place. โ€ข Catalogs: postgres works with the trino profile alone. iceberg, kafka, clickhouse and valkey load without their backing services, so they appear in SHOW CATALOGS and then fail at query time unless catalog, storage, kafka-lite, ch-lite or valkey is also up. ``` | Service | Image | Host:container ports | Memory limit | | ------- | ------------------- | -------------------- | ------------ | | `trino` | `trinodb/trino:484` | `8080:8080` | `4g` | ## Metabase Profiles: `metabase`. Stack `analytics`, defined in `compose-analytics.yml`. ### metabase Depends on: `postgres`. `odctl up metabase` also starts: `deps`, `postgres`. ```text โ€ข Metabase Dashboard Engine: * External (host): http://127.0.0.1:3000 * Internal (Docker): http://metabase:3000 ``` | Service | Image | Host:container ports | Memory limit | | ---------- | ----------------------------- | -------------------- | ------------ | | `metabase` | `metabase/metabase:v0.64.3.5` | `3000:3000` | `1500m` | # Messaging ## Kafka Profiles: `kafka-lite`, `kafka-full`. Stack `kafka`, defined in `compose-kafka.yml`. ### kafka-lite Depends on: `deps`. `odctl up kafka-lite` also starts: `deps`. ```text โ€ข Kafka UI: - External (host): http://127.0.0.1:8086 - Internal (Docker): http://kafka-ui:8080 โ€ข Schema Registry & Connect: - External (host): http://127.0.0.1:8081 / http://127.0.0.1:8083 โ€ข Bootstrap Servers (1-Node Lite): - External (host): 127.0.0.1:9092 - Internal (Docker): broker-1:19092 ``` | Service | Image | Host:container ports | Memory limit | | ---------- | ----------------------------------- | -------------------- | ------------ | | `kafka` | `apache/kafka:4.3.2` | `9092:9092` | `1500m` | | `connect` | `apache/kafka:4.3.2` | `8083:8083` | `2g` | | `karapace` | `ghcr.io/aiven-open/karapace:6.2.4` | `8081:8081` | `512m` | | `kafka-ui` | `ghcr.io/kafbat/kafka-ui:v1.5.0` | `8086:8080` | `512m` | ### kafka-full Depends on: `deps`. `odctl up kafka-full` also starts: `deps`. ```text โ€ข Kafka UI: - External (host): http://127.0.0.1:8086 - Internal (Docker): http://kafka-ui:8080 โ€ข Schema Registry & Connect: - External (host): http://127.0.0.1:8081 / http://127.0.0.1:8083 โ€ข Bootstrap Servers (3-Node Cluster): - External (host): 127.0.0.1:9092, 127.0.0.1:9093, 127.0.0.1:9094 - Internal (Docker): broker-1:19092, broker-2:19093, broker-3:19094 ``` | Service | Image | Host:container ports | Memory limit | | ---------- | ----------------------------------- | -------------------- | ------------ | | `kafka-1` | `apache/kafka:4.3.2` | `9092:9092` | `1500m` | | `kafka-2` | `apache/kafka:4.3.2` | `9093:9093` | `1500m` | | `kafka-3` | `apache/kafka:4.3.2` | `9094:9094` | `1500m` | | `connect` | `apache/kafka:4.3.2` | `8083:8083` | `2g` | | `karapace` | `ghcr.io/aiven-open/karapace:6.2.4` | `8081:8081` | `512m` | | `kafka-ui` | `ghcr.io/kafbat/kafka-ui:v1.5.0` | `8086:8080` | `512m` | # Metadata and Lineage ## OpenMetadata Profiles: `metadata`. Stack `metadata`, defined in `compose-metadata.yml`. ### metadata Depends on: `postgres`, `storage`. `odctl up metadata` also starts: `deps`, `postgres`, `storage`. ```text โ€ข OpenMetadata UI & REST API (Credentials: admin@odctl.io / password, or user@odctl.io / password, on a new database; a database from 1.3.0 to 1.6.0 keeps admin@odctl.io / admin, and one from an earlier release keeps admin@open-data.local and user@open-data.local, both with the password admin): - External (host): http://127.0.0.1:8585 - Internal (Docker): http://openmetadata-server:8585 โ€ข Ingestion Pipeline UI (Credentials: user / password): - External (host): http://127.0.0.1:8087 - Internal (Docker): http://openmetadata-ingestion:8080 โ€ข Custom connectors: files in s3://openmetadata/connectors, checked every 15 seconds and copied into the ingestion container when they change. โ€ข AutoPilot (default agents for a service added in the UI) is off unless METADATA_AUTOPILOT="true" is set in .odctl/.env. ``` | Service | Image | Host:container ports | Memory limit | | ------------------------ | ----------------------------------------------------- | ------------------------ | ------------ | | `openmetadata-es` | `docker.elastic.co/elasticsearch/elasticsearch:9.3.0` | `9200:9200`, `9300:9300` | `2g` | | `openmetadata-migrate` | `docker.getcollate.io/openmetadata/server:2.0.8` | none | `1g` | | `openmetadata-server` | `docker.getcollate.io/openmetadata/server:2.0.8` | `8585:8585`, `8586:8586` | `2g` | | `openmetadata-ingestion` | `docker.getcollate.io/openmetadata/ingestion:2.0.8` | `8087:8080` | `1500m` | | `openmetadata-init` | `docker.getcollate.io/openmetadata/ingestion:2.0.8` | none | `64m` | ## Marquez Profiles: `lineage`. Stack `obsv`, defined in `compose-obsv.yml`. ### lineage Depends on: `postgres`. `odctl up lineage` also starts: `deps`, `postgres`. ```text โ€ข Marquez API Metadata Endpoint: * External (host): http://127.0.0.1:5002 * Internal (Docker): http://marquez-api:5000 โ€ข Marquez Lineage Graph Web UI: * External (host): http://127.0.0.1:3003 * Internal (Docker): http://marquez-web:3000 ``` | Service | Image | Host:container ports | Memory limit | | ------------- | ----------------------------------- | ------------------------ | ------------ | | `marquez-api` | `marquezproject/marquez:0.51.1` | `5002:5000`, `5001:5001` | `1g` | | `marquez-web` | `marquezproject/marquez-web:0.51.1` | `3003:3000` | `512m` | # MLOps ## MLflow Profiles: `mlflow`. Stack `mlops`, defined in `compose-mlops.yml`. ### mlflow Depends on: `postgres`, `storage`. `odctl up mlflow` also starts: `deps`, `postgres`, `storage`. ```text โ€ข MLflow Tracking UI & Artifact Registry Server: * External (host): http://127.0.0.1:5004 * Internal (Docker): http://mlflow:5000 โ€ข MLflow Model Scoring Server, idle until MODEL_URI in .env names a model; run `odctl recreate mlflow` after setting it. It loads the model once, when it starts, so a new version behind the same alias needs `odctl restart mlflow`, and the server answers nothing for about 16 seconds: * External (host): http://127.0.0.1:5003/invocations * Internal (Docker): http://mlflow-serve:5003/invocations โ€ข Liveness: http://127.0.0.1:5003/ping and http://127.0.0.1:5003/health ``` | Service | Image | Host:container ports | Memory limit | | -------------- | ------------------------------------------------- | -------------------- | ------------ | | `mlflow` | `ghcr.io/jaehyeon-kim/odctl/mlflow:` | `5004:5000` | `2500m` | | `mlflow-serve` | `ghcr.io/jaehyeon-kim/odctl/mlflow:` | `5003:5003` | `1500m` | ## Feast Profiles: `feast`. Stack `mlops`, defined in `compose-mlops.yml`. ### feast Depends on: `catalog`, `valkey`. `odctl up feast` also starts: `catalog`, `deps`, `postgres`, `storage`, `valkey`. ```text โ€ข Feast UI (registry browser, not data): * External (host): http://127.0.0.1:8890 * Internal (Docker): http://feast-ui:8888 โ€ข Upload your feature repository to s3://feast/repo. The services check it every 15 seconds and restart Feast when it changes. Run `feast apply` from your own environment. odctl ships no definitions. โ€ข Your environment needs: feast[duckdb,iceberg,redis,postgres] โ€ข On an IcebergSource against catalog, pass warehouse="" and catalog_name="odctl". The REST client puts warehouse in the URL path, and Iceberg's REST fixture server serves the spec with no prefix. โ€ข Online feature server for get_online_features: * External (host): http://127.0.0.1:6566 * Internal (Docker): http://feast-serve:6566 โ€ข MCP endpoint, for agents that list Feast's tools and read features: http://127.0.0.1:6566/mcp โ€ข Liveness: http://127.0.0.1:6566/health ``` | Service | Image | Host:container ports | Memory limit | | ------------- | ---------------------------------------- | -------------------- | ------------ | | `feast-ui` | `quay.io/feastdev/feature-server:0.68.0` | `8890:8888` | `1g` | | `feast-serve` | `quay.io/feastdev/feature-server:0.68.0` | `6566:6566` | `1g` | ## Evidently Profiles: `evidently`. Stack `mlops`, defined in `compose-mlops.yml`. ### evidently Depends on: `postgres`. `odctl up evidently` also starts: `deps`, `postgres`. ```text โ€ข Evidently UI & API (no login, and writes are open): * External (host): http://127.0.0.1:8089 * Internal (Docker): http://evidently:8000 โ€ข Push reports with RemoteWorkspace("http://127.0.0.1:8089") from the host, or RemoteWorkspace("http://evidently:8000") from a container. Reports are computed by the client, so it needs evidently==0.7.23 and pandas<3.1. In Airflow, set _AIRFLOW_PIP_DEPS="evidently==0.7.23" in .env. โ€ข Projects, reports, dashboards and datasets: the evidently database on Postgres. โ€ข Settings: evidently/config.yaml in the workspace. ``` | Service | Image | Host:container ports | Memory limit | | ----------- | ------------------------------------ | -------------------- | ------------ | | `evidently` | `evidently/evidently-service:0.7.23` | `8089:8000` | `1g` | ## Label Studio Profiles: `label-studio`. Stack `mlops`, defined in `compose-mlops.yml`. ### label-studio Depends on: `postgres`. `odctl up label-studio` also starts: `deps`, `postgres`. ```text โ€ข Label Studio UI & API (user user@odctl.io, password password): * External (host): http://127.0.0.1:8088 * Internal (Docker): http://label-studio:8080 โ€ข API token: odctl-label-studio-token, sent as the header Authorization: Token odctl-label-studio-token โ€ข Projects, tasks and annotations: the labelstudio database on Postgres. โ€ข S3 storage: the label-studio bucket on SeaweedFS (start storage), with the endpoint field left empty and pre-signed URLs off. ``` | Service | Image | Host:container ports | Memory limit | | -------------- | --------------------------------- | -------------------- | ------------ | | `label-studio` | `heartexlabs/label-studio:1.23.2` | `8088:8080` | `768m` | # Observability ## Telemetry (grafana/otel-lgtm) Profiles: `telemetry`. Stack `obsv`, defined in `compose-obsv.yml`. ### telemetry Depends on: `deps`. `odctl up telemetry` also starts: `deps`. ```text โ€ข Prometheus Time Series Engine (also takes OTLP at /api/v1/otlp/v1/metrics): * External (host): http://127.0.0.1:19090 * Internal (Docker): http://prometheus:9090 โ€ข Grafana Dashboards (Credentials: user / password): * External (host): http://127.0.0.1:3004 * Internal (Docker): http://grafana:3000 * A dashboard per service in the odctl folder, which fills in once that service's profile is up. * Alert rules from .odctl/grafana/alerting, loaded when telemetry starts: one fires after a failed Airflow DAG run. No contact point is configured, so a firing rule shows only on the Alerting page. โ€ข OTLP Receiver (metrics, traces and logs): * External (host): 127.0.0.1:4317 (gRPC), http://127.0.0.1:4318 (HTTP) * Internal (Docker): otel-lgtm:4317 (gRPC), http://otel-lgtm:4318 (HTTP) ``` | Service | Image | Host:container ports | Memory limit | | ----------- | -------------------------- | --------------------------------------------------- | ------------ | | `otel-lgtm` | `grafana/otel-lgtm:0.40.0` | `19090:9090`, `3004:3000`, `4317:4317`, `4318:4318` | `2g` | # Orchestration ## Apache Airflow Profiles: `airflow`. Stack `orch`, defined in `compose-orch.yml`. ### airflow Depends on: `postgres`, `storage`. `odctl up airflow` also starts: `deps`, `postgres`, `storage`. ```text โ€ข Airflow Web UI (Credentials: user / password): * External (host): http://127.0.0.1:8085 * Internal (Docker): http://airflow:8080 โ€ข DAGs: read from s3://airflow/dags every 15 seconds. โ€ข Plugins: copied from s3://airflow/plugins at start. Airflow loads plugins only once, so run `odctl restart airflow` after changing one. ``` | Service | Image | Host:container ports | Memory limit | | --------- | -------------------------------------------------- | -------------------- | ------------ | | `airflow` | `ghcr.io/jaehyeon-kim/odctl/airflow:` | `8085:8080` | `3g` | ## Temporal Profiles: `temporal`. Stack `orch`, defined in `compose-orch.yml`. ### temporal Depends on: `deps`. `odctl up temporal` also starts: `deps`. ```text โ€ข Temporal Web UI: * External (host): http://127.0.0.1:8233 * Internal (Docker): http://temporal:8233 โ€ข gRPC endpoint for SDK clients and workers (namespace "default"): * External (host): 127.0.0.1:7233 * Internal (Docker): temporal:7233 โ€ข Your environment needs: temporalio. odctl runs only the server, so your own process runs the worker that executes workflows and activities. โ€ข Development server: the server, the UI and a SQLite database in one process. History survives `odctl restart` and is lost on `odctl down`, like every odctl service's data. โ€ข Prometheus metrics: http://temporal:9090/metrics, inside the network. ``` | Service | Image | Host:container ports | Memory limit | | ---------- | --------------------------- | ------------------------ | ------------ | | `temporal` | `temporalio/temporal:1.9.1` | `7233:7233`, `8233:8233` | `1g` | # Storage and Catalog ## PostgreSQL Profiles: `postgres`. Stack `postgres`, defined in `compose-infra.yml`. ### postgres Depends on: `deps`. `odctl up postgres` also starts: `deps`. ```text โ€ข Postgres JDBC: - External (host): jdbc:postgresql://127.0.0.1:5432/odctl - Internal (Docker): jdbc:postgresql://postgres:5432/odctl - Credentials: user / password โ€ข Extensions, created in the `vector` database: - vector (pgvector): vector columns and HNSW or IVFFlat indexes - pg_textsearch: BM25 keyword ranking, for hybrid search with pgvector - postgis: geometry and geography types and spatial queries Run CREATE EXTENSION in another database to use them there. ``` | Service | Image | Host:container ports | Memory limit | | ---------- | --------------------------------------------------- | -------------------- | ------------ | | `postgres` | `ghcr.io/jaehyeon-kim/odctl/postgres:` | `5432:5432` | `1g` | ## SeaweedFS Profiles: `storage`. Stack `storage`, defined in `compose-infra.yml`. ### storage Depends on: `deps`. `odctl up storage` also starts: `deps`. ```text โ€ข S3 Browser UI: - External (host): http://127.0.0.1:8889 - Internal (Docker): http://seaweed:8888 โ€ข S3 API Endpoint: - External (host): http://127.0.0.1:8333 - Internal (Docker): http://seaweed:8333 - Default AWS Keys: user / password ``` | Service | Image | Host:container ports | Memory limit | | -------------- | -------------------------- | ------------------------------------- | ------------ | | `seaweed` | `chrislusf/seaweedfs:4.52` | `8333:8333`, `8889:8888`, `9333:9333` | `1024m` | | `seaweed-init` | `chrislusf/seaweedfs:4.52` | none | not set | ## Iceberg REST Catalog Profiles: `catalog`. Stack `catalog`, defined in `compose-infra.yml`. ### catalog Depends on: `postgres`, `storage`. `odctl up catalog` also starts: `deps`, `postgres`, `storage`. ```text โ€ข REST API Endpoint: - External (host): http://127.0.0.1:8181 - Internal (Docker): http://catalog:8181 ``` | Service | Image | Host:container ports | Memory limit | | --------- | ----------------------------------- | -------------------- | ------------ | | `catalog` | `azul/zulu-openjdk:17-jre-headless` | `8181:8181` | `512m` | ## Valkey Profiles: `valkey`. Stack `store`, defined in `compose-store.yml`. ### valkey Depends on: `deps`. `odctl up valkey` also starts: `deps`. ```text โ€ข Valkey (Multi-Model Data Store: Cache, JSON, Vector Search): * External (host): 127.0.0.1:6379 * Internal (Docker): valkey:6379 * Authentication: username 'user', password 'password' ``` | Service | Image | Host:container ports | Memory limit | | -------- | ----------------------------------- | -------------------- | ------------ | | `valkey` | `valkey/valkey-bundle:9.1.4-alpine` | `6379:6379` | `512m` | ## Apache Fluss Profiles: `fluss`. Stack `store`, defined in `compose-store.yml`. ### fluss Depends on: `deps`. `odctl up fluss` also starts: `deps`. ```text โ€ข Fluss Coordinator API: * External (host): 127.0.0.1:9123 * Internal (Docker): fluss-coordinator:9123 โ€ข The servers advertise their container names, which resolve only inside the odctl network, so run Fluss clients in a container, such as Flink. ``` | Service | Image | Host:container ports | Memory limit | | ------------------- | -------------------- | -------------------- | ------------ | | `fluss-zookeeper` | `zookeeper:3.9.6` | none | `512m` | | `fluss-coordinator` | `apache/fluss:1.0.0` | `9123:9123` | `1g` | | `fluss-tablet-1` | `apache/fluss:1.0.0` | `9124:9123` | `2g` | ## Shared Dependencies Profiles: `deps`. Stack `deps`, defined in `compose-deps.yml`. ### deps Depends on: nothing. `odctl up deps` also starts: nothing else. ```text โ€ข Runs automatically on 'odctl up'. โ€ข No persistent UI or ports exposed. ``` | Service | Image | Host:container ports | Memory limit | | ----------- | ----------------------------------------------- | -------------------- | ------------ | | `init-deps` | `ghcr.io/jaehyeon-kim/odctl/deps:` | none | not set | # Stream and Batch Processing ## Apache Flink Profiles: `flink-lite`, `flink-full`. Stack `flink`, defined in `compose-flink.yml`. ### flink-lite Depends on: `catalog`. `odctl up flink-lite` also starts: `catalog`, `deps`, `postgres`, `storage`. ```text โ€ข Flink JobManager UI (1 TaskManager): - External (host): http://127.0.0.1:8082 - Internal (Docker): http://jobmanager:8081 โ€ข SQL Gateway REST: - External (host): http://127.0.0.1:8084 - Internal (Docker): http://flink-sql-gateway:8083 ``` | Service | Image | Host:container ports | Memory limit | | --------------- | ------------------ | -------------------- | ------------ | | `jobmanager` | `flink:2.2-java17` | `8082:8081` | `1500m` | | `taskmanager-a` | `flink:2.2-java17` | none | `3500m` | | `sql-gateway` | `flink:2.2-java17` | `8084:8083` | `1g` | ### flink-full Depends on: `catalog`. `odctl up flink-full` also starts: `catalog`, `deps`, `postgres`, `storage`. ```text โ€ข Flink JobManager UI (3 TaskManagers): - External (host): http://127.0.0.1:8082 - Internal (Docker): http://jobmanager:8081 โ€ข SQL Gateway REST: - External (host): http://127.0.0.1:8084 - Internal (Docker): http://flink-sql-gateway:8083 ``` | Service | Image | Host:container ports | Memory limit | | --------------- | ------------------ | -------------------- | ------------ | | `jobmanager` | `flink:2.2-java17` | `8082:8081` | `1500m` | | `taskmanager-a` | `flink:2.2-java17` | none | `3500m` | | `taskmanager-b` | `flink:2.2-java17` | none | `3500m` | | `taskmanager-c` | `flink:2.2-java17` | none | `3500m` | | `sql-gateway` | `flink:2.2-java17` | `8084:8083` | `1g` | ## Apache Spark Profiles: `spark-lite`, `spark-full`. Stack `spark`, defined in `compose-spark.yml`. ### spark-lite Depends on: `catalog`. `odctl up spark-lite` also starts: `catalog`, `deps`, `postgres`, `storage`. ```text โ€ข Spark Master UI (1 Worker): - External (host): http://127.0.0.1:9080 - Internal (Docker): http://spark-master:8080 โ€ข Master RPC: - External (host): spark://127.0.0.1:7077 - Internal (Docker): spark://spark-master:7077 โ€ข Spark History Server: - External (host): http://127.0.0.1:18080 - Internal (Docker): http://spark-history:18080 ``` | Service | Image | Host:container ports | Memory limit | | ---------------- | ------------------------------------------------ | ------------------------ | ------------ | | `spark-master` | `ghcr.io/jaehyeon-kim/odctl/spark:` | `9080:8080`, `7077:7077` | `1g` | | `spark-worker-1` | `ghcr.io/jaehyeon-kim/odctl/spark:` | `9081:8081` | `2g` | | `spark-history` | `ghcr.io/jaehyeon-kim/odctl/spark:` | `18080:18080` | `1g` | ### spark-full Depends on: `catalog`. `odctl up spark-full` also starts: `catalog`, `deps`, `postgres`, `storage`. ```text โ€ข Spark Master UI (3 Workers): - External (host): http://127.0.0.1:9080 - Internal (Docker): http://spark-master:8080 โ€ข Master RPC: - External (host): spark://127.0.0.1:7077 - Internal (Docker): spark://spark-master:7077 โ€ข Spark History Server: - External (host): http://127.0.0.1:18080 - Internal (Docker): http://spark-history:18080 ``` | Service | Image | Host:container ports | Memory limit | | ---------------- | ------------------------------------------------ | ------------------------ | ------------ | | `spark-master` | `ghcr.io/jaehyeon-kim/odctl/spark:` | `9080:8080`, `7077:7077` | `1g` | | `spark-worker-1` | `ghcr.io/jaehyeon-kim/odctl/spark:` | `9081:8081` | `2g` | | `spark-worker-2` | `ghcr.io/jaehyeon-kim/odctl/spark:` | `9082:8081` | `2g` | | `spark-worker-3` | `ghcr.io/jaehyeon-kim/odctl/spark:` | `9083:8081` | `2g` | | `spark-history` | `ghcr.io/jaehyeon-kim/odctl/spark:` | `18080:18080` | `1g` | # Guides # Guides Each guide starts the profiles one task needs, then shows how to use their services from the host and from a container. The guides are in groups by area, in the same order as the sidebar. A guide may start from a stack that an earlier guide in its group set up, so the pages of a group read in order. ______________________________________________________________________ ## Streaming | Guide | What it shows | | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | [Kafka](https://jaehyeon.me/odctl/latest/guides/kafka/index.md) | the Kafka broker, the schema registry and Kafka UI of `kafka-lite` | | [Kafka Connect](https://jaehyeon.me/odctl/latest/guides/kafka-connect/index.md) | the Kafka Connect worker and the connectors it comes with | | [Flink SQL](https://jaehyeon.me/odctl/latest/guides/flink-sql/index.md) | a Flink SQL job, sent through the SQL Gateway, that reads a Kafka topic and writes totals to PostgreSQL | | [Rolling Windows](https://jaehyeon.me/odctl/latest/guides/rolling-windows/index.md) | features over the last 10 minutes, 1 hour and 1 day, with `OVER` windows | | [Fluss](https://jaehyeon.me/odctl/latest/guides/fluss/index.md) | Fluss tables that Flink writes and reads with low latency | ## Lakehouse and Batch | Guide | What it shows | | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | [Ingest into Iceberg](https://jaehyeon.me/odctl/latest/guides/iceberg-ingestion/index.md) | Kafka Connect, Flink and Spark writing Iceberg tables through one REST catalog | | [Spark](https://jaehyeon.me/odctl/latest/guides/spark/index.md) | Spark jobs on the `iceberg` catalog, and the Spark History Server | ## Analytics | Guide | What it shows | | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | [ClickHouse](https://jaehyeon.me/odctl/latest/guides/clickhouse/index.md) | a ClickHouse cluster with shards, replicas and ClickHouse Keeper | | [Trino and Metabase](https://jaehyeon.me/odctl/latest/guides/trino-metabase/index.md) | one SQL interface over PostgreSQL, Iceberg, ClickHouse, Kafka and Valkey, and charts in Metabase | ## Orchestration | Guide | What it shows | | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | [Airflow DAGs from S3](https://jaehyeon.me/odctl/latest/guides/airflow-dags/index.md) | Airflow reading DAG files that you upload to SeaweedFS | | [Temporal Workers](https://jaehyeon.me/odctl/latest/guides/temporal-workers/index.md) | a worker of your own that runs workflows and activities, and a workflow that waits for an approval | ## ML and AI | Guide | What it shows | | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | [Feast with Iceberg](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/index.md) | a feature store with its offline store on Iceberg and its online store on Valkey | | [Feast Vector Retrieval](https://jaehyeon.me/odctl/latest/guides/feast-vector-retrieval/index.md) | the stored rows whose embeddings are nearest to a query embedding | | [MLflow](https://jaehyeon.me/odctl/latest/guides/mlflow/index.md) | runs and models in the tracking server, and a registered model scored over HTTP | | [Evidently Reports](https://jaehyeon.me/odctl/latest/guides/evidently-reports/index.md) | data drift and data quality reports pushed to the Evidently UI | | [Label Studio](https://jaehyeon.me/odctl/latest/guides/label-studio/index.md) | a labelling project, its tasks and labels, and the labels loaded into PostgreSQL | | [Hybrid Search](https://jaehyeon.me/odctl/latest/guides/hybrid-search/index.md) | vector search and BM25 keyword ranking in PostgreSQL, and their rankings combined | | [Valkey](https://jaehyeon.me/odctl/latest/guides/valkey/index.md) | the Valkey Bundle, with its modules for JSON documents, Bloom filters and vector search | | [Langfuse](https://jaehyeon.me/odctl/latest/guides/langfuse/index.md) | traces and evaluations of an application built on an LLM | ## Metadata and Operations | Guide | What it shows | | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | [OpenLineage and Marquez](https://jaehyeon.me/odctl/latest/guides/lineage/index.md) | OpenLineage events from jobs, and the lineage graph Marquez draws | | [OpenMetadata](https://jaehyeon.me/odctl/latest/guides/openmetadata/index.md) | a data catalogue of the stack's databases, tables and columns | | [Telemetry](https://jaehyeon.me/odctl/latest/guides/telemetry/index.md) | metrics, traces and logs sent with OpenTelemetry and read in Grafana | The MCP server, which gives an AI assistant odctl's commands as tools, is in [AI Assistants](https://jaehyeon.me/odctl/latest/ai/index.md). # Produce and Consume with Kafka The `kafka-lite` profile runs one Kafka broker in KRaft mode, the Karapace schema registry, a [Kafka Connect](https://jaehyeon.me/odctl/latest/guides/kafka-connect/index.md) worker and Kafka UI. KRaft mode means the brokers keep the cluster's metadata themselves, with no ZooKeeper. `kafka-full` runs three brokers with the same services beside them. The end-to-end tests run the commands below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md), which installs odctl and says where the code on this page comes from. Start the profile: ```bash odctl up kafka-lite ``` | Endpoint | Address | | -------------------------------------------- | -------------------------------------------------------------------------- | | Bootstrap servers from the host | `127.0.0.1:9092` (`kafka-full` adds `127.0.0.1:9093` and `127.0.0.1:9094`) | | Bootstrap servers inside the `odctl` network | `broker-1:19092` | | Kafka UI | `http://127.0.0.1:8086` | | Schema registry (Karapace) | `http://127.0.0.1:8081` | | Kafka Connect REST API | `http://127.0.0.1:8083` | ## Create a Topic The Kafka command-line tools are in the broker container, which is `kafka` in `kafka-lite` and `kafka-1` in `kafka-full`. This creates the topic `demo` with one partition: ```bash docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:19092 \ --create --topic demo --partitions 1 --replication-factor 1 ``` The command prints `Created topic demo.` ## Produce and Consume The producer writes five JSON records to `demo`, and the consumer reads them back from the start of the topic: ```bash for i in 1 2 3 4 5; do echo "{\"n\":$i}"; done | docker exec -i kafka \ /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server broker-1:19092 --topic demo docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server broker-1:19092 \ --topic demo --from-beginning --max-messages 5 --timeout-ms 60000 ``` The consumer prints the five records, `{"n":1}` to `{"n":5}`, one per line, and then `Processed a total of 5 messages`. It reads through a consumer group, so its offsets are stored in the broker's `__consumer_offsets` topic. Kafka UI at `http://127.0.0.1:8086` shows the topics, their messages, the consumer groups and the schemas in Karapace. ## Keep Messages Longer The brokers delete old messages. A partition keeps data for one day or up to 500 MB, and a segment is deleted once either limit is passed. Both limits are set today in `.odctl/compose-kafka.yml`, where an edit can change them: `KAFKA_LOG_RETENTION_MS` is `86400000` and `KAFKA_LOG_RETENTION_BYTES` is `524288000`. A topic created without settings takes these values, which this command reads: ```bash docker exec kafka /opt/kafka/bin/kafka-configs.sh --bootstrap-server broker-1:19092 \ --describe --all --entity-type topics --entity-name demo | grep -E '^ *retention\.(ms|bytes)=' ``` The output shows `retention.ms=86400000` and `retention.bytes=524288000`. To keep everything in one topic, set `retention.ms` and `retention.bytes` to `-1` when you create it. `-1` means no limit. Other topics keep the broker defaults. This creates the topic `history` with no limits and describes it: ```bash docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:19092 \ --create --topic history --partitions 1 --replication-factor 1 \ --config retention.ms=-1 --config retention.bytes=-1 docker exec kafka /opt/kafka/bin/kafka-configs.sh --bootstrap-server broker-1:19092 \ --describe --entity-type topics --entity-name history ``` The describe command lists the settings of that topic: `retention.ms=-1` and `retention.bytes=-1`. `retention.bytes` applies to each partition, so a topic with three partitions at 500 MB can hold 1.5 GB. To change an existing topic, use `kafka-configs.sh --alter --add-config` with the same two settings. If older messages are missing, see [Troubleshooting](https://jaehyeon.me/odctl/latest/troubleshooting/#older-messages-are-missing). ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), clients reach the broker at `broker-1:19092`, and `kafka-full` adds `broker-2:19093` and `broker-3:19094`. Karapace is `http://karapace:8081` and Kafka UI is `http://kafka-ui:8080`. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It starts `kafka-lite`, creates `demo` if it does not exist, writes five records and reads five back: ```bash #!/usr/bin/env bash # Starts kafka-lite, creates the topic demo, writes five records and reads them back. set -euo pipefail odctl up kafka-lite docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:19092 \ --create --if-not-exists --topic demo --partitions 1 --replication-factor 1 for i in 1 2 3 4 5; do echo "{\"n\":$i}"; done | docker exec -i kafka \ /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server broker-1:19092 --topic demo docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server broker-1:19092 \ --topic demo --from-beginning --max-messages 5 --timeout-ms 60000 ``` It prints `{"n":1}` to `{"n":5}` and `Processed a total of 5 messages`. When you run it again, it adds five more records and still reads the first five. # Kafka Connect Both Kafka profiles run one Kafka Connect worker, `connect`, beside the brokers. Kafka Connect runs connectors: a source connector writes records from another system into Kafka, and a sink connector writes records from Kafka into another system. The worker loads its connectors from the shared volume that the `deps` profile fills, and you manage them through its REST API. The end-to-end tests run the commands below on every release. This guide builds on [Kafka](https://jaehyeon.me/odctl/latest/guides/kafka/index.md), which starts `kafka-lite` and reads a topic with the console consumer. Start the profile: ```bash odctl up kafka-lite ``` | Setting | Value | | ----------------------------------- | -------------------------------------- | | REST API from the host | `http://127.0.0.1:8083` | | REST API inside the `odctl` network | `http://connect:8083` | | Plugin folder | `/mnt/shared-deps/connect` | | Default converters | JSON with schemas, for keys and values | | Offset flush interval | 10 seconds | The converters and the flush interval are set today in the `connect` service of `.odctl/compose-kafka.yml` (`offset.flush.interval.ms=10000`), where an edit can change them. ## Plugins The `deps` image puts these connectors in the plugin folder: the Apache Iceberg sink, the Debezium PostgreSQL source, the ClickHouse sink, the Aiven JDBC connector, the Aiven S3 sink, the Redis connector and the MSK data generator. This lists the classes Connect has loaded: ```bash curl -s http://127.0.0.1:8083/connector-plugins ``` The reply is a JSON list with one entry per class. The tests check that it includes `org.apache.iceberg.connect.IcebergSinkConnector`, `io.debezium.connector.postgresql.PostgresConnector` and `com.clickhouse.kafka.connect.ClickHouseSinkConnector`. A plugin missing from the volume does not stop Connect from starting. It only leaves the class out of this list. ## Register a Connector A connector is a name and a `config` object, posted to `/connectors`. The MSK data generator is a source connector that needs nothing but Kafka: it writes made-up records to a topic. Save this configuration as `demo-datagen.json`. It writes a record to the topic `customers` every half second, with an ISBN-10 number as its key, a first name and a number of visits from 1 to 10: ```json { "name": "demo-datagen", "config": { "connector.class": "com.amazonaws.mskdatagen.GeneratorSourceConnector", "tasks.max": "1", "genkp.customers.with": "#{Code.isbn10}", "genv.customers.name.with": "#{Name.first_name}", "genv.customers.visits.with": "#{number.number_between '1','10'}", "global.throttle.ms": "500", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false" } } ``` The worker's default converters expect JSON with a schema envelope, so the connector sets its own: plain strings for keys, and JSON without a schema for values. Post the file from the folder that holds it: ```bash curl -X POST -H 'Content-Type: application/json' \ http://127.0.0.1:8083/connectors -d @demo-datagen.json ``` The reply repeats the configuration, with `"tasks":[]` and `"type":"source"`. The worker starts the task right after it answers. ## Read Its Records The broker creates `customers` when the connector first writes to it, because automatic topic creation is on by default. This reads the first three records with their keys: ```bash docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server broker-1:19092 \ --topic customers --from-beginning --max-messages 3 --timeout-ms 60000 \ --formatter-property print.key=true ``` Each line is a key, a tab and a value such as `{"name":"Patrina","visits":"3"}`. The names and numbers are random. The consumer ends with `Processed a total of 3 messages`. ## Check and Remove It This reads the connector's status, then deletes it: ```bash curl -s http://127.0.0.1:8083/connectors/demo-datagen/status curl -X DELETE http://127.0.0.1:8083/connectors/demo-datagen ``` The status shows `"state":"RUNNING"` for the connector and for task `0`. A failed task shows `FAILED` with its error. The worker's log has the rest: ```bash docker logs --tail 40 connect ``` After the delete, the last lines of the log say that the worker stopped the task of `demo-datagen`. Kafka UI at `http://127.0.0.1:8086` lists the registered connectors under Kafka Connect, Connectors. [Ingest into Iceberg](https://jaehyeon.me/odctl/latest/guides/iceberg-ingestion/#kafka-connect-sink) registers a sink connector, the Iceberg sink. ## PostgreSQL for Debezium The `postgres` profile runs PostgreSQL with `wal_level=logical`. Its init script creates the `cdc` schema and the `cdc_pub` publication for every table in that schema, so a Debezium source can read changes to tables you create in `cdc`. Start both profiles to use it: ```bash odctl up postgres kafka-lite ``` ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), the REST API is `http://connect:8083`, and connectors reach the brokers at `broker-1:19092`, the catalog at `http://catalog:8181` and SeaweedFS at `http://seaweed:8333`. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It holds the configuration above, so it needs no other file: ```bash #!/usr/bin/env bash # Starts kafka-lite, registers a data generator connector, reads three of its # records, prints the connector's status and removes it. set -euo pipefail odctl up kafka-lite # The REST API answers a little after the worker starts. until curl -fs http://127.0.0.1:8083/connectors >/dev/null; do sleep 2; done curl -s -X POST -H 'Content-Type: application/json' http://127.0.0.1:8083/connectors -d @- <<'JSON' { "name": "demo-datagen", "config": { "connector.class": "com.amazonaws.mskdatagen.GeneratorSourceConnector", "tasks.max": "1", "genkp.customers.with": "#{Code.isbn10}", "genv.customers.name.with": "#{Name.first_name}", "genv.customers.visits.with": "#{number.number_between '1','10'}", "global.throttle.ms": "500", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false" } } JSON echo docker exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server broker-1:19092 \ --topic customers --from-beginning --max-messages 3 --timeout-ms 60000 \ --formatter-property print.key=true curl -s http://127.0.0.1:8083/connectors/demo-datagen/status echo curl -s -X DELETE http://127.0.0.1:8083/connectors/demo-datagen ``` It prints the reply to the registration, three records, `Processed a total of 3 messages` and the status with `"state":"RUNNING"`. # Flink SQL: Kafka to PostgreSQL The `flink-lite` profile runs a Flink JobManager, one TaskManager and the SQL Gateway. `flink-full` runs three TaskManagers. Flink's connector jars for Kafka, Avro with the schema registry, JDBC, Iceberg and HTTP come from the shared volume that the `deps` profile fills. The end-to-end tests run the commands below on every release. This guide reads Avro records from Kafka through the Karapace schema registry and writes an aggregate to PostgreSQL over JDBC. It builds on [Kafka](https://jaehyeon.me/odctl/latest/guides/kafka/index.md), which starts `kafka-lite` and creates a topic. [Ingest into Iceberg](https://jaehyeon.me/odctl/latest/guides/iceberg-ingestion/index.md) shows Flink writing an Iceberg table, [Rolling Windows](https://jaehyeon.me/odctl/latest/guides/rolling-windows/index.md) computes features over several window sizes, and [Feast with Iceberg](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#push-features) sends rows to Feast over HTTP. Follow the arrows from the TaskManager: it writes Avro records to the `orders` topic, reads them back and writes the totals to PostgreSQL. Start both profiles: ```bash odctl up flink-lite kafka-lite ``` `flink-lite` starts PostgreSQL itself, because it depends on the Iceberg catalog, which keeps its metadata there. | Endpoint | Address | | ----------------------------------------- | ------------------------------- | | Flink UI and REST API from the host | `http://127.0.0.1:8082` | | SQL Gateway REST API from the host | `http://127.0.0.1:8084` | | Flink REST API inside the `odctl` network | `http://jobmanager:8081` | | SQL Gateway inside the `odctl` network | `http://flink-sql-gateway:8083` | ## Create the Topic and the Target Table This creates the Kafka topic `orders` and the PostgreSQL table `supplier_stats` that the job writes to: ```bash docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:19092 \ --create --if-not-exists --topic orders --partitions 1 --replication-factor 1 docker exec postgres psql -U user -d odctl -c " CREATE TABLE supplier_stats (supplier TEXT PRIMARY KEY, orders BIGINT, total DOUBLE PRECISION);" ``` The commands print `Created topic orders.` and `CREATE TABLE`. ## Run the SQL Open the SQL client in the JobManager container: ```bash docker exec -it flink-jobmanager /opt/flink/bin/sql-client.sh ``` Run these statements in the client. They declare the topic and the PostgreSQL table as Flink tables, write three orders and sum them by supplier: ```sql SET 'execution.runtime-mode' = 'batch'; SET 'table.dml-sync' = 'true'; CREATE TABLE orders (supplier STRING, price DOUBLE) WITH ('connector'='kafka','topic'='orders','properties.bootstrap.servers'='broker-1:19092','scan.startup.mode'='earliest-offset','scan.bounded.mode'='latest-offset','format'='avro-confluent','avro-confluent.url'='http://karapace:8081'); INSERT INTO orders VALUES ('a', 10.0), ('a', 5.0), ('b', 7.5); CREATE TABLE stats (supplier STRING, orders BIGINT, total DOUBLE, PRIMARY KEY (supplier) NOT ENFORCED) WITH ('connector'='jdbc','url'='jdbc:postgresql://postgres:5432/odctl','table-name'='supplier_stats','username'='user','password'='password'); INSERT INTO stats SELECT supplier, COUNT(*), SUM(price) FROM orders GROUP BY supplier; ``` Each statement prints `[INFO] Execute statement succeeded.` or, for an `INSERT`, `[INFO] Complete execution of the SQL update statement.` The first `INSERT` writes three Avro records to Kafka and registers their schema in Karapace. The second reads them back, groups them by supplier and writes the result to PostgreSQL. `scan.bounded.mode` set to `latest-offset` makes the Kafka source stop at the end of the topic, so the batch job finishes. Leave it out and set `execution.runtime-mode` to `streaming` for a job that keeps running. Right after Kafka starts, Karapace can answer its health check and still time out on a schema write for a short while. If the first `INSERT` fails on the schema registry, run it again. ## Check the Result This reads the table the job wrote: ```bash docker exec postgres psql -U user -d odctl -c "SELECT * FROM supplier_stats ORDER BY supplier" ``` Supplier `a` has 2 orders totalling 15, and supplier `b` has 1 order totalling 7.5: ```text supplier | orders | total ----------+--------+------- a | 2 | 15 b | 1 | 7.5 (2 rows) ``` The Flink UI at `http://127.0.0.1:8082` shows the jobs the SQL client submitted. The two `INSERT` statements run as two batch jobs, listed under Completed Jobs. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), Flink's REST API is `http://jobmanager:8081` and the SQL Gateway is `http://flink-sql-gateway:8083`. In Flink SQL, Kafka is `broker-1:19092`, Karapace is `http://karapace:8081` and PostgreSQL is `jdbc:postgresql://postgres:5432/odctl`. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It runs the steps above with a topic and a table of its own for each run, so it can run again: ```bash #!/usr/bin/env bash # Starts flink-lite and kafka-lite, writes three Avro records to a Kafka topic # with Flink SQL, sums them by supplier into PostgreSQL and prints the result. # Each run uses its own topic and table names, so the file can run again. set -euo pipefail odctl up flink-lite kafka-lite # Karapace answers its health check before it can store a schema, so wait # until it accepts one. until curl -fs -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' \ -d '{"schema": "{\"type\": \"string\"}"}' http://127.0.0.1:8081/subjects/ready/versions >/dev/null; do sleep 5 done run=$(date +%s) docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:19092 \ --create --topic "orders-$run" --partitions 1 --replication-factor 1 docker exec postgres psql -U user -d odctl -c " CREATE TABLE supplier_stats_$run (supplier TEXT PRIMARY KEY, orders BIGINT, total DOUBLE PRECISION);" docker exec -i flink-jobmanager /opt/flink/bin/sql-client.sh </dev/null | grep '^\[0' >/dev/null; do sleep 2 done run=$(date +%s) docker exec -i flink-jobmanager /opt/flink/bin/sql-client.sh </dev/null; do sleep 2; done # A namespace, which may exist already, and an empty table for the sink. curl -s -o /dev/null -X POST -H 'Content-Type: application/json' -d '{"namespace":["demo"]}' "$api" curl -fs -o /dev/null -X POST -H 'Content-Type: application/json' \ -d "{\"name\":\"kc_$run\",\"schema\":{\"type\":\"struct\",\"schema-id\":0,\"fields\":[{\"id\":1,\"name\":\"id\",\"required\":false,\"type\":\"long\"},{\"id\":2,\"name\":\"name\",\"required\":false,\"type\":\"string\"}]}}" \ "$api/demo/tables" # Create the topic and the sink's control topic, then write five JSON records. for topic in "iceberg-$run" "control-$run"; do docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:19092 \ --create --topic "$topic" --partitions 1 --replication-factor 1 done for i in 1 2 3 4 5; do echo "{\"id\":$i,\"name\":\"n$i\"}"; done | docker exec -i kafka \ /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server broker-1:19092 --topic "iceberg-$run" curl -fs -o /dev/null -X POST -H 'Content-Type: application/json' http://127.0.0.1:8083/connectors -d @- </dev/null; do sleep 5; done echo "demo.kc_$run: 5 records written by the Kafka Connect sink" curl -s -o /dev/null -X DELETE "http://127.0.0.1:8083/connectors/iceberg-$run" docker exec -i flink-jobmanager /opt/flink/bin/sql-client.sh </dev/null docker exec kafka /opt/kafka/bin/kafka-topics.sh --bootstrap-server broker-1:19092 \ --create --topic "spark-$run" --partitions 1 --replication-factor 1 for i in 1 2 3; do echo "{\"n\":$i}"; done | docker exec -i kafka \ /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server broker-1:19092 --topic "spark-$run" docker exec -i spark-master sh -c "cat > /tmp/read_kafka_$run.py" </dev/null ``` It prints `3`, `Created topic` with the topic's name, and a table with the values `{"n":1}`, `{"n":2}` and `{"n":3}`. # ClickHouse ClickHouse is a column-oriented database for analytical queries. A shard holds part of a table's rows, and each replica of a shard holds a copy of that part. ClickHouse Keeper is the coordination service that keeps the replicas of a shard in step. The `ch-lite` profile runs one shard with two replicas, `ch-11` and `ch-12`, coordinated by ClickHouse Keeper in `ch-keeper`. `ch-full` adds a second shard, `ch-21` and `ch-22`. The end-to-end tests run the commands below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). Start the profile: ```bash odctl up ch-lite ``` | Endpoint | Address | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | HTTP from the host | `http://127.0.0.1:8123` (`ch-11`) and `http://127.0.0.1:8124` (`ch-12`); `ch-full` adds `8125` (`ch-21`) and `8126` (`ch-22`) | | Native protocol from the host | `127.0.0.1:9000` (`ch-11`) and `127.0.0.1:9001` (`ch-12`); `ch-full` adds `9002` (`ch-21`) and `9003` (`ch-22`) | | Users | `default` and `user`, both with the password `password` | Every server creates the `feature_store` and `odctl` databases when it starts with no data. ## Create a Table and Query It `clickhouse-client` is in each server container. This lists the databases, then creates a table, writes 10 rows and counts them: ```bash docker exec ch-11 clickhouse-client --password password -q "SHOW DATABASES" docker exec ch-11 clickhouse-client --password password -q \ "CREATE TABLE default.demo (id UInt32) ENGINE=MergeTree ORDER BY id" docker exec ch-11 clickhouse-client --password password -q \ "INSERT INTO default.demo SELECT number FROM numbers(10)" docker exec ch-11 clickhouse-client --password password -q \ "SELECT count() FROM default.demo" ``` The first command lists `default`, `feature_store`, `odctl` and the system databases. The last prints `10`. ClickHouse also serves a query page at `http://127.0.0.1:8123/play`. Enter `default` and `password` in the top right, write a query and press Run. ## Replicate Across the Cluster In `ch-full`, `{cluster}` names `odctl_cluster`, which has two shards of two replicas. A table created `ON CLUSTER` with `ReplicatedMergeTree` is created on all four servers, and Keeper keeps the two replicas of each shard in step: ```bash odctl up ch-full docker exec ch-11 clickhouse-client --password password -q \ "CREATE TABLE default.repl ON CLUSTER '{cluster}' (id UInt32) ENGINE=ReplicatedMergeTree ORDER BY id" docker exec ch-11 clickhouse-client --password password -q \ "INSERT INTO default.repl SELECT number FROM numbers(50)" docker exec ch-12 clickhouse-client --password password -q "SELECT count() FROM default.repl" docker exec ch-21 clickhouse-client --password password -q "SELECT count() FROM default.repl" ``` The `CREATE` prints one line for each of the four servers, with status `0` for success. `ch-12`, the other replica of shard 1, returns `50` once replication has caught up. `ch-21` returns `0`, because shard 2 holds only the rows written to it. In `ch-lite`, `{cluster}` names `odctl_single_shard`, which holds `ch-11` and `ch-12` only, so `ON CLUSTER` statements finish without waiting for the missing shard. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), connect to `ch-11:8123` over HTTP or `ch-11:9000` with the native protocol, with the same users. Trino's `clickhouse` catalog reads the `feature_store` database on `ch-11`, as [Trino and Metabase](https://jaehyeon.me/odctl/latest/guides/trino-metabase/index.md) shows. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It runs on `ch-lite`, where `{cluster}` holds the two replicas of one shard, and uses tables of its own for each run: ```bash #!/usr/bin/env bash # Starts ch-lite, writes 10 rows to a MergeTree table and 50 rows to a # replicated table, and counts the replicated rows on the other replica. # Each run uses its own table names. set -euo pipefail odctl up ch-lite run=$(date +%s) ch() { docker exec "$1" clickhouse-client --password password -q "$2"; } ch ch-11 "CREATE TABLE default.demo_$run (id UInt32) ENGINE=MergeTree ORDER BY id" ch ch-11 "INSERT INTO default.demo_$run SELECT number FROM numbers(10)" ch ch-11 "SELECT count() FROM default.demo_$run" ch ch-11 "CREATE TABLE default.repl_$run ON CLUSTER '{cluster}' (id UInt32) ENGINE=ReplicatedMergeTree ORDER BY id" >/dev/null ch ch-11 "INSERT INTO default.repl_$run SELECT number FROM numbers(50)" # ch-12 is the other replica of the same shard; replication takes a moment. until [ "$(ch ch-12 "SELECT count() FROM default.repl_$run")" = "50" ]; do sleep 1; done echo "ch-12 holds 50 rows of repl_$run" ``` It prints `10`, then `ch-12 holds 50 rows of` with the replicated table's name. # Trino and Metabase The `trino` profile runs Trino, which queries PostgreSQL, Iceberg, ClickHouse, Kafka and Valkey through one SQL interface. The `metabase` profile runs Metabase, which draws charts and dashboards over those databases. Metabase keeps its own settings and dashboards in PostgreSQL, so both profiles start `postgres`. The end-to-end tests run the commands below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). The catalogs read the services of earlier guides, such as [Kafka](https://jaehyeon.me/odctl/latest/guides/kafka/index.md), [Ingest into Iceberg](https://jaehyeon.me/odctl/latest/guides/iceberg-ingestion/index.md) and [ClickHouse](https://jaehyeon.me/odctl/latest/guides/clickhouse/index.md), when their profiles are running. Start both profiles: ```bash odctl up trino metabase ``` | Endpoint | Address | | ------------------------------- | ----------------------- | | Trino UI and JDBC from the host | `http://127.0.0.1:8080` | | Metabase from the host | `http://127.0.0.1:3000` | ## Trino Catalogs A Trino catalog is a configured connection to one data source. This lists them: ```bash docker exec trino trino --execute "SHOW CATALOGS" ``` The command prints `"clickhouse"`, `"iceberg"`, `"kafka"`, `"postgres"`, `"system"` and `"valkey"`, one per line, with a `Picked up JAVA_TOOL_OPTIONS` line before them: | Catalog | Reads | Needs | | ------------ | ----------------------------------------------------- | ---------------------------- | | `postgres` | the `odctl` database | nothing more | | `iceberg` | tables in the REST catalog, with files on SeaweedFS | `catalog` | | `clickhouse` | the `feature_store` database on `ch-11` | `ch-lite` or `ch-full` | | `kafka` | topics, with fields from the Karapace schema registry | `kafka-lite` or `kafka-full` | | `valkey` | keys in Valkey | `valkey` | Every catalog loads when Trino starts, even when its service is not running. A query against such a catalog fails until you start that profile. Catalogs finish loading after Trino's health endpoint starts answering, so `SHOW CATALOGS` can take a minute to list them all. This runs a query with no table: ```bash docker exec trino trino --execute "SELECT 1" ``` It prints `"1"`. Trino has no password authentication, so it accepts any user name. `docker exec trino trino --execute ''` runs as `trino`. Pass `--user user` to use the name the rest of the stack uses. [Examples](https://jaehyeon.me/odctl/latest/how-it-works/#examples) on How It Works explains the rules in `trino/rules.json`. The Trino UI at `http://127.0.0.1:8080` asks only for a user name. It shows the cluster and the queries that ran in the last few minutes. ## Metabase Open `http://127.0.0.1:3000` and create the first admin account. Then add each database under Admin settings, Databases, with these settings: | Database type | Host | Port | Other settings | | --------------------- | ---------- | ------ | --------------------------------------------------------------- | | PostgreSQL | `postgres` | `5432` | database `odctl`, user `user`, password `password` | | ClickHouse | `ch-11` | `8123` | database `default`, user `default`, password `password`, no SSL | | Starburst (for Trino) | `trino` | `8080` | catalog `postgres`, schema `public`, user `admin`, no SSL | Metabase runs inside the `odctl` network, so it reaches each database by container name. Use the Starburst driver for Trino, because it is the Trino driver Metabase ships. It needs a catalog, and any catalog in the table above works. Metabase takes a while to start. It is ready when this request answers `{"status":"ok"}`: ```bash curl http://127.0.0.1:3000/api/health ``` A SQL question on the PostgreSQL connection, shown as a bar chart: ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), Trino is at `http://trino:8080` and Metabase at `http://metabase:3000`. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It waits for the catalogs and for Metabase before it runs the commands above: ```bash #!/usr/bin/env bash # Starts trino and metabase, waits until every Trino catalog has loaded and # Metabase answers, then lists the catalogs and runs a query. set -euo pipefail odctl up trino metabase # Catalogs finish loading after Trino's health endpoint answers. until docker exec trino trino --execute "SHOW CATALOGS" 2>/dev/null | grep '"valkey"' >/dev/null; do sleep 5; done docker exec trino trino --execute "SHOW CATALOGS" docker exec trino trino --execute "SELECT 1" until curl -fs http://127.0.0.1:3000/api/health; do sleep 5; done echo ``` It prints the six catalogs, `"1"` and `{"status":"ok"}`. # Airflow DAGs from S3 The `airflow` profile runs `airflow standalone` in one container against odctl's PostgreSQL. It reads DAG files from `s3://airflow/dags` on SeaweedFS, so you deploy a DAG by uploading it. No volume or sync container is involved. The end-to-end tests run the steps below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). Look at the left of the diagram: DAG files go to SeaweedFS, and Airflow reads them from there. Start the profile: ```bash odctl up airflow ``` | Endpoint | Address | | ---------------------------------------------- | --------------------------------------------------------- | | Web UI and REST API from the host | `http://127.0.0.1:8085`, user `user`, password `password` | | Web UI and REST API inside the `odctl` network | `http://airflow:8080` | | DAG folder | `s3://airflow/dags` on SeaweedFS | ## Upload a DAG Any S3 client pointed at `http://127.0.0.1:8333` with key `user` and secret `password` can upload. This uses boto3 inside the Airflow container to write the DAG `hello`, with one task that runs `echo hello`: ```bash docker exec -i airflow python - <<'PY' import boto3 body = ''' from airflow.sdk import DAG from airflow.providers.standard.operators.bash import BashOperator import pendulum with DAG( dag_id="hello", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), schedule=None, catchup=False, ): BashOperator(task_id="say_hello", bash_command="echo hello") ''' boto3.client("s3", endpoint_url="http://seaweed:8333").put_object( Bucket="airflow", Key="dags/hello.py", Body=body.encode()) PY ``` The upload prints nothing. Airflow checks the bucket every 15 seconds and then parses the new file. The interval is set today by `refresh_interval` in `AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST` in `.odctl/compose-orch.yml`, where an edit can change it. Two other profiles also read their code from a bucket; [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#where-data-lives) compares the three. ## Run It New DAGs start paused. Wait until the DAG is listed, then unpause it, trigger it and list its runs: ```bash docker exec airflow airflow dags list | grep hello docker exec airflow airflow dags unpause hello docker exec airflow airflow dags trigger hello docker exec airflow airflow dags list-runs hello ``` The first command prints the `hello` line of the list. The last lists the run with its `run_id`, such as `manual__2026-10-24T04:55:03.340843+00:00`, in state `queued` or `running`, and listing it again a few seconds later shows `success`. The DAG page shows the run and the `say_hello` task. ## Plugins and Packages Plugins are copied from `s3://airflow/plugins` when the container starts, because Airflow loads plugins only once. Run `odctl restart airflow` after changing one. Tasks already have the MLflow and Feast clients, and `MLFLOW_TRACKING_URI` points at the `mlflow` profile. [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) shows how to install extra Python packages with `_AIRFLOW_PIP_DEPS`. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), the Airflow API is `http://airflow:8080`, and tasks reach SeaweedFS at `http://seaweed:8333` and MLflow at `http://mlflow:5000`. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It runs the steps above and waits for each one: ```bash #!/usr/bin/env bash # Starts airflow, uploads a DAG to s3://airflow/dags, waits until Airflow has # parsed it, triggers a run and waits until the run finishes. set -euo pipefail odctl up airflow docker exec -i airflow python - <<'PY' import boto3 body = ''' from airflow.sdk import DAG from airflow.providers.standard.operators.bash import BashOperator import pendulum with DAG( dag_id="hello", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), schedule=None, catchup=False, ): BashOperator(task_id="say_hello", bash_command="echo hello") ''' boto3.client("s3", endpoint_url="http://seaweed:8333").put_object( Bucket="airflow", Key="dags/hello.py", Body=body.encode()) PY # Airflow checks the bucket every 15 seconds, then parses the new file. until docker exec airflow airflow dags list 2>/dev/null | grep hello >/dev/null; do sleep 5; done docker exec airflow airflow dags unpause hello docker exec airflow airflow dags trigger hello >/dev/null until docker exec airflow airflow dags list-runs hello -o plain 2>/dev/null | grep -E 'success|failed' >/dev/null; do sleep 5; done docker exec airflow airflow dags list-runs hello -o plain ``` It prints the `hello` line from `airflow dags unpause` and then the run, in state `success`. # Temporal Workers Temporal runs long-running code that survives failures: a workflow is a function whose progress the server records, and an activity is a step of it that can fail and be retried. The `temporal` profile runs the Temporal development server: the server, the Web UI and a SQLite database in one container. odctl runs only the server. Your own process runs the worker that executes workflows and activities. The end-to-end tests run the code below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). Start the profile and install the Python SDK at the version the tests use: ```bash odctl up temporal pip install temporalio==1.34.0 ``` | Endpoint | Address | | --------------------------------------- | ----------------------- | | gRPC from the host, namespace `default` | `127.0.0.1:7233` | | gRPC inside the `odctl` network | `temporal:7233` | | Web UI | `http://127.0.0.1:8233` | ## A Worker and a Workflow Save this code as `greet.py` and run it with `python greet.py`. It starts a worker, runs the workflow `Greet` on it and prints the result. The activity fails on its first attempt, so the result shows that the retry policy ran it again: ```python import asyncio from datetime import timedelta from temporalio import activity, workflow from temporalio.client import Client from temporalio.common import RetryPolicy from temporalio.worker import Worker @activity.defn async def greet(name: str) -> str: if activity.info().attempt < 2: raise RuntimeError("first attempt fails on purpose") return f"hello {name}" @workflow.defn class Greet: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_activity( greet, name, start_to_close_timeout=timedelta(seconds=30), retry_policy=RetryPolicy(initial_interval=timedelta(seconds=1), maximum_attempts=3), ) async def main() -> None: client = await Client.connect("127.0.0.1:7233") async with Worker(client, task_queue="demo", workflows=[Greet], activities=[greet]): print(await client.execute_workflow(Greet.run, "odctl", id="greet-1", task_queue="demo")) # The workflow sandbox imports this file again, so run main only as a script. if __name__ == "__main__": asyncio.run(main()) ``` The worker logs the `RuntimeError` of the first attempt with its traceback, and then the script prints `hello odctl`. ## Wait for an Approval A workflow can wait for a signal, for example a person approving an agent's action. The workflow below waits until its `approve` signal arrives, and then returns who approved it: ```python @workflow.defn class Approval: def __init__(self) -> None: self.approver = None @workflow.signal def approve(self, approver: str) -> None: self.approver = approver @workflow.run async def run(self) -> str: await workflow.wait_condition(lambda: self.approver is not None) return f"approved by {self.approver}" ``` Add the class to `greet.py` and replace `main` with this one, which registers both workflows with the worker, runs `Greet`, starts `Approval` and sends its signal: ```python async def main() -> None: client = await Client.connect("127.0.0.1:7233") async with Worker(client, task_queue="demo", workflows=[Greet, Approval], activities=[greet]): print(await client.execute_workflow(Greet.run, "odctl", id="greet-1", task_queue="demo")) handle = await client.start_workflow(Approval.run, id="approval-1", task_queue="demo") await handle.signal(Approval.approve, "alice") print(await handle.result()) ``` The script prints `hello odctl` and then `approved by alice`. The complete file is the [runnable example](#runnable-example) below. Until the signal arrives, the Web UI lists the workflow as running. The Web UI at `http://127.0.0.1:8233` lists each workflow. Here `approval-2` was started without a signal, so it is still running: ## History Workflow history is in the database file inside the container. It survives `odctl restart temporal`, and the tests check that a workflow left waiting for its signal resumes and completes afterwards. [`odctl down`](https://jaehyeon.me/odctl/latest/how-it-works/#where-data-lives) removes the history. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), a worker connects to `temporal:7233` with the namespace `default`. ## Runnable Example Save this file as `example.py` and run it with `python example.py` while the `temporal` profile is up, in an environment with the SDK from the install step. It is `greet.py` with both steps above: ```python """A Temporal worker with two workflows: one retries a failing activity, the other waits for an approval signal. Run it with odctl's temporal profile up.""" import asyncio from datetime import timedelta from temporalio import activity, workflow from temporalio.client import Client from temporalio.common import RetryPolicy from temporalio.worker import Worker @activity.defn async def greet(name: str) -> str: if activity.info().attempt < 2: raise RuntimeError("first attempt fails on purpose") return f"hello {name}" @workflow.defn class Greet: @workflow.run async def run(self, name: str) -> str: return await workflow.execute_activity( greet, name, start_to_close_timeout=timedelta(seconds=30), retry_policy=RetryPolicy(initial_interval=timedelta(seconds=1), maximum_attempts=3), ) @workflow.defn class Approval: def __init__(self) -> None: self.approver = None @workflow.signal def approve(self, approver: str) -> None: self.approver = approver @workflow.run async def run(self) -> str: await workflow.wait_condition(lambda: self.approver is not None) return f"approved by {self.approver}" async def main() -> None: client = await Client.connect("127.0.0.1:7233") async with Worker(client, task_queue="demo", workflows=[Greet, Approval], activities=[greet]): print(await client.execute_workflow(Greet.run, "odctl", id="greet-1", task_queue="demo")) handle = await client.start_workflow(Approval.run, id="approval-1", task_queue="demo") await handle.signal(Approval.approve, "alice") print(await handle.result()) # The workflow sandbox imports this file again, so run main only as a script. if __name__ == "__main__": asyncio.run(main()) ``` It prints `hello odctl` and then `approved by alice`. It can run again, because Temporal accepts a new run of a workflow id whose last run has completed. # Feast with Iceberg Feast is a feature store: it keeps the definitions of a model's input features and serves their values. The offline store holds the history of each feature, for training. The online store holds each entity's latest value, for serving a prediction quickly. Materialising copies values from the offline store to the online store. The `feast` profile runs the Feast UI and the online feature server. Feast stores no feature values of its own. Feature definitions live in a SQL registry in PostgreSQL, offline values come from Iceberg tables through the REST catalog, and online values are served from Valkey. You run `feast apply` and the other client commands from your own environment. The end-to-end tests run the steps below on every release. This guide builds on [Ingest into Iceberg](https://jaehyeon.me/odctl/latest/guides/iceberg-ingestion/index.md), which writes Iceberg tables through the catalog, [Flink SQL](https://jaehyeon.me/odctl/latest/guides/flink-sql/index.md), whose SQL client the Flink push uses, and [Airflow DAGs from S3](https://jaehyeon.me/odctl/latest/guides/airflow-dags/index.md), which uploads and runs a DAG. Follow the arrows from the `feast` CLI on the left: definitions go to the registry, and values go from Iceberg to Valkey. An Airflow DAG at the bottom centre also materialises into Valkey, on a schedule. A client at the bottom right can also push a value through the feature server into Valkey. The Feast client needs Python 3.11 or newer. Start the profile and install the client, with the extras that read Iceberg, PostgreSQL and Valkey: ```bash odctl up feast pip install "feast[duckdb,iceberg,redis,postgres]==0.68.0" pyarrow ``` | Endpoint | Address | | ----------------------------------------- | ---------------------------------- | | Feast UI | `http://127.0.0.1:8890` | | Feature server from the host | `http://127.0.0.1:6566` | | Feature server inside the `odctl` network | `http://feast-serve:6566` | | MCP endpoint from the host | `http://127.0.0.1:6566/mcp` | | Registry | the `feast` database on PostgreSQL | ## Point the Clients at the Stack PyIceberg, which Feast uses to read Iceberg, takes its catalog settings from the environment. Run these in the shell where you run the next steps: ```bash export PYICEBERG_CATALOG__ODCTL__TYPE=rest export PYICEBERG_CATALOG__ODCTL__URI=http://127.0.0.1:8181 export PYICEBERG_CATALOG__ODCTL__WAREHOUSE=s3://warehouse/ export PYICEBERG_CATALOG__ODCTL__S3__ENDPOINT=http://127.0.0.1:8333 export PYICEBERG_CATALOG__ODCTL__S3__ACCESS_KEY_ID=user export PYICEBERG_CATALOG__ODCTL__S3__SECRET_ACCESS_KEY=password export PYICEBERG_CATALOG__ODCTL__S3__PATH_STYLE_ACCESS=true export PYICEBERG_CATALOG__ODCTL__S3__REGION=us-east-1 ``` ## Create an Iceberg Table Save this code as `create_table.py` and run it with `python create_table.py`. It creates `demo.driver_stats` if it does not exist and writes one hour-old row for each of two drivers: ```python import datetime, pyarrow as pa from pyiceberg.catalog import load_catalog cat = load_catalog("odctl") cat.create_namespace_if_not_exists("demo") schema = pa.schema([ pa.field("driver_id", pa.int64(), nullable=False), pa.field("event_timestamp", pa.timestamp("us", tz="UTC"), nullable=False), pa.field("conv_rate", pa.float32(), nullable=False), ]) t = cat.create_table_if_not_exists("demo.driver_stats", schema=schema) now = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0) t.append(pa.Table.from_pydict({ "driver_id": [1001, 1002], "event_timestamp": [now - datetime.timedelta(hours=1)] * 2, "conv_rate": [0.5, 0.75], }, schema=schema)) ``` ## Define the Feature Repository A feature repository is a folder of configuration and definitions. Save this file as `feature_repo/feature_store.yaml`. It names the registry, the offline store and the online store: ```yaml project: demo provider: local registry: registry_type: sql path: postgresql+psycopg://user:password@127.0.0.1:5432/feast offline_store: type: duckdb online_store: type: redis connection_string: "127.0.0.1:6379,username=user,password=password" entity_key_serialization_version: 3 ``` Save the definitions as `feature_repo/definitions.py`. They declare the entity `driver` and the feature view `driver_stats` over the Iceberg table: ```python from datetime import timedelta from feast import Entity, FeatureView, Field from feast.types import Float32 from feast.value_type import ValueType from feast.infra.data_sources.contrib.iceberg_catalog.iceberg_source import IcebergSource driver = Entity(name="driver", join_keys=["driver_id"], value_type=ValueType.INT64) src = IcebergSource( warehouse="", namespace="demo", table="driver_stats", catalog_type="rest", catalog_name="odctl", timestamp_field="event_timestamp", ) fv = FeatureView( name="driver_stats", entities=[driver], ttl=timedelta(days=365), schema=[Field(name="conv_rate", dtype=Float32)], source=src, online=True, ) ``` Pass `catalog_name="odctl"`, so that PyIceberg reads the `PYICEBERG_CATALOG__ODCTL__*` variables. Pass `warehouse=""`, so that PyIceberg takes the warehouse from `PYICEBERG_CATALOG__ODCTL__WAREHOUSE`. The source has no `endpoint`, so the registry stores no catalog address, and PyIceberg takes it from `PYICEBERG_CATALOG__ODCTL__URI`. Each program that reads the registry therefore uses its own address: `http://127.0.0.1:8181` in your shell and `http://catalog:8181` in the Airflow container, set today in `x-airflow-common` of `.odctl/compose-orch.yml`, where an edit can change it. The entity names its `value_type`, so that `apply` need not read the table to find the type of `driver_id`. ## Register and Materialise Run these from the folder that holds `feature_repo`. `apply` writes the definitions to the registry, and `materialize-incremental` copies the latest values into Valkey. `--skip-source-validation` stops `apply` from checking the Iceberg source with Feast's own catalog client, which would need the catalog address that the source leaves out: ```bash feast -c feature_repo apply --skip-source-validation feast -c feature_repo materialize-incremental "$(date -u +%Y-%m-%dT%H:%M:%S)" ``` `apply` prints `Applying changes for project demo` and `Deploying infrastructure for driver_stats`, and `materialize-incremental` names the feature view and the time range it copied. ### On a Schedule The DAG uses the table and the feature view from the steps above, so run them first. Materialising by hand copies only the values that exist at that moment. An Airflow DAG can run `materialize_incremental` on a schedule. That method is the one behind `feast materialize-incremental`. It copies the rows whose `event_timestamp` falls between the end of the previous materialisation and `end_date`. A row written later with an older timestamp is therefore not copied. The `airflow` image already has the Feast client, so the DAG installs nothing. It builds the feature store in code from three settings: - the registry address in `FEAST_REGISTRY`, which the Airflow container sets; - the DuckDB offline store; - the Valkey online store at the container address `valkey:6379`. Save it as `feast_materialize.py`: ```python from datetime import datetime, timezone from airflow.sdk import dag, task @dag( dag_id="feast_materialize", start_date=datetime(2026, 1, 1, tzinfo=timezone.utc), schedule="@hourly", catchup=False, ) def feast_materialize(): """Copy the latest feature values to Valkey every hour.""" @task def materialize(): """Materialise the feature views up to now.""" # Imported here so that parsing the file does not load Feast. import os from feast import FeatureStore, RepoConfig store = FeatureStore(config=RepoConfig( project="demo", provider="local", registry={"registry_type": "sql", "path": os.environ["FEAST_REGISTRY"]}, offline_store={"type": "duckdb"}, online_store={ "type": "redis", "connection_string": "valkey:6379,username=user,password=password", }, entity_key_serialization_version=3, )) store.materialize_incremental(end_date=datetime.now(timezone.utc)) materialize() feast_materialize() ``` `schedule="@hourly"` runs the DAG at the start of each hour. `catchup=False` skips the hours missed while Airflow was stopped, because one incremental run covers them. Start Airflow beside Feast: ```bash odctl up airflow ``` Copy the DAG into the Airflow container and upload it to `s3://airflow/dags`, as [Airflow DAGs from S3](https://jaehyeon.me/odctl/latest/guides/airflow-dags/#upload-a-dag) does: ```bash docker cp feast_materialize.py airflow:/tmp/feast_materialize.py docker exec airflow python -c ' import boto3 boto3.client("s3", endpoint_url="http://seaweed:8333").upload_file( "/tmp/feast_materialize.py", "airflow", "dags/feast_materialize.py")' ``` Airflow reads the bucket every 15 seconds, so the DAG appears in its list a short time after the upload. To give the next run a value to copy, save this code as `append.py` and run it with `python append.py` in the shell where you set the catalog variables. It appends the value 0.6 for driver 1001, with the current time, to the Iceberg table: ```python import datetime, pyarrow as pa from pyiceberg.catalog import load_catalog t = load_catalog("odctl").load_table("demo.driver_stats") now = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0) t.append(pa.Table.from_pydict({ "driver_id": [1001], "event_timestamp": [now], "conv_rate": [0.6], }, schema=t.schema().as_arrow())) ``` Then unpause the DAG and trigger a run. Repeat the first command until it prints the DAG's line. Before that, `unpause` prints `No paused DAGs were found` and changes nothing, and `trigger` and `list-runs` fail. Unpausing also starts one scheduled run, for the latest hour that has ended, so the last command can list two runs: ```bash docker exec airflow airflow dags list | grep feast_materialize docker exec airflow airflow dags unpause feast_materialize docker exec airflow airflow dags trigger feast_materialize docker exec airflow airflow dags list-runs feast_materialize ``` Run the last command again until each run shows `success`. Run the read from [Read the Features](#read-the-features) below. After the DAG run it prints `{'driver_id': [1001, 1002], 'conv_rate': [0.6000000238418579, 0.75]}`, because the DAG copied 0.6, which is a 32-bit float, into Valkey. ## Read the Features `FeatureStore(repo_path="feature_repo").get_historical_features(...)` takes a point-in-time join from Iceberg through DuckDB, for training. This reads the materialised values from Valkey instead, as a model server would: ```python from feast import FeatureStore fs = FeatureStore(repo_path="feature_repo") print(fs.get_online_features( features=["driver_stats:conv_rate"], entity_rows=[{"driver_id": 1001}, {"driver_id": 1002}], ).to_dict()) ``` It prints `{'driver_id': [1001, 1002], 'conv_rate': [0.5, 0.75]}`. Driver 1001 reads `0.6000000238418579` instead if you ran the DAG in [On a Schedule](#on-a-schedule). ## Serve from the Stack The UI at `http://127.0.0.1:8890` and the online feature server at `http://127.0.0.1:6566` load their repository from `s3://feast/repo`. To serve your features, upload the repository there with container addresses in its `feature_store.yaml`: `postgres:5432` and `valkey:6379`. The definitions need no change, because the services take the catalog address from `PYICEBERG_CATALOG__ODCTL__URI`, which is `http://catalog:8181` in `x-feast-env` of `.odctl/compose-mlops.yml` today. The services check that prefix every 15 seconds and restart Feast when it changes. The interval is set today by `time.sleep(15)` in `x-feast-repo-sync` in `.odctl/compose-mlops.yml`. Two other profiles also read their code from a bucket; [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#where-data-lives) compares the three. Once the repository is uploaded, the UI lists the project and its feature views: ## Push Features A push source lets a client write a feature value straight to the online store through the feature server's `/push` endpoint, instead of materialising it. A client can be `curl`, a Flink SQL job, or a small Python process that reads Kafka and calls `FeatureStore.push` with `PushMode.ONLINE`. That mode writes to the online store only, as `"to": "online"` does over HTTP. [Keep Messages Longer](https://jaehyeon.me/odctl/latest/guides/kafka/#keep-messages-longer) says how long Kafka keeps a stream for such a process to replay. The slower route writes the stream to Iceberg with Flink and runs `feast materialize-incremental` on a schedule, as [On a Schedule](#on-a-schedule) shows. A value then reaches the online store only after the next run. Feast needs a push source before it accepts a push. The feature server serves the project that its own `feature_store.yaml` names, `odctl`. You therefore apply the definitions from inside the `feast-serve` container. Save these definitions as `push_definitions.py`. They declare the push source `driver_push` and the feature view `driver_pushed`: ```python from datetime import timedelta from feast import Entity, FeatureView, Field, FileSource, PushSource from feast.types import Float32, Int64 from feast.value_type import ValueType driver = Entity(name="driver", join_keys=["driver_id"], value_type=ValueType.INT64) # Nothing reads this file. A push source needs a batch source behind it. batch = FileSource(path="/tmp/driver_stats.parquet", timestamp_field="event_timestamp") push = PushSource(name="driver_push", batch_source=batch) driver_pushed = FeatureView( name="driver_pushed", entities=[driver], ttl=timedelta(days=1), schema=[Field(name="driver_id", dtype=Int64), Field(name="conv_rate", dtype=Float32)], source=push, online=True, ) ``` Copy the definitions into the Feast container and apply them. `--skip-source-validation` stops Feast from looking for the file behind the batch source: ```bash docker cp push_definitions.py feast-serve:/tmp/feature_repo/push_definitions.py docker exec feast-serve feast -c /tmp/feature_repo apply --skip-source-validation ``` `apply` prints `Applying changes for project odctl` and `Deploying infrastructure for driver_pushed`. ### Over HTTP This sends the value 0.7 for driver 1001 to `/push`. The body names the push source and holds `df`, a table of columns in which each field is an array, and `"to": "online"` writes to the online store only: ```bash # Push 0.7 for driver 1001, stamped with the current time. curl -s -w '%{http_code}\n' -X POST http://127.0.0.1:6566/push \ -H 'Content-Type: application/json' -d @- <, conv_rate ARRAY, event_timestamp ARRAY>, allow_registry_cache BOOLEAN ) WITH ( 'connector' = 'http-async-sink', 'url' = 'http://feast-serve:6566/push', 'format' = 'json', 'http.sink.writer.request.mode' = 'single', 'http.sink.header.Content-Type' = 'application/json' ); INSERT INTO feast_push SELECT 'driver_push', ROW(ARRAY[CAST(1001 AS BIGINT)], ARRAY[CAST(0.9 AS DOUBLE)], ARRAY[CURRENT_TIMESTAMP]), FALSE; ``` The connector is `http-async-sink`. Each setting of the table has a reason: - `http.sink.writer.request.mode` is `single`, one request for each row, because Feast's `/push` takes one JSON object and the connector's default sends a batch as one array. - `http.sink.header.Content-Type` sets the header, because the sink sends none and Feast answers 422 without it. - The column names are the field names of the request body, and each field of `df` holds an array, as under Over HTTP. - `allow_registry_cache` is `FALSE` so that the push reads the registry again, as under Over HTTP. - The statement uses `CURRENT_TIMESTAMP` so that the timestamp is newer than the one the HTTP push stored. The Flink job finishes and prints `Complete execution of the SQL update statement.` whether or not Feast accepts the request. A rejected request shows only in the TaskManager log, as `Http Sink failed to write`. Run the read from Over HTTP again to check the push. The answer now holds `0.8999999761581421` for `conv_rate`, which is 0.9 as a 32-bit float. ## Connect an MCP Client MCP, the Model Context Protocol, lets an AI agent discover and call the tools of a server. `feast-serve` serves an MCP endpoint at `http://127.0.0.1:6566/mcp`, and each route of the feature server is a tool. The bundled repository sets `mcp_transport: http`, which is Streamable HTTP: the client sends each request as an HTTP POST to the one `/mcp` address. The MCP specification of 2025-03-26 put it in place of HTTP with server-sent events. odctl does not install an MCP client, so install the Python one in your environment: ```bash pip install "mcp>=1.24,<2" ``` Save this code as `mcp_client.py` and run it with `python mcp_client.py`. It lists the tools, then calls the tool for `/get-online-features` to read `driver_pushed:conv_rate`, the value pushed above. Run [Push Features](#push-features) first: without `driver_pushed`, the call fails and the script prints Feast's error, which names the missing feature view. ```python import asyncio import json from mcp import ClientSession from mcp.client.streamable_http import streamable_http_client async def main(): """List Feast's tools and read one feature through them. Returns: The error text when the tool call fails, otherwise None. """ async with streamable_http_client("http://127.0.0.1:6566/mcp") as (read, write, _): async with ClientSession(read, write) as session: await session.initialize() for tool in (await session.list_tools()).tools: print(tool.name) result = await session.call_tool( "get_online_features_get_online_features_post", {"features": ["driver_pushed:conv_rate"], "entities": {"driver_id": [1001]}}, ) if result.isError: return result.content[0].text reply = json.loads(result.content[0].text) names = reply["metadata"]["feature_names"] print(reply["results"][names.index("conv_rate")]["values"]) error = asyncio.run(main()) if error: raise SystemExit(error) ``` It prints the name of each tool, such as `push_push_post`, which is the route function, its path and its HTTP method, and then the value as `[0.8999999761581421]`. It prints `[0.699999988079071]` if you pushed over HTTP only. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), the feature server is `http://feast-serve:6566`. A repository for the services uses `postgres:5432` for the registry and `valkey:6379` for the online store, and its Iceberg source names no address. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace, in an environment with the Feast and MCP clients installed as above. It writes the repository files into `feature_repo` in that folder, and appends two rows to `demo.driver_stats` on each run: ```bash #!/usr/bin/env bash # Starts feast and flink-lite, builds an Iceberg table of driver features, # registers a feature view over it, materialises it into Valkey and reads it # back, then pushes a value to the feature server over HTTP and through Flink # SQL, reads that back, and lists the feature server's MCP tools. Run it in an # environment with the packages from the guide's two install steps. set -euo pipefail odctl up feast flink-lite export PYICEBERG_CATALOG__ODCTL__TYPE=rest export PYICEBERG_CATALOG__ODCTL__URI=http://127.0.0.1:8181 export PYICEBERG_CATALOG__ODCTL__WAREHOUSE=s3://warehouse/ export PYICEBERG_CATALOG__ODCTL__S3__ENDPOINT=http://127.0.0.1:8333 export PYICEBERG_CATALOG__ODCTL__S3__ACCESS_KEY_ID=user export PYICEBERG_CATALOG__ODCTL__S3__SECRET_ACCESS_KEY=password export PYICEBERG_CATALOG__ODCTL__S3__PATH_STYLE_ACCESS=true export PYICEBERG_CATALOG__ODCTL__S3__REGION=us-east-1 python - <<'PY' import datetime, pyarrow as pa from pyiceberg.catalog import load_catalog cat = load_catalog("odctl") cat.create_namespace_if_not_exists("demo") schema = pa.schema([ pa.field("driver_id", pa.int64(), nullable=False), pa.field("event_timestamp", pa.timestamp("us", tz="UTC"), nullable=False), pa.field("conv_rate", pa.float32(), nullable=False), ]) t = cat.create_table_if_not_exists("demo.driver_stats", schema=schema) now = datetime.datetime.now(datetime.timezone.utc).replace(microsecond=0) t.append(pa.Table.from_pydict({ "driver_id": [1001, 1002], "event_timestamp": [now - datetime.timedelta(hours=1)] * 2, "conv_rate": [0.5, 0.75], }, schema=schema)) PY mkdir -p feature_repo cat > feature_repo/feature_store.yaml <<'YAML' project: demo provider: local registry: registry_type: sql path: postgresql+psycopg://user:password@127.0.0.1:5432/feast offline_store: type: duckdb online_store: type: redis connection_string: "127.0.0.1:6379,username=user,password=password" entity_key_serialization_version: 3 YAML cat > feature_repo/definitions.py <<'PY' from datetime import timedelta from feast import Entity, FeatureView, Field from feast.types import Float32 from feast.value_type import ValueType from feast.infra.data_sources.contrib.iceberg_catalog.iceberg_source import IcebergSource driver = Entity(name="driver", join_keys=["driver_id"], value_type=ValueType.INT64) src = IcebergSource( warehouse="", namespace="demo", table="driver_stats", catalog_type="rest", catalog_name="odctl", timestamp_field="event_timestamp", ) fv = FeatureView( name="driver_stats", entities=[driver], ttl=timedelta(days=365), schema=[Field(name="conv_rate", dtype=Float32)], source=src, online=True, ) PY feast -c feature_repo apply --skip-source-validation feast -c feature_repo materialize-incremental "$(date -u +%Y-%m-%dT%H:%M:%S)" python - <<'PY' from feast import FeatureStore fs = FeatureStore(repo_path="feature_repo") print(fs.get_online_features( features=["driver_stats:conv_rate"], entity_rows=[{"driver_id": 1001}, {"driver_id": 1002}], ).to_dict()) PY # The push source goes into the feature server's own repository. cat > push_definitions.py <<'PY' from datetime import timedelta from feast import Entity, FeatureView, Field, FileSource, PushSource from feast.types import Float32, Int64 from feast.value_type import ValueType driver = Entity(name="driver", join_keys=["driver_id"], value_type=ValueType.INT64) # Nothing reads this file. A push source needs a batch source behind it. batch = FileSource(path="/tmp/driver_stats.parquet", timestamp_field="event_timestamp") push = PushSource(name="driver_push", batch_source=batch) driver_pushed = FeatureView( name="driver_pushed", entities=[driver], ttl=timedelta(days=1), schema=[Field(name="driver_id", dtype=Int64), Field(name="conv_rate", dtype=Float32)], source=push, online=True, ) PY docker cp push_definitions.py feast-serve:/tmp/feature_repo/push_definitions.py docker exec feast-serve feast -c /tmp/feature_repo apply --skip-source-validation # Push 0.7 for driver 1001 over HTTP, stamped with the current time. curl -s -w '%{http_code}\n' -X POST http://127.0.0.1:6566/push \ -H 'Content-Type: application/json' -d @- <, conv_rate ARRAY, event_timestamp ARRAY>, allow_registry_cache BOOLEAN ) WITH ( 'connector' = 'http-async-sink', 'url' = 'http://feast-serve:6566/push', 'format' = 'json', 'http.sink.writer.request.mode' = 'single', 'http.sink.header.Content-Type' = 'application/json' ); INSERT INTO feast_push SELECT 'driver_push', ROW(ARRAY[CAST(1001 AS BIGINT)], ARRAY[CAST(0.9 AS DOUBLE)], ARRAY[CURRENT_TIMESTAMP]), FALSE; SQL # The feature server can take up to a minute to see a new feature view. tries=0 until curl -s -X POST http://127.0.0.1:6566/get-online-features \ -H 'Content-Type: application/json' \ -d '{"features": ["driver_pushed:conv_rate"], "entities": {"driver_id": [1001]}}' \ | grep 0.8999999761581421; do tries=$((tries + 1)) [ "$tries" -lt 24 ] || { echo "Flink's push did not arrive. Look for 'Http Sink failed to write' in the TaskManager log." >&2; exit 1; } sleep 5 done python - <<'PY' import asyncio import json from mcp import ClientSession from mcp.client.streamable_http import streamable_http_client async def main(): async with streamable_http_client("http://127.0.0.1:6566/mcp") as (read, write, _): async with ClientSession(read, write) as session: await session.initialize() for tool in (await session.list_tools()).tools: print(tool.name) result = await session.call_tool( "get_online_features_get_online_features_post", {"features": ["driver_pushed:conv_rate"], "entities": {"driver_id": [1001]}}, ) if result.isError: return result.content[0].text reply = json.loads(result.content[0].text) names = reply["metadata"]["feature_names"] print(reply["results"][names.index("conv_rate")]["values"]) error = asyncio.run(main()) if error: raise SystemExit(error) PY ``` It prints `{'driver_id': [1001, 1002], 'conv_rate': [0.5, 0.75]}`, the status `200` of the HTTP push and the feature server's answer with `0.8999999761581421`. Then it prints the name of each MCP tool and the value `[0.8999999761581421]`. Driver 1001 reads `0.6000000238418579` in the first value instead if you ran the DAG in [On a Schedule](#on-a-schedule). # Vector Retrieval with Feast Feast can return the stored rows whose embeddings are nearest to a query embedding, which is how a retrieval step finds the documents that match a question. An embedding is a list of numbers that represents a text, and the nearest embeddings belong to the most similar texts. This guide runs a second Feast repository, `vector_repo`, whose online store is PostgreSQL with pgvector, the extension that stores vectors and finds the nearest ones. Feast's `retrieve_online_documents_v2` is the call that returns them. The end-to-end tests run the steps below on every release. This guide builds on [Feast with Iceberg](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/index.md), which defines feature views and a feature repository, and on [Hybrid Search](https://jaehyeon.me/odctl/latest/guides/hybrid-search/index.md), which describes the `vector` database and its pgvector extension. | Endpoint | Address | | ------------------------------------- | --------------------------------------------------------- | | PostgreSQL from the host | `127.0.0.1:5432`, user `user`, password `password` | | PostgreSQL inside the `odctl` network | `postgres:5432` | | Database for the vectors | `vector` | | Registry | the `feast` database on PostgreSQL, project `vector_demo` | The Feast client needs Python 3.11 or newer. Start PostgreSQL and install the client, with the extras that read DuckDB and PostgreSQL: ```bash odctl up postgres pip install "feast[duckdb,postgres]==0.68.0" ``` ## Define a Second Repository A feature repository has one online store. The default repository of the `feast` profile uses Feast's Redis online store for Valkey, and that store has no document retrieval method. Vectors therefore go in a second repository whose online store is PostgreSQL. The `valkey` profile also has a 512 MB memory limit, set by `mem_limit` of `valkey` in `.odctl/compose-store.yml`, so the vectors of a large corpus belong in PostgreSQL too. Save this file as `vector_repo/feature_store.yaml`. `vector_enabled: true` makes the online store keep vectors in pgvector columns. `sslmode: disable` is needed because Feast asks for SSL by default and odctl's PostgreSQL does not serve it: ```yaml project: vector_demo provider: local registry: registry_type: sql path: postgresql+psycopg://user:password@127.0.0.1:5432/feast offline_store: type: duckdb online_store: type: postgres host: 127.0.0.1 port: 5432 database: vector user: user password: password sslmode: disable vector_enabled: true entity_key_serialization_version: 3 ``` Save the definitions as `vector_repo/definitions.py`. They declare the entity `doc` and the feature view `docs`. The field `embedding` holds three numbers, as in Hybrid Search, and `vector_index=True` marks it as the field to search: ```python from datetime import timedelta from feast import Entity, FeatureView, Field, FileSource from feast.types import Array, Float32, String from feast.value_type import ValueType doc = Entity(name="doc", join_keys=["doc_id"], value_type=ValueType.INT64) # Nothing reads this file. A feature view needs a batch source, and the rows go # straight to the online store. batch = FileSource(path="/tmp/docs.parquet", timestamp_field="event_timestamp") docs = FeatureView( name="docs", entities=[doc], ttl=timedelta(days=365), schema=[ Field(name="embedding", dtype=Array(Float32), vector_index=True, vector_length=3), Field(name="title", dtype=String), ], source=batch, online=True, ) ``` Run this from the folder that holds `vector_repo`. `apply` writes the definitions to the registry and creates the table for `docs` in the `vector` database. `--skip-source-validation` stops Feast from looking for the file behind the batch source: ```bash feast -c vector_repo apply --skip-source-validation ``` `apply` prints `Applying changes for project vector_demo` and `Deploying infrastructure for docs`. ## Write and Retrieve Save this code as `write.py` and run it with `python write.py` in the same folder. It writes three documents to the online store, each with its embedding. The `event_timestamp` column is required, and a second run replaces the three rows: ```python import datetime import pandas as pd from feast import FeatureStore fs = FeatureStore(repo_path="vector_repo") now = datetime.datetime.now(datetime.timezone.utc) fs.write_to_online_store("docs", pd.DataFrame({ "doc_id": [1, 2, 3], "title": ["Feast", "Flink", "Iceberg"], "embedding": [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.9, 0.1, 0.0]], "event_timestamp": [now] * 3, })) ``` Then save this code as `retrieve.py` and run it with `python retrieve.py`. It asks for the two documents nearest to `[1, 0, 0]` by Euclidean distance (`L2`): ```python from feast import FeatureStore fs = FeatureStore(repo_path="vector_repo") result = fs.retrieve_online_documents_v2( features=["docs:embedding", "docs:title"], query=[1.0, 0.0, 0.0], top_k=2, distance_metric="L2", ).to_dict() print(result["title"]) ``` It prints `['Feast', 'Iceberg']`. The query is `[1, 0, 0]`, the same vector as `Feast`. `Iceberg` at `[0.9, 0.1, 0]` is the next nearest. `Flink` at `[0, 1, 0]` is the farthest, and `top_k=2` leaves it out. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), a repository for the services sets `host: postgres` in `online_store` and uses `postgres:5432` in the registry path, with the same user and password. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace, in an environment with the client installed as above. It writes the repository files into `vector_repo` in that folder. A second run registers the same definitions again and replaces the three rows: ```bash #!/usr/bin/env bash # Starts postgres, writes a second feature repository whose online store is # PostgreSQL with pgvector, registers a feature view of three documents, writes # their embeddings and retrieves the two nearest to a query vector. Run it in an # environment with the packages from the guide's install step. set -euo pipefail odctl up postgres mkdir -p vector_repo cat > vector_repo/feature_store.yaml <<'YAML' project: vector_demo provider: local registry: registry_type: sql path: postgresql+psycopg://user:password@127.0.0.1:5432/feast offline_store: type: duckdb online_store: type: postgres host: 127.0.0.1 port: 5432 database: vector user: user password: password sslmode: disable vector_enabled: true entity_key_serialization_version: 3 YAML cat > vector_repo/definitions.py <<'PY' from datetime import timedelta from feast import Entity, FeatureView, Field, FileSource from feast.types import Array, Float32, String from feast.value_type import ValueType doc = Entity(name="doc", join_keys=["doc_id"], value_type=ValueType.INT64) # Nothing reads this file. A feature view needs a batch source, and the rows go # straight to the online store. batch = FileSource(path="/tmp/docs.parquet", timestamp_field="event_timestamp") docs = FeatureView( name="docs", entities=[doc], ttl=timedelta(days=365), schema=[ Field(name="embedding", dtype=Array(Float32), vector_index=True, vector_length=3), Field(name="title", dtype=String), ], source=batch, online=True, ) PY feast -c vector_repo apply --skip-source-validation python - <<'PY' import datetime import pandas as pd from feast import FeatureStore fs = FeatureStore(repo_path="vector_repo") now = datetime.datetime.now(datetime.timezone.utc) fs.write_to_online_store("docs", pd.DataFrame({ "doc_id": [1, 2, 3], "title": ["Feast", "Flink", "Iceberg"], "embedding": [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.9, 0.1, 0.0]], "event_timestamp": [now] * 3, })) result = fs.retrieve_online_documents_v2( features=["docs:embedding", "docs:title"], query=[1.0, 0.0, 0.0], top_k=2, distance_metric="L2", ).to_dict() print(result["title"]) PY ``` It prints the lines of `apply` and then `['Feast', 'Iceberg']`. # MLflow Tracking and Model Serving The `mlflow` profile runs two containers. `mlflow` is the tracking server and model registry, with runs on PostgreSQL and artifacts on SeaweedFS under `s3://mlflow`. `mlflow-serve` scores a registered model over HTTP. The end-to-end tests run the commands below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). Its last section reads features from [Feast with Iceberg](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#push-features). Your code logs runs and models to the tracking server, and `mlflow-serve` loads a registered model from it: Start the profile: ```bash odctl up mlflow ``` | Endpoint | Address | | --------------------------------------- | -------------------------------------- | | Tracking UI and API from the host | `http://127.0.0.1:5004` | | Tracking API inside the `odctl` network | `http://mlflow:5000` | | Model server from the host | `http://127.0.0.1:5003/invocations` | | Model server inside the `odctl` network | `http://mlflow-serve:5003/invocations` | The server runs MLflow 3.18.0, so install the same client version on the host. PostgreSQL keeps the `mlflow` database across `odctl recreate` ([Where Data Lives](https://jaehyeon.me/odctl/latest/how-it-works/#where-data-lives)), and a server refuses a database whose schema an older MLflow created. So the `mlflow` command in `.odctl/compose-mlops.yml` runs `mlflow db upgrade` before the server on every start. The upgrade creates the tables on an empty database, upgrades an older one and leaves a current one unchanged. It adds about a second to each start. The server also runs MLflow's AI Gateway and automatic evaluation, whose judges run in the server's job workers. The `mlflow` service sets `MLFLOW_GATEWAY_URI=http://127.0.0.1:5000`, so a judge with a `gateway:/` model reaches the gateway at an address that `--allowed-hosts` accepts. ## Log a Run with an Artifact This logs a run with a parameter, a metric and a small file as an artifact, then prints where the artifacts are stored and the run's artifact list: ```python import pathlib import mlflow mlflow.set_tracking_uri("http://127.0.0.1:5004") mlflow.set_experiment("demo") p = pathlib.Path("notes.txt") p.write_text("hello") with mlflow.start_run() as run: mlflow.log_param("k", "v") mlflow.log_metric("m", 1.0) mlflow.log_artifact(str(p)) print(mlflow.get_run(run.info.run_id).info.artifact_uri) print([f.path for f in mlflow.MlflowClient().list_artifacts(run.info.run_id)]) ``` The code prints the artifact URI and the list `['notes.txt']`. The artifact URI starts with `mlflow-artifacts:/`. The server proxies artifacts, so the client uploads through the tracking server and needs no S3 address or keys. Downloads go through the tracking server too, as [Load a Model from Your Machine](#load-a-model-from-your-machine) shows. The run and its artifact appear in the UI. The runs are listed under the experiment in the Model training view: ## Register a Model This trains a small XGBoost classifier, registers it as `demo-model` and points the alias `champion` at the latest version: ```python import mlflow, numpy as np, xgboost as xgb from mlflow import MlflowClient mlflow.set_tracking_uri("http://127.0.0.1:5004") mlflow.set_experiment("demo-model") X, y = np.array([[0.0], [1.0], [2.0], [3.0]]), np.array([0, 0, 1, 1]) with mlflow.start_run(): model = xgb.XGBClassifier(n_estimators=5, max_depth=2).fit(X, y) mlflow.xgboost.log_model(model, name="model", registered_model_name="demo-model") client = MlflowClient() version = max(int(mv.version) for mv in client.search_model_versions("name='demo-model'")) client.set_registered_model_alias("demo-model", "champion", version) ``` The run prints `Created version '1' of model 'demo-model'`. This needs XGBoost on the host. The model server has XGBoost 3.2, so install the same minor version to log a model it can load. The registry lists each version, with the `champion` alias on the latest: ## Serve the Model `mlflow-serve` starts with the profile and waits idle while `MODEL_URI` is empty. `MODEL_URI` is set in `.odctl/.env`, which `odctl init` writes. Change its line there to: ```bash MODEL_URI="models:/demo-model@champion" ``` Then replace the containers with `odctl recreate mlflow`, so `mlflow-serve` starts with the new value, and score two rows: ```bash odctl recreate mlflow curl -X POST http://127.0.0.1:5003/invocations \ -H 'Content-Type: application/json' -d '{"inputs": [[0.0], [3.0]]}' ``` The response is `{"predictions": [0, 0]}`, one prediction per input row. Check that the body holds `predictions`: a scoring failure answers 400 with the error in the body, and `curl` without `--fail` still exits 0. `http://127.0.0.1:5003/ping` answers once the model is loaded. ## Move the Alias to a New Version `mlflow-serve` resolves `MODEL_URI` once, when its container starts. A new version registered behind an unchanged alias such as `champion` is not served until you run `odctl restart mlflow`. Nothing reports an error in the meantime, and the predictions stay those of the old version. To see the change in `.env`, use a version URI such as `models:/demo-model/2` and run `odctl recreate mlflow`. Run the code from [Register a Model](#register-a-model) again with `min_child_weight=0` added to `XGBClassifier`. The trees then split, so the new version predicts differently. It registers version 2 and moves `champion` to it. Then restart the containers and score again: ```bash odctl restart mlflow # The server answers nothing while the model loads: 16 seconds in one run. until curl -fs http://127.0.0.1:5003/ping >/dev/null; do sleep 2; done curl -X POST http://127.0.0.1:5003/invocations \ -H 'Content-Type: application/json' -d '{"inputs": [[0.0], [3.0]]}' ``` The predictions now come from version 2: `{"predictions": [0, 1]}` instead of `[0, 0]`. ## Load a Model from Your Machine Downloads go through the tracking server, as uploads do, so `mlflow.artifacts.download_artifacts` and a `load_model` call such as `mlflow.sklearn.load_model` work from your machine with no setting: ```python import os import mlflow mlflow.set_tracking_uri("http://127.0.0.1:5004") path = mlflow.artifacts.download_artifacts("models:/demo-model@champion") print(sorted(os.listdir(path))) ``` The output is a listing of the model files that includes `MLmodel`. ## Serve a Model That Reads Online Features The MLflow image has no Feast client. A model can still read online features from `feast-serve` over HTTP, with an `mlflow.pyfunc.PythonModel` whose `predict` method sends the request. This one reads `conv_rate` of the `driver_pushed` feature view and scores a driver 1 when it is above 0.5. Apply the push definitions as [Push Features](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#push-features) shows, then run [From Flink SQL](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#from-flink-sql), so the online store holds `conv_rate` 0.9 for driver 1001, then save the model class as `driver_score.py`: ```python import mlflow import requests class DriverScore(mlflow.pyfunc.PythonModel): """Scores a driver 1 when the online store holds a conversion rate above 0.5.""" def predict(self, context, model_input, params=None): reply = requests.post( "http://feast-serve:6566/get-online-features", json={ "features": ["driver_pushed:conv_rate"], "entities": {"driver_id": [int(i) for i in model_input["driver_id"]]}, }, timeout=10, ).json() names = reply["metadata"]["feature_names"] rates = reply["results"][names.index("conv_rate")]["values"] return [int(rate is not None and rate > 0.5) for rate in rates] mlflow.models.set_model(DriverScore()) ``` Then register it from a file, so that the server loads the class from the code and not from a copy pickled on your machine. Run this in the folder that holds `driver_score.py`. It gives the model a signature and no input example, because an example would make MLflow call `predict` on your machine, where `feast-serve` does not resolve: ```python import mlflow, pandas as pd from mlflow import MlflowClient from mlflow.models import infer_signature mlflow.set_tracking_uri("http://127.0.0.1:5004") mlflow.set_experiment("driver-score") with mlflow.start_run(): mlflow.pyfunc.log_model( name="model", python_model="driver_score.py", pip_requirements=["requests"], signature=infer_signature(pd.DataFrame({"driver_id": [1001]}), [1]), registered_model_name="driver-score", ) client = MlflowClient() version = max(int(mv.version) for mv in client.search_model_versions("name='driver-score'")) client.set_registered_model_alias("driver-score", "champion", version) ``` The run prints three warnings, about integer columns, type hints on `predict` and the missing input example, and then `Created version '1' of model 'driver-score'`. The warnings do not stop the registration. Serve it as in [Serve the Model](#serve-the-model), with this line in `.odctl/.env`: ```bash MODEL_URI="models:/driver-score@champion" ``` Run `odctl recreate mlflow`, then score the driver: ```bash curl -X POST http://127.0.0.1:5003/invocations \ -H 'Content-Type: application/json' \ -d '{"dataframe_split": {"columns": ["driver_id"], "data": [[1001]]}}' ``` The reply is `{"predictions": [1]}`. Right after `apply`, `feast-serve` may not know the feature view yet (see [Over HTTP](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#over-http)), and the reply is an error. Run the command again after a minute. The model costs one HTTP request to `feast-serve` for each prediction request, and `mlflow-serve` keeps its 1,500 MB memory limit, which a model with large libraries can reach. The limit is set today by `mem_limit` in the `mlflow-serve` service of `.odctl/compose-mlops.yml`, where an edit can change it. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), set the tracking URI to `http://mlflow:5000` and score at `http://mlflow-serve:5003/invocations`. Airflow tasks use these addresses too. Extra Python packages for both MLflow containers go in `_MLOPS_PIP_DEPS` in `.odctl/.env`, followed by `odctl recreate mlflow`. [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) says when those packages are installed again. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace, in an environment with the MLflow and XGBoost versions named above installed. It logs a run, registers `demo-model`, sets `MODEL_URI` in `.odctl/.env` and scores two rows: ```bash #!/usr/bin/env bash # Starts mlflow, logs a run with an artifact, registers an XGBoost model with # the alias champion, serves it with mlflow-serve and scores two rows. Run it # in the folder that holds your .odctl workspace, in an environment with the # MLflow and XGBoost versions that the guide names. set -euo pipefail odctl up mlflow python - <<'PY' import pathlib import mlflow, numpy as np, xgboost as xgb from mlflow import MlflowClient mlflow.set_tracking_uri("http://127.0.0.1:5004") mlflow.set_experiment("demo") p = pathlib.Path("notes.txt") p.write_text("hello") with mlflow.start_run() as run: mlflow.log_param("k", "v") mlflow.log_metric("m", 1.0) mlflow.log_artifact(str(p)) print(mlflow.get_run(run.info.run_id).info.artifact_uri) mlflow.set_experiment("demo-model") X, y = np.array([[0.0], [1.0], [2.0], [3.0]]), np.array([0, 0, 1, 1]) with mlflow.start_run(): model = xgb.XGBClassifier(n_estimators=5, max_depth=2).fit(X, y) mlflow.xgboost.log_model(model, name="model", registered_model_name="demo-model") client = MlflowClient() version = max(int(mv.version) for mv in client.search_model_versions("name='demo-model'")) client.set_registered_model_alias("demo-model", "champion", version) PY # Serve the champion: set MODEL_URI in the workspace and replace the containers. sed -i.bak 's|^MODEL_URI=.*|MODEL_URI="models:/demo-model@champion"|' .odctl/.env odctl recreate mlflow curl -s -X POST http://127.0.0.1:5003/invocations \ -H 'Content-Type: application/json' -d '{"inputs": [[0.0], [3.0]]}' echo ``` It prints the artifact URI, `Created version` with the new version number, the recreate messages and `{"predictions": [0, 0]}`. The `sed` command keeps the old `.env` as `.odctl/.env.bak`. # Evidently Reports The `evidently` profile runs the Evidently UI and API. Evidently checks data drift and data quality. Data drift is a change in the distribution of a column between a reference dataset and current data. Your client computes each report and pushes the result to the server. The server keeps projects, reports, dashboards and datasets in the `evidently` database on PostgreSQL. The end-to-end tests run the code below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). | Endpoint | Address | | ------------------------------ | ----------------------- | | UI and API from the host | `http://127.0.0.1:8089` | | API inside the `odctl` network | `http://evidently:8000` | Start the profile and install the client: ```bash odctl up evidently pip install evidently==0.7.23 "pandas<3.1" ``` The install command pins the client to the server's version. That version fails to import with pandas 3.1, so the command keeps pandas below 3.1. The UI has no login. ## Push a Data Drift Report This creates the project `demo`, builds a reference dataset and a current one, computes a data drift report and pushes it to the server: ```python import numpy as np, pandas as pd from evidently import Report from evidently.presets import DataDriftPreset from evidently.ui.workspace import RemoteWorkspace ws = RemoteWorkspace("http://127.0.0.1:8089") project = ws.create_project("demo") rng = np.random.default_rng(7) reference = pd.DataFrame({"x": rng.normal(0, 1, 500), "y": rng.normal(0, 1, 500)}) current = pd.DataFrame({"x": rng.normal(3, 1, 500), "y": rng.normal(0, 1, 500)}) snapshot = Report([DataDriftPreset()]).run(current_data=current, reference_data=reference) ws.add_run(project.id, snapshot) ``` The code prints nothing. Column `x` moves by three standard deviations and `y` does not, so the report in the UI shows one drifted column. Open the report under the project's Reports tab: ## Store a Dataset This stores a small table as a dataset in the same project and reads it back: ```python from evidently import DataDefinition, Dataset data = pd.DataFrame({"id": [1, 2, 3], "score": [0.1, 0.5, 0.9]}) dataset_id = ws.add_dataset(project.id, Dataset.from_pandas(data, data_definition=DataDefinition()), "demo") print(ws.load_dataset(dataset_id).as_dataframe()) ``` It prints the dataset read back from the server, with the columns `id` and `score` and three rows. The dataset is stored in the `evidently` database, with the projects and reports. ## Performance Without Outcomes Evidently gives no estimate of a model's performance while the outcomes are unknown. NannyML gives one. NannyML 0.13.1 supports Python 3.9 to 3.12, and the Airflow image runs Python 3.13. Run NannyML in a Python 3.12 process on your machine instead. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), connect with `RemoteWorkspace("http://evidently:8000")`. For Airflow tasks, set `_AIRFLOW_PIP_DEPS="evidently==0.7.23"` in `.odctl/.env` and run `odctl recreate airflow`. The server's settings are in `evidently/config.yaml` in the workspace. ## Runnable Example Save this file as `example.py`. Start the profile and install the client as above, then run it with `python example.py`. It creates the project `demo-example`: ```python """Pushes a data drift report to Evidently, stores a dataset and reads it back. Start the profile with `odctl up evidently` and install the client first, as the guide's install step shows. """ import numpy as np import pandas as pd from evidently import DataDefinition, Dataset, Report from evidently.presets import DataDriftPreset from evidently.ui.workspace import RemoteWorkspace ws = RemoteWorkspace("http://127.0.0.1:8089") project = ws.create_project("demo-example") # Column x moves by three standard deviations and y does not. rng = np.random.default_rng(7) reference = pd.DataFrame({"x": rng.normal(0, 1, 500), "y": rng.normal(0, 1, 500)}) current = pd.DataFrame({"x": rng.normal(3, 1, 500), "y": rng.normal(0, 1, 500)}) snapshot = Report([DataDriftPreset()]).run(current_data=current, reference_data=reference) ws.add_run(project.id, snapshot) print(f"report pushed to project {project.id}") data = pd.DataFrame({"id": [1, 2, 3], "score": [0.1, 0.5, 0.9]}) dataset = Dataset.from_pandas(data, data_definition=DataDefinition()) dataset_id = ws.add_dataset(project.id, dataset, "demo") print(ws.load_dataset(dataset_id).as_dataframe()) ``` It prints `report pushed to project` with the project's id, and then the dataset with its three rows. # Labelling Data with Label Studio The `label-studio` profile runs Label Studio, where people label data: a person reads each task and picks its label. A project holds the tasks and the labelling setup, and each label a person gives is an annotation. Label Studio keeps projects, tasks and annotations in the `labelstudio` database on odctl's PostgreSQL. The end-to-end tests run the commands below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). Its samples are requests to the driver model of the [MLflow guide](https://jaehyeon.me/odctl/latest/guides/mlflow/#serve-a-model-that-reads-online-features), which scores a driver 1 when the `conv_rate` feature is above 0.5. Run the commands in the folder that holds your `.odctl` workspace, and in one terminal, because later steps use the variables that earlier ones set. Start the profile: ```bash odctl up label-studio ``` | Endpoint | Address | | -------------------------------------------------------------------- | -------------------------- | | UI and API from the host (user `user@odctl.io`, password `password`) | `http://127.0.0.1:8088` | | API inside the `odctl` network | `http://label-studio:8080` | | API token, sent as `Authorization: Token ` | `odctl-label-studio-token` | `odctl up label-studio` starts PostgreSQL first. Label Studio creates the user and its token on its first start, from the settings in `.odctl/compose-mlops.yml`, where you can change them before the first start. Labels go from Label Studio to a table that your training and monitoring code can join to its predictions: Both green boxes are the same `postgres` container: Label Studio's own data is in the `labelstudio` database, and the label table is `dev.labels` in the `odctl` database. ## Log In Open and log in with the user and password in the table above. The login takes an email address, not a user name. The API steps below use the same user's token instead of the password. ## Create a Project A project's label config is the labelling setup, written in Label Studio's XML tags. This one shows each task's `text` and offers two choices, `1` and `0`, the two scores the driver model gives. The command creates the project and keeps its ID in `PROJECT_ID`: ```bash # A project that labels each request 1 or 0, the two scores the driver model # gives. It prints the project ID. $text names each task's text field, which # Label Studio fills in, so it stays in single quotes. The shellcheck line tells # the repository's shell linter that this is intended. TOKEN=odctl-label-studio-token # shellcheck disable=SC2016 PROJECT_ID=$(curl -s -X POST http://127.0.0.1:8088/api/projects \ -H "Authorization: Token $TOKEN" -H "Content-Type: application/json" \ -d '{"title": "Driver requests", "label_config": ""}' \ | python3 -c "import json, sys; print(json.load(sys.stdin)['id'])") echo "$PROJECT_ID" ``` It prints the project ID, which is `1` for the first project: ```text 1 ``` ## Import Tasks Each task is a JSON object. Its keys become the task's data: `text` is the field the label config shows, and `sample_id` is the ID of the request, so the label can be joined back to the request's prediction later: ```bash # Two tasks, each a request's ID and a text that describes the request. curl -s -X POST "http://127.0.0.1:8088/api/projects/$PROJECT_ID/import" \ -H "Authorization: Token $TOKEN" -H "Content-Type: application/json" \ -d '[{"sample_id": "r-1", "text": "Driver 1001, conv_rate 0.9"}, {"sample_id": "r-2", "text": "Driver 1002, conv_rate 0.75"}]' echo ``` ```text {"task_count":2,"annotation_count":0,"prediction_count":0,"duration":0.013175487518310547,"file_upload_ids":[],"could_be_tasks_list":false,"found_formats":[],"data_columns":[]} ``` Read the tasks back: ```bash # This prints the sample ID and text of each task in the project. curl -s "http://127.0.0.1:8088/api/tasks?project=$PROJECT_ID" -H "Authorization: Token $TOKEN" \ | python3 -c "import json, sys; [print(t['data']['sample_id'], t['data']['text']) for t in json.load(sys.stdin)['tasks']]" ``` ```text r-1 Driver 1001, conv_rate 0.9 r-2 Driver 1002, conv_rate 0.75 ``` ## Label the Tasks Open the project at , or use your `PROJECT_ID` in place of `1`. Open a task, pick a choice and click Submit. This is the labelling view of the first task after it was labelled `1`. The task list is on the left, and the task's text and the two choices are in the middle: If you labelled the tasks in the UI, skip the next command, because it adds a second annotation to each task. Otherwise a script can add the annotations through the API, which is how the end-to-end tests label them. This labels `r-1` with `1` and `r-2` with `0`. A label is what a person judges, so it can differ from the model's score: ```bash # Labels the first task 1 and the second 0 from a script, as a person does in # the UI. read -r FIRST SECOND < <(curl -s "http://127.0.0.1:8088/api/tasks?project=$PROJECT_ID" -H "Authorization: Token $TOKEN" \ | python3 -c "import json, sys; print(*[t['id'] for t in json.load(sys.stdin)['tasks']])") annotate() { curl -s -o /dev/null -w "%{http_code}\n" -X POST "http://127.0.0.1:8088/api/tasks/$1/annotations" \ -H "Authorization: Token $TOKEN" -H "Content-Type: application/json" \ -d "{\"result\": [{\"from_name\": \"label\", \"to_name\": \"text\", \"type\": \"choices\", \"value\": {\"choices\": [\"$2\"]}}]}" } annotate "$FIRST" 1 annotate "$SECOND" 0 ``` Label Studio answers 201 for each annotation it creates: ```text 201 201 ``` ## Load the Labels into PostgreSQL Export the project's tasks with their annotations as JSON. The file `annotations.json` holds one object for each task that has at least one annotation, with its data and its list of annotations: ```bash # Exports the project's tasks with their annotations as JSON. curl -s "http://127.0.0.1:8088/api/projects/$PROJECT_ID/export?exportType=JSON" \ -H "Authorization: Token $TOKEN" -o annotations.json ``` The label table, `dev.labels`, has four columns: the sample ID, the label, the source of the label and its arrival time, which is when the label reached the table. Labels can come from more than one source, such as people in Label Studio and a script that labels by rule, so the table's primary key is the sample ID and the source together. The table is in the `dev` schema of the `odctl` database. The database's search path starts with `cdc`, and a Debezium source reads every table in `cdc`, as the [Kafka Connect guide](https://jaehyeon.me/odctl/latest/guides/kafka-connect/index.md) shows, so the SQL names `dev`. Save this script as `load.py`. It creates the label table if it does not exist, and writes one row for each labelled task, from its latest annotation, with the source `label-studio`. A task you skipped in the UI gives no row. Loading a later export again changes a row only when its label changed, and then sets its arrival time again: ```python import json import subprocess def quote(text): """Quote a value as an SQL string literal.""" return "'" + text.replace("'", "''") + "'" statements = [ "SET client_min_messages = warning;", """CREATE TABLE IF NOT EXISTS dev.labels ( sample_id text NOT NULL, label text NOT NULL, source text NOT NULL, arrived_at timestamptz NOT NULL DEFAULT now(), PRIMARY KEY (sample_id, source) );""", ] with open("annotations.json") as file: tasks = json.load(file) for task in tasks: # A task skipped in the UI has an annotation with no result. A task with # more than one annotation keeps its latest. done = [a for a in task["annotations"] if not a["was_cancelled"] and a["result"]] if not done: continue latest = max(done, key=lambda annotation: annotation["updated_at"]) label = latest["result"][0]["value"]["choices"][0] statements.append( "INSERT INTO dev.labels (sample_id, label, source)" f" VALUES ({quote(task['data']['sample_id'])}, {quote(label)}, 'label-studio')" " ON CONFLICT (sample_id, source) DO UPDATE" " SET label = EXCLUDED.label, arrived_at = now()" " WHERE labels.label <> EXCLUDED.label;" ) subprocess.run( ["docker", "exec", "-i", "postgres", "psql", "-q", "-v", "ON_ERROR_STOP=1", "-U", "user", "-d", "odctl"], input="\n".join(statements), text=True, check=True, ) ``` Run it and read the table: ```bash python3 load.py && docker exec postgres psql -U user -d odctl -c "SELECT sample_id, label, source FROM dev.labels ORDER BY sample_id" ``` ```text sample_id | label | source -----------+-------+-------------- r-1 | 1 | label-studio r-2 | 0 | label-studio (2 rows) ``` ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), call the API at `http://label-studio:8080` with the same token. The container calls Label Studio's own port, 8080, not the port published on the host: ```bash # Inside the odctl network, Label Studio is label-studio:8080. This prints the # email of the user that the token belongs to. docker run --rm --network odctl curlimages/curl:8.17.0 -s \ -H "Authorization: Token odctl-label-studio-token" http://label-studio:8080/api/current-user/whoami \ | python3 -c "import json, sys; print(json.load(sys.stdin)['email'])" ``` ```text user@odctl.io ``` ## Connect Other Services Label Studio 1.23.1 and later refuse calls to private addresses, and every container on the `odctl` network has one. odctl sets `ML_BLOCK_LOCAL_IP` and `USE_DEFAULT_BANNED_SUBNETS` to `false` in `.odctl/compose-mlops.yml`, so Label Studio can call an ML backend or a webhook receiver that runs as a container on that network. The UI and API are published on `127.0.0.1` only. Give that container a network alias with a dot, such as `docker run --network odctl --network-alias my-backend.odctl ...`, and use `http://my-backend.odctl:9090`. Label Studio's URL check refuses a host name with no dot: a webhook at `http://my-backend:9090` is refused with "Enter a valid URL", and an ML backend at that address connects but gets no automatic webhook, so annotation events never reach it. The Start Training button still works, because it calls the backend's address directly. For S3 storage, start the `storage` profile with `odctl up storage`, which creates the `label-studio` bucket. odctl sets `LABEL_STUDIO_S3_ENDPOINT` to `http://seaweed:8333`, an address Label Studio trusts, so leave the endpoint field of a storage connection empty and give the bucket `label-studio` with the access key `user` and the secret `password`. Turn off pre-signed URLs (`presign=False` in the API) for files that a browser loads: a pre-signed URL names `seaweed:8333`, which the browser on your host cannot open. With them off, Label Studio sends each file itself. ## Runnable Example Save this file as `example.py`. It creates a project, imports two tasks, labels them through the API and loads the labels into the label table. Start the profile first, then run it with `python3 example.py`: ```python """Labels two samples in Label Studio and loads the labels into PostgreSQL. It creates a project, imports two tasks, labels them through the API, exports the annotations and loads them into the label table, `dev.labels`, in the odctl database. Start the profile with `odctl up label-studio` first, then run `python3 example.py`. It uses only the Python standard library and `docker exec` for psql. """ import json import subprocess import urllib.request URL = "http://127.0.0.1:8088" TOKEN = "odctl-label-studio-token" LABEL_CONFIG = ( '' '' '' ) # Each sample is a request to the driver model: its ID, a text that describes it, # and the score a person gives it. SAMPLES = [ ("e-1", "Driver 1001, conv_rate 0.9", "1"), ("e-2", "Driver 1002, conv_rate 0.75", "0"), ] def call(path, body=None): """Send one request to the Label Studio API and return its JSON answer.""" request = urllib.request.Request( URL + path, data=None if body is None else json.dumps(body).encode(), headers={"Authorization": f"Token {TOKEN}", "Content-Type": "application/json"}, ) with urllib.request.urlopen(request) as response: return json.load(response) def quote(text): """Quote a value as an SQL string literal.""" return "'" + text.replace("'", "''") + "'" project = call("/api/projects", {"title": "Example", "label_config": LABEL_CONFIG}) call( f"/api/projects/{project['id']}/import", [{"sample_id": sample_id, "text": text} for sample_id, text, _ in SAMPLES], ) tasks = call(f"/api/tasks?project={project['id']}")["tasks"] labels = {sample_id: label for sample_id, _, label in SAMPLES} for task in tasks: # A person labels each task in the UI; this script stands in for them. choice = {"choices": [labels[task["data"]["sample_id"]]]} result = {"from_name": "label", "to_name": "text", "type": "choices", "value": choice} call(f"/api/tasks/{task['id']}/annotations", {"result": [result]}) statements = [ "SET client_min_messages = warning;", "CREATE TABLE IF NOT EXISTS dev.labels (sample_id text NOT NULL, label text NOT NULL," " source text NOT NULL, arrived_at timestamptz NOT NULL DEFAULT now()," " PRIMARY KEY (sample_id, source));", ] for task in call(f"/api/projects/{project['id']}/export?exportType=JSON"): done = [a for a in task["annotations"] if not a["was_cancelled"] and a["result"]] if not done: continue latest = max(done, key=lambda annotation: annotation["updated_at"]) label = latest["result"][0]["value"]["choices"][0] statements.append( "INSERT INTO dev.labels (sample_id, label, source)" f" VALUES ({quote(task['data']['sample_id'])}, {quote(label)}, 'label-studio')" " ON CONFLICT (sample_id, source) DO UPDATE" " SET label = EXCLUDED.label, arrived_at = now()" " WHERE labels.label <> EXCLUDED.label;" ) statements.append( "SELECT sample_id, label, source FROM dev.labels WHERE sample_id LIKE 'e-%' ORDER BY sample_id;" ) subprocess.run( ["docker", "exec", "-i", "postgres", "psql", "-q", "-v", "ON_ERROR_STOP=1", "-U", "user", "-d", "odctl"], input="\n".join(statements), text=True, check=True, ) ``` It prints the two rows it loaded: ```text sample_id | label | source -----------+-------+-------------- e-1 | 1 | label-studio e-2 | 0 | label-studio (2 rows) ``` # Hybrid Search in PostgreSQL odctl's PostgreSQL image adds pgvector for vector search and pg_textsearch for BM25 keyword ranking. Vector search finds the rows whose embeddings are nearest to a query embedding. BM25 is a ranking function that scores a text by how often it holds the query's words, weighted by how rare each word is. Both extensions are created in the `vector` database. Hybrid search runs both queries and combines their rankings, which your application or a final SQL query does. The end-to-end tests run the SQL below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). | Endpoint | Address | | ------------------------------------- | -------------------------------------------------- | | PostgreSQL from the host | `127.0.0.1:5432`, user `user`, password `password` | | PostgreSQL inside the `odctl` network | `postgres:5432` | | Database with both extensions | `vector` | Start the profile and open `psql` in the container: ```bash odctl up postgres docker exec -it postgres psql -U user -d vector ``` ## Vector Similarity with pgvector This creates a table of three-dimensional vectors with an HNSW index and returns the row nearest to `[1,0,0]`: ```sql CREATE TABLE items (id bigserial primary key, embedding vector(3)); INSERT INTO items (embedding) VALUES ('[1,0,0]'), ('[0,1,0]'), ('[0.9,0.1,0]'); CREATE INDEX items_hnsw ON items USING hnsw (embedding vector_l2_ops); SELECT id FROM items ORDER BY embedding <-> '[1,0,0]' LIMIT 1; ``` `<->` is the Euclidean distance, so the nearest row comes first, and the query returns id `1`. The index is an HNSW index: HNSW (hierarchical navigable small world) is a graph of the vectors in layers, which finds near neighbours without comparing the query with every row. ## Keyword Ranking with pg_textsearch This creates a table of three texts with a BM25 index and returns the best match for `ranking function`: ```sql CREATE TABLE docs (id int primary key, content text); INSERT INTO docs VALUES (1, 'PostgreSQL is a powerful database system'), (2, 'BM25 is an effective ranking function'), (3, 'Full text search with custom scoring'); CREATE INDEX docs_bm25 ON docs USING bm25(content) WITH (text_config='english'); SELECT id FROM docs ORDER BY content <@> 'ranking function' LIMIT 1; ``` Ordering by `content <@> 'query'` puts the best BM25 match first, so the query returns id `2`. pg_textsearch works only when PostgreSQL preloads it, and odctl's compose file sets `shared_preload_libraries` to do that. `psql` shows each query's result: To use the extensions in another database, run `CREATE EXTENSION vector;` or `CREATE EXTENSION pg_textsearch;` there. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), connect to `postgres:5432` with the same user and password, for example `postgresql://user:password@postgres:5432/vector`. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It runs both queries on tables named `demo_items` and `demo_docs`, which it drops and creates again on every run: ```bash #!/usr/bin/env bash # Starts postgres and runs a vector query and a BM25 query in the vector # database, on tables it creates again on every run. set -euo pipefail odctl up postgres docker exec -i postgres psql -U user -d vector -v ON_ERROR_STOP=1 <<'SQL' DROP TABLE IF EXISTS demo_items, demo_docs; CREATE TABLE demo_items (id bigserial primary key, embedding vector(3)); INSERT INTO demo_items (embedding) VALUES ('[1,0,0]'), ('[0,1,0]'), ('[0.9,0.1,0]'); CREATE INDEX demo_items_hnsw ON demo_items USING hnsw (embedding vector_l2_ops); SELECT id FROM demo_items ORDER BY embedding <-> '[1,0,0]' LIMIT 1; CREATE TABLE demo_docs (id int primary key, content text); INSERT INTO demo_docs VALUES (1, 'PostgreSQL is a powerful database system'), (2, 'BM25 is an effective ranking function'), (3, 'Full text search with custom scoring'); CREATE INDEX demo_docs_bm25 ON demo_docs USING bm25(content) WITH (text_config='english'); SELECT id FROM demo_docs ORDER BY content <@> 'ranking function' LIMIT 1; SQL ``` It prints the statements' results, with id `1` from the vector query and id `2` from the BM25 query. The BM25 index build also prints `NOTICE` lines. # Valkey The `valkey` profile runs the Valkey Bundle image as one container named `valkey`. Valkey is an in-memory key-value store that speaks the Redis protocol. The Bundle image adds modules for JSON documents, Bloom filters and vector search. The end-to-end tests run the commands below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). Start the profile: ```bash odctl up valkey ``` | Setting | Value | | ---------------------------------- | --------------------------- | | Address from the host | `127.0.0.1:6379` | | Address inside the `odctl` network | `valkey:6379` | | User | `user`, password `password` | | Memory limit | 512 MB | The memory limit is set today by `mem_limit` in the `valkey` service of `.odctl/compose-store.yml`, where an edit can change it. ## Users and Permissions The unauthenticated `default` user is turned off, so every client logs in as `user`. That user has every command, every key pattern and every Pub/Sub channel. Channels are a separate permission in Valkey, and the profile grants them explicitly, so `PUBLISH` and `SUBSCRIBE` work as well as key commands. ## Write and Read Keys `valkey-cli` is in the container. Log in with a URL that carries the user and password: ```bash vk=redis://user:password@localhost:6379 docker exec valkey valkey-cli --no-auth-warning -u "$vk" ping docker exec valkey valkey-cli --no-auth-warning -u "$vk" mset 'model:1' '{"a":1}' 'model:2' '{"a":2}' docker exec valkey valkey-cli --no-auth-warning -u "$vk" mget 'model:1' 'model:2' docker exec valkey valkey-cli --no-auth-warning -u "$vk" del 'model:1' 'model:2' ``` `--no-auth-warning` stops `valkey-cli` printing a warning about the password on the command line. `ping` answers `PONG`. `mset` writes both keys in one command and prints `OK`. `mget` reads them back in one round trip and prints `{"a":1}` and `{"a":2}`. `del` prints `2`, the number of keys it removed. ## Persistence Valkey writes an append-only file and saves a snapshot every 60 seconds when at least one key has changed. Both settings are in the `command` of the `valkey` service in `.odctl/compose-store.yml` (`--appendonly yes` and `--save 60 1`). Both files are inside the container, so a restart keeps the data and `odctl down` removes it. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), connect to `valkey:6379` with the same user and password, for example `redis://user:password@valkey:6379` for a Redis client. Feast's online store uses this address, as [Feast with Iceberg](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/index.md) shows. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It also stores a JSON document with the JSON module and reads one field of it: ```bash #!/usr/bin/env bash # Starts valkey, writes two keys and reads them back, stores a JSON document # and reads one field of it, then deletes the keys. set -euo pipefail odctl up valkey vk=redis://user:password@localhost:6379 docker exec valkey valkey-cli --no-auth-warning -u "$vk" ping docker exec valkey valkey-cli --no-auth-warning -u "$vk" mset 'model:1' '{"a":1}' 'model:2' '{"a":2}' docker exec valkey valkey-cli --no-auth-warning -u "$vk" mget 'model:1' 'model:2' # Store a JSON document with the JSON module and read one field by its path. docker exec valkey valkey-cli --no-auth-warning -u "$vk" JSON.SET 'doc:1' '$' '{"name":"odctl","stars":5}' docker exec valkey valkey-cli --no-auth-warning -u "$vk" JSON.GET 'doc:1' '$.stars' docker exec valkey valkey-cli --no-auth-warning -u "$vk" del 'model:1' 'model:2' 'doc:1' ``` It prints `PONG`, `OK`, the two values, `OK`, `[5]` for the `stars` field and `3` for the three deleted keys. # Tracing LLM Apps with Langfuse The `langfuse` profile runs Langfuse, which traces and evaluates applications built on LLMs (large language models) and agents. An application sends Langfuse one trace for each request: the record of what it did to answer, such as a call to a model. OpenTelemetry is an open standard for traces, metrics and logs; it calls each step of a trace a span. Langfuse stores each span it receives as an observation. The profile runs Langfuse 4. It has `langfuse-web`, which serves the UI and receives traces, and `langfuse-worker`, which processes them. Both use odctl's PostgreSQL, SeaweedFS, Valkey and `ch-lite` (ClickHouse), which `odctl up` starts first. An application sends traces over OTLP, the protocol that the [Telemetry guide](https://jaehyeon.me/odctl/latest/guides/telemetry/index.md) describes. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). The Send to Telemetry Too section also builds on the [Telemetry guide](https://jaehyeon.me/odctl/latest/guides/telemetry/index.md). Run the commands in the folder that holds your `.odctl` workspace. Start the profile: ```bash odctl up langfuse ``` | Endpoint | Address | | ------------------------------------------------------------ | ------------------------------------------ | | UI from the host (user `user@odctl.io`, password `password`) | `http://127.0.0.1:3005` | | OTLP endpoint from the host | `http://127.0.0.1:3005/api/public/otel` | | OTLP endpoint inside the `odctl` network | `http://langfuse-web:3000/api/public/otel` | | Project keys | public `pk-lf-odctl`, secret `sk-lf-odctl` | Three facts about the profile: - It starts `ch-lite`, whose two ClickHouse servers have a limit of 4 GB each, set by `mem_limit` in `.odctl/compose-analytics.yml`. - Langfuse needs its databases in UTC, so keep `TZ` in `.odctl/.env` at its default, `UTC`. - `ch-lite` and `ch-full` use the same container names, so stop `ch-full` with `odctl down ch-full` before `odctl up langfuse`. PostgreSQL reads `TZ` only when it creates its data folder, as [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) explains. A trace takes this path. `langfuse-web` writes each event it receives to SeaweedFS and puts a reference on a Valkey queue. `langfuse-worker` takes the reference, reads the event and writes it to ClickHouse: `langfuse-web` keeps users, projects and keys in the `langfuse` database on PostgreSQL. ClickHouse holds the traces in its own `langfuse` database. The worker has no host port, because only `langfuse-web` receives requests. ## Log In Open `http://127.0.0.1:3005` and sign in as `user@odctl.io` with the password `password`. `langfuse-web` creates this user, the project `odctl` and its two keys when it first starts. Once you have sent the trace in the next section, the Tracing page lists it, and the trace opens like this: The tree on the left lists the two observations of the trace. The panel on the right shows the input and output of the one you select. ## Send a Trace This script sends one trace with the OpenTelemetry SDK for Python, the library that creates spans and exports them over OTLP. A question and its answer make one trace of two observations: the root `answer-question`, and its child `model-call`. odctl runs no model, so a fixed answer stands in for the model call. Save the script as `send.py`: ```python import base64 import json from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor ENDPOINT = "http://127.0.0.1:3005/api/public/otel/v1/traces" AUTH = base64.b64encode(b"pk-lf-odctl:sk-lf-odctl").decode() provider = TracerProvider(resource=Resource.create({"service.name": "demo"})) provider.add_span_processor( SimpleSpanProcessor( OTLPSpanExporter( endpoint=ENDPOINT, headers={ "Authorization": f"Basic {AUTH}", "x-langfuse-ingestion-version": "4", }, ) ) ) tracer = provider.get_tracer("demo") def send_trace(question): """Send one trace of a question and its answer, and return its trace ID.""" with tracer.start_as_current_span("answer-question") as root: root.set_attribute("langfuse.observation.input", json.dumps(question)) with tracer.start_as_current_span("model-call") as call: call.set_attribute("gen_ai.request.model", "stand-in-model") call.set_attribute("langfuse.observation.input", json.dumps(question)) # odctl runs no model, so a fixed answer stands in for the model call. answer = "Langfuse stores traces." call.set_attribute("langfuse.observation.output", json.dumps(answer)) root.set_attribute("langfuse.observation.output", json.dumps(answer)) return format(root.get_span_context().trace_id, "032x") print(send_trace("What does Langfuse store?")) ``` Run it. The commands use `uv run --with`, which needs uv; with pip, install the two packages with `pip install` and run the script with `python`. `uv run` installs the two OpenTelemetry packages for this run only: ```bash uv run --with "opentelemetry-sdk>=1.45,<2" --with "opentelemetry-exporter-otlp-proto-http>=1.45,<2" python send.py ``` It prints the trace ID, a different one on every run: ```text 532ec9ccb17361a9cdd312a414119ade ``` The exporter sends each span to the OTLP endpoint with Basic auth: an `Authorization` header that holds `Basic` and the base64 text of the public key, a colon and the secret key. The header `x-langfuse-ingestion-version` with the value `4` makes Langfuse 4 show the data at once. Without it, Langfuse 4 can show the data up to 10 minutes late. The attributes `langfuse.observation.input` and `langfuse.observation.output` become the input and output of an observation. A span with the attribute `gen_ai.request.model` becomes a generation, which is the kind of observation Langfuse uses for a call to a model. ## Read It Back Langfuse 4 answers `GET /api/public/traces` with 404, so this reads the trace with the observations API, version 2. Set `TRACE_ID` to the ID the script printed, such as `TRACE_ID=532ec9ccb17361a9cdd312a414119ade`, then run the command from the same terminal: ```bash # This prints the name and type of each observation of the trace. curl -s -u pk-lf-odctl:sk-lf-odctl -G http://127.0.0.1:3005/api/public/v2/observations \ --data-urlencode "traceId=$TRACE_ID" \ | python3 -c "import json, sys; [print(o['name'], o['type']) for o in sorted(json.load(sys.stdin)['data'], key=lambda o: o['name'])]" ``` ```text answer-question SPAN model-call GENERATION ``` The trace has two observations. The root `answer-question` is a span, and `model-call` is a generation. `langfuse-worker` writes a trace to ClickHouse a moment after `langfuse-web` receives it, so an empty answer means the trace has not arrived yet. Run the command again after a few seconds. ## Send to Telemetry Too With `odctl up telemetry` running, the same application can send each trace to Langfuse and to the telemetry collector. Add these lines to the end of `send.py`. They add a second span processor with an exporter that sends to the collector, and then send another trace: ```python # The collector from `odctl up telemetry` receives OTLP over HTTP on port 4318. provider.add_span_processor( SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://127.0.0.1:4318/v1/traces")) ) print(send_trace("Does this trace reach Tempo too?")) ``` Run the script again with the same command. It prints two IDs. The first trace goes to Langfuse only, because the script adds the second processor after it. The second trace goes to both. Tempo is the trace store inside the `otel-lgtm` container that the `telemetry` profile runs. Set `TRACE_ID` to the second ID and ask Tempo for the trace: ```bash # Set TRACE_ID to the second ID the script printed. Tempo answers 200 when it # holds the trace and 404 when it does not. docker exec otel-lgtm curl -s -o /dev/null -w "%{http_code}\n" "http://localhost:3200/api/traces/$TRACE_ID" ``` ```text 200 ``` The read command above returns the same two observations for this ID, so the trace is in both places. Tempo can answer 404 for a few seconds after the script ends. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), send OTLP to `http://langfuse-web:3000/api/public/otel`, with the same keys and header. The container sends to port 3000, which `langfuse-web` listens on, and not to the host port 3005. This command sends an empty batch from a container on the network: ```bash # An empty batch is enough to check the address, the keys and the header. # Langfuse answers 200. docker run --rm --network odctl curlimages/curl:8.17.0 -s -o /dev/null -w "%{http_code}\n" \ -u pk-lf-odctl:sk-lf-odctl -H "x-langfuse-ingestion-version: 4" \ -H "Content-Type: application/json" -d '{"resourceSpans":[]}' \ http://langfuse-web:3000/api/public/otel/v1/traces ``` ```text 200 ``` ## Runnable Example Save this file as `example.py`. It sends one trace and asks the observations API until both observations are there. Start the profile first, then run it with the same `uv run` command as above, ending in `python example.py`: ```python """Sends one trace to Langfuse and waits until its two observations can be read back. Start the profile with `odctl up langfuse` first, then run this file with `uv run --with "opentelemetry-sdk>=1.45,<2" --with "opentelemetry-exporter-otlp-proto-http>=1.45,<2" python example.py`. """ import base64 import json import time import urllib.parse import urllib.request from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import SimpleSpanProcessor BASE = "http://127.0.0.1:3005/api/public" AUTH = base64.b64encode(b"pk-lf-odctl:sk-lf-odctl").decode() provider = TracerProvider() provider.add_span_processor( SimpleSpanProcessor( OTLPSpanExporter( endpoint=f"{BASE}/otel/v1/traces", headers={ "Authorization": f"Basic {AUTH}", "x-langfuse-ingestion-version": "4", }, ) ) ) tracer = provider.get_tracer("demo") with tracer.start_as_current_span("answer-question") as root: with tracer.start_as_current_span("model-call") as call: # odctl runs no model, so this span only carries a model name. call.set_attribute("gen_ai.request.model", "stand-in-model") trace_id = format(root.get_span_context().trace_id, "032x") query = urllib.parse.urlencode({"traceId": trace_id}) request = urllib.request.Request( f"{BASE}/v2/observations?{query}", headers={"Authorization": f"Basic {AUTH}"} ) # Langfuse writes the trace to ClickHouse a moment after it receives it. for _ in range(30): with urllib.request.urlopen(request) as response: observations = json.load(response)["data"] if len(observations) == 2: break time.sleep(2) print(f"trace {trace_id} has {len(observations)} observations") ``` It prints the trace ID and the number of observations Langfuse returned for it: ```text trace 731b2cf2f10d39660f7e4bb75ff472eb has 2 observations ``` # OpenLineage and Marquez The `lineage` profile runs Marquez. Marquez has an API that receives OpenLineage events and keeps jobs, runs and datasets on PostgreSQL, and a web UI that draws the lineage graph. OpenLineage is an open standard for events that say which job read and wrote which datasets in each run. Anything that sends OpenLineage events over HTTP can report to Marquez. A section below sets up each of Airflow, a Spark job and `feast apply`. The end-to-end tests run the commands below on every release. They skip the `up` commands and the `.env` edit, because the workflow starts each profile itself and sets the transport in its shell. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). Your job, Airflow, Spark and Feast send their events to `marquez-api`, and `marquez-web` reads the API to draw the graph: Start the profile: ```bash odctl up lineage ``` | Endpoint | Address | | -------------------------------------- | ------------------------- | | Marquez API from the host | `http://127.0.0.1:5002` | | Marquez API inside the `odctl` network | `http://marquez-api:5000` | | Lineage graph UI | `http://127.0.0.1:3003` | ## Send OpenLineage Events A run is reported as a `START` event and a `COMPLETE` event with the same run id. The job reads one dataset and writes another: ```bash RUN_ID=a1b2c3d4-0000-4000-8000-000000000001 NOW=$(date -u +%Y-%m-%dT%H:%M:%S.000Z) for event in START COMPLETE; do curl -s -X POST http://127.0.0.1:5002/api/v1/lineage \ -H 'Content-Type: application/json' \ -d "{\"eventType\":\"$event\",\"eventTime\":\"$NOW\", \"producer\":\"demo\", \"run\":{\"runId\":\"$RUN_ID\"}, \"job\":{\"namespace\":\"demo\",\"name\":\"demo-job\"}, \"inputs\":[{\"namespace\":\"demo\",\"name\":\"raw-orders\"}], \"outputs\":[{\"namespace\":\"demo\",\"name\":\"demo-dataset\"}]}" done ``` The run id is a UUID, as OpenLineage requires. Each request prints nothing when Marquez accepts the event. ## Read the Lineage Back These read the namespaces, the job and the output dataset back from the API: ```bash curl http://127.0.0.1:5002/api/v1/namespaces curl http://127.0.0.1:5002/api/v1/namespaces/demo/jobs/demo-job curl http://127.0.0.1:5002/api/v1/namespaces/demo/datasets/demo-dataset ``` Marquez writes the job, the run and the dataset from the events, so all three read back. The first command lists the namespaces, which include `demo`. The second returns the job `demo-job` as JSON, with `raw-orders` among its inputs, `demo-dataset` among its outputs and a `latestRun` in state `COMPLETED`. The third returns the dataset. The UI shows them under the `demo` namespace. The UI draws the job between the datasets it read and wrote: ## Lineage from Airflow The `airflow` image includes Airflow's OpenLineage provider. It sends a `START` event and a `COMPLETE` event for each DAG run and each task, and for a SQL task it reads the tables from the SQL. It sends events only when it has a transport, which is the setting that says where to send them. odctl leaves the transport empty, so with no change Airflow sends nothing and logs no error. This section builds on [Airflow DAGs from S3](https://jaehyeon.me/odctl/latest/guides/airflow-dags/index.md), which explains how to upload and run a DAG. Set the transport in the `.env` key `AIRFLOW_OPENLINEAGE_TRANSPORT`. Its value is JSON: `type` is `http`, and `url` is the address of `marquez-api` inside the `odctl` network, from the table at the top of this page. `sed -i.bak` works with both GNU and macOS `sed`, and keeps the original as `.env.bak`. Run this in the folder that holds your `.odctl` workspace, then start Airflow: ```bash sed -i.bak "s|^AIRFLOW_OPENLINEAGE_TRANSPORT=.*|AIRFLOW_OPENLINEAGE_TRANSPORT='{\"type\": \"http\", \"url\": \"http://marquez-api:5000\"}'|" .odctl/.env ``` ```bash odctl up airflow ``` If Airflow is already running, run `odctl recreate airflow` instead, because a restart does not read `.env` again, as [Apply a Change](https://jaehyeon.me/odctl/latest/how-it-works/#apply-a-change) explains. The DAG has two tasks that run SQL on the `odctl` database of the `postgres` service. The first creates `raw_orders`, and the second builds `daily_orders` from it. Both tables are in the `dev` schema. The `odctl` database looks in `cdc` first, so the SQL names `dev`. Save it as `lineage_orders.py`: ```python import pendulum from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator from airflow.sdk import DAG with DAG( dag_id="lineage_orders", start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), schedule=None, catchup=False, ): seed_raw_orders = SQLExecuteQueryOperator( task_id="seed_raw_orders", conn_id="odctl_postgres", sql=[ "DROP TABLE IF EXISTS dev.raw_orders", "CREATE TABLE dev.raw_orders AS SELECT * FROM " "(VALUES ('a', 5.0), ('a', 10.0), ('b', 7.5)) AS t(supplier, price)", ], ) build_daily_orders = SQLExecuteQueryOperator( task_id="build_daily_orders", conn_id="odctl_postgres", sql=[ "DROP TABLE IF EXISTS dev.daily_orders", "CREATE TABLE dev.daily_orders AS " "SELECT supplier, sum(price) AS total FROM dev.raw_orders GROUP BY supplier", ], ) seed_raw_orders >> build_daily_orders ``` The provider parses each statement. The highlighted one gives `build_daily_orders` its datasets: it reads `dev.raw_orders` and writes `dev.daily_orders`. The tasks reach PostgreSQL through an Airflow connection, a stored address and login that a task names with `conn_id`. This adds the connection `odctl_postgres`, with the user and password of odctl's PostgreSQL: ```bash docker exec airflow airflow connections add odctl_postgres \ --conn-type postgres --conn-host postgres --conn-login user --conn-password password \ --conn-schema odctl --conn-port 5432 ``` The command prints a few `SyntaxWarning` lines, then `` Successfully added `conn_id`=odctl_postgres `` and a table with the connection. The warnings do not affect the result. Copy the DAG into the Airflow container and upload it to `s3://airflow/dags`, as [Airflow DAGs from S3](https://jaehyeon.me/odctl/latest/guides/airflow-dags/#upload-a-dag) does: ```bash docker cp lineage_orders.py airflow:/tmp/lineage_orders.py docker exec airflow python -c ' import boto3 boto3.client("s3", endpoint_url="http://seaweed:8333").upload_file( "/tmp/lineage_orders.py", "airflow", "dags/lineage_orders.py")' ``` Airflow reads the bucket every 15 seconds, so the DAG appears in its list a short time after the upload. Wait until the first command lists it, then unpause the DAG and trigger a run: ```bash docker exec airflow airflow dags list | grep lineage_orders docker exec airflow airflow dags unpause lineage_orders docker exec airflow airflow dags trigger lineage_orders ``` The last command prints the new run in state `queued`. Airflow sends the events while the run executes, and the run takes a few seconds, so list the runs until the state is `success` with `docker exec airflow airflow dags list-runs lineage_orders`. Then read the jobs back. A namespace groups jobs and datasets in Marquez. This lists the jobs of one namespace from the same API that [Read the Lineage Back](#read-the-lineage-back) uses, cut to each job's name and the datasets it read and wrote: ```bash # Jobs of the namespace default, with the datasets each read and wrote. curl -s http://127.0.0.1:5002/api/v1/namespaces/default/jobs | python3 -c ' import json, sys for job in json.load(sys.stdin)["jobs"]: if job["inputs"] or job["outputs"]: print(job["name"], [d["name"] for d in job["inputs"]], [d["name"] for d in job["outputs"]])' ``` ```text lineage_orders.build_daily_orders ['odctl.dev.raw_orders'] ['odctl.dev.daily_orders'] lineage_orders.seed_raw_orders [] ['odctl.dev.raw_orders'] ``` Airflow's provider names each job after the DAG and the task, and each dataset after the database, the schema and the table. Four more facts explain the output: - Airflow puts its jobs in the namespace `default`, which `AIRFLOW__OPENLINEAGE__NAMESPACE` in `.odctl/compose-orch.yml` changes after `odctl recreate airflow`. - Airflow gives each table the namespace `postgres://postgres:5432`, the address of the database. - The DAG is a third job, `lineage_orders`, with no datasets, so the read-back leaves it out. - The first task writes `raw_orders` and the second reads it, so the graph joins the two jobs through that table. The UI draws the graph of the second task at : The graph runs from `seed_raw_orders` through the table `raw_orders` to `build_daily_orders` and `daily_orders`, because the second task reads the table the first one wrote. ## Lineage from Spark The OpenLineage Spark listener is a Spark listener, a class that Spark calls as a job runs. It sends the events with the datasets each statement read and wrote. The listener's jar is `openlineage.jar` in `/mnt/shared-deps/spark`. The Spark containers already put that folder on the classpath, so the job needs no `--packages` option. This section builds on [Spark](https://jaehyeon.me/odctl/latest/guides/spark/index.md). Start the profile: ```bash odctl up spark-lite ``` `spark-full` works the same way, because it runs the same `spark-master` container. Save this PySpark job as `orders_job.py`. It writes the Iceberg table `raw_orders` and then builds `daily_orders` from it: ```python from pyspark.sql import SparkSession spark = SparkSession.builder.appName("spark-orders").getOrCreate() spark.sql("CREATE NAMESPACE IF NOT EXISTS iceberg.demo") spark.sql(""" CREATE OR REPLACE TABLE iceberg.demo.raw_orders (supplier STRING, price DOUBLE) USING iceberg""") spark.sql(""" INSERT INTO iceberg.demo.raw_orders VALUES ('a', 5.0), ('a', 10.0), ('b', 7.5)""") spark.sql(""" CREATE OR REPLACE TABLE iceberg.demo.daily_orders (supplier STRING, total DOUBLE) USING iceberg""") spark.sql(""" INSERT INTO iceberg.demo.daily_orders SELECT supplier, sum(price) FROM iceberg.demo.raw_orders GROUP BY supplier""") ``` The job has no lineage code. The listener reads each statement's plan, and the highlighted statement is the one that reads `raw_orders` and writes `daily_orders`. Copy the file into the master container and submit it with the listener's settings: ```bash docker cp orders_job.py spark-master:/tmp/orders_job.py # Submit the job with the OpenLineage listener sending to marquez-api. docker exec spark-master /opt/spark/bin/spark-submit \ --driver-class-path '/mnt/shared-deps/spark/openlineage.jar:/mnt/shared-deps/shared/*:/mnt/shared-deps/spark/*' \ --conf spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener \ --conf spark.openlineage.transport.type=http \ --conf spark.openlineage.transport.url=http://marquez-api:5000 \ --conf spark.openlineage.namespace=odctl \ /tmp/orders_job.py ``` Each setting does one thing: - `--driver-class-path` puts `openlineage.jar` first on the driver's classpath, ahead of the other jars of the shared volume. - `spark.extraListeners` names the listener class, so Spark calls it for every statement. - `spark.openlineage.transport.type` and `spark.openlineage.transport.url` send the events over HTTP to `marquez-api`. - `spark.openlineage.namespace` is the namespace of the jobs in Marquez. The jar comes first because the OpenMetadata agent jar in the same folder holds a second copy of the listener's classes. With Spark's default classpath the driver loads that copy. The listener then fails with `NoSuchMethodError`, Spark stops the session, and the job fails. The `--driver-class-path` value repeats the classpath from `spark/spark-defaults.conf` in the workspace, with the jar in front. The job prints Spark's usual log, with some OpenLineage warnings and one `ContextFactory` error line, and the events still reach Marquez. Spark sends them as each statement ends, so read the jobs of the namespace `odctl` back a few seconds after the job ends: ```bash # Jobs of the namespace odctl, with the datasets each read and wrote. curl -s http://127.0.0.1:5002/api/v1/namespaces/odctl/jobs | python3 -c ' import json, sys for job in json.load(sys.stdin)["jobs"]: if job["inputs"] or job["outputs"]: print(job["name"], [d["name"] for d in job["inputs"]], [d["name"] for d in job["outputs"]])' ``` ```text spark_orders.append_data.iceberg_demo_daily_orders ['demo/raw_orders'] ['demo/daily_orders'] spark_orders.atomic_replace_table.demo_daily_orders [] ['demo/daily_orders'] spark_orders.append_data.iceberg_demo_raw_orders [] ['demo/raw_orders'] spark_orders.atomic_replace_table.demo_raw_orders [] ['demo/raw_orders'] ``` Spark reports one job for each statement that writes a table. The listener names each job after the application name, with `-` turned into `_` (`spark_orders`), the kind of write and the table. The first line is the job that read `raw_orders` and wrote `daily_orders`. The listener names an Iceberg table by its place in the warehouse bucket, so the datasets are `demo/raw_orders` and `demo/daily_orders` in the namespace `s3://warehouse`. The application is a fifth job, `spark_orders`, with no datasets, so the read-back leaves it out. ## Lineage from Feast `feast apply` can send an event for the feature views it applies. It does so when the `openlineage` block of `feature_store.yaml` says `enabled: true`. The bundled repository of the `feast` profile has no such block, so Feast sends nothing until you add one. The block needs the OpenLineage extra of the Feast client. The `airflow` image has it and the `feast` services do not, so this section runs `feast apply` in the `airflow` container, started in [Lineage from Airflow](#lineage-from-airflow). It builds on [Feast with Iceberg](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/index.md), whose `feature_repo/definitions.py` it uses: save that file first if you have not. The feature view `driver_stats` is over the Iceberg table `demo.driver_stats`, and `apply` does not read the table. Save this file as `lineage_repo/feature_store.yaml`. It differs from the Feast guide's file in three places: the project, `lineage_demo`, which keeps these feature views apart from the guide's project `demo`; the addresses, which are the container addresses because `feast apply` runs inside the `odctl` network; and the `openlineage` block: ```yaml project: lineage_demo provider: local registry: registry_type: sql path: postgresql+psycopg://user:password@postgres:5432/feast offline_store: type: duckdb online_store: type: redis connection_string: "valkey:6379,username=user,password=password" entity_key_serialization_version: 3 openlineage: enabled: true transport_type: http transport_url: http://marquez-api:5000 ``` `transport_type` and `transport_url` send the events over HTTP to `marquez-api`, and Feast adds the path `api/v1/lineage` itself. Without `namespace`, Feast uses the project name as the namespace. Run these from the folder that holds `feature_repo` and `lineage_repo`. Copy both files into the container and apply them. `--skip-source-validation` is the option that [Feast with Iceberg](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#register-and-materialise) explains: ```bash docker exec airflow mkdir -p /tmp/lineage_repo docker cp lineage_repo/feature_store.yaml airflow:/tmp/lineage_repo/feature_store.yaml docker cp feature_repo/definitions.py airflow:/tmp/lineage_repo/definitions.py docker exec airflow feast -c /tmp/lineage_repo apply --skip-source-validation ``` ```text INFO:openlineage.client.client:OpenLineageClient will use `http` transport No project found in the repository. Using project name lineage_demo defined in feature_store.yaml Applying changes for project lineage_demo Deploying infrastructure for driver_stats ``` The first line shows that the OpenLineage client is on. Read the jobs of the namespace `lineage_demo`, the project name, back: ```bash # Jobs of the namespace lineage_demo, the name of the Feast project, with the datasets each read and wrote. curl -s http://127.0.0.1:5002/api/v1/namespaces/lineage_demo/jobs | python3 -c ' import json, sys for job in json.load(sys.stdin)["jobs"]: if job["inputs"] or job["outputs"]: print(job["name"], [d["name"] for d in job["inputs"]], [d["name"] for d in job["outputs"]])' ``` ```text feast_feature_views_lineage_demo ['driver', '.demo.driver_stats'] ['driver_stats'] ``` Feast reports one job for the project's feature views. Its inputs are the entity `driver` and the data source, and its output is the feature view `driver_stats`. The source is named `.demo.driver_stats` because an `IcebergSource` with no `name` joins its `warehouse`, `namespace` and `table`, and the definitions pass an empty `warehouse`. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), send events to `http://marquez-api:5000/api/v1/lineage`. For an OpenLineage client's HTTP transport, set the URL to `http://marquez-api:5000`, because the transport adds `api/v1/lineage` by default. ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It gives each run a new id, so it reports a new run every time: ```bash #!/usr/bin/env bash # Starts lineage, reports one run of a job that reads raw-orders and writes # demo-dataset, and prints the state of the job's latest run. set -euo pipefail odctl up lineage RUN_ID=$(python3 -c 'import uuid; print(uuid.uuid4())') NOW=$(date -u +%Y-%m-%dT%H:%M:%S.000Z) for event in START COMPLETE; do curl -s -X POST http://127.0.0.1:5002/api/v1/lineage \ -H 'Content-Type: application/json' \ -d "{\"eventType\":\"$event\",\"eventTime\":\"$NOW\", \"producer\":\"demo\", \"run\":{\"runId\":\"$RUN_ID\"}, \"job\":{\"namespace\":\"demo\",\"name\":\"demo-job\"}, \"inputs\":[{\"namespace\":\"demo\",\"name\":\"raw-orders\"}], \"outputs\":[{\"namespace\":\"demo\",\"name\":\"demo-dataset\"}]}" done curl -s http://127.0.0.1:5002/api/v1/namespaces/demo/jobs/demo-job \ | python3 -c 'import json, sys; run = json.load(sys.stdin)["latestRun"]; print(run["id"], run["state"])' ``` It prints the new run's id and `COMPLETED`. # Catalogue a Database with OpenMetadata A data catalogue lists the databases, tables and columns of an organisation, so people can search them and see who owns each one. The `metadata` profile runs OpenMetadata, a data catalogue: the server with its UI and REST API, Elasticsearch for search, and an ingestion container. The ingestion container is an Airflow that runs the ingestion pipelines the server deploys, which the OpenMetadata UI calls agents (API path `ingestionPipelines`). Its own tables are in the `omt` database on PostgreSQL. The end-to-end tests run the commands below on every release. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). It uses the PostgreSQL that the profile starts as the database to catalogue. The profile also starts `storage`, SeaweedFS, whose `openmetadata` bucket holds [custom connectors](#add-a-custom-connector). Start the profile: ```bash odctl up metadata ``` | Endpoint | Address | | ------------------------------------------ | --------------------------------- | | UI and REST API from the host | `http://127.0.0.1:8585` | | UI and REST API inside the `odctl` network | `http://openmetadata-server:8585` | | Ingestion Airflow from the host | `http://127.0.0.1:8087` | Log in as `admin@odctl.io` or `user@odctl.io`, both with the password `password`. A user's password is set when the database creates the user, so a database from an earlier release keeps its own logins. From 1.3.0 to 1.6.0, `admin@odctl.io` has the password `admin`. Before 1.3.0, the users are `admin@open-data.local` and `user@open-data.local`, both with the password `admin`. A wrong domain fails with an invalid username or password error. The ingestion Airflow takes `user` and `password`. ## Get a Token The login API takes the password base64-encoded and returns a JWT, a signed token that later requests send in the `Authorization` header. This sets `TOKEN` and `AUTH` for the next steps, and prints nothing: ```bash TOKEN=$(curl -s -H 'Content-Type: application/json' \ -d "{\"email\":\"admin@odctl.io\",\"password\":\"$(printf password | base64)\"}" \ http://127.0.0.1:8585/api/v1/users/login \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["accessToken"])') AUTH="Authorization: Bearer $TOKEN" ``` On a database from 1.3.0 to 1.6.0, encode `admin` instead of `password`. On a database from before 1.3.0, also use `admin@open-data.local` instead of `admin@odctl.io`. ## Register PostgreSQL as a Database Service PostgreSQL is already running, so it serves as the source. The server reaches it by its container name. This sets `SVC` to the id of the new service: ```bash SVC=$(curl -s -X PUT -H "$AUTH" -H 'Content-Type: application/json' \ -d '{"name":"demo_pg","serviceType":"Postgres","connection":{"config":{"type":"Postgres","scheme":"postgresql+psycopg2","username":"user","authType":{"password":"password"},"hostPort":"postgres:5432","database":"omt"}}}' \ http://127.0.0.1:8585/api/v1/services/databaseServices \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])') ``` OpenMetadata has an app called AutoPilot. When a service is added in the UI, AutoPilot creates and runs default agents for it. A service created through the API never starts AutoPilot. odctl turns AutoPilot off, so a service added in the UI also gets only the agents you create. The `METADATA_AUTOPILOT` key in `.odctl/.env` sets it, `false` by default, and the `openmetadata-init` service applies it to AutoPilot's `active` value each time the profile starts. This shows the current value: ```bash curl -s -H "$AUTH" \ "http://127.0.0.1:8585/api/v1/apps/name/AutoPilotApplication?fields=appConfiguration" \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["appConfiguration"]["active"])' ``` It prints `False`. To turn AutoPilot on, set `METADATA_AUTOPILOT="true"` in `.odctl/.env` and run `odctl recreate metadata`. The command then prints `True`. `openmetadata-init` tries the admin login of a new database first, then the admin logins of earlier releases; when all of them fail, `docker logs openmetadata-init` ends with `AutoPilot could not be set`. ## Run a Metadata Ingestion Create an agent, deploy it as a DAG into the ingestion Airflow, and trigger it: ```bash PIPE=$(curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json' \ -d "{\"name\":\"demo_pg_metadata\",\"pipelineType\":\"metadata\",\"service\":{\"id\":\"$SVC\",\"type\":\"databaseService\"},\"sourceConfig\":{\"config\":{\"type\":\"DatabaseMetadata\"}},\"airflowConfig\":{\"startDate\":\"2026-01-01T00:00:00.000Z\"}}" \ http://127.0.0.1:8585/api/v1/services/ingestionPipelines \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])') curl -s -X POST -H "$AUTH" http://127.0.0.1:8585/api/v1/services/ingestionPipelines/deploy/"$PIPE" curl -s -X POST -H "$AUTH" http://127.0.0.1:8585/api/v1/services/ingestionPipelines/trigger/"$PIPE" ``` The deploy answers that the agent's workflow has been created. It works once `odctl up metadata` has returned, because the profile waits until the ingestion Airflow answers. Airflow has to parse the new DAG file before the trigger works, so repeat the trigger every few seconds until it answers that the workflow has been triggered. Then read the run's state: ```bash curl -s -H "$AUTH" \ "http://127.0.0.1:8585/api/v1/services/ingestionPipelines/name/demo_pg.demo_pg_metadata?fields=pipelineStatuses" ``` The reply is the agent as JSON. The `pipelineState` of the first entry of `pipelineStatuses` reaches `success` when the tables are catalogued. They then appear in the UI under the `demo_pg` service. ## Search the Catalogue Rebuild the search index, then query it: ```bash curl -s -X POST -H "$AUTH" http://127.0.0.1:8585/api/v1/apps/trigger/SearchIndexingApplication curl -s -H "$AUTH" "http://127.0.0.1:8585/api/v1/search/query?q=*&index=table_search_index&size=1" ``` The search reply is JSON, and `hits.total.value` counts the searchable tables. The index fills over a few minutes after the rebuild, and then holds more than 100 tables: OpenMetadata's own tables in the `omt` database. ## Add a Custom Connector A custom connector is a Python class that reports databases, schemas and tables from a system OpenMetadata has no connector for. The ingestion container copies every file under `s3://openmetadata/connectors` into `/opt/airflow/connectors`, which is on its `PYTHONPATH`, so an agent can import a module or a package from there. [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#where-data-lives) compares the three profiles that read code from a bucket. A connector module must not have the name of an installed package such as `metadata` or `airflow`, because the connectors folder comes before the installed packages on the Python path. This connector reports one database, one schema and one table, written in the code. Save it as `demo_connector.py`: ```python """A custom database connector that reports one database, one schema and one table.""" from typing import Iterable, Optional from metadata.generated.schema.api.data.createDatabase import CreateDatabaseRequest from metadata.generated.schema.api.data.createDatabaseSchema import ( CreateDatabaseSchemaRequest, ) from metadata.generated.schema.api.data.createTable import CreateTableRequest from metadata.generated.schema.entity.data.table import Column, DataType from metadata.generated.schema.metadataIngestion.workflow import ( Source as WorkflowSource, ) from metadata.ingestion.api.models import Either from metadata.ingestion.api.steps import Source from metadata.ingestion.ometa.ometa_api import OpenMetadata class DemoSource(Source): """Reads no external system: the table it reports is written in the code.""" def __init__(self, config: WorkflowSource, metadata: OpenMetadata): super().__init__() self.config = config self.metadata = metadata self.service = config.serviceName @classmethod def create( cls, config_dict: dict, metadata: OpenMetadata, pipeline_name: Optional[str] = None, ) -> "DemoSource": return cls(WorkflowSource.model_validate(config_dict), metadata) def prepare(self) -> None: pass def test_connection(self) -> None: pass def _iter(self, *_, **__) -> Iterable[Either]: yield Either(right=CreateDatabaseRequest(name="demo_db", service=self.service)) yield Either( right=CreateDatabaseSchemaRequest( name="public", database=f"{self.service}.demo_db" ) ) yield Either( right=CreateTableRequest( name="orders", databaseSchema=f"{self.service}.demo_db.public", columns=[ Column(name="id", dataType=DataType.INT), Column(name="amount", dataType=DataType.DOUBLE), ], ) ) def close(self) -> None: pass ``` Upload it from the folder that holds the file. The command runs boto3 inside the ingestion container, which has the S3 settings, and reaches SeaweedFS at `http://seaweed:8333`, its S3 address inside the `odctl` network. It prints nothing: ```bash docker exec -i openmetadata-ingestion python -c ' import boto3, sys boto3.client("s3", endpoint_url="http://seaweed:8333").put_object( Bucket="openmetadata", Key="connectors/demo_connector.py", Body=sys.stdin.buffer.read()) ' < demo_connector.py ``` The container checks S3 every 15 seconds and copies the file within about 15 seconds of the upload. The interval is set today by `INTERVAL_SECONDS` in `.odctl/openmetadata/connector_sync.py`, where an edit can change it. List the connectors folder until `demo_connector.py` is there: ```bash docker exec openmetadata-ingestion ls /opt/airflow/connectors ``` Create the service and its agent through the API, as in the PostgreSQL steps above, with the same `AUTH`. The service is of type `CustomDatabase`, and `sourcePythonClass` names the module and the class. This sets `SVC` to the id of the new service: ```bash SVC=$(curl -s -X PUT -H "$AUTH" -H 'Content-Type: application/json' \ -d '{"name":"demo_connector","serviceType":"CustomDatabase","connection":{"config":{"type":"CustomDatabase","sourcePythonClass":"demo_connector.DemoSource"}}}' \ http://127.0.0.1:8585/api/v1/services/databaseServices \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])') ``` Create the agent, deploy it as a DAG into the ingestion Airflow, and trigger it. As in the PostgreSQL steps, repeat the trigger every few seconds until it answers that the workflow has been triggered: ```bash PIPE=$(curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json' \ -d "{\"name\":\"demo_connector_metadata\",\"pipelineType\":\"metadata\",\"service\":{\"id\":\"$SVC\",\"type\":\"databaseService\"},\"sourceConfig\":{\"config\":{\"type\":\"DatabaseMetadata\"}},\"airflowConfig\":{\"startDate\":\"2026-01-01T00:00:00.000Z\"}}" \ http://127.0.0.1:8585/api/v1/services/ingestionPipelines \ | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])') curl -s -X POST -H "$AUTH" http://127.0.0.1:8585/api/v1/services/ingestionPipelines/deploy/"$PIPE" curl -s -X POST -H "$AUTH" http://127.0.0.1:8585/api/v1/services/ingestionPipelines/trigger/"$PIPE" ``` Read the run's state until the `pipelineState` of the first entry of `pipelineStatuses` reaches `success`: ```bash curl -s -H "$AUTH" \ "http://127.0.0.1:8585/api/v1/services/ingestionPipelines/name/demo_connector.demo_connector_metadata?fields=pipelineStatuses" ``` The table is then in the catalogue: ```bash curl -s -H "$AUTH" \ "http://127.0.0.1:8585/api/v1/tables/name/demo_connector.demo_db.public.orders?fields=columns" ``` The reply is the table `orders` with its columns `id` and `amount`. The UI shows the same table under the service `demo_connector`, with its columns and the agent run that created it: A change to the connector needs no restart. Rename the table in the file: ```bash sed -i.bak 's/name="orders"/name="orders_v2"/' demo_connector.py ``` Upload it again with the upload command above. The container has copied it when this prints `1`: ```bash docker exec openmetadata-ingestion grep -c orders_v2 /opt/airflow/connectors/demo_connector.py ``` Trigger the agent again: ```bash curl -s -X POST -H "$AUTH" http://127.0.0.1:8585/api/v1/services/ingestionPipelines/trigger/"$PIPE" ``` A run that starts after the copy reports `orders_v2`. Read the run's state as above, then the changed table: ```bash curl -s -H "$AUTH" \ "http://127.0.0.1:8585/api/v1/tables/name/demo_connector.demo_db.public.orders_v2" ``` ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), call the API at `http://openmetadata-server:8585`. The server also answers the Model Context Protocol at `http://127.0.0.1:8585/mcp` with the same token. ## Remove the Example This deletes the services with their agents and the tables they catalogued, and the uploaded connector. The container removes the file from `/opt/airflow/connectors` within about 15 seconds: ```bash for service in demo_pg demo_connector; do curl -s -X DELETE -H "$AUTH" \ "http://127.0.0.1:8585/api/v1/services/databaseServices/name/$service?hardDelete=true&recursive=true" done docker exec openmetadata-ingestion python -c ' import boto3 boto3.client("s3", endpoint_url="http://seaweed:8333").delete_object( Bucket="openmetadata", Key="connectors/demo_connector.py") ' ``` ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace and `demo_connector.py`. It runs every step above except changing the connector, waits where a step needs time, and removes both services and the uploaded connector at the end. It prints `autopilot active: False`, the state of each ingestion, `custom table: orders`, the number of searchable tables, and `removed demo_pg and demo_connector`: ```bash #!/usr/bin/env bash # Starts metadata, shows AutoPilot's state, registers PostgreSQL as a database # service and runs a metadata ingestion of the omt database. Then it uploads a # custom connector, runs its agent, reads its table, counts the searchable # tables and removes both services and the connector again. Run it in the folder # that holds .odctl and demo_connector.py, the file of the Add a Custom # Connector section. set -euo pipefail odctl up metadata om=http://127.0.0.1:8585/api/v1 json() { python3 -c "import json, sys; print($1)"; } TOKEN=$(curl -s -H 'Content-Type: application/json' \ -d "{\"email\":\"admin@odctl.io\",\"password\":\"$(printf password | base64)\"}" \ "$om/users/login" | json 'json.load(sys.stdin)["accessToken"]') AUTH="Authorization: Bearer $TOKEN" echo "autopilot active: $(curl -s -H "$AUTH" "$om/apps/name/AutoPilotApplication?fields=appConfiguration" \ | json 'json.load(sys.stdin)["appConfiguration"]["active"]')" # Registers a database service, creates its agent, deploys and triggers it, and # waits for the run. Usage: ingest ingest() { local svc pipe state="" svc=$(curl -s -X PUT -H "$AUTH" -H 'Content-Type: application/json' -d "$2" \ "$om/services/databaseServices" | json 'json.load(sys.stdin)["id"]') pipe=$(curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json' \ -d "{\"name\":\"$1_metadata\",\"pipelineType\":\"metadata\",\"service\":{\"id\":\"$svc\",\"type\":\"databaseService\"},\"sourceConfig\":{\"config\":{\"type\":\"DatabaseMetadata\"}},\"airflowConfig\":{\"startDate\":\"2026-01-01T00:00:00.000Z\"}}" \ "$om/services/ingestionPipelines" | json 'json.load(sys.stdin)["id"]') until curl -s -X POST -H "$AUTH" "$om/services/ingestionPipelines/deploy/$pipe" \ | grep 'has been created' >/dev/null; do sleep 5; done # Airflow parses a new DAG file before the trigger works, so the trigger is repeated. until curl -s -X POST -H "$AUTH" "$om/services/ingestionPipelines/trigger/$pipe" \ | grep 'has been triggered' >/dev/null; do sleep 5; done until [ "$state" = success ] || [ "$state" = failed ]; do sleep 10 state=$(curl -s -H "$AUTH" \ "$om/services/ingestionPipelines/name/$1.$1_metadata?fields=pipelineStatuses" \ | json '(json.load(sys.stdin).get("pipelineStatuses") or [{}])[0].get("pipelineState", "")') done echo "$1 ingestion: $state" } ingest demo_pg '{"name":"demo_pg","serviceType":"Postgres","connection":{"config":{"type":"Postgres","scheme":"postgresql+psycopg2","username":"user","authType":{"password":"password"},"hostPort":"postgres:5432","database":"omt"}}}' # The container copies the connector from SeaweedFS within about 15 seconds. docker exec -i openmetadata-ingestion python -c ' import boto3, sys boto3.client("s3", endpoint_url="http://seaweed:8333").put_object( Bucket="openmetadata", Key="connectors/demo_connector.py", Body=sys.stdin.buffer.read()) ' < demo_connector.py until docker exec openmetadata-ingestion test -f /opt/airflow/connectors/demo_connector.py; do sleep 5; done ingest demo_connector '{"name":"demo_connector","serviceType":"CustomDatabase","connection":{"config":{"type":"CustomDatabase","sourcePythonClass":"demo_connector.DemoSource"}}}' echo "custom table: $(curl -s -H "$AUTH" "$om/tables/name/demo_connector.demo_db.public.orders" \ | json 'json.load(sys.stdin)["name"]')" # The search index fills over a few minutes after it is rebuilt. curl -s -X POST -H "$AUTH" "$om/apps/trigger/SearchIndexingApplication" >/dev/null hits=0 until [ "$hits" -gt 100 ]; do sleep 5 hits=$(curl -s -H "$AUTH" "$om/search/query?q=*&index=table_search_index&size=1" \ | json 'json.load(sys.stdin)["hits"]["total"]["value"]') done echo "searchable tables: $hits" for service in demo_pg demo_connector; do curl -s -X DELETE -H "$AUTH" \ "$om/services/databaseServices/name/$service?hardDelete=true&recursive=true" >/dev/null done docker exec openmetadata-ingestion python -c ' import boto3 boto3.client("s3", endpoint_url="http://seaweed:8333").delete_object( Bucket="openmetadata", Key="connectors/demo_connector.py") ' echo "removed demo_pg and demo_connector" ``` The searchable tables number more than 100, and the whole run takes several minutes. # Telemetry The `telemetry` profile runs the grafana/otel-lgtm image: an OpenTelemetry Collector, Prometheus, Tempo, Loki and Grafana in one container. OpenTelemetry is an open standard for metrics, traces and logs, and OTLP is its protocol for sending them. Tempo stores traces, Loki stores logs, and Grafana draws dashboards from all three stores. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md). Start the profile next to the profiles you want to watch: ```bash odctl up telemetry kafka-lite flink-lite ``` Prometheus reads metrics from the services, the collector receives OTLP and passes each kind of data to its store, and Grafana reads the stores and evaluates its alert rule against Prometheus: | Endpoint | Address | | ------------------------------------------ | ------------------------------------------------------- | | Grafana (user `user`, password `password`) | `http://127.0.0.1:3004` | | Prometheus | `http://127.0.0.1:19090` | | OTLP through the collector | `127.0.0.1:4317` (gRPC), `http://127.0.0.1:4318` (HTTP) | ## Where the Metrics Come From Prometheus scrapes the services that publish a metrics endpoint, by container name on the `odctl` network. Airflow pushes its metrics over OTLP to the collector, and the collector reads PostgreSQL and Valkey directly. A service that is not running is a target that is down. ## Dashboards Grafana has a dashboard per service in a folder named `odctl`. A dashboard fills in once its service's profile is up. The dashboards are JSON files in `.odctl/grafana/dashboards/`. Edit a file and run `odctl restart telemetry` to load it. Changes made in the Grafana UI are not kept. For example, the PostgreSQL dashboard: ## Send Your Own Metrics Point an OpenTelemetry SDK at `http://127.0.0.1:4318`, its default HTTP port. The end-to-end tests post one metric with curl, which shows the path: ```bash ts=$(python3 -c "import time;print(int(time.time()*1e9))") cat > /tmp/otlp.json < DRIFT_THRESHOLD: raise ValueError(f"x drifted: {distance:.2f} is above {DRIFT_THRESHOLD}") check_drift() ``` The DAG needs Evidently in the Airflow container. odctl installs the packages in `_AIRFLOW_PIP_DEPS` when Airflow starts, as [Evidently Reports](https://jaehyeon.me/odctl/latest/guides/evidently-reports/#from-another-container) explains, so set the key and start Airflow. Run this in the folder that holds your `.odctl` workspace: ```bash sed -i.bak 's|^_AIRFLOW_PIP_DEPS=.*|_AIRFLOW_PIP_DEPS="evidently==0.7.23"|' .odctl/.env ``` The command prints nothing and leaves the old file as `.odctl/.env.bak`. ```bash odctl up airflow ``` The command prints each container it starts and ends with `All requested profiles successfully started!`. If Airflow is already running, run `odctl recreate airflow` instead, because a restart does not read `.env` again, as [Apply a Change](https://jaehyeon.me/odctl/latest/how-it-works/#apply-a-change) explains. Copy the DAG into the Airflow container and upload it to `s3://airflow/dags`, as [Airflow DAGs from S3](https://jaehyeon.me/odctl/latest/guides/airflow-dags/#upload-a-dag) does: ```bash docker cp drift_check.py airflow:/tmp/drift_check.py docker exec airflow python -c ' import boto3 boto3.client("s3", endpoint_url="http://seaweed:8333").upload_file( "/tmp/drift_check.py", "airflow", "dags/drift_check.py")' ``` The upload prints nothing. Airflow reads the bucket every 15 seconds, so wait until the first command lists the DAG. Then unpause it and trigger a run named `drifted`: ```bash docker exec airflow airflow dags list | grep drift_check docker exec airflow airflow dags unpause drift_check docker exec airflow airflow dags trigger drift_check --run-id drifted ``` The last command prints the new run in state `queued`. The task fails after a few seconds, so list the runs with `docker exec airflow airflow dags list-runs drift_check` until `drifted` has the state `failed`. Airflow sends its metrics to the collector every 15 seconds, set by `OTEL_METRIC_EXPORT_INTERVAL` in `.odctl/compose-orch.yml`. Grafana evaluates the rule every minute, so the alert appears within about 2 minutes. This lists the alerts that are firing: ```bash # Alerts that are firing, with the DAG each names. curl -s -u user:password http://127.0.0.1:3004/api/prometheus/grafana/api/v1/alerts \ | python3 -c ' import json, sys for alert in json.load(sys.stdin)["data"]["alerts"]: if alert["state"] == "Alerting": print(alert["labels"]["alertname"], alert["labels"].get("dag_id", ""), alert["state"])' ``` ```text Airflow DAG run failed drift_check Alerting ``` The alert carries the label `dag_id` with the DAG that failed. Grafana calls the state `Alerting` in its API and Firing in its UI. Open and log in with `user` and `password`: The destination is `empty`, Grafana's default contact point. It sends nothing, because odctl configures no contact point of its own. A contact point is where Grafana sends a notification, such as an email address or a chat channel. A firing alert therefore shows only in Grafana. The summary under the rule's title is the template, because one rule covers every DAG. Each alert has it filled in with its DAG, which the instance's details and the alerts API show. To be told when an alert fires, you need a contact point and a notification policy that routes alerts to it. Grafana's database lives inside the `otel-lgtm` container and has no volume, so `odctl down` deletes a contact point made in the UI. Keep them in a file instead. Grafana loads a contact point and a notification policy from a file in `.odctl/grafana/alerting/` at each start, in the format of [Grafana's file provisioning](https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/). Add the file and restart `telemetry`, as above. Deleting a file and restarting leaves its rule, contact point or policy in Grafana. `odctl down telemetry` and then `odctl up telemetry` remove them. To tell whoever receives the alert what to do, add a `runbook_url` annotation to the rule in its file. The alert clears when the failed run falls out of the window. A run below the threshold raises none. This triggers a run named `steady` with `shift` 0 and prints the new run in state `queued`. The second command prints the distance of each run from its task log: ```bash docker exec airflow airflow dags trigger drift_check --run-id steady --conf '{"shift": 0}' ``` ```bash # Distance each run printed, from its task log. docker exec airflow sh -c 'cd /opt/airflow/logs/dag_id=drift_check && grep -o "Wasserstein distance[^\"]*" run_id=*/task_id=check_drift/attempt=1.log' ``` ```text run_id=drifted/task_id=check_drift/attempt=1.log:Wasserstein distance of x: 3.35, threshold 0.5 run_id=steady/task_id=check_drift/attempt=1.log:Wasserstein distance of x: 0.16, threshold 0.5 ``` Both runs draw their samples from fixed seeds, so the distances repeat on every run. The drifted run is above the threshold and fails, and the steady run is below it and succeeds. A rule of your own is another file in the same folder. Copy it in and restart `telemetry`, as above. To write one, start from the sample below and change its `uid`, `title`, query and limit. This sample fires when the model calls of the last hour have used more tokens than a budget. odctl does not provision it, because the budget is yours: change the highlighted limit. `gen_ai.client.token.usage` is the OpenTelemetry metric for the tokens that one model call used. Prometheus writes its name with `_` in place of `.`. For a histogram with explicit buckets, as the OpenTelemetry SDKs send by default, Prometheus keeps the sum in the series that ends in `_sum`. The query therefore sums `gen_ai_client_token_usage_sum`. An SDK set to exponential histograms sends one native histogram series with no `_sum`, as Airflow's does, and this query then returns nothing. The query counts each series in one of two ways: - A series that began within the last hour counts by its last value, which is all its tokens since it began, even after the application stops sending. - A series that is more than an hour old counts by its increase over the hour, while the application exports at least every 5 minutes, as the OpenTelemetry SDKs do by default. `increase()` counts only the change after a series' first sample. Alone, it would miss the tokens in that first sample, and every token of an application that exports once. When `telemetry` starts while the application is already running, the first hour counts all tokens since the application started. Save it as `token_budget.yaml`: ```yaml # A sample alert rule: it fires when the model calls of the last hour have used more # tokens than the budget set below. Copy it into .odctl/grafana/alerting/ and run # odctl restart telemetry to load it. apiVersion: 1 groups: - orgId: 1 name: model-calls folder: odctl interval: 1m rules: - uid: odctl-token-budget title: Model call tokens over budget condition: C data: # Tokens that gen_ai.client.token.usage recorded in the last hour, input # and output together. Prometheus keeps the histogram's sum in the series # gen_ai_client_token_usage_sum. A series that began within the last hour # counts by its last value, which is all its tokens since it began, also # when the application stopped sending more than 5 minutes ago, because # increase() counts only the change after a series' first sample. A series # more than an hour old counts by its increase over the hour, while the # application exports at least every 5 minutes. When `telemetry` starts # while the application is already running, the first hour counts all # tokens since the application started. - refId: A relativeTimeRange: from: 3600 to: 0 datasourceUid: prometheus model: refId: A instant: true expr: sum((last_over_time(gen_ai_client_token_usage_sum[1h]) unless gen_ai_client_token_usage_sum offset 1h) or increase(gen_ai_client_token_usage_sum[1h])) - refId: B datasourceUid: __expr__ model: refId: B type: reduce expression: A reducer: last - refId: C datasourceUid: __expr__ model: refId: C type: threshold expression: B conditions: - evaluator: type: gt # The budget: change this number to your own limit. params: [1000000] # No model calls in the last hour is under budget, not an alert. noDataState: OK execErrState: Error for: 0s labels: severity: warning annotations: summary: "Model calls used more tokens in the last hour than the budget" ``` ```bash cp token_budget.yaml .odctl/grafana/alerting/ odctl restart telemetry ``` The copy prints nothing, and the restart ends with `Restart complete.` Grafana and the collector take a few seconds to answer after the restart. Run the listing command from the start of this section again until it prints both rules: ```text Airflow DAG run failed Model call tokens over budget ``` The budget rule has no data until an application sends the metric, and no data counts as no alert. [Send Your Own Metrics](#send-your-own-metrics) shows how to send a metric to the collector. ## From Another Container Inside the [`odctl` network](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service), send OTLP to the collector at `otel-lgtm:4317` (gRPC) or `http://otel-lgtm:4318` (HTTP), as Airflow does. Prometheus is `http://otel-lgtm:9090` and Grafana is `http://otel-lgtm:3000`. This command sends an empty batch to the collector from a container on the network: ```bash # An empty batch is enough to check the address. The collector answers 200. docker run --rm --network odctl curlimages/curl:8.17.0 -s -o /dev/null -w "%{http_code}\n" \ -H "Content-Type: application/json" -d '{}' http://otel-lgtm:4318/v1/traces ``` ```text 200 ``` ## Runnable Example Save this file as `example.sh` and run it with `bash example.sh` in the folder that holds your `.odctl` workspace. It waits until Prometheus returns the metric: ```bash #!/usr/bin/env bash # Starts telemetry, sends one OTLP metric to the collector and waits until # Prometheus returns it. set -euo pipefail odctl up telemetry ts=$(python3 -c "import time;print(int(time.time()*1e9))") curl -s -X POST -H "Content-Type: application/json" http://127.0.0.1:4318/v1/metrics -d @- </dev/null; do sleep 2 done curl -s "http://127.0.0.1:19090/api/v1/query?query=demo_total" echo ``` It prints the collector's answer and then the query result, with `demo_total` and the value `"42"`. # AI Assistants # AI Assistants An AI coding assistant can start and use an odctl stack when it has three things: the rules for choosing profiles and reaching their services, odctl's commands as tools it can call, and odctl's documentation as text. This section describes each and how to install them. The parts relate as follows: | Part | What it gives the assistant | Page | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | Agent skill | a folder, `skills/odctl/`, of rules in the Agent Skills format: which profile to start, how to reach each service from the host and from a container, and how to run odctl in a project's tests | [Agent Skill](https://jaehyeon.me/odctl/latest/ai/agent-skill/index.md) | | MCP server | `odctl mcp`, which offers odctl's commands to an MCP client as tools; MCP, the Model Context Protocol, is the standard an assistant uses to call a tool | [MCP Server](https://jaehyeon.me/odctl/latest/guides/mcp-server/index.md) | | Plugin | one install for Claude Code, Codex or Antigravity that adds the skill and the MCP server together | [Plugin](https://jaehyeon.me/odctl/latest/ai/plugin/index.md) | | `llms.txt` | the pages of this site as text, for an assistant that reads documentation | below | The plugin is a package of the other two parts. It starts the MCP server read-only, so the assistant can list, explain, inspect and read logs, and cannot run `up` or `down`. Install the skill without the plugin when your tool has no plugin support, and set up the MCP server by hand to allow `up` and `down`. The plugin does not install odctl, and its MCP server needs the `mcp` extra: see [Before You Install](https://jaehyeon.me/odctl/latest/ai/plugin/#before-you-install). ## Documentation as Text The site publishes two files built from the sidebar: - [`llms.txt`](https://jaehyeon.me/odctl/llms.txt) is an index of the pages with a link to each. - [`llms-full.txt`](https://jaehyeon.me/odctl/llms-full.txt) holds every page in one file. Give an assistant one of them when its tool cannot install a skill. The skill is shorter and tells the assistant what to do, while the pages describe every profile and command. # Plugin The odctl plugin adds the [agent skill](https://jaehyeon.me/odctl/latest/ai/agent-skill/index.md) and the [MCP server](https://jaehyeon.me/odctl/latest/guides/mcp-server/index.md) to Claude Code, Codex or Antigravity with one install. The plugin starts the server as `odctl mcp`, without `--allow-changes`, so the assistant gets the five tools that only read and cannot start or stop profiles. The repository holds the plugin's files at its root: a manifest for each tool, a marketplace file for the tools that have one, and the MCP entry. Codex copies the plugin into its cache, `~/.codex/plugins/cache/odctl/odctl//`, and Antigravity copies it to `~/.gemini/config/plugins/odctl/`, where it failed on a symlink that points outside the folder, such as one in a local clone's `.venv`. Claude Code copies a GitHub marketplace into `~/.claude/plugins/` and reads a local-path marketplace in place. ## Before You Install The plugin does not install odctl. The MCP server it starts runs the `odctl` command, which needs the `mcp` extra, so install odctl first: ```bash uv tool install "odctl[mcp]" # or, if you installed odctl with pipx pipx install --force "odctl[mcp]" ``` Without the extra, `odctl mcp` exits with an error that names this command. ## Install Run the lines of your tool. Each installs the plugin from the repository's `main` branch. Claude Code: ```bash claude plugin marketplace add jaehyeon-kim/odctl claude plugin install odctl@odctl ``` Codex: ```bash codex plugin marketplace add jaehyeon-kim/odctl codex plugin add odctl@odctl ``` For Claude Code and Codex, `odctl@odctl` is the plugin's name, an `@`, and the marketplace's name, which are both `odctl`. Antigravity: ```bash agy plugin install https://github.com/jaehyeon-kim/odctl ``` Start a new session after the install, so the tool loads the skill and the server. ## Check the Install The skill is named `odctl:odctl` in Claude Code and Codex, which join the plugin's name and the skill's name with a colon. Antigravity uses the skill's own name, `odctl`. - **Claude Code:** `claude plugin list` shows `odctl@odctl` as enabled, and `claude plugin details odctl` lists the skill and the MCP server. - **Codex:** `codex plugin list` shows odctl with its status, and `/plugins` in `codex` shows it under its marketplace. - **Antigravity:** `agy plugin list` shows the plugin, its version and its skill count, and `agy plugin list --json` lists the skill's name. Then ask the assistant a request that needs a stack, such as "start Kafka and Flink on my machine with odctl". It loads the skill, and it can call the MCP tool `list` to read the profiles. In print mode (`claude -p`, `agy -p`), the tool cannot ask before the assistant's first call to an MCP tool, so allow the odctl tools for the run: - **Claude Code:** name each tool in `--allowedTools`, such as `--allowedTools mcp__plugin_odctl_odctl__list`. - **Antigravity:** pass `--settings '{"permissions":{"allow":["mcp(odctl_odctl/list)"]}}'`, which holds for that run only. `codex exec` called the tool without this. ## Update A new odctl release changes the plugin's version. Update with the same tool: - **Claude Code:** `claude plugin update odctl@odctl`. - **Codex:** `codex plugin marketplace upgrade odctl`, which reinstalls the plugin. - **Antigravity:** run `agy plugin install https://github.com/jaehyeon-kim/odctl` again. ## Allow Up and Down The plugin's server is read-only. To let an assistant start and stop profiles, add `odctl mcp --allow-changes` to your tool's MCP settings by hand, as [Allow Up and Down](https://jaehyeon.me/odctl/latest/guides/mcp-server/#allow-up-and-down) explains, and read what the tools can remove first. ## Sources The install commands are those of each tool's own help (`claude plugin install --help`, `codex plugin add --help` and `agy plugin help`), [Create a marketplace](https://code.claude.com/docs/en/plugin-marketplaces) for Claude Code, and [Plugins](https://antigravity.google/docs/plugins?tab=cli) for Antigravity. # Agent Skill The odctl skill gives an AI coding assistant the rules for using odctl, so it starts the right profiles, reaches their services at the right address and stops only what it started, instead of guessing. A skill is a folder in the Agent Skills format that Claude Code, Codex and other assistants read. The assistant loads the skill when a request needs one of the technologies odctl runs locally, such as Kafka, Spark, Trino or MLflow, or code that connects to them. The folder is `skills/odctl/` in the repository: | File | What it covers | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SKILL.md` | a checklist for the assistant to copy into its reply; the rules for starting, waiting for, stopping and reaching services, each with a wrong and a right example; and the commands, options and `.env` keys of odctl | | `references/profiles.md` | for each profile, its addresses from the host and from a container, its logins, what `odctl up` also starts, its services, ports and memory limits, which services `odctl up` waits for, and where its data lives | | `references/testing.md` | odctl in a project's tests and CI: pinning the version, starting and stopping the profiles a job needs, and what a teardown removes | The commands, options, keys and profile notes are generated from odctl's CLI, `registry.yml` and compose files, so the folder lists what the release has. The rules and `references/testing.md` are written by hand in `scripts/skill/`. The skill does not install odctl or Docker. The assistant runs the `odctl` commands, so install odctl first, as [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#install) shows. ## Install with the Plugin The [plugin](https://jaehyeon.me/odctl/latest/ai/plugin/index.md) installs the skill for Claude Code, Codex and Antigravity, together with the MCP server. ## Install without the Plugin Copy `skills/odctl/` into the skills folder of your assistant. Take it from the repository at the tag of the release you installed, which is `v` followed by the `cli_version` that `odctl info --json` prints. For Claude Code, `~/.claude/skills/odctl/` is for every project and `.claude/skills/odctl/` in a project is for that project. These lines copy it for every project: ```bash # The source of the installed release, which holds the skill version=$(odctl info --json | python3 -c 'import json, sys; print(json.load(sys.stdin)["cli_version"])') src=$(mktemp -d) git clone --depth 1 --branch "v$version" https://github.com/jaehyeon-kim/odctl "$src" && mkdir -p ~/.claude/skills && rm -rf ~/.claude/skills/odctl && cp -R "$src/skills/odctl" ~/.claude/skills/odctl rm -rf "$src" ``` The lines clone into a new private folder, replace an earlier copy of the skill only when the clone succeeds, and remove the clone, so they also update the skill to a newer release. If `CLAUDE_CONFIG_DIR` is set, use that folder in place of `~/.claude`. With the GitHub CLI 2.90.0 or later, one command does the copy. Without `--pin`, it takes the latest tagged release, which can be newer than the odctl you installed, so the lines pin the installed release: ```bash # The release that is installed version=$(odctl info --json | python3 -c 'import json, sys; print(json.load(sys.stdin)["cli_version"])') gh skill install jaehyeon-kim/odctl odctl --pin "v$version" --agent claude-code --scope user ``` `--scope user` installs for every project, in your home folder. `--scope project`, the default, installs for the current project, inside its git repository. Another assistant takes its own `--agent` value, listed in `gh skill install --help`. ## Check the Install Check that the folder holds `SKILL.md`: ```bash ls ~/.claude/skills/odctl/SKILL.md ``` In Claude Code, `/skills` lists the loaded skills, and `odctl` must be among them. After a project copy, check `.claude/skills/odctl/SKILL.md` in the project instead. Then ask the assistant for a stack, such as "run Kafka and Flink on my machine with odctl". It reads the skill when the request names one of the technologies. ## Limits - The skill's rules cover odctl's container lifecycle: starting, waiting, stopping, addresses and logins. - `references/testing.md` covers how a test starts odctl's stack, not what the tests check. # Connect an AI Agent with MCP MCP, the Model Context Protocol, lets an AI agent discover and call the tools of a server. A tool is a function with a name and named parameters. `odctl mcp` starts an MCP server that offers odctl's commands as tools. An MCP client, such as an AI assistant, starts `odctl mcp` as a process. It exchanges messages with the process over its standard input and output, which MCP calls stdio. The agent calls a tool and odctl runs the command. The server offers five tools that only read, each running the command of the same name: - `list` returns every profile with its services and ports. - `explain` takes `profile` and returns that profile's services, images, ports, volumes and usage. - `ps` takes `profiles` and returns their containers and the active profiles, or every container odctl manages when `profiles` is left out. - `info` returns the odctl version, the image tag, the workspace, whether Docker is reachable and odctl's own images. - `logs` takes `profiles`, `service`, `tail`, `timestamps`, `since` and `until`, and returns the logs of the containers of those profiles. `tail` defaults to 100 lines, so one call cannot fill the agent's context with a long log. `odctl logs` prints all lines, and a call can pass another number. Results and errors: - `list`, `explain`, `ps` and `info` return the documents on the [JSON Output](https://jaehyeon.me/odctl/latest/json-output/index.md) page, and `list`'s array arrives under the key `result` because a tool's structured result is an object. - `info` reports Docker as not reachable with `reachable: false`, not as an error. - `logs` returns the lines that `odctl logs` prints. - A tool that fails returns the error message of the command. - Tools run one at a time. The server offers the tools `up` and `down` only when you start it with `--allow-changes`, as [Allow Up and Down](#allow-up-and-down) explains. The [plugin](https://jaehyeon.me/odctl/latest/ai/plugin/index.md) starts the server without that option, so an assistant that uses the plugin gets the five tools that only read. The tool names and their parameter names stay the same in every 1.x release, as [What Stays Stable in 1.x](https://jaehyeon.me/odctl/latest/how-it-works/#what-stays-stable-in-1x) says. This guide builds on [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md) and [JSON Output](https://jaehyeon.me/odctl/latest/json-output/index.md). ## Install The server needs the `mcp` extra of odctl. This installs odctl with it: ```bash uv tool install "odctl[mcp]" ``` Without the extra, `odctl mcp` exits with an error that names this command. `uv tool install` keeps odctl in an environment of its own, so the client scripts below need the `mcp` package in an environment of their own. This makes one in the current folder: `python -m venv .venv`, then `source .venv/bin/activate` and `pip install "mcp>=2.3.0,<3"`. Keep it out of an environment that holds the client of the [Feast guide](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#connect-an-mcp-client), which needs `mcp` below 2. ## Add It to an MCP Client Claude Code is Anthropic's command-line coding agent, and it is an MCP client. The scripts in the next sections start the server themselves and need no client set up, so skip this section if you use only them. Run this in the folder of your project to add odctl to the project: ```bash # Writes .mcp.json in this folder. claude mcp add --scope project odctl -- odctl mcp ``` The command writes this entry to `.mcp.json`: ```json { "mcpServers": { "odctl": { "type": "stdio", "command": "odctl", "args": [ "mcp" ], "env": {} } } } ``` Any MCP client that starts a server over stdio needs the same two things, the command `odctl` and the argument `mcp`. odctl reads the `.odctl` workspace of the folder where the client starts it. When your client starts servers in another folder, put `"--workspace", ""` before `"mcp"` in the highlighted `args`. The `--workspace` option sets the workspace for every tool, because it is read once, when the server starts. ## List the Tools This script starts the server command from its arguments and prints the name and description of each tool. Save it as `mcp_client.py` and run it in the environment that has `mcp`: ```python import sys import anyio from mcp import Client, StdioServerParameters async def main(command): """Print the name and description of each tool the server offers. Args: command: The command that starts the server, as a list of words. """ server = StdioServerParameters(command=command[0], args=command[1:]) async with Client(server) as client: for tool in (await client.list_tools()).tools: print(f"{tool.name}: {tool.description}") anyio.run(main, sys.argv[1:]) ``` This lists the five tools that only read: ```bash python mcp_client.py odctl mcp ``` ```text list: List every profile with its services and ports, as `odctl list --json` does. explain: Describe one profile: its services, images, ports, volumes and usage, as `odctl explain --json` does. ps: Show the containers odctl manages and the active profiles, as `odctl ps --json` does. Without profiles it shows every container. info: Show the odctl version, the image tag, the workspace, whether Docker is reachable and odctl's images, as `odctl info --json` does. logs: Fetch the logs of the containers of some profiles, as `odctl logs` prints them, the last 100 lines of each by default. ``` ## Allow Up and Down The tools `up` and `down` start and stop profiles, so an agent that can call them changes what runs on your machine. They exist only when the server starts with the option `--allow-changes`. An option shows in the client's configuration, where an environment variable left set in a shell could allow changes without anyone seeing it. This lists the seven tools: ```bash python mcp_client.py odctl mcp --allow-changes ``` ```text list: List every profile with its services and ports, as `odctl list --json` does. explain: Describe one profile: its services, images, ports, volumes and usage, as `odctl explain --json` does. ps: Show the containers odctl manages and the active profiles, as `odctl ps --json` does. Without profiles it shows every container. info: Show the odctl version, the image tag, the workspace, whether Docker is reachable and odctl's images, as `odctl info --json` does. logs: Fetch the logs of the containers of some profiles, as `odctl logs` prints them, the last 100 lines of each by default. up: Start profiles and their dependencies, as `odctl up` does. down: Stop the named profiles. With `volumes`, also remove their volumes, and the shared jar volume `odctl-shared-deps` once no other profile has a container, as `odctl down --volumes` does. ``` `up` takes `profiles` and `pull`, and `down` takes `profiles` and `volumes`. Each returns what the command prints. The server marks both as destructive, so a client can ask before it runs them. `up` replaces a container whose definition changed, and the data inside it goes with it. `down` takes named profiles only. `odctl down --all --volumes` asks before it removes the data of every profile, and so does `odctl down --volumes` with names that cover every profile, but a server cannot ask. So `down` with `volumes` refuses names that cover every profile `odctl down --all --volumes` would stop. The names need not include `deps`, because a named teardown with `volumes` also removes it once no other profile has a container. The guard compares profile names, not volumes, so one call can still remove a volume another profile uses: `down` of `spark-lite` with `volumes` removes `spark-events-data`, which `spark-full` uses too. ## Runnable Example Save this file as `example.py` and run it with `python example.py` in the folder that holds your `.odctl` workspace, made by `odctl init` as [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md) shows, in the environment that has `mcp`. It starts the server with `--allow-changes`, starts `kafka-lite`, reads `info`, the containers of `kafka-lite` and the last five lines of the `connect` service, then stops `kafka-lite`. A tool that fails returns its error as the result, so the script checks each result, stops at the first error and still stops `kafka-lite`: ```python import os import anyio from mcp import Client, StdioServerParameters async def call(client, name, arguments): """Call one tool, print its name and its text result, and stop on an error. A tool that fails returns its error as the result, so the call checks for it. Args: client: The connected MCP client. name: The tool to call. arguments: The tool's parameters. Raises: RuntimeError: When the tool returns an error. """ result = await client.call_tool(name, arguments) print(f"--- {name}") print(result.content[0].text) if result.is_error: raise RuntimeError(f"the tool {name} failed") async def main(): """Start kafka-lite, read it through four tools, then stop it.""" # Docker reads variables such as DOCKER_HOST, which a client passes to the # server only when asked. server = StdioServerParameters( command="odctl", args=["mcp", "--allow-changes"], env=dict(os.environ) ) async with Client(server) as client: try: await call(client, "up", {"profiles": ["kafka-lite"]}) await call(client, "info", {}) await call(client, "ps", {"profiles": ["kafka-lite"]}) await call(client, "logs", {"profiles": ["kafka-lite"], "service": "connect", "tail": 5}) finally: # Stop kafka-lite even when a tool failed. await call(client, "down", {"profiles": ["kafka-lite"]}) anyio.run(main) ``` It prints the result of each tool under a line with the tool's name. The output below shortens the path of the `.odctl` folder to `/home/you/project/.odctl`: ```text --- up โš™๏ธ Launching ๐Ÿงฑ Infrastructure (deps)... โš™๏ธ Launching ๐Ÿš€ Target (kafka-lite)... โœ“ All requested profiles successfully started! --- info { "cli_version": "", "tag": "", "workspace": { "path": "/home/you/project/.odctl", "exists": true, "stale_tag": null }, "docker": { "reachable": true }, "images": [ { "reference": "ghcr.io/jaehyeon-kim/odctl/airflow:", "local": false }, { "reference": "ghcr.io/jaehyeon-kim/odctl/deps:", "local": true }, { "reference": "ghcr.io/jaehyeon-kim/odctl/mlflow:", "local": false }, { "reference": "ghcr.io/jaehyeon-kim/odctl/postgres:", "local": false }, { "reference": "ghcr.io/jaehyeon-kim/odctl/spark:", "local": false } ] } --- ps { "active_profiles": [ "kafka-lite" ], "containers": [ { "name": "connect", "service": "connect", "state": "running", "health": null, "ports": [ { "host": 8083, "container": 8083, "protocol": "tcp" } ] }, { "name": "kafka", "service": "kafka", "state": "running", "health": null, "ports": [ { "host": 9092, "container": 9092, "protocol": "tcp" } ] }, { "name": "kafka-ui", "service": "kafka-ui", "state": "running", "health": null, "ports": [ { "host": 8086, "container": 8080, "protocol": "tcp" } ] }, { "name": "karapace", "service": "karapace", "state": "running", "health": "healthy", "ports": [ { "host": 8081, "container": 8081, "protocol": "tcp" } ] } ] } --- logs connect | [2026-10-30 21:50:24,917] INFO [Worker clientId=connect-connect:8083, groupId=odctl-connect-cluster] Cluster ID: 42RglGpZQwy9D5VzTVpCWA (org.apache.kafka.clients.Metadata:365) connect | [2026-10-30 21:50:24,919] INFO [Worker clientId=connect-connect:8083, groupId=odctl-connect-cluster] Discovered group coordinator broker-1:19092 (id: 2147483646 rack: null isFenced: false) (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:943) connect | [2026-10-30 21:50:24,920] INFO [Worker clientId=connect-connect:8083, groupId=odctl-connect-cluster] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:239) connect | [2026-10-30 21:50:24,920] INFO [Worker clientId=connect-connect:8083, groupId=odctl-connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:611) connect | [2026-10-30 21:50:24,923] INFO [Worker clientId=connect-connect:8083, groupId=odctl-connect-cluster] (Re-)joining group (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:611) --- down ๐Ÿ›‘ Stopping kafka-lite... โœ“ Teardown complete. ``` # CLI Reference # CLI Reference This page holds the output of `odctl --help` and of `odctl --help`, captured when the site was built. ## odctl ```text Usage: odctl [OPTIONS] COMMAND [ARGS]... odctl: run an open data stack on your machine, with its services already connected, from one command. Commands list the profiles, start and stop them with Docker Compose, and show what is running. โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --install-completion Install completion for the current shell. โ”‚ โ”‚ --show-completion Show completion for the current shell, to copy โ”‚ โ”‚ it or customize the installation. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Global Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --verbose Enable debug-level logging across all commands. โ”‚ โ”‚ --workspace -w Path to the odctl workspace directory (default: โ”‚ โ”‚ ./.odctl). โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Inspection & Info โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ list List all available profiles and their capabilities. โ”‚ โ”‚ explain Explain the details, services, images, and dependencies of a โ”‚ โ”‚ profile. โ”‚ โ”‚ ps List Docker containers managed by the Open Data Stack. โ”‚ โ”‚ info Show the odctl version, whether Docker is reachable and, with โ”‚ โ”‚ `--json`, the image tag, the workspace and odctl's own images. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Workspace โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ init Initialize a local .odctl workspace for custom configurations. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Cluster Lifecycle โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ pull Pre-fetch Docker images without starting the containers. โ”‚ โ”‚ up Launch Open Data profiles. โ”‚ โ”‚ down Stop and remove profile containers and networks. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Management โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ logs Fetch the logs of containers managed by specific profiles. โ”‚ โ”‚ restart Restart one or more specific profiles. โ”‚ โ”‚ recreate Replace one or more profiles' containers, applying compose file โ”‚ โ”‚ changes. โ”‚ โ”‚ mcp Serve odctl's commands to an AI agent as MCP tools over stdio. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Examples: # View all profiles and exposed ports $ odctl list -d # See exactly what the kafka-lite profile provisions $ odctl explain kafka-lite # Launch specific profiles and their dependencies $ odctl up flink-lite kafka-lite spark-lite # Complete teardown and wipe all data $ odctl down --all --volumes ``` ## odctl list ```text Usage: odctl list [OPTIONS] List all available profiles and their capabilities. A profile is a specific capability or technology (for example, `kafka-lite`, `spark-lite`, `airflow`) provided by the Open Data Stack. This command lists them alongside their parent stack. โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --details -d Inspect docker-compose files to show exact services and โ”‚ โ”‚ exposed host ports. โ”‚ โ”‚ --json Print the output as JSON, for scripts and AI โ”‚ โ”‚ assistants. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Examples: # View all basic profiles $ odctl list # View profiles, underlying services, and ports $ odctl list -d ``` ## odctl explain ```text Usage: odctl explain [OPTIONS] {profile} Explain the details, services, images, and dependencies of a profile. Displays a detailed breakdown of what a profile provisions, its container images, exposed host ports, and any prerequisite profiles it depends on. โ•ญโ”€ Arguments โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ * profile The target profile to inspect (for example, โ”‚ โ”‚ 'airflow', 'kafka-lite'). โ”‚ โ”‚ [required] โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --json Print the output as JSON, for scripts and AI assistants. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Examples: # See what the Spark profile provisions $ odctl explain spark-lite ``` ## odctl init ```text Usage: odctl init [OPTIONS] Initialize a local .odctl workspace for custom configurations. Copies the bundled docker-compose files, configs, and `.env` template into a local `./.odctl/` directory. You can then edit these files directly to customize the stack. โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --force -f Wipe out the existing workspace and recreate it from โ”‚ โ”‚ scratch. โ”‚ โ”‚ --yes -y Answer yes to the confirmation prompt, for scripts and โ”‚ โ”‚ CI. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Examples: # Initialize a new workspace in ./.odctl/ $ odctl init # Recreate workspace, overwriting any local changes $ odctl init --force # The same, without the prompt, as in a script $ odctl init --force --yes ``` ## odctl pull ```text Usage: odctl pull [OPTIONS] [profiles]... Pre-fetch Docker images without starting the containers. Useful for downloading heavy images (like Spark, Flink, Kafka) ahead of time, or for fetching a tag that changed in the registry before launching. โ•ญโ”€ Arguments โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ profiles Specific profiles to pull images for. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --all -a Pull images for ALL available profiles in the registry. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Examples: # Pre-fetch images for all profiles $ odctl pull --all # Pre-fetch images just for Flink and Kafka $ odctl pull flink-lite kafka-lite ``` ## odctl up ```text Usage: odctl up [OPTIONS] {profiles}... Launch Open Data profiles. Resolves dependencies for the requested profiles and brings up the required Docker Compose stacks in the correct topological order. Refuses to start if the images for the resolved TAG are not published, naming the tag rather than leaving Docker to report a missing manifest. Warns about any service whose declared profile belongs to another compose file, since the planner will never start it. โ•ญโ”€ Arguments โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ * profiles One or more profiles to launch (for example, โ”‚ โ”‚ 'ch-lite', 'kafka-lite'). โ”‚ โ”‚ [required] โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --dry-run Preview the execution plan without actually starting โ”‚ โ”‚ anything. โ”‚ โ”‚ --pull Pull the images again before starting, to fetch a tag โ”‚ โ”‚ that changed in the registry. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Examples: # Launch Clickhouse, Flink, and their dependencies $ odctl up ch-lite flink-lite # Preview what would be launched for Airflow $ odctl up airflow --dry-run # Pull the images again before launching $ odctl up kafka-lite --pull ``` ## odctl down ```text Usage: odctl down [OPTIONS] [profiles]... Stop and remove profile containers and networks. Tears down the requested profiles. Data kept inside a container is removed with it. Unnamed volumes stay on disk, unused. Use --volumes to also remove the volumes, such as the shared jar volume. โ•ญโ”€ Arguments โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ profiles Specific profiles to stop. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --all -a Stop ALL running profiles. โ”‚ โ”‚ --volumes -v Also remove the volumes of the stopped containers, โ”‚ โ”‚ named and unnamed, and the shared jar volume once no โ”‚ โ”‚ other profile uses it. โ”‚ โ”‚ --dry-run Preview the teardown plan, with the volumes `--volumes` โ”‚ โ”‚ would remove, without stopping anything. โ”‚ โ”‚ --yes -y Answer yes to the confirmation prompt, for scripts and โ”‚ โ”‚ CI. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Examples: # Stop specific profile(s) $ odctl down kafka-lite # Stop all running profiles $ odctl down --all # Complete teardown and wipe all data $ odctl down --all -v # The same, without the prompt, as in a script $ odctl down --all -v --yes ``` ## odctl ps ```text Usage: odctl ps [OPTIONS] [profiles]... List Docker containers managed by the Open Data Stack. Filters out system containers and only shows those belonging to requested profiles. โ•ญโ”€ Arguments โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ profiles Specific profiles to inspect. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --all -a Show all containers managed by odctl. โ”‚ โ”‚ --json Print the output as JSON, for scripts and AI assistants. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Examples: # Show all running containers managed by odctl $ odctl ps --all # Show containers just for specific profiles $ odctl ps kafka-lite spark-lite ``` ## odctl info ```text Usage: odctl info [OPTIONS] Show the odctl version, whether Docker is reachable and, with `--json`, the image tag, the workspace and odctl's own images. โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --json Print the output as JSON, for scripts and AI assistants. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` ## odctl logs ```text Usage: odctl logs [OPTIONS] {profiles}... Fetch the logs of containers managed by specific profiles. โ•ญโ”€ Arguments โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ * profiles Profiles to fetch logs for. [required] โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --service -s Filter to a specific compose service (Use โ”‚ โ”‚ 'odctl ps' to find names). โ”‚ โ”‚ --follow -f Follow log output in real-time. โ”‚ โ”‚ --tail -n Number of lines to show from the end of the โ”‚ โ”‚ logs. โ”‚ โ”‚ [default: all] โ”‚ โ”‚ --timestamps -t Show timestamps. โ”‚ โ”‚ --since Show logs since timestamp or relative (for โ”‚ โ”‚ example '42m'). โ”‚ โ”‚ --until Show logs before a timestamp or relative. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ Examples: # Tail the last 50 lines of all Flink containers and follow live $ odctl logs flink-lite -n 50 -f # View logs strictly for the JobManager service $ odctl logs flink-lite -s jobmanager # Show logs with timestamps for the last 10 minutes $ odctl logs kafka-lite --since 10m -t ``` ## odctl restart ```text Usage: odctl restart [OPTIONS] {profiles}... Restart one or more specific profiles. โ•ญโ”€ Arguments โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ * profiles Profiles to restart. [required] โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` ## odctl recreate ```text Usage: odctl recreate [OPTIONS] {profiles}... Replace one or more profiles' containers, applying compose file changes. โ•ญโ”€ Arguments โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ * profiles Profiles to recreate. [required] โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --pull Pull images before recreating. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` ## odctl mcp ```text Usage: odctl mcp [OPTIONS] Serve odctl's commands to an AI agent as MCP tools over stdio. โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ โ”‚ --allow-changes Also serve `up` and `down`, which start and stop โ”‚ โ”‚ profiles. โ”‚ โ”‚ --help Show this message and exit. โ”‚ โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ ``` # JSON Output # JSON Output `odctl list`, `odctl explain`, `odctl ps` and `odctl info` print one JSON document with `--json`, for scripts and AI assistants. The `list`, `explain`, `ps` and `info` tools of the [MCP server](https://jaehyeon.me/odctl/latest/guides/mcp-server/index.md) return the same documents. This page holds the JSON Schema of each document, captured when the site was built. A JSON Schema lists the keys of a document, the type of each value and which keys are always present. Standard output holds only the JSON document. Warnings and errors go to standard error. A usage error, such as an unknown option, exits with code 2. Any other failure exits with code 1 and prints nothing on standard output. [What Stays Stable in 1.x](https://jaehyeon.me/odctl/latest/how-it-works/#what-stays-stable-in-1x) says which changes to these keys wait for 2.0. ## odctl list `odctl list --json` prints an array with one object for each profile, sorted by profile name. Its schema: ```json { "$defs": { "Port": { "properties": { "host": { "description": "The port on the host, a number.", "title": "Host", "type": "integer" }, "container": { "description": "The port inside the container, a number.", "title": "Container", "type": "integer" }, "protocol": { "default": "tcp", "description": "The transport protocol, `tcp` or `udp`, as Docker or the compose file gives it.", "title": "Protocol", "type": "string" } }, "required": [ "host", "container", "protocol" ], "title": "Port", "type": "object" }, "ProfileInfo": { "properties": { "profile": { "description": "The profile name, which `odctl up` takes.", "title": "Profile", "type": "string" }, "stack": { "description": "The stack the profile belongs to, such as `kafka` for `kafka-lite`. When the registry gives the stack a `parent` group, the value is that group instead, such as `infra` for `postgres`, `storage` and `catalog`.", "title": "Stack", "type": "string" }, "description": { "description": "What the stack provides.", "title": "Description", "type": "string" }, "compose_file": { "description": "The compose file that defines the profile's services.", "title": "Compose File", "type": "string" }, "depends_on": { "description": "The profiles the registry lists as direct prerequisites, sorted by name. Empty when there are none.", "items": { "type": "string" }, "title": "Depends On", "type": "array" }, "starts": { "description": "Every profile `odctl up` starts for this one: the profile itself and all its prerequisites, sorted by name.", "items": { "type": "string" }, "title": "Starts", "type": "array" }, "services": { "description": "The services the profile starts, sorted by name.", "items": { "$ref": "#/$defs/ServiceInfo" }, "title": "Services", "type": "array" } }, "required": [ "profile", "stack", "description", "compose_file", "depends_on", "starts", "services" ], "title": "ProfileInfo", "type": "object" }, "ServiceInfo": { "properties": { "name": { "description": "The compose service name.", "title": "Name", "type": "string" }, "image": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The image as it will run, with `TAG` resolved. `null` when the compose file gives the service no `image`, as for a service that is built.", "title": "Image" }, "ports": { "description": "The ports the service publishes on the host, sorted by host port. Compose variables are expanded first, as compose does. A port range is left out, and so is a port with no host part, which compose publishes on a random host port. Empty when the service publishes none.", "items": { "$ref": "#/$defs/Port" }, "title": "Ports", "type": "array" } }, "required": [ "name", "image", "ports" ], "title": "ServiceInfo", "type": "object" } }, "items": { "$ref": "#/$defs/ProfileInfo" }, "type": "array" } ``` ## odctl explain `odctl explain --json` prints one object for the profile you name. Its schema: ```json { "$defs": { "Port": { "properties": { "host": { "description": "The port on the host, a number.", "title": "Host", "type": "integer" }, "container": { "description": "The port inside the container, a number.", "title": "Container", "type": "integer" }, "protocol": { "default": "tcp", "description": "The transport protocol, `tcp` or `udp`, as Docker or the compose file gives it.", "title": "Protocol", "type": "string" } }, "required": [ "host", "container", "protocol" ], "title": "Port", "type": "object" }, "ServiceInfo": { "properties": { "name": { "description": "The compose service name.", "title": "Name", "type": "string" }, "image": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The image as it will run, with `TAG` resolved. `null` when the compose file gives the service no `image`, as for a service that is built.", "title": "Image" }, "ports": { "description": "The ports the service publishes on the host, sorted by host port. Compose variables are expanded first, as compose does. A port range is left out, and so is a port with no host part, which compose publishes on a random host port. Empty when the service publishes none.", "items": { "$ref": "#/$defs/Port" }, "title": "Ports", "type": "array" } }, "required": [ "name", "image", "ports" ], "title": "ServiceInfo", "type": "object" } }, "properties": { "profile": { "description": "The profile name, which `odctl up` takes.", "title": "Profile", "type": "string" }, "stack": { "description": "The stack the profile belongs to, such as `kafka` for `kafka-lite`. When the registry gives the stack a `parent` group, the value is that group instead, such as `infra` for `postgres`, `storage` and `catalog`.", "title": "Stack", "type": "string" }, "description": { "description": "What the stack provides.", "title": "Description", "type": "string" }, "compose_file": { "description": "The compose file that defines the profile's services.", "title": "Compose File", "type": "string" }, "depends_on": { "description": "The profiles the registry lists as direct prerequisites, sorted by name. Empty when there are none.", "items": { "type": "string" }, "title": "Depends On", "type": "array" }, "starts": { "description": "Every profile `odctl up` starts for this one: the profile itself and all its prerequisites, sorted by name.", "items": { "type": "string" }, "title": "Starts", "type": "array" }, "services": { "description": "The services the profile starts, sorted by name.", "items": { "$ref": "#/$defs/ServiceInfo" }, "title": "Services", "type": "array" }, "usage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The text that says how to reach the profile's services. `null` when the registry has none for this profile.", "title": "Usage" }, "role": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "What the profile's stack does in a data platform. `null` when the registry gives none.", "title": "Role" }, "volumes": { "description": "The named volumes the profile's services mount, sorted by name. Bind mounts are left out, and so are volumes the compose file declares as `external`, so `odctl-shared-deps` appears only for the `deps` profile. Empty when there are none.", "items": { "type": "string" }, "title": "Volumes", "type": "array" } }, "required": [ "profile", "stack", "description", "compose_file", "depends_on", "starts", "services", "usage", "role", "volumes" ], "title": "ProfileDetails", "type": "object" } ``` ## odctl ps `odctl ps --json` prints one object with the active profiles and the matching containers. Its schema: ```json { "$defs": { "ContainerInfo": { "properties": { "name": { "description": "The container name.", "title": "Name", "type": "string" }, "service": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The compose service. `null` when the container has no compose service label.", "title": "Service" }, "state": { "description": "The state as Docker reports it, for example `running` or `exited`.", "title": "State", "type": "string" }, "health": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The health status as Docker reports it, for example `healthy`. `null` when the container has no health check.", "title": "Health" }, "ports": { "description": "The ports published on the host, sorted by host port, then container port and protocol. Empty when the container publishes none.", "items": { "$ref": "#/$defs/Port" }, "title": "Ports", "type": "array" } }, "required": [ "name", "service", "state", "health", "ports" ], "title": "ContainerInfo", "type": "object" }, "Port": { "properties": { "host": { "description": "The port on the host, a number.", "title": "Host", "type": "integer" }, "container": { "description": "The port inside the container, a number.", "title": "Container", "type": "integer" }, "protocol": { "default": "tcp", "description": "The transport protocol, `tcp` or `udp`, as Docker or the compose file gives it.", "title": "Protocol", "type": "string" } }, "required": [ "host", "container", "protocol" ], "title": "Port", "type": "object" } }, "properties": { "active_profiles": { "description": "The profiles whose services are all running, sorted. Only the profiles the command names are considered, with their prerequisites in the same compose file, such as `postgres` and `storage` for `catalog`. With `--all`, every profile is considered. `deps` is never listed. A profile whose services are a strict subset of another listed profile's is left out, so `spark-lite` is not listed beside `spark-full`. Empty when no profile is fully running.", "items": { "type": "string" }, "title": "Active Profiles", "type": "array" }, "containers": { "description": "The matching containers, sorted by name. Empty when none match.", "items": { "$ref": "#/$defs/ContainerInfo" }, "title": "Containers", "type": "array" } }, "required": [ "active_profiles", "containers" ], "title": "PsReport", "type": "object" } ``` ## odctl info `odctl info --json` prints one object with the CLI version, the image tag, the workspace, Docker and odctl's own images. Its schema: ```json { "$defs": { "DockerInfo": { "properties": { "reachable": { "description": "Whether the daemon answers.", "title": "Reachable", "type": "boolean" } }, "required": [ "reachable" ], "title": "DockerInfo", "type": "object" }, "ImageInfo": { "properties": { "reference": { "description": "The full image reference, with its tag.", "title": "Reference", "type": "string" }, "local": { "description": "Whether the image is already on the Docker host. `false` when Docker is not reachable.", "title": "Local", "type": "boolean" } }, "required": [ "reference", "local" ], "title": "ImageInfo", "type": "object" }, "WorkspaceInfo": { "properties": { "path": { "description": "The workspace path, an absolute path.", "title": "Path", "type": "string" }, "exists": { "description": "Whether the folder exists. When it does not, odctl reads the files bundled with the CLI.", "title": "Exists", "type": "boolean" }, "stale_tag": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The workspace's `TAG` when it differs from the CLI version and no `TAG` is set in the shell. Otherwise `null`.", "title": "Stale Tag" } }, "required": [ "path", "exists", "stale_tag" ], "title": "WorkspaceInfo", "type": "object" } }, "properties": { "cli_version": { "description": "The installed odctl version.", "title": "Cli Version", "type": "string" }, "tag": { "description": "`TAG` as Docker Compose resolves it.", "title": "Tag", "type": "string" }, "workspace": { "$ref": "#/$defs/WorkspaceInfo", "description": "The workspace folder." }, "docker": { "$ref": "#/$defs/DockerInfo", "description": "The Docker daemon." }, "images": { "description": "odctl's own images at `tag` for every profile, sorted by reference.", "items": { "$ref": "#/$defs/ImageInfo" }, "title": "Images", "type": "array" } }, "required": [ "cli_version", "tag", "workspace", "docker", "images" ], "title": "Info", "type": "object" } ``` # Troubleshooting # Troubleshooting ## Containers Exit or Docker Runs Out of Memory Each service has a memory limit, listed on its [profile page](https://jaehyeon.me/odctl/latest/profiles/index.md). The limits of everything you start add up, and Docker needs that much memory available. Give Docker the memory that [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/#requirements) names, start fewer profiles at once, or choose a lite profile over a full one. To lower a limit, edit `mem_limit` in the compose file in `.odctl` and run `odctl recreate `, as [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) shows for SeaweedFS. ## Uploads Fail with HTTP 500 and Seaweed Exited with Code 137 `seaweed` reached its memory limit while it buffered a large upload, and the kernel killed it. The client sees HTTP 500 and every profile that uses S3 loses its storage until the container starts again. `docker inspect seaweed --format '{{.State.OOMKilled}}'` prints `true` after such a kill. A model or file of several hundred MB can reach the limit. Raise `mem_limit` and `GOMEMLIMIT` together, as [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) shows for 2 GB. A workspace made by a release before `GOMEMLIMIT` was added lacks it until you run `odctl init --force`, which [Start Again](https://jaehyeon.me/odctl/latest/how-it-works/#start-again) describes. ## A Port Is Already in Use Every service publishes fixed host ports, listed on its profile page. If another program holds one, `odctl up` fails to start that service. Stop the other program, or change the host side of the port mapping in `.odctl/compose-*.yml` and run `odctl recreate `, as [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) shows for Trino. Containers still reach each other on their internal ports, so only clients on your host need the new port. ## Use 127.0.0.1, Not localhost Host addresses in `odctl explain` and on these pages use `127.0.0.1`. With IPv6 enabled in Docker, `localhost` can resolve to a service's IPv6 address, and services that listen on IPv4 only reset the connection. `localhost` works when Docker's IPv6 is off. ## No Published Image for a TAG `odctl up` checks the images odctl builds before it starts anything. If the registry says an image for the resolved `TAG` does not exist, it stops with `No published image for TAG=...` and lists the missing images. An image already on your machine passes without a registry call. If the registry cannot be reached, odctl starts anyway and Docker reports what it finds. The usual causes are a CLI version whose images are not published yet, or a `TAG` in `.odctl/.env` that you changed by hand. Set `TAG` to a published version in `.odctl/.env`, or install a CLI version whose images exist. ## Workspace TAG Differs from the CLI Version After you upgrade the CLI, every command except `odctl init` warns that `.odctl/.env` sets an older `TAG`. The workspace keeps the old profiles, images and compose files until you run `odctl init --force`. [Start Again](https://jaehyeon.me/odctl/latest/how-it-works/#start-again) says what that command removes. ## A Service Is Up but Does Not Work `odctl logs ` shows the logs of a profile's containers. Add `-s ` for one service, or `-f` to follow them. `odctl ps --all` shows every container odctl manages and its state. ## Older Messages Are Missing The Kafka brokers delete old messages, so a consumer that starts from the beginning of a topic sees only what is left. [Keep Messages Longer](https://jaehyeon.me/odctl/latest/guides/kafka/#keep-messages-longer) gives the limits, shows how to read them and shows how to set `retention.ms` and `retention.bytes` to `-1` on a topic that must keep everything. # About # Why odctl I like building demo apps, and each one needs a local environment to try things in. Docker and Docker Compose are good for that. Across several projects, though, it is not easy, and the setup drifts from one project to the next. odctl is a CLI that manages Docker Compose services. It runs Compose files and selects services with Compose profiles. For one small project with two or three services, plain Compose is fine. It starts to hurt when you have several projects. One needs Kafka, Flink and Iceberg. Another needs Spark and Trino but not Kafka. Each project ends up with its own Compose file, and over time they drift apart, for example when you upgrade image tags in one and forget the others. With odctl, the files live in one place. Each release fixes the image versions, and an end-to-end test runs every profile before it is published. A project just pins an odctl version. The second part is the wiring, and odctl does it for you. When PostgreSQL first starts, it creates a database for each service that needs one, such as the Iceberg catalog, Airflow and MLflow. The Iceberg REST catalog keeps its tables in S3 storage (SeaweedFS), and Spark and Trino are already configured to use it. Kafka Connect, Flink and Spark also need connector jars with matching versions. odctl builds them once into a shared image, and each service takes the jars it needs from there. `odctl up spark-lite` starts PostgreSQL, the storage, the catalog and the shared jars first, and Prometheus already scrapes Kafka, Flink, Spark and ClickHouse. The third part is day-to-day use. With several Compose files, it is hard to remember what is running and where. odctl has commands for that: - `odctl list` shows every profile and what it runs. - `odctl explain spark-lite` shows a profile's services, images, ports and the profiles it depends on. - `odctl up spark-lite --dry-run` shows what would start, in order, before anything starts. - `odctl ps --all` lists the running containers and their ports. - `odctl logs spark-lite` shows the logs of one profile. - `odctl down --all` stops everything. Getting all of this right by hand takes time, and it is easy to break across projects. I use odctl in my own projects: [desflow](https://jaehyeon.me/desflow/) starts the services of its tests and examples with it, and so do the demos in [benchtop](https://github.com/jaehyeon-kim/benchtop). It is useful for three kinds of project: - **Data engineering.** Streaming and batch pipelines on Kafka, Kafka Connect, Flink, Fluss and Spark, an Iceberg lakehouse queried with Trino, ClickHouse for analytics, Airflow and Temporal to run them, and lineage in Marquez and OpenMetadata. - **ML and MLOps.** MLflow tracks experiments, registers models and serves them. Feast serves features from Iceberg tables and Valkey, and Evidently stores drift reports. The Airflow image has the MLflow and Feast clients, with the tracking URI and the Feast registry already set. - **AI.** PostgreSQL has pgvector and pg_textsearch for [hybrid search](https://jaehyeon.me/odctl/latest/guides/hybrid-search/index.md), and Valkey has vector search. An AI assistant can choose a profile and reach its services with the odctl skill and `llms.txt`, as [Getting started](https://jaehyeon.me/odctl/latest/getting-started/#use-it-with-an-ai-assistant) describes. `odctl mcp` serves odctl's commands to AI agents as MCP tools. The `langfuse` profile traces LLM and agent applications. The [Roadmap](https://jaehyeon.me/odctl/latest/about/roadmap/index.md) adds a desflow server profile after desflow 1.0. # Roadmap Now is the work being built, and Planned lists the rest in no order. Neither has dates or version numbers: an item gets its version when it ships. Each item links to its GitHub issue where one exists. ## Now Nothing is being built now. The next item comes from Planned. ## Planned - **Guides** ______________________________________________________________________ Iceberg table maintenance with Spark, vector search in Valkey, and traces and prompts in MLflow. [#152](https://github.com/jaehyeon-kim/odctl/issues/152) - **MLOps guides** ______________________________________________________________________ A prediction log and labels in PostgreSQL, feature and leakage checks, hyperparameter search, ONNX quantisation, serving a second model or many models, serving latency sent over OTLP, Feast dataset versions and reads of large Iceberg tables. - **Agents guides** ______________________________________________________________________ Traces, token metrics, feedback and evaluation of agent and model calls, an agent in a Temporal workflow, agent services beside the stack, and agent state in PostgreSQL. - **Retrieval guides** ______________________________________________________________________ Hybrid search with rank fusion, graph queries in PostgreSQL, ingestion with change detection, Valkey as a cache, and media with signed URLs. - **Testing and CI** ______________________________________________________________________ The end-to-end tests run on every pull request, including image updates. Each release is also tested as an upgrade from the one before it, and on arm64. [#102](https://github.com/jaehyeon-kim/odctl/issues/102) - **Kafka Connect transforms** ______________________________________________________________________ Transforms built into the Connect image: make a record schema required, replace personal fields with a hash, and send lineage events to Marquez for any connector. [#143](https://github.com/jaehyeon-kim/odctl/issues/143) - **Memory check and models on the host** ______________________________________________________________________ `odctl up` warns when Docker has less memory than the profiles it starts need. One setting lets containers reach a model server on the host with an OpenAI-compatible API, such as Ollama or LM Studio. [#155](https://github.com/jaehyeon-kim/odctl/issues/155), [#109](https://github.com/jaehyeon-kim/odctl/issues/109) - **desflow server** ______________________________________________________________________ When the desflow server ships: a profile that runs the [desflow](https://jaehyeon.me/desflow/) simulation server, so a project can start simulations that write to Kafka, PostgreSQL or Iceberg. [#144](https://github.com/jaehyeon-kim/odctl/issues/144) ## Released Newest first. 1.10.0: Agent skill and plugin The odctl agent skill is rewritten around the container lifecycle, with a checklist, rules that each show a wrong and a right example, and reference pages, and a plugin installs it with the read-only MCP server in Claude Code, Codex and Antigravity ([AI Assistants](https://jaehyeon.me/odctl/latest/ai/index.md)). [#177](https://github.com/jaehyeon-kim/odctl/issues/177) A workspace compose file that cannot be read stops each command with an error that says why, `odctl explain` names the role of the Langfuse server and worker, Feast's feature server and Label Studio, and the Trino notes give a `docker exec` command that works without a terminal. 1.9.0: Label Studio The `label-studio` profile runs Label Studio on odctl's PostgreSQL, and its guide loads the exported labels into a label table in PostgreSQL, which the MLOps guides then use ([Label Studio](https://jaehyeon.me/odctl/latest/guides/label-studio/index.md)). MLflow is 3.18.0 and Feast is 0.68.0. [#169](https://github.com/jaehyeon-kim/odctl/issues/169), [#184](https://github.com/jaehyeon-kim/odctl/issues/184) 1.8.0: Lineage and alerts Airflow, Spark and Feast send their lineage to Marquez ([OpenLineage and Marquez](https://jaehyeon.me/odctl/latest/guides/lineage/index.md)), and the Airflow image has Feast's `openlineage` extra. The `telemetry` profile loads Grafana alert rules from the workspace and ships one for a failed DAG run, and the Telemetry guide adds sample rules for data drift and a token budget ([Add an Alert Rule](https://jaehyeon.me/odctl/latest/guides/telemetry/#add-an-alert-rule)). [#149](https://github.com/jaehyeon-kim/odctl/issues/149) 1.7.0: Langfuse The `langfuse` profile traces LLM and agent applications in Langfuse, on odctl's PostgreSQL, ClickHouse, Valkey and SeaweedFS ([Langfuse](https://jaehyeon.me/odctl/latest/guides/langfuse/index.md)). On a new database, `admin@odctl.io` logs in to OpenMetadata with the password `password` ([OpenMetadata](https://jaehyeon.me/odctl/latest/guides/openmetadata/index.md)). [#120](https://github.com/jaehyeon-kim/odctl/issues/120), [#181](https://github.com/jaehyeon-kim/odctl/issues/181) 1.6.0: MCP server `odctl mcp` gives an AI agent `list`, `explain`, `ps`, `info` and `logs` as MCP tools, and `up` and `down` with `--allow-changes` ([MCP Server](https://jaehyeon.me/odctl/latest/guides/mcp-server/index.md)). The dry run of `odctl down --volumes` names the volumes it would remove. [#141](https://github.com/jaehyeon-kim/odctl/issues/141), [#170](https://github.com/jaehyeon-kim/odctl/issues/170) 1.5.0: JSON output and what stays stable in 1.x `odctl list`, `explain`, `ps` and `info` print JSON with `--json`, and [What Stays Stable in 1.x](https://jaehyeon.me/odctl/latest/how-it-works/#what-stays-stable-in-1x) lists what every 1.x release keeps. [#140](https://github.com/jaehyeon-kim/odctl/issues/140) 1.4.0: Feast Feast's feature server serves an MCP endpoint for agents, the Feast guide pushes features over HTTP and from Flink SQL and materialises from an Airflow DAG, and a new guide retrieves the nearest documents from PostgreSQL with pgvector. The `metadata` e2e group is split into three that run in parallel. [#150](https://github.com/jaehyeon-kim/odctl/issues/150), [#168](https://github.com/jaehyeon-kim/odctl/issues/168) 1.3.0: OpenMetadata connectors Custom OpenMetadata connectors uploaded to S3 are picked up by the ingestion container with no restart, AutoPilot is off by default, extra packages can be installed in the ingestion container, and OpenMetadata runs 2.0.7. 1.2.2: newer images Trino, Metabase, `otel-lgtm` and the PostgreSQL JDBC driver in the deps image are updated, and the Grafana dashboard descriptions no longer name the exact version of the component they show. 1.2.1: recreate on several files `odctl recreate`, `odctl restart` and `odctl logs` work on profiles from more than one compose file, `odctl init` and `odctl down` can skip their prompt, and the docs show every command with a runnable example. 1.2.0: Streaming Flink writes to HTTP APIs such as Feast's push endpoint, Spark reads Kafka without downloading packages, and new guides cover Kafka retention and several rolling windows in Flink SQL. The release also fixes the MLflow and storage guides and sets `GOMEMLIMIT`, a soft limit for the Go runtime, on `seaweed`, so large model uploads do not hit its container memory limit. 1.1.0: localhost-only services Every service is reachable from your machine only, Kafka no longer publishes the minikube listener ports, an odctl skill and `llms.txt` help AI assistants, and the docs have this Roadmap and a [Changelog](https://jaehyeon.me/odctl/latest/about/changelog/index.md). 1.0.2: Python 3.15 odctl is tested on Python 3.15, ZooKeeper and Metabase are updated, and Renovate groups image updates by compose file. 1.0.1: upgrade fixes Old jars no longer stay in the shared deps volume after an upgrade, and without a workspace odctl's images run with odctl's own version instead of a cached `latest`. 1.0.0: documentation site and compatibility promise A documentation site with guides, profile pages and the CLI reference. Semantic versioning with a compatibility promise for 1.x. `odctl up` works straight after `odctl recreate`. What each release changed is in the [Changelog](https://jaehyeon.me/odctl/latest/about/changelog/index.md) and on [GitHub Releases](https://github.com/jaehyeon-kim/odctl/releases). # Changelog Each release, newest first: what changed, then under Upgrading what to change in commands, workspaces or clients that use the previous release. Releases up to 1.0.2 are described on [GitHub Releases](https://github.com/jaehyeon-kim/odctl/releases). ## Version 1.10.0 A plugin that installs the odctl skill and the MCP server in Claude Code, Codex and Antigravity, a rewritten skill, clearer errors, and fixes from a review of the CLI. - A plugin installs the odctl skill and the read-only MCP server in Claude Code, Codex and Antigravity ([Plugin](https://jaehyeon.me/odctl/latest/ai/plugin/index.md)). Install odctl with `uv tool install 'odctl[mcp]'` first, then: - Claude Code: `claude plugin marketplace add jaehyeon-kim/odctl`, then `claude plugin install odctl@odctl`. - Codex: `codex plugin marketplace add jaehyeon-kim/odctl`, then `codex plugin add odctl@odctl`. - Antigravity: `agy plugin install https://github.com/jaehyeon-kim/odctl`. - The docs have an AI Assistants section for the plugin, the skill, the MCP server and `llms.txt` ([AI Assistants](https://jaehyeon.me/odctl/latest/ai/index.md)). The MCP Server guide keeps its address. - The agent skill is rewritten around the container lifecycle: a checklist, rules that each show a wrong and a right example, and reference pages for each profile's addresses and logins and for odctl in a project's tests ([Agent Skill](https://jaehyeon.me/odctl/latest/ai/agent-skill/index.md), [Use It with an AI Assistant](https://jaehyeon.me/odctl/latest/getting-started/#use-it-with-an-ai-assistant)). - OpenMetadata is 2.0.8, Confluent's converters are 8.4.1 and Debezium is 3.7.1. - The Trino notes of `odctl explain trino` give `docker exec trino trino --execute ''`, which works without a terminal ([Trino and Metabase](https://jaehyeon.me/odctl/latest/guides/trino-metabase/index.md)). - The MLflow notes of `odctl explain mlflow` say to run `odctl recreate mlflow` after setting `MODEL_URI` in `.env`, the command the [MLflow](https://jaehyeon.me/odctl/latest/guides/mlflow/index.md) guide runs. Before, they said `odctl up mlflow`. - A compose file in your workspace that is missing, is not UTF-8 text or is not valid YAML now stops every command that reads it with `Cannot read : ...`, which says which of these it is. A dry run of `up` or `down` without `--volumes` still prints the plan. Before, `down --volumes --dry-run` ended in a traceback, `info --json` left that file's images out, and the tables listed `File Error` as that file's services. - A volume such as `${PG_VOLUME:-pgdata}:/var/lib/postgresql/data`, in the short or the long form, is expanded before it is read, and a volume without `name:` is named after the project Compose runs, so `down --volumes --dry-run` lists the volume the real run removes. A source starting with `~` counts as a folder on the host, as Compose reads it. - A service with no `profiles:` key, which Compose starts with every profile of its file, is now listed by `ps`, `explain`, `logs --service` and `down`. - odctl passes each compose file's project name to Compose, so a `COMPOSE_PROJECT_NAME` in the shell or in `.env` no longer puts every file into one project, where `odctl down kafka-lite` removed the containers of the other profiles. A container counts for a file only when both its project and its service are that file's, so `ps`, `down` and the idle check of the shared jar volume no longer mix two files' services of the same name. - `odctl up` no longer reads a missing credential helper, or any other registry error whose text holds `not found`, as an image that is not published. - The check that warns about an old TAG in the workspace `.env` reads the file as Compose reads it, so `export TAG=1.9.0` and `TAG = 1.9.0` give the same answer as the image tag odctl runs. - A workspace whose `.env` has an empty TAG, or no TAG line, now runs the images of the CLI's version. Before, Compose ran the `latest` images. - `odctl init` and `odctl init --force` refuse a folder that holds files and no `registry.yml`, and a path that is a file. Its question names the folder it removes. `odctl init` copies `registry.yml` first, so an `init` that fails part way can be run again. - A `registry.yml` that is missing, empty, not UTF-8 text, not valid YAML or not in the registry's form now stops each command with `Cannot read : ...`. Before, `explain` and the dry runs of `up` and `down` ended in a traceback. - A registry whose profile is listed under two stacks or twice in one stack, whose stack lists dependencies for a profile of another stack, or whose dependency names no profile, now stops with a message that names the profile. - `odctl up` starts the file of a dependency before the file of the profile that depends on it, read from `depends_on` in the registry. Before, a workspace profile could start before a dependency in another file. - A failed pull in `odctl up --pull` prints `Failed to pull images for ` and exits 1. Before, it ended in a traceback. - `restart`, `recreate` and `logs` print `Failed to ` with Docker's message when a Compose call fails, and `restart` and `recreate` print each file before they act on it. Before, they ended in a traceback. - `odctl recreate deps` no longer waits on the container that copies the jars and exits. `odctl up` and `odctl recreate` stop with `Failed to compose-deps.yml` when that container exits with a code other than 0. Before, they went on as if the jars had been copied. - A followed log stream that fails, as in `odctl logs kafka-lite -f`, prints `Failed to fetch logs for ` and exits 1 as soon as it fails, while other streams still follow, and also when Ctrl+C comes after the failure. Before, it printed a thread traceback and exited 0. - `odctl logs --service ` with a service that none of the profiles defines prints an error that names the service and the profiles, and exits 1. Before, it printed nothing and exited 0. - The MCP tool `down` with `volumes` set refuses a call that names every profile except `deps`, because that call removes all data, which only `odctl down --all --volumes` in a terminal does after asking. A tool call that reaches a question of the command returns the command's text as its error, where before it returned only `Error executing tool down`. - `odctl down --volumes` asks the same question as `odctl down --all --volumes`, unless `--yes` is given. - `odctl down deps --volumes` keeps the shared jar volume while other profiles still have containers that use it, and says so, as the help of `--volumes` describes. Its dry run without Docker shows that deps is stopped in every case and that only the volume depends on the other profiles. The dry run of `down --volumes` also lists, under `compose-deps.yml`, the unnamed volumes of the stopped containers, if any. - A registry description, role, file or profile name, a compose service, port or volume name, or an error from Docker, that holds square brackets is printed as written, in `list`, `explain`, the dry runs and the progress lines. Before, `list` and `explain` could end in a markup error, and text such as `[bold]` disappeared. - The Network Mapping of `odctl explain` prints the ports with variables expanded, as `explain --json` does, leaves out a port with no host part or a port range, and adds the protocol only when it is not tcp. Before, `${KAFKA_PORT:-19092}` printed as `Host -19092}`. - `odctl explain` lists a service built from source, with no image, as `built locally`. Before, it was left out when another service had an image. - `odctl ps` counts a profile as active only when its services are running, or, for a one-shot service that another service waits on, have exited with code 0. Before, a stopped container counted as running. - `odctl explain` names the role of the Langfuse server and worker, Feast's feature server, ZooKeeper, the ClickHouse shard 2 stand-in and Label Studio. Each role matches the image's repository rather than any text in the image name, with or without a tag or a digest. - The skill's description names Label Studio and Metabase, so an assistant loads it for a task that needs them. - odctl reads the workspace `.env` as Compose does: it drops a UTF-8 byte order mark, strips an inline comment after a space, removes quotes, applies escapes inside double quotes and expands `${OTHER}`, and `${OTHER}` in a value reads a variable set in the shell before one set earlier in the file. Before, `TAG=1.9.0 # pinned` made odctl check the registry for the tag `1.9.0 # pinned`. - `odctl down --volumes` for a profile your workspace adds to `compose-deps.yml`, such as `deps-extra`, now removes that profile's own volumes, as `docker compose down --volumes` does. Before, they stayed while any other profile had a container. A profile that mounts the shared jar volume still keeps it until no other profile has a container. The dry run lists the same volumes, with `--all` too. - odctl reads your compose files as Docker Compose reads them. A port written without quotes, such as `2222:22`, now shows in `list -d`, `explain` and `ps`, where it used to be read as a number and left out. `yes`, `on` and `off` stay text. A key tagged `!reset` under a `<<` merge key now takes the merged value, as Compose does. - A workspace profile name with square brackets no longer stops `odctl up --pull` or `odctl ps` with a markup error. - `odctl --help` no longer prints the internal `Args:` notes of the `info` command. ### Upgrading - When a compose file in your workspace is missing, is not UTF-8 text or is not valid YAML, odctl now stops with exit code 1 and `Cannot read : ...`, which says which of these it is. Before 1.10.0, the commands without `--json` listed `File Error` or `Err: ` as that file's service, port, image and volume, `info --json` left that file's images out, and `down --volumes --dry-run` ended in a traceback. A dry run of `up` or `down` without `--volumes` still prints the plan. Fix the file, or run `odctl init --force` to copy the packaged one; it replaces your edits, so keep a copy of them. - `odctl explain` names the role of the Langfuse server and worker, Feast's feature server, ZooKeeper, the ClickHouse shard 2 stand-in and Label Studio, which it showed as `Service Container`. A script that reads that line sees the new role. A workspace whose compose file uses the image `valkey/valkey`, which only 0.0.1 and 0.0.2 shipped, now shows `Service Container` for it; the packaged `valkey/valkey-bundle` keeps its role. - The Trino notes that `odctl explain trino` prints now give `docker exec trino trino --execute ''`, which works without a terminal. A workspace made by `odctl init` before 1.10.0 keeps its old `registry.yml`, so run `odctl init --force` to copy the new text; it replaces your edits, so keep a copy of them. - OpenMetadata is 2.0.8, up from 2.0.7. A workspace made by `odctl init` before 1.10.0 keeps its old `compose-metadata.yml`, which runs 2.0.7: run `odctl init --force` to copy the new file; it replaces your edits, so keep a copy of them. On the next `odctl up` with the metadata profile, `openmetadata-migrate` applies 2.0.8's database migrations to an existing OpenMetadata database. - A workspace service with no `profiles:` now appears in the `services` of `list --json`, `explain --json` and `ps --json`, and `down` stops it with its profile. - odctl now ignores `COMPOSE_PROJECT_NAME`. Each compose file runs as the project its `name:` names. If you changed `name:` in a workspace file, odctl now manages the containers of that project. If you set `COMPOSE_PROJECT_NAME` for an earlier version, its containers stay in that project, which odctl no longer manages, and `odctl up` fails while their fixed container names are in use. Remove them with `docker compose -p down` before you run `odctl up`. - A workspace whose `.env` has an empty TAG, or no TAG line, now runs the images of the CLI's version, not `latest`. Set TAG in `.env` to run another version. - `odctl init` and `odctl init --force` now refuse a `--workspace` that names a file, or a folder that holds files and no `registry.yml`. Name a new or empty folder, or an existing workspace. - A workspace registry with a profile listed under two stacks or twice in one stack, a `depends_on` key that is a profile of another stack, or a dependency that no stack lists, loaded before and now stops every command with a message. Fix the entry it names. - `odctl up` and `odctl recreate` now exit 1 when the container that copies the jars into `odctl-shared-deps` exits with a code other than 0, where they went on and exited 0. - A failed `odctl logs -f` now exits 1, where it exited 0. - `odctl logs --service ` with a service that none of the profiles defines now exits 1, where it printed nothing and exited 0. - The MCP tool `down` now refuses a call with `volumes` set that names every profile except `deps`, which it accepted. To remove every profile's data, run `odctl down --all --volumes` in a terminal, which asks first, or name fewer profiles. - A script that runs `odctl down` with every profile and `--volumes` now needs `--yes`, because the command asks first. - `odctl down deps --volumes` now keeps the shared jar volume `odctl-shared-deps` while other profiles have containers. Stop them first, or run `odctl down --all --volumes`, to remove it. - `active_profiles` in `odctl ps --json` no longer lists a profile whose containers are stopped. A one-shot container that exited with code 0 still counts. - odctl now reads `.env` as Compose does. A value with an inline comment, quotes, escapes or `${OTHER}`, such as `TAG=1.9.0 # pinned`, now gives the value Compose uses, and `${OTHER}` in a value reads a variable set in the shell before one set earlier in the file, so `odctl info` and the check for a published image can show a different tag than before. ## Version 1.9.0 Label Studio, and MLflow 3.18.0 and Feast 0.68.0. - The `label-studio` profile runs Label Studio on odctl's PostgreSQL, and its guide loads the exported labels into a label table in PostgreSQL ([Label Studio](https://jaehyeon.me/odctl/latest/guides/label-studio/index.md)). - Label Studio can reach an ML backend, a webhook receiver and SeaweedFS on the `odctl` network, and the `storage` profile creates a `label-studio` bucket for its S3 storage ([Connect Other Services](https://jaehyeon.me/odctl/latest/guides/label-studio/#connect-other-services)). - MLflow is 3.18.0. On each start, `mlflow` upgrades the schema of a database that an earlier MLflow created. - A download of a model from the host works with no client setting ([Load a Model from Your Machine](https://jaehyeon.me/odctl/latest/guides/mlflow/#load-a-model-from-your-machine)). - MLflow's automatic evaluation scores traces with a judge that uses an AI Gateway endpoint. Before, every assessment failed with "Invalid Host header" ([MLflow](https://jaehyeon.me/odctl/latest/guides/mlflow/index.md)). - Feast is 0.68.0. - `feast-serve` and `feast-ui` keep running when `s3://feast/repo` changes while they load it. Before, a file deleted during the load stopped the container until the next `odctl up`. - The docs site lists one version per minor release, such as 1.2, with the docs of its last patch. Patch versions are no longer listed, so an address with one, such as `https://jaehyeon.me/odctl/1.2.0/`, now returns 404; use the minor, `https://jaehyeon.me/odctl/1.2/`. ### Upgrading - A new profile, `label-studio`, in the `mlops` stack. It starts PostgreSQL and uses host port 8088. - A workspace made by `odctl init` before 1.9.0 has an older `compose-mlops.yml`, `compose-infra.yml` and PostgreSQL init script. Run `odctl init --force` to copy the new files; it replaces your edits, including `.env`, so keep a copy of them. Put your settings back into the new `.env` before you start the services, because a `postgres` container keeps the user and password it was created with. A `postgres` container started before 1.9.0 does not run the init script again, so it has no `labelstudio` database. Start it with `odctl up postgres`, then create the database once before `odctl up label-studio`: `docker exec postgres psql -U user -d postgres -c 'CREATE DATABASE labelstudio OWNER "user"'`. The `postgres` database always exists, whatever `POSTGRES_DB` is set to. If you set `POSTGRES_USER`, put that name in place of `user` in both places. - MLflow is 3.18.0, up from 3.16.1. On its next start, `mlflow` upgrades the schema of your existing `mlflow` database. Going back to 3.16.1 afterwards is not tested. Install `mlflow==3.18.0` on the host to match the server. - A download from the host no longer needs `MLFLOW_ENABLE_PROXY_MULTIPART_DOWNLOAD=false`; you can remove it. - Feast is 0.68.0, up from 0.66.0, and needs Python 3.11 or newer. Install `feast[...]==0.68.0` on the host. The feature server's latency metric labels `feature_count` with a range, such as `1-10`, so a query that matches a single count changes. ## Version 1.8.0 Lineage from Airflow, Spark and Feast in Marquez, and Grafana alert rules in the `telemetry` profile. - Airflow sends each DAG run's lineage to Marquez when `AIRFLOW_OPENLINEAGE_TRANSPORT` is set in `.env`; the key is empty by default ([Lineage from Airflow](https://jaehyeon.me/odctl/latest/guides/lineage/#lineage-from-airflow)). - A Spark job sends its lineage to Marquez through the OpenLineage Spark listener, whose jar the Spark containers already load ([Lineage from Spark](https://jaehyeon.me/odctl/latest/guides/lineage/#lineage-from-spark)). - `feast apply` sends the feature views it applies to Marquez when `feature_store.yaml` enables it, and the `airflow` image has Feast's `openlineage` extra for that ([Lineage from Feast](https://jaehyeon.me/odctl/latest/guides/lineage/#lineage-from-feast)). - The `telemetry` profile loads Grafana alert rules from `.odctl/grafana/alerting/` and ships one that fires when an Airflow DAG run fails ([Add an Alert Rule](https://jaehyeon.me/odctl/latest/guides/telemetry/#add-an-alert-rule)). - The Telemetry guide has two sample rules: one for data drift found by an Evidently check in an Airflow DAG, and one for a budget of model tokens. ### Upgrading - A new `.env` key, `AIRFLOW_OPENLINEAGE_TRANSPORT`, is empty by default, so nothing changes until you set it. - A workspace made by `odctl init` before 1.8.0 keeps its old `compose-orch.yml` and `compose-obsv.yml`, has no `grafana/alerting` folder, and has no `AIRFLOW_OPENLINEAGE_TRANSPORT` line in its `.env`. Run `odctl init --force` to copy the new files and write a new `.env` that has the key; it replaces your edits, so keep a copy of them. Then set the key as [Lineage from Airflow](https://jaehyeon.me/odctl/latest/guides/lineage/#lineage-from-airflow) shows, and run `odctl recreate` for the profiles you use, such as `odctl recreate airflow telemetry`. ## Version 1.7.0 A Langfuse profile for tracing LLM and agent applications, Valkey starting earlier, the OpenMetadata admin's password changed to `password`, and newer otel-lgtm and Confluent converters. - The `langfuse` profile runs Langfuse in a new AI Engineering area, with its UI and OTLP endpoint on host port 3005 ([Langfuse](https://jaehyeon.me/odctl/latest/guides/langfuse/index.md)). - The `storage` profile also creates a `langfuse` bucket. - `valkey` now starts before the other profiles that sort after it, so `odctl up` and its `--dry-run` list it earlier ([Dependencies and the Start Order](https://jaehyeon.me/odctl/latest/how-it-works/#dependencies-and-the-start-order)). - On a new database, `admin@odctl.io` logs in to OpenMetadata with the password `password`, the same as `user@odctl.io` ([OpenMetadata](https://jaehyeon.me/odctl/latest/guides/openmetadata/index.md)). - The `telemetry` profile runs `grafana/otel-lgtm:0.40.0`, up from 0.37.0. The deps image has the Confluent Avro, Protobuf and JSON Schema converters 8.4.0, up from 8.3.2. ### Upgrading - A workspace made by `odctl init` before 1.7.0 keeps its old files: it has no `langfuse` profile, `telemetry` keeps `otel-lgtm` 0.37.0, and a new OpenMetadata database gives `admin@odctl.io` the password `admin`. Run `odctl init --force` to copy the new files; it replaces your edits, so keep a copy of them. Then run `odctl recreate` for the profiles you use, such as `odctl recreate telemetry metadata`. - A `ch-lite` with data from an earlier release does not run its init file again, so it has no `langfuse` database. Start it with `odctl up ch-lite` and create the database: `docker exec ch-11 clickhouse-client --password password --query "CREATE DATABASE IF NOT EXISTS langfuse"`. Then run `odctl up langfuse`: `seaweed-init` creates the `langfuse` bucket, and Langfuse creates its PostgreSQL database. - An OpenMetadata database created by 1.3.0 to 1.6.0 keeps the admin password `admin`, so a script that logs in as `admin@odctl.io` uses `admin` there and `password` on a new database. `openmetadata-init` accepts both. ## Version 1.6.0 An MCP server that gives AI agents odctl's profiles, containers and logs, and a dry run of `odctl down --volumes` that names the volumes. - `odctl mcp` serves `list`, `explain`, `ps`, `info` and `logs` as tools to an MCP client, such as Claude Code, after `uv tool install "odctl[mcp]"` ([MCP Server](https://jaehyeon.me/odctl/latest/guides/mcp-server/index.md)). - All tools but `logs` return the same JSON as `--json`, and their names and parameter names stay the same in every 1.x release ([What Stays Stable in 1.x](https://jaehyeon.me/odctl/latest/how-it-works/#what-stays-stable-in-1x)). - `odctl mcp --allow-changes` also serves `up` and `down`, so an agent can start and stop profiles. - The `down` tool with `volumes` refuses profiles that cover all that `odctl down --all --volumes` would remove, because that command asks first and the server cannot ask. - `odctl down --volumes --dry-run` names the volumes the real run would remove: the named volumes, the number of unnamed volumes of each container, and `odctl-shared-deps` once no other profile has a container. - `odctl down --all --dry-run` reports that Docker is not reachable instead of ending with a traceback. - odctl reads Compose variables in a workspace file in every form Compose supports, such as `$VAR`, `${VAR:+word}` and a nested default, so the images, ports and volume names it shows match what Compose runs. ## Version 1.5.0 JSON output from `list`, `explain`, `ps` and `info`, a written list of what 1.x keeps stable, and newer SeaweedFS, Valkey and Spark images. - `odctl list`, `odctl explain`, `odctl ps` and `odctl info` take `--json` and print one JSON document for scripts and AI assistants, with warnings and errors on stderr ([JSON Output](https://jaehyeon.me/odctl/latest/json-output/index.md), [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md)). - `odctl explain` names a dependency the registry lists but no stack defines, instead of ending with a traceback; it still exits 1. - `odctl info --json` also shows the image tag, the workspace and which of odctl's own images are on your machine. - [What Stays Stable in 1.x](https://jaehyeon.me/odctl/latest/how-it-works/#what-stays-stable-in-1x) lists what every 1.x release keeps: profile names, published ports, `.env` keys, volumes, the workspace files and the `--json` keys. - odctl now needs Typer 0.18.0 or later, because older Typer releases misread odctl's options with newer Click releases; with some pairs `odctl init` replaced an existing workspace without asking. - The base of odctl's `postgres` image, `postgres:18-trixie`, is pinned by digest, so a release that does not change the image no longer gives it new layers to download. - SeaweedFS runs `chrislusf/seaweedfs:4.52`, up from 4.48. Valkey runs `valkey/valkey-bundle:9.1.4-alpine`, up from 9.1.3. odctl's Spark image is built on `apache/spark:4.1.4`, up from 4.1.3, and the deps image has the Spark Kafka connector 4.1.4. - The odctl skill (`skills/odctl/SKILL.md`) names each image without its tag; `odctl explain ` shows the tag. ### Upgrading - A workspace made by `odctl init` with an earlier release keeps its old `compose-infra.yml` and `compose-store.yml`, and so the old SeaweedFS and Valkey images. Run `odctl init --force` to copy the new files; it replaces your edits, so keep a copy of them. Then run `odctl recreate` for the profiles you use, such as `odctl recreate storage valkey`. ## Version 1.4.0 Feast's MCP endpoint, pushing features over HTTP and from Flink, materialising from Airflow, and vector retrieval on PostgreSQL. - `feast-serve` serves an MCP endpoint at `http://127.0.0.1:6566/mcp` with each route of the feature server as a tool ([Connect an MCP Client](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#connect-an-mcp-client)). - [Push Features](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#push-features) pushes a value to the feature server [Over HTTP](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#over-http) and [From Flink SQL](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#from-flink-sql), then reads it back. - [On a Schedule](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#on-a-schedule) materialises the feature view into Valkey from an Airflow DAG every hour. - [Vector Retrieval with Feast](https://jaehyeon.me/odctl/latest/guides/feast-vector-retrieval/index.md) writes document embeddings to PostgreSQL with pgvector and retrieves the nearest rows. - `feast apply` in the Feast guide now runs with `--skip-source-validation`, so the same definitions work from your shell and from the Airflow container. ### Upgrading - A workspace made by `odctl init` before 1.4.0 keeps its `feast/feature_store.yaml` without the MCP endpoint, and its `registry.yml` without the MCP line in `odctl explain feast`. Run `odctl init --force` to copy the new files (it replaces your edits), or add the `feature_server` block yourself. Then run `odctl recreate feast`. - If you applied the guide's definitions before 1.4.0, their Iceberg source stores `endpoint`, the catalog address of your shell, which the Airflow container cannot reach. Remove `endpoint` from the source and run `feast apply --skip-source-validation` again before you run the DAG. ## Version 1.3.0 Custom connectors for OpenMetadata, extra packages for its ingestion container, AutoPilot off by default, OpenMetadata 2.0.7, and newer Kafka and Metabase images. - A Python connector uploaded to `s3://openmetadata/connectors` is importable in the ingestion container within about 15 seconds, and a change to it is used by the next agent run with no restart. [Add a Custom Connector](https://jaehyeon.me/odctl/latest/guides/openmetadata/#add-a-custom-connector) creates its service and agent through the API. - OpenMetadata's AutoPilot, which creates and runs default agents for a service added in the UI, is off by default. The new `.env` key `METADATA_AUTOPILOT` turns it on, and the new `openmetadata-init` service applies it each time `metadata` starts. See [Register PostgreSQL as a Database Service](https://jaehyeon.me/odctl/latest/guides/openmetadata/#register-postgresql-as-a-database-service). - `_METADATA_PIP_DEPS` in `.odctl/.env` installs extra Python packages in the `openmetadata-ingestion` container, as `_AIRFLOW_PIP_DEPS` does for Airflow. See [Examples](https://jaehyeon.me/odctl/latest/how-it-works/#examples). - `openmetadata-ingestion` has a healthcheck, so `odctl up` and `odctl recreate` return once its Airflow answers. - [Where Data Lives](https://jaehyeon.me/odctl/latest/how-it-works/#where-data-lives) shows the three profiles that read their code from a bucket: `airflow`, `feast` and `metadata`. - OpenMetadata and its ingestion container run 2.0.7, up from 2.0.1. - On a new database the sign-in domain is `odctl.io`, and a second admin `user@odctl.io` has the password `password`. The ingestion Airflow takes `user` and `password`. - Kafka and Kafka Connect run `apache/kafka:4.3.2`, up from 4.3.1. Metabase runs v0.64.3.5, up from v0.64.3.1. The deps image has `log4j-slf4j2-impl` 2.26.1, up from 2.23.1. ### Upgrading - The `metadata` profile now also starts `storage`, SeaweedFS on 127.0.0.1 ports 8333, 8889 and 9333, from which the ingestion container takes connectors. - Two new `.env` keys, `_METADATA_PIP_DEPS` and `METADATA_AUTOPILOT`, are written by `odctl init`. An existing `.env` works without them: add the lines, or get them with `odctl init --force`. - AutoPilot is now off unless `METADATA_AUTOPILOT="true"`, so a service added in the OpenMetadata UI no longer gets default agents. An existing `.env` has no key, which means off. Set `METADATA_AUTOPILOT="true"` in `.odctl/.env` to keep the old behaviour. - `openmetadata-migrate` runs `openmetadata-ops.sh migrate` against your existing database on the next start of `metadata`, by `odctl up` or `odctl recreate`, for the move from 2.0.1 to 2.0.7. Going back to 2.0.1 afterwards is not tested. - The sign-in domain `odctl.io` and the password `password` of `user` apply to a new database only. A database from an earlier release keeps its users `admin@open-data.local` and `user@open-data.local`, both with the password `admin`, because the server restores a user's stored email on every update (`UserRepository.java`, `entitySpecificUpdate`, at `2.0.7-release`) and keeps a stored password. On that database `admin@odctl.io` and `user@odctl.io` fail with HTTP 400, "You have entered an invalid username or password." - The ingestion Airflow login changes from `admin` / `admin` to `user` / `password`. - A workspace made by `odctl init` with an earlier release keeps its old `compose-metadata.yml`, `compose-infra.yml`, `compose-kafka.yml`, `compose-analytics.yml` and `registry.yml`, has no `openmetadata/connector_sync.py` or `openmetadata/autopilot_init.py`, and so has none of the changes above. Run `odctl init --force` to copy the new files; it replaces your edits, so keep a copy of them. Then run `odctl recreate metadata storage`, and `odctl recreate` for the Kafka and Metabase profiles you use, such as `odctl recreate kafka-lite metabase`. ## Version 1.2.2 Newer images for Trino, Metabase and the observability stack. - Trino runs 484, up from 483. Metabase runs v0.64.3.1, up from v0.63.19.3. `otel-lgtm` runs `grafana/otel-lgtm:0.37.0`, up from 0.35.0. The deps image has the PostgreSQL JDBC driver 42.7.14, up from 42.7.13. - The Grafana dashboard descriptions no longer name the exact version of the component they show, so they stay true when an image is updated. They name a major version only where it tells two kinds apart: Flink 2, whose metrics differ from Flink 1, and OpenMetadata 2, which is maintained beside OpenMetadata 1. ### Upgrading - A workspace made by `odctl init` with an earlier release keeps its old compose files, and so the old images. Run `odctl init --force` to copy the new files; it replaces your edits, so keep a copy of them. Then run `odctl recreate` for the profiles you use, such as `odctl recreate trino metabase`. ## Version 1.2.1 `recreate`, `restart` and `logs` work on profiles from several compose files, `init` and `down` can skip their prompt, and the docs show every command with a runnable example. - `odctl recreate`, `odctl restart` and `odctl logs` work on profiles from more than one compose file, such as `odctl recreate storage kafka-lite`. They ran every file as one project, so `recreate` failed with `Conflict. The container name "/seaweed" is already in use`, and `restart` and `logs` left out the containers of all but one file. Each file now runs as its own project, as with `odctl up`. [Apply a Change](https://jaehyeon.me/odctl/latest/how-it-works/#apply-a-change) uses `odctl recreate`. - `odctl init --force --yes` and `odctl down --all --volumes --yes` run without asking `Are you sure?`, for scripts and CI. `-y` is the short form. See [Stop a Profile](https://jaehyeon.me/odctl/latest/getting-started/#stop-a-profile) and [Start Again](https://jaehyeon.me/odctl/latest/how-it-works/#start-again). - [Getting Started](https://jaehyeon.me/odctl/latest/getting-started/index.md) runs every command on `kafka-lite`, with its output, and the [home page](https://jaehyeon.me/odctl/latest/#commands) lists each command with a link to its example. - Every guide ends with a Runnable Example that names the files to run. [Rolling Windows](https://jaehyeon.me/odctl/latest/guides/rolling-windows/index.md) is a page of its own, and the Flink push to Feast moved to [From Flink SQL](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#from-flink-sql) on the Feast guide. - The animation on [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#what-odctl-up-does) shows its four steps again. Since 1.1.0 it showed steps 1 to 3 for a moment each and then stayed on step 4, because one step was exported 2 pixels narrower and ffmpeg stopped at the change in size. - The Labs page is hidden until its first lab is published. ## Version 1.2.0 Flink can send rows to an HTTP API, Spark reads Kafka with no extra options, SeaweedFS survives uploads of large models, and the guides cover cases that real projects met. - Flink's SQL client has the HTTP connector. [From Flink SQL](https://jaehyeon.me/odctl/latest/guides/feast-iceberg/#from-flink-sql) on the Feast guide pushes a feature value to Feast. - Spark has the Kafka connector and the Avro module, so a job reads a topic, including Avro records written with the schema registry, with no `--packages` option. See [Read a Kafka Topic](https://jaehyeon.me/odctl/latest/guides/spark/#read-a-kafka-topic) and [Read Avro Records Written with the Schema Registry](https://jaehyeon.me/odctl/latest/guides/spark/#read-avro-records-written-with-the-schema-registry). - [Keep Messages Longer](https://jaehyeon.me/odctl/latest/guides/kafka/#keep-messages-longer) shows the broker's one-day and 500 MB retention and how to turn both off for a topic. [Troubleshooting](https://jaehyeon.me/odctl/latest/troubleshooting/#older-messages-are-missing) lists the symptom. - `seaweed` sets `GOMEMLIMIT` to three quarters of its memory limit, so uploads of a model of a few hundred MB no longer end with HTTP 500 and a killed container. [Troubleshooting](https://jaehyeon.me/odctl/latest/troubleshooting/#uploads-fail-with-http-500-and-seaweed-exited-with-code-137) says what to do when the limit is still reached, and [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) changes both values together. - The [MLflow guide](https://jaehyeon.me/odctl/latest/guides/mlflow/index.md) has three new sections: [Move the Alias to a New Version](https://jaehyeon.me/odctl/latest/guides/mlflow/#move-the-alias-to-a-new-version), which needs `odctl restart mlflow`, [Load a Model from Your Machine](https://jaehyeon.me/odctl/latest/guides/mlflow/#load-a-model-from-your-machine) and [Serve a Model That Reads Online Features](https://jaehyeon.me/odctl/latest/guides/mlflow/#serve-a-model-that-reads-online-features). - The docs now say that the packages in `_MLOPS_PIP_DEPS` are installed from PyPI again in every new `mlflow` and `mlflow-serve` container. [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) lists when a new container is made. - [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/#examples) shows how to create a bucket before its profile exists. - [Rolling Windows](https://jaehyeon.me/odctl/latest/guides/rolling-windows/index.md) shows that Flink SQL refuses two different `OVER` windows in one streaming query, and the two ways around it. - The [Evidently guide](https://jaehyeon.me/odctl/latest/guides/evidently-reports/index.md) says what Evidently does not estimate and why NannyML needs Python 3.12. ### Upgrading - A workspace made by `odctl init` with an earlier release keeps its old `compose-infra.yml`, which has no `GOMEMLIMIT` for `seaweed`. Run `odctl init --force` to copy the new files; it replaces your edits, so keep a copy of them. Then run `odctl recreate storage`. ## Version 1.1.0 Every service is now reachable from your machine only, and the docs have a [Roadmap](https://jaehyeon.me/odctl/latest/about/roadmap/index.md) and a [Changelog](https://jaehyeon.me/odctl/latest/about/changelog/index.md). - Every published port binds to `127.0.0.1`, on macOS, Linux and WSL2, so other machines on your network cannot reach the services. [How You Reach a Service](https://jaehyeon.me/odctl/latest/how-it-works/#how-you-reach-a-service) says how to open a port when you need to. - Kafka no longer publishes ports 29092, 29093 and 29094, which were listeners for a minikube cluster. - An odctl skill and `llms.txt` help an AI assistant choose a profile and reach its services. See [Use It with an AI Assistant](https://jaehyeon.me/odctl/latest/getting-started/#use-it-with-an-ai-assistant). - Eight guides show in a diagram how data moves between their services, and [How It Works](https://jaehyeon.me/odctl/latest/how-it-works/index.md) shows what `odctl up` checks and the order it starts services in. The end-to-end tests run every command the guides show. - The tables of `odctl ps` and `odctl explain` draw straight right borders on rows with port mappings. - `odctl down --help` says what `--volumes` removes. ### Upgrading - To reach a service from another machine, run `odctl init`, remove `127.0.0.1:` from the service's port line in `.odctl/compose-*.yml`, and run `odctl recreate `. - A Kafka client in minikube cannot use `host.minikube.internal:29092` any more. Add a listener for it to the Kafka service in `.odctl/compose-kafka.yml`. - A workspace made by `odctl init` with an earlier release keeps its old compose files, which publish on all interfaces and include port 29092. Run `odctl init --force` to copy the new files; it replaces your edits.