Skip to content

Guides

Each guide starts the profiles one task needs, then shows how to use their services from the host and from a container. The guides are in groups by area, in the same order as the sidebar. A guide may start from a stack that an earlier guide in its group set up, so the pages of a group read in order.


Streaming

Guide What it shows
Kafka the Kafka broker, the schema registry and Kafka UI of kafka-lite
Kafka Connect the Kafka Connect worker and the connectors it comes with
Flink SQL a Flink SQL job, sent through the SQL Gateway, that reads a Kafka topic and writes totals to PostgreSQL
Rolling Windows features over the last 10 minutes, 1 hour and 1 day, with OVER windows
Fluss Fluss tables that Flink writes and reads with low latency

Lakehouse and Batch

Guide What it shows
Ingest into Iceberg Kafka Connect, Flink and Spark writing Iceberg tables through one REST catalog
Spark Spark jobs on the iceberg catalog, and the Spark History Server

Analytics

Guide What it shows
ClickHouse a ClickHouse cluster with shards, replicas and ClickHouse Keeper
Trino and Metabase one SQL interface over PostgreSQL, Iceberg, ClickHouse, Kafka and Valkey, and charts in Metabase

Orchestration

Guide What it shows
Airflow DAGs from S3 Airflow reading DAG files that you upload to SeaweedFS
Temporal Workers a worker of your own that runs workflows and activities, and a workflow that waits for an approval

ML and AI

Guide What it shows
Feast with Iceberg a feature store with its offline store on Iceberg and its online store on Valkey
Feast Vector Retrieval the stored rows whose embeddings are nearest to a query embedding
MLflow runs and models in the tracking server, and a registered model scored over HTTP
Evidently Reports data drift and data quality reports pushed to the Evidently UI
Label Studio a labelling project, its tasks and labels, and the labels loaded into PostgreSQL
Hybrid Search vector search and BM25 keyword ranking in PostgreSQL, and their rankings combined
Valkey the Valkey Bundle, with its modules for JSON documents, Bloom filters and vector search
Langfuse traces and evaluations of an application built on an LLM

Metadata and Operations

Guide What it shows
OpenLineage and Marquez OpenLineage events from jobs, and the lineage graph Marquez draws
OpenMetadata a data catalogue of the stack's databases, tables and columns
Telemetry metrics, traces and logs sent with OpenTelemetry and read in Grafana

The MCP server, which gives an AI assistant odctl's commands as tools, is in AI Assistants.