odctl on Jaehyeon Kim

odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command

Mon, 12 Oct 2026 00:00:00 +0000

Trying a modern data tool on your own machine is rarely about the tool. Writing Iceberg tables from Flink needs a catalog, the catalog needs a database and object storage, Avro on Kafka needs a schema registry, and every piece needs the right connector jars, ports and network settings. Most of an afternoon goes into making them talk to each other before you write a line of your own code.

odctl removes that work. It is a command-line tool that runs a set of open-source data and machine learning services on your laptop with Docker, already connected to each other. You say what you want, for example Kafka and Flink, and it starts them together with everything they depend on. It has now reached 1.0, with a documentation site that has a guide for every service.

Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL

Fri, 02 Oct 2026 00:00:00 +0000

A mobile game produces a score every time a player finishes a round, and players expect the leaderboards to move as they play. In this post, a simulation plays the game and sends each score to Kafka. Four Flink SQL jobs keep four top 10 leaderboards up to date in PostgreSQL, and a web dashboard shows them as they change. Everything runs on your own machine.

Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect

Thu, 01 Oct 2026 00:00:00 +0000

Most databases change all day: users sign up, orders are placed, and each order moves from one status to the next. Change data capture (CDC) turns those changes into a stream of events that other systems can read as they happen. In this post, a simulated online shop writes to PostgreSQL in real time, Debezium streams every change to Kafka, and a sink connector saves the changes as files in object storage. Everything runs on your own machine.

Data Streaming and Machine Learning Projects That Run on Your Laptop

Wed, 30 Sep 2026 00:00:00 +0000

Reading about a data tool only goes so far. The hard part is getting several tools to run together and seeing how data moves between them. Benchtop is a collection of small, hands-on demos for data engineering, stream processing, machine learning, AI engineering and MLOps. Each demo runs locally on the odctl stack, and works from a fresh clone of the repository, with nothing to install beforehand but Docker and uv. Each one builds a working system from open-source tools and explains the ideas behind it, so you learn a tool by running it rather than only reading about it.

Learning MLOps with a Feature Store: A New Series

Mon, 28 Sep 2026 00:00:00 +0000

Learning MLOps one tool at a time leaves out the hard part: how features, training and serving fit together as one system. This series learns it by building complete systems. I follow Jim Dowling’s Building Machine Learning Systems with a Feature Store and rebuild its hands-on projects with open-source tools, running on my laptop.

Building an Agentic Analytics System over an Iceberg Lakehouse

Sat, 18 Jul 2026 00:00:00 +0000

Generative AI has made conversational analytics feel within reach, yet direct text-to-SQL systems remain hard to operate reliably. A database schema tells you the tables, columns, and types, but it says nothing about which datasets are canonical, which join paths are approved, how a governed metric is calculated, or what the business actually means by “revenue” or “active customer”. Ask a language model to infer all of that from raw tables and, sooner or later, it will confidently invent an answer.

I built a proof of concept to explore a more disciplined approach: put a semantic layer between the model and the data, and let the agent reason over governed business concepts instead of guessing at raw schemas. It runs entirely locally on an open-source stack, and it happens to tie together two of my other projects along the way.

Running Kafka, Flink, Spark, Trino and Iceberg Locally with One CLI

Thu, 16 Jul 2026 00:00:00 +0000

Anyone who has tried to stand up a realistic data platform on their laptop knows the pain. You want Kafka talking to Flink, Spark writing to Iceberg, Trino querying the result, and maybe a catalog and a lineage tool watching over all of it. What you actually get is an afternoon lost to dependency conflicts, port clashes, and a Docker Compose file that grows a new bug every time you touch it.

I built odctl to make that afternoon disappear. It is a curated collection of open-source data technologies wrapped in a small CLI, and it is now available on PyPI.

Flink Table API - Declarative Analytics for Supplier Stats in Real Time

Tue, 17 Jun 2025 00:00:00 +0000

In the last post, we explored the fine-grained control of Flink’s DataStream API. Now, we’ll approach the same problem from a higher level of abstraction using the Flink Table API. This post demonstrates how to build a declarative analytics pipeline that processes our continuous stream of Avro-formatted order events. We will define a Table on top of a DataStream and use SQL-like expressions to perform windowed aggregations. This example highlights the power and simplicity of the Table API for analytical tasks and showcases Flink’s seamless integration between its different API layers to handle complex requirements like late data.

Flink DataStream API - Scalable Event Processing for Supplier Stats

Tue, 10 Jun 2025 00:00:00 +0000

Building on our exploration of stream processing, we now transition from Kafka’s native library to Apache Flink, a powerful, general-purpose distributed processing engine. In this post, we’ll dive into Flink’s foundational DataStream API. We will tackle the same supplier statistics problem - analyzing a stream of Avro-formatted order events - but this time using Flink’s robust features for stateful computation. This example will highlight Flink’s sophisticated event-time processing with watermarks and its elegant, built-in mechanisms for handling late-arriving data through side outputs.

Kafka Streams - Lightweight Real-Time Processing for Supplier Stats

Tue, 03 Jun 2025 00:00:00 +0000

A Kotlin application analyses a continuous stream of Avro-formatted order events, calculates supplier statistics in tumbling windows, and handles late-arriving data. This post shifts our focus from basic Kafka clients to real-time stream processing with Kafka Streams. The example demonstrates the power of Kafka Streams for building lightweight, yet robust, stream processing applications directly within your Kafka ecosystem, using event-time processing and custom logic.

Kafka Clients with Avro - Schema Registry and Order Events

Tue, 27 May 2025 00:00:00 +0000

A Kafka producer that generates mock order data and a consumer that processes these orders are built with Kotlin, Apache Avro for data serialization, and Gradle for build management. This example highlights best practices such as schema management with Avro, robust error handling, and graceful shutdown, providing a solid foundation for your own Kafka-based projects. We’ll dive into the build configuration, the Avro schema definition, utility functions for Kafka administration, and the core logic of both the producer and consumer applications.

Kafka Clients with JSON - Producing and Consuming Order Events

Tue, 20 May 2025 00:00:00 +0000

A Kafka producer application generates and sends order data, and a Kafka consumer application receives and processes those orders. This post explores that Kotlin-based Kafka project, detailing the construction and operation of both applications. We’ll go through each component, from build configuration to message handling, to understand how they work together in an event-driven system.