desflow on Jaehyeon Kim

Defining Data-Streaming Simulations in YAML, Without Writing Python

Tue, 06 Oct 2026 00:00:00 +0000

A pipeline is easier to test when its input behaves like production: orders arrive at random, queues grow, a machine slows down. A simulation can produce that data, but when the simulation is a Python program, every change is a code change. Someone who only wants to double an arrival rate has to read and edit code, and a reviewer has to check that nothing else moved. A file that holds only settings is easier to read, to compare in a pull request and to run in CI.

dynamic-des 0.16.0 adds that file. A YAML blueprint describes the whole simulation, and the ddes command runs it. Each of the seven examples in the repository now has a YAML version in plain YAML. The documentation was rebuilt around the three ways to write a simulation.

Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL

Fri, 02 Oct 2026 00:00:00 +0000

A mobile game produces a score every time a player finishes a round, and players expect the leaderboards to move as they play. In this post, a simulation plays the game and sends each score to Kafka. Four Flink SQL jobs keep four top 10 leaderboards up to date in PostgreSQL, and a web dashboard shows them as they change. Everything runs on your own machine.

Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect

Thu, 01 Oct 2026 00:00:00 +0000

Most databases change all day: users sign up, orders are placed, and each order moves from one status to the next. Change data capture (CDC) turns those changes into a stream of events that other systems can read as they happen. In this post, a simulated online shop writes to PostgreSQL in real time, Debezium streams every change to Kafka, and a sink connector saves the changes as files in object storage. Everything runs on your own machine.

Data Streaming and Machine Learning Projects That Run on Your Laptop

Wed, 30 Sep 2026 00:00:00 +0000

Reading about a data tool only goes so far. The hard part is getting several tools to run together and seeing how data moves between them. Benchtop is a collection of hands-on demos for data engineering, stream processing, machine learning, AI engineering and MLOps. Each demo runs locally on the odctl stack, and works from a fresh clone of the repository, with nothing to install beforehand but Docker and uv. Each one builds a working system from open-source tools and explains the ideas behind it, so you learn a tool by running it rather than only reading about it.

Learning MLOps with a Feature Store: A New Series

Mon, 28 Sep 2026 00:00:00 +0000

Learning MLOps one tool at a time leaves out the hard part: how features, training and serving fit together as one system. This series learns it by building complete systems. I follow Jim Dowling’s Building Machine Learning Systems with a Feature Store and rebuild its hands-on projects with open-source tools, running on my laptop.

Building an Agentic Analytics System over an Iceberg Lakehouse

Sat, 18 Jul 2026 00:00:00 +0000

Generative AI has made conversational analytics feel within reach, yet direct text-to-SQL systems remain hard to operate reliably. A database schema tells you the tables, columns, and types, but it says nothing about which datasets are canonical, which join paths are approved, how a governed metric is calculated, or what the business actually means by “revenue” or “active customer”. Ask a language model to infer all of that from raw tables and, sooner or later, it will confidently invent an answer.

I built a proof of concept to explore a more disciplined approach: put a semantic layer between the model and the data, and let the agent reason over governed business concepts instead of guessing at raw schemas. It runs entirely locally on an open-source stack, and it happens to tie together two of my other projects along the way.

Dynamic DES: A Declarative API with Postgres and Redis Connectors

Fri, 17 Jul 2026 00:00:00 +0000

Simulated data is most useful when the code that produces it is easy to read, and when it reaches the same places as production data, such as a PostgreSQL table or a Redis stream. Since I wrote about Dynamic DES v0.8.1 and its data lake support, the project has grown in both directions: a cleaner way to write a simulation, and more places to send and receive data.

Here is what changed on the way to v0.11.1.

One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES

Mon, 25 May 2026 00:00:00 +0000

Training a machine learning model on simulated data is straightforward until you try to deploy it. The disconnect usually happens at the pipeline level: training requires massive, historical batch data (like Parquet files in an S3 bucket), but production inference requires real-time, event-driven streams (like Kafka or Redis). Maintaining two separate simulation codebases, one for generating training data and another for streaming live events, introduces friction, schema mismatches, and duplicated engineering effort.

Building an Event-Driven Hybrid Digital Twin with dynamic-des

Wed, 29 Apr 2026 00:00:00 +0000

Asynchronous Gap A true Hybrid Digital Twin does more than just mirror reality. It actively forecasts the future by running a simulation against live operational states. We established this in Part 1. If you have ever tried to build one of these systems from scratch, you immediately hit a fundamental architectural clash. Standard simulation clocks (like those in traditional SimPy implementations) are logically synchronous and not designed to handle high-frequency asynchronous I/O without explicit decoupling.

Why Digital Twins Are Rewiring Industry 4.0

Wed, 22 Apr 2026 00:00:00 +0000

A simulation and a digital twin are often treated as the same thing, but they are built differently. A traditional simulation runs offline on assumed inputs. An operational digital twin stays aligned with one real system as it changes. This post walks through the layers that separate the two, and an event-driven hybrid that combines them. Beyond CAD Models There is a project by Dassault Systèmes called the Living Heart that illustrates the trajectory of this technology.

Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning

Tue, 21 Apr 2026 00:00:00 +0000

Overview Imagine using a rolling pin to flatten out a thick piece of dough. A Hot Strip Mill does the exact same thing, but with glowing red-hot steel slabs (often heated over 1000°C) and massive mechanical rollers. The steel is passed through a series of these rollers, crushing it down from a thick block into a long, thin sheet. Calculating the exact Rolling Force required to crush the steel is critical.