Benchtop on Jaehyeon Kim

Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL

Fri, 02 Oct 2026 00:00:00 +0000

A mobile game produces a score every time a player finishes a round, and players expect the leaderboards to move as they play. In this post, a simulation plays the game and sends each score to Kafka. Four Flink SQL jobs keep four top 10 leaderboards up to date in PostgreSQL, and a web dashboard shows them as they change. Everything runs on your own machine.

Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect

Thu, 01 Oct 2026 00:00:00 +0000

Most databases change all day: users sign up, orders are placed, and each order moves from one status to the next. Change data capture (CDC) turns those changes into a stream of events that other systems can read as they happen. In this post, a simulated online shop writes to PostgreSQL in real time, Debezium streams every change to Kafka, and a sink connector saves the changes as files in object storage. Everything runs on your own machine.

Data Streaming and Machine Learning Projects That Run on Your Laptop

Wed, 30 Sep 2026 00:00:00 +0000

Reading about a data tool only goes so far. The hard part is getting several tools to run together and seeing how data moves between them. Benchtop is a collection of small, hands-on demos for data engineering, stream processing, machine learning, AI engineering and MLOps. Each demo runs locally on the odctl stack, and works from a fresh clone of the repository, with nothing to install beforehand but Docker and uv. Each one builds a working system from open-source tools and explains the ideas behind it, so you learn a tool by running it rather than only reading about it.

Productionizing an Online Product Recommender using Event Driven Architecture

Mon, 23 Feb 2026 00:00:00 +0000

Real-world recommendation systems require low-latency inference for users and high-throughput training for model updates. A monolithic script cannot handle production scale, while it is effective for testing algorithms locally. In Part 1, we built a contextual bandit prototype using Python and Mab2Rec.

This post demonstrates how to decouple these concerns using an event-driven architecture with Apache Flink, Kafka, and Valkey.

Prototyping an Online Product Recommender in Python

Tue, 27 Jan 2026 00:00:00 +0000

Overview Traditional recommendation approaches such as Collaborative Filtering remain widely adopted, yet they come with notable constraints. They are particularly vulnerable to the cold-start problem, where new users lack sufficient interaction history, and they depend heavily on long-term behavioral data. As a result, they frequently overlook real-time contextual signals, including time of day, device type, location, or session intent. This can prevent them from capturing situational preferences, such as someone preferring coffee in the morning but pizza in the evening.

Flink Table API - Declarative Analytics for Supplier Stats in Real Time

Tue, 17 Jun 2025 00:00:00 +0000

In the last post, we explored the fine-grained control of Flink’s DataStream API. Now, we’ll approach the same problem from a higher level of abstraction using the Flink Table API. This post demonstrates how to build a declarative analytics pipeline that processes our continuous stream of Avro-formatted order events. We will define a Table on top of a DataStream and use SQL-like expressions to perform windowed aggregations. This example highlights the power and simplicity of the Table API for analytical tasks and showcases Flink’s seamless integration between its different API layers to handle complex requirements like late data.

Flink DataStream API - Scalable Event Processing for Supplier Stats

Tue, 10 Jun 2025 00:00:00 +0000

Building on our exploration of stream processing, we now transition from Kafka’s native library to Apache Flink, a powerful, general-purpose distributed processing engine. In this post, we’ll dive into Flink’s foundational DataStream API. We will tackle the same supplier statistics problem - analyzing a stream of Avro-formatted order events - but this time using Flink’s robust features for stateful computation. This example will highlight Flink’s sophisticated event-time processing with watermarks and its elegant, built-in mechanisms for handling late-arriving data through side outputs.

Kafka Streams - Lightweight Real-Time Processing for Supplier Stats

Tue, 03 Jun 2025 00:00:00 +0000

A Kotlin application analyses a continuous stream of Avro-formatted order events, calculates supplier statistics in tumbling windows, and handles late-arriving data. This post shifts our focus from basic Kafka clients to real-time stream processing with Kafka Streams. The example demonstrates the power of Kafka Streams for building lightweight, yet robust, stream processing applications directly within your Kafka ecosystem, using event-time processing and custom logic.

Kafka Clients with Avro - Schema Registry and Order Events

Tue, 27 May 2025 00:00:00 +0000

A Kafka producer that generates mock order data and a consumer that processes these orders are built with Kotlin, Apache Avro for data serialization, and Gradle for build management. This example highlights best practices such as schema management with Avro, robust error handling, and graceful shutdown, providing a solid foundation for your own Kafka-based projects. We’ll dive into the build configuration, the Avro schema definition, utility functions for Kafka administration, and the core logic of both the producer and consumer applications.

Kafka Clients with JSON - Producing and Consuming Order Events

Tue, 20 May 2025 00:00:00 +0000

A Kafka producer application generates and sends order data, and a Kafka consumer application receives and processes those orders. This post explores that Kotlin-based Kafka project, detailing the construction and operation of both applications. We’ll go through each component, from build configuration to message handling, to understand how they work together in an event-driven system.

Next.js Dashboard - Realtime Dashboard with FastAPI, Streamlit and Next.js Part 3

Tue, 04 Mar 2025 00:00:00 +0000

A real-time monitoring dashboard connects to the WebSocket server from Part 1 to continuously fetch and visualize key metrics such as order counts, sales data, and revenue by traffic source and country. With interactive bar charts and dynamic metrics, users can monitor sales trends and other critical business KPIs in real-time. In this post, we build it using Next.js, a React framework that supports server-side rendering, static site generation, and full-stack capabilities with built-in performance optimizations. It is similar to the Streamlit app we developed in Part 2.

Streamlit Dashboard - Realtime Dashboard with FastAPI, Streamlit and Next.js Part 2

Tue, 25 Feb 2025 00:00:00 +0000

A real-time monitoring dashboard is developed using Streamlit, an open-source Python framework that allows data scientists and AI/ML engineers to create interactive data apps. The app connects to the WebSocket server we developed in Part 1 and continuously fetches data to visualize key metrics such as order counts, sales data, and revenue by traffic source and country. With interactive bar charts and dynamic metrics, users can monitor sales trends and other important business KPIs in real-time.

Realtime Dashboard with FastAPI, Streamlit and Next.js - Part 1 Data Producer

Tue, 18 Feb 2025 00:00:00 +0000

A data generating app is created with Python, and it ingests the theLook eCommerce data continuously into a PostgreSQL database. A WebSocket server, built by FastAPI, periodically queries the data to serve its clients. In this series, we develop real-time monitoring dashboard applications, and this post walks through the data generation app and backend API. The monitoring dashboards will be developed using Streamlit and Next.js, with Apache ECharts for visualization. They will be discussed in later posts.