Data Engineering on Jaehyeon Kim

odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command

Mon, 12 Oct 2026 00:00:00 +0000

Trying a modern data tool on your own machine is rarely about the tool. Writing Iceberg tables from Flink needs a catalog, the catalog needs a database and object storage, Avro on Kafka needs a schema registry, and every piece needs the right connector jars, ports and network settings. Most of an afternoon goes into making them talk to each other before you write a line of your own code.

odctl removes that work. It is a command-line tool that runs a set of open-source data and machine learning services on your laptop with Docker, already connected to each other. You say what you want, for example Kafka and Flink, and it starts them together with everything they depend on. It has now reached 1.0, with a documentation site that has a guide for every service.

Defining Data-Streaming Simulations in YAML, Without Writing Python

Tue, 06 Oct 2026 00:00:00 +0000

A pipeline is easier to test when its input behaves like production: orders arrive at random, queues grow, a machine slows down. A simulation can produce that data, but when the simulation is a Python program, every change is a code change. Someone who only wants to double an arrival rate has to read and edit code, and a reviewer has to check that nothing else moved. A file that holds only settings is easier to read, to compare in a pull request and to run in CI.

dynamic-des 0.16.0 adds that file. A YAML blueprint describes the whole simulation, and the ddes command runs it. Each of the seven examples in the repository now has a YAML version in plain YAML. The documentation was rebuilt around the three ways to write a simulation.

Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL

Fri, 02 Oct 2026 00:00:00 +0000

A mobile game produces a score every time a player finishes a round, and players expect the leaderboards to move as they play. In this post, a simulation plays the game and sends each score to Kafka. Four Flink SQL jobs keep four top 10 leaderboards up to date in PostgreSQL, and a web dashboard shows them as they change. Everything runs on your own machine.

Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect

Thu, 01 Oct 2026 00:00:00 +0000

Most databases change all day: users sign up, orders are placed, and each order moves from one status to the next. Change data capture (CDC) turns those changes into a stream of events that other systems can read as they happen. In this post, a simulated online shop writes to PostgreSQL in real time, Debezium streams every change to Kafka, and a sink connector saves the changes as files in object storage. Everything runs on your own machine.

Data Streaming and Machine Learning Projects That Run on Your Laptop

Wed, 30 Sep 2026 00:00:00 +0000

Reading about a data tool only goes so far. The hard part is getting several tools to run together and seeing how data moves between them. Benchtop is a collection of small, hands-on demos for data engineering, stream processing, machine learning, AI engineering and MLOps. Each demo runs locally on the odctl stack, and works from a fresh clone of the repository, with nothing to install beforehand but Docker and uv. Each one builds a working system from open-source tools and explains the ideas behind it, so you learn a tool by running it rather than only reading about it.

Building an Agentic Analytics System over an Iceberg Lakehouse

Sat, 18 Jul 2026 00:00:00 +0000

Generative AI has made conversational analytics feel within reach, yet direct text-to-SQL systems remain hard to operate reliably. A database schema tells you the tables, columns, and types, but it says nothing about which datasets are canonical, which join paths are approved, how a governed metric is calculated, or what the business actually means by “revenue” or “active customer”. Ask a language model to infer all of that from raw tables and, sooner or later, it will confidently invent an answer.

I built a proof of concept to explore a more disciplined approach: put a semantic layer between the model and the data, and let the agent reason over governed business concepts instead of guessing at raw schemas. It runs entirely locally on an open-source stack, and it happens to tie together two of my other projects along the way.

Dynamic DES: A Declarative API with Postgres and Redis Connectors

Fri, 17 Jul 2026 00:00:00 +0000

Simulated data is most useful when the code that produces it is easy to read, and when it reaches the same places as production data, such as a PostgreSQL table or a Redis stream. Since I wrote about Dynamic DES v0.8.1 and its data lake support, the project has grown in both directions: a cleaner way to write a simulation, and more places to send and receive data.

Here is what changed on the way to v0.11.1.

Running Kafka, Flink, Spark, Trino and Iceberg Locally with One CLI

Thu, 16 Jul 2026 00:00:00 +0000

Anyone who has tried to stand up a realistic data platform on their laptop knows the pain. You want Kafka talking to Flink, Spark writing to Iceberg, Trino querying the result, and maybe a catalog and a lineage tool watching over all of it. What you actually get is an afternoon lost to dependency conflicts, port clashes, and a Docker Compose file that grows a new bug every time you touch it.

I built odctl to make that afternoon disappear. It is a curated collection of open-source data technologies wrapped in a small CLI, and it is now available on PyPI.

One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES

Mon, 25 May 2026 00:00:00 +0000

Training a machine learning model on simulated data is straightforward until you try to deploy it. The disconnect usually happens at the pipeline level: training requires massive, historical batch data (like Parquet files in an S3 bucket), but production inference requires real-time, event-driven streams (like Kafka or Redis). Maintaining two separate simulation codebases, one for generating training data and another for streaming live events, introduces friction, schema mismatches, and duplicated engineering effort.

Current London 2026: Building End-to-End Data Lineage

Fri, 22 May 2026 00:00:00 +0000

When a number in a report looks wrong, the first question is where it came from, and the next is what else a change will break. Answering both needs data lineage: a record of which jobs read and wrote which datasets, across every system the data passes through. At Current London 2026 I presented a session on this, Building End-to-End Data Lineage with Kafka, Flink, and Spark, which captures that metadata across parallel pipelines using OpenLineage.

Building an Event-Driven Hybrid Digital Twin with dynamic-des

Wed, 29 Apr 2026 00:00:00 +0000

Asynchronous Gap A true Hybrid Digital Twin does more than just mirror reality. It actively forecasts the future by running a simulation against live operational states. We established this in Part 1. If you have ever tried to build one of these systems from scratch, you immediately hit a fundamental architectural clash. Standard simulation clocks (like those in traditional SimPy implementations) are logically synchronous and not designed to handle high-frequency asynchronous I/O without explicit decoupling.

Self-service Data Platform via a Multi-tenant SQL Gateway

Thu, 17 Jul 2025 00:00:00 +0000

In the modern data stack, providing direct access to powerful engines like Apache Spark and Flink is a double-edged sword. While it empowers users, it often leads to chaos: resource contention from “noisy neighbors,” inconsistent security enforcement, and operational fragility. The core problem is the lack of a robust control plane between users and the raw compute power. The solution, therefore, isn’t to take power away from users, but to manage it through an intelligent intermediary.

Meet the Streamhouse Trio - Paimon, Fluss, and Iceberg for Unified Data Architectures

Tue, 06 May 2025 00:00:00 +0000

The world of data is converging. The traditional divide between batch processing for historical analytics and stream processing for real-time insights is becoming increasingly blurry. Businesses demand architectures that handle both seamlessly. Enter the “Streamhouse” - an evolution of the Lakehouse concept, designed with streaming as a first-class citizen.

Today, we’ll introduce three key open-source technologies shaping this space: Apache Paimon™, Fluss, and Apache Iceberg. While each has unique strengths, their true power lies in how they can be integrated to build robust, flexible, and performant data platforms.

Guide to Running DBT in Production

Fri, 13 Sep 2024 00:00:00 +0000

Deploying a dbt project in multiple environments is the focus here, and we walk through the entire CI/CD process step-by-step. In the previous post, we started discussing a continuous integration/continuous delivery (CI/CD) process of a dbt project by introducing two GitHub Actions workflows - slim-ci and deploy. The former is triggered when a pull request is created to the main branch, and it builds only modified models and its first-order children in a ci dataset, followed by performing tests on them. The second workflow gets triggered once a pull request is merged. Beginning with running unit tests, it packages the dbt project as a Docker container and publishes to Artifact Registry.

DBT CI/CD Demo with BigQuery and GitHub Actions

Thu, 05 Sep 2024 00:00:00 +0000

Continuous integration (CI) is the process of ensuring new code integrates with the larger code base, and it puts a great emphasis on testing automation to check that the application is not broken whenever new commits are integrated into the main branch. Continuous delivery (CD) is an extension of continuous integration since it automatically deploys all code changes to a testing and/or production environment after the build stage. CI/CD helps development teams avoid bugs and code failures while maintaining a continuous cycle of software development and updates. In this post, we discuss how to set up a CI/CD pipeline for a data build tool (dbt) project using GitHub Actions where BigQuery is used as the target data warehouse.

Apache Beam Local Development with Python - Part 5 Testing Pipelines

Thu, 09 May 2024 00:00:00 +0000

We developed batch and streaming pipelines in Part 2 and Part 4. Often it is faster and simpler to identify and fix bugs on the pipeline code by performing local unit testing. Moreover, especially when it comes to creating a streaming pipeline, unit testing cases can facilitate development further by using TestStream as it allows us to advance watermarks or processing time according to different scenarios. In this post, we discuss how to perform unit testing of the batch and streaming pipelines that we developed earlier.

Data Build Tool (dbt) Pizza Shop Demo - Part 6 ETL on Amazon Athena via Airflow

Thu, 14 Mar 2024 00:00:00 +0000

An ETL process is set up on the pizza shop dbt project using Apache Airflow. In Part 5, we developed a dbt project that that targets Apache Iceberg where transformations are performed on Amazon Athena. Two dimension tables that keep product and user records are created as Type 2 slowly changing dimension (SCD Type 2) tables, and one transactional fact table is built to keep pizza orders. To improve query performance, the fact table is denormalized to pre-join records from the dimension tables using the array and struct data types.

Data Build Tool (dbt) Pizza Shop Demo - Part 5 Modelling on Amazon Athena

Thu, 07 Mar 2024 00:00:00 +0000

The data is modelled by SCD type 2 dimension tables and one transactional fact table. In Part 1 and Part 3, we developed data build tool (dbt) projects that target PostgreSQL and BigQuery using fictional pizza shop data. While the order records should be joined with dimension tables to get complete details for PostgreSQL, the fact table is denormalized using nested and repeated fields to improve query performance for BigQuery.

Data Build Tool (dbt) Pizza Shop Demo - Part 4 ETL on BigQuery via Airflow

Thu, 22 Feb 2024 00:00:00 +0000

An ETL process is set up with Apache Airflow on a dbt project that targets Google BigQuery with fictional pizza shop data. We developed that project in Part 3. Two dimension tables that keep product and user records are created as Type 2 slowly changing dimension (SCD Type 2) tables, and one transactional fact table is built to keep pizza orders. The fact table is denormalized using nested and repeated fields for improving query performance.

Data Build Tool (dbt) Pizza Shop Demo - Part 3 Modelling on BigQuery

Thu, 08 Feb 2024 00:00:00 +0000

We create a new dbt project that targets Google BigQuery in this post. While the dimension tables are kept by the same SCD type 2 approach, the fact table is denormalized using nested and repeated fields, which potentially can improve query performance by pre-joining corresponding dimension records. In this series, we discuss practical examples of data warehouse and lakehouse development where data transformation is performed by the data build tool (dbt) and ETL is managed by Apache Airflow.

Data Build Tool (dbt) Pizza Shop Demo - Part 2 ETL on PostgreSQL via Airflow

Thu, 25 Jan 2024 00:00:00 +0000

We set up an ETL process on a dbt project using Apache Airflow in this post. In this series of posts, we discuss data warehouse/lakehouse examples using data build tool (dbt) including ETL orchestration with Apache Airflow. In Part 1, we developed a dbt project on PostgreSQL with fictional pizza shop data. Two dimension tables that keep product and user records are created as Type 2 slowly changing dimension (SCD Type 2) tables, and one transactional fact table is built to keep pizza orders.

Data Build Tool (dbt) Pizza Shop Demo - Part 1 Modelling on PostgreSQL

Thu, 18 Jan 2024 00:00:00 +0000

The data build tool (dbt) is a popular data transformation tool for data warehouse development. Moreover, it can be used for data lakehouse development thanks to open table formats such as Apache Iceberg, Apache Hudi and Delta Lake. dbt supports key AWS analytics services and I wrote a series of posts that discuss how to utilise dbt with Redshift, Glue, EMR on EC2, EMR on EKS, and Athena. Those posts focus on platform integration, however, they do not show realistic ETL scenarios.

Setup Local Development Environment for Apache Flink and Spark Using EMR Container Images

Thu, 07 Dec 2023 00:00:00 +0000

Apache Flink became generally available for Amazon EMR on EKS from the EMR 6.15.0 releases, and we are able to pull the Flink (as well as Spark) container images from the ECR Public Gallery. As both of them can be integrated with the Glue Data Catalog, it can be particularly useful if we develop real time data ingestion/processing via Flink and build analytical queries using Spark (or any other tools or services that can access to the Glue Data Catalog).

Data Build Tool (dbt) for Effective Data Transformation on AWS - Part 5 Athena

Tue, 06 Dec 2022 00:00:00 +0000

Data models on Amazon Athena can be built with the data build tool (dbt) and the dbt-athena adapter. In the previous posts, we discussed benefits of a common data transformation tool and the potential of dbt to cover a wide range of data projects from data warehousing to data lake to data lakehouse. Demo data projects that target Redshift Serverless, Glue, EMR on EC2 and EMR on EKS are illustrated as well.

EMR on EKS - Data Build Tool (dbt) for Effective Data Transformation on AWS Part 4

Tue, 01 Nov 2022 00:00:00 +0000

Data models on Amazon EMR on EKS can be built with the data build tool (dbt) through a Spark Thrift server that runs on Kubernetes. In the previous posts, we discussed benefits of a common data transformation tool and the potential of dbt to cover a wide range of data projects from data warehousing to data lake to data lakehouse. Demo data projects that target Redshift Serverless, Glue and EMR on EC2 are illustrated as well.

EMR on EC2 - Data Build Tool (dbt) for Effective Data Transformation on AWS Part 3

Wed, 19 Oct 2022 00:00:00 +0000

Data models on Amazon EMR on EC2 can be built with the data build tool (dbt) through a Spark Thrift server started on the cluster. In the previous posts, we discussed benefits of a common data transformation tool and the potential of dbt to cover a wide range of data projects from data warehousing to data lake to data lakehouse. Demo data projects that target Redshift Serverless and Glue are illustrated as well.

Data Build Tool (dbt) for Effective Data Transformation on AWS - Part 2 Glue

Sun, 09 Oct 2022 00:00:00 +0000

Data models on AWS Glue can be built with the data build tool (dbt) through Glue interactive sessions and the dbt-glue adapter. In part 1, we discussed benefits of a common data transformation tool and the potential of dbt to cover a wide range of data projects from data warehousing to data lake to data lakehouse. A demo data project that targets Redshift Serverless is illustrated as well. In part 2 of the dbt on AWS series, we discuss data transformation pipelines using dbt on AWS Glue.

Data Build Tool (dbt) for Effective Data Transformation on AWS - Part 1 Redshift

Wed, 28 Sep 2022 00:00:00 +0000

Data models in Amazon Redshift Serverless can be built with the data build tool (dbt), which also supports Glue, EMR and Athena. In part 1 of the dbt on AWS series, we discuss data transformation pipelines using dbt on Redshift Serverless. Subsets of IMDb data are used as source and data models are developed in multiple layers according to the dbt best practices. Part 1 Redshift (this post) Part 2 Glue Part 3 EMR on EC2 Part 4 EMR on EKS Part 5 Athena Motivation In our experience delivering data solutions for our customers, we have observed a desire to move away from a centralised team function, responsible for the data collection, analysis and reporting, towards shifting this responsibility to an organisation’s lines of business (LOB) teams.

Develop and Test Apache Spark Apps for EMR Remotely Using Visual Studio Code

Wed, 07 Sep 2022 00:00:00 +0000

When we develop a Spark application on EMR, we can use docker for local development or notebooks via EMR Studio (or EMR Notebooks). However, the local development option is not viable if the size of data is large. Also, I am not a fan of notebooks as it is not possible to utilise the features my editor supports such as syntax highlighting, autocomplete and code formatting. Moreover, it is not possible to organise code into modules and to perform unit testing properly with that option.

Manage EMR on EKS with Terraform

Fri, 26 Aug 2022 00:00:00 +0000

Amazon EMR on EKS is a deployment option for Amazon EMR that allows you to automate the provisioning and management of open-source big data frameworks on EKS. While eksctl is popular for working with Amazon EKS clusters, it has limitations when it comes to building infrastructure that integrates multiple AWS services. Also, it is not straightforward to update EKS cluster resources incrementally with it. On the other hand Terraform can be an effective tool for managing infrastructure that includes not only EKS and EMR virtual clusters but also other AWS resources.

Revisit AWS Lambda Invoke Function Operator of Apache Airflow

Sat, 06 Aug 2022 00:00:00 +0000

Apache Airflow is a popular workflow management platform. A wide range of AWS services are integrated with the platform by Amazon AWS Operators. AWS Lambda is one of the integrated services, and it can be used to develop workflows efficiently. The current Lambda Operator, however, just invokes a Lambda function, and it can fail to report the invocation result of a function correctly and to record the exact error message from failure.

Data Warehousing ETL Demo with Apache Iceberg on EMR Local Environment

Sun, 26 Jun 2022 00:00:00 +0000

Unlike traditional Data Lake, new table formats (Iceberg, Hudi and Delta Lake) support features that can be used to apply data warehousing patterns, which can bring a way to be rescued from Data Swamp. In this post, we’ll discuss how to implement ETL using retail analytics data. It has two dimension data (user and product) and a single fact data (order). The dimension data sets have different ETL strategies depending on whether to track historical changes.

Develop and Test Apache Spark Apps for EMR Locally Using Docker

Sun, 08 May 2022 00:00:00 +0000

Amazon EMR is a managed service that simplifies running Apache Spark on AWS. It has multiple deployment options that cover EC2, EKS, Outposts and Serverless. For development and testing, EMR Notebooks or EMR Studio can be an option. Both provide a Jupyter Notebook environment and the former is only available for EMR on EC2. There are cases, however, that development (and learning) is performed in a local environment more efficiently. The AWS Glue team understands this demand, and they illustrate how to make use of a custom Docker image for Glue in a recent blog post.

EMR on EKS by Example

Mon, 17 Jan 2022 00:00:00 +0000

EMR on EKS provides a deployment option for Amazon EMR that allows you to automate the provisioning and management of open-source big data frameworks on Amazon EKS. While a wide range of open source big data components are available in EMR on EC2, only Apache Spark is available in EMR on EKS. It is more flexible, however, that applications of different EMR versions can be run in multiple availability zones on either EC2 or Fargate.

Implement Data Lake - Data Lake Demo using Change Data Capture (CDC) on AWS Part 3

Sun, 19 Dec 2021 00:00:00 +0000

As the last part of this series, we’ll build an Apache Hudi DeltaStreamer app on Amazon EMR and use the resulting Hudi table with Amazon Athena and Amazon Quicksight to build a dashboard. In the previous post, we created a VPC that has private and public subnets in 2 availability zones in order to build and deploy the data lake solution on AWS. NAT instances are created to forward outbound traffic to the internet and a VPN bastion host is set up to facilitate deployment.

Data Lake Demo using Change Data Capture (CDC) on AWS - Part 2 Implement CDC

Sun, 12 Dec 2021 00:00:00 +0000

We build the CDC part of the data lake solution on AWS using Amazon MSK and MSK Connect. In the previous post, we discussed a data lake solution where data ingestion is performed using change data capture (CDC) and the output files are upserted to an Apache Hudi table. Being registered to Glue Data Catalog, it can be used for ad-hoc queries and report/dashboard creation. The Northwind database is used as the source database and, following the transactional outbox pattern, order-related changes are _upserted _to an outbox table by triggers.

Data Lake Demo using Change Data Capture (CDC) on AWS - Part 1 Local Development

Sun, 05 Dec 2021 00:00:00 +0000

Change data capture (CDC) is a proven data integration pattern that has a wide range of applications. Among those, data replication to data lakes is a good use case in data engineering. Coupled with best-in-breed data lake formats such as Apache Hudi, we can build an efficient data replication solution. This is the first post of the data lake demo series. Over time, we’ll build a data lake that uses CDC.

Local Development of AWS Glue 3.0 and Later

Sun, 14 Nov 2021 00:00:00 +0000

In an earlier post, I demonstrated how to set up a local development environment for AWS Glue 1.0 and 2.0 using a docker image that is published by the AWS Glue team and the Visual Studio Code Remote – Containers extension. Recently AWS Glue 3.0 was released, but a docker image for this version is not published. In this post, I’ll illustrate how to create a development environment for AWS Glue 3.

AWS Glue Local Development with Docker and Visual Studio Code

Fri, 20 Aug 2021 00:00:00 +0000

As described in the product page, AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. For development, a development endpoint is recommended, but it can be costly, inconvenient or unavailable (for Glue 2.0). The AWS Glue team published a Docker image that includes the AWS Glue binaries and all the dependencies packaged together. After inspecting it, I find some modifications are necessary in order to build a development environment on it.

Thoughts on Apache Airflow AWS Lambda Operator

Mon, 13 Apr 2020 00:00:00 +0000

Apache Airflow is a popular open-source workflow management platform. Typically tasks run remotely by Celery workers for scalability. In AWS, however, scalability can also be achieved using serverless computing services in a simpler way. For example, the ECS Operator allows to run dockerized tasks and, with the Fargate launch type, they can run in a serverless environment. The ECS Operator alone is not sufficient because it can take up to several minutes to pull a Docker image and to set up network interface (for the case of Fargate launch type).

Boost SparkR with Hive

Sat, 30 Apr 2016 00:00:00 +0000

While SparkR is in active development, it is yet to fully support Spark’s key libraries such as MLlib and Spark Streaming. Even, as a data processing engine, this R API is still limited as it is not possible to manipulate RDDs directly but only via Spark SQL/DataFrame API. In the previous post, it is demonstrated how to start SparkR in local and cluster mode. As can be checked in the API doc, SparkR rebuilds many existing R functions to work with Spark DataFrame and notably it borrows some functions from the dplyr package.

Quick Start SparkR in Local and Cluster Mode

Wed, 02 Mar 2016 00:00:00 +0000

A BitBucket repository is introduced, which is a R project that includes Spark 1.6.0 Pre-built for Hadoop 2.0 and later and hadoop-common 2.2.0 - the latter is necessary if it is tested on Windows. In the previous post, a Spark cluster is set up using 2 VirtualBox Ubuntu guests. While this is a viable option for many, it is not always for others. For those who find setting-up such a cluster is not convenient, there’s still another option, which is relying on the local mode of Spark.

Spark Cluster Setup on VirtualBox

Mon, 22 Feb 2016 00:00:00 +0000

We discuss how to set up a Spark cluser between 2 Ubuntu guests. Firstly it begins with machine preparation. Once a machine is baked, its image file (VDI) is be copied for the second one. Then how to launch a cluster by standalone mode is discussed. Let’s get started. Status, September 2026. The Spark 1.6.0 download link used below no longer resolves, so the cluster cannot be built by following these commands as written.