Jaehyeon Kim
Jaehyeon Kim

  • Blog
    • Categories

      List of categories.

    • Tags

      List of tags.

    • Series

      List of series.

    • Archives

  • Projects
  • Slides

/

  • Github Linkedin RSS

  • Font Size
  • Palette
  • Mode
  1. Home
  2. blogs

Current London 2026: Building End-to-End Data Lineage

Current London 2026: Building End-to-End Data Lineage
May 22, 20263 min read Data-EngineeringApache FlinkApache KafkaApache SparkData LineageOpenLineage

Tracking data across Kafka, Flink and Spark pipelines with OpenLineage, to show where each dataset came from and what reads it. Slides from my Current London 2026 session.

Read More: Current London 2026: Building End-to-End Data Lineage

Building an Event-Driven Hybrid Digital Twin with dynamic-des

Building an Event-Driven Hybrid Digital Twin with dynamic-des
April 29, 20269 min read Data-Engineering Open Source System-Architecture Building Real-Time Digital Twins With Dynamic-DesApache KafkaDigital TwinDiscrete Event SimulationDynamic-DesIoTPythonSimPy

Keeping a simulation in step with live data: change a running SimPy model's parameters from Kafka and stream its events out, so it acts as a digital twin.

Read More: Building an Event-Driven Hybrid Digital Twin with dynamic-des

Why Digital Twins Are Rewiring Industry 4.0

Why Digital Twins Are Rewiring Industry 4.0
April 22, 20265 min read Open Source System-Architecture Building Real-Time Digital Twins With Dynamic-DesDigital TwinDiscrete Event SimulationDynamic-DesIoTPythonSimPy

Why a simulation is not a digital twin: the layers that separate an offline model, an operational twin and an event-driven hybrid of the two.

Read More: Why Digital Twins Are Rewiring Industry 4.0

Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning

Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
April 21, 20269 min read Data-Streaming Machine-Learning Open Source Building Real-Time Digital Twins With Dynamic-DesApache FlinkApache KafkaDigital TwinDiscrete Event SimulationDynamic-DesKotlinOnline Machine LearningPython

An Apache Flink and Kotlin streaming architecture where online machine learning detects concept drift from machinery wear, controlled by a shadow mode router.

Read More: Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning

Slides as Code: Integrating Reveal.js into my Hugo Blog

Slides as Code: Integrating Reveal.js into my Hugo Blog
March 9, 20265 min read Web DevelopmentHugoJavaScriptMarkdownMarpReveal.js

Reveal.js slides written in Markdown and built by Hugo alongside blog posts, with code walkthroughs, fragments, MathJax and Mermaid diagrams.

Read More: Slides as Code: Integrating Reveal.js into my Hugo Blog

Productionizing an Online Product Recommender using Event Driven Architecture

Productionizing an Online Product Recommender using Event Driven Architecture
February 23, 202610 min read Machine-Learning From Prototype to Production: Real-Time Product Recommendation With Contextual BanditsApache FlinkApache KafkaBenchtopContextual BanditsOnline Machine LearningOpen Data StackPythonRecommender SystemValkey

Apache Flink, Kafka and Valkey turn a contextual bandit recommender into an event driven service with stateful training and low latency inference.

Read More: Productionizing an Online Product Recommender using Event Driven Architecture

Prototyping an Online Product Recommender in Python

Prototyping an Online Product Recommender in Python
January 27, 202620 min read Machine-Learning From Prototype to Production: Real-Time Product Recommendation With Contextual BanditsBenchtopContextual BanditsMab2RecMABWiserOnline Machine LearningPythonRecommender SystemReinforcement Learning

A Python prototype of a contextual multi-armed bandit recommender that simulates user behaviour and validates the algorithm against cold-start users.

Read More: Prototyping an Online Product Recommender in Python

Stream Processing with Flink in Kotlin

Stream Processing with Flink in Kotlin
December 10, 20255 min read Data-StreamingApache FlinkGradleKotlinKtor

Examples from the book Stream Processing with Apache Flink ported to Kotlin, updated to current Flink APIs and built with Gradle after PyFlink lacked features.

Read More: Stream Processing with Flink in Kotlin

Guide to Building Integrated Web Applications with FastAPI and NiceGUI

Guide to Building Integrated Web Applications with FastAPI and NiceGUI
November 19, 202511 min read DevelopmentDockerFastAPINiceGUIPostgreSQLPydanticPythonSoftware ArchitectureSQLModel

FastAPI and NiceGUI served from one Python process give a backend API and a web UI in a single codebase, compared here with React and with Streamlit.

Read More: Guide to Building Integrated Web Applications with FastAPI and NiceGUI

Self-service Data Platform via a Multi-tenant SQL Gateway

Self-service Data Platform via a Multi-tenant SQL Gateway
July 17, 202515 min read Big Data Data Architecture Data-Engineering Data-StreamingApache FlinkApache KyuubiApache RangerApache SparkMarquezOpenLineageSQL GatewayTrino

Apache Kyuubi as a multi-tenant SQL gateway that provisions on-demand Spark, Flink and Trino engines, giving self-service analytics with central governance.

Read More: Self-service Data Platform via a Multi-tenant SQL Gateway
  • ««
  • «
  • 1
  • 2
  • 3
  • 4
  • 5
  • »
  • »»
Profile
Jaehyeon Kim
Jaehyeon Kim
Data Engineer | Data Streaming | Powering ML & AI in Real Time
Taxonomies
Data Streaming 70 Data Engineering 43 Development 28 Data Analysis 17 Open Source 13 Data Integration 12 Machine Learning 7 Kubernetes 6 Security 5 Data Architecture 3 Data Processing 3 Big Data 2 System Architecture 2 Web Development 2
Python 77 Apache Kafka 73 AWS 50 Docker 50 Apache Flink 39 Apache Beam 17 Apache Spark 17 Kafka Connect 16 Amazon MSK 14 AWS Lambda 14 Benchtop 13 dbt 13 odctl 12 Amazon EMR 11 dynamic-des 11 PostgreSQL 10 Kubernetes 8 PyFlink 8 Apache Iceberg 7 Change Data Capture (CDC) 7 Debezium 7 Kotlin 7 Apache Airflow 6 Discrete Event Simulation 6 Kafka UI 6 Amazon DynamoDB 5 PySpark 5 SimPy 5 Amazon API Gateway 4 Amazon Athena 4 AWS Glue 4 AWS Glue Schema Registry 4 BigQuery 4 Digital Twin 4 FastAPI 4 Minikube 4 Amazon EKS 3 Amazon QuickSight 3 Amazon S3 3 Apache Hudi 3 ALL 143
Kafka Development with Docker 11 Apache Beam Python Examples 10 Real Time Streaming with Kafka and Flink 7 Building Real-Time Digital Twins with dynamic-des 6 dbt Pizza Shop Demo 6 Apache Beam Local Development with Python 5 dbt for Effective Data Transformation on AWS 5 Getting Started with Real-Time Streaming in Kotlin 5 Kafka Connect for AWS Services Integration 5 Serverless Data Product 4 Data Lake Demo Using Change Data Capture 3 Getting Started with PyFlink on AWS 3 Kafka Development on Kubernetes 3 Parallel processing on single machine 3 Realtime Dashboard with FastAPI, Streamlit and Next.js 3 API development with R 2 dbt Guide for Production 2 Deploy Python Stream Processing App on Kubernetes 2 Download Stock Data 2 From Prototype to Production: Real-Time Product Recommendation with Contextual Bandits 2 ALL 25
2026 17 2025 13 2024 29 2023 39 2022 15 2021 7 2020 1 2019 5 2018 2 2017 6 2016 6 2015 15 2014 5
Posts
  • odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    October 12, 2026
  • Defining Data-Streaming Simulations in YAML, Without Writing Python
    Defining Data-Streaming Simulations in YAML, Without Writing Python
    October 6, 2026
  • Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    October 2, 2026
  • Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    October 1, 2026
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Current London 2026: Building End-to-End Data Lineage
    Current London 2026: Building End-to-End Data Lineage
    May 22, 2026
  • Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    April 21, 2026
  • Productionizing an Online Product Recommender using Event Driven Architecture
    Productionizing an Online Product Recommender using Event Driven Architecture
    February 23, 2026
  • Stream Processing with Flink in Kotlin
    Stream Processing with Flink in Kotlin
    December 10, 2025
  • Guide to Building Integrated Web Applications with FastAPI and NiceGUI
    Guide to Building Integrated Web Applications with FastAPI and NiceGUI
    November 19, 2025
  • odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    October 12, 2026
  • Defining Data-Streaming Simulations in YAML, Without Writing Python
    Defining Data-Streaming Simulations in YAML, Without Writing Python
    October 6, 2026
  • Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    October 2, 2026
  • Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    October 1, 2026
  • Data Streaming and Machine Learning Projects That Run on Your Laptop
    Data Streaming and Machine Learning Projects That Run on Your Laptop
    September 30, 2026
  • Learning MLOps with a Feature Store: A New Series
    Learning MLOps with a Feature Store: A New Series
    September 28, 2026
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Dynamic DES: A Declarative API with Postgres and Redis Connectors
    Dynamic DES: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • Running Kafka, Flink, Spark, Trino and Iceberg Locally with One CLI
    Running Kafka, Flink, Spark, Trino and Iceberg Locally with One CLI
    July 16, 2026
  • One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES
    May 25, 2026
Actions
Go back Reload Copy URL

Jaehyeon Kim

Data Engineer | Data Streaming | Powering ML & AI in Real Time

Copyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.