Kpow on Jaehyeon Kimhttps://jaehyeon.me/tags/kpow/Recent content in Kpow on Jaehyeon KimHugo -- gohugo.ioenCopyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.Mon, 23 Feb 2026 00:00:00 +0000Productionizing an Online Product Recommender using Event Driven Architecturehttps://jaehyeon.me/blog/2026-02-23-productionize-recommender-with-eda/Mon, 23 Feb 2026 00:00:00 +0000https://jaehyeon.me/blog/2026-02-23-productionize-recommender-with-eda/In Part 1, we built a contextual bandit prototype using Python and Mab2Rec. While effective for testing algorithms locally, a monolithic script cannot handle production scale. Real-world recommendation systems require low-latency inference for users and high-throughput training for model updates.

This post demonstrates how to decouple these concerns using an event-driven architecture with Apache Flink, Kafka, and Redis.

]]>Flink Table API - Declarative Analytics for Supplier Stats in Real Timehttps://jaehyeon.me/blog/2025-06-17-kotlin-getting-started-flink-table/Tue, 17 Jun 2025 00:00:00 +0000https://jaehyeon.me/blog/2025-06-17-kotlin-getting-started-flink-table/In the last post, we explored the fine-grained control of Flink’s DataStream API. Now, we’ll approach the same problem from a higher level of abstraction using the Flink Table API. This post demonstrates how to build a declarative analytics pipeline that processes our continuous stream of Avro-formatted order events. We will define a Table on top of a DataStream and use SQL-like expressions to perform windowed aggregations. This example highlights the power and simplicity of the Table API for analytical tasks and showcases Flink’s seamless integration between its different API layers to handle complex requirements like late data.

]]>Flink DataStream API - Scalable Event Processing for Supplier Statshttps://jaehyeon.me/blog/2025-06-10-kotlin-getting-started-flink-datastream/Tue, 10 Jun 2025 00:00:00 +0000https://jaehyeon.me/blog/2025-06-10-kotlin-getting-started-flink-datastream/Building on our exploration of stream processing, we now transition from Kafka’s native library to Apache Flink, a powerful, general-purpose distributed processing engine. In this post, we’ll dive into Flink’s foundational DataStream API. We will tackle the same supplier statistics problem - analyzing a stream of Avro-formatted order events - but this time using Flink’s robust features for stateful computation. This example will highlight Flink’s sophisticated event-time processing with watermarks and its elegant, built-in mechanisms for handling late-arriving data through side outputs.

]]>Kafka Streams - Lightweight Real-Time Processing for Supplier Statshttps://jaehyeon.me/blog/2025-06-03-kotlin-getting-started-kafka-streams/Tue, 03 Jun 2025 00:00:00 +0000https://jaehyeon.me/blog/2025-06-03-kotlin-getting-started-kafka-streams/In this post, we shift our focus from basic Kafka clients to real-time stream processing with Kafka Streams. We’ll explore a Kotlin application designed to analyze a continuous stream of Avro-formatted order events, calculate supplier statistics in tumbling windows, and intelligently handle late-arriving data. This example demonstrates the power of Kafka Streams for building lightweight, yet robust, stream processing applications directly within your Kafka ecosystem, leveraging event-time processing and custom logic.

]]>Kafka Clients with Avro - Schema Registry and Order Eventshttps://jaehyeon.me/blog/2025-05-27-kotlin-getting-started-kafka-avro-clients/Tue, 27 May 2025 00:00:00 +0000https://jaehyeon.me/blog/2025-05-27-kotlin-getting-started-kafka-avro-clients/In this post, we’ll explore a practical example of building Kafka client applications using Kotlin, Apache Avro for data serialization, and Gradle for build management. We’ll walk through the setup of a Kafka producer that generates mock order data and a consumer that processes these orders. This example highlights best practices such as schema management with Avro, robust error handling, and graceful shutdown, providing a solid foundation for your own Kafka-based projects. We’ll dive into the build configuration, the Avro schema definition, utility functions for Kafka administration, and the core logic of both the producer and consumer applications.

]]>Kafka Clients with JSON - Producing and Consuming Order Eventshttps://jaehyeon.me/blog/2025-05-20-kotlin-getting-started-kafka-json-clients/Tue, 20 May 2025 00:00:00 +0000https://jaehyeon.me/blog/2025-05-20-kotlin-getting-started-kafka-json-clients/<p>This post explores a Kotlin-based Kafka project, meticulously detailing the construction and operation of both a Kafka producer application, responsible for generating and sending order data, and a Kafka consumer application, designed to receive and process these orders. We&rsquo;ll delve into each component, from build configuration to message handling, to understand how they work together in an event-driven system.</p>Real Time Streaming with Kafka and Flink - Lab 6 Consume data from Kafka using Lambdahttps://jaehyeon.me/blog/2023-12-14-real-time-streaming-with-kafka-and-flink-7/Thu, 14 Dec 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-12-14-real-time-streaming-with-kafka-and-flink-7/Amazon MSK can be configured as an event source of a Lambda function. Lambda internally polls for new messages from the event source and then synchronously invokes the target Lambda function. With this feature, we can develop a Kafka consumer application in serverless environment where developers can focus on application logic. In this lab, we will discuss how to create a Kafka consumer using a Lambda function. Introduction Lab 1 Produce data to Kafka using Lambda Lab 2 Write data to Kafka from S3 using Flink Lab 3 Transform and write data to S3 from Kafka using Flink Lab 4 Clean, Aggregate, and Enrich Events with Flink Lab 5 Write data to DynamoDB using Kafka Connect Lab 6 Consume data from Kafka using Lambda (this post) Architecture Fake taxi ride data is sent to a Kafka topic by the Kafka producer application that is discussed in Lab 1.Real Time Streaming with Kafka and Flink - Lab 5 Write data to DynamoDB using Kafka Connecthttps://jaehyeon.me/blog/2023-11-30-real-time-streaming-with-kafka-and-flink-6/Thu, 30 Nov 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-11-30-real-time-streaming-with-kafka-and-flink-6/[UPDATE 2025-10-01] Bitnami&rsquo;s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka.Real Time Streaming with Kafka and Flink - Lab 3 Transform and write data to S3 from Kafka using Flinkhttps://jaehyeon.me/blog/2023-11-16-real-time-streaming-with-kafka-and-flink-4/Thu, 16 Nov 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-11-16-real-time-streaming-with-kafka-and-flink-4/[UPDATE 2025-10-01] Bitnami&rsquo;s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 In this lab, we will create a Pyflink application that exports Kafka topic messages into a S3 bucket. The app enriches the records by adding a new column using a user defined function and writes them via the FileSystem SQL connector.Real Time Streaming with Kafka and Flink - Lab 2 Write data to Kafka from S3 using Flinkhttps://jaehyeon.me/blog/2023-11-09-real-time-streaming-with-kafka-and-flink-3/Thu, 09 Nov 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-11-09-real-time-streaming-with-kafka-and-flink-3/In this lab, we will create a Pyflink application that reads records from S3 and sends them into a Kafka topic. A custom pipeline Jar file will be created as the Kafka cluster is authenticated by IAM, and it will be demonstrated how to execute the app in a Flink cluster deployed on Docker as well as locally as a typical Python app. We can assume the S3 data is static metadata that needs to be joined into another stream, and this exercise can be useful for data enrichment.Kafka Connect for AWS Services Integration - Part 5 Deploy Aiven OpenSearch Sink Connectorhttps://jaehyeon.me/blog/2023-10-30-kafka-connect-for-aws-part-5/Mon, 30 Oct 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-10-30-kafka-connect-for-aws-part-5/In the previous post, we discussed how to develop a data pipeline from Apache Kafka into OpenSearch locally using Docker. The pipeline will be deployed on AWS using Amazon MSK, Amazon MSK Connect and Amazon OpenSearch Service using Terraform in this post. First the infrastructure will be deployed that covers a Virtual Private Cloud (VPC), Virtual Private Network (VPN) server, MSK Cluster and OpenSearch domain. Then Kafka source and sink connectors will be deployed on MSK Connect, followed by performing quick data analysis.Real Time Streaming with Kafka and Flink - Lab 1 Produce data to Kafka using Lambdahttps://jaehyeon.me/blog/2023-10-26-real-time-streaming-with-kafka-and-flink-2/Thu, 26 Oct 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-10-26-real-time-streaming-with-kafka-and-flink-2/In this lab, we will create a Kafka producer application using AWS Lambda, which sends fake taxi ride data into a Kafka topic on Amazon MSK. A configurable number of the producer Lambda function will be invoked by an Amazon EventBridge schedule rule. In this way we are able to generate test data concurrently based on the desired volume of messages. Introduction Lab 1 Produce data to Kafka using Lambda (this post) Lab 2 Write data to Kafka from S3 using Flink Lab 3 Transform and write data to S3 from Kafka using Flink Lab 4 Clean, Aggregate, and Enrich Events with Flink Lab 5 Write data to DynamoDB using Kafka Connect Lab 6 Consume data from Kafka using Lambda [Update 2023-11-06] Initially I planned to deploy Pyflink applications on Amazon Managed Service for Apache Flink, but I changed the plan to use a local Flink cluster deployed on Docker.Kafka Connect for AWS Services Integration - Part 4 Develop Aiven OpenSearch Sink Connectorhttps://jaehyeon.me/blog/2023-10-23-kafka-connect-for-aws-part-4/Mon, 23 Oct 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-10-23-kafka-connect-for-aws-part-4/[UPDATE 2025-10-01] Bitnami&rsquo;s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 OpenSearch is a popular search and analytics engine and its use cases cover log analytics, real-time application monitoring, and clickstream analysis. OpenSearch can be deployed on its own or via Amazon OpenSearch Service.Kafka, Flink and DynamoDB for Real Time Fraud Detection - Part 2 Deployment via AWS Managed Flinkhttps://jaehyeon.me/blog/2023-09-14-fraud-detection-part-2/Thu, 14 Sep 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-09-14-fraud-detection-part-2/This series aims to help those who are new to Apache Flink and Amazon Managed Service for Apache Flink by re-implementing a simple fraud detection application that is discussed in an AWS workshop titled AWS Kafka and DynamoDB for real time fraud detection. In part 1, I demonstrated how to develop the application locally, and the app will be deployed via Amazon Managed Service for Apache Flink in this post.Getting Started with Pyflink on AWS - Part 3 AWS Managed Flink and MSKhttps://jaehyeon.me/blog/2023-09-04-getting-started-with-pyflink-on-aws-part-3/Mon, 04 Sep 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-09-04-getting-started-with-pyflink-on-aws-part-3/In this series of posts, we discuss a Flink (Pyflink) application that reads/writes from/to Kafka topics. In the previous posts, I demonstrated a Pyflink app that targets a local Kafka cluster as well as a Kafka cluster on Amazon MSK. The app was executed in a virtual environment as well as in a local Flink cluster for improved monitoring. In this post, the app will be deployed via Amazon Managed Service for Apache Flink, which is the easiest option to run Flink applications on AWS.Getting Started with Pyflink on AWS - Part 2 Local Flink and MSKhttps://jaehyeon.me/blog/2023-08-28-getting-started-with-pyflink-on-aws-part-2/Mon, 28 Aug 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-08-28-getting-started-with-pyflink-on-aws-part-2/In this series of posts, we discuss a Flink (Pyflink) application that reads/writes from/to Kafka topics. In part 1, an app that targets a local Kafka cluster was created. In this post, we will update the app by connecting a Kafka cluster on Amazon MSK. The Kafka cluster is authenticated by IAM and the app has additional jar dependency. As Amazon Managed Service for Apache Flink does not allow you to specify multiple pipeline jar files, we have to build a custom Uber Jar that combines multiple jar files.Getting Started with Pyflink on AWS - Part 1 Local Flink and Local Kafkahttps://jaehyeon.me/blog/2023-08-17-getting-started-with-pyflink-on-aws-part-1/Thu, 17 Aug 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-08-17-getting-started-with-pyflink-on-aws-part-1/[UPDATE 2025-10-01] Bitnami&rsquo;s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 Apache Flink is an open-source, unified stream-processing and batch-processing framework. Its core is a distributed streaming data-flow engine that you can use to run real-time stream processing on high-throughput data sources. Currently, it is widely used to build applications for fraud/anomaly detection, rule-based alerting, business process monitoring, and continuous ETL to name a few.Kafka, Flink and DynamoDB for Real Time Fraud Detection - Part 1 Local Developmenthttps://jaehyeon.me/blog/2023-08-10-fraud-detection-part-1/Thu, 10 Aug 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-08-10-fraud-detection-part-1/[UPDATE 2025-10-01] Bitnami&rsquo;s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 Apache Flink is an open-source, unified stream-processing and batch-processing framework. Its core is a distributed streaming data-flow engine that you can use to run real-time stream processing on high-throughput data sources. Currently, it is widely used to build applications for fraud/anomaly detection, rule-based alerting, business process monitoring, and continuous ETL to name a few.Kafka Connect for AWS Services Integration - Part 3 Deploy Camel DynamoDB Sink Connectorhttps://jaehyeon.me/blog/2023-07-03-kafka-connect-for-aws-part-3/Mon, 03 Jul 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-07-03-kafka-connect-for-aws-part-3/As part of investigating how to utilize Kafka Connect effectively for AWS services integration, I demonstrated how to develop the Camel DynamoDB sink connector using Docker in Part 2. Fake order data was generated using the MSK Data Generator source connector, and the sink connector was configured to consume the topic messages to ingest them into a DynamoDB table. In this post, I will illustrate how to deploy the data ingestion applications using Amazon MSK and MSK Connect.Kafka Development with Docker - Part 7 Producer and Consumer with Glue Schema Registryhttps://jaehyeon.me/blog/2023-06-22-kafka-development-with-docker-part-7/Thu, 22 Jun 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-06-22-kafka-development-with-docker-part-7/[UPDATE 2025-10-01] Bitnami&rsquo;s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 In Part 4, we developed Kafka producer and consumer applications using the kafka-python package. The Kafka messages are serialized as Json, but are not associated with a schema as there was not an integrated schema registry.Kafka Development with Docker - Part 6 Kafka Connect with Glue Schema Registryhttps://jaehyeon.me/blog/2023-06-15-kafka-development-with-docker-part-6/Thu, 15 Jun 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-06-15-kafka-development-with-docker-part-6/[UPDATE 2025-10-01] Bitnami&rsquo;s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 In Part 3, we developed a data ingestion pipeline with fake online order data using Kafka Connect source and sink connectors. Schemas are not enabled on both of them as there was not an integrated schema registry.Kafka Development with Docker - Part 2 Management Apphttps://jaehyeon.me/blog/2023-05-18-kafka-development-with-docker-part-2/Thu, 18 May 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-05-18-kafka-development-with-docker-part-2/In the previous post, I illustrated how to create a topic and to produce/consume messages using the command utilities provided by Apache Kafka. It is not convenient, however, for example, when you consume serialised messages where their schemas are stored in a schema registry. Also, the utilities don&rsquo;t support to browse or manage related resources such as connectors and schemas. Therefore, a Kafka management app can be a good companion for development, which helps monitor and manage resources on an easy-to-use user interface.Integrate Glue Schema Registry with Your Python Kafka Apphttps://jaehyeon.me/blog/2023-04-12-integrate-glue-schema-registry/Wed, 12 Apr 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-04-12-integrate-glue-schema-registry/[UPDATE 2025-10-01] Bitnami&rsquo;s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 As Kafka producer and consumer apps are decoupled, they operate on Kafka topics rather than communicating with each other directly. As described in the Confluent document, Schema Registry provides a centralized repository for managing and validating schemas for topic message data, and for serialization and deserialization of the data over the network.