Jaehyeon Kim
Jaehyeon Kim

  • Blog
    • Categories

      List of categories.

    • Tags

      List of tags.

    • Series

      List of series.

    • Archives

  • Projects
  • Slides

/

  • Github Linkedin RSS

  • Font Size
  • Palette
  • Mode
  1. Home
  2. Tags
  3. AWS

Real Time Streaming with Kafka and Flink - Introduction

Real Time Streaming with Kafka and Flink - Introduction
October 5, 20236 min read Data-Streaming Real Time Streaming With Kafka and FlinkAmazon MSKApache FlinkApache KafkaAWSPyFlink

Kafka and PyFlink replace Amazon Kinesis in a real time analytics app from an AWS workshop. The original and the new architecture are compared.

Read More: Real Time Streaming with Kafka and Flink - Introduction

Deployment via AWS Managed Flink - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 2

Deployment via AWS Managed Flink - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 2
September 14, 202317 min read Data-Streaming Kafka, Flink and DynamoDB for Real Time Fraud DetectionAmazon DynamoDBApache FlinkApache KafkaAWSKpowPython

Deploy the Kafka, Flink and DynamoDB fraud detection app to Amazon Managed Service for Apache Flink, after developing it locally on Docker.

Read More: Deployment via AWS Managed Flink - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 2

Getting Started with PyFlink on AWS - Part 3 AWS Managed Flink and MSK

Getting Started with PyFlink on AWS - Part 3 AWS Managed Flink and MSK
September 4, 202314 min read Data-Streaming Getting Started With PyFlink on AWSAmazon MSKApache KafkaAWSDockerKpowPyFlinkPython

Deploy a PyFlink app that reads and writes Kafka topics on Amazon MSK to Amazon Managed Service for Apache Flink, the managed Flink runtime.

Read More: Getting Started with PyFlink on AWS - Part 3 AWS Managed Flink and MSK

Getting Started with PyFlink on AWS - Part 2 Local Flink and MSK

Getting Started with PyFlink on AWS - Part 2 Local Flink and MSK
August 28, 202320 min read Data-Streaming Getting Started With PyFlink on AWSAmazon MSKApache FlinkApache KafkaAWSKpowPyFlinkPython

Connect a PyFlink app to an IAM authenticated MSK cluster, building a custom uber jar because Amazon Managed Service for Apache Flink takes only one jar.

Read More: Getting Started with PyFlink on AWS - Part 2 Local Flink and MSK

Local Development - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 1

Local Development - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 1
August 10, 202317 min read Data-Streaming Kafka, Flink and DynamoDB for Real Time Fraud DetectionAmazon DynamoDBApache FlinkApache KafkaAWSDockerKpowPython

Develop a fraud detection app locally on Docker with Kafka, Flink and DynamoDB, re-implementing a solution taken from an AWS workshop.

Read More: Local Development - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 1

Deploy Camel DynamoDB Sink Connector - Kafka Connect for AWS Services Integration Part 3

Deploy Camel DynamoDB Sink Connector - Kafka Connect for AWS Services Integration Part 3
July 3, 202314 min read Data Integration Data-Streaming Kafka Connect for AWS Services IntegrationAmazon DynamoDBAmazon MSKApache CamelApache KafkaAWSKafka ConnectKpow

Deploy the Camel DynamoDB sink connector and its data generator source on Amazon MSK and MSK Connect, moving the local pipeline onto AWS.

Read More: Deploy Camel DynamoDB Sink Connector - Kafka Connect for AWS Services Integration Part 3

Producer and Consumer with Glue Schema Registry - Kafka Development with Docker Part 7

Producer and Consumer with Glue Schema Registry - Kafka Development with Docker Part 7
June 22, 202313 min read Data-Streaming Kafka Development With DockerApache KafkaAWSAWS Glue Schema RegistryDockerKpowPython

Schema registry support added to kafka-python producer and consumer apps, which serialise and deserialise through AWS Glue Schema Registry.

Read More: Producer and Consumer with Glue Schema Registry - Kafka Development with Docker Part 7

Kafka Development with Docker - Part 6 Kafka Connect with Glue Schema Registry

Kafka Development with Docker - Part 6 Kafka Connect with Glue Schema Registry
June 15, 202313 min read Data Integration Data-Streaming Kafka Development With DockerApache KafkaAWSAWS Glue Schema RegistryDockerKafka ConnectKpow

Schema registry support added to a Kafka Connect ingestion pipeline, so the source and sink connectors serialise through AWS Glue Schema Registry.

Read More: Kafka Development with Docker - Part 6 Kafka Connect with Glue Schema Registry

Kafka Development with Docker - Part 5 Glue Schema Registry

Kafka Development with Docker - Part 5 Glue Schema Registry
June 8, 20237 min read Data-Streaming Kafka Development With DockerApache KafkaAWSAWS Glue Schema RegistryKafka ConnectSchema Registry

Build the Glue Schema Registry client library and use it to manage and enforce schemas in Kafka producer and consumer applications.

Read More: Kafka Development with Docker - Part 5 Glue Schema Registry

Develop Camel DynamoDB Sink Connector - Kafka Connect for AWS Services Integration Part 2

Develop Camel DynamoDB Sink Connector - Kafka Connect for AWS Services Integration Part 2
June 4, 202314 min read Data-Streaming Kafka Connect for AWS Services IntegrationAmazon DynamoDBApache CamelApache KafkaAWSDockerKafka Connect

Develop the Camel DynamoDB sink connector on Docker, consuming fake order records from a Kafka topic and writing them into a DynamoDB table.

Read More: Develop Camel DynamoDB Sink Connector - Kafka Connect for AWS Services Integration Part 2
  • ««
  • «
  • 1
  • 2
  • 3
  • 4
  • 5
  • »
  • »»
Profile
Jaehyeon Kim
Jaehyeon Kim
Data Engineer | Data Streaming | Powering ML & AI in Real Time
Taxonomies
Data Streaming 70 Data Engineering 41 Development 28 Data Analysis 17 Data Integration 12 Open Source 11 Machine Learning 7 Kubernetes 6 Security 5 Data Architecture 3 Data Processing 3 Big Data 2 System Architecture 2 Web Development 2
Python 75 Apache Kafka 71 AWS 50 Docker 50 Apache Flink 36 Apache Beam 17 Apache Spark 16 Kafka Connect 16 Amazon MSK 14 AWS Lambda 14 dbt 13 Amazon EMR 11 odctl 11 dynamic-des 10 PostgreSQL 9 Kubernetes 8 PyFlink 8 Debezium 7 Kotlin 7 Apache Airflow 6 Change Data Capture (CDC) 6 Kafka UI 6 Amazon DynamoDB 5 Apache Iceberg 5 Discrete Event Simulation 5 PySpark 5 Amazon API Gateway 4 Amazon Athena 4 AWS Glue 4 AWS Glue Schema Registry 4 BigQuery 4 Digital Twin 4 FastAPI 4 Minikube 4 SimPy 4 Amazon EKS 3 Amazon QuickSight 3 Amazon S3 3 Apache Hudi 3 EMR on EKS 3 ALL 142
Kafka Development with Docker 11 Apache Beam Python Examples 10 Real Time Streaming with Kafka and Flink 7 dbt Pizza Shop Demo 6 Apache Beam Local Development with Python 5 Building Real-Time Digital Twins with dynamic-des 5 dbt for Effective Data Transformation on AWS 5 Getting Started with Real-Time Streaming in Kotlin 5 Kafka Connect for AWS Services Integration 5 Serverless Data Product 4 Data Lake Demo Using Change Data Capture 3 Getting Started with PyFlink on AWS 3 Kafka Development on Kubernetes 3 Parallel processing on single machine 3 Realtime Dashboard with FastAPI, Streamlit and Next.js 3 API development with R 2 dbt Guide for Production 2 Deploy Python Stream Processing App on Kubernetes 2 Download Stock Data 2 From Prototype to Production: Real-Time Product Recommendation with Contextual Bandits 2 ALL 25
2026 15 2025 13 2024 29 2023 39 2022 15 2021 7 2020 1 2019 5 2018 2 2017 6 2016 6 2015 15 2014 5
Posts
  • Live Game Leaderboards with Kafka, Flink SQL and a Discrete-Event Simulation
    Live Game Leaderboards with Kafka, Flink SQL and a Discrete-Event Simulation
    October 2, 2026
  • Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    October 1, 2026
  • Introducing Benchtop: Data Streaming and Machine Learning Projects You Can Run Locally
    Introducing Benchtop: Data Streaming and Machine Learning Projects You Can Run Locally
    September 30, 2026
  • Learning MLOps with a Feature Store: A New Series
    Learning MLOps with a Feature Store: A New Series
    September 28, 2026
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • Introducing odctl: One CLI for a Local Open Data Stack
    Introducing odctl: One CLI for a Local Open Data Stack
    July 16, 2026
  • One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    May 25, 2026
  • Building an Event-Driven Hybrid Digital Twin with dynamic-des
    Building an Event-Driven Hybrid Digital Twin with dynamic-des
    April 29, 2026
  • Why Digital Twins Are Rewiring Industry 4.0
    Why Digital Twins Are Rewiring Industry 4.0
    April 22, 2026
  • Live Game Leaderboards with Kafka, Flink SQL and a Discrete-Event Simulation
    Live Game Leaderboards with Kafka, Flink SQL and a Discrete-Event Simulation
    October 2, 2026
  • Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    October 1, 2026
  • Introducing Benchtop: Data Streaming and Machine Learning Projects You Can Run Locally
    Introducing Benchtop: Data Streaming and Machine Learning Projects You Can Run Locally
    September 30, 2026
  • Learning MLOps with a Feature Store: A New Series
    Learning MLOps with a Feature Store: A New Series
    September 28, 2026
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • Introducing odctl: One CLI for a Local Open Data Stack
    Introducing odctl: One CLI for a Local Open Data Stack
    July 16, 2026
  • One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    May 25, 2026
  • Current London 2026: Building End-to-End Data Lineage
    Current London 2026: Building End-to-End Data Lineage
    May 22, 2026
  • Building an Event-Driven Hybrid Digital Twin with dynamic-des
    Building an Event-Driven Hybrid Digital Twin with dynamic-des
    April 29, 2026
Actions
Go back Reload Copy URL

Jaehyeon Kim

Data Engineer | Data Streaming | Powering ML & AI in Real Time

Copyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.