Jaehyeon Kim
Jaehyeon Kim

  • Blog
    • Categories

      List of categories.

    • Tags

      List of tags.

    • Series

      List of series.

    • Archives

  • Projects
  • Slides

/

  • Github Linkedin RSS

  • Font Size
  • Palette
  • Mode
  1. Home
  2. Tags
  3. Python

Transform and write data to S3 from Kafka using Flink - Real Time Streaming with Kafka and Flink Lab 3

featured.png
November 16, 202316 min read Data-Streaming Real Time Streaming With Kafka and FlinkAmazon S3Apache FlinkApache KafkaAWSKpowPyFlinkPython

Export Kafka topic messages to S3 with PyFlink, enriching records through a user defined function and querying them on Athena through a Glue table.

Read More

Produce data to Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 1

featured.png
October 26, 202314 min read Data-Streaming Real Time Streaming With Kafka and FlinkAmazon EventBridgeAmazon MSKApache KafkaAWSAWS LambdaKpowPython

Produce fake taxi ride data into a Kafka topic on Amazon MSK from an AWS Lambda function invoked repeatedly by an EventBridge schedule rule.

Read More

Building Apache Flink Applications in Python

featured.png
October 19, 20236 min read Data-StreamingApache FlinkApache KafkaDockerPyFlinkPython

Three Flink DataStream applications from a Confluent Java course converted to PyFlink, from ingesting source data to calculating usage statistics.

Read More

Deployment via AWS Managed Flink - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 2

featured.png
September 14, 202317 min read Data-Streaming Kafka, Flink and DynamoDB for Real Time Fraud DetectionAmazon DynamoDBApache FlinkApache KafkaAWSKpowPython

Deploy the Kafka, Flink and DynamoDB fraud detection app to Amazon Managed Service for Apache Flink, after developing it locally on Docker.

Read More

Getting Started with PyFlink on AWS - Part 3 AWS Managed Flink and MSK

featured.png
September 4, 202314 min read Data-Streaming Getting Started With PyFlink on AWSAmazon MSKApache KafkaAWSDockerKpowPyFlinkPython

Deploy a PyFlink app that reads and writes Kafka topics on Amazon MSK to Amazon Managed Service for Apache Flink, the managed Flink runtime.

Read More

Getting Started with PyFlink on AWS - Part 2 Local Flink and MSK

featured.png
August 28, 202320 min read Data-Streaming Getting Started With PyFlink on AWSAmazon MSKApache FlinkApache KafkaAWSKpowPyFlinkPython

Connect a PyFlink app to an IAM authenticated MSK cluster, building a custom uber jar because Amazon Managed Service for Apache Flink takes only one jar.

Read More

Getting Started with PyFlink on AWS - Part 1 Local Flink and Local Kafka

featured.png
August 17, 202317 min read Data-Streaming Getting Started With PyFlink on AWSApache FlinkApache KafkaDockerKpowPyFlinkPython

Develop a PyFlink app against a Kafka cluster on Docker, running it both in a virtual environment and in a local Flink cluster for better monitoring.

Read More

Local Development - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 1

featured.png
August 10, 202317 min read Data-Streaming Kafka, Flink and DynamoDB for Real Time Fraud DetectionAmazon DynamoDBApache FlinkApache KafkaAWSDockerKpowPython

Develop a fraud detection app locally on Docker with Kafka, Flink and DynamoDB, re-implementing a solution taken from an AWS workshop.

Read More

Kafka Development with Docker - Part 11 Kafka Authorization

featured.png
July 20, 202314 min read Data-Streaming Security Kafka Development With DockerApache KafkaDockerPython

Control user access to Kafka resources with authorization rules, keeping SASL for client authentication, with Java and Python client examples.

Read More

Kafka Development with Docker - Part 10 SASL Authentication

featured.png
July 13, 202311 min read Data-Streaming Security Kafka Development With DockerApache KafkaDockerPythonSASL

Authenticate Kafka clients with SASL instead of client certificates, with Java and Python client examples of the SASL mechanisms.

Read More
  • ««
  • «
  • 3
  • 4
  • 5
  • 6
  • 7
  • »
  • »»
Profile
Jaehyeon Kim
Jaehyeon Kim
Data Engineer | Data Streaming Enthusiast | Powering ML & AI in Real Time
Taxonomies
Data Streaming 70 Data Engineering 38 Development 28 Data Analysis 17 Data Integration 12 Open Source 7 Kubernetes 6 Machine Learning 5 Security 5 Data Architecture 3 Data Processing 3 Big Data 2 System Architecture 2 Web Development 2
Python 75 Apache Kafka 68 AWS 50 Docker 50 R 37 Apache Flink 34 Kpow 22 Apache Beam 17 Apache Spark 16 Kafka Connect 15 Amazon MSK 14 AWS Lambda 14 dbt 13 Amazon EMR 11 Kubernetes 8 PyFlink 8 Kotlin 7 PostgreSQL 7 Change Data Capture (CDC) 6 Debezium 6 dynamic-des 6 Amazon DynamoDB 5 Apache Airflow 5 Discrete Event Simulation 5 Factor House Local 5 PySpark 5 Amazon API Gateway 4 Amazon Athena 4 Apache Iceberg 4 AWS Glue 4 AWS Glue Schema Registry 4 BigQuery 4 Digital Twin 4 FastAPI 4 Minikube 4 R Shiny 4 RServe 4 SimPy 4 Amazon EKS 3 Amazon QuickSight 3 ALL 137
Kafka Development with Docker 11 Apache Beam Python Examples 10 Real Time Streaming with Kafka and Flink 7 dbt Pizza Shop Demo 6 Tree Based Methods in R 6 Apache Beam Local Development with Python 5 Building Real-Time Digital Twins with dynamic-des 5 dbt for Effective Data Transformation on AWS 5 Getting Started with Real-Time Streaming in Kotlin 5 Kafka Connect for AWS Services Integration 5 Serverless Data Product 4 Data Lake Demo Using Change Data Capture 3 Getting Started with PyFlink on AWS 3 Kafka Development on Kubernetes 3 Parallel processing on single machine 3 Realtime Dashboard with FastAPI, Streamlit and Next.js 3 API development with R 2 dbt Guide for Production 2 Deploy Python Stream Processing App on Kubernetes 2 Download Stock Data 2 ALL 24
2026 11 2025 13 2024 29 2023 39 2022 15 2021 7 2020 1 2019 5 2018 2 2017 6 2016 6 2015 15 2014 5
Posts
  • featured.png
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • featured.png
    Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • featured.png
    Introducing odctl: One CLI for a Local Open Data Stack
    July 16, 2026
  • featured.png
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    May 25, 2026
  • featured.png
    Building an Event-Driven Hybrid Digital Twin with dynamic-des
    April 29, 2026
  • featured.png
    Why Digital Twins Are Rewiring Industry 4.0
    April 22, 2026
  • featured.png
    Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    April 21, 2026
  • featured.png
    Slides as Code: Integrating Reveal.js into my Hugo Blog
    March 9, 2026
  • featured.gif
    Productionizing an Online Product Recommender using Event Driven Architecture
    February 23, 2026
  • featured.png
    Stream Processing with Flink in Kotlin
    December 10, 2025
  • featured.png
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • featured.png
    Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • featured.png
    Introducing odctl: One CLI for a Local Open Data Stack
    July 16, 2026
  • featured.png
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    May 25, 2026
  • featured.jfif
    Current London 2026: Building End-to-End Data Lineage
    May 22, 2026
  • featured.png
    Building an Event-Driven Hybrid Digital Twin with dynamic-des
    April 29, 2026
  • featured.png
    Why Digital Twins Are Rewiring Industry 4.0
    April 22, 2026
  • featured.png
    Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    April 21, 2026
  • featured.png
    Slides as Code: Integrating Reveal.js into my Hugo Blog
    March 9, 2026
  • featured.gif
    Productionizing an Online Product Recommender using Event Driven Architecture
    February 23, 2026
Actions
Go back Reload Copy URL

Jaehyeon Kim

Data Engineer | Data Streaming Enthusiast | Powering ML & AI in Real Time

Copyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.