Jaehyeon Kim
Jaehyeon Kim

  • Blog
    • Categories

      List of categories.

    • Tags

      List of tags.

    • Series

      List of series.

    • Archives

  • Projects
  • Slides

/

  • Github Linkedin RSS

  • Font Size
  • Palette
  • Mode
  1. Home
  2. Tags
  3. Python

Kafka Development on Kubernetes - Part 2 Producer and Consumer

Kafka Development on Kubernetes - Part 2 Producer and Consumer
January 4, 20248 min read Data-Streaming Kubernetes Kafka Development on KubernetesApache KafkaDockerKubernetesMinikubePythonStrimzi

Send and read Kafka topic records from Python client applications built with the kafka-python package and deployed on Kubernetes.

Read More: Kafka Development on Kubernetes - Part 2 Producer and Consumer

Kafka Development on Kubernetes - Part 1 Cluster Setup

Kafka Development on Kubernetes - Part 1 Cluster Setup
December 21, 20237 min read Data-Streaming Kubernetes Kafka Development on KubernetesApache KafkaDockerKubernetesMinikubePythonStrimzi

Create a Kafka cluster on Kubernetes with Strimzi, the first step in a series on Python clients and Kafka Connect pipelines on Kubernetes.

Read More: Kafka Development on Kubernetes - Part 1 Cluster Setup

Consume data from Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 6

Consume data from Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 6
December 14, 20236 min read Data-Streaming Real Time Streaming With Kafka and FlinkAmazon MSKApache KafkaAWSAWS LambdaKpowPython

Consume Kafka messages with an AWS Lambda function, using Amazon MSK as an event source so that Lambda polls the topic and invokes the function.

Read More: Consume data from Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 6

Clean, Aggregate, and Enrich Events with Flink - Real Time Streaming with Kafka and Flink Lab 4

Clean, Aggregate, and Enrich Events with Flink - Real Time Streaming with Kafka and Flink Lab 4
November 23, 202315 min read Data-Streaming Real Time Streaming With Kafka and FlinkApache FlinkApache KafkaOpenSearchPyFlinkPython

Aggregate taxi ride counts and trip durations by vendor over five second windows with PyFlink, then chart them in an OpenSearch dashboard.

Read More: Clean, Aggregate, and Enrich Events with Flink - Real Time Streaming with Kafka and Flink Lab 4

Transform and write data to S3 from Kafka using Flink - Real Time Streaming with Kafka and Flink Lab 3

Transform and write data to S3 from Kafka using Flink - Real Time Streaming with Kafka and Flink Lab 3
November 16, 202316 min read Data-Streaming Real Time Streaming With Kafka and FlinkAmazon S3Apache FlinkApache KafkaAWSKpowPyFlinkPython

Export Kafka topic messages to S3 with PyFlink, enriching records through a user defined function and querying them on Athena through a Glue table.

Read More: Transform and write data to S3 from Kafka using Flink - Real Time Streaming with Kafka and Flink Lab 3

Produce data to Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 1

Produce data to Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 1
October 26, 202314 min read Data-Streaming Real Time Streaming With Kafka and FlinkAmazon EventBridgeAmazon MSKApache KafkaAWSAWS LambdaKpowPython

Produce fake taxi ride data into a Kafka topic on Amazon MSK from an AWS Lambda function invoked repeatedly by an EventBridge schedule rule.

Read More: Produce data to Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 1

Building Apache Flink Applications in Python

Building Apache Flink Applications in Python
October 19, 20236 min read Data-StreamingApache FlinkApache KafkaDockerPyFlinkPython

Three Flink DataStream applications from a Confluent Java course converted to PyFlink, from ingesting source data to calculating usage statistics.

Read More: Building Apache Flink Applications in Python

Deployment via AWS Managed Flink - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 2

Deployment via AWS Managed Flink - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 2
September 14, 202317 min read Data-Streaming Kafka, Flink and DynamoDB for Real Time Fraud DetectionAmazon DynamoDBApache FlinkApache KafkaAWSKpowPython

Deploy the Kafka, Flink and DynamoDB fraud detection app to Amazon Managed Service for Apache Flink, after developing it locally on Docker.

Read More: Deployment via AWS Managed Flink - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 2

Getting Started with PyFlink on AWS - Part 3 AWS Managed Flink and MSK

Getting Started with PyFlink on AWS - Part 3 AWS Managed Flink and MSK
September 4, 202314 min read Data-Streaming Getting Started With PyFlink on AWSAmazon MSKApache KafkaAWSDockerKpowPyFlinkPython

Deploy a PyFlink app that reads and writes Kafka topics on Amazon MSK to Amazon Managed Service for Apache Flink, the managed Flink runtime.

Read More: Getting Started with PyFlink on AWS - Part 3 AWS Managed Flink and MSK

Getting Started with PyFlink on AWS - Part 2 Local Flink and MSK

Getting Started with PyFlink on AWS - Part 2 Local Flink and MSK
August 28, 202320 min read Data-Streaming Getting Started With PyFlink on AWSAmazon MSKApache FlinkApache KafkaAWSKpowPyFlinkPython

Connect a PyFlink app to an IAM authenticated MSK cluster, building a custom uber jar because Amazon Managed Service for Apache Flink takes only one jar.

Read More: Getting Started with PyFlink on AWS - Part 2 Local Flink and MSK
  • ««
  • «
  • 3
  • 4
  • 5
  • 6
  • 7
  • »
  • »»
Profile
Jaehyeon Kim
Jaehyeon Kim
Data Engineer | Data Streaming | Powering ML & AI in Real Time
Taxonomies
Data Streaming 70 Data Engineering 45 Development 28 Data Analysis 17 Open Source 15 Data Integration 12 Machine Learning 7 Kubernetes 6 Security 5 Data Architecture 3 Data Processing 3 Big Data 2 System Architecture 2 Web Development 2
Python 79 Apache Kafka 73 AWS 50 Docker 50 Apache Flink 39 Apache Beam 17 Apache Spark 17 Kafka Connect 16 Amazon MSK 14 AWS Lambda 14 Benchtop 13 dbt 13 desflow 13 odctl 12 Amazon EMR 11 PostgreSQL 10 Discrete Event Simulation 8 Kubernetes 8 PyFlink 8 Apache Iceberg 7 Change Data Capture (CDC) 7 Debezium 7 Kotlin 7 SimPy 7 Apache Airflow 6 Kafka UI 6 Amazon DynamoDB 5 PySpark 5 Amazon API Gateway 4 Amazon Athena 4 AWS Glue 4 AWS Glue Schema Registry 4 BigQuery 4 Digital Twin 4 FastAPI 4 Minikube 4 Amazon EKS 3 Amazon QuickSight 3 Amazon S3 3 Apache Hudi 3 ALL 144
Kafka Development with Docker 11 Apache Beam Python Examples 10 Building Real-Time Digital Twins with desflow 8 Real Time Streaming with Kafka and Flink 7 dbt Pizza Shop Demo 6 Apache Beam Local Development with Python 5 dbt for Effective Data Transformation on AWS 5 Getting Started with Real-Time Streaming in Kotlin 5 Kafka Connect for AWS Services Integration 5 Serverless Data Product 4 Data Lake Demo Using Change Data Capture 3 Getting Started with PyFlink on AWS 3 Kafka Development on Kubernetes 3 Parallel processing on single machine 3 Realtime Dashboard with FastAPI, Streamlit and Next.js 3 API development with R 2 dbt Guide for Production 2 Deploy Python Stream Processing App on Kubernetes 2 Download Stock Data 2 From Prototype to Production: Real-Time Product Recommendation with Contextual Bandits 2 ALL 25
2026 19 2025 13 2024 29 2023 39 2022 15 2021 7 2020 1 2019 5 2018 2 2017 6 2016 6 2015 15 2014 5
Posts
  • desflow from the Command Line: One Model, Every Command
    desflow from the Command Line: One Model, Every Command
    October 30, 2026
  • From dynamic-des to desflow: What Changed in Five Releases
    From dynamic-des to desflow: What Changed in Five Releases
    October 28, 2026
  • odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    October 12, 2026
  • Defining Data-Streaming Simulations in YAML, Without Writing Python
    Defining Data-Streaming Simulations in YAML, Without Writing Python
    October 6, 2026
  • Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    October 2, 2026
  • Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    October 1, 2026
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Current London 2026: Building End-to-End Data Lineage
    Current London 2026: Building End-to-End Data Lineage
    May 22, 2026
  • Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    April 21, 2026
  • Productionizing an Online Product Recommender using Event Driven Architecture
    Productionizing an Online Product Recommender using Event Driven Architecture
    February 23, 2026
  • desflow from the Command Line: One Model, Every Command
    desflow from the Command Line: One Model, Every Command
    October 30, 2026
  • From dynamic-des to desflow: What Changed in Five Releases
    From dynamic-des to desflow: What Changed in Five Releases
    October 28, 2026
  • odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    October 12, 2026
  • Defining Data-Streaming Simulations in YAML, Without Writing Python
    Defining Data-Streaming Simulations in YAML, Without Writing Python
    October 6, 2026
  • Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    October 2, 2026
  • Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    October 1, 2026
  • Data Streaming and Machine Learning Projects That Run on Your Laptop
    Data Streaming and Machine Learning Projects That Run on Your Laptop
    September 30, 2026
  • Learning MLOps with a Feature Store: A New Series
    Learning MLOps with a Feature Store: A New Series
    September 28, 2026
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • desflow: A Declarative API with Postgres and Redis Connectors
    desflow: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
Actions
Go back Reload Copy URL

Jaehyeon Kim

Data Engineer | Data Streaming | Powering ML & AI in Real Time

Copyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.