Jaehyeon Kim
Jaehyeon Kim

  • Blog
    • Categories

      List of categories.

    • Tags

      List of tags.

    • Series

      List of series.

    • Archives

  • Projects
  • Slides

/

  • Github Linkedin RSS

  • Font Size
  • Palette
  • Mode
  1. Home
  2. Tags
  3. AWS Lambda

Consume data from Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 6

Consume data from Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 6
December 14, 20236 min read Data-Streaming Real Time Streaming With Kafka and FlinkAmazon MSKApache KafkaAWSAWS LambdaKpowPython

Consume Kafka messages with an AWS Lambda function, using Amazon MSK as an event source so that Lambda polls the topic and invokes the function.

Read More: Consume data from Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 6

Produce data to Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 1

Produce data to Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 1
October 26, 202314 min read Data-Streaming Real Time Streaming With Kafka and FlinkAmazon EventBridgeAmazon MSKApache KafkaAWSAWS LambdaKpowPython

Produce fake taxi ride data into a Kafka topic on Amazon MSK from an AWS Lambda function invoked repeatedly by an EventBridge schedule rule.

Read More: Produce data to Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 1

Integrate Glue Schema Registry with Your Python Kafka App

Integrate Glue Schema Registry with Your Python Kafka App
April 12, 202326 min read Data-StreamingApache KafkaAWS Glue Schema RegistryAWS LambdaKpowPython

Integrate Python Kafka producer and consumer apps running on AWS Lambda with the Glue Schema Registry, which manages and validates message schemas.

Read More: Integrate Glue Schema Registry with Your Python Kafka App

Simplify Streaming Ingestion on AWS - Part 2 MSK and Athena

Simplify Streaming Ingestion on AWS - Part 2 MSK and Athena
March 14, 202313 min read Data-Streaming Simplify Streaming Ingestion on AWSAmazon AthenaAmazon MSKApache KafkaAWSAWS LambdaPython

Stream Kafka records from Amazon MSK into Athena through its direct integration, with a producer Lambda built locally using AWS SAM and Terraform.

Read More: Simplify Streaming Ingestion on AWS - Part 2 MSK and Athena

Simplify Streaming Ingestion on AWS - Part 1 MSK and Redshift

Simplify Streaming Ingestion on AWS - Part 1 MSK and Redshift
February 8, 202318 min read Data-Streaming Simplify Streaming Ingestion on AWSAmazon MSKAmazon RedshiftApache KafkaAWSAWS LambdaPython

Stream Kafka records from Amazon MSK into Redshift through its direct integration, with a producer Lambda built locally using AWS SAM and Terraform.

Read More: Simplify Streaming Ingestion on AWS - Part 1 MSK and Redshift

Revisit AWS Lambda Invoke Function Operator of Apache Airflow

Revisit AWS Lambda Invoke Function Operator of Apache Airflow
August 6, 202215 min read Data-EngineeringApache AirflowAWSAWS LambdaDockerPython

Extend the Apache Airflow Lambda invoke function operator with a correlation ID, so it reports the invocation result and the exact error message.

Read More: Revisit AWS Lambda Invoke Function Operator of Apache Airflow

Serverless Application Model (SAM) for Data Professionals

Serverless Application Model (SAM) for Data Professionals
July 18, 20228 min read DevelopmentAmazon S3AWSAWS LambdaAWS SAMPython

Build a serverless data processing app with AWS SAM. A Lambda function runs when an S3 object is created and gets its third party packages from a layer.

Read More: Serverless Application Model (SAM) for Data Professionals

Yet another serverless solution for invoking AWS Lambda at a sub-minute frequency

Yet another serverless solution for invoking AWS Lambda at a sub-minute frequency
October 13, 20216 min read DevelopmentAmazon EventBridgeAmazon SQSAWSAWS LambdaNode.jsServerless Framework

Schedule an AWS Lambda function more often than once a minute, using Amazon SQS to get past the one invocation per minute limit of EventBridge rules.

Read More: Yet another serverless solution for invoking AWS Lambda at a sub-minute frequency

Thoughts on Apache Airflow AWS Lambda Operator

Thoughts on Apache Airflow AWS Lambda Operator
April 13, 202010 min read Data-EngineeringApache AirflowAWSAWS LambdaDockerPython

In this post, it is demonstrated how AWS Lambda can be integrated with Apache Airflow using a custom operator inspired by the ECS Operator.

Read More: Thoughts on Apache Airflow AWS Lambda Operator

AWS Local Development with LocalStack

AWS Local Development with LocalStack
July 20, 20197 min read DevelopmentAWSAWS LambdaDockerFlaskLocalStackPython

Develop and test AWS applications against LocalStack, a mocking framework that runs AWS services on a local machine, demonstrated with a web service.

Read More: AWS Local Development with LocalStack
  • ««
  • «
  • 1
  • 2
  • »
  • »»
Profile
Jaehyeon Kim
Jaehyeon Kim
Data Engineer | Data Streaming | Powering ML & AI in Real Time
Taxonomies
Data Streaming 70 Data Engineering 38 Development 28 Data Analysis 17 Data Integration 12 Open Source 7 Kubernetes 6 Machine Learning 5 Security 5 Data Architecture 3 Data Processing 3 Big Data 2 System Architecture 2 Web Development 2
Python 75 Apache Kafka 68 AWS 50 Docker 50 Apache Flink 34 Apache Beam 17 Apache Spark 16 Kafka Connect 15 Amazon MSK 14 AWS Lambda 14 dbt 13 Amazon EMR 11 Kubernetes 8 PyFlink 8 Kotlin 7 PostgreSQL 7 Change Data Capture (CDC) 6 Debezium 6 dynamic-des 6 Amazon DynamoDB 5 Apache Airflow 5 Discrete Event Simulation 5 Factor House Local 5 PySpark 5 Amazon API Gateway 4 Amazon Athena 4 Apache Iceberg 4 AWS Glue 4 AWS Glue Schema Registry 4 BigQuery 4 Digital Twin 4 FastAPI 4 Minikube 4 SimPy 4 Amazon EKS 3 Amazon QuickSight 3 Amazon S3 3 Apache Hudi 3 EMR on EKS 3 GCP 3 ALL 137
Kafka Development with Docker 11 Apache Beam Python Examples 10 Real Time Streaming with Kafka and Flink 7 dbt Pizza Shop Demo 6 Apache Beam Local Development with Python 5 Building Real-Time Digital Twins with dynamic-des 5 dbt for Effective Data Transformation on AWS 5 Getting Started with Real-Time Streaming in Kotlin 5 Kafka Connect for AWS Services Integration 5 Serverless Data Product 4 Data Lake Demo Using Change Data Capture 3 Getting Started with PyFlink on AWS 3 Kafka Development on Kubernetes 3 Parallel processing on single machine 3 Realtime Dashboard with FastAPI, Streamlit and Next.js 3 API development with R 2 dbt Guide for Production 2 Deploy Python Stream Processing App on Kubernetes 2 Download Stock Data 2 From Prototype to Production: Real-Time Product Recommendation with Contextual Bandits 2 ALL 24
2026 11 2025 13 2024 29 2023 39 2022 15 2021 7 2020 1 2019 5 2018 2 2017 6 2016 6 2015 15 2014 5
Posts
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • Introducing odctl: One CLI for a Local Open Data Stack
    Introducing odctl: One CLI for a Local Open Data Stack
    July 16, 2026
  • One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    May 25, 2026
  • Building an Event-Driven Hybrid Digital Twin with dynamic-des
    Building an Event-Driven Hybrid Digital Twin with dynamic-des
    April 29, 2026
  • Why Digital Twins Are Rewiring Industry 4.0
    Why Digital Twins Are Rewiring Industry 4.0
    April 22, 2026
  • Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    April 21, 2026
  • Slides as Code: Integrating Reveal.js into my Hugo Blog
    Slides as Code: Integrating Reveal.js into my Hugo Blog
    March 9, 2026
  • Productionizing an Online Product Recommender using Event Driven Architecture
    Productionizing an Online Product Recommender using Event Driven Architecture
    February 23, 2026
  • Stream Processing with Flink in Kotlin
    Stream Processing with Flink in Kotlin
    December 10, 2025
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • Introducing odctl: One CLI for a Local Open Data Stack
    Introducing odctl: One CLI for a Local Open Data Stack
    July 16, 2026
  • One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    May 25, 2026
  • Current London 2026: Building End-to-End Data Lineage
    Current London 2026: Building End-to-End Data Lineage
    May 22, 2026
  • Building an Event-Driven Hybrid Digital Twin with dynamic-des
    Building an Event-Driven Hybrid Digital Twin with dynamic-des
    April 29, 2026
  • Why Digital Twins Are Rewiring Industry 4.0
    Why Digital Twins Are Rewiring Industry 4.0
    April 22, 2026
  • Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    April 21, 2026
  • Slides as Code: Integrating Reveal.js into my Hugo Blog
    Slides as Code: Integrating Reveal.js into my Hugo Blog
    March 9, 2026
  • Productionizing an Online Product Recommender using Event Driven Architecture
    Productionizing an Online Product Recommender using Event Driven Architecture
    February 23, 2026
Actions
Go back Reload Copy URL

Jaehyeon Kim

Data Engineer | Data Streaming | Powering ML & AI in Real Time

Copyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.