Jaehyeon Kim
Jaehyeon Kim

  • Blog
    • Categories

      List of categories.

    • Tags

      List of tags.

    • Series

      List of series.

    • Archives

  • Projects
  • Slides

/

  • Github Linkedin RSS

  • Font Size
  • Palette
  • Mode
  1. Home
  2. Tags
  3. Python

Revisit AWS Lambda Invoke Function Operator of Apache Airflow

Revisit AWS Lambda Invoke Function Operator of Apache Airflow
August 6, 202215 min read Data-EngineeringApache AirflowAWSAWS LambdaDockerPython

Extend the Apache Airflow Lambda invoke function operator with a correlation ID, so it reports the invocation result and the exact error message.

Read More: Revisit AWS Lambda Invoke Function Operator of Apache Airflow

Serverless Application Model (SAM) for Data Professionals

Serverless Application Model (SAM) for Data Professionals
July 18, 20228 min read DevelopmentAmazon S3AWSAWS LambdaAWS SAMPython

Build a serverless data processing app with AWS SAM. A Lambda function runs when an S3 object is created and gets its third party packages from a layer.

Read More: Serverless Application Model (SAM) for Data Professionals

Data Warehousing ETL Demo with Apache Iceberg on EMR Local Environment

Data Warehousing ETL Demo with Apache Iceberg on EMR Local Environment
June 26, 202212 min read Data-EngineeringAmazon EMRApache IcebergApache SparkAWSPySparkPython

Run a data warehousing ETL job with Apache Iceberg for storage and PySpark for processing in an EMR local environment, then verify results in Athena.

Read More: Data Warehousing ETL Demo with Apache Iceberg on EMR Local Environment

Local Development of AWS Glue 3.0 and Later

Local Development of AWS Glue 3.0 and Later
November 14, 20218 min read Data-EngineeringAWSAWS GlueDockerPySparkPython

Create a development environment for AWS Glue 3.0 and later by building a custom Docker image, because AWS publishes no image for those versions.

Read More: Local Development of AWS Glue 3.0 and Later

AWS Glue Local Development with Docker and Visual Studio Code

AWS Glue Local Development with Docker and Visual Studio Code
August 20, 20219 min read Data-EngineeringApache SparkAWSAWS GlueDockerPySparkPython

Build development environments for AWS Glue 1.0 and 2.0 with the published Docker image and the Visual Studio Code Remote Containers extension.

Read More: AWS Glue Local Development with Docker and Visual Studio Code

Thoughts on Apache Airflow AWS Lambda Operator

Thoughts on Apache Airflow AWS Lambda Operator
April 13, 202010 min read Data-EngineeringApache AirflowAWSAWS LambdaDockerPython

Run frequent Apache Airflow tasks on AWS Lambda instead of Celery workers or ECS, with a custom operator modelled on the ECS operator.

Read More: Thoughts on Apache Airflow AWS Lambda Operator

Dynamic Routing and Centralized Auth with Traefik, Python and R Example

Dynamic Routing and Centralized Auth with Traefik, Python and R Example
November 29, 20199 min read DevelopmentDockerFastAPIPythonRTraefik

Set up path-based routing with Traefik and Docker for Python and R services, then add centralized authentication with Forward Authentication.

Read More: Dynamic Routing and Centralized Auth with Traefik, Python and R Example

Distributed Task Queue with Python and R Example

Distributed Task Queue with Python and R Example
November 15, 20199 min read DevelopmentFastAPIPythonRRServe

A FastAPI web service dispatches tasks to Celery and RServe workers, with Redis as broker and store, running on Docker Compose and Kubernetes.

Read More: Distributed Task Queue with Python and R Example

Linux Dev Environment on Windows

Linux Dev Environment on Windows
November 1, 201913 min read DevelopmentDockerKubernetesMinikubePythonRWSL

Build a Linux development environment on Windows with WSL, then run an RServe web service with a sidecar container on Minikube as the example app.

Read More: Linux Dev Environment on Windows

AWS Local Development with LocalStack

AWS Local Development with LocalStack
July 20, 20197 min read DevelopmentAWSAWS LambdaDockerFlaskLocalStackPython

Develop and test AWS applications against LocalStack, a mocking framework that runs AWS services on a local machine, demonstrated with a web service.

Read More: AWS Local Development with LocalStack
  • ««
  • «
  • 4
  • 5
  • 6
  • 7
  • 8
  • »
  • »»
Profile
Jaehyeon Kim
Jaehyeon Kim
Data Engineer | Data Streaming | Powering ML & AI in Real Time
Taxonomies
Data Streaming 70 Data Engineering 43 Development 28 Data Analysis 17 Open Source 13 Data Integration 12 Machine Learning 7 Kubernetes 6 Security 5 Data Architecture 3 Data Processing 3 Big Data 2 System Architecture 2 Web Development 2
Python 77 Apache Kafka 73 AWS 50 Docker 50 Apache Flink 39 Apache Beam 17 Apache Spark 17 Kafka Connect 16 Amazon MSK 14 AWS Lambda 14 Benchtop 13 dbt 13 odctl 12 Amazon EMR 11 dynamic-des 11 PostgreSQL 10 Kubernetes 8 PyFlink 8 Apache Iceberg 7 Change Data Capture (CDC) 7 Debezium 7 Kotlin 7 Apache Airflow 6 Discrete Event Simulation 6 Kafka UI 6 Amazon DynamoDB 5 PySpark 5 SimPy 5 Amazon API Gateway 4 Amazon Athena 4 AWS Glue 4 AWS Glue Schema Registry 4 BigQuery 4 Digital Twin 4 FastAPI 4 Minikube 4 Amazon EKS 3 Amazon QuickSight 3 Amazon S3 3 Apache Hudi 3 ALL 143
Kafka Development with Docker 11 Apache Beam Python Examples 10 Real Time Streaming with Kafka and Flink 7 Building Real-Time Digital Twins with dynamic-des 6 dbt Pizza Shop Demo 6 Apache Beam Local Development with Python 5 dbt for Effective Data Transformation on AWS 5 Getting Started with Real-Time Streaming in Kotlin 5 Kafka Connect for AWS Services Integration 5 Serverless Data Product 4 Data Lake Demo Using Change Data Capture 3 Getting Started with PyFlink on AWS 3 Kafka Development on Kubernetes 3 Parallel processing on single machine 3 Realtime Dashboard with FastAPI, Streamlit and Next.js 3 API development with R 2 dbt Guide for Production 2 Deploy Python Stream Processing App on Kubernetes 2 Download Stock Data 2 From Prototype to Production: Real-Time Product Recommendation with Contextual Bandits 2 ALL 25
2026 17 2025 13 2024 29 2023 39 2022 15 2021 7 2020 1 2019 5 2018 2 2017 6 2016 6 2015 15 2014 5
Posts
  • odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    October 12, 2026
  • Defining Data-Streaming Simulations in YAML, Without Writing Python
    Defining Data-Streaming Simulations in YAML, Without Writing Python
    October 6, 2026
  • Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    October 2, 2026
  • Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    October 1, 2026
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Current London 2026: Building End-to-End Data Lineage
    Current London 2026: Building End-to-End Data Lineage
    May 22, 2026
  • Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    April 21, 2026
  • Productionizing an Online Product Recommender using Event Driven Architecture
    Productionizing an Online Product Recommender using Event Driven Architecture
    February 23, 2026
  • Stream Processing with Flink in Kotlin
    Stream Processing with Flink in Kotlin
    December 10, 2025
  • Guide to Building Integrated Web Applications with FastAPI and NiceGUI
    Guide to Building Integrated Web Applications with FastAPI and NiceGUI
    November 19, 2025
  • odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command
    October 12, 2026
  • Defining Data-Streaming Simulations in YAML, Without Writing Python
    Defining Data-Streaming Simulations in YAML, Without Writing Python
    October 6, 2026
  • Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    Keeping Game Leaderboards Up to Date in Real Time with Kafka and Flink SQL
    October 2, 2026
  • Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    Change Data Capture on a Simulated Online Shop with Debezium and Kafka Connect
    October 1, 2026
  • Data Streaming and Machine Learning Projects That Run on Your Laptop
    Data Streaming and Machine Learning Projects That Run on Your Laptop
    September 30, 2026
  • Learning MLOps with a Feature Store: A New Series
    Learning MLOps with a Feature Store: A New Series
    September 28, 2026
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Dynamic DES: A Declarative API with Postgres and Redis Connectors
    Dynamic DES: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • Running Kafka, Flink, Spark, Trino and Iceberg Locally with One CLI
    Running Kafka, Flink, Spark, Trino and Iceberg Locally with One CLI
    July 16, 2026
  • One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES
    May 25, 2026
Actions
Go back Reload Copy URL

Jaehyeon Kim

Data Engineer | Data Streaming | Powering ML & AI in Real Time

Copyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.