Jaehyeon Kim
Jaehyeon Kim

  • Blog
    • Categories

      List of categories.

    • Tags

      List of tags.

    • Series

      List of series.

    • Archives

  • Projects
  • Slides

/

  • Github Linkedin RSS

  • Font Size
  • Palette
  • Mode
  1. Home
  2. Tags
  3. AWS

Data Lake Demo using Change Data Capture (CDC) on AWS - Part 1 Local Development

Data Lake Demo using Change Data Capture (CDC) on AWS - Part 1 Local Development
December 5, 202118 min read Data-Engineering Data Integration Data-Streaming Data Lake Demo Using Change Data CaptureAmazon EMRAmazon MSKApache HudiApache KafkaAWSChange Data Capture (CDC)DebeziumKafka Connect

Set up the source PostgreSQL database with an outbox table, then run Debezium and an S3 sink connector locally with Docker Compose.

Read More: Data Lake Demo using Change Data Capture (CDC) on AWS - Part 1 Local Development

Local Development of AWS Glue 3.0 and Later

Local Development of AWS Glue 3.0 and Later
November 14, 20218 min read Data-EngineeringAWSAWS GlueDockerPySparkPython

Create a development environment for AWS Glue 3.0 and later by building a custom Docker image, because AWS publishes no image for those versions.

Read More: Local Development of AWS Glue 3.0 and Later

Yet another serverless solution for invoking AWS Lambda at a sub-minute frequency

Yet another serverless solution for invoking AWS Lambda at a sub-minute frequency
October 13, 20216 min read DevelopmentAmazon EventBridgeAmazon SQSAWSAWS LambdaNode.jsServerless Framework

Schedule an AWS Lambda function more often than once a minute, using Amazon SQS to get past the one invocation per minute limit of EventBridge rules.

Read More: Yet another serverless solution for invoking AWS Lambda at a sub-minute frequency

AWS Glue Local Development with Docker and Visual Studio Code

AWS Glue Local Development with Docker and Visual Studio Code
August 20, 20219 min read Data-EngineeringApache SparkAWSAWS GlueDockerPySparkPython

Build development environments for AWS Glue 1.0 and 2.0 with the published Docker image and the Visual Studio Code Remote Containers extension.

Read More: AWS Glue Local Development with Docker and Visual Studio Code

Thoughts on Apache Airflow AWS Lambda Operator

Thoughts on Apache Airflow AWS Lambda Operator
April 13, 202010 min read Data-EngineeringApache AirflowAWSAWS LambdaDockerPython

In this post, it is demonstrated how AWS Lambda can be integrated with Apache Airflow using a custom operator inspired by the ECS Operator.

Read More: Thoughts on Apache Airflow AWS Lambda Operator

AWS Local Development with LocalStack

AWS Local Development with LocalStack
July 20, 20197 min read DevelopmentAWSAWS LambdaDockerFlaskLocalStackPython

Develop and test AWS applications against LocalStack, a mocking framework that runs AWS services on a local machine, demonstrated with a web service.

Read More: AWS Local Development with LocalStack

Serverless Data Product POC Backend Part 4 - Serving R ML Model via S3

Serverless Data Product POC Backend Part 4 - Serving R ML Model via S3
April 17, 201710 min read Development Serverless Data ProductAmazon API GatewayAWSAWS LambdaPythonR

Host the web application that calls an R machine learning model on Amazon S3, so both the front end and the Lambda backend run without a server.

Read More: Serverless Data Product POC Backend Part 4 - Serving R ML Model via S3

Serverless Data Product POC Backend Part 3 - Exposing R ML Model via APIG

Serverless Data Product POC Backend Part 3 - Exposing R ML Model via APIG
April 13, 201711 min read Development Serverless Data ProductAmazon API GatewayAWSAWS LambdaPythonR

Expose an R machine learning model packaged in AWS Lambda through Amazon API Gateway, giving the model a callable HTTP endpoint on AWS.

Read More: Serverless Data Product POC Backend Part 3 - Exposing R ML Model via APIG

Serverless Data Product POC Backend Part 2 - Deploying R ML Model via Lambda

Serverless Data Product POC Backend Part 2 - Deploying R ML Model via Lambda
April 11, 20179 min read Development Serverless Data ProductAmazon API GatewayAWSAWS LambdaPythonR

In the previous post, it is discuss how to develop and package an R machine learning model. In this post, I'll illustrate how to deploy the model via AWS Lambda.

Read More: Serverless Data Product POC Backend Part 2 - Deploying R ML Model via Lambda

Serverless Data Product POC Backend Part 1 - Packaging R ML Model for Lambda

Serverless Data Product POC Backend Part 1 - Packaging R ML Model for Lambda
April 8, 201712 min read Development Serverless Data ProductAmazon API GatewayAWSAWS LambdaPythonR

In this post, I'll demonstrate how to test and develop a logistic regression model developed in R. Also the model will be packaged for AWS Lambda.

Read More: Serverless Data Product POC Backend Part 1 - Packaging R ML Model for Lambda
  • ««
  • «
  • 1
  • 2
  • 3
  • 4
  • 5
  • »
  • »»
Profile
Jaehyeon Kim
Jaehyeon Kim
Data Engineer | Data Streaming | Powering ML & AI in Real Time
Taxonomies
Data Streaming 70 Data Engineering 38 Development 28 Data Analysis 17 Data Integration 12 Open Source 7 Kubernetes 6 Machine Learning 5 Security 5 Data Architecture 3 Data Processing 3 Big Data 2 System Architecture 2 Web Development 2
Python 75 Apache Kafka 68 AWS 50 Docker 50 Apache Flink 34 Apache Beam 17 Apache Spark 16 Kafka Connect 15 Amazon MSK 14 AWS Lambda 14 dbt 13 Amazon EMR 11 Kubernetes 8 PyFlink 8 Kotlin 7 PostgreSQL 7 Change Data Capture (CDC) 6 Debezium 6 dynamic-des 6 Amazon DynamoDB 5 Apache Airflow 5 Discrete Event Simulation 5 Factor House Local 5 PySpark 5 Amazon API Gateway 4 Amazon Athena 4 Apache Iceberg 4 AWS Glue 4 AWS Glue Schema Registry 4 BigQuery 4 Digital Twin 4 FastAPI 4 Minikube 4 SimPy 4 Amazon EKS 3 Amazon QuickSight 3 Amazon S3 3 Apache Hudi 3 EMR on EKS 3 GCP 3 ALL 137
Kafka Development with Docker 11 Apache Beam Python Examples 10 Real Time Streaming with Kafka and Flink 7 dbt Pizza Shop Demo 6 Apache Beam Local Development with Python 5 Building Real-Time Digital Twins with dynamic-des 5 dbt for Effective Data Transformation on AWS 5 Getting Started with Real-Time Streaming in Kotlin 5 Kafka Connect for AWS Services Integration 5 Serverless Data Product 4 Data Lake Demo Using Change Data Capture 3 Getting Started with PyFlink on AWS 3 Kafka Development on Kubernetes 3 Parallel processing on single machine 3 Realtime Dashboard with FastAPI, Streamlit and Next.js 3 API development with R 2 dbt Guide for Production 2 Deploy Python Stream Processing App on Kubernetes 2 Download Stock Data 2 From Prototype to Production: Real-Time Product Recommendation with Contextual Bandits 2 ALL 24
2026 11 2025 13 2024 29 2023 39 2022 15 2021 7 2020 1 2019 5 2018 2 2017 6 2016 6 2015 15 2014 5
Posts
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • Introducing odctl: One CLI for a Local Open Data Stack
    Introducing odctl: One CLI for a Local Open Data Stack
    July 16, 2026
  • One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    May 25, 2026
  • Building an Event-Driven Hybrid Digital Twin with dynamic-des
    Building an Event-Driven Hybrid Digital Twin with dynamic-des
    April 29, 2026
  • Why Digital Twins Are Rewiring Industry 4.0
    Why Digital Twins Are Rewiring Industry 4.0
    April 22, 2026
  • Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    April 21, 2026
  • Slides as Code: Integrating Reveal.js into my Hugo Blog
    Slides as Code: Integrating Reveal.js into my Hugo Blog
    March 9, 2026
  • Productionizing an Online Product Recommender using Event Driven Architecture
    Productionizing an Online Product Recommender using Event Driven Architecture
    February 23, 2026
  • Stream Processing with Flink in Kotlin
    Stream Processing with Flink in Kotlin
    December 10, 2025
  • Building an Agentic Analytics System over an Iceberg Lakehouse
    Building an Agentic Analytics System over an Iceberg Lakehouse
    July 18, 2026
  • Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    Dynamic DES v0.11.1: A Declarative API with Postgres and Redis Connectors
    July 17, 2026
  • Introducing odctl: One CLI for a Local Open Data Stack
    Introducing odctl: One CLI for a Local Open Data Stack
    July 16, 2026
  • One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES v0.8.1
    May 25, 2026
  • Current London 2026: Building End-to-End Data Lineage
    Current London 2026: Building End-to-End Data Lineage
    May 22, 2026
  • Building an Event-Driven Hybrid Digital Twin with dynamic-des
    Building an Event-Driven Hybrid Digital Twin with dynamic-des
    April 29, 2026
  • Why Digital Twins Are Rewiring Industry 4.0
    Why Digital Twins Are Rewiring Industry 4.0
    April 22, 2026
  • Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning
    April 21, 2026
  • Slides as Code: Integrating Reveal.js into my Hugo Blog
    Slides as Code: Integrating Reveal.js into my Hugo Blog
    March 9, 2026
  • Productionizing an Online Product Recommender using Event Driven Architecture
    Productionizing an Online Product Recommender using Event Driven Architecture
    February 23, 2026
Actions
Go back Reload Copy URL

Jaehyeon Kim

Data Engineer | Data Streaming | Powering ML & AI in Real Time

Copyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.