https://jaehyeon.me/tags/aws-lambda/Recent content in AWS Lambda on Jaehyeon KimHugo -- gohugo.ioenCopyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.Thu, 14 Dec 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-12-14-real-time-streaming-with-kafka-and-flink-7/Thu, 14 Dec 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-12-14-real-time-streaming-with-kafka-and-flink-7/Amazon MSK can be configured as an event source of a Lambda function. Lambda internally polls for new messages from the event source and then synchronously invokes the target Lambda function. With this feature, we can develop a Kafka consumer application in serverless environment where developers can focus on application logic. In this lab, we will discuss how to create a Kafka consumer using a Lambda function. Introduction Lab 1 Produce data to Kafka using Lambda Lab 2 Write data to Kafka from S3 using Flink Lab 3 Transform and write data to S3 from Kafka using Flink Lab 4 Clean, Aggregate, and Enrich Events with Flink Lab 5 Write data to DynamoDB using Kafka Connect Lab 6 Consume data from Kafka using Lambda (this post) Architecture Fake taxi ride data is sent to a Kafka topic by the Kafka producer application that is discussed in Lab 1.https://jaehyeon.me/blog/2023-10-26-real-time-streaming-with-kafka-and-flink-2/Thu, 26 Oct 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-10-26-real-time-streaming-with-kafka-and-flink-2/In this lab, we will create a Kafka producer application using AWS Lambda, which sends fake taxi ride data into a Kafka topic on Amazon MSK. A configurable number of the producer Lambda function will be invoked by an Amazon EventBridge schedule rule. In this way we are able to generate test data concurrently based on the desired volume of messages. Introduction Lab 1 Produce data to Kafka using Lambda (this post) Lab 2 Write data to Kafka from S3 using Flink Lab 3 Transform and write data to S3 from Kafka using Flink Lab 4 Clean, Aggregate, and Enrich Events with Flink Lab 5 Write data to DynamoDB using Kafka Connect Lab 6 Consume data from Kafka using Lambda [Update 2023-11-06] Initially I planned to deploy Pyflink applications on Amazon Managed Service for Apache Flink, but I changed the plan to use a local Flink cluster deployed on Docker.https://jaehyeon.me/blog/2023-04-12-integrate-glue-schema-registry/Wed, 12 Apr 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-04-12-integrate-glue-schema-registry/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 As Kafka producer and consumer apps are decoupled, they operate on Kafka topics rather than communicating with each other directly. As described in the Confluent document, Schema Registry provides a centralized repository for managing and validating schemas for topic message data, and for serialization and deserialization of the data over the network.https://jaehyeon.me/blog/2023-03-14-simplify-streaming-ingestion-athena/Tue, 14 Mar 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-03-14-simplify-streaming-ingestion-athena/In Part 1, we discussed a streaming ingestion solution using EventBridge, Lambda, MSK and Redshift Serverless. Athena provides the MSK connector to enable SQL queries on Apache Kafka topics directly, and it can also facilitate the extraction of insights without setting up an additional pipeline to store data into S3. In this post, we discuss how to update the streaming ingestion solution so that data in the Kafka topic can be queried by Athena instead of Redshift.https://jaehyeon.me/blog/2023-02-08-simplify-streaming-ingestion-redshift/Wed, 08 Feb 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-02-08-simplify-streaming-ingestion-redshift/Apache Kafka is a popular distributed event store and stream processing platform. Previously loading data from Kafka into Redshift and Athena usually required Kafka connectors (e.g. Amazon Redshift Sink Connector and Amazon S3 Sink Connector). Recently these AWS services provide features to ingest data from Kafka directly, which facilitates a simpler architecture that achieves low-latency and high-speed ingestion of streaming data. In part 1 of the simplify streaming ingestion on AWS series, we discuss how to develop an end-to-end streaming ingestion solution using EventBridge, Lambda, MSK and Redshift Serverless on AWS.https://jaehyeon.me/blog/2022-08-06-revisit-lambda-operator/Sat, 06 Aug 2022 00:00:00 +0000https://jaehyeon.me/blog/2022-08-06-revisit-lambda-operator/Apache Airflow is a popular workflow management platform. A wide range of AWS services are integrated with the platform by Amazon AWS Operators. AWS Lambda is one of the integrated services, and it can be used to develop workflows efficiently. The current Lambda Operator, however, just invokes a Lambda function, and it can fail to report the invocation result of a function correctly and to record the exact error message from failure.https://jaehyeon.me/blog/2022-07-18-sam-for-data-professionals/Mon, 18 Jul 2022 00:00:00 +0000https://jaehyeon.me/blog/2022-07-18-sam-for-data-professionals/AWS Lambda provides serverless computing capabilities, and it can be used for performing validation or light processing/transformation of data. Moreover, with its integration with more than 140 AWS services, it facilitates building complex systems employing event-driven architectures. There are many ways to build serverless applications and one of the most efficient ways is using specialised frameworks such as the AWS Serverless Application Model (SAM) and Serverless Framework. In this post, I’ll demonstrate how to build a serverless data processing application using SAM.https://jaehyeon.me/blog/2021-10-13-lambda-schedule/Wed, 13 Oct 2021 00:00:00 +0000https://jaehyeon.me/blog/2021-10-13-lambda-schedule/Triggering a Lambda function by an EventBridge Events rule can be used as a _serverless _replacement of cron job. The highest frequency of it is one invocation per minute so that it cannot be used directly if you need to schedule a Lambda function more frequently. For example, it may be refreshing an application with real time metrics from an Amazon Connect instance where some metrics are updated every 15 seconds.https://jaehyeon.me/blog/2020-04-13-airflow-lambda-operator/Mon, 13 Apr 2020 00:00:00 +0000https://jaehyeon.me/blog/2020-04-13-airflow-lambda-operator/Apache Airflow is a popular open-source workflow management platform. Typically tasks run remotely by Celery workers for scalability. In AWS, however, scalability can also be achieved using serverless computing services in a simpler way. For example, the ECS Operator allows to run dockerized tasks and, with the Fargate launch type, they can run in a serverless environment. The ECS Operator alone is not sufficient because it can take up to several minutes to pull a Docker image and to set up network interface (for the case of Fargate launch type).https://jaehyeon.me/blog/2019-07-20-aws-localstack/Sat, 20 Jul 2019 00:00:00 +0000https://jaehyeon.me/blog/2019-07-20-aws-localstack/LocalStack provides an easy-to-use test/mocking framework for developing AWS applications. In this post, I’ll demonstrate how to utilize LocalStack for development using a web service. Specifically a simple web service built with Flask-RestPlus is used. It supports simple CRUD operations against a database table. It is set that SQS and Lambda are used for creating and updating a record. When a POST or PUT request is made, the service sends a message to a SQS queue and directly returns 204 reponse.https://jaehyeon.me/blog/2017-04-17-serverless-data-product-4/Mon, 17 Apr 2017 00:00:00 +0000https://jaehyeon.me/blog/2017-04-17-serverless-data-product-4/In the previous posts, it is discussed how to package/deploy an R model with AWS Lambda and to expose the Lambda function via Amazon API Gateway. Main benefits of serverless architecture is cost-effectiveness and being hassle-free from provisioning/managing servers. While the API returns a predicted admission status value given GRE, GPA and Rank, there is an issue if it is served within a web application: Cross-Origin Resource Sharing (CORS). This post discusses how to resolve this issue by updating API configuration and the Lambda function handler with a simple web application.https://jaehyeon.me/blog/2017-04-13-serverless-data-product-3/Thu, 13 Apr 2017 00:00:00 +0000https://jaehyeon.me/blog/2017-04-13-serverless-data-product-3/In Part I of this series, R and necessary libraries/packages together with a Lambda function handler are packaged and saved to Amazon S3. Then, in Part II, the package is deployed at AWS Lambda after creating and assigning a role to the Lambda function. Although the Lambda function can be called via the Invoke API, it’ll be much more useful if the function can be called as a web service (or API).https://jaehyeon.me/blog/2017-04-11-serverless-data-product-2/Tue, 11 Apr 2017 00:00:00 +0000https://jaehyeon.me/blog/2017-04-11-serverless-data-product-2/In the previous post, serverless event-driven application development is introduced. Also how to package R, necessary libraries/packages and a Lambda function handler is discussed. No need of provisioning/managing servers is one of the key benefits of the architecture. It is also a cost-effective way of delivering a data product as functions are executed on-demand rather than in servers that run 24/7. Furthermore AWS Lambda free tier includes 1M free requests per month and 400,000 GB-seconds of compute time per month, which is available to both existing and new AWS customers indefinitely.https://jaehyeon.me/blog/2017-04-08-serverless-data-product-1/Sat, 08 Apr 2017 00:00:00 +0000https://jaehyeon.me/blog/2017-04-08-serverless-data-product-1/