<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Amazon MSK on Jaehyeon Kim</title><link>https://jaehyeon.me/tags/amazon-msk/</link><description>Recent content in Amazon MSK on Jaehyeon Kim</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>Copyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.</copyright><lastBuildDate>Thu, 14 Dec 2023 00:00:00 +0000</lastBuildDate><atom:link href="https://jaehyeon.me/tags/amazon-msk/index.xml" rel="self" type="application/rss+xml"/><item><title>Consume data from Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 6</title><link>https://jaehyeon.me/blog/2023-12-14-real-time-streaming-with-kafka-and-flink-7/</link><pubDate>Thu, 14 Dec 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-12-14-real-time-streaming-with-kafka-and-flink-7/</guid><description>Amazon MSK can be configured as an event source of a Lambda function. Lambda internally polls for new messages from the event source and then synchronously invokes the target Lambda function. With this feature, we can develop a Kafka consumer application in serverless environment where developers can focus on application logic. In this lab, we will discuss how to create a Kafka consumer using a Lambda function.
Introduction Lab 1 Produce data to Kafka using Lambda Lab 2 Write data to Kafka from S3 using Flink Lab 3 Transform and write data to S3 from Kafka using Flink Lab 4 Clean, Aggregate, and Enrich Events with Flink Lab 5 Write data to DynamoDB using Kafka Connect Lab 6 Consume data from Kafka using Lambda (this post) Architecture Fake taxi ride data is sent to a Kafka topic by the Kafka producer application that is discussed in Lab 1.</description><enclosure url="https://jaehyeon.me/blog/2023-12-14-real-time-streaming-with-kafka-and-flink-7/featured.png" length="138986" type="image/png"/></item><item><title>Deploy Aiven OpenSearch Sink Connector - Kafka Connect for AWS Services Integration Part 5</title><link>https://jaehyeon.me/blog/2023-10-30-kafka-connect-for-aws-part-5/</link><pubDate>Mon, 30 Oct 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-10-30-kafka-connect-for-aws-part-5/</guid><description>A data pipeline from Apache Kafka into OpenSearch is deployed on AWS using Amazon MSK, Amazon MSK Connect and Amazon OpenSearch Service with Terraform in this post. First the infrastructure will be deployed that covers a Virtual Private Cloud (VPC), Virtual Private Network (VPN) server, MSK Cluster and OpenSearch domain. Then Kafka source and sink connectors will be deployed on MSK Connect, followed by performing quick data analysis. In the previous post, we discussed how to develop that pipeline locally using Docker.</description><enclosure url="https://jaehyeon.me/blog/2023-10-30-kafka-connect-for-aws-part-5/featured.png" length="85575" type="image/png"/></item><item><title>Produce data to Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 1</title><link>https://jaehyeon.me/blog/2023-10-26-real-time-streaming-with-kafka-and-flink-2/</link><pubDate>Thu, 26 Oct 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-10-26-real-time-streaming-with-kafka-and-flink-2/</guid><description>In this lab, we will create a Kafka producer application using AWS Lambda, which sends fake taxi ride data into a Kafka topic on Amazon MSK. A configurable number of the producer Lambda function will be invoked by an Amazon EventBridge schedule rule. In this way we are able to generate test data concurrently based on the desired volume of messages.
Introduction Lab 1 Produce data to Kafka using Lambda (this post) Lab 2 Write data to Kafka from S3 using Flink Lab 3 Transform and write data to S3 from Kafka using Flink Lab 4 Clean, Aggregate, and Enrich Events with Flink Lab 5 Write data to DynamoDB using Kafka Connect Lab 6 Consume data from Kafka using Lambda [Update 2023-11-06] Initially I planned to deploy Pyflink applications on Amazon Managed Service for Apache Flink, but I changed the plan to use a local Flink cluster deployed on Docker.</description><enclosure url="https://jaehyeon.me/blog/2023-10-26-real-time-streaming-with-kafka-and-flink-2/featured.png" length="138560" type="image/png"/></item><item><title>Real Time Streaming with Kafka and Flink - Introduction</title><link>https://jaehyeon.me/blog/2023-10-05-real-time-streaming-with-kafka-and-flink-1/</link><pubDate>Thu, 05 Oct 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-10-05-real-time-streaming-with-kafka-and-flink-1/</guid><description>Real Time Streaming with Amazon Kinesis is an AWS workshop that helps users build a streaming analytics application on AWS. Incoming events are stored in a number of streams of the Amazon Kinesis Data Streams service, and various other AWS services and tools are used to process and analyse data.
Apache Kafka is a popular distributed event store and stream processing platform, and it stores incoming events in topics. As part of learning real time streaming analytics on AWS, we can rebuild the analytics applications by replacing the Kinesis streams with Kafka topics.</description><enclosure url="https://jaehyeon.me/blog/2023-10-05-real-time-streaming-with-kafka-and-flink-1/featured.png" length="138141" type="image/png"/></item><item><title>Getting Started with PyFlink on AWS - Part 3 AWS Managed Flink and MSK</title><link>https://jaehyeon.me/blog/2023-09-04-getting-started-with-pyflink-on-aws-part-3/</link><pubDate>Mon, 04 Sep 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-09-04-getting-started-with-pyflink-on-aws-part-3/</guid><description>Amazon Managed Service for Apache Flink is the easiest option to run Flink applications on AWS, and the Pyflink app is deployed via it in this post. In this series of posts, we discuss a Flink (Pyflink) application that reads/writes from/to Kafka topics. In the previous posts, I demonstrated a Pyflink app that targets a local Kafka cluster as well as a Kafka cluster on Amazon MSK. The app was executed in a virtual environment as well as in a local Flink cluster for improved monitoring.</description><enclosure url="https://jaehyeon.me/blog/2023-09-04-getting-started-with-pyflink-on-aws-part-3/featured.png" length="74618" type="image/png"/></item><item><title>Getting Started with PyFlink on AWS - Part 2 Local Flink and MSK</title><link>https://jaehyeon.me/blog/2023-08-28-getting-started-with-pyflink-on-aws-part-2/</link><pubDate>Mon, 28 Aug 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-08-28-getting-started-with-pyflink-on-aws-part-2/</guid><description>A Kafka cluster on Amazon MSK is authenticated by IAM, and the app needs an additional jar dependency for it. As Amazon Managed Service for Apache Flink does not allow you to specify multiple pipeline jar files, we have to build a custom Uber Jar that combines multiple jar files. In this series of posts, we discuss a Flink (Pyflink) application that reads/writes from/to Kafka topics. In part 1, an app that targets a local Kafka cluster was created.</description><enclosure url="https://jaehyeon.me/blog/2023-08-28-getting-started-with-pyflink-on-aws-part-2/featured.png" length="64005" type="image/png"/></item><item><title>Deploy Camel DynamoDB Sink Connector - Kafka Connect for AWS Services Integration Part 3</title><link>https://jaehyeon.me/blog/2023-07-03-kafka-connect-for-aws-part-3/</link><pubDate>Mon, 03 Jul 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-07-03-kafka-connect-for-aws-part-3/</guid><description>As part of investigating how to utilize Kafka Connect effectively for AWS services integration, I demonstrated how to develop the Camel DynamoDB sink connector using Docker in Part 2. Fake order data was generated using the MSK Data Generator source connector, and the sink connector was configured to consume the topic messages to ingest them into a DynamoDB table. In this post, I will illustrate how to deploy the data ingestion applications using Amazon MSK and MSK Connect.</description><enclosure url="https://jaehyeon.me/blog/2023-07-03-kafka-connect-for-aws-part-3/featured.png" length="76240" type="image/png"/></item><item><title>Kafka Connect for AWS Services Integration - Part 1 Introduction</title><link>https://jaehyeon.me/blog/2023-05-03-kafka-connect-for-aws-part-1/</link><pubDate>Wed, 03 May 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-05-03-kafka-connect-for-aws-part-1/</guid><description>Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka (MSK) are two managed streaming services offered by AWS. Many resources on the web indicate Kinesis Data Streams is better when it comes to integrating with AWS services. However, it is not necessarily the case with the help of Kafka Connect. According to the documentation of Apache Kafka, Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems.</description><enclosure url="https://jaehyeon.me/blog/2023-05-03-kafka-connect-for-aws-part-1/featured.png" length="22272" type="image/png"/></item><item><title>Simplify Streaming Ingestion on AWS - Part 2 MSK and Athena</title><link>https://jaehyeon.me/blog/2023-03-14-simplify-streaming-ingestion-athena/</link><pubDate>Tue, 14 Mar 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-03-14-simplify-streaming-ingestion-athena/</guid><description>Athena provides the MSK connector to enable SQL queries on Apache Kafka topics directly, and it can also facilitate the extraction of insights without setting up an additional pipeline to store data into S3. In this post, we discuss how to update the streaming ingestion solution so that data in the Kafka topic can be queried by Athena instead of Redshift. In Part 1, we discussed a streaming ingestion solution using EventBridge, Lambda, MSK and Redshift Serverless.</description><enclosure url="https://jaehyeon.me/blog/2023-03-14-simplify-streaming-ingestion-athena/featured.png" length="43403" type="image/png"/></item><item><title>Simplify Streaming Ingestion on AWS - Part 1 MSK and Redshift</title><link>https://jaehyeon.me/blog/2023-02-08-simplify-streaming-ingestion-redshift/</link><pubDate>Wed, 08 Feb 2023 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2023-02-08-simplify-streaming-ingestion-redshift/</guid><description>Apache Kafka is a popular distributed event store and stream processing platform. Previously loading data from Kafka into Redshift and Athena usually required Kafka connectors (e.g. Amazon Redshift Sink Connector and Amazon S3 Sink Connector). Recently these AWS services provide features to ingest data from Kafka directly, which facilitates a simpler architecture that achieves low-latency and high-speed ingestion of streaming data. In part 1 of the simplify streaming ingestion on AWS series, we discuss how to develop an end-to-end streaming ingestion solution using EventBridge, Lambda, MSK and Redshift Serverless on AWS.</description><enclosure url="https://jaehyeon.me/blog/2023-02-08-simplify-streaming-ingestion-redshift/featured.png" length="32864" type="image/png"/></item><item><title>Use External Schema Registry with MSK Connect - Part 2 MSK Deployment</title><link>https://jaehyeon.me/blog/2022-04-03-schema-registry-part2/</link><pubDate>Sun, 03 Apr 2022 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2022-04-03-schema-registry-part2/</guid><description>We build the Change Data Capture (CDC) solution with a schema registry on AWS using MSK, MSK Connect, Aurora PostgreSQL and ECS. In the previous post, we discussed that solution in a local development environment set up using Docker Compose. The Debezium and Confluent S3 connectors are deployed with the Confluent Avro converter and the Apicurio registry is used as the schema registry service. A quick example is shown to illustrate how schema evolution can be managed by the schema registry.</description><enclosure url="https://jaehyeon.me/blog/2022-04-03-schema-registry-part2/featured.png" length="59689" type="image/png"/></item><item><title>Implement Data Lake - Data Lake Demo using Change Data Capture (CDC) on AWS Part 3</title><link>https://jaehyeon.me/blog/2021-12-19-datalake-demo-part3/</link><pubDate>Sun, 19 Dec 2021 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2021-12-19-datalake-demo-part3/</guid><description>As the last part of this series, we&amp;rsquo;ll build an Apache Hudi DeltaStreamer app on Amazon EMR and use the resulting Hudi table with Amazon Athena and Amazon Quicksight to build a dashboard. In the previous post, we created a VPC that has private and public subnets in 2 availability zones in order to build and deploy the data lake solution on AWS. NAT instances are created to forward outbound traffic to the internet and a VPN bastion host is set up to facilitate deployment.</description><enclosure url="https://jaehyeon.me/blog/2021-12-19-datalake-demo-part3/featured.png" length="164526" type="image/png"/></item><item><title>Data Lake Demo using Change Data Capture (CDC) on AWS - Part 2 Implement CDC</title><link>https://jaehyeon.me/blog/2021-12-12-datalake-demo-part2/</link><pubDate>Sun, 12 Dec 2021 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2021-12-12-datalake-demo-part2/</guid><description>We build the CDC part of the data lake solution on AWS using Amazon MSK and MSK Connect. In the previous post, we discussed a data lake solution where data ingestion is performed using change data capture (CDC) and the output files are upserted to an Apache Hudi table. Being registered to Glue Data Catalog, it can be used for ad-hoc queries and report/dashboard creation. The Northwind database is used as the source database and, following the transactional outbox pattern, order-related changes are _upserted _to an outbox table by triggers.</description><enclosure url="https://jaehyeon.me/blog/2021-12-12-datalake-demo-part2/featured.png" length="164526" type="image/png"/></item><item><title>Data Lake Demo using Change Data Capture (CDC) on AWS - Part 1 Local Development</title><link>https://jaehyeon.me/blog/2021-12-05-datalake-demo-part1/</link><pubDate>Sun, 05 Dec 2021 00:00:00 +0000</pubDate><guid>https://jaehyeon.me/blog/2021-12-05-datalake-demo-part1/</guid><description>Change data capture (CDC) is a proven data integration pattern that has a wide range of applications. Among those, data replication to data lakes is a good use case in data engineering. Coupled with best-in-breed data lake formats such as Apache Hudi, we can build an efficient data replication solution. This is the first post of the data lake demo series. Over time, we&amp;rsquo;ll build a data lake that uses CDC.</description><enclosure url="https://jaehyeon.me/blog/2021-12-05-datalake-demo-part1/featured.png" length="164526" type="image/png"/></item></channel></rss>