Data Lake Demo Using Change Data Capture (CDC) on AWS – Part 3 Implement Data Lake

featured.png
December 19, 202111 min read Data Engineering Data Lake Demo Using Change Data CaptureAmazon EMRAmazon MSKAmazon MSK ConnectApache HudiApache KafkaApache SparkAWSChange Data CaptureData LakeDockerKafka ConnectTerraform

Change data capture (CDC) on Amazon MSK and ingesting data using Apache Hudi on Amazon EMR can be used to build an efficient data lake solution. In this post, we'll build a Hudi DeltaStramer app on Amazon EMR and use the resulting Hudi table with Athena and Quicksight to build a dashboard.

Data Lake Demo Using Change Data Capture (CDC) on AWS – Part 2 Implement CDC

featured.png
December 12, 202117 min read Data Engineering Data Lake Demo Using Change Data CaptureAmazon EMRAmazon MSKAmazon MSK ConnectApache HudiApache KafkaApache SparkAWSChange Data CaptureData LakeDockerKafka ConnectTerraform

Change data capture (CDC) on Amazon MSK and ingesting data using Apache Hudi on Amazon EMR can be used to build an efficient data lake solution. In this post, we'll build CDC with Amazon MSK and MSK Connect.

Data Lake Demo Using Change Data Capture (CDC) on AWS – Part 1 Local Development

featured.png
December 5, 202118 min read Data Engineering Data Lake Demo Using Change Data CaptureAmazon EMRAmazon MSKAmazon MSK ConnectApache HudiApache KafkaApache SparkAWSChange Data CaptureData LakeDockerDocker ComposeKafka Connect

Change data capture (CDC) on Amazon MSK and ingesting data using Apache Hudi on Amazon EMR can be used to build an efficient data lake solution. As a starting point, we’ll discuss the source database and CDC streaming infrastructure in the local environment.