Stream Kafka records from Amazon MSK into Redshift through its direct integration, with a producer Lambda built locally using AWS SAM and Terraform.

Configure a Kafka consumer to seek offsets by timestamp while partitions are assigned by subscription, on a single node cluster in Docker Compose.

Amazon Athena data transformation pipelines with dbt, closing the dbt on AWS series. Subsets of IMDb data feed models developed in multiple layers.

Amazon EMR on EKS data transformation pipelines with dbt. Subsets of IMDb data feed models developed in multiple layers following dbt best practices.

Amazon EMR on EC2 data transformation pipelines with dbt. Subsets of IMDb data feed models developed in multiple layers following dbt best practices.

AWS Glue data transformation pipelines with dbt. Subsets of IMDb data feed models developed in multiple layers following dbt best practices.

Redshift Serverless data transformation pipelines with dbt. Subsets of IMDb data feed models developed in multiple layers following dbt best practices.

Develop Spark apps on an EMR cluster in a private subnet over VPN and the VS Code remote SSH extension, with the cluster shared by several users.

Provision EMR on EKS with Terraform and EKS Blueprints, then compare two Spark jobs with and without Dynamic Resource Allocation under Karpenter.

Extend the Apache Airflow Lambda invoke function operator with a correlation ID, so it reports the invocation result and the exact error message.