Orchestrate the Amazon Athena and Iceberg pizza shop dbt project with Apache Airflow, running the dimension and denormalised fact table builds.

Model pizza shop data as Apache Iceberg tables transformed on Amazon Athena with dbt, using array and struct types to denormalise the fact table.

Consume Kafka messages with an AWS Lambda function, using Amazon MSK as an event source so that Lambda polls the topic and invokes the function.

A local Flink and Spark environment built from EMR container images, where Flink ingests data in real time and Spark queries it via the Glue Data Catalog.

Ingest records from a Kafka topic into a DynamoDB table using the Camel DynamoDB sink connector, defined and run on Kafka Connect.

Export Kafka topic messages to S3 with PyFlink, enriching records through a user defined function and querying them on Athena through a Glue table.

Read records from S3 and send them into a Kafka topic with PyFlink, using a custom pipeline jar for IAM authentication, which is useful for enrichment.

Deploy a Kafka to OpenSearch pipeline on AWS with Terraform, covering a VPC, VPN server, MSK cluster, OpenSearch domain and the MSK Connect connectors.

Produce fake taxi ride data into a Kafka topic on Amazon MSK from an AWS Lambda function invoked repeatedly by an EventBridge schedule rule.

Build a local Kafka to OpenSearch pipeline on Docker with the Aiven sink connector, ingesting fake impressions and clicks for near real time analytics.