Orchestrate the PostgreSQL pizza shop dbt project with Apache Airflow, scheduling the SCD type 2 dimension and fact table builds as an ETL process.

Model fictional pizza shop data on PostgreSQL with dbt, building two SCD type 2 dimension tables and one transactional fact table.

Ingest fake customer and order data into Kafka with the MSK Data Generator and save it to S3, with connectors deployed as Strimzi custom resources.

Send and read Kafka topic records from Python client applications built with the kafka-python package and deployed on Kubernetes.

Create a Kafka cluster on Kubernetes with Strimzi, the first step in a series on Python clients and Kafka Connect pipelines on Kubernetes.

Consume Kafka messages with an AWS Lambda function, using Amazon MSK as an event source so that Lambda polls the topic and invokes the function.

A local Flink and Spark environment built from EMR container images, where Flink ingests data in real time and Spark queries it via the Glue Data Catalog.

Ingest records from a Kafka topic into a DynamoDB table using the Camel DynamoDB sink connector, defined and run on Kafka Connect.

Aggregate taxi ride counts and trip durations by vendor over five second windows with PyFlink, then chart them in an OpenSearch dashboard.

Export Kafka topic messages to S3 with PyFlink, enriching records through a user defined function and querying them on Athena through a Glue table.