Unit test batch and streaming Apache Beam pipelines in Python, using TestStream to advance watermarks and processing time across scenarios.

A streaming Apache Beam pipeline in Python that aggregates page visits by user in fixed 20 second windows, written with and without Beam SQL.

The Apache Beam portability layer explained, with bash scripts that manage local Flink and Kafka clusters and a streaming pipeline on the Flink Runner.

Batch Apache Beam pipelines in Python that aggregate website visit logs by user and time, written with and without Beam SQL and run in notebooks.

Run a basic Apache Beam pipeline in Python, then develop interactively in Jupyter notebooks with Beam SQL and Beam DataFrames examples.

Orchestrate the Amazon Athena and Iceberg pizza shop dbt project with Apache Airflow, running the dimension and denormalised fact table builds.

Model pizza shop data as Apache Iceberg tables transformed on Amazon Athena with dbt, using array and struct types to denormalise the fact table.

Orchestrate the BigQuery pizza shop dbt project with Apache Airflow, running the SCD type 2 dimension tables and the denormalised fact table.

Model pizza shop data on Google BigQuery with dbt, keeping SCD type 2 dimensions and denormalising the fact table with nested and repeated fields.

Orchestrate the PostgreSQL pizza shop dbt project with Apache Airflow, scheduling the SCD type 2 dimension and fact table builds as an ETL process.