One simulation that writes Parquet for model training and streams Kafka events for live inference, so training and serving use the same simulation code.

Tracking data across Kafka, Flink and Spark pipelines with OpenLineage, to show where each dataset came from and what reads it. Slides from my Current London 2026 session.

Keeping a simulation in step with live data: change a running SimPy model's parameters from Kafka and stream its events out, so it acts as a digital twin.

Apache Kyuubi as a multi-tenant SQL gateway that provisions on-demand Spark, Flink and Trino engines, giving self-service analytics with central governance.

Apache Paimon, Fluss and Apache Iceberg compared as table layers for streaming and batch, then combined into one architecture with Flink and Spark.

Deploying a dbt project to dev and prod on BigQuery, covering slim CI, unit tests and a write audit publish step that builds on a cloned dataset.

GitHub Actions gives a dbt project on BigQuery a slim CI run on pull requests and a deploy job that publishes the project as a container image.

Unit test batch and streaming Apache Beam pipelines in Python, using TestStream to advance watermarks and processing time across scenarios.

Orchestrate the Amazon Athena and Iceberg pizza shop dbt project with Apache Airflow, running the dimension and denormalised fact table builds.

Model pizza shop data as Apache Iceberg tables transformed on Amazon Athena with dbt, using array and struct types to denormalise the fact table.