odctl is a CLI on PyPI that launches Kafka, Flink, Spark, Trino, Iceberg, Airflow and an MLOps and observability suite as one local stack.

A comprehensive walkthrough from my session at Current London 2026 on capturing and visualizing data lineage across a production-style data stack.

Apache Kyuubi as a multi-tenant SQL gateway that provisions on-demand Spark, Flink and Trino engines, giving self-service analytics with central governance.

A local Flink and Spark environment built from EMR container images, where Flink ingests data in real time and Spark queries it via the Glue Data Catalog.

Amazon EMR on EKS data transformation pipelines with dbt. Subsets of IMDb data feed models developed in multiple layers following dbt best practices.

Amazon EMR on EC2 data transformation pipelines with dbt. Subsets of IMDb data feed models developed in multiple layers following dbt best practices.

AWS Glue data transformation pipelines with dbt. Subsets of IMDb data feed models developed in multiple layers following dbt best practices.

Develop Spark apps on an EMR cluster in a private subnet over VPN and the VS Code remote SSH extension, with the cluster shared by several users.

Provision EMR on EKS with Terraform and EKS Blueprints, then compare two Spark jobs with and without Dynamic Resource Allocation under Karpenter.

Run a data warehousing ETL job with Apache Iceberg for storage and PySpark for processing in an EMR local environment, then verify results in Athena.