Direct text-to-SQL is hard to trust in production because raw schemas do not capture governed metrics or business meaning. This post walks through a local, open-source proof of concept that puts a semantic layer between the language model and the lakehouse, combining Strands, WrenAI, Trino, Iceberg, and long-term agent memory.

Since the v0.8.1 release, Dynamic DES has gained a declarative SimulationContext API, native Postgres and Redis ingress and egress connectors, and broader object storage support. This post walks through what changed and why the new builder pattern makes real-time digital twins far easier to write.

Provisioning a local data platform usually means fighting dependency conflicts, port clashes, and brittle Docker Compose files. odctl is a small CLI that turns Kafka, Flink, Spark, Trino, Iceberg, Airflow, and a full MLOps and observability suite into a cohesive stack you can launch with a single command. Now available on PyPI.

Dynamic DES v0.8.1 introduces native Data Lake integration. Learn how to use a single SimPy codebase to generate batch Parquet data for ML training, and seamlessly transition to streaming live Kafka events for production inference.

A comprehensive walkthrough from my session at Current London 2026 on capturing and visualizing data lineage across a production-style data stack.

In Part 2 of our series, we dive into the code and architecture of dynamic-des. Learn how to use the Switchboard pattern, mutable resources, and dynamic topic routing to transform a static model into a synchronized forecasting engine.

Providing direct access to big data engines like Spark and Flink often creates chaos. A gateway-centric architecture solves this by introducing a robust control plane. This article presents a detailed blueprint using Apache Kyuubi, a multi-tenant SQL gateway, to provision and manage on-demand Spark, Flink, and Trino engines. Learn how this model delivers true self-service analytics with centralized governance, finally resolving the conflict between user empowerment and platform stability.

The world of data is converging. The traditional divide between batch processing for historical analytics and stream processing for real-time insights is becoming increasingly blurry. Businesses demand architectures that handle both seamlessly. Enter the “Streamhouse” - an evolution of the Lakehouse concept, designed with streaming as a first-class citizen.

Today, we’ll introduce three key open-source technologies shaping this space: Apache Paimon™, Fluss, and Apache Iceberg. While each has unique strengths, their true power lies in how they can be integrated to build robust, flexible, and performant data platforms.

In the previous post, we started discussing a continuous integration/continuous delivery (CI/CD) process of a dbt project by introducing two GitHub Actions workflows - slim-ci and deploy. The former is triggered when a pull request is created to the main branch, and it builds only modified models and its first-order children in a ci dataset, followed by performing tests on them. The second workflow gets triggered once a pull request is merged. Beginning with running unit tests, it packages the dbt project as a Docker container and publishes to Artifact Registry. In this post, we focus on how to deploy a dbt project in multiple environments while walking through the entire CI/CD process step-by-step.

Continuous integration (CI) is the process of ensuring new code integrates with the larger code base, and it puts a great emphasis on testing automation to check that the application is not broken whenever new commits are integrated into the main branch. Continuous delivery (CD) is an extension of continuous integration since it automatically deploys all code changes to a testing and/or production environment after the build stage. CI/CD helps development teams avoid bugs and code failures while maintaining a continuous cycle of software development and updates. In this post, we discuss how to set up a CI/CD pipeline for a data build tool (dbt) project using GitHub Actions where BigQuery is used as the target data warehouse.