I built a proof of concept to explore a more disciplined approach: put a semantic layer between the model and the data, and let the agent reason over governed business concepts instead of guessing at raw schemas. It runs entirely locally on an open-source stack, and it happens to tie together two of my other projects along the way.

Why a Semantic Layer

A semantic layer defines models, relationships, reusable metrics, business rules, and approved query patterns independently of the language model. That separation is the whole point. When the definition of a metric lives in a governed contract rather than in a prompt, the agent cannot quietly redefine it, and the same question produces the same number every time.

I evaluated several options (WrenAI, Vanna AI, Nao, and MetricFlow) and selected WrenAI. It uses a Modeling Definition Language (MDL) to define models, relationships, views, calculated fields, and cubes as a structured, executable contract, and critically it generates SQL without executing it. That last property lets the orchestrator own authorization, validation, and controlled execution as a separate concern, which is exactly the boundary you want when adapting a system for enterprise governance.

Architecture

The stack is fully decoupled. An AI orchestrator interprets requests and talks to the semantic engine over the Model Context Protocol (MCP). The semantic engine plans deterministic queries from its business models and vector memory, and those queries execute against the physical lakehouse.

The diagram above traces a request end to end, and the table below maps each component to its role.

The infrastructure is launched with odctl, and the sample e-commerce dataset (customers, products, orders, order items, payments, returns) is generated with dynamic-des running a fast-forward clock and writing Parquet straight to SeaweedFS. Both projects earn their keep here: one stands up the platform, the other fills it with realistic data in seconds.

Two Layers That Keep the Model Honest

A major failure point for generative AI in data work is raw SQL hallucination. If you hand a model raw tables and ask it to join them on the fly, it will eventually produce a plausible but wrong metric. This system separates the problem into two layers so the model never sees a raw table.

Layer 1, the semantic compiler (WrenAI): supplies governed context (models, relationships, cubes, metrics), expands modeled SQL through MDL with dry_plan, validates against the live backend with dry_run, and executes through Trino with run_sql.

Layer 2, the autonomous agent (Strands): interprets the question, calls tools such as get_context and recall_queries, selects a cube or constructs modeled SQL, decides which execution tool to invoke, and maintains conversational context. Strands spawns WrenAI’s native MCP server as a subprocess and discovers the semantic tools automatically, so there is no hardcoded database wiring.

Because the agent only ever interacts with WrenAI’s governed semantic API, the set of available models and joins is constrained by design, which is what reduces the hallucination risk.

A Query, End to End

1. Context check: the orchestrator queries Mem0 on Valkey to pull long-term preferences.
2. Semantic translation: the request goes to WrenAI, which uses its MDL and LanceDB memory to map intent to accurate SQL.
3. Validation and execution: the agent validates the physical schema against live metadata, then executes against Iceberg through Trino and returns structured data.

For a governed question like “what is the revenue for delivered orders?”, the agent discovers the daily_revenue cube and maps “revenue” to gross_revenue. It never writes the SUM() or GROUP BY itself, because WrenAI compiles the deterministic SQL defined in the cube.

Memory That Turns Usage into Learning

Two forms of memory make the system better over time.

WrenAI builds a local LanceDB index over schema items, seed queries, and approved SQL examples, so it can retrieve only the relevant models for a question and offer proven queries as few-shot context. Most text-to-SQL systems treat every question as the first one. Adding a retrieval layer means successful work improves future work as the knowledge base grows.

On top of that, Mem0 gives the agent long-term memory for subjective business logic. Ask “how many high-value orders did we have yesterday?” with no memory, and the model will invent a threshold. Teach it once that high-value means a total amount greater than $1,000, and that definition is stored in Valkey and applied consistently from then on, without editing the underlying semantic model.

Evaluation

Measuring an agent means judging both its decision-making and its safety. The repository includes a golden test suite and an automated harness that spins up an isolated agent per case and grades the output with an LLM-as-a-judge, covering three things: cube discovery (does it route governed questions to governed metrics), raw table navigation (can it handle joins and aggregations when no cube fits), and hallucination prevention (does it refuse gracefully when asked for data that does not exist, such as a missing return_reason column, rather than fabricating it).

Where This Is Going

This is a foundational phase, deliberately scoped to batch data over cold storage. Real-time integration and dynamic routing between hot and cold storage are natural next steps, and the decoupled architecture is designed to absorb them.

The interesting takeaway for me is that a reliable data assistant is less about a smarter model and more about the boundaries you draw around it. A governed semantic layer, a clear split between generation and execution, and a memory that learns do far more for trustworthiness than any single prompt.

The full walkthrough, including infrastructure setup, semantic modeling, the orchestrator, and the evaluation harness, is on GitHub.

• GitHub: jaehyeon-kim/agentic-analytics-system

Here is what changed on the way to v0.11.1.

A Declarative SimulationContext API

The biggest change is how you write a simulation. Earlier versions exposed a low-level, imperative API where you wired up the environment, registry, connectors, and processes by hand. That is still available and fully supported for advanced control flows, but it is now the low-level paradigm rather than the default one.

The new Standard API is built around SimulationContext, a declarative builder that describes the whole simulation in one readable block:

 1from dynamic_des import SimulationContext, ConsoleEgress, LocalIngress
 2
 3# Schedule capacity to jump to 3 at t=10s, then drop to 2 at t=20s
 4app = (
 5    SimulationContext(sim_id="Line_A", factor=1.0, random_seed=42)
 6    .add_resource("lathe", current_cap=1, max_cap=5)
 7    .add_arrival("standard", dist="exponential", rate=1.0)
 8    .add_service("milling", dist="normal", mean=3.0, std=0.5)
 9    .add_ingress(LocalIngress(
10        schedule=[
11            (10.0, "Line_A.resources.lathe.current_cap", 3),
12            (20.0, "Line_A.resources.lathe.current_cap", 2),
13        ]
14    ))
15    .add_egress(ConsoleEgress())
16)

Simulation logic is then attached with decorators, so the mechanics of queuing, resource locking, duration sampling, and telemetry emission are handled for you:

 1@app.arrival_loop("standard")
 2def arrival_process(context: SimulationContext):
 3    task_id = 0
 4    while True:
 5        yield context.wait_for_arrival("standard")
 6        context.spawn(work_task(task_id))
 7        task_id += 1
 8
 9@app.task(service_id="milling", resource_id="lathe")
10def work_task(task_id: int):
11    return {"part_id": task_id}
12
13app.run(until=25.0)

The value here is separation of concerns. The builder declares parameters, connectors, and configuration, while the decorators express only the behavior that is unique to your model. Swapping a console for Kafka, or a fast-forward clock for a real-time one, does not touch the simulation logic at all.

Postgres and Redis Connectors

Dynamic DES uses a Switchboard Pattern that decouples data sourcing from simulation logic. Background threads handle heavy I/O, a central registry flattens incoming data into dot-notation paths, and SimPy resources wake up only when the registry signals a change. Because that boundary is clean, adding a new backend is mostly a connector implementation.

Recent releases added two important ones, each with matching ingress and egress:

• PostgreSQL: stream telemetry and lifecycle events into a relational store, or drive simulation parameters from a table. Useful when your control plane or your analytics already live in Postgres.
• Redis: a low-latency, in-memory option for pushing state out to dashboards or reading live parameter updates, alongside the existing Kafka path.

These sit next to the connectors that were already there, so a single simulation can now ingest from and emit to Kafka, Redis, or PostgreSQL, and export historical data to local disk, AWS S3, Google Cloud Storage, Azure Blob, or SeaweedFS through PyArrow’s virtual filesystem, all with strict schema drift prevention.

Strict Data Contracts

Regardless of the backend, outbound payloads are validated with Pydantic, so external consumers receive a predictable shape. There are two stream types. Telemetry carries scalar metrics such as utilization and queue length:

1{
2  "stream_type": "telemetry",
3  "path_id": "Line_A.resources.lathe.utilization",
4  "value": 85.5,
5  "sim_ts": 120.5,
6  "timestamp": "2023-10-25T14:30:00.000Z"
7}

Events carry discrete lifecycle transitions such as a task arriving, queuing, or finishing:

 1{
 2  "stream_type": "event",
 3  "key": "task-001",
 4  "value": {
 5    "status": "finished",
 6    "duration": 45.2,
 7    "path_id": "Line_A.service.lathe"
 8  },
 9  "sim_ts": 125.0,
10  "timestamp": "2023-10-25T14:30:04.500Z"
11}

Stability

The v0.11.1 patch closed out threading race conditions in the egress providers during environment teardown. When a simulation shuts down, the background I/O threads now drain and stop cleanly instead of racing the main thread, which matters for both test suites and long-running twins.

Try It Out

Everything above is available now:

1# Core library
2pip install dynamic-des
3
4# Everything: Kafka, Redis, Postgres, dashboard, Avro, Parquet
5pip install "dynamic-des[all]"

The repository ships zero-setup demos for the local, Kafka, Postgres, and Redis paths in both the declarative and imperative styles.

• PyPI: pypi.org/project/dynamic-des
• GitHub: jaehyeon-kim/dynamic-des
• Documentation: jaehyeon.me/dynamic-des

I built odctl to make that afternoon disappear. It is a curated collection of open-source data technologies wrapped in a small CLI, and it is now available on PyPI.

What odctl Is

odctl provides a cohesive, Docker-based blueprint for experimenting with modern data architecture and MLOps locally. Instead of hand-writing Compose files, you describe what you want in terms of profiles, and the CLI resolves the dependency graph, network routing, and configuration for you.

Install it as an isolated CLI tool:

1uv tool install odctl
2# or
3pipx install odctl

Bundled Technologies

Rather than shipping one monolithic cluster, the stack is organized into profiles you can launch independently or together:

• Messaging: Kafka (KRaft), Schema Registry (Karapace), Kafka Connect
• Stream Processing: Apache Flink
• Data Processing: Apache Spark
• Analytics: ClickHouse, Trino, Metabase
• Orchestration: Apache Airflow
• MLOps: MLflow, Ray Serve
• Metadata: OpenMetadata
• Observability: Prometheus, Alertmanager, Grafana
• Lineage: OpenLineage, Marquez
• Foundational storage and catalog: PostgreSQL (pgvector), SeaweedFS (S3), Iceberg REST Catalog, Valkey, Apache Fluss

That covers ingestion, streaming, batch, OLAP, orchestration, machine learning, governance, and observability, all from a single tool.

Dependencies Resolve Themselves

Manual Compose setups fall apart because the interesting engines depend on foundational infrastructure that you have to remember to start first. odctl treats those dependencies as a graph and walks it for you.

Getting a full data engineering environment running takes three steps:

1# 1. Copy the default Compose files and configs into a local .odctl workspace
2odctl init
3
4# 2. See what you can launch
5odctl list
6
7# 3. Launch the streaming and batch engines
8odctl up kafka-lite flink1-lite spark-lite

You do not need to memorize what depends on what. When you ask for Kafka, Flink, and Spark, the CLI detects that they require foundational infrastructure and quietly launches PostgreSQL, SeaweedFS (S3), and the Iceberg REST Catalog before starting the compute engines.

A Small, Predictable Command Surface

Commands are grouped by intent, and every one accepts --help:

• Inspect: odctl list, odctl explain <profile>, odctl ps, odctl info
• Lifecycle: odctl pull, odctl up, odctl down
• Manage: odctl logs, odctl restart

A few examples:

1# View all profiles and exposed ports
2odctl list -d
3
4# See exactly what the kafka profile provisions
5odctl explain kafka-lite
6
7# Complete teardown and wipe all data
8odctl down --all --volumes

Hackable by Design

odctl does not hide the underlying configuration from you. Running odctl init generates a local ./.odctl/ workspace containing everything that powers the stack:

• compose-*.yml: the actual Docker Compose definitions. Edit these to change exposed ports, adjust memory limits, or inject environment variables.
• registry.yml: the internal dependency graph.
• .env: shared environment variables such as default credentials and timezones.

The CLI always prioritizes the files in your local workspace, so you can tweak freely. If you break something, odctl init --force restores the pristine defaults.

Try It Out

odctl is on PyPI now. If you build demos, write technical content, or just want a realistic data platform to experiment with, it removes the setup tax so you can get to the interesting part.

• PyPI: pypi.org/project/odctl
• GitHub: jaehyeon-kim/odctl

It is licensed under Apache 2.0, and contributions are welcome.

Maintaining two separate simulation codebases, one for generating training data and another for streaming live events, introduces friction, schema mismatches, and duplicated engineering effort.

Dynamic DES was originally designed to solve the real-time problem by acting as a control plane for SimPy models. With the release of v0.8.1, it now natively handles high-performance historical data generation, allowing you to use the exact same simulation code for both batch training and live inference.

Step 1: Batch Training (Historical Fast-Forward)

To generate training data efficiently, the simulation must be decoupled from the real-world clock. By setting the environment’s factor to 0.0, the engine initiates a “fast-forward,” executing logical events as fast as the CPU allows.

To capture this massive data throughput without exhausting memory or fragmenting storage, v0.8.1 introduces ParquetStorageEgress and JsonlStorageEgress. Built on PyArrow, these connectors handle the complexities of Data Lake ingestion natively:

• Schema Drift Protection: The engine infers the schema from the initial data batch and strictly casts all subsequent records. This guarantees deterministic data types, preventing downstream ML pipelines from failing due to unexpected schema mutations.
• Native Data Lake Integration: Data is automatically chunked, compressed, and written directly to local disk, AWS S3, or SeaweedFS clusters via PyArrow’s virtual filesystem.

You can simulate months of system operations in seconds, outputting optimized Parquet chunks ready for model training.

Step 2: Live Inference (Production Streaming)

Once the model is trained on the historical Parquet data, transitioning to production testing requires zero changes to your core simulation logic.

By changing the environment’s factor from 0.0 to 1.0, logical time synchronizes with wall-clock time. You simply swap the ParquetStorageEgress for a KafkaEgress. The simulation now runs in real-time, streaming the exact same schema to your message broker, providing a live data feed for your deployed ML model to consume.

This provides an end-to-end data engineering toolkit for simulation-based Machine Learning, from historical batch generation to live streaming pipelines.

Try it out

The v0.8.1 release and the new storage connectors are available now. You can view the source code and run the historical data generation example below.

• GitHub Repository: jaehyeon-kim/dynamic-des
• Documentation & Examples: Historical Data Generation Guide

Building End-to-End Data Lineage

With Kafka, Flink, and Spark

What is Data Lineage?

The journey of data - where it comes from, how it’s transformed, and where it ends up.

Why Data Lineage Matters

• Debugging & Root Cause Analysis: Quickly trace production bugs back to the source.
• Impact Analysis & Governance: See downstream dependencies before changing schemas.
• Compliance & Audit: Document data provenance for strict regulatory requirements.
• Trust & Reliability: Increase stakeholder confidence in data products.

What is OpenLineage?

An open standard for capturing lineage metadata from jobs in execution.

Two Lineage Paradigms

Batch Lineage (Retrospective)

• Data: Bounded Sets
• Lifecycle: Finite, Scheduled
• Capture: At Job Completion
• Result: Historical Audit Trail

Streaming Lineage (Operational)

• Challenge: Unbounded Streams
• Challenge: Continuous Jobs
• Opportunity: Capture During Execution
• Opportunity: A Live, Observable System

Kafka: Lineage with Connect

Use a custom Single Message Transform (SMT) as a “pass-through” lineage agent.

• Lifecycle Hook: Intercepts connector states (RUNNING, FAIL, COMPLETE) without altering records.
• Column-Level Depth: Resolves fields via Avro schemas in the Schema Registry.
• Consistent Namespacing: Normalizes physical dataset naming (e.g., kafka://..., s3://...) for job linking.

Kafka Pipeline Mapping

One lineage job tracking data flow per active connector.

Flink: Integration Patterns

Native JobListener(DataStream API)

• Method: OpenLineageFlinkJobListener.
• Pros: Simple, out-of-the-box.
• Cons: Fails to capture final ABORT transition.

Manual Orchestration(Table API / SQL)

• Method: OpenLineage Java client.
• Pros: Full lifecycle event tracking.
• Cons: Requires custom wrapper code.

Flink Pipeline Mapping

One lineage job mapped directly per active Flink application cluster.

Spark: Final Picture

Batch Spark job reading from Flink-written Iceberg tables.

• Agent: Tracked via OpenLineage Java agent in spark.extraListeners.
• Query Plan: Auto-detects inputs/outputs from Logical/Physical plans.
• Granular Actions: Parent job with child jobs per Spark action.
• 💡 Namespace Alignment: Engines must agree on the same physical root namespace (e.g., s3://warehouse).

Spark Pipeline Mapping

One lineage job registered for each execution action block.

Conclusion: Key Takeaways

Bridging real-time events and historical batch processing.

• Choose the Right Pattern
  • Trade-off: Simplicity vs. Tracing Durability.
  • Standard: Use Native Listeners.
  • Critical: Use Manual Orchestration for strict lifecycle tracking.
• Align Namespaces
  • Essential for cross-tech lineage.
• Start Small
  • Instrument one critical pipeline first.
  • Establish metadata visibility, then scale outward.

Project Repositories & Guides

• OpenLineage Project
• Data Linage Lab

In modern data ecosystems, understanding data provenance, transformation steps, and final destinations is necessary for governance and root-cause analysis. My session, Building End-to-End Data Lineage with Kafka, Flink, and Spark, detailed a way to capture this metadata across parallel pipelines using OpenLineage.

Presentation Slides

Below are the slides from my session, rendered natively here on the blog.

You can use your arrow keys to navigate the slides above, or click the “View Full Screen” button for the complete experience.

Session Breakdown: Tracking the Data Lifecycle

The presentation focused on tracking data from a single Kafka topic as it distributed across concurrent architectural paths. To demonstrate the lineage graph, we analyzed a standard production stack.

1. Real-Time & Archival Fan-Out

The architecture begins with a primary Kafka topic. From there, the data splits into distinct operational domains:

• Archival: A Kafka S3 Sink Connector handling raw data offloading to object storage.
• Live Analytics: A real-time Flink DataStream job consuming events for stateful processing.
• Data Lakehouse: A Flink Table API job ingesting the continuous stream into an Apache Iceberg table for query engines.

2. Completing the Picture with Spark

To demonstrate a complete end-to-end lifecycle, the session traced the lineage as a batch Apache Spark job consumed from the populated Iceberg table to generate downstream aggregations.

3. Instrumenting with OpenLineage

Visualizing this multi-path journey, including column-level details, was achieved using Marquez as the lineage repository and visualization layer. The metadata extraction was handled through OpenLineage. Integrating these systems to output a unified lineage graph requires specific strategies:

• Kafka Connect: Lineage is established at the connector level using a custom Single Message Transform (SMT) to capture operational state without altering the payload.
• Apache Flink: Two distinct patterns were evaluated: a low-overhead listener-based approach, and a manual orchestration method necessary for capturing application cancellations.
• Apache Spark: Spark’s extraListeners were configured to auto-detect inputs and outputs, linking the batch jobs to upstream Flink outputs via aligned physical namespaces.

Moving Forward

By instrumenting event streams, streaming compute, and batch processing engines with a unified standard like OpenLineage, organizations can establish observable and reliable data architectures.

Thank you to everyone who attended the session at Current London. For those unable to attend, the slides are provided above for reference.

In Part 1, we established that a true Hybrid Digital Twin does more than just mirror reality. It actively forecasts the future by running a simulation against live operational states.

If you have ever tried to build one of these systems from scratch, you immediately hit a fundamental architectural clash.

Standard simulation clocks (like those in traditional SimPy implementations) are logically synchronous and not designed to handle high-frequency asynchronous I/O without explicit decoupling. They step through logical time deterministically. Real-world telemetry (like a Kafka stream or a Redis Pub/Sub channel) is asynchronous and non-deterministic in arrival timing and ordering at the system level. If you naively wire a live data feed directly into a standard simulation loop, you will stall the simulation loop while awaiting network I/O. The simulation drifts from real-world time, making your live twin useless.

To solve this, the compute layer must be explicitly decoupled from the network layer. This is the exact design problem the open-source dynamic-des package addresses.

Core Architecture: Switchboard Pattern

dynamic-des operates as a real-time control plane wrapped around a discrete-event simulation core. It bridges the asynchronous gap using the Switchboard Pattern, which isolates the network from the math:

1. Connectors (I/O Layer): Background async I/O workers handle heavy network traffic (Kafka, Redis, Postgres) entirely independently of the simulation clock.
2. Simulation Registry (Switchboard): This is the central state manager. It intercepts incoming network data, safely casts types, and flattens complex system schemas into addressable paths that act as reactive state bindings, allowing any component in the simulation to subscribe to and react to state transitions.
3. Simulation Objects: The actual mathematical models and resources that consume these state transitions.

Because the background I/O workers and the simulation clock communicate through thread-safe queues, network latency does not block your forecasting logic. However, decoupling is not a magic bullet. If ingestion rates exceed processing rates, drift can still occur, which is why system observability is critical.

Physics of Change: Mutable Resources

With the data flowing safely, the simulation objects themselves must be able to react. In a traditional DES model, resources are static. If you initialize simpy.Resource(capacity=5), it stays 5 forever. In the real world, machines break, operators go on break, and routing logic shifts dynamically.

Instead of polling the network, dynamic-des introduces mutable resources (DynamicResource, DynamicContainer, DynamicStore) that adapt safely:

• Growing is easy: If capacity increases from 5 to 10, new tokens are immediately added to the pool, instantly unblocking pending requests.
• Shrinking is complex: If capacity drops from 5 to 2 while 4 tasks are actively being processed, what happens? Work-In-Progress (WIP) is never destroyed. The resource enters a temporary over-capacity state. It waits for the active tasks to naturally finish, intercepting their returned tokens until the physical pool matches the new, smaller capacity limit.

System Observability

When running a real-time simulation, your biggest operational risk is simulation drift. If your compute layer cannot process the incoming event stream fast enough, the simulation clock will fall behind relative to event time or wall-clock time.

To solve this, dynamic-des includes built-in observability. The egress layer automatically calculates this drift and continuously streams a system.simulation.lag_seconds telemetry metric. This makes it easy to set up Grafana dashboards and trigger alerts if the forecasting engine begins lagging behind reality.

Designing for Failure Modes

Moving from a static offline script to a live, event-driven twin introduces distributed systems problems. A production-grade deployment must account for specific failure modes:

• Backlog Growth: Queues prevent the network from blocking the simulation loop, but if your computational logic is too heavy, the event backlog will grow and cause severe simulation drift.
• Event Ordering: In distributed systems, events can arrive out of order. You must implement proper Kafka partitioning strategies to ensure causal events are processed sequentially within a partition; cross-partition ordering must be handled at the application level.
• State Recovery: If a simulation node crashes, it must be able to restart, rebuild its state tree from the Registry or by replaying the event log, and resume processing from the correct Kafka offsets without losing WIP logic.

Developer Journey: From Laptop to Cluster

A good digital twin framework must be portable. You should be able to validate your business logic locally before introducing the complexity of enterprise message brokers. dynamic-des achieves this by making the I/O layer entirely pluggable.

Phase 1: Local Deterministic Testing

Before touching Kafka, you can use LocalIngress to schedule mutations deterministically. This allows you to unit-test exactly how your factory floor reacts to a breakdown at a specific timestamp.

 1from dynamic_des import (
 2    DynamicRealtimeEnvironment, DynamicResource, 
 3    LocalIngress, ConsoleEgress, SimParameter, CapacityConfig
 4)
 5
 6# 1. Define the initial state
 7params = SimParameter(
 8    sim_id="Line_A",
 9    resources={"lathe": CapacityConfig(current_cap=1, max_cap=5)},
10)
11
12# 2. Schedule a deterministic capacity jump (1 -> 3) at t=5.0s
13ingress = LocalIngress([
14    (5.0, "Line_A.resources.lathe.current_cap", 3)
15])
16egress = ConsoleEgress()
17
18# 3. Initialize Environment & Registry
19env = DynamicRealtimeEnvironment(factor=1.0)
20env.registry.register_sim_parameter(params)
21env.setup_ingress([ingress])
22env.setup_egress([egress])
23
24res = DynamicResource(env, "Line_A", "lathe")
25
26env.run(until=10)

Notice the factor parameter in the environment setup. The environment can run in real-time or accelerated logical time, depending on the configured factor. A factor of 1.0 syncs the simulation strictly to the wall-clock, while lower factors accelerate it for rapid scenario testing.

Phase 2: Live Kafka Integration

Once your logic is sound, transitioning to an operational digital twin requires changing exactly two lines of code. You swap the local connectors for Kafka connectors.

While this example is kept clean for readability, real production deployments will naturally require additional configurations for retries, authentication, and backpressure handling.

 1from dynamic_des import KafkaIngress, KafkaEgress
 2
 3# Swap connectors: no changes needed to business logic
 4ingress = KafkaIngress(
 5    topic="factory-config", 
 6    bootstrap_servers="broker.internal:9092"
 7)
 8
 9egress = KafkaEgress(
10    telemetry_topic="sim-vitals", 
11    event_topic="sim-events", 
12    bootstrap_servers="broker.internal:9092"
13)
14
15env.setup_ingress([ingress])
16env.setup_egress([egress])
17
18env.run()

Evolution into a State-Machine Engine

Traditional Discrete-Event Simulation is strictly quantitative, focusing mostly on queue lengths, processing times, and resource capacities. However, dynamic-des has evolved beyond this to blend DES with reactive programming, behaving much like a state-machine engine.

By utilizing the variables registry and the wait_for_change() observer pattern, developers can track arbitrary logical states. Instead of just yielding on time delays, simulation processes can yield on external network events. A Kafka payload no longer just updates a number. It acts as a state-transition trigger that dictates the logical flow of the entire application.

This flexibility means digital twins can incorporate sophisticated modeling well beyond tracking discrete items in a queue. You can map real-time telemetry directly to custom mathematical logic. For a practical example, you can view the Hot Strip Mill: Real-Time Online Machine Learning & Digital Twin project, which demonstrates how to build a real-time tracking dashboard for steel manufacturing using advanced physical simulations.

💡 Domain-Agnostic Applications

Because it behaves as a reactive state engine, dynamic-des is not restricted to factory floors. I am currently building out additional demo projects to highlight its versatility across entirely different domains:

• Continuous-Time Crypto Trading Bot: This demo proves the package can handle high-velocity financial simulations. It utilizes the variables registry to track live BTC prices and portfolio balances, while using dynamic resources to model exchange API rate limits. Concurrent processes simulate the market maker (random walk price updates), strategy monitor (moving average crossovers), and order execution (yielding realistic network latency delays).
• RPG Adventure Game State Machine: This demo highlights the stochastic and time-delay strengths of the engine by modeling complex entity interactions over continuous time. Primitive game states (player HP, enemy HP, current room) live in the variables registry. Continuous processes handle room exploration with travel time delays and active combat loops. The combat process yields attack cooldown timeouts and uses the global Sampler for randomized damage, tracking causality back to the root encounter event.

Enforcing Data Contracts and Routing

A hybrid twin is only as useful as the data it outputs. To ensure downstream analytics systems do not break, dynamic-des natively enforces strict data contracts and provides granular control over serialization.

Type Safety and Pydantic Integration

Data hygiene works in both directions. On the ingress side, the framework performs dynamic type casting to prevent malformed sensor data from crashing your active simulation.

On the egress side, you can yield strongly typed Pydantic models directly from your simulation logic. You do not need to manually convert objects to dictionaries. The egress layer automatically detects, extracts, and serializes your models:

 1from pydantic import BaseModel
 2from dynamic_des import DynamicRealtimeEnvironment
 3
 4class TaskEvent(BaseModel):
 5    path_id: str
 6    status: str
 7
 8def work_task(env: DynamicRealtimeEnvironment, task_id: int):
 9    # Pass the Pydantic model directly to the framework
10    event = TaskEvent(path_id="Line_A.lathe", status="queued")
11    env.publish_event(f"task-{task_id}", event)

Pluggable Routing and Avro Schemas

In an enterprise architecture, you rarely dump all events into a single Kafka topic. You might want standard telemetry routed as lightweight JSON, while high-velocity machine learning payloads are serialized as binary Avro.

dynamic-des handles this through pluggable topic routers and serializers. You can map specific topics to a ConfluentAvroSerializer or a GlueAvroSerializer, and any unmapped topics will safely fall back to default JSON.

Because hardcoding Avro schemas is prone to drift, the framework natively supports dataclasses-avroschema. This allows you to auto-generate your schema directly from your Pydantic model and route it seamlessly:

 1from pydantic import Field
 2from dataclasses_avroschema.pydantic import AvroBaseModel
 3from dynamic_des import KafkaEgress
 4from dynamic_des.connectors.egress.kafka import ConfluentAvroSerializer
 5
 6# 1. Define your model and auto-generate the schema
 7class MLPrediction(AvroBaseModel):
 8    """High-velocity ML payload"""
 9    task_id: str = Field(...)
10    confidence: float = Field(...)
11
12    class Meta:
13        namespace = "com.dynamic_des.ml"
14        schema_name = "PredictionEvent"
15
16# 2. Define a topic router to separate standard events from ML events
17def ml_topic_router(data: dict) -> str:
18    if data.get("value", {}).get("event_type") == "prediction":
19        return "ml-predictions"
20    return "sim-lifecycle"
21
22# 3. Configure the Egress Connector
23egress = KafkaEgress(
24    bootstrap_servers="localhost:9092",
25    topic_router=ml_topic_router,
26    topic_serializers={
27        # Map the auto-generated Avro serializer to the ML topic
28        "ml-predictions": ConfluentAvroSerializer(
29            registry_url="http://localhost:8081",
30            schema_str=MLPrediction.avro_schema() 
31        ),
32    }
33)

Moving to Production

By decoupling network I/O from the simulation clock, enforcing strict data schemas, and blending DES with reactive programming, dynamic-des provides a clean architectural pattern for modern event-driven systems.

This enables a new class of systems beyond static models and passive dashboards. You can build resilient, synchronized forecasting engines that adapt to reality the moment it changes.

To explore the source code, view the full API documentation, or test out the zero-setup Docker examples, check out the dynamic-des repository on GitHub.

There is a project by Dassault Systèmes called the Living Heart that illustrates the trajectory of this technology. Instead of relying on standard 2D scans, surgeons can pull up a high-fidelity 3D model of a patient’s heart that simulates blood flow, mechanics, and electricity based on imaging-derived reconstructions and population-based physiological calibration.

While the Living Heart is closer to a personalized, highly-parameterized simulation than a continuously streaming IoT system, it highlights the core philosophy of a modern digital twin: moving past static CAD files to create models that are fundamentally aligned with a specific, real-world physical instance.

Industry Applications

While manufacturing initially drove this technology, digital twins, systems that periodically or continuously align a digital model with physical reality, exist on a wide spectrum. Some are event-driven, some are periodically synchronized, and others are simulation-first with occasional calibration.

• Smart Manufacturing: On automated factory floors, robotic arms stream telemetry to virtual models. This allows plant managers to track maintenance needs and test routing scenarios based on the current state of the factory.
• Supply Chain and Logistics: Global supply chains use digital twins to track everything from warehouse inventory levels to fleet movements, achieving end-to-end visibility.
• Cybersecurity and Infrastructure: Network digital twins are increasingly used to replicate network topology and traffic models. Security teams use these digital replicas as testbeds to safely test rate-limiting and AI behavioral rules against simulated DDoS attacks without risking live infrastructure.
• Next-Generation Telecommunications: Telecom operators build digital twins of their networks to manage dynamic bandwidth allocation and network slicing, analyzing traffic spikes across thousands of nodes before they cause outages.

Reality Gap: Simulations vs. Digital Twins

The tech industry frequently uses the terms “simulation” and “digital twin” interchangeably. Clarifying how these systems handle state and time is critical for evaluating modern control architectures. To understand this practically, let’s look at a factory floor where “Machine B” suddenly breaks down.

Here is how different architectural patterns handle that exact event:

1. Traditional Simulation (Isolated Execution)

A simulation is a predictive model designed to explore “what if” scenarios.

• Scenario: On Monday morning, an engineer runs a Discrete-Event Simulation (DES) of the factory. On Tuesday, Machine B physically breaks.
• Limitation: A traditional simulation run is an isolated execution model. Unless the system is built to re-seed its parameters from a database and trigger a new batch run, the ongoing execution remains unaware of the real-world breakdown. It will incorrectly continue assuming Machine B is running at full capacity.

2. Operational Digital Twin (State Synchronization)

An operational digital twin extends a model by aligning it with reality using time-series databases and IoT connectivity.

• Scenario: The moment Machine B breaks, sensors ping the database. The twin immediately updates the machine’s live status to “Offline.”
• Limitation: In practice, a digital twin is usually a stack of overlapping layers. The core synchronization layer provides state fidelity, while separate analytics layers handle forecasting. However, if a system only utilizes the synchronization layer as a dashboard without integrating predictive models, it often relies on separate analytics modules, external simulation tools, or human operators to forecast future cascading consequences.

3. Event-Driven Hybrid Execution (Dynamic Adaptation)

Advanced architectures blend these layers into unified ecosystems, feeding real-world states directly into predictive engines. While many systems accomplish this through windowed re-execution or micro-batch propagation, another opinionated architectural pattern is event-driven mutation.

• Scenario: The operational twin registers the breakdown and emits a live Kafka event. A background hybrid simulation engine ingests that event while it is actively running. It dynamically mutates Machine B’s capacity parameter to zero and recalculates its logic on the fly. It immediately outputs new telemetry: “Machine B’s queue is backing up, routing logic must shift, and a bottleneck will form at Machine C in exactly 45 minutes.”

Upgraded Simulation Fallacy

Because of overlapping definitions, organizations often fall into what we might call the “Upgraded Simulation Fallacy”. They purchase standard simulation software, attach a 3D dashboard, and market the project as a digital twin. If the system lacks the data pipelines to periodically align with live operational states, it sits in an ambiguous middle zone, delivering some monitoring value, but falling short of a fully integrated twin ecosystem.

Transitioning to an operational twin introduces strict engineering realities:

• Cost and Complexity: Building them requires heavy investments in sensors, edge computing, and complex data architecture.
• Data Pipelines and Standardization: Getting disparate sensors to speak the same language and pipe into a central engine requires robust schemas (like Pydantic, Avro, or Protobuf).

This last point is where many hybrid projects stall. If you naively force real-time asynchronous updates directly into a standard simulation loop, the engine’s internal clock can bottleneck while waiting for network requests. To avoid this design anti-pattern, teams must explicitly decouple their network I/O from the simulation execution or rely on rolling micro-batches.

Taking Control with dynamic-des

Organizations shouldn’t have to choose between an isolated simulation and an operational dashboard. The goal is to combine them, running “what-if” forecasts against synchronized, real-world states.

But if you want to implement an event-driven architecture where a running mathematical model ingests Kafka streams and alters its parameters on the fly without stopping, you face a complex I/O challenge.

One purpose-built approach to solving this specific pattern is the dynamic-des package. In Part 2 of this series, we will look at how this open-source tool uses a Switchboard pattern and dynamic mutable resources to turn a static SimPy script into a synchronized, event-driven engine.

Imagine using a rolling pin to flatten out a thick piece of dough. A Hot Strip Mill does the exact same thing, but with glowing red-hot steel slabs (often heated over 1000°C) and massive mechanical rollers. The steel is passed through a series of these rollers, crushing it down from a thick block into a long, thin sheet.

Calculating the exact Rolling Force required to crush the steel is critical. If the machine pushes too hard, it can severely damage the rollers; if it doesn’t push hard enough, the steel doesn’t reach the target thickness. Because the rollers are constantly grinding against raw steel, their physical shape slowly degrades over time. As the machinery wears down, the legacy mathematical formulas used to predict that perfect force slowly become inaccurate. This physical degradation is the root of the Concept Drift our real-time ML pipeline is solving.

In heavy industrial manufacturing, such as steel hot strip rolling, deterministic physics formulas are the traditional standard for calculating the exact force required to deform a slab of steel. However, these pure physics models share a fatal flaw: they assume a pristine factory state. As physical rollers grind against red-hot steel over hours of production, they experience mechanical wear.

As the machinery degrades, the actual physical force required drifts away from the theoretical prediction. In data science, this is a classic manifestation of Concept Drift.

To tackle this, I recently built a real-time, fault-tolerant Online Machine Learning (OML) pipeline and Digital Twin. By combining Apache Kafka, Apache Flink (written in Kotlin), and the Massive Online Analysis (MOA) framework, the system learns the new physical reality of the worn machinery on the fly, autonomously correcting the physics baseline safely behind a deterministic Shadow Mode router.

Here is an overview of the architecture and the engineering challenges solved along the way. The complete source code for this project, including the Flink stream processor and the Python Digital Twin simulation, is available on GitHub.

Architecture at a Glance

The project is split into three highly decoupled domains:

1. Digital Twin (Python): Utilizing the Dynamic DES Python package, this layer generates synthetic rolling events, applies simulated mechanical wear, calculates theoretical/actual physics, and pushes the data to Kafka.
2. Message Broker (Kafka): Handles the asynchronous, high-throughput streaming of prediction requests and delayed ground-truth target forces.
3. Stream Processor (Flink/Kotlin): The core engine. It aligns asynchronous streams, trains the machine learning models dynamically, evaluates safety guardrails, and sinks metrics to ClickHouse for evaluation in real-time.

Moving data from the Digital Twin through Kafka to Flink looks straightforward in a diagram, but real-world industrial physics and distributed systems are highly complex. To make this pipeline robust enough for the unpredictability of a live factory floor, we had to overcome four major engineering challenges.

Challenges

Taking an online machine learning model out of the lab and deploying it into a live industrial environment introduces a unique set of hurdles. From managing the physical delays of factory sensors to ensuring the model never commands a physically unsafe action, here is a breakdown of the four primary engineering challenges we tackled to make this architecture more robust.

Challenge 1: Asynchronous Industrial Streams

Industrial data streams are inherently asynchronous. The factory floor requests a prediction before the steel enters the rolling stand (Event A), but the ground-truth sensor data confirming the actual required force arrives after the steel is crushed (Event B).

To execute machine learning, these two events must be perfectly joined. However, network latency and partition skew in Kafka mean that these events might arrive out of order.

In Flink, this is solved using a KeyedCoProcessFunction. By keying the streams on a composite Slab ID + Pass Number, Flink guarantees both events route to the exact same TaskManager. The EventMatchProcessFunction utilizes Flink’s ValueState to buffer whichever event arrives first. It then registers a processing-time timer (Time-To-Live). If the matching event arrives, they are joined and emitted. If the timer expires (e.g., a physical sensor failure), the orphaned state is safely purged to prevent memory leaks.

Challenge 2: Online Residual Learning

Instead of training a batch model offline on historical data, which would instantly become obsolete the moment the rollers wore down further, the Flink pipeline trains on the continuous stream of slabs using a strict Test-then-Train (prequential) paradigm.

Crucially, the ML models do not predict the absolute rolling force from scratch. Instead, they utilize Residual Learning. They predict the residual error between the theoretical formula and the actual physical force.

To execute this, the pipeline integrates the Java MOA framework. The primary production model is AMRules (Adaptive Model Rules), a streaming rule-learning algorithm. It builds an ensemble of rules that calculate a linear combination of physical attributes, continuously updating weights via the Delta rule.

Unlike static models, AMRules runs an internal Page-Hinkley test to detect sudden physical shocks (like a roller bearing breaking) and instantly prunes obsolete rules, allowing rapid convergence to new physical realities. Furthermore, because ML models require normalized inputs, the pipeline implements Welford’s Online Algorithm in a custom Flink state object to calculate streaming Z-scores on the fly without needing to load full datasets into memory.

Challenge 3: Shadow Mode Router

Industrial Machine Learning cannot operate without strict safety boundaries. A model error that generates excessive rolling force could severely damage a multi-million-dollar rolling stand or create a major production bottleneck in the factory.

Before any ML-adjusted prediction is allowed to influence the factory floor, it must pass through a deterministic Shadow Mode Router consisting of two guardrails:

1. Stateless Mechanical Limits: Calculates the residual difference between the model’s requested force and the physics baseline. If the model requests a deviation outside of physical safety bounds (e.g., > +25% or < -20%), the prediction is instantly rejected.
2. Stateful Trust Score: Using an Exponentially Weighted Moving Average (EWMA), the router continuously tracks the Absolute Percentage Error (APE) of both the model and the pure physics baseline. If the model’s average error trails the physics baseline by a defined “Trust Deficit” margin, it is benched.

If either guardrail is triggered, the router safely falls back to the deterministic Physics Baseline. The ML model continues to train in the background (Shadow Mode) until it re-learns the physical reality, improves its EWMA trust score, and is autonomously promoted back to active control.

Challenge 4: Checkpointing Complex ML State

For Flink to provide exactly-once processing guarantees, it must asynchronously snapshot the state of operators to a durable backend like RocksDB.

While simple tracking states serialize perfectly into basic Flink ValueState<Double>, the AMRules model is a deeply complex, dynamic tree structure. Allowing Flink’s default Kryo serializers to traverse this massive object graph during checkpointing causes severe performance degradation and frequent serialization crashes.

To bypass this, the MoaEvaluationProcessFunction interacts with the MOA model as a standard Java object in memory for high-performance execution. However, upon every state update, it manually serializes the model into a raw byte array (ValueState<ByteArray>) using Java’s native ObjectOutputStream. When Flink triggers a checkpoint, it simply flushes these pre-serialized byte arrays to disk. Upon recovery, Flink deserializes the bytes, instantly restoring the exact computational brain-state of the algorithm.

System in Action

You can easily spin up the entire architecture locally to see the system in action. The project repository includes a complete Docker Compose environment that provisions Kafka, Flink, and ClickHouse.

To get started, simply clone the repository, build the Flink application using the provided Gradle wrapper, and bring up the infrastructure. Once the environment is running, you can launch the Python-based data generator to simulate the steel rolling process and open the NiceGUI control dashboard to monitor the machine learning metrics in real-time.

For detailed, step-by-step instructions on cloning the repository, configuring licenses, and deploying the Flink job, refer to the Getting Started section in the repository.

Simulation Scenarios

Using the NiceGUI Dashboard, you can actively manipulate the physical state of the Digital Twin to observe the Flink pipeline’s reaction in real-time.

Here is how the pipeline handles different industrial scenarios:

Abrupt Drift (Mechanical Shock)

Simulates a sudden mechanical failure (e.g., a roller bearing breaking), instantly altering the physics of the mill.

• Simulation Settings: Trigger Abrupt Shock (Wear Level: 60.0).
• Observation: The pure physics baseline error instantly spikes and remains high (often >10% APE) because the physical reality no longer matches the math. The AMRules model initially spikes alongside it, but its Page-Hinkley change detector immediately drops obsolete rules, allowing it to rapidly converge back to lower error as it learns the new broken state.

Gradual Drift (Standard Wear)

Simulates the continuous, bi-directional cycle of slow roller degradation and subsequent maintenance recovery over hours of production.

• Simulation Settings: Gradual Wear (Step Size: 5.0 units, Frequency: 30 seconds).
• Observation: The physics baseline error slowly and persistently creeps upward/download over time (e.g. ranging from 2% to 7% APE) as the wear level drifts gradually. The AMRules model gracefully tracks this changing reality, updating its linear weights incrementally to maintain a smooth error rate.

No Drift (Pristine State)

Simulates a pristine factory state, such as immediately after a maintenance shift replaces the rollers.

• Simulation Settings: Wear Level: 0.0.
• Observation: The physical reality of the factory floor perfectly aligns with the deterministic mathematical formulas. The physics baseline maintains a highly accurate, near-zero error rate (< 0.3% APE). AMRules remains stable under this condition.

Conclusion

Combining Apache Flink with Online Machine Learning bridges the gap between theoretical physics and the harsh reality of a degrading factory floor. To make this safe and effective, the architecture relies on three core ideas: predicting residual errors instead of absolute forces, isolating models by product line to prevent forgetting, and enforcing strict safety guardrails. Together, these techniques ensure that heavy machinery operates optimally, even as it breaks down over time.

You can explore the complete codebase, run the simulation locally, and dive deeper into the architecture on the project’s GitHub repository.

Building a Real-Time Product Recommender

Contextual Bandits & Event-Driven Architecture

Why Contextual Bandits?

• Problem: Conventional recommenders (e.g., Collaborative Filtering)
  • Ignore situational context (e.g., Time of Day, Location, Device).
  • Struggle with “Cold Starts” for new items/users.
• Solution: Contextual Multi-Armed Bandits (CMAB).
  • Exploitation: Maximize immediate reward using current knowledge.
  • Exploration: Gather information on uncertain items to improve future performance.

Part 1: Prototype | Part 2: Productionization

Part 1: Prototype

Prototype an online product recommender with Python

–

Python Ecosystem

• Vowpal Wabbit and River ML are well-known for CMAB.
  • Gap: Lack of end-to-end examples integrating feature engineering and offline policy evaluation.
• Fidelity Investments Open Source
  • MABWiser: Algorithm implementation.
  • Mab2Rec: Offline policy evaluation.
  • TextWiser: Text featurization.

–

Prototyping Workflow

From synthetic data generation to live simulation.

–

Live Demo & Walkthrough

Let’s dive into the code.

–

Offline Policy Evaluation

Why LinUCB?

• Best Trade-off: High Ranking (AUC) + High Engagement (CTR).
• Beats LinGreedy: Explores effectively (CTR 0.20 vs 0.11).
• Beats LinTS: Ranks accurately (AUC 0.86 vs 0.64).

–

LinUCB Algorithm

Balancing Exploitation and Exploration

$$ \text{Score}_a = \color{cyan}{x^T \theta_a} \mathbin{\color{white}{+}} \color{orange}{\alpha \sqrt{x^T A_a^{-1} x}} \color{white}{, \quad \text{where } \theta_a = A_a^{-1} b_a} $$

• ● Exploitation: Predicted reward ($x^T \theta_a$).
• ● Exploration: Uncertainty bonus (UCB).
• $\theta_a = A_a^{-1} b_a$: Model weights estimated via Ridge Regression.

–

Limitations

A monolithic Python script isn’t built for scale.

• Latency: Training blocks inference.
• Scalability: Matrix math in memory limits the catalog size.
• Fault Tolerance: If the script crashes, the learned state is lost.

Back to Start | Jump to Productionization

Part 2: Productionization

Scaling with an Event-Driven Architecture

–

Architecture

Training (Apache Flink)

• Stateful Processing:

  • Flink acts as “Online Memory”.

• Asynchronous Updates:

  • Fast Path: Updates $A$ and $b$.
    • ($A \leftarrow A + x x^T, b \leftarrow b + r x$)
  • Slow Path: Every 5s, computes $A^{-1}$.

• Sync to Redis: Emits $A^{-1}$ and $b$.

–

Architecture

Serving (Python & Redis)

• Stateless Inference:

  • The client does not train.

• Low Latency:

  • Fetches pre-computed LinUCB parameters from Redis.

• Action:

  • Calculates scores,
  • Ranks items, and
  • Sends feedback to Kafka.

–

Architecture

Transport (Apache Kafka)

• Asynchronous Buffer:

  • Decouples user-facing app speed from backend training.

• Durability:

  • Stores “Ground Truth” events safely for replay or analytics.

–

Live Demo & Walkthrough

Let’s dive into the code.

Back to Start | Jump to Takeaways

Key Takeaways

Bridging the gap between Data Science and Data Engineering.

–

Start Small, Evaluate Offline

• Before touching infrastructure, evaluate policies.
• Using tools like MABWiser and Mab2Rec allows you to simulate user behavior and validate algorithms on historical data safely.

–

Decouple to Scale

• A monolithic architecture forces a trade-off between model accuracy and user latency.
• Event-Driven Architecture (EDA) solves this by separating high-speed inference (Redis) from heavy stateful training (Flink).

–

Real-Time Adaptability

• By integrating Kafka and Flink, the system learns from user behavior instantly.
  • Dynamic Personalization: Optimizes for specific real-time user context with every click.
  • Continuous Learning: Eliminates the “Cold Start” problem for new items without batch-job downtime.

Thank You!

Code & Resources:

• GitHub Repository
• Blog Posts: Part 1: Prototype | Part 2: Productionization
• Youtube Playlist

Back to Start

Demonstrating Reveal.js Featuresfor Engineering Presentation

1. Image Embedding

Standard Markdown or HTML backgrounds

–

Full Background Slide

(Use – for vertical slides)

Cosmic Background

This slide has a remote image background.

2. LaTeX Math

Rendered via MathJax

The Probability Density Function for a Normal Distribution:

$$ f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{ -\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2 } $$

3. HTML Tables

Markdown and HTML mix

4. Charts (Mermaid.js)

Diagrams from Code

5. Code Highlighting

Highlight specific lines (Click next)

1def process_stream(data):
2    # Step 1: Clean
3    data = data.strip()
4    data = data.lower()
5    
6    # Step 2: Return
7    return data

6. Fragments

Step-by-step visibility

• Item 1 (Fades in)
• Item 2 (Fades in second)
• Item 3 (Grows larger)
• Item 4 (Turns red)
• Item 5 (Fades out)

Choosing between Marp and Reveal.js

When looking for a solution, I initially considered Marp.

Marp is undeniably easier to author initially; its VS Code extension with live preview makes designing slides feel like magic. However, it presented a hurdle for my Hugo integration as it requires separate compilation.

Marp requires a separate build step to convert Markdown into standalone HTML. To integrate it, I would have to manually export each deck and move the resulting HTML into my static/ folder. This felt “disconnected” from the Hugo ecosystem.

Reveal.js, on the other hand, is a browser-based engine. By using Reveal.js, I can:

1. Stay Native: Write pure Markdown in my content/ folder just like a blog post.
2. Zero Manual Export: Hugo handles the routing, and the browser handles the rendering at runtime.
3. Automatic Updates: When I git push a Markdown file, the slides are live instantly. No separate compilation, no manual file moving.

See it in Action

Before diving into the configuration, you can interact with the live demo below. This is the actual slide deck, rendered natively by Hugo and embedded directly into this post:

You can use your arrow keys to navigate the slides above, or click the “View Full Screen” button for the complete experience.

How It Works

I chose to keep slides independent from my main blog posts to avoid cluttering my main feed. Using Hugo’s Leaf Bundles, each presentation gets its own folder for local assets like images.

1content/slides/
2└── 2026-03-06-revealjs-demo/            
3    ├── index.md                          # The slide content
4    └── featured.jpg                      # Thumbnail for list view

Custom “Shell” Layout

The secret is a custom layout at layouts/slides/single.html. Instead of using my theme’s standard blog layout (which includes headers and sidebars), I created a clean “shell” that loads the Reveal.js engine from a CDN.

Crucially, I use {{ .RawContent }} instead of {{ .Content }}. This is because Reveal.js needs the raw Markdown syntax (like --- for slide breaks) to calculate transitions in the browser, rather than the pre-rendered HTML Hugo usually provides.

Advanced Technical Features

Because Reveal.js is a web-based runtime, it allows for high-end technical storytelling that traditional slide software can’t easily handle:

Interactive Code Storytelling

Standard syntax highlighting is good, but Reveal.js allows for directed walkthroughs. Using a specific syntax like [1|3-5|7], I can guide the audience through a specific block of code:

• Workflow: First, I highlight the function definition, then step into the processing logic, and finally focus on the return statement. This “guided tour” of the code prevents the audience from getting overwhelmed by a large block of text.

Precision Media Styling

Using the .element comment syntax, I have granular control over CSS directly within the Markdown.

• Dynamic Backgrounds: I can switch the entire slide context using the data-background attribute (like a high-impact “Cosmic” space background).
• CSS Overrides: I can apply borders, rounded corners, and specific alignments, for example, centering a gold-bordered image without ever leaving the Markdown file.

Managing Cognitive Load with Fragments

To keep the audience focused on the current talking point, I use Fragments. This provides progressive disclosure of information. Beyond just “appearing,” I can use classes like grow to emphasize a specific item or highlight-red to signal a warning, all triggered by a simple keypress.

Mathematical Precision (LaTeX/MathJax)

For engineering-heavy talks, raw text equations look unprofessional. By integrating MathJax, I can render complex mathematical models, such as Probability Density Functions, with the same precision found in a LaTeX paper. They remain sharp at any zoom level because they are rendered as vectors in the browser.

Diagrams as Code (Mermaid.js)

Instead of embedding static PNGs of architecture diagrams that inevitably become outdated, I use Mermaid.js. This allows me to define data pipelines, like a Kafka-to-Flink-to-Redis flow, directly in the Markdown text.

• Engineering Value: Diagrams are now version-controlled and searchable. If the architecture changes, I update a few lines of text, and the visual layout re-renders automatically.

Road Ahead: Professional Diagrams with Draw.io

While Mermaid.js is excellent for rapid, text-based flowcharts, complex system architectures often require the precision of a dedicated design tool. I am currently investigating the best way to integrate Draw.io into this “Slides-as-Code” workflow without losing the benefits of version control.

Currently, I am evaluating two brief paths:

• Editable SVGs: Exporting diagrams as SVGs with embedded XML. This allows the slides to treat the diagram as a standard image while keeping the source file fully editable and version-controlled.
• The Draw.io JS Viewer: Utilizing the official Diagrams.net JavaScript library to render .drawio XML files dynamically. This would enable interactive features like zooming and layer toggling directly during a presentation.

Moving forward, the goal is to ensure that even the most complex architectural designs remain as easy to maintain as a line of Markdown.

Why this works for me

By leveraging Hugo’s Directory Inference, any file I drop into content/slides/ automatically inherits this powerful shell. I no longer “design” slides; I engineer them.

While Marp might be slightly easier to write, the Reveal.js integration is much easier to maintain. It turns my site into a single source of truth for both my articles and my presentations.

You can check out my latest presentations in the Slides section!

This post demonstrates how to decouple these concerns using an event-driven architecture with Apache Flink, Kafka, and Redis.

System Architecture

To move from prototype to production, we split the application into two distinct layers: Serving and Training.

• Python Client (Serving): A lightweight, stateless client responsible for inference. It fetches pre-calculated model parameters from Redis, computes scores locally to make product recommendations, and captures user feedback.
• Kafka (Transport): Buffers feedback events asynchronously, decoupling the speed of serving from the speed of training.
• Flink (Training): A stateful streaming application. It consumes feedback events, updates the model parameters (LinUCB matrices $A$ and $b$), and pushes the inverted matrices back to Redis.
  • ❗ Unlike Part 1, where training relied on MABWiser, here it is performed via explicit matrix operations.
• Redis (Model Store): Stores the latest model parameters ($A^{-1}$ and $b$) for low-latency access by the client.

📂 Source Code for the Post

The source code for this post is available in the product-recommender folder of the streaming-demos GitHub repository.

Flink Application Design

The Flink job (recsys-trainer) ties these concepts together using a few specific patterns.

Stateful Model Training

The core challenge in distributed online learning is managing state. The LinUCBUpdater function in the Flink trainer acts as the system’s memory. It implements a disjoint LinUCB model, meaning it maintains a completely independent set of matrices for each unique product.

❗The matrices are used to calculate scores for making recommendations.

For each product_id, Flink maintains two pieces of state in RocksDB:

1. Matrix $A$ ($d \times d$): Represents the covariance of features seen so far. It tracks Exposure, recording how many times specific user contexts (e.g., “Morning Users” or “Weekend Users”) have been seen for a specific product.
2. Vector $b$ ($d \times 1$): Represents the accumulated reward. It tracks Success, recording which features actually led to a click.

The matrix $A$ is initialized as a scaled identity matrix $A_0 = \lambda I$ to ensure invertibility and to encode an initial prior of uniform uncertainty across feature dimensions.

When a feedback event arrives (Context $x$, Reward $r$), Flink performs the updates:

• Update A: $A \leftarrow A + x x^T$. The outer product $x x^T$ increases covariance along the observed feature directions. As similar contexts repeat, $A$ grows in those directions, reflecting increased confidence.
• Update b: $b \leftarrow b + r x$. If the user clicked ($r=1$), we add their feature vector to $b$, reinforcing that preference pattern.

The updated $A$ and $b$ are stored immediately in Flink keyed state (backed by RocksDB). With checkpointing enabled, this state is durably persisted and recovered in case of failure.

Optimization 1: Inversion on Write

To generate a score, we need the inverse matrix $A^{-1}$, which is computationally expensive. If we performed this inversion inside the Python client for every recommendation request, latency would increase significantly. Instead, the Flink training job periodically loads $A$ from state, factorizes it using LU decomposition, computes $A^{-1}$, and stores the inverse in Redis. Because the contextual feature dimension in this demo is small, recomputing the inverse periodically remains efficient while keeping the serving layer lightweight.

Optimization 2: Batched Updates

In a high-traffic environment, a popular product might receive thousands of clicks per second. Inverting the matrix and writing to Redis for every single click would be inefficient.

To solve this, we use Flink timers to buffer updates. The model state ($A$ and $b$) is updated immediately for every event, while the expensive inversion and Redis write are triggered periodically (e.g., every 5 seconds). This drastically reduces CPU load and network traffic while keeping the model fresh.

Scalable Inference Logic

The Python client (eda_recommender.py) is responsible for ranking items. It uses the Upper Confidence Bound (UCB) formula to balance exploiting known good items and exploring uncertain ones.

For a given user context vector $x$ and product $a$, the score is calculated as:

$$ \text{Score}_a = x^T \theta_a + \alpha \sqrt{x^T A_a^{-1} x}, \quad \text{where } \theta_a = A_a^{-1} b_a $$

Prediction ($x^T \theta_a$)
This is the standard Linear Regression prediction. It asks: “Based on historical data, how likely is this user to click?” If the user matches features stored in vector $b$ (features that previously led to clicks), this term is high.

Exploration ($\alpha \sqrt{x^T A_a^{-1} x}$)

• Familiar User: If we have seen this user type many times, the matrix $A$ accumulates repeated contributions of $x x^T$. This increases the magnitude of $A$ in those feature directions. Because the exploration term depends on $x^T A^{-1} x$, a larger $A$ leads to a smaller quadratic form, shrinking the confidence bound. The model therefore relies more on exploitation.
• Cold Start: If we have rarely (or never) observed this feature pattern, $A$ remains close to its initial regularized identity matrix. After inversion, these directions yield larger values of $x^T A^{-1} x$, increasing the confidence bound and encouraging exploration to reduce uncertainty.
• ❗ $\alpha$ is a hyperparameter and it is set to 0.1 as determined in Part 1.

Hybrid Source for Warm Start

Contextual bandits suffer from the “Cold Start” problem. To mitigate this, we implement a Hybrid Source.

1. File Source: Reads the historical CSV (training_log.csv) generated in Part 1 to bootstrap the model state.
2. Kafka Source: Automatically switches to the live feedback-events topic once the historical data is processed.

Custom Redis Sink

We implement a custom Sink using the Sink V2 API and Jedis. This allows us to perform efficient SET operations to update the model parameters in Redis directly from the Flink stream. Because each update overwrites the full parameter vector, repeated writes remain logically safe under at-least-once delivery semantics. Besides, because the upstream LinUCBUpdater batches the emissions, this sink receives highly aggregated model updates, preventing Redis from being overwhelmed by write operations.

Recommender Simulation Design

To validate the architecture without live user traffic, we designed a Python client (eda_recommender.py) that simulates the entire lifecycle of a recommendation request. This script plays two roles simultaneously: it acts as the Recommendation Service (serving predictions) and the User (providing feedback).

Serving Logic

In a production environment, this logic would live in a high-performance API. For this simulation, the Python client:

1. Context Generation: Creates a synthetic user profile (Age, Gender) and derives key temporal features (e.g., Morning, Weekend) from a simulated timestamp to form the full context.
2. Model Retrieval: Fetches the latest LinUCB parameters ($A^{-1}$ and $b$) for all active products directly from Redis.
3. Scoring and Ranking: Calculates the UCB score for every product, ranks them in descending order, and returns the top 5 highest-scoring items as the recommendation set.

Feedback Generation

To prove the model is learning, the simulation follows the same “Ground Truth” logic used in Part 1:

• Morning Routine: Users click “Drinks & Desserts” (Coffee) between 6 AM and 11 AM.
• Weekend Treats: On Saturdays and Sundays, users prefer “Pizzas” or “Burgers.”
• Price Sensitivity: Users under 25 avoid expensive items.

If any of the recommended top 5 items matches the user’s current context (e.g., showing a Latte on a Tuesday morning), the script generates a Reward (1). Otherwise, it generates no reward (0). This feedback is serialized to Avro and produced to Kafka, completing the loop.

Environment Setup

We use Docker Compose to orchestrate the infrastructure (Kafka, Flink, Redis) and Gradle to build the Kotlin application.

Prerequisites

Clone the repository and infrastructure utilities, then download the required connectors (Kafka, Flink, Avro).

 1git clone https://github.com/jaehyeon-kim/streaming-demos.git
 2cd streaming-demos
 3
 4# Clone Factor House Local for infrastructure definitions
 5git clone https://github.com/factorhouse/factorhouse-local.git
 6
 7# Download Kafka/Flink Dependencies
 8./factorhouse-local/resources/setup-env.sh
 9
10cd product-recommender

Build and Launch

We bootstrap the environment by generating training data, building the Flink JAR, and launching the cluster. We use Kpow and Flex to monitor the Kafka and Flink clusters; these tools require a Factor House community license. Visit the Factor House License Portal to generate your license, save the details in a file (e.g., license.env), and export the associated environment variables (KPOW_LICENSE and FLEX_LICENSE).

With the license configured, launch the Docker Compose services as shown below.

❗ You do not need Kotlin or Gradle installed locally. The ./gradlew script handles all build dependencies.

 1# Setup Python and Generate Bootstrap Data
 2python3 -m venv venv
 3source venv/bin/activate
 4pip install -r requirements.txt
 5python recsys-engine/prepare_data.py
 6
 7# Build Flink Application (Shadow Jar)
 8cd recsys-trainer
 9./gradlew shadowJar
10cd ..
11
12# Launch Infrastructure (Kafka, Flink, Redis, Kpow)
13export KPOW_SUFFIX="-ce"
14export FLEX_SUFFIX="-ce"
15export KPOW_LICENSE=<path-to-license-file>
16export FLEX_LICENSE=<path-to-license-file>
17
18docker compose -p kpow -f ../factorhouse-local/compose-kpow.yml up -d \
19  && docker compose -p stripped -f ./compose-stripped.yml up -d \
20  && docker compose -p flex -f ./compose-flex.yml up -d

Live Recommender Simulation

Once the infrastructure is running, Flink will first process the historical events to warm up. Once the historical processing is complete, we can run the Python client to simulate live traffic.

To visualize the system in action, open two terminals.

Terminal 1: Client

Run the Python script. It acts as the user, receiving recommendations and sending feedback (clicks) to Kafka.

1python recsys-engine/eda_recommender.py

Terminal 2: Trainer

Watch the Flink TaskManager logs. You will see the application reacting to the events sent to the feedback-events topic in real-time.

1docker logs taskmanager -f

Result

You will see a series of feedback events generated by users on the left-hand side. On the right-hand side, you can see the logs confirming that model parameters are being sent to Redis in batches.

This confirms the closed loop: Read (Redis) -> Act (Kafka) -> Learn (Flink) -> Write (Redis).

You can inspect feedback events on Kpow at http://localhost:3000.

Teardown

To stop the cluster and remove resources:

1docker compose -p flex -f ./compose-flex.yml down \
2  && docker compose -p stripped -f ./compose-stripped.yml down \
3  && docker compose -p kpow -f ../factorhouse-local/compose-kpow.yml down

Conclusion

Traditional recommendation systems such as Collaborative Filtering rely on long-term interaction history and often treat user preferences as static. As a result, they struggle to incorporate immediate context, missing situational shifts like a user preferring coffee in the morning but pizza in the evening.

To overcome this, we use Contextual Multi-Armed Bandits (CMAB), an online learning approach that balances exploitation and exploration using real-time contextual signals. While our Python prototype in Part 1 validated the concept, it was not built for scale.

We then evolved it into a production-ready event-driven architecture: Kafka streams feedback events, Flink handles distributed stateful training, and Redis serves precomputed parameters for low-latency inference. This design enables horizontal scalability and real-time adaptation to user behavior.

Traditional recommendation approaches such as Collaborative Filtering remain widely adopted, yet they come with notable constraints. They are particularly vulnerable to the cold-start problem, where new users lack sufficient interaction history, and they depend heavily on long-term behavioral data. As a result, they frequently overlook real-time contextual signals, including time of day, device type, location, or session intent. This can prevent them from capturing situational preferences, such as someone preferring coffee in the morning but pizza in the evening.

Contextual Multi-Armed Bandits (CMAB) address these gaps through online learning.

As a practical form of reinforcement learning, CMAB balances two goals in real time:

1. Exploitation: Recommending what is known to work.
2. Exploration: Trying less-tested options to discover new favorites.

By conditioning decisions on live context, CMAB adapts instantly to changing user behavior.

Why CMAB?

• Beyond A/B Testing: Instead of finding a single global winner, CMAB enables 1:1 personalization, selecting the best option for this user in this context.
• Real-Time Adaptation: Unlike batch-trained models that quickly become stale, CMAB updates incrementally, making it ideal for news/products recommendation, dynamic pricing, or inventory-aware ranking.

Several CMAB implementations exist, including Vowpal Wabbit and River ML. In this post, we use Mab2Rec for offline policy evaluation and MABWiser to build the product recommender prototype.

Data Streaming Opportunity

CMAB performs well in data streaming environments. Integrated with platforms like Kafka and Flink, it learns directly from event streams, creating a feedback loop that responds to trends and shifts in user intent in sub-seconds.

In this series, Part 1 (this post) builds a complete Python prototype to validate the algorithm and simulate user behavior. Part 2 will scale this to a distributed, event-driven architecture.

Tech Stack

We are building this prototype using Python 3.11.

Engineering Note: We explicitly chose Python 3.11 because parts of our stack (specifically mabwiser dependencies) rely on older versions of pandas (< 2.0). On Python 3.12+, installing these dependencies often triggers long compilation times or failures due to missing binary wheels.

We use uv for Python environment management. The core libraries include:

• MABWiser: The engine. It implements the core Contextual Bandit algorithms.
• Mab2Rec: The vehicle. A high-level wrapper that streamlines Recommender System pipelines.
• TextWiser: For converting raw text features into numerical embeddings.
• scikit-learn: For feature scaling and encoding.
• Faker & Pandas: For synthetic data generation and simulation.

The development environment can be constructed as follows:

 1$ git clone https://github.com/jaehyeon-kim/streaming-demos.git
 2$ cd streaming-demos
 3$ uv python install 3.11
 4$ uv venv --python 3.11 venv
 5$ source venv/bin/activate
 6(venv) $ uv pip install -r product-recommender/requirements.txt
 7(venv) $ uv pip list | grep -E "mab|wiser|panda|numpy|scikit|faker"
 8# Using Python 3.11.14 environment at: venv
 9# faker                              40.1.2
10# mab2rec                            1.3.1
11# mabwiser                           2.7.4
12# numpy                              1.26.4
13# pandas                             1.5.3
14# scikit-learn                       1.8.0
15# textwiser                          2.0.2

📂 Source Code for the Post

The source code for this post is available in the product-recommender folder of the streaming-demos GitHub repository.

Data Generation

We first need product and user data to generate the required features.

Products

We utilize a set of 200 raw products, each containing a product ID, name, text description, price, and high-level category.

Here is a list of sample products:

Users

We generate 1,000 Synthetic Users using Faker. Each user is assigned static attributes like Age, Gender, Location, and Traffic Source. These attributes will serve as the “Context” for our Bandit.

Here is a sample of our user base:

Note that street address, postal code, city, state, and country are omitted, as only latitude and longitude are used for feature generation.

Feature Engineering

Bandit algorithms operate on numerical vectors, not raw text. In other words, they cannot interpret "Burger" unless it is converted into numbers. To address this, we developed a transformation pipeline to properly prepare our data:

1. Product Features: We used TextWiser to convert raw product descriptions into vector embeddings. This allows the model to understand that “Burger” and “Sandwich” are semantically closer than “Burger” and “Headphones”. We also applied One-Hot Encoding to categories (Product Category) and MinMax scaling to the price. Finally, we added a binary feature, is_coffee, which is set to 1 for coffee products (e.g., espresso, cappuccino) and 0 otherwise.
2. User Features: Similar to the product features, we applied One-Hot Encoding to categories (Gender and Traffic Source) and MinMax scaling to numerical fields (Age, Latitude, and Longitude).
3. Pipeline Artifacts: We save these transformers as preprocessing_artifacts.pkl. This allows our system to instantly transform any new user/product record into a compatible feature vector during inference.

Sample Processed Product Features:

Notice how the description is now represented by txt_0…txt_9 embeddings.

Sample Processed User Features:

Notice that Age, Latitude and Longitude are normalized between 0 and 1, and categorical fields are binary.

Bandit History Simulation

To evaluate whether our model can truly learn user behavior, we need a controlled Ground Truth, which is an Oracle that determines the likelihood of a simulated user clicking on a recommendation.

Crucially, this Oracle is hidden from the model. The model’s task is to infer these patterns purely from trial and error.

We also inject Dynamic Context features like Time of Day and Day of Week into the user profile at the moment of interaction. These temporal signals create realistic, fluctuating patterns that the model must adapt to.

Simulation Logic

The simulation is implemented as a class GroundTruth, and we define specific rules that govern user behaviour:

• Start from a low base logit (−2.5) to model generally low click probability.
• Rule 1: Morning coffee preference: if the user is browsing in the morning and the item is a coffee product, add a strong positive boost to the score.
• Rule 2: Weekend comfort food: if the session is on a weekend and the item is Pizza or Burgers & Sandwiches, add a moderate positive boost.
• Rule 3: Budget sensitivity: if the user is young (normalized age < 0.25) and the item is expensive (normalized price > 0.8), apply a strong negative penalty.
• Rule 4: Traffic source bias: if the user arrived via Search, add a small intent-based boost.
• Convert the final logit score into a click probability using a sigmoid function, then sample a Bernoulli trial to simulate whether a click occurs.

 1# product-recommender/recsys-engine/src/bandit_simulator.py
 2class GroundTruth:
 3    """
 4    The HIDDEN FORMULA (Ground Truth) for click simulation.
 5    Determines user click behavior based on context and item features.
 6    """
 7
 8    @staticmethod
 9    def calculate_probability(user_ctx: dict, item_ctx: dict) -> float:
10        """
11        Computes the probability that a user clicks an item.
12        Uses logistic regression-style scoring with domain-specific rules.
13        """
14        score = -2.5  # Base logit: starts with a low probability
15
16        # Rule 1: Morning Coffee
17        # Users are more likely to click coffee in the morning
18        if user_ctx.get("is_morning") == 1 and item_ctx.get("is_coffee") == 1:
19            score += 2.5
20
21        # Rule 2: Weekend Comfort Food
22        # Users tend to choose Pizza or Burgers on weekends
23        if user_ctx.get("is_weekend") == 1:
24            if item_ctx.get("cat_Pizzas") == 1 or item_ctx.get("cat_Burgers & Sandwiches") == 1:
25                score += 1.8
26
27        # Rule 3: Budget Constraint
28        # Young users (<25 years) avoid expensive items (normalized price > 0.8)
29        user_age = user_ctx.get("age", 0.5)  # normalized age 0-1
30        item_price = item_ctx.get("price", 0.5)  # normalized price 0-1
31        if user_age < 0.25 and item_price > 0.8:
32            score -= 3.0
33
34        # Rule 4: Traffic Bias
35        # Users arriving via Search have a slightly higher propensity to click
36        if user_ctx.get("traffic_source_Search") == 1:
37            score += 0.5
38
39        # Convert logit score to probability using sigmoid function
40        return 1 / (1 + np.exp(-score))
41
42    def will_click(self, user_ctx: dict, item_ctx: dict, fake: Faker) -> int:
43        """
44        Simulates a Bernoulli trial (click = 1, no click = 0) based on probability.
45        """
46        prob = self.calculate_probability(user_ctx, item_ctx)
47        return 1 if fake.random.random() < prob else 0

Data Preparation

We generate 10,000 historical events to serve as our “Offline Training” dataset. This process involves picking a random user and a random product, then asking the Oracle “Did they click?”.

Because the user and product are matched randomly (not by a recommender), the Average Click Rate (CTR) is naturally low. In this example, it is around 13.65%, and this serves as our baseline.

💡 There are three main scripts for this post: prepare_data.py for feature engineering and bandit history simulation, evalue.py for offline policy evaluation, and local_recommender.py for running product recommendation locally. Each script accepts a --seed argument, which defaults to 1237. As long as the seed remains the same, running the scripts will produce identical outputs.

 1(venv) $ python product-recommender/recsys-engine/prepare_data.py
 2[2026-01-26 19:16:09] INFO    : Generating 1000 synthetic users...
 3[2026-01-26 19:16:09] INFO    : Saved raw users to: .../users.csv
 4[2026-01-26 19:16:09] INFO    : Starting Feature Engineering...
 5[2026-01-26 19:16:09] INFO    : Saved User Features: (1000, 11)
 6[2026-01-26 19:16:10] INFO    : Saved Product Features: (200, 21)
 7[2026-01-26 19:16:10] INFO    : Saved Pipeline Artifacts to: .../preprocessing_artifacts.pkl
 8[2026-01-26 19:16:10] INFO    : Loaded 1000 users and 200 products.
 9[2026-01-26 19:16:10] INFO    : Generating 10000 events...
10[2026-01-26 19:16:10] INFO    : Done. Saved Training Log to .../training_log.csv
11[2026-01-26 19:16:10] INFO    : Avg Click Rate: 13.65%
12[2026-01-26 19:16:10] INFO    : Data Preparation Complete.

The main dataset (training_log.csv) combines user features, dynamic context (e.g., is_morning), product ID, and the interaction result (response):

Offline Policy Evaluation

We benchmarked several policies using Mab2Rec on the 10,000 historical events.

The Candidates

• Random: The baseline. Recommends items blindly.
• Popularity: Recommends items with the highest global click rate.
  • Result: Mediocre (AUC ~0.59). While better than random, it still fails to capture specific rules, such as “Morning Coffee” vs. “Weekend Pizza.”
• LinGreedy: Disjoint Linear Regression with $\epsilon$-greedy exploration.
• LinUCB (The Winner): Disjoint Linear Regression with Upper Confidence Bound.
• LinTS (Thompson Sampling): Bayesian regression that samples from a probability distribution.

Winner: LinUCB

While LinGreedy achieved the highest theoretical ranking accuracy (AUC ~0.88), it suffered from a low click rate (CTR ~11%) because it exploited “safe” choices too early.

LinUCB is the practical winner. It achieved a comparable ranking accuracy (AUC ~0.86) but nearly double the engagement (CTR ~20.5%) to LinGreedy.

This algorithm excels because it balances two competing goals:

1. Exploitation: It uses the predicted probability of a click ($x^T \theta$) to find good items.
2. Exploration: It adds a confidence interval ($\alpha \sqrt{x^T A^{-1} x}$) to the score. If the model is uncertain about a specific context (e.g., “I haven’t seen a user drink Coffee at 8 PM before”), the interval widens, boosting the score and forcing the model to test that hypothesis.

This allows LinUCB to discover high-value opportunities that the conservative LinGreedy model misses.

 1(venv) $ python product-recommender/recsys-engine/evaluate.py
 2Running Benchmark... (This trains and scores all models automatically)
 3--------------------------------------------------------------------------------
 4Available Metrics: ['AUC(score)@5', 'CTR(score)@5', 'Precision@5', 'Recall@5']
 5            AUC(score)@5  CTR(score)@5  Precision@5  Recall@5
 6Random          0.550000      0.102041     0.003876  0.019380
 7Popularity      0.592857      0.192308     0.007752  0.038760
 8LinGreedy       0.885185      0.117647     0.004651  0.023256
 9LinUCB          0.860317      0.204545     0.006977  0.034884
10LinTS           0.640798      0.211538     0.008527  0.042636
11ClustersTS      0.550505      0.153846     0.004651  0.023256
12--------------------------------------------------------------------------------

Why LinUCB Outperforms the Baseline in CTR

This is the core concept of Offline Policy Evaluation.

The benchmark does not test on every single row of your history. It uses a technique called Rejection Sampling (or simply “Matching”).

Here is exactly how mab2rec calculates that 20.5%:

1. The Log (History): Contains a mix of “Good Decisions” and “Bad Decisions” because it was generated randomly.

   • Row A: Morning User $\to$ Show Pizza $\to$ No Click (Bad Random Choice)
   • Row B: Morning User $\to$ Show Coffee $\to$ Click (Lucky Random Choice)

2. The Test (LinUCB): The model is smart. It knows Morning users want Coffee.

   • For Row A, LinUCB says: “I would recommend Coffee.”
     • Mismatch! The history shows Pizza. We cannot know what would have happened if we showed Coffee. This row is IGNORED.
   • For Row B, LinUCB says: “I would recommend Coffee.”
     • Match! The history shows Coffee. We know the result (Click). This row is COUNTED.

The dataset average (13.7%) includes all the “Bad Random Choices” (Row A). The LinUCB score (20.5%) filters out the bad choices. It effectively says: “On the rare occasions where the random history actually showed the right product (Row B), did the user click?” Since LinUCB focuses only on the “Right Products,” the click rate for those specific matches is much higher than the average of the random pile.

Simulation of the Selected Product Recommender Locally

With the model selected, we built a script to simulate the product recommender locally. This script acts as the Server, the User, and the Trainer simultaneously in a continuous loop.

Step 1: Pre-training (Offline Replay)

We don’t want to start with a “dumb” model. We load the 10,000 historical events (training_log.csv) and run model.fit(). This gives the bandit a baseline knowledge of the world before the live loop begins.

Step 2: The Online Loop

We simulate a sequence of user visits:

1. User Arrival: Pick a random user from the pool.
2. Contextualize: Inject a simulated timestamp (e.g., varying between Mon 08:00 AM and Sat 09:00 PM). This is the key “Context” the model must react to.
3. Recommend: LinUCB calculates scores for all 200 products and returns the Top 5.
4. Reaction: The GroundTruth Oracle decides if the user clicks.
5. Online Update: We call model.partial_fit(). This updates the matrices ($A$ and $b$) instantly. The very next recommendation will reflect this new learning.

Here is a sample of 30 recommendation records from the local simulation.

 1(venv) $ python product-recommender/recsys-engine/local_recommender.py 
 2[2026-02-05 15:47:43] INFO    : Loaded 1000 users
 3[2026-02-05 15:47:43] INFO    : Loading artifacts...
 4[2026-02-05 15:47:48] INFO    : Loaded 200 products.
 5[2026-02-05 15:47:48] INFO    : Pre-training model from history...
 6[2026-02-05 15:47:48] INFO    : Model pre-trained on 10000 events.
 7
 8--- STARTING LIVE LOOP (30 visits) ---
 9
10User 0153 (56 yo) @ Fri 00:58 -> Recs: [200, 124, 015, 058, 011] -> Clicked: 200 (❌)
11User 0909 (21 yo) @ Sat 15:53 -> Recs: [038, 040, 017, 020, 046] -> Clicked: 038 (❌)
12User 0406 (30 yo) @ Sat 05:24 -> Recs: [020, 041, 008, 055, 040] -> Clicked: 020 (✅)
13User 0317 (31 yo) @ Sat 05:38 -> Recs: [008, 055, 057, 059, 139] -> Clicked: 055 (✅)
14User 0246 (44 yo) @ Mon 02:04 -> Recs: [015, 058, 057, 011, 124] -> Clicked: 015 (❌)
15User 0974 (61 yo) @ Fri 01:16 -> Recs: [058, 073, 124, 074, 051] -> Clicked: 058 (❌)
16User 0234 (26 yo) @ Thu 12:16 -> Recs: [036, 103, 002, 186, 070] -> Clicked: 036 (❌)
17User 0360 (35 yo) @ Sat 20:23 -> Recs: [058, 051, 008, 042, 018] -> Clicked: 042 (✅)
18User 0513 (51 yo) @ Sun 05:37 -> Recs: [051, 059, 043, 014, 020] -> Clicked: 051 (✅)
19User 0640 (33 yo) @ Mon 00:49 -> Recs: [073, 124, 023, 147, 074] -> Clicked: 073 (❌)
20User 0363 (31 yo) @ Fri 23:35 -> Recs: [200, 126, 085, 058, 018] -> Clicked: 018 (✅)
21User 0718 (58 yo) @ Sat 23:05 -> Recs: [018, 036, 040, 042, 020] -> Clicked: 018 (✅)
22User 0390 (49 yo) @ Tue 00:56 -> Recs: [147, 165, 020, 089, 047] -> Clicked: 147 (❌)
23User 0425 (39 yo) @ Thu 03:59 -> Recs: [147, 165, 062, 028, 055] -> Clicked: 147 (❌)
24User 0792 (21 yo) @ Sun 23:28 -> Recs: [042, 056, 018, 043, 046] -> Clicked: 042 (✅)
25User 0190 (41 yo) @ Sat 10:54 -> Recs: [192, 139, 189, 008, 055] -> Clicked: 192 (✅)
26User 0544 (41 yo) @ Tue 20:42 -> Recs: [018, 058, 090, 043, 147] -> Clicked: 018 (❌)
27User 0192 (17 yo) @ Sat 18:38 -> Recs: [042, 056, 018, 046, 043] -> Clicked: 042 (✅)
28User 0757 (55 yo) @ Thu 16:08 -> Recs: [015, 171, 165, 037, 126] -> Clicked: 126 (✅)
29User 0904 (60 yo) @ Sat 02:07 -> Recs: [103, 041, 017, 042, 057] -> Clicked: 042 (✅)
30User 0552 (39 yo) @ Tue 11:33 -> Recs: [192, 190, 189, 194, 193] -> Clicked: 192 (✅)
31User 0540 (36 yo) @ Sat 08:56 -> Recs: [043, 041, 192, 014, 073] -> Clicked: 043 (✅)
32User 0326 (26 yo) @ Wed 13:04 -> Recs: [015, 171, 165, 023, 126] -> Clicked: 015 (❌)
33User 0834 (29 yo) @ Sat 22:58 -> Recs: [051, 002, 042, 058, 036] -> Clicked: 051 (✅)
34User 0290 (21 yo) @ Mon 19:09 -> Recs: [058, 200, 004, 018, 085] -> Clicked: 018 (✅)
35User 0275 (18 yo) @ Wed 11:10 -> Recs: [189, 002, 160, 078, 103] -> Clicked: 189 (❌)
36User 0327 (23 yo) @ Wed 19:54 -> Recs: [200, 126, 018, 085, 058] -> Clicked: 200 (❌)
37User 0144 (67 yo) @ Thu 12:31 -> Recs: [087, 126, 047, 103, 034] -> Clicked: 087 (❌)
38User 0497 (60 yo) @ Sun 08:26 -> Recs: [192, 139, 008, 189, 059] -> Clicked: 192 (✅)
39User 0508 (64 yo) @ Tue 12:41 -> Recs: [165, 087, 026, 171, 037] -> Clicked: 165 (❌)
40
41--- END LOOP ---

Evaluation of Simulation

The system is behaving exactly as a Contextual Bandit should. It is aggressively exploiting known high-probability zones while struggling (realistically) in neutral zones.

The “Weekend Pizza” Strategy is Dominant

The model has learned that Weekends (Sat/Sun) are for Pizzas (Category 40s).

• User 0360 (Sat 20:23): Recommended [..., 042 (Aussie Pizza), ...] $\to$ Clicked ✅.
• User 0513 (Sun 05:37): Recommended [051 (Buffalo Pizza), 059 (Lamb Pizza)...] $\to$ Clicked ✅.
• User 0834 (Sat 22:58): Recommended [051 (Buffalo Pizza)... 042 (Aussie Pizza)] $\to$ Clicked ✅.
• Insight: The model pushes Pizzas hard on weekends regardless of the specific hour, resulting in a very high conversion rate for these users.

The “Morning Coffee” Precision

The model correctly switches strategies based on the hour, even distinguishing “Weekend Morning” from “Weekend Night”.

• User 0552 (Tue 11:33): It is a Weekday Morning. The model recommended 5 Coffees [192, 190, 189, 194, 193]. The user clicked 192 (Long Black). ✅
• User 0497 (Sun 08:26): It is a Weekend, but it is Morning. The model prioritized Coffee (192, 189) over Pizza. The user clicked 192. ✅
• User 0508 (Tue 12:41): This is a great edge case. It is 41 minutes past the “Morning” cutoff (12:00). The model stopped recommending Coffee and switched to Lunch items (Steamed Veggies, Burritos). The user did not click ❌, but the behavior change proves the features are working perfectly.

Perfect Recommendation, No Interaction (Realism)

• User 0275 (Wed 11:10): The model recommended 189 (Flat White). However, the user ignored it, and the result was ❌ (No Reward).
• Why? This mimics real life. Even if the recommendation is perfect, users don’t always convert. In the GroundTruth, the probability caps at ~50% (sigmoid of 0). This “Bad Luck” outcome confirms your evaluation pipeline is honest.

Conclusion

Across 30 simulated visits, CTR reached 53%.

This elevated CTR indicates that the model has internalized dominant temporal and demographic patterns encoded in the simulation. It aggressively exploits strong signals such as weekend pizza and morning coffee while still exploring less-certain regions.

Importantly, the simulation includes realistic noise. Even strong recommendations do not always convert, reflecting probabilistic user behavior.

What’s Next?

We have successfully prototyped a Contextual Bandit that learns time-based preferences. However, this Python script has major limitations for a production environment:

1. Scalability: Disjoint LinUCB maintains a matrix for every product. With 10 million products, a single server will run out of memory.
2. Latency: The training (partial_fit) blocks the inference (recommend). In a real system, you cannot make a user wait for the model to update.
3. Fault Tolerance: If the script crashes, the learned state is lost.
4. Concurrency: A single Python process cannot handle thousands of concurrent requests.

In Part 2, we will transform this prototype into an Event-Driven Architecture:

• Kafka will transport click events asynchronously.
• Flink will handle distributed, stateful model training.
• Redis will serve the model matrices for sub-millisecond inference.

Stay tuned!

In porting the examples to Kotlin, I also took the opportunity to align the code with modern Flink practices. The complete source for this post is available in the stream-processing-with-flink directory of the flink-demos GitHub repository.

Updating the Code and APIs

The book, while conceptually valuable, is a bit dated. As I worked through the examples, I updated several deprecated features to use their modern equivalents.

• ListCheckpointed to CheckpointedFunction: Replaced the older checkpointing interface with the more flexible CheckpointedFunction.
• SourceFunction to Source API: Migrated from the legacy SourceFunction to the newer, more robust Source API.
• SinkFunction to Sink V2 API: Updated the SinkFunction to the Sink V2 API.
• Queryable State: Ignored as it has been deprecated since Flink 1.18. These examples are built using Flink 1.20.1.

Optimizing the Build with Gradle

Figuring out the Gradle build was a valuable lesson in itself. I learned how to create a single build.gradle.kts to handle two different scenarios: producing a lean production JAR and keeping local execution simple.

For the production JAR, Flink dependencies are declared with compileOnly. This correctly excludes them from the final artifact, as the Flink cluster provides these libraries at runtime.

❗The test environment also needs the Flink APIs to compile and run. This is handled by testImplementation. This standard Gradle configuration provides the Flink libraries only to the test classpath, keeping them completely separate from the production JAR and the local run task.

1dependencies {
2    // Flink Dependencies are not bundled into the JAR
3    compileOnly("org.apache.flink:flink-streaming-java:$flinkVersion")
4    compileOnly("org.apache.flink:flink-clients:$flinkVersion")
5
6    // Flink is available for test compilation and execution
7    testImplementation("org.apache.flink:flink-streaming-java:$flinkVersion")
8    testImplementation("org.apache.flink:flink-clients:$flinkVersion")
9}

However, this creates a problem for local development, as the Flink libraries are now missing from the default runtime classpath. The key technique I learned was how to solve this by creating a custom configuration, localRunClasspath.

This configuration rebuilds the full classpath specifically for the run task by adding the compileOnly dependencies back in, along with the standard implementation and runtimeOnly scopes. This makes local development seamless.

1val localRunClasspath by configurations.creating {
2    extendsFrom(configurations.implementation.get(), configurations.compileOnly.get(), configurations.runtimeOnly.get())
3}
4
5// ...
6
7tasks.named<JavaExec>("run") {
8    classpath = localRunClasspath + sourceSets.main.get().output
9}

Main Chapters

So far, I have translated the examples from the following main chapters:

• Chapter 5: Basic and Keyed Transformations: Covers fundamental data manipulation, including map, filter, keyBy, and rolling sum aggregations, as well as multi-stream transformations.
• Chapter 6: Event Time and Windowing: Focuses on time-based operations, including ProcessFunction timers, watermark generation strategies, window functions, custom window logic, and side outputs for late data handling.
• Chapter 7: State Management: Explores different types of state in Flink, such as ValueState, ListState, MapState, and BroadcastState, along with operator state.
• Chapter 8: Asynchronous I/O and Custom Connectors: Demonstrates how to interact with external systems asynchronously and build custom sources and sinks.

How to Build and Run the Examples

The Flink applications can be run directly from the command line for local testing and development. This is useful for quick debugging without needing a full Flink cluster. Moreover, each Flink app has detailed documentation so it is easy to understand, for example:

 1/**
 2 * This Flink job demonstrates transformations on a `KeyedStream`.
 3 *
 4 * It showcases the `reduce` operator, a powerful tool for maintaining running aggregates
 5 * for each key in a stream.
 6 *
 7 * The pipeline is as follows:
 8 * 1. **Source**: Ingests a stream of `SensorReading` events.
 9 * 2. **KeyBy**: Partitions the stream by the `id` of each sensor. All subsequent
10 *    operations will run independently for each sensor.
11 * 3. **Reduce**: For each key, this operator maintains a running state of the `SensorReading`
12 *    with the maximum temperature seen so far. For every new reading that arrives, it
13 *    compares it to the current maximum and emits the new maximum downstream.
14 * 4. **Sink**: Prints the continuous stream of running maximums for each sensor to the console.
15 */
16object KeyedTransformations {
17    // ...
18}

To launch the apps, use the run task and set the desired main class with the -PmainClass project property. Here are the full examples:

1. Run the Chapter 5 examples.

1./gradlew run -PmainClass=me.jaehyeon.chapter5.BasicTransformations
2./gradlew run -PmainClass=me.jaehyeon.chapter5.KeyedTransformations
3./gradlew run -PmainClass=me.jaehyeon.chapter5.RollingSum
4./gradlew run -PmainClass=me.jaehyeon.chapter5.MultiStreamTransformations

2. Run the Chapter 6 examples.

1./gradlew run -PmainClass=me.jaehyeon.chapter6.ProcessFunctionTimers
2./gradlew run -PmainClass=me.jaehyeon.chapter6.PeriodicWatermarkGeneration
3./gradlew run -PmainClass=me.jaehyeon.chapter6.MarkerBasedWatermarkGeneration
4./gradlew run -PmainClass=me.jaehyeon.chapter6.CoProcessFunctionTimers
5./gradlew run -PmainClass=me.jaehyeon.chapter6.WindowFunctions --args="min1" # min2, avg, minmax1, or minmax2
6./gradlew run -PmainClass=me.jaehyeon.chapter6.CustomWindows
7./gradlew run -PmainClass=me.jaehyeon.chapter6.SideOutputs
8./gradlew run -PmainClass=me.jaehyeon.chapter6.LateDataHandling --args="filter" # sideout or update

3. Run the Chapter 7 examples.

1./gradlew run -PmainClass=me.jaehyeon.chapter7.KeyedStateFunction
2./gradlew run -PmainClass=me.jaehyeon.chapter7.StatefulProcessFunction
3./gradlew run -PmainClass=me.jaehyeon.chapter7.BroadcastStateFunction
4./gradlew run -PmainClass=me.jaehyeon.chapter7.OperatorStateFunction
5./gradlew run -PmainClass=me.jaehyeon.chapter7.KeyedAndOperatorStateFunction

4. Run the Chapter 8 examples.

1./gradlew run -PmainClass=me.jaehyeon.chapter8.AsyncFunction
2./gradlew run -PmainClass=me.jaehyeon.chapter8.CustomConnectors

Conclusion

Working through the examples in Kotlin has been an effective way to dive deeper into Apache Flink. Translating the examples to Kotlin not only forced me to understand the concepts more thoroughly but also provided a great opportunity to get hands-on with the latest APIs and build practices. For those looking to learn Apache Flink through up-to-date examples, I hope sharing my experience and code proves helpful. It’s been a fun and effective learning journey.

This article explores an alternative approach: an integrated architecture where the backend API and the frontend UI are served from a single, cohesive Python application.

We will provide a technical analysis of this architecture’s implementation by integrating FastAPI and NiceGUI, referencing the complete project template available at jaehyeon-kim/nicegui-fastapi-template.

💡 Version 2.0: Unified Application Architecture

The initial version was designed with a distinct separation between a FastAPI backend and a NiceGUI frontend, which communicated over HTTP. This new version consolidates the application, leveraging the fact that NiceGUI is built on top of FastAPI. The result is a more tightly integrated structure that allows the UI and API logic to coexist in the same process.

Key Architectural Changes

• Single FastAPI Instance: The separate FastAPI server process has been removed. The application now operates on the single FastAPI instance provided by nicegui.app.
• Direct Function Calls: UI event handlers no longer make HTTP requests (httpx) to the backend. They now import and call the necessary Python functions from the repository layer directly, removing the network layer for UI-to-backend communication.
• Preserved API Endpoints: The original API, intended for external clients, is maintained. It is mounted using FastAPI’s APIRouter onto the main NiceGUI application, ensuring that JSON endpoints remain available.
• Consolidated Codebase: The frontend and backend directories have been merged into a single application package (e.g., app or src). A run.py script at the project root now serves as the single entry point.
• Shared Logic: Business logic, such as permission checks and database operations, has been centralized in the repository layer, where it is called by both the UI event handlers and the API endpoints.

This updated architecture provides a more direct and cohesive way to build full-stack applications where the UI and backend logic are tightly coupled.

Project Structure

The application is structured with a single entry point, app.py, at the project root, and a main source package, src/, which contains all the application’s logic. This design provides a clear separation between the runnable script and the installable source code.

• app.py: This script is the single entry point for the application. It is responsible for creating the main NiceGUI app instance, including all the API routers from src/backend/, importing the UI pages from src/frontend/ to register their routes, and starting the web server.

• src/: This directory is the main Python package for the application. It contains all the core logic, API definitions, and UI code, organized into the following modules:

  • backend/: Contains the code for the data-only API, intended for external clients.
    • endpoints/: Each file defines a set of related API routes (e.g., for items, users, login) using FastAPI’s APIRouter.
    • deps.py: Manages FastAPI’s dependency injection system for the API, such as providing database sessions or the current authenticated user.
  • core/: Holds application-wide configuration (config.py) and security-related functions like password hashing and token creation (security.py).
  • db/: Manages all database interactions, including engine creation, session management (session.py), and initial database setup (init_db.py).
  • frontend/: Contains all the NiceGUI code for the user interface.
    • components/: Holds reusable UI elements and utilities, such as header.py, footer.py, notifications.py, and authentication helpers in auth_utils.py.
    • layouts/: Defines the overall page structure, like the main dashboard frame, ensuring a consistent look and feel.
    • pages/: Each file represents a specific UI view, such as the login screen (login.py) or the item management page (items.py), using the @ui.page decorator.
    • state.py: A module for managing UI-specific state, like the user’s authentication token.
  • models/: Contains the SQLModel (and Pydantic) schemas that define database tables and data structures used across the entire application.
  • repositories/: This is the core business logic and data access layer. It abstracts all database queries and contains functions for data manipulation. In this unified architecture, its functions are now called directly by both the API endpoints in src/backend/ and the UI event handlers in src/frontend/.

FastAPI Backend

FastAPI provides a robust foundation for the backend due to several key technical features:

• ASGI Foundation: Built on the Asynchronous Server Gateway Interface (ASGI), FastAPI natively supports asynchronous operations. This allows it to handle high-concurrency, I/O-bound tasks, such as network requests and database queries, efficiently without blocking the server.
• Schema Generation and Data Validation: FastAPI uses Pydantic models for strict, type-hint-based data validation and serialization. These models automatically generate OpenAPI schemas, which power the interactive API documentation (via Swagger UI and ReDoc) and ensure that the API contract is clearly defined and enforced.
• Dependency Injection System: Its dependency injection system is a core feature that enhances modularity and testability. It allows for the management of dependencies like database sessions and authentication credentials, ensuring that resources are correctly provisioned and cleaned up for each request.

NiceGUI Frontend

NiceGUI serves as the frontend component, allowing for UI development entirely within Python.

• Pythonic Abstraction of Web Technologies: NiceGUI functions as an abstraction layer that generates the necessary HTML, CSS, and JavaScript from Python objects and methods. This allows developers to define complex user interfaces without writing client-side code directly.
• Server-Side Event Handling: The framework employs an event-driven model where UI components are bound to Python callback functions. User interactions (e.g., button clicks, form submissions) trigger these functions, which execute on the server. This creates a direct and clear link between a UI event and its corresponding backend logic.
• Server-Maintained State: Unlike JavaScript frontend frameworks that manage state on the client, NiceGUI maintains the UI state within the server’s Python process. This simplifies application logic, as there is no need for complex state synchronization mechanisms between client and server.

Architectural Advantages and Framework Comparisons

The Integrated Server Approach

In this architecture, the NiceGUI application is mounted directly onto the FastAPI instance. This is typically done with a single function call, creating a unified application that serves both API endpoints and the user interface from one process.

The primary benefit is the ability to use shared data models (e.g., SQLModel or Pydantic) across the entire stack. A model defined once can be used to structure a database table, validate an API request payload, and define the data contract for the UI. This ensures end-to-end data consistency and reduces code duplication, as all parts of the application are built around the same data structures.

NiceGUI vs. JavaScript Frameworks (e.g., React, Vue)

• Bridging the Expertise Gap: Acquiring deep expertise in both backend Python and a modern JavaScript frontend framework is a significant undertaking. NiceGUI directly addresses this by enabling Python developers to build sophisticated user interfaces without leaving the Python ecosystem. This allows them to leverage their existing skills rather than learning a new language and its complex toolchain.
• Leveraging Mature Frontend Technologies: NiceGUI is not a proprietary UI system built from scratch. It is built on top of the robust and widely-used Vue and Quasar frameworks. This provides the best of both worlds: developers interact with a simple Pythonic API while benefiting from the power, performance, and rich component library of a mature frontend technology.
• Trade-offs: The server-side rendering approach is highly efficient for internal tools and data-heavy applications. However, because every interaction requires a round-trip to the server, it may introduce latency on highly interactive UIs compared to a client-side Single Page Application (SPA), which can handle many state changes without network requests.

NiceGUI vs. Streamlit

• Control and Layout: NiceGUI provides more granular control over UI component placement and application layout, using rows, columns, and grids. This makes it well-suited for building applications with a structured, traditional design. Streamlit is more opinionated, favoring a simple, top-to-bottom script execution model that is excellent for linear data narratives but offers less layout flexibility.
• Event Handling and Execution Model: Both frameworks use callbacks, but their underlying execution models differ significantly. NiceGUI uses a persistent component model where an event, such as a button click, executes a specific callback function (on_click=handle_click). This function is the only code that runs and is responsible for explicitly updating any UI elements. This aligns closely with traditional GUI programming paradigms. In contrast, Streamlit uses a script re-run model. While it also has an on_click callback, this function typically modifies a session state object. After the callback completes, Streamlit re-runs the entire application script from top to bottom. The UI is then re-rendered based on the new values in the session state. This model simplifies the creation of linear, data-centric apps but can be less direct and potentially less performant for managing complex, multi-state interfaces compared to NiceGUI’s explicit event-driven approach.
• Integration: NiceGUI is designed to be a component that can be integrated with standard web frameworks like FastAPI. Streamlit is generally used as a self-contained application server and is less straightforward to embed within another ASGI application.

Implementation Overview

To provide a concrete example of this architecture, the reference repository at jaehyeon-kim/nicegui-fastapi-template contains a fully functional application. Let’s examine its structure and key components.

Project Structure

The project is organized into two main directories, ensuring a clear separation of concerns between the backend logic and the user interface:

• backend/: This directory contains all the FastAPI source code. It is further subdivided into modules for handling specific responsibilities:
  • api/: Defines the API endpoints for resources like users and items.
  • db/: Manages the database session and engine configuration.
  • models/: Contains the SQLModel (and Pydantic) data schemas that define our database tables and API data structures.
  • repositories/: Implements the data access layer, abstracting the database queries from the API endpoints.
• frontend/: This directory holds all the NiceGUI code for the user interface, organized by function into the following subdirectories and modules:
  • components/: Contains reusable UI elements and helper functions. This includes visual components like header.py and footer.py, as well as utility modules like notifications.py for displaying messages to the user and form_helpers.py for handling common form logic.
  • layouts/: Defines the overall structure of the application’s pages. The default.py file, for instance, assembles the header, drawer, and footer to ensure a consistent look and feel across different views.
  • pages/: Each file in this directory represents a specific page or view within the application, such as the login screen (login.py), the item management page (items.py), and the user creation form (create_user.py).
  • state.py: A dedicated module for managing the application’s client-side state, such as user authentication status. This allows different parts of the UI to react consistently to changes in state.
• backend/main.py: This file serves as the central integration point. It initializes the FastAPI application, includes the API routers from the backend/api/ directory, and finally, mounts the entire NiceGUI frontend. This is where the two parts of the application become one.

Application Demo

The repository provides a complete user and item management application that showcases role-based access control. The functionality is divided between two user roles:

• Standard User: After logging in, a standard user has full CRUD (Create, Read, Update, Delete) permissions over their own items. They can add new items, view their list, and edit or delete them as needed.

• Superuser: A superuser has elevated privileges. In addition to managing their own items, they can also create new user accounts. Crucially, they have a global view of the system and can manage the items belonging to any user, making this role suitable for administrative purposes.

The demo showcases these distinct workflows, illustrating how the NiceGUI frontend dynamically adapts to the user’s permissions, which are enforced by the FastAPI backend.

Automatic API Documentation

One of the most powerful features of FastAPI is its ability to automatically generate interactive API documentation from the Pydantic models and endpoint definitions. The application provides two documentation interfaces out-of-the-box:

• Swagger UI (/docs): A feature-rich, interactive interface that allows developers to not only view the API endpoints but also test them directly from the browser by sending live requests.

• ReDoc (/redoc): A clean, read-only documentation page that presents the API in a more traditional, hierarchical format. It is excellent for quickly referencing endpoints and their schemas.

These auto-generated documents are invaluable for development, testing, and collaboration, and they are created without any extra effort, thanks to FastAPI’s adherence to the OpenAPI standard.

Summary and Use Cases

The integration of FastAPI and NiceGUI provides a robust architecture for building web applications entirely in Python. It streamlines development by creating a unified environment, simplifies deployment to a single process, and ensures strong data consistency through the use of shared models.

This architecture is exceptionally well-suited for:

• Internal Tools and Administrative Dashboards: Where rapid development and ease of maintenance are critical.
• Rapid Prototyping and MVPs: To quickly build and validate a functional application.
• Machine Learning and Data Science Demos: To create interactive interfaces for models without requiring frontend expertise.

While it may not be the optimal choice for every project, particularly those requiring complex client-side interactivity, it offers a powerful and efficient alternative for a significant class of web applications. For a practical implementation, refer to the project code at jaehyeon-kim/nicegui-fastapi-template.

This is where a gateway-centric architecture comes in. By placing a specialized gateway at the heart of the platform, we can transform this chaos into a stable, governed, and self-service system. The Apache Kyuubi project provides a perfect blueprint for this approach, defining its role as:

Apache Kyuubi is a distributed, multi-tenant gateway providing unified access to big data engines like Spark and Flink via JDBC/ODBC for interactive queries and REST APIs for programmatic submissions.

This definition is more than just a product description; it’s a strategic vision. By embracing this model, we can finally deliver on the promise of self-service data access without sacrificing stability or governance. Let’s explore exactly how each layer of the platform is designed to support this powerful concept.

Architecture: A Layer-by-Layer Breakdown

1. Client Layer: Simplicity Through Standardization

How do users connect? Via interactive SQL clients or programmatic APIs.

The primary goal of a self-service platform is to meet users where they are. The gateway makes this possible by providing multiple, standardized entry points, ensuring that both interactive users and automated systems can leverage the platform’s power. Kyuubi provides two primary connection methods: the Thrift API for interactive SQL and the REST API for programmatic batch workloads.

Thrift API: Gateway for Interactive Analytics

This is the most established and widely used connection method, ideal for analysts and data scientists using standard BI and SQL tools.

• Primary Use Case: Interactive, ad-hoc queries and data exploration.
• How it Works: Kyuubi implements the HiveServer2 Thrift protocol, which is the standard for SQL-on-Big-Data. This allows any tool that can communicate with Hive or Spark SQL to connect seamlessly to Kyuubi using a standard JDBC/ODBC driver.
• Tools: Tableau, PowerBI, DBeaver, Jupyter Notebooks (with PyHive), and other SQL-native clients.
• Gateway’s Role: From the user’s perspective, they are connecting to a traditional database. They don’t need engine-specific drivers or complex connection strings. The immense complexity of the underlying Spark or Flink engines is completely abstracted away behind one familiar JDBC/ODBC endpoint. The Thrift interface uses long-lived connections, making it well-suited for the back-and-forth nature of an interactive session.

💡 Note: While tools typically use JDBC or ODBC drivers, these drivers communicate over the Thrift protocol—hence the term “Thrift API” is technically accurate.

REST API: Interface for Programmatic Workflows

This interface is designed for automation, batch job submission, and integration into larger data pipelines.

• Primary Use Case: Submitting and managing batch jobs (e.g., SQL, Scala, Python, or JAR files) programmatically. This is perfect for ETL/ELT workflows, scheduled reports, and machine learning model training.
• How it Works: The REST API operates over standard, short-lived HTTP connections. Clients can submit a job definition (including the code and execution parameters) to a REST endpoint. Kyuubi then manages the entire lifecycle of the batch job on the user’s behalf. This stateless approach is ideal for automation and works well with load balancers in a high-availability setup.
• Tools:
  • Programmatic Clients: Custom applications, workflow orchestrators like Apache Airflow, or scripts using tools like curl.
  • Command-Line Tool: Kyuubi provides kyuubi-ctl, a command-line interface that uses the REST API for managing batch jobs via YAML files.

💡 Tip: Because of its stateless nature, the REST API is particularly well-suited for distributed environments and works seamlessly with CI/CD systems and job schedulers.

2. Gateway Layer: Multi-Tenant Control Plane

This is the heart of the architecture, where raw compute power is transformed into a stable, governed service.

This layer is where Apache Kyuubi lives. It is not merely a pass-through proxy; it is an intelligent, distributed control plane responsible for security, stability, and resource management. It acts as the indispensable intermediary between a diverse set of users and a powerful but complex set of backend engines.

High Availability and Fault Tolerance

A gateway cannot be a single point of failure. Kyuubi is designed for resilience and can be deployed in a high-availability (HA) configuration. You run multiple Kyuubi server instances, stateless by nature, behind a load balancer. Session information and operational metadata are stored in a shared, external state store (like ZooKeeper or Etcd). If one Kyuubi server goes down, the load balancer redirects clients to a healthy instance, which can then recover the user’s session state from the shared store, providing seamless failover for end-users.

Centralized Authentication Hub

Before any query is processed, the user must be authenticated. The gateway serves as the single, secure entry point for the entire platform. It integrates directly with standard enterprise authentication protocols like LDAP and Kerberos. This centralizes all authentication logic. Instead of exposing multiple engines and configuring security on each one, you secure one endpoint: the gateway. This simplifies client configuration and dramatically reduces the security attack surface.

True Multi-Tenancy through Dynamic Resource Isolation

This is Kyuubi’s most critical feature. It enforces multi-tenancy not just logically, but physically.

• Engine-per-User Isolation: When a user connects and runs a query, Kyuubi does not send it to a shared, monolithic cluster. Instead, it dynamically provisions a dedicated compute engine (e.g., a Spark application on Kubernetes) scoped specifically to that user or their session. This is the key to true multi-tenancy: one user’s poorly written query consuming 100% of its engine’s CPU has zero impact on the performance or stability of another user’s engine.

• Configurable Sharing Levels: Kyuubi also offers flexible engine-sharing modes to balance isolation and performance. While the default is to isolate by user (USER level), you can configure it to share an engine at the CONNECTION level or even create server-scoped engines for specific use cases, giving architects fine-grained control over resource trade-offs.

Unified Interface and Intelligent Routing

The gateway abstracts away the diversity of the backend. A user connects to a single JDBC/ODBC endpoint and speaks standard SQL. They don’t need to know which engine is best for their job.

Based on session properties set by the user or administrator (e.g., kyuubi.engine.type=FLINK), the gateway intelligently routes the request. It accepts the standard SQL, identifies the target engine type, manages the entire lifecycle of that engine (provisioning, query submission, termination), and streams the results back. This turns a complex ecosystem of Spark, Flink, and Trino into a single, unified, and easy-to-use SQL service for the entire organization.

3. Compute Layer: Powerhouse Engines

What does the gateway connect to? Isolated, on-demand Spark, Flink, and Trino engines.

This is where the actual data processing occurs. The compute layer consists of powerful, specialized engines, but they are no longer directly exposed to users. Instead, they are treated as a backend resource, managed entirely by the gateway based on the workload.

• Engines:
  • Apache Spark: The workhorse for large-scale batch SQL processing and ETL.
  • Apache Flink: The engine for real-time, continuous SQL queries on streaming data.
  • Trino: The engine for high-performance, interactive federated queries across disparate data sources.

Gateway’s Role: A Lifecycle Manager for Compute

Kyuubi’s most important function is to act as an intelligent and dynamic lifecycle manager for these powerful engines. It doesn’t just proxy queries; it completely abstracts the complexity of resource management away from the end-user. Here’s how:

• On-Demand, User-Scoped Provisioning: When a user connects and runs their first query, Kyuubi intercepts the request. It authenticates the user and, based on predefined rules, submits a request to a resource manager (like Kubernetes or YARN) to launch a brand new engine instance specifically for that user or session. This is the core of multi-tenancy: the engine is created on behalf of the user, runs with their permissions, and is completely isolated from other users’ engines. This eliminates the “noisy neighbor” problem, where one user’s heavy query can destabilize the entire platform.

• Intelligent Caching and Sharing: While isolating engines is key, constantly launching new ones can introduce latency. Kyuubi manages this intelligently. It can be configured to keep a warm pool of engines ready or to share an engine across multiple queries from the same user within a single session. This provides the performance of a long-running session without sacrificing the isolation between different users.

• Automatic Termination and Cost Control: Kyuubi is also responsible for cleanup. It constantly monitors the engines it has launched. If an engine sits idle for a configured period (e.g., 30 minutes), Kyuubi will automatically terminate it and release its resources back to the cluster. This is absolutely critical for cost-efficiency in a cloud environment, ensuring you only pay for compute resources when they are actively being used.

• Seamless Abstraction: From the user’s perspective, this entire lifecycle is invisible. They write and execute standard SQL. They don’t need to know how to write a spark-submit command, craft a Kubernetes pod YAML file, or worry about memory allocation. Kyuubi handles the translation from a simple SQL query to a complex, resource-managed job on a distributed engine, ensuring the right tool is used for the right job without exposing any of the underlying complexity.

4. Governance Layer: Centralized Control at the Gateway

How do we enforce rules? The gateway is the natural “chokepoint” for governance.

In a self-service world, governance must be automated. The gateway architecture makes this feasible because every single query and every user session must pass through it. This creates a natural control point for applying rules, though the specific capabilities can vary by the backend engine.

Authorization: An Engine-Specific Approach

The gateway acts as the central point for authenticating users and integrating with an authorization engine like Apache Ranger. However, the actual enforcement of security policies is delegated to the backend compute engines. This approach leverages the native security features of each engine, meaning the depth of integration differs across the platform, which is a critical architectural consideration.

• For Apache Spark: The integration is the most mature and powerful. Kyuubi provides a specialized Kyuubi Spark Authz Plugin that deeply integrates with Apache Ranger. This plugin allows Ranger to enforce fine-grained policies—including row-level filtering and column-level masking—directly within the Spark engine for all Spark SQL queries. The gateway authenticates the user, launches a Spark engine on their behalf, and the engine then uses the Authz plugin to check with Ranger and apply policies before execution. This provides robust, centralized security for all Spark-based workloads.
• For Apache Flink: The story is more about isolation and coarse-grained control. Kyuubi provides essential security for Flink through robust authentication and session isolation. This guarantees that only authenticated users can submit jobs and that their Flink sessions are isolated from others. However, fine-grained, policy-based authorization for Flink jobs is not handled by a Kyuubi plugin. Instead, security must be enforced at the data source level. A common pattern is to use Flink’s HiveCatalog to read data from Hive tables, where Apache Ranger’s existing Hive-level policies can be applied for access control.
• For Trino: Authorization is handled by delegating to Trino’s own powerful security model. Trino has a native Apache Ranger plugin that provides comprehensive, fine-grained access control. This plugin supports policies for catalogs, schemas, tables, and columns, as well as row-level filtering and column masking. In this architecture, Kyuubi’s role is to authenticate the user and then pass the user’s identity securely to the Trino engine. Trino then uses its own Ranger plugin to enforce the centralized policies. This allows organizations to manage Spark and Trino permissions within the same Ranger UI, while the enforcement happens natively within each respective engine.

Lineage: A Universal, Automated View of Your Data’s Journey

Unlike authorization, which can be engine-specific, data lineage is applied universally and automatically across the platform. This creates a complete, trustworthy audit trail for every query, which is essential for data governance, impact analysis, and debugging complex data flows.

• How It Works: Automated Agent Injection The gateway-centric design is the key to automating lineage collection. Kyuubi is configured to automatically inject a lightweight OpenLineage agent into the runtime of every single Spark and Flink engine it provisions. Users do not need to add any special libraries or modify their code; lineage capture is a guaranteed part of the platform’s execution process. For Trino, lineage is captured via a separate, native OpenLineage plugin configured directly within the Trino coordinator.
• What is Captured? Rich, Actionable Metadata As jobs run, the OpenLineage agent observes the execution plan and collects detailed metadata. This isn’t just a high-level overview. The agent captures:
  • Job Information: The name of the job, its start and end times, and whether it succeeded or failed.
  • Dataset Information (Inputs and Outputs): The physical sources (e.g., S3 paths, Kafka topics, database tables) and destinations of the data.
  • Column-Level Lineage (for Spark and Trino): For supported engines, the agent can trace the dependencies between specific columns, showing exactly how an output column was derived from one or more input columns. This is invaluable for tracking sensitive data and understanding complex transformations.
  • Operational Statistics: Metadata about the run itself, such as the number of rows written or bytes processed.
• Governance Backend: Visualizing the Flow The agent sends this stream of standardized JSON-formatted lineage events to a compatible metadata service, with Marquez being the reference implementation for OpenLineage. Marquez consumes this metadata and builds a comprehensive, interactive graph of your entire data ecosystem. This provides a single, unified map showing how all datasets are created and consumed across Spark, Flink, and Trino, answering critical questions:
  • “If I change this table schema, what downstream jobs and dashboards will break?”
  • “This report looks wrong. Where did the data it uses come from?”
  • “Which jobs are processing PII data?”

By making lineage collection an automatic, non-negotiable part of the platform architecture, governance is no longer an afterthought. It becomes a reliable, built-in feature that provides a complete and auditable view of how data moves and transforms across the entire organization.

5. Catalog & Storage Layer: Foundation of Truth

Where does the data live? In a standard data lake, accessed via the gateway’s managed engines.

This layer is the physical and logical foundation of the platform. The gateway itself doesn’t store data, but it manages the engines that access it.

• Storage: Cloud object stores like Amazon S3, GCS, or ADLS.
• Table Formats: Open table formats like Apache Iceberg, Apache Hudi, Delta Lake, and Apache Paimon, which provide transactional (ACID) capabilities on top of raw data files in your data lake.
• Catalog: A universal Hive Metastore.
• Gateway’s Role: The gateway ensures that all access to the data lake is mediated. A user cannot simply spin up their own Spark session to bypass rules. For example, to read an Iceberg table, they must submit a SQL query through Kyuubi. The gateway then authenticates them and, particularly for Spark, ensures they are authorized before provisioning an engine to perform the read.

Use Case 1: Interactive Batch Query (Spark)

Let’s trace a classic analytics workflow to see the architecture in action.

1. An analyst connects Tableau to the Kyuubi JDBC endpoint.
2. They write a SQL query joining several large Iceberg tables. Tableau sends this standard SQL to the Kyuubi Gateway.
3. Kyuubi receives the query. It authenticates the user and checks their detailed permissions with Apache Ranger.
4. Seeing this is the user’s first query, Kyuubi provisions a new, dedicated Spark application for them on Kubernetes, injecting the OpenLineage agent.
5. Kyuubi forwards the SQL to the user’s isolated Spark engine.
6. The Spark engine uses the Hive Metastore to find the data’s location on S3 and executes the query.
7. Results are streamed back through the Spark engine, to the Kyuubi gateway, and finally to Tableau. The entire process is isolated, governed, and transparent to the user.

Use Case 2: Real-Time Streaming Query (Flink & Kafka)

The platform’s true power is its ability to handle more than just batch analytics. Let’s see how an analyst can query a live stream of clickstream data from Apache Kafka using Flink SQL, all through the same gateway.

The user connects their SQL client to the exact same Kyuubi JDBC endpoint. They then define a table backed by a Kafka topic and query it continuously:

 1-- Step 1: Map a table structure onto a live Kafka topic.
 2-- The underlying Flink engine uses its Kafka connector to handle this.
 3CREATE TABLE ClickStream (
 4    `user_id` BIGINT,
 5    `url` STRING,
 6    `event_timestamp` TIMESTAMP(3) METADATA FROM 'timestamp',
 7    WATERMARK FOR `event_timestamp` AS `event_timestamp` - INTERVAL '5' SECOND
 8) WITH (
 9    'connector' = 'kafka',
10    'topic' = 'clickstream_events',
11    'properties.bootstrap.servers' = 'kafka-broker-1:9092,...',
12    'scan.startup.mode' = 'latest-offset',
13    'format' = 'json'
14);
15
16-- Step 2: Run a continuous query against the live stream.
17-- This query will run indefinitely, pushing new results to the client as they arrive.
18SELECT
19    TUMBLE_START(event_timestamp, INTERVAL '1' MINUTE) as window_start,
20    url,
21    COUNT(*) as clicks
22FROM
23    ClickStream
24GROUP BY
25    TUMBLE(event_timestamp, INTERVAL '1' MINUTE),
26    url;

How it Works:

1. The user sends the CREATE TABLE and SELECT statements to the Kyuubi JDBC endpoint.
2. Kyuubi authenticates the user. Based on session properties, it recognizes this as a Flink workload and provisions a dedicated Flink session cluster on Kubernetes, again injecting the OpenLineage agent.
3. Kyuubi submits the SQL to the Flink cluster.
4. Flink’s SQL engine parses the query. It uses its Kafka connector to connect to the clickstream_events topic and begins consuming the JSON data stream.
5. The SELECT query runs continuously. As new data arrives in Kafka, Flink processes it, calculates the tumbling window aggregates, and streams the updated results back through the Kyuubi gateway to the analyst’s SQL client.
6. Simultaneously, the OpenLineage agent reports to Marquez that a new data flow has been established, drawing a lineage graph from the Kafka topic to this Flink SQL job.

Conclusion: The Power of a Gateway-Centric Design

Building a successful self-service platform is not about exposing raw power, but about providing controlled, stable, and governed access. By embracing a gateway-centric architecture built around the principles embodied by Apache Kyuubi, you can finally resolve the conflict between user empowerment and platform stability. The gateway acts as the indispensable control plane, turning a potential “Wild West” of big data engines into a well-regulated, multi-tenant, and powerful SQL service for the entire organization—for both batch and real-time workloads.

• Kafka Clients with JSON - Producing and Consuming Order Events
• Kafka Clients with Avro - Schema Registry and Order Events
• Kafka Streams - Lightweight Real-Time Processing for Supplier Stats
• Flink DataStream API - Scalable Event Processing for Supplier Stats
• Flink Table API - Declarative Analytics for Supplier Stats in Real Time (this post)

Flink Table Application

We develop a Flink application that uses Flink’s Table API and SQL-like expressions to perform real-time analytics. This application:

• Consumes Avro-formatted order events from a Kafka topic.
• Uses a mix of the DataStream and Table APIs to prepare data, handle late events, and define watermarks.
• Defines a table over the streaming data, complete with an event-time attribute and watermarks.
• Runs a declarative, SQL-like query to compute supplier statistics (total price and count) in 5-second tumbling windows.
• Splits the stream to route late-arriving records to a separate “skipped” topic for analysis.
• Sinks the aggregated results to a Kafka topic using the built-in avro-confluent format connector.

The source code for the application discussed in this post can be found in the orders-stats-flink folder of this GitHub repository.

The Build Configuration

The build.gradle.kts file sets up the project, its dependencies, and packaging. It’s shared between the DataStream and Table API applications - see the previous post for the Flink application that uses the DataStream API.

• Plugins:
  • kotlin("jvm"): Enables Kotlin language support.
  • com.github.davidmc24.gradle.plugin.avro: Compiles Avro schemas into Java classes.
  • com.github.johnrengelman.shadow: Creates an executable “fat JAR” with all dependencies.
  • application: Configures the project to be runnable via Gradle.
• Dependencies:
  • Flink Core & Table APIs: flink-streaming-java, flink-clients, and crucially, flink-table-api-java-bridge, flink-table-planner-loader, and flink-table-runtime for the Table API.
  • Flink Connectors: flink-connector-kafka for Kafka integration.
  • Flink Formats: flink-avro and flink-avro-confluent-registry for handling Avro data with Confluent Schema Registry.
  • Note on Dependency Scope: The Flink dependencies are declared with implementation. This allows the application to be run directly with ./gradlew run. For production deployments on a Flink cluster (where the Flink runtime is already provided), these dependencies should be changed to compileOnly to significantly reduce the size of the final JAR.
• Application Configuration:
  • The application block sets the mainClass and passes necessary JVM arguments. The run task is configured with environment variables to specify Kafka and Schema Registry connection details.
• Avro & Shadow JAR:
  • The avro block configures code generation.
  • The shadowJar task configures the output JAR name and merges service files, which is crucial for Flink connectors to work correctly.

 1plugins {
 2    kotlin("jvm") version "2.1.20"
 3    id("com.github.davidmc24.gradle.plugin.avro") version "1.9.1"
 4    id("com.github.johnrengelman.shadow") version "8.1.1"
 5    application
 6}
 7
 8group = "me.jaehyeon"
 9version = "1.0-SNAPSHOT"
10
11repositories {
12    mavenCentral()
13    maven("https://packages.confluent.io/maven")
14}
15
16dependencies {
17    // Flink Core and APIs
18    implementation("org.apache.flink:flink-streaming-java:1.20.1")
19    implementation("org.apache.flink:flink-table-api-java:1.20.1")
20    implementation("org.apache.flink:flink-table-api-java-bridge:1.20.1")
21    implementation("org.apache.flink:flink-table-planner-loader:1.20.1")
22    implementation("org.apache.flink:flink-table-runtime:1.20.1")
23    implementation("org.apache.flink:flink-clients:1.20.1")
24    implementation("org.apache.flink:flink-connector-base:1.20.1")
25    // Flink Kafka and Avro
26    implementation("org.apache.flink:flink-connector-kafka:3.4.0-1.20")
27    implementation("org.apache.flink:flink-avro:1.20.1")
28    implementation("org.apache.flink:flink-avro-confluent-registry:1.20.1")
29    // Json
30    implementation("com.fasterxml.jackson.module:jackson-module-kotlin:2.13.0")
31    // Logging
32    implementation("io.github.microutils:kotlin-logging-jvm:3.0.5")
33    implementation("ch.qos.logback:logback-classic:1.5.13")
34    // Kotlin test
35    testImplementation(kotlin("test"))
36}
37
38kotlin {
39    jvmToolchain(17)
40}
41
42application {
43    mainClass.set("me.jaehyeon.MainKt")
44    applicationDefaultJvmArgs =
45        listOf(
46            "--add-opens=java.base/java.util=ALL-UNNAMED",
47        )
48}
49
50avro {
51    setCreateSetters(true)
52    setFieldVisibility("PUBLIC")
53}
54
55tasks.named("compileKotlin") {
56    dependsOn("generateAvroJava")
57}
58
59sourceSets {
60    named("main") {
61        java.srcDirs("build/generated/avro/main")
62        kotlin.srcDirs("src/main/kotlin")
63    }
64}
65
66tasks.withType<com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar> {
67    archiveBaseName.set("orders-stats-flink")
68    archiveClassifier.set("")
69    archiveVersion.set("1.0")
70    mergeServiceFiles()
71}
72
73tasks.named("build") {
74    dependsOn("shadowJar")
75}
76
77tasks.named<JavaExec>("run") {
78    environment("TO_SKIP_PRINT", "false")
79    environment("BOOTSTRAP", "localhost:9092")
80    environment("REGISTRY_URL", "http://localhost:8081")
81}
82
83tasks.test {
84    useJUnitPlatform()
85}

Avro Schema for Supplier Statistics

The SupplierStats.avsc file defines the structure for the aggregated output data. This schema is used by the Flink Table API’s Kafka connector with the avro-confluent format to serialize the final Row results into Avro, ensuring type safety for downstream consumers.

• Type: A record named SupplierStats in the me.jaehyeon.avro namespace.
• Fields:
  • window_start and window_end (string): The start and end times of the aggregation window.
  • supplier (string): The supplier being aggregated.
  • total_price (double): The sum of order prices within the window.
  • count (long): The total number of orders within the window.

 1{
 2  "type": "record",
 3  "name": "SupplierStats",
 4  "namespace": "me.jaehyeon.avro",
 5  "fields": [
 6    { "name": "window_start", "type": "string" },
 7    { "name": "window_end", "type": "string" },
 8    { "name": "supplier", "type": "string" },
 9    { "name": "total_price", "type": "double" },
10    { "name": "count", "type": "long" }
11  ]
12}

Shared Utilities

These utility files provide common functionality used by both the DataStream and Table API applications.

Kafka Admin Utilities

This file provides two key helper functions for interacting with the Kafka ecosystem:

• createTopicIfNotExists(...): Uses Kafka’s AdminClient to programmatically create topics. It’s designed to be idempotent, safely handling cases where the topic already exists to prevent application startup failures.
• getLatestSchema(...): Connects to the Confluent Schema Registry using CachedSchemaRegistryClient to fetch the latest Avro schema for a given subject. This is essential for the Flink source to correctly deserialize incoming Avro records without hardcoding the schema in the application.

 1package me.jaehyeon.kafka
 2
 3import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient
 4import mu.KotlinLogging
 5import org.apache.avro.Schema
 6import org.apache.kafka.clients.admin.AdminClient
 7import org.apache.kafka.clients.admin.AdminClientConfig
 8import org.apache.kafka.clients.admin.NewTopic
 9import org.apache.kafka.common.errors.TopicExistsException
10import java.util.Properties
11import java.util.concurrent.ExecutionException
12import kotlin.use
13
14private val logger = KotlinLogging.logger { }
15
16fun createTopicIfNotExists(
17    topicName: String,
18    bootstrapAddress: String,
19    numPartitions: Int,
20    replicationFactor: Short,
21) {
22    val props =
23        Properties().apply {
24            put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress)
25            put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, "5000")
26            put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, "3000")
27            put(AdminClientConfig.RETRIES_CONFIG, "1")
28        }
29
30    AdminClient.create(props).use { client ->
31        val newTopic = NewTopic(topicName, numPartitions, replicationFactor)
32        val result = client.createTopics(listOf(newTopic))
33
34        try {
35            logger.info { "Attempting to create topic '$topicName'..." }
36            result.all().get()
37            logger.info { "Topic '$topicName' created successfully!" }
38        } catch (e: ExecutionException) {
39            if (e.cause is TopicExistsException) {
40                logger.warn { "Topic '$topicName' was created concurrently or already existed. Continuing..." }
41            } else {
42                throw RuntimeException("Unrecoverable error while creating a topic '$topicName'.", e)
43            }
44        }
45    }
46}
47
48fun getLatestSchema(
49    schemaSubject: String,
50    registryUrl: String,
51    registryConfig: Map<String, String>,
52): Schema {
53    val schemaRegistryClient =
54        CachedSchemaRegistryClient(
55            registryUrl,
56            100,
57            registryConfig,
58        )
59    logger.info { "Fetching latest schema for subject '$schemaSubject' from $registryUrl" }
60    try {
61        val latestSchemaMetadata = schemaRegistryClient.getLatestSchemaMetadata(schemaSubject)
62        logger.info {
63            "Successfully fetched schema ID ${latestSchemaMetadata.id} version ${latestSchemaMetadata.version} for subject '$schemaSubject'"
64        }
65        return Schema.Parser().parse(latestSchemaMetadata.schema)
66    } catch (e: Exception) {
67        logger.error(e) { "Failed to retrieve schema for subject '$schemaSubject' from registry $registryUrl" }
68        throw RuntimeException("Failed to retrieve schema for subject '$schemaSubject'", e)
69    }
70}

Flink Kafka Connectors

This file centralizes the creation of Flink’s Kafka sources and sinks. The Table API application uses createOrdersSource and createSkippedSink from this file. The sink for aggregated statistics is defined declaratively using a TableDescriptor instead of the createStatsSink function.

• createOrdersSource(...): Configures a KafkaSource to consume GenericRecord Avro data.
• createSkippedSink(...): Creates a generic KafkaSink for late records, which are handled as simple key-value string pairs.

  1package me.jaehyeon.kafka
  2
  3import me.jaehyeon.avro.SupplierStats
  4import org.apache.avro.Schema
  5import org.apache.avro.generic.GenericRecord
  6import org.apache.flink.connector.base.DeliveryGuarantee
  7import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema
  8import org.apache.flink.connector.kafka.sink.KafkaSink
  9import org.apache.flink.connector.kafka.source.KafkaSource
 10import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer
 11import org.apache.flink.formats.avro.registry.confluent.ConfluentRegistryAvroDeserializationSchema
 12import org.apache.flink.formats.avro.registry.confluent.ConfluentRegistryAvroSerializationSchema
 13import org.apache.kafka.clients.consumer.ConsumerConfig
 14import org.apache.kafka.clients.producer.ProducerConfig
 15import java.nio.charset.StandardCharsets
 16import java.util.Properties
 17
 18fun createOrdersSource(
 19    topic: String,
 20    groupId: String,
 21    bootstrapAddress: String,
 22    registryUrl: String,
 23    registryConfig: Map<String, String>,
 24    schema: Schema,
 25): KafkaSource<GenericRecord> =
 26    KafkaSource
 27        .builder<GenericRecord>()
 28        .setBootstrapServers(bootstrapAddress)
 29        .setTopics(topic)
 30        .setGroupId(groupId)
 31        .setStartingOffsets(OffsetsInitializer.earliest())
 32        .setValueOnlyDeserializer(
 33            ConfluentRegistryAvroDeserializationSchema.forGeneric(
 34                schema,
 35                registryUrl,
 36                registryConfig,
 37            ),
 38        ).setProperties(
 39            Properties().apply {
 40                put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, "500")
 41            },
 42        ).build()
 43
 44fun createStatsSink(
 45    topic: String,
 46    bootstrapAddress: String,
 47    registryUrl: String,
 48    registryConfig: Map<String, String>,
 49    outputSubject: String,
 50): KafkaSink<SupplierStats> =
 51    KafkaSink
 52        .builder<SupplierStats>()
 53        .setBootstrapServers(bootstrapAddress)
 54        .setKafkaProducerConfig(
 55            Properties().apply {
 56                setProperty(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true")
 57                setProperty(ProducerConfig.LINGER_MS_CONFIG, "100")
 58                setProperty(ProducerConfig.BATCH_SIZE_CONFIG, (64 * 1024).toString())
 59                setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4")
 60            },
 61        ).setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
 62        .setRecordSerializer(
 63            KafkaRecordSerializationSchema
 64                .builder<SupplierStats>()
 65                .setTopic(topic)
 66                .setKeySerializationSchema { value: SupplierStats ->
 67                    value.supplier.toByteArray(StandardCharsets.UTF_8)
 68                }.setValueSerializationSchema(
 69                    ConfluentRegistryAvroSerializationSchema.forSpecific<SupplierStats>(
 70                        SupplierStats::class.java,
 71                        outputSubject,
 72                        registryUrl,
 73                        registryConfig,
 74                    ),
 75                ).build(),
 76        ).build()
 77
 78fun createSkippedSink(
 79    topic: String,
 80    bootstrapAddress: String,
 81): KafkaSink<Pair<String?, String>> =
 82    KafkaSink
 83        .builder<Pair<String?, String>>()
 84        .setBootstrapServers(bootstrapAddress)
 85        .setKafkaProducerConfig(
 86            Properties().apply {
 87                setProperty(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true")
 88                setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4")
 89                setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArraySerializer")
 90                setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArraySerializer")
 91            },
 92        ).setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
 93        .setRecordSerializer(
 94            KafkaRecordSerializationSchema
 95                .builder<Pair<String?, String>>()
 96                .setTopic(topic)
 97                .setKeySerializationSchema { pair: Pair<String?, String> ->
 98                    pair.first?.toByteArray(StandardCharsets.UTF_8)
 99                }.setValueSerializationSchema { pair: Pair<String?, String> ->
100                    pair.second.toByteArray(StandardCharsets.UTF_8)
101                }.build(),
102        ).build()

Table API Processing Logic

The following files contain the core logic specific to the Table API implementation.

Manual Late Data Routing

Because the Table API does not have a direct equivalent to the DataStream API’s .sideOutputLateData(), we must handle late records manually. This ProcessFunction is a key component. It inspects each record’s timestamp against the current watermark and an allowedLatenessMillis threshold. Records deemed “too late” are routed to a side output, while on-time records are passed downstream to be converted into a Table.

 1package me.jaehyeon.flink.processing
 2
 3import org.apache.flink.streaming.api.functions.ProcessFunction
 4import org.apache.flink.util.Collector
 5import org.apache.flink.util.OutputTag
 6
 7class LateDataRouter(
 8    private val lateOutputTag: OutputTag<RecordMap>,
 9    private val allowedLatenessMillis: Long,
10) : ProcessFunction<RecordMap, RecordMap>() {
11
12    init {
13        require(allowedLatenessMillis >= 0) {
14            "allowedLatenessMillis cannot be negative. Got: $allowedLatenessMillis"
15        }
16    }
17
18    @Throws(Exception::class)
19    override fun processElement(
20        value: RecordMap,
21        ctx: ProcessFunction<RecordMap, RecordMap>.Context,
22        out: Collector<RecordMap>,
23    ) {
24        val elementTimestamp: Long? = ctx.timestamp()
25        val currentWatermark: Long = ctx.timerService().currentWatermark()
26
27        // Element has no timestamp or watermark is still at its initial value
28        if (elementTimestamp == null || currentWatermark == Long.MIN_VALUE) {
29            out.collect(value)
30            return
31        }
32
33        // Element has a timestamp and watermark is active.
34        // An element is "too late" if its timestamp is older than current watermark - allowed lateness.
35        if (elementTimestamp < currentWatermark - allowedLatenessMillis) {
36            ctx.output(lateOutputTag, value)
37        } else {
38            out.collect(value)
39        }
40    }
41}

Timestamp and Watermark Strategy for Rows

After the initial stream processing and before converting the DataStream to a Table, we need a watermark strategy that operates on Flink’s Row type. This strategy extracts the timestamp from a specific field index in the Row (in this case, field 1, which holds the bid_time as a Long) and generates watermarks, allowing the Table API to correctly perform event-time windowing.