There is a project by Dassault Systèmes called the Living Heart that illustrates the trajectory of this technology. Instead of relying on standard 2D scans, surgeons can pull up a 3D model of a patient’s heart that simulates blood flow, mechanics, and electricity based on real patient data. It allows doctors to test therapeutic interventions before surgery begins.

This highlights what a modern digital twin looks like in practice. It has moved past being a static 3D CAD file and serves as a persistent link between physical and digital environments, updating continuously based on operational data.

Industry Applications

While manufacturing initially drove this technology, digital twins are now expanding across a wider spectrum of advanced applications.

• Smart Manufacturing: On automated factory floors, robotic arms stream live telemetry to virtual models. This allows plant managers to predict maintenance needs and test routing scenarios without halting physical production lines.
• Supply Chain and Logistics: Global supply chains use digital twins to track everything from warehouse inventory levels to fleet movements, achieving end-to-end visibility. Operators can simulate supply chain shocks and dynamically resolve bottlenecks before they impact customers.
• Cybersecurity and Infrastructure: Network digital twins are increasingly used to replicate API gateways and load balancers. Subjecting a live production server to a simulated DDoS attack carries unacceptable downtime risks. Instead, security teams use virtual replicas as isolated environments to safely test rate-limiting and AI behavioral rules.
• Next-Generation Telecommunications: Modern 5G networks are incredibly complex. Telecom operators build digital twins of their networks to manage dynamic bandwidth allocation and network slicing, simulating massive traffic spikes across thousands of nodes before they cause cellular outages.

Reality Gap: Simulations vs. Digital Twins

The tech industry frequently uses the terms “simulation” and “digital twin” interchangeably. Clarifying the technical difference between the two is critical for evaluating modern control architectures. To understand this difference practically, let’s look at a factory floor where “Machine B” suddenly breaks down and its capacity drops to zero.

Here is how three different architectural systems handle that exact event:

Traditional Simulation (Sealed Box)

A simulation is a predictive model designed to explore “what if” scenarios within bounded parameters, relying heavily on static inputs and batch processing. Many traditional simulation environments, including standard Discrete-Event Simulation (DES) tools, were not designed for continuous live operational updates.

• Scenario: On Monday morning, an engineer runs a DES model of the factory. On Tuesday, Machine B physically breaks. Because the traditional simulation operates as a sealed box, it is unaware of real-world changes once the mathematical execution begins. Unless an engineer manually stops the simulation, rewrites the parameters, and restarts it, the model will incorrectly continue assuming Machine B is running at full capacity.

Operational Digital Twin (Live Mirror)

At the advanced end of the spectrum, an operational digital twin maintains a continuous, two-way connection to reality. Instead of relying on static snapshots, it utilizes distributed computing, time-series databases, and IoT connectivity to assess the exact current operational state.

• Scenario: The moment Machine B breaks, sensors ping the cloud database. The digital twin dashboard immediately turns red, updating the machine’s live status to “Offline” and its current capacity to zero. However, it only tells you what is happening right now. It does not automatically recalculate the mathematical impact to simulate the future consequences of that breakdown.

Hybrid Execution (Dynamic Adaptation)

The hybrid approach bridges this gap by wiring the live state of the digital twin directly into a running simulation engine.

• Scenario: The operational twin registers that Machine B is broken and emits a live Kafka event. A background hybrid simulation engine ingests that event while it is still running. It dynamically mutates Machine B’s capacity parameter to zero and recalculates the logic on the fly. The system now automatically outputs simulated telemetry showing the immediate cascading consequences: “Machine B’s queue is backing up, routing logic must shift to Machine D, and a critical bottleneck will form at Machine C in exactly 45 minutes if no intervention is taken.”

Upgraded Simulation Fallacy

Because of this market confusion, organizations often fall into the “Upgraded Simulation Fallacy.” They purchase simulation software, attach a 3D dashboard, and classify the project as a digital twin. However, if the system cannot process continuous data streams or synchronize with live states, it fundamentally remains a static simulation.

Transitioning to an operational digital twin introduces strict engineering realities:

• Cost and Complexity: Building them requires heavy investments in sensors, edge computing, and complex system architectures.
• Cybersecurity Risks: Connecting physical infrastructure to cloud-based models expands the attack surface.
• Data Pipelines and Standardization: Getting disparate sensors to speak the same language and pipe into a central engine is an architectural challenge.

This last point is where many projects stall. If you force real-time updates into a legacy simulation engine, the simulation clock often bottlenecks while waiting for network requests, or the system fails due to unpredictable data. To build a functional digital twin, your simulation engine must be designed to handle asynchronous network I/O so the underlying math model runs uninterrupted, while enforcing strict data schemas (like Pydantic or Avro) to maintain system stability.

Taking Control with a Hybrid Approach

Ultimately, organizations shouldn’t have to choose between a static simulation and a live twin. The future of Industry 4.0 is a hybrid approach: running “what-if” simulations against live digital twin states.

By feeding the current, real-time state of a digital twin directly into a simulation engine, engineers can forecast the next four hours of production or network traffic based on exact current conditions.

But this introduces a major architectural problem: How do you take a live digital twin state, ingest an asynchronous Kafka stream, and dynamically alter the capacity of a running virtual machine without lagging behind real-time?

One approach to solving this class of problem is the dynamic-des package. In Part 2 of this series, we will look at how the package’s Switchboard pattern and dynamic resources can help turn a static mathematical model into a synchronized, event-driven digital twin.

References

1. Attaran, M., & Celik, B. G. (2023). Digital Twin: Benefits, use cases, challenges, and opportunities. Decision Analytics Journal, 6, 100165.
2. Javaid, M., Haleem, A., & Suman, R. (2023). Digital Twin applications toward Industry 4.0: A Review. Cognitive Robotics, 3, 71-92.
3. Niantic Spatial. (2025). Simulations vs. Digital Twins. Niantic Spatial Campaigns.

Imagine using a rolling pin to flatten out a thick piece of dough. A Hot Strip Mill does the exact same thing, but with glowing red-hot steel slabs (often heated over 1000°C) and massive mechanical rollers. The steel is passed through a series of these rollers, crushing it down from a thick block into a long, thin sheet.

Calculating the exact Rolling Force required to crush the steel is critical. If the machine pushes too hard, it can severely damage the rollers; if it doesn’t push hard enough, the steel doesn’t reach the target thickness. Because the rollers are constantly grinding against raw steel, their physical shape slowly degrades over time. As the machinery wears down, the legacy mathematical formulas used to predict that perfect force slowly become inaccurate. This physical degradation is the root of the Concept Drift our real-time ML pipeline is solving.

In heavy industrial manufacturing, such as steel hot strip rolling, deterministic physics formulas are the traditional standard for calculating the exact force required to deform a slab of steel. However, these pure physics models share a fatal flaw: they assume a pristine factory state. As physical rollers grind against red-hot steel over hours of production, they experience mechanical wear.

As the machinery degrades, the actual physical force required drifts away from the theoretical prediction. In data science, this is a classic manifestation of Concept Drift.

To tackle this, I recently built a real-time, fault-tolerant Online Machine Learning (OML) pipeline and Digital Twin. By combining Apache Kafka, Apache Flink (written in Kotlin), and the Massive Online Analysis (MOA) framework, the system learns the new physical reality of the worn machinery on the fly, autonomously correcting the physics baseline safely behind a deterministic Shadow Mode router.

Here is an overview of the architecture and the engineering challenges solved along the way. The complete source code for this project, including the Flink stream processor and the Python Digital Twin simulation, is available on GitHub.

Architecture at a Glance

The project is split into three highly decoupled domains:

1. Digital Twin (Python): Utilizing the Dynamic DES Python package, this layer generates synthetic rolling events, applies simulated mechanical wear, calculates theoretical/actual physics, and pushes the data to Kafka.
2. Message Broker (Kafka): Handles the asynchronous, high-throughput streaming of prediction requests and delayed ground-truth target forces.
3. Stream Processor (Flink/Kotlin): The core engine. It aligns asynchronous streams, trains the machine learning models dynamically, evaluates safety guardrails, and sinks metrics to ClickHouse for evaluation in real-time.

Moving data from the Digital Twin through Kafka to Flink looks straightforward in a diagram, but real-world industrial physics and distributed systems are highly complex. To make this pipeline robust enough for the unpredictability of a live factory floor, we had to overcome four major engineering challenges.

Challenges

Taking an online machine learning model out of the lab and deploying it into a live industrial environment introduces a unique set of hurdles. From managing the physical delays of factory sensors to ensuring the model never commands a physically unsafe action, here is a breakdown of the four primary engineering challenges we tackled to make this architecture more robust.

Challenge 1: Asynchronous Industrial Streams

Industrial data streams are inherently asynchronous. The factory floor requests a prediction before the steel enters the rolling stand (Event A), but the ground-truth sensor data confirming the actual required force arrives after the steel is crushed (Event B).

To execute machine learning, these two events must be perfectly joined. However, network latency and partition skew in Kafka mean that these events might arrive out of order.

In Flink, this is solved using a KeyedCoProcessFunction. By keying the streams on a composite Slab ID + Pass Number, Flink guarantees both events route to the exact same TaskManager. The EventMatchProcessFunction utilizes Flink’s ValueState to buffer whichever event arrives first. It then registers a processing-time timer (Time-To-Live). If the matching event arrives, they are joined and emitted. If the timer expires (e.g., a physical sensor failure), the orphaned state is safely purged to prevent memory leaks.

Challenge 2: Online Residual Learning

Instead of training a batch model offline on historical data, which would instantly become obsolete the moment the rollers wore down further, the Flink pipeline trains on the continuous stream of slabs using a strict Test-then-Train (prequential) paradigm.

Crucially, the ML models do not predict the absolute rolling force from scratch. Instead, they utilize Residual Learning. They predict the residual error between the theoretical formula and the actual physical force.

To execute this, the pipeline integrates the Java MOA framework. The primary production model is AMRules (Adaptive Model Rules), a streaming rule-learning algorithm. It builds an ensemble of rules that calculate a linear combination of physical attributes, continuously updating weights via the Delta rule.

Unlike static models, AMRules runs an internal Page-Hinkley test to detect sudden physical shocks (like a roller bearing breaking) and instantly prunes obsolete rules, allowing rapid convergence to new physical realities. Furthermore, because ML models require normalized inputs, the pipeline implements Welford’s Online Algorithm in a custom Flink state object to calculate streaming Z-scores on the fly without needing to load full datasets into memory.

Challenge 3: Shadow Mode Router

Industrial Machine Learning cannot operate without strict safety boundaries. A model error that generates excessive rolling force could severely damage a multi-million-dollar rolling stand or create a major production bottleneck in the factory.

Before any ML-adjusted prediction is allowed to influence the factory floor, it must pass through a deterministic Shadow Mode Router consisting of two guardrails:

1. Stateless Mechanical Limits: Calculates the residual difference between the model’s requested force and the physics baseline. If the model requests a deviation outside of physical safety bounds (e.g., > +25% or < -20%), the prediction is instantly rejected.
2. Stateful Trust Score: Using an Exponentially Weighted Moving Average (EWMA), the router continuously tracks the Absolute Percentage Error (APE) of both the model and the pure physics baseline. If the model’s average error trails the physics baseline by a defined “Trust Deficit” margin, it is benched.

If either guardrail is triggered, the router safely falls back to the deterministic Physics Baseline. The ML model continues to train in the background (Shadow Mode) until it re-learns the physical reality, improves its EWMA trust score, and is autonomously promoted back to active control.

Challenge 4: Checkpointing Complex ML State

For Flink to provide exactly-once processing guarantees, it must asynchronously snapshot the state of operators to a durable backend like RocksDB.

While simple tracking states serialize perfectly into basic Flink ValueState<Double>, the AMRules model is a deeply complex, dynamic tree structure. Allowing Flink’s default Kryo serializers to traverse this massive object graph during checkpointing causes severe performance degradation and frequent serialization crashes.

To bypass this, the MoaEvaluationProcessFunction interacts with the MOA model as a standard Java object in memory for high-performance execution. However, upon every state update, it manually serializes the model into a raw byte array (ValueState<ByteArray>) using Java’s native ObjectOutputStream. When Flink triggers a checkpoint, it simply flushes these pre-serialized byte arrays to disk. Upon recovery, Flink deserializes the bytes, instantly restoring the exact computational brain-state of the algorithm.

System in Action

You can easily spin up the entire architecture locally to see the system in action. The project repository includes a complete Docker Compose environment that provisions Kafka, Flink, and ClickHouse.

To get started, simply clone the repository, build the Flink application using the provided Gradle wrapper, and bring up the infrastructure. Once the environment is running, you can launch the Python-based data generator to simulate the steel rolling process and open the NiceGUI control dashboard to monitor the machine learning metrics in real-time.

For detailed, step-by-step instructions on cloning the repository, configuring licenses, and deploying the Flink job, refer to the Getting Started section in the repository.

Simulation Scenarios

Using the NiceGUI Dashboard, you can actively manipulate the physical state of the Digital Twin to observe the Flink pipeline’s reaction in real-time.

Here is how the pipeline handles different industrial scenarios:

Abrupt Drift (Mechanical Shock)

Simulates a sudden mechanical failure (e.g., a roller bearing breaking), instantly altering the physics of the mill.

• Simulation Settings: Trigger Abrupt Shock (Wear Level: 60.0).
• Observation: The pure physics baseline error instantly spikes and remains high (often >10% APE) because the physical reality no longer matches the math. The AMRules model initially spikes alongside it, but its Page-Hinkley change detector immediately drops obsolete rules, allowing it to rapidly converge back to lower error as it learns the new broken state.

Gradual Drift (Standard Wear)

Simulates the continuous, bi-directional cycle of slow roller degradation and subsequent maintenance recovery over hours of production.

• Simulation Settings: Gradual Wear (Step Size: 5.0 units, Frequency: 30 seconds).
• Observation: The physics baseline error slowly and persistently creeps upward/download over time (e.g. ranging from 2% to 7% APE) as the wear level drifts gradually. The AMRules model gracefully tracks this changing reality, updating its linear weights incrementally to maintain a smooth error rate.

No Drift (Pristine State)

Simulates a pristine factory state, such as immediately after a maintenance shift replaces the rollers.

• Simulation Settings: Wear Level: 0.0.
• Observation: The physical reality of the factory floor perfectly aligns with the deterministic mathematical formulas. The physics baseline maintains a highly accurate, near-zero error rate (< 0.3% APE). AMRules remains stable under this condition.

Conclusion

Combining Apache Flink with Online Machine Learning bridges the gap between theoretical physics and the harsh reality of a degrading factory floor. To make this safe and effective, the architecture relies on three core ideas: predicting residual errors instead of absolute forces, isolating models by product line to prevent forgetting, and enforcing strict safety guardrails. Together, these techniques ensure that heavy machinery operates optimally, even as it breaks down over time.

You can explore the complete codebase, run the simulation locally, and dive deeper into the architecture on the project’s GitHub repository.

Building a Real-Time Product Recommender

Contextual Bandits & Event-Driven Architecture

Why Contextual Bandits?

• Problem: Conventional recommenders (e.g., Collaborative Filtering)
  • Ignore situational context (e.g., Time of Day, Location, Device).
  • Struggle with “Cold Starts” for new items/users.
• Solution: Contextual Multi-Armed Bandits (CMAB).
  • Exploitation: Maximize immediate reward using current knowledge.
  • Exploration: Gather information on uncertain items to improve future performance.

Part 1: Prototype | Part 2: Productionization

Part 1: Prototype

Prototype an online product recommender with Python

–

Python Ecosystem

• Vowpal Wabbit and River ML are well-known for CMAB.
  • Gap: Lack of end-to-end examples integrating feature engineering and offline policy evaluation.
• Fidelity Investments Open Source
  • MABWiser: Algorithm implementation.
  • Mab2Rec: Offline policy evaluation.
  • TextWiser: Text featurization.

–

Prototyping Workflow

From synthetic data generation to live simulation.

–

Live Demo & Walkthrough

Let’s dive into the code.

–

Offline Policy Evaluation

Why LinUCB?

• Best Trade-off: High Ranking (AUC) + High Engagement (CTR).
• Beats LinGreedy: Explores effectively (CTR 0.20 vs 0.11).
• Beats LinTS: Ranks accurately (AUC 0.86 vs 0.64).

–

LinUCB Algorithm

Balancing Exploitation and Exploration

$$ \text{Score}_a = \color{cyan}{x^T \theta_a} \mathbin{\color{white}{+}} \color{orange}{\alpha \sqrt{x^T A_a^{-1} x}} \color{white}{, \quad \text{where } \theta_a = A_a^{-1} b_a} $$

• ● Exploitation: Predicted reward ($x^T \theta_a$).
• ● Exploration: Uncertainty bonus (UCB).
• $\theta_a = A_a^{-1} b_a$: Model weights estimated via Ridge Regression.

–

Limitations

A monolithic Python script isn’t built for scale.

• Latency: Training blocks inference.
• Scalability: Matrix math in memory limits the catalog size.
• Fault Tolerance: If the script crashes, the learned state is lost.

Back to Start | Jump to Productionization

Part 2: Productionization

Scaling with an Event-Driven Architecture

–

Architecture

Training (Apache Flink)

• Stateful Processing:

  • Flink acts as “Online Memory”.

• Asynchronous Updates:

  • Fast Path: Updates $A$ and $b$.
    • ($A \leftarrow A + x x^T, b \leftarrow b + r x$)
  • Slow Path: Every 5s, computes $A^{-1}$.

• Sync to Redis: Emits $A^{-1}$ and $b$.

–

Architecture

Serving (Python & Redis)

• Stateless Inference:

  • The client does not train.

• Low Latency:

  • Fetches pre-computed LinUCB parameters from Redis.

• Action:

  • Calculates scores,
  • Ranks items, and
  • Sends feedback to Kafka.

–

Architecture

Transport (Apache Kafka)

• Asynchronous Buffer:

  • Decouples user-facing app speed from backend training.

• Durability:

  • Stores “Ground Truth” events safely for replay or analytics.

–

Live Demo & Walkthrough

Let’s dive into the code.

Back to Start | Jump to Takeaways

Key Takeaways

Bridging the gap between Data Science and Data Engineering.

–

Start Small, Evaluate Offline

• Before touching infrastructure, evaluate policies.
• Using tools like MABWiser and Mab2Rec allows you to simulate user behavior and validate algorithms on historical data safely.

–

Decouple to Scale

• A monolithic architecture forces a trade-off between model accuracy and user latency.
• Event-Driven Architecture (EDA) solves this by separating high-speed inference (Redis) from heavy stateful training (Flink).

–

Real-Time Adaptability

• By integrating Kafka and Flink, the system learns from user behavior instantly.
  • Dynamic Personalization: Optimizes for specific real-time user context with every click.
  • Continuous Learning: Eliminates the “Cold Start” problem for new items without batch-job downtime.

Thank You!

Code & Resources:

• GitHub Repository
• Blog Posts: Part 1: Prototype | Part 2: Productionization
• Youtube Playlist

Back to Start

Demonstrating Reveal.js Featuresfor Engineering Presentation

1. Image Embedding

Standard Markdown or HTML backgrounds

–

Full Background Slide

(Use – for vertical slides)

Cosmic Background

This slide has a remote image background.

2. LaTeX Math

Rendered via MathJax

The Probability Density Function for a Normal Distribution:

$$ f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{ -\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2 } $$

3. HTML Tables

Markdown and HTML mix

4. Charts (Mermaid.js)

Diagrams from Code

5. Code Highlighting

Highlight specific lines (Click next)

1def process_stream(data):
2    # Step 1: Clean
3    data = data.strip()
4    data = data.lower()
5    
6    # Step 2: Return
7    return data

6. Fragments

Step-by-step visibility

• Item 1 (Fades in)
• Item 2 (Fades in second)
• Item 3 (Grows larger)
• Item 4 (Turns red)
• Item 5 (Fades out)

Choosing between Marp and Reveal.js

When looking for a solution, I initially considered Marp.

Marp is undeniably easier to author initially; its VS Code extension with live preview makes designing slides feel like magic. However, it presented a hurdle for my Hugo integration as it requires separate compilation.

Marp requires a separate build step to convert Markdown into standalone HTML. To integrate it, I would have to manually export each deck and move the resulting HTML into my static/ folder. This felt “disconnected” from the Hugo ecosystem.

Reveal.js, on the other hand, is a browser-based engine. By using Reveal.js, I can:

1. Stay Native: Write pure Markdown in my content/ folder just like a blog post.
2. Zero Manual Export: Hugo handles the routing, and the browser handles the rendering at runtime.
3. Automatic Updates: When I git push a Markdown file, the slides are live instantly. No separate compilation, no manual file moving.

See it in Action

Before diving into the configuration, you can interact with the live demo below. This is the actual slide deck, rendered natively by Hugo and embedded directly into this post:

You can use your arrow keys to navigate the slides above, or click the “View Full Screen” button for the complete experience.

How It Works

I chose to keep slides independent from my main blog posts to avoid cluttering my main feed. Using Hugo’s Leaf Bundles, each presentation gets its own folder for local assets like images.

1content/slides/
2└── 2026-03-06-revealjs-demo/            
3    ├── index.md                          # The slide content
4    └── featured.jpg                      # Thumbnail for list view

Custom “Shell” Layout

The secret is a custom layout at layouts/slides/single.html. Instead of using my theme’s standard blog layout (which includes headers and sidebars), I created a clean “shell” that loads the Reveal.js engine from a CDN.

Crucially, I use {{ .RawContent }} instead of {{ .Content }}. This is because Reveal.js needs the raw Markdown syntax (like --- for slide breaks) to calculate transitions in the browser, rather than the pre-rendered HTML Hugo usually provides.

Advanced Technical Features

Because Reveal.js is a web-based runtime, it allows for high-end technical storytelling that traditional slide software can’t easily handle:

Interactive Code Storytelling

Standard syntax highlighting is good, but Reveal.js allows for directed walkthroughs. Using a specific syntax like [1|3-5|7], I can guide the audience through a specific block of code:

• Workflow: First, I highlight the function definition, then step into the processing logic, and finally focus on the return statement. This “guided tour” of the code prevents the audience from getting overwhelmed by a large block of text.

Precision Media Styling

Using the .element comment syntax, I have granular control over CSS directly within the Markdown.

• Dynamic Backgrounds: I can switch the entire slide context using the data-background attribute (like a high-impact “Cosmic” space background).
• CSS Overrides: I can apply borders, rounded corners, and specific alignments, for example, centering a gold-bordered image without ever leaving the Markdown file.

Managing Cognitive Load with Fragments

To keep the audience focused on the current talking point, I use Fragments. This provides progressive disclosure of information. Beyond just “appearing,” I can use classes like grow to emphasize a specific item or highlight-red to signal a warning, all triggered by a simple keypress.

Mathematical Precision (LaTeX/MathJax)

For engineering-heavy talks, raw text equations look unprofessional. By integrating MathJax, I can render complex mathematical models, such as Probability Density Functions, with the same precision found in a LaTeX paper. They remain sharp at any zoom level because they are rendered as vectors in the browser.

Diagrams as Code (Mermaid.js)

Instead of embedding static PNGs of architecture diagrams that inevitably become outdated, I use Mermaid.js. This allows me to define data pipelines, like a Kafka-to-Flink-to-Redis flow, directly in the Markdown text.

• Engineering Value: Diagrams are now version-controlled and searchable. If the architecture changes, I update a few lines of text, and the visual layout re-renders automatically.

Road Ahead: Professional Diagrams with Draw.io

While Mermaid.js is excellent for rapid, text-based flowcharts, complex system architectures often require the precision of a dedicated design tool. I am currently investigating the best way to integrate Draw.io into this “Slides-as-Code” workflow without losing the benefits of version control.

Currently, I am evaluating two brief paths:

• Editable SVGs: Exporting diagrams as SVGs with embedded XML. This allows the slides to treat the diagram as a standard image while keeping the source file fully editable and version-controlled.
• The Draw.io JS Viewer: Utilizing the official Diagrams.net JavaScript library to render .drawio XML files dynamically. This would enable interactive features like zooming and layer toggling directly during a presentation.

Moving forward, the goal is to ensure that even the most complex architectural designs remain as easy to maintain as a line of Markdown.

Why this works for me

By leveraging Hugo’s Directory Inference, any file I drop into content/slides/ automatically inherits this powerful shell. I no longer “design” slides; I engineer them.

While Marp might be slightly easier to write, the Reveal.js integration is much easier to maintain. It turns my site into a single source of truth for both my articles and my presentations.

You can check out my latest presentations in the Slides section!

This post demonstrates how to decouple these concerns using an event-driven architecture with Apache Flink, Kafka, and Redis.

System Architecture

To move from prototype to production, we split the application into two distinct layers: Serving and Training.

• Python Client (Serving): A lightweight, stateless client responsible for inference. It fetches pre-calculated model parameters from Redis, computes scores locally to make product recommendations, and captures user feedback.
• Kafka (Transport): Buffers feedback events asynchronously, decoupling the speed of serving from the speed of training.
• Flink (Training): A stateful streaming application. It consumes feedback events, updates the model parameters (LinUCB matrices $A$ and $b$), and pushes the inverted matrices back to Redis.
  • ❗ Unlike Part 1, where training relied on MABWiser, here it is performed via explicit matrix operations.
• Redis (Model Store): Stores the latest model parameters ($A^{-1}$ and $b$) for low-latency access by the client.

📂 Source Code for the Post

The source code for this post is available in the product-recommender folder of the streaming-demos GitHub repository.

Flink Application Design

The Flink job (recsys-trainer) ties these concepts together using a few specific patterns.

Stateful Model Training

The core challenge in distributed online learning is managing state. The LinUCBUpdater function in the Flink trainer acts as the system’s memory. It implements a disjoint LinUCB model, meaning it maintains a completely independent set of matrices for each unique product.

❗The matrices are used to calculate scores for making recommendations.

For each product_id, Flink maintains two pieces of state in RocksDB:

1. Matrix $A$ ($d \times d$): Represents the covariance of features seen so far. It tracks Exposure, recording how many times specific user contexts (e.g., “Morning Users” or “Weekend Users”) have been seen for a specific product.
2. Vector $b$ ($d \times 1$): Represents the accumulated reward. It tracks Success, recording which features actually led to a click.

The matrix $A$ is initialized as a scaled identity matrix $A_0 = \lambda I$ to ensure invertibility and to encode an initial prior of uniform uncertainty across feature dimensions.

When a feedback event arrives (Context $x$, Reward $r$), Flink performs the updates:

• Update A: $A \leftarrow A + x x^T$. The outer product $x x^T$ increases covariance along the observed feature directions. As similar contexts repeat, $A$ grows in those directions, reflecting increased confidence.
• Update b: $b \leftarrow b + r x$. If the user clicked ($r=1$), we add their feature vector to $b$, reinforcing that preference pattern.

The updated $A$ and $b$ are stored immediately in Flink keyed state (backed by RocksDB). With checkpointing enabled, this state is durably persisted and recovered in case of failure.

Optimization 1: Inversion on Write

To generate a score, we need the inverse matrix $A^{-1}$, which is computationally expensive. If we performed this inversion inside the Python client for every recommendation request, latency would increase significantly. Instead, the Flink training job periodically loads $A$ from state, factorizes it using LU decomposition, computes $A^{-1}$, and stores the inverse in Redis. Because the contextual feature dimension in this demo is small, recomputing the inverse periodically remains efficient while keeping the serving layer lightweight.

Optimization 2: Batched Updates

In a high-traffic environment, a popular product might receive thousands of clicks per second. Inverting the matrix and writing to Redis for every single click would be inefficient.

To solve this, we use Flink timers to buffer updates. The model state ($A$ and $b$) is updated immediately for every event, while the expensive inversion and Redis write are triggered periodically (e.g., every 5 seconds). This drastically reduces CPU load and network traffic while keeping the model fresh.

Scalable Inference Logic

The Python client (eda_recommender.py) is responsible for ranking items. It uses the Upper Confidence Bound (UCB) formula to balance exploiting known good items and exploring uncertain ones.

For a given user context vector $x$ and product $a$, the score is calculated as:

$$ \text{Score}_a = x^T \theta_a + \alpha \sqrt{x^T A_a^{-1} x}, \quad \text{where } \theta_a = A_a^{-1} b_a $$

Prediction ($x^T \theta_a$)
This is the standard Linear Regression prediction. It asks: “Based on historical data, how likely is this user to click?” If the user matches features stored in vector $b$ (features that previously led to clicks), this term is high.

Exploration ($\alpha \sqrt{x^T A_a^{-1} x}$)

• Familiar User: If we have seen this user type many times, the matrix $A$ accumulates repeated contributions of $x x^T$. This increases the magnitude of $A$ in those feature directions. Because the exploration term depends on $x^T A^{-1} x$, a larger $A$ leads to a smaller quadratic form, shrinking the confidence bound. The model therefore relies more on exploitation.
• Cold Start: If we have rarely (or never) observed this feature pattern, $A$ remains close to its initial regularized identity matrix. After inversion, these directions yield larger values of $x^T A^{-1} x$, increasing the confidence bound and encouraging exploration to reduce uncertainty.
• ❗ $\alpha$ is a hyperparameter and it is set to 0.1 as determined in Part 1.

Hybrid Source for Warm Start

Contextual bandits suffer from the “Cold Start” problem. To mitigate this, we implement a Hybrid Source.

1. File Source: Reads the historical CSV (training_log.csv) generated in Part 1 to bootstrap the model state.
2. Kafka Source: Automatically switches to the live feedback-events topic once the historical data is processed.

Custom Redis Sink

We implement a custom Sink using the Sink V2 API and Jedis. This allows us to perform efficient SET operations to update the model parameters in Redis directly from the Flink stream. Because each update overwrites the full parameter vector, repeated writes remain logically safe under at-least-once delivery semantics. Besides, because the upstream LinUCBUpdater batches the emissions, this sink receives highly aggregated model updates, preventing Redis from being overwhelmed by write operations.

Recommender Simulation Design

To validate the architecture without live user traffic, we designed a Python client (eda_recommender.py) that simulates the entire lifecycle of a recommendation request. This script plays two roles simultaneously: it acts as the Recommendation Service (serving predictions) and the User (providing feedback).

Serving Logic

In a production environment, this logic would live in a high-performance API. For this simulation, the Python client:

1. Context Generation: Creates a synthetic user profile (Age, Gender) and derives key temporal features (e.g., Morning, Weekend) from a simulated timestamp to form the full context.
2. Model Retrieval: Fetches the latest LinUCB parameters ($A^{-1}$ and $b$) for all active products directly from Redis.
3. Scoring and Ranking: Calculates the UCB score for every product, ranks them in descending order, and returns the top 5 highest-scoring items as the recommendation set.

Feedback Generation

To prove the model is learning, the simulation follows the same “Ground Truth” logic used in Part 1:

• Morning Routine: Users click “Drinks & Desserts” (Coffee) between 6 AM and 11 AM.
• Weekend Treats: On Saturdays and Sundays, users prefer “Pizzas” or “Burgers.”
• Price Sensitivity: Users under 25 avoid expensive items.

If any of the recommended top 5 items matches the user’s current context (e.g., showing a Latte on a Tuesday morning), the script generates a Reward (1). Otherwise, it generates no reward (0). This feedback is serialized to Avro and produced to Kafka, completing the loop.

Environment Setup

We use Docker Compose to orchestrate the infrastructure (Kafka, Flink, Redis) and Gradle to build the Kotlin application.

Prerequisites

Clone the repository and infrastructure utilities, then download the required connectors (Kafka, Flink, Avro).

 1git clone https://github.com/jaehyeon-kim/streaming-demos.git
 2cd streaming-demos
 3
 4# Clone Factor House Local for infrastructure definitions
 5git clone https://github.com/factorhouse/factorhouse-local.git
 6
 7# Download Kafka/Flink Dependencies
 8./factorhouse-local/resources/setup-env.sh
 9
10cd product-recommender

Build and Launch

We bootstrap the environment by generating training data, building the Flink JAR, and launching the cluster. We use Kpow and Flex to monitor the Kafka and Flink clusters; these tools require a Factor House community license. Visit the Factor House License Portal to generate your license, save the details in a file (e.g., license.env), and export the associated environment variables (KPOW_LICENSE and FLEX_LICENSE).

With the license configured, launch the Docker Compose services as shown below.

❗ You do not need Kotlin or Gradle installed locally. The ./gradlew script handles all build dependencies.

 1# Setup Python and Generate Bootstrap Data
 2python3 -m venv venv
 3source venv/bin/activate
 4pip install -r requirements.txt
 5python recsys-engine/prepare_data.py
 6
 7# Build Flink Application (Shadow Jar)
 8cd recsys-trainer
 9./gradlew shadowJar
10cd ..
11
12# Launch Infrastructure (Kafka, Flink, Redis, Kpow)
13export KPOW_SUFFIX="-ce"
14export FLEX_SUFFIX="-ce"
15export KPOW_LICENSE=<path-to-license-file>
16export FLEX_LICENSE=<path-to-license-file>
17
18docker compose -p kpow -f ../factorhouse-local/compose-kpow.yml up -d \
19  && docker compose -p stripped -f ./compose-stripped.yml up -d \
20  && docker compose -p flex -f ./compose-flex.yml up -d

Live Recommender Simulation

Once the infrastructure is running, Flink will first process the historical events to warm up. Once the historical processing is complete, we can run the Python client to simulate live traffic.

To visualize the system in action, open two terminals.

Terminal 1: Client

Run the Python script. It acts as the user, receiving recommendations and sending feedback (clicks) to Kafka.

1python recsys-engine/eda_recommender.py

Terminal 2: Trainer

Watch the Flink TaskManager logs. You will see the application reacting to the events sent to the feedback-events topic in real-time.

1docker logs taskmanager -f

Result

You will see a series of feedback events generated by users on the left-hand side. On the right-hand side, you can see the logs confirming that model parameters are being sent to Redis in batches.

This confirms the closed loop: Read (Redis) -> Act (Kafka) -> Learn (Flink) -> Write (Redis).

You can inspect feedback events on Kpow at http://localhost:3000.

Teardown

To stop the cluster and remove resources:

1docker compose -p flex -f ./compose-flex.yml down \
2  && docker compose -p stripped -f ./compose-stripped.yml down \
3  && docker compose -p kpow -f ../factorhouse-local/compose-kpow.yml down

Conclusion

Traditional recommendation systems such as Collaborative Filtering rely on long-term interaction history and often treat user preferences as static. As a result, they struggle to incorporate immediate context, missing situational shifts like a user preferring coffee in the morning but pizza in the evening.

To overcome this, we use Contextual Multi-Armed Bandits (CMAB), an online learning approach that balances exploitation and exploration using real-time contextual signals. While our Python prototype in Part 1 validated the concept, it was not built for scale.

We then evolved it into a production-ready event-driven architecture: Kafka streams feedback events, Flink handles distributed stateful training, and Redis serves precomputed parameters for low-latency inference. This design enables horizontal scalability and real-time adaptation to user behavior.

Traditional recommendation approaches such as Collaborative Filtering remain widely adopted, yet they come with notable constraints. They are particularly vulnerable to the cold-start problem, where new users lack sufficient interaction history, and they depend heavily on long-term behavioral data. As a result, they frequently overlook real-time contextual signals, including time of day, device type, location, or session intent. This can prevent them from capturing situational preferences, such as someone preferring coffee in the morning but pizza in the evening.

Contextual Multi-Armed Bandits (CMAB) address these gaps through online learning.

As a practical form of reinforcement learning, CMAB balances two goals in real time:

1. Exploitation: Recommending what is known to work.
2. Exploration: Trying less-tested options to discover new favorites.

By conditioning decisions on live context, CMAB adapts instantly to changing user behavior.

Why CMAB?

• Beyond A/B Testing: Instead of finding a single global winner, CMAB enables 1:1 personalization, selecting the best option for this user in this context.
• Real-Time Adaptation: Unlike batch-trained models that quickly become stale, CMAB updates incrementally, making it ideal for news/products recommendation, dynamic pricing, or inventory-aware ranking.

Several CMAB implementations exist, including Vowpal Wabbit and River ML. In this post, we use Mab2Rec for offline policy evaluation and MABWiser to build the product recommender prototype.

Data Streaming Opportunity

CMAB performs well in data streaming environments. Integrated with platforms like Kafka and Flink, it learns directly from event streams, creating a feedback loop that responds to trends and shifts in user intent in sub-seconds.

In this series, Part 1 (this post) builds a complete Python prototype to validate the algorithm and simulate user behavior. Part 2 will scale this to a distributed, event-driven architecture.

Tech Stack

We are building this prototype using Python 3.11.

Engineering Note: We explicitly chose Python 3.11 because parts of our stack (specifically mabwiser dependencies) rely on older versions of pandas (< 2.0). On Python 3.12+, installing these dependencies often triggers long compilation times or failures due to missing binary wheels.

We use uv for Python environment management. The core libraries include:

• MABWiser: The engine. It implements the core Contextual Bandit algorithms.
• Mab2Rec: The vehicle. A high-level wrapper that streamlines Recommender System pipelines.
• TextWiser: For converting raw text features into numerical embeddings.
• scikit-learn: For feature scaling and encoding.
• Faker & Pandas: For synthetic data generation and simulation.

The development environment can be constructed as follows:

 1$ git clone https://github.com/jaehyeon-kim/streaming-demos.git
 2$ cd streaming-demos
 3$ uv python install 3.11
 4$ uv venv --python 3.11 venv
 5$ source venv/bin/activate
 6(venv) $ uv pip install -r product-recommender/requirements.txt
 7(venv) $ uv pip list | grep -E "mab|wiser|panda|numpy|scikit|faker"
 8# Using Python 3.11.14 environment at: venv
 9# faker                              40.1.2
10# mab2rec                            1.3.1
11# mabwiser                           2.7.4
12# numpy                              1.26.4
13# pandas                             1.5.3
14# scikit-learn                       1.8.0
15# textwiser                          2.0.2

📂 Source Code for the Post

The source code for this post is available in the product-recommender folder of the streaming-demos GitHub repository.

Data Generation

We first need product and user data to generate the required features.

Products

We utilize a set of 200 raw products, each containing a product ID, name, text description, price, and high-level category.

Here is a list of sample products:

Users

We generate 1,000 Synthetic Users using Faker. Each user is assigned static attributes like Age, Gender, Location, and Traffic Source. These attributes will serve as the “Context” for our Bandit.

Here is a sample of our user base:

Note that street address, postal code, city, state, and country are omitted, as only latitude and longitude are used for feature generation.

Feature Engineering

Bandit algorithms operate on numerical vectors, not raw text. In other words, they cannot interpret "Burger" unless it is converted into numbers. To address this, we developed a transformation pipeline to properly prepare our data:

1. Product Features: We used TextWiser to convert raw product descriptions into vector embeddings. This allows the model to understand that “Burger” and “Sandwich” are semantically closer than “Burger” and “Headphones”. We also applied One-Hot Encoding to categories (Product Category) and MinMax scaling to the price. Finally, we added a binary feature, is_coffee, which is set to 1 for coffee products (e.g., espresso, cappuccino) and 0 otherwise.
2. User Features: Similar to the product features, we applied One-Hot Encoding to categories (Gender and Traffic Source) and MinMax scaling to numerical fields (Age, Latitude, and Longitude).
3. Pipeline Artifacts: We save these transformers as preprocessing_artifacts.pkl. This allows our system to instantly transform any new user/product record into a compatible feature vector during inference.

Sample Processed Product Features:

Notice how the description is now represented by txt_0…txt_9 embeddings.

Sample Processed User Features:

Notice that Age, Latitude and Longitude are normalized between 0 and 1, and categorical fields are binary.

Bandit History Simulation

To evaluate whether our model can truly learn user behavior, we need a controlled Ground Truth, which is an Oracle that determines the likelihood of a simulated user clicking on a recommendation.

Crucially, this Oracle is hidden from the model. The model’s task is to infer these patterns purely from trial and error.

We also inject Dynamic Context features like Time of Day and Day of Week into the user profile at the moment of interaction. These temporal signals create realistic, fluctuating patterns that the model must adapt to.

Simulation Logic

The simulation is implemented as a class GroundTruth, and we define specific rules that govern user behaviour:

• Start from a low base logit (−2.5) to model generally low click probability.
• Rule 1: Morning coffee preference: if the user is browsing in the morning and the item is a coffee product, add a strong positive boost to the score.
• Rule 2: Weekend comfort food: if the session is on a weekend and the item is Pizza or Burgers & Sandwiches, add a moderate positive boost.
• Rule 3: Budget sensitivity: if the user is young (normalized age < 0.25) and the item is expensive (normalized price > 0.8), apply a strong negative penalty.
• Rule 4: Traffic source bias: if the user arrived via Search, add a small intent-based boost.
• Convert the final logit score into a click probability using a sigmoid function, then sample a Bernoulli trial to simulate whether a click occurs.

 1# product-recommender/recsys-engine/src/bandit_simulator.py
 2class GroundTruth:
 3    """
 4    The HIDDEN FORMULA (Ground Truth) for click simulation.
 5    Determines user click behavior based on context and item features.
 6    """
 7
 8    @staticmethod
 9    def calculate_probability(user_ctx: dict, item_ctx: dict) -> float:
10        """
11        Computes the probability that a user clicks an item.
12        Uses logistic regression-style scoring with domain-specific rules.
13        """
14        score = -2.5  # Base logit: starts with a low probability
15
16        # Rule 1: Morning Coffee
17        # Users are more likely to click coffee in the morning
18        if user_ctx.get("is_morning") == 1 and item_ctx.get("is_coffee") == 1:
19            score += 2.5
20
21        # Rule 2: Weekend Comfort Food
22        # Users tend to choose Pizza or Burgers on weekends
23        if user_ctx.get("is_weekend") == 1:
24            if item_ctx.get("cat_Pizzas") == 1 or item_ctx.get("cat_Burgers & Sandwiches") == 1:
25                score += 1.8
26
27        # Rule 3: Budget Constraint
28        # Young users (<25 years) avoid expensive items (normalized price > 0.8)
29        user_age = user_ctx.get("age", 0.5)  # normalized age 0-1
30        item_price = item_ctx.get("price", 0.5)  # normalized price 0-1
31        if user_age < 0.25 and item_price > 0.8:
32            score -= 3.0
33
34        # Rule 4: Traffic Bias
35        # Users arriving via Search have a slightly higher propensity to click
36        if user_ctx.get("traffic_source_Search") == 1:
37            score += 0.5
38
39        # Convert logit score to probability using sigmoid function
40        return 1 / (1 + np.exp(-score))
41
42    def will_click(self, user_ctx: dict, item_ctx: dict, fake: Faker) -> int:
43        """
44        Simulates a Bernoulli trial (click = 1, no click = 0) based on probability.
45        """
46        prob = self.calculate_probability(user_ctx, item_ctx)
47        return 1 if fake.random.random() < prob else 0

Data Preparation

We generate 10,000 historical events to serve as our “Offline Training” dataset. This process involves picking a random user and a random product, then asking the Oracle “Did they click?”.

Because the user and product are matched randomly (not by a recommender), the Average Click Rate (CTR) is naturally low. In this example, it is around 13.65%, and this serves as our baseline.

💡 There are three main scripts for this post: prepare_data.py for feature engineering and bandit history simulation, evalue.py for offline policy evaluation, and local_recommender.py for running product recommendation locally. Each script accepts a --seed argument, which defaults to 1237. As long as the seed remains the same, running the scripts will produce identical outputs.

 1(venv) $ python product-recommender/recsys-engine/prepare_data.py
 2[2026-01-26 19:16:09] INFO    : Generating 1000 synthetic users...
 3[2026-01-26 19:16:09] INFO    : Saved raw users to: .../users.csv
 4[2026-01-26 19:16:09] INFO    : Starting Feature Engineering...
 5[2026-01-26 19:16:09] INFO    : Saved User Features: (1000, 11)
 6[2026-01-26 19:16:10] INFO    : Saved Product Features: (200, 21)
 7[2026-01-26 19:16:10] INFO    : Saved Pipeline Artifacts to: .../preprocessing_artifacts.pkl
 8[2026-01-26 19:16:10] INFO    : Loaded 1000 users and 200 products.
 9[2026-01-26 19:16:10] INFO    : Generating 10000 events...
10[2026-01-26 19:16:10] INFO    : Done. Saved Training Log to .../training_log.csv
11[2026-01-26 19:16:10] INFO    : Avg Click Rate: 13.65%
12[2026-01-26 19:16:10] INFO    : Data Preparation Complete.

The main dataset (training_log.csv) combines user features, dynamic context (e.g., is_morning), product ID, and the interaction result (response):

Offline Policy Evaluation

We benchmarked several policies using Mab2Rec on the 10,000 historical events.

The Candidates

• Random: The baseline. Recommends items blindly.
• Popularity: Recommends items with the highest global click rate.
  • Result: Mediocre (AUC ~0.59). While better than random, it still fails to capture specific rules, such as “Morning Coffee” vs. “Weekend Pizza.”
• LinGreedy: Disjoint Linear Regression with $\epsilon$-greedy exploration.
• LinUCB (The Winner): Disjoint Linear Regression with Upper Confidence Bound.
• LinTS (Thompson Sampling): Bayesian regression that samples from a probability distribution.

Winner: LinUCB

While LinGreedy achieved the highest theoretical ranking accuracy (AUC ~0.88), it suffered from a low click rate (CTR ~11%) because it exploited “safe” choices too early.

LinUCB is the practical winner. It achieved a comparable ranking accuracy (AUC ~0.86) but nearly double the engagement (CTR ~20.5%) to LinGreedy.

This algorithm excels because it balances two competing goals:

1. Exploitation: It uses the predicted probability of a click ($x^T \theta$) to find good items.
2. Exploration: It adds a confidence interval ($\alpha \sqrt{x^T A^{-1} x}$) to the score. If the model is uncertain about a specific context (e.g., “I haven’t seen a user drink Coffee at 8 PM before”), the interval widens, boosting the score and forcing the model to test that hypothesis.

This allows LinUCB to discover high-value opportunities that the conservative LinGreedy model misses.

 1(venv) $ python product-recommender/recsys-engine/evaluate.py
 2Running Benchmark... (This trains and scores all models automatically)
 3--------------------------------------------------------------------------------
 4Available Metrics: ['AUC(score)@5', 'CTR(score)@5', 'Precision@5', 'Recall@5']
 5            AUC(score)@5  CTR(score)@5  Precision@5  Recall@5
 6Random          0.550000      0.102041     0.003876  0.019380
 7Popularity      0.592857      0.192308     0.007752  0.038760
 8LinGreedy       0.885185      0.117647     0.004651  0.023256
 9LinUCB          0.860317      0.204545     0.006977  0.034884
10LinTS           0.640798      0.211538     0.008527  0.042636
11ClustersTS      0.550505      0.153846     0.004651  0.023256
12--------------------------------------------------------------------------------

Why LinUCB Outperforms the Baseline in CTR

This is the core concept of Offline Policy Evaluation.

The benchmark does not test on every single row of your history. It uses a technique called Rejection Sampling (or simply “Matching”).

Here is exactly how mab2rec calculates that 20.5%:

1. The Log (History): Contains a mix of “Good Decisions” and “Bad Decisions” because it was generated randomly.

   • Row A: Morning User $\to$ Show Pizza $\to$ No Click (Bad Random Choice)
   • Row B: Morning User $\to$ Show Coffee $\to$ Click (Lucky Random Choice)

2. The Test (LinUCB): The model is smart. It knows Morning users want Coffee.

   • For Row A, LinUCB says: “I would recommend Coffee.”
     • Mismatch! The history shows Pizza. We cannot know what would have happened if we showed Coffee. This row is IGNORED.
   • For Row B, LinUCB says: “I would recommend Coffee.”
     • Match! The history shows Coffee. We know the result (Click). This row is COUNTED.

The dataset average (13.7%) includes all the “Bad Random Choices” (Row A). The LinUCB score (20.5%) filters out the bad choices. It effectively says: “On the rare occasions where the random history actually showed the right product (Row B), did the user click?” Since LinUCB focuses only on the “Right Products,” the click rate for those specific matches is much higher than the average of the random pile.

Simulation of the Selected Product Recommender Locally

With the model selected, we built a script to simulate the product recommender locally. This script acts as the Server, the User, and the Trainer simultaneously in a continuous loop.

Step 1: Pre-training (Offline Replay)

We don’t want to start with a “dumb” model. We load the 10,000 historical events (training_log.csv) and run model.fit(). This gives the bandit a baseline knowledge of the world before the live loop begins.

Step 2: The Online Loop

We simulate a sequence of user visits:

1. User Arrival: Pick a random user from the pool.
2. Contextualize: Inject a simulated timestamp (e.g., varying between Mon 08:00 AM and Sat 09:00 PM). This is the key “Context” the model must react to.
3. Recommend: LinUCB calculates scores for all 200 products and returns the Top 5.
4. Reaction: The GroundTruth Oracle decides if the user clicks.
5. Online Update: We call model.partial_fit(). This updates the matrices ($A$ and $b$) instantly. The very next recommendation will reflect this new learning.

Here is a sample of 30 recommendation records from the local simulation.

 1(venv) $ python product-recommender/recsys-engine/local_recommender.py 
 2[2026-02-05 15:47:43] INFO    : Loaded 1000 users
 3[2026-02-05 15:47:43] INFO    : Loading artifacts...
 4[2026-02-05 15:47:48] INFO    : Loaded 200 products.
 5[2026-02-05 15:47:48] INFO    : Pre-training model from history...
 6[2026-02-05 15:47:48] INFO    : Model pre-trained on 10000 events.
 7
 8--- STARTING LIVE LOOP (30 visits) ---
 9
10User 0153 (56 yo) @ Fri 00:58 -> Recs: [200, 124, 015, 058, 011] -> Clicked: 200 (❌)
11User 0909 (21 yo) @ Sat 15:53 -> Recs: [038, 040, 017, 020, 046] -> Clicked: 038 (❌)
12User 0406 (30 yo) @ Sat 05:24 -> Recs: [020, 041, 008, 055, 040] -> Clicked: 020 (✅)
13User 0317 (31 yo) @ Sat 05:38 -> Recs: [008, 055, 057, 059, 139] -> Clicked: 055 (✅)
14User 0246 (44 yo) @ Mon 02:04 -> Recs: [015, 058, 057, 011, 124] -> Clicked: 015 (❌)
15User 0974 (61 yo) @ Fri 01:16 -> Recs: [058, 073, 124, 074, 051] -> Clicked: 058 (❌)
16User 0234 (26 yo) @ Thu 12:16 -> Recs: [036, 103, 002, 186, 070] -> Clicked: 036 (❌)
17User 0360 (35 yo) @ Sat 20:23 -> Recs: [058, 051, 008, 042, 018] -> Clicked: 042 (✅)
18User 0513 (51 yo) @ Sun 05:37 -> Recs: [051, 059, 043, 014, 020] -> Clicked: 051 (✅)
19User 0640 (33 yo) @ Mon 00:49 -> Recs: [073, 124, 023, 147, 074] -> Clicked: 073 (❌)
20User 0363 (31 yo) @ Fri 23:35 -> Recs: [200, 126, 085, 058, 018] -> Clicked: 018 (✅)
21User 0718 (58 yo) @ Sat 23:05 -> Recs: [018, 036, 040, 042, 020] -> Clicked: 018 (✅)
22User 0390 (49 yo) @ Tue 00:56 -> Recs: [147, 165, 020, 089, 047] -> Clicked: 147 (❌)
23User 0425 (39 yo) @ Thu 03:59 -> Recs: [147, 165, 062, 028, 055] -> Clicked: 147 (❌)
24User 0792 (21 yo) @ Sun 23:28 -> Recs: [042, 056, 018, 043, 046] -> Clicked: 042 (✅)
25User 0190 (41 yo) @ Sat 10:54 -> Recs: [192, 139, 189, 008, 055] -> Clicked: 192 (✅)
26User 0544 (41 yo) @ Tue 20:42 -> Recs: [018, 058, 090, 043, 147] -> Clicked: 018 (❌)
27User 0192 (17 yo) @ Sat 18:38 -> Recs: [042, 056, 018, 046, 043] -> Clicked: 042 (✅)
28User 0757 (55 yo) @ Thu 16:08 -> Recs: [015, 171, 165, 037, 126] -> Clicked: 126 (✅)
29User 0904 (60 yo) @ Sat 02:07 -> Recs: [103, 041, 017, 042, 057] -> Clicked: 042 (✅)
30User 0552 (39 yo) @ Tue 11:33 -> Recs: [192, 190, 189, 194, 193] -> Clicked: 192 (✅)
31User 0540 (36 yo) @ Sat 08:56 -> Recs: [043, 041, 192, 014, 073] -> Clicked: 043 (✅)
32User 0326 (26 yo) @ Wed 13:04 -> Recs: [015, 171, 165, 023, 126] -> Clicked: 015 (❌)
33User 0834 (29 yo) @ Sat 22:58 -> Recs: [051, 002, 042, 058, 036] -> Clicked: 051 (✅)
34User 0290 (21 yo) @ Mon 19:09 -> Recs: [058, 200, 004, 018, 085] -> Clicked: 018 (✅)
35User 0275 (18 yo) @ Wed 11:10 -> Recs: [189, 002, 160, 078, 103] -> Clicked: 189 (❌)
36User 0327 (23 yo) @ Wed 19:54 -> Recs: [200, 126, 018, 085, 058] -> Clicked: 200 (❌)
37User 0144 (67 yo) @ Thu 12:31 -> Recs: [087, 126, 047, 103, 034] -> Clicked: 087 (❌)
38User 0497 (60 yo) @ Sun 08:26 -> Recs: [192, 139, 008, 189, 059] -> Clicked: 192 (✅)
39User 0508 (64 yo) @ Tue 12:41 -> Recs: [165, 087, 026, 171, 037] -> Clicked: 165 (❌)
40
41--- END LOOP ---

Evaluation of Simulation

The system is behaving exactly as a Contextual Bandit should. It is aggressively exploiting known high-probability zones while struggling (realistically) in neutral zones.

The “Weekend Pizza” Strategy is Dominant

The model has learned that Weekends (Sat/Sun) are for Pizzas (Category 40s).

• User 0360 (Sat 20:23): Recommended [..., 042 (Aussie Pizza), ...] $\to$ Clicked ✅.
• User 0513 (Sun 05:37): Recommended [051 (Buffalo Pizza), 059 (Lamb Pizza)...] $\to$ Clicked ✅.
• User 0834 (Sat 22:58): Recommended [051 (Buffalo Pizza)... 042 (Aussie Pizza)] $\to$ Clicked ✅.
• Insight: The model pushes Pizzas hard on weekends regardless of the specific hour, resulting in a very high conversion rate for these users.

The “Morning Coffee” Precision

The model correctly switches strategies based on the hour, even distinguishing “Weekend Morning” from “Weekend Night”.

• User 0552 (Tue 11:33): It is a Weekday Morning. The model recommended 5 Coffees [192, 190, 189, 194, 193]. The user clicked 192 (Long Black). ✅
• User 0497 (Sun 08:26): It is a Weekend, but it is Morning. The model prioritized Coffee (192, 189) over Pizza. The user clicked 192. ✅
• User 0508 (Tue 12:41): This is a great edge case. It is 41 minutes past the “Morning” cutoff (12:00). The model stopped recommending Coffee and switched to Lunch items (Steamed Veggies, Burritos). The user did not click ❌, but the behavior change proves the features are working perfectly.

Perfect Recommendation, No Interaction (Realism)

• User 0275 (Wed 11:10): The model recommended 189 (Flat White). However, the user ignored it, and the result was ❌ (No Reward).
• Why? This mimics real life. Even if the recommendation is perfect, users don’t always convert. In the GroundTruth, the probability caps at ~50% (sigmoid of 0). This “Bad Luck” outcome confirms your evaluation pipeline is honest.

Conclusion

Across 30 simulated visits, CTR reached 53%.

This elevated CTR indicates that the model has internalized dominant temporal and demographic patterns encoded in the simulation. It aggressively exploits strong signals such as weekend pizza and morning coffee while still exploring less-certain regions.

Importantly, the simulation includes realistic noise. Even strong recommendations do not always convert, reflecting probabilistic user behavior.

What’s Next?

We have successfully prototyped a Contextual Bandit that learns time-based preferences. However, this Python script has major limitations for a production environment:

1. Scalability: Disjoint LinUCB maintains a matrix for every product. With 10 million products, a single server will run out of memory.
2. Latency: The training (partial_fit) blocks the inference (recommend). In a real system, you cannot make a user wait for the model to update.
3. Fault Tolerance: If the script crashes, the learned state is lost.
4. Concurrency: A single Python process cannot handle thousands of concurrent requests.

In Part 2, we will transform this prototype into an Event-Driven Architecture:

• Kafka will transport click events asynchronously.
• Flink will handle distributed, stateful model training.
• Redis will serve the model matrices for sub-millisecond inference.

Stay tuned!

In porting the examples to Kotlin, I also took the opportunity to align the code with modern Flink practices. The complete source for this post is available in the stream-processing-with-flink directory of the flink-demos GitHub repository.

Updating the Code and APIs

The book, while conceptually valuable, is a bit dated. As I worked through the examples, I updated several deprecated features to use their modern equivalents.

• ListCheckpointed to CheckpointedFunction: Replaced the older checkpointing interface with the more flexible CheckpointedFunction.
• SourceFunction to Source API: Migrated from the legacy SourceFunction to the newer, more robust Source API.
• SinkFunction to Sink V2 API: Updated the SinkFunction to the Sink V2 API.
• Queryable State: Ignored as it has been deprecated since Flink 1.18. These examples are built using Flink 1.20.1.

Optimizing the Build with Gradle

Figuring out the Gradle build was a valuable lesson in itself. I learned how to create a single build.gradle.kts to handle two different scenarios: producing a lean production JAR and keeping local execution simple.

For the production JAR, Flink dependencies are declared with compileOnly. This correctly excludes them from the final artifact, as the Flink cluster provides these libraries at runtime.

❗The test environment also needs the Flink APIs to compile and run. This is handled by testImplementation. This standard Gradle configuration provides the Flink libraries only to the test classpath, keeping them completely separate from the production JAR and the local run task.

1dependencies {
2    // Flink Dependencies are not bundled into the JAR
3    compileOnly("org.apache.flink:flink-streaming-java:$flinkVersion")
4    compileOnly("org.apache.flink:flink-clients:$flinkVersion")
5
6    // Flink is available for test compilation and execution
7    testImplementation("org.apache.flink:flink-streaming-java:$flinkVersion")
8    testImplementation("org.apache.flink:flink-clients:$flinkVersion")
9}

However, this creates a problem for local development, as the Flink libraries are now missing from the default runtime classpath. The key technique I learned was how to solve this by creating a custom configuration, localRunClasspath.

This configuration rebuilds the full classpath specifically for the run task by adding the compileOnly dependencies back in, along with the standard implementation and runtimeOnly scopes. This makes local development seamless.

1val localRunClasspath by configurations.creating {
2    extendsFrom(configurations.implementation.get(), configurations.compileOnly.get(), configurations.runtimeOnly.get())
3}
4
5// ...
6
7tasks.named<JavaExec>("run") {
8    classpath = localRunClasspath + sourceSets.main.get().output
9}

Main Chapters

So far, I have translated the examples from the following main chapters:

• Chapter 5: Basic and Keyed Transformations: Covers fundamental data manipulation, including map, filter, keyBy, and rolling sum aggregations, as well as multi-stream transformations.
• Chapter 6: Event Time and Windowing: Focuses on time-based operations, including ProcessFunction timers, watermark generation strategies, window functions, custom window logic, and side outputs for late data handling.
• Chapter 7: State Management: Explores different types of state in Flink, such as ValueState, ListState, MapState, and BroadcastState, along with operator state.
• Chapter 8: Asynchronous I/O and Custom Connectors: Demonstrates how to interact with external systems asynchronously and build custom sources and sinks.

How to Build and Run the Examples

The Flink applications can be run directly from the command line for local testing and development. This is useful for quick debugging without needing a full Flink cluster. Moreover, each Flink app has detailed documentation so it is easy to understand, for example:

 1/**
 2 * This Flink job demonstrates transformations on a `KeyedStream`.
 3 *
 4 * It showcases the `reduce` operator, a powerful tool for maintaining running aggregates
 5 * for each key in a stream.
 6 *
 7 * The pipeline is as follows:
 8 * 1. **Source**: Ingests a stream of `SensorReading` events.
 9 * 2. **KeyBy**: Partitions the stream by the `id` of each sensor. All subsequent
10 *    operations will run independently for each sensor.
11 * 3. **Reduce**: For each key, this operator maintains a running state of the `SensorReading`
12 *    with the maximum temperature seen so far. For every new reading that arrives, it
13 *    compares it to the current maximum and emits the new maximum downstream.
14 * 4. **Sink**: Prints the continuous stream of running maximums for each sensor to the console.
15 */
16object KeyedTransformations {
17    // ...
18}

To launch the apps, use the run task and set the desired main class with the -PmainClass project property. Here are the full examples:

1. Run the Chapter 5 examples.

1./gradlew run -PmainClass=me.jaehyeon.chapter5.BasicTransformations
2./gradlew run -PmainClass=me.jaehyeon.chapter5.KeyedTransformations
3./gradlew run -PmainClass=me.jaehyeon.chapter5.RollingSum
4./gradlew run -PmainClass=me.jaehyeon.chapter5.MultiStreamTransformations

2. Run the Chapter 6 examples.

1./gradlew run -PmainClass=me.jaehyeon.chapter6.ProcessFunctionTimers
2./gradlew run -PmainClass=me.jaehyeon.chapter6.PeriodicWatermarkGeneration
3./gradlew run -PmainClass=me.jaehyeon.chapter6.MarkerBasedWatermarkGeneration
4./gradlew run -PmainClass=me.jaehyeon.chapter6.CoProcessFunctionTimers
5./gradlew run -PmainClass=me.jaehyeon.chapter6.WindowFunctions --args="min1" # min2, avg, minmax1, or minmax2
6./gradlew run -PmainClass=me.jaehyeon.chapter6.CustomWindows
7./gradlew run -PmainClass=me.jaehyeon.chapter6.SideOutputs
8./gradlew run -PmainClass=me.jaehyeon.chapter6.LateDataHandling --args="filter" # sideout or update

3. Run the Chapter 7 examples.

1./gradlew run -PmainClass=me.jaehyeon.chapter7.KeyedStateFunction
2./gradlew run -PmainClass=me.jaehyeon.chapter7.StatefulProcessFunction
3./gradlew run -PmainClass=me.jaehyeon.chapter7.BroadcastStateFunction
4./gradlew run -PmainClass=me.jaehyeon.chapter7.OperatorStateFunction
5./gradlew run -PmainClass=me.jaehyeon.chapter7.KeyedAndOperatorStateFunction

4. Run the Chapter 8 examples.

1./gradlew run -PmainClass=me.jaehyeon.chapter8.AsyncFunction
2./gradlew run -PmainClass=me.jaehyeon.chapter8.CustomConnectors

Conclusion

Working through the examples in Kotlin has been an effective way to dive deeper into Apache Flink. Translating the examples to Kotlin not only forced me to understand the concepts more thoroughly but also provided a great opportunity to get hands-on with the latest APIs and build practices. For those looking to learn Apache Flink through up-to-date examples, I hope sharing my experience and code proves helpful. It’s been a fun and effective learning journey.

This article explores an alternative approach: an integrated architecture where the backend API and the frontend UI are served from a single, cohesive Python application.

We will provide a technical analysis of this architecture’s implementation by integrating FastAPI and NiceGUI, referencing the complete project template available at jaehyeon-kim/nicegui-fastapi-template.

💡 Version 2.0: Unified Application Architecture

The initial version was designed with a distinct separation between a FastAPI backend and a NiceGUI frontend, which communicated over HTTP. This new version consolidates the application, leveraging the fact that NiceGUI is built on top of FastAPI. The result is a more tightly integrated structure that allows the UI and API logic to coexist in the same process.

Key Architectural Changes

• Single FastAPI Instance: The separate FastAPI server process has been removed. The application now operates on the single FastAPI instance provided by nicegui.app.
• Direct Function Calls: UI event handlers no longer make HTTP requests (httpx) to the backend. They now import and call the necessary Python functions from the repository layer directly, removing the network layer for UI-to-backend communication.
• Preserved API Endpoints: The original API, intended for external clients, is maintained. It is mounted using FastAPI’s APIRouter onto the main NiceGUI application, ensuring that JSON endpoints remain available.
• Consolidated Codebase: The frontend and backend directories have been merged into a single application package (e.g., app or src). A run.py script at the project root now serves as the single entry point.
• Shared Logic: Business logic, such as permission checks and database operations, has been centralized in the repository layer, where it is called by both the UI event handlers and the API endpoints.

This updated architecture provides a more direct and cohesive way to build full-stack applications where the UI and backend logic are tightly coupled.

Project Structure

The application is structured with a single entry point, app.py, at the project root, and a main source package, src/, which contains all the application’s logic. This design provides a clear separation between the runnable script and the installable source code.

• app.py: This script is the single entry point for the application. It is responsible for creating the main NiceGUI app instance, including all the API routers from src/backend/, importing the UI pages from src/frontend/ to register their routes, and starting the web server.

• src/: This directory is the main Python package for the application. It contains all the core logic, API definitions, and UI code, organized into the following modules:

  • backend/: Contains the code for the data-only API, intended for external clients.
    • endpoints/: Each file defines a set of related API routes (e.g., for items, users, login) using FastAPI’s APIRouter.
    • deps.py: Manages FastAPI’s dependency injection system for the API, such as providing database sessions or the current authenticated user.
  • core/: Holds application-wide configuration (config.py) and security-related functions like password hashing and token creation (security.py).
  • db/: Manages all database interactions, including engine creation, session management (session.py), and initial database setup (init_db.py).
  • frontend/: Contains all the NiceGUI code for the user interface.
    • components/: Holds reusable UI elements and utilities, such as header.py, footer.py, notifications.py, and authentication helpers in auth_utils.py.
    • layouts/: Defines the overall page structure, like the main dashboard frame, ensuring a consistent look and feel.
    • pages/: Each file represents a specific UI view, such as the login screen (login.py) or the item management page (items.py), using the @ui.page decorator.
    • state.py: A module for managing UI-specific state, like the user’s authentication token.
  • models/: Contains the SQLModel (and Pydantic) schemas that define database tables and data structures used across the entire application.
  • repositories/: This is the core business logic and data access layer. It abstracts all database queries and contains functions for data manipulation. In this unified architecture, its functions are now called directly by both the API endpoints in src/backend/ and the UI event handlers in src/frontend/.

FastAPI Backend

FastAPI provides a robust foundation for the backend due to several key technical features:

• ASGI Foundation: Built on the Asynchronous Server Gateway Interface (ASGI), FastAPI natively supports asynchronous operations. This allows it to handle high-concurrency, I/O-bound tasks, such as network requests and database queries, efficiently without blocking the server.
• Schema Generation and Data Validation: FastAPI uses Pydantic models for strict, type-hint-based data validation and serialization. These models automatically generate OpenAPI schemas, which power the interactive API documentation (via Swagger UI and ReDoc) and ensure that the API contract is clearly defined and enforced.
• Dependency Injection System: Its dependency injection system is a core feature that enhances modularity and testability. It allows for the management of dependencies like database sessions and authentication credentials, ensuring that resources are correctly provisioned and cleaned up for each request.

NiceGUI Frontend

NiceGUI serves as the frontend component, allowing for UI development entirely within Python.

• Pythonic Abstraction of Web Technologies: NiceGUI functions as an abstraction layer that generates the necessary HTML, CSS, and JavaScript from Python objects and methods. This allows developers to define complex user interfaces without writing client-side code directly.
• Server-Side Event Handling: The framework employs an event-driven model where UI components are bound to Python callback functions. User interactions (e.g., button clicks, form submissions) trigger these functions, which execute on the server. This creates a direct and clear link between a UI event and its corresponding backend logic.
• Server-Maintained State: Unlike JavaScript frontend frameworks that manage state on the client, NiceGUI maintains the UI state within the server’s Python process. This simplifies application logic, as there is no need for complex state synchronization mechanisms between client and server.

Architectural Advantages and Framework Comparisons

The Integrated Server Approach

In this architecture, the NiceGUI application is mounted directly onto the FastAPI instance. This is typically done with a single function call, creating a unified application that serves both API endpoints and the user interface from one process.

The primary benefit is the ability to use shared data models (e.g., SQLModel or Pydantic) across the entire stack. A model defined once can be used to structure a database table, validate an API request payload, and define the data contract for the UI. This ensures end-to-end data consistency and reduces code duplication, as all parts of the application are built around the same data structures.

NiceGUI vs. JavaScript Frameworks (e.g., React, Vue)

• Bridging the Expertise Gap: Acquiring deep expertise in both backend Python and a modern JavaScript frontend framework is a significant undertaking. NiceGUI directly addresses this by enabling Python developers to build sophisticated user interfaces without leaving the Python ecosystem. This allows them to leverage their existing skills rather than learning a new language and its complex toolchain.
• Leveraging Mature Frontend Technologies: NiceGUI is not a proprietary UI system built from scratch. It is built on top of the robust and widely-used Vue and Quasar frameworks. This provides the best of both worlds: developers interact with a simple Pythonic API while benefiting from the power, performance, and rich component library of a mature frontend technology.
• Trade-offs: The server-side rendering approach is highly efficient for internal tools and data-heavy applications. However, because every interaction requires a round-trip to the server, it may introduce latency on highly interactive UIs compared to a client-side Single Page Application (SPA), which can handle many state changes without network requests.

NiceGUI vs. Streamlit

• Control and Layout: NiceGUI provides more granular control over UI component placement and application layout, using rows, columns, and grids. This makes it well-suited for building applications with a structured, traditional design. Streamlit is more opinionated, favoring a simple, top-to-bottom script execution model that is excellent for linear data narratives but offers less layout flexibility.
• Event Handling and Execution Model: Both frameworks use callbacks, but their underlying execution models differ significantly. NiceGUI uses a persistent component model where an event, such as a button click, executes a specific callback function (on_click=handle_click). This function is the only code that runs and is responsible for explicitly updating any UI elements. This aligns closely with traditional GUI programming paradigms. In contrast, Streamlit uses a script re-run model. While it also has an on_click callback, this function typically modifies a session state object. After the callback completes, Streamlit re-runs the entire application script from top to bottom. The UI is then re-rendered based on the new values in the session state. This model simplifies the creation of linear, data-centric apps but can be less direct and potentially less performant for managing complex, multi-state interfaces compared to NiceGUI’s explicit event-driven approach.
• Integration: NiceGUI is designed to be a component that can be integrated with standard web frameworks like FastAPI. Streamlit is generally used as a self-contained application server and is less straightforward to embed within another ASGI application.

Implementation Overview

To provide a concrete example of this architecture, the reference repository at jaehyeon-kim/nicegui-fastapi-template contains a fully functional application. Let’s examine its structure and key components.

Project Structure

The project is organized into two main directories, ensuring a clear separation of concerns between the backend logic and the user interface:

• backend/: This directory contains all the FastAPI source code. It is further subdivided into modules for handling specific responsibilities:
  • api/: Defines the API endpoints for resources like users and items.
  • db/: Manages the database session and engine configuration.
  • models/: Contains the SQLModel (and Pydantic) data schemas that define our database tables and API data structures.
  • repositories/: Implements the data access layer, abstracting the database queries from the API endpoints.
• frontend/: This directory holds all the NiceGUI code for the user interface, organized by function into the following subdirectories and modules:
  • components/: Contains reusable UI elements and helper functions. This includes visual components like header.py and footer.py, as well as utility modules like notifications.py for displaying messages to the user and form_helpers.py for handling common form logic.
  • layouts/: Defines the overall structure of the application’s pages. The default.py file, for instance, assembles the header, drawer, and footer to ensure a consistent look and feel across different views.
  • pages/: Each file in this directory represents a specific page or view within the application, such as the login screen (login.py), the item management page (items.py), and the user creation form (create_user.py).
  • state.py: A dedicated module for managing the application’s client-side state, such as user authentication status. This allows different parts of the UI to react consistently to changes in state.
• backend/main.py: This file serves as the central integration point. It initializes the FastAPI application, includes the API routers from the backend/api/ directory, and finally, mounts the entire NiceGUI frontend. This is where the two parts of the application become one.

Application Demo

The repository provides a complete user and item management application that showcases role-based access control. The functionality is divided between two user roles:

• Standard User: After logging in, a standard user has full CRUD (Create, Read, Update, Delete) permissions over their own items. They can add new items, view their list, and edit or delete them as needed.

• Superuser: A superuser has elevated privileges. In addition to managing their own items, they can also create new user accounts. Crucially, they have a global view of the system and can manage the items belonging to any user, making this role suitable for administrative purposes.

The demo showcases these distinct workflows, illustrating how the NiceGUI frontend dynamically adapts to the user’s permissions, which are enforced by the FastAPI backend.

Automatic API Documentation

One of the most powerful features of FastAPI is its ability to automatically generate interactive API documentation from the Pydantic models and endpoint definitions. The application provides two documentation interfaces out-of-the-box:

• Swagger UI (/docs): A feature-rich, interactive interface that allows developers to not only view the API endpoints but also test them directly from the browser by sending live requests.

• ReDoc (/redoc): A clean, read-only documentation page that presents the API in a more traditional, hierarchical format. It is excellent for quickly referencing endpoints and their schemas.

These auto-generated documents are invaluable for development, testing, and collaboration, and they are created without any extra effort, thanks to FastAPI’s adherence to the OpenAPI standard.

Summary and Use Cases

The integration of FastAPI and NiceGUI provides a robust architecture for building web applications entirely in Python. It streamlines development by creating a unified environment, simplifies deployment to a single process, and ensures strong data consistency through the use of shared models.

This architecture is exceptionally well-suited for:

• Internal Tools and Administrative Dashboards: Where rapid development and ease of maintenance are critical.
• Rapid Prototyping and MVPs: To quickly build and validate a functional application.
• Machine Learning and Data Science Demos: To create interactive interfaces for models without requiring frontend expertise.

While it may not be the optimal choice for every project, particularly those requiring complex client-side interactivity, it offers a powerful and efficient alternative for a significant class of web applications. For a practical implementation, refer to the project code at jaehyeon-kim/nicegui-fastapi-template.

This is where a gateway-centric architecture comes in. By placing a specialized gateway at the heart of the platform, we can transform this chaos into a stable, governed, and self-service system. The Apache Kyuubi project provides a perfect blueprint for this approach, defining its role as:

Apache Kyuubi is a distributed, multi-tenant gateway providing unified access to big data engines like Spark and Flink via JDBC/ODBC for interactive queries and REST APIs for programmatic submissions.

This definition is more than just a product description; it’s a strategic vision. By embracing this model, we can finally deliver on the promise of self-service data access without sacrificing stability or governance. Let’s explore exactly how each layer of the platform is designed to support this powerful concept.

Architecture: A Layer-by-Layer Breakdown

1. Client Layer: Simplicity Through Standardization

How do users connect? Via interactive SQL clients or programmatic APIs.

The primary goal of a self-service platform is to meet users where they are. The gateway makes this possible by providing multiple, standardized entry points, ensuring that both interactive users and automated systems can leverage the platform’s power. Kyuubi provides two primary connection methods: the Thrift API for interactive SQL and the REST API for programmatic batch workloads.

Thrift API: Gateway for Interactive Analytics

This is the most established and widely used connection method, ideal for analysts and data scientists using standard BI and SQL tools.

• Primary Use Case: Interactive, ad-hoc queries and data exploration.
• How it Works: Kyuubi implements the HiveServer2 Thrift protocol, which is the standard for SQL-on-Big-Data. This allows any tool that can communicate with Hive or Spark SQL to connect seamlessly to Kyuubi using a standard JDBC/ODBC driver.
• Tools: Tableau, PowerBI, DBeaver, Jupyter Notebooks (with PyHive), and other SQL-native clients.
• Gateway’s Role: From the user’s perspective, they are connecting to a traditional database. They don’t need engine-specific drivers or complex connection strings. The immense complexity of the underlying Spark or Flink engines is completely abstracted away behind one familiar JDBC/ODBC endpoint. The Thrift interface uses long-lived connections, making it well-suited for the back-and-forth nature of an interactive session.

💡 Note: While tools typically use JDBC or ODBC drivers, these drivers communicate over the Thrift protocol—hence the term “Thrift API” is technically accurate.

REST API: Interface for Programmatic Workflows

This interface is designed for automation, batch job submission, and integration into larger data pipelines.

• Primary Use Case: Submitting and managing batch jobs (e.g., SQL, Scala, Python, or JAR files) programmatically. This is perfect for ETL/ELT workflows, scheduled reports, and machine learning model training.
• How it Works: The REST API operates over standard, short-lived HTTP connections. Clients can submit a job definition (including the code and execution parameters) to a REST endpoint. Kyuubi then manages the entire lifecycle of the batch job on the user’s behalf. This stateless approach is ideal for automation and works well with load balancers in a high-availability setup.
• Tools:
  • Programmatic Clients: Custom applications, workflow orchestrators like Apache Airflow, or scripts using tools like curl.
  • Command-Line Tool: Kyuubi provides kyuubi-ctl, a command-line interface that uses the REST API for managing batch jobs via YAML files.

💡 Tip: Because of its stateless nature, the REST API is particularly well-suited for distributed environments and works seamlessly with CI/CD systems and job schedulers.

2. Gateway Layer: Multi-Tenant Control Plane

This is the heart of the architecture, where raw compute power is transformed into a stable, governed service.

This layer is where Apache Kyuubi lives. It is not merely a pass-through proxy; it is an intelligent, distributed control plane responsible for security, stability, and resource management. It acts as the indispensable intermediary between a diverse set of users and a powerful but complex set of backend engines.

High Availability and Fault Tolerance

A gateway cannot be a single point of failure. Kyuubi is designed for resilience and can be deployed in a high-availability (HA) configuration. You run multiple Kyuubi server instances, stateless by nature, behind a load balancer. Session information and operational metadata are stored in a shared, external state store (like ZooKeeper or Etcd). If one Kyuubi server goes down, the load balancer redirects clients to a healthy instance, which can then recover the user’s session state from the shared store, providing seamless failover for end-users.

Centralized Authentication Hub

Before any query is processed, the user must be authenticated. The gateway serves as the single, secure entry point for the entire platform. It integrates directly with standard enterprise authentication protocols like LDAP and Kerberos. This centralizes all authentication logic. Instead of exposing multiple engines and configuring security on each one, you secure one endpoint: the gateway. This simplifies client configuration and dramatically reduces the security attack surface.

True Multi-Tenancy through Dynamic Resource Isolation

This is Kyuubi’s most critical feature. It enforces multi-tenancy not just logically, but physically.

• Engine-per-User Isolation: When a user connects and runs a query, Kyuubi does not send it to a shared, monolithic cluster. Instead, it dynamically provisions a dedicated compute engine (e.g., a Spark application on Kubernetes) scoped specifically to that user or their session. This is the key to true multi-tenancy: one user’s poorly written query consuming 100% of its engine’s CPU has zero impact on the performance or stability of another user’s engine.

• Configurable Sharing Levels: Kyuubi also offers flexible engine-sharing modes to balance isolation and performance. While the default is to isolate by user (USER level), you can configure it to share an engine at the CONNECTION level or even create server-scoped engines for specific use cases, giving architects fine-grained control over resource trade-offs.

Unified Interface and Intelligent Routing

The gateway abstracts away the diversity of the backend. A user connects to a single JDBC/ODBC endpoint and speaks standard SQL. They don’t need to know which engine is best for their job.

Based on session properties set by the user or administrator (e.g., kyuubi.engine.type=FLINK), the gateway intelligently routes the request. It accepts the standard SQL, identifies the target engine type, manages the entire lifecycle of that engine (provisioning, query submission, termination), and streams the results back. This turns a complex ecosystem of Spark, Flink, and Trino into a single, unified, and easy-to-use SQL service for the entire organization.

3. Compute Layer: Powerhouse Engines

What does the gateway connect to? Isolated, on-demand Spark, Flink, and Trino engines.

This is where the actual data processing occurs. The compute layer consists of powerful, specialized engines, but they are no longer directly exposed to users. Instead, they are treated as a backend resource, managed entirely by the gateway based on the workload.

• Engines:
  • Apache Spark: The workhorse for large-scale batch SQL processing and ETL.
  • Apache Flink: The engine for real-time, continuous SQL queries on streaming data.
  • Trino: The engine for high-performance, interactive federated queries across disparate data sources.

Gateway’s Role: A Lifecycle Manager for Compute

Kyuubi’s most important function is to act as an intelligent and dynamic lifecycle manager for these powerful engines. It doesn’t just proxy queries; it completely abstracts the complexity of resource management away from the end-user. Here’s how:

• On-Demand, User-Scoped Provisioning: When a user connects and runs their first query, Kyuubi intercepts the request. It authenticates the user and, based on predefined rules, submits a request to a resource manager (like Kubernetes or YARN) to launch a brand new engine instance specifically for that user or session. This is the core of multi-tenancy: the engine is created on behalf of the user, runs with their permissions, and is completely isolated from other users’ engines. This eliminates the “noisy neighbor” problem, where one user’s heavy query can destabilize the entire platform.

• Intelligent Caching and Sharing: While isolating engines is key, constantly launching new ones can introduce latency. Kyuubi manages this intelligently. It can be configured to keep a warm pool of engines ready or to share an engine across multiple queries from the same user within a single session. This provides the performance of a long-running session without sacrificing the isolation between different users.

• Automatic Termination and Cost Control: Kyuubi is also responsible for cleanup. It constantly monitors the engines it has launched. If an engine sits idle for a configured period (e.g., 30 minutes), Kyuubi will automatically terminate it and release its resources back to the cluster. This is absolutely critical for cost-efficiency in a cloud environment, ensuring you only pay for compute resources when they are actively being used.

• Seamless Abstraction: From the user’s perspective, this entire lifecycle is invisible. They write and execute standard SQL. They don’t need to know how to write a spark-submit command, craft a Kubernetes pod YAML file, or worry about memory allocation. Kyuubi handles the translation from a simple SQL query to a complex, resource-managed job on a distributed engine, ensuring the right tool is used for the right job without exposing any of the underlying complexity.

4. Governance Layer: Centralized Control at the Gateway

How do we enforce rules? The gateway is the natural “chokepoint” for governance.

In a self-service world, governance must be automated. The gateway architecture makes this feasible because every single query and every user session must pass through it. This creates a natural control point for applying rules, though the specific capabilities can vary by the backend engine.

Authorization: An Engine-Specific Approach

The gateway acts as the central point for authenticating users and integrating with an authorization engine like Apache Ranger. However, the actual enforcement of security policies is delegated to the backend compute engines. This approach leverages the native security features of each engine, meaning the depth of integration differs across the platform, which is a critical architectural consideration.

• For Apache Spark: The integration is the most mature and powerful. Kyuubi provides a specialized Kyuubi Spark Authz Plugin that deeply integrates with Apache Ranger. This plugin allows Ranger to enforce fine-grained policies—including row-level filtering and column-level masking—directly within the Spark engine for all Spark SQL queries. The gateway authenticates the user, launches a Spark engine on their behalf, and the engine then uses the Authz plugin to check with Ranger and apply policies before execution. This provides robust, centralized security for all Spark-based workloads.
• For Apache Flink: The story is more about isolation and coarse-grained control. Kyuubi provides essential security for Flink through robust authentication and session isolation. This guarantees that only authenticated users can submit jobs and that their Flink sessions are isolated from others. However, fine-grained, policy-based authorization for Flink jobs is not handled by a Kyuubi plugin. Instead, security must be enforced at the data source level. A common pattern is to use Flink’s HiveCatalog to read data from Hive tables, where Apache Ranger’s existing Hive-level policies can be applied for access control.
• For Trino: Authorization is handled by delegating to Trino’s own powerful security model. Trino has a native Apache Ranger plugin that provides comprehensive, fine-grained access control. This plugin supports policies for catalogs, schemas, tables, and columns, as well as row-level filtering and column masking. In this architecture, Kyuubi’s role is to authenticate the user and then pass the user’s identity securely to the Trino engine. Trino then uses its own Ranger plugin to enforce the centralized policies. This allows organizations to manage Spark and Trino permissions within the same Ranger UI, while the enforcement happens natively within each respective engine.

Lineage: A Universal, Automated View of Your Data’s Journey

Unlike authorization, which can be engine-specific, data lineage is applied universally and automatically across the platform. This creates a complete, trustworthy audit trail for every query, which is essential for data governance, impact analysis, and debugging complex data flows.

• How It Works: Automated Agent Injection The gateway-centric design is the key to automating lineage collection. Kyuubi is configured to automatically inject a lightweight OpenLineage agent into the runtime of every single Spark and Flink engine it provisions. Users do not need to add any special libraries or modify their code; lineage capture is a guaranteed part of the platform’s execution process. For Trino, lineage is captured via a separate, native OpenLineage plugin configured directly within the Trino coordinator.
• What is Captured? Rich, Actionable Metadata As jobs run, the OpenLineage agent observes the execution plan and collects detailed metadata. This isn’t just a high-level overview. The agent captures:
  • Job Information: The name of the job, its start and end times, and whether it succeeded or failed.
  • Dataset Information (Inputs and Outputs): The physical sources (e.g., S3 paths, Kafka topics, database tables) and destinations of the data.
  • Column-Level Lineage (for Spark and Trino): For supported engines, the agent can trace the dependencies between specific columns, showing exactly how an output column was derived from one or more input columns. This is invaluable for tracking sensitive data and understanding complex transformations.
  • Operational Statistics: Metadata about the run itself, such as the number of rows written or bytes processed.
• Governance Backend: Visualizing the Flow The agent sends this stream of standardized JSON-formatted lineage events to a compatible metadata service, with Marquez being the reference implementation for OpenLineage. Marquez consumes this metadata and builds a comprehensive, interactive graph of your entire data ecosystem. This provides a single, unified map showing how all datasets are created and consumed across Spark, Flink, and Trino, answering critical questions:
  • “If I change this table schema, what downstream jobs and dashboards will break?”
  • “This report looks wrong. Where did the data it uses come from?”
  • “Which jobs are processing PII data?”

By making lineage collection an automatic, non-negotiable part of the platform architecture, governance is no longer an afterthought. It becomes a reliable, built-in feature that provides a complete and auditable view of how data moves and transforms across the entire organization.

5. Catalog & Storage Layer: Foundation of Truth

Where does the data live? In a standard data lake, accessed via the gateway’s managed engines.

This layer is the physical and logical foundation of the platform. The gateway itself doesn’t store data, but it manages the engines that access it.

• Storage: Cloud object stores like Amazon S3, GCS, or ADLS.
• Table Formats: Open table formats like Apache Iceberg, Apache Hudi, Delta Lake, and Apache Paimon, which provide transactional (ACID) capabilities on top of raw data files in your data lake.
• Catalog: A universal Hive Metastore.
• Gateway’s Role: The gateway ensures that all access to the data lake is mediated. A user cannot simply spin up their own Spark session to bypass rules. For example, to read an Iceberg table, they must submit a SQL query through Kyuubi. The gateway then authenticates them and, particularly for Spark, ensures they are authorized before provisioning an engine to perform the read.

Use Case 1: Interactive Batch Query (Spark)

Let’s trace a classic analytics workflow to see the architecture in action.

1. An analyst connects Tableau to the Kyuubi JDBC endpoint.
2. They write a SQL query joining several large Iceberg tables. Tableau sends this standard SQL to the Kyuubi Gateway.
3. Kyuubi receives the query. It authenticates the user and checks their detailed permissions with Apache Ranger.
4. Seeing this is the user’s first query, Kyuubi provisions a new, dedicated Spark application for them on Kubernetes, injecting the OpenLineage agent.
5. Kyuubi forwards the SQL to the user’s isolated Spark engine.
6. The Spark engine uses the Hive Metastore to find the data’s location on S3 and executes the query.
7. Results are streamed back through the Spark engine, to the Kyuubi gateway, and finally to Tableau. The entire process is isolated, governed, and transparent to the user.

Use Case 2: Real-Time Streaming Query (Flink & Kafka)

The platform’s true power is its ability to handle more than just batch analytics. Let’s see how an analyst can query a live stream of clickstream data from Apache Kafka using Flink SQL, all through the same gateway.

The user connects their SQL client to the exact same Kyuubi JDBC endpoint. They then define a table backed by a Kafka topic and query it continuously:

 1-- Step 1: Map a table structure onto a live Kafka topic.
 2-- The underlying Flink engine uses its Kafka connector to handle this.
 3CREATE TABLE ClickStream (
 4    `user_id` BIGINT,
 5    `url` STRING,
 6    `event_timestamp` TIMESTAMP(3) METADATA FROM 'timestamp',
 7    WATERMARK FOR `event_timestamp` AS `event_timestamp` - INTERVAL '5' SECOND
 8) WITH (
 9    'connector' = 'kafka',
10    'topic' = 'clickstream_events',
11    'properties.bootstrap.servers' = 'kafka-broker-1:9092,...',
12    'scan.startup.mode' = 'latest-offset',
13    'format' = 'json'
14);
15
16-- Step 2: Run a continuous query against the live stream.
17-- This query will run indefinitely, pushing new results to the client as they arrive.
18SELECT
19    TUMBLE_START(event_timestamp, INTERVAL '1' MINUTE) as window_start,
20    url,
21    COUNT(*) as clicks
22FROM
23    ClickStream
24GROUP BY
25    TUMBLE(event_timestamp, INTERVAL '1' MINUTE),
26    url;

How it Works:

1. The user sends the CREATE TABLE and SELECT statements to the Kyuubi JDBC endpoint.
2. Kyuubi authenticates the user. Based on session properties, it recognizes this as a Flink workload and provisions a dedicated Flink session cluster on Kubernetes, again injecting the OpenLineage agent.
3. Kyuubi submits the SQL to the Flink cluster.
4. Flink’s SQL engine parses the query. It uses its Kafka connector to connect to the clickstream_events topic and begins consuming the JSON data stream.
5. The SELECT query runs continuously. As new data arrives in Kafka, Flink processes it, calculates the tumbling window aggregates, and streams the updated results back through the Kyuubi gateway to the analyst’s SQL client.
6. Simultaneously, the OpenLineage agent reports to Marquez that a new data flow has been established, drawing a lineage graph from the Kafka topic to this Flink SQL job.

Conclusion: The Power of a Gateway-Centric Design

Building a successful self-service platform is not about exposing raw power, but about providing controlled, stable, and governed access. By embracing a gateway-centric architecture built around the principles embodied by Apache Kyuubi, you can finally resolve the conflict between user empowerment and platform stability. The gateway acts as the indispensable control plane, turning a potential “Wild West” of big data engines into a well-regulated, multi-tenant, and powerful SQL service for the entire organization—for both batch and real-time workloads.

• Kafka Clients with JSON - Producing and Consuming Order Events
• Kafka Clients with Avro - Schema Registry and Order Events
• Kafka Streams - Lightweight Real-Time Processing for Supplier Stats
• Flink DataStream API - Scalable Event Processing for Supplier Stats
• Flink Table API - Declarative Analytics for Supplier Stats in Real Time (this post)

Flink Table Application

We develop a Flink application that uses Flink’s Table API and SQL-like expressions to perform real-time analytics. This application:

• Consumes Avro-formatted order events from a Kafka topic.
• Uses a mix of the DataStream and Table APIs to prepare data, handle late events, and define watermarks.
• Defines a table over the streaming data, complete with an event-time attribute and watermarks.
• Runs a declarative, SQL-like query to compute supplier statistics (total price and count) in 5-second tumbling windows.
• Splits the stream to route late-arriving records to a separate “skipped” topic for analysis.
• Sinks the aggregated results to a Kafka topic using the built-in avro-confluent format connector.

The source code for the application discussed in this post can be found in the orders-stats-flink folder of this GitHub repository.

The Build Configuration

The build.gradle.kts file sets up the project, its dependencies, and packaging. It’s shared between the DataStream and Table API applications - see the previous post for the Flink application that uses the DataStream API.

• Plugins:
  • kotlin("jvm"): Enables Kotlin language support.
  • com.github.davidmc24.gradle.plugin.avro: Compiles Avro schemas into Java classes.
  • com.github.johnrengelman.shadow: Creates an executable “fat JAR” with all dependencies.
  • application: Configures the project to be runnable via Gradle.
• Dependencies:
  • Flink Core & Table APIs: flink-streaming-java, flink-clients, and crucially, flink-table-api-java-bridge, flink-table-planner-loader, and flink-table-runtime for the Table API.
  • Flink Connectors: flink-connector-kafka for Kafka integration.
  • Flink Formats: flink-avro and flink-avro-confluent-registry for handling Avro data with Confluent Schema Registry.
  • Note on Dependency Scope: The Flink dependencies are declared with implementation. This allows the application to be run directly with ./gradlew run. For production deployments on a Flink cluster (where the Flink runtime is already provided), these dependencies should be changed to compileOnly to significantly reduce the size of the final JAR.
• Application Configuration:
  • The application block sets the mainClass and passes necessary JVM arguments. The run task is configured with environment variables to specify Kafka and Schema Registry connection details.
• Avro & Shadow JAR:
  • The avro block configures code generation.
  • The shadowJar task configures the output JAR name and merges service files, which is crucial for Flink connectors to work correctly.

 1plugins {
 2    kotlin("jvm") version "2.1.20"
 3    id("com.github.davidmc24.gradle.plugin.avro") version "1.9.1"
 4    id("com.github.johnrengelman.shadow") version "8.1.1"
 5    application
 6}
 7
 8group = "me.jaehyeon"
 9version = "1.0-SNAPSHOT"
10
11repositories {
12    mavenCentral()
13    maven("https://packages.confluent.io/maven")
14}
15
16dependencies {
17    // Flink Core and APIs
18    implementation("org.apache.flink:flink-streaming-java:1.20.1")
19    implementation("org.apache.flink:flink-table-api-java:1.20.1")
20    implementation("org.apache.flink:flink-table-api-java-bridge:1.20.1")
21    implementation("org.apache.flink:flink-table-planner-loader:1.20.1")
22    implementation("org.apache.flink:flink-table-runtime:1.20.1")
23    implementation("org.apache.flink:flink-clients:1.20.1")
24    implementation("org.apache.flink:flink-connector-base:1.20.1")
25    // Flink Kafka and Avro
26    implementation("org.apache.flink:flink-connector-kafka:3.4.0-1.20")
27    implementation("org.apache.flink:flink-avro:1.20.1")
28    implementation("org.apache.flink:flink-avro-confluent-registry:1.20.1")
29    // Json
30    implementation("com.fasterxml.jackson.module:jackson-module-kotlin:2.13.0")
31    // Logging
32    implementation("io.github.microutils:kotlin-logging-jvm:3.0.5")
33    implementation("ch.qos.logback:logback-classic:1.5.13")
34    // Kotlin test
35    testImplementation(kotlin("test"))
36}
37
38kotlin {
39    jvmToolchain(17)
40}
41
42application {
43    mainClass.set("me.jaehyeon.MainKt")
44    applicationDefaultJvmArgs =
45        listOf(
46            "--add-opens=java.base/java.util=ALL-UNNAMED",
47        )
48}
49
50avro {
51    setCreateSetters(true)
52    setFieldVisibility("PUBLIC")
53}
54
55tasks.named("compileKotlin") {
56    dependsOn("generateAvroJava")
57}
58
59sourceSets {
60    named("main") {
61        java.srcDirs("build/generated/avro/main")
62        kotlin.srcDirs("src/main/kotlin")
63    }
64}
65
66tasks.withType<com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar> {
67    archiveBaseName.set("orders-stats-flink")
68    archiveClassifier.set("")
69    archiveVersion.set("1.0")
70    mergeServiceFiles()
71}
72
73tasks.named("build") {
74    dependsOn("shadowJar")
75}
76
77tasks.named<JavaExec>("run") {
78    environment("TO_SKIP_PRINT", "false")
79    environment("BOOTSTRAP", "localhost:9092")
80    environment("REGISTRY_URL", "http://localhost:8081")
81}
82
83tasks.test {
84    useJUnitPlatform()
85}

Avro Schema for Supplier Statistics

The SupplierStats.avsc file defines the structure for the aggregated output data. This schema is used by the Flink Table API’s Kafka connector with the avro-confluent format to serialize the final Row results into Avro, ensuring type safety for downstream consumers.

• Type: A record named SupplierStats in the me.jaehyeon.avro namespace.
• Fields:
  • window_start and window_end (string): The start and end times of the aggregation window.
  • supplier (string): The supplier being aggregated.
  • total_price (double): The sum of order prices within the window.
  • count (long): The total number of orders within the window.

 1{
 2  "type": "record",
 3  "name": "SupplierStats",
 4  "namespace": "me.jaehyeon.avro",
 5  "fields": [
 6    { "name": "window_start", "type": "string" },
 7    { "name": "window_end", "type": "string" },
 8    { "name": "supplier", "type": "string" },
 9    { "name": "total_price", "type": "double" },
10    { "name": "count", "type": "long" }
11  ]
12}

Shared Utilities

These utility files provide common functionality used by both the DataStream and Table API applications.

Kafka Admin Utilities

This file provides two key helper functions for interacting with the Kafka ecosystem:

• createTopicIfNotExists(...): Uses Kafka’s AdminClient to programmatically create topics. It’s designed to be idempotent, safely handling cases where the topic already exists to prevent application startup failures.
• getLatestSchema(...): Connects to the Confluent Schema Registry using CachedSchemaRegistryClient to fetch the latest Avro schema for a given subject. This is essential for the Flink source to correctly deserialize incoming Avro records without hardcoding the schema in the application.

 1package me.jaehyeon.kafka
 2
 3import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient
 4import mu.KotlinLogging
 5import org.apache.avro.Schema
 6import org.apache.kafka.clients.admin.AdminClient
 7import org.apache.kafka.clients.admin.AdminClientConfig
 8import org.apache.kafka.clients.admin.NewTopic
 9import org.apache.kafka.common.errors.TopicExistsException
10import java.util.Properties
11import java.util.concurrent.ExecutionException
12import kotlin.use
13
14private val logger = KotlinLogging.logger { }
15
16fun createTopicIfNotExists(
17    topicName: String,
18    bootstrapAddress: String,
19    numPartitions: Int,
20    replicationFactor: Short,
21) {
22    val props =
23        Properties().apply {
24            put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress)
25            put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, "5000")
26            put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, "3000")
27            put(AdminClientConfig.RETRIES_CONFIG, "1")
28        }
29
30    AdminClient.create(props).use { client ->
31        val newTopic = NewTopic(topicName, numPartitions, replicationFactor)
32        val result = client.createTopics(listOf(newTopic))
33
34        try {
35            logger.info { "Attempting to create topic '$topicName'..." }
36            result.all().get()
37            logger.info { "Topic '$topicName' created successfully!" }
38        } catch (e: ExecutionException) {
39            if (e.cause is TopicExistsException) {
40                logger.warn { "Topic '$topicName' was created concurrently or already existed. Continuing..." }
41            } else {
42                throw RuntimeException("Unrecoverable error while creating a topic '$topicName'.", e)
43            }
44        }
45    }
46}
47
48fun getLatestSchema(
49    schemaSubject: String,
50    registryUrl: String,
51    registryConfig: Map<String, String>,
52): Schema {
53    val schemaRegistryClient =
54        CachedSchemaRegistryClient(
55            registryUrl,
56            100,
57            registryConfig,
58        )
59    logger.info { "Fetching latest schema for subject '$schemaSubject' from $registryUrl" }
60    try {
61        val latestSchemaMetadata = schemaRegistryClient.getLatestSchemaMetadata(schemaSubject)
62        logger.info {
63            "Successfully fetched schema ID ${latestSchemaMetadata.id} version ${latestSchemaMetadata.version} for subject '$schemaSubject'"
64        }
65        return Schema.Parser().parse(latestSchemaMetadata.schema)
66    } catch (e: Exception) {
67        logger.error(e) { "Failed to retrieve schema for subject '$schemaSubject' from registry $registryUrl" }
68        throw RuntimeException("Failed to retrieve schema for subject '$schemaSubject'", e)
69    }
70}

Flink Kafka Connectors

This file centralizes the creation of Flink’s Kafka sources and sinks. The Table API application uses createOrdersSource and createSkippedSink from this file. The sink for aggregated statistics is defined declaratively using a TableDescriptor instead of the createStatsSink function.

• createOrdersSource(...): Configures a KafkaSource to consume GenericRecord Avro data.
• createSkippedSink(...): Creates a generic KafkaSink for late records, which are handled as simple key-value string pairs.

  1package me.jaehyeon.kafka
  2
  3import me.jaehyeon.avro.SupplierStats
  4import org.apache.avro.Schema
  5import org.apache.avro.generic.GenericRecord
  6import org.apache.flink.connector.base.DeliveryGuarantee
  7import org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchema
  8import org.apache.flink.connector.kafka.sink.KafkaSink
  9import org.apache.flink.connector.kafka.source.KafkaSource
 10import org.apache.flink.connector.kafka.source.enumerator.initializer.OffsetsInitializer
 11import org.apache.flink.formats.avro.registry.confluent.ConfluentRegistryAvroDeserializationSchema
 12import org.apache.flink.formats.avro.registry.confluent.ConfluentRegistryAvroSerializationSchema
 13import org.apache.kafka.clients.consumer.ConsumerConfig
 14import org.apache.kafka.clients.producer.ProducerConfig
 15import java.nio.charset.StandardCharsets
 16import java.util.Properties
 17
 18fun createOrdersSource(
 19    topic: String,
 20    groupId: String,
 21    bootstrapAddress: String,
 22    registryUrl: String,
 23    registryConfig: Map<String, String>,
 24    schema: Schema,
 25): KafkaSource<GenericRecord> =
 26    KafkaSource
 27        .builder<GenericRecord>()
 28        .setBootstrapServers(bootstrapAddress)
 29        .setTopics(topic)
 30        .setGroupId(groupId)
 31        .setStartingOffsets(OffsetsInitializer.earliest())
 32        .setValueOnlyDeserializer(
 33            ConfluentRegistryAvroDeserializationSchema.forGeneric(
 34                schema,
 35                registryUrl,
 36                registryConfig,
 37            ),
 38        ).setProperties(
 39            Properties().apply {
 40                put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, "500")
 41            },
 42        ).build()
 43
 44fun createStatsSink(
 45    topic: String,
 46    bootstrapAddress: String,
 47    registryUrl: String,
 48    registryConfig: Map<String, String>,
 49    outputSubject: String,
 50): KafkaSink<SupplierStats> =
 51    KafkaSink
 52        .builder<SupplierStats>()
 53        .setBootstrapServers(bootstrapAddress)
 54        .setKafkaProducerConfig(
 55            Properties().apply {
 56                setProperty(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true")
 57                setProperty(ProducerConfig.LINGER_MS_CONFIG, "100")
 58                setProperty(ProducerConfig.BATCH_SIZE_CONFIG, (64 * 1024).toString())
 59                setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4")
 60            },
 61        ).setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
 62        .setRecordSerializer(
 63            KafkaRecordSerializationSchema
 64                .builder<SupplierStats>()
 65                .setTopic(topic)
 66                .setKeySerializationSchema { value: SupplierStats ->
 67                    value.supplier.toByteArray(StandardCharsets.UTF_8)
 68                }.setValueSerializationSchema(
 69                    ConfluentRegistryAvroSerializationSchema.forSpecific<SupplierStats>(
 70                        SupplierStats::class.java,
 71                        outputSubject,
 72                        registryUrl,
 73                        registryConfig,
 74                    ),
 75                ).build(),
 76        ).build()
 77
 78fun createSkippedSink(
 79    topic: String,
 80    bootstrapAddress: String,
 81): KafkaSink<Pair<String?, String>> =
 82    KafkaSink
 83        .builder<Pair<String?, String>>()
 84        .setBootstrapServers(bootstrapAddress)
 85        .setKafkaProducerConfig(
 86            Properties().apply {
 87                setProperty(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true")
 88                setProperty(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4")
 89                setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArraySerializer")
 90                setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.ByteArraySerializer")
 91            },
 92        ).setDeliveryGuarantee(DeliveryGuarantee.AT_LEAST_ONCE)
 93        .setRecordSerializer(
 94            KafkaRecordSerializationSchema
 95                .builder<Pair<String?, String>>()
 96                .setTopic(topic)
 97                .setKeySerializationSchema { pair: Pair<String?, String> ->
 98                    pair.first?.toByteArray(StandardCharsets.UTF_8)
 99                }.setValueSerializationSchema { pair: Pair<String?, String> ->
100                    pair.second.toByteArray(StandardCharsets.UTF_8)
101                }.build(),
102        ).build()

Table API Processing Logic

The following files contain the core logic specific to the Table API implementation.

Manual Late Data Routing

Because the Table API does not have a direct equivalent to the DataStream API’s .sideOutputLateData(), we must handle late records manually. This ProcessFunction is a key component. It inspects each record’s timestamp against the current watermark and an allowedLatenessMillis threshold. Records deemed “too late” are routed to a side output, while on-time records are passed downstream to be converted into a Table.

 1package me.jaehyeon.flink.processing
 2
 3import org.apache.flink.streaming.api.functions.ProcessFunction
 4import org.apache.flink.util.Collector
 5import org.apache.flink.util.OutputTag
 6
 7class LateDataRouter(
 8    private val lateOutputTag: OutputTag<RecordMap>,
 9    private val allowedLatenessMillis: Long,
10) : ProcessFunction<RecordMap, RecordMap>() {
11
12    init {
13        require(allowedLatenessMillis >= 0) {
14            "allowedLatenessMillis cannot be negative. Got: $allowedLatenessMillis"
15        }
16    }
17
18    @Throws(Exception::class)
19    override fun processElement(
20        value: RecordMap,
21        ctx: ProcessFunction<RecordMap, RecordMap>.Context,
22        out: Collector<RecordMap>,
23    ) {
24        val elementTimestamp: Long? = ctx.timestamp()
25        val currentWatermark: Long = ctx.timerService().currentWatermark()
26
27        // Element has no timestamp or watermark is still at its initial value
28        if (elementTimestamp == null || currentWatermark == Long.MIN_VALUE) {
29            out.collect(value)
30            return
31        }
32
33        // Element has a timestamp and watermark is active.
34        // An element is "too late" if its timestamp is older than current watermark - allowed lateness.
35        if (elementTimestamp < currentWatermark - allowedLatenessMillis) {
36            ctx.output(lateOutputTag, value)
37        } else {
38            out.collect(value)
39        }
40    }
41}

Timestamp and Watermark Strategy for Rows

After the initial stream processing and before converting the DataStream to a Table, we need a watermark strategy that operates on Flink’s Row type. This strategy extracts the timestamp from a specific field index in the Row (in this case, field 1, which holds the bid_time as a Long) and generates watermarks, allowing the Table API to correctly perform event-time windowing.

 1package me.jaehyeon.flink.watermark
 2
 3import mu.KotlinLogging
 4import org.apache.flink.api.common.eventtime.WatermarkStrategy
 5import org.apache.flink.types.Row
 6
 7object RowWatermarkStrategy {
 8    private val logger = KotlinLogging.logger {}
 9
10    val strategy: WatermarkStrategy<Row> =
11        WatermarkStrategy
12            .forBoundedOutOfOrderness<Row>(java.time.Duration.ofSeconds(5))
13            .withTimestampAssigner { row: Row, _ ->
14                try {
15                    // Get the field by index. Assumes bid_time is at index 1 and is Long.
16                    val timestamp = row.getField(1) as? Long
17                    if (timestamp != null) {
18                        timestamp
19                    } else {
20                        logger.warn { "Null or invalid timestamp at index 1 in Row: $row. Using current time." }
21                        System.currentTimeMillis() // Fallback
22                    }
23                } catch (e: Exception) {
24                    // Catch potential ClassCastException or other issues
25                    logger.error(e) { "Error accessing timestamp at index 1 in Row: $row. Using current time." }
26                    System.currentTimeMillis() // Fallback
27                }
28            }.withIdleness(java.time.Duration.ofSeconds(10)) // Same idleness
29}

Not Applicable Source Code

The files SupplierWatermarkStrategy, SupplierStatsAggregator, and SupplierStatsFunction are used exclusively by the Flink DataStream API application for its specific watermark and aggregation logic. They are not relevant to this Table API implementation.

Core Table API Application

This is the main driver for the Table API application. It demonstrates the powerful integration between the DataStream and Table APIs.

1. Environment Setup: It initializes both a StreamExecutionEnvironment and a StreamTableEnvironment.
2. Data Ingestion and Preparation:
   • It consumes Avro GenericRecords using a KafkaSource and maps them to a DataStream<RecordMap>.
   • It applies a WatermarkStrategy to the RecordMap stream so that the subsequent LateDataRouter can function correctly based on event time.
3. Late Data Splitting: It uses the custom LateDataRouter ProcessFunction to split the stream into an on-time stream and a late-data side output.
4. DataStream-to-Table Conversion:
   • The on-time DataStream<RecordMap> is converted to a DataStream<Row>. This step transforms the data into the structured, columnar format required by the Table API.
   • A second WatermarkStrategy (RowWatermarkStrategy) is applied to the DataStream<Row>.
   • tEnv.createTemporaryView registers the DataStream<Row> as a table named “orders”. A Schema is defined, crucially marking the bid_time column as the event-time attribute (TIMESTAMP_LTZ(3)) and telling Flink to use the watermarks generated by the DataStream (SOURCE_WATERMARK()).
5. Declarative Query: A high-level, declarative query is executed on the “orders” table. It uses Tumble to define 5-second windows and performs groupBy and select operations with aggregate functions (sum, count) to calculate the statistics.
6. Sinking:
   • A TableDescriptor is defined for the Kafka sink. It specifies the sink schema and, most importantly, the avro-confluent format, which handles serialization to Avro and integration with Schema Registry automatically.
   • statsTable.executeInsert() writes the results of the query to the sink.
   • The separate late data stream is processed and sunk to its own topic.
7. Execution: env.execute() starts the Flink job.

  1@file:Suppress("ktlint:standard:no-wildcard-imports")
  2
  3package me.jaehyeon
  4
  5import com.fasterxml.jackson.databind.ObjectMapper
  6import com.fasterxml.jackson.module.kotlin.registerKotlinModule
  7import me.jaehyeon.flink.processing.LateDataRouter
  8import me.jaehyeon.flink.processing.RecordMap
  9import me.jaehyeon.flink.watermark.RowWatermarkStrategy
 10import me.jaehyeon.flink.watermark.SupplierWatermarkStrategy
 11import me.jaehyeon.kafka.createOrdersSource
 12import me.jaehyeon.kafka.createSkippedSink
 13import me.jaehyeon.kafka.createTopicIfNotExists
 14import me.jaehyeon.kafka.getLatestSchema
 15import mu.KotlinLogging
 16import org.apache.avro.generic.GenericRecord
 17import org.apache.flink.api.common.eventtime.WatermarkStrategy
 18import org.apache.flink.api.common.typeinfo.TypeHint
 19import org.apache.flink.api.common.typeinfo.TypeInformation
 20import org.apache.flink.api.java.typeutils.RowTypeInfo
 21import org.apache.flink.streaming.api.datastream.DataStream
 22import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator
 23import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
 24import org.apache.flink.streaming.connectors.kafka.table.KafkaConnectorOptions
 25import org.apache.flink.table.api.DataTypes
 26import org.apache.flink.table.api.Expressions.*
 27import org.apache.flink.table.api.Expressions.lit
 28import org.apache.flink.table.api.FormatDescriptor
 29import org.apache.flink.table.api.Schema
 30import org.apache.flink.table.api.Table
 31import org.apache.flink.table.api.TableDescriptor
 32import org.apache.flink.table.api.Tumble
 33import org.apache.flink.table.api.bridge.java.StreamTableEnvironment
 34import org.apache.flink.types.Row
 35import org.apache.flink.util.OutputTag
 36import java.time.Instant
 37import java.time.LocalDateTime
 38import java.time.ZoneId
 39import java.time.format.DateTimeFormatter
 40import java.time.format.DateTimeParseException
 41
 42object TableApp {
 43    private val toSkipPrint = System.getenv("TO_SKIP_PRINT")?.toBoolean() ?: true
 44    private val bootstrapAddress = System.getenv("BOOTSTRAP") ?: "kafka-1:19092"
 45    private val inputTopicName = System.getenv("TOPIC") ?: "orders-avro"
 46    private val registryUrl = System.getenv("REGISTRY_URL") ?: "http://schema:8081"
 47    private val registryConfig =
 48        mapOf(
 49            "basic.auth.credentials.source" to "USER_INFO",
 50            "basic.auth.user.info" to "admin:admin",
 51        )
 52    private const val INPUT_SCHEMA_SUBJECT = "orders-avro-value"
 53    private const val NUM_PARTITIONS = 3
 54    private const val REPLICATION_FACTOR: Short = 3
 55    private val logger = KotlinLogging.logger {}
 56
 57    // ObjectMapper for converting late data Map to JSON
 58    private val objectMapper: ObjectMapper by lazy {
 59        ObjectMapper().registerKotlinModule()
 60    }
 61
 62    fun run() {
 63        // Create output topics if not existing
 64        val outputTopicName = "$inputTopicName-ktl-stats"
 65        val skippedTopicName = "$inputTopicName-ktl-skipped"
 66        listOf(outputTopicName, skippedTopicName).forEach { name ->
 67            createTopicIfNotExists(
 68                name,
 69                bootstrapAddress,
 70                NUM_PARTITIONS,
 71                REPLICATION_FACTOR,
 72            )
 73        }
 74
 75        val env = StreamExecutionEnvironment.getExecutionEnvironment()
 76        env.parallelism = 3
 77        val tEnv = StreamTableEnvironment.create(env)
 78
 79        val inputAvroSchema = getLatestSchema(INPUT_SCHEMA_SUBJECT, registryUrl, registryConfig)
 80        val ordersGenericRecordSource =
 81            createOrdersSource(
 82                topic = inputTopicName,
 83                groupId = "$inputTopicName-flink-tl",
 84                bootstrapAddress = bootstrapAddress,
 85                registryUrl = registryUrl,
 86                registryConfig = registryConfig,
 87                schema = inputAvroSchema,
 88            )
 89
 90        // 1. Stream of GenericRecords from Kafka
 91        val genericRecordStream: DataStream<GenericRecord> =
 92            env
 93                .fromSource(ordersGenericRecordSource, WatermarkStrategy.noWatermarks(), "KafkaGenericRecordSource")
 94
 95        // 2. Convert GenericRecord to Map<String, Any?> (RecordMap)
 96        val recordMapStream: DataStream<RecordMap> =
 97            genericRecordStream
 98                .map { genericRecord ->
 99                    val map = mutableMapOf<String, Any?>()
100                    genericRecord.schema.fields.forEach { field ->
101                        val value = genericRecord.get(field.name())
102                        map[field.name()] = if (value is org.apache.avro.util.Utf8) value.toString() else value
103                    }
104                    map as RecordMap // Cast to type alias
105                }.name("GenericRecordToMapConverter")
106                .returns(TypeInformation.of(object : TypeHint<RecordMap>() {}))
107
108        // 3. Define OutputTag for late data (now carrying RecordMap)
109        val lateMapOutputTag =
110            OutputTag(
111                "late-order-records",
112                TypeInformation.of(object : TypeHint<RecordMap>() {}),
113            )
114
115        // 4. Split late records from on-time ones
116        val statsStreamOperator: SingleOutputStreamOperator<RecordMap> =
117            recordMapStream
118                .assignTimestampsAndWatermarks(SupplierWatermarkStrategy.strategy)
119                .process(LateDataRouter(lateMapOutputTag, allowedLatenessMillis = 5000))
120                .name("LateDataRouter")
121
122        // 5. Create source table (statsTable)
123        val statsStream: DataStream<RecordMap> = statsStreamOperator
124        val rowStatsStream: DataStream<Row> =
125            statsStream
126                .map { recordMap ->
127                    val orderId = recordMap["order_id"] as? String
128                    val price = recordMap["price"] as? Double
129                    val item = recordMap["item"] as? String
130                    val supplier = recordMap["supplier"] as? String
131
132                    val bidTimeString = recordMap["bid_time"] as? String
133                    var bidTimeInstant: Instant? = null // Changed from bidTimeLong to bidTimeInstant
134
135                    if (bidTimeString != null) {
136                        try {
137                            val formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
138                            val localDateTime = LocalDateTime.parse(bidTimeString, formatter)
139                            // Convert to Instant
140                            bidTimeInstant =
141                                localDateTime
142                                    .atZone(ZoneId.systemDefault()) // Or ZoneOffset.UTC
143                                    .toInstant()
144                        } catch (e: DateTimeParseException) {
145                            logger.error(e) { "Failed to parse bid_time string '$bidTimeString'. RecordMap: $recordMap" }
146                        } catch (e: Exception) {
147                            logger.error(e) { "Unexpected error parsing bid_time string '$bidTimeString'. RecordMap: $recordMap" }
148                        }
149                    } else {
150                        logger.warn { "bid_time string is null in RecordMap: $recordMap" }
151                    }
152
153                    Row.of(
154                        orderId,
155                        bidTimeInstant,
156                        price,
157                        item,
158                        supplier,
159                    )
160                }.returns(
161                    RowTypeInfo(
162                        arrayOf<TypeInformation<*>>(
163                            TypeInformation.of(String::class.java),
164                            TypeInformation.of(Instant::class.java), // bid_time (as Long milliseconds for TIMESTAMP_LTZ)
165                            TypeInformation.of(Double::class.java),
166                            TypeInformation.of(String::class.java),
167                            TypeInformation.of(String::class.java),
168                        ),
169                        arrayOf("order_id", "bid_time", "price", "item", "supplier"),
170                    ),
171                ).assignTimestampsAndWatermarks(RowWatermarkStrategy.strategy)
172                .name("MapToRowConverter")
173        val tableSchema =
174            Schema
175                .newBuilder()
176                .column("order_id", DataTypes.STRING())
177                .column("bid_time", DataTypes.TIMESTAMP_LTZ(3)) // Event time attribute
178                .column("price", DataTypes.DOUBLE())
179                .column("item", DataTypes.STRING())
180                .column("supplier", DataTypes.STRING())
181                .watermark("bid_time", "SOURCE_WATERMARK()") // Use watermarks from DataStream
182                .build()
183        tEnv.createTemporaryView("orders", rowStatsStream, tableSchema)
184
185        val statsTable: Table =
186            tEnv
187                .from("orders")
188                .window(Tumble.over(lit(5).seconds()).on(col("bid_time")).`as`("w"))
189                .groupBy(col("supplier"), col("w"))
190                .select(
191                    col("supplier"),
192                    col("w").start().`as`("window_start"),
193                    col("w").end().`as`("window_end"),
194                    col("price").sum().round(2).`as`("total_price"),
195                    col("order_id").count().`as`("count"),
196                )
197
198        // 6. Create sink table
199        val sinkSchema =
200            Schema
201                .newBuilder()
202                .column("supplier", DataTypes.STRING())
203                .column("window_start", DataTypes.TIMESTAMP(3))
204                .column("window_end", DataTypes.TIMESTAMP(3))
205                .column("total_price", DataTypes.DOUBLE())
206                .column("count", DataTypes.BIGINT())
207                .build()
208        val kafkaSinkDescriptor: TableDescriptor =
209            TableDescriptor
210                .forConnector("kafka")
211                .schema(sinkSchema) // Set the schema for the sink
212                .option(KafkaConnectorOptions.TOPIC, listOf(outputTopicName))
213                .option(KafkaConnectorOptions.PROPS_BOOTSTRAP_SERVERS, bootstrapAddress)
214                .format(
215                    FormatDescriptor
216                        .forFormat("avro-confluent")
217                        .option("url", registryUrl)
218                        .option("basic-auth.credentials-source", "USER_INFO")
219                        .option("basic-auth.user-info", "admin:admin")
220                        .option("subject", "$outputTopicName-value")
221                        .build(),
222                ).build()
223
224        // 7. Handle late data as a pair of key and value
225        val lateDataMapStream: DataStream<RecordMap> = statsStreamOperator.getSideOutput(lateMapOutputTag)
226        val lateKeyPairStream: DataStream<Pair<String?, String>> =
227            lateDataMapStream
228                .map { recordMap ->
229                    val mutableMap = recordMap.toMutableMap()
230                    mutableMap["late"] = true
231                    val orderId = mutableMap["order_id"] as? String
232                    try {
233                        val value = objectMapper.writeValueAsString(mutableMap)
234                        Pair(orderId, value)
235                    } catch (e: Exception) {
236                        logger.error(e) { "Error serializing late RecordMap to JSON: $mutableMap" }
237                        val errorJson = "{ \"error\": \"json_serialization_failed\", \"data_keys\": \"${
238                            mutableMap.keys.joinToString(
239                                ",",
240                            )}\" }"
241                        Pair(orderId, errorJson)
242                    }
243                }.returns(TypeInformation.of(object : TypeHint<Pair<String?, String>>() {}))
244        val skippedSink =
245            createSkippedSink(
246                topic = skippedTopicName,
247                bootstrapAddress = bootstrapAddress,
248            )
249
250        if (!toSkipPrint) {
251            tEnv
252                .toDataStream(statsTable)
253                .print()
254                .name("SupplierStatsPrint")
255            lateKeyPairStream
256                .map { it.second }
257                .print()
258                .name("LateDataPrint")
259        }
260
261        statsTable.executeInsert(kafkaSinkDescriptor)
262        lateKeyPairStream.sinkTo(skippedSink).name("LateDataSink")
263        env.execute("SupplierStats")
264    }
265}