Python on Jaehyeon Kim

odctl 1.0: Kafka, Flink, Spark, Iceberg, Trino and MLflow on Your Laptop with One Command

Mon, 12 Oct 2026 00:00:00 +0000

Trying a modern data tool on your own machine is rarely about the tool. Writing Iceberg tables from Flink needs a catalog, the catalog needs a database and object storage, Avro on Kafka needs a schema registry, and every piece needs the right connector jars, ports and network settings. Most of an afternoon goes into making them talk to each other before you write a line of your own code.

odctl removes that work. It is a command-line tool that runs a set of open-source data and machine learning services on your laptop with Docker, already connected to each other. You say what you want, for example Kafka and Flink, and it starts them together with everything they depend on. It has now reached 1.0, with a documentation site that has a guide for every service.

Defining Data-Streaming Simulations in YAML, Without Writing Python

Tue, 06 Oct 2026 00:00:00 +0000

A pipeline is easier to test when its input behaves like production: orders arrive at random, queues grow, a machine slows down. A simulation can produce that data, but when the simulation is a Python program, every change is a code change. Someone who only wants to double an arrival rate has to read and edit code, and a reviewer has to check that nothing else moved. A file that holds only settings is easier to read, to compare in a pull request and to run in CI.

dynamic-des 0.16.0 adds that file. A YAML blueprint describes the whole simulation, and the ddes command runs it. Each of the seven examples in the repository now has a YAML version in plain YAML. The documentation was rebuilt around the three ways to write a simulation.

Dynamic DES: A Declarative API with Postgres and Redis Connectors

Fri, 17 Jul 2026 00:00:00 +0000

Simulated data is most useful when the code that produces it is easy to read, and when it reaches the same places as production data, such as a PostgreSQL table or a Redis stream. Since I wrote about Dynamic DES v0.8.1 and its data lake support, the project has grown in both directions: a cleaner way to write a simulation, and more places to send and receive data.

Here is what changed on the way to v0.11.1.

Running Kafka, Flink, Spark, Trino and Iceberg Locally with One CLI

Thu, 16 Jul 2026 00:00:00 +0000

Anyone who has tried to stand up a realistic data platform on their laptop knows the pain. You want Kafka talking to Flink, Spark writing to Iceberg, Trino querying the result, and maybe a catalog and a lineage tool watching over all of it. What you actually get is an afternoon lost to dependency conflicts, port clashes, and a Docker Compose file that grows a new bug every time you touch it.

I built odctl to make that afternoon disappear. It is a curated collection of open-source data technologies wrapped in a small CLI, and it is now available on PyPI.

One Simulation, Two Pipelines: Batch Training and Live Inference with Dynamic DES

Mon, 25 May 2026 00:00:00 +0000

Training a machine learning model on simulated data is straightforward until you try to deploy it. The disconnect usually happens at the pipeline level: training requires massive, historical batch data (like Parquet files in an S3 bucket), but production inference requires real-time, event-driven streams (like Kafka or Redis). Maintaining two separate simulation codebases, one for generating training data and another for streaming live events, introduces friction, schema mismatches, and duplicated engineering effort.

Building an Event-Driven Hybrid Digital Twin with dynamic-des

Wed, 29 Apr 2026 00:00:00 +0000

Asynchronous Gap A true Hybrid Digital Twin does more than just mirror reality. It actively forecasts the future by running a simulation against live operational states. We established this in Part 1. If you have ever tried to build one of these systems from scratch, you immediately hit a fundamental architectural clash. Standard simulation clocks (like those in traditional SimPy implementations) are logically synchronous and not designed to handle high-frequency asynchronous I/O without explicit decoupling.

Why Digital Twins Are Rewiring Industry 4.0

Wed, 22 Apr 2026 00:00:00 +0000

A simulation and a digital twin are often treated as the same thing, but they are built differently. A traditional simulation runs offline on assumed inputs. An operational digital twin stays aligned with one real system as it changes. This post walks through the layers that separate the two, and an event-driven hybrid that combines them. Beyond CAD Models There is a project by Dassault Systèmes called the Living Heart that illustrates the trajectory of this technology.

Building a Real-Time Industrial Digital Twin with Apache Flink and Online Machine Learning

Tue, 21 Apr 2026 00:00:00 +0000

Overview Imagine using a rolling pin to flatten out a thick piece of dough. A Hot Strip Mill does the exact same thing, but with glowing red-hot steel slabs (often heated over 1000°C) and massive mechanical rollers. The steel is passed through a series of these rollers, crushing it down from a thick block into a long, thin sheet. Calculating the exact Rolling Force required to crush the steel is critical.

Productionizing an Online Product Recommender using Event Driven Architecture

Mon, 23 Feb 2026 00:00:00 +0000

Real-world recommendation systems require low-latency inference for users and high-throughput training for model updates. A monolithic script cannot handle production scale, while it is effective for testing algorithms locally. In Part 1, we built a contextual bandit prototype using Python and Mab2Rec.

This post demonstrates how to decouple these concerns using an event-driven architecture with Apache Flink, Kafka, and Valkey.

Prototyping an Online Product Recommender in Python

Tue, 27 Jan 2026 00:00:00 +0000

Overview Traditional recommendation approaches such as Collaborative Filtering remain widely adopted, yet they come with notable constraints. They are particularly vulnerable to the cold-start problem, where new users lack sufficient interaction history, and they depend heavily on long-term behavioral data. As a result, they frequently overlook real-time contextual signals, including time of day, device type, location, or session intent. This can prevent them from capturing situational preferences, such as someone preferring coffee in the morning but pizza in the evening.

Guide to Building Integrated Web Applications with FastAPI and NiceGUI

Wed, 19 Nov 2025 00:00:00 +0000

The standard architecture for modern web applications involves a decoupled frontend, typically built with a JavaScript framework, and a backend API. This pattern is powerful but introduces complexity in managing two separate codebases, development environments, and the API contract between them.

This article explores an alternative approach: an integrated architecture where the backend API and the frontend UI are served from a single, cohesive Python application.

Streamlit Dashboard - Realtime Dashboard with FastAPI, Streamlit and Next.js Part 2

Tue, 25 Feb 2025 00:00:00 +0000

A real-time monitoring dashboard is developed using Streamlit, an open-source Python framework that allows data scientists and AI/ML engineers to create interactive data apps. The app connects to the WebSocket server we developed in Part 1 and continuously fetches data to visualize key metrics such as order counts, sales data, and revenue by traffic source and country. With interactive bar charts and dynamic metrics, users can monitor sales trends and other important business KPIs in real-time.

Realtime Dashboard with FastAPI, Streamlit and Next.js - Part 1 Data Producer

Tue, 18 Feb 2025 00:00:00 +0000

A data generating app is created with Python, and it ingests the theLook eCommerce data continuously into a PostgreSQL database. A WebSocket server, built by FastAPI, periodically queries the data to serve its clients. In this series, we develop real-time monitoring dashboard applications, and this post walks through the data generation app and backend API. The monitoring dashboards will be developed using Streamlit and Next.js, with Apache ECharts for visualization. They will be discussed in later posts.

Develop Streaming File Reader using Splittable DoFn - Apache Beam Python Examples Part 10

Thu, 19 Dec 2024 00:00:00 +0000

We can extend a batch file reader so that, instead of listing files once at the beginning, it scans an input folder periodically for new files and processes whenever new files are created in the folder. In Part 9, we developed two Apache Beam pipelines using Splittable DoFn (SDF), and one of them is that batch file reader, which reads a list of files in an input folder followed by processing them in parallel. The techniques used here can be quite useful as they can be applied to developing I/O connectors that target other unbounded (or streaming) data sources (eg Kafka) using the Python SDK.

Develop Batch File Reader and PiSampler using Splittable DoFn - Apache Beam Python Examples Part 9

Thu, 05 Dec 2024 00:00:00 +0000

A Splittable DoFn (SDF) is a generalization of a DoFn that enables Apache Beam developers to create modular and composable I/O components. Also, it can be applied in advanced non-I/O scenarios such as Monte Carlo simulation. In this post, we develop two Apache Beam pipelines. The first pipeline is an I/O connector, and it reads a list of files in a folder followed by processing each of the file objects in parallel. The second pipeline estimates the value of $\pi$ by performing Monte Carlo simulation.

Enhance Sport Activity Tracker with Runner Motivation - Apache Beam Python Examples Part 8

Thu, 21 Nov 2024 00:00:00 +0000

We develop a new pipeline that tracks user activities and reports pacing details by comparing short term metrics to their long term counterparts. In Part 3, we developed a Beam pipeline that tracks sport activities of users and outputs their speeds periodically. While reporting such values is useful for users on its own, we can provide more engaging information to users if we have a pipeline that reports pacing of their activities over periods. For example, we can send a message to encourage a user to work harder if he/she has a performance goal and is underperforming for some periods.

Apache Beam Python Examples - Part 7 Separate Droppable Data into Side Output

Thu, 24 Oct 2024 00:00:00 +0000

We develop an Apache Beam pipeline that separates droppable elements from the rest of the data. Droppable elements are those that come later when the watermark passes the window max timestamp plus allowed lateness. Using a timer in a Stateful DoFn, droppable data is separated from normal data and dispatched into a side output rather than being discarded silently, which is the default behaviour. Note that this pipeline works in a situation where droppable elements do not appear often, and thus the chance that a droppable element is delivered as the first element in a particular window is low.

Call RPC Service in Batch with Defined Batch Size using Stateful DoFn - Apache Beam Python Examples Part 6

Wed, 02 Oct 2024 00:00:00 +0000

A stateful DoFn improves the pipeline so that the number elements to process and maximum wait seconds can be controlled by state and timers. In the previous post, we continued discussing an Apache Beam pipeline that arguments input data by calling a Remote Procedure Call (RPC) service. A pipeline was developed that makes a single RPC call for a bundle of elements. The bundle size is determined by the runner, however, we may encounter an issue e.g. if an RPC service becomes quite slower if many elements are included in a single request. Note that, although the stateful DoFn used in this post solves the data augmentation task well, in practice, we should use the built-in transforms such as BatchElements and GroupIntoBatches whenever possible.

Call RPC Service in Batch using Stateless DoFn - Apache Beam Python Examples Part 5

Wed, 18 Sep 2024 00:00:00 +0000

We enhance the pipeline so that a single RPC call is made for a bundle of elements, which can save a significant amount time compared to making a call for each element. In the previous post, we developed an Apache Beam pipeline where the input data is augmented by a Remote Procedure Call (RPC) service. Each input element performs an RPC call and the output is enriched by the response. This is not an efficient way of accessing an external service provided that the service can accept more than one element.

Cache Data on Apache Beam Pipelines Using a Shared Object

Thu, 22 Aug 2024 00:00:00 +0000

I recently contributed to Apache Beam by adding a common pipeline pattern - Cache data using a shared object. Both batch and streaming pipelines are introduced, and they utilise the Shared class of the Python SDK to enrich PCollection elements. This pattern can be more memory-efficient than side inputs, simpler than a stateful DoFn, and more performant than calling an external service, because it does not have to access an external service for every element or bundle of elements. In this post, we discuss this pattern in more details with batch and streaming use cases. For the latter, we configure the cache gets refreshed periodically.

Apache Beam Python Examples - Part 4 Call RPC Service for Data Augmentation

Thu, 15 Aug 2024 00:00:00 +0000

Each input element performs a Remote Procedure Call (RPC) and the output is enriched by the response. This is not an efficient way of accessing an external service provided that the service can accept more than one element. In this post, we develop an Apache Beam pipeline where the input data is augmented by an RPC service. We begin with illustrating how to manage development resources followed by demonstrating the RPC service that we use in this series. Finally, we develop a Beam pipeline that accesses the external service to augment the input elements. In the subsequent two posts, we will discuss updated pipelines that make RPC calls more efficiently.

Build Sport Activity Tracker with/without SQL - Apache Beam Python Examples Part 3

Thu, 01 Aug 2024 00:00:00 +0000

Two Apache Beam pipelines that track sport activities of users and output their speed periodically are developed in this post. The first pipeline uses native transforms and Beam SQL is used for the latter. While Beam SQL can be useful in some situations, its features in the Python SDK are not complete compared to the Java SDK. Therefore, we are not able to build the required tracking pipeline using it. We end up discussing potential improvements of Beam SQL so that it can be used for building competitive applications with the Python SDK.

Calculate Average Word Length with/without Fixed Look back - Apache Beam Python Examples Part 2

Thu, 18 Jul 2024 00:00:00 +0000

Two Apache Beam pipelines calculate average word lengths from input texts that are ingested by a Kafka topic. They obtain the statistics in different angles. The first pipeline emits the global average lengths whenever a new input text arrives while the latter triggers those values in a sliding time window. We develop both pipelines in this post.

Calculate K Most Frequent Words and Max Word Length - Apache Beam Python Examples Part 1

Thu, 04 Jul 2024 00:00:00 +0000

Beginning with setting up the development environment, we build two pipelines that obtain top K most frequent words and the word that has the longest word length in this post. In this series, we develop Apache Beam Python pipelines. The majority of them are from Building Big Data Pipelines with Apache Beam by Jan Lukavský. Mainly relying on the Java SDK, the book teaches fundamentals of Apache Beam using hands-on tasks, and we convert those tasks using the Python SDK. We focus on streaming pipelines, and they are deployed on a local (or embedded) Apache Flink cluster using the Apache Flink Runner.

Beam Pipeline on Flink Runner - Deploy Python Stream Processing App on Kubernetes Part 2

Thu, 06 Jun 2024 00:00:00 +0000

An Apache Beam pipeline built with the Python SDK runs on an Apache Flink cluster via the Apache Flink Runner. We develop the pipeline as a Python package and add the package to a custom Docker image so that Python user code can be executed externally. In this post, we create a Flink session cluster via the Flink Kubernetes Operator, and deploy the pipeline using a Kubernetes job. Same as Part I, we deploy a Kafka cluster using the Strimzi Operator on a minikube cluster as the pipeline uses Apache Kafka topics for its data source and sink. Finally, we check the output of the application by sending messages to the input Kafka topic using a Python producer application.

Deploy Python Stream Processing App on Kubernetes - Part 1 PyFlink Application

Thu, 30 May 2024 00:00:00 +0000

Flink Kubernetes Operator acts as a control plane to manage the complete deployment lifecycle of Apache Flink applications. With the operator, we can simplify deployment and management of Python stream processing applications. In this series, we discuss how to deploy a PyFlink application and Python Apache Beam pipeline on the Flink Runner on Kubernetes. In Part 1, we first deploy a Kafka cluster on a minikube cluster as the source and sink of the PyFlink application are Kafka topics. Then, the application source is packaged in a custom Docker image and deployed on the minikube cluster using the Flink Kubernetes Operator. Finally, the output of the application is checked by sending messages to the input Kafka topic using a Python producer application.

Apache Beam Local Development with Python - Part 5 Testing Pipelines

Thu, 09 May 2024 00:00:00 +0000

We developed batch and streaming pipelines in Part 2 and Part 4. Often it is faster and simpler to identify and fix bugs on the pipeline code by performing local unit testing. Moreover, especially when it comes to creating a streaming pipeline, unit testing cases can facilitate development further by using TestStream as it allows us to advance watermarks or processing time according to different scenarios. In this post, we discuss how to perform unit testing of the batch and streaming pipelines that we developed earlier.

Apache Beam Local Development with Python - Part 4 Streaming Pipelines

Thu, 02 May 2024 00:00:00 +0000

We build a streaming pipeline that aggregates page visits by user in a fixed time window of 20 seconds. Two versions of the pipeline are created with/without relying on Beam SQL. In Part 3, we discussed the portability layer of Apache Beam as it helps understand (1) how Python pipelines run on the Flink Runner and (2) how multiple SDKs can be used in a single pipeline, followed by demonstrating local Flink and Kafka cluster creation for developing streaming pipelines.

Apache Beam Local Development with Python - Part 3 Flink Runner

Thu, 18 Apr 2024 00:00:00 +0000

In this series, we discuss local development of Apache Beam pipelines using Python. In the previous posts, we mainly talked about Batch pipelines with/without Beam SQL. Beam pipelines are portable between batch and streaming semantics, and we will discuss streaming pipeline development in this and the next posts. While there are multiple Beam Runners, not every Runner supports Python or some Runners have too limited features in streaming semantics - see Beam Capability Matrix for details.

Apache Beam Local Development with Python - Part 2 Batch Pipelines

Thu, 04 Apr 2024 00:00:00 +0000

We discuss Batch pipelines that aggregate website visit log by user and time in this post. The pipelines are developed with and without Beam SQL. Additionally, each pipeline is implemented on a Jupyter notebook for demonstration. In this series, we discuss local development of Apache Beam pipelines using Python. A basic Beam pipeline was introduced in Part 1, followed by demonstrating how to utilise Jupyter notebooks, Beam SQL and Beam DataFrames.

Pipeline, Notebook, SQL and DataFrame - Apache Beam Local Development with Python Part 1

Thu, 28 Mar 2024 00:00:00 +0000

Apache Beam and Apache Flink are open-source frameworks for parallel, distributed data processing at scale. Flink has DataStream and Table/SQL APIs and the former has more capacity to develop sophisticated data streaming applications. The DataStream API of PyFlink, Flink’s Python API, however, is not as complete as its Java counterpart, and it doesn’t provide enough capability to extend when there are missing features in Python. Recently I had a chance to look through Apache Beam and found it supports more possibility to extend and/or customise its features.

Data Build Tool (dbt) Pizza Shop Demo - Part 6 ETL on Amazon Athena via Airflow

Thu, 14 Mar 2024 00:00:00 +0000

An ETL process is set up on the pizza shop dbt project using Apache Airflow. In Part 5, we developed a dbt project that that targets Apache Iceberg where transformations are performed on Amazon Athena. Two dimension tables that keep product and user records are created as Type 2 slowly changing dimension (SCD Type 2) tables, and one transactional fact table is built to keep pizza orders. To improve query performance, the fact table is denormalized to pre-join records from the dimension tables using the array and struct data types.

Data Build Tool (dbt) Pizza Shop Demo - Part 5 Modelling on Amazon Athena

Thu, 07 Mar 2024 00:00:00 +0000

The data is modelled by SCD type 2 dimension tables and one transactional fact table. In Part 1 and Part 3, we developed data build tool (dbt) projects that target PostgreSQL and BigQuery using fictional pizza shop data. While the order records should be joined with dimension tables to get complete details for PostgreSQL, the fact table is denormalized using nested and repeated fields to improve query performance for BigQuery.

Data Build Tool (dbt) Pizza Shop Demo - Part 4 ETL on BigQuery via Airflow

Thu, 22 Feb 2024 00:00:00 +0000

An ETL process is set up with Apache Airflow on a dbt project that targets Google BigQuery with fictional pizza shop data. We developed that project in Part 3. Two dimension tables that keep product and user records are created as Type 2 slowly changing dimension (SCD Type 2) tables, and one transactional fact table is built to keep pizza orders. The fact table is denormalized using nested and repeated fields for improving query performance.

Data Build Tool (dbt) Pizza Shop Demo - Part 3 Modelling on BigQuery

Thu, 08 Feb 2024 00:00:00 +0000

We create a new dbt project that targets Google BigQuery in this post. While the dimension tables are kept by the same SCD type 2 approach, the fact table is denormalized using nested and repeated fields, which potentially can improve query performance by pre-joining corresponding dimension records. In this series, we discuss practical examples of data warehouse and lakehouse development where data transformation is performed by the data build tool (dbt) and ETL is managed by Apache Airflow.

Data Build Tool (dbt) Pizza Shop Demo - Part 2 ETL on PostgreSQL via Airflow

Thu, 25 Jan 2024 00:00:00 +0000

We set up an ETL process on a dbt project using Apache Airflow in this post. In this series of posts, we discuss data warehouse/lakehouse examples using data build tool (dbt) including ETL orchestration with Apache Airflow. In Part 1, we developed a dbt project on PostgreSQL with fictional pizza shop data. Two dimension tables that keep product and user records are created as Type 2 slowly changing dimension (SCD Type 2) tables, and one transactional fact table is built to keep pizza orders.

Data Build Tool (dbt) Pizza Shop Demo - Part 1 Modelling on PostgreSQL

Thu, 18 Jan 2024 00:00:00 +0000

The data build tool (dbt) is a popular data transformation tool for data warehouse development. Moreover, it can be used for data lakehouse development thanks to open table formats such as Apache Iceberg, Apache Hudi and Delta Lake. dbt supports key AWS analytics services and I wrote a series of posts that discuss how to utilise dbt with Redshift, Glue, EMR on EC2, EMR on EKS, and Athena. Those posts focus on platform integration, however, they do not show realistic ETL scenarios.

Kafka Development on Kubernetes - Part 3 Kafka Connect

Thu, 11 Jan 2024 00:00:00 +0000

Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka. In this post, we discuss how to set up a data ingestion pipeline using Kafka connectors. Fake customer and order data is ingested into Kafka topics using the MSK Data Generator. Also, we use the Confluent S3 sink connector to save the messages of the topics into a S3 bucket.

Kafka Development on Kubernetes - Part 2 Producer and Consumer

Thu, 04 Jan 2024 00:00:00 +0000

Apache Kafka has five core APIs, and we can develop applications to send/read streams of data to/from topics in a Kafka cluster using the producer and consumer APIs. While the main Kafka project maintains only the Java APIs, there are several open source projects that provide the Kafka client APIs in Python. In this post, we discuss how to develop Kafka client applications using the kafka-python package on Kubernetes. [UPDATE 2025-10-01]

Kafka Development on Kubernetes - Part 1 Cluster Setup

Thu, 21 Dec 2023 00:00:00 +0000

Apache Kafka is one of the key technologies for implementing data streaming architectures. Strimzi provides a way to run an Apache Kafka cluster and related resources on Kubernetes in various deployment configurations. In this series of posts, we will discuss how to create a Kafka cluster, to develop Kafka client applications in Python and to build a data pipeline using Kafka connectors on Kubernetes. Part 1 Cluster Setup (this post) Part 2 Producer and Consumer Part 3 Kafka Connect Setup Kafka Cluster The Kafka cluster is deployed using the Strimzi Operator on a Minikube cluster.

Consume data from Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 6

Thu, 14 Dec 2023 00:00:00 +0000

Amazon MSK can be configured as an event source of a Lambda function. Lambda internally polls for new messages from the event source and then synchronously invokes the target Lambda function. With this feature, we can develop a Kafka consumer application in serverless environment where developers can focus on application logic. In this lab, we will discuss how to create a Kafka consumer using a Lambda function. Introduction Lab 1 Produce data to Kafka using Lambda Lab 2 Write data to Kafka from S3 using Flink Lab 3 Transform and write data to S3 from Kafka using Flink Lab 4 Clean, Aggregate, and Enrich Events with Flink Lab 5 Write data to DynamoDB using Kafka Connect Lab 6 Consume data from Kafka using Lambda (this post) Architecture Fake taxi ride data is sent to a Kafka topic by the Kafka producer application that is discussed in Lab 1.

Clean, Aggregate, and Enrich Events with Flink - Real Time Streaming with Kafka and Flink Lab 4

Thu, 23 Nov 2023 00:00:00 +0000

The value of data can be maximised when it is used without delay. With Apache Flink, we can build streaming analytics applications that incorporate the latest events with low latency. In this lab, we will create a Pyflink application that writes accumulated taxi rides data into an OpenSearch cluster. It aggregates the number of trips/passengers and trip durations by vendor ID for a window of 5 seconds. The data is then used to create a chart that monitors the status of taxi rides in the OpenSearch Dashboard.

Transform and write data to S3 from Kafka using Flink - Real Time Streaming with Kafka and Flink Lab 3

Thu, 16 Nov 2023 00:00:00 +0000

In this lab, we will create a Pyflink application that exports Kafka topic messages into a S3 bucket. The app enriches the records by adding a new column using a user defined function and writes them via the FileSystem SQL connector. This allows us to achieve a simpler architecture compared to the original lab where the records are sent into Amazon Kinesis Data Firehose, enriched by a separate Lambda function and written to a S3 bucket afterwards.

Produce data to Kafka using Lambda - Real Time Streaming with Kafka and Flink Lab 1

Thu, 26 Oct 2023 00:00:00 +0000

In this lab, we will create a Kafka producer application using AWS Lambda, which sends fake taxi ride data into a Kafka topic on Amazon MSK. A configurable number of the producer Lambda function will be invoked by an Amazon EventBridge schedule rule. In this way we are able to generate test data concurrently based on the desired volume of messages. Introduction Lab 1 Produce data to Kafka using Lambda (this post) Lab 2 Write data to Kafka from S3 using Flink Lab 3 Transform and write data to S3 from Kafka using Flink Lab 4 Clean, Aggregate, and Enrich Events with Flink Lab 5 Write data to DynamoDB using Kafka Connect Lab 6 Consume data from Kafka using Lambda [Update 2023-11-06] Initially I planned to deploy Pyflink applications on Amazon Managed Service for Apache Flink, but I changed the plan to use a local Flink cluster deployed on Docker.

Building Apache Flink Applications in Python

Thu, 19 Oct 2023 00:00:00 +0000

Building Apache Flink Applications in Java is a course to introduce Apache Flink through a series of hands-on exercises, and it is provided by Confluent. Utilising the Flink DataStream API, the course develops three Flink applications that populate multiple source data sets, collect them into a standardised data set, and aggregate it to produce usage statistics. As part of learning the Flink DataStream API in Pyflink, I converted the Java apps into Python equivalent while performing the course exercises in Pyflink.

Deployment via AWS Managed Flink - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 2

Thu, 14 Sep 2023 00:00:00 +0000

This series aims to help those who are new to Apache Flink and Amazon Managed Service for Apache Flink by re-implementing a simple fraud detection application that is discussed in an AWS workshop titled AWS Kafka and DynamoDB for real time fraud detection. In part 1, I demonstrated how to develop the application locally, and the app will be deployed via Amazon Managed Service for Apache Flink in this post.

Getting Started with PyFlink on AWS - Part 3 AWS Managed Flink and MSK

Mon, 04 Sep 2023 00:00:00 +0000

Amazon Managed Service for Apache Flink is the easiest option to run Flink applications on AWS, and the Pyflink app is deployed via it in this post. In this series of posts, we discuss a Flink (Pyflink) application that reads/writes from/to Kafka topics. In the previous posts, I demonstrated a Pyflink app that targets a local Kafka cluster as well as a Kafka cluster on Amazon MSK. The app was executed in a virtual environment as well as in a local Flink cluster for improved monitoring.

Getting Started with PyFlink on AWS - Part 2 Local Flink and MSK

Mon, 28 Aug 2023 00:00:00 +0000

A Kafka cluster on Amazon MSK is authenticated by IAM, and the app needs an additional jar dependency for it. As Amazon Managed Service for Apache Flink does not allow you to specify multiple pipeline jar files, we have to build a custom Uber Jar that combines multiple jar files. In this series of posts, we discuss a Flink (Pyflink) application that reads/writes from/to Kafka topics. In part 1, an app that targets a local Kafka cluster was created.

Getting Started with PyFlink on AWS - Part 1 Local Flink and Local Kafka

Thu, 17 Aug 2023 00:00:00 +0000

Apache Flink is an open-source, unified stream-processing and batch-processing framework. Its core is a distributed streaming data-flow engine that you can use to run real-time stream processing on high-throughput data sources. Currently, it is widely used to build applications for fraud/anomaly detection, rule-based alerting, business process monitoring, and continuous ETL to name a few. On AWS, we can deploy a Flink application via Amazon Kinesis Data Analytics (KDA), Amazon EMR and Amazon EKS.

Local Development - Kafka, Flink and DynamoDB for Real Time Fraud Detection Part 1

Thu, 10 Aug 2023 00:00:00 +0000

Kafka Development with Docker - Part 11 Kafka Authorization

Thu, 20 Jul 2023 00:00:00 +0000

In the previous posts, we discussed how to implement client authentication by TLS (SSL or TLS/SSL) and SASL authentication. One of the key benefits of client authentication is achieving user access control. Kafka ships with a pluggable, out-of-the box authorization framework, which is configured with the authorizer.class.name property in the server configuration and stores Access Control Lists (ACLs) in the cluster metadata (either Zookeeper or the KRaft metadata log). In this post, we will discuss how to configure Kafka authorization with Java and Python client examples while SASL is kept for client authentication.

Kafka Development with Docker - Part 10 SASL Authentication

Thu, 13 Jul 2023 00:00:00 +0000

In the previous post, we discussed TLS (SSL or TLS/SSL) authentication to improve security. It enforces two-way verification where a client certificate is verified by Kafka brokers. Client authentication can also be enabled by Simple Authentication and Security Layer (SASL), and we will discuss how to implement SASL authentication with Java and Python client examples in this post. [UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository.

Kafka Development with Docker - Part 9 SSL Authentication

Thu, 06 Jul 2023 00:00:00 +0000

In the previous post, we discussed how to configure TLS (SSL or TLS/SSL) encryption with Java and Python client examples. SSL encryption is a one-way verification process where a server certificate is verified by a client via SSL Handshake. To improve security, we can add client authentication either by enforcing two-way verification where a client certificate is verified by Kafka brokers (SSL authentication). Or we can choose a separate authentication mechanism, which is typically Simple Authentication and Security Layer (SASL).

Kafka Development with Docker - Part 8 SSL Encryption

Thu, 29 Jun 2023 00:00:00 +0000

By default, Apache Kafka communicates in PLAINTEXT, which means that all data is sent without being encrypted. To secure communication, we can configure Kafka clients and other components to use Transport Layer Security (TLS) encryption. Note that TLS is also referred to Secure Sockets Layer (SSL) or TLS/SSL. SSL is the predecessor of TLS, and has been deprecated since June 2015. However, it is used in configuration and code instead of TLS for historical reasons.

Producer and Consumer with Glue Schema Registry - Kafka Development with Docker Part 7

Thu, 22 Jun 2023 00:00:00 +0000

In Part 4, we developed Kafka producer and consumer applications using the kafka-python package. The Kafka messages are serialized as Json, but are not associated with a schema as there was not an integrated schema registry. Later we discussed how producers and consumers to Kafka topics can use schemas to ensure data consistency and compatibility as schemas evolve in Part 5. In this post, I’ll demonstrate how to enhance the existing applications by integrating AWS Glue Schema Registry.

Kafka Development with Docker - Part 4 Producer and Consumer

Thu, 01 Jun 2023 00:00:00 +0000

In the previous post, we discussed Kafka Connect to stream data to/from a Kafka cluster. Kafka also includes the Producer/Consumer APIs that allow client applications to send/read streams of data to/from topics in a Kafka cluster. While the main Kafka project maintains only the Java clients, there are several open source projects that provide the Kafka client APIs in Python. In this post, I’ll demonstrate how to develop producer/consumer applications using the kafka-python package.

Integrate Glue Schema Registry with Your Python Kafka App

Wed, 12 Apr 2023 00:00:00 +0000

As Kafka producer and consumer apps are decoupled, they operate on Kafka topics rather than communicating with each other directly. As described in the Confluent document, Schema Registry provides a centralized repository for managing and validating schemas for topic message data, and for serialization and deserialization of the data over the network. Producers and consumers to Kafka topics can use schemas to ensure data consistency and compatibility as schemas evolve. In AWS, the Glue Schema Registry supports features to manage and enforce schemas on data streaming applications using convenient integrations with Apache Kafka, Amazon Managed Streaming for Apache Kafka, Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics for Apache Flink, and AWS Lambda.

Simplify Streaming Ingestion on AWS - Part 2 MSK and Athena

Tue, 14 Mar 2023 00:00:00 +0000

Athena provides the MSK connector to enable SQL queries on Apache Kafka topics directly, and it can also facilitate the extraction of insights without setting up an additional pipeline to store data into S3. In this post, we discuss how to update the streaming ingestion solution so that data in the Kafka topic can be queried by Athena instead of Redshift. In Part 1, we discussed a streaming ingestion solution using EventBridge, Lambda, MSK and Redshift Serverless.

Simplify Streaming Ingestion on AWS - Part 1 MSK and Redshift

Wed, 08 Feb 2023 00:00:00 +0000

Apache Kafka is a popular distributed event store and stream processing platform. Previously loading data from Kafka into Redshift and Athena usually required Kafka connectors (e.g. Amazon Redshift Sink Connector and Amazon S3 Sink Connector). Recently these AWS services provide features to ingest data from Kafka directly, which facilitates a simpler architecture that achieves low-latency and high-speed ingestion of streaming data. In part 1 of the simplify streaming ingestion on AWS series, we discuss how to develop an end-to-end streaming ingestion solution using EventBridge, Lambda, MSK and Redshift Serverless on AWS.

How to configure Kafka consumers to seek offsets by timestamp

Tue, 10 Jan 2023 00:00:00 +0000

Normally we consume Kafka messages from the beginning/end of a topic or last committed offsets. For backfilling or troubleshooting, however, we need to consume messages from a certain timestamp occasionally. If we know which topic partition to choose e.g. by assigning a topic partition, we can easily override the fetch offset to a specific timestamp. When we deploy multiple consumer instances together, however, we make them subscribe to a topic and topic partitions are dynamically assigned, which means we cannot determine which fetch offset to use for an instance.

Revisit AWS Lambda Invoke Function Operator of Apache Airflow

Sat, 06 Aug 2022 00:00:00 +0000

Apache Airflow is a popular workflow management platform. A wide range of AWS services are integrated with the platform by Amazon AWS Operators. AWS Lambda is one of the integrated services, and it can be used to develop workflows efficiently. The current Lambda Operator, however, just invokes a Lambda function, and it can fail to report the invocation result of a function correctly and to record the exact error message from failure.

Serverless Application Model (SAM) for Data Professionals

Mon, 18 Jul 2022 00:00:00 +0000

AWS Lambda provides serverless computing capabilities, and it can be used for performing validation or light processing/transformation of data. Moreover, with its integration with more than 140 AWS services, it facilitates building complex systems employing event-driven architectures. There are many ways to build serverless applications and one of the most efficient ways is using specialised frameworks such as the AWS Serverless Application Model (SAM) and Serverless Framework. In this post, I’ll demonstrate how to build a serverless data processing application using SAM.

Data Warehousing ETL Demo with Apache Iceberg on EMR Local Environment

Sun, 26 Jun 2022 00:00:00 +0000

Unlike traditional Data Lake, new table formats (Iceberg, Hudi and Delta Lake) support features that can be used to apply data warehousing patterns, which can bring a way to be rescued from Data Swamp. In this post, we’ll discuss how to implement ETL using retail analytics data. It has two dimension data (user and product) and a single fact data (order). The dimension data sets have different ETL strategies depending on whether to track historical changes.

Local Development of AWS Glue 3.0 and Later

Sun, 14 Nov 2021 00:00:00 +0000

In an earlier post, I demonstrated how to set up a local development environment for AWS Glue 1.0 and 2.0 using a docker image that is published by the AWS Glue team and the Visual Studio Code Remote – Containers extension. Recently AWS Glue 3.0 was released, but a docker image for this version is not published. In this post, I’ll illustrate how to create a development environment for AWS Glue 3.

AWS Glue Local Development with Docker and Visual Studio Code

Fri, 20 Aug 2021 00:00:00 +0000

As described in the product page, AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. For development, a development endpoint is recommended, but it can be costly, inconvenient or unavailable (for Glue 2.0). The AWS Glue team published a Docker image that includes the AWS Glue binaries and all the dependencies packaged together. After inspecting it, I find some modifications are necessary in order to build a development environment on it.

Thoughts on Apache Airflow AWS Lambda Operator

Mon, 13 Apr 2020 00:00:00 +0000

Apache Airflow is a popular open-source workflow management platform. Typically tasks run remotely by Celery workers for scalability. In AWS, however, scalability can also be achieved using serverless computing services in a simpler way. For example, the ECS Operator allows to run dockerized tasks and, with the Fargate launch type, they can run in a serverless environment. The ECS Operator alone is not sufficient because it can take up to several minutes to pull a Docker image and to set up network interface (for the case of Fargate launch type).

Dynamic Routing and Centralized Auth with Traefik, Python and R Example

Fri, 29 Nov 2019 00:00:00 +0000

Ingress in Kubernetes exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. By setting rules, it routes requests to appropriate services (precisely requests are sent to individual Pods by Ingress Controller). Rules can be set up dynamically and I find it’s more efficient compared to traditional reverse proxy. Traefik is a modern HTTP reverse proxy and load balancer and it can be used as a Kubernetes Ingress Controller.

Distributed Task Queue with Python and R Example

Fri, 15 Nov 2019 00:00:00 +0000

While I’m looking into Apache Airflow, a workflow management tool, I thought it would be beneficial to get some understanding of how Celery works. To do so, I built a simple web service that sends tasks to Celery workers and collects the results from them. FastAPI is used for developing the web service and Redis is used for the message broker and result backend. During the development, I thought it would be possible to implement similar functionality in R with Rserve.

Linux Dev Environment on Windows

Fri, 01 Nov 2019 00:00:00 +0000

I use Linux containers a lot for development. Having Windows computers at home and work, I used to use Linux VMs on VirtualBox or VMWare Workstation. It’s not a bad option but it requires a lot of resources. Recently, after my home computer was updated, I was not able to start my hypervisor anymore. Also I didn’t like huge resource consumption of it so that I began to look for a different development environment.

AWS Local Development with LocalStack

Sat, 20 Jul 2019 00:00:00 +0000

LocalStack provides an easy-to-use test/mocking framework for developing AWS applications. In this post, I’ll demonstrate how to utilize LocalStack for development using a web service. Status, September 2026. The Lambda function below is created with the python3.6 runtime, which AWS stopped supporting, so the deployment commands fail as written. Use a currently supported Python runtime and a current LocalStack release, whose service endpoints also differ from the per-service ports used here.

Serverless Data Product POC Backend Part 4 - Serving R ML Model via S3

Mon, 17 Apr 2017 00:00:00 +0000

While the API returns a predicted admission status value given GRE, GPA and Rank, there is an issue if it is served within a web application: Cross-Origin Resource Sharing (CORS). This post discusses how to resolve this issue by updating API configuration and the Lambda function handler with a simple web application. Also it is illustrated how to host the application in a serverless environment. In the previous posts, it is discussed how to package/deploy an R model with AWS Lambda and to expose the Lambda function via Amazon API Gateway.

Serverless Data Product POC Backend Part 3 - Exposing R ML Model via APIG

Thu, 13 Apr 2017 00:00:00 +0000

In Part I of this series, R and necessary libraries/packages together with a Lambda function handler are packaged and saved to Amazon S3. Then, in Part II, the package is deployed at AWS Lambda after creating and assigning a role to the Lambda function. Although the Lambda function can be called via the Invoke API, it’ll be much more useful if the function can be called as a web service (or API).

Serverless Data Product POC Backend Part 2 - Deploying R ML Model via Lambda

Tue, 11 Apr 2017 00:00:00 +0000

No need of provisioning/managing servers is one of the key benefits of serverless event-driven application development. It is also a cost-effective way of delivering a data product as functions are executed on-demand rather than in servers that run 24/7. In the previous post, that kind of application development is introduced, and how to package R, necessary libraries/packages and a Lambda function handler is discussed. Furthermore AWS Lambda free tier includes 1M free requests per month and 400,000 GB-seconds of compute time per month, which is available to both existing and new AWS customers indefinitely.

Serverless Data Product POC Backend Part 1 - Packaging R ML Model for Lambda

Sat, 08 Apr 2017 00:00:00 +0000

Let say you’ve got a prediction model built in R and you’d like to productionize it, for example, by serving it in a web application. One way is exposing the model through an API that returns the predicted result as a web service. However there are many issues. Firstly R is not a language for API development although there may be some ways - eg the plumber package. More importantly developing an API is not the end of the story as the API can’t be served in a production system if it is not deployed/managed/upgraded/patched/… appropriately in a server or if it is not scalable, protected via authentication/authorization and so on.

Quick Test to Wrap Python in R

Sat, 21 Nov 2015 00:00:00 +0000

In relation to s3, although there are a number of existing packages, many of them seem to be deprecated, premature or platform-dependent. As mentioned in an earlier post, things that are not easy in R can be relatively simple in other languages, and connecting to Amazon Web Services is another example. (I consider the cloudyr project looks promising though.) If there isn’t a comprehensive R-way of doing something yet, it may be necessary to create it from scratch.

Some Thoughts on Python for R Users

Sun, 09 Aug 2015 00:00:00 +0000

There seem to be growing interest in Python in the R cummunity. While there can be a range of opinions about using R over Python (or vice versa) for exploratory data analysis, fitting statistical/machine learning algorithms and so on, I consider one of the strongest attractions of using Python comes from the fact that Python is a general purpose programming language. As more developers are involved in, it can provide a way to get jobs done easily, which can be tricky in R.

Some Thoughts on Python

Sat, 08 Aug 2015 00:00:00 +0000

When I bagan to teach myself C# 3 years ago, I only had some experience in interactive analysis tools such as MATLAB and R - I didn’t consider R as a programming language at that time. The general purpose programming language shares some common features (data type, loop, if…) but it is rather different in the way how code is written/organized, which is object oriented. Therefore, while it was not a problem to grap the common features, it took quite some time to understand and keep my code in an object oriented way.