Development on Jaehyeon Kim

Guide to Building Integrated Web Applications with FastAPI and NiceGUI

Wed, 19 Nov 2025 00:00:00 +0000

The standard architecture for modern web applications involves a decoupled frontend, typically built with a JavaScript framework, and a backend API. This pattern is powerful but introduces complexity in managing two separate codebases, development environments, and the API contract between them.

This article explores an alternative approach: an integrated architecture where the backend API and the frontend UI are served from a single, cohesive Python application.

Realtime Dashboard with FastAPI, Streamlit and Next.js - Part 3 Next.js Dashboard

Tue, 04 Mar 2025 00:00:00 +0000

In this post, we build a real-time monitoring dashboard using Next.js, a React framework that supports server-side rendering, static site generation, and full-stack capabilities with built-in performance optimizations. Similar to the Streamlit app we developed in Part 2, this dashboard connects to the WebSocket server from Part 1 to continuously fetch and visualize key metrics such as order counts, sales data, and revenue by traffic source and country. With interactive bar charts and dynamic metrics, users can monitor sales trends and other critical business KPIs in real-time.

Realtime Dashboard with FastAPI, Streamlit and Next.js - Part 2 Streamlit Dashboard

Tue, 25 Feb 2025 00:00:00 +0000

In this post, we develop a real-time monitoring dashboard using Streamlit, an open-source Python framework that allows data scientists and AI/ML engineers to create interactive data apps. The app connects to the WebSocket server we developed in Part 1 and continuously fetches data to visualize key metrics such as order counts, sales data, and revenue by traffic source and country. With interactive bar charts and dynamic metrics, users can monitor sales trends and other important business KPIs in real-time.

Realtime Dashboard with FastAPI, Streamlit and Next.js - Part 1 Data Producer

Tue, 18 Feb 2025 00:00:00 +0000

In this series, we develop real-time monitoring dashboard applications. A data generating app is created with Python, and it ingests the theLook eCommerce data continuously into a PostgreSQL database. A WebSocket server, built by FastAPI, periodically queries the data to serve its clients. The monitoring dashboards will be developed using Streamlit and Next.js, with Apache ECharts for visualization. In this post, we walk through the data generation app and backend API, while the monitoring dashboards will be discussed in later posts.

Setup Local Development Environment for Apache Flink and Spark Using EMR Container Images

Thu, 07 Dec 2023 00:00:00 +0000

[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 Apache Flink became generally available for Amazon EMR on EKS from the EMR 6.15.0 releases, and we are able to pull the Flink (as well as Spark) container images from the ECR Public Gallery.

Serverless Application Model (SAM) for Data Professionals

Mon, 18 Jul 2022 00:00:00 +0000

AWS Lambda provides serverless computing capabilities, and it can be used for performing validation or light processing/transformation of data. Moreover, with its integration with more than 140 AWS services, it facilitates building complex systems employing event-driven architectures. There are many ways to build serverless applications and one of the most efficient ways is using specialised frameworks such as the AWS Serverless Application Model (SAM) and Serverless Framework. In this post, I’ll demonstrate how to build a serverless data processing application using SAM.

Simplify Your Development on AWS with Terraform

Sun, 06 Feb 2022 00:00:00 +0000

When I wrote my data lake demo series (part 1, part 2 and part 3) recently, I used an Aurora PostgreSQL, MSK and EMR cluster. All of them were deployed to private subnets and dedicated infrastructure was created using CloudFormation. Using the infrastructure as code (IaC) tool helped a lot, but it resulted in creating 7 CloudFormation stacks, which was a bit harder to manage in the end. Then I looked into how to simplify building infrastructure and managing resources on AWS and decided to use Terraform instead.

Yet another serverless solution for invoking AWS Lambda at a sub-minute frequency

Wed, 13 Oct 2021 00:00:00 +0000

Triggering a Lambda function by an EventBridge Events rule can be used as a _serverless _replacement of cron job. The highest frequency of it is one invocation per minute so that it cannot be used directly if you need to schedule a Lambda function more frequently. For example, it may be refreshing an application with real time metrics from an Amazon Connect instance where some metrics are updated every 15 seconds.

Adding Authorization to a Graphql API

Tue, 20 Jul 2021 00:00:00 +0000

Authorization is the mechanism that controls who can do what on which resource in an application. Although it is a critical part of an application, there are limited resources available on how to build authorization into an app effectively. In this post, I’ll be illustrating how to set up authorization in a GraphQL API using a custom directive and Oso, an open-source authorization library. This tutorial covers the NodeJS variant of Oso, but it also supports Python and other languages.

Dynamic Routing and Centralized Auth with Traefik, Python and R Example

Fri, 29 Nov 2019 00:00:00 +0000

Ingress in Kubernetes exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. By setting rules, it routes requests to appropriate services (precisely requests are sent to individual Pods by Ingress Controller). Rules can be set up dynamically and I find it’s more efficient compared to traditional reverse proxy. Traefik is a modern HTTP reverse proxy and load balancer and it can be used as a Kubernetes Ingress Controller.

Distributed Task Queue with Python and R Example

Fri, 15 Nov 2019 00:00:00 +0000

While I’m looking into Apache Airflow, a workflow management tool, I thought it would be beneficial to get some understanding of how Celery works. To do so, I built a simple web service that sends tasks to Celery workers and collects the results from them. FastAPI is used for developing the web service and Redis is used for the message broker and result backend. During the development, I thought it would be possible to implement similar functionality in R with Rserve.

Linux Dev Environment on Windows

Fri, 01 Nov 2019 00:00:00 +0000

I use Linux containers a lot for development. Having Windows computers at home and work, I used to use Linux VMs on VirtualBox or VMWare Workstation. It’s not a bad option but it requires a lot of resources. Recently, after my home computer was updated, I was not able to start my hypervisor anymore. Also I didn’t like huge resource consumption of it so that I began to look for a different development environment.

AWS Local Development with LocalStack

Sat, 20 Jul 2019 00:00:00 +0000

LocalStack provides an easy-to-use test/mocking framework for developing AWS applications. In this post, I’ll demonstrate how to utilize LocalStack for development using a web service. Specifically a simple web service built with Flask-RestPlus is used. It supports simple CRUD operations against a database table. It is set that SQS and Lambda are used for creating and updating a record. When a POST or PUT request is made, the service sends a message to a SQS queue and directly returns 204 reponse.

Cronicle Multi Server Setup

Fri, 19 Jul 2019 00:00:00 +0000

Accroding to the project GitHub repository, Cronicle is a multi-server task scheduler and runner, with a web based front-end UI. It handles both scheduled, repeating and on-demand jobs, targeting any number of slave servers, with real-time stats and live log viewer. By default, Cronicle is configured to launch a single master server - task scheduling is controlled by the master server. For high availability, it is important that another server takes the role of master when the existing master server fails.

Shiny to Vue.js

Sat, 26 May 2018 00:00:00 +0000

In the last post, the async feature of Shiny was discussed. Although it is a remarkable step forward to web development in R, it is not to the full extent that a Javascript application can bring. In fact, (long running) requests of a user (or session) are not impacted by those of other users (or sessions) but, for a given user, all requests are handled sequentially. On the other hand, it is not the case for a Javascript-backed app where all requests are processed asynchronously.

Async Shiny and Its Limitation

Sat, 19 May 2018 00:00:00 +0000

A Shiny app is served by one (single-threaded blocking) process by Open Source Shiny Server. This causes a scalability issue because all requests are handled one by one in a queue. Recently the creator of Shiny introduced the promises package, which brings asynchronous programming capabilities to R. This is a remarkable step forward to web development in R. In this post, it’ll be demonstrated how to implement the async feature of Shiny.

API Development with R Part II

Sun, 19 Nov 2017 00:00:00 +0000

In Part I, it is discussed how to serve an R function with plumber, Rserve and rApache. In this post, the APIs are deployed in a Docker container and, after showing example requests, their performance is compared. The rocker/r-ver:3.4 is used as the base image and each of the APIs is added to it. For simplicity, the APIs are served by Supervisor. For performance testing, Locust is used. The source of this post can be found in this GitHub repository.

API Development with R Part I

Sat, 18 Nov 2017 00:00:00 +0000

API is an effective way of distributing analysis outputs to external clients. When it comes to API development with R, however, there are not many choices. Probably development would be made with plumber, Rserve, rApache or OpenCPU if a client or bridge layer to R is not considered. This is 2 part series in relation to API development with R. In this post, serving an R function with plumber, Rserve and rApache is discussed.

Serverless Data Product POC Backend Part IV - Serving R ML Model via S3

Mon, 17 Apr 2017 00:00:00 +0000

In the previous posts, it is discussed how to package/deploy an R model with AWS Lambda and to expose the Lambda function via Amazon API Gateway. Main benefits of serverless architecture is cost-effectiveness and being hassle-free from provisioning/managing servers. While the API returns a predicted admission status value given GRE, GPA and Rank, there is an issue if it is served within a web application: Cross-Origin Resource Sharing (CORS). This post discusses how to resolve this issue by updating API configuration and the Lambda function handler with a simple web application.

Serverless Data Product POC Backend Part III - Exposing R ML Model via APIG

Thu, 13 Apr 2017 00:00:00 +0000

In Part I of this series, R and necessary libraries/packages together with a Lambda function handler are packaged and saved to Amazon S3. Then, in Part II, the package is deployed at AWS Lambda after creating and assigning a role to the Lambda function. Although the Lambda function can be called via the Invoke API, it’ll be much more useful if the function can be called as a web service (or API).

Serverless Data Product POC Backend Part II - Deploying R ML Model via Lambda

Tue, 11 Apr 2017 00:00:00 +0000

In the previous post, serverless event-driven application development is introduced. Also how to package R, necessary libraries/packages and a Lambda function handler is discussed. No need of provisioning/managing servers is one of the key benefits of the architecture. It is also a cost-effective way of delivering a data product as functions are executed on-demand rather than in servers that run 24/7. Furthermore AWS Lambda free tier includes 1M free requests per month and 400,000 GB-seconds of compute time per month, which is available to both existing and new AWS customers indefinitely.

Serverless Data Product POC Backend Part I - Packaging R ML Model for Lambda

Sat, 08 Apr 2017 00:00:00 +0000

Let say you’ve got a prediction model built in R and you’d like to productionize it, for example, by serving it in a web application. One way is exposing the model through an API that returns the predicted result as a web service. However there are many issues. Firstly R is not a language for API development although there may be some ways - eg the plumber package. More importantly developing an API is not the end of the story as the API can’t be served in a production system if it is not deployed/managed/upgraded/patched/… appropriately in a server or if it is not scalable, protected via authentication/authorization and so on.

Some Thoughts on Shiny Open Source - Render Multiple Pages

Mon, 27 Jun 2016 00:00:00 +0000

R Shiny applications are served as a single page application and it is not built to render multiple pages. There are benefits of rendering multiple pages such as code management and implement authentication. In this page, we discuss how to implement multi-page rendering in a Shiny app. As indicated above, Shiny is not designed to render multiple pages and, in general, the UI is rendered on the fly as defined in ui.

Some Thoughts on Shiny Open Source - Internal Load Balancing

Mon, 23 May 2016 00:00:00 +0000

Shiny is an interesting web framework that helps create a web application quickly. If it targets a large number of users, however, there are several limitations and it is so true when the open source version of Shiny is in use. It would be possible to tackle down some of the limitations with the enterprise version but it is not easy to see enough examples of Shiny applications in production environment.

Asynchronous Processing Using Job Queue

Thu, 12 May 2016 00:00:00 +0000

In this post, a way to overcome one of R’s limitations (lack of multi-threading) is discussed by job queuing using the jobqueue package - a generic asynchronous job queue implementation for R. See the package description below. The jobqueue package is meant to provide an easy-to-use interface that allows to queue computations for background evaluation while the calling R session remains responsive. It is based on a 1-node socket cluster from the parallel package.

Quick Test to Wrap Python in R

Sat, 21 Nov 2015 00:00:00 +0000

As mentioned in an earlier post, things that are not easy in R can be relatively simple in other languages. Another example would be connecting to Amazon Web Services. In relation to s3, although there are a number of existing packages, many of them seem to be deprecated, premature or platform-dependent. (I consider the cloudyr project looks promising though.) If there isn’t a comprehensive R-way of doing something yet, it may be necessary to create it from scratch.

Some Thoughts on Python for R Users

Sun, 09 Aug 2015 00:00:00 +0000

There seem to be growing interest in Python in the R cummunity. While there can be a range of opinions about using R over Python (or vice versa) for exploratory data analysis, fitting statistical/machine learning algorithms and so on, I consider one of the strongest attractions of using Python comes from the fact that Python is a general purpose programming language. As more developers are involved in, it can provide a way to get jobs done easily, which can be tricky in R.

Some Thoughts on Python

Sat, 08 Aug 2015 00:00:00 +0000

When I bagan to teach myself C# 3 years ago, I only had some experience in interactive analysis tools such as MATLAB and R - I didn’t consider R as a programming language at that time. The general purpose programming language shares some common features (data type, loop, if…) but it is rather different in the way how code is written/organized, which is object oriented. Therefore, while it was not a problem to grap the common features, it took quite some time to understand and keep my code in an object oriented way.