16 posts in total
- Apache Beam Python Examples - Part 10 Develop Streaming File Reader using Splittable DoFn
December 19, 2024
- Apache Beam Python Examples - Part 9 Develop Batch File Reader and PiSampler using Splittable DoFn
December 5, 2024
- Apache Beam Python Examples - Part 8 Enhance Sport Activity Tracker with Runner Motivation
November 21, 2024
- Apache Beam Python Examples - Part 7 Separate Droppable Data into Side Output
October 24, 2024
- Apache Beam Python Examples - Part 6 Call RPC Service in Batch with Defined Batch Size using Stateful DoFn
October 2, 2024
- Apache Beam Python Examples - Part 5 Call RPC Service in Batch using Stateless DoFn
September 18, 2024
- Cache Data on Apache Beam Pipelines Using a Shared Object
August 22, 2024
- Apache Beam Python Examples - Part 4 Call RPC Service for Data Augmentation
August 15, 2024
- Apache Beam Python Examples - Part 3 Build Sport Activity Tracker with/without SQL
August 1, 2024
- Apache Beam Python Examples - Part 2 Calculate Average Word Length with/without Fixed Look back
July 18, 2024
- Apache Beam Python Examples - Part 1 Calculate K Most Frequent Words and Max Word Length
July 4, 2024
- Apache Beam Local Development with Python - Part 5 Testing Pipelines
May 9, 2024
- Apache Beam Local Development with Python - Part 4 Streaming Pipelines
May 2, 2024
- Apache Beam Local Development with Python - Part 3 Flink Runner
April 18, 2024
- Apache Beam Local Development with Python - Part 2 Batch Pipelines
April 4, 2024
- Apache Beam Local Development with Python - Part 1 Pipeline, Notebook, SQL and DataFrame
March 28, 2024
14 posts in total
- Kafka Development on Kubernetes - Part 3 Kafka Connect
January 11, 2024
- Kafka Development on Kubernetes - Part 2 Producer and Consumer
January 4, 2024
- Kafka Development on Kubernetes - Part 1 Cluster Setup
December 21, 2023
- Kafka Development with Docker - Part 11 Kafka Authorization
July 20, 2023
- Kafka Development with Docker - Part 10 SASL Authentication
July 13, 2023
- Kafka Development with Docker - Part 9 SSL Authentication
July 6, 2023
- Kafka Development with Docker - Part 8 SSL Encryption
June 29, 2023
- Kafka Development with Docker - Part 7 Producer and Consumer with Glue Schema Registry
June 22, 2023
- Kafka Development with Docker - Part 6 Kafka Connect with Glue Schema Registry
June 15, 2023
- Kafka Development with Docker - Part 5 Glue Schema Registry
June 8, 2023
- Kafka Development with Docker - Part 4 Producer and Consumer
June 1, 2023
- Kafka Development with Docker - Part 3 Kafka Connect
May 25, 2023
- Kafka Development with Docker - Part 2 Management App
May 18, 2023
- Kafka Development with Docker - Part 1 Cluster Setup
May 4, 2023
26 posts in total
- Guide to Running DBT in Production
September 13, 2024
- DBT CI/CD Demo with BigQuery and GitHub Actions
September 5, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 6 ETL on Amazon Athena via Airflow
March 14, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 5 Modelling on Amazon Athena
March 7, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 4 ETL on BigQuery via Airflow
February 22, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 3 Modelling on BigQuery
February 8, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 2 ETL on PostgreSQL via Airflow
January 25, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 1 Modelling on PostgreSQL
January 18, 2024
- Setup Local Development Environment for Apache Flink and Spark Using EMR Container Images
December 7, 2023
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 5 Athena
December 6, 2022
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 4 EMR on EKS
November 1, 2022
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 3 EMR on EC2
October 19, 2022
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 2 Glue
October 9, 2022
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 1 Redshift
September 28, 2022
- Develop and Test Apache Spark Apps for EMR Remotely Using Visual Studio Code
September 7, 2022
- Manage EMR on EKS with Terraform
August 26, 2022
- Revisit AWS Lambda Invoke Function Operator of Apache Airflow
August 6, 2022
- Data Warehousing ETL Demo with Apache Iceberg on EMR Local Environment
June 26, 2022
- Develop and Test Apache Spark Apps for EMR Locally Using Docker
May 8, 2022
- EMR on EKS by Example
January 17, 2022
29 posts in total
- Run Flink SQL Cookbook in Docker
April 15, 2025
- Change Data Capture (CDC) Local Development with PostgreSQL, Debezium Server and Pub/Sub Emulator
November 7, 2024
- Deploy Python Stream Processing App on Kubernetes - Part 2 Beam Pipeline on Flink Runner
June 6, 2024
- Deploy Python Stream Processing App on Kubernetes - Part 1 PyFlink Application
May 30, 2024
- Real Time Streaming with Kafka and Flink - Lab 6 Consume data from Kafka using Lambda
December 14, 2023
- Real Time Streaming with Kafka and Flink - Lab 5 Write data to DynamoDB using Kafka Connect
November 30, 2023
- Real Time Streaming with Kafka and Flink - Lab 4 Clean, Aggregate, and Enrich Events with Flink
November 23, 2023
- Real Time Streaming with Kafka and Flink - Lab 3 Transform and write data to S3 from Kafka using Flink
November 16, 2023
- Real Time Streaming with Kafka and Flink - Lab 2 Write data to Kafka from S3 using Flink
November 9, 2023
- Benefits and Opportunities of Stateful Stream Processing
November 2, 2023
- Kafka Connect for AWS Services Integration - Part 5 Deploy Aiven OpenSearch Sink Connector
October 30, 2023
- Real Time Streaming with Kafka and Flink - Lab 1 Produce data to Kafka using Lambda
October 26, 2023
- Kafka Connect for AWS Services Integration - Part 4 Develop Aiven OpenSearch Sink Connector
October 23, 2023
- Building Apache Flink Applications in Python
October 19, 2023
- Real Time Streaming with Kafka and Flink - Introduction
October 5, 2023
- Kafka, Flink and DynamoDB for Real Time Fraud Detection - Part 2 Deployment via AWS Managed Flink
September 14, 2023
- Getting Started with Pyflink on AWS - Part 3 AWS Managed Flink and MSK
September 4, 2023
- Getting Started with Pyflink on AWS - Part 2 Local Flink and MSK
August 28, 2023
- Getting Started with Pyflink on AWS - Part 1 Local Flink and Local Kafka
August 17, 2023
- Kafka, Flink and DynamoDB for Real Time Fraud Detection - Part 1 Local Development
August 10, 2023
19 posts in total
- Serverless Application Model (SAM) for Data Professionals
July 18, 2022
- Simplify Your Development on AWS with Terraform
February 6, 2022
- Yet another serverless solution for invoking AWS Lambda at a sub-minute frequency
October 13, 2021
- Adding Authorization to a Graphql API
July 20, 2021
- Dynamic Routing and Centralized Auth with Traefik, Python and R Example
November 29, 2019
- Distributed Task Queue with Python and R Example
November 15, 2019
- Linux Dev Environment on Windows
November 1, 2019
- AWS Local Development with LocalStack
July 20, 2019
- Cronicle Multi Server Setup
July 19, 2019
- Shiny to Vue.js
May 26, 2018
- Async Shiny and Its Limitation
May 19, 2018
- API Development with R Part II
November 19, 2017
- API Development with R Part I
November 18, 2017
- Serverless Data Product POC Backend Part IV - Serving R ML Model via S3
April 17, 2017
- Serverless Data Product POC Backend Part III - Exposing R ML Model via APIG
April 13, 2017
- Serverless Data Product POC Backend Part II - Deploying R ML Model via Lambda
April 11, 2017
- Serverless Data Product POC Backend Part I - Packaging R ML Model for Lambda
April 8, 2017
- Some Thoughts on Shiny Open Source - Render Multiple Pages
June 27, 2016
- Some Thoughts on Shiny Open Source - Internal Load Balancing
May 23, 2016
15 posts in total
- Asynchronous Processing Using Job Queue
May 12, 2016
- Boost SparkR with Hive
April 30, 2016
- Quick Start SparkR in Local and Cluster Mode
March 2, 2016
- Spark Cluster Setup on VirtualBox
February 22, 2016
- Quick Test to Wrap Python in R
November 21, 2015
- Packaging Analysis
March 24, 2015
- Parallel Processing on Single Machine - Part III
March 19, 2015
- Parallel Processing on Single Machine - Part II
March 17, 2015
- Parallel Processing on Single Machine - Part I
March 14, 2015
- Quick Trial of Adding Column
January 14, 2015
- Looping without for
December 17, 2014
- Short R Examples
December 3, 2014
- Summarise Stock Returns from Multiple Files
November 27, 2014
- Download Stock Data - Part II
November 21, 2014
- Download Stock Data - Part I
November 20, 2014