17 posts in total
- Apache Beam Python Examples - Part 10 Develop Streaming File Reader using Splittable DoFn
December 19, 2024
- Apache Beam Python Examples - Part 9 Develop Batch File Reader and PiSampler using Splittable DoFn
December 5, 2024
- Apache Beam Python Examples - Part 8 Enhance Sport Activity Tracker with Runner Motivation
November 21, 2024
- Apache Beam Python Examples - Part 7 Separate Droppable Data into Side Output
October 24, 2024
- Apache Beam Python Examples - Part 6 Call RPC Service in Batch with Defined Batch Size using Stateful DoFn
October 2, 2024
- Apache Beam Python Examples - Part 5 Call RPC Service in Batch using Stateless DoFn
September 18, 2024
- Cache Data on Apache Beam Pipelines Using a Shared Object
August 22, 2024
- Apache Beam Python Examples - Part 4 Call RPC Service for Data Augmentation
August 15, 2024
- Apache Beam Python Examples - Part 3 Build Sport Activity Tracker with/without SQL
August 1, 2024
- Apache Beam Python Examples - Part 2 Calculate Average Word Length with/without Fixed Look back
July 18, 2024
- Apache Beam Python Examples - Part 1 Calculate K Most Frequent Words and Max Word Length
July 4, 2024
- Deploy Python Stream Processing App on Kubernetes - Part 2 Beam Pipeline on Flink Runner
June 6, 2024
- Apache Beam Local Development with Python - Part 5 Testing Pipelines
May 9, 2024
- Apache Beam Local Development with Python - Part 4 Streaming Pipelines
May 2, 2024
- Apache Beam Local Development with Python - Part 3 Flink Runner
April 18, 2024
- Apache Beam Local Development with Python - Part 2 Batch Pipelines
April 4, 2024
- Apache Beam Local Development with Python - Part 1 Pipeline, Notebook, SQL and DataFrame
March 28, 2024
13 posts in total
- Run Flink SQL Cookbook in Docker
April 15, 2025
- Deploy Python Stream Processing App on Kubernetes - Part 1 PyFlink Application
May 30, 2024
- Setup Local Development Environment for Apache Flink and Spark Using EMR Container Images
December 7, 2023
- Real Time Streaming with Kafka and Flink - Lab 4 Clean, Aggregate, and Enrich Events with Flink
November 23, 2023
- Real Time Streaming with Kafka and Flink - Lab 3 Transform and write data to S3 from Kafka using Flink
November 16, 2023
- Real Time Streaming with Kafka and Flink - Lab 2 Write data to Kafka from S3 using Flink
November 9, 2023
- Benefits and Opportunities of Stateful Stream Processing
November 2, 2023
- Building Apache Flink Applications in Python
October 19, 2023
- Real Time Streaming with Kafka and Flink - Introduction
October 5, 2023
- Kafka, Flink and DynamoDB for Real Time Fraud Detection - Part 2 Deployment via AWS Managed Flink
September 14, 2023
- Getting Started with Pyflink on AWS - Part 2 Local Flink and MSK
August 28, 2023
- Getting Started with Pyflink on AWS - Part 1 Local Flink and Local Kafka
August 17, 2023
- Kafka, Flink and DynamoDB for Real Time Fraud Detection - Part 1 Local Development
August 10, 2023
38 posts in total
- Kafka Development on Kubernetes - Part 3 Kafka Connect
January 11, 2024
- Kafka Development on Kubernetes - Part 2 Producer and Consumer
January 4, 2024
- Kafka Development on Kubernetes - Part 1 Cluster Setup
December 21, 2023
- Real Time Streaming with Kafka and Flink - Lab 6 Consume data from Kafka using Lambda
December 14, 2023
- Real Time Streaming with Kafka and Flink - Lab 5 Write data to DynamoDB using Kafka Connect
November 30, 2023
- Real Time Streaming with Kafka and Flink - Lab 3 Transform and write data to S3 from Kafka using Flink
November 16, 2023
- Real Time Streaming with Kafka and Flink - Lab 2 Write data to Kafka from S3 using Flink
November 9, 2023
- Kafka Connect for AWS Services Integration - Part 5 Deploy Aiven OpenSearch Sink Connector
October 30, 2023
- Real Time Streaming with Kafka and Flink - Lab 1 Produce data to Kafka using Lambda
October 26, 2023
- Kafka Connect for AWS Services Integration - Part 4 Develop Aiven OpenSearch Sink Connector
October 23, 2023
- Real Time Streaming with Kafka and Flink - Introduction
October 5, 2023
- Kafka, Flink and DynamoDB for Real Time Fraud Detection - Part 2 Deployment via AWS Managed Flink
September 14, 2023
- Getting Started with Pyflink on AWS - Part 2 Local Flink and MSK
August 28, 2023
- Getting Started with Pyflink on AWS - Part 1 Local Flink and Local Kafka
August 17, 2023
- Kafka, Flink and DynamoDB for Real Time Fraud Detection - Part 1 Local Development
August 10, 2023
- Kafka Development with Docker - Part 11 Kafka Authorization
July 20, 2023
- Kafka Development with Docker - Part 10 SASL Authentication
July 13, 2023
- Kafka Development with Docker - Part 9 SSL Authentication
July 6, 2023
- Kafka Connect for AWS Services Integration - Part 3 Deploy Camel DynamoDB Sink Connector
July 3, 2023
- Kafka Development with Docker - Part 8 SSL Encryption
June 29, 2023
13 posts in total
- Setup Local Development Environment for Apache Flink and Spark Using EMR Container Images
December 7, 2023
- Develop and Test Apache Spark Apps for EMR Remotely Using Visual Studio Code
September 7, 2022
- Manage EMR on EKS with Terraform
August 26, 2022
- Data Warehousing ETL Demo with Apache Iceberg on EMR Local Environment
June 26, 2022
- Develop and Test Apache Spark Apps for EMR Locally Using Docker
May 8, 2022
- Data Lake Demo using Change Data Capture (CDC) on AWS – Part 3 Implement Data Lake
December 19, 2021
- Data Lake Demo using Change Data Capture (CDC) on AWS – Part 2 Implement CDC
December 12, 2021
- Data Lake Demo using Change Data Capture (CDC) on AWS – Part 1 Local Development
December 5, 2021
- Local Development of AWS Glue 3.0 and Later
November 14, 2021
- AWS Glue Local Development with Docker and Visual Studio Code
August 20, 2021
- Boost SparkR with Hive
April 30, 2016
- Quick Start SparkR in Local and Cluster Mode
March 2, 2016
- Spark Cluster Setup on VirtualBox
February 22, 2016
26 posts in total
- Guide to Running DBT in Production
September 13, 2024
- DBT CI/CD Demo with BigQuery and GitHub Actions
September 5, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 6 ETL on Amazon Athena via Airflow
March 14, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 5 Modelling on Amazon Athena
March 7, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 4 ETL on BigQuery via Airflow
February 22, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 3 Modelling on BigQuery
February 8, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 2 ETL on PostgreSQL via Airflow
January 25, 2024
- Data Build Tool (dbt) Pizza Shop Demo - Part 1 Modelling on PostgreSQL
January 18, 2024
- Setup Local Development Environment for Apache Flink and Spark Using EMR Container Images
December 7, 2023
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 5 Athena
December 6, 2022
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 4 EMR on EKS
November 1, 2022
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 3 EMR on EC2
October 19, 2022
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 2 Glue
October 9, 2022
- Data Build Tool (dbt) for Effective Data Transformation on AWS – Part 1 Redshift
September 28, 2022
- Develop and Test Apache Spark Apps for EMR Remotely Using Visual Studio Code
September 7, 2022
- Manage EMR on EKS with Terraform
August 26, 2022
- Revisit AWS Lambda Invoke Function Operator of Apache Airflow
August 6, 2022
- Data Warehousing ETL Demo with Apache Iceberg on EMR Local Environment
June 26, 2022
- Develop and Test Apache Spark Apps for EMR Locally Using Docker
May 8, 2022
- EMR on EKS by Example
January 17, 2022
52 posts in total
- Meet the Streamhouse Trio - Paimon, Fluss, and Iceberg for Unified Data Architectures
May 6, 2025
- Apache Beam Python Examples - Part 10 Develop Streaming File Reader using Splittable DoFn
December 19, 2024
- Apache Beam Python Examples - Part 9 Develop Batch File Reader and PiSampler using Splittable DoFn
December 5, 2024
- Apache Beam Python Examples - Part 8 Enhance Sport Activity Tracker with Runner Motivation
November 21, 2024
- Change Data Capture (CDC) Local Development with PostgreSQL, Debezium Server and Pub/Sub Emulator
November 7, 2024
- Apache Beam Python Examples - Part 7 Separate Droppable Data into Side Output
October 24, 2024
- Apache Beam Python Examples - Part 6 Call RPC Service in Batch with Defined Batch Size using Stateful DoFn
October 2, 2024
- Apache Beam Python Examples - Part 5 Call RPC Service in Batch using Stateless DoFn
September 18, 2024
- Apache Beam Python Examples - Part 4 Call RPC Service for Data Augmentation
August 15, 2024
- Apache Beam Python Examples - Part 3 Build Sport Activity Tracker with/without SQL
August 1, 2024
- Apache Beam Python Examples - Part 2 Calculate Average Word Length with/without Fixed Look back
July 18, 2024
- Apache Beam Python Examples - Part 1 Calculate K Most Frequent Words and Max Word Length
July 4, 2024
- Deploy Python Stream Processing App on Kubernetes - Part 2 Beam Pipeline on Flink Runner
June 6, 2024
- Deploy Python Stream Processing App on Kubernetes - Part 1 PyFlink Application
May 30, 2024
- Kafka Development on Kubernetes - Part 3 Kafka Connect
January 11, 2024
- Kafka Development on Kubernetes - Part 2 Producer and Consumer
January 4, 2024
- Kafka Development on Kubernetes - Part 1 Cluster Setup
December 21, 2023
- Real Time Streaming with Kafka and Flink - Lab 6 Consume data from Kafka using Lambda
December 14, 2023
- Real Time Streaming with Kafka and Flink - Lab 5 Write data to DynamoDB using Kafka Connect
November 30, 2023
- Real Time Streaming with Kafka and Flink - Lab 4 Clean, Aggregate, and Enrich Events with Flink
November 23, 2023
22 posts in total
- Realtime Dashboard with FastAPI, Streamlit and Next.js - Part 3 Next.js Dashboard
March 4, 2025
- Realtime Dashboard with FastAPI, Streamlit and Next.js - Part 2 Streamlit Dashboard
February 25, 2025
- Realtime Dashboard with FastAPI, Streamlit and Next.js - Part 1 Data Producer
February 18, 2025
- Serverless Application Model (SAM) for Data Professionals
July 18, 2022
- Simplify Your Development on AWS with Terraform
February 6, 2022
- Yet another serverless solution for invoking AWS Lambda at a sub-minute frequency
October 13, 2021
- Adding Authorization to a Graphql API
July 20, 2021
- Dynamic Routing and Centralized Auth with Traefik, Python and R Example
November 29, 2019
- Distributed Task Queue with Python and R Example
November 15, 2019
- Linux Dev Environment on Windows
November 1, 2019
- AWS Local Development with LocalStack
July 20, 2019
- Cronicle Multi Server Setup
July 19, 2019
- Shiny to Vue.js
May 26, 2018
- Async Shiny and Its Limitation
May 19, 2018
- API Development with R Part II
November 19, 2017
- API Development with R Part I
November 18, 2017
- Serverless Data Product POC Backend Part IV - Serving R ML Model via S3
April 17, 2017
- Serverless Data Product POC Backend Part III - Exposing R ML Model via APIG
April 13, 2017
- Serverless Data Product POC Backend Part II - Deploying R ML Model via Lambda
April 11, 2017
- Serverless Data Product POC Backend Part I - Packaging R ML Model for Lambda
April 8, 2017