Jaehyeon Kim
Jaehyeon Kim

  • Blog
    • Archives

    • Series

      List of series.

    • Categories

      List of categories.

    • Tags

      List of tags.


/

  • Github Linkedin Paypal RSS

  • Font Size
  • Palette
  • Mode
  1. Home
  2. Tags
  3. Redis

Distributed Task Queue With Python and R Example

featured.png
November 15, 20198 min read Data ProductCeleryDockerDocker ComposeFastAPIKubernetesPythonRRedisRserve

In this post, I'll illustrate how a web service is created using FastAPI framework where tasks are sent to multiple workers. The workers are built with Celery and Rserve. Redis is used as a message broker/result backend for Celery and a key-value store for Rserve. Demos can be run in both Docker Compose and Kubernetes.

Read More
Profile
Jaehyeon Kim
Jaehyeon Kim
Consultant at Cevo 🇦🇺 ☁ AWS Community Builder 💡 Blogger ⚡ Stateful Stream Processing Enthusiast
Taxonomies
Apache Kafka 20 R 15 Data Product 11 Data Streaming 11 Data Engineering 8 Machine Learning 7 Data Build Tool (DBT) 5 Development 5 General 5 Apache Flink 3 Change Data Capture (CDC) 3 Engineering 3 Apache Airflow 2 Stateful Stream Processing 1
Docker 43 Apache Kafka 40 AWS 40 Docker Compose 38 Python 37 R 36 Amazon MSK 20 Apache Spark 17 Kafka Connect 16 AWS Lambda 14 Terraform 14 Apache Flink 12 Amazon EMR 11 Pyflink 11 Amazon MSK Connect 9 PySpark 6 Amazon DynamoDB 5 Data Build Tool (DBT) 5 Kubernetes 5 Rserve 5 Visual Studio Code 5 Amazon API Gateway 4 Amazon Athena 4 Amazon Managed Flink 4 Amazon Managed Service for Apache Flink 4 Amazon QuickSight 4 AWS Glue 4 OpenSearch 4 Security 4 Shiny 4 Amazon EKS 3 Amazon OpenSearch Service 3 Amazon S3 3 Apache Camel 3 Apache Hudi 3 AWS SAM 3 Change Data Capture 3 Data Lake 3 FastAPI 3 Glue Schema Registry 3 ALL 91
Kafka Development with Docker 11 Real Time Streaming with Kafka and Flink 6 Tree based methods in R 6 DBT for Effective Data Transformation on AWS 5 Kafka Connect for AWS Services Integration 5 Serverless Data Product 4 Data Lake Demo Using Change Data Capture 3 Getting Started with Pyflink on AWS 3 Parallel processing on single machine 3 API development with R 2 Integrate Schema Registry with MSK Connect 2 Kafka, Flink and DynamoDB for Real Time Fraud Detection 2 Simplify Streaming Ingestion on AWS 2
2023 37 2022 15 2021 7 2020 1 2019 5 2018 2 2017 6 2016 6 2015 15 2014 5
Posts
  • featured.png
    Setup Local Development Environment for Apache Flink and Spark Using EMR Container Images
    December 7, 2023
  • featured.png
    Building Apache Flink Applications in Python
    October 19, 2023
  • featured.png
    Real Time Streaming With Kafka and Flink - Introduction
    October 5, 2023
  • featured.png
    Getting Started With Pyflink on AWS - Part 1 Local Flink and Local Kafka
    August 17, 2023
  • featured.png
    Kafka, Flink and DynamoDB for Real Time Fraud Detection - Part 1 Local Development
    August 10, 2023
  • featured.png
    Kafka Development With Docker - Part 1 Cluster Setup
    May 4, 2023
  • featured.png
    Kafka Connect for AWS Services Integration - Part 1 Introduction
    May 3, 2023
  • featured.png
    Integrate Glue Schema Registry With Your Python Kafka App
    April 12, 2023
  • featured.png
    Simplify Streaming Ingestion on AWS – Part 1 MSK and Redshift
    February 8, 2023
  • featured.png
    Data Build Tool (Dbt) for Effective Data Transformation on AWS – Part 1 Redshift
    September 28, 2022
  • featured.png
    Setup Local Development Environment for Apache Flink and Spark Using EMR Container Images
    December 7, 2023
  • featured.png
    Real Time Streaming With Kafka and Flink - Lab 5 Write Data to DynamoDB Using Kafka Connect
    November 30, 2023
  • featured.png
    Real Time Streaming With Kafka and Flink - Lab 4 Clean, Aggregate, and Enrich Events With Flink
    November 23, 2023
  • featured.png
    Real Time Streaming With Kafka and Flink - Lab 3 Transform and Write Data to S3 From Kafka Using Flink
    November 16, 2023
  • featured.png
    Real Time Streaming With Kafka and Flink - Lab 2 Write Data to Kafka From S3 Using Flink
    November 9, 2023
  • featured.png
    Benefits and Opportunities of Stateful Stream Processing
    November 2, 2023
  • featured.png
    Kafka Connect for AWS Services Integration - Part 5 Deploy Aiven OpenSearch Sink Connector
    October 30, 2023
  • featured.png
    Real Time Streaming With Kafka and Flink - Lab 1 Produce Data to Kafka Using Lambda
    October 26, 2023
  • featured.png
    Kafka Connect for AWS Services Integration - Part 4 Develop Aiven OpenSearch Sink Connector
    October 23, 2023
  • featured.png
    Building Apache Flink Applications in Python
    October 19, 2023
Actions
Go back Reload Copy URL

Jaehyeon Kim

Consultant at Cevo 🇦🇺 ☁ AWS Community Builder 💡 Blogger ⚡ Stateful Stream Processing Enthusiast

Copyright © 2023-2023 Jaehyeon Kim. All Rights Reserved.