A streaming Apache Beam pipeline in Python that aggregates page visits by user in fixed 20 second windows, written with and without Beam SQL.

The Apache Beam portability layer explained, with bash scripts that manage local Flink and Kafka clusters and a streaming pipeline on the Flink Runner.

A local Flink and Spark environment built from EMR container images, where Flink ingests data in real time and Spark queries it via the Glue Data Catalog.

Ingest records from a Kafka topic into a DynamoDB table using the Camel DynamoDB sink connector, defined and run on Kafka Connect.

Aggregate taxi ride counts and trip durations by vendor over five second windows with PyFlink, then chart them in an OpenSearch dashboard.

Export Kafka topic messages to S3 with PyFlink, enriching records through a user defined function and querying them on Athena through a Glue table.

Read records from S3 and send them into a Kafka topic with PyFlink, using a custom pipeline jar for IAM authentication, which is useful for enrichment.

Stateful stream processing set against traditional data infrastructure, showing which application patterns it improves and what opportunities it opens.

Three Flink DataStream applications from a Confluent Java course converted to PyFlink, from ingesting source data to calculating usage statistics.

Kafka and PyFlink replace Amazon Kinesis in a real time analytics app from an AWS workshop. The original and the new architecture are compared.