https://jaehyeon.me/series/kafka-development-with-docker/Recent content in Kafka Development With Docker on Jaehyeon KimHugo -- gohugo.ioenCopyright © 2023-2026 Jaehyeon Kim. All Rights Reserved.Thu, 20 Jul 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-07-20-kafka-development-with-docker-part-11/Thu, 20 Jul 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-07-20-kafka-development-with-docker-part-11/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 In the previous posts, we discussed how to implement client authentication by TLS (SSL or TLS/SSL) and SASL authentication. One of the key benefits of client authentication is achieving user access control. Kafka ships with a pluggable, out-of-the box authorization framework, which is configured with the authorizer.https://jaehyeon.me/blog/2023-07-13-kafka-development-with-docker-part-10/Thu, 13 Jul 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-07-13-kafka-development-with-docker-part-10/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 In the previous post, we discussed TLS (SSL or TLS/SSL) authentication to improve security. It enforces two-way verification where a client certificate is verified by Kafka brokers. Client authentication can also be enabled by Simple Authentication and Security Layer (SASL), and we will discuss how to implement SASL authentication with Java and Python client examples in this post.https://jaehyeon.me/blog/2023-07-06-kafka-development-with-docker-part-9/Thu, 06 Jul 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-07-06-kafka-development-with-docker-part-9/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 In the previous post, we discussed how to configure TLS (SSL or TLS/SSL) encryption with Java and Python client examples. SSL encryption is a one-way verification process where a server certificate is verified by a client via SSL Handshake.https://jaehyeon.me/blog/2023-06-29-kafka-development-with-docker-part-8/Thu, 29 Jun 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-06-29-kafka-development-with-docker-part-8/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 By default, Apache Kafka communicates in PLAINTEXT, which means that all data is sent without being encrypted. To secure communication, we can configure Kafka clients and other components to use Transport Layer Security (TLS) encryption.https://jaehyeon.me/blog/2023-06-22-kafka-development-with-docker-part-7/Thu, 22 Jun 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-06-22-kafka-development-with-docker-part-7/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 In Part 4, we developed Kafka producer and consumer applications using the kafka-python package. The Kafka messages are serialized as Json, but are not associated with a schema as there was not an integrated schema registry.https://jaehyeon.me/blog/2023-06-15-kafka-development-with-docker-part-6/Thu, 15 Jun 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-06-15-kafka-development-with-docker-part-6/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 In Part 3, we developed a data ingestion pipeline with fake online order data using Kafka Connect source and sink connectors. Schemas are not enabled on both of them as there was not an integrated schema registry.https://jaehyeon.me/blog/2023-06-08-kafka-development-with-docker-part-5/Thu, 08 Jun 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-06-08-kafka-development-with-docker-part-5/As described in the Confluent document, Schema Registry provides a centralized repository for managing and validating schemas for topic message data, and for serialization and deserialization of the data over the network. Producers and consumers to Kafka topics can use schemas to ensure data consistency and compatibility as schemas evolve. In AWS, the Glue Schema Registry supports features to manage and enforce schemas on data streaming applications using convenient integrations with Apache Kafka, Amazon Managed Streaming for Apache Kafka, Amazon Kinesis Data Streams, Amazon Kinesis Data Analytics for Apache Flink, and AWS Lambda.https://jaehyeon.me/blog/2023-06-01-kafka-development-with-docker-part-4/Thu, 01 Jun 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-06-01-kafka-development-with-docker-part-4/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 In the previous post, we discussed Kafka Connect to stream data to/from a Kafka cluster. Kafka also includes the Producer/Consumer APIs that allow client applications to send/read streams of data to/from topics in a Kafka cluster.https://jaehyeon.me/blog/2023-05-25-kafka-development-with-docker-part-3/Thu, 25 May 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-05-25-kafka-development-with-docker-part-3/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 According to the documentation of Apache Kafka, Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It makes it simple to quickly define connectors that move large collections of data into and out of Kafka.https://jaehyeon.me/blog/2023-05-18-kafka-development-with-docker-part-2/Thu, 18 May 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-05-18-kafka-development-with-docker-part-2/In the previous post, I illustrated how to create a topic and to produce/consume messages using the command utilities provided by Apache Kafka. It is not convenient, however, for example, when you consume serialised messages where their schemas are stored in a schema registry. Also, the utilities don’t support to browse or manage related resources such as connectors and schemas. Therefore, a Kafka management app can be a good companion for development, which helps monitor and manage resources on an easy-to-use user interface.https://jaehyeon.me/blog/2023-05-04-kafka-development-with-docker-part-1/Thu, 04 May 2023 00:00:00 +0000https://jaehyeon.me/blog/2023-05-04-kafka-development-with-docker-part-1/[UPDATE 2025-10-01] Bitnami’s public Docker images have been moved to the Bitnami Legacy repository. To ensure continued access and compatibility, please update your Docker image references accordingly. For example: bitnami/kafka:2.8.1 → bitnamilegacy/kafka:2.8.1 bitnami/zookeeper:3.7.0 → bitnamilegacy/zookeeper:3.7.0 bitnami/python:3.9.0 → bitnamilegacy/python:3.9.0 I’m teaching myself modern data streaming architectures on AWS, and Apache Kafka is one of the key technologies, which can be used for messaging, activity tracking, stream processing and so on. While applications tend to be deployed to cloud, it can be much easier if we develop and test those with Docker and Docker Compose locally.