Send one Kafka topic as Avro, with the schema generated from a Pydantic model, while the other topics stay JSON.
KafkaEgress writes every record as JSON by default. A consumer that needs a compact, schema-checked format, such as a machine learning service reading predictions, can read Avro instead. Each topic can have its own serializer, so JSON and Avro topics share one producer.
Before You Start
Egress explains KafkaEgress, and Route Records to Topics explains __topic__, which this page uses to pick the topic.
The code publishes to Kafka and to a schema registry, so start both with odctl up kafka-lite. See Installation for the one-time odctl install. Stop it with odctl down kafka-lite --volumes.
The model needs dataclasses-avroschema with its Pydantic support: pip install "dataclasses-avroschema[pydantic]".
The serializer needs its extra: pip install "desflow[confluent]" for the Confluent Schema Registry, or pip install "desflow[glue]" for the AWS Glue Schema Registry. Both include kafka.
Steps
Generate the Schema from a Model
A schema written by hand as a JSON string stops matching the data when someone changes the model and not the schema. AvroBaseModel of dataclasses-avroschema is a Pydantic model that also writes its Avro schema. KafkaEgress serializes the whole record, not only its value: key, task, instance, sim_ts, timestamp and value, without stream_type. task and instance are null on an event published without them, so the schema makes both nullable. PredictionRecord is that record, with the prediction MLPrediction as its value:
fromtypingimportOptionalfromdataclasses_avroschema.pydanticimportAvroBaseModelfrompydanticimportFieldclassMLPrediction(AvroBaseModel):"""The prediction, published as the event value."""event_type:str="prediction"task_id:str=Field(...)confidence:float=Field(...)classPredictionRecord(AvroBaseModel):"""The record KafkaEgress serializes for an event."""key:strtask:Optional[str]=Noneinstance:Optional[int]=Nonesim_ts:floattimestamp:strvalue:MLPredictionclassMeta:namespace="com.desflow.ml"schema_name="PredictionEvent"schema_string=PredictionRecord.avro_schema()
A serializer takes the generated string as its schema. ConfluentAvroSerializer uses a Confluent Schema Registry, and GlueAvroSerializer uses an AWS Glue Schema Registry. A registry that needs a user name, a certificate or a proxy takes more arguments: see Connect to a Secured Schema Registry.
fromdesflow.connectors.egress.kafkaimportConfluentAvroSerializergenerated_serializer=ConfluentAvroSerializer(registry_url="http://127.0.0.1:8081",schema_str=PredictionRecord.avro_schema(),# follows the model)
A blueprint reads the schema from a file, so write the generated schema beside it:
topics gives the settings of each topic, and its serializer names the serializer of that topic. A topic with no serializer uses the default, JsonSerializer. The mapping of a blueprint has a type, here confluent_avro, the registry address and the schema. For the Glue registry the type is glue_avro and the registry is named by registry_name.
In this shop each packed order's finished event is a prediction, and it names the Avro topic with __topic__. The queued and started events name no topic, so they go to sim-events as JSON. A blueprint writes the fields of MLPrediction in its payload. In the code forms the model checks the fields and model_dump() gives the payload.
resources:packers:{current_cap:1, max_cap:1}services:packing:{dist:constant, value:1.5}arrivals:orders:{dist:exponential, rate:0.5, spawn:pack_order}tasks:pack_order:service:packingresource:packers# The finished event is a prediction for the order: the fields of MLPredictionpayload:__topic__:ml-predictionsevent_type:predictiontask_id:{gen:entity}confidence:0.9
app=(SimulationContext(sim_id="Shop",factor=0,random_seed=42).add_resource("packers",current_cap=1,max_cap=1).add_service("packing",dist="constant",value=1.5).add_arrival("orders",dist="exponential",rate=0.5).add_egress(egress)# predictions as Avro, the rest as JSON)@app.task(service_id="packing",resource_id="packers")defpack_order(task_id:int,entity):# The finished event is a prediction for the order: the fields of MLPredictionprediction=MLPrediction(task_id=entity.key,confidence=0.9)return{**prediction.model_dump(),"__topic__":"ml-predictions"}@app.arrival_loop("orders")deforders(context:SimulationContext):whileTrue:order=yieldcontext.wait_for_arrival("orders")context.spawn(pack_order(context.next_instance("pack_order"),entity=order))
defpack_order(env,packers,sampler,key,task_id):path_id="Shop.service.packing"env.publish_event(key,{"path_id":path_id,"sim_id":"Shop","service":"packing","status":"queued",},"pack_order",task_id,)withpackers.request()asunit:yieldunitenv.publish_event(key,{"path_id":path_id,"sim_id":"Shop","service":"packing","status":"started",},"pack_order",task_id,)yieldenv.timeout(sampler.sample(env.registry.get_config(path_id)))# The finished event is a prediction for the order: the fields of MLPredictionprediction=MLPrediction(task_id=key,confidence=0.9)payload={**prediction.model_dump(),"__topic__":"ml-predictions"}env.publish_event(key,payload,"pack_order",task_id)
deforders(env,packers,sampler):gap=env.registry.get_config("Shop.arrival.orders")fornumberinitertools.count(1):yieldenv.timeout(sampler.sample(gap))env.process(pack_order(env,packers,sampler,f"orders-{number}",number-1))defshop(sinks:list,factor:float=0.0)->DynamicRealtimeEnvironment:"""The shop, publishing to the sinks given."""env=DynamicRealtimeEnvironment(factor=factor)env.registry.register_sim_parameter(PARAMS)env.setup_egress(sinks)packers=DynamicResource(env,"Shop","packers")env.process(orders(env,packers,Sampler(seed=42)))returnenv
A task can also return the model itself, and publish_event turns a Pydantic V2 model into a dictionary before any sink sees it. Pydantic V1 models, including pydantic.v1, are not supported and raise a PydanticSerializationError. A model cannot hold __topic__, so choose the topic of such a record with a topic_router, as Route Records to Topics shows.
Check the Result
Run desflow run avro_and_pydantic.yaml, python avro_and_pydantic.py or python avro_and_pydantic_low_level.py while kafka-lite is up. Each runs for one simulated minute and ends at once. The serializer registers the generated schema with the registry of kafka-lite on its first send, under the subject ml-predictions-value, which curl http://127.0.0.1:8081/subjects lists. The run sends 20 predictions to ml-predictions, and every one fits the generated schema. The queued and started events go to sim-events as JSON.
To read a prediction back, run this script after a shop has run. It reads ml-predictions and decodes the first message with a ConfluentAvroDeserializer, which fetches the schema that the message names from the registry:
importasynciofromaiokafkaimportAIOKafkaConsumerfromdesflow.connectors.ingress.kafkaimportConfluentAvroDeserializerasyncdefshow_first_prediction()->None:"""Prints the first message of ml-predictions, decoded with the schema it names."""consumer=AIOKafkaConsumer("ml-predictions",bootstrap_servers="127.0.0.1:9092",auto_offset_reset="earliest",)awaitconsumer.start()try:message=awaitasyncio.wait_for(consumer.getone(),timeout=10)reader=ConfluentAvroDeserializer(registry_url="http://127.0.0.1:8081")print(reader.deserialize("ml-predictions",message.value))finally:awaitconsumer.stop()if__name__=="__main__":asyncio.run(show_first_prediction())
"""The Pydantic models and the Avro serializer of the Use Avro and Pydantic page.Every code form of the shop imports the models and the serializer from this file:avro_and_pydantic.py (builder) and avro_and_pydantic_low_level.py. It builds nosimulation. Install `dataclasses-avroschema[pydantic]` for the models.`python avro_and_pydantic_models.py` prints the Avro schema of the models, which is thefile prediction_event.avsc of the blueprint."""# Each section carries its own imports, as the page shows it.# ruff: noqa: E402importjsonfromtypingimportOptionalfromdataclasses_avroschema.pydanticimportAvroBaseModelfrompydanticimportFieldclassMLPrediction(AvroBaseModel):"""The prediction, published as the event value."""event_type:str="prediction"task_id:str=Field(...)confidence:float=Field(...)classPredictionRecord(AvroBaseModel):"""The record KafkaEgress serializes for an event."""key:strtask:Optional[str]=Noneinstance:Optional[int]=Nonesim_ts:floattimestamp:strvalue:MLPredictionclassMeta:namespace="com.desflow.ml"schema_name="PredictionEvent"schema_string=PredictionRecord.avro_schema()fromdesflow.connectors.egress.kafkaimportConfluentAvroSerializergenerated_serializer=ConfluentAvroSerializer(registry_url="http://127.0.0.1:8081",schema_str=PredictionRecord.avro_schema(),# follows the model)if__name__=="__main__":print(json.dumps(json.loads(schema_string),indent=2))
# The Use Avro and Pydantic guide: each packed order publishes a prediction, sent to# ml-predictions as Avro; every other record goes to its topic as JSON. Needs# prediction_event.avsc beside it, and a Kafka broker with a schema registry:# odctl up kafka-lite. Run it with: desflow run avro_and_pydantic.yamlsimulation:{sim_id:Shop, factor:0, random_seed:42}egress:-type:Kafkaconfig:bootstrap_servers:${KAFKA_BOOTSTRAP_SERVERS:-127.0.0.1:9092}topics:ml-predictions:# Avro; sim-events and sim-telemetry stay JSONserializer:type:confluent_avroregistry_url:${SCHEMA_REGISTRY_URL:-http://127.0.0.1:8081}schema_file:prediction_event.avscresources:packers:{current_cap:1, max_cap:1}services:packing:{dist:constant, value:1.5}arrivals:orders:{dist:exponential, rate:0.5, spawn:pack_order}tasks:pack_order:service:packingresource:packers# The finished event is a prediction for the order: the fields of MLPredictionpayload:__topic__:ml-predictionsevent_type:predictiontask_id:{gen:entity}confidence:0.9run:{until:1 min}
{"type":"record","name":"PredictionEvent","fields":[{"name":"key","type":"string"},{"name":"task","type":["null","string"],"default":null},{"name":"instance","type":["null","long"],"default":null},{"name":"sim_ts","type":"double"},{"name":"timestamp","type":"string"},{"name":"value","type":{"type":"record","name":"MLPrediction","fields":[{"name":"event_type","type":"string","default":"prediction"},{"name":"task_id","type":"string"},{"name":"confidence","type":"double"}],"doc":"The prediction, published as the event value."}}],"doc":"The record KafkaEgress serializes for an event.","namespace":"com.desflow.ml"}
"""The shop of the Use Avro and Pydantic page, built with the builder API.The shop publishes predictions as Avro and every other record as JSON. It needs a Kafkabroker and a schema registry: start both with `odctl up kafka-lite`, then run`python avro_and_pydantic.py`. avro_and_pydantic.yaml and avro_and_pydantic_low_level.pyare the same shop in YAML and the low-level API. The code forms import the models andthe serializer of avro_and_pydantic_models.py, which must be beside them."""# ruff: noqa: E402importloggingimportosfromavro_and_pydantic_modelsimportMLPrediction,generated_serializerfromdesflowimportKafkaEgress,SimulationContextegress=KafkaEgress(bootstrap_servers=os.environ.get("KAFKA_BOOTSTRAP_SERVERS","127.0.0.1:9092"),topics={"ml-predictions":{"serializer":generated_serializer}},# Avro)app=(SimulationContext(sim_id="Shop",factor=0,random_seed=42).add_resource("packers",current_cap=1,max_cap=1).add_service("packing",dist="constant",value=1.5).add_arrival("orders",dist="exponential",rate=0.5).add_egress(egress)# predictions as Avro, the rest as JSON)@app.task(service_id="packing",resource_id="packers")defpack_order(task_id:int,entity):# The finished event is a prediction for the order: the fields of MLPredictionprediction=MLPrediction(task_id=entity.key,confidence=0.9)return{**prediction.model_dump(),"__topic__":"ml-predictions"}@app.arrival_loop("orders")deforders(context:SimulationContext):whileTrue:order=yieldcontext.wait_for_arrival("orders")context.spawn(pack_order(context.next_instance("pack_order"),entity=order))if__name__=="__main__":logging.basicConfig(level=logging.INFO,format="%(levelname)s%(message)s")app.run(until="1 min")
"""The Use Avro and Pydantic guide, low-level API: each packed order publishes aprediction, sent to ml-predictions as Avro, and every other record goes to its topicas JSON.The model and the serializer are those of avro_and_pydantic_models.py, which must bebeside this script. It needs a Kafka broker and a schema registry: start both with`odctl up kafka-lite`, then run `python avro_and_pydantic_low_level.py`."""importitertoolsimportloggingimportosfromavro_and_pydantic_modelsimportMLPrediction,generated_serializerfromdesflowimport(CapacityConfig,DistributionConfig,DynamicRealtimeEnvironment,DynamicResource,KafkaEgress,Sampler,SimParameter,)PARAMS=SimParameter(sim_id="Shop",arrival={"orders":DistributionConfig(dist="exponential",rate=0.5)},service={"packing":DistributionConfig(dist="constant",value=1.5)},resources={"packers":CapacityConfig(current_cap=1,max_cap=1)},)egress=KafkaEgress(bootstrap_servers=os.environ.get("KAFKA_BOOTSTRAP_SERVERS","127.0.0.1:9092"),topics={"ml-predictions":{"serializer":generated_serializer}},# Avro)defpack_order(env,packers,sampler,key,task_id):path_id="Shop.service.packing"env.publish_event(key,{"path_id":path_id,"sim_id":"Shop","service":"packing","status":"queued",},"pack_order",task_id,)withpackers.request()asunit:yieldunitenv.publish_event(key,{"path_id":path_id,"sim_id":"Shop","service":"packing","status":"started",},"pack_order",task_id,)yieldenv.timeout(sampler.sample(env.registry.get_config(path_id)))# The finished event is a prediction for the order: the fields of MLPredictionprediction=MLPrediction(task_id=key,confidence=0.9)payload={**prediction.model_dump(),"__topic__":"ml-predictions"}env.publish_event(key,payload,"pack_order",task_id)deforders(env,packers,sampler):gap=env.registry.get_config("Shop.arrival.orders")fornumberinitertools.count(1):yieldenv.timeout(sampler.sample(gap))env.process(pack_order(env,packers,sampler,f"orders-{number}",number-1))defshop(sinks:list,factor:float=0.0)->DynamicRealtimeEnvironment:"""The shop, publishing to the sinks given."""env=DynamicRealtimeEnvironment(factor=factor)env.registry.register_sim_parameter(PARAMS)env.setup_egress(sinks)packers=DynamicResource(env,"Shop","packers")env.process(orders(env,packers,Sampler(seed=42)))returnenvif__name__=="__main__":logging.basicConfig(level=logging.INFO,format="%(levelname)s [%(asctime)s] %(message)s")env=shop([egress])# predictions as Avro, the rest as JSONtry:env.run(until=60)finally:env.teardown()