Send each record of a simulation to the Kafka topic that suits it: failed orders to an alarm topic, telemetry to its own topic and every other event to the event topic.
Before You Start
Egress explains KafkaEgress and its two default topics.
Install the Kafka connectors: pip install "desflow[kafka]".
The scripts publish to Kafka, so start a broker with odctl up kafka-lite. Installation shows the one-time odctl install. Tear it down afterwards with odctl down kafka-lite --volumes.
Steps
Name the Topic in the Payload
A payload key __topic__ names the topic of a record. The sink sends the record to that topic and removes the key from the message. An event without the key goes to the event topic, and telemetry always goes to the telemetry topic.
This shop packs an order about every 2 seconds, and every tenth order fails. A failed order names system-alarms, and a packed order names shop-events:
app=SimulationContext(sim_id="Shop",factor=0,random_seed=42)app.add_egress(egress)app.add_resource("packers",current_cap=1,max_cap=1)app.add_arrival("orders","exponential",rate=0.5,spawn="pack_order")app.add_service("packing","constant",value=1.5)app.with_stats("minimal",interval=10)# readings for shop-telemetrypayload=Template({"status":{"gen":"when","condition":"failed","then":"packing_failed","else":"packed",},"__topic__":{"gen":"when","condition":"failed","then":"system-alarms","else":"shop-events",},},name="tasks.pack_order",vars={"failed":{"gen":"expr","expr":"process.instance % 10 == 9"}},)@app.task(name="pack_order",service_id="packing",resource_id="packers")defpack_order(task_id):"""Gives one order record."""returnpayload
defpack_order(env,packers,sampler,key,task_id):path_id="Shop.service.packing"stats=env.statsqueued={"path_id":path_id,"sim_id":"Shop","service":"packing"}env.publish_event(key,{**queued,"status":"queued"},"pack_order",task_id)stats.task_queued("pack_order",task_id)withpackers.request()asunit:yieldunitenv.publish_event(key,{**queued,"status":"started"},"pack_order",task_id)stats.task_started("pack_order",task_id)yieldenv.timeout(sampler.sample(env.registry.get_config(path_id)))failed=task_id%10==9# every tenth orderpayload={"status":"packing_failed"iffailedelse"packed","__topic__":"system-alarms"iffailedelse"shop-events",}env.publish_event(key,payload,"pack_order",task_id)stats.task_finished("pack_order",task_id)deforders(env,packers,sampler):gap=env.registry.get_config("Shop.arrival.orders")fornumberinitertools.count(1):yieldenv.timeout(sampler.sample(gap))env.process(pack_order(env,packers,sampler,f"orders-{number}",number-1))
The queued and started records of an order name no topic, so they go to the event topic.
Create the Topics
The sink creates the event and telemetry topics when the run starts. A topic that only a payload names is unknown to the sink, so list it in topics. topic_policy says what the sink does with its topics: create, the default, makes a missing topic and leaves an existing one, drop_and_create deletes each topic and creates it again at the start of a run, and manual creates nothing.
defpayload_egress()->KafkaEgress:"""Sends each record to the topic its payload names, and creates the topics."""returnKafkaEgress(bootstrap_servers=BOOTSTRAP,event_topic="shop-events",telemetry_topic="shop-telemetry",topic_policy="create",topics={"system-alarms":{}},)
defpayload_egress()->KafkaEgress:"""Sends each record to the topic its payload names, and creates the topics."""returnKafkaEgress(bootstrap_servers=BOOTSTRAP,event_topic="shop-events",telemetry_topic="shop-telemetry",topic_policy="create",topics={"system-alarms":{}},)
Pick the Topic With a Function
Use a function when the topic depends on more than a payload can say. A topic_router receives one record, as a dictionary with its stream_type, key or path_id, task, instance and value, and returns the topic name. It sees telemetry too, so it must always return a topic. It wins over __topic__, and the sink then creates only the topics in topics. To drop records, give the sink a when on its entry: it filters them before the router sees them. Egress shows the forms of when.
This router sends the shop's records to the same three topics. Every form uses it unchanged, from complex_routing_logic.py:
defcustom_kafka_router(data:Dict[str,Any])->str:"""Picks a topic for each record by its kind and its value."""ifdata.get("stream_type")=="telemetry":return"shop-telemetry"value=data.get("value")ifisinstance(value,dict)and(value.get("status")=="packing_failed"orvalue.get("level")=="ERROR"):return"system-alarms"# Every other event, including a payload that is not a dict.return"shop-events"
egress:-type:Kafkaconfig:bootstrap_servers:${KAFKA_BOOTSTRAP_SERVERS:-127.0.0.1:9092}topic_router:!pythoncomplex_routing_logic.custom_kafka_routertopics:# with a router, name the topics it usesshop-events:{}shop-telemetry:{}system-alarms:{}
defrouter_egress()->KafkaEgress:"""Sends each record to the topic a function picks."""returnKafkaEgress(bootstrap_servers=BOOTSTRAP,topic_router=custom_kafka_router,topics={"shop-events":{},"shop-telemetry":{},"system-alarms":{}},)
defrouter_egress()->KafkaEgress:"""Sends each record to the topic a function picks."""returnKafkaEgress(bootstrap_servers=BOOTSTRAP,topic_router=custom_kafka_router,topics={"shop-events":{},"shop-telemetry":{},"system-alarms":{}},)
Run the shop with the topic in the payload as desflow run complex_routing.yaml, python complex_routing.py or python complex_routing_low_level.py. Run it with the function as desflow run complex_routing_router.yaml, or set ROUTING=router in front of python complex_routing.py or python complex_routing_low_level.py. Each runs for one simulated minute and ends at once.
Check the Result
While kafka-lite is up, the Kafka UI at http://127.0.0.1:8086 lists the topics shop-events, shop-telemetry and system-alarms. In one simulated minute system-alarms receives two messages, for orders-10 and orders-20, the instances 9 and 19 of pack_order, and each has the value {"status": "packing_failed"}. No message holds __topic__. The three forms use seed 42, so they send the same messages, and the router sends each message to the same topic as the payload form.
# The Route Records to Topics guide: a shop whose failed orders go to their own Kafka# topic. Needs a broker on 127.0.0.1:9092 (odctl up kafka-lite). Run it with# desflow run complex_routing.yamlsimulation:{sim_id:Shop, factor:0, random_seed:42}egress:-type:Kafkaconfig:bootstrap_servers:${KAFKA_BOOTSTRAP_SERVERS:-127.0.0.1:9092}event_topic:shop-eventstelemetry_topic:shop-telemetrytopic_policy:createtopics:system-alarms:{}resources:packers:{current_cap:1, max_cap:1}services:packing:{dist:constant, value:1.5 s}arrivals:orders:{dist:exponential, rate:0.5, spawn:pack_order}tasks:pack_order:service:packingresource:packersvars:failed:{gen:expr, expr:"process.instance%10==9"}# every tenth orderpayload:status:{gen:when, condition:failed, then:packing_failed, else:packed}__topic__:{gen:when, condition:failed, then:system-alarms, else:shop-events}stats:{level:minimal, interval:10}# readings for shop-telemetryrun:{until:1 min}
# The Route Records to Topics guide: the same shop, with a function that picks the# topic. Needs a broker on 127.0.0.1:9092 (odctl up kafka-lite). Run it with# desflow run complex_routing_router.yaml, beside complex_routing_logic.pysimulation:{sim_id:Shop, factor:0, random_seed:42}egress:-type:Kafkaconfig:bootstrap_servers:${KAFKA_BOOTSTRAP_SERVERS:-127.0.0.1:9092}topic_router:!pythoncomplex_routing_logic.custom_kafka_routertopics:# with a router, name the topics it usesshop-events:{}shop-telemetry:{}system-alarms:{}resources:packers:{current_cap:1, max_cap:1}services:packing:{dist:constant, value:1.5 s}arrivals:orders:{dist:exponential, rate:0.5, spawn:pack_order}tasks:pack_order:service:packingresource:packersvars:failed:{gen:expr, expr:"process.instance%10==9"}# every tenth orderpayload:status:{gen:when, condition:failed, then:packing_failed, else:packed}stats:{level:minimal, interval:10}# readings for shop-telemetryrun:{until:1 min}
"""The Route Records to Topics guide: the function that picks a topic for each record."""fromtypingimportAny,Dictdefcustom_kafka_router(data:Dict[str,Any])->str:"""Picks a topic for each record by its kind and its value."""ifdata.get("stream_type")=="telemetry":return"shop-telemetry"value=data.get("value")ifisinstance(value,dict)and(value.get("status")=="packing_failed"orvalue.get("level")=="ERROR"):return"system-alarms"# Every other event, including a payload that is not a dict.return"shop-events"
"""The Route Records to Topics guide: a shop whose failed orders go to their own topic.It needs a Kafka broker on 127.0.0.1:9092: start one with `odctl up kafka-lite`. Run itwith `python complex_routing.py`, beside complex_routing_logic.py; it runs for oneminute. With ROUTING=router in the environment, `python complex_routing.py` sends therecords through a function instead, which wins over the topic in the payload."""importloggingimportosfromcomplex_routing_logicimportcustom_kafka_routerfromdesflowimportKafkaEgress,SimulationContextfromdesflow.valuesimportTemplateBOOTSTRAP=os.environ.get("KAFKA_BOOTSTRAP_SERVERS","127.0.0.1:9092")defpayload_egress()->KafkaEgress:"""Sends each record to the topic its payload names, and creates the topics."""returnKafkaEgress(bootstrap_servers=BOOTSTRAP,event_topic="shop-events",telemetry_topic="shop-telemetry",topic_policy="create",topics={"system-alarms":{}},)defrouter_egress()->KafkaEgress:"""Sends each record to the topic a function picks."""returnKafkaEgress(bootstrap_servers=BOOTSTRAP,topic_router=custom_kafka_router,topics={"shop-events":{},"shop-telemetry":{},"system-alarms":{}},)defbuild(egress:KafkaEgress)->SimulationContext:"""Builds the shop, whose payload names a topic, with `egress` as its one sink."""app=SimulationContext(sim_id="Shop",factor=0,random_seed=42)app.add_egress(egress)app.add_resource("packers",current_cap=1,max_cap=1)app.add_arrival("orders","exponential",rate=0.5,spawn="pack_order")app.add_service("packing","constant",value=1.5)app.with_stats("minimal",interval=10)# readings for shop-telemetrypayload=Template({"status":{"gen":"when","condition":"failed","then":"packing_failed","else":"packed",},"__topic__":{"gen":"when","condition":"failed","then":"system-alarms","else":"shop-events",},},name="tasks.pack_order",vars={"failed":{"gen":"expr","expr":"process.instance % 10 == 9"}},)@app.task(name="pack_order",service_id="packing",resource_id="packers")defpack_order(task_id):"""Gives one order record."""returnpayloadreturnappif__name__=="__main__":logging.basicConfig(level=logging.INFO,format="%(levelname)s%(message)s")chosen=(router_egress()ifos.environ.get("ROUTING")=="router"elsepayload_egress())build(chosen).run(until="1 min")
"""The Route Records to Topics guide, low-level API: a shop whose failed orders go totheir own topic.It needs a Kafka broker on 127.0.0.1:9092: start one with `odctl up kafka-lite`. Run itwith `python complex_routing_low_level.py`, beside complex_routing_logic.py; it runsfor one minute. With ROUTING=router in the environment, it sends the records through afunction instead, which wins over the topic in the payload."""importitertoolsimportloggingimportosfromcomplex_routing_logicimportcustom_kafka_routerfromdesflowimport(CapacityConfig,DistributionConfig,DynamicRealtimeEnvironment,DynamicResource,KafkaEgress,Sampler,SimParameter,)BOOTSTRAP=os.environ.get("KAFKA_BOOTSTRAP_SERVERS","127.0.0.1:9092")PARAMS=SimParameter(sim_id="Shop",arrival={"orders":DistributionConfig(dist="exponential",rate=0.5)},service={"packing":DistributionConfig(dist="constant",value=1.5)},resources={"packers":CapacityConfig(current_cap=1,max_cap=1)},)defpayload_egress()->KafkaEgress:"""Sends each record to the topic its payload names, and creates the topics."""returnKafkaEgress(bootstrap_servers=BOOTSTRAP,event_topic="shop-events",telemetry_topic="shop-telemetry",topic_policy="create",topics={"system-alarms":{}},)defrouter_egress()->KafkaEgress:"""Sends each record to the topic a function picks."""returnKafkaEgress(bootstrap_servers=BOOTSTRAP,topic_router=custom_kafka_router,topics={"shop-events":{},"shop-telemetry":{},"system-alarms":{}},)defpack_order(env,packers,sampler,key,task_id):path_id="Shop.service.packing"stats=env.statsqueued={"path_id":path_id,"sim_id":"Shop","service":"packing"}env.publish_event(key,{**queued,"status":"queued"},"pack_order",task_id)stats.task_queued("pack_order",task_id)withpackers.request()asunit:yieldunitenv.publish_event(key,{**queued,"status":"started"},"pack_order",task_id)stats.task_started("pack_order",task_id)yieldenv.timeout(sampler.sample(env.registry.get_config(path_id)))failed=task_id%10==9# every tenth orderpayload={"status":"packing_failed"iffailedelse"packed","__topic__":"system-alarms"iffailedelse"shop-events",}env.publish_event(key,payload,"pack_order",task_id)stats.task_finished("pack_order",task_id)deforders(env,packers,sampler):gap=env.registry.get_config("Shop.arrival.orders")fornumberinitertools.count(1):yieldenv.timeout(sampler.sample(gap))env.process(pack_order(env,packers,sampler,f"orders-{number}",number-1))defrun(until=60.0,egress=None):"""Runs the shop unpaced into `egress`, which is `payload_egress()` unless given."""env=DynamicRealtimeEnvironment(factor=0)env.registry.register_sim_parameter(PARAMS)env.setup_egress([egressorpayload_egress()])env.setup_stats("Shop","minimal",interval=10)# readings for shop-telemetrypackers=DynamicResource(env,"Shop","packers")env.process(orders(env,packers,Sampler(seed=42)))try:env.run(until=until)finally:env.teardown()if__name__=="__main__":logging.basicConfig(level=logging.INFO,format="%(levelname)s%(message)s")run(router_egress()ifos.environ.get("ROUTING")=="router"elsepayload_egress())