IcebergStorageEgress appends records to an Apache Iceberg table through a catalog, so any engine that reads Iceberg, such as Spark, Trino or DuckDB, can query them. This page appends an hour of a shop's orders to the table shop.events through a catalog you pick, then reads the table back.
Before You Start
Egress explains the sink, the columns of a table it creates, and upserts.
Install the extra: pip install "desflow[iceberg]". It covers every catalog on this page.
The SQLite catalog needs no service. The REST catalog needs odctl up catalog, and the PostgreSQL catalog odctl up postgres. See Installation for the one-time odctl install. Stop a profile with odctl down <profile> --volumes.
The Glue catalog needs AWS credentials and an S3 bucket for the table files.
Steps
Choose the Catalog
catalog is a mapping of PyIceberg catalog properties, which desflow passes to PyIceberg's load_catalog on the first write. type picks the kind of catalog, and the optional name key names it. The script keeps one mapping for each catalog, and ICEBERG_CATALOG picks one:
importosfrompathlibimportPath# The SQLite database and the local table files go in the folder iceberg/FOLDER=Path("iceberg").resolve()FOLDER.mkdir(exist_ok=True)# One mapping of PyIceberg catalog properties for each catalogCATALOGS={# A SQL catalog on SQLite, with no service, for a first run"sqlite":{"type":"sql","uri":f"sqlite:///{FOLDER}/catalog.db","warehouse":f"file://{FOLDER}/warehouse",},# The REST catalog of `odctl up catalog`, with the table files on its SeaweedFS"rest":{"type":"rest","uri":"http://127.0.0.1:8181","warehouse":"s3://warehouse/","s3.endpoint":"http://127.0.0.1:8333","s3.access-key-id":"user","s3.secret-access-key":"password","s3.region":"us-east-1",},# A SQL catalog on the PostgreSQL of `odctl up postgres`, with local table files"postgres":{"type":"sql","uri":"postgresql+psycopg2://user:password@127.0.0.1:5432/odctl","warehouse":f"file://{FOLDER}/warehouse",},# The AWS Glue Data Catalog, with the table files in S3 and the standard AWS settings"glue":{"type":"glue","glue.region":"us-east-1","s3.region":"us-east-1","warehouse":os.getenv("ICEBERG_WAREHOUSE","s3://my-bucket/warehouse"),},}name=os.getenv("ICEBERG_CATALOG","sqlite")catalog={"name":name,**CATALOGS[name]}
The Glue mapping gives no credentials, so PyIceberg uses the standard AWS settings: environment variables such as AWS_ACCESS_KEY_ID, the files in ~/.aws, or the role the machine runs as. ICEBERG_WAREHOUSE names the S3 folder of the table files. PyIceberg's configuration lists the properties of each catalog.
Write the Table
The sink creates the table from the first batch, with a column for each key of the records. Each flush is one commit, and every commit writes new metadata, so the sink gets a large batch of its own:
fromdesflowimportIcebergStorageEgressdeficeberg_sink(catalog:dict,table:str="shop.events")->IcebergStorageEgress:"""Every event as a row of `table`, which the sink creates from the first batch."""returnIcebergStorageEgress(catalog=catalog,default_table=table)sink=iceberg_sink(catalog)# One commit per flush, so the sink gets a large batch of its ownBATCH_SIZE=200_000
A new table takes the type of each column from schemas when it is named there, and otherwise from the first batch. A table that exists keeps its own types. A column that a later batch adds is added to the table, with two exceptions. A table named in schemas never gains a column. A table with upsert_keys that is not named there refuses that batch. Egress has the column and type rules.
A table that has an integer column refuses a later batch with a fraction in it. The sink logs an error, such as could not write 1 rows to shop.events: row 1 holds 2.5 for the int64 column 'amount', which would lose its fraction. Iceberg cannot change the type of a column that exists. Name the column double in schemas before the table is created (for a list or struct column, in a PyArrow schema), or create the table again: run once with table_policy: drop_and_create, then set it back to create, and the rows of that batch are not written. A table that exists keeps its long column, so a schemas entry changes it only when the table is created again. To fix it, name every column the table needs in schemas, such as schemas: {shop.events: {stream_type: string, sim_ts: double, timestamp: string, key: string, task: string, instance: long, path_id: string, sim_id: string, service: string, status: string, order_id: long, amount: double}}. A table named there is created with exactly those columns and leaves any other field out. Then create the table again. Run once with table_policy: drop_and_create, which drops the table and deletes its rows. Then set table_policy back to create. The policy drops the table at the first write of every run, as Egress says, so a later run left on it deletes the new rows too.
A statistic (a number) and an event or a reading (a mapping) cannot share a table or a file, because the value column has one type. If a table_router or a path_router sends both to one place, the error names this cause. Send each kind to its own table.
Each form runs the shop for an hour of simulated time at factor=0.0. The run flushes once, at its end, so the hour of orders is one commit:
egress:# Every event as a row of shop.events, which the sink creates from the first batch-type:Icebergconfig:default_table:shop.eventscatalog:!includeiceberg_tables_catalogs/${ICEBERG_CATALOG:-sqlite}.yaml# One commit per flush, so the sink gets a large batch of its ownbatch_size:200000
The YAML form keeps the same four mappings in the files sqlite.yaml, rest.yaml, postgres.yaml and glue.yaml of the folder iceberg_tables_catalogs. !include iceberg_tables_catalogs/${ICEBERG_CATALOG:-sqlite}.yaml reads the one that ICEBERG_CATALOG names, so the variable picks the catalog in every form. The Builder and Low-level scripts create the folder iceberg for the SQLite files. For the YAML form, run mkdir iceberg first, because SQLite does not create the folder of its database.
importloggingfromiceberg_tables_catalogimportBATCH_SIZE,sinkfromdesflowimportIcebergStorageEgress,SimulationContextdefbuild_shop(sink:IcebergStorageEgress)->SimulationContext:"""The shop, appending to the table of the sink given, as fast as it can."""app=(SimulationContext(sim_id="Shop",factor=0.0,random_seed=42).add_resource("packers",current_cap=1,max_cap=1).add_service("packing",dist="constant",value=1.5).add_arrival("orders",dist="exponential",rate=0.5).add_egress(sink,batch_size=BATCH_SIZE))@app.task(service_id="packing",resource_id="packers")defpack_order(task_id:int,entity):# The finished event is the order recordreturn{"order_id":task_id,"amount":19.99}@app.arrival_loop("orders")deforders(context:SimulationContext):whileTrue:order=yieldcontext.wait_for_arrival("orders")context.spawn(pack_order(context.next_instance("pack_order"),entity=order))returnappapp=build_shop(sink)if__name__=="__main__":logging.basicConfig(level=logging.INFO,format="%(levelname)s [%(asctime)s] %(message)s")app.run(until="1 hour")
shop takes the sink as an argument, and the script passes it sink from iceberg_tables_catalog.py. shop sets the batch size of the sink, BATCH_SIZE from the same file, in batch_sizes:
defpack_order(env,packers,sampler,key,task_id):path_id="Shop.service.packing"env.publish_event(key,{"path_id":path_id,"sim_id":"Shop","service":"packing","status":"queued",},"pack_order",task_id,)withpackers.request()asunit:yieldunitenv.publish_event(key,{"path_id":path_id,"sim_id":"Shop","service":"packing","status":"started",},"pack_order",task_id,)yieldenv.timeout(sampler.sample(env.registry.get_config(path_id)))# The finished event is the order recordorder={"order_id":task_id,"amount":19.99}env.publish_event(key,order,"pack_order",task_id)deforders(env,packers,sampler):gap=env.registry.get_config("Shop.arrival.orders")fornumberinitertools.count(1):yieldenv.timeout(sampler.sample(gap))env.process(pack_order(env,packers,sampler,f"orders-{number}",number-1))defshop(sink,factor:float=0.0)->DynamicRealtimeEnvironment:"""The shop, appending to the table of the sink given, as fast as it can."""env=DynamicRealtimeEnvironment(factor=factor)env.registry.register_sim_parameter(PARAMS)# One commit per flush, so the sink gets a large batch of its ownenv.setup_egress([sink],batch_sizes=[BATCH_SIZE])packers=DynamicResource(env,"Shop","packers")env.process(orders(env,packers,Sampler(seed=42)))returnenv
Read the Table Back
read_events loads the table through the same catalog and reads every row:
defread_events(catalog:dict,table:str="shop.events"):"""Every row of `table`, as one PyArrow table."""frompyiceberg.catalogimportload_catalogproperties={key:valueforkey,valueincatalog.items()ifkey!="name"}return(load_catalog(catalog["name"],**properties).load_table(table).scan().to_arrow())
A second run appends its rows to the same table. To replace the rows of an earlier run instead, give the sink upsert_keys, as Egress shows. A table with upsert_keys that is not named in schemas refuses a batch that brings a column the table lacks. A table named there leaves such a field out without an error. So name every column in schemas before the table is created. For a table that exists, name them and create the table again. Run once with table_policy: drop_and_create, which deletes its rows. Then set table_policy back to create, or every later run drops the table again. Egress has the rest of the column rules. A sink with upsert_keys skips a record that has no destination or that has a null or missing key column, and logs one warning at the end of each flush with the number skipped and the reason. An entry that names its tables in schemas, locations or upsert_keys needs no default_table or router: a record routes by its __table__. A record whose __table__ is null or empty goes to default_table. A record whose __table__ names no namespace, such as events in place of shop.events, is skipped, and the flush logs a warning with the number skipped.
Check the Result
Run python iceberg_tables.py, python iceberg_tables_low_level.py or desflow run iceberg_tables.yaml. With no variables set, the catalog is SQLite, and its database and table files go in the folder iceberg. With the catalog profile up,
writes through the REST catalog instead. In every form, shop.events gains one snapshot, with a row for each event: the queued and started events of each order, with path_id and status, and its finished event, with order_id and amount. A seeded run writes the same events in every form. Only timestamp differs, because it is the clock time at the start of the run plus the simulation time.
To count the rows, run the catalog file with the same variables. Its last lines call read_events and print the count:
The count of one run is 5,465 rows. A second run appends its rows to the table, so the count is then the sum of both runs.
Full Files
The Builder and Low-level forms use the catalog and the sink of iceberg_tables_catalog.py, which builds no shop. The YAML form uses the folder iceberg_tables_catalogs instead. Keep the files of the form you run in one folder.
"""The catalogs, sink and reader of the Write Iceberg Tables page.The builder and low-level forms of the shop import the catalog and the sink from this file:iceberg_tables.py and iceberg_tables_low_level.py. iceberg_tables.yaml reads the samemappings from the folder iceberg_tables_catalogs. It builds no simulation.ICEBERG_CATALOG picks the catalog: sqlite (the default), rest, postgres or glue."""# Each section carries its own imports, as the page shows it.# ruff: noqa: E402importosfrompathlibimportPath# The SQLite database and the local table files go in the folder iceberg/FOLDER=Path("iceberg").resolve()FOLDER.mkdir(exist_ok=True)# One mapping of PyIceberg catalog properties for each catalogCATALOGS={# A SQL catalog on SQLite, with no service, for a first run"sqlite":{"type":"sql","uri":f"sqlite:///{FOLDER}/catalog.db","warehouse":f"file://{FOLDER}/warehouse",},# The REST catalog of `odctl up catalog`, with the table files on its SeaweedFS"rest":{"type":"rest","uri":"http://127.0.0.1:8181","warehouse":"s3://warehouse/","s3.endpoint":"http://127.0.0.1:8333","s3.access-key-id":"user","s3.secret-access-key":"password","s3.region":"us-east-1",},# A SQL catalog on the PostgreSQL of `odctl up postgres`, with local table files"postgres":{"type":"sql","uri":"postgresql+psycopg2://user:password@127.0.0.1:5432/odctl","warehouse":f"file://{FOLDER}/warehouse",},# The AWS Glue Data Catalog, with the table files in S3 and the standard AWS settings"glue":{"type":"glue","glue.region":"us-east-1","s3.region":"us-east-1","warehouse":os.getenv("ICEBERG_WAREHOUSE","s3://my-bucket/warehouse"),},}name=os.getenv("ICEBERG_CATALOG","sqlite")catalog={"name":name,**CATALOGS[name]}fromdesflowimportIcebergStorageEgressdeficeberg_sink(catalog:dict,table:str="shop.events")->IcebergStorageEgress:"""Every event as a row of `table`, which the sink creates from the first batch."""returnIcebergStorageEgress(catalog=catalog,default_table=table)sink=iceberg_sink(catalog)# One commit per flush, so the sink gets a large batch of its ownBATCH_SIZE=200_000defread_events(catalog:dict,table:str="shop.events"):"""Every row of `table`, as one PyArrow table."""frompyiceberg.catalogimportload_catalogproperties={key:valueforkey,valueincatalog.items()ifkey!="name"}return(load_catalog(catalog["name"],**properties).load_table(table).scan().to_arrow())if__name__=="__main__":events=read_events(catalog)print(f"{events.num_rows} rows in shop.events")
# The Write to Iceberg Tables guide: the shop appends an hour of orders to the Iceberg# table shop.events, as fast as the machine allows. ICEBERG_CATALOG picks the catalog# (sqlite, the default, rest, postgres or glue). Needs the folder# iceberg_tables_catalogs beside it, and for sqlite an empty folder iceberg. Run it# with: desflow run iceberg_tables.yamlsimulation:{sim_id:Shop, factor:0.0, random_seed:42}resources:packers:{current_cap:1, max_cap:1}services:packing:{dist:constant, value:1.5}arrivals:orders:{dist:exponential, rate:0.5, spawn:pack_order}tasks:pack_order:service:packingresource:packers# The finished event is the order recordpayload:order_id:{gen:sequence}amount:19.99egress:# Every event as a row of shop.events, which the sink creates from the first batch-type:Icebergconfig:default_table:shop.eventscatalog:!includeiceberg_tables_catalogs/${ICEBERG_CATALOG:-sqlite}.yaml# One commit per flush, so the sink gets a large batch of its ownbatch_size:200000run:{until:1 hour}
# A SQL catalog on SQLite, with no service, for a first run. The folder iceberg must# existname:sqlitetype:sqluri:sqlite:///iceberg/catalog.dbwarehouse:iceberg/warehouse
# The REST catalog of `odctl up catalog`, with the table files on its SeaweedFSname:resttype:resturi:http://127.0.0.1:8181warehouse:s3://warehouse/s3.endpoint:http://127.0.0.1:8333s3.access-key-id:users3.secret-access-key:passwords3.region:us-east-1
# A SQL catalog on the PostgreSQL of `odctl up postgres`, with local table filesname:postgrestype:sqluri:postgresql+psycopg2://user:password@127.0.0.1:5432/odctlwarehouse:iceberg/warehouse
# The AWS Glue Data Catalog, with the table files in S3 and the standard AWS settingsname:gluetype:glueglue.region:us-east-1s3.region:us-east-1warehouse:${ICEBERG_WAREHOUSE:-s3://my-bucket/warehouse}
"""The shop of the Write Iceberg Tables page, built with the builder API.The shop appends an hour of orders to the Iceberg table shop.events, as fast as themachine allows. Run it with `python iceberg_tables.py`; rest needs `odctl up catalog`,postgres needs `odctl up postgres`, and glue needs AWS credentials and ICEBERG_WAREHOUSE.iceberg_tables.yaml and iceberg_tables_low_level.py are the same shop in YAML and thelow-level API. All three import the catalog and the sink of iceberg_tables_catalog.py, whichmust be beside them."""# ruff: noqa: E402importloggingfromiceberg_tables_catalogimportBATCH_SIZE,sinkfromdesflowimportIcebergStorageEgress,SimulationContextdefbuild_shop(sink:IcebergStorageEgress)->SimulationContext:"""The shop, appending to the table of the sink given, as fast as it can."""app=(SimulationContext(sim_id="Shop",factor=0.0,random_seed=42).add_resource("packers",current_cap=1,max_cap=1).add_service("packing",dist="constant",value=1.5).add_arrival("orders",dist="exponential",rate=0.5).add_egress(sink,batch_size=BATCH_SIZE))@app.task(service_id="packing",resource_id="packers")defpack_order(task_id:int,entity):# The finished event is the order recordreturn{"order_id":task_id,"amount":19.99}@app.arrival_loop("orders")deforders(context:SimulationContext):whileTrue:order=yieldcontext.wait_for_arrival("orders")context.spawn(pack_order(context.next_instance("pack_order"),entity=order))returnappapp=build_shop(sink)if__name__=="__main__":logging.basicConfig(level=logging.INFO,format="%(levelname)s [%(asctime)s] %(message)s")app.run(until="1 hour")
"""The Write to Iceberg Tables guide, low-level API: the shop appends an hour of ordersto the Iceberg table shop.events, as fast as the machine allows.The catalog and the sink are those of iceberg_tables_catalog.py, which must be beside thisscript. ICEBERG_CATALOG picks the catalog: sqlite (the default), rest, postgres or glue.Run it with `python iceberg_tables_low_level.py`."""importitertoolsimportloggingfromiceberg_tables_catalogimportBATCH_SIZE,sinkfromdesflowimport(CapacityConfig,DistributionConfig,DynamicRealtimeEnvironment,DynamicResource,Sampler,SimParameter,)PARAMS=SimParameter(sim_id="Shop",arrival={"orders":DistributionConfig(dist="exponential",rate=0.5)},service={"packing":DistributionConfig(dist="constant",value=1.5)},resources={"packers":CapacityConfig(current_cap=1,max_cap=1)},)defpack_order(env,packers,sampler,key,task_id):path_id="Shop.service.packing"env.publish_event(key,{"path_id":path_id,"sim_id":"Shop","service":"packing","status":"queued",},"pack_order",task_id,)withpackers.request()asunit:yieldunitenv.publish_event(key,{"path_id":path_id,"sim_id":"Shop","service":"packing","status":"started",},"pack_order",task_id,)yieldenv.timeout(sampler.sample(env.registry.get_config(path_id)))# The finished event is the order recordorder={"order_id":task_id,"amount":19.99}env.publish_event(key,order,"pack_order",task_id)deforders(env,packers,sampler):gap=env.registry.get_config("Shop.arrival.orders")fornumberinitertools.count(1):yieldenv.timeout(sampler.sample(gap))env.process(pack_order(env,packers,sampler,f"orders-{number}",number-1))defshop(sink,factor:float=0.0)->DynamicRealtimeEnvironment:"""The shop, appending to the table of the sink given, as fast as it can."""env=DynamicRealtimeEnvironment(factor=factor)env.registry.register_sim_parameter(PARAMS)# One commit per flush, so the sink gets a large batch of its ownenv.setup_egress([sink],batch_sizes=[BATCH_SIZE])packers=DynamicResource(env,"Shop","packers")env.process(orders(env,packers,Sampler(seed=42)))returnenvif__name__=="__main__":logging.basicConfig(level=logging.INFO,format="%(levelname)s [%(asctime)s] %(message)s")env=shop(sink)# rows of shop.events in the catalog ICEBERG_CATALOG namestry:env.run(until=3600)finally:env.teardown()