In this post, we develop a real-time monitoring dashboard using Streamlit, an open-source Python framework that allows data scientists and AI/ML engineers to create interactive data apps. The app connects to the WebSocket server we developed in Part 1 and continuously fetches data to visualize key metrics such as order counts, sales data, and revenue by traffic source and country. With interactive bar charts and dynamic metrics, users can monitor sales trends and other important business KPIs in real-time.
- Part 1 Data Producer
- Part 2 Streamlit Dashboard (this post)
- Part 3 Next.js Dashboard
Streamlit Frontend
This Streamlit dashboard is designed to process and display real-time theLook eCommerce data using pandas for data manipulation, Streamlit’s built-in metric component for KPIs, and Apache ECharts for visualizations. The source code for this post can be found in this GitHub repository.
Components
The dashboard components and data processing logic are managed by functions in streamlit/utils.py
. Below are the details of those functions.
Loading Records:
- The
load_records()
function accepts a list of records, converts them into a pandas DataFrame, and ensures that the necessary columns are present. - It then processes data by converting the age column to integers and the cost and sale_price columns to floating-point values, rounding the sale price to one decimal place.
- Key metrics like the number of orders, number of order items, and total sales are calculated from the DataFrame and returned for display.
- The
Generating Metrics:
- The
create_metric_items()
function creates a list of dictionaries containing metrics and their respective changes (delta
), comparing the current and previous values of orders, order items, and total sales. - The
generate_metrics()
function takes the calculated metrics and displays them in Streamlit’s metric components within theplaceholder
container. It uses a column layout to show the metrics side-by-side.
- The
Creating Chart Options:
- The
create_options_items()
function generates configuration data for bar charts that display revenue by country and revenue by traffic source. - It groups the data by country and traffic source, sums the sale_price for each, and sorts it in descending order.
- ECharts options are defined for each chart with custom colors, titles, and axis settings. The charts are configured to show tooltips when hovering over the bars.
- The
Displaying Charts:
- The
generate_charts()
function takes the ECharts options and renders each chart within a container using thest_echarts
component of the streamlit-echarts package. - Each chart is placed in its own column, with dynamic layout adjustments based on the number of charts being displayed. The charts are rendered with a fixed height of 500px.
- The
1## producer/streamlit/utils.py
2from uuid import uuid4
3
4import pandas as pd
5import streamlit as st
6from streamlit.delta_generator import DeltaGenerator
7from streamlit_echarts import st_echarts
8
9
10def load_records(records: list):
11 df = pd.DataFrame(records)
12 assert set(
13 [
14 "order_id",
15 "item_id",
16 "country",
17 "traffic_source",
18 "age",
19 "cost",
20 "sale_price",
21 ]
22 ).issubset(df.columns)
23 df["age"] = df["age"].astype(int)
24 df["cost"] = df["cost"].astype(float)
25 df["sale_price"] = df["sale_price"].astype(float).round(1)
26 metric_values = {
27 "num_orders": df["order_id"].nunique(),
28 "num_order_items": df["item_id"].nunique(),
29 "total_sales": round(df["sale_price"].sum()),
30 }
31 return metric_values, df
32
33
34def create_metric_items(metric_values, prev_values):
35 return [
36 {
37 "label": "Number of Orders",
38 "value": metric_values["num_orders"],
39 "delta": (metric_values["num_orders"] - prev_values["num_orders"]),
40 },
41 {
42 "label": "Number of Order Items",
43 "value": metric_values["num_order_items"],
44 "delta": (
45 metric_values["num_order_items"] - prev_values["num_order_items"]
46 ),
47 },
48 {
49 "label": "Total Sales",
50 "value": f"$ {metric_values['total_sales']}",
51 "delta": (metric_values["total_sales"] - prev_values["total_sales"]),
52 },
53 ]
54
55
56def generate_metrics(placeholder: DeltaGenerator, metric_items: list = None):
57 if metric_items is None:
58 metric_items = [
59 {"label": "Number of Orders", "value": 0, "delta": 0},
60 {"label": "Number of Order Items", "value": 0, "delta": 0},
61 {"label": "Total Sales", "value": 0, "delta": 0},
62 ]
63 with placeholder.container():
64 for i, col in enumerate(st.columns(len(metric_items))):
65 metric = metric_items[i]
66 col.metric(
67 label=metric["label"], value=metric["value"], delta=metric["delta"]
68 )
69
70
71def create_options_items(df: pd.DataFrame):
72 colors = [
73 "#00008b",
74 "#b22234",
75 "#00247d",
76 "#f00",
77 "#ffde00",
78 "#002a8f",
79 "#000",
80 "#003580",
81 "#ed2939",
82 "#003897",
83 "#f93",
84 "#bc002d",
85 "#024fa2",
86 "#000",
87 "#00247d",
88 "#ef2b2d",
89 "#dc143c",
90 "#d52b1e",
91 "#e30a17",
92 ]
93 chart_cols = [
94 {"x": "country", "y": "sale_price"},
95 {"x": "traffic_source", "y": "sale_price"},
96 ]
97 option_items = []
98 for col in chart_cols:
99 data = (
100 df[[col["x"], col["y"]]]
101 .groupby(col["x"])
102 .sum()
103 .reset_index()
104 .sort_values(by=col["y"], ascending=False)
105 )
106 options = {
107 "title": {"text": f"Revenue by {col['x'].replace('_', ' ').title()}"},
108 "xAxis": {
109 "type": "category",
110 "data": data[col["x"]].to_list(),
111 "axisLabel": {"show": True, "rotate": 75},
112 },
113 "yAxis": {"type": "value"},
114 "series": [
115 {
116 "data": [
117 {"value": d, "itemStyle": {"color": colors[i]}}
118 for i, d in enumerate(data[col["y"]].to_list())
119 ],
120 "type": "bar",
121 }
122 ],
123 "tooltip": {"trigger": "axis", "axisPointer": {"type": "shadow"}},
124 }
125 option_items.append(options)
126 return option_items
127
128
129def generate_charts(placeholder: DeltaGenerator, option_items: list):
130 with placeholder.container():
131 for i, col in enumerate(st.columns(len(option_items))):
132 options = option_items[i]
133 with col:
134 st_echarts(options=options, height="500px", key=str(uuid4()))
Application
The dashboard connects to the WebSocket server to fetch and display real-time theLook eCommerce data. Here’s a detailed breakdown of its functionality:
WebSocket Connection:
- The
generate()
function is an asynchronous task that establishes a connection to a WebSocket server (ws://localhost:8000/ws
) using an asynchronous HTTP client by the aiohttp package. It listens for incoming messages from the server, which contain the eCommerce data. - As each message is received, the function loads the data using the
load_records()
function, which processes the data into pandas DataFrame format and computes key metrics.
- The
Generating and Displaying Metrics:
- The
create_metric_items()
function calculates and prepares key metrics (such as number of orders, order items, and total sales) along with the delta (changes) from the previous values. - The
generate_metrics()
function updates the Streamlit dashboard by displaying these metrics in themetric_placeholder
.
- The
Creating and Displaying Charts:
- The
create_options_items()
function processes the data and generates configuration options for bar charts, displaying revenue by country and traffic source. - The
generate_charts()
function renders the charts in thechart_placeholder
container, using ECharts for interactive data visualizations.
- The
Real-time Updates:
- The loop continuously listens for new data from the WebSocket server. As data is received, it updates both the metrics and charts in real-time.
User Interface:
- The app sets a wide layout with the title “theLook eCommerce Dashboard”.
- A checkbox (
Connect to WS Server
) lets the user choose whether to connect to the WebSocket server. When checked, the dashboard fetches data live and updates metrics and charts accordingly. - If the checkbox is unchecked, only the static metrics are displayed.
This setup provides a dynamic dashboard that pulls and visualizes real-time eCommerce data, making it interactive and responsive for monitoring sales and performance metrics.
1## producer/streamlit/app.py
2import json
3import asyncio
4
5import aiohttp
6import streamlit as st
7from streamlit.delta_generator import DeltaGenerator
8
9from utils import (
10 load_records,
11 create_metric_items,
12 generate_metrics,
13 create_options_items,
14 generate_charts,
15)
16
17
18async def generate(
19 metric_placeholder: DeltaGenerator, chart_placeholder: DeltaGenerator
20):
21 prev_values = {"num_orders": 0, "num_order_items": 0, "total_sales": 0}
22 async with aiohttp.ClientSession() as session:
23 async with session.ws_connect("ws://localhost:8000/ws") as ws:
24 async for msg in ws:
25 metric_values, df = load_records(json.loads(msg.json()))
26 generate_metrics(
27 metric_placeholder, create_metric_items(metric_values, prev_values)
28 )
29 generate_charts(chart_placeholder, create_options_items(df))
30 prev_values = metric_values
31
32
33st.set_page_config(
34 page_title="theLook eCommerce",
35 page_icon="✅",
36 layout="wide",
37)
38
39st.title("theLook eCommerce Dashboard")
40
41connect = st.checkbox("Connect to WS Server")
42metric_placeholder = st.empty()
43chart_placeholder = st.empty()
44
45if connect:
46 asyncio.run(
47 generate(
48 metric_placeholder=metric_placeholder, chart_placeholder=chart_placeholder
49 )
50 )
51else:
52 generate_metrics(metric_placeholder, None)
Deployment
Data Producer and WebSocket Server
As discussed in Part 1, the data generator and WebSocket server can be deployed using Docker Compose with the command docker-compose -f producer/docker-compose.yml up -d
. Once started, the server can be checked with a WebSocket client by executing ws listen ws://localhost:8000/ws
, and its logs can be monitored by running docker logs -f producer
.
Frontend Dashboard
The dashboard can be started by running the Streamlit app as shown below. Once started, it can be accessed in a browser at http://localhost:8501.
1## create and activate a virtual environment
2# https://docs.python.org/3/library/venv.html
3
4## install pip packages
5$ pip install -r requirements.txt
6
7## start the app
8$ streamlit run streamlit/app.py
Comments