Skip to content

Commit 01251d1

Browse files
networmixnetwormix
committed
added descriptions of all metrics
1 parent 9a75626 commit 01251d1

3 files changed

Lines changed: 303 additions & 8 deletions

File tree

‎metrics.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# Topline Project Metrics and Plots
2+
3+
This document describes the project-level metrics and produced figures. All results for each multi-scenario run are stored in a set of nested folders inside a per-run top folder.
4+
5+
## Metrics (project-level)
6+
7+
- **scenario**: Scenario name (directory name under the per-run top-level folder)
8+
- What it is: Human-readable identifier for the network topology/config being analyzed. Taken directly from the scenario directory name under the per-run top-level folder. In the overall pipeline, scenario directory names come from the names of the TopoGen config files used for the run.
9+
10+
- **seeds**: Number of seeds included for the scenario
11+
- What it is: Count of independent runs of each of the scenarios contributing to the metrics.
12+
- How it's calculated: From discovered per-seed summaries.
13+
14+
- **node_count**: Median node count across seeds for the scenario
15+
- What it is: Number of nodes in the topology, per seed; summarized by median.
16+
- How it's calculated: Read from `network_stats_summary.csv` per scenario and take the median.
17+
18+
- **link_count**: Median link count across seeds for the scenario
19+
- What it is: Number of links in the topology, per seed; summarized by median.
20+
- How it's calculated: Read from `network_stats_summary.csv` per scenario and take the median.
21+
22+
- **alpha_star**: Maximum supported demand multiplier found by the MSD step (per seed), then median across seeds.
23+
- What it is: The largest traffic scaling factor (relative to the baseline traffic matrix) the network can place without violating placement constraints.
24+
- How it's calculated: From the `msd_baseline` step per seed; we take the median across seeds.
25+
- How to interpret: Higher is better; measures capacity headroom given topology and constraints. By design, TopoGen tries to preserve capacity invariant in the model to make it easier to compare across scenarios, hence, this should be the same or very similar for all scenarios in a run. If it's not, it's a sign of a problem with the generated model.
26+
- Pseudocode:
27+
- per seed: `alpha_star_seed = results.steps.msd_baseline.data.alpha_star`
28+
- scenario: `alpha_star = median(alpha_star_seed over seeds)`
29+
30+
- **bw_p99**: Bandwidth at 99% availability, normalized by baseline-offered (per seed), median across seeds.
31+
- What it is: Delivered bandwidth level met or exceeded in 99% of iterations, divided by baseline delivered (offered); capped at 1.0.
32+
- How it's calculated: Lower-tail quantile of total delivered series at q = 1 − 0.99; divide by total offered; cap at 1.0; then median across seeds.
33+
- How to interpret: Higher is better; tail reliability of delivered capacity.
34+
35+
- **bw_p999**: Bandwidth at 99.9% availability, normalized by baseline-offered (per seed), median across seeds.
36+
- What it is: Same as `bw_p99` with 99.9% reliability threshold.
37+
- How it's calculated: q = 1 − 0.999; divide by offered; cap at 1.0; then median across seeds.
38+
- How to interpret: Higher is better; stringent tail SLO.
39+
40+
- **bac_auc**: BAC AUC (area under curve) — mean of delivered/offered clipped to 1.0 across iterations (per seed), median across seeds.
41+
- What it is: Average normalized delivered capacity across iterations; equivalent to area under the availability curve (with uniform iteration weight).
42+
- How it's calculated: For each iteration, compute delivered/offered, cap at 1.0; average across iterations; median across seeds.
43+
- How to interpret: Higher is better; summarizes resilience beyond single tail points.
44+
45+
- **lat_base_p50**: Baseline latency stretch p50 (volume-weighted) per seed; median across seeds.
46+
- What it is: Typical latency stretch in the baseline (no failure) case.
47+
- How it's calculated: Compute per-flow stretch relative to baseline min-cost path, volume-weighted; take weighted p50; median across seeds.
48+
- How to interpret: Near 1.0 is ideal; if more than half the volume is above min-cost, the p50 would exceed 1.0.
49+
50+
- **lat_fail_p99**: Median across failure iterations (within a seed) of latency stretch p99; then median across seeds.
51+
- What it is: Tail latency stretch under failures.
52+
- How it's calculated: For each failure iteration, compute weighted p99 stretch; median across iterations; median across seeds.
53+
- How to interpret: Lower is better. Shows the close-to-worst-case latency stretch under failures.
54+
55+
- **lat_TD99**: Tail degradation at 99% = (failure p99) / (baseline p99), per seed; median across seeds.
56+
- What it is: Ratio of failure to baseline tail latency.
57+
- How it's calculated: `TD99_seed = fail_p99_seed / base_p99_seed`; then median across seeds.
58+
- How to interpret: 1.0 = parity; >1.0 worse tails under failure.
59+
60+
- **lat_SLO_1_2_drop**: Drop in share meeting SLO=1.2 stretch: baseline_share(≤1.2) − failures_share(≤1.2) per seed; median across seeds.
61+
- What it is: Lost fraction of traffic volume that meets 1.2× stretch SLO moving from baseline to failures.
62+
- How it's calculated: Compute volume share with stretch ≤1.2 in baseline and in failures (per-iteration median), then baseline − failures; median across seeds.
63+
- How to interpret: Closer to 0.0 is better.
64+
65+
- **lat_best_path_drop**: Drop in best-path share: baseline_best_share − failures_best_share per seed; median across seeds.
66+
- What it is: Reduction in volume fraction still on the baseline minimum-cost path under failures.
67+
- How it's calculated: Share equal to baseline min-cost in baseline and failures; baseline − failures; median across seeds.
68+
- How to interpret: Closer to 0.0 is better.
69+
70+
- **lat_WES_delta**: Weighted Excess Stretch delta: failures_WES − baseline_WES per seed; median across seeds.
71+
- What it is: Change in average excess stretch beyond 1.0, volume-weighted.
72+
- How it's calculated: `WES = E[(stretch−1)+]` with volume weights; failures median − baseline; median across seeds.
73+
- How to interpret: It shows average “extra stretch” beyond ideal (1.0) that traffic experiences, volume-weighted, under failures vs baseline. Closer to 0.0 is better; positive means worse stretch. It combines both how much and how often paths are longer than ideal. Small inflations on lots of traffic and rare big inflations both move it upward in proportion to severity and volume.
74+
75+
- **spf_calls_per_iter**, **flows_created_per_iter**, **reopt_calls_per_iter**: Median across seeds of per-iteration SPF calls, flows created, reoptimization calls.
76+
- What they are: Control-plane and placement effort proxies per iteration.
77+
- How they're calculated: From `iterops_summary.csv` per scenario; median across seeds.
78+
- How to interpret: Lower SPF/reopt generally better; flows created reflects demand fragmentation and might be a proxy for required number of LSPs to avoid stranded traffic or capacity.
79+
80+
- **tm_duration_total_sec**, **tm_duration_per_iter_sec**: tm_placement timing (seconds), total and per-iteration medians across seeds.
81+
- What they are: Runtime diagnostics for placement, dominated by SPF calls.
82+
- How they're calculated: From per-seed timing summaries; median across seeds.
83+
- How to interpret: Lower is better. In combination with the number of SPF calls, it shows the relative cost of the placement step.
84+
85+
- **USD_per_Gbit_offered**, **Watt_per_Gbit_offered**: Cost/Power divided by offered at alpha*; median across seeds.
86+
- What they are: Efficiency per offered Gbps at `alpha*`.
87+
- How they're calculated: `capex_total / (alpha_star * base_total_demand)`; similarly for watts.
88+
- How to interpret: Lower is better.
89+
90+
- **USD_per_Gbit_p999**, **Watt_per_Gbit_p999**: Cost/Power divided by reliable bandwidth at p99.9; median across seeds.
91+
- What they are: Dollars or watts per Gbps available with 99.9% probability.
92+
- How they're calculated: `capex_total / bw_at_probability_abs[99.9]`; similar for watts.
93+
- How to interpret: Lower is better.
94+
95+
## Project-level summaries
96+
97+
It is a text report generated from the project CSVs and includes:
98+
99+
- A table titled “Consolidated project metrics” with columns:
100+
- `seeds`, `node_count`, `link_count`, `alpha_star`, `bw_p99`, `bw_p999`, `bac_auc`, `lat_base_p50`, `lat_fail_p99`, `lat_TD99`, `lat_SLO_1_2_drop`, `lat_best_path_drop`, `lat_WES_delta`, `spf_calls_per_iter`, `flows_created_per_iter`, `reopt_calls_per_iter`, `tm_duration_total_sec`, `tm_duration_per_iter_sec`, `USD_per_Gbit_offered`, `Watt_per_Gbit_offered`, `USD_per_Gbit_p999`, `Watt_per_Gbit_p999`, `capex_total`.
101+
- A table titled “Baseline-normalized metrics (scenario / baseline)” with normalized ratios/deltas per scenario:
102+
- Ratios (`*_r`) vs 1.0: `bw_p99_pct_r`, `bw_p999_pct_r`, `auc_norm_r`, `USD_per_Gbit_offered_r`, `Watt_per_Gbit_offered_r`, `USD_per_Gbit_p999_r`, `Watt_per_Gbit_p999_r`, plus node/link ratios.
103+
- Deltas (`*_d`) vs 0.0: `lat_SLO_1_2_drop_d`, `lat_best_path_drop_d`, `lat_WES_delta_d`.
104+
- A section “All baseline-normalized comparisons vs target (mean, n, adj_p)” summarizing paired t-tests vs target (1.0 for ratios, 0.0 for deltas). Columns show mean value, sample size (common seeds), and Holm-adjusted p-values per scenario.
105+
106+
## Project-level figures
107+
108+
- **BAC.png**
109+
- What it shows: Cross-seed pooled BAC availability curves per scenario: A(x) = P(delivered ≥ x), x in % of offered. Includes IQR bands across seeds when ≥3 seeds.
110+
- How it’s computed: Pool normalized delivered samples across seeds from `seed*/bac.json`, form empirical availability (1 − CDF), and plot stepwise curves. See `metrics/plot_cross_seed_bac.py`.
111+
- How to read: Curves further upper-right are better. Narrow IQR band indicates stable behavior. Arguably, this is the most important figure for scenario comparison.
112+
113+
- **BAC_delta_vs_baseline.png**
114+
- What it shows: Availability improvement vs baseline (default 80–100% band). Positive = better than baseline at that delivered threshold.
115+
- How it’s computed: Pooled availability for each scenario and baseline are interpolated on a common grid; delta = scenario − baseline availability. See `metrics/plot_bac_delta_vs_baseline.py`.
116+
- How to read: Lines above zero indicate higher availability vs baseline; guides at 80/90/95% aid comparison.
117+
118+
- **Latency_p99.png**
119+
- What it shows: Cross-seed pooled latency exceedance curves for p99 stretch (baseline-referenced). Y = P(stretch > x) = 1 − CDF(stretch).
120+
- How it’s computed: From `seed*/latency.json` per-iteration p99 over failures; pool across seeds, compute exceedance (1 − CDF). IQR bands from per-seed curves on a common grid. See `metrics/plot_cross_seed_latency.py`.
121+
- How to read: For a fixed stretch x (e.g., 5), lower curve is better (smaller fraction of iterations exceed x). For a fixed tail probability y (e.g., 0.01), lower x is better (smaller p99-stretch at that exceedance).
122+
123+
- **effects_heatmap.png**
124+
- What it shows: Baseline-normalized effect sizes per metric per scenario, with significance markers for p < 0.05.
125+
- How it’s computed: From `normalized_insights.csv`, effect = mean − 1.0 for ratio metrics and mean − 0.0 for delta metrics; Holm-adjusted p-values used for annotation. See `metrics/plot_significance_heatmap.py`.
126+
- How to read: Red/blue indicates direction and magnitude vs baseline; dots mark statistically significant differences at p < 0.05.
127+
128+
## Other project artifacts
129+
130+
- Absolute distributions with per-seed jitter and scenario medians:
131+
- `abs_AUC.png` (BAC AUC), `abs_BW_p99.png`, `abs_Latency_fail_p99.png`, `abs_USD_per_Gbit_offered.png`, `abs_USD_per_Gbit_p999.png`, `abs_nodes.png`, `abs_links.png`, `abs_CapEx.png`.
132+
- Baseline-normalized distributions with per-seed jitter and scenario medians:
133+
- `norm_AUC.png`, `norm_BW_p99.png`, `norm_Latency_fail_p99.png`, `norm_USD_per_Gbit_offered.png`, `norm_USD_per_Gbit_p999.png`, `norm_nodes.png`, `norm_links.png`.
134+
- Cross-seed iteration ops summary:
135+
- `IterationOps.png` (SPF/iter, flows created/iter, reopt/iter) across scenarios.
136+
137+
## CSV artifacts
138+
139+
- `project.csv`: Consolidated project metrics table (one row per scenario). Columns include those described in Metrics.
140+
- `project_baseline_normalized.csv`: Per-scenario metrics normalized to baseline (ratios and deltas). Used for normalized plots.
141+
- `normalized_insights.csv`: Wide-format table of baseline-normalized comparisons vs target with columns per metric: `<metric>__mean`, `<metric>__n`, `<metric>__p`, plus `<metric>__p_adj` (Holm-adjusted within metric).
142+
- `project_per_seed_abs.csv`: Per-seed absolute metrics across scenarios (e.g., `auc_norm`, `bw_p99_pct`, `lat_fail_p99`, `USD_per_Gbit_offered`, `USD_per_Gbit_p999`, `capex_total`, `node_count`, `link_count`). Used for “abs_*.png” figures.
143+
- `project_baseline_normalized_per_seed.csv`: Per-seed baseline-normalized metrics (ratios `_r` vs 1.0, deltas `_d` vs 0.0) for non-baseline scenarios. Used for “norm_*.png” figures.
144+
145+
## Scenario and seed-level outputs
146+
147+
Within each scenario directory (e.g., `Clos_L16_S4/`):
148+
149+
- Scenario-level summaries: `alpha_summary.json`, `bac_summary.json`, `latency_summary.csv`, `iterops_summary.csv`, `costpower_summary.csv`, `network_stats_summary.csv`.
150+
- Scenario-level figures: `BAC.png`, `Latency_p99.png`, `IterationOps.png` (single-scenario views).
151+
152+
All per-seed artifacts are stored in the per-seed folder within the scenario directory.

‎metrics/plot_cross_seed_latency.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
1-
"""Cross-seed latency availability-style curves (publishable figure).
1+
"""Cross-seed latency exceedance curves (publishable figure).
22
33
Approach: pooled empirical (like BAC). For each seed, take per-iteration
44
latency tail (e.g., p99 stretch) across failure iterations, pool across seeds,
5-
and plot availability of "stretch ≤ x" (1 − CDF of stretch). Also show IQR band
6-
by computing seed-wise availability on a common x-grid.
5+
and plot exceedance probability P(stretch > x) = 1 − CDF(stretch). Also show an
6+
IQR band by computing seed-wise exceedance on a common x-grid.
77
88
Usage:
99
python3 -m metrics.plot_cross_seed_latency scenarios_metrics \
@@ -117,10 +117,10 @@ def plot_cross_seed_latency(
117117
ax.step(grid, sc, where="post", color=color, alpha=0.12, linewidth=0.6)
118118

119119
ax.set_xlabel("Latency stretch (× baseline)")
120-
ax.set_ylabel("Availability (stretch ≤ x)")
120+
ax.set_ylabel("Exceedance probability (stretch > x)")
121121
ax.set_ylim(0.0, 1.0)
122122
ax.legend(title="Scenario", loc="lower right", frameon=True)
123-
ax.set_title(f"Cross-seed latency availability (metric={metric})")
123+
ax.set_title(f"Cross-seed latency exceedance (metric={metric})")
124124
if save_to is not None:
125125
save_to = save_to.resolve()
126126
save_to.parent.mkdir(parents=True, exist_ok=True)

0 commit comments

Comments
 (0)