Skip to content

Treat NaN as absent in v1 arithmetic #627

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
FBumann wants to merge 6 commits into
base: harmonize-linopy-operations-mixed
Choose a base branch
from
Draft

Treat NaN as absent in v1 arithmetic #627

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e7b7dc2
Treat NaN as absent in v1 arithmetic instead of raising ValueError
FBumann Mar 18, 2026
6c33232
Fix Variable * NaN inconsistency, restore constraint RHS NaN raise
FBumann Mar 18, 2026
e9a5a40
Show .sel() masking approach in missing-data notebook demo
FBumann Mar 18, 2026
099277b
Add mask= approach alongside .sel() in constraint NaN demo
FBumann Mar 18, 2026
833f346
Split constraint NaN demo into separate cells
FBumann Mar 18, 2026
42a8855
Require fill_value when alignment introduces NaN in mul/div
FBumann Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
382 changes: 195 additions & 187 deletions examples/arithmetic-convention.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

206 changes: 78 additions & 128 deletions examples/missing-data.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

296 changes: 296 additions & 0 deletions examples/mixed-coordinate-arithmetic.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,296 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "2d7hs5iptwn",
"metadata": {},
"source": "# Mixed-Coordinate Arithmetic\n\nA common pattern in energy modeling: variables cover **different subsets** of a shared dimension, but cost parameters span the full set. This notebook shows how to combine them cleanly under the v1 arithmetic convention.\n\n**Scenario:** Three capacity variables (`cap_a`, `cap_b`, `cap_c`) cover different technology subsets. Cost coefficients are defined over all technologies. We want a single cost expression over the union of technologies."
},
{
"cell_type": "code",
"id": "vnmxvu41lk",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:11:58.624527Z",
"start_time": "2026-03-18T15:11:58.002434Z"
}
},
"source": [
"import xarray as xr\n",
"\n",
"import linopy\n",
"\n",
"linopy.options[\"arithmetic_convention\"] = \"v1\""
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "2bvmgbym644",
"metadata": {},
"source": "## Setup\n\nThree technology groups with overlapping cost data:"
},
{
"cell_type": "code",
"id": "3fe7y8gn5a2",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:11:58.665108Z",
"start_time": "2026-03-18T15:11:58.627517Z"
}
},
"source": [
"m = linopy.Model()\n",
"\n",
"tech_a = [\"wind\", \"solar\"]\n",
"tech_b = [\"gas\"]\n",
"tech_all = [\"wind\", \"solar\", \"gas\"]\n",
"\n",
"cap_a = m.add_variables(lower=0, coords=[tech_a], dims=[\"tech\"], name=\"cap_a\")\n",
"cap_b = m.add_variables(lower=0, coords=[tech_b], dims=[\"tech\"], name=\"cap_b\")\n",
"cap_c = m.add_variables(lower=0, coords=[tech_all], dims=[\"tech\"], name=\"cap_c\")\n",
"\n",
"# Cost parameters span all technologies — NaN where a variable doesn't apply\n",
"cost_a = xr.DataArray([7, 9, float(\"nan\")], coords=[(\"tech\", tech_all)])\n",
"cost_b = xr.DataArray([float(\"nan\"), float(\"nan\"), 11], coords=[(\"tech\", tech_all)])\n",
"cost_c = xr.DataArray([13, 17, 19], coords=[(\"tech\", tech_all)])"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "8j05vog5umk",
"metadata": {},
"source": "## Approach 1: `fillna(0)` + explicit joins\n\nThe most explicit approach. Since `cost_a` has NaN at \"gas\" (where `cap_a` doesn't exist), fill NaN with 0 before multiplying. Use `join=\"left\"` so the product keeps only the variable's coordinates, then `join=\"outer\"` when adding to build the union.\n\n`fillna(0)` on a cost means \"this technology has no cost contribution from this variable\" — a safe, intentional choice."
},
{
"cell_type": "code",
"id": "biw39h6a1e",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:11:58.688387Z",
"start_time": "2026-03-18T15:11:58.667722Z"
}
},
"source": [
"combined = (\n",
" cap_a.mul(cost_a.fillna(0), join=\"left\")\n",
" .add(cap_b.mul(cost_b.fillna(0), join=\"left\"), join=\"outer\")\n",
" .add(cap_c.mul(cost_c, join=\"left\"), join=\"outer\")\n",
")\n",
"combined"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "3z14qxy8l2g",
"metadata": {},
"source": "Expected result:\n```\n[gas]: +11 cap_b[gas] + 19 cap_c[gas]\n[solar]: +9 cap_a[solar] + 17 cap_c[solar]\n[wind]: +7 cap_a[wind] + 13 cap_c[wind]\n```"
},
{
"cell_type": "markdown",
"id": "u3bhml209b9",
"metadata": {},
"source": "## Approach 2: `dropna()` on costs first\n\nInstead of filling NaN with 0, drop the irrelevant entries from the cost arrays. Then multiply with `join=\"left\"` (the variable's coords are always a subset of the cost's coords after dropping), and combine with `join=\"outer\"`."
},
{
"cell_type": "code",
"id": "hb8n0uzb1u",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:11:58.726974Z",
"start_time": "2026-03-18T15:11:58.698597Z"
}
},
"source": [
"combined_v2 = (\n",
" cap_a.mul(cost_a.dropna(\"tech\"), join=\"left\")\n",
" .add(cap_b.mul(cost_b.dropna(\"tech\"), join=\"left\"), join=\"outer\")\n",
" .add(cap_c.mul(cost_c, join=\"left\"), join=\"outer\")\n",
")\n",
"combined_v2"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "jw46qdqpzhg",
"metadata": {},
"source": "## Approach 3: Scope costs to each variable upfront\n\nThe cleanest option when you control the data: define costs only over the relevant technologies from the start, eliminating NaN entirely."
},
{
"cell_type": "code",
"id": "311s75nab7q",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:11:58.753074Z",
"start_time": "2026-03-18T15:11:58.732344Z"
}
},
"source": [
"# Costs scoped to each variable's technologies — no NaN needed\n",
"cost_a_scoped = xr.DataArray([7, 9], coords=[(\"tech\", tech_a)])\n",
"cost_b_scoped = xr.DataArray([11], coords=[(\"tech\", tech_b)])\n",
"cost_c_scoped = xr.DataArray([13, 17, 19], coords=[(\"tech\", tech_all)])\n",
"\n",
"combined_v3 = (\n",
" (cap_a * cost_a_scoped)\n",
" .add(cap_b * cost_b_scoped, join=\"outer\")\n",
" .add(cap_c * cost_c_scoped, join=\"outer\")\n",
")\n",
"combined_v3"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "7su6bb0lk2o",
"metadata": {},
"source": "## Approach 4: Pre-align with `linopy.align()`\n\nUse `linopy.align()` to reindex all variables and cost arrays to the same coordinates upfront. After alignment, all operands share the same `tech` dimension, so arithmetic uses exact matching — no per-operation `join=` needed.\n\nVariables get absent slots at coordinates they don't cover; cost arrays get NaN. Since NaN in a multiplicative constant acts as a mask, the NaN entries naturally produce absent terms — no `fillna` needed."
},
{
"cell_type": "code",
"id": "azddqkp858",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:11:58.782598Z",
"start_time": "2026-03-18T15:11:58.761871Z"
}
},
"source": [
"# Align all variables and costs to the union of tech coordinates\n",
"cap_a_al, cap_b_al, cap_c_al, cost_a_al, cost_b_al, cost_c_al = linopy.align(\n",
" cap_a, cap_b, cap_c, cost_a, cost_b, cost_c, join=\"outer\"\n",
")\n",
"\n",
"# NaN in costs naturally masks — no fillna needed!\n",
"combined_v4 = cap_a_al * cost_a_al + cap_b_al * cost_b_al + cap_c_al * cost_c_al\n",
"combined_v4"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "tnpb928aup",
"metadata": {},
"source": "---\n\n## Adding a partial scaling factor\n\nNow extend the example: a `rate` parameter applies only to gas technologies. We want `cap_c * cost_c * rate`, where `rate` defaults to 1 for technologies it doesn't cover.\n\nExpected result — same as before, but the gas entry for `cap_c` is scaled by 1.04:\n```\n[gas]: +11 cap_b[gas] + 19.76 cap_c[gas]\n[solar]: +9 cap_a[solar] + 17 cap_c[solar]\n[wind]: +7 cap_a[wind] + 13 cap_c[wind]\n```"
},
{
"cell_type": "code",
"id": "p92dqoyi8d",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:11:58.788715Z",
"start_time": "2026-03-18T15:11:58.787060Z"
}
},
"source": [
"rate = xr.DataArray([1.04], coords=[(\"tech\", [\"gas\"])])"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "qah2n8cbiic",
"metadata": {},
"source": "### Option A: `fill_value=1` on `.mul()`\n\nThe `fill_value` parameter tells linopy what to use for technologies not covered by `rate`. Since `rate` is a scaling factor, `1` is the natural identity — \"no scaling\"."
},
{
"cell_type": "code",
"id": "8pw3s5xra62",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:14:37.549415Z",
"start_time": "2026-03-18T15:14:37.516481Z"
}
},
"source": "combined_rate_a = (\n cap_a.mul(cost_a.fillna(0), join=\"left\")\n .add(cap_b.mul(cost_b.fillna(0), join=\"left\"), join=\"outer\")\n .add(\n cap_c.mul(cost_c, join=\"left\").mul(rate, join=\"left\", fill_value=1),\n join=\"outer\",\n )\n)\ncombined_rate_a",
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "5sob69uofr5",
"metadata": {},
"source": "### Option B: Prepare the parameter with xarray first\n\nPre-multiply the cost and rate arrays using standard xarray operations before passing to linopy. This keeps the linopy arithmetic simple."
},
{
"cell_type": "code",
"id": "rtyit39tuj",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:11:58.835196Z",
"start_time": "2026-03-18T15:11:58.815621Z"
}
},
"source": [
"# Extend rate to all techs (fill with 1 = no scaling), then multiply with cost\n",
"cost_c_rated = cost_c * rate.reindex(tech=tech_all).fillna(1)\n",
"print(\"cost_c_rated:\", cost_c_rated.values) # [13, 17, 19.76]\n",
"\n",
"combined_rate_b = (\n",
" cap_a.mul(cost_a.fillna(0), join=\"left\")\n",
" .add(cap_b.mul(cost_b.fillna(0), join=\"left\"), join=\"outer\")\n",
" .add(cap_c * cost_c_rated, join=\"outer\")\n",
")\n",
"combined_rate_b"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "e7k602xaqwc",
"metadata": {},
"source": "---\n\n## NaN as mask\n\nUnder the v1 convention, NaN in a multiplicative constant **masks** the corresponding term — the position becomes absent. This means you can use NaN-containing cost arrays directly with `join=\"left\"` and the NaN entries will naturally drop out:"
},
{
"cell_type": "code",
"id": "tymb0e9grj",
"metadata": {
"ExecuteTime": {
"end_time": "2026-03-18T15:11:58.857705Z",
"start_time": "2026-03-18T15:11:58.838815Z"
}
},
"source": [
"# NaN in cost_a at \"gas\" naturally masks cap_a at \"gas\" — no fillna needed!\n",
"combined_nan_mask = (\n",
" cap_a.mul(cost_a, join=\"left\")\n",
" .add(cap_b.mul(cost_b, join=\"left\"), join=\"outer\")\n",
" .add(cap_c * cost_c, join=\"outer\")\n",
")\n",
"combined_nan_mask"
],
"outputs": [],
"execution_count": null
},
{
"cell_type": "markdown",
"id": "5nw136646y2",
"metadata": {},
"source": "---\n\n## Summary of patterns\n\n| Situation | Solution |\n|---|---|\n| Cost array has NaN for irrelevant techs | NaN in mul/div **masks** the term (makes it absent) — no cleanup needed |\n| NaN in additive constant | NaN treated as 0 (additive identity) — no cleanup needed |\n| Variables have different coord subsets | Use `.add(..., join=\"outer\")` to build the union |\n| Pre-align all operands | `linopy.align(*vars, *costs, join=\"outer\")`, then use `+`/`*` directly |\n| Multiplication with matching coords | `var * cost` (exact match, no join needed) |\n| Multiplication with superset cost | `var.mul(cost, join=\"left\")` to keep var's coords |\n| Partial scaling factor (e.g., rate for some techs) | `expr.mul(rate, join=\"left\", fill_value=1)` |\n| Partial scaling factor (alternative) | Pre-compute `cost * rate.reindex_like(cost).fillna(1)` in xarray |\n\n**Key principle:** NaN in multiplicative constants means \"no term here\" (absent). NaN in additive constants means \"no contribution\" (zero). For scaling factors where missing means \"identity,\" use `fill_value=1` explicitly."
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading
Loading