← ag_risk documentation

`twx-risk-severity-library` — explainer & how-to

What the library is · how calibration is performed · how upstream yield/other work feeds in · how to add and calibrate a curve

The HOW BAD catalogue: for a given peril, crop and place, it maps a peril's signal to a 0–1 severity on a shared loss ruler (1 = total attributable loss). Sibling of twx-phenoweight (the WHEN). It stores and resolves finished curves — it does not fit; shapes come from literature or consequence data upstream, and calibration re-anchors a shape onto a region's data so a pipeline head scores directly.

The plain-language part is sections 1–2. Sections 3–5 are the technical detail (calibration, upstream feed-in, adding a curve); 6 is the guardrails, 7 the gaps. Deep reference lives in the repo: README.md, docs/CALIBRATION.md, docs/ONSET_CUMSUM_REANCHOR.md, and the full docs/EXPLAINER.md this page mirrors.

1. What the library is

The model reads signal → phenoweight (WHEN) → severity (HOW BAD) → combine. This library is the HOW-BAD box: one shared, versioned, installable place that turns a peril's dose into a 0–1 severity on a common loss ruler. ag_risk is its first consumer.

Three things it is deliberately not:

Not all damage curves. A severity function is produced by one of three methods: curve (dose-response / damage curve), fraction (observed affected-area footprint), or return_period (probabilistic exposure). Only curve is a damage curve. There is no "model emits severity" method — a detection model is a data source whose output becomes a curve upstream.
It does not fit at read time. It stores finished, ruler-native curves and resolves the right one; harmonisation and consequence-calibration happen upstream (or in the opt-in offline build step) and deposit a finished curve.
It does not decide WHEN a hazard matters — that is twx-phenoweight.

Every entry carries three independent axes (none implies another):

Axis	Values	Means
verdict	`Directional` · `Exposure` · `model`	How the number was derived
status	`literature-anchored` → `Validated`	How well-grounded it is
scope	`global` → `ADM0` → `ADM2` → `location`	How specific it is

At run time the resolver returns the most-specific entry for the location and falls back up to the global curve, the guaranteed catch-all.

2. What's in the catalogue today

Version 0.9.0. Twelve curves ship as YAML in src/severity_library/data/, all currently scope: global, covering maize (the v1 dimension set) and coffee. The key distinction when wiring a config: some maize curves are head-anchored and some are physical-axis, and the two are not interchangeable — the wrong one silently produces wrong severities.

Head-anchored curve	metric	dimension	notes
`maize_heat_edd_schlenker`	edd	Heat	Schlenker & Roberts EDD
`maize_hot_nights_bayer_5level`	hot_days_min	Heat
`maize_water_deficit_flux`	water_deficit	Drought	v1 PROVISIONAL shape
`maize_cuml_rain_bayer_5level`	cuml_rain	Drought	decreasing (rainfall supply); ratio-to-normal axis
`maize_dry_spell`	dry_spell	Drought	v1 PROVISIONAL shape
`maize_waterlogging_surplus`	precip_excess	Water	Ren waterlogging
`maize_frost`	frost	Cold	UNCALIBRATED (~nil frost in KE)

Physical axis (consume via the derived/raw dose): maize hot_days_max, consecutive_roasting_days, ndvi_condition; and all coffee curves.

The machine-readable marker is calibration.method: onset_cumsum_reanchor or onset_cumsum_ratio_to_normal = head-anchored; absent = physical. A curve not fit to score can carry a gated reason — resolve() then raises GatedCurveError rather than returning it, while load_catalog() still loads it so the gate stays inspectable.

3. Where a curve comes from — literature vs consequence data

There is no single required source of a dose→loss mapping; the library carries the best available per peril and place. In rough order of strength:

Directional Literature / agronomy. A published dose-response or expert judgement (Schlenker EDD, Ren waterlogging, Bayer thresholds, FAO). A legitimate, production-usable basis — often the only one, and not a placeholder.
Validated Consequence-calibrated. When you have outcome data, fit the dose to an actual consequence: observed/modelled yield (gold standard), feature importance / SHAP from a trained loss model, futures / financial data, or any custom local dataset.
v1 PROVISIONAL Provisional ramp. Only when you have neither: a documented placeholder shape, flagged, wanting any real anchor. "Provisional" ≠ "literature".

How upstream yield / MI work feeds in. This library does not run those models — it is the deposit point. An MI activity (yield model, hindcast, EO/detection model, financial study) produces a dose→consequence relationship; that becomes a curve shape (fp); it is harmonised onto the 0–1 ruler and deposited here with verdict: Validated, its source/citation, and a scope at the level the data supports (usually ADM0 or finer). Because resolution is most-specific-wins, that calibrated regional entry automatically overrides the global literature curve where it applies, with the literature curve remaining the fallback elsewhere. Validation is an upgrade per peril and place — never a gate every curve must pass.

4. How calibration is currently performed

"Calibration" here has a narrow meaning, separate from "where the shape came from". Every head-consumed curve is built in two steps:

Shape (fp) — the form of the dose→loss (section 3).
Calibration (xp) — re-anchor that shape onto the region's actual head axis so a pipeline head scores directly. Mechanical, reproducible, and every head curve needs it. Calibration is not an alternative to literature; it is the step after the shape.

Why it's needed: ag_risk consumes the weather pipeline's normalized, season-to-date maize_<peril>_phenoweight_onset_cumsum heads, which live on a different, region-specific scale from the literature's physical units (the EDD head tops ~5–8 vs Schlenker's 0–120 °C-days — a ~15× compression). Without moving the axis, every pixel pegs at severity 1.0. Calibration is offline and opt-in (pip install -e '.[calibrate]'); runtime consumers never pull those deps.

4a. Default method — `onset_cumsum_reanchor`

xp_head[i] = base_anchor + original_xp[i] × (anchor − base_anchor) / original_xp_at_top

anchor — the season-end head the curve's top breakpoint maps to, a percentile (not max; heads are heavy-tailed) of the per-pixel-year season-end head over the region's realised climatology. P99.9 for heat/water, P99 for drought (P99.9 sat in the single-pixel tail and mapped the body to ~0).
base_anchor — the head that dose 0 maps to; 0.0 for an increasing curve.
fp is unchanged, including direction. Orientation lives in fp, never in xp — the head axis must stay increasing.

calibrate.season_end_anchor takes the per-pixel-per-year max of the cumulative head, then the requested percentile across pixel-years; its tail (high/low) picks which end and therefore what the anchor means.

4b. The decreasing (supply) case

maize_cuml_rain is the one decreasing curve: despite the "precip_deficit" head name it is phenoweight(base_signal=precip) — accumulated rainfall supply — so fp runs 1 → 0 and total loss sits at the bottom of the distribution. Read a head's base_signal, never its name. Directions are pinned in tests/test_orientation.py.

4c. When a pooled anchor can't work — `onset_cumsum_ratio_to_normal`

A pooled percentile only anchors a head whose variance is seasonal. If spread is mostly climatological (dry pixels always low, wet always high), the pooled P99 is "a wet pixel", not "a wet season". Check first with calibrate.variance_decomposition(...). maize_cuml_rain is the worked example — ~81% of its variance is between pixels — so the fix is to change the axis: divide the season-end head by that pixel's own baseline normal and score the dimensionless ratio (1.0 = normal). This makes the curve legitimately scope: global (portable to TZ/ZM) and absorbs multiplicative forecast bias.

As of v0.9.0 the per-pixel reference is produced offline by calibrate.climatology_normal and read by the consumer as a static input (like a crop mask) — not recomputed at scoring time (a forecast store carries one vintage, not a 30-year climatology, so it can't be). The reference is the mean of per-year month-end head values over the baseline, on the same crop-AND-plausible pixel set as the anchors, and is matched by position in season, not calendar month (a season_month axis — a normal season reads ≈1.0 every month; dividing a season-to-date dose by a season-end reference would make early months read catastrophic). The consumer still applies the ratio and cross-checks intent: the curve declares calibration.method: onset_cumsum_ratio_to_normal, the consumer declares normalise: climatology_ratio (verified in check_coverage). A mismatch must fail the dry run.

4d. How the Kenya maize curves were actually built

Shape. cuml_rain reused the Bayer 5-level shape; water_deficit the waterlogging shape; dry_spell a 5-level ramp (the two with no literature form are v1 PROVISIONAL).
Calibrate via the reproducible driver, over crop pixels only:

python scripts/calibrate_maize.py \
    --store s3://.../weather-dagster/processed/indices/kenya_gridded_maize.zarr \
    --crop-calendar s3://.../commodity-growth-stages/v1/processed/gridded/ggcmi \
    --adm0 KEN --baseline 1980-2026 --profile platform-qa   # print-only; add --write to emit YAMLs

The anchor, pixel set and provenance are stamped into each curve's calibration: block.

5. Adding a curve (and calibrating it)

5a. Any curve — the common steps

Pick method & axis (curve/fraction/return_period; physical vs head-anchored).
Get the shape (fp) — literature, consequence data (→ Validated), or a flagged provisional ramp.
Write the YAML into src/severity_library/data/ as <crop>_<peril-or-metric>[_<variant>].yaml:

crop: maize
peril: heat
metric: edd
mode: curve                 # curve | fraction | return_period
interp: pchip               # linear | pchip
verdict: Directional        # Directional | Exposure | model
status: literature-anchored
scope: {level: global}      # or {level: ADM0, id: KEN}
source: literature
citation: <study / model run>
unit: <units of xp>
label: <human-readable label>
curve:
  xp: [0.0, 30, 60, 90, 120]
  fp: [0.0, 0.25, 0.5, 0.75, 1.0]

Ensure a global fallback exists for the (crop, peril, metric) or the resolver raises KeyError; regional entries are ADM0/ADM2 overrides.
Bump the version in pyproject.toml — any change to a scored severity means consumers re-pin deliberately.

5b. Head-anchored curves — the extra calibration steps

Check the head's variance first — variance_decomposition(store, head_var, baseline=(1991,2020)). Between-pixel dominates → use ratio-to-normal, not a pooled anchor.
Confirm orientation from base_signal, and add a row to tests/test_orientation.py.
Anchor over CROP pixels (and, for precip heads, plausible crop pixels) — season_end_anchor(store, head_var, percentile=…, tail=…, mask=…).
Produce the anchored curve — reanchor_curve(...) (pooled) or build.onset_cumsum_ratio_to_normal (climatological); the calibration block records method, head_var, anchor(s), original xp/unit and pixel set.
Gate it with a gated: reason if not yet fit; clear once it is.

5c. Calibrating for a new country (Tanzania, Zambia)

When a country's realised store lands, run the driver per country to emit ADM0-scoped curves:

python scripts/calibrate_maize.py \
    --store s3://.../<country>_gridded_maize.zarr \
    --crop-calendar s3://.../commodity-growth-stages/v1/processed/gridded/ggcmi \
    --adm0 TZA --baseline <window> --profile platform-qa   # then --write

A head axis is region-specific, so a KE-anchored curve mis-scores TZ/ZM — each country gets its own scope: {level: ADM0, id: <ISO3>} entry with the global literature curve as fallback. The exception is a ratio_to_normal curve (cuml_rain): dimensionless, so legitimately global and portable without a per-country refit.

6. Guardrails / gotchas

Never mix axes — head-anchored vs physical; the calibration.method marker is the discriminator.
Orientation lives in fp, read from base_signal — cuml_rain shipped increasing for two versions and scored the wettest seasons as most drought-stressed.
Anchor over crop pixels (plausible crop pixels for precip heads) — structural zeros and implausible highs both corrupt an anchor.
A pooled percentile only anchors a seasonal head — run variance_decomposition first.
Re-anchor when a head's base_signal/formula changes upstream — the anchor goes stale silently even if the name is unchanged.
PCHIP holds the endpoint outside the knots (fixed in 0.4.0) — it previously dived below zero, reading total loss as no loss.

7. What is not built yet

CLI is a stub — severity-library raises NotImplementedError; planned check · show · plot · validate-lineage · coverage not implemented.
Scope resolution has no geometry yet — all curves global; the lat/lon → ADM spatial join (the geo extra) activates once regional overrides are added.
fraction is single-input — assumes conditional-damage = 1; the fraction × conditional-damage upgrade is a TODO.
Validation — head curves' absolute severity climatology is pending yield validation; several shapes are v1 PROVISIONAL.

Source of truth is the repo (twx-risk-severity-library): README.md, docs/EXPLAINER.md, docs/CALIBRATION.md, docs/ONSET_CUMSUM_REANCHOR.md. This page mirrors docs/EXPLAINER.md.

twx-risk-severity-library — explainer & how-to