Kenya maize — validation

How a Directional risk model is validated with no yield data — and the Tier-1 result on real ERA5 · 2026-07-16

ag_risk emits a Directional risk score — calibrated to published dose-response, not fitted to local yield — so "skill against yield" can't be the test. Instead we validate against independent proxies on a ladder: does the model light up in documented bad years, in the right places and months, and does it co-move with outcome proxies (vegetation greenness, reported production, food-security classes)? The Tier-1 face-validity gate PASSES on real ERA5 (2010–2024), across perils.

1. What we validate, and how — with no yield

Because the score is Directional, we don't measure error against observed yield. A credible Directional model must instead clear a ladder of proxy checks, each using references that exist for Kenya without yield: FEWS NET / IPC drought classifications, FAO / KNBS maize production, MODIS NDVI/EVI anomalies, and the documented event record.

Light up in documented bad years and stay low in good years — face validity.
Hit the right places and months — spatial / temporal agreement.
Co-move with independent outcome proxies — greenness, production, food-security class.

2. Tier-1 — the known-bad-year gate PASSED

Method. Ran Kenya maize on the real ERA5 gridded feed for 2010–2024 (15 seasons), scored the weather dimensions, and ranked each year by season-mean drought over the crop belt (n = 758 crop pixels). No thresholds are fitted — the test is whether the years separate and the documented events surface at the top.

Season-mean drought z-score by year. The three documented droughts sort to the top; the two documented good-rains years sit ~2σ below the mean.

documented drought elevated near-mean documented good / wet

rank	year	drought	z	note
1	2011	0.818	+1.28	Horn of Africa drought / famine
2	2017	0.817	+1.27	failed 2017 long rains
3	2022	0.804	+1.07	2020–23 multi-season drought
4	2014	0.781	+0.73
5	2019	0.770	+0.56	poor/late 2019 long rains
6	2023	0.768	+0.54
7	2021	0.756	+0.36	2020–23 sequence (bad, not worst)
8	2016	0.730	−0.02
9	2015	0.724	−0.11
10	2012	0.719	−0.18
11	2010	0.717	−0.22
12	2024	0.697	−0.50
13	2013	0.691	−0.61
14	2018	0.592	−2.07	good long rains (post-2017 recovery)
15	2020	0.590	−2.09	decent long rains

Interpretation. The top-3 drought years are 2011, 2017 and 2022 — all documented Kenya droughts — and the two lowest (2018, 2020) are known good-rains years. Known-bad years sit +1 to +1.3σ above the mean; good years −2σ below. The model catches the real events and the years separate cleanly. Face validity holds.

2.1 The other perils

The gate extends to all four dimensions — with an internal-consistency cross-check that doesn't depend on any external reference.

Water (waterlogging) passes — and it's the strongest cross-check. Ranked by season-mean water (excess), the top two years are 2020 (+2.14σ) and 2018 (+1.45σ) — both documented wet/flood years — and the bottom are the dry drought years (2011, 2014, 2022). Crucially, drought and water (waterlogging) are anti-correlated across years at r = −0.89: wet years are simultaneously low-drought and high-water (excess), drought years the reverse. The two water perils behave physically opposite, correctly — a strong consistency signal, independent of the drought reference.

Cold (frost) correctly ~0 — nothing to rank. Frost reads 0.000 in every year, everywhere, which is right: the equatorial Kenya maize belt (~1500–2000 m) doesn't frost. Kenya's frost "validation" is that it correctly reads zero; to stress-test the frost curve itself needs a frost-prone region (SH highlands / Zambia), not Kenya.

Heat suggestive, weaker. Heat years are rankable, but Kenya heat-stress events aren't cleanly documented and heat partly co-varies with drought (hot-and-dry) — so 2017/2011 read hot while 2022 does not. Treat heat's gate as suggestive, not conclusive; it needs a heat-specific reference (or a hotter region) to validate firmly.

3. The validation ladder

Tier	Check	Reference	Status
1	Known-bad-year gate (per peril) — bad years rank high, good low	droughts · wet years · frost≈0	✓ passed drought + water (waterlogging) strong; heat weak
2	Spatial / temporal — right regions + right months	agro-ecological boxes · drought-yr vs wet-yr contrast	✓ implemented runs now (`spatial.py`)
3	Proxy-consequence agreement — risk rank-correlates with an outcome	FAO/KNBS production · NDVI · FEWS NET/IPC	ready production coded; needs the FAOSTAT CSV filled (NDVI data-gated)
4	Yield skill — LOYO vs detrended yield, ROC/AUC	GDHY / FAOSTAT yield	later needs yield + ADM roll-up

How the tiers run in code. Validation is a package — validation/ — with a rerunnable driver that scores a span of seasons then runs Tier-2 + Tier-3, writing a summary JSON + a comparison PNG. Tier-1/3 (correlation.py) reduce each season's risk zarr to one scalar (crop-masked spatial mean → season peak) and Spearman rank-correlate it against a continuous "healthier = higher" consequence — a detrended FAO/KNBS production anomaly (reference.py) now, a MODIS NDVI anomaly next, and eventually (Tier-4) a detrended-yield anomaly through the same function. Tier-2 (spatial.py) aggregates the season-peak field over agro-ecological boxes and contrasts a drought year vs a wet year. A tracking model gives a clearly negative rho (worse weather → higher risk → lower greenness/yield); there is deliberately no pass/fail threshold — a human reads the rho and n. Tier-3 reports skipped, with reason until data/kenya_maize_production.csv is filled from FAOSTAT.

4. Run provenance

Item	Value
Feed	ERA5 gridded indices — `s3://…/weather-dagster/…/kenya_gridded_maize.zarr` (1980→2026, 0.25°)
Config	`kenya_maize`, weather-only, native two-driver drought (demand + store), static planting DOY 105 (~mid-April, held constant across years for a clean comparison)
Mask	WorldCover cropland stopgap (over-inclusive; maize-specific upgrade queued)
Metric	per-year season-mean drought averaged over crop pixels (`n = 758`)
Combine	weight-free soft-max, `softmax_p = 3`

Caveats — stated, not hidden. The absolute drought values are high (0.59–0.82), inflated by the over-inclusive WorldCover cropland mask (not the tight maize belt) and the two-driver max(demand, store) drought — immaterial for a ranking validation (the year-to-year signal is what's tested, and it's clean); a maize-specific mask (WorldCereal / MapSPAM) will sharpen the absolute belt values. This is proxy / face-validity, not skill-against-yield. And the gate is per-peril: drought + water (waterlogging) validate well, frost is trivially ~0 for Kenya, heat is weaker.

5. Verdict

A Directional Kenya maize model whose Tier-1 face-validity gate passes on real ERA5, across perils: it flags the documented 2011 / 2017 / 2022 droughts as its worst drought years, the 2018 / 2020 wet years as its worst water (waterlogging) years, and — the strongest single check — drought and water (waterlogging) are anti-correlated at r = −0.89. Frost correctly reads ~0 (equatorial Kenya); heat is suggestive but weaker. Remaining work is proxy-agreement (NDVI / FEWS / production), the spatial check, and eventual yield skill (§3). Curves stay Directional; validation, not calibration, carries the beta (calibration is a later workstream — see docs/VALIDATION.md).

Source: model_foundry/models/ag_risk/docs/VALIDATION.md (commit 60ce56de) · method: the model_foundry/models/ag_risk/validation/ package (correlation · spatial · reference · driver, commit 66754930). Tier-1 gate over 2010–2024 real ERA5.