Data sources & the wider-MI fit

How external data / models feed the risk framework, what new dimensions they unlock, and how it sits alongside Market Insights · 2026-07-16

The framework is source-agnostic: anything that emits a field on the grid — a pipeline, an EO product, a detection model — can become a dimension, once its output is turned into a severity curve. This doc maps what the tf_environmental_factors_preparation GEE workflow can feed in (by severity-curve role), the new dimensions it unlocks beyond weather, and — as an open review question — how the approach sits alongside MI's own yield / area models.

1. How it wires into MI — the live core + the upstream feed

The live v1 path is deliberately simple and self-contained (solid below): weather feeds plus the two libraries — twx-phenoweight (WHEN) and twx-risk-severity-library (HOW BAD) — drive the config-driven engine to gridded scores. The wider MI interaction sits upstream (dashed, blue): MI's yield / hindcast models and the EO / anomaly work do the feature-importance + calibration, and that output becomes a curve deposited into the HOW-BAD library — the engine never scores from a model directly. It's drawn dashed on purpose: this feed is the roadmap, not wired in v1 — today's curves are literature-anchored, and the MI→library loop is designed but not yet built.

Solid = live v1 (self-contained scoring). Dashed blue = upstream MI: feature-importance + calibration turn MI's models and EO work into curves that land in the HOW-BAD library — the engine consumes the curve, never the model. The dashed arrow is the roadmap; v1 curves are literature-anchored.

2. What the GEE workflow produces, by severity-curve role

tf_environmental_factors_preparation (the gee_prep toolkit) pulls Sentinel-1/2, MODIS, ERA5-Land, SMAP, Landsat, CHIRPS, GRIDMET and more, and emits gridded feature fields. Mapped to how each could feed a severity curve:

Factor (source)	Role
LST · EDD · GDD · VPD · CDD (MODIS/VIIRS, GRIDMET, ERA5-Land)	dose → damage curve heat / cold
Precip · SPI / SPEI / EDDI / PDSI · SMAP soil moisture · ESI = 1−ET/PET (CHIRPS, gridmet_drought, SMAP, OpenET/MOD16/WaPOR)	dose → damage curve drought / water
VHI = 0.5·VCI + 0.5·TCI (MODIS NDVI + LST)	dose + threshold → exposure fraction
Flood mask (Sentinel-1 backscatter drop − JRC permanent water)	exposure fraction (new)
Burn ratio NBR / dNBR (Landsat)	exposure (fire — building block, not built)
Pest/disease z-score (red-edge NDVI + LST anomaly)	dose (screening-grade, low confidence)
NDVI · EVI · NDWI · LAI / FPAR · phenology (SOS/POS/EOS)	consequence / validation — outcome proxy, not a driver
Soil (SoilGrids/OpenLandMap) · terrain (SRTM)	susceptibility modifiers (could scale a curve), not doses
Land cover / crop mask (CDL, WorldCover, Dynamic World) · AlphaEarth embeddings	masks / denominators / ML features — not risk doses

3. New dimensions it unlocks (beyond ag_risk's weather-only v1)

ag_risk today scores four weather dimensions (drought / heat / cold / water). The temperature/precip families here largely overlap those. The genuinely new ones:

Flood (exposure) — observed Sentinel-1 inundation footprint, distinct from the modelled water (waterlogging) dimension.
Observed soil moisture (SMAP) — a measured water-store signal, complementing ERA5's modelled soil water.
Evaporative stress (ESI = 1−ET/PET) — a distinct water-stress dose (OpenET / MOD16 / WaPOR).
VHI stressed-fraction — a packaged vegetation+thermal drought severity with an established threshold ladder.
Disease / pest proxy — a new biotic dimension (red-edge/LST anomaly), screening-grade only.
Fire / burn severity — NBR exists as a building block; dNBR burn severity would need building.
Vegetation condition (NDVI etc.) — the key consequence/validation signal ag_risk's weather-only model has no equivalent of; powerful for calibrating/validating curves against real outcomes.

Scope note: this GEE env-factors path (fire / flood / disease — the EO dimensions) is not part of the weather-only v1 / Model Foundry production model. It lives on a separate testbed branch for testing and expansion; the weather-only v1 shipped in Model Foundry does not depend on it.

A ready severity-engine template already exists in that repo. The Vietnam-coffee cascade (anomaly.py) already turns each layer into a robust z-anomaly, maps z → [0,1] stress, and phenology-weights it into a composite score — directly analogous to a severity curve, and a concrete pattern for turning any of the doses above into a curve.

4. Caveats — what makes a factor hard to turn into a curve

No damage calibration for most — VHI thresholds are heuristic; the pest proxy can't separate disease vs nutrient vs water stress (screening only); flood is a hard dB threshold, not graded.
Short baselines — SMAP ~9 yr, dense Sentinel-2 ~2018+; anomalies are indicative, not climatological.
Coarse / mixed-pixel — SMAP 9 km, GRIDMET 4 km, MODIS LST 1 km; sub-field detail needs downscaling.
Cloud gaps in optical/thermal; regional coverage limits (OpenET & gridmet_drought are CONUS-only; WaPOR is Africa/Near-East).
Fire not implemented; AlphaEarth embeddings aren't physically interpretable.

5. How MI models feed the library — three routes

The split is the whole point (and it's what the repos already do): the scoring engine stays simple downstream; the region-specific science — "how bad is this, really?" — happens upstream in MI and is deposited into the library as a finished curve. Three concrete routes, each grounded in code that exists today:

Route	What it produces	Where it lives (today)
a · curve shape feature importance	which drivers matter + their effect sizes → the shape of a damage curve	`crop_yield` `StackedResidual` (Ridge coefficients on the yield residual); `commodity_hindcast` PCA-ridge signed components + Cook's-distance influence
b · calibration consequence fit	driver-vs-outcome relationship → anchors a curve onto the 0–1 loss ruler (and validates it)	`crop_yield` `plot_stress_vs_residual` (stress vs log-detrended yield, Ridge fit); `commodity_hindcast` `run_meta_models` (bias-correct + conformal calibration against observed NASS/WASDE yield)
c · new signal EO product	a new exposure fraction or a new dimension's driver field	`gee_prep` — 29 EO/climate/soil datasets; the `anomaly.py` robust-z → stress → phenology-weighted `cascade_score` (a worked composite)

Two things to keep honest about this picture:

It's the intended architecture, not a live v1 data flow. Both libraries' charters say curves are stored, not fitted — "harmonisation and consequence-calibration happen upstream." But today the library is literature-anchored (FAO-33/56, Schlenker, Bayer); crop_yield and commodity_hindcast currently emit yield forecasts as their own products and do not yet feed the severity library. The loop is designed and the ingredients exist — the wiring is roadmap.
RISK ≠ STRESS — don't let the anomaly cascade become severity. gee_prep's cascade_score is a how-unusual (z-anomaly) composite; the library holds how-bad (dose → damage) curves. A stress/anomaly product feeds in as a pre-derived signal or becomes a curve only after upstream calibration — it never enters as severity directly. This is the "no model emits severity" rule.
Alignment across MI's own models — the open review question for Tommy & Wei. Whether twx-risk-severity-library + twx-phenoweight is genuinely modular and shareable across MI's yield / area models — and where it should share vs diverge — is their call on review.

The design rule (from the §4 note): we do not want competing scoring frameworks. All the hard, fun, region-specific work — working out how bad a peril is, per crop and place, from models and consequence data — happens upstream and feeds the shared libraries. The engine downstream just reads them.

Sources: review of treefera-market-insights (crop_yield, commodity_hindcast), tf_environmental_factors_preparation (gee_prep + anomaly.py cascade), and the twx-risk-severity-library / ag_risk charters (EXTENDING.md, OUTPUT_CONTRACT.md, VALIDATION.md). The stress maths itself lives in tf-data-ml-utils/weather (Tommy).