Downstream integration & output format

How ag_risk scores are written, what the files contain, and how they reach the platform, app and LLM · derived from OUTPUT_CONTRACT.md · 2026-07-16

ag_risk writes two append-only zarr stores per (region, commodity) — one realised, one forecast. The zarr is a complete, self-describing, append-only record — not the app's query surface. Engineering converts it downstream into raster/COG + a tidy ADM mart, which the app and LLM consume. Any change to this contract goes via Chris & Laurence.

1. The flow

The engine writes two zarr stores; the platform turns them into the app/LLM query surface. The stores are the record, not the query layer.

2. The two stores

Store	Holds	Grid	Append axis
`forecast_risk_<region>_<commodity>.zarr`	every forecast vintage, forward-looking months only	forecast-native (~1.0° / 100 km, SEAS5)	`init_time` — one vintage per run
`realised_risk_<region>_<commodity>.zarr`	what actually happened, one value per elapsed month	0.25° / 25 km (ERA5)	`valid_time` — one month per run

Two principles baked into the split:

Physical forecast/realised split. A run's elapsed months go to the realised store; its forward months go to the forecast store. No blended store. The platform stitches at query time: realised up to today, latest vintage beyond. Each cell is stored in exactly one place.
Vintage retention is a KEY requirement. Every forecast vintage is kept forever and is immutable once written — sel(init_time="2026-06-01") returns exactly what the June init produced. That's how "look back at previous predictions" works.

It saves the way the weather pipeline does. This isn't a bespoke store — it mirrors how the wx pipeline persists weather. The realised store appends one month-slab per run along valid_time (like the pipeline's ERA5 obs-areal store); the forecast store uses the canonical init_time × horizon vintage layout (the SEAS5 / blend_latest convention). Each run appends, never rewrites — so consuming ag_risk output is the same pattern as consuming the weather pipeline's.

3. The file format

One score variable, risk_score (0–1). combined is an entry on the hazard axis, not a separate variable. Same variable in both stores; only the axes differ.

Forecast store

risk_score   (init_time, horizon, hazard, latitude, longitude)   float32   0–1, NaN beyond reach / off-crop
valid_time   (init_time, horizon)                                datetime  derived = init_time + horizon (months)
source       (hazard,)                                           str       which upstream produced each hazard
spatial_ref  ()                                                  int32     CRS holder (EPSG:4326)

Realised store

risk_score   (hazard, valid_time, latitude, longitude)           float32   0–1, NaN off-crop
source       (hazard,)                                           str
spatial_ref  ()                                                  int32

Coordinates

Coord	Dtype	Meaning
`init_time`	datetime64	SEAS5 initialisation — the vintage key & append axis (forecast only)
`horizon`	int32	months ahead of `init_time` (`0…forecast_horizon_months`); fixed-length (forecast only)
`valid_time`	datetime64	calendar month scored. Realised: a dimension. Forecast: a derived 2-D coord = `init_time + horizon`
`hazard`	str	`[drought, heat, cold, water, …, combined]` — a dimension; new hazard = longer coord
`source`	str (per hazard)	upstream producer of that hazard's score
`latitude / longitude`	float64	cell centres, EPSG:4326

Semantics

risk_score = crop-loss risk, 0.0 (fine) → 1.0 (total loss). Continuous.
combined = the weight-free soft-max headline across the hazards named in the combined_hazards attr; per-hazard entries are the decomposition.
NaN = off-crop pixel, or (forecast) a horizon beyond that vintage's reach. NaN is "no statement", never "no risk".

Deliberately NOT in the file (so the app owns presentation, and format never breaks): no *_cat band layers — banding is app-side using the published band_threshold_medium / band_threshold_high attrs; no ensemble member dim — one point score per cell (uncertainty, if added later, arrives as extra vars risk_score_q10/q90, never a dim). A new hazard is a longer hazard coord + one more source tag — never a schema migration.

Global attrs the app needs

Emitted by the writer; these drive presentation without touching the data: role (realised/forecast), region, commodity, grid_resolution_deg, combined_hazards, band_threshold_medium, band_threshold_high, combine_operator (e.g. softmax_p3 — the operator and its exponent), forecast_horizon_months, verdict (Directional), plus CF-1.10 conventions and a scalar spatial_ref CRS (EPSG:4326).

Format: zarr v2, Blosc, consolidated metadata; forecast chunked init_time=1, realised chunked valid_time=1 — so an append writes new chunks only, never rewriting prior data.

4. The vintage matrix (how forecast grows)

Each vintage is a row indexed by init_time, laid out over horizon; its valid_time = init_time + horizon. Nothing is ever overwritten.

                          horizon (months ahead) →   valid_time = init_time + horizon
 init_time ↓     0    1    2    3    4    5    6
   2026-06-01    •    •    •    •    •    •    •     ← June vintage  (valid Jun…Dec)
   2026-07-01    •    •    •    •    •    •    •     ← July vintage  (valid Jul…Jan)
   2026-08-01    •    •    •    •    •    •    •     ← Aug vintage   (valid Aug…Feb)
                 (· = NaN where a horizon is beyond the season / feed reach)

5. What happens each month

Monthly, after the SEAS5 release. One run per (region, commodity) does two appends and nothing else — history only grows. Worked example, the run of 6 Aug 2026, Kenya maize:

Realised append. July 2026 has elapsed → score it from the realised feeds → append the valid_time=2026-07-01 slab to the realised store.
Forecast append. The August SEAS5 init is out → score Aug 2026 → Feb 2027 → append the whole outlook as the init_time=2026-08-01 vintage to the forecast store. June/July vintages untouched.

Net per month: +1 vintage in forecast, +1 month in realised. The verification pairing (what we forecast for a month vs what happened) falls out for free.

6. How the app & LLM consume it

Platform converts, then serves. Engineering turns the zarr into raster/COG + a tidy ADM mart — that's the app's query surface, not the zarr directly.
The app shows the latest forecast vintage (sel(init_time=max)), stitched with realised up to today.
Banding is app-side using the band_threshold_* attrs (low/medium/high) — per-hazard/per-region thresholds are an open item Laurence is confirming.
The combined headline is the default view; per-hazard entries drive the "why" (which hazard is driving the score).
The LLM templates responses over the scores (combined + decomposition + band) from the mart — downstream of the platform.

7. Query patterns

import xarray as xr
fc = xr.open_zarr(".../forecast_risk_kenya_maize.zarr")   # forecast archive
re = xr.open_zarr(".../realised_risk_kenya_maize.zarr")   # actuals

fc.sel(init_time=fc.init_time.max())["risk_score"]           # latest outlook (what the app shows)
fc.sel(init_time="2026-06-01")["risk_score"]                 # a specific vintage, over horizon 0…H
fc["risk_score"].sel(hazard="combined")                      # headline …
fc["risk_score"].sel(hazard="drought")                       # … vs one hazard (the "why")
re["risk_score"].sel(valid_time="2026-06-01")                # what actually happened in June
dict(zip(re.hazard.values, re.source.values))                # which upstream produced each hazard

8. Change control

The contract is frozen; changes go via Chris & Laurence. The shape is built so most evolution is not a breaking change: a new hazard extends a coord; uncertainty arrives as extra variables; a denser cadence is just a denser valid_time. Open items: app-side banding thresholds (Laurence), scale check on the growing forecast store (Laurence), monthly-vs-daily valid_time, and a *_latest convenience copy.

Source of truth: model-foundry: model_foundry/models/ag_risk/ — render/store.py + MODEL_CARD.md. The output contract is unchanged by the migration. Interrogate every dim, coord, dtype and attr in the synthetic example stores under docs/examples/.