WFP Ethiopia AA MAM Trigger Documentation and Metrics - Somali Region
Last Updated: 18 July, 2024
0. Introduction
Purpose of Document and Outline
This document presents documentation and skill metrics for the WFP Forecast-Based Anticipatory Action Plan for the March-April-May (MAM) rainfall season in Somali region.
This document pulls its data from the IRI AA Design Maptool for Somali Region:
http://iridl.ldeo.columbia.edu/fbfmaproom2/ethiopia
This Maptool was also used by the TWG to determine the specific criteria for anticipatory action described below. Note that the Maptool is intended for exploratory data analysis and teaching / illustration, and does not necessary reflect the detailed AAP described in this document.
1. Forecast Trigger Rule
The Forecast-Based Anticipatory Action Plan is triggered according to the following rule.
The TWG designates a historical Frequency with which anticipatory action is intended to trigger. For this project, actions associated with a “Moderate” severity drought are meant to trigger in 30% of years, and actions associated with a “Severe” drought are meant to trigger in only 20% of years.
For each year, 1991-present, the IRI forecast database retrieves the forecasted probability of non-exceedence associated with the chosen Frequency. For example, the forecast probability at 20% frequency indicates the likelihood during a given year that cumulative MAM precipitation will be at least as dry as the 20th percentile in the historical record, or roughly a 1/5 year drought.
The IRI database calculates the Threshold for triggering action. This is also based on the Frequency; for example, at the 20% frequency, the threshold is set such that the most probable 1/5th of forecasts in the historical record would have triggered action.
Since the forecast model has different levels of skill at different lead times, the Threshold corresponding to a given frequency of action will not necessarily be the same for each issue month. Consult the Appendix for more detailed documentation of model construction and skill.
This also means that as more years of history are added to the record, the Threshold will change from year to year, since it is based on percentiles of the historical distribution from 1990 to the most recent year.
To account for the fact that consecutive seasons of drought can have compounding affects in arid areas like Somali Region, two adjustments are made to the forecast threshold.
If the previous OND season’s rainfall (as measured by the Standardized Precipitation Index, SPI) was below its long-term average, 3.5 percentage points are subtracted from the forecast threshold.
If the most recent monthly measurement of MODIS NDVI (for example, November for the December issue forecast) is below its long-term average, 1.5 percentage points are subtracted from the forecast threshold.
The forecast probability for a given year is compared against the adjusted Threshold. If it is equal to or higher than the Threshold, anticipatory action is triggered for that month. If it is less than the Threshold, no action is taken that month.
This process is followed for each forecast issue month - December, January and February.
This trigger rule is based on the average forecast over Somali region. However, the tables in this document also present information on the forecast measured at the Woreda level, to assist in the targeting of anticipatory action programs within the region.
NB: For the December issue forecast only, CHIRP data for the period October 1 - November 16 is used to calculate the previous season’s SPI, since the full CHIRPS OND data is not yet available.
2. Guide to AA Design Map Tool
A detailed user guide to IRI’s AA Design Map Tool can be found here:
https://fist.iri.columbia.edu/publications/docs/ethiopiaFbFJul2022/
3. Forecast Model Documentation
3.1 Technical Overview
The forecasts integrated in the IRI decision tool represent a set of multi-model seasonal forecasts of rainfall in Ethiopia during the periods of October-November-December (OND) and March-April-May (MAM). The developed tool allows users to spatially visualize historical forecast (Hindcast) information as well as with a table displaying the forecast together with information on observed historical rainfall, and bad years reported by the agricultural sector since 1991. Preliminary forecasting efforts have focused on predicting seasonal precipitation from the precipitation fields of the North American Multi-Model Ensemble (NMME) global climate model suite and the European Copernicus model suite. Further research is ongoing on the prediction of other potential predictors (e.g., frequency of rainy days, frequency of dry spells, date of onset of monsoon, etc) and use of other model variables as predictors (e.g., air temperature at 2 meters, sea surface temperatures, winds, etc.).
3.2 NEXTGEN
Forecasts are developed using the NextGen method as discussed in Acharya et al. 2021 for different seasons over Ethiopia. NextGen is a systematic approach to establishing, calibrating, and verifying objective climate predictions based on multi-models. Forecast development is made easier with the NextGen approach. By assessing past model performance, forecasters are able to determine how to correct and combine different global climate models. Additionally, it facilitates the selection of climate models for any region of interest, it automates the generation of forecasts at regional, national, or even at subnational scales.
The calibration method of NextGen means that the probabilities of non-exceedence presented in the forecast output reflect the historical skill of that forecast for predicting rainfall for a given season, lead time and location. This is why the probability threshold associated with a certain action frequency may vary over seasons, lead times and locations, as mentioned in Section 1.
For example, for a high-skill forecast, the 1 out of 5 year frequency trigger threshold may be around a probability of 30%, meaning that any forecast leading to action would be at least 50% (i.e., (30-20)/30) more confident than guessing based on climatological odds (this concept, known as the “sharpness ratio”, is discussed further in Appendix A3). In contrast, a lower-skill forecast may have a 1 out of 5 year frequency trigger closer to a probability of 25%, meaning a triggered forecast would only be at least 25% more confident than guessing; however, the overall frequency of trigger actions remains constant in both scenarios.
3.3 Design and calibration data
Season: Oct-Nov-Dec (OND) and Mar-Apr-May (MAM)
Predictor (GCM Data): Precipitation (simulated by GCMs)
Predictor Domain: 5S-20S, 20E-60E
Predictand (Observed Data): Precipitation (observation from CHIRPS)
Predictand Domain: 2.5S-15.5S, 32.5E-48.52E
Models: EU-C3S-ECMWF-SEAS5, GFDL-SPEAR, COLA-RSMAS-CCSM4, NCEP-CFSv2, NASA-GEOSS2S, CanSIPS-IC3
MOS Method: CCA
Initialization month: Jul, Aug, Sep (OND), Dec, Jan, Apr (MAM)
Training period: 1991-2020
Forecast period: 2021-2022
Presentation: flexible format
3.4 ENACTS
Historical observations are taken from ENACTS data. ENACTS includes most weather stations and merges station data with high-resolution (in this case, 0.05 degrees) satellite information from 1981-1519.
4. Historical Skill Analysis
4.2 Bad Year Matching Rate (Humanitarian Predictive Skill)
We can also evaluate forecasts based on their skill at predicting known humanitarian catastrophes during which AA would have been merited. Data on these “historical bad years” comes from an interactive exercise conducted with the Regional Technical Working Group in Somali region. Participants were asked to identify and rank the worst 10 years from 1990-2021.
From this information, we can obtain the following metrics of humanitarian predictive skill:
Worthy-action - number of years in which a drought was forecasted, and a ‘bad year’ occurred
Act-in-vain - number of years in which adrought was forecasted, but a ‘bad year’ did not occur
Fail-to-act - number of years in which a no drought was forecasted, but a ‘bad year’ occurred
Worthy-Inaction: number of years in which a no drought was forecasted, and no ‘bad year’ occurred
Rate - Overall performance (the number of correct action years - i.e., both worthy-action and worthy-inaction - over the total number of years)
In this case, we present this skill metric for a trigger rule based on the forecast alone, as well as for the more sophisticated trigger rule which incorporates the previous season’s conditions (described in Section 1).