ForecastWatch Methods and Scientific Approach

ForecastWatch utilizes a comprehensive and exhaustive process to produce reliable analyses that can be trusted for both scientific studies and for comparative accuracy assessment among weather providers. The following documents the scientifically rigorous methods employed to produce these analyses.

Obtaining Quality Observations

High-quality observations are essential to producing meaningful, useful data. Observations are retrieved from quality-controlled local information provided by the National Climactic Data Center (NCDC) and from other meteorological agencies whose data has met specific quality standards.

The quality control process on this dataset includes more than 50 validity checks, which include extreme value checks, internal (within observation) consistency checks, and temporal (versus another observation for the same station) continuity checks. Observations that don’t pass these checks are flagged as invalid.

Retrieving Geographically-Appropriate Forecasts from Weather Providers

Best efforts are made to ensure that provider forecasts are for the appropriate, corresponding observation location. Latitude/longitude coordinates or actual weather station names are commonly used to ensure that forecasts cover the location being analyzed. This process is applied to more than 1200 U.S. and international locations.

Forecasts are typically retrieved in one of two ways. The most common method is to query providers’ public websites and APIs. In other instances, forecasts are retrieved from private feeds provided to ForecastWatch directly by the weather provider. Forecasts for a given location are retrieved at the same time from all providers, and location order is randomized each day.

Use of Standard Scientific Statistics

Hundreds of standard statistics are produced to reflect provider performance and skill across a variety of forecast scenarios. Standard statistics are used rather than arbitrary or proprietary statistics to ensure that assessments are objective and universally acceptable. Statistics used to describe variance and error include bias, absolute error, equitable threat score, false alarm rate and Brier score.

Monitoring and Auditing of Forecast and Observation Validity

A series of audits are performed to ensure that forecasts and observations that are highly improbable or obviously not valid are excluded from analysis. This is accomplished through the use of pre-set thresholds that allow for identification of observations and forecasts falling outside reasonable ranges.

Examples of data excluded from analysis include: 1) a forecast that’s considered impossible, such as a prediction of -200°F for a particular location, or 2) observations or forecasts that are highly unlikely — such as a temperature of 27°F in Los Angeles. The goal is to eliminate from analysis forecasts that are obviously wrong, but not forecasts that are simply bad.

Use of Advanced Systems to Interpret Text and Icon Forecasts

A sophisticated system is used to parse and interpret text and icon forecasts. For example, if a text forecast calls for “a mix of sun and clouds,” logical assumptions are made about what that means regarding the degree of cloud cover. Similar logic is applied to icons depicting varying degrees of sunshine and clouds. The text and icon interpretation system allows forecasts to be placed into canonical categorizations, which assist with analysis and scoring.

A Note on ForecastWatch Methods

ForecastWatch methods and procedures are fully transparent and well documented. As a result, analyses can be readily replicated. Meteorologists and others are encouraged to evaluate these processes and provide any feedback for improvement.