Object-based verification of 1-h probabilistic mesocyclone guidance from the NSSL Experimental Warn-on-Forecast System
Although the skill of the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e) probabilistic guidance has been subjectively demonstrated, an objective assessment is warranted. Naturally, gridded forecast probabilities have been evaluated in a grid-point framework. However, small spatial displacements between probabilistic forecasts and observations leads to poor reliability and the infamous double penalty (i.e., missed observation and false alarm forecast). To relax the condition of an exact match and improve reliability, forecast probabilities and observations are post-processed (e.g., Gaussian and maximum filtering) and verified using neighborhood techniques. However, smoothing probabilistic forecasts effectively eliminates the connection between forecast probabilities and the underlying, individual thunderstorms which violates a fundamental aspect of NEWS-e probabilistic guidance. Therefore, we argue for using an object-based framework established in Skinner et al. (2018) to evaluate grid-scale NEWS-e probabilistic guidance.
In the object-based framework, grid-scale (no additional post-processing) probability swaths associated with individual thunderstorms are treated as forecast objects from which a single, representative probability value is extracted. Forecast objects are matched in neighborhoods of 0, 9, 15, and 30 km to observed rotation tracks objects derived from Multi-Radar Multi-Sensor (MRMS) azimuthal wind shear. Object matching allows for verification based on traditional contingency table statistics (i.e., hits, misses, and false alarms) and reliability, which are intuitive and easily interpreted. To demonstrate this method, NEWS-e probabilistic mesocyclone guidance was generated for the following periods: 0-60 min, 30-90 min, 60-120 min, and 90-150 min. Two model diagnostics for mesocyclone existence are used: 2-5-km AGL (mid-level) and 0-2-km AGL (low-level) updraft helicity. Results suggest both mid- and low-level updraft helicity probabilities are skillful up to 90 min lead times. Reliability, however, degrades after 30 min in association with increasing under-forecast bias at higher probabilities.