Patrick Skinner-May 11th

Development and application of an object-based verification system for Warn-on-Forecast

A goal of the Warn-on-Forecast project is to produce probabilistic numerical weather prediction guidance of thunderstorm hazards between the watch and warning scales.  Working towards this goal, a prototype system dubbed the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e) has been used to produce forecasts for 32 days between December 2015 and June 2017.  NEWS-e produces 18-member ensemble forecasts with up to 36 forecast output times 17 times per case.  This large amount of data generated by NEWS-e makes subjective forecast verification unfeasible and necessitates the development of an automated verification system in order to assess system performance.

This talk will detail the motivation, development, and application of an object-based verification system for NEWS-e forecasts.  Multi-Radar Multi-Sensor (MRMS) composite reflectivity and rotation track observations are used as a verification dataset and provide proxies for thunderstorm and mesocyclone occurrence.  Objects in MRMS observations are matched to corresponding objects in NEWS-e forecasts allowing contingency table-based verification metrics to be generated on a storm-to-storm basis.  Aggregation of contingency table elements allow NEWS-e skill to be quantified for different forecast minutes, cases, or initialization times.  Results will be presented that establish a baseline of NEWS-e skill for general and severe thunderstorms and examine differences in skill for different storm environments and system configurations.  Finally, limitations and expansion of the current verification system will be discussed.  Specifically, preliminary verification of 2018 forecasts and classification of forecast skill across different mesoscale environments using machine learning will be presented.