Resolving Complexities in Linking Forecast Data to Observational Data

Linking forecast data with observational data may sound simple, but comes with a myriad of data issues. Mine-Weather addresses the different types of data issues encountered in linking the forecast data with the observational data, and the type of solutions proposed to handle them.

Weather forecasting is predicting the future based on past and present data using different analytical methods. Forecast data is sourced from a number of data sources collected at regular intervals over time. Forecast feeds may include forecast variables in addition to summery descriptions or synopses of the overall weather conditions. Weather variables could be like maximum/minimum temperature, wet bulb, dew point, wind speed/direction/gust, pressure, humidity, cloud cover, light intensity, snow/rain accumulation, and alike. Synopsis is an important field used by the Mine-Weather system to extract valuable information that may be used at a later stage of analysis.

Observational weather data is obtained from a wide range of observing stations across the utility service area, the National Weather Services, and other private providers. Automatic Weather Stations (AWSs) are used at majority of the locations to send data frequently usually on an hourly basis. Observational feeds may include weather variables usually associated with an earlier time period, such as pressure changes during the past 3 hours, maximum air temperature from the past 6 – 24 hours, and alike. Observational data shows the actual values, whereas forecast data shows the predicted values. There are various data nuances associated with linking both these values. Mine-Weather simplifies this process as it adheres to a set of general rules regarding capturing and storing forecast and observational data, migration of historical data and calculating scores at the most granular levels. By following these rules various data issues can be resolved even before system startup:

  • Device automated process to capture all data (both forecast and observational data feeds).
  • Retain all data in the database for an extended period of time to address critical weather issues.
  • Historical data migration provides the analytical system with extended timeframes for doing analysis immediately after the new system is setup.
  • Do not include any additional forecast data entries to match the hourly observational data as this will have a negative impact on the accuracy of the output.
  • Ensure scores are calculated at the lowest levels of granularity as this helps in neutralizing many of the data issues.

Data Mapping – Linking Forecast Data with Observational Data

Mine-Weather uses advanced table mapping techniques and takes a standardized approach in comparing forecasts and actual data. On comparing the forecast and observational data, it is evident that there are only a few similarities that can be easily linked. Hence, it is important to invest time and attention for mapping the right fields between forecasts and observations in order to obtain accurate scores.

Issues to be dealt with while linking forecast data with observational data are:

  • Delve into the data dictionaries for an in-depth understanding of individual weather variables.
  • Remember the exact scoring formulas needed to validate the accuracy of the forecast data.
  • Consider the transition from hourly data to daily data that causes a discrepancy between forecast data and actual data.
  • Methodical decision driven approach when there are gaps between the availability of actual data feed or when the actual data takes on an unrealistic value.
  • Ensure that separate scores are calculated and later aggregated for observational data that are mapped to individual forecast feeds.

Download our Whitepaper to take a detailed look on how Mine-Weather resolves the various issues in linking Forecast Data with Observational Data.

Mapping is a critical process which if not done properly could lead to erroneous results with companies ending-up with huge losses. By adhering to the set of general rules and appropriately following all the data linking process enables calculating scores that are meaningful in judging the accuracy of various forecasts. With Mine-Weather simplifying various data linking complexities, companies can be assured of running their production line on a more reliable forecast.

Download Whitepaper

About the Author

Dr. Ronald Mueller

Ron is CEO and Founder of Macrosoft, Inc.. Ron heads up all company strategic activities, and directs day-to-day work of the Leadership Team at Macrosoft. Ron is also Macrosoft’s Chief Scientist, defining and structuring Macrosoft’s path forward on new technologies and products, such as Cloud; Big Data; and AI. Ron has a Ph.D. in Theoretical Physics from New York University, and worked in physics for over a decade at Yale University, The Fusion Energy Institute in Princeton, NJ, and at Argonne National Laboratory. Ron also worked at Bell Laboratories in Murray Hill, NJ., where he managed a group on Big Data, including very early work on neural networks. Ron has a career-long passion in ultra-large-scale data processing and analysis including: predictive analytics; data mining, machine learning and neural networks.

Leave a Reply

Your email address will not be published. Required fields are marked *