MagNet: Model the Geomagnetic Field

Help NOAA better forecast changes in Earth’s magnetic field! Improved models can provide more advanced warning of geomagnetic storms and reduce errors in magnetic navigation systems. #science

$30,000 in prizes
feb 2021
616 joined

Information provided by the Earth's magnetic field is of primary importance for navigation and the pointing of technical devices such as antennas, satellites and smartphones. The excellent results from this challenge hold immediate promise for the space weather community.

— Manoj Nair, Research Scientist, NOAA/CIRES Geomagnetism Group

Why

The transfer of energy from solar wind to Earth's magnetic field can cause massive geomagnetic storms, wreaking havoc on key infrastructure systems like GPS, satellite communication, and electric power transmission.

The severity of these geomagnetic storms is measured by the Disturbance Storm-time Index, or Dst. In the past three decades, empirical, physics-based, and machine learning models have made advances in forecasting Dst from real-time solar wind data. However, predicting extreme geomagnetic events remains especially hard, and robust solutions are needed that can work with raw, real-time data streams under realistic conditions like sensor malfunctions and noise.

The Solution

The goal of this challenge was to develop models for forecasting Dst that 1) push the boundary of predictive performance, 2) under operationally viable constraints, and 3) using specified real-time solar-wind data feeds.

This is a hard problem where the best approaches are not evident at the outset. Competitors were tasked with improving forecasts both for the current Dst value (t0) and Dst one hour in the future (t1). Participants needed to submit code that could execute in a simulated real-time environment with operations constraints on runtime and programming inputs.

The Results

Over the course of the competition, we saw over 600 participants and 1,200 submissions. The top four prize-winners were able to achieve 11.1 - 11.5 nT RMSE on the private test set, beating the benchmark of 15.2 nT. Interestingly, an ensemble of the top four models does best of all with an RMSE of 10.6 nT, achieving a 30% reduction from the benchmark!

RMSE graph

NOAA was also able to verify the winning models by performing inference on an unseen dataset, collected between Nov 1, 2020 through March 4, 2021. This was a fantastic opportunity to see how winning solutions perform on completely new data. During this relatively quiet period, NOAA compared the top four models against its latest, experimental NCEI (National Centers for Environmental Information) Dst forecasting model, and they found that winners were able to push the state-of-the-art on this new unseen data! First place achieved 5.9 nT RMSE to the NCEI model's 6.5 nT, while the ensemble of the four winners again performed best with an RMSE of just 5.6 nT.

All the prize-winning solutions from this competition, including detailed reports, have been made available on Github for anyone to use and learn from.


RESULTS ANNOUNCEMENT + MEET THE WINNERS

WINNING MODELS ON GITHUB

CHALLENGE DATASET ON NOAA WEBSITE

CHALLENGE RESEARCH PUBLICATION IN AGU SPACE WEATHER