As the GF Challenge enters its final weeks, we are watching the leaderboard with excitement. The Solver community has been doing an incredible job producing forecasts against a very broad portfolio of individual forecasting problems (IFPs). We are pleased to announce a new tier of “Second Chance” Prizes that may be awarded in the event that fewer than 5 methods qualify for Overall Prizes. As described in the GF Challenge Rules Document, the 1st through 5th Place Overall Prizes will be awarded for those methods with the highest Net Brier Points (NBPs), conditional on participating in at least 70% of IFPs and receiving positive NBPs on more than half of attempted IFPs. (Only each Solver’s best scoring method will be considered).

If, however, fewer than 5 winners receive Overall Prizes, IARPA will award “Second Chance” Prizes to recognize methods that achieved high NBPs, while participating in at least 70% of IFPs, but did not meet the 50% positive NBP minimum. The Second Chance award tier will be awarded as follows:

  • Methods that qualify for 1st through 5th Place Overall Prizes (and Ultimate Forecaster and Star Forecaster) will be awarded according to the originally stated criteria.
  • If fewer than 5 winning Solver-methods receive Overall Prizes, the remaining slots will become available in the Second Chance tier (e.g., if only 1st and 2nd place Overall Prizes are awarded, then 3rd through 5th place slots will be eligible for Second Chance award).
  • Second Chance prizes will be awarded using the same standards as Overall Prizes (i.e., highest NBPs, at least 70% participation) with the exception that methods will need to have positive NBPs on a minimum of only 33.3333…% of attempted IFPs.
  • Second Chance prizes will be worth half as much as their equivalent Overall Prize amounts, as shown in the table below, and will not be eligible for Ultimate Forecaster or Star Forecaster awards.
  • Power Forecaster(s) Award: Only awarded if both the Ultimate and Star Forecaster Overall Prizes are not awarded. Must participate in at least 70% of the IFPs.  Must have positive NBPs on a minimum of one third (33.3333…%) of the IFPs you attempt.  Must have a lower Brier score than the Benchmark on the Beat HFC Leaderboard.  Have a lower Brier score than all HFC Performers on the Beat HFC Leaderboard.  Uses the best scoring method, regardless of the Solver’s overall method used in final scoring. This will be awarded at $25,000 and is split amongst all eligible Solvers / teams.
  • Each Solver will only be eligible for one prize (either Overall or Second Chance) based on their single best performing method.
  • Please see the updated Rules Document for full eligibility requirements and prize breakdowns

Position

Overall Prize

Second Chance Prizes (if applicable)

1st

$20,000

$10,000

Ultimate Forecaster

$40,000

N/A

2nd

$12,000

$6,000

3rd

$10,000

$5,000

4th

$7,000

$3,500

5th

$4,000

$2,000

Star Forecaster(s)

$30,000

N/A

Power Forecaster(s)

N/A

$25,000 (split amongst all eligible Solvers / teams)

 

How were the percentage and prize allocations determined?

The team examined the challenge from many different perspectives—trends across time, attempt rates, success rates, as well as comparison evaluation with HFC. We recognize that we set an extremely high bar and are appreciative of the community’s achievements to date. An evaluation of the leaderboard and IFPs assisted in setting the “Second Chance” Prize percentage, which then informed the “Second Chance” prize amount awards.