Geopolitical Forecasting [GF] Challenge 2

There are a few differences between the first GF Challenge and GF Challenge 2. These differences are summarized here.

The Challenge is expected to launch on May 15, 2019. Please follow this Challenge to get the latest updates on the Challenge launch date and more information as it becomes available.

The GF Challenge 2 is looking for forecasting methods; solutions that are capable of processing data and making forecasts. While this may include investigating how a combination of both computer models and human judgments can be used to make accurate forecasts, it need not do so. If indeed a very good human forecaster (or a small group thereof), can do just as well as hybrid solutions or the IARPA Aggregative Contingent Estimation ACE baseline, then we have learned something. However, we believe this is extremely unlikely due to the number of individual forecasting problems (IFPs) in the Challenge. Moreover, methods will be compared, for many of the prizes, against a benchmark crowd of human forecasters using the validated state-of-the-art method from the ACE program. While your solution need not be fully algorithmic in nature, you will need to submit your forecasts throughout the Challenge through an API and be able to describe the solution for your forecasts at the end of the Challenge. Therefore, some data science literacy is likely required.

Yes, this is a U.S. government funded project. The Intelligence Advanced Research Projects Activity (IARPA) invests in high-risk/high-pay off research programs that have the potential to provide our nation with an overwhelming intelligence advantage.

In order to register for the Challenge, you will need to log into the HeroX GF Challenge 2 page and "Accept the Challenge". This includes accepting the Challenge rules and acknowledging that you adhere to the eligibility requirements for the Challenge, in addition to accepting the Intellectual Property (IP) Agreement. Once you have completed that step, you will be sent a link to the Cultivate Labs platform to register for your, or your team's, API token.

The GF Challenge 2 is a means for IARPA to solicit solutions that are not under development in HFC. While HFC Performers may have other forecasting approaches that are not under development in the HFC Program, to ensure that HFC approaches are not used at all in GF Challenge 2, HFC Performers are not eligible to participate in the Challenge.

This is a prize Challenge, not an IARPA research program. No IARPA contracts are anticipated to arise from this Challenge.

For more information on doing business with IARPA outside of this Challenge, visit the following link: www.iarpa.gov/index.php/working-with-iarpa

GF Challenge 2 Solvers are welcome to compete with their pre-existing products or tools to address the Individual Forecasting Problems (IFPs) that are issued during the Challenge.

We anticipate asking approximately 310 IFPs over the course of the Challenge. The intent is to resolve all IFPs by the end of the competition.

No, you do not need to forecast against every IFP, but you will be scored on every question, regardless of whether you submitted a forecast or not. To clarify, choosing not to submit a forecast against an individual IFP will likely result in a very poor score because for each unanswered question, the Challenge scoring protocol enters an "ignorance prior" in the form of equal probability assigned to each of the possible forecast outcomes (e.g., 0.5 to YES, 0.5 to NO for a binary question). More about how responses, or non- responses, may impact a score can be found in the Scoring section of the Challenge Rules Document.

Yes, every day is a scored day. The Challenge Rules Document describes what forecast days are.

Each day, there is a reporting deadline: 14:00:59 U.S. Eastern Time. Forecasts in the system at that time will be scored. Until an initial forecast is submitted, the ignorance prior (equal probability to each of the possible forecast outcomes) is assigned as the forecast. Updated forecasts may be submitted throughout the day, until the 14:00:59 U.S. Eastern Time. The latest forecast submitted ahead of the deadline will be scored.

There will be some variation among IFPs. We anticipate durations ranging from one day to the entire length of the Challenge.

Scores will be tracked on a leaderboard on the Cultivate Labs Platform. You can log into your account or Team account on the platform to see how well you are doing in the overall Challenge, or on the individual prizes/milestones throughout the Challenge.

Once it has been determined that you have won a prize, you will be contacted via email with instructions on where to send your tax and payment information in order to receive your funds. As a reminder, the Solver is solely responsible for paying any taxes or fees that may be incurred based on the award of prize.

You may enter the Challenge at any point during the competition. However, if you wait too long it is unlikely that you will be able to perform well enough to qualify for Overall or Bonus prizes. You will still have a chance to win Milestone prizes.

Most Solvers used the provided human forecast stream and benchmark consensus forecast stream as their only inputs.  They applied techniques that were similar to the IARPA ACE Program winning approach.  These included:

• Determining the most accurate forecasters and upweighting them.
• Applying decay weighting to favor more recent updates.
• Extremizing the resulting forecast.

Some Solvers did use newer techniques for determining forecaster weights such as neural networks and machine learning.  Only two Solvers used exogeneous data or models in their forecasts, with mixed success.  Overall, the best methods were comparable with the benchmark method derived from the ACE Program but no method was significantly better than the benchmark.

It is the opinion of the Challenge Sponsor that the accuracy weighting, decay weighting, and extremization approach has been mined to exhaustion in ACE and GF Challenge 1 and that the likelihood of significant performance improvement beyond the benchmark using these techniques alone is extremely low.

Yes, with a few exceptions, international Solvers may participate in this Challenge. Within the Challenge Rules Document, there is a section outlining the guidelines for international participation, and additional information. If you still need clarification on your eligibility to participate, please send an email to:

No. IARPA's HFC Performer team members, Government-provided Forecasters for RCT-B, and other individuals who have access to HFC Performer Hybrid Forecasting Systems during the competition, as well as members of such persons' immediate families (spouses, children, siblings, parents), and persons living in the same household as such persons, whether or not related, are not eligible to participate in the competition.

Former HFC Performer team members may participate but will need to forego prize money. In order to compete, an email must be sent to with the team name, organization represented, and team member information. 

Former HFC Performer forecasters may participate as long as they do not have access to HFC Performer Hybrid Forecasting Systems during the competition.

Yes, with a few stipulations. First, Solvers from FFRDCs or UARCs must forego the prize money for the Challenge. Second, the entity, along with anyone working on the Challenge, must meet the other eligibility requirements in the Challenge Rules Document. Third, they may not have supported IARPA's HFC Program in any manner. Please see the Challenge Rules Document for complete information on how to proceed.

The IFP data stream and human forecaster data stream are provided for the sole purpose of participating in GF Challenge 2. If you would like to use the IFPs or Human Forecaster stream for a different purpose, please email describing your use case and the team will determine if additional use will be possible.

Except as described in the IP agreement that is required of all Challenge Solvers, participating in this Challenge does not alter title or license rights Solvers. The IP Agreement can be found here.

Except as described in the IP agreement that is required of all Challenge Solvers, participating in this Challenge does not alter title or license rights to the methods you use to generate your forecasts. IARPA is supportive of further scientific research in accordance with the Office of Science and Technology Policy (OSTP) memorandum of 2013 called “Increasing Access to the Results of Federally Funded Scientific Work". IARPA asks that you send us a courtesy copy of the publication if you cite the Challenge or IARPA within it.

In accordance with the IP Agreement required of all Solvers in the Challenge, IARPA will retain a copy of the forecasts that you provide and a description of your method. There is no transfer of title or license to the IP, so there is no expectation that IARPA or the U.S. government will pay royalties to Challenge Solvers. Prize-winners may be approached about other opportunities, but such opportunities are outside the scope of the GF Challenge 2.

A team can be composed of either one or many Solvers. In other words, you may participate as a lone Solver (a team of one) or in a group of many Solvers.

Once the Challenge launches on HeroX, you will have an opportunity to register as a team and invite members to join the competition and join your team. If you decide to create a team after you have already registered, you still have the ability to create a team within the community tab of the HeroX GF Challenge 2 page. All team members must meet the general eligibility requirements, register, and acknowledge the Challenge rules through the HeroX GF Challenge 2 page. Upon registering for the Challenge, the team captain will need to provide a breakdown of all team members along with the agreed upon prize percentage allocation between all team members. Each team will need to determine who will be setting up their team's API token, as there can only be one API token granted per Solver/Team. You are only permitted to participate in the Challenge as an individual or as part of one team (membership on multiple teams is not allowed).

No, participants can only join as a lone Solver or as a member of a Solver Team comprising two a more Solvers. Joining as a lone Solver and as a member of Solver Team or joining more than one Solver team is strictly prohibited. 

No, once you have registered for a team and your team has begun making forecasts, you cannot switch teams, or join a new team. Similarly, if you are a lone Solver and have begun making forecasts, you can no longer join a team.

Yes, individuals can form teams after the first IFP is issued, but you cannot form a team after you have submitted a forecast as an individual or the intended team has submitted a forecast. The first IFPs will be released on May 15—do not start forecasting until you have your team solidified. If you are thinking about teaming, it would be prudent to wait to submit your forecast until you determine how you would like to participate in this Challenge (as part of a team or as an individual).

We suggest that you post to the HeroX forum in order to seek out other team members.

Teaming presents an opportunity to divide the labor and conquer. However, many individuals prefer to work independently and may have the necessary skill sets to be successful as an individual. It is up to your personal preference and working style.

These IFPs are developed by an experienced team working for IARPA. The team’s experience includes developing similar questions for the earlier IARPA Aggregative Contingent Estimation (ACE) program, HFC, and last year's GF Challenge. Each IFP includes an authoritative source and/or procedure for determining the outcome. In our experience, we have found these IFPs to be well posed and to have unambiguous outcomes. It may take up to a few weeks after the IFP closes for it to resolve at which time the official data source is updated, and the outcome is confirmed. Embedded in the API stream for each IFP is a description of how each IFP will be resolved to include which data source will be used for the ground truth resolution of the IFP. If additional clarification information is deemed necessary, the clarification details will be incorporated into the API.

No suggestions for or nominations of questions/IFPs are solicited nor will any be accepted.

Challenge Solvers will have access to a stream of de-identified individual-level human forecast data. From this, a baseline aggregate will be constructed using a method based on the state-of-the-art from IARPA's ACE program. Solvers will also be given access to this aggregate. Baseline data (and the associated de-identified individual forecast data) will be available to Solvers via the Cultivate Labs API.

The human forecaster stream is provided by a crowd of hundreds of non-expert forecasters. Forecasters are encouraged and incentivized to provide frequent and accurate forecasts.

The human forecasters are incentivized to provide frequent and accurate forecasts. Forecasts and their rationales are monitored as part of a quality control process. Low quality forecasters (e.g., forecasters who consistently fail to provide meaningful rationales) may be removed.

In GF Challenge 2, Domain and Region information for a given IFP will be available through the API once an IFP has closed.

Yes. In order to maintain account security, and to ensure that Solvers read and understand the updated rules associated with GF Challenge 2, all Solvers and Teams will need to register for GF Challenge 2 and be issued a new API key for GF Challenge 2.

Each Solver will be given an authentication code in the form of an API token upon registration. Forecast submissions are authenticated with this code and are not visible to others. If you keep your API token secure, it is unlikely that your submission will be intercepted or hacked.

All submissions must be made through the API on the Cultivate Labs Platform.

Once you accept the Challenge Rules and IP Agreement for the Challenge on the HeroX GF Challenge 2 page, you will be sent a link to register for your API token which will provide you access to the Cultivate Labs Platform. Please note that only one API account will be allowed for each lone Solver or team of Solvers.

Yes. Each Solver team member may sign up for an individual API token to access IFPs and forecasting data. Solver Teams also have the option to access IFPs and forecasting data using one API token shared among the team members. However, only one API token can be used by a Solver team for submission.

Log into your API account on the Cultivate Labs Platform and from the Home Page click the "View API Token Management" button to retrieve or re-generate an API token.

The complete API documentation will be available on the Cultivate Labs Platform which is accessible to registrants for the Challenge on HeroX.

The Challenge will only provide all solvers with the IFP stream, human forecaster data, and baseline aggregation. No additional data will be provided. We understand that not all Solvers may want to use this data. Solvers are able to use additional or alternative data sources that they deem useful to their solutions.

As the challenge progresses, you may track the performance of individual forecasters by comparing their forecasts from the human forecast data stream with the IFP resolutions. Additionally, there are two sources of "pre-season" training data available: The Cultivate staging server has 86 resolved IFPs with human forecasts, and the Cultivate production server will have roughly 6 weeks of historical forecasts available in the system for the start of the Challenge. This will be available through the Cultivate Labs API. Data on the staging server will not have human forecaster IDs that match those in the GF Challenge 2, but the production server pre-season data will include forecaster IDs that match those in the GF Challenge 2.

The final score will be based solely on the accuracy of your forecasts. In order to receive your prize, you will have to submit a document that describes the technical approach behind your method. Your score is based on the quality of your forecasts.

We will make the exact requirements clear, but the expectation is that the document that describes your solution will be no more than four pages long. We may request more information to gain a better understanding of your method or to provide clarity. Any parts of your solution that are proprietary need to be clearly marked as such. In the event that you are a prize winner, we need to be able to describe your methodology in greater detail to a public audience so that the public can have confidence that the prize funds were awarded properly.

The results of this Challenge will be publicized through public communication channels, the IARPA website and Challenge HeroX Page. We will also host a Workshop at the end of the Challenge where select winning Solvers will be invited to share their solutions with a government audience and HFC Performers.

Yes, quick and easy registration on HeroX.com gets you access to follow, comment and register to compete. Just click the "Join Us" button at the top of the page and follow the instructions to complete your registration. All you need to provide is your name and email address.

If you have a question not answered in the FAQ, we recommend that you post it in the Forum. A response will be posted on the Forum, so others who may have the same question will be able to see it. IARPA will not respond privately to individual questions submitted via email except for specific questions regarding eligibility to compete.

You will submit your forecasts via the Cultivate Labs platform API. Once the Challenge has officially launched, you will be directed to the challenge platform where you can receive an API key.

For questions involving topics and events that are widely reported in the media, we consult credible open source media sources (e.g., Reuters, BBC, AP) to determine the outcome of a question. Some questions may specify that the resolution will be determined by specific sources (e.g., UNHCR data, Moody's). When in doubt, please check the information provided in the Description and Clarification fields of each question for guidance specific to that question.

Lone Solvers do not have to register as team. There are no real advantages to registering as a team of one if you are a lone Solver. If you decide to register as a team of one, you will have to complete and submit the Prize Distribution form.

If you are participating as a lone Solver you do not need to submit the Prize Distribution form. However, all registered teams, including teams comprised of one Solver, are required to submit the Prize Distribution form.

The number of team members are not limited by the number of rows in the form and Solvers can append additional rows as needed.

Yes, every person on the team needs to sign the form to ensure the team is in agreement on how to split the prize money, should the team win. For signatures, you can always have the team affix a digital signature to eliminate the need for travel. 

Forecasting questions are released to the Human Forecasters at the same time they are released into the system.  It is possible that Human Forecasters may not respond immediately to an IFP.  Given that their responses also impact the calculation of the baseline, Solvers will be on equal footing with the baseline based on Human Forecaster Responses.  We are hoping that Solvers will incorporate additional data sources beyond the Human Forecaster stream into their solutions.

When there are no specific resolution criteria listed for an IFP, the IFP will be resolved by reviewing open-source news coverage. Any number of news outlets may offer credible reporting sufficient to issue outcome resolution. Although the Challenge Team will generally ascribe greater credibility to major English-language news outlets (e.g., Reuters, BBC, Associated Press, Agence France-Press), other sources will be judged credible (i.e., sufficient to determine a resolution) on a case-by-case basis.