Loading...
 8,105

Fast & Furious Fact Check Challenge

Let's create a nearly-instant system for fact checking the content of text. Read Overview...
$50,000
Overview

Today’s “always on” environment, together with social media, really does give us the ability to hear anything said by anyone, anywhere, anytime. Ironically, this flood of material makes it difficult to know what is actually true! Knowing the believability and accuracy of what we read, hear and see is important around the world -- and no less important for us here in the world’s leading democracy.

Fact checking is the process of verifying what someone has said, and then receiving a rating about the accuracy of the ‘fact.’ Fact checking enables us to sort through a tidal wave of massive information and communication.  

Some fact checking services exist, but none are instant.  

Fact checking today is done mostly by qualified humans. It’s a laborious, time-consuming process that is not easy, quick, cheap or comprehensive. There simply aren’t enough journalism researchers with the skills to verify all the claims made by our political candidates and public figures. It often takes a day or more to verify the accuracy of statements, especially in the context that they were made. And as time elapses, the truth moves further and further away from us.

The critical time to know if political claims and statements are accurate is now -- as we read or view it.  Therefore, the breakthroughs sought in this prize are those that improve speed of results in fact checking.

Guidelines
Timeline
Updates 19
Forum 8
Community 90
Entries
Claims
Practise Claims

Practice Claims: presented by Full Fact

We’re Full Fact, the UK’s independent fact checking organization, and the authors of The State of Automated Factchecking.

The goal of the challenge is for innovators to create a tool that can fact check 80% of claims correctly.

There are four primary types of claims we will be tackling

  • Numerical claims
  • Verification of quotes
  • Position statements
  • Objects, properties and events

This is a non exhaustive list. We may publish different claim types at a later date.

This post sets out a bit more about how we are devising claims to check that they are challenging enough that a winning tool would actually be useful, but not so hard that no team wins.

  1. We will be trying to devise claims that cannot just be looked up on a search engine or a human fact checking website. The winning software will need to be able to figure out new knowledge.
  2. Every claim will need to be rated on the scale of True, Somewhat True, Somewhat False, False. That limits the kinds of claims we can give you as that answer needs to be fairly unambiguous.

So we’re going to give you a fairly clear steer on the kinds of claims you might see, and what counts as more and less complex.

Justifications

Your tool will provide a rating for each claim and an optional justification. If you get the rating wrong, but give what the judges consider to be a sound justification for your rating, you may still get points for a claim.
 

For example:

Angelina Jolie and Brad Pitt are married.

This claim is (currently) True. That is what the official records would show. But they have separated and Angelina Jolie has filed for divorce. We would therefore expect that a rating of Somewhat True or even Somewhat False might be accepted by the judges with an appropriate justification.

So tools that provide intelligent justifications may have an advantage (and they definitely serve audiences better).

We have included ratings for the example claims below. We are open to feedback if any of the ratings are confusing and will take that into account in the claims for the challenge.

 

Numerical Claims

This type of claim will encompass numerical and statistical claims. It will not include interpretation of statistics or statistical analysis, nor will it include scrutinizing methodologies.

At their simplest, these might include claims like “The US population is 320 million.”

We will consider three broad categories of numerical claims –

  • Reference numbers—that can be looked up in a database, obvious examples include population, employment rates, and other standard official statistics. The best justification is likely to be a link to an authoritative source.
  • Analysis of numbers—where the numbers have to be related to other concept. A complex example might be a claim such as “young people are finding it harder than ever to own their own home”, which needs an understanding of young people, incomes, and house prices (but this example could probably be verified through other means). The best justification is likely to be a link to any sources used, and may also include any definitions or assumptions used.
  • Judgements using numbers—one type of school gets better exam results than another. No set of statistics can answer this claim, it’s actually a research question dressed up in numbers. A good analysis would need to consider how comparable different schools are and lots of other confounding factors. The best justification is likely to be something that identifies different sources and reaches a balanced conclusion, whether that’s computer-generated or a computer-identified meta-analysis.
     

Examples

  1. The Obama administration has doubled the US national debt in eight years. (Somewhat True)
  2. The USA won the most medals at the 2016 Rio Olympics (True)
  3. The US employment rate is rising (True)
  4. The US population is 320 million (True)
  5. USA has the lowest self employment rate in the OECD countries (False)
  6. The Forbes billionaires are richer than Japan (True)
  7. Income inequality is 0.396 in the United States (True)
  8. There are more unemployed white women than there are unemployed black women in the US workforce (True)
  9. In America, white women are nearly half as likely to be unemployed than black women (True)
  10. 5,000 workers earned less than minimum wage in Hawaii, in 2013 (True)
  11. In Fairbanks Alaska there were 2017 farms in 2012 (False)
  12. In the USA there are 3 hospital beds per 1,000 people (True)
  13. In 2015 the USA produced 294,000 tonnes of carrots (True)
  14. When I was secretary of state we increased american exports globally 30% (Somewhat True)
  15. 5.0% of the US labor force is unemployed (Somewhat True)
  16. In Westchester, New York, the average wage is almost 10 times the national average. (False)
  17. Spain has the most unemployed young men (True)
  18. Warren Buffett's 2015 tax deductions totalled almost $105.5m (False)
  19. On average each person is the US is responsible for 16.2 tonnes of CO2 emissions (True)
  20. In 1984 there were 682,800 adults in prison in the USA (Somewhat True)
     

 

Examples of justifications

 

 

Complexity

Numerical claims vary in complexity. Factors that make them more complex include –

  • Comparisons
  • Different ways of referring to the same thing (‘unemployment’, ‘joblessness’, ‘people without jobs’)
  • Participation (Obama, we, the opposition, the government, experts)
  • Time periods described in human terms (‘the Obama administration’, ‘since the recession’)
  • Imprecise adjectives (plummeting, soaring)
  • Proportionalities (the majority, three quarters, for every five houses sold only one is built)
  • Hedges (roughly, generally, broadly)
  • Geographies (New Jersey, New York, my state, the east coast, the western world, G20 countries, OECD)

 

Verification of quotes

This type of claim will encompass verifying the published source of a quote, the accuracy of a quote or the event at which a quote supposedly occurred.

Examples

  1. “The concept of global warming was created by and for the Chinese in order to make US manufacturing non-competitive.” Donald Trump (True)
  2. “This TPP sets the gold standard in trade agreements to open free, transparent, fair trade, the kind of environment that has the rule of law and a level playing field.” Hillary Clinton (True)
  3. Hillary Clinton claimed that "[Donald Trump] was one of the people who rooted for the housing crisis. He said back in 2006, ‘Gee, I hope it does collapse because then I can go in and buy some and make some money.’" (True)
  4. Obama said “we know that having millions of people in the criminal justice system, without any ability to find a job after release, is unsustainable.  It’s bad for communities and it’s bad for our economy.” (True)
  5. Hillary Clinton said “I gratefully accept your nomination for the presidency of the United States.” (Somewhat False)
  6. In his acceptance speech Mike Pence said “This is the legacy of Hillary Clinton: death, destruction and weakness.” (False)

 

Examples of justifications

 

  • Claim: Mark Twain said "A lie can travel halfway around the world while the truth is putting on its shoes."
  • Rating: False
  • Justification: This is misattributed to Mark Twain. Jonathan Swift said something similar http://quoteinvestigator.com/2014/07/13/truth/

 

Complexity

Factors that make these claims more complex include –

  • Quotes within quotes
  • Paraphrasing
  • Lack of authoritative sources
  • Widely shared but discredited
  • Misattributed quotes

 

Position statements

This type of claim will encompass position statements, whether someone supported a change in policy, or voted a certain a way, or endorsed a candidate. This will also include claims about the state of public opinion.

Examples

  1. Donald Trump did not support the Iraq War (False)
  2. President Obama signed an executive order banning the Pledge of Allegiance in public schools (False)
  3. Rep. Betty McCollum voted to pass the Water Resources Reform and Development Act of 2014. (Somewhat True)
  4. Barack Obama sponsored the Mercury Export Ban Act of 2008 (True)
  5. Hillary Clinton voted to raise taxes on workers earning as little as $41,500. (Somewhat False)
  6. Heineken sponsors dog fighting events (False)

Examples of justification

 

 

 

Complexity

Factors that make these claims more complex include –

  • Lack of definitive sources
  • Ambiguous claims about the effect of law or legal decisions
  • Conflicting records of behaviour
  • Comparisons
  • On public opinion, different polls with different methodology, understanding that not all source are equal

 

Objects, properties, or events

This type of claim will encompass different properties such as the ages of leading figures, qualifications they may have, whether they attended a certain university or event.

Examples

  1. Corey Lewandowski is Donald Trump’s campaign manager (False)
  2. Hillary Clinton is the Democratic candidate for President (True)
  3. Angelina Jolie and Brad Pitt are married (True)
  4. Donald Trump is the next president of the United States of America. (False)
  5. Texas has 37 electoral votes (False)
  6. Sen. Dianne Feinstein is the chairman for the Senate Select Committee on Intelligence (False)
  7. Houston is the most diverse city in America (Somewhat True)
  8. The President talked with the Ukrainian Deputy Prime Minister Volodymyr Groysman on May 13th 2016 (Somewhat False)
  9. Louisiana Sen. Mary Landrieu "received almost $1.8 million from BP over the last decade." (False)

Complexity

Factors that make these claims more complex include –

  • Lack of obvious or dominant source
  • Informal naming, such as ‘Brangelina are getting a divorce’
  • Widespread misperceptions, such as ‘there are nine planets in the solar system’
  • Having to use inferences or fill in gaps to justify the conclusion: Catherine Middleton’s husband is a helicopter pilot [Catherine, or Kate, Middleton married William, Prince of Wales, who also flies helicopters]