The Fairness of Fact-Checking and Its Impact on Social Media

Over the last two decades, social media has played a pivotal role in driving social and political transformations around the world. By allowing citizens to create and share information freely, social media has subverted a more top-down public discourse. This has proven particularly beneficial to marginalized communities who can challenge singular narratives that historically have dominated mainstream media. We see this complex dynamic play out during significant global events, such as the ongoing Israel-Palestine conflict.

Social media platforms do not aim to be sources or arbiters of truth, but regardless, they host, influence, and profit from conversations on trending issues and events. In recent years, the integrity of these conversations has been called into question, as they are riddled with individual perspectives, biases, beliefs, and values that may or may not be accurate, genuine, or fact-based. This has inadvertently contributed to political echo chambers, polarization, and ‘internet toxicity. There are also bad actors who manipulate online discourse by leveraging confusion and polarization to intentionally spread false information to favor their opinions or cause. These online arguments can incite offline violence or create other forms of safety risks for users, such as exposing them to hate speech, harassment, and cyberbullying, among others.

Whether of their own volition or due to government regulation, online platforms are in the throes of solving this “trust problem” and other information integrity concerns, especially in this worldwide election year. Fact-checking has become a commonly used approach by platforms to tackle the challenge of trust and misinformation. While there are mixed opinions on the effectiveness of fact-checking, it has been shown to be effective at correcting false beliefs and perceptions or combatting misinformation in some cases.

Is fact-checking fair?

Fact-checking is generally described as a process to verify the accuracy of a claim, and the goal is usually to prevent mis/disinformation. It is used on verifiable content such as statistical, scientific, and historical claims or assertions rather than opinions or subjective statements. Journalists and researchers have historically used fact-checking, but social media companies started implementing fact-checking measures more prominently between 2016 and 2020 to combat election and health-related mis/disinformation.

The fact-checking process involves verifying information against multiple reliable sources, evaluating the context within which the statement is made, and rating its accuracy. On online platforms, these ratings are used to label/add warning notices to content you may have seen and inform various content moderation decisions. A key component of every fact-checking process is identifying credible sources, a step we think requires deeper examination and consideration.

Researchers define source credibility as the extent to which an information source is perceived to be believable, competent, unbiased, and trustworthy. This is often measured by criteria that likely include reputation, editorial control and independence, statements of fact versus opinions, expertise levels of journalists, and transparency, among others. A fact-checker’s assessment of the credibility of a news source may also vary based on individual experiences, political leanings, ideological beliefs, and, in some cases, geographical location. Thus, the assessment of source credibility can be subjective, intuitive, and susceptible to bias. For example, a news outlet being transparent about its editorial process or having “expert” talent or a “good” reputation does not necessarily mean it is free from bias.

Building credibility requires time, capacity, and resources, and thus, it tends to favor larger and established networks with institutional power and position. These “big news” networks are also mostly Western, making up most of the world’s top media outlets, and have the most power in the industry. This gives Western media outlets prominence and control of the global news market.

These biases and inequities in how a source’s credibility is judged impact the use of local news outlets for fact-checking as they often lack the resources to build their reputation and “credibility.” This can be a challenge even in their local jurisdictions - for example, in some emerging markets, Western news outlets still dominate the coverage of local stories and rely on Western sources, resulting in imbalanced coverage of stories that lack local context and perspective. Some argue that since news source credibility is about position and power, this implies that locals, who are in some ways outside of power, are most likely to remain largely unused as sources in media reports.

While Western news dominance has been mostly attributed to power and resources, others maintain it is rooted in media imperialism and neo-colonization, where colonial power imbalances, dependencies, and legacies still influence media coverage in former colonies.

This creates an asymmetrical power dynamic (as it relates to power, resources, market control, etc.) between media agencies in emerging markets and Western countries, thus making Western sources seem more credible, authoritative, or of a “higher quality” which hinders the credibility of local outlets and journalists.

Who checks the fact-checker?

Fact-checkers offer a unique service to social media and other platforms that need to solve content governance in parts of the world where they do not possess deep cultural, political, and historical context. Meta’s Third-Party Fact-Checker and YouTube’s informational panel work with the International Fact-Checking Network (IFCN), which expands across 90 organizations and over 60 languages, to address viral misinformation. This network of fact-checking agencies taps into its local news networks to verify information coming from various regions.

However, the majority of the IFCN signatories are based in the West; out of the IFCN’s current 116 active signatories, only about seven are based in Africa and nine in South America. A quick scan of the 378 fact-checking organizations listed in the Duke Reporter's Lab database also shows over 50% are based in Western countries.

According to Mozilla’s recently published elections casebook, tech platforms rely on fact-checking as a first line of defense for global election misinformation, but these efforts often do not include partners who understand the local contexts and nuances. For example, in Liberia’s 2023 presidential elections, Meta did not have a trusted local flagger partner, resulting in election-related information largely missing or lacking local insights.

In the instance that local fact-checkers flag false content, their decision may have to be corroborated by foreign fact-checkers unfamiliar with the local context, which may have the effect of undermining the very expertise provided by a localized effort. The Liberia example further underscores how a ‘credibility’ bias against local media outlets can be shaped by financial constraints that limit their ability to hire journalists, run investigations, or diversify their coverage.

Since fact-checkers are highly reliant on news sources for their work, source credibility biases can impact their work. This is evidenced by how perceptions of source credibility differ among fact-checkers. These variations in accuracy ratings can be attributed to organizational or individual biases. The perceived credibility of a news source has been shown to impact the effectiveness of the work fact-checkers do and vice versa, albeit minimal. This is all to say fact-checkers are only as credible as the sources they rely on to fact-check. Fact-checkers are also susceptible to Western bias and could thus give more weight to Western media outlets than local ones based on “credibility.”

The impact of biased fact-checking and how to address it

News sources shape how information is obtained and how issues and events are reported. Sources cited in the media or on platforms do more than back up facts; they ascribe meaning to events and shape public perception. As a result, the credibility—real or perceived— of these sources, impacts the news landscape and influences the fact-checking process, particularly in deciding which perspectives are considered accurate.

This influence raises the risks of a ‘single story,’ as author Chimamanda Adichie warned in her TED talk, “...the single story creates stereotypes, and the problem with stereotypes is not that they are untrue, but that they are incomplete." This is particularly true in scenarios where the stories of the global majority world are being told by Western media. These stories can be rooted in cultural chauvinism and imperial stereotypes and result in imbalanced stories, spreading further ignorance and worsening stereotypes.

As a society, we may not be able to fully eliminate credibility biases. However, we can work towards having a breadth of sources that reflect a diversity of perspectives and a more balanced narrative, ensuring that all sides of a story are told. This is also true for fact-checkers - having a diverse group of fact-checkers helps to minimize bias with source credibility.

Fact-checking organizations are often small and scrappy and face significant challenges. However, they can become more diverse, balanced, and credible by making some small changes:

• Transparency: Clearly stating their methodology, sources, and the criteria used for fact-checking. Greater transparency evokes higher credibility perceptions.
• Diverse team: Ensuring a diverse team of fact-checkers with varying perspectives and backgrounds.
• Peer review: Subjecting their fact-checks to internal or external peer review for scrutiny.
• Corrections and updates: Being transparent about any corrections or updates to their fact-checks.
• Diversify the database: There are enormous benefits to having a diverse repository of news sources to verify information on social media. When there are global, sensitive events such as conflicts and elections, local media agencies may have firsthand accounts of what is happening, and could be well suited to capture cultural nuances and historical context.
• Local partnerships: Partnering with and amplifying the voices of local media agencies and stories, thus “boosting” their credibility.
• Funding: Fund smaller, local media agencies that do not have the same resources as large corporate agencies. This can help contribute to the diversity of thought in the media.
• Allocate more resources: Social media companies should allocate more resources to inclusive fact-checking. They should ensure that fact-checking guidelines like fact sheets, fact checks, and explainers are translated to reflect a country’s linguistic diversity so that everyone has equitable access to the information they need to perform more effective fact-checking.
• Comprehensive coverage: Focus on telling as many credible sides of the story as possible, versus a single story from a credible source.

Online content, news, and stories will always reflect society's inherent biases. By challenging and reassessing how we define credible news sources, we can begin to account for those biases. In doing so, fact-checkers can work towards publishing more balanced narratives that draw on perspectives from representative, diverse voices and encourage readers to form their own views.