Algorithms are insufficient to curb disinformation

Analysis

The accuracy of AI-powered content screening tools is constantly being improved, transitioning from early filters based on keyword detection towards more advanced deep learning-based models.

Most notably, the Fake News Challenge, organised by a team of volunteer journalists, scientists and academics since 2017, pitches disinformation detection AI models against each other.

The project tests the ability of AI models accurately to label the stance of social media posts or news articles. Stance detection is crucial in detecting and countering disinformation as it labels what side of the argument the content argues for.

Since the challenge's launch, the accuracy of the various models entered in the competition has improved, from around 82% in 2017 to 88% in 2019. A promising potential contender, from DarwinAI and the University of Waterloo, even exceeded 90% accuracy.

Yet such models at the moment can assist fact-checkers but cannot entirely replace them. Although some simple factual statements can be reliably checked by AI (eg, "COVID-19 is just a flu"), allegations that require a nuanced contextualisation are still out of reach (eg, "COVID-19 vaccines can be dangerous").

Indeed, current algorithms are poor at detecting disinformation precisely because they struggle with evaluating nuance, humour and sarcasm, and because in many instances it is hard to assess what is true and what is not. By contrast, screening terrorist propaganda, graphic content or posts inciting violence are relatively easier to detect accurately.

Technical solutionism

Although the efficacy of such tools will improve, the challenge of disinformation is likely to intensify as deep fakes become harder to detect, bots become more genuine, and state actors sink more resources into their influence operations (see INTERNATIONAL: Disinformation risks national security - August 16, 2021).

For example, language prediction models such as GPT-3 are getting effective at summarising large volumes of text reasonably accurately, which is useful for stance detection. However, they are also increasingly effective at writing stories like a human.

It is likely that while machine learning and text processing models will steadily keep improving, they will be leveraged by both promoters and fighters of disinformation.

Moreover, an excessive focus on technological solutions risks distracting policy attention from the only reliable inoculation from disinformation: critical thinking and media literacy skills. To that end, Twitter has launched a partnership with Reuters and Associated Press to add potential misinformation labels and contextualisation on news pieces circulating on the social media platform.

However, rolling out broader educational programmes is a politically fraught exercise; the EU is alone among governments to have concerted plans for this.

Risks of over-relying on AI

Relying on machine learning models and other AI-powered tools to fight disinformation allows the processing of vast amounts of data, something that would be impossible to do manually.

However, excessive reliance carries its own risks.

Automation bias

Over-reliance on automated systems means that journalists and fact-checkers may gradually lose the ability to question the algorithm's judgment and, in effect, their own critical judgment. This is the well-known problem of automation bias.

The presence and impact of this bias has been widely recorded in other contexts, for example in areas such as the use of auto-pilot tools in transport. It is deemed partially responsible for catastrophic decision-making errors, such as the 1988 USS Vincennes incident (when the US missile cruiser mistakenly shot down a civilian Iranian aeroplane during the Iran-Iraq war) and the 2020 shooting down of the Ukrainian Air Flight 752.

Therefore, some level of human monitoring is likely to remain essential for the foreseeable future, including an appeals process.

Introducing bias

As with all algorithms, there is a risk of embedding human biases into AI models, especially how they assess language, humour or expressions -- in particular among minority groups. The bias is attributable both to the lack of sufficient data on minority groups in the training data and the lack of diversity among AI engineers.

Such biases have already been uncovered in areas such as employment applications (where male candidates can be favoured over females) and some face recognition algorithms, for instance, that identify faces of people of colour incorrectly much more frequently than white faces.

Inexplicability

A related problem is that algorithms are often black boxes, especially those based on deep neural networks. The design of such algorithms is not always conducive to auditing, making their decisions hard to explain and making some of their biases hard to correct systematically (see INT: Explainability issue requires judicious use of AI - August 6, 2021).

Adversarial manipulation

AI models are always at risk of adversarial manipulation, where an adversary could purposefully take advantage of biases in the algorithm to make it greenlight false information and, in the worst case, modify models themselves by manipulating their learning capabilities.

Like image and speech recognition, text classification systems have been shown to be vulnerable to adversarial input that alters their output in ways that are obviously erroneous to human observers. For example, targeted alteration of just one word in a text can lead a negative review to be classified as positive by a sentiment analysis system, and the addition of an apparently insignificant sentence into a text can lead a question-answering system to extract the wrong answer from the text.

While these instances refer to input that is deliberately misleading, it is likely that deep learning systems for content classification will make errors of this nature without manipulation. Such systems may fail to identify misinformation if it is worded differently or has other superficial dissimilarities to the data used to train the system. Similarly, the system may misidentify legitimate content if it has superficial similarities with examples of misinformation in the training examples.

Outlook

As with most machine learning and other AI-powered algorithms, excessive reliance on technical solutions risks introducing a variety of biases and inaccuracies into how social media content is filtered and regulated, which could prove detrimental to the very purpose they were designed for.