by Sahil Shah, Say No to Disinfo, UK.
Advances in AI are making the creation, dissemination and amplification of mis and disinformation significantly easier, quicker and cheaper, with 60% of participants in a 2024 study saying they had encountered a deepfake at least once in the past year. In the current political environment, there may be limited success for regulatory and platform interventions in the short term. In this environment where disinformation is more prevalent, more personalised and harder to discern, educational interventions such as media literacy are becoming increasingly important, though the current baseline level of media literacy is low, with a study showing only 2% of British children and young people have the skills to spot fake news. However, the evidence base of quantitative experiments on the efficacy of interventions to combat mis/disinformation and build media literacy is disparate, disaggregated and lacks a shared ontology. On media literacy interventions, there has historically been a lack of robust measurement and evaluation, or a lack of publication of results for government run programmes, leading to a scarcity of data on what works. In addition, this evidence base is hard to find, navigate and interpret so tends to be underutilised by the practitioner community.
At Say No to Disinfo, we are building the world’s first and most complete online living database to counter disinformation and build information resilience. It aggregates, categorises, curates and extracts key information from empirical studies on interventions to counter mis/disinformation and build media literacy. In our database, we extract key data from thousands of empirical counter disinformation experiments around the world across topics and fields. This includes: methods, results, characteristics of people targeted, topics, claims and factors that determine intervention effectiveness. Our in-house ML algorithms use extracted data from our database to rank effectiveness for different interventions based on what has worked for similar demographics on similar issues. We use LLMs to analyse factors that determine counter campaign effectiveness which helps guide campaign design and implementation.
What makes an effective media literacy campaign? What we have learnt from our database:
When designing and implementing a media literacy intervention, it is important to start with a clear definition of media literacy in the context of the intervention, in order to guide design decisions. When setting the objectives of the intervention, key considerations include the effect type (this may be knowledge based or behavioural for a media literacy intervention, and it is often most useful to focus on one), effect timing (to ensure measurement techniques capture the effect appropriately), change vs reinforcement effects, direction and degree of the change, and the scope of change across the target group. There is also a need for interventions to shift from simply making targets better technology users, as many emergent technologies and applications of AI are ‘used on’ rather than ‘used by’ people, it is critical to ensure that interventions also help users navigate these technologies, for example, helping users understanding data-driven automated systems and identifying when they are being used.
The effectiveness of an intervention depends on the level of personalisation to the target group, it is important to gather and analyse target information to improve intervention design. As the information environment is so rapidly changing, it is important to understand the information seeking behaviours and habits of the target group. Research has noted that some people are still more susceptible to misinformation than others. For example, those with a more extreme political orientation have also consistently shown themselves to be more susceptible to misinformation, even when the misinformation in question is non-political. Other factors such as greater numeracy skills and cognitive and analytic thinking styles have consistently been revealed to have a negative correlation with misinformation susceptibility—although other scholars have identified partisanship as a potential moderating factor. Targeting education towards the most vulnerable groups can significantly increase its impact.
This can be done through a variety of channels including through schools, workplaces, broader civic education and public communications, in order to reach diverse groups of people. This could also come through community-led initiatives to engage harder to reach individuals. Critically, in order to be effective, this education needs to come from different nodes within a network, over a prolonged period of time, and from trusted actors. Material design needs to include embedded context to personalise to different audiences, including the use of specific case studies as different vulnerable groups are susceptible to manipulation from different actors and narratives. Our database helps identify the most effective techniques and implementation considerations based on similarities of the target demographics.
But implementation is not the end. In order to iterate and improve interventions, as well as to build up the evidence base further, measurement and analysis is crucial. Measurement design should be guided by the intervention objectives, and measurement at multiple points in time is critical to understand the immediacy and the longevity of the effects. It is important to test a variety of messages, formats, channels and networks to identify the most effective approaches for particular sub groups, in order to scale effectively.

Author
Sahil Shah, co-founder of Say No to Disinfo