by Anne-Katelijne Rotteveel, Koninklijke Bibliotheek & Elke Muller, De Haagse Hogeschool, The Netherlands.
Between March 2021 and April 2023, a consortium (consisting of in the Netherlands: the National Library of the Netherlands/Koninklijke Bibliotheek, The Hague University of Applied Sciences and the Netherlands Institute for Sound and Vision; in Belgium: Media & Learning Association and Public Libraries 2030; and in Spain: Fundación Platoniq) carried out an Erasmus+ funded research project on news media literacy among Dutch, Belgian and Spanish young people aged 12-15. The acronym SMILES, which stands for ‘innovative methodS for Media & Information Literacy Education involving schools and librarieS‘, was chosen for the project title.
Main goals and research method
One of the main objectives was developing teaching materials for Dutch, Belgian and Spanish pupils aged 12-15 with the aim of making them recognise disinformation and making them more resilient against it. These teaching materials consist of five building blocks, each with a different theme. Another goal was forming pairs between librarians and secondary school teachers in the three European countries, who were empowered through train-the-trainer workshops to teach these building blocks to secondary school pupils. The approach, key objectives and an overview of the learning materials have been described in a previous article.
Another part of the project was to scientifically evaluate the effectiveness of the implemented lessons through impact measurement using ‘pre-knowledge tests’ and ‘post-knowledge tests’. Feedback from the pupils, teachers and librarians through surveys and interviews were used to evaluate the materials further.
The SMILES project started with conducting five so-called ‘Baseline studies’, or literature reviews. The focus was on what the different educational approaches in Spain, Belgium and the Netherlands were regarding disinformation and how these approaches can be linked. Based on this literature on disinformation, the knowledge tests were designed to conduct an impact measurement of the workshops among the trainers (teaching pairs) and pupils, respectively. These knowledge tests contained statements on disinformation that were answered correctly or incorrectly by respondents. The number of correctly answered statements prior to the lessons was compared to the number of correctly answered statements after the lessons. In this way, an attempt was made to prove a positive learning effect of the deployed lessons.
In addition to these quantitative analyses, qualitative results were also used to analyse and look at the extent to which the training provided to trainers (teaching pairs) and the learning materials for pupils proved effective in teaching, recognising and becoming more resilient to disinformation. In doing so, we also reflected on whether the methodology tested has been effective in the three countries: what are the best practices and where do we see areas for improvement? The results can be found in the evaluation report.
Results of the research
The analysis shows that the statements from the knowledge tests do not sufficiently match the building blocks to establish a significant learning curve. However, the knowledge tests also show that pupils self-report that they can tell the difference between disinformation and reliable news after following the lessons. In the Netherlands, 55% of pupils say so, in Belgium 63% and in Spain 44%. In addition, the pupils say they learned new things and want to know more about the subject of disinformation. A quarter of pupils in the Netherlands and Belgium state that they would like to learn more about the topic of disinformation; in Spain the figure is as high as 35%. The quantitative analysis also shows that over 40% of the pupils in the three countries enjoyed participating in the building blocks. Around 40-50% of the pupils say they will apply what they have learned in practice.
The qualitative results show that pupils found the building blocks interesting and that the SMILES lessons were a starting point for discussion, further activities, and deepening.
The knowledge tests filled in among the trainers (the teaching pairs) also show no significant differences between the number of correctly answered statements in the pre-training knowledge test and the post-training knowledge test. But the data do show that most trainers say that after attending the train-the-trainer workshops they felt confident to give media literacy training on disinformation to young people. In the Netherlands, 71% agreed with this statement, in Belgium 60% and in Spain 67% agreed. The qualitative results of the study also show that the implemented lessons have positive effects. Teachers indicate that pupils were interested in the teaching materials and felt that they gained knowledge around the topic. Teachers remark that the building blocks are well constructed and add to the existing teaching materials. All trainers therefore indicate that they will continue to teach the building blocks developed by SMILES, although they will sometimes do so in modified forms in the near future.
All resources, materials and tools are available in English along with versions suitable for delivery in the Netherlands, Belgium and Spain on the SMILES website (see Digital Toolkit).
Anne-Katelijne Rotteveel, Impact Researcher, Koninklijke Bibliotheek & Elke Muller, Teacher and Senior Researcher, De Haagse Hogeschool, The Netherlands.
