{"id":32442,"date":"2024-05-02T14:56:44","date_gmt":"2024-05-02T12:56:44","guid":{"rendered":"https:\/\/media-and-learning.eu\/type\/featured-articles\/opening-doors-empowering-educators-students-with-oer-open-pedagogy-copy\/"},"modified":"2024-05-02T17:03:36","modified_gmt":"2024-05-02T15:03:36","slug":"opening-doors-empowering-educators-students-with-oer-open-pedagogy-copy","status":"publish","type":"post","link":"https:\/\/media-and-learning.eu\/type\/featured-articles\/opening-doors-empowering-educators-students-with-oer-open-pedagogy-copy\/","title":{"rendered":"From theory to practice \u2013 lessons learned from Ofcom\u2019s evaluation toolkit"},"content":{"rendered":"\n

by Kate Morris<\/strong>, Making Sense of Media (MSOM) programme, Ofcom, UK.<\/p>\n\n\n\n

When we started work on our evaluation toolkit<\/a> for organisations delivering media literacy interventions, we were committed to basing our work on the realities of our potential audience.\u00a0 We believed our toolkit would have most value for those organisations with little or no experience of evaluation and that knowledge pushed us to keep things as simple as we could.\u00a0 We stripped back jargon, explained terminology and created a fictional media literacy programme to illustrate key points related to the evaluation process, which we knew could feel complex and overwhelming.\u00a0We were keen to land the message that evaluation should not be seen as a pass or fail judgement on a project, but as a way of proving and improving interventions. And we also wanted to encourage organisations to think about using evaluation methods to demonstrate societal impact \u2013 using outcomes and indicators – rather than only reporting engagement numbers.\u00a0There were other aims too \u2013 that our guidance and free online workshops would build capacity in organisations with little time or funds to do evaluation work, and that interventions would be designed with evaluation embedded from the start.\u00a0<\/p>\n\n\n\n

What came next was a period of action research where we trialled this approach with 13 organisations that we commissioned to deliver pilot projects across the UK<\/a>.\u00a0Each organisation was required to produce an evaluation report and was offered our toolkit, and the support of an independent evaluation expert, throughout the year that the contract ran.\u00a0 At the heart of this approach was the way that the organisations would design their pilot programmes around evaluation frameworks and would appreciate the utility of evaluation in programme building. It was also a valuable opportunity to trial the toolkit and understand its strengths, and where we could improve it.\u00a0<\/p>\n\n\n\n

Lessons learned \u2013 the organisations<\/strong> <\/p>\n\n\n\n

Halfway through the year, the 13 organisations came together to share their learning, and some themes emerged.\u00a0
<\/p>\n\n\n\n

The value of embedding evaluation from the start<\/strong>: One organisation redesigned their programme after realising it was too easy for their participants, following responses to their pre-intervention survey. They reflected that using a robust evaluation approach on previous programmes they had delivered could have helped them avoid this.\u00a0<\/p>\n\n\n\n

The value of learning from your mistakes<\/strong>: One organisation shared that they forgot to put identifiers on survey responses so couldn\u2019t match up the pre and post surveys. They reflected it was a useful lesson for next time.\u00a0<\/p>\n\n\n\n

The value of the evaluation process overall<\/strong>: Some found the notion of drawing up an evaluation framework as they designed their programme challenging but this shifted when they started work on their evaluation reports. This demonstrated to them that the benefits of evaluation might only become apparent once they had gone through the process for the first time.\u00a0<\/p>\n\n\n\n

Lessons learned – Ofcom <\/strong><\/p>\n\n\n\n

We also took the time to reflect on our approach. <\/p>\n\n\n\n

Evaluation requires investment\u00a0<\/strong>: This trial had evaluation baked in and included specialist\u202fsupport. Despite this, some\u202forganisations\u202fstill struggled to find\u202ftime to\u202fprioritise\u202fevaluation alongside project delivery.\u00a0<\/p>\n\n\n\n

Evaluation is challenging<\/strong>: Organisations\u202fwere not necessarily comfortable with the\u202fevaluation process, and its specialist language \u2013 even with the\u202fsupport of an independent evaluation expert and our toolkit. Challenges included mapping outcomes to survey questions, writing outcomes, writing survey questions and finding suitable indicators.\u00a0<\/p>\n\n\n\n

Measuring impact is tough for smaller organisations\u00a0<\/strong>: Gathering longitudinal data from some of the cohorts in this\u202fprogramme was challenging for a range of reasons – from staffing\u202fshortages to the difficulty of getting back in touch with audiences.\u00a0<\/p>\n\n\n\n

Key takeaways <\/strong><\/p>\n\n\n\n

In one sense, this action research confirmed what we learned from the scoping work we did in preparation for our toolkit work \u2013 that funding, and lack of expertise, meant organisations were not able to evaluate as fully as they might like.\u00a0\u00a0 But it also raised another theme we think is worth investigating, linked to how smaller media literacy organisations might demonstrate the impact of their projects.\u00a0 From what we observed over the year, it was not realistic to expect a small charity to produce a robust evaluation akin to an academic project given expertise levels, budgets, time constraints and the size of data sets available.\u00a0 The research suggests that appropriate evaluation methods are contingent on the size of the organisation and available budgets \u2013 in other words, that it is all about proportionality.\u00a0<\/p>\n\n\n\n

Next steps <\/strong><\/p>\n\n\n\n

For now, we are working on our evaluation toolkit to add more examples, and explainer text to help people better understand how to devise outcomes, and design pre and post survey questions.\u00a0 We will also be publishing a report summarising lessons learned from our pilot projects – because we ourselves are committed to following our own advice: that course-correction is a key ingredient of successful delivery.\u00a0<\/p>\n\n\n\n

\"\"<\/figure>
\n

Kate Morris<\/strong>, Senior Associate, Evaluation, Ofcom, UK<\/p>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"

by Kate Morris, Making Sense of Media (MSOM) programme, Ofcom, UK. When we started work on our evaluation toolkit for organisations delivering media literacy interventions, we were committed to basing our work on the realities of our potential audience.\u00a0 We believed our toolkit would have most value for those organisations with little or no experience […]<\/p>\n","protected":false},"author":7,"featured_media":16873,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_mo_disable_npp":"","footnotes":""},"categories":[4,275,272],"tags":[45],"class_list":["post-32442","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-featured-articles","category-higher-education","category-media-literacy","tag-media-literacy"],"featured_image_src":"https:\/\/media-and-learning.eu\/files\/2022\/02\/bigstock-Modern-Benchmarking-As-Busines-433764122-scaled.jpg","author_info":{"display_name":"Shirin Izadpanah","author_link":"https:\/\/media-and-learning.eu\/author\/shirin-izadpanah\/"},"_links":{"self":[{"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/posts\/32442","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/comments?post=32442"}],"version-history":[{"count":7,"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/posts\/32442\/revisions"}],"predecessor-version":[{"id":32456,"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/posts\/32442\/revisions\/32456"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/media\/16873"}],"wp:attachment":[{"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/media?parent=32442"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/categories?post=32442"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/media-and-learning.eu\/api-json\/wp\/v2\/tags?post=32442"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}