by Daniel Köhler, Hasso Plattner Institute, Germany.
Do you recall the last time you learned something from an online video? Results of our study indicate that the video type you might have watched can influence how and what you learned from it. In cybersecurity, our results show differing performance between recall and transfer tasks of learners after watching different types of videos covering the same content.
Over the past years, the popularity of online (video-) based education has been rising tremendously. Educational content is available on almost all topics and fields in online platforms. What does, however, differentiate a good video from a bad one? Various kinds of educational formats have been researched in the past years. Traditional videos featuring Presenters with Slides or Interviews have often primarily been found in (academic) online courses. In contrast, more modern Explainer-style videos are usually observed on platforms like YouTube. While differences between, e.g., learner engagement between different video types, previous research has seldomly looked at the impact of different types towards different types of tasks presented to the learners.
In the context of cybersecurity awareness education, we revisited the investigation of the impact of different video types on learning outcomes across three different types of tasks. Cybersecurity awareness can be learned and understood across three different levels: (1) Knowledge of the risks present in the digital world, (2) Understanding Protective mechanisms and how they are applied, and (3) Adjusting one’s Behavior accordingly. In a study with more than 1,500 participants, we investigated three different video types and their impact on learning across the three dimensions of cybersecurity awareness.
Our study was conducted alongside a free cybersecurity Massive Open Online Course (MOOC) run in October of 2022 on openHPI, an online educational platform. After completing a pretest to baseline participants’ level of knowledge, they were randomized into either of three study groups, each being shown a different type of video:
- Presenter with slides
- Interview without slides
- Animation with voiceover
To ensure the study’s validity, the content of all videos needed to be as identical as possible. As the animation video was integrated from an external resource, we built the slides for the first variant based on this animation. The interview was later conducted using the same slides as orientation for the topics of the discussion. Still, it covered a few additional aspects the interviewees encountered during the recording.
Figure 1 Screenshots of the three video types, left to right: Presenter and Slides, Interview, Animation
We did not observe any differences between the later assigned groups in the pretests for knowledge. This was expected as we randomized the groups. After consuming the videos, we prepared three different tasks for learners. All learners were distributed to two separate exercises. One in which they were asked to recall Knowledge obtainable during the videos. In the second task, the other group of learners was tasked with categorizing four emails into phishing or not phishing to test the skill of Protection. Finally, as the third task, those learners who enrolled in a phishing test were sent up to three phishing emails for them to identify.
When comparing the performances of the three study groups, it became apparent that for the Knowledge, learners who had watched the Presenter with slides performed significantly better than their peers, as confirmed by statistical tests. For the exercises targeting learners’ Protection skills, however, those who had watched the Animation performed significantly better than their peers. However, the skills targeted with both exercises had to be put into practice in our phishing exercise. Across three months, we sent enrolled learners up to three phishing emails, which they had to identify correctly and, in turn, ignore or delete. For our participants, we tracked interaction with the phishing content in the stages of Email Opened, Phishing Link Clicked, and Data Submitted on our phishing webpage. Learners who had previously watched the Animation videos reacted best to the phishing emails, with only 34% of them opening the phishing mail, while from the other two groups, 41% (Interview) and 47% (Slides) opened the emails.
Our results indicate different learning outcomes achieved by different video types. While videos with Slides help learners remember and recite knowledge, other more interactive video types, such as Animations, help them better prepare for in-world situations in which the knowledge has to be applied to practice. With that observation, we want to motivate educators to rethink if and why they opt for a specific video type. Variation in presentation means has already proven beneficial for learner interest. It can be used to define the learning outcomes further.
Author
Daniel Köhler, Hasso Plattner Institute, GermanyEditor’s note: This article has been inspired and adapted from: Köhler, D., Pünter, W., Meinel, C. (2023). The “How” Matters: Evaluating Different Video Types for Cybersecurity MOOCs. In: Viberg, O., Jivet, I., Muñoz-Merino, P., Perifanou, M., Papathoma, T. (eds) Responsive and Sustainable Educational Futures. EC-TEL 2023. Lecture Notes in Computer Science, vol 14200. Springer, Cham. https://doi.org/10.1007/978-3-031-42682-7_11
