Many educational researchers and practitioners appreciate Augmented Reality (AR) and Virtual Reality (VR) as promising tools to cognitively support learning processes. One of the strengths of AR in terms of learning lies with the ability to merge digital instructions onto a physical object. In this way, it is possible to make instructions only appear when and where they are needed, and eliminate the need to repeatedly switch attention between an object and the instruction. Cameras can provide additional benefits in the form of feedback loops. We can use a depth camera to check if a learner picks a part from the correct picking box.
The strength of VR lies in immersing the learner in a three-dimensional world. In this way, it is possible to really learn by doing. In addition, the immersive space makes VR well-suited to make abstract concepts more simple (imagine trying to explain to a newcomer how planets orbit around the sun, what a human heart looks like or – in the context of our project COSMO – how to assemble a big agricultural machine). Furthermore, learning in VR is a multi-sensory experience that includes visual, auditory, haptic and kinesthetic feedback, which enables a richer learning experience.
These advantages mean that AR and VR may offer interesting opportunities for education. But do we already exploit the full learning potential of these technologies?
In the imec.icon project COSMO, partners from industry and research centers are cooperating in order to exploit the full learning potential of AR and VR, through personalised instructions. The COSMO project takes place in an assembly context, but the idea of personalised instructions may equally apply to any other domain.
In many cases where AR/VR is used in a learning context, all users undertake exactly the same AR/VR training. From a pedagogical perspective, this approach raises a few concerns. The first of these is that having completed the AR/VR training, users may still not understand a certain concept, or to master a specific skill. When we remove the instructional support, will the learner still master the skill in the real world? Someone may be able to complete a complex task supported by detailed instructions in AR/VR, but may eventually be stuck once this digital support is gone. As a comparison, imagine being able to drive to a certain destination guided by a GPS, but then experiencing difficulties to repeat the same journey once your GPS is no longer available. A second concern is that the same instructions are given to people with different levels of ability. As a consequence, a certain instruction may not be sufficiently detailed for newcomers, so they may not be able to complete the task. On the other hand, that same instruction may be too detailed for advanced learners, potentially slowing them down, or causing frustration.
These are the concerns that motivated our team of researchers in COSMO to develop and study personalised AR/VR instructions. These are instructions in which the level of detail is adapted to the ability of the user, at any stage in his/her learning curve and stress level. The approach in which instructions gradually fade away as a learner gets to master a task better is known in educational sciences to foster learning.
Our approach uses three different levels of instructional detail (instructions with a low, a medium and a high level of detail). In AR, as the level of detail increases, there are more lights, arrows and text that indicate which part a user should pick and what to do with it. In a similar approach, in VR, as the level of detail increases, there will be more virtual tools to guide the user: indications of the location of interest, contours that show how something should be assembled, haptic feedback to guide the user, audio instructions, etc.
Before designing personalised instructions, we wanted to know if the level of detail of AR/VR instructions makes any difference at all. We introduced about 100 second and third grade secondary education special needs students to a variety of AR tasks, and about 100 third grade technical-oriented secondary education students to VR tasks. Students alternately got support with a low, a medium or a high level of detail. We found that a high level of detail led to faster completion times, lower error-making and a lower perceived complexity.
Figure 1: a student assembling an agricultural machine in VR (figure on the left) and a student assembling a wooden bird house supported by AR instruction (figure on the right)
Building on these findings, we are now making the tasks adaptive, in that the level of support can change within a task. We will compare different types of adaptivity, and investigate if they are advantageous compared to a non-adaptive control condition. We are keen to find out if adaptive AR/VR tasks could be advantageous in terms of completion times, error-making, perceived complexity, well-being and could lead to steeper learning curves or a better retention. Interested? Follow us via the project website.
Editor’s note: The imec.icon project COSMO is a collaboration between partners from industry (Mariasteen, Rhinox, Azumuta and CNHi), research centers (imec research groups Itec, IPI and ID Lab, and Flanders Make) and four Flemish schools. It was co-financed by imec and received project support from Flanders Innovation & Entrepreneurship.
Authors: Pieter Vanneste, Kim Dekeyser, Luis Alberto Pinos Ullauri, Dries Debeer,
Annelies Raes, Fien Depaepe, Wim Van den Noortgate
