Comprehending products with mixed reality

Nowadays, the virtual reality (VR) market has made significant strides. VR is widely used in different applications, and education and training are one of the leading areas which are expected to grow to US $2.2 billion in revenue by 2023. Mixed reality (MR) is also known as hybrid reality that encompasses both VR and augmented reality. It allows interaction and feedback from the users, which is particularly important to enhance the experience in teaching and learning. MR not only merges the real and virtual worlds into a new environment, which combines physical and digital content, but also allows users to interact with the digital content dynamically in real time. In support of teaching and training, MR technology has been applied in diverse manufacturing and industrial sectors to visualize the digital models. Nevertheless, most of these MR projects are mainly focused on users’ experience. The effectiveness in comprehending product design is seldom investigated. Therefore, in this project, an MR application is developed for teaching product design to university students. The application is developed on the HoloLens system. A practicable application has been designed that allows students to visualize the geometry of 3-D objects as well as the exploded diagrams of selected components. The students can command the system through the command manual or can signal the system through gazing, gesturing, and voice to implement instructions. To investigate the effectiveness of the MR application in teaching product design, a pretest and a posttest of similar difficulties were designed to access students’ abilities in five main areas: the ability of students in comprehending design, understanding product functions, visualizing 3-D geometry, understanding geometric relationship, and students’ creativity. In the investigation, 45 students were participated in the tests. Experimental results show that the proposed design positively improves students’ understanding in geometric relationships and creativity. In the future, more features will be added to the MR application to further improve students’ abilities in learning and training in other aspects.


Introduction
In recent years, the virtual reality (VR) market has made significant strides.VR is widely used in various applications such as health care, fashion, sports, education, commercial, and entertainment.The compound annual growth rate for VR revenue is expected to grow at 54.84% from 2018 to 2023.It is anticipated that in the leading sector of education and training, it is expected to grow to US$2.2 billion in revenue by 2023. 1 VR is not only able to provide immersive stereoscopic visualization of virtual environments and sound effects, but participants can also interact with virtual objects with haptic feedback. 2For the education and training in various disciplines, VR gives an opportunity to connect with learners and teachers in a novel and meaningful way.VR enables experiential learning by simulating virtual environments.According to the experience from educators, it was found that many students can increase their levels of engagement and improve their test scores with the aid of VR education programs. 3Hands-on VR learning approaches can also contribute to increase cognitive and memory abilities. 4A meta-analysis was conducted by Zahira et al. 5 who suggested that games, simulations, and virtual worlds in VR were effective in improving learning outcome gains of the students in higher education.Despite VR providing an artificial and computer-generated immersive simulation of real-life situation by stimulating their multisensory experience such as vision and audition, it is important to blend digital components into the real world in such a way that they enhance one another rather than the purely virtual environment. 6Therefore, augmented reality (AR) is developing very quickly in recent years.Before the popularity of VR, virtual collaboration tasks were becoming more significant for employees in commercial global enterprises or companies, and a virtual meeting system called V-ROOM 7 was developed to exchange and process instant metadata and information via the virtual environment. 8The proposed V-ROOM system integrated with an intelligent system and meeting summarization feature can improve the quality of summaries produced in the virtual meeting.
AR technology is currently used in a number of fields, such as medicine, education, and simulated training.It is also used within the tourism sector, aiming to improve the tourist experience, and so on. 9Among the VR/AR stakeholders, it is expected that the manufacturing and industrial sectors are key industries that will be impacted the most in the future. 10AR technologies are usually used for supporting maintenance operations 11 and product design in manufacturing and industrial industries. 12,13It is because AR provides a relatively new form of human-machine interaction that overlays virtual data or models a real product.Users can easily and rapidly correlate the AR context such as size and positions of the digital content with real products.Shen et al. 14 presented the application of AR to support concurrent engineering among multidisciplinary team members for collaborative product design.AR is believed to facilitate the design process by visualizing the product structure and information.It can enhance users' perception of product positions and sizes.However, teaching the concepts of products and design is not easy tasks involving a lot of engineering knowledge and the students' sense of geometric relationships and creativity. 15For students or individuals with learning difficulties, mixed reality (MR) allows interaction and feedback from the users, which is particularly important for enhancing experience in teaching and learning.MR is also known as hybrid reality, encompassing both VR and AR.MR not only merges the real and virtual worlds into a new environment, which combines physical and digital content, but also allows users to interact with the digital content dynamically in real time.The benefits of incorporating MR technology into educational experiences include better engagement and the opportunity for students to experience and better remember what they have learned. 168][19][20] Wide use of MR for simulated teaching and training is prevalent nowadays.Jad and Steven 21 presented the application and influence of MR on the productivity and quality of electrical conduit construction.An experimental methodology demonstrated the benefit of MR as enabling faster conduit construction with fewer errors compared to 2-D plans.Rouse et al. 22 explored the design strategies for MR in relation to Milgram's definition and explained most of the technical focus of Milgram's definition to capture the experiential dimensions from MR to MRx.The proposed strategies provide a humanistic framework of MRx for advanced experiences with the collaboration of three characteristics of MRx applications (esthetic, performative, and social).It can be employed as a generative tool for pushing the boundaries of contemporary interaction design in MR.Ke et al. 16 suggested that an MR-integrated learning environment, with the benefits of reinforcement of the sense of presence, potentially acts as an immersive platform for active teaching training.Quint et al. 23 proposed a system architecture for a MR-based learning environment, which combines physical objects and visualization of its digital content via AR.It allows dynamic interrelations between real and digital factory visible and tangible.Tamura et al. 24 explored the importance and advantages of using MR to augment the real world with synthetic electronic data for different kinds of applications.Bordegoni and Caruso 25 proposed an innovative methodology in MR for the automotive industry that allows a closer collaborative design review and modification of some components of automotive interiors, between designers, engineers, and end users.Professionals can work together via an interactive MR distributed design platform.The proposed methodology can effectively reduce the review and validation time during the car interior design stage.Kurilovas 26 presented a systematic review and analysis of the problem of quality evaluation and personalization of VR/AR/MR.A VR/AR/MR learning systems/environments quality evaluation and personalization framework was proposed and explained with quality evaluation methods and quality criteria models.The personalization framework of the proposed environments was concluded and based on learners "models/profiles using students" learning styles, intelligent technologies, and Semantic Web applications.Kirkley and Kirkley 27 explored the theoretical and design principles of constructivist learning environments for MR and addressed the use of methodologies and authoring systems with various tools to support the design process for MR application.In computer graphics, Knecht et al. 28 proposed a novel plausible rendering method for MR systems that can be applied in various real-life application scenarios.The proposed method combines the instant radiosity algorithm with differential rendering supporting more complex light-path combinations.Virtual objects can be seamlessly blended into the real environment.Amouri and Ababsa 29 proposed the design of a six-degree-of-freedom low-cost platform, which brings more realism to sliding movement.This system is applicable to many other applications like rehabilitation for people with motor disabilities, rescue environments and educative issues, and to reproduce natural motion feeling when the human is completely merged in MR environments.Besides, Pan et al. 30 demonstrated educational uses of the virtual learning environment (VLE) concerned with the issues of learning, training, and entertainment.VLE can be efficiently used for motivating and stimulating learners' understanding of traditional instruction-based learning, which has been proved inappropriate or difficult in the past, without MR.Besides, Mateu et al. 31 presented the potential use of an assistive tool of VirtualTouch for developing MR educational applications.The use of virtual worlds and tangible user interfaces was proposed to offer a "mixed reality" experience.The promising results indicated that teachers can easily design and implement instant learning modules in the classroom.Nevertheless, most of these MR projects mainly focus on users' experience.The effectiveness in comprehending product design is seldom investigated.Many research in education assessment have been studied to understand the effectiveness of different educational approaches in tertiary academic institutions. 32Therefore, in this project, an MR application was proposed and developed to help students in comprehending product design in tertiary education.The pretest and posttest were used to investigate the effectiveness of the MR application.We found that the current design of the MR application can not only help students in understanding geometric relationships but also improve their creativity, based on the experimental results.
This article aims to develop an MR application on a HoloLens system for the teaching and learning of product design.The geometry of 3-D objects can be visualized.The methodology of the proposed MR application is presented in the "Methodology" section.The hardware for suited MR tools, software for application development, and criteria for the experiments are also presented and demonstrated in this manuscript.A case scenario of an aircraft turbofan engine is developed and presented to implement the proposed MR application in order to teach students the complicated concepts of product design.The outcomes and recommendations for future work of the proposed MR application are discussed and explained in the "Results and discussion" section.Finally, the conclusions summarize the results, outcomes, and contributions of the MR proposed application.

Methodology
In this project, an MR application is developed for teaching product design.The application is also developed on the HoloLens system.HoloLens uses MR head-mounted display smart glasses.It is a self-contained, holographic computer that can engage with digital content and interact with holograms. 33Figure 1 shows an overview of the designed system configurations.It encompasses a wide range of experience, including AR and VR.People, places, and objects from the physical and virtual worlds can be merged together in a blended environment.It provides an immersive virtual environment for students to explore the concept of reality and human perception in a real-time perspective.Figure 2 shows an example of visualizing and exploring a design with the HoloLens system.The MR application allows students to visualize the size of product and to merge and compare with a real object.
Comparing with the VR and AR, MR not only allows interaction between the digital model and the users, but the digital model also interacts seamlessly with the real environment.For instance, MR can simulate the collision between the digital model and the objects in the real environment.To allow interaction with the real environment, the environment understanding cameras and the depth camera perform surface reconstruction of the real environment.The cameras are also used to perform hand tracking for gesture control by the users.
An application was designed that allows students to visualize the geometry of 3-D objects, as well as the exploded diagrams of selected components.A command manual is displayed at the bottom of the holographic screen.The students can command the system through the command manual or can signal the system through gazing, gesturing, and voice to implement instructions.We provided several commands and features in the application, including object transformation, reset, explode, and invisible view of selected components.Students can signal the system, as well as hearing the voice navigation from the system at the same time.In the voice navigation features, the system explains the functions and features of the selected components.Figure 3 summarizes the command provided in the developed application.
An aircraft turbofan engine is employed as the case scenario in the developed application to teach students the complicated concepts of product design.The application allows users to explore various major components of the aircraft engine using the exploded view or hidden parts.It can be signaled by gesture and voice commands.In order to signal the system, students need to select the application context through gazing, which acts as the hologram cursor.It can tie the user's intention with the application context.For instance, if you want to hide a holographic model, such as the fan part of the turbofan engine, you need to gaze at the fan part.In this case, the system can recognize that you are intended to signal the fan because users are usually intended to control the model that is gazing at.Then you can command the model through gestures or say the command "invisible" which is predefined in the application.In this application, we make use of the GestureManager script in the HoloToolkit to set up the air-tap gesture.Figure 4 shows the common air-tap gestures used in the HoloLens application.Bloom gesture is used to open the main menu of the HoloLens application, the ready and tap gestures are used to select the virtual model in the application context, and the hold and drag gestures are used to transform the model.This project makes use of Unity 36 as the game engine for application development.To increase the engagement of the students, we make use of Vuforia 37 as the Software Development Kit to enable the creation of the MR application that can recognize a specified teaching material.C# is used as the programming languages to develop the key features of the application.The model is displayed on the HoloLens once the student looks at the teaching material.Figure 5 shows the scenario of the model displayed on the HoloLens when the student looks at a specified teaching material.This not only allows students to visualize the digital model corresponding to the specific teaching material, but also enhances the students' engagement on the MR application.
In order to investigate the outcomes of the MR application, a group of students will be recruited and invited to participate in the investigation voluntarily.At the beginning of the investigation, students will be required to participate in a pretest to measure the quantitative result that the students can achieve in respect of a certain level of understanding in the five main areas of the assessment of components: the abilities of students in comprehending design, understanding product functions, visualizing 3-D geometry, understanding geometric relationship, and students' creativity.A car engine is used as the case study to setup the questions in the pretest.To determine the number of questions and time required for the tests, a preliminary rehearsal was conducted and based on the experience from the teachers, we decided to make use of nine questions for the test with 1 min for each question allowed.Two questions were used to assess each area of the students' ability, and the score for each question was equally weighted except the question used to assess the students' creativity.The test consisted of open-ended and multiple choice questions.The total time required for the students to answer the whole test paper was also recorded.Table 1 summarizes the question types and areas used to assess the students' abilities in the pretest and posttest.The sample questions used in the pretest is included in the Appendix 1.
Students will then be allowed to make use of the MR application to learn the product features, functions, and design.An aircraft turbofan engine model is used in the HoloLens in our investigation.At the beginning of the MR  experience sessions, a brief introduction on the operations of the HoloLens will be given and students will be given 5-10 min to familiarize with its operations.After students are familiarized with the HoloLens basic operations, they will be allowed to look at the given turbofan engine and understand its function and structures with 10 min time.Right after the MR experience session, a posttest with similar difficulty will be used to assess the students' abilities in the five main areas, as in the pretest.Students are required to answer the questions related to the turbofan engine model used in the HoloLens.Figure 6 shows the flowchart of the evaluation test.The time and the average scores for the students to complete the pre and posttests are denoted by t 1 and t 2 , and M 1 and M 2 , respectively.These quantitative data will be stored in the database for further processing.Figure 7 shows a sample question used in the pretest.The time bar indicating how much time is left to answer the test is shown at the top of the test paper.Only one attempt is allowed for each student and they are not allowed to go back to the previous question after submitting the answer for each question.

Results and discussion
In this investigation, the learning support to be provided to the students is the HoloLens MR learning application.To recruit students into participating in the tests, a Holo-Lens MR experience session has been organized and   opened to all students in different faculties of the Hong Kong Polytechnic University, including Faculty of Engineering, Faculty of Applied Science and Textiles, School of Design, and so on; 68 students were registered for the MR experience session and 45 of them attended the tests.Figure 8 shows the distribution of the students from the different faculties who participated in the MR experience session.From Figure 8, it was found that most students participated in the HoloLens experience session were studying in the School of Design and the Faculty of Engineering, which accounting for 38% and 31% of all participating students, respectively.The figures indicated that there is a high demand for the use of MR technology to assist in product design in the Faculty of Engineering and School of Design.The third most students participating in the experience session were from the Faculty of Business, which accounting for 13%.We collected students' opinions and feedback in participating in the experience session and summarized some of their reasons in joining the session as follows.At the commencement of the experience workshop, students were required to participate in the pretest to measure the quantitative result of five main areas of students' abilities: comprehending design, understand the product functions, visualizing 3-D geometry, understanding geometric relationship, and students' creativity.The average marks of each area were calculated and evaluated.Figure 9 compares the average marks obtained by the students in the pretest and the posttest.It indicated that the marks in understanding the product function are very similar for the pretest and the posttest which are 3.67% and 3.68%, respectively.Comparing the students' abilities in comprehending design and visualizing 3-D geometry, the marks obtained in the pretest were higher than in the posttest.The mark difference was around 0.86% and 0.56% in comprehending design and understanding product function, respectively.It may be due to the posttest made use of a more complex turbofan engine as the case scenario.The turbofan engine involves much more complicated 3-D geometries and functions compared with the car engine used in the pretest.Figure 10 compared the mark difference between pretest and posttest for each area of students' abilities.Students' abilities in understanding geometric relationships and creativity in the posttest were 2.77% and 1.07% higher than the pretest, respectively, whereas the tests scores were very similar for visualizing product 3-D geometry.The results reflect the fact that the transformation functions (i.e.rotation) in the designed HoloLens application can assist students to visualize a 3-D model in different angles and positions.Therefore, it helps the students in understanding the product geometric relationship.We believe that students understanding in geometric relationships have a positive and pronounced effect in students' creativity.On the other hand, the average time for students to complete the tests were measured and evaluated.The average time of students to complete the pretest and posttest was 5 h 17 min and 4 h 8 min, respectively.The results show that the average time for students to complete the posttest was over 1 min faster than the pretest.It is believed that the HoloLens is an effective way for students to learn and study the product design.
Despite experimental results showing that the current HoloLens application can improve students' understanding in geometric relationship and creativity, there is no significant improvement in visualizing 3-D geometry.On the other hand, students' abilities in comprehending design and understanding product functions are even lower in the posttest.Although it may be due to the posttest making use of a more complicated turbofan engine as the case study, improvement in these two areas of students' abilities is needed.In the future, additional features in the HoloLens application will be designed and integrated, such as providing more explanations and display some key vocabulary when the users look at a certain component.
In addition, the results showed that students' understanding in geometric relationships also improved creativity.In the future, more research work will be conducted and explored to investigate the relationship between understanding geometric relationships and creativity and the key factors that improve students' creativity in product design.The results in this study reflected that the MR HoloLens application is an appropriate supplementary tool to assist teachers in teaching product design, and it can combine and integrate with other teaching materials such as exercises, tests, videos, and so on to synergy further new teaching pedagogies.

Conclusion
In this project, the MR application was developed for teaching product design to university students to enhance the multisensory learning experience and to improve the teaching effectiveness on complicated product design from the traditional teaching and learning methodology in a lesson.The MR application was developed on the HoloLens system, and a practicable application has been designed that allows students to visualize the geometry of 3-D objects.Pretests and posttests of similar difficulties were designed to investigate the effectiveness of the MR application in teaching product design and to assess the students' abilities in five main areas.The experimental results illustrate that the proposed application design positively improves students' understanding in geometric relationships and creativity.Besides, the results of the average time for students to complete the tests illustrate that the pretest is faster than the posttest.The results reflect that the Holo-Lens tool is an effective way for students to learn and study complicated product design, particularly in understanding the geometric relationships in a design and in improving students' creativity.In the future, more features will be added to the MR application to further improve students' abilities in learning and training in other aspects.

Figure 1 .
Figure 1.The overview of the HoloLens system configuration. 34

Figure 2 .
Figure 2.An example of visualizing and exploring a design with the HoloLens system. 35

Figure 4 .
Figure 4.The common air-tap gesture used in the HoloLens application.

Figure 3 .
Figure 3. Summary of the features provided in the developed application.

Figure 5 .
Figure 5.The model is displayed on the HoloLens when the student looks at a specified teaching material.

Figure 6 .
Figure 6.The flowchart of the evaluation test.

Student 1 from
Faculty of Engineering: I am doing a VR project and I would like to see how this MR experience session can help in the development of various applications.

Figure 7 .
Figure 7. Sample question used in the pretest.

Figure 8 .
Figure 8. Distribution of students from different faculties participated in the MR HoloLens experience session.MR: mixed reality.

Figure 9 .Figure 10 .
Figure 9. Average marks for different categories of questions in the pretest and posttest.

Table 1 .
Questions type and categories used in the tests.