WO2023148530A1

WO2023148530A1 - Generic xr game-based system for industrial training

Info

Publication number: WO2023148530A1
Application number: PCT/IB2022/051386
Authority: WO
Inventors: Eliana Sofia DIAS OLIVEIRA; Andreia Carina SOUSA REIS; Luís GONZAGA MENDES MAGALHÃES; Rui Manuel ALVES DA SILVA E SOUSA; Rui Manuel DE SÁ PEREIRA DE LIMA; Nuno Miguel EIRA DE SOUSA; Telmo Miguel OLIVEIRA ADÃO; Marcelo António ARAÚJO ALVES PIRES; Luis Filipe DE CARVALHO EVANGELISTA; Cristiano DE JESUS; Ingrid TEIXEIRA DE SOUZA; Rui Pedro NEVES GUIMARÃES; Aléxis Francisco MACHADO SANTOS
Original assignee: Bosch Car Multimedia Portugal, S.A.; Universidade Do Minho
Priority date: 2022-02-02
Filing date: 2022-02-16
Publication date: 2023-08-10

Abstract

The present invention describes a system and method to train industrial production processes operators. The developed system allows a complete training session to a trainee, said system possessing and guaranteeing the capability to adapt and evolve along time, following changes in the processes and/or machines/tools in an industrial environment. In order to increase the engagement and consequently the development of competences of the operators/trainees, a Game Based Learning (GBL) methodology provided along with Virtual Reality (VR) and Augmented Reality AR) is used in the training context.

Description

DESCRIPTION

"Generic XR game-based system for industrial training"

Technical Field

The present application describes a system and method to train industrial production processes operators .

Background art

Document US2020410286A1 , entitled Learned Interaction with A Virtual Scenario , provides a description on learned interactions in a virtual reality or augmented reality ( collectively, a computer-mediated reality) can include determining a probable physical action of a user interacting with a computer-mediated reality ( CMR) environment . A learned interaction corresponding to the probable physical action can be generated based on a CMR-physical action ( CMRPA) model that correlates phys ical actions with results of the physical actions in a CMR scenario of the CMR environment . In response to determining, based on at least one identi fied characteristic of the user, a statistical likelihood of benefiting the user by providing the learned interaction, a learned interaction corresponding to the probable physical action can be provided to the user .

Summary

The present invention describes a system ( 100 ) for training industrial production operators comprising a Virtual Reality module ; an Augmented Reality module ; and a training management module ; the Virtual Reality module , the Augmented Reality module and the training management module being configured to interchange data with a File System and with a database through a Web Service database access ; and wherein the Virtual Reality module and the Augmented Reality module are further configured to simultaneously provide to a trainee a game-based learning model comprised of scenarios that develop the trainee professional competence skil ls , said scenarios comprising challenges to be developed and problems to be solved based on industrial machine operation simulations .

In a proposed embodiment of present invention, the Virtual Reality module comprises a physical interactions simulator, a communications module , a gami f ication module , a Virtual Reality task executor and a machine simulator .

Yet in another proposed embodiment of present invention, the Augmented Reality module comprises Augmented Reality application, a noti fication viewer, a performance viewer and a content viewer .

Yet in another proposed embodiment of present invention, the content viewer comprises a selection of training content topics .

Yet in another proposed embodiment of present invention, the Augmented Reality application is configured to interchange data with the communication system component , which further communicates with the Web Service database access and the gami fication module .

Yet in another proposed embodiment of present invention, the training management module comprises a main application linked to a set of training plans management , trainers management , machine management , Virtual Reality training management , Augmented Reality training management , communication management , content management and trainee management .

Yet in another proposed embodiment of present invention, the training management module , the Virtual Reality module and the Augmented Reality module are configured to customi ze and manage the scenarios and problems of the game-based learning model by means of a training manager .

The present invention additionally describes the operating method of the system for training industrial production operators according to the previous description, comprising the steps of the trainee interacting with the Virtual Reality module and with the Augmented Reality module ; and wherein both interactions comprise steps to train and learn to operate a machine in an industrial environment .

In a proposed embodiment , the operating method of the system for training industrial production operators comprises the steps of the training manager interacting with the training management module to access the profile of the trainee ; the training manager interacting with the Virtual Reality module and with the Augmented Reality module to customi ze , add, edit or adapt the training sessions based on the trainee profile .

The present invention further describes a computer program configured to carry out every step of the described operating method .

The present invention further describes a (non-transitory ) machine-readable storage device , on which the computer program configured to carry every step of the described operating method is stored .

The present invention further describes a data processing system, comprising the necessary physical means for the execution of the computer program configured to carry every step of the described operating method .

The present invention further describes an electronic control unit , configured to carry out every step of the operating method herein disclosed .

General Description

The herein disclosed invention describes a system and method to train industrial production processes operators involving Game Based Learning ( GBL ) supported by Virtual Reality (VR) and Augmented Reality (AR) .

Some of present existing training augmented reality solutions allow to create training guides based on existing technical documentation and re-use any existing models . They also encrypt all the data to increase its protection and allow users to make an automatic maintenance report improving management quality, comprising also interactive touchpoints that lets the user interact with clickable points of interest that display information about it . Some other features are included as cross platform support , of f line/online data, AR mode , VR mode ( allowing to view 3D information without the physical obj ect or in environments where the machine is not fully accessible from all angles ) , component filter, detail view, Optical Character Recognition ( OCR) , Quick Response ( QR) and barcode scanning and multi-language support . Additional existing solutions are based on cloud AR, VR & 3D presentation platforms that enable users to quickly produce photorealistic, interactive , multimedia experiences for their audiences by simply dragging and dropping obj ects to populate their scene , edit or add animations and interactivity, and then publish to the desired platform, requiring no coding or programming .

Few solutions provide means to configure the training sessions but with a limited set of configurations . Mostly they focus on providing the possibility to define the content to be shown . When it comes to the simulation of the machines behavior it is even more limited .

Although there are some similar solution approaches in the market , the known solutions are designed solely as VR or AR- based learning environments , and few refer or clearly anticipate solutions for both combined learning environments . Additionally, existing somehow similar solutions do not have clear links with the production system training requirements neither with ef fective learning personalised approaches .

Therefore , the developed system allows a complete training session possessing and guaranteeing the capability to adapt and evolve along time , following changes in the processes and/or machines/ tools . In order to increase the engagement and consequently the development of competences of the operators/ trainees , a Game Based Learning ( GBL ) methodology is used in the training context . A Virtual Reality (VR) training environment is able to provide a safe learning environment where trainees perform tasks without being worried about damaging the machines or tools or harming themselves . It is also possible to simulate any kind of machine and all possible situations including hazard ones . On the other hand, the Augmented Reality (AR) environment allows trainees to work in a real environment and receive instructions on how to perform tasks . VR and AR learning environments provide di f ferent features and usually are used with di f ferent aims . Both possess advantages , so a learning system that provides both is a more complete one . The VR learning environment can be used in an earlier stage of the training program, when the trainees do not have yet enough knowledge and experience to work in a real environment . The AR learning environment can be used in more advanced stages of the training program, when the trainees already have the necessary knowledge and experience to work/train in a real environment but still need some help .

Although combining the previously mentioned features , additionally, the developed platform includes a diverse file support , a 3D visuali zer, an asset and template library, no plugins required, being optimized for all devices , allowing an interactive presentation, and performing real time rendering .

Several technical concepts and techniques are involved in the present invention, namely :

( i ) Instructional Design, Work-Based Learning and Gami fication;

( ii ) Virtual Reality (VR) ; and

( iii ) Augmented Reality (AR) . The integration of all the above-mentioned concepts in order to develop and deliver an ef ficient training system (methodology and platform) is in itsel f an innovation with regard to known technologies . Additionally, an innovative aspect of the methodology for creating training courses or procedures in the present development is related with the fact that it incorporates some tools that are widely used in the context of production processes , more speci fically, the Business Process Modelling and Notation (BPMN) diagrams and the I shikawa diagrams . An innovative process for the definition of competences to be developed, based on the analysis of the production process itsel f (using work instructions , BPMN models of production process and I shikawa diagrams of relevant problems ) , is proposed .

The methodology how the game-based learning model is defined is also innovative . In this part of the training design process , scenarios are defined as challenges related to the "competences" to be developed (BPMN diagrams of the production process ) and the problems to be solved ( I shikawa diagrams ) . These scenarios/challenges are the essence of the Game Based Learning model .

The VR component is responsible for training the new industrial machine operators , i . e . , the future users of the industrial machines , leading them to perform predetermined tasks and receive scores depending on their performance , thus materiali zing a gami fication component , in order to be evaluated and improve their skills .

The AR component is configured to display information about a certain machine and help the operator to perform its operations . The system has a training management component , implemented through a Backof fice web application that feeds a remote database , allowing to def ine/conf igure the training sessions , which can be tailored to the trainee profile , configure the VR training sessions , configure each machine/tool simulation (which allows the introduction of new machines in the training or the reconfiguration of the existing ones ) , and finally has the possibility to configure the AR training sessions .

The use of Game Based Learning will be implemented in the training itsel f in order to keep learning motivation high . The goal is to engage learners from the training process beginning .

The training system will include learning obj ects ( e . g . scenarios/challenges ) adapted to the di f ferent learning styles (profiles ) of the trainees , allowing thus the personali zation of learning paths , which also consider the performance of the trainee during the training process . This is another innovative aspect of the system that will contribute to increase the ef ficiency of the training process .

Brief description of the drawings

For better understanding of the present application, figures representing preferred embodiments are herein attached which, however, are not intended to limit the technique disclosed herein . Figure 1 - illustrates the overall Training Platform

Architecture of the developed system (100) .

Figure 2 - illustrates the VR application module (102) .

Figure 3 - illustrates the AR application module (104) .

Figure 4 - illustrates the training management application module (106) .

Description of Embodiments

With reference to the figures, some embodiments are now described in more detail, which are however not intended to limit the scope of the present application.

The proposed system (100) discloses a software implemented platform and an operating method thereof particularly developed to train future industrial production machine/ tool users .

Supported by Figure 1, which discloses the overall Training Platform Architecture (100) of the system, it is possible to identify the data flow between a VR application module (102) and an AR application module (104) with a Web Service database access (105) . Both the AR application module (104) and VR application module (102) also perform data exchanges with a File System (101) , which in turn exchanges data with a training management application module (106) . The Web Service database access (105) is also configured to exchange data with the training management application module (106) and with a database (108) . Additionally, the system (100) will be operated by a trainee (103) , being managed if required by a training manager (107) . By application, referred throughout the constituting modules of the system, let's consider a software that bundles together certain features in a way that is accessible to a user.

Figure 2 depicts the VR application module (102) which comprises a physical interactions simulator (1021) , a communications module (1022) , a gamification module (1023) and a VR task executor (1024) . The interaction with said module (102) if carried out by the trainee (103) .

On Figure 3 is illustrated the the AR application module (104) , which in a similar way will be available to the trainee (103) and to the training manager (107) for the determined proposes, comprised of a communications module (1022) and a gamification module (1023) , receiving data from an AR application (1041) , that further is data-driven by a notification viewer (1042) , a performance viewer (1043) and a content viewer (1044) , being this last module (1044) provided with a quick selection of training content topics (1045) .

Finally, Figure 4 illustrates the training management application module (106) , being centrally managed by a main application (1061) that interacts with a training plans management (1062) , a trainers management (1063) , a machine management (1064) , a VR training management (1065) , a AR training management (1066) , a communication management (1067) and a trainee management (1069) modules.

The system (100) was developed based on instructional design and on a game-based learning strategy. The proposed methodology is oriented by Instructional Design, which is based on the ADDIE concept, an acronym for Analyse, Design, Develop, Implement, and Evaluate, which are the stages for a training program. Thus, the development and creation process were mapped so that operations are organized and structured in phases foreseen in the ADDIE concept.

The detailed survey of requirements for the development of the training system (100) for future user trainees (103) , as well as the identification and characterization of the competences to be developed by those trainees (103) , incorporates the use of tools that were never used in this technical field, namely the Business Process Modelling and Notation (BPNM) and Ishikawa Diagrams.

The competences to be developed by the trainees (103) will be identified from the process characterization, represented from the BPMN modeling procedure, and also based on the survey and classification of occurrences (e.g. problems in the production process) through an Ishikawa Diagram.

The creation of scenarios, or game challenges, is carried out taking into account not only the desired competences of the trainees (103) , identified through BPMN and Ishikawa diagrams, but also the trainees' learning styles, which are classified according to the Felder-Silverman model. Scenarios will be developed so that trainees (103) develop competences to deal not only with the normal operation of the process, but also with the problematic situations that may occur, e.g. machine breakdown or malfunction. The gamebased learning approach will be used, incorporating virtual and augmented reality to simulate machine operation scenarios . The proposed system (100) has two fundamental characteristics: personalization and adaptability. The personalization of the training scenarios corresponds to the presentation of an environment according to the preferences and characteristics of the trainees (103) learning style. Adaptability includes the training environment configuration according to the performance and needs of the trainees (103) as they interact with the system (100) .

Learning objects are the elements that make it possible to implement these characteristics and may include: virtual reality and augmented reality resources, videos, demonstrations, exercises, assessments, statements, references, topics, lists, graphics, images, emails, discussion forums, among others. The environment will be geared both to classroom training and to support the trainee in the execution of operations in a real environment.

The system (100) is composed by three main modules: the VR application module (102) , the AR application module (104) and the training management application module (106) .

Additionally, the system (100) is also comprised by a File System (101) , a Web Service database access (105) and a database (108) . These modules, combined together, define the developed system (100) , satisfying all the training platform needs. Each module is divided into other components that, through the external Communication Management component (1067) , communicate with each other using Web Services.

The system was developed to operate with two main user types: the training manager (107) , which is responsible for def ining/conf iguring the training courses, and the trainee (103) , which will attend the trainings.

The training manager (107) uses the training management application (106) to access the profile of the trainee (103) in order to obtain relevant data about him, to customize VR/AR training sessions in order to add or edit content and adapt as necessary, and to edit the machines according to the needs of the trainings.

The trainee (103) will use the VR and AR applications (102,

104) to learn and train the steps needed to operate the machines. All data involved at this stage will be obtained or entered into the database through web services.

The VR training module corresponds to the VR application module (102) , and it is divided into five components. The Gamification component (1023) is the component that will give the application its game characteristics, such as a points system, and will interact directly with the VR Task Executor (1024) and Communications components (1022) . The VR Task Executor component (1024) is responsible for managing the training tasks, which will worth a certain number of points, depending on how well the trainee performs. The Machine Simulator component (1025) will receive the files related to the machine, such as 3D models and animation setup files, to create a virtual machine with characteristics similar to the real machine. The Physical Interactions Simulator component (1021) will use the files related to the physical feedback that the user/trainee (103) will receive, when interacting with different virtual objects, through VR gloves with force feedback. The Communications component (1022) will interact with the other components of the system (100) , like the Web Services (105) and File System (101) components. This File System component (101) is needed in order to have access to files that are being hosted in the database. The Machine Simulator (1025) and the Physical Interactions Simulator (1021) components will then use those files to perform their part of the job.

The AR training module (104) is mainly composed by an AR application (1041) , a notification viewer (1042) , a performance viewer (1043) and a content viewer (1044) . Here, the trainees (103) are able to visualize the contents (1044) associated with their training and may select the training content topics (1045) that they need or want to see at a given moment. All data generated from this AR application (1041) is interchanged with the communication system component (1022) , which further communicates with the Web Service system component (105) and the gamification module (1023) .

The training management application module (106) , implemented as a web platform, allows to customize the training courses and thus manage them. It is mainly comprised of a main application (1061) linked to a set of training plans management (1062) , trainers management (1063) , machine management (1064) , VR training management (1065) , AR training management (1066) , communication management (1067) , content management (1068) and trainee management (1069) blocks. The main application (1061) corresponds to the entry point of the application and is responsible for user authentication and presenting the main menu. The training plans management (1062) allows to create, edit and delete training plans, and the trainers management (1063) allows to create, edit and delete information regarding trainers. The machine management (1064) allows to create, edit and delete information regarding machines, and the VR training management (1065) allows to configure each VR training session by defining the tasks that the trainee should perform, the points that he gets from each task when successfully complete it, and the content to be shown. The AR training management (1066) allows to configure each AR training session by defining which information and how and when it is displayed. The communication management (1067) is responsible for managing the communication between this module and the web services component and other third-party services, if needed. The content management (1068) allows to add, edit and delete content building a content Library; the VR training management and the AR training management will use this library to associate content to the tasks. Finally, the trainee management (1069) allows to create, edit and delete information regarding trainees

In both VR and AR Training Management Modules (102, 104) , the training manager (107) is able to manage the sessions and the feedback associated to the objects as desired. Everything at this point can be customized according to the needs of the training course.

The embodiments of this invention constitute a complete, flexible and generic training system (100) that provides an engaging learning environment, which supports the trainee (103) during all stages of the training program and allows to conf igure/personalize and monitor the training programmes. The herein disclosed invention allows decreasing the amount of time needed to train new operators and decreases the number of resources needed. This overcomes existing training systems since these have only a limited set of training configuration possibilities and use a more traditional learning approach, which is less engaging .

Claims

1. System (100) for training industrial production operators comprising a Virtual Reality module (102) ; an Augmented Reality module (104) ; and a training management module (106) ; the Virtual Reality module (102) , the Augmented Reality module (104) and the training management module (106) being configured to interchange data with a File System (101) and with a database (108) through a Web Service database access (105) ; and wherein the Virtual Reality module (102) and the Augmented Reality module (104) are further configured to simultaneously provide to a trainee (103) a game-based learning model comprised of scenarios that develop the trainee (103) professional competence skills, said scenarios comprising challenges to be developed and problems to be solved based on industrial machine operation simulations.

2. System (100) according to the previous claim, wherein the Virtual Reality module (102) comprises a physical interactions simulator (1021) , a communications module (1022) , a gamification module (1023) , a Virtual Reality task executor (1024) and a machine simulator (1025) .

3. System (100) according to any of the previous claims, wherein the Augmented Reality module (104) comprises Augmented Reality application (1041) , a notification viewer

(1042) , a performance viewer (1043) and a content viewer (1044) .

4. System (100) according to any of the previous claims, wherein the content viewer (1044) comprises a selection of training content topics (1045) .

5. System (100) according to any of the previous claims, wherein the Augmented Reality application (1041) is configured to interchange data with the communication system component (1022) , which further communicates with the Web Service database access (105) and the gamification module (1023) .

6. System (100) according to any of the previous claims, wherein the training management module (106) comprises a main application (1061) linked to a set of training plans management (1062) , trainers management (1063) , machine management (1064) , Virtual Reality training management (1065) , Augmented Reality training management (1066) , communication management (1067) , content management (1068) and trainee management (1069) .

7. System (100) according to any of the previous claims, wherein the training management module (106) , the Virtual Reality module (102) and the Augmented Reality module (104) are configured to customize and manage the scenarios and problems of the game-based learning model by means of a training manager (107) .

8. Operating method of the system (100) for training industrial production operators according to any of the previous claims, comprising the steps of the trainee (103) interacting with the Virtual Reality module (102) and with the Augmented Reality module (104) ; and wherein both interactions comprise steps to train and learn to operate a machine in an industrial environment.

9. Operating method of the system (100) for training industrial production operators according to any of the previous claims, comprising the steps of the training manager (107) interacting with the training management module (106) to access the profile of the trainee (103) ; the training manager (107) interacting with the Virtual Reality module (102) and with the Augmented Reality module (104) to customize, add, edit or adapt the training sessions based on the trainee (103) profile.

10. Computer program configured to carry out every step of one of the operating methods described in claims 9 and 10.

11. (Non-transitory ) Machine-readable storage device in which the computer program of claim 10 is stored.

12. Data processing system comprising the necessary physical means for the execution of the computer program of claim 10.

13. Electronic control unit, configured to carry out every step of one of the operating methods of claims 9 to 10.