KR101671320B1 - Virtual network training processing unit included client system of immersive virtual training system that enables recognition of respective virtual training space and collective and organizational cooperative training in shared virtual workspace of number of trainees through multiple access and immersive virtual training method using thereof - Google Patents

Abstract

 The present invention provides a virtual space on the screen displayed on the client HMD device, which is provided to each trainee and is connected to the client HMD device and the client motion recognition device, in cooperation with the master system 300 that is provided to the training manager to monitor the training process The client HMD apparatus and the client motion recognition apparatus are initialized to the virtual training processing apparatus 200 of the client system for performing the training in the virtual work space and the motion recognition / A display unit 210; A client system processing unit 220 for performing network connection with the master system 300 and loading training content composed of 3D contents, receiving and processing motion data recognized from the client motion recognition apparatus, And a training scenario processor 230 for initializing training contents composed of 3D contents, creating and loading a character of a trainee in the virtual work space, and controlling the operation of the training contents in the virtual work space to proceed .

Description

A network virtual training processing apparatus constituting a client system of a immersive network virtual training system capable of recognizing individual virtual training spaces of multiple trainees through a multiple access and collectively and systematically collaborating in a shared virtual work space and a device using the same A virtual network training method is proposed. {Virtual network training processing unit included client system of immersive virtual training system that enables each virtual training space and collective and organizational cooperative training in shared virtual workspace of number of trainees through multiple access and immersive virtual training method using them}

The present invention relates to a network virtual training apparatus for constructing a client system of a virtual training system and a method for immersive network virtual training using the same, and more particularly, For example, it is possible for a plurality of trainers participating in virtual space education to recognize a virtual training space including individual trainees for performing unexpected disaster response training, The purpose of this study was to develop a system for training and supervising the immersive network virtual training system which enables the training manager to supervise the contents of the training and to measure the results. Work environments that require action, for example, (VR) to create a virtual training space by integrating computer graphics technology and computational application technology, and to share the virtual training space composed of each training participant, and training including training purpose and content Scenario and Training Manager Based on the behavioral guidelines, the multiple interactions allow multiple trainers to recognize the individual virtual training spaces and collectively organize collaborative training in a shared virtual training space. To a network virtual training processing apparatus constituting a client system of a training system, and to a immersive network virtual training method using the same.

Currently, the virtual training system and the virtual training method through it are being used to train work in various industries (game, medical, architecture, aviation, nuclear power), and in particular, disaster (human disaster and natural disaster) Through various training scenarios, job training in a virtual training space, which is provided through a digital display, can be used for training in situations, training for technical security, or work in dangerous industrial sites. , Lack of training and lack of experience to prevent human errors that can occur, resulting in tremendous economic losses and loss of life is being used as an effective training method.

This virtual training system has been linked with various hardware and software to the virtual training system due to the development of the display and software market since 2010. In particular, HMD (Head Mounted Display), which was expensive equipment as a virtual reality (VR) And the motion recognition device capable of recognizing the motion of the person has improved performance and the price is lowered so that the HMD (Head Mounted Display) and the virtual training system in which the motion recognition device is installed as hardware It can be commercialized.

On the other hand, in spite of the hardware and software developments, the virtual training system is generally a one - person virtual training system in which one trainee connects and experiences and trains by himself.

On the other hand, according to the results of analyzing the accident caused by the human error in the industrial field, about 70% of the incidents and accidents did not utilize the resources available to the team members directly or indirectly And in the unexperienced environment of disasters, it has caused enormous loss of life and economic loss due to improper decision making and ineffective communication between each other. These results have been found to be related to inadequate leadership and poor management skills, and there is a strong demand for training of team members' resource management skills based on cooperation and communication among team members.

Therefore, it is possible for a plurality of users who are team members to simultaneously experience and work on the same work environment and work in a virtual space, recognize and share the virtual work space among the team members at the same time, communicate with each other, It is urgently required to develop an immersive network virtual training system capable of recognizing individual virtual training spaces of a plurality of trainees through a plurality of trains capable of training and collective and systematic cooperative training in a shared virtual training space.

Accordingly, the present invention enables training of multiple users who are team members at the same time in real time in a virtual work space, training of communication between team members and between team members and team members by recognizing and sharing the virtual work space among team members at their own time , An immersive network virtual training system capable of recognizing individual virtual training spaces of a plurality of trainees through a multiple access and collective and systematic cooperative training in a shared virtual training space, and an immersive network virtual training method using the same The purpose.

In addition, through the above-mentioned multiple connection, it is possible to recognize the individual virtual training space of a plurality of trainees and to realize a network system capable of constituting a client system of each trainee in a immersive network virtual training system capable of collective and systematic cooperation training in a shared virtual training space A virtual training processing apparatus and an immersive network virtual training method using the same are provided.

Furthermore, the present invention provides an immersive network virtual training system capable of collaborative training in a virtual work space based on multiple clients and a immersive network virtual training method using the same, thereby enabling installation and use at a low cost, It is an object of the present invention to provide a immersive network virtual training system capable of preventing overload of a server and thus delaying connection and program processing and capable of configuring a network in various ways so as to be applicable to various collaborative training and immersive network virtual training method .

 In order to achieve the above object, in one aspect, the present invention is provided to a training system, which is provided to a training manager so as to monitor a training process, is provided to each trainee and connected to a client HMD apparatus and a client motion recognition apparatus The client HMD device and the client motion recognition device are initialized by the virtual training processing device 200 of the client system that performs training in the virtual work space which is a virtual space on the screen displayed on the client HMD device, A motion recognition / display unit 210 for setting each function in conjunction with the master system 300, a network connection with the master system 300, loading training content composed of 3D contents, receiving motion data recognized from the client motion recognition apparatus, A client system processing unit 220 for processing the 3D content, And a training scenario processing unit (230) for initializing the training contents composed of the training contents in the virtual work space and generating and loading the characters of the trainee in the virtual work space, 210) includes a client HMD tracking module (212) for computing data according to a movement of a trainee head recognized from an HMD tracking module sensor of a client HMD device and generating motion data of a trainee head; A client HMD camera link module (213) linking a camera view display of the client HMD device to a camera of a trainee character located in a virtual work space; A display module 214 for transmitting to the screen of the HMD device a screen to be rendered on the camera of the trainee character at the time and the time of the visual field seen by the trainee character in the virtual work space; A motion tracking module (216) for receiving data according to a movement of the trainee from the clinic motion recognition device and extracting motion data of the trainee and tracking the movement of the trainee; And a motion link module (217) for linking the motion data of the trainee to the joint data of the trainee character of the virtual work space, wherein the client system processing unit (220) And a motion process module (224) for comparing the motion of the trainee recognized from the data to determine whether the motion is matched, extracting motion accuracy for the motion of the trainee based on whether or not the determined motion is matched, The processing unit 230 includes a training data / message receiving module 233 for receiving information on the task and the task authority from the master system 300 so that the training data / message receiving module 233 Receives data relating to the task authorization from the master system 300 and via the training data / message receiving module 233 The motion tracking module 216 is activated and performs motion tracking to transmit the tracked motion data values to the motion link module 217. [ The screen of the character of the other trainee who is granted the task right at the viewpoint and time of the sight of the character of the trainee in the virtual work space is displayed on the screen of the HMD device The motion tracking module 216 is disabled and does not perform the motion tracking. If there is a message to be delivered to another trainee, the motion tracking module 216 receives the message and transmits the message through chat or voice communication. Through chat or voice communication.

delete

delete

delete

delete

Preferably, the training scenario processing unit 230 includes: a training content initialization module 231 for initializing the training content in the virtual space; And a trainee character loading module 232 for receiving characters from the master system 300 and creating the trainee character in the initialized virtual workspace. The trainee character loading module 232 includes a character object allocated to the trainee, And loads the character object to be displayed at a specific position in the virtual work space according to position information of the character object in the virtual work space.

delete

The training scenario processing unit 230 preferably includes a training result information generation module 235 for generating a training result including a motion accuracy and an operation time in the form of data and transmitting the data to the master system.

Further, preferably, the virtual workspace is configured to include objects for a work environment, a work character, a work tool, and a work object.

Here, it is preferable that an object for the work object is set as a collider.

In another aspect, the present invention provides a method and system for collaborating in a virtual space using a immersive network virtual training system including a virtual training processor of a client system capable of collective spatial recognition and collective and systematic cooperative training in a shared virtual workspace through multiple connections The client HMD device and the motion recognition device are initialized to receive and confirm the ID to establish a connection, set a GUI screen for connection to a virtual work space, and transmit the master system 300 A system initialization step (SlOO) for providing a GUI screen that can be connected to the GUI screen; A network setup / connection step (S200) of setting up a network connection environment with the master server (300) according to information inputted and inputting the IP of the client system, the connectioner ID and the connection room through the provided GUI, and connecting to the network; A content initialization step (S300) of loading training content composed of 3D contents, setting a loaded 3D object to a preset value, and placing the loaded 3D object on a virtual space; Receives the task authorization from the master system, receives the task authorization, receives a message related to the task content, and displays the task authorization message to the trainee. If it is determined that the task authorization is granted, A training event processing step (S400) of changing a position value, a rotation value, and a magnitude value of an object constituting a virtual work space of a trainee, which is recognized according to motion input and motion analysis, to each object of a virtual training environment of a trainee ); A performance data transmission / confirmation step (S500) of transmitting training data and final training result information to the master system; (S600) for generating and providing screen data to a trainee client system whose training has been terminated so as to provide a virtual work environment of another trainee who is given training authority to the trainee, In the training event processing step S400, when it is determined that the task authority is not granted, the task recognition is not performed for the trainee, but the task authority is set at the viewpoint and the time of the sight that the character of the trainee A training image data generation step of generating a training data including a work time, a movement path deviation, and a training gesture motion, The performance data transmission / confirmation step (S500) may transmit the training data to the master system The operation monitoring step S600 may include a step of not performing the motion recognition for the trainee who has finished the training but the operation of recognizing the character of the other trainee When the trainee has received a message to be transmitted to another trainee, the message is transmitted through chat or voice communication, and a message of another trainee or training manager is transmitted to the trainee Chat or voice communication.

delete

Preferably, the objects constituting the virtual work space include objects of a work surrounding environment, a work character, a work tool, and a work object.

Here, the content initialization step S300 may be performed by setting a call reader in the work object.

Preferably, the training event processing step (S400) includes generating a training result information, and generating data on whether a work time, a movement path deviation, and a training gesture operation match.

delete

According to the present invention, multiple users who are team members can experience real-time experience and training in a virtual work space at the same time, and can recognize the virtual work space among team members at the same time and can train communication between team members and between team leaders and team members It is possible to provide an immersive network virtual training system capable of collaborative training in a virtual work space through a multiple access and a immersive network virtual training method through the same.

According to the present invention, there is provided an immersive network virtual training system capable of cooperative training in a virtual work space based on multiple clients and a immersive network virtual training method using the same, It prevents the delay of program processing and can be applied to various collaborative training because network configuration is possible in various ways.

Furthermore, by applying the technique of the present invention, it is possible to easily incorporate into programs such as disaster training training and hazardous facility safety education simulation, and it is possible to expect practical training effect by applying to disaster and educational safety contents produced by one person training It is possible to provide a technology that can be upgraded immersed through multiple networks.

1 is a block diagram of an immersive network virtual training system capable of collaborative training in a virtual workspace through multiple connections of the present invention.
2A is a configuration diagram of a client motion recognition / display unit 210 according to an embodiment of the present invention.
2B is an output screen of each joint name recognized and motion-mapped by the client motion recognition / display unit 210 according to an embodiment of the present invention.
3 is a configuration diagram of a client system processing unit 220 according to an embodiment of the present invention.
4 is a configuration diagram of a training scenario processing unit 230 according to an embodiment of the present invention.
5 is a configuration diagram of a master system 300 according to an embodiment of the present invention.
FIG. 6 illustrates an actual implementation of an immersive network virtual training system capable of collaborative training in a virtual workspace based on multiple clients of the present invention; FIG.
FIG. 7 is an implementation screen of a virtual work environment according to an embodiment of the present invention; FIG.
8 is a flow diagram of a immersive network virtual training method using the immersive network virtual training system of the present invention.
9 is a time table of the immersive network virtual training method using the immersive network virtual training system of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which the claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

1 is a block diagram of an immersive network virtual training system capable of collaborative training in a virtual workspace through multiple connections of the present invention.

Referring to FIG. 1, the immersive network virtual training system capable of cooperative training in a virtual work space through multiple accesses of the present invention includes a plurality of trainer client systems including a client HMD (Head Mounted Display) device and a client motion device 100) (client system 1, client system 2 ... client system n), network virtual training processing device 200, and master system 300.

Each of the trainer client systems 100 includes a client terminal (not shown) connected to the client HMD device and the client motion recognition device. The client motion recognition device includes a contact type motion recognition device (or a contact type motion recognition controller) And a non-contact motion recognition device (or contact type motion recognition controller).

The network virtual training processing apparatus 200 includes the trainee client systems (client system 1, client system 2, ..., client system n), and is included in the form of a program driven in each client terminal, (Not shown). In this case, it is collectively referred to as a client program.

The master system 300 is connected to the client terminals of the respective client systems through a network so that the trainer connected to each client system can experience the training in the same virtual work space. (Not shown) for the purpose of providing a UI and a necessary function for synchronizing and transmitting commands necessary for training and outputting necessary training information to the training manager.

The client HMD device is capable of head tracing in the X, Y, and Z axes, and the head tracking method of the HMD may vary depending on the configuration of the selected hardware device. Normally, X-axis rotation checks the up / down rotation value of the head with the Pitch value. Y-axis rotation checks the camera's left / right rotation value by Yaw rotation and z value is the rotation value when the head is shifted to the left / And outputs the movement and state of the wearer's head in the form of X, Y, Z values. To this end, HMD tracking module sensors such as a gyro sensor, a gravity sensor, and the like are built in the client HMD device. The X, Y, and Z values are X, Y, and Z axis rotation values, and three values of Pitch (X axis rotation value), Yaw (y axis rotation value), and Roll Data.

The client motion recognition device is a motion tracking device for recognizing / tracking the movement of a user. The motion tracking device includes one of a contact type motion recognition controller (e.g., Virtuix Omni, Cyberith Virtualizer, PrioVR and the like) and a contactless motion recognition controller Both of which are selected and used. In this embodiment, both the contact controller and the contactless controller are used as the client motion recognition device.

The contact type motion recognition controller recognizes the movement and rotation of the trainee, and the movement and rotation of the trainee recognized through the contact type motion recognition controller are mapped to the movement and rotation of the character in the virtual work space.

In the present invention, a motion controller such as a Virtuix Omni or Cyberith Virtualizer, which is commercially available as a contact type controller, can be used, which includes a sensor of a foot plate and a sensor located near the waist of a trainee, When a footstool is rolled, it receives a signal through the sensor of the footstep, and the contact controller outputs it as motion data for the forward movement during the behavior of the particular trainer. The trainee character performs a movement operation in the virtual work space according to the motion data.

On the other hand, the rotation sensor located at the waist recognizes the Y axis angle every time the trainee twists the body to the left and right, and outputs it as motion data. The rotation value is mapped in real time to the character of the trainer in the screen, The upper body and / or the lower body of the character inside is rotated in the Y axis.

The contactless controller is responsible for recognizing and analyzing the training behavior of the character, and the training action is the movement of the hands and arms of the trainee, recognizing two motions or motions and outputting them as motion data.

The first is the gripping motion of the object, that is, the grip motion recognition. When there is equipment such as a tool on the screen, when the trainer stretches his hand and grasps the hand, the hand of the trainer character mapped in the virtual workspace If the position of the tool matches the position of the tool, the virtual workspace should be recognized as having the tool held by the hand. This is like a mouse control on the screen of a virtual workspace. That is, the mouse pointer on the screen is mapped to the motion of the right hand of the motion recognition. Not all objects can be caught, but are recognized as objects that are needed only in the training scenarios, such as those that only capture the working tools (eg, tools such as drivers, pliers, and hammer).

The second is motion recognition of motion of the trainer's arm. Depending on the training scenario, the trainer must follow the action. In the case of hammering, the trainer performs an operation of shaking his / her arm back and forth, thereby recognizing that the character of the trainee in the virtual work space is hammered, and performing an operation of hammering on the screen of the virtual work space. These motions are performed in such a manner that the operations necessary for the training scenarios are preset, and whether they are in agreement with the motion or motion-tracked motion.

Referring to FIG. 1, the network virtual training apparatus 200 of the present invention includes a client motion recognition / display unit 210, a client system processing unit 220, and a training scenario processing unit 230.

2A is a configuration diagram of an embodiment of the client motion recognition and display unit of the present invention.

2A, the client motion recognition / display unit 210 initializes a client HMD device connected to a client system and each motion recognition controller, sets each function in association with the client system processing unit 220, Module 211, a client HMD tracking module 212, a client HMD camera link module 213, a display module 214, a motion recognition controller initialization module 215, a motion tracking module 216, a motion link module 217, .

The client HMD initialization module 211 proceeds to initialize the client HMD device. The client HMD initialization module 211 checks the connection status of the client HMD device connected to each client terminal, extracts the ID of the connected client HMD device, and transmits the ID of the client HMD device of the client system to the client system processing module 220 do.

The ID value is transmitted to the client system processing unit 220 in the form of String data as a hardware unique number of the client HMD device.

The client system processing unit 220 confirms and stores the transmitted client HMD device ID, and if the corresponding ID does not exist or is not received in comparison with the pre-stored client HMD device ID list, the connection of the device is incorrect And sends a warning string (Alert String) as a message to the display module so that the trainee can check again whether the equipment is connected or not.

The client system processing unit 220 checks the client HMD device ID and sets the viewing angle (float-type real number value) of the camera in the displayed virtual work space when the connection status is confirmed. Normally, the viewing angle is set to a viewing angle of 90 to 110 degrees, which is similar to a human viewing angle. In addition, according to the characteristics of the HMD device, two cameras are set for each character in the virtual work space (left eye, right eye). In this case, the distance between two cameras is set as a float type real value. The distance between the cameras is the same as the distance of the human eye and is usually set to a distance of 3 to 6 cm. When the setting of the camera is completed, the client HMD tracking module 212 is activated and prepares for tracking. In response, the client system processing unit 220 confirms the connection status, receives screen size information for displaying the current virtual work space from the client system processing unit 220, and information on the viewing angle to be displayed, extracts and stores the screen size , The viewing angle to be displayed is set. In addition, the value between the left lens and the right lens of the HMD device, the magnification value, and the like are adjusted. Preferably, the HMD device is set to have a viewing angle of 90 degrees to 110 degrees. 212 are activated to prepare for tracking.

The client HMD tracking module 212 computes data on the head movement of the trainee, which is recognized from the HMD tracking module sensor of the client HMD, and generates HMD tracking module sensors such as a gyro sensor, a gravity sensor (X, Y, Z axis rotation values) of the head of the trainee wearing the HMD tracking module sensor in the HMD device having the built-in HMD device, And transmits it to the client processing unit 220.

Here, the X, Y and Z axis rotation values are obtained by real-time checking three data of Pitch (X axis rotation value), Yaw (y axis rotation value) and Roll (z axis rotation value) through the gyro sensor of the HMD device, Is a value calculated by the tracking module 212. The calculated value is transmitted to the camera link module 213, and the camera link module 213 sets the three rotation data to the rotation value (rotation value) of the camera. The transmitted X, Y, Z axis rotation values are values in the range of -1 to 1, respectively, as Float type real numbers, and the received values are converted into angular values in the virtual work space, Means 180 degrees.

The client HMD camera link module 213 links the camera field of view display of the client HMD device to the camera of the trainee character located within the virtual workspace.

These links transmit the X, Y, and Z axis rotation values received from the HMD tracking module 212 to the rotation values (rotation values) of both cameras (left eye and right eye) set in the trainee's own character in the virtual work space And performs real-time processing (mapping).

To this end, the client HMD camera link module 213 receives the X, Y, and Z axis rotation value data of the client HMD device in real time, and converts the X, Y, and Z axis rotation value data into a rotation value (rotation value) , The camera in the virtual work space is rotated according to the rotation value (rotation value) of the camera in the virtual work space, and the screen displayed in accordance with the rotation of the camera is changed in real time. The field of view of the character viewed by both cameras (eyes) is transmitted to the display module 214 and displayed by the 3D rendering engine.

Accordingly, the 3D environment in which two cameras of the virtual space character are viewed is displayed on the two lenses (left and right) of the client HMD device. That is, if the x, y and z axis rotation values of the character camera in the virtual space are linked with the gyro sensor x, y and z axis rotation values of the HMD in real time and the trainer rotates the head by wearing the HMD device, A virtual workspace trainer maps to his character's camera. The character's camera is set to two. One serving as the left lens of the HMD device and the other as the right lens.

The display module 214 transmits a screen rendered on the camera of the trainee character to the screen of the HMD device at the time and the time of the visual field viewed by the trainee character in the virtual work space, So that the trainer can view the trainee through the screen of the HMD device at the viewpoint and the viewpoint of the view. A rendering screen of the virtual work space processed by the 3D engine is displayed on the HMD lens, and the viewpoints and times of the two cameras (left and right) in the character in the virtual space are transmitted to the left lens and the right lens of the HMD device and displayed.

The motion recognition controller initialization module 215 initializes the kinesthetic motion recognition device, that is, the contact type motion recognition controller and the noncontact motion recognition controller. Contact type motion recognition controller and the non-contact type motion recognition controller, extracts the ID of the connected motion controller, and transmits the extracted ID to the client system processing unit 220. The motion recognition controller initialization module 214 receives the device unique ID (string string) from the hardware of the motion recognition device and transfers it to the client system processing unit 220. Whether the motion recognition SDK (software development kit) And transmits it to the client system processing unit 220. If the hardware connection is not correct (the hardware ID can not be extracted) or the motion recognition SDK is not installed in the system, the client system processing unit 220 transmits a warning string (Alert String) to the display module The trainer should check whether the equipment is connected again. Here, an SDK (Window Kinect, Omni Device SDK, etc.) provided by each hardware manufacturer is used for a motion recognition SDK (software development kit). When the connection is confirmed, the sensor of the contact type motion recognition controller and the sensor of the non-contact type controller are activated.

The motion tracking module 216 receives data according to the motion of the trainee from the kinetic motion recognition device, extracts the motion data of the trainee, and tracks the motion of the trainee. Motion data obtained through the SDK of the client motion recognition devices set in each client system, that is, the rotation value and the position value of each joint are extracted and tracked by the trainee's motion or motion to perform motion recognition.

FIG. 2B is an output view of each joint name recognized and motion-mapped by the client motion recognition / display unit 210 according to an exemplary embodiment of the present invention. Referring to FIG. 2, The position and rotation values of 23 joints are tracked and used. The position value of each joint is the sum of three float-type real numbers at x position, y position, and z position, and is transferred to the data type processed by Vector3's rendering engine. The rotation value is also transmitted as a Vector3 type in which three float-type real numbers of the x rotation value, the y-axis rotation value, and the z-axis rotation value are combined. On the other hand, the motion tracking module 216 does not always operate, and the motion tracking module 216 is executed only when the task authorization is received from the master system 300. That is, the task right is received from the training data / message receiving module 233, and when the task right is received from the master system 300 during training, the motion tracking of the motion tracking module 216 proceeds, The value is passed to the motion link module 217.

The motion link module 217 links the motion data obtained by the motion recognition of the motion recognition device, that is, the joint data, with the joint data of the virtual workspace character. The motion data of the trainee traced in the motion tracking module 216, that is, the 23 joint motion data, is mapped to the rotation value (Vector3) and the position value (Vector3) of the bone of the character of the virtual work space corresponding trainee. That is, work is performed to connect the joint work data of the virtual work space character so that the movement of the trainee is mapped in real time to the trainee character of the virtual work space. For this, the motion link module 217 receives the motion data obtained through the motion or motion recognition of the trainee through the sensor of the non-contact type motion recognition controller in the case of the motion recognition controller in this embodiment, that is, the rotation value and the position value of each joint, 220, and when a situation in which the motion is to be compared during the training occurs, the client system processing unit 220 uses the motion data received from the motion link module 216, / RTI > < RTI ID = 0.0 > and / or < / RTI >

3 is a configuration diagram of a client system processing unit 220 according to an embodiment of the present invention.

Referring to FIG. 3, the client system processing unit 220 performs network connection with the master system 300, loads training content composed of 3D contents, and receives and processes motion data recognized from the client motion recognition apparatus. That is, the non-contact type motion controller received from the motion link module 217 and the HMD camera link module 213 receives and processes the motion data recognized from the contact type controller, and the 3D virtual environment loading and training system Preparation / processing tasks, and network connection with the master system.

In other words. The client system processing unit 220 manages / controls the overall process flow of the client. (Data reception, data transmission) with the master system 300 when the network connection with the master system is completed.

Data to be transmitted / received to / from the master system 300 includes work turnable variable data (Bool Type), chat message data (String Type), work time float type data, and training result data (Array Type) .

3, the client system processing unit 220 includes a system initialization module 221, a network setting module 222, a system initialization module 221, a network setting module 222, a 3D content loading module 223, A process module 224, and a termination module 225.

The system initialization module 221 initializes the 3D engine and the necessary program in each client system, initializes the 3D engine among the necessary programs of the virtual training system and initializes the client program, that is, The network virtual training apparatus 200 of FIG.

The system initialization module 221 receives the graphic driver necessary for driving the 3D engine from the window system as the initialization of the program and proceeds with the 3D engine initialization operation necessary for the program operation. If there is no graphics driver or the system is unstable, Provides an alert window when it can not run, and sends an error message to the user. The initialization process of the 3D engine is performed by including the process of initializing the mesh data through the vertex, preparing the input / output system, and initializing the rendering engine. When the client program is activated through the 3D engine initialization process, Confirm the ID of the device (client HMD device and client motion recognition device) and connect. The IDs of the hardware devices are received from the client HMD initialization module 211 and the motion recognition controller initialization module 214, respectively.

The network setting module 222 sets the network connection environment in accordance with the information input from the trainee, thereby performing the network connection, and is responsible for starting / completing the network connection. The setting for connection to the network is performed in a manner that a trainee provides a GUI screen for inputting information.

The trainer inputs data of the IP, connection port and user ID to be connected through the GUI screen and tries to connect to the network by pressing the input button UI, for example, "Join" button, and transmits the data to the master system. When the master system 300 is driven by the IP to be connected and the connection port number, the connection is successful and the 3D content loading module 233 is activated and the 3D content is loaded. If there is no response from the server (i.e., the master system 300), the input IP and port number are waited for a predetermined time, for example, about 30 seconds, and then the connection is stopped and the user is requested to re-input. Access to the master system 300 of the trainee may be restricted depending on the number of connectable persons set by the master system 300. [

The 3D content loading module 223 is a module for loading 3D virtual training content loaded in the client program. The 3D virtual training contents are configured and stored in a database (not shown) in the client program of each client system 100.

When the connection to the master system 300 is successful through the network setting module 223, the 3D content configured in the client program is loaded. 3D contents are 3D objects. In detail, it consists of work environment, working tools, work objects and so on. GUI environment such as a button UI, a chat window UI, a work turn management UI, a Timer UI, and a motion display UI. In addition, the 3D content includes a scenario configuration (an event script, a conflict processing script, and an input / output script) necessary for proceeding with a training scenario. This virtual workspace is equipped with training scenarios, and it has an interaction structure that responds whenever the character performs training. The portion of the training course, which is related to the training course, is described in detail with respect to the training scenario processing unit 230.

The motion process module 224 compares motion of the trainee, which is recognized by the motion data, with motion of the motion requested by the predetermined trainee, Receives the motion data, and performs a function of comparing / processing the motion required for the training scenario.

The motion process module 224 extracts the motion accuracy of the trainee's motion based on whether or not the determined motion is matched. Specifically, the motion process module 224 extracts the motion accuracy of each joint from the non-contact motion controller and the contact- Value data is received. Also, when an operation request is received in the training scenario, the corresponding module judges / processes whether or not the motion is matched by comparing the operation with the data to be received. The process of calculating the motion / mismatch accuracy of the motion is as follows.

The required motion during the trainer training process is preformed in animation form in the client program. In the motion input step during the training system, the animation motion data already prepared in the program is compared with the motion data received from the motion link module 217 in real time. The position and rotation values of the joint to be tracked are compared with the position and rotation values of the previously created animation and the accuracy is measured. In case of Kinect motion recognition, error range may occur depending on light or angle. Therefore, it is calculated in a form that recognizes the range of large motion, not the accurate position value calculation. The error range should be within 20cm. The result of training such as a float type, a percent type, and the like, which is performed through a motion comparison, is extracted in data form, and the extracted data is included in the training result information through the training result information generation module 235 And transmitted to the master system 300.

The termination module 225 terminates the network connection upon completion of the training process, clears the loaded content from the memory, and then terminates the program.

4 is a configuration diagram of a training scenario processing unit 230 according to an embodiment of the present invention.

Referring to FIG. 4, the training scenario processing unit 230 initializes training contents composed of 3D contents, creates and loads a character of a trainee in a virtual work space, controls work to be performed as a training content in a virtual work space, The 3D virtual content loaded from the 3D content loading module 223 is initialized in the order necessary for the training progress and the virtual virtual content is loaded from the master system 300 to the virtual work It receives characters in space, and supervises / controls / processes the progress of the training. A training content initialization module 231, a trainer character loading module 232, a training data / message receiving module 233, an event processing module 234, and a training result information generating module 235.

The training content initialization module 231 is a module for initializing training content composed of a collection of 3D objects in a virtual space. The training content is a virtual space implemented in 3D, and objects modeled in 3D such as work objects and work tools are constructed therein. (Details in Layout) As the initialization of the training content, the training content initialization module 231 sets the Light object to the 3D virtual content generated by loading from the 3D content loading module 223, so that the texture can be rendered , And sets a call reader (Collider) on the collision processing object necessary for the training process. Here, the call reader is an invisible object (object) provided by Unity3d that performs collision processing such that, when a trainee moves beyond the range of movement during training, touches an object that is not a work object, and so on. Also set a call reader on the floor object so that the character can move without going down the floor. In the case of an object requiring moving animation during training, setting of necessary animation is also performed in the training content initialization module 231. [

The trainer character loading module 232 is a module for receiving a character from the master system 300 in the virtual work space initialized in the initialization module 231 and generating the character. For this, the master system 300 transmits the character object allocated to the trainee and the position information in the virtual space, and the trainer character loading module 232 transmits the character object allocated to the trainee transmitted from the master system and the virtual And loads the character object so as to be displayed at a specific position in the virtual work space according to position information in the work space.

Specifically, when the network connection with the master system 300 is performed through the client system processing unit, the character assigned to the trainee and the position in the virtual work space are set according to the selection input of the training manager in the master system 300 The position value is transmitted to the client system, and the character position in the loaded virtual workspace is initially set to the position value (Vector3 type) set in the master system 300. [ This is because, in the master system 300, the training manager has to set its initial position according to the training role of each trainee, and the character is loaded to the position determined in the master system by the initial position. The position where the character of the trainee is generated is mainly set to a position around the workpiece. The trainer's character has the same joint structure as the actual person, as we have already seen, and has the model shape and animation designed with 3D Max Biped Animation structure. The animation of the character is made up of 23 joints that can be linked with motion controllers such as external equipment (HMD, Kinect, Omni).

 The training data / message receiving module 233 receives information on tasks and data on task privileges from the master system. The trainer enters the training work environment and is instructed by the master system to receive tasks and receive data regarding task privileges .

The task information is received as a string text message through the chat window, and the task right is received as data (bool type) about the task authorization from the master system including true or false values. When a value of true is received, it is determined that the corresponding trainee has authority to work. When true is received from the master system 300 regarding the task authorization, the event processing module 234 is activated and proceeds. In the case of receiving false as the data regarding the task authority, the other trainee monitors through the HDM display that the task proceeds through the activation of the event processing module 234. That is, in each client system, motion tracking and HMD tracking can be performed at the same time when the data regarding the received task right is true, and only HMD tracking is set to be possible when the data regarding the received task right is false.

The event processing module 234 is a module for allowing a trainee to perform a task in a virtual work space. When the training data / message receiving module 233 receives a true value as data from the master system 300 This is the module in progress. The start of the training work starts with the movement of the trainee, and the trainee moves along the movement path of the floor. The movement is made by using the contact-type controller to roll the floor, and the character is rotated to the left and right using the waistband of the contact-type motion controller. At this time, the working environment of the virtual workspace is rendered in real time through the camera lens of the HMD. Upon reaching the training area, the user can use the tool, the tool is present around the workpiece in the virtual workspace, and the acquisition of the work tool is performed by grasping motion, i.e., gripping motion recognition of the hand. Grip motion recognition means that when the hand is extended when the tool is on the screen, when the position of the hand matches the position of the equipment (2D coordinate system of the x and y axes) when the hand is grasped, . When the tool is gripped, the tool's position coordinate data is set as the child object of the trainee's character's hand, and moves as the character's hand moves. Once the tool is acquired, the trainee will be presented with the required training gesture in the virtual workspace in the form of a message received from the master system. The training gesture is represented in the form of a 2D animation at the bottom left of Figure 7d, and the trainer performs the task in the same manner as the training gesture presented before the workpiece. The motion recognition process is calculated through the motion process module 234 of the client system.

The training result information generation module 235 generates data in the form of data for training and transmits the data to the master system. The training result information generation module 235 generates data for the training results including the motion accuracy and the working time. Data on the result of the training generated from the motion process module 224, that is, the time that has elapsed, and the motion accuracy data are transmitted to the master system. Upon completion of the transfer, a false value is received from the server system as data regarding the task authority, and the next trainee watches the event processing module 234 to proceed.

5 is a block diagram of a master system 300 according to an embodiment of the present invention.

Referring to FIG. 5, the master system 300 serves as a server for allowing a user client to perform cooperative training in a virtual work space through multiple connections. It defines the beginning and end of training, controlling the work order and work authority for the work, monitoring the training situation in which the training is conducted, and evaluating the training or work in the virtual work space of the trainee through the monitored training course And the like.

On the other hand, the network configuration in the present invention is based on a turn-based network. In other words, if one client is authorized to work, the other trainee only monitors the work performed by the trained trainee. In other words, only the trainee who has the use authority can proceed the interactive work. This network configuration is necessary because it is more important for individual step-by-step procedures than for concurrent work if it is a risky task scenario. To this end, it is necessary to transmit playtone data (true / false), which is data relating to the task authority, to the client system of the connected trainee so that the event processing module 234 of each client system As shown in FIG. In addition, the master system 300 provides a function for the master to input a score into the evaluation table while the client is working, and enables the monitoring of the work situation in real time. The evaluation is conducted by judging the appropriateness of the actions, procedures, and time that the trainee is going through. In addition, it may be configured to perform a function of receiving a message or voice from a training manager so that the message can be delivered to each client through a chat system or voice communication in the form of a message such as a status delivery command.

 For this, the master system 300 includes a training environment initialization / setting unit 310, a training control data transmitting unit 320, and a training performance receiving / evaluating unit 330.

The training environment initialization / setting unit 310 sets the IP and port number of the master system 300 (or the server on which the master system is installed), waits for the connection of the trainee client system, It distributes / positions character. When a specific trainer accesses the IP network and the port number set in the master system through the network virtual training processing apparatus 200 of his client system, the training environment initialization / setting unit 310 sets the position Value to the trainee character loading module 232 of the client system of the corresponding trainee. The position value is composed of position values of x, y, and z in the virtual space as a type of Vector 3, and may be stored in advance according to the contents of the training, or may be automatically set or set according to the selection input of the training manager.

Also, the training environment initialization / setting unit 310 transmits a character object (3DObject type) prepared and stored in advance in the master system to the character loading module 232 of the client, As shown in FIG.

In addition, the training environment initialization / setting unit 310 transmits a work task to the training data / message receiving module 233 of the client system that receives the character position value and the character. Work tasks are basically delivered via a chat system (text format), and can be sent in the form of a 2DImage type if necessary.

The training control data transmitting unit 320 receives the input of the training manager and transmits the training data and messages required for training in the virtual work space to the network virtual training processing apparatus 200 of the client system of the connected trainee , And distributes a play turn among the operations. And transmits play data (true / false), which is data on the task authority, to the trainee to perform a turn distribution function to advance / observe the event processing module 234. The real-time chatting and voice chatting system also performs transmission / reception processing in the corresponding module.

The training performance receiving / evaluating unit 330 receives from the training result information generating module 235 of the network virtual training processing apparatus 200 of the client system of the connected trainee the task given the trainee the time, the motion accuracy, etc. The training result data is received and stored, and the training manager receives and displays the screen data of the virtual work space on which the trainees work, thereby allowing the training manager to monitor the work situation on the screen in real time.

Hereinafter, an actual implementation of a immersive network virtual training system capable of collaborative training in a virtual work space based on the multi-client of the present invention, and a virtual environment used are supplemented.

FIG. 6 is a diagram for explaining an actual implementation of an immersive network virtual training system capable of collaborative training in a virtual work space based on the multi-client of the present invention.

The movement of the trainee's head is recognized through the client HMD device, and the virtual movement and movement of the trainee are recognized through the client motion recognition device such as trainee's back and forth movement, waist rotation, arm movement, Movement and movement of the head of the trainee's character in the virtual work space based on the recognized motion data, and the movement of the head of the trainee's character in the virtual work space The screen is implemented as an immersive virtual training system that is visually presented to the trainee through the display of the client HMD device.

Referring to FIG. 7A, in this embodiment, a virtual work environment is defined and defined as a work environment, a work character, a work tool, and a work object. All components are composed of 3D objects (3D products produced by 3ds max, maya etc.), imported into Unity3D (VR & game development engine), and environment is set. In other words, training scenarios are created through the Unity3D engine, and non-contact motion recognition devices such as Kinect and contact motion recognition devices and HMD devices will also be set up in the Unity3D environment.

FIG. 7A is a configuration screen of the virtual environment of the embodiment of the present invention.

The work environment is the space in which the virtual avatar will work. The work environment configuration is configured by 3D modeling of the indoor space (3D results produced by modeling tools such as 3ds max and maya). It is an essential component of the virtual environment. It is composed of trainee 's movement path, wall to prevent trainee' s deviation from work environment, and other objects suitable for work environment. The breadcrumb display is displayed on the floor as an image that tells users how to move to the workpiece. The wall surrounds the workpiece, and the trainee collides against the work environment. In addition to these objects, the surrounding environment also sets objects such as a camera that displays a certain area and a light that expresses color in a space. Camera objects are set one by one for each avatar object.

7B is an exemplary image of a working tool used in an embodiment of the present invention.

7C is a screen displayed as a character in a virtual work environment when multiple users of the embodiment of the present invention are connected.

① No. is difficult to contact with training objects and express dangerous objects in 3D form. ② The trainee is a trainee, and from the first person point of view, multiple trainees are connected. The position value of the trainee is assigned a position value suitable for the trainee in the scenario, and the trainer is created at the position value designated by the trainee at the connection. When the trainer is created, the first person view is shown, and if it is not a work turn, the other trainer will only watch the progress of the trainer, and simultaneous work is impossible. Tools may vary from one trainer scenario to another and are not required.

Referring to FIG. 7C, the work character is an object that is mapped to the trainee himself / herself to perform work in the virtual work environment space. The trainer's own motion is mapped to the character by the motion recognition sensor and the HMD (Head Mounted Display) device, and the motion is controlled. The work character has a position value to be positioned in the work space, and has one camera (visual field charge) per character. The work character is largely represented in the virtual work space by three external device information values. First, the head part is rotated by the HMD device and the display output function. The HMD device uses a coordinate tracking device with a gyro sensor. In other words, if the user wears the HMD device and turns his or her head to the left or right, the left and right cameras applied to the character in the virtual space move in the same manner, and the virtual space in which the character's camera looks is also displayed do. The Y-axis rotation of the head of the person wearing the HMD device checks the rotation value of the camera with the Yaw rotation. The X-axis rotation checks the vertical rotation value of the head with the Pitch value. z Rotation value is the roll rotation value, and the rotation value is checked when the head is shifted left and right. This rotation input value is mapped to the work character. The data structure when mapped is Vector3 Type, which is represented by a single data type, in which three floating-point floating-point values of x, y, and z axes are gathered. In other words, three Float values of yaw, pitch, and roll are received as a Vector3 type, and the head rotation value of the character is displayed on the character's camera in the virtual workspace with the input value. Second, The input device uses a motion recognition controller of the Kinect SDK (Microsoft Corporation). Kinect is a non-contact controller that uses two cameras to recognize human joints and Depth values and pass them on. Recognized joints consist of head, body, arms (3 joints), legs (3 joints), and pelvis. Motion recognition is performed by inputting position and rotation values of each joint. The position value and the rotation value are input from the Kinect to the Vector3 data type, and when I assign to each joint of the character, my motion is expressed as a character in the virtual work space. Third, the matter of moving the character uses the contact controller. The contact controller uses the commercially available Virtuix Omni or Cyberith Virtualizer hardware. There are two types of contact type controllers. One is the velocity value when the character in the footstep squeezes the foot, and one is the rotation value of the waistband. You will receive two input. When a person holds a footstool, the character moves in the Z direction according to the speed of the roll. The speed at which the character moves in the virtual space also changes depending on the speed at which the foot is rolled. The band at the waist determines the direction of the character that moves the character in what direction. When the waist is turned to the left, the character rotates in the left direction of the Y axis, and the character rotates in the Y axis in the right direction when the waist is moved to the right. The velocity value of the footplate is transmitted as a float (real value), and the waistband is transmitted as the rotation value of Vecotr3. As described above, the view, action, and movement of the virtual workspace character are mapped and represented through the three hardware devices.

A work tool is a tool object that is needed when a trainer performs a training task in a virtual workspace. For example, if the trainee needs to proceed with disassembly of the workpiece, it is necessary to unscrew the workpiece. At this time, the necessary tools are work tools such as scanners. In this scenario, necessary work tools are arranged and configured in advance in the work environment. Work tools in space are made up of 3D objects and have position values in space. When the virtual space character moves around the work tool and is positioned in the character's hand, the acquisition of the work tool is performed by checking the position value of the hand with the motion recognition and taking the action of grabbing the hand, . As shown in FIG. 7B, various work tool objects are pre-created and used, and the work tools can be changed according to the scenario of the content such as the contents of the work assigned to the trainee. It is the target object. Generate 3D objects in dangerous and difficult objects that are universally encountered in real life, and implement them in a virtual workspace. Work objects can be of various types, depending on the training scenario. There is a procedural process for the work of work objects in most industrial settings. In other words, the object can be disassembled / assembled according to the order, and it can be a trouble-shooting trouble shooting.

The workpiece is normally located at the center of the virtual workspace of the work environment, the position value is located at (0, 0, 0), and the actual size (size existing in real life) is equally implemented in the virtual workspace. The work object is divided into a portion where an event occurs and a portion where an event does not occur. Collider exists in the part where the event occurs, and collision with the trainee is detected. This collision is called with the OnTriggerEnter function when the Collider of the trainer tool and the Collider of the workpiece collide in the virtual workspace. When a collision is determined, an event (a predetermined event such as an animation play, a movement, or a rotation) is generated in the workpiece. Objects that do not require an object in the work object are set so that the position and rotation values do not change. In order to prevent the character from penetrating the workpiece, the collider collider is set, as in [workpiece object class sample] below.

[Workpiece object class sample]

* Collider: Collision class object provided by Unity3D's Physics engine. * OnTriggerEnter: Function that occurs when objects with Collider collide with each other.

Hereinafter, the immersive network virtual training method using the immersive network virtual training system according to the present invention will be described in detail with reference to FIG. 8 and FIG.

Figure 8 is a flow diagram of an immersive network virtual training method using the immersive network virtual training system of the present invention.

8, the immersive network virtual training method is a method performed in a specific client system in conjunction with the master system of the immersive network virtual training system of the present invention. The system initialization step S100, the network setting / S200, training content initialization step S300, event processing step S400, training performance data transmission / confirmation step S500, and task observation step S600.

The system initialization step S100 initializes the connected client HMD (Head Mounted Display) device and the motion recognition device at the start of the network virtual training device 200 in the form of a training program installed in the client system 100, And establishes a connection, sets a GUI screen for connection to a specific virtual workspace, for example, a training work room, and provides a GUI screen that can be connected to the master system 300. [ Is performed by the client HMD initialization module 211, the motion recognition controller initialization module 214, and the system initialization module 221 of the network virtual training apparatus 200 described above.

The network setting / connection step (S200) receives the IP of the client system, the contact person ID, and the user name, room name, and access room information to be accessed by the trainee through the provided GUI screen, And setting up a network connection environment with the network 300, and connecting the network to the master system, is performed by the network setting module 222 of the network virtual training apparatus 200 described above.

The content initialization step S300 is a process of loading training content composed of 3D contents, setting a loaded 3D object to a preset value, and placing the 3D object on a virtual work space. After the process of loading the training content composed of the 3D contents constituting the training scenario, which is the training content, is performed, the loaded 3D object, that is, the position value of the work environment, the work character, the work tool, Values and the like are set to predetermined values and placed in the virtual work space. And a training data / message receiving process for receiving and displaying a task character loading process and a data or message indicating a task as a description of a task and a play turn as a task sequence from the master system 300.

The content initialization step S300 includes a 3D content loading module 223, a training content initialization module 231, a trainee character loading module 232, a training data / message receiving module 233 of the network virtual training processing device 200 ) Is performed in conjunction with the master system 300.

The work character loading process is a process of creating a character of the trainee in the virtual work space of the virtual work environment. When the network connection with the master system 300 is performed through the client system processing unit, the trainee character loading module 232 reads the master A character in the virtual training content is created and loaded based on the character received from the system 300 and the position value of the character. The position where the trainee character is created is set to the position around the workpiece. The trainee's character has the same joint structure as the real person, and has the model shape and animation designed with 3D Max Biped Animation structure. The animation of the character is linked with the motion controller of the external equipment (HMD, Kinect, Omni), and the rotation value of the joint is expressed.

The process of receiving the training data / message is a process in which the trainer enters a training work environment and receives data or a message indicating a task, which is a description of a task to be instructed from the master server and a play turn, It is a process of delivering to a trainee using a chat window and a 2D image. As a result, the trainer learns the task through the message, waits in the virtual workspace, and performs the task when it comes to his / her work order. And a training data / message receiving module 233 is interlocked with the master system 300.

In the content initialization step S300, a light object capable of displaying a color to an object in a virtual work space is set. A process of setting a collider of an object on which conflict processing is performed, that is, For example, when the trainer moves beyond the range of the movement path, it performs a dispatch process for disabling the object, such as touching an object that is not a work object, and preparing a warning message or the like to be output.

In the training event processing step (S400), the plaintone data, which is data related to the task authority, is received from the master system through the training data / message receiving process, the task authority is granted, the message regarding the task content is received and displayed to the trainee, A positional value, a rotation value, a magnitude value of an object in a work environment, a work character, a work tool, and an object of a workpiece in a virtual training environment of a trainee recognized by a trainee's motion input and motion analysis Etc., and maps them to objects of a virtual training environment of a trainee, which is a virtual space. A series of processes of destination movement, tool acquisition, training gesture input, information transmission and training termination are performed, The virtual training environment of the changing virtual workspace is rendered in real time through the camera lens of the HMD device, So it should show. In addition, the training event processing step S400 includes generating the training result information, and generating training data as data on the operation time, the deviation of the movement path, and the training gesture operation. The event processing module 234 of the above-described network virtual training processing device 200 is mainly performed.

The training event processing (S400) will be described with reference to the HMD device display screen in the training event processing step of FIG. 7D. When playturn data, which is data on the task authority, is received from the master system by the specific trainee client, As shown in item ① of the screen, green is displayed to indicate the play turn. When the green light comes on, the trainee confirms his training turn, allowing manipulation in the virtual training environment through the client HMD device and the client motion device. In the virtual training environment, the camera viewpoint is always provided as the first person viewpoint, and it is possible to perform motion tracking with respect to the character of the client in association with the client HMD device and the client motion device. The trainer performs a motion for moving from the virtual work space to a predetermined position where the workpiece is located, and data on whether or not the movement path is deviated is generated and included in the training result information. When the trainer reaches the designated work position, the item (2) displays the action required by the trainee, that is, the action required in the task scenario, in the form of a two-dimensional animation. The trainer inputs his motion through the client HMD device and the client motion device according to the motion shown in ②. In the input motion, the trainee's motion is recognized based on the measured motion data, and the motion of the recognized trainee is matched with the motion required in the task scenario. The input motion is generated as training data, which is training gesture motion matching data, Included in the result information. On the other hand, all the training situations are not limited to the motion input, and according to the embodiment, the specific task of the task scenario can be performed by simple input such as button click and object touch.

On the other hand, when the work is started, the work time is measured and displayed as in item ④, the work time is calculated in seconds, and is transmitted to the training supervisor of the master server as the training result information after completion of the work. In addition, as in item ⑤, communication between all trainees can be confirmed in real time through chat or voice communication by using the communication function. Through this, it is possible to exchange confirmation of work, progress, and communication through chat. The client is mainly used before and after the start of the work because it is difficult to input a chat during the motion work, and the instruction of another user or the training supervisor is received and displayed in the form of a message, .

When the work is completed, the training of the corresponding trainee is automatically terminated or the training is terminated according to the input of the termination button of the trainee or the training supervisor.

The training performance data transmission / confirmation step (S500) is a process of transmitting training data and final training result information to a master server. The training data transmission / confirmation step transmits training data generated in the course of training event processing (S400) At the end of the training, it is transmitted to the master server as the final training result information, and the training ending process can be set after the process of transmitting to the master server as the final training result information.

In the job observation step S600, the training data is generated by the trainee client system whose training has been terminated so as to provide the virtual work environment of the other trainee who is given training authority to the trainee whose training is completed, And changes the position values, the rotation values, the magnitude values, and the like of each work environment, the work character, the work tool, and the object of the workpiece in the virtual training environment of the trainee recognized by the trainee's motion input and motion analysis This is a process of mapping to each object of the virtual training environment of the trainee, which is a virtual space, so that the function of motion recognition and the like for the trainee who has been trained as an observer is not activated, The virtual work environment of the HMD is displayed in real time through the camera lens of the HMD. All. As described above, the communication function is activated, and when a trainee whose training in the task observation step has been completed has a message to be delivered to another trainee, the message is transmitted through chat or voice communication, and the message of another trainee Or via voice communication.

FIG. 9 is a time table of an embodiment of the immersive network virtual training method using the immersive network virtual training system of the present invention. The immersive network virtual training method of the present invention is capable of cooperative training in a virtual work space on the basis of multiple clients And the client system connected to the master system through the network is configured to perform the above-described immersive network virtual training method in the order designated by the training manager via the master server in the turn mode do.

100: client system 200: network virtual training processing device
300: master system
210: client motion recognition / display unit 220: client system processing unit
230: training scenario processing unit 310: training environment initialization / setting unit
320: training control data transmission unit 320: training performance receiving /

Claims (15)

The virtual HMM device is provided to each trainee and is connected to the client HMD device and the client motion recognition device in cooperation with the master system 300 that is provided to the training manager to monitor the training process. A virtual training processing apparatus (200) of a client system for performing training in a virtual training machine (200)
A motion recognition / display unit 210 for initializing the client HMD apparatus and the client motion recognition apparatus and setting each function in association with the client system processing unit 220,
A client system processing unit 220 for performing network connection with the master system 300 and loading training content composed of 3D contents, receiving and processing motion data recognized from the client motion recognition apparatus,
A training scenario processing unit (230) for initializing training contents composed of 3D contents, creating and loading characters of a trainee in a virtual work space, and controlling the operation of training contents in a virtual work space to proceed,
The motion recognition / display unit 210,
A client HMD tracking module 212 for calculating data according to the movement of the head of the trainee recognized from the HMD tracking module sensor of the client HMD device to generate motion data of the head of the trainee; A client HMD camera link module (213) linking a camera view display of the client HMD device to a camera of a trainee character located in a virtual work space; A display module 214 for transmitting to the screen of the HMD device a screen to be rendered on the camera of the trainee character at the time and the time of the visual field seen by the trainee character in the virtual work space; A motion tracking module (216) for receiving data according to a movement of the trainee from the clinic motion recognition device and extracting motion data of the trainee and tracking the movement of the trainee; And a motion link module (217) linking the trainee's motion data to the joint data of the trainee's character in the virtual workspace
The client system processing unit 220,
And a motion process module (224) for comparing the motion of the work required by the preset trainee with the motion of the trainee recognized from the motion data to determine whether the motion is matched or not, Extract the motion accuracy for the motion,
The training scenario processing unit 230,
And a training data / message receiving module (233) for receiving, from the master system (300), information relating to the task and data relating to the task entitlement,
Receives data relating to task privilege from the master system 300 through the training data / message receiving module 233, and receives data regarding task privilege received from the master system through the training data / message receiving module 233 Accordingly, when it is determined that the task authority is granted, the motion tracking module 216 activates and performs motion tracking to transfer the tracked motion data values to the motion link module 217, A scene to be rendered on the camera of the character of another trainee to which the task right is given is transmitted to the screen of the HMD device at the viewpoint and the viewpoint of the trainee's character in the virtual work space, 216 are deactivated and do not perform motion tracking, and if there is a message to be delivered to other trainees Receives the message through chat or voice communication, and receives a message from another trainer or training manager through chat or voice communication. delete delete delete delete The method according to claim 1,
The training scenario processing unit 230,
A training content initialization module (231) for initializing the training content in the virtual space;
And a trainee character loading module 232 for receiving characters from the master system 300 and creating the trainee character in the initialized virtual workspace. The trainee character loading module 232 includes a character object allocated to the trainee, And the character object is loaded so as to be displayed at a specific position in a virtual work space according to position information in the virtual work space of the character object. delete The method according to claim 6,
The training scenario processing unit 230,
And a training result information generation module (235) for generating a result of the training, including the motion accuracy and the working time, in a data form and transmitting the data to the master system. The method according to claim 1,
Wherein the virtual workspace includes objects for a work environment, a work character, a work tool, and a work object. 10. The method of claim 9,
Wherein the object for the work object is a call reader (Collider). An immersive network capable of collaborative training in a virtual space using immersive network virtual training system that includes a virtual training processor of a client system capable of collective and systematic cooperation training in a shared virtual workspace through multiple accesses In the virtual training method,
A system for initializing a client HMD device and a motion recognition device, receiving and confirming the ID to establish a connection, setting a GUI screen for connection to a virtual work space, and providing a GUI screen capable of accessing the master system 300 Initialization step S100;
A network setup / connection step (S200) of setting up a network connection environment with the master server (300) according to information inputted and inputting the IP of the client system, the connectioner ID and the connection room through the provided GUI, and connecting to the network;
A content initialization step (S300) of loading training content composed of 3D contents, setting a loaded 3D object to a preset value, and placing the loaded 3D object on a virtual space;
Receives the task authorization from the master system, receives the task authorization, receives a message related to the task content, and displays the task authorization message to the trainee. If it is determined that the task authorization is granted, A training event processing step (S400) of changing a position value, a rotation value, and a magnitude value of an object constituting a virtual work space of a trainee, which is recognized according to motion input and motion analysis, to each object of a virtual training environment of a trainee );
A performance data transmission / confirmation step (S500) of transmitting training data and final training result information to the master system;
(S600) for generating and providing screen data to a trainee client system whose training has been terminated so as to provide a virtual work environment of another trainee who is given training authority to a trainee who is given training authority,
In the training event processing step S400, if it is determined that the task right is not granted, the motion recognition is not performed on the trainee, And the training result information generation process is performed to generate training data on whether the operation time, the movement path deviation, and the training gesture motion match are satisfied ,
The performance data transmission / confirmation step (S500) transmits the training data to the master system in real time,
The operation observation step S600 may include a step of performing a motion recognition on the trainee who has finished the training and a scene in which the trainee is trained on the camera of another trainee And transmits the message to another HMD device when the trainee has finished sending the message to the other trainee. If there is a message to be transmitted to the other trainee, the message is transmitted through chat or voice communication. The method of claim 1, wherein the virtual training is a virtual training. 12. The method of claim 11,
Wherein the virtual work space comprises an object of a work environment, a work character, a work tool, and an object of a work object. 13. The method of claim 12,
In the content initialization step S300,
And setting a call reader in the work object. The method of claim 1, wherein the virtual object is a virtual object. delete delete

Images