KR102277358B1 - Method and appratus for vr training - Google Patents

Abstract

According to an embodiment of the present invention, in a method of operating a VR (virtual reality) device, performing a VR training operation, acquiring information on immersion while performing the VR training operation, and measuring the immersion A method comprising comparing information with a preset threshold value and performing a first operation for improving user's immersion when the information on the degree of immersion is less than the preset threshold value, and the method comprising the steps of: You can provide a device to do it.

Description

Method and apparatus for virtual reality training {METHOD AND APPRATUS FOR VR TRAINING}

The present invention relates to a method and system for virtual reality (VR) training. In addition, the present invention relates to a method and apparatus for performing a training simulation using a VR device.

With the development of wearable devices that can be worn on the body, various systems using VR electronic devices are being developed. The wearable device may be in a form that can be attached or detached to the body or clothing that can be worn on the body, such as a head-mounted device (HMD), a smart watch, and a wristband, and includes a plurality of wearable devices. As devices are used, VR systems that combine functions of a plurality of devices are also being developed.

In addition, with the development of display technology, the spread of HDM (head mounted display) devices is increasing, and accordingly, various VR technologies using head mounted devices are being developed. In the case of a head-mounted device, a realistic VR service can be provided using a display placed in front of the user's eyes and the user's movement.

With the development of VR technology, recently virtual reality hardware/software technology has been developed and popularized in various fields. With VR technology, you can safely experience situations in a virtual environment that require real space/goods, are expensive, or are dangerous or difficult to experience in a real environment. Therefore, there is a need for an attempt to utilize VR technology as a tool for education or training in various fields.

Republic of Korea Patent Publication No. 10-2018-0028033 (2018.03.15)

A technical problem to be achieved in an embodiment of the present invention is to provide a method and an apparatus for performing a training simulation using a VR device.

In addition, a technical task to be achieved in an embodiment of the present invention is to provide a method and an apparatus for performing a training simulation based on VR immersion.

According to an embodiment of the present invention, in a method of operating a VR (virtual reality) device, performing a VR training operation; acquiring information on a degree of immersion while performing the VR training operation; Comparing the information and a preset threshold value, and if the information on the immersion level is less than the preset threshold value, performing a first operation for improving the user's immersion level. can provide

Further, according to an embodiment of the present invention, in a VR (virtual reality) device, a VR training operation is performed through an output unit and the output unit, and information on the degree of immersion is obtained while the VR training operation is performed, and the A VR device comprising a processor that compares immersion level information with a preset threshold, and controls to perform a first operation for improving user immersion when the immersion level information is less than the preset threshold value can provide

The technical problems to be achieved in the embodiment of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned are clear to those of ordinary skill in the art to which the present invention belongs from the description below. will be able to be understood

According to an embodiment of the present invention, it is possible to provide a method and apparatus for efficiently performing training simulation using a VR device.

In addition, according to an embodiment of the present invention, it is possible to provide a new method and apparatus for providing a user experience (UX) for a training simulation based on the degree of immersion obtained through various devices.

1 is a diagram illustrating a system using VR according to an embodiment of the present invention.
2 is a diagram illustrating a VR device according to an embodiment of the present invention.
3 is a diagram illustrating an operation of a VR device according to an embodiment of the present invention.
4 is a diagram illustrating a method of acquiring immersion in a VR device according to various embodiments of the present disclosure.
5 is a diagram illustrating a method of determining whether information on effective immersion level has been obtained by a VR device according to various embodiments of the present disclosure;
6 is a diagram illustrating a method of setting a viewpoint for acquiring immersion in a VR device according to various embodiments of the present disclosure.
7 is a diagram illustrating a method of determining whether a threshold condition is satisfied based on information on a plurality of levels of immersion according to various embodiments of the present disclosure;
8 is a diagram illustrating a method of recording a VR training result according to various embodiments of the present disclosure.
9 is a diagram illustrating a method of sequentially controlling VR output based on immersion according to various embodiments of the present disclosure.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. At this time, it should be noted that in the accompanying drawings, the same components are denoted by the same reference numerals as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted.

In describing embodiments in the present specification, descriptions of technical contents that are well known in the technical field to which the present invention pertains and are not directly related to the present invention will be omitted. This is to more clearly convey the gist of the present invention without obscuring the gist of the present invention by omitting unnecessary description.

For the same reason, some components are exaggerated, omitted, or schematically illustrated in the accompanying drawings. In addition, the size of each component does not fully reflect the actual size. In each figure, the same or corresponding elements are assigned the same reference numerals.

Advantages and features of the present invention, and a method for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains. It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

In various embodiments of the present disclosure, virtual reality may refer to a specific environment or environment or the technology itself that is similar to reality created by artificial technology using a computer, etc. but is not real. In addition, virtual reality according to various embodiments of the present invention is immersive virtual reality (ex., virtual reality using special equipment such as HDM, data gloves, data clothes, etc.), remote robotics (immersive virtual reality and robot technology are combined virtual reality) and desktop virtual reality (a computer system with stereoscopic glasses, a joystick, etc.) may correspond to at least one of.

In the embodiment of the present invention, the term is mainly described using the terms virtual reality or VR, but this is for convenience of explanation and does not exclude augmented reality (AR) or mixed reality (MR), The technology can also be applied to systems and methods related to augmented reality or mixed reality.

In various embodiments of the present invention, VR training includes VR education (mining industry education, construction education, etc.) using a VR device, VR practice, VR training (fire training, etc.), VR simulation (driving a car, driving an airplane, etc.) can do. However, embodiments of the present invention may be applied to VR training as well as various environments using various VRs.

In various embodiments of the present disclosure, information on immersion may be used interchangeably with terms such as immersion or immersion information.

1 is a diagram illustrating a system using VR according to an embodiment of the present invention.

The system using VR of FIG. 1 may include a VR device 110 , a sensor device 120 , and an electronic device 130 .

The VR device 110 may be a device that provides virtual reality to a user. For example, the VR device 110 may be a device that provides visual information on virtual reality to a user by displaying information on virtual reality. Also, the VR device 110 may include a configuration for providing five sense information for virtual reality. In addition, the VR device 110 may include various configurations for applying manipulations or commands for user interaction with the virtual space. A detailed configuration of the VR device 110 will be described later with reference to FIG. 2 .

The sensor device 120 senses an external environment (eg, a user state) according to the operating state of the VR device 110 , and directly displays information corresponding to the sensed state to the user or to the VR device 110 . may be transmitted or may be transmitted to the electronic device 130 . The sensor device includes at least one of a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor. may include Meanwhile, the biosensor may include a sensor function for measuring at least one signal among electroencephalogram, respiration, body temperature, blood pressure, blood flow, current skin resistance, eyeball, electrocardiogram, electromyography, pulse, and the like. Meanwhile, the sensor device 120 may be included in the VR device 110 .

The electronic device 130 may interwork with the VR device 110 , may indicate an operating state of the VR device 110 , and may perform a user input for controlling the VR device 110 . Also, it may be a device for providing an alarm according to the VR device 110 and a user state. The electronic device 130 may be various electronic devices capable of interworking with the VR 110 device, such as a terminal, a mobile terminal, a smart phone, a mobile computer, a computer, or a wearable device. The electronic device 130 may perform an operation of a training device. The training apparatus may provide information for performing a preset training operation to the VR apparatus 110 . For example, when training for fire drills is performed, the VR device 110 may be controlled to perform a preset training operation. Meanwhile, the training device may be included in the VR device 110 .

2 is a diagram illustrating a VR device according to an embodiment of the present invention.

Referring to FIG. 2 , the VR device includes a communication circuit 210 , an input unit 220 , a connector 230 , an eye tracking unit 240 , a focus control unit 250 , a processor 260 , and an output unit 270 . may include at least one of

The communication circuit 210 may be a device for transmitting and receiving signals, information, and data when the VR device is connected to an electronic device or a sensor device.

The input unit 220 may obtain an input for controlling the VR device and transmit it to the processor 260 . The input unit 220 may include a touch pad and a button. The touchpad may recognize a touch input using, for example, at least one of a capacitive type, a pressure sensitive type, an infrared type, and an ultrasonic type.

The connector 230 may correspond to an interface for charging or connecting a VR device. For example, the connector may include at least one of an electrical connection unit, a universal serial bus (USB) wired interface, a high definition multimedia interface (HDMI) wired interface, and the like.

The eye tracking unit 240 may track the gaze of the user using the VR device, and may transmit information on the gaze tracking to the processor 260 . The gaze tracking unit 240 may include a camera for tracking the gaze.

The focus adjusting unit 250 may measure an inter-pupil distance (IPD) of the user so that the user can enjoy an image suitable for his/her eyesight, and adjust the distance of the lens and the display of the VR device.

The processor 260 may control an overall operation of the VR device and a signal flow between internal components of the VR device, and may perform data processing. In addition, the processor 260 may process signals, data, information, and the like, transmitted and received by the VR device to and from an electronic device, a sensor device, and a training device. The processor 260 may be defined as a controller or a controller, and may control the operation of the VR device according to various embodiments of the present disclosure.

In addition, according to an embodiment of the present invention, the processor 260 performs a VR training operation through the output unit, obtains information on the degree of immersion while performing the VR training operation, and includes information on the degree of immersion and a preset threshold value. , and if the information on the level of immersion is less than the preset threshold value, it is possible to control to perform a first operation for improving the level of user's level of commitment. In this case, the first operation is at least one of an operation of outputting an immersion alarm to the user, an operation of providing the user with a rest time during VR training, or an operation of controlling the output of the VR device based on the characteristics of the currently provided virtual reality. may contain one. Also, the processor 260 may control to determine whether to continue the VR training by re-evaluating the user's level of immersion after performing the first operation.

In addition, the processor 260 may determine whether a preset condition is satisfied for acquiring the immersion-related information, and if the condition is satisfied, control to acquire the immersion-related information, and the preset condition is a viewpoint , a period, a priority of indirect information, and may include at least one of a prerequisite of indirect information.

The processor 260 obtains indirect information for the immersion-related information, determines the validity of the indirect information based on validity determination information set in relation to the indirect information, and if the indirect information is not valid, the indirect information The information may be discarded, and if the indirect information is valid, it is possible to control to acquire the information on the degree of immersion based on the indirect information.

Also, the processor 260 may control to set the type and time point of indirect information for acquiring the immersion-related information. The viewpoint may be set by an administrator according to importance among all viewpoints of VR training, and the type of indirect information may be determined based on characteristics of virtual reality provided at the viewpoint.

In addition, the processor 260 determines whether a plurality of immersion information is obtained, and if the plurality of immersion information is obtained, determines whether immersion information satisfying an absolute condition among the plurality of immersion information is included, and , if immersion information that satisfies the absolute condition is included, other immersion information is excluded and immersion information that satisfies the absolute condition is processed as the immersion-related information, and the immersion information that satisfies the absolute condition is If not included, an average value obtained from the plurality of immersion information may be controlled to be processed as the immersion-related information.

In addition, the processor 260 stores the immersion information and training section information corresponding to the point in time at which the immersion information is acquired, determines whether the training section is an essential training section, and based on whether the essential training section is determined It is possible to control to perform re-training for the training section.

In addition, the processor 260 determines the type of indirect information based on the immersion-related information, adjusts the output of the VR device based on the type of the indirect information, and adjusts the output of the VR device according to the output adjustment of the VR device. It can be controlled to determine whether the user's immersion level is improved.

The output unit 270 may be a device that outputs virtual reality provided by the VR device to the user. For example, the output unit 270 may output information on at least one of the five senses for virtual reality to the user.

3 is a diagram illustrating an operation of a VR device according to an embodiment of the present invention.

Referring to FIG. 3 , the VR training system may be started in operation 310 . The VR training system may be started in the VR device, and may be started under control from an electronic device associated with the VR device. When starting in the VR device, it may be started when a preset condition is satisfied (a preset time, a preset number of people, a preset location, etc.), or may be started according to a user's start command. The VR device may be started according to the control of the electronic device related to the VR device, if a preset condition is satisfied, or according to a start command of an administrator who manages the electronic device. Meanwhile, in the embodiment of FIG. 3 , the VR training system is described as an example, but in various embodiments of the present invention, the VR training system may correspond to an example, and various embodiments of the present invention are immersive in the VR system and the VR system. It can also be applied to various application examples to which the diagram can be applied.

In operation 320, the VR device performs an operation for VR training. The VR device may provide the user with information about virtual reality corresponding to VR training. For example, in the case of VR training for the mining industry, virtual reality can be provided for training to respond to dangerous situations that may occur at the mine site, and in the case of VR training for fire drills, emergency measures for fires in fire situations It can provide virtual reality for training to evacuate from a fire situation by performing or confirming an evacuation route, and in the case of VR training for military training such as reserve force training, various virtual reality assuming military education or wartime situations can be provided. can In addition, VR training may be used for training such as various education and training. The VR training may be conducted with one user participating, or a plurality of users may participate in the VR training and perform training together. The user is not only immersed in the virtual reality provided by the VR device, but also uses the input unit of the VR device or an input device connected to the VR device to manipulate or command the virtual reality, etc. They can interact, and the VR device can provide virtual reality that responds to a user's manipulation or command.

In operation 330, the VR device may acquire information about the user's immersion level. The degree of immersion may be obtained directly from the VR device or from another electronic device connected to the VR device. The user's immersion level may correspond to a numerical value indicating the degree of concentration or immersion in the virtual reality provided by the user who is experiencing virtual reality through the VR device. The information on the degree of immersion may be obtained from indirect information for evaluating the degree of immersion.

The indirect information for evaluating the degree of immersion may include at least one of various biometric information obtainable from a user using the VR device. The biometric information may include at least one of information such as electroencephalogram, respiration, body temperature, blood pressure, blood flow, current skin resistance, eyeball, electrocardiogram, electromyography, and pulse. The biometric information may be obtained from a sensor included in the VR device or may be obtained from an electronic device connected to the VR device and provided to the VR device.

Also, the indirect information may include at least one of information obtained from a manipulation or input command of a user using the VR device. For example, the information may include at least one of information such as the number of input of a command from the user, a command input period, a command input frequency, and a user's reaction speed.

Meanwhile, the indirect information for determining the degree of immersion is not limited to the above example, and may further include other information obtainable from a user using the VR device.

The VR device may acquire information on the degree of immersion of the user based on at least one of the indirect information. For example, in the case of using pulse information among biometric information, the degree of immersion may be evaluated based on information such as a pulse rate smaller than a preset threshold value or a change in pulse for a preset time. For other biometric information, a preset threshold value and its size may be directly compared, or the degree of immersion may be evaluated based on information such as an amount of change during a preset period as described above. In various embodiments of the present disclosure, a preset threshold value for comparison with indirect information may be defined as a first threshold value. The first threshold value may be set to a different value according to the type of indirect information.

Indirect information obtained from a user's manipulation or input command may also be evaluated for immersion in a similar manner. For example, the number of input of commands from the user may be compared with a preset threshold, and the degree of immersion may be evaluated based on information such as the amount of change in the number of input of commands by the user for a preset time.

In the above description, it has been described that information on the degree of immersion is obtained based on indirect information, but the indirect information itself may be used by substituting information on the degree of immersion. That is, the procedure of obtaining information on the degree of immersion from the indirect information may be omitted, and operation 340 may be performed according to the comparison result of the indirect information itself and the threshold value.

When information on effective immersion is obtained in operation 330 , the process may proceed to operation 340 . For example, when at least one piece of information on immersion is obtained, it may be determined that information on effective immersion has been obtained, and information on immersion that satisfies a condition set by a user or a manager of a training system is obtained. In this case, it may proceed to operation 340 . For a detailed operation thereof, refer to the additional embodiment of FIG. 5 .

In operation 340 , the VR device may compare the information on the degree of immersion acquired in operation 330 with threshold value information for an additional operation. Threshold value information for an additional operation is defined as second threshold value information below. The second threshold value may be set to a different value according to the type of indirect information based on obtaining the information on the degree of immersion. The VR device may perform different operations according to a result of comparing the acquired information on the degree of immersion with the second threshold value. Meanwhile, there may be a plurality of pieces of information on the degree of immersion acquired in the VR device. In this case, it is possible to determine whether operation 350 or operation 360 is performed according to whether the information on the at least one degree of immersion satisfies the threshold condition, and whether the information on the at least one level of immersion does not satisfy the threshold condition Accordingly, it may be determined whether operation 350 or operation 360 is performed. In addition, it may be determined whether a threshold condition is satisfied by applying information on a predetermined priority or weight to a plurality of pieces of immersion information. For a specific example of using a plurality of weight information, refer to the additional embodiment of FIG. 7 .

If it is determined that the degree of immersion is higher than the second threshold value, operation 350 may be performed. In operation 350 , the VR device determines whether VR training has ended. If the VR training is not finished, proceed to operation 320 and perform the previously described operation again. When the VR training is finished, operation 355 may be performed to end the operation of the VR device. Also, when the operation of the VR device is terminated, the VR training result may be recorded in the VR device or transmitted to an electronic device connected to the VR device to record the VR training result. For a detailed operation of recording the VR training result, refer to the additional embodiment of FIG. 8 .

When it is determined that the degree of immersion is lower than the second threshold value, operation 360 may be performed. When proceeding to the 360 motion, an additional motion related to immersion may be performed. In this case, since the user's level of immersion is low, the VR device performs at least one of an operation to improve the user's immersion level, an operation to provide a rest to the user, and an operation to provide an alarm for the level of immersion. can do.

The operation for improving immersion may include an operation of changing an output of the VR device. For example, the amount of visual information among virtual reality information of the VR device may be changed. For example, by changing the brightness, brightness, saturation, etc. corresponding to the visual information, it is possible to induce the improvement of the user's immersion. For example, when the auditory information is changed, it is possible to induce the improvement of the user's immersion level by changing the volume level. The operation for improving immersion may selectively change visual information, auditory information, or other information according to characteristics of a currently played virtual reality. In addition, by increasing the number of outputs for inducing a user command, it is also possible to induce improvement in user immersion. The size of the output may be determined according to the previously acquired degree of immersion of the user.

The operation of providing a rest to the user may be an operation for stopping the operation of the VR device for a preset time or for notifying the user of a break for a preset time so that the user can use the VR device after resting for a preset time or longer. The break time may be determined based on the previously acquired level of user immersion. The lower the immersion, the longer the break may be required. However, although the VR device has started an operation for providing relaxation, an input requesting re-evaluation of the user's level of immersion may be input to the VR apparatus. In this case, the VR apparatus re-evaluates the user's level of immersion to rest or 320 motion. can proceed with The operation of providing an alarm to the user may indicate that the current level of immersion is less than or equal to a preset threshold value, or may indicate that the efficiency for VR training is falling.

In operation 370 , it may be determined whether to continue VR training. For example, after the 360 operation, information on the user's immersion level may be acquired again, and based on this, it is possible to determine whether to proceed with VR training. When the degree of immersion obtained in operation 370 is equal to or less than the third threshold value, operation 360 may be additionally performed again, or it may be determined that VR training cannot be performed any more and the procedure may be terminated. When the procedure ends, operation 355 may be performed. When the degree of immersion exceeds the third threshold, it is determined that VR training can be continued, and operation 350 may be performed. Operation 370 may be omitted.

Meanwhile, in the embodiment of FIG. 3 , the operation of the VR device has been described based on operation 340 when the user's level of immersion is less than a preset threshold, but the scope of the present invention is not limited thereto, and The same can be applied. If the user is over-immersed in the virtual reality provided by the VR device, it may affect the safety of the user, and thus an action to suppress the over-immersion situation may be required.

For example, in operation 340 , the VR device may determine whether the degree of immersion exceeds a preset threshold value (a fourth threshold value) for determining over-immersion. If the fourth threshold is not exceeded, the above-described operation 350 may be performed.

When the degree of immersion exceeds the fourth threshold, operation 360 and operation 370 according to operation 360 may be performed. In operation 360, the VR device may perform at least one of a notification of over-immersion, provision of a break time, and an operation for lowering the user's immersion level.

The operation for reducing immersion may include an operation of changing an output of the VR device. For example, the amount of visual information among virtual reality information of the VR device may be changed. For example, it is possible to induce a decrease in user immersion by changing brightness, brightness, saturation, etc. corresponding to visual information. For example, when auditory information is changed, it is possible to induce a decrease in user immersion by changing the volume level. The operation for improving immersion may selectively change visual information, auditory information, or other information according to characteristics of a currently played virtual reality. In addition, by reducing the number of outputs for inducing a user command, it is also possible to induce an improvement in the user's immersion. According to the degree of immersion of the user obtained in the above manner, the size of an output provided by the VR device or an input required by the VR device for reducing the degree of immersion of the user may be adjusted.

The operation of providing a rest to the user may be an operation for stopping the operation of the VR device for a preset time or for notifying the user of a break for a preset time so that the user can use the VR device after resting for a preset time or longer. The break time may be determined based on the previously acquired level of user immersion. The higher the immersion, the longer the rest period may be required. However, although the VR device has started an operation for providing relaxation, an input requesting re-evaluation of the user's level of immersion may be input to the VR apparatus. In this case, the VR apparatus re-evaluates the user's level of immersion to rest or 320 motion. can proceed with The operation of providing an alarm to the user may indicate that the current level of immersion exceeds a preset threshold value, may indicate that the efficiency for VR training may decrease, or that a risk may occur to the user.

The user's over-immersion control operation as described above is not limited to the above embodiment, and various operations of the present invention described below can be applied not only when the level of immersion is low, but also in an over-immersion state in which the level of immersion exceeds a preset threshold value. can

4 is a diagram illustrating a method of acquiring immersion in a VR device according to various embodiments of the present disclosure.

The embodiment of FIG. 4 may be applied as an additional embodiment to operation 330 of FIG. 3 .

In operation 410, a condition for obtaining information on the degree of immersion may be set. The condition may be at least one of a preset period, a preset specific time point, a preset level of indirect information, a priority of indirect information, or a setting for a precedence condition of indirect information, and the like.

For example, if the preset condition is a periodic condition, the process proceeds to operation 420, and in operation 420, the VR device determines whether it is a preset period for acquiring information on the degree of immersion. If it is determined that it is a preset period, the VR device may acquire information on the degree of immersion from at least one of indirect information on biosignals and indirect information on user command input.

For example, if the preset specific time point is the condition, operation 430 proceeds. In operation 430, the VR device determines whether it is a specific point in time for acquiring information on immersion. The specific time point may be aperiodic, and may be determined as an arbitrary time point according to the characteristics of virtual reality provided by the VR device or the characteristics of the VR training system. For example, it may be set to acquire information on immersion at an important point in the VR training system, or at a point in time when a specific virtual reality for moving to the next stage is provided.

If the preset level of indirect information is the condition, operation 440 may be performed. When the preset level of indirect information satisfies a preset condition, the VR device may acquire information on the degree of immersion. For example, when the eye movement is a preset condition, when the eye movement is less than or equal to the preset movement, it may be determined that it is necessary to obtain information on the degree of immersion. If it is determined that information on the degree of immersion is required according to operation 440, the process proceeds to operation 445 to obtain information on the degree of immersion from at least one of indirect information on biosignals or indirect information on user command input or preset indirect information. can

Meanwhile, the first indirect information used in operation 440 and the second indirect information used in operation 445 may be indirect information for mutual complementation. For example, if it is determined in operation 440 that information on immersion is necessary based on eye movements, in operation 445, information on immersion is provided by including at least one of indirect information about a user command input rather than a body signal. can be obtained In addition, if it is determined in operation 440 that information on the degree of immersion is necessary based on the movement of the eyeball, in operation 445 , information on the degree of immersion may be acquired by including at least one other bio-signal other than visual information.

For example, when the priority of indirect information is a condition, operation 450 may be performed. If the level of indirect information of the preset priority satisfies the preset condition in operation 450 , the VR device may acquire information on the degree of immersion in operation 455 . For example, it may be determined that the biosignal has a higher priority than the user command input, and it may be determined that the user command input has a higher priority than the biosignal. If the priority of the biosignal is high, even if there is a very low input to the user command input, the process does not proceed to operation 455 because the preset condition is not satisfied in operation 450, and the VR device does not acquire information on immersion. does not If the level of indirect information related to the biosignal satisfies the preset condition, it is determined that the condition of operation 450 is satisfied and proceeds to operation 455, and the VR device additionally determines the level of immersion from indirect information about the biosignal or user command input. information can be obtained.

For example, if the prerequisite of indirect information is a condition for acquiring information on immersion, the process proceeds to operation 460 . If the preset prerequisite is satisfied in operation 460, the VR device may acquire information on the degree of immersion in operation 465 . For example, the preceding condition may be set as obtaining indirect information on input of a user command when a preset condition is satisfied in relation to a biosignal, and when a preset condition for input of a user command is satisfied It may be configured to acquire information related to biosignals. For example, if the heart rate as a biosignal is less than or equal to a preset threshold, it is determined whether the frequency of user command input is less than or equal to a preset threshold, and if both conditions are satisfied, proceed to operation 465 to determine the level of immersion. information can be obtained.

Conditions or settings for acquiring information on immersion in the above-described VR device may be set according to self-setting of the VR device, setting of the VR training system, input of a VR training system manager, and the like. Also, it may be determined according to the type of virtual reality provided through the VR device. For example, if the user's participation in the virtual reality provided through the VR device or the user's command input is not required, or if the frequency is remarkably low, the conditions, priorities, and prerequisites for the biosignals are determined in consideration of the characteristics. You can set and obtain information about immersion. For example, if virtual reality provided through a VR device consistently requires a user's command input, the user's command input may be set as a basic condition, priority, or precedence condition.

Meanwhile, although the conditions of 420, 430, 440, 450, and 460 have been described above for convenience of description, it is also possible to use a combination of at least two of the conditions.

In various embodiments of the present disclosure, there may be limitations to the sensor in the VR device. Accordingly, when information about a user command input is obtained from the VR device, the VR device may request the electronic device to which the VR device is connected to provide user biometric information, and the VR device may measure the electronic device having a sensor or the user's biosignal. The information collected from the sensor can be acquired by receiving it through electronic politics. In addition, although user biometric information is received every time a preset period or a specific event occurs, it may be considered only when a specific condition for inputting a user command is satisfied. Conversely, it may be considered to determine the user command input when a specific condition for the biosignal is satisfied, and in this case, the type of indirect information may be preset. In addition, when specific information is collected while monitoring a part of a biosignal or only a part of a user command, other biometric information and other user command information may be collected. A method of obtaining information on immersion according to various embodiments of the present disclosure is not limited to the above example.

5 is a diagram illustrating a method of determining whether information on effective immersion level has been obtained by a VR device according to various embodiments of the present disclosure;

Indirect information used to determine the degree of immersion may not always be valid. Invalid indirect information may be obtained from a defect in the electronic device, a defect in a sensor, and the like, and information on immersion obtained from the invalid indirect information may be determined to be invalid. For example, when the heart rate is 0, it is difficult to determine that indirect information such as when the body temperature is 30 degrees or less is a valid biosignal of the user. Based on this, it is difficult to determine the level of immersion and control the VR device using the level of immersion. It may not correspond to the spirit of the embodiment of the invention. Therefore, an effective range can be set for each indirect information in advance, and when information out of this is collected, it is not determined that information on effective immersion has been collected.

In operation 510 , the type of indirect information and validity determination information to be used for acquiring immersion in the VR device may be set. Validity determination information may be set to a value within a specific range and determined to be invalid if it is outside a specific range, and may be determined to be invalid if it is set to a specific value and is less than or equal to or greater than a specific value, It may be the number of inputs during a set period (eg, 1000 inputs per minute, etc.).

For example, as indirect information, 35 to 42 can be set in a specific range for the body temperature of the biosignal, 30 can be set as the pulse rate, and the validity of input 10 or more times per second as user input corresponds to indirect information. information can be set.

When the VR device acquires indirect information, in operation 520 , the VR device may determine whether the acquired indirect information is indirect information preset in relation to validity determination. In the case of preset indirect information, it may be determined whether the corresponding information is valid information using validity determination information corresponding to the indirect information. For example, when body temperature information obtained as indirect information is 20 degrees, it may be determined as invalid because it is out of a preset range. For example, when the pulse rate obtained as indirect information is 50, it may be determined to be valid because it is a value greater than or equal to a preset threshold value. For example, if the number of user command input per second obtained as indirect information is 30, since it is a value that exceeds a preset condition, there may be a problem with the input device, and the indirect information may be determined to be invalid. have.

If it is determined that the corresponding indirect information is valid, operation 530 is performed. In operation 530, the VR device may acquire information on the degree of immersion by using the acquired indirect information.

If it is determined that the corresponding indirect information is invalid, operation 540 is performed. In operation 540, since the obtained indirect information is invalid, the VR device discards the corresponding information, and does not trigger information for acquiring the immersion information based on the indirect information. If the invalid indirect information is a prerequisite for acquiring information on immersion according to various embodiments of the present invention, it can be informed that resetting of the prerequisite is necessary, and when performing an immersion-related operation You may use only other indirect information, excluding that condition.

Meanwhile, even if one or more invalid indirect information is collected, when other valid information is collected, information on the degree of immersion may be acquired based on the valid indirect information.

6 is a diagram illustrating a method of setting a viewpoint for acquiring immersion in a VR device according to various embodiments of the present disclosure.

The manager of the VR training system may not be able to reflect the requirements of always acquiring the degree of immersion, or that the point of obtaining the degree of immersion is determined according to the user's situation (user's biosignal or user's command input). In addition, monitoring the user's situation at all times can act as a load on the VR device. Therefore, according to the setting of the VR training manager or the setting of the user, the time point or condition for acquiring immersion may be preset.

Efficiency according to various embodiments of the present disclosure may be improved when immersion information is obtained using indirect information most suitable for evaluating immersion according to the environment provided by the VR device at a time desired by the administrator.

For example, in FIG. 6 , it is assumed that VR training is designed such that VR training starts at time 610 and VR training ends at time 650 in FIG. 6 . The VR training is set to acquire information on the degree of immersion based on the first indirect information at time 620, and is set to acquire information on the level of immersion based on the second indirect information at time 630, and at time 640 may be set to acquire information on the degree of immersion based on the third indirect information and the fourth indirect information.

For example, visual information provided by the VR device may be the most important viewpoint at viewpoint 620, and indirect information related to visual information such as eye movement and pupil state may be set as the first information. For example, if a user command input according to user participation is most actively requested at time 630, the user command input related to user participation may be set as indirect information. In addition, since only one indirect information needs to be set, the third indirect information and the fourth indirect information may be considered together as in the 640 time point.

In the above, the section or time point for acquiring immersion-related information is set by the administrator according to the importance among the entire time points of VR training, and the type of indirect information is determined by the VR device based on the characteristics of the virtual reality provided at the time point. may be In addition, it does not exclude that the administrator sets the type of the indirect type.

According to the above setting, an administrator or a user can preset a section in which user concentration is important, and can obtain a higher training effect by determining the level of immersion in the section and controlling the operation of the VR device.

7 is a diagram illustrating a method of determining whether a threshold condition is satisfied based on information on a plurality of levels of immersion according to various embodiments of the present disclosure;

In operation 710, the VR device determines whether there are a plurality of pieces of information on the degree of immersion acquired at a specific point in time or at a specific section. The information on the degree of immersion obtained from different indirect information may be determined as information about a plurality of degrees of immersion with respect to different degrees of immersion. If there are not a plurality, operation 720 proceeds, and operation 340 of FIG. 3 is performed using information on the degree of immersion and threshold value information corresponding to the information.

If there are a plurality of pieces of information on the degree of immersion obtained, it may proceed to operation 730 to determine whether information on the degree of immersion satisfying an absolute condition is included. The absolute condition refers to a condition in which, when the information on the corresponding level of immersion satisfies a preset condition, other conditions for the level of immersion need not be considered. For example, if information on the pupil is an absolute condition, but information on the pupil is not input for a preset period of time, it can be determined that the user's sleep state or the user cannot receive visual information about virtual reality. However, information on other immersion levels may not be judged. If information on the degree of immersion that satisfies the absolute condition is included, operation 735 may be performed. In operation 735, the VR device performs an operation corresponding to operation 340 of FIG. 3 by using information on the degree of immersion corresponding to the absolute condition without considering other information even when information on other levels of immersion is obtained. can

If information on the degree of immersion that satisfies the absolute condition is not included, operation 740 may be performed. In case of proceeding to operation 740, the VR device may calculate average immersion level information by using a priority or weight condition set for each piece of immersion level information. Thereafter, the VR device may proceed to operation 745 and perform an operation corresponding to operation 340 of FIG. 3 using the average immersion information. For example, the number of immersion information to be used for averaging may be set, and information on high priority immersion may be selected for averaging by using the priority. In addition, if there is a weight, a weight may be applied to each information on the degree of immersion when calculating for averaging.

In the embodiment of FIG. 7 , operations 730 and 735 may be omitted.

On the other hand, in the case of VR training in which a plurality of users participate together, information on a plurality of immersion levels may be obtained from a plurality of users, and in this case, the method of processing information on the plurality of levels of immersion may be equally applied. have.

8 is a diagram illustrating a method of recording a VR training result according to various embodiments of the present disclosure.

The VR device may record VR training results. It may be reasonable to re-perform VR training for a section in which the user's immersion has fallen. To this end, when a specific event occurs in relation to the user's immersion, the VR training device may input a result in the corresponding section of the VR training, and perform an operation of additionally providing the user's evaluation or VR training based on the input result can do.

In operation 810 , when a specific event related to immersion occurs, the VR device may determine a VR training section corresponding to the degree of immersion and store information about the section. For example, the specific event may be a case in which the degree of immersion is less than or equal to a preset threshold, and the section may be a section in which the degree of immersion is less than or equal to a preset threshold.

In operation 820 , the VR device may determine whether the immersion lowering period collected in operation 810 is an essential training period. If it is determined as the required training section, the process proceeds to operation 830, and VR training can be started again for the corresponding section. In addition, it can be recorded that VR training is not completed due to the omission of the required training section. If VR training is not completed, the user may need to perform VR training for the corresponding section again later.

If it is not determined as the required training period, operation 840 may be performed. In operation 840, the VR device may calculate the user's VR training result and record the training result. If the level of immersion was low in a specific section, but the calculated VR training result satisfies a preset condition, it may be determined that the VR training was successful, and this may be recorded. If the calculated VR training result does not satisfy a preset condition due to low immersion multiple times or for a long time, it may be determined that the VR training has failed, and this may be recorded.

9 is a diagram illustrating a method of sequentially controlling VR output based on immersion according to various embodiments of the present disclosure.

In operation 910 , the VR device may determine the type of indirect information for which the degree of immersion is less than or equal to a preset threshold. For example, when it is determined that the degree of immersion based on eye movement is equal to or less than a preset threshold value, eye movement or relevance of visual information may be determined.

In operation 920, the VR device may control the output of the VR device based on the determined type of indirect information. For example, in the case of eye movement or visual information, the VR device may control the output amount of visual information that provides virtual reality, or control the output to stimulate eye movement or the user's vision. For example, an output amount related to visual information may be controlled by adjusting brightness, saturation, brightness, screen size, and the like.

Also, the VR device may specifically request user input. For example, the user's eye movement may be induced. For example, the user may be induced to move the eyeball left and right up and down. In addition, it is also possible to command the user input more than n times.

In operation 930, the VR device again acquires information on the user's level of immersion, and determines whether the re-acquired level of immersion is less than or equal to a preset threshold value. If the threshold value is exceeded, operation 940 proceeds. In this case, since the level of immersion is restored to a normal state, an operation corresponding to operation 320 of FIG. 3 may be performed.

When the re-acquired level of immersion is less than or equal to the preset threshold, since the level of immersion is still not recovered, operation 920 may be additionally performed. The VR device may change the amount of output or the type of output in performing operation 920 . For example, if the output was previously changed in relation to the visual information, the output of the auditory information may be additionally changed, and the amount of change in the output of the visual information may be further adjusted.

According to the above method, in the present invention, indirect information having a problem in immersion can be identified, the output of the VR device can be changed based on the indirect information, and the VR device related to the indirect information affecting the current immersion By changing the output, the user can quickly regain immersion.

In addition, the embodiments disclosed in the present specification and drawings are merely provided for specific examples in order to easily explain and understand the contents of the present invention, and are not intended to limit the scope of the present invention. Therefore, the scope of the present invention should be construed as including all changes or modifications derived based on the technical characteristics of the present invention in addition to the embodiments disclosed herein are included in the scope of the present invention.

110: VR device
120: sensor device
130: electronic device

Claims (16)

A method of operating a virtual reality (VR) device, the method comprising:
performing a VR training operation;
acquiring information on immersion while performing the VR training operation;
comparing the information on the degree of immersion with a preset threshold value; and
When the information on the level of immersion is less than the preset threshold, performing a first operation for improving the level of user engagement;
The step of obtaining information on the degree of immersion is
determining whether a plurality of immersion information has been acquired;
if the plurality of immersion information is obtained, determining whether immersion information satisfying an absolute condition is included among the plurality of immersion information;
If the immersion information that satisfies the absolute condition is included, excluding other immersion information and processing the immersion information satisfying the absolute condition as the immersion-related information, and
and processing, as the immersion-related information, an average value calculated using a priority or weight condition set for the plurality of immersion information if the immersion information that satisfies the absolute condition is not included. how to do it According to claim 1, wherein the first operation,
At least one of an operation of outputting an alarm for immersion to the user, an operation of providing a break time during VR training to the user, or an operation of controlling the output of a VR device based on the characteristics of the currently provided virtual reality,
The method further comprising the step of determining whether to continue the VR training by re-evaluating the level of immersion of the user after performing the first operation. According to claim 1, wherein the step of obtaining information on the degree of immersion,
determining whether a preset condition is satisfied for obtaining the immersion-related information;
Comprising the step of obtaining information on the degree of immersion when the condition is satisfied,
The preset condition comprises at least one of a time point, a period, a priority of indirect information, and a precedence condition of indirect information. According to claim 1,
obtaining indirect information for the immersion-related information;
determining the validity of the indirect information based on validity determination information set in relation to the indirect information;
discarding the indirect information if the indirect information is invalid; and
If the indirect information is valid, the method further comprising the step of acquiring information on the degree of immersion based on the indirect information. According to claim 1,
Further comprising the step of setting the type and time point of indirect information for acquiring the immersion-related information,
The viewpoint is set by an administrator according to importance among all viewpoints of VR training, and the type of indirect information is determined based on characteristics of virtual reality provided at the viewpoint. delete According to claim 1,
Storing immersion information and training section information corresponding to a time point at which the immersion information is acquired;
determining whether the training interval is a required training interval; and
The method further comprising the step of performing re-training for the training period based on whether the required training period is determined. The method of claim 1, wherein performing the first operation comprises:
determining the type of indirect information based on the immersion-related information;
adjusting the output of the VR device based on the type of the indirect information;
The method further comprising the step of determining whether the user's immersion level is improved according to the output adjustment of the VR device. In a virtual reality (VR) device,
output unit; and
A VR training operation is performed through the output unit, information on the degree of immersion is obtained during the execution of the VR training operation, the information on the degree of immersion is compared with a preset threshold value, and the information on the degree of immersion is obtained from the group. If it is less than the set threshold, including a processor that controls to perform the first operation for improving the user's immersion,
The processor is
It is determined whether a plurality of immersion information is obtained, and if the plurality of immersion information is obtained, it is determined whether immersion information that satisfies an absolute condition is included among the plurality of immersion information, and the degree of immersion that satisfies the absolute condition If information is included, other immersion information is excluded and immersion information satisfying the absolute condition is processed as the immersion-related information, and if immersion information satisfying the absolute condition is not included, the plurality of A VR device that controls to process an average value calculated using a priority or weight condition set for the immersion information as the immersion-related information. 10. The method of claim 9, wherein the first operation,
At least one of an operation of outputting an alarm for immersion to the user, an operation of providing a break time during VR training to the user, or an operation of controlling the output of a VR device based on the characteristics of the currently provided virtual reality,
VR device, characterized in that the processor controls to determine whether to continue the VR training by re-evaluating the level of immersion of the user after performing the first operation. 10. The method of claim 9, wherein the processor,
determining whether a preset condition is satisfied for obtaining the immersion-related information, and controlling to obtain the information on the immersion level if the condition is satisfied;
The preset condition includes at least one of a viewpoint, a period, a priority of indirect information, and a precedence condition of indirect information. 10. The method of claim 9, wherein the processor,
obtaining indirect information for the immersion-related information, determining the validity of the indirect information based on validity determination information set in relation to the indirect information, and discarding the indirect information if the indirect information is not valid; If the indirect information is valid, the VR device, characterized in that controlling to acquire the information on the degree of immersion based on the indirect information. 10. The method of claim 9, wherein the processor,
Control to set the type and time point of indirect information for acquiring the immersion-related information,
The viewpoint is set by an administrator according to importance among all viewpoints of VR training, and the type of indirect information is determined based on characteristics of virtual reality provided at the viewpoint. delete 10. The method of claim 9, wherein the processor,
Storing immersion information and training section information corresponding to the point at which the immersion information is acquired, determining whether the training section is a required training section, and re-training for the training section based on whether the required training section is determined VR device, characterized in that it controls to perform. 10. The method of claim 9, wherein the processor,
Determine the type of indirect information based on the immersion-related information, adjust the output of the VR device based on the type of indirect information, and determine whether the user's immersion is improved according to the output adjustment of the VR device VR device, characterized in that it controls to judge.

Images