KR101756326B1 - Method, system and computer-readable recording medium for virtual reality game control simulation based on natural user interface and natural user experience - Google Patents

Abstract

The present invention relates to a method and system for game control simulation of a VR environment based on NUI / NUX, comprising the steps of: defining a set command as at least one gesture and storing the command in a database; receiving the user's operation and EMG data Recognizing a gesture that is mapped to the input operation of the user, evaluating the gesture recognized using the input operation of the user, the EMG data, and evaluating the recognized gesture Wherein the step of evaluating the gesture comprises the steps of: receiving a recognition rate at which a gesture is recognized according to a user's operation; inputting a user's operation preference with respect to at least one or more gestures; Calculating the muscle fatigue, and determining the recognition rate, Calculating a UX (User Experience) score using the meat fatigue, and selecting a gesture to be provided to the user based on the UX score. According to the NUI / NUX-based VR environment game control simulation method and system according to the present invention configured as described above, gesture evaluation is performed on a defined gesture, thereby providing a gesture most suitable for a user.
In addition, according to the NUI / NUX-based VR environment game control simulation method and system according to the present invention, by selecting a gesture best suited to a user by measuring gestures, body movements, and electromyograms, have.

Description

METHOD, SYSTEM AND COMPUTER READABLE RECORDING MEDIUM FOR VIRTUAL REALITY GAME CONTROL SIMULATION BASED ON NATURAL USER INTERFACE AND NATURAL USER EXPERIENCE BACKGROUND OF THE INVENTION 1. Field of the Invention [0001]

The present invention relates to a method and system for simulating a VR environment game control based on NUI / NUX, and more particularly, to a system and method for controlling a game by using a NUI And more particularly,

Conventional game control methods mainly use game controllers, and game controllers can be operated by sitting or lying down, which is recognized as the most suitable game control device for human engineering.

Recently, research on the control method of VR game has been progressed in accordance with development of VR (Virtual Reality) HMD (Head Mounted Display). In order to increase the realism, the VR game control method recognizes the motion by holding the tool in the hand or recognizes the motion through the wearable device.

Most of the researches are conducted in such a manner that the game is controlled based on the standing of the user, and this method has a problem that it is difficult to enjoy the game for a long time because the user uses the entire body.

Therefore, it is necessary to control the game through the recognition of the motion of the body or the hand in the sitting state.

Korean Patent No. 10-1231989 (February 04, 2013)

SUMMARY OF THE INVENTION It is an object of the present invention to provide a gesture best suited to a user by performing a gesture evaluation on a defined gesture.

It is another object of the present invention to control a game easily even when a user is sitting by selecting a gesture best suited to a user by measuring a hand gesture, a body motion, and an EMG.

According to another aspect of the present invention, there is provided a method for controlling an operation of a user, the method comprising: defining a set command as at least one gesture and storing the command in a database; inputting a user's operation and EMG data of the user; The method comprising the steps of: recognizing a gesture; evaluating the gesture recognized using the input user activity and the EMG data; and evaluating the recognized gesture, wherein evaluating the gesture comprises: The method comprising the steps of: receiving a recognition rate at which a gesture is recognized according to an operation of a user; inputting a user's operation preference with respect to at least one or more gestures; calculating muscle fatigue of a user using EMG data; UX (User Experience) scores using preference and muscle fatigue Stage and based on the UX score that is configured to include the steps of selecting a gesture to be provided to the user.

delete

In the method for simulating the game control of the VR environment based on NUI / NUX according to the present invention, the step of evaluating the gesture includes calculating the UX score by assigning different weights to the recognition rate, the motion preference, and the muscle fatigue, .

In the method of simulating a game control based on NUI / NUX based on NUI / NUX according to the present invention, the step of selecting a gesture to be provided to the user may be performed when the command corresponding to the recognized gesture is stored in only one gesture And if the score is less than the set threshold value, a new operation is input to the command corresponding to the gesture and the gesture is updated.

A method for simulating a VR environment game control based on NUI / NUX according to the present invention, the step of selecting a gesture to be provided to the user may include a step of, when a command corresponding to the recognized gesture is stored as a plurality of gestures, The gesture having the highest score is selected as the most suitable gesture for the user.

In the method for simulating NUI / NUX-based VR environment game control according to the present invention, in the step of selecting a gesture to be provided to the user, gestures having the highest value as a result of calculation of the UX score for the plurality of gestures The gesture having the lowest muscle fatigue is selected as the most suitable gesture for the user.

According to another aspect of the present invention, there is provided a computer program product including a database for defining and storing a set command as at least one gesture, a sensor unit for measuring an operation of the user and an electromyogram signal of the user, And an analyzer for analyzing the evaluation of the gesture recognized by using the measured user operation and the electromyogram signal and the gesture recognized and selecting an optimum gesture for the user .

A system for simulating a VR environment game control based on NUI / NUX according to the present invention is characterized in that the evaluation unit receives a recognition rate at which the gesture is recognized according to an operation of the user, Calculating a user's muscle fatigue using the EMG signal, and calculating a UX (User Experience) score using the recognition rate, the operation preference, and the muscle fatigue.

In the system for simulating NUI / NUX-based VR environment game control according to the present invention, the evaluator calculates the UX score by assigning different weights to the recognition rate, the operation preference, and the muscle fatigue, respectively do.

In the system for simulating the NUI / NUX-based VR environment game control according to the present invention, when the command corresponding to the recognized gesture is stored in only one gesture, if the UX score of the gesture is less than a preset threshold value, And the gesture is updated by receiving a new operation for a command corresponding to the gesture.

In the system for simulating NUI / NUX-based VR environment game control according to the present invention, when the command corresponding to the recognized gesture is stored as a plurality of gestures, the analyzing unit calculates UX scores for the plurality of gestures And the gesture having the highest score is selected as the most suitable gesture for the user.

In the system for simulating VU environment game control based on NUI / NUX according to the present invention, when a plurality of gestures having the highest value as a result of calculation of the UX score for the plurality of gestures are present, And a gesture is selected as a gesture most suitable for the user.

According to the NUI / NUX-based VR environment game control simulation method and system according to the present invention configured as described above, gesture evaluation is performed on a defined gesture, thereby providing a gesture most suitable for a user.

In addition, according to the NUI / NUX-based VR environment game control simulation method and system according to the present invention, by selecting a gesture best suited to a user by measuring gestures, body movements, and electromyograms, have.

1 is a block diagram illustrating a system for simulating a VR environment game control based on NUI / NUX according to an embodiment of the present invention.
2 is a diagram illustrating a configuration of a sensor unit for recognizing a user's motion and measuring a user's EMG according to an embodiment of the present invention.
3 is a flowchart illustrating a method of simulating a VR environment game control based on NUI / NUX according to a preferred embodiment of the present invention.
4 is a flow chart illustrating a method for evaluating a gesture in accordance with an embodiment of the present invention.
5 is a diagram illustrating various candidate gestures used in game control according to an embodiment of the present invention.

The present invention may have various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a system for simulating a VR environment game control based on NUI / NUX according to an embodiment of the present invention. Referring to FIG. 1, a VR environment game control simulation system 100 includes a database 110, a sensor unit 120, a recognition unit 130, an evaluation unit 140, and an analysis unit 150.

The database 110 defines and stores the set command as at least one gesture. That is, the developer defines commands on the VR environment game control simulation system 100 as at least one gesture, learns the defined gestures, and stores the learned gestures in the database 110.

The sensor unit 120 includes a sensor for measuring a user's operation and a sensor for measuring a user's EMG signal. Here, the sensor unit 120 will be described in more detail with reference to FIG.

2 is a diagram illustrating a configuration of a sensor unit for recognizing a user's motion and measuring a user's EMG according to an embodiment of the present invention. Referring to FIG. 2, it is composed of a first sensor, a second sensor, and an EMG sensor. The first sensor and the second sensor can be implemented as a 3D sensor, which is not limited to a 3D sensor.

The sensor unit configured as shown in FIG. 2 can measure the hand motion using the first sensor and measure the body motion using the second sensor. In addition, an EMG sensor can be attached to a user's muscle to measure an EMG signal.

Here, the first sensor is mounted on an HMD (Head Mounted Display) to measure a finger-based operation by capturing an operation range of the hand, and the second sensor is installed at a user's position and a remote location to measure a user's body motion have. In addition, the electromyogram sensor can measure an electromagnetic wave, that is, an electromyogram signal, generated in a user's muscle by being attached to a user's muscle region.

As described above, the sensor unit 120 can measure the operation of the user using two sensors, and three or more sensors can be utilized for more precise measurement. In addition, the user's operation can be measured with only one sensor, which can change the number of sensors for measuring body motion according to the user's setting.

1, the recognition unit 130 will be described. The recognition unit 130 recognizes a gesture that is mapped to the operation of the user measured by the sensor unit 120. [ That is, the recognition unit 130 recognizes whether a gesture stored in the database 110 is mapped to a user operation measured using at least one sensor in the sensor unit 120, and recognizes a gesture mapped with a user's operation do.

The evaluation unit 140 evaluates the gesture recognized by the recognition unit 130 using the user's activity measured by the sensor unit 120 and the EMG signal. That is, the gesture recognized by the recognition unit 130 can be calculated by the user using the user's action measured by the sensor unit 120 and the EMG signal.

Next, the analysis unit 150 analyzes the contents evaluated by the evaluation unit 140 with respect to the gesture recognized by the recognition unit 130, and selects a gesture best suited to the user. That is, the analyzing unit 150 can check the evaluation of the recognized gesture, analyze it in various ways to determine whether there is no more suitable gesture for the user or whether there is a gesture to be replaced, and select and guide the most suitable gesture to the user.

As described above, a system for simulating VR environment game control based on NUI / NUX has been described. Next, referring to FIG. 3, a method of simulating a game control of a VR environment based on NUI / NUX will be described.

3 is a flowchart illustrating a method of simulating a VR environment game control based on NUI / NUX according to a preferred embodiment of the present invention.

Referring to FIG. 3, the VR environment game control simulation system defines a set command as at least one gesture and stores it in a database (S10). Here, the step (S10) of storing at least one gesture in the database defines a gesture by the developer as to the operation of the command set in the VR environment game control simulation system, and learns the defined gesture and stores it in the database.

In particular, a developer may store only a single gesture for a set command, and may include a plurality of candidate groups. Therefore, the developer defines and stores at least one gesture for the set command of the VR environment game control simulation system.

Next, the VR environment game control simulation system receives the user's operation and the user's electromyogram data (S20). More specifically, the VR environment game control simulation system receives a user's motion and EMG data from a sensor unit that measures a user's motion and a user's EMG signal.

Here, the sensor unit measures the operation of the user using at least one sensor, and the detailed description is omitted in FIG. 2 because it is described in FIG.

Next, the VR environment game control simulation system recognizes a gesture that is mapped to the input user's operation (S30). More specifically, the VR environment game control simulation system analyzes the operation of the user input from the sensor unit to check whether there is a mapped gesture among the gestures stored in the database, and recognizes the action of the user as a corresponding gesture if it exists. Here, various motion recognition algorithms can be used as a method of analyzing a user's motion.

The VR environment game control simulation system recognizes the gesture mapped to the user's operation, and then evaluates the recognized gesture using the user's operation input from the sensor unit and the EMG data (S40). The step (S40) of evaluating the more specifically recognized gesture will be described with reference to FIG.

4 is a flow chart illustrating a method for evaluating a gesture in accordance with an embodiment of the present invention. Referring to FIG. 4, in step S40 of evaluating a gesture, a recognition rate at which a gesture is recognized according to a user's operation is input (S41).

More specifically, the user repeatedly performs an operation on the recognized gesture to calculate a recognition rate at which the gesture is recognized, and inputs the recognition rate to the VR environment game control simulation system. Here, the recognition rate can be input as a percentage of the number of operations perceived by the recognized gesture and perceived as a gesture, thereby allowing the user to confirm how well the gesture is recognized.

Next, the user's operation preference is input to at least one gesture (S42). Here, the operation preference is input to the user's subjective score with respect to the gesture of the VR environment game control simulation system.

In other words, a user can enter a numerical value in favor of a moving motion in order to recognize a gesture in a VR environment game control simulation system by giving a subjective score to a moving motion in order to use the stored gesture.

Also, when there is a candidate group gesture for a command set in the gesture to be used, the user can input the operation preference for the candidate group gesture. This allows the user to compare which of the recognized gestures and candidate group gestures is better or better suited to them.

After inputting the motion preference, the VR environment game control simulation system calculates the user's muscle fatigue using the received EMG data (S43). Here, the EMG data are characterized in that the amplitude and frequency of the EMG are reduced according to the fatigue of the human muscle.

Therefore, when the gesture recognition rate is input, it is possible to record how many times the gesture is used and store the data such as the amplitude and the frequency of the EMG, thereby confirming how quickly the frequency of the EMG decreases while taking the gesture. Using this feature, you can calculate muscle fatigue during gestures by performing the same number of times for different gestures.

When the muscle fatigue calculation is completed, the VR environment game control simulation system calculates the UX (User Experience) score using the received recognition rate, the motion preference, and the calculated muscle fatigue (S44). Here, the UX score can be calculated as shown in Equation (1).

The UX score is calculated using the recognition rate, the motion preference, and the muscle fatigue calculated and input in the step of evaluating the gesture (S40) as shown in Equation (1). Here, the score of muscle fatigue is higher as the fatigue effect is lower, and the UX score can be obtained by using the percentage value of motion preference and recognition rate.

In addition, each value can be weighted according to the setting. That is, different weights are assigned to each of the muscle fatigue, the motion preference, and the recognition rate according to the user, and it is possible to adjust the index reflected in the UX score by multiplying the corresponding weight values.

Also, the UX score can be adjusted according to the weight, so that the user can assign a higher weight to the indicator that he thinks is most important and evaluate the recognized gesture with the adjusted UX score according to the weight.

As described above, the step of evaluating the gesture (S40) has been described in detail. I will go back to Figure 3 and explain the later steps.

After evaluating the gesture, the evaluation of the recognized gesture is analyzed to select the most suitable gesture for the user (S50). More specifically, the game control simulation system of the VR environment can select a gesture best suited to the user by using the UX score calculated in the step of evaluating the gesture (S40).

Here, when a command corresponding to a recognized gesture is stored only as a single gesture when a gesture best suited to the user is selected, if the UX score of the recognized gesture is less than the set threshold value, a new operation is input to the command corresponding to the recognized gesture You can update the gesture.

In other words, if only the gesture for moving the hand from top to down is stored for the command 'scrolling', if the UX score of the gesture is below the set threshold value, it is determined that the gesture is inappropriate. You can update the gesture for the command "scrolling".

Also, when a command corresponding to a recognized gesture is stored as a plurality of gestures when selecting a gesture best suited to the user, UX scores for a plurality of gestures are respectively calculated to select a gesture having the highest score as a most suitable gesture .

In this way, the VR environment game control simulation system can select a suitable gesture. Personalized gestures can be set for each user by selecting a suitable gesture for the user, and the personalized gesture setting will be described with reference to FIG.

5 is a diagram illustrating various candidate gestures used in game control according to an embodiment of the present invention. Referring to FIG. 5, there are three predefined gestures (a), (b), and (c) for 'driving'.

Here, when the user experiences the VR driving game using each of the three gestures, the user subjective evaluates the motion preference of which of the three gestures is the most suitable for driving. Then, by combining the recognition rate and the muscle fatigue of the gesture, the most suitable gesture can be selected.

For example, the user's subjective preference ratings are UX a = 90%, UX b = 80%, UX c = 90%, recognition rates a = 97%, b = 98%, c = 96%, and muscle fatigue When a = 61.11, b = 74.48, and c = 64.8, UX scores are shown as a = 61.11, b = 74.48, c = 64.8. Therefore, by calculating the UX score in this manner, it can be confirmed that the gesture most suitable for the user is b. In this way, each user can select the best gesture for himself / herself.

Also, when a plurality of gestures having the highest value as a result of UX score calculation for a plurality of gestures are found, a gesture having the least muscle fatigue can be selected as a gesture most suitable for a user.

The reason why the gesture with the lowest muscle fatigue is selected as the most suitable gesture for the user is that, even if the UX score is the same, even if the game is performed for a long time, the user is no longer able to play the game due to muscle fatigue This is to prevent situations that can not be avoided. Also, if the muscle fatigue is high, the user may be less motivated to play the game in the future. Therefore, it is to prevent such situation comprehensively.

As described above, the method of simulating the game control of the VR environment based on the NUI / NUX according to the preferred embodiment of the present invention has been described.

According to the NUI / NUX-based VR environment game control simulation method and system according to the present invention configured as described above, gesture evaluation is performed on a defined gesture, thereby providing a gesture most suitable for a user.

In addition, according to the NUI / NUX-based VR environment game control simulation method and system according to the present invention, by selecting a gesture best suited to a user by measuring gestures, body movements, and electromyograms, have.

The embodiments of the present invention described in the present specification and the configurations shown in the drawings relate to the most preferred embodiments of the present invention and are not intended to encompass all of the technical ideas of the present invention so that various equivalents It should be understood that there may be water and variations. Therefore, it is to be understood that the present invention is not limited to the above-described embodiments, and that various modifications may be made without departing from the spirit and scope of the invention as defined in the appended claims. , Such changes shall be within the scope of the claims set forth in the claims.

100: VR environment game control simulation system
110: Database
120:
130:
140:
150: Analytical Department

Claims (13)

VR environment game control simulation system A method for simulating NUI / NUX based VR environment game control,
Defining a set command as at least one gesture and storing the command in a database;
Receiving an operation of a user and EMG data of the user;
Recognizing a gesture that is mapped to an input operation of the user;
Evaluating the gesture recognized using the input user operation and the EMG data; And
And evaluating for the recognized gesture,
Wherein evaluating the gesture comprises:
Receiving a recognition rate at which the gesture is recognized according to an operation of the user;
Receiving an operation preference of the user for the at least one gesture;
Calculating muscle fatigue of the user using the EMG data;
Calculating a UX (User Experience) score using the recognition rate, the operation preference, and the muscle fatigue; And
And selecting a gesture to be provided to the user based on the UX score. A method for simulating a game control of a VR environment based on NUI / NUX. delete The method according to claim 1,
Wherein the step of evaluating the gesture by the VR environment game control simulation system comprises:
Wherein the UX score is calculated by assigning different weights to the recognition rate, the motion preference, and the muscle fatigue, respectively, in the NUI / NUX-based VR environment. The method according to claim 1,
Wherein the step of selecting a gesture to be provided to the user by the VR environment game control simulation system comprises:
If the UX score of the gesture is less than a predetermined threshold value when the recognized command corresponding to the gesture is stored as only one gesture,
And the gesture is updated by receiving a new operation for a command corresponding to the gesture. The method according to claim 1,
Wherein the step of selecting a gesture to be provided to the user by the VR environment game control simulation system comprises:
When the command corresponding to the recognized gesture is stored as a plurality of gestures, the gesture having the highest score is selected as the gesture most suitable for the user, by calculating UX scores for the plurality of gestures, respectively How to simulate VR environment game controls. 6. The method of claim 5,
Wherein the step of selecting a gesture to be provided to the user comprises:
Wherein the gesture having the lowest muscle fatigue is selected as a gesture best suited to the user when the plurality of gestures having the highest value as a result of calculation of the UX score for the plurality of gestures are selected as the gesture having the highest muscle fatigue, How to simulate a control. A computer-readable recording medium having recorded thereon a program for executing a method according to any one of claims 1 and 6 to a computer. A database for defining and storing a set command as at least one gesture;
A sensor unit for measuring an operation of a user and an EMG signal of the user;
A recognition unit for recognizing a gesture that is mapped to the measured operation of the user;
An evaluator for evaluating the gesture recognized using the measured user operation and the EMG signal; And
Wherein the controller is configured to receive a recognition rate at which a gesture is recognized according to an operation of the user, receive a user's operation preference for at least one gesture, calculate muscle fatigue of a user using EMG data, A system for calculating a UX (User Experience) score using muscle fatigue and an analyzer for selecting a gesture to be provided to a user based on the UX score, and a system for simulating a VR environment game control based on NUI / NUX. delete 9. The method of claim 8,
The evaluating unit,
Wherein the UX score is calculated by assigning different weights to the recognition rate, the operation preference, and the muscle fatigue, respectively, in the NUI / NUX-based VR environment. 9. The method of claim 8,
The analyzing unit,
If the UX score of the gesture is less than a predetermined threshold value when the recognized command corresponding to the gesture is stored as only one gesture,
And the gesture is updated by receiving a new operation for a command corresponding to the gesture. 9. The method of claim 8,
The analyzing unit,
When the command corresponding to the recognized gesture is stored as a plurality of gestures, the gesture having the highest score is selected as the gesture most suitable for the user, by calculating UX scores for the plurality of gestures, respectively VR environment to simulate game control system. 13. The method of claim 12,
The analyzing unit,
Wherein the gesture having the lowest muscle fatigue is selected as a gesture best suited to the user when the plurality of gestures having the highest value as a result of calculation of the UX score for the plurality of gestures are selected as the gesture having the highest muscle fatigue, A system that simulates a control.

Images