KR102502195B1 - Method and system for operating virtual training content using user-defined gesture model - Google Patents

Abstract

It relates to the operation of virtual training content using a user-defined gesture model, and more specifically, recognizes a user-defined hand gesture instead of a controller that must always be held in the hand when using virtual training content to enable customized functions according to the user. It relates to a method and system for operating virtual training contents using a user-defined gesture model that operates training contents.

Description

Method and system for operating virtual training content using user-defined gesture model

The present invention relates to the operation of virtual training content using a user-defined gesture model, and more particularly, enables customized functions according to the user by recognizing a user-defined hand gesture instead of a controller that must always be held in the hand when using virtual training content. It relates to a method and system for operating virtual training contents using a user-defined gesture model that operates virtual training contents by doing so.

As contents through virtual environments increase, educational programs in virtual reality are also being actively developed. In particular, when using virtual reality, it is possible to realistically experience situations that are difficult to reproduce in reality, and since it is possible to provide contents without restrictions even in a remote situation, education can overcome realistic limitations and provide various educational environments and contents. Program applications are rapidly increasing. In particular, in the case of vocational training, the improvement of training effect through increased immersion and safe interaction is emerging as a practical advantage.

Virtual reality technology, realized through the convergence of various IT technologies, is a technology that expands the user's experience area and reduces physical energy and various costs, and has recently attracted a lot of attention, and global companies are competitively promoting technology development. The reason for this is that the replacement life of high-tech equipment that responds to technological changes is short, so there is a limit to continuous financial input. Virtual training systems that can replace it are very effective and are developing high demand, especially high-risk and high-cost industrial training. Demand for experiential and hands-on virtual training content for equipment practice that replaces is increasing, and learning systems that can interact in virtual space are related to realistic interaction of virtual reality systems and natural user interface technology. This is because new-concept interface devices and software technologies are expected to be applied in many industries, such as various industries, education, medical care, defense, and entertainment, and the market is expected to expand.

Therefore, along with the development of virtual reality devices, the development of educational programs in virtual reality is becoming active, and virtual reality training contents that can realistically experience and practice remote situations or situations that are difficult to reproduce realistically overcome realistic limitations. In terms of providing various educational environments and contents, the application fields are rapidly increasing. In particular, when applying virtual reality technology to vocational training, the increase in immersion and the improvement of training effect and learning transfer through interaction are emerging as practical advantages.

However, during virtual reality-based training, the user's wearing of equipment acts as a great burden, which has the disadvantage of lowering the sense of presence and immersion in training.

KR 10-2003-0065620 A

The present invention was invented to solve this problem, and when using virtual training contents, it recognizes a user-defined hand gesture instead of a controller that must be held in the hand at all times to enable customized functions according to the user, thereby increasing the sense of reality and immersion. Its purpose is to provide a method and system for operating virtual training contents using a user-defined gesture model that operates training contents.

In order to achieve the above object, a method of operating virtual training content using a user-defined gesture model according to the present invention includes: (a) receiving a hand gesture image obtained from a camera; (b) generating a pattern consisting of a trajectory and subregions corresponding to the trajectory from the hand gesture image received in step (a), and comparing the pattern with a pre-learned user-defined gesture model; and (c) driving virtual training content according to the recognition result of the hand gesture according to the comparison in step (b), wherein the hand gesture includes a hand shape and a hand motion, and the step (b) The comparison may include (b1) finding the location of the center point of each of the partial regions; (b2) calculating a relative distance using the location value of the central point found in step (b1); and (b3) a Wangner Fischer Algorithm in which a result is recognized as a pattern having the smallest relative distance calculated in step (b2).

In the step (b), the trajectory corresponds to the hand motion.

The comparison is made by the clustering method of the ART algorithm.

If the relative distance value of step (b3) is greater than the threshold value, the result is not recognized.

Another aspect of the present invention for achieving the above object is an apparatus for operating virtual training content using a user-defined gesture model, comprising: at least one processor; and at least one memory for storing computer-executable instructions, wherein the computer-executable instructions stored in the at least one memory are, by the at least one processor, (a) a hand gesture image obtained from a camera. receiving; (b) generating a pattern consisting of a trajectory and subregions corresponding to the trajectory from the hand gesture image received in step (a), and comparing the pattern with a pre-learned user-defined gesture model; and (c) driving virtual training content according to the recognition result of the hand gesture according to the comparison in step (b), wherein the hand gesture includes a hand shape and a hand motion, and the step (b) The comparison may include (b1) finding the location of the center point of each of the partial regions; (b2) calculating a relative distance using the location value of the central point found in step (b1); and (b3) the Wangner Fischer Algorithm, which is a step of recognizing a pattern having the smallest relative distance calculated in step (b2), is executed.

The device for operating virtual training contents using the user-defined gesture model is connected to the camera module.

The device for operating virtual training contents using the user-defined gesture model is connected to the monitor module.

Another aspect of the present invention for achieving this object is a computer program for operating virtual training content using a user-defined gesture model, stored in a non-temporary storage medium, by a processor, (a) obtained from a camera receiving a hand gesture image; (b) generating a pattern consisting of a trajectory and subregions corresponding to the trajectory from the hand gesture image received in step (a), and comparing the pattern with a pre-learned user-defined gesture model; and (c) driving virtual training content according to the recognition result of the hand gesture according to the comparison in step (b), wherein the hand gesture includes a hand shape and a hand motion, and the step (b) The comparison of is, (b1) finding the position of the center point of each of the sub-regions; (b2) calculating a relative distance using the location value of the central point found in step (b1); and (b3) a command for executing the Wangner Fischer Algorithm, which is a step of recognizing a pattern having the smallest relative distance calculated in step (b2).

According to the present invention, there is an effect of reducing the user's burden of wearing equipment during virtual reality-based training.

In addition, it has the effect of enhancing the sense of presence and immersion in virtual reality training.

1 is a diagram showing a learning method of a user-defined gesture model for the operation of virtual training content according to the present invention.
2 and 3 are diagrams for helping understanding of a learning method of a user-defined gesture model for the operation of virtual training content according to FIG. 1;
Figure 4 is a flow chart showing the operation of virtual training content using a user-defined gesture model according to the present invention.
5 to 7 are views for helping understanding of the present invention according to FIG. 4;
8 to 9 are screens showing mouse functions, menus, and buttons used in the operation of virtual training contents using the user-defined gesture model according to FIG. 4;
10 to 14 are diagrams showing interface screens through which a user can directly acquire and learn data for the operation of virtual training contents using the user-defined gesture model of the present invention according to FIG. 4 .
15 is a diagram showing the configuration of an operation system for virtual training contents using a user-defined gesture model according to the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to this, the terms or words used in this specification and claims should not be construed as being limited to the usual or dictionary meaning, and the inventor appropriately uses the concept of the term in order to explain his/her invention in the best way. It should be interpreted as a meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined. Therefore, since the embodiments described in this specification and the configurations shown in the drawings are only one of the most preferred embodiments of the present invention and do not represent all of the technical ideas of the present invention, various alternatives may be used at the time of this application. It should be understood that there may be equivalents and variations.

4 is a flowchart showing a method of operating virtual training contents using a user-defined gesture model according to the present invention.

Prior to the description of the present invention, a user-defined gesture model used in the present invention will be described through FIGS. 1 to 3 as follows.

1 is a diagram showing a method of learning a user-defined gesture model for operating virtual training content according to the present invention, and FIGS. 2 and 3 are a method of learning a user-defined gesture model for operating virtual training content according to FIG. 1 As a diagram to aid understanding, FIG. 2 is a diagram showing area division according to data trajectories to help understand machine learning according to FIG. 1, and FIG. 3 is Levenshitein between the two trajectories to help understand machine learning according to FIG. 1 It is a drawing for explaining distance.

Learning of a user-defined gesture model is formed through machine learning, and at this time, machine learning utilizes the clustering technique of the ART algorithm. ART is basically unsupervised clustering, which is suitable because clustering is performed automatically and quickly whenever data is added, and it does not require repetitive learning for optimization. Since the ART algorithm distinguishes data areas from inside to outside, whenever there is data input, it is possible to immediately determine whether or not new data is included in the previously formed area and whether to expand the existing area.

First, as shown in FIG. 1, learning of a user-defined gesture model receives a hand gesture image acquired from a camera (S300). Here, the hand gesture includes a hand shape and a hand motion.

In step S300, machine learning for forming a gesture model is performed using the received hand gesture image data (S310). Here, in machine learning, as shown in FIG. 2, the red line is the trajectory of continuously input input, and the blue box is the result of automatically dividing the area according to the input, as shown in the right (b) of FIG. At this time, regions outside the blue box are also classified according to the order of proximity, and if there is a distance, the degree of region membership is calculated by reflecting this. Since the region is immediately updated whenever hand gesture image data is received, the update of the regions indicated by the blue box where the trajectory input ends is immediately completed at the same time.

On the other hand, there are times when continuous input repeatedly enters one area. For example, up to 60 hand gesture information is input from the camera per second. This means that 60 hand data came in from the area. If it took 0.1 seconds, it means that the hand data came in 6 times. If all inputs repeatedly received within one area are considered, the speed of hand movement can be considered with the number of hand data samples repeated within the area. That is, even if the hand motion has the same pattern, it is possible to distinguish between a fast movement and a slow movement. However, even if the same user repeats the same gesture, it is difficult to repeat the same gesture at the same speed, so hand data repeated within the region is not redundantly processed. That is, until the area of the trajectory is changed, hand motion information is regarded as the same.

After that, the pattern of the hand motion gesture is determined depending on which sequence the input data has. At this time, the pattern is determined by comparing the similarity between the two sequences. Levenshitein Distance, called Edit Distance, counts how many corrections are needed to compare the difference between two strings and defines it as a distance. For example, the difference between "kitten" and "sitting" is that Levenshitein Distance is 3 because it can be the same letter with a total of 3 modifications, and "saturday" and "sunday" can be the same letter with 3 modifications, so Levenshitein Distance becomes 3. At this time, the comparison of character strings has been exemplified, but in the present invention, similarity (distance) can be calculated in the same way if the sequence of numbers in the partial region is assumed to be a character string and compared. The following [Equation 1] is a method for calculating the similarity value.

[Equation 1]

However, in the case of recursive calculation like the method of [Equation 1], the amount of calculation increases, so the present invention uses the Wagner Fischer Algorithm.

FIG. 3 is a diagram for explaining the Levenshitein Distance between two trajectories to help understanding of machine learning according to FIG. 1 .

Referring to FIG. 3, as shown in the figure on the left (a) of FIG. 3, the trajectories 1 and 2 are has the same difference as

Trajectory 1: (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)

Trajectory 2: (0, 1, 10, 11, 12, 13, 8, 9))

The Levenshitein Distance (Edit Distance) between the two trajectories is 6

Since the relative distance of the subregions of the two trajectories is not considered here, the distance is calculated based on the relative distance between the subregions rather than unconditionally counting the distance as 1 just because the subregions are different. That is, when the two subregions are different, the information on the distance indicated by the green arrow is calculated and reflected as shown in the drawing on the right (b).

Then, the gesture model performed in step S310 is stored (S320). At this time, the storage is divided into categories for each pattern and stored. Through this process, learning of a user-defined gesture model for the operation of virtual training content proceeds.

4 is a flow chart showing the operation of virtual training content using a user-defined gesture model according to the present invention, and FIGS. 5 to 7 are diagrams to aid understanding of the present invention according to FIG. 4 .

First, referring to FIG. 4 , a hand gesture image obtained from a camera is received (S100). At this time, the hand gesture includes a hand shape and a hand motion. Here, when a hand gesture image is input, a trajectory (red) is generated as shown in FIG. 4, and partial regions (blue) corresponding to the trajectory are selected according to the ART algorithm. Then, the indices of the selected subregions are made into a sequence and called C = (c ₁ ,...C _n ).

Subsequently, the hand gesture image received in step S100 is used as an input and compared with a pre-learned user-defined gesture model (S110). At this time, as shown in FIG. 6 , since C may not match a previously learned gesture, C is compared with previously learned gesture patterns P _i = (P ₁ ,...Pm _i ). At this time, when comparing C and _Pi , the Levenshtein Distance lev _s,t is calculated, but the Wangner Fischer Algorithm of FIG. 7 is used.

In the Wangner Fischer Algorithm of FIG. 7, it is considered that s matches P _i and t matches C, and the case where the substitution cost should be 1 is improved to add distance information taking advantage of spatial characteristics.

At this time, the calculation of the Levenshtein Distance by the algorithm of FIG. 7 is as follows.

First, the distance between C _i and P _j is first found, and the position of the center point z of c _i and p _j of each cluster is calculated through the following [Equation 2].

[Equation 2]

,

는 클러스터 k의 웨이트 벡터이다. here,

,

is the weight vector of cluster k.

In addition, the distance D(c _i , p _j ) of c _i and p _j uses the following [Equation 3], which is a distance calculation formula in the ART algorithm, and divides by the length value of the pattern to reduce the difference according to the distance of the pattern. .

[Equation 3]

이다. here,

am.

And if the dimension of the hand gesture input x is M, lev _c,pi has a value between 0 and M.

Finally, the pattern with the smallest value of lev _c,pi calculated by reflecting the value of D(.) is found and selected as the result value according to the following [Equation 4].

[Equation 4]

값보다 크다면, 아무런 손 제스쳐 패턴도 선택하지 않는다. If the threshold set by the value of lev _c,pi

If greater than this value, no hand gesture pattern is selected.

Then, virtual training content is driven according to the recognition result of the hand gesture according to the comparison in step S110 (S120). Here, the virtual training contents are, for example, vocational training contents related to plumbing and LED production, and most of the interfaces can be operated with a mouse, but in the present invention, the virtual training contents are driven by recognized hand gestures. In the present invention, functions shown in FIG. 8 can be substituted in addition to a mouse event for clicking a button, etc., and parts that are interlocked with internal functions of the content shown in FIG. 9 can be implemented.

10 to 14 are views showing interface screens through which a user can directly acquire and learn data for the operation of virtual training contents using the user-defined gesture model of the present invention according to FIG. 4 .

FIG. 10 is a diagram for explaining the basic buttons. As long as the camera sensor is connected through the Start Camera and Stop Camera buttons of FIG. 10, the operation of turning on and off can be confirmed. In addition, if you want to save a gesture learned through the user interface of the present invention, you can save it in a desired location by pressing the Save Model button. The file to be saved is basically saved with the extension .gr (Gesture Recognition Model). And if you want to load a model trained in the past, click the Load Model button to load the model trained in the target directory. Only .gr (Gesture Recognition Model) extensions of trained models can be loaded. When you click the Load Model button, a warning message appears because the existing trained model may disappear. For reference, it is possible to save/load as a file with hand shapes and hand gestures.

11 is a diagram for explaining recognition and registration of a hand gesture. When a sensor is turned on by pressing button (1) in FIG. 10, the hand is recognized as shown in screen (2). At this time, if there is a model that has already been learned, from the moment the hand is recognized on the screen, hand gesture recognition is performed every time sensor data is received, and results are output in steps (3) and (4). At this time, the order of the hands (first hand number 3, second hand number 4) is the order of the first hand seen on the camera image, and gesture recognition is possible for up to two at the same time. Since it is not easy to distinguish the front and back sides of the hand, the left hand and right hand can be distinguished according to the left/right position on the image.

And, if you want to learn additional hand gestures, press button (5) to check the list of previously stored gestures and select the desired gesture. In addition, when you want to register a gesture that did not exist before, you can directly input the gesture name in window (6) and use it. If the gesture to be learned is selected, place the hand shape corresponding to the gesture in front of the sensor and press the Train button (7) to start learning. At this time, the minimum/maximum time required for learning is not fixed. However, even if the shape of the hand is the same, it may look different from the sensor depending on various angles, so during learning, the hand is slightly rotated and moved to learn to be strong against changes. If you think that the learning is over, press the (7) button again to end the learning. If you want to initialize all of the current hand gesture models learned so far, press the Reset button (8).

12 is a diagram for explaining recognition and registration of hand gestures. In the case of hand gestures, since a pattern must be input, it is not possible to learn every frame in front of a sensor, but one gesture pattern input must be performed.

First, after turning on the sensor with button (1), as in (2), when the hand appears on the screen, hand motion information is automatically recorded. When the gesture is recognized, the result is output in window (3). Currently, motion recognition is possible only for the first hand.

As explained in the previous section, the timing at which the gesture is recognized is output when the hand is still for 1 second. You can check whether the hand is moving or not, in the Status window (4), and when the hand is moving, moving is displayed, and when it is determined that the hand is stopped, ready is displayed.

If you want to learn additional hand gestures, press (5) to select a gesture from the list, or if it is a non-existent gesture, directly enter the gesture name in (6). Then place your hand on the sensor and press button (7) to start the gesture. When one gesture is finished, press the button (7) again to confirm that the gesture is finished. If you want to learn additional gestures, you can repeat this process.

If you want to delete one of the learned gestures, press (5) to select the corresponding gesture from the gesture list, and press the Delete button (8) to delete it. If you want to initialize all hand gesture information, press the Reset button (9).

FIG. 13 is a diagram for explaining function connection. If the hand shape and hand gestures have been registered previously, an operation to connect the function for interlocking with actual content must be performed. 13 is a close-up picture of the lower left corner of the UI screen.

When button (1) is pressed, a list of functions that can be interlocked by the user is basically provided. As described in the previous section, functions related to the mouse and specific menus that can be used within the content are displayed and can be selected. If you select a function to link in (1), you can check the currently connected hand shape and hand motion in (6) and (9). Since the hand shape and hand motion were learned differently, it was configured to be added separately here as well. If there is a hand shape or motion that you want to add here, press (2) to select a hand shape that is not yet connected to any function, and if you want to add a hand gesture, press (3) to open the list and select it. . In the case of the hand shape, after selecting a gesture from the list (2), you can press the arrow button (4) to connect to the function selected in (1). If there is something you want to remove, the hand you want to remove from the list (6) After selecting the shape, press the arrow button (5). Similarly, in the case of hand gestures, after selecting a gesture from the list (3), you can press the arrow button (7) to connect to the function selected in (1), and if there is something you want to remove, select it from the list (9) ) by pressing the arrow button.

15 is a diagram showing the configuration of an operating system for virtual training content using a user-defined gesture model according to the present invention. Referring to FIG. 15, the components of the virtual training content operating device 20 will be briefly summarized and described. do.

As shown in FIG. 15, the virtual training content operating device 20 includes a processor 110, a non-volatile storage unit 120 for storing programs and data, a volatile memory 130 for storing running programs, and other devices. It consists of a communication unit 140 for performing communication, a bus that is an internal communication path between these devices, and the like. Programs that are being executed may include device drivers, operating systems, and various applications. Although not shown, it includes a power supply unit.

The virtual training content operation device 20 receives hand gesture image data from the camera 30 by the image data reception driver 150 and inputs it to a user-defined gesture model in the virtual training content operation application, and the user The virtual training content is operated by the output of the defined gesture model and outputted through the monitor.

In addition, the communication unit 140 performs a role of transmitting and receiving data with the user-defined gesture model learning device 10 as described with reference to FIG. The received image data is transmitted to the user-defined gesture model learning device, and the learned user-defined gesture model is received from the user-defined gesture model learning device (10).

As described above, although the present invention has been described by the limited embodiments and drawings, the present invention is not limited thereto, and the technical spirit of the present invention and the following by those skilled in the art to which the present invention belongs Of course, various modifications and variations are possible within the scope of equivalents of the claims to be described.

10: User-defined gesture model learning device
20: virtual training content operation device
30: camera
40: monitor
110: monitor
120: storage unit
130: memory
140: communication department
150: image data reception driver
160: UX/UI output driver

Claims (11)

As a method of operating virtual training content using a user-defined gesture model,
(a) receiving a hand gesture image obtained from a camera;
(b) generating a pattern consisting of a trajectory and subregions corresponding to the trajectory from the hand gesture image received in step (a), and comparing the pattern with a pre-learned user-defined gesture model; and
(c) driving virtual training content according to the recognition result of the hand gesture according to the comparison in step (b)
Including,
The hand gesture includes a hand shape and a hand motion,
The comparison of step (b) above,
(b1) finding the location of the center point of each of the partial regions;
(b2) calculating a relative distance using the location value of the central point found in step (b1); and,
(b3) Recognizing the result as a pattern with the smallest value of the relative distance calculated in step (b2)
which includes the Wagner Fischer Algorithm
Operation method of virtual training contents using user-defined gesture model. The method of claim 1,
In the step (b), the trajectory corresponds to the hand motion
Method of operating virtual training content using a user-defined gesture model, characterized in that. The method of claim 1,
The comparison is made by the clustering method of the ART algorithm
Method of operating virtual training content using a user-defined gesture model, characterized in that. delete delete The method of claim 1,
If the relative distance value of step (b3) is greater than the threshold value, the result is not recognized
Method of operating virtual training content using a user-defined gesture, characterized in that. A device for operating virtual training content using a user-defined gesture model,
at least one processor; and
including at least one memory for storing computer-executable instructions;
The computer-executable instructions stored in the at least one memory are, by the at least one processor,
(a) receiving a hand gesture image obtained from a camera;
(b) generating a pattern consisting of a trajectory and subregions corresponding to the trajectory from the hand gesture image received in step (a), and comparing the pattern with a pre-learned user-defined gesture model; and
(c) driving virtual training content according to the recognition result of the hand gesture according to the comparison in step (b)
Including,
The hand gesture includes a hand shape and a hand motion,
The comparison of step (b) above,
(b1) finding the location of the center point of each of the partial regions;
(b2) calculating a relative distance using the location value of the central point found in step (b1); and,
(b3) Recognizing the result as a pattern with the smallest value of the relative distance calculated in step (b2)
which allows the Wagner Fischer Algorithm to be executed.
Operation device of virtual training contents using user-defined gesture model. The method of claim 7,
The device for operating virtual training content using a user-defined gesture model is connected to a camera module. in claim 7
The device for operating virtual training content using a user-defined gesture model is characterized in that connected to the monitor module. As a computer program for operating virtual training content using a user-defined gesture model,
It is stored in a non-temporary storage medium, and by the processor,
(a) receiving a hand gesture image obtained from a camera;
(b) generating a pattern consisting of a trajectory and subregions corresponding to the trajectory from the hand gesture image received in step (a), and comparing the pattern with a pre-learned user-defined gesture model; and
(c) driving virtual training content according to the recognition result of the hand gesture according to the comparison in step (b)
Including,
The hand gesture includes a hand shape and a hand motion.
The comparison of step (b) above,
(b1) finding the location of the center point of each of the partial regions;
(b2) calculating a relative distance using the location value of the central point found in step (b1); and,
(b3) Recognizing the result as a pattern with the smallest value of the relative distance calculated in step (b2)
contains instructions that cause the Wagner Fischer Algorithm to be executed.
A computer program that operates virtual training contents using a user-defined gesture model. in claim 1
The location of the center point in step (b1) is

is calculated by
here,

is,

Is the weight vector of the subregion k, and the relative distance calculation in step (b2) is,

is calculated by
here,

is,
Recognizing the result of step (b3),

recognized by
Method of operating virtual training content using a user-defined gesture model, characterized in that.

Images