KR20210155600A - Method and system for operating virtual training content using user-defined gesture model - Google Patents

Abstract

The present invention relates to an operation of virtual training content using a user-defined gesture model and, more specifically, to an operating method of virtual training content using a user-defined gesture model and a system thereof which recognize user-defined hand gestures instead of a controller, which should always be held in a hand when using virtual training content, to function as a customized function according to a user, thereby operating the virtual training content.

Description

Method and system for operating virtual training content using user-defined gesture model

The present invention relates to the operation of virtual training content using a user-defined gesture model, and more particularly, by recognizing a user-defined hand gesture instead of a controller that should always be held in the hand when using the virtual training content, a customized function according to the user is possible It relates to a method and system for operating virtual training content using a user-defined gesture model that operates the virtual training content.

As content through virtual environments increases, educational programs in virtual reality are being actively developed. In particular, if virtual reality is used, it is possible to realistically experience situations that are difficult to reproduce in reality, and because content can be provided without restrictions even in remote situations, education is provided in terms of providing various educational environments and contents by overcoming realistic limitations. Program applications are growing rapidly. In particular, in the case of vocational training, improvement in training effectiveness through increased immersion and safe interaction is emerging as a practical advantage.

Virtual reality technology, implemented through the convergence of various IT technologies, has recently attracted a lot of attention as a technology that expands the user's experience area and reduces physical energy and various costs, and global companies are competitively promoting technology development. The reason is that the replacement life of high-tech equipment that responds to technological change is short, and there is a limit to continuous financial input. There is an increasing demand for experiential and practical virtual training content for equipment practice that replaces the This is because the new concept of interface device and software technology is expected to be applied in many industries, such as various industries, education, medical care, defense, and entertainment in the future, and thus the market is expected to expand.

Therefore, along with the development of virtual reality devices, the development of educational programs in virtual reality is becoming more active. In terms of providing various educational environments and contents, the field of application is rapidly increasing. In particular, when virtual reality technology is applied to vocational training, an increase in immersion and an improvement in training effect and learning transfer through interaction are emerging as practical advantages.

However, the wear of the user's equipment during virtual reality-based training is a great burden, and this has the disadvantage of lowering the sense of reality and immersion in training.

KR 10-2003-0065620 A

The present invention was devised to solve such a problem, and when using virtual training content, it recognizes a user-defined hand gesture instead of a controller that should always be held in the hand and enables a customized function according to the user to increase the sense of presence and immersion in virtual reality. An object of the present invention is to provide a method and system for operating virtual training content using a user-defined gesture model for operating the training content.

In order to achieve the above object, there is provided a method of operating virtual training content using a user-defined gesture model according to the present invention, comprising the steps of: (a) receiving a hand gesture image obtained from a camera; (b) comparing the hand gesture image received in step (a) with a previously learned user-defined gesture model; and (c) driving the virtual training content according to the recognition result of the hand gesture according to the comparison of step (b).

The hand gesture includes a hand gesture and a hand gesture.

The comparison is made by the clustering method of the ART algorithm.

The input hand gesture image is divided into a trajectory and a pattern consisting of clusters corresponding to the trajectory.

The comparison of step (b) includes the steps of: (b1) finding the position of the center point of each cluster; (b2) calculating a relative distance using the position value of the central point found in step (b1); and (b3) recognizing the result as a pattern having the smallest relative distance calculated in step (b2).

If the relative distance value in step (b3) is greater than the threshold, the result is not recognized.

Another aspect of the present invention for achieving the above object is an apparatus for operating virtual training content using a user-defined gesture model, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions stored in the at least one memory are configured by the at least one processor: (a) a hand gesture image obtained from a camera receiving; (b) comparing the hand gesture image received in step (a) with a previously learned user-defined gesture model; And (c) the step of driving the virtual training content according to the recognition result of the hand gesture according to the comparison of step (b) is executed.

The virtual training content operation device using the user-defined gesture model is connected to the camera module.

The virtual training content operation device using the user-defined gesture model is connected to the monitor module.

Another aspect of the present invention for achieving the above object is a computer program for operating virtual training content using a user-defined gesture model, stored in a non-transitory storage medium, by the processor, (a) obtained from the camera receiving a hand gesture image; (b) comparing the hand gesture image received in step (a) with a previously learned user-defined gesture model; and (c) a command for executing the step of driving the virtual training content according to the recognition result of the hand gesture according to the comparison of step (b).

According to the present invention, there is an effect of reducing the burden of the user wearing equipment during virtual reality-based training.

In addition, it has the effect of increasing the sense of reality and immersion in virtual reality training.

1 is a view showing a learning method of a user-defined gesture model for the operation of virtual training content according to the present invention.
2 to 3 are diagrams to help understand the learning method of the user-defined gesture model for the operation of the virtual training content according to FIG.
4 is a flowchart showing the operation of virtual training content using a user-defined gesture model according to the present invention.
5 to 7 are diagrams for helping understanding of the present invention according to FIG.
8 to 9 are screens showing mouse functions, menus and buttons utilized in the operation of virtual training contents using the user-defined gesture model according to FIG.
10 to 14 are diagrams showing an interface screen through which a user can directly acquire and learn data for the operation of virtual training content using the user-defined gesture model of the present invention according to FIG. 4;
15 is a diagram showing the configuration of an operation system of virtual training content using a user-defined gesture model according to the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to this, the terms or words used in the present specification and claims should not be construed as being limited to conventional or dictionary meanings, and the inventor should properly understand the concept of the term in order to best describe his invention. Based on the principle that it can be defined, it should be interpreted as meaning and concept consistent with the technical idea of the present invention. Accordingly, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiment of the present invention and do not represent all the technical spirit of the present invention, so at the time of the present application, various It should be understood that there may be equivalents and variations.

4 is a flowchart illustrating a method of operating virtual training content using a user-defined gesture model according to the present invention.

First, the user-defined gesture model used in the present invention will be described with reference to FIGS. 1 to 3 before the present invention is described.

1 is a diagram showing a learning method of a user-defined gesture model for operation of virtual training content according to the present invention, and FIGS. 2 to 3 are a learning method of a user-defined gesture model for operation of virtual training content according to FIG. 1 As a diagram to aid understanding, FIG. 2 is a diagram showing region division according to data trajectories to help understand machine learning according to FIG. 1 , and FIG. 3 is a Levenshitein between two trajectories to help understand machine learning according to FIG. 1 . It is a diagram for explaining Distance.

User-defined gesture model learning is formed through machine learning, where machine learning utilizes the clustering technique of the ART algorithm. This is suitable because ART is basically unsupervised clustering, where clustering is performed automatically and quickly whenever data is added, and iterative learning is not required for optimization. Since the ART algorithm divides the data area into and out, whenever there is data input, it is possible to immediately determine whether or not new data is included in the previously formed area and whether the existing area is expanded.

First, as shown in FIG. 1 , in learning the user-defined gesture model, a hand gesture image obtained from a camera is received ( S300 ). Here, the hand gesture includes a hand gesture and a hand gesture.

And in step S300, machine learning for forming a gesture model is performed using the received hand gesture image data (S310). Here, in machine learning, as shown in FIG. 2 , as in the right (b) of FIG. 2 , the red line is the trajectory of the continuously incoming input, and the blue box is the result of automatically dividing the area according to the input. At this time, the areas outside the blue box are also divided according to the order of proximity, and if there is a distance, the degree of belonging to the area is calculated by reflecting this. Since the area is updated every time hand gesture image data is received, the area marked with a blue box where the trajectory input ends is also immediately completed at the same time.

On the other hand, there are times when continuous input repeatedly enters one area. For example, hand gesture information is inputted from the camera up to 60 times per second. This means that 60 hand data came in from the area. If it took 0.1 seconds, it means that 6 hand data have been entered. In this way, if all repetitive input in one area is considered, the speed of hand movement can be considered with the number of repeated hand data samples in the area. That is, even with the same pattern of hand gestures, it is possible to distinguish between a fast movement and a slow movement. However, even if the same user repeats the same gesture, there is a difficulty in repeating the same gesture at the same speed. That is, until the area of the trajectory is changed, the information of the hand gesture is regarded as the same.

Thereafter, the pattern of the hand gesture gesture is determined based on which sequence the input data has. In this case, the pattern is determined by comparing the similarity between the two sequences. The Levenshitein Distance, called Edit Distance, is defined as the distance by counting how many corrections are needed to compare the difference between two strings. For example, the difference between "kitten" and "sitting" is that the Levenshitein Distance becomes 3 because the same letter can be modified 3 times in total, and "saturday" and "sunday" can become the same letter with 3 modifications, so Levenshitein Distance becomes 3. In this case, the comparison of character strings is an example, but in the present invention, if the number sequence of the cluster is assumed as a character string and compared, the similarity (distance) can be calculated in the same way. The following [Equation 1] is a method of calculating the similarity value.

[Equation 1]

However, in the case of recurring calculation as in the method of [Equation 1], the amount of calculation increases, so Wagner Fischer Algorithm is used in the present invention.

FIG. 3 is a diagram for explaining the Levenshitein Distance between two trajectories to help understanding machine learning according to FIG. 1 .

Referring to FIG. 3, as shown in the figure on the left (a) of FIG. 3, trajectories 1 and 2 are as follows according to the cluster (blue box) that is updated immediately when two different patterns (red trajectories 1 and 2) are input. have the same difference.

Trajectory 1: (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)

Trajectory 2: (0, 1, 10, 11, 12, 13, 8, 9))

The Levenshitein Distance (Edit Distance) between the two trajectories is 6

Here, since the relative distance of the clusters among the two trajectories is not considered, the distance is calculated based on the relative distance between the clusters rather than counting the distance as 1 just because the clusters are different. That is, when the two clusters are different, the information on the distance indicated by the green arrow is calculated and reflected as shown in the figure on the right (b).

Then, the gesture model performed in step S310 is stored (S320), where the storage is divided into categories for each pattern and stored, and through this process, learning of a user-defined gesture model for the operation of virtual training content is carried out.

4 is a flowchart illustrating the operation of virtual training content using a user-defined gesture model according to the present invention, and FIGS. 5 to 7 are diagrams to help the understanding of the present invention according to FIG. 4 .

First, referring to FIG. 4 , an image of a hand gesture obtained from a camera is received ( S100 ). In this case, the hand gesture includes a hand gesture and a hand gesture. Here, when a hand gesture image is input, a trajectory (red) is generated as shown in FIG. 4, and clusters (blue) matching the trajectory are selected according to the ART algorithm. And, by making the indices of the selected clusters into a sequence, it is called C = (c ₁ ,...C _n ).

Next, the hand gesture image received in step S100 is input and compared with a previously learned user-defined gesture model (S110). At this time, as shown in FIG. 6 , since C may not match the previously learned gesture, C is _{compared with previously learned gesture patterns P i} = (P ₁ ,...Pm _i ). At this time, _{when comparing C and P i} , the Levenshtein Distance lev _s,t is calculated, but the Wangner Fischer Algorithm of FIG. 7 is used.

In the Wangner Fischer Algorithm of FIG. 7, it is seen that s matches P _i and t matches C, and the case where the substitution cost should be 1 is improved to take advantage of spatial characteristics to add distance information.

At this time, the calculation of the Levenshtein Distance by the algorithm of FIG. 7 is as follows.

First finds the location of the C _i and P _j distance first, the center point of each cluster of c _i and p _j in the z, which is computed by the following [formula 2].

[Equation 2]

,

는 클러스터 k의 웨이트 벡터이다. here,

,

is the weight vector of cluster k.

And c _i to the distance p _j D (c _i, p _j) gives divided by the length value of the pattern to take advantage of the [formula 3], and the distance calculation in the ART algorithm, and reduce the difference in the distance of the pattern .

[Equation 3]

이다. here,

to be.

And if the dimension of hand gesture input x is M, lev _c,pi has a value between 0 and M.

_{Finally, find the pattern with the smallest lev c,pi} value calculated by reflecting the D(.) value and select it as the result value according to the following [Equation 4].

[Equation 4]

값보다 크다면, 아무런 손 제스쳐 패턴도 선택하지 않는다. If the lev _{c, pi} value is a set threshold,

If greater than the value, no hand gesture pattern is selected.

Afterwards, the virtual training content is driven according to the recognition result of the hand gesture according to the comparison in step S110 (S120). Here, the virtual training content is, for example, vocational training content related to piping wiring and LED production. Most interfaces can be operated with a mouse, etc., but in the present invention, the virtual training content is driven by a recognized hand gesture. In the present invention, in addition to the mouse event of clicking a button or the like, the functions shown in FIG. 8 can be substituted, and parts linked to the internal functions of the content shown in FIG. 9 can be implemented.

10 to 14 are diagrams showing an interface screen through which a user can directly acquire and learn data for the operation of virtual training contents using the user-defined gesture model of the present invention according to FIG. 4 .

FIG. 10 is a diagram for explaining the basic button, and if the camera sensor is connected through the Start Camera and Stop Camera buttons of FIG. 10, the on/off operation can be checked. In addition, if you want to save the gesture learned through the user interface of the present invention, you can save it in a desired location by pressing the Save Model button. The file to be saved is basically saved with the extension .gr (Gesture Recognition Model). And if you want to load the model trained in the past, click the Load Model button to load the model trained in the target directory. The extension of the trained model can only be called .gr (Gesture Recognition Model). If you press the Load Model button, a warning message appears because the existing training model may disappear. For reference, it is possible to save/load hand gestures and hand gestures together as a file.

11 is a diagram for explaining the recognition and registration of a hand gesture. When the sensor is turned on by pressing the button (1) of FIG. 10, the hand is recognized as in the screen (2). At this time, if there is a model that has already been trained, from the moment the hand is recognized on the screen, hand gesture recognition is performed every time sensor data is received, and the results are output in steps (3) and (4). In this case, the hand order (No. 3 First Hand, No. 4 Second Hand) is the order of the first hand seen on the camera image, and up to two gestures can be recognized simultaneously. Since it is not easy to distinguish the front and back sides of the hand, the left hand and the right hand can be distinguished according to the left/right positions on the image.

And if you want to learn the hand gesture additionally, you can check the list of gestures stored in the past by pressing the button (5) and select the desired gesture. In addition, if you want to register a gesture that has not existed before, you can use it by directly entering the gesture name in the window (6). If you have selected a gesture to learn, place the hand corresponding to the gesture in front of the sensor and press the Train button (7) to start learning. At this time, the minimum/maximum time required for learning is not determined. However, even the same hand shape can be seen differently by the sensor depending on various angles, so while learning, the hand is rotated slightly while learning to be strong against change. If you think the learning is over, press button (7) again to end the learning. If you want to initialize the entire current hand gesture model learned so far, press the Reset button (8).

12 is a diagram for explaining the recognition and registration of hand gestures. In the case of hand gestures, since a pattern must be input, learning cannot be performed every frame in front of the sensor, but one gesture pattern input must be performed.

First, after turning on the sensor with button (1), as in (2), when the hand appears on the screen, the information of the hand motion starts to be recorded automatically. When the gesture is recognized, the result is output in the window (3). Currently, motion recognition is available only for the first hand.

The timing at which the gesture is recognized is as described in the previous section, if the hand is still for 1 second, the result is output. Whether the hand is moving or not can be checked in the status window (4). When the hand is moving, it is moving, and when it is determined that the hand is stopped, it is displayed as ready.

If you want to learn hand gestures additionally, press (5) to select a gesture from the list, or directly enter the gesture name in (6) if there is no existing gesture. Then, place your hand on the sensor and press button (7) to start the gesture. When one gesture is finished, press button (7) again to confirm that the gesture is finished. If you want to learn additional gestures, repeat this process.

If you want to delete one of the learned gestures, press (5) to select the gesture from the gesture list, and press (8) Delete button to delete it. If you want to initialize all hand gesture information, press (9) Reset button.

FIG. 13 is a diagram for explaining functional connection. When the registration of hand shapes and hand gestures is completed, it is necessary to connect the functions with the actual content. 13 is a close-up photo of the lower left corner of the UI screen earlier.

When button (1) is pressed, a list of functions that can be linked by the user is provided by default. As explained in the previous section, mouse-related functions and specific menus that can be used within the content are displayed and can be selected. If you select a function to be linked in (1), you can check the hand shape and hand gestures currently connected in (6) and (9). Since the shape of the hand and the movement of the hand were learned differently, it was configured to be added separately here as well. If there is a hand shape or hand gesture that you want to add here, press (2) to select a hand that is not connected to any function yet. If you want to add a hand gesture, press (3) to open the list and select. . In the case of a hand, after selecting a gesture from the list (2), press the arrow button (4) to connect to the function selected in (1). If there is something you want to remove, the hand you want to remove from the list (6) After selecting a shape, press the (5) arrow button. Similarly, in the case of hand gestures, after selecting a gesture from the list (3), press the arrow button (7) to connect to the function selected in (1). ) by pressing the arrow button.

15 is a diagram showing the configuration of a system for operating virtual training content using a user-defined gesture model according to the present invention. do.

15, the virtual training content operation device 20 includes a processor 110, a non-volatile storage unit 120 for storing programs and data, a volatile memory 130 for storing running programs, and other devices. It consists of a communication unit 140 for performing communication, a bus serving as an internal communication path between these devices, and the like. The running program may include a device driver, an operating system, and various applications. Although not shown, it includes a power supply unit.

The virtual training content operation device 20 receives the hand gesture image data from the camera 30 by the image data reception driver 150 and inputs it to a user-defined gesture model in the virtual training content operation application, and the user The virtual training content is operated by the output of the definition gesture model, and it is output through the monitor.

In addition, the communication unit 140 performs a data transmission/reception role with the user-defined gesture model learning apparatus 10 as described with reference to FIG. The received image data is transmitted to the user-defined gesture model learning apparatus, and the learned user-defined gesture model is received from the user-defined gesture model learning apparatus (10).

As described above, although the present invention has been described with reference to limited embodiments and drawings, the present invention is not limited thereto, and the technical idea of the present invention and the following by those skilled in the art to which the present invention pertains. Of course, various modifications and variations are possible within the scope of equivalents of the claims to be described.

10: Custom Gesture Model Training Apparatus
20: virtual training content operation device
30: camera
40: monitor
110: monitor
120: storage
130: memory
140: communication department
150: image data receiving driver
160: UX/UI output driver

Claims (10)

As an operation method of virtual training content using a user-defined gesture model,
(a) receiving a hand gesture image obtained from a camera;
(b) comparing the hand gesture image received in step (a) with a previously learned user-defined gesture model; and
(c) driving the virtual training content according to the recognition result of the hand gesture according to the comparison of step (b);
Operation method of virtual training content using a user-defined gesture model comprising a. The method according to claim 1,
The hand gesture is a hand gesture and a hand gesture
Operation method of virtual training content using a user-defined gesture model, characterized in that it comprises a. The method according to claim 1,
The comparison is made in the clustering method of the ART algorithm.
Operation method of virtual training content using a user-defined gesture model, characterized in that. The method according to claim 1
The input hand gesture image is divided into a trajectory and a pattern consisting of clusters corresponding to the trajectory.
Operation method of virtual training content using a user-defined gesture, characterized in that. The method according to claim 1,
The comparison of step (b) is,
(b1) finding the position of the center point of each cluster;
(b2) calculating a relative distance using the position value of the central point found in step (b1); and
(b3) step of recognizing the result as a pattern having the smallest relative distance calculated in step (b2)
Operation method of virtual training content using a user-defined gesture, characterized in that it comprises a. 6. The method of claim 5,
If the relative distance value of step (b3) is greater than the threshold, the result is not recognized
Operation method of virtual training content using a user-defined gesture, characterized in that. As a device for operating virtual training content using a user-defined gesture model,
at least one processor; and
at least one memory for storing computer-executable instructions;
The computer-executable instructions stored in the at least one memory are executed by the at least one processor,
(a) receiving a hand gesture image obtained from a camera;
(b) comparing the hand gesture image received in step (a) with a previously learned user-defined gesture model; and
(c) driving the virtual training content according to the recognition result of the hand gesture according to the comparison of step (b);
Operation device of virtual training content using a user-defined gesture model that allows to be executed. 8. The method of claim 7,
The device for operating virtual training contents using the user-defined gesture model is an operating device for virtual training contents using a user-defined gesture model, characterized in that it is connected to a camera module. 8. The method of claim 7
The device for operating virtual training contents using the user-defined gesture model is an operating device for virtual training contents using a user-defined gesture model, characterized in that it is connected to a monitor module. As a computer program for operating virtual training content using a user-defined gesture model,
It is stored in a non-transitory storage medium, and by the processor,
(a) receiving a hand gesture image obtained from a camera;
(b) comparing the hand gesture image received in step (a) with a previously learned user-defined gesture model; and
(c) driving the virtual training content according to the recognition result of the hand gesture according to the comparison of step (b)
A computer program for operating virtual training content using a user-defined gesture model including instructions to be executed.

Images