KR100733964B1 - A game Apparatus and method using motion capture and voice recognition - Google Patents

Abstract

1. TECHNICAL FIELD OF THE INVENTION

The present invention relates to a game device and method using motion recognition and voice recognition.

2. The technical problem to be solved by the invention

According to the present invention, after extracting an object from a binocular image having different camera information, depth information is calculated using binocular parallax, and the position of the object is calculated in a three-dimensional space using the calculated depth information. (Inverse Kinematics) After classifying the phrases of sentences using the motion recognition result generated by recognizing the entire motion data and the feature point extracted from the voice (sentence) and the motion recognition result received from the motion recognition means It is an object of the present invention to provide a game apparatus and method using motion recognition and voice recognition for controlling the movement of a character and the corresponding sound after recognizing a command by combining the voice recognition results.

3. Summary of Solution to Invention

The present invention provides a game device using motion recognition and voice recognition, comprising: storage means for storing user-specific command data; After extracting an object from a binocular image having different camera information, depth information is calculated using binocular disparity, the position of the object is calculated in three-dimensional space using the calculated depth information, and inverse kinematics. Motion recognition means for recognizing motion by generating full motion data through an algorithm; Isolated word recognition means for recognizing a phrase after dividing a phrase of speech using a feature point extracted from a speech composed of at least one phrase and a motion recognition result received from the motion recognition means; Command recognition means for recognizing the command by combining the recognition result in the isolated word recognition means and the recognition result in the motion recognition means; And central processing means for controlling the operation of the character and a corresponding sound according to the recognition result (command data) in the command recognition means, and storing the recognition result in the storage means.

4. Important uses of the invention

The present invention is used in game devices and the like.

Motion recognition, speech recognition, combination, virtual space, backward motion algorithm, isolated word recognition, game device

Description

Game apparatus and method using motion recognition and voice recognition {A game Apparatus and method using motion capture and voice recognition}

1 is a configuration diagram of a game device using motion recognition and voice recognition according to the present invention;

2 is a diagram illustrating an embodiment of a correction object method among camera correction methods used in the present invention;

3 is a diagram illustrating an embodiment of an autonomous correction method among camera correction methods used in the present invention;

Figure 4 is an embodiment implementation of a game device using motion recognition and speech recognition according to the present invention,

5 is a diagram illustrating an embodiment of a boundary classification process between phrases using motion information according to the present invention;

6 is a diagram illustrating an embodiment of a game method using a network according to the present invention;

7 is a diagram illustrating an embodiment of a backward operation algorithm used in the present invention;

8 is a flowchart illustrating an embodiment of a game method using motion recognition and voice recognition according to the present invention.

* Explanation of symbols for the main parts of the drawings

10: storage unit 20: input unit

30: motion recognition unit 40: speech recognition unit

50: command recognition unit 60: output unit

70: internet connection 80: central processing unit

The present invention relates to a game device and method using motion recognition and voice recognition, and more particularly, to distinguish between the various motions and direction, intensity, other mode changes, etc. by combining the advantages and disadvantages of motion recognition and voice recognition through video The present invention relates to a game apparatus and method using motion recognition and voice recognition to provide a more dynamic and interesting game than a game based only on conventional motion recognition or voice recognition.

The basic principle of human or animal movement is rotation of the joints. That is, the movement of the joint is generally limited, and if this is defined in the joint set in the model, when the lower part of the model is moved, the upper joint moves accordingly. Inverse Kinematics is an algorithm that automatically calculates the movement of the upper joint within a limited range according to the movement of the lower joint, rather than the movement of the upper joint.

When a person raises or moves an arm, the nerve usually moves in a willingness to place the hand in a position, and rarely shows a willingness to grab that object by turning the shoulder blade a few degrees. Therefore, moving forward is not an intuitive way of working for an animator. The method that compensates for these shortcomings is the backward operation algorithm.

HMM (Hidden Markov Model) is a method of modeling the basic unit (phoneme) for speech recognition. That is, most of the Korean voice recognition engine companies use the phoneme coming into the voice recognition engine and the phoneme that has a database in the voice recognition engine to make words and sentences.

HMM is widely used in speech recognition as a double probability processing method that estimates unobservable processes through other observable processes. Therefore, using the HMM method in speech recognition refers to constructing a speech recognition system by modeling a minimum unit (phoneme) of speech recognition. Therefore, the advantage of HMM is that recognition rate is higher than other methods. However, if the current HMM training sample is not sufficient, there are problems such as accurate model estimation and knowledge of phoneme context.

Meanwhile, motion recognition and voice recognition have been studied for a long time in their respective fields.

First, motion recognition is based on the ultrasonic method using the sensor and the receiver to generate ultrasonic waves according to the equipment used, the prosthetic method using a potentiometer and slider to measure the joint movement of the user, a magnetic field sensor is attached to each joint area of the user It is classified into a magnetic method for measuring movement by calculating the change of the spatial difference and an optical method using an image obtained from a charge coupled device (CCD) camera.

The dual optical method has the advantage that a relatively accurate result can be obtained through simple equipment. Until now, a lot of researches have been conducted to attach markers to the body, to see the movement of these markers, and to infer joint movement. This method is cumbersome to attach a marker to the body every time it is used, making it unsuitable for practical game content.

Therefore, as an approach for recognizing motion in an optical manner without markers, there is a method of inferring posture by extracting silhouette information that has been studied at Oxford University, and this method has a weak performance in overlapping motions of body parts. .

In general, the camera calibration process looks for external variables such as the position of the camera looking at an object, angles such as angles, internal variables such as the camera's inherent skew, aspect ratio, and focal length. This is called.

Camera correction can be largely classified into a method using a rectangular parallelepiped object and an autonomous correction method obtained by tracking a position of a natural feature automatically detected from an image.

The method of using a correcting object is a method of using a correcting object as shown in FIG. 2, which is widely used until now, and uses a correcting object having a three-dimensional cuboid shape or a plane correcting object that knows coordinates of points on a plane. That is, the correction object is photographed to calculate and correct a camera projection matrix from a known geometric relationship of the object.

The autonomous correction method can calculate the position of these feature points in 3D space in metric space if they know the correspondence of the feature points observed in the images from different angles, and at the same time, automatically correct the camera. (Auto-Calibration) follows the mathematical logic that the camera can see the projection matrix P = KR [I | -C] looking at the object, and "Richard Hartley" and "Marc Pollefeys" are each organized in their own way. I've done it.

Despite the wide range of applications, these autonomous correction methods are still very vulnerable to the noise of feature points, and the existing methods cannot realize real time.

FIG. 3 shows the movement paths and angles of the cameras obtained after the process of the feature points extracted from the boujou 3.0, which is known to show the best performance in the autonomous correction method, and after the correction process. In this way, once the camera is calibrated, the 3D spatial coordinates of the corresponding feature points can be calculated by a "Triangulation" method including a mid-point algorithm.

Next, speech recognition extracts feature points through pre-processing when the first user inputs them, sees them as HMM models, and compares them with existing Bayesian theorem which defines conditional probabilities. Likelihood) and get the final result.

At this time, it is divided into a word unit isolated word recognizer, a phoneme unit isolated word recognizer, and a continuous speech recognizer according to the HMM modeling unit. The word-based isolated word recognizer is a structure suitable for small-scale recognizers that recognize isolated digits or a target vocabulary.

Since the phoneme-based isolated word recognizer divides the phonemes into HMM units, arbitrary words can be formed by combining phoneme models, so that a variable vocabulary recognizer whose recognition target vocabulary changes according to circumstances can be implemented. Even if the amount of voice data is small, the models having the same phonetic environment can share parameters, so that the amount of voice data is at least advantageous.

The problem of continuous speech recognition is to find the most probable word sequence given the observation sequence, acoustic model, pronunciation model, and language model. Since the word boundary information is not given, every word can be started every frame. The search space is larger.

In addition, due to the reflection of the speaker's voice reflected in the surrounding environment, the difficulty of finding the boundary point becomes greater, and much research is still underway, but a system that has high performance enough for practical use has not been devised yet.

In other words, continuous speech recognition occurs when a user attempts continuous operation to pronounce several words or uses a spell consisting of sentences, which makes it difficult to find the boundary between words and causes the 'echo' phenomenon in the voice to act as noise. There is this.

Accordingly, the present invention proposes a method of complementing the incomplete parts between the motion information and the voice information and providing a dynamic game by combining existing methods that have been relatively proven in the area of motion recognition or voice recognition.

The present invention has been proposed to solve the above problems, after extracting an object from a binocular image having different camera information, calculating depth information using binocular parallax, and using the calculated depth information in a three-dimensional space The position of the object is obtained, and the motion recognition result generated by recognizing and generating the entire motion data through an inverse kinematics algorithm, the feature point extracted from the voice composed of at least one phrase, and the motion recognition result received from the motion recognition means. The game apparatus and method using motion recognition and voice recognition for controlling the movement of the character and the corresponding sound after recognizing the command by combining the recognized speech recognition result after dividing the phrase of the sentence by using The purpose is to provide.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

According to an aspect of the present invention, there is provided a game apparatus using motion recognition and voice recognition, comprising: storage means for storing user-specific command data; After extracting an object from a binocular image having different camera information, depth information is calculated using binocular disparity, the position of the object is calculated in three-dimensional space using the calculated depth information, and inverse kinematics. Motion recognition means for recognizing motion by generating full motion data through an algorithm; Isolated word recognition means for recognizing a phrase after dividing a phrase of speech using a feature point extracted from a speech composed of at least one phrase and a motion recognition result received from the motion recognition means; Command recognition means for recognizing the command by combining the recognition result in the isolated word recognition means and the recognition result in the motion recognition means; And central processing means for controlling the operation of the character and its corresponding sound in accordance with the recognition result (command data) in the command recognition means, and for storing the recognition result in the storage means.

Meanwhile, in the game method using motion recognition and voice recognition, the method of the present invention extracts an object as a motion recognition means receives a binocular image having different camera information, and then uses depth binocular disparity to extract depth information. Calculating; Recognizing the motion by generating the entire motion data through an inverse kinematics algorithm after obtaining the position of the object in a three-dimensional space by using the calculated depth information by the motion recognizing means; Extracting a feature point according to an input of a voice composed of at least one phrase by the isolated word recognizing means, and recognizing a phrase after dividing a phrase of the voice using the feature point and the motion recognition result; Command recognition means for recognizing a command by combining the phrase recognition result and the motion recognition result; And controlling, by the central processing means, the motion of the character and the corresponding sound according to the recognition result (command data).

In addition, the present invention combines the advantages and disadvantages of motion recognition and voice recognition through video, and is effective for producing a game that can have dynamic interaction with the contents of the game and a large number of users connected while taking various actions. Provide a model and specific implementation.

In addition, the present invention solves the biggest disadvantage of the backward motion algorithm in which the motion ambiguity exists because the whole motion is inferred from a limited number of body parts by combining with the voice data. For example, when a user attempts to perform a specific action or magic in the game, the user may call out the title, the name of the martial art, or the spell, to correctly recognize the command.

In addition, the present invention easily solves a number of functions that are cumbersome to solve by motion recognition alone. For example, when a user walks in a game and wants to switch modes to run or fly, there is no need to click a button with the mouse controls or click a button on the display screen in motion like a traditional game. If you pronounce the command, you can switch immediately. In addition, special actions such as selecting specific items in the game or exchanging weapons or equipment can be simply indicated by voice.

In addition, the present invention has a database of the recognized commands (motion data and corresponding voice data), the longer the user the game, the easier it is to communicate to the game device, which is reflected in the increase in the 'experience value' in the game Can be.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of a game device using motion recognition and voice recognition according to the present invention.

As shown in FIG. 1, a game device using motion recognition and voice recognition according to the present invention includes a storage unit 10 for storing user-specific command data (motion data and corresponding voice data), and different cameras. After extracting the object from the input unit 20 for receiving the binocular image having the information and the audio (sentence), the binocular image received from the input unit 20, depth information is calculated using binocular disparity, and the calculated A motion for obtaining the position of the object in a three-dimensional space using depth information and generating all motion data through an inverse kinematics algorithm to recognize the motion and deliver the result to the isolated word recognition unit 40. Recognition unit 30, using the feature point extracted from the voice (sentence) received from the input unit 20 and the motion of the sentence using the motion recognition result received from the motion recognition unit 30 After distinguishing between the isolated word recognition unit 40 for recognizing a phrase, the recognition result in the isolated word recognition unit 40 and the recognition result in the motion recognition unit 30 for recognizing a command The command recognition unit 50, the output unit 60 for outputting the motion of the character and the corresponding sound under the control of the central processing unit 80, the Internet connection unit 70 for connecting to the Internet, and the command recognition unit And a central processing unit 80 for controlling the operation of the character and the corresponding sound according to the recognition result (command data) in 50, and for storing the recognition result in the storage unit 10.

Here, the Internet connection 70 is an additional element of the present invention.

In addition, the camera information includes camera three-dimensional position information (x, y, z coordinates), camera direction information, focal length information.

The input unit 20 includes an image sensor 201 for receiving a binocular image having different camera information, and a voice sensor 202 for receiving a voice (sentence).

On the other hand, the motion recognition unit 30 is an object (for example, in the binocular image received through the motion sensor 300, the image sensor 201, which stores the motion data used in the game for each age, gender and body type) Depth information (distance information in the direction perpendicular to the camera lens) is calculated by using the object extractor 301 for extracting the head, the hand, the foot, and the binocular parallax of the object extracted by the object extractor 301. In the three-dimensional space calculated by the position calculator 302 for calculating the position of the object in the three-dimensional space using the calculated depth information, the position calculator 302 using an inverse kinematics algorithm The backward motion processor 303 for generating the entire motion data by calculating the position and state of other parts such as the elbow, knee, and pelvis through the position of the object of the object, and generated by the backward motion processor 303. And a motion recognizer 304 for comparing the motion data with the corresponding motion data on the motion database 300 and recognizing the motion and delivering the result to the phrase boundary delimiter 402.

Here, as shown in FIG. 7, the backward motion processor 303 receives a position b of a specific body part as an input and calculates the entire motion data c as a result.

In addition, since the motion input from the user is not always the same, the motion database 300 stores a plurality of similar motion data for one accurate motion data.

On the other hand, the voice recognition unit 40, the voice database 400 that stores the voice data used in the game, a feature point extractor for extracting feature points from the voice (sentence) received through the voice sensor 202 ( 401, a phrase boundary boundary separator 402 for classifying phrases of sentences using the feature points extracted by the feature point extractor 401 and the motion recognition result from the motion recognizer 304, and the phrase boundary separator An isolated word recognizer 403 for recognizing the phrase classified at 402 is included.

In this case, since the voice input from the user is not always the same, the voice database 400 stores a plurality of similar voice data for one accurate voice data.

In addition, as shown in FIG. 5, the phrase boundary delimiter 402 separates a phrase from a voice composed of at least one phrase. At this time, the motion information used to distinguish between the phrases so that the start and end of each operation can be clearly recognized than the voice information.

In addition, the isolated word recognizer 403 selects a result having a maximum likelihood through a feature point extraction step and comparing each word with a database of one HMM model.

On the other hand, the command recognizer 50, a command recognizer 501 for recognizing a command by combining the recognition result in the isolated word recognizer 403 and the recognition result in the motion recognizer 304, and the command recognizer A user database 500 for storing voice and motion corresponding to the command recognized by 501 is included.

Figure 4 is an embodiment implementation of a game device using motion recognition and speech recognition according to the present invention.

As shown in FIG. 4, a game device using motion recognition and voice recognition according to the present invention includes a user 41, a display device (for example, a TV) 42 for outputting an output image of a game, and a preliminary correction. Two cam cameras (43) acting as stereo cameras whose position and angle are precisely fixed so that the results can be used, microphones (44) for voice input, inputs and outputs, and game machines for performing game operations. A main body 45 and an internet connection 46.

Here, the Internet connection unit 46 allows the user to enjoy the game in the same electronic space as other users through the network, and by providing the operation and technology available in the game through the online upgrade, users do not demonstrate to the game Keep it interesting.

8 is a flowchart illustrating an embodiment of a game method using motion recognition and voice recognition according to the present invention.

First, in order to apply Inverse Kinematics, at the beginning of the first game, the user raises his arm in a cruciform position in front of the camera so that his palm is facing forward to recognize the figure.

Then, after receiving the binocular image having different camera information, the object is extracted, depth information is calculated using binocular parallax, and the position of the object is calculated in the three-dimensional space using the calculated depth information. The entire motion data is generated through an inverse kinematics algorithm to recognize the motion (801).

Then, after extracting the feature point according to the pressure (sentence) under the pressure, and using the feature point and the motion recognition result to distinguish the phrase of the sentence and recognizes the phrase (802).

Thereafter, the command is recognized by combining the voice recognition result with the motion recognition result (step 803).

Then, the operation of the character and the corresponding sound is controlled according to the recognition result (command data) (804).

On the other hand, in the present invention, the camera correction method is preferable to use the 'correction method using the compensation object' for noise-resistant and seamless service.

In addition, in the present invention, when receiving a long spell to utilize a fantasy element such as magic in the game as well as a short-named attack means, the word is classified into a speech recognizer using motion recognition data. It is preferable to use the unit isolated word recognizer 403.

In addition, the present invention, as shown in Figure 6, through the network, a plurality of users in the same virtual space to enjoy each other's actions in real time through the characters in the game through the game, and compete with each other higher It can be interesting.

In addition, the present invention is periodically provided with upgrades to operations, spells, and the like through an Internet connection, so that users can continue to utilize new attacks, defenses and magic spells and enjoy a long time without demonstrating the same game.

In addition, in the present invention, the image information is also used to distinguish the motion, but mainly used to determine the direction, agility, strength, and the like of the motion, and the voice information is used to accurately classify the motion and intensity (joint sound), other item selection, or mode selection. It is used to give specific instructions.

In addition, in the present invention, the user stands in front of the display device and takes various actions, and on the display screen, the user's appearance and various attack targets and defense targets appear, and the game content includes a fantasy game in which action and magic appear. do. Therefore, the user has active interaction with the game by utilizing both the body and the voice, such as shouting a specific action name or spell while putting various sounds such as attack and defense, and putting in the sound of a fire.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form. Since this process can be easily implemented by those skilled in the art will not be described in more detail.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

As described above, the present invention extracts an object from a binocular image having different camera information, calculates depth information using binocular parallax, and calculates a position of the object in a three-dimensional space using the calculated depth information. , Using the inverse kinematics algorithm to generate the entire motion data and classify the sentence phrase using the motion recognition result and the feature point extracted from the voice (sentence) and the motion recognition result received from the motion recognition means After recognizing the command by combining the recognized voice recognition results, and controlling the motion of the character and the corresponding sound, there is an effect that can provide a game with a small error rate.

In addition, the present invention enables to realize dynamic and realistic action and fantasy game contents that cannot be realized by the conventional method by combining the advantages and disadvantages of motion recognition and voice recognition. You can check the action in real time and have the effect of playing the game realistically.

Claims (5)

In the game device using motion recognition and voice recognition, Storage means for storing user-specific command data; After extracting an object from a binocular image having different camera information, depth information is calculated using binocular disparity, the position of the object is calculated in three-dimensional space using the calculated depth information, and inverse kinematics. Motion recognition means for recognizing motion by generating full motion data through an algorithm; Isolated word recognition means for recognizing a phrase after dividing a phrase of speech using a feature point extracted from a speech composed of at least one phrase and a motion recognition result received from the motion recognition means; Command recognition means for recognizing the command by combining the recognition result in the isolated word recognition means and the recognition result in the motion recognition means; And Central processing means for controlling the operation of the character and the corresponding sound in accordance with the recognition result (command data) in the command recognition means, and for storing the recognition result in the storage means Game device using motion recognition and speech recognition comprising a. The method of claim 1, Internet connection means for connecting to the internet Game device using motion recognition and voice recognition further comprising. The method according to claim 1 or 2, The motion recognition means, A motion database that stores motion data used in the game for each age, gender, and body type; An object extractor for extracting an object from the binocular image; Calculate depth information (distance information in a direction perpendicular to the camera lens) using binocular parallax of the object extracted by the object extractor, and calculate a position for obtaining the position of the object in a three-dimensional space using the calculated depth information group; Using the inverse kinematics algorithm to calculate the position and state of other parts such as elbows, knees, pelvis, etc. from the position of the object in the three-dimensional space calculated by the position calculator to generate the entire motion data A reverse motion processor for; And A motion recognizer for comparing the motion data generated by the backward motion processor with the corresponding motion data on the motion database to recognize the motion and to transmit the result to the isolated word recognition means. Game device using motion recognition and speech recognition comprising a. The method of claim 3, wherein The isolated word recognition means, A voice database for storing voice data utilized in the game; A feature point extractor for extracting feature points from the speech composed of the at least one phrase; A phrase boundary separator for classifying speech phrases of speech using feature points extracted by the feature point extractor and motion recognition results from the motion recognizer; And Isolated word recognizer for recognizing phrases classified by the boundary delimiter between phrases Game device using motion recognition and speech recognition comprising a. In the game method using motion recognition and voice recognition, Calculating depth information using binocular disparity after extracting an object according to a binocular image having different camera information inputted by the motion recognition means; Recognizing the motion by generating the entire motion data through an inverse kinematics algorithm after obtaining the position of the object in a three-dimensional space by using the calculated depth information by the motion recognizing means; Extracting a feature point according to an input of a voice composed of at least one phrase by the isolated word recognizing means, and recognizing a phrase after dividing a phrase of the voice using the feature point and the motion recognition result; Command recognition means for recognizing a command by combining the phrase recognition result and the motion recognition result; And Controlling, by the central processing means, the movement of the character and the corresponding sound according to the recognition result (command data) Game method using a motion recognition and speech recognition comprising a.

Images