JP2010511958A - Gesture / voice integrated recognition system and method - Google Patents

Abstract

  The present invention relates to a gesture / speech integrated recognition system and method, and relates to a voice and speech recognition method that detects the start and end points of a command word from input voices in order to improve the recognition performance of the command word through voice and gesture integration in a noisy environment. A voice feature extraction unit for extracting feature information, a gesture feature extraction unit for detecting a command section from a gesture of a captured video using information on the detected start point and end point, and extracting gesture feature information; and the extracted voice The system includes an integrated recognition unit that outputs feature information and gesture feature information as integrated recognition data using preset learning parameters, and can easily and accurately recognize user commands.

Description

  The present invention relates to integrated recognition technology, and in particular, extracts feature information of gestures using speech EPD values and integrates them with speech feature information in order to recognize user commands with high performance in an actual noise environment. And a gesture / voice integrated recognition system and method capable of recognizing a user's command.

  The present invention is derived from research conducted as part of the IT New Growth Dynamic Core Technology Development Project of the Information Communication Department and the Information Communication Research Promotion Agency [Problem Management Number: 2006-S-031-01, Name of Project] : Development of five-sense information processing technology for network-based real-world services].

  Recently, with the development of multimedia technology and interface technology, facial expression and direction, lip shape, gaze tracking, hand gesture, voice, etc. are used to easily and easily realize human-machine interface. However, research on the recognition of multi-modal forms has been actively conducted.

  In particular, among the current Man-Machine interface technologies, speech recognition technology and gesture recognition technology are used as the most convenient interface technologies. However, although the voice recognition technology and the gesture recognition technology show a high recognition rate in a limited environment, there is a problem that the performance cannot be sufficiently exhibited in an actual noise environment. This is because, in speech recognition, environmental noise has the greatest effect on performance, and in camera-based gesture recognition technology, there are many differences in performance depending on lighting changes and gesture types. Therefore, it is necessary to develop a technology that can recognize speech recognition technology using a noise-resistant algorithm, and gesture recognition technology needs technology development that can extract a specific section of a gesture including recognition information. It became. In addition, when using a general gesture, it is difficult to recognize a specific section of the gesture because it cannot be easily divided.

  In the case where voice and gesture are integrated and recognized, the processing speed of the audio frame is about 10 ms / frame and the processing speed of the video frame is about 66.7 ms / frame. In addition, since the gesture section generally takes more time than the voice section, there is a difference between the length of the generated voice section and the length of the gesture section, and the voice and gesture are synchronized. Problems arise.

  Therefore, in order to solve the above problems, an algorithm that is resistant to environmental noise is used, a command word section is searched from the user's voice, feature information is extracted, and information about the start point of the voice command word is used. There is a need for a means that can detect a feature section of a gesture and easily recognize a command even for a gesture that is not clearly divided.

  In addition, there is a need for means for matching synchronization by applying an optimal frame preset in the gesture command section detected by the voice EPD value to the problem regarding the synchronization difference that occurs in the integrated recognition of voice and gesture. .

  An integrated gesture / speech recognition system according to the present invention for solving the above-described problem includes a speech feature extraction unit that detects a start point and an end point of a command word from input speech and extracts speech feature information; Using the information about the detected start and end points, a gesture feature extraction unit that detects a command section from a gesture of a captured video and extracts gesture feature information; and the extracted speech feature information and gesture feature information with preset learning parameters It is characterized by comprising an integrated recognition unit that is used and output as integrated recognition data.

  On the other hand, the integrated gesture / speech recognition system uses the detected start point to detect the start point of the gesture from the captured video and applies the optimum number of frames set in advance from the start point of the gesture. And a synchronization module including an optimum frame application module for calculating and extracting the video frame. At this time, the gesture start point detection module detects the start point of the gesture by checking the detected start point (EPD: End Point Detection) plug in the captured video.

  The voice feature extraction unit uses an EPD (End Point Detection) detection module that detects a start point and an end point of a command word from the input voice, and an algorithm based on an auditory model, and detects the detection. A speech feature extraction module based on an auditory model for extracting speech feature information included in the command word from the command word, and further removing noise from the extracted speech feature information.

  The gesture feature extraction module uses a hand tracking module that tracks hand movements from video captured by a camera and transmits the motion to the synchronization module, and an optimal video frame extracted by the synchronization module. It is characterized by comprising a gesture feature extraction module for extracting feature information.

  In addition, the integrated recognition unit includes an integrated learning DB control module that generates a learning parameter based on a preset integrated learning model and an integrated learning database, the extracted speech feature information and gesture feature information as the generated learning parameter And an integrated recognition module for generating a result controlled by the integrated feature control module as a recognition result. In this case, the integrated feature control module is an input. The feature vector of the extracted speech feature information and gesture feature information is controlled through expansion and reduction of the number of nodes of the vector to be performed.

  In order to achieve the above object, the gesture / speech integrated recognition method of the present invention includes a step of detecting speech feature information by detecting a start point (EPD value) and an end point of a command word from input speech. , Using the detected start point of the command word, detecting the command section from the gesture of the video input by the camera, and extracting the gesture feature information; and the extracted voice feature information and gesture feature information are already set It is characterized by comprising three stages of outputting as integrated recognition data using learning parameters.

  At this time, the step 1 is characterized in that voice feature information is extracted based on an auditory model from a command word section by a start point and an end point of the command word.

  In addition, the two steps are preset as A step for tracking a hand gesture from the input image of the camera and B step for detecting a command interval by the hand gesture using the transmitted EPD value. And C stage for determining the optimum frame from the command section by the gesture and D stage for extracting gesture feature information from the determined optimum frame.

  As described above, the gesture / speech integrated recognition system and method according to the present invention detects the gesture command word section using the EPD value that is the starting point of the voice command word section, and also in the case of a gesture that cannot be clearly distinguished. The recognition rate can be increased, and integrated recognition by voice and gesture can be realized by applying an optimum frame to the command word section of gesture and synchronizing the voice and gesture.

1 is a diagram illustrating a concept of an integrated gesture / voice recognition system according to the present invention. 1 is a diagram illustrating a configuration of a gesture / voice integrated recognition system according to the present invention. 3 is a flowchart illustrating a gesture / voice integrated recognition method according to the present invention.

  DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to the presently preferred embodiments of the invention, which are readily practiced by those skilled in the art to which the invention pertains. However, in describing the operating principles for the preferred embodiment of the present invention in detail, it is determined that specific descriptions of known functions or configurations may unnecessarily obscure the subject matter of the present invention. Detailed description thereof will be omitted.

  FIG. 1 is a diagram showing a concept of a gesture / voice integrated recognition system according to the present invention.

  Referring to FIG. 1, the gesture / speech integrated recognition technology recognizes a human voice and a command based on a gesture, and controls a device that expresses the five senses using a control command generated according to the recognition result.

  Specifically, the person 100 gives a command with a voice 110 and a gesture 120. Here, an example is given to an order given by a person. When a person purchases an item in cyberspace, the command to select a specific bread from the displayed items is called “corn bread”. Can act to point to corn bread.

  When the person 100 issues a command with the voice 110 and the gesture 120, the feature information related to the voice command of the person is recognized through the voice recognition 111, and the feature information based on the person's gesture is recognized through the gesture recognition 121. The recognition information of the speech and gesture recognized in this way is recognized as one user command by the integrated recognition 130 in order to improve the recognition rate for the gesture that cannot be clearly distinguished from the speech weak to environmental noise. To do.

  The present invention is a technique relating to integrated recognition of human speech and gestures. The command recognized in this way is transmitted to the speaker 170, the display device 171, the aroma generator 172, the tactile device 173, and the taste device 174, which are output devices for individual sensations, by the control unit, and controls the respective devices. It is also possible to transmit the recognition result to the network, transmit the five sense data for the result, and control each output device. However, the present invention relates to integrated recognition, and since the configuration after recognition can be applied in various ways, description thereof will be omitted.

  FIG. 2 is a diagram showing a configuration of a gesture / voice integrated recognition system according to the present invention.

  Referring to FIG. 2, the gesture / speech integrated recognition system detects a start point and an end point of a command word from speech input by a microphone 211 and extracts speech feature information, and a speech feature extraction unit. The information on the start point and end point detected by 210 is used to detect the command section from the gesture of the video shot by the camera, and the gesture feature extraction unit 220 that extracts the gesture feature information, and the start point detected by the voice feature extraction unit 210 A synchronization module 230 that detects the start point of the gesture from the captured video using the image, applies the optimal number of frames set in advance from the start point of the gesture thus detected, and extracts the video frame in this way Integrated voice feature information and gesture feature information using preset learning parameters Characterized in that it comprises an integrated recognition section 240 to output as identification data. Hereinafter, each component will be described in detail.

  The voice feature extraction unit 210 is detected by a microphone 211 into which a user inputs voice, an EPD (End Point Detection) detection module 212 that detects the start point and end point of a command word section from the user's voice, and an EPD detection module 212. The speech feature extraction module 213 based on the auditory model extracts voice feature information based on the auditory model for the command word section of speech. In addition, a channel noise removal module that removes noise included in the extracted voice feature information can be included (not shown).

  The EPD detection module 212 analyzes the voice input by the wired / wireless microphone and detects the start point and the end point of the command word.

  Specifically, the EPD detection module 212 acquires a voice signal, calculates an energy value necessary for detecting the end point of the voice signal, determines a section to be calculated as a command word from the input voice signal, and determines a command. Detect the start and end points of words.

  The EPD detection module 212 first acquires an audio signal from a microphone, and converts the acquired audio into a form for frame calculation. When voice is input wirelessly in this process, problems such as data loss and signal distortion due to signal interference may occur, and a processing process for this is necessary when acquiring a signal.

  In the EPD detection module 212, calculation of the energy value necessary for detecting the end point of the audio signal is obtained as follows, for example. The frame size for analyzing the audio signal is based on 160 sample, and the frame energy is calculated by the following equation.

  S (n): vocal cord signal sample, N: number of samples in one frame

  The frame energy obtained in this way is used as a parameter for the subsequent end point detection.

  The EPD detection module 212 determines a section to be actually calculated as a command word after calculating the frame energy value. For example, the process of calculating the start point and the end point of an audio signal is determined by four energy threshold values using frame energy and ten conditions. Here, the four energy critical values and the ten conditions can be set in various ways. Preferably, the most appropriate one for obtaining the command word section is selected by experiment. The four critical values are determined for each frame by the end point detection algorithm.

  The EPD detection module 212 transmits information on the start point of the instruction word thus detected (hereinafter referred to as “EPD value”) to the gesture start point detection module 231 of the synchronization module 230.

  Further, the EPD detection module 212 transmits information on the command word section from the input speech to the speech feature extraction module 213 based on the auditory model, and extracts speech feature information.

  The voice feature extraction module 213 based on the auditory model that has received information on the voice command word section extracts feature information from the command word section detected by the EPD detection module 212 based on the auditory model. Algorithms used to extract voice feature information based on an auditory model include an EIH algorithm and a ZCPA algorithm.

  The voice feature information extracted by the voice feature extraction module 213 based on the auditory model is transmitted to the integrated recognition unit 240 after removing noise by a channel noise removal module (not shown).

  The gesture feature extraction unit 220 tracks a face and hand detection module 222 that detects a face and a hand from an image captured by the camera 221, and detects and detects the movement of the hand, and transmits the detected motion to the synchronization module 230. The gesture feature extraction module 224 extracts the feature information of the gesture using the optimal frame calculated by the above.

  The face and hand detection module 222 detects the face and hand to be gestured from the video, and the hand tracking module 223 continuously tracks the movement of the hand in the video. However, although the hand tracking module 223 has been described as limited to hands, it can track various body parts that can be recognized as gestures by those skilled in the art.

  The hand tracking module 223 continuously stores hand movements as time progresses, and a part that can be recognized as a gesture command from the hand movements uses the EPD value transmitted from the speech feature extraction unit 210 by the synchronization module 230. Detected. Hereinafter, a synchronization module 230 that detects an interval recognized as a gesture command from the hand movement using the EPD value and applies an optimal frame for synchronizing the voice and the gesture will be described.

  The synchronization module 230 is necessary for integrated recognition using the gesture start point detection module 231 that detects the start point of the gesture using the EPD value and the image of the hand movement, and the gesture start point frame calculated from the detected gesture start point. An optimal frame application module 232 for calculating an optimal video frame is included.

  When the EPD value of the audio is detected by the EPD detection module 212 while the audio signal and the video signal are input in real time, the synchronization module 230 checks the audio EPD plug from the video signal. In this way, the gesture start point detection module 231 calculates the start frame of the gesture. Also, using the calculated start frame of the gesture, the optimum frame application module 232 calculates an optimum video frame necessary for the integrated recognition and transmits it to the gesture feature extraction module 224. The optimal video frame necessary for integrated recognition applied by the optimal frame application module 232 is preset with the number of frames determined to have the highest gesture recognition rate, and the gesture start point detection module 231 calculates the gesture start point frame. Then, determine the optimal video frame.

  The integrated recognition unit 240 is adapted to the development of an integrated model generation module 242 that generates an integrated model for efficiently integrating speech feature information and gesture feature information based on a learning model, and an integrated recognition algorithm based on a statistical model. Integrated learning DB 244 constructed in the form, integrated learning DB control module 243 for controlling learning and learning parameters by the integrated model generation module 242 and the integrated learning DB 244, and features of the speech feature information and gesture feature information inputted as learning parameters An integrated feature control module 241 that controls vectors and an integrated recognition model 245 that generates recognition results and provides various functions.

  The integrated model generation module 242 generates a high-performance integrated model in order to efficiently integrate voice feature information and gesture feature information. In order to determine a high-performance integrated model, various existing learning algorithms (Hidden Markov Model (HMM), Neural Network (NN), Dynamic Time Wapping (DTW), etc.) can be implemented and determined through experiments. In particular, the present invention can use a method of determining an integrated model based on NN and optimizing NN parameters that can exhibit high performance in integrated recognition. However, one of the biggest problems in generating a high-performance integrated model is how to synchronize two modalities with different numbers of frames in the learning model.

  The synchronization problem within the learning model is the same as the learning model optimization problem. The present invention includes an integration layer, and optimizes a voice and gesture connection method within the integration layer. In order to optimize, the speech and gesture overlap length is calculated based on the time axis and then synchronized based on the calculated length. Such a superposition length is searched for a connection method showing the highest recognition rate through an experiment on the recognition rate.

  The integrated learning DB 244 constructs an integrated recognition database in a form that is compatible with the development of an integrated recognition algorithm based on a statistical model.

  For example, data of various age groups are synchronized and collected using a stereo camera and a wireless microphone for 10 words. Table 1 shows a command group defined for the integration of gesture and voice. The defined command groups are intended for natural gestures that humans can generally understand without much learning.

  At this time, the audio sampling rate is 16 bits at 16 kHz, and recording is performed using a Pulse Coded Modulation (PCM) Waveform with 1 channel (mono). The video was recorded using a STH-DCSG-C stereo camera, and a 24-bit BITMAP image of 15 frames per second and 320 × 240 size was recorded under illumination with a blue screen background and four fluorescent lamp boxes. Since there is no audio interface in stereo cameras, audio and video acquisition modules are created independently, and video and audio are synchronized by controlling the video acquisition process through IPC (Inter-Process Communications) with an audio recording program. A program was created and data was collected. The video collection module was configured using an Open CV (Computer Vision) library and an SVS (Small Vision System).

  Stereo camera images must be applied to the actual recording environment through a separate calibration process, and the gain, exposure, brightness, red, and blue parameter values are modified to obtain the optimal image. The color feeling, exposure and WB value were adjusted. Calibration information and parameter information are saved in a separate ini file and called by the video saving module for reference.

  The integrated learning DB control module 243 generates learning parameters based on the integrated learning DB 244 generated and stored in advance in cooperation with the integrated model generation module 242.

  The integrated feature control module 241 controls the learning parameters generated by the integrated learning DB control module 243, the speech extracted by the speech feature extraction unit 210, the gesture feature extraction unit 220, and the feature vector of the gesture feature information. . Such control involves expansion and reduction of the number of nodes in the input vector. The integrated feature control module 241 is characterized by having an integrated layer, which is developed to efficiently integrate different sized speech and gesture lengths to present a single recognition rate.

  The integrated recognition module 245 generates a recognition result using the control result from the integrated feature control module 241. It also provides various functions for interacting with an integrated expression device or network.

  FIG. 3 is a flowchart illustrating a gesture / voice integrated recognition method according to the present invention.

  Referring to FIG. 3, the gesture / speech integrated recognition method is composed of three threads and operates. The three threads include a speech feature extraction thread 10 that extracts speech features, a gesture feature extraction thread 20 that extracts gesture features, and an integrated recognition thread 30 that performs integrated recognition of speech and gestures. The three threads 10, 20, and 30 are generated at the time when the learning parameters are loaded, and operate organically using a thread plug. Hereinafter, a gesture / speech integrated recognition method through an organic operation of the three threads 10, 20, and 30 will be described.

  When the user gives an instruction using voice and gesture, the voice feature extraction thread 10 continues to receive voice using the wired / wireless microphone (S311). In addition, the gesture feature extraction thread 20 continuously receives an image including a gesture using the camera (S320). The EPD detection module 212 detects the start point and the end point (speech EPD value) of the command word included in the voice while calculating the voice frame of the voice that is continuously input using the microphone (S313). When the voice EPD value is detected, the voice EPD value is transmitted to the synchronization stage 40 of the gesture feature extraction thread. When the voice command word section is determined by the start point and the end point of the command word included in the voice, the voice feature extraction module 213 based on the auditory model extracts the voice feature from the command word section based on the auditory model. Communicate to the thread 30.

  The gesture feature extraction thread 20 detects a hand and a face from images continuously input through the camera (S321). When the hand and face are detected in this way, the user's gesture is tracked (S322). Since the user's gesture continues to change, a certain length of gesture is stored in the buffer S323.

  If the voice EPD value is detected and transmitted in the process of storing the gesture in the buffer, the voice EPD plug in the gesture video stored in the buffer is checked S324. The starting point and the ending point of the gesture including the feature information of the video are searched by the voice EPD plug in S325, and the gesture feature thus searched is stored in S326. Since the gesture features stored in this manner are different in synchronization with the voice, the optimum frame set in advance is applied and the optimum frame is calculated from the start frame of the gesture. The calculated optimum frame is extracted using the gesture feature extraction module 224 and is transmitted to the integrated recognition thread.

  When the voice and gesture feature information is successfully extracted by the voice feature extraction thread 10 and the gesture feature extraction thread 20, the voice / gesture feature extraction threads 10 and 20 are stopped while the integrated recognition thread 30 checks the recognition result. S328 and S315 that are in the (Sleep) state.

  The integrated recognition thread 30 generates a high-performance integrated model by the integrated model generation module 242 in advance before receiving the speech feature information and the gesture feature information, and controls the generated integrated model and the integrated learning DB 244. The integrated learning DB control module 243 generates and loads learning parameters S331. When the learning parameters are loaded in this way, the integrated recognition thread 30 is maintained in a stopped state until the voice / gesture feature information is transmitted S332.

  The integrated recognition thread 30 in the stopped state as described above completes the extraction of the feature information of the voice and gesture S333, and when receiving the signal related to the feature information, loads each feature into the memory S334. When the voice and gesture feature information is loaded, a recognition result is calculated using a preset optimized integrated learning model and learning parameters (S335).

  When the recognition result is calculated by the integrated recognition unit 240, the voice feature extraction thread 10 and the gesture feature extraction thread 20 in the stopped state perform the work of extracting feature information from the input voice and video again.

  The present invention described above is not limited by the above-described embodiments and the accompanying drawings, and various replacements, modifications and changes can be made without departing from the technical idea of the present invention. It is obvious to a person skilled in the art having ordinary knowledge in the technical field to which the present invention belongs.

Claims (15)

A voice feature extraction unit for detecting voice feature information by detecting the start point and end point of the command word from the input voice;
A gesture feature extraction unit for detecting a command section from a gesture of a captured video using information on the detected start point and end point, and extracting gesture feature information;
A gesture / speech integrated recognition system, comprising: an integrated recognition unit that outputs the extracted voice feature information and gesture feature information as integrated recognition data using preset learning parameters. A gesture start point detection module for detecting a start point of a gesture from the captured video using the detected start point;
The gesture according to claim 1, further comprising a synchronization module including an optimal frame application module that applies an optimal number of frames set in advance from a starting point of the gesture and calculates and extracts an optimal video frame. / Integrated speech recognition system. The gesture start point detection module includes:
The gesture / speech integrated recognition system according to claim 2, wherein the detected start point (EPD: End Point Detection) plug is checked in the captured video to detect a start point of a gesture. The voice feature extraction unit
An EPD (End Point Detection) detection module for detecting a start point and an end point of a command word from the input voice;
2. A speech feature extraction module based on an auditory model that extracts speech feature information contained in the command word from the detected command word using an algorithm based on an auditory model. The gesture / speech integrated recognition system according to any one of Items 3 to 3. The voice feature extraction unit
5. The gesture / voice integrated recognition system according to claim 4, wherein noise is removed from the extracted voice feature information. The gesture feature extraction unit
A hand tracking module that tracks hand movements from video captured by the camera and transmits it to the synchronization module;
4. The gesture / speech integrated recognition system according to claim 3, further comprising a gesture feature extraction module that extracts gesture feature information using an optimal video frame extracted by the synchronization module. The integrated recognition unit
An integrated learning DB control module for generating learning parameters based on a preset integrated learning model and an integrated learning database;
An integrated feature control module for controlling the extracted voice feature information and gesture feature information using the generated learning parameter;
The gesture / voice integrated recognition system according to claim 1, further comprising an integrated recognition module that generates a result controlled by the integrated feature control module as a recognition result. The integrated learning model is
The integrated gesture / speech recognition system according to claim 7, wherein the gesture / speech integrated recognition system is generated based on a neural network (NN) learning algorithm. The integrated learning database is
The feature information for speech and gestures of various ages is integrated using a stereo camera and a wireless microphone, and is constructed in a form applicable to an integrated recognition algorithm based on a statistical model. Gesture / voice integrated recognition system. The gesture / voice integrated recognition system according to claim 7, wherein the integrated recognition module includes an integration layer that integrates the extracted voice feature information and gesture feature information. The integrated feature control module includes:
The gesture / speech integrated recognition system according to claim 7, wherein the feature vector of the extracted speech feature information and gesture feature information is controlled through expansion and reduction of the number of nodes of the input vector. A step of detecting voice feature information by detecting a start point (EPD value) and an end point of a command word from input voice;
Using the start point of the detected command word, detecting a command section from a gesture of a video input by a camera, and extracting gesture feature information;
A gesture / speech integrated recognition method comprising three steps of outputting the extracted speech feature information and gesture feature information as integrated recognition data using preset learning parameters. The one stage includes
13. The gesture / voice integrated recognition method according to claim 12, wherein voice feature information is extracted based on an auditory model from a command word section defined by a start point and an end point of the command word. The two steps are:
A stage for tracking hand gestures from the input video of the camera;
B stage for detecting a command interval by the hand gesture using the transmitted EPD value;
C stage for applying an optimal frame set in advance and determining an optimal frame from the command section by the gesture;
13. The integrated gesture / speech recognition method according to claim 12, further comprising a step D of extracting gesture feature information from the determined optimum frame. The method of claim 12, wherein the one step further includes a step of removing noise from the extracted voice feature information.

Images