JPH1173297A - Recognition method using timely relation of multi-modal expression with voice and gesture - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make a voice indicating the meaning of gesture correspond to gesture performed with the voice. SOLUTION: Stop positions T1-T3 of an operation are detected based on a moving image. Also, a voice is word-recognized, and utterance start and end positions T11 and T12 of the word are detected. Stop positions T1 T2 which are the closest to the utterance start and end positions T11 and T12 are defined as gesture start and end positions, and a voice and the gesture are made correspond to each other by the start and end positions T1, T2, T11, and T12.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gesture recognition method for photographing a human gesture and identifying the content of the gesture based on the photographed image, and more particularly to a recognition method using a temporal relationship between a voice and a multimodal expression by the gesture. About the method.

[0002]

2. Description of the Related Art It is not enough to increase the recognition rate of a speech recognition device or a gesture recognition device for each modal in order to make modals such as voices and gestures available, such as communication between humans. is there. For example, the classification of gestures (Ekman, P. Friesen, W.
V., "The reportoire of nonverbval behaivior-categor
ies, origins, usage, and coding ", Semiotical, pp.4
As pointed out in 9-98 (1969).), Gestures often indicate the position or degree of a gesture while saying "this way" or "this much". Thus, there are many cases that can be interpreted for the first time by a combination of modals.

In order to automatically recognize such a modal combination expression by a recognition device, it is necessary to find a corresponding part between a voice and a gesture.

[0004]

A speech recognition apparatus for recognizing a voice and a gesture recognition apparatus for recognizing a gesture based on a photographed image have been proposed.
No method has been proposed, and when trying to recognize such a multi-modal expression, in the above example, it is possible to recognize the voice "here" and convert it to a character string etc. There is a problem to be solved in which it cannot be determined where a gesture starts and ends in a moving image without visual confirmation by a human.

[0005] In view of the above, it is an object of the present invention to automatically associate a voice with a gesture part when audio data relating to the gesture and moving image data obtained by photographing the operation of the gesture are given. It is an object of the present invention to provide a recognition method which can be performed in an efficient manner.

[0006]

In order to achieve the above object, a first aspect of the present invention is a recognition method for recognizing a gesture accompanied by a voice indicating the content of a gesture. , In the word recognition,
The utterance start time and the utterance end time of the word are detected, respectively, and the stop time at which the operation of the gesture is stopped is detected based on the moving image of the gesture, and the utterance start time and the utterance are detected among the detected stop times. The gesture is associated with the corresponding voice by detecting a stop point closest to the end point.

According to a second aspect of the present invention, in the recognition method according to the first aspect, a difference value between two adjacent frame images in a continuous frame image constituting the moving image is obtained, and the difference value of the difference value is obtained. The minimum position in the time series change is set as the stop point.

According to a third aspect of the present invention, in the recognition method according to the first aspect, the voice is a continuous voice, and words in the continuous voice are recognized by RIFCDP.

[0009]

Embodiments of the present invention will be described below in detail with reference to the drawings.

A typical gesture classification is Ekman's classification (see Table 1).

[0011]

[Table 1]

The exemplifiers in Table 1 are categories that explicitly supplement audio content, and correspond to so-called modal combination expressions. In particular, in the case of gestures such as instruction, baton, space, and elephant, corresponding voice expressions such as "this" and "this much" are types that appear simultaneously with the gesture. The inventor of the present application has paid attention to this simultaneity, and has invented a method of searching for a corresponding portion between modals (between voice and moving image).

[0013] As a classification different from the classification in Table 1, the concept of dividing into time space (Phase) (McNeil, D: Psychol)
inguistics, Harper & Row (1987), see Table 2).

[0014]

[Table 2]

As a result of the field of cognitive psychology, it has already been reported that these phases are related to the power of fundamental frequencies of co-occurring language utterances and the timing of phonemes and the like.

In the present invention, among the features of the gestures shown in Table 2, it has been found that the gesture stops at the start and end of the gesture. Based on such knowledge, the operation stop part of the gesture image closest to the start point and the end point of the audio part is regarded as the start point and the end point of the gesture, and the start point and the end point of the gesture of the audio and the moving image are associated with each other. There is a feature.

The majority of the "exemplator" category is a collection of relatively simple linear motions. Therefore, it is assumed that (1) the temporal change of the hand position is simple acceleration and deceleration, and (2) the change of the hand motion is substantially proportional to the temporal change of the image. From this, the phrase is automatically segmented into the phases of the above-mentioned McNeil classification from the temporal change pattern of the moving image (Nagaya, Seki, Oka, "A Proposal of Gesture Traje
ctry Feature for Gesture Spotthing Recognition ", Te
chnical Report of IEICE, Vol.PRU95 (142), pp45-50. (19
95)).

FIG. 1 schematically shows an example of a time series of inter-frame difference values, a moving average thereof, and a frame image at a minimum time (codes T1, T2, T3).

In an actual speech recognition scene, it is possible to detect features such as the power of the fundamental frequency and the phoneme peak, but it is generally difficult to detect it stably. Therefore, in this embodiment, RIFCDP (Itoh, Kiyama, Kojima, Sei
ki, Oka: Reference Interval-Free Continuous Dynamic
Programming (RIFCDP) for spotting speech waves byar
itary parts of a reference pattern, IEICE Tech.Repo
A method called rt, Vol. SP95-34 (1995)) examined the relationship between time intervals and these phases at the same time as recognition from continuous speech. As a result, the following relationship was confirmed between the two timings (see FIG. 2).

(Rule 1) The gesture stroke stops immediately after or at the same time as the utterance expression as a key ends.

(Rule 2) The starting point of the gesture is the shortest stroke section including the key voice expression.

In order to verify the temporal relationship between voices and gestures, experiments were conducted using data actually collected for these voice / gesture combination expressions. What was used was a multimodal database created by RWC. This is a technique in which men and women in their late teens and early fifties express multimodal expressions, and their voices and moving images are used as a database. The number of participants is 22, the number of expressions is 25, and the experimental data of four times is stored for each subject.

In the experiment, the result almost coincided with the result of manual discrimination (association of a gesture image and audio data by visual observation) (93.4%). Disagreement cases were limited to those where the subject's behavior was clearly unnatural.

Next, an experiment was conducted to determine whether or not a gesture stroke section can be correctly determined when a key voice word is cut out. The experimental data used was the combination expression that was just imported into the movie file. The RIFCDP method was used to cut out speech words, and the time section (start point and end point) of a movie (moving image) that matches the voice input in real time was used as a key for a gesture / stroke section search. 96% accuracy was obtained for 10 subjects.

The above-described recognition method will be described more clearly. FIG. 3 shows the system configuration of the recognition device. In FIG. 1, reference numeral 1 denotes an information processing device such as a personal computer. The personal computer 1 executes the processing program of FIG. 4 to associate each multimodal multimodal (voice and gesture).

Reference numeral 2 denotes a microphone which inputs a voice uttered by the subject and outputs a voice signal to the personal computer 1. Reference numeral 3 denotes a video camera which captures a gesture of a subject and outputs moving image data (a so-called movie) in frame units to the personal computer 1 in the form of electric signals.

The personal computer 1 executes the following processing based on the processing program shown in FIG. The audio data output from the microphone 2 and the frame image output from the video camera 3 are stored in the device (step 10). FIG. 1 shows a frame image at a specific time point when a subject performs a gesture corresponding to this [No] while saying "No", and FIG. 2 shows an audio waveform (reference numeral 200). .

After acquiring the voice data and the frame image for a predetermined time, the personal computer 1 recognizes the words of the voice data by the RIFCDP method (step S20). More specifically, features are extracted from voice data, and a plurality of sets of features prepared as a standard pattern are compared in advance using a matching method called continuous DP. Speech recognition is performed by detecting the features of the standard pattern. In this matching process, the word section of the word to be recognized in the continuous voice (in this case, “No”), that is, the start position (start time, T11 in FIG. 2) and the end position (end time T12) are also detected. (Step 3
0).

Next, the personal computer 1 detects the stop position (time) of the subject's operation based on the moving image (a plurality of continuous frame images). Because of this, two adjacent
The difference between the image data at the same pixel position for one frame image is calculated (step S40). For a still portion, the difference value between the two image data is extremely close to 0 (zero).

The difference value of each pixel is summed. The sum is the above-described inter-frame difference value. 1 and FIG. 2 are obtained by taking the average of the inter-frame difference values for a certain time from a certain time point, and taking the average value as the moving average value at the above time point, and plotting the moving average value at each time point in time series. A curve denoted by reference numeral 100 is obtained. Since the movement (stroke) of the subject's hand or head stops at the gesture stop position, the inter-frame difference value is close to zero. By utilizing such a property, a position (time) at which the moving average value changes with time is determined as a gesture stop position (rule 1, step S50 described above). The calculation of the inter-frame difference value, the moving average, and the detection of the minimum point are well known and need not be described in detail.

Times T1, T2 and T3 in FIG. 2 are obtained as a result of detecting the gesture stop position. Note that only the stop position of the gesture operation can be detected from this frame image, and it cannot be determined whether the stop position is the start point or the end point of the gesture. Therefore, in the present embodiment, in the speech recognition of the word, the utterance start time of the word (the time T in FIG.
11) and stroke stop times T1, T2, and T3, respectively, to detect a stroke time having the shortest interval with respect to the word utterance start time T11. In the example of FIG. 2, the time T1 is detected as the shortest time, and the time T1 is determined as the start time of the gesture here (rule 2 described above).

Similarly, when the shortest stroke stop time is detected at the utterance end time T12 of the word, the time T2 (see FIG. 2) is obtained, and the end time of the gesture is determined (step S60). When a certain word in the voice is recognized in this way, the voice section (start time T11 and end time 12) of the word and the gesture section (start time T)
1 and the end time T2), and these time data are stored in the apparatus. If necessary, a speech recognition result is stored together with the time data.

The following embodiment can be carried out in addition to the above embodiment.

1) As a method of detecting the utterance start time and the utterance end time of the voice, in addition to the above-mentioned RIFCDP, the time when the voltage level of the voice signal is changed from the threshold value to the threshold value or more and vice versa. Methods for detecting are known.
This method can be used when the subject utters only one word. The RIFCDP method can detect a section of each word in a continuous speech in which a plurality of words (including phrases) are continuous. Therefore, when performing a plurality of gestures continuously, the speech is also continuous. RIFC
By using the DP, each gesture can be associated with a voice indicating its meaning. As a result, when a gesture is performed by a subject even when a plurality of types of gestures are performed, an operation stop occurs at a boundary between the gesture and another gesture. Therefore, according to the present invention, a plurality of continuous gestures can be recognized.

[0035]

As described above, according to the first aspect of the present invention, the start time and the end time of the gesture can be detected based on the word utterance time and the word utterance end time of the voice. It is possible to automatically associate a gesture with a voice section indicating the meaning of the gesture based on the start and end points.

According to the second aspect of the present invention, it is possible to detect the point at which the operation of the gesture is stopped by using the difference value between the frame images. In addition, by detecting the minimum portion of the temporal change in the difference value, even if a plurality of types of gestures are performed in series, it is possible to reliably detect a break in the gesture.

According to the third aspect of the present invention, since the word recognition by RIFCDP can detect the utterance section of each word in the continuous voice, it can cope with a case where a plurality of types of gestures are recognized. Can be.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a relationship between a frame image at a specific point in time and image characteristics according to an embodiment of the present invention.

FIG. 2 is a diagram showing a correspondence relationship between image features and audio according to the embodiment of the present invention.

FIG. 3 is a block diagram showing a system configuration according to the embodiment of the present invention.

FIG. 4 is a flowchart showing a processing procedure according to the embodiment of the present invention.

[Explanation of symbols]

 1 personal computer 2 microphone 3 video camera

──────────────────────────────────────────────────の Continuation of the front page (51) Int.Cl. ⁶ Identification code FI G10L 3/00 571 G06F 15/62 380 (72) Inventor Shigeki Nagaya 1-280 Higashi-Koigakubo, Kokubunji-shi, Tokyo Hitachi, Ltd. In-house (72) Inventor Kiyoshi Furukawa 1-9-4 Otemachi, Chiyoda-ku, Tokyo Keidanren Kaikan Within the Iron and Steel Federation of Japan (72) Inventor Ryuichi Oka 1-6-1-1, Takezono, Tsukuba-shi, Ibaraki Tsukuba Mitsui Building Technology Research Association New Information Processing Development Organization Tsukuba Research Center

Claims (3)

[Claims] 1. A recognition method for recognizing a gesture accompanied by a voice indicating the contents of a gesture, performing word recognition on the voice, and detecting a start time and a finish time of utterance of the word in the word recognition, respectively. Detecting a stop point at which the operation of the gesture stops based on the moving image taken of the gesture, and detecting a stop point closest to the utterance start point and the utterance end point, respectively, among the detected stop points, thereby defining the gesture as A recognition method characterized by associating a corresponding voice. 2. The recognition method according to claim 1, wherein a difference value between two adjacent frame images in a continuous frame image forming the moving image is obtained, and a difference value of the difference value in a time series is obtained. Wherein the minimum position is set as the stop time. 3. The recognition method according to claim 1, wherein the speech is continuous speech, and words in the continuous speech are converted to RIF.
A recognition method characterized by word recognition by CDP.