JP2006127541A - Gesture recognition method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To permit recognition of gestures of a plurality of persons by speeding up gesture recognition processing. <P>SOLUTION: With a multidirectional visual sensor, a plurality of photographic subjects are imaged, individual photographic subject image is picked up from the imaging result, and gesture recognition is performed with an image processor. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a gesture recognition method and apparatus for recognizing a gesture of a subject by photographing the subject with an imaging device and analyzing a feature pattern of the photographed image in an image processing device.

  The technology for recognizing human gestures is important in building a flexible Man-Mchine Interface System (Man Machine Interface System) (Takao Kurokawa, Nonverbal Interface, Ohm, 1994). In particular, gesture recognition using moving images that capture human movements without attempting to attach contact sensors or markers such as data gloves to the operator has been attempted (Katsuhiko Takahashi, Susumu Seki, Hiroshi Kojima, Ryuichi Oka , Spotting recognition of gesture moving images, theory of theory (D-II), Vol. J77-D-Iino.8, pp1552-1561, 1994.).

  So far, gesture recognition of one person has been attempted, but in recent years, multi-modal dialogue between multiple persons and computers (Ito Yoshiaki, Kiyama Jiro, Seki Susumu, Kojima Hiroshi, Book Kenshin, Oka Ryuichi, simultaneous multiple (Novel Interface System, spoken language processing 7-3, pp 17-22, 1995) based on real-time integrated understanding of conversational voices and gestures of dialogue persons is important. That is, there is a need for a system that recognizes and understands a dialogue between gestures and voices between a plurality of persons and provides information necessary for facilitating the dialogue from a database. For that purpose, it was necessary to recognize gestures and voices of a plurality of persons simultaneously in real time. Integration has been realized for voices between multiple persons and gesture recognition of a single person. Nagaya et al. (Shigeki Nagaya, Seki Susumu, Ryuichi Oka, Gesture recognition by multi-resolution features, IEICE Technical Report, PRU95-99, pp121-126 and Shigeki Nagaya, Seki Susumu, Ryuichi Oka, Motion trajectory features for gesture recognition , IEICE Technical Report, NLC95-37, PRU95-142, pp45-50), a method for specifying the positions and number of operators is proposed. However, the realization of a real-time gesture recognition system for a plurality of persons has not yet been realized.

  The first problem in realizing such a system is whether to prepare a camera and a recognition system equal to the number of human subjects. Because, as described in the above-mentioned report of spotting recognition of gesture moving images, a general-purpose image processing board (Imaging Technology Series 150/40) and a workstation ( This is because the burden of using (Iris Crimson) is forced.

  Furthermore, it is extremely inconvenient to capture a specific person for each camera, and there are situations in which it is even more difficult to capture them when dialoguers look at each other's faces.

  As a conventional technique, several studies on the gesture recognition of a single person have been reported. Yamato et al. (J. Yamato, J. Ohya, K. Isii, Recognizing Human Action in Time-Sequential Images Using Hidden Markov Model, Proc. CVPR, pp 379-385, 1992) A motion recognition method using the Markov Model has been proposed. In this method, it is possible to learn an action model, but it is necessary to manually segment the time space in which the action is performed.

  Darrell et al. (TJ Darell and AP Pentland, Space-Time Gestures, Proc. IJCAI '93 Looking at People Workshop (Aug. 1993)) expresses the movement of the palm as a transition sequence of its appearance. A plurality of gestures were recognized by associating the observed appearance series with the behavior model by using Dynamic Time Warping. However, in this proposal, the background of the input image needs to be plain, and the method of temporal segmentation between the same works is not shown.

  In addition, Ishii et al. (H. Ishi, K. Mochizuki and F. Kshino, A Motion Recognition Method from Stereo Images for Human Image Synthesis, 180-97-TheD. 08)) obtains a three-dimensional position of a skin color portion such as a hand or a face by color image processing and stereo matching, and measures the movement amount. However, motion recognition at the video rate has not been realized even in the systems of Ishii and Darell et al. Using a dedicated image processing apparatus.

  When simultaneously recognizing gestures of a plurality of persons, it is necessary to satisfy the following two constraints in order to make the use environment natural and favorable.

Constraint 1 The ability to use a single camera that guarantees an environment where multiple people can interact in a natural way. This is because, for example, when shooting multiple people with a single camera, the multiple people are in a line with their faces in front. This must be a natural environment. For example, if you try to shoot three people facing each other from the side, you will not be able to shoot the gestures of the two people with the camera.

Constraint 2 Recognize gestures of multiple people in real time This is not limited to gesture recognition of multiple people, but as the number of recognition targets increases, the recognition result is obtained in real time unless the gesture recognition process is performed at high speed. It is not possible.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a communication photographing method and apparatus, an image processing apparatus, and a gesture recognition method and apparatus suitable for photographing gestures of a plurality of persons facing each other with a single camera.

  In order to achieve the object of the present invention, the invention of claim 1 is characterized in that a communication state is photographed by a single omnidirectional visual sensor installed at a position where a plurality of persons can be imaged.

  According to a second aspect of the present invention, there is provided a single omnidirectional visual sensor installed at a position where a plurality of persons can be imaged, and the state of communication is photographed by the omnidirectional visual sensor.

  The invention of claim 3 divides an area in which each person appears from image data obtained by capturing a communication state by a single omnidirectional visual sensor installed at a position where a plurality of persons can be imaged. It is characterized by processing.

  The invention according to claim 4 divides the area in which each person is shown from the video data that captures the state of communication by a single omnidirectional visual sensor installed at a position where a plurality of persons can image. It is characterized by recognition.

  According to the invention of claim 5, a single omnidirectional visual sensor installed at a position where a plurality of persons can be imaged, and video data obtained by capturing the state of communication by the omnidirectional visual sensor are displayed. And a gesture recognizing unit for recognizing a gesture of a person shown in the divided area.

  According to the first to fifth aspects of the present invention, since the gesture images of a plurality of subjects are captured without overlapping by the omnidirectional visual sensor, the images of the individual subjects can be acquired by dividing the imaging results. As a result, gestures of a plurality of subjects can be recognized even with a single imaging device.

  According to the first to fifth aspects of the invention, it is possible to recognize all the gestures of a plurality of humans arranged in a circle, for example, to recognize a conversation in sign language, to recognize the contents of a market message, and to communicate the recognition result remotely. The local people can be informed of the contents of the place.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. This embodiment has a first feature in that a plurality of subjects are imaged using an omnidirectional visual sensor.

  An omnidirectional visual sensor is an imaging device (see reference numeral 106 in FIG. 13) that guides an omnidirectional image to a CCD camera via a hyperboloidal mirror (conical mirror). Recently, it has been proposed to be mounted on a mobile robot. (Kazumasa Yamazawa, Yasushi Yagi et al., Proposal of Hyper Omni Vision for Navigation of Mobile Robots, Theory of Science (D-II), Vol. J79-D-II, no5, pp698-707, 1996). This omnidirectional visual sensor is applied to gesture recognition. A second feature is that saturation processing (described later) is performed when feature patterns are extracted.

  FIG. 1 shows an installation example of an omnidirectional visual sensor. Reference numeral (a) indicates that the omnidirectional visual sensor 1 placed in the center in an environment during a meeting on a round table captures gestures of all attendees in one image. Reference numeral (b) indicates that the omnidirectional visual sensor 1 is installed above the autonomous mobile robot 3. Gestures of a plurality of persons around the autonomous mobile robot 3 are photographed in one image. For reference, an image taken by the omnidirectional visual sensor 1 is shown in FIG.

  The image acquired in this manner is divided for each subject in the image processing apparatus, and then a feature pattern of the gesture is extracted from each subject image. This extraction process will be described in comparison with a conventional method. FIG. 2 shows a conventional processing process. FIG. 3 shows the processing process of the present embodiment. For example, the imaging device acquires image data of i × j pixels at time t.

  When an input image I (i, j, t) at a certain time t is obtained, a time difference image from the input image I (i, j, t-1) at the previous time t-1 is created. When this time difference image is compared with a threshold value, a binary image Ib (i, j, t) (0 ≦ i, j <L, 0 ≦ t) is obtained. When this is expressed by a mathematical formula, the following formula 1 is obtained.

Here, hc is a threshold value that determines whether the pixel value has changed. Further, the binary image Ib (i, j, t) is spatially reduced to N ₂ × N ₂ to obtain a feature vector f (k, v, t) (0 ≦ k, v <N ₂ ).

Here, p and q are both integers, and h = N ₁ / N ₂ . This feature vector f (k, v, t) indicates the rate at which the pixel value changes in each region in the size N ₂ × N ₂ reduction image, that is, the rate at which the body part of the subject moves. The feature vector f (k, v, t) is averaged over three frames, and the logarithm thereof is a feature pattern used for gesture recognition.

  The example of FIG. 2 shows an example in which an input image of 64 × 64 pixels is compressed into 16 × 16 data.

  On the other hand, in the present embodiment, among the data of each area subjected to spatial reduction, that is, the value of the feature vector, a value larger than the threshold value hm is saturated and held to the threshold value. This is expressed by the following equation.

  In this way, the feature pattern at time t is obtained, and the same processing is performed at the next time t + 1 to extract the feature pattern. For the standard pattern used for gesture recognition, the feature pattern is extracted by the same process.

  In the gesture recognition, matching is performed with a plurality of sets of feature patterns in time series of the standard pattern and a plurality of sets of feature patterns extracted from the video of the subject to be recognized. As this matching method, a method called continuous DP is well known.

  In spotting recognition by continuous DP, feature extraction is performed from an input image as described above to obtain a feature vector. Next, the input feature vector sequence is matched with a standard pattern corresponding to each gesture by continuous DP. This standard pattern is a feature vector sequence created by a feature extraction method similar to that of an input image from an image sequence obtained by capturing standard operations in advance. Finally, the results of continuous DP matching with all the standard patterns are compared, and the best match is output as the matching result at that time.

Hereinafter, a feature vector sequence matching method based on continuous DP will be specifically described. The cumulative distance S (t, τ) is appropriately set as follows, where d (t, τ) is the distance between frames of the input feature pattern and the standard pattern.
Initial conditions:

  Here, t represents an input discrete time, τ is a parameter corresponding to the length of the standard pattern, and 1 ≦ τ ≦ T (T is a pattern length). The output A (t) of the continuous DP is

Determine as

  As an experimental device for performing such gesture recognition, Indy (R4400 200 MHz) manufactured by SGI and an attached camera called IndyCom were used. The experiment was performed on one subject sitting on a chair in the office. The field of view of the camera was set so that the subject's gesture entered properly. For the lighting, only fluorescent lamps installed on the ceiling of the building were used.

An image obtained by AD conversion of the output image of the CCD camera is an RGB image having a size of 160 × 120 and one pixel and 256 gradations, but only a green component that has a relatively strong influence on luminance is used for recognition. This image was spatially reduced and an image of size N ₁ × N ₁ was used as an input to the feature extraction unit. Further, hm for saturating the feature vector of Equation 3 was set to 0.3.

  The gestures used in the experiment are: (1) Banzai (both hands), (2) Bye Bye (right hand), (3) Maru (both hands), (4) Clapping hands (both hands), (5) To here (right hand), There are eight types: (6) left (left hand), (7) right (right hand), (8) no (right hand). This is expressed as a gesture v (v = 1, 2,..., 8). FIG. 4 shows a snapshot of each gesture, and FIG. 5 shows an image series of the gesture “Banzai”. The subject performed each operation at normal speed, and the image was sampled at 15 Hz. Further, the threshold value hc in the equation 1 is set to 10 in consideration of the thermal noise of the camera.

  A standard pattern v (v = 1, 2,..., 8) was created by manually cutting out only a gesture portion from an image series capturing each gesture. The frame length T of the standard pattern used in this experiment was 11 to 15. Moreover, the input image sequence v which repeated the same gesture 20 times was created. Next, the input image sequence v was input to the recognition system, and the first recognition rate and the correct candidate rate were obtained.

  Here, the number v of correct answer gestures is the number of gestures that can be correctly recognized among the 20 gestures in the input image sequence v. In addition, it is assumed that “detection” is made when the same recognition result is obtained continuously for three frames or more.

Here, assuming that the input image size N ₁ = 64, the optimal value of the dimension number (N ₂ × N ₂ ) of the feature vector is obtained, so that N ₂ = {1, 2, 3, 4, 5, 7, 10, 16}. Using the optimum value obtained here, the input size was changed to N ₁ = {3, 6, 9, 12, _{15, 30,} 64} to obtain the optimum value. Here, to investigate the effects of clothing and background,
S1 When the standard pattern is created and the clothes and the background are the same S2 When the standard pattern is created and when the brightness of the clothes and the background are different (FIG. 6). The color of the clothes was gray when S1 and yellow when S2. The standard pattern was created in the case of S1, and the threshold value hv was manually set so that the first-rank recognition rate of S1 was as large as possible. In S2, a recognition experiment was performed using the standard pattern and threshold value created in S1.

The result of the recognition experiment is shown in FIG. Even when the clothes and the background are different (S2), a high first-order recognition rate of about 80% was obtained with N ₂ = 3, 4, and 5, indicating that this method is robust to changes in clothes and background. .

An error of about 20% occurs when the clothes and background are different. (1) Differences in how the clothes are wrinkled, (2) Differences in hand shadows, and (3) Differences in the size of the person due to swelling. Considering the amount of calculation, when N ₂ is 3, it is an optimal recognition system for the eight types of gestures used this time. In addition, when N ₂ is 7 or more, the first-order recognition rate is lowered, but this is considered to be because the reduction size is too large to absorb the fluctuation of the motion trajectory.

Next, Table 1 shows the recognition results when N ₁ is changed with N ₂ = 3 fixed.

From this result, the recognition rate is about 80% when 12 ≦ N ₁ , and the recognition rate is lowered when N ₁ ≦ 9. When N ₂ = 12, the feature vector value is N ₁ / N ₂ = 12/3 = 4, which is 4 × 4 = 16 stages, and the recognition rate deteriorates because it is sufficient for recognition by continuous DP. Probably not.

From this result, it was shown that gestures can be recognized with a high recognition rate even from a small human image of N ₁ = 12.

  A recognition experiment of multiple persons was also conducted. In the experiment, an omnidirectional visual sensor was installed on an autonomous mobile robot (see FIG. 1B), and four subjects sitting on a chair were placed around the autonomous mobile robot. The writing subject faces the autonomous mobile robot and makes a gesture. Experimental conditions such as the light source, image size, threshold value, and the like were the same as those in the above recognition experiment. An example of the image of the omnidirectional visual sensor at this time is schematically shown in FIG.

The four persons are shown as shown in FIG. For segmentation of each person, the person range was equally divided into 3 × 3, and feature extraction was performed using pixels having a center of gravity in each divided area. As shown in FIG. 8, since the size of the person differs depending on the distance to the omnidirectional visual sensor, the input size N ₁ × N ₁ to the feature extraction unit b of each person is different. The distance to the farthest person (reference numeral 51) was 4 m, and the image size of the person at that time was 18 × 15. A 3 × 3-dimensional feature vector was calculated from the input image by the above-described feature pattern extraction method. Note that the distortion of the omnidirectional visual sensor is not corrected.

  The gestures used for the experiment were the same as the above-mentioned experiment, and eight types were used. FIG. 10 shows an image series for every three frames of the gesture “Banzai”. The input image series was shot with four people in the same clothes as when shooting the standard pattern. The number of frames of this input image series was 457, and the gestures performed by four people during this period were 10 to 13 times.

  Table 2 shows the recognition rate of each person.

  Although the clothes and the background are the same as those at the time of creating the standard pattern, they can be recognized with a high recognition rate of about 80%, indicating that the gesture recognition method of the present embodiment is effective.

Furthermore, FIG. 11 shows an example of the output value of continuous DP for four person actions. The horizontal axis is the number of frames, and the horizontal line drawn above shows the gesture actually performed by the subject and the time interval. The CDP (continuous DP) output on the vertical axis is a value obtained by subtracting the respective threshold values. Therefore, it is recognized when the value of the CDP output becomes negative. It can be seen from FIG. 11 that the CDP output of the appropriate standard pattern is decreased even in the case of recognition failure.
A real-time gesture recognition system using this method was created using one Indy. FIG. 12 shows its appearance.
N ₁ = 12, N ₂ = 3, the sampling rate is 15 Hz, the input image is displayed and recognized in real time, and the result is displayed. As a result of real-time recognition experiments, a recognition rate of about 70% was obtained even if the clothes and background were different.

  FIG. 13 shows a system configuration of the present embodiment. In FIG. 13, a CPU 100 controls the entire system based on a system program in a system memory 101 and executes a gesture recognition process according to the present invention in accordance with a gesture recognition program loaded in the system memory 101.

  The system memory 101 includes a ROM and a RAM, and stores the system program, parameters used for system control, input / output data for the CPU 100, and an image to be displayed on the display 102. The display 102 displays the captured image input from the omnidirectional visual sensor 106. In addition, information input from the input device 105 described later, a gesture recognition result, and the like are displayed.

  A hard disk storage device (HDD) 103 stores a gesture recognition program for storage (FIGS. 14 and 15) and a standard pattern used for gesture recognition. The standard pattern is obtained by performing a gesture in which the movement of one subject is known in advance and using the above-described feature pattern extraction method from the captured image. The standard pattern is composed of characteristic patterns with different gestures and identification codes indicating the corresponding gesture contents.

  An input / output interface (I / O) 104 is connected to the omnidirectional visual sensor 106 and delivers a captured image to the CPU 100. The input device 105 has a keyboard and a mouse and inputs information.

  In such a system configuration, gesture recognition processing to be executed will be described with reference to FIGS. When activation of the gesture recognition process is instructed by the input device 105, the CPU 100 reads a gesture recognition program from the HDD 103, loads it into the system memory 101, and starts execution.

  First, the CPU 100 initializes various parameters used in the gesture recognition process (step S10). The CPU 100 captures a captured image (also referred to as a frame) for one screen via the I / O 104, divides a plurality of subjects into images, and temporarily stores them in the system memory 101 (step S20).

  Next, the second captured image is stored in the system memory 101 in which the image is similarly divided. The CPU 100 creates an image of the first subject in the first acquired frame and a time difference image of the first subject in the second acquired frame, and stores the creation result in the system memory 101 (step S30). → 40).

  The CPU 100 compares the time difference image with the threshold value for binarization and converts it into bit 1/0 data. Assuming that one screen is composed of 16 × 16 pixels, the 16 × 16 pixel group is divided into four regions, that is, four vertically and horizontally divided according to a compression rate prepared as a parameter in advance. Divide into areas. Thus, 8 × 8 pixels are included in one area. The CPU 100 counts the number of bits 1 included in this one area. Similarly, the number of bits 1 is counted for all areas. It is assumed that 1,201, 100, 59 are obtained as this number. As a result, 16 × 16 multi-value (for example, 16 bits) image data is spatially compressed into four 16-bit data (step S60).

  Next, the CPU 100 compares the spatially compressed data (1, 201, 100, 59) with the threshold value 150 individually. As a result, the value 201 exceeding 150 is converted to the same value as the threshold value, and the number of bits 1 is regarded as 150. Therefore, the spatially compressed data after such saturation processing is (1, 150, 100, 59). This spatially compressed data is stored in the system memory 101 as a feature pattern of the gesture of the first subject at time t.

  The CPU 100 performs pattern matching of the standard pattern using the continuous DP method, the acquired feature pattern, and the feature pattern acquired at the previous time. The pattern matching process for gesture recognition when there is only one subject has been briefly described above, but is well known and will not require detailed description (step S80).

  As a result of pattern matching, when it is determined that the pattern is similar to a specific standard pattern, the identification code is displayed on the display 102 (step S90 → S100).

  Thereafter, gesture recognition of another subject is executed for the frame at time t by the loop processing of steps S40 to S110. When the gesture recognition processing for all subjects at time t is performed in this way, the captured image of the omnidirectional visual sensor 106 at time t + 1 is next captured, and gesture recognition processing is performed as described above (step S30). Loop processing of S120).

  The processing procedures in FIGS. 14 and 15 are terminated in response to the termination instruction from the input device 105.

In addition to the embodiments described above, the following embodiments can be implemented.
1) In the above embodiment, when the image is divided into individual subjects, the division position is informed manually, but the image processing apparatus can automatically perform the image division. As an example, the still image portion is removed from the time difference image. The removed portion appears as bit 0 in the binarized image. If this property is used, a gap between the subject and another subject is a still image, and this gap portion is a set of bits 0 in the binary image. Therefore, a pixel group in which bit 0 continues from the center position of the image to the outer edge of the image is detected by the image processing apparatus. Since this pixel group becomes a delimiter for each subject, image data delimited by the delimiter is extracted.
2) The omnidirectional visual sensor uses a mirror that guides the subject image to the sensor. However, the shape of the mirror is not limited to a cone, and various types known so far can be used.
3) In the processing procedure of FIG. 13, the image division processing for each subject is performed on the input image from the omnidirectional visual sensor 106, but the binarization processing is performed for the entire screen, and the binary image is processed for each subject. Image segmentation can also be performed.
4) In this embodiment, the number of pixels of the input image and the number of data after spatial reduction are fixedly used, but can be variably set to arbitrary values. In this case, a desired value may be input from the input device 105 and stored in the HDD 103 as a parameter.

It is a perspective view which shows the example of installation of an omnidirectional vision sensor. It is explanatory drawing which shows the image processing process when not performing a saturation process. It is explanatory drawing which shows the image processing process in the case of performing a saturation process. It is explanatory drawing which shows a time-sequential imaging result. It is explanatory drawing which shows a time-sequential imaging result. It is explanatory drawing for demonstrating the standard pattern from which clothes and a background differ. It is explanatory drawing which shows the relationship between a reduction size and a recognition rate. It is a photograph which shows the imaging result of the omnidirectional visual sensor used as the image processing target of a computer. It is the photograph which shows typically the position of the to-be-photographed object in the imaging result of an omnidirectional vision sensor. It is explanatory drawing which shows the gesture image of Banzai. It is explanatory drawing which shows the output value of continuous DP. It is a front view which shows the front external appearance of a real-time gesture recognition system. It is a block diagram which shows the system configuration | structure of a real-time gesture recognition system. It is a flowchart which shows the feature pattern extraction process procedure and the gesture recognition process procedure. It is a flowchart which shows the feature pattern extraction process procedure and the gesture recognition process procedure.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Omnidirectional visual sensor 2 Dialogue person 3 Autonomous traveling robot 100 CPU
101 System memory 103 HDD
104 I / O
105 Input device 106 Omnidirectional visual sensor

Claims (5)

  A communication photographing method, wherein a communication state is photographed by a single omnidirectional visual sensor installed at a position where a plurality of persons can be photographed.   A communication photographing apparatus having a single omnidirectional visual sensor installed at a position where a plurality of persons can be imaged, and photographing a state of communication by the omnidirectional visual sensor.   A feature is that image processing is performed by dividing a region in which each person is shown from video data that captures the state of communication by a single omnidirectional visual sensor installed at a position where a plurality of persons can be imaged. Image processing device.   A feature is that gesture recognition is performed by dividing an area where each person is shown from video data that captures the state of communication using a single omnidirectional visual sensor installed at a position where multiple persons can be imaged. Gesture recognition method. A single omnidirectional visual sensor installed at a position where multiple persons can image;
A dividing means for dividing an area in which each person is shown out of the video data capturing the state of communication by the omnidirectional visual sensor;
A gesture recognition device comprising gesture recognition means for recognizing a gesture of a person shown in the divided area.

Images