JPH0734229B2 - Video recognition device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION (Field of Industrial Application) The present invention analyzes a moving image in which a subject is a moving object such as movement or deformation, and extracts a feature amount of the subject. The present invention relates to a moving picture recognition device. The feature amount here is a parameter indicating the shape of part or all of the subject.

By using such a moving image recognition device, it is possible to recognize the movements of various objects. As an example, when analyzing a moving image in which a human face is a subject, the feature amount to be extracted is the amount of change in the shape of the portion (for example, eyes, nose, mouth, etc.) to be recognized in the human face. . That is, it is a change amount indicating the degree of opening of the eyes, the direction of the eyeball, the degree of opening of the mouth, and the like.

Such a moving image recognition device that extracts a feature amount from a face image is used in a communication field such as a video conference or a man-machine
It can be effectively used in various ways in the field of information processing aiming to improve the interface.

For example, a video conference system having a much narrower band than conventional can be configured by using such a moving image recognition device. That is, instead of transmitting the actual face image, by transmitting the feature amount extracted from the face image, it becomes possible to reproduce the movement of the face on the receiving side. The information amount of the feature amount of the face image is much smaller than the information amount of the actual face image. Therefore, the amount of information to be transmitted can be greatly reduced.

Further, by analyzing the movement of the face of the operator of the computer by using such a moving image recognition device, it is possible to input information to the computer by the movement of the face of the operator. As a simple example, it is possible to configure an input method in which “yes” is input when the operator shakes the face vertically, and “no” is input when the operator shakes the face horizontally. By providing such input means,
The operability of conventional computers can be greatly improved.

In addition to such face image recognition, the moving image recognition device can be used to recognize various objects. For example, it can be used for analysis of movement of animals and plants.

For the sake of simplicity, only the recognition of face images will be described below, but this does not impair generality.

(Prior Art and Problems Thereof) In a conventional face image recognition apparatus, in order to recognize an image of an object having a very complicated shape such as a face, a portion to be recognized in the face image (for example, eyes, nose) , Mouth, etc.), and the recognition processing is performed for one input face image by using the knowledge about the specified portion.

As described above, in the conventional face image recognition apparatus, since the processing is performed only on the limited information of one face image, it is necessary to prepare very detailed knowledge about a specific part of the face in advance. I had to keep it. that time,
Since the shape of a face is extremely complicated and changes flexibly, the amount of knowledge information required to accurately extract a feature amount from such a face image becomes enormous. Therefore, there has been a problem that the labor for creating this becomes enormous.

Further, since the recognition processing is performed using such a huge amount of information, there is also a problem that the time required for the processing becomes large.

Moreover, in the conventional face image recognition device, when creating knowledge about a specific part of the face, various recognition conditions are increased by limiting various conditions such as the direction and size of the face in the target image. It was For this reason, if face images under different conditions have to be recognized, a huge amount of knowledge corresponding to each condition must be created again.

(Object of the Invention) The present invention is intended to solve such a problem, and an object of the present invention is to provide a moving image recognition apparatus that can accurately recognize a moving image with less knowledge and processing time.

(Structure of the Invention) A moving image recognition apparatus of the present invention includes a moving image storage unit that stores an input moving image, a position and a shape of a plurality of portions of a subject in the moving image that are analyzed by the moving image. In the moving image recognition apparatus including a moving image analysis unit that extracts a feature amount indicating, the moving image analysis unit is cut out from the first image that has already been analyzed so as to include a plurality of portions of the subject, respectively. A plurality of first partial images and a partial image storage unit that stores cutout position information of each of these partial images; and a second image stored in the moving image storage unit that is temporally continuous with the first image. In the corresponding position determining unit that determines the position information corresponding to the plurality of first partial images in, the position of the plurality of parts in the second image by analyzing the second image based on the corresponding position information And shape A plurality of feature amounts, each of which includes a plurality of portions of the subject based on positions and shapes in the second image of the plurality of portions of the subject represented by the feature amount. Of the second partial image of the second image is determined from the second image and the partial image storage unit writes the plurality of second partial images and the cut-out position information to the partial image storage unit. ing.

(Embodiment) A moving image recognition apparatus according to an embodiment of the present invention will be described below with reference to the drawings. For simplicity of description, a case of recognizing a moving image with a face as a subject will be described, but the generality is not impaired.

FIG. 1 is a block diagram showing the entire moving image recognition apparatus as an embodiment of the present invention. As shown in FIG. 1, a moving image storage unit 2 for temporarily storing a moving image signal 102 input from the outside one frame at a time is provided and is composed of a frame memory. An image reading line 103 for reading out a necessary portion of the image P stored in the moving image storage unit 2 is provided. The moving image analysis unit 3 is provided for analyzing the image P read out via the image readout line 103 and outputting the feature amount 106 regarding the image P as the recognition result.

The moving image analysis unit 3 stores n partial images Qi (i = 1, 2, ..., N) cut out from the already analyzed image and cut-out position information of the partial images Qi. For this purpose, a partial image storage unit 4 is provided. As the partial image Qi, a portion having a feature in the face such as eyes, nose, mouth, etc. is selected.

FIG. 2 is an explanatory diagram showing the contents of the partial image storage unit 4. Further, FIG. 3 is an explanatory diagram showing an example of cutout positions (Xi, Yi) and sizes (SXi, SYi) as cutout position information of the partial image Qi stored in the partial image storage unit 4. As shown in FIG. 2, the partial image storage unit 4 stores a partial image Qi as a digital image in the partial image storage memory 4a, a cutout position (Xi, Yi) of the cutout partial image Qi, and a partial image Qi. And a cutout position information storage memory 4b for storing the size (SXi, SYi) of the partial image Qi as cutout position information.

FIG. 6 is an explanatory diagram showing an example of the input moving image. When a moving image as shown in FIG. 6 is input, first,
A partial image Qi is determined from the first face image P1 and stored in the partial image storage unit 4. To determine the partial image Qi in the first face image P1, a person refers to the face image P1 and determines the size (SXi, SYi) of the cutout position (Xi, Yi) of each partial image Qi. It is executed manually. The cut out partial image Qi is transferred via the initial partial image writing line 108.
It is written in the partial image storage unit 4.

When it is difficult to perform such manual work, a standard partial image is stored in advance in the partial image storage unit 4 as initial data via the initial partial image writing line 108, and this embodiment described below is performed. Using the corresponding position determination unit 5, the feature amount extraction unit 6, and the partial image cutout unit 7 of the face image P1
It is also possible to cut out the partial image most similar to this standard partial image from among the partial images Qi. Also,
The position of each part of the face may be recognized using only the first face image P1 by using the conventional face image recognition device, and the partial image Qi may be cut out. The partial image Qi for the face image P1 cut out in this way is stored in the partial image storage unit 4.

In this way, after the partial image Qi cut out from the face image P1 is stored in the partial image storage unit 4, the second face image P2
Is stored in the moving image storage unit 2 as the moving image signal 102.

Then, the partial image Qi stored in the partial image storage unit 4 is compared with the face image P2 stored in the moving image storage unit 2, and the corresponding position (X'i, Y) of the partial image Qi on the face image P2 is compared. ′ I)
Corresponding position determining unit 5 is provided for determining. The corresponding position (X'i, Y'i) is the position of the partial image determined to have the highest correlation with the partial image Qi on the face image P2.

FIG. 4 is a block diagram showing the details of the corresponding position determining unit 5. As shown in FIG. 4, the corresponding position determination unit 5
It is connected to the partial image storage unit 4 via the partial image read line 104. At the same time, from the corresponding position determination unit 5,
The determined corresponding position (X′i, Y′i) and the partial image size (SXi, SYi) are output as corresponding position information 105.

In order to calculate such corresponding position information 105, the corresponding position determining unit 5 includes a partial image storage 5a and a correlation calculator 5b.
A best position selector 5c, a best position memory 5d, and a search range setting device 5e are provided.

FIG. 7 is an explanatory diagram showing an example of the processing performed by the corresponding position determination unit 5. As shown in FIG. 7, the search range setting unit 5e is preset with a value (DX, DY) indicating the width of the search range of the corresponding position (X'i, Y'i). This value (D
X, DY) is output to the best position selector 5c via the connection line 114.

In this best position selector 5c, first, with this value (DX, DY),
The position (WX, WY) and size (WSX, WSY) of the search range W are calculated and stored from the cut-out position information of the partial image Qi. However, the cut-out position information of the partial image Qi is read from the partial image storage unit 4 through the partial image reading line 104.

These values are calculated as follows.

WX = Xi-DX WY = Yi-DY WSX = DX + DX + SX WSY = DY + DY + SY The position (WX, WY) and size (WSX, WSY) of the search range W thus obtained are stored as partial images via the connection line 112. vessel
It is output to 5a. The partial image storage unit 5a reads a part of the face image P2 indicated by these values from the moving image storage unit 2 via the image reading line 103 and stores it.

Next, in the best position selector 5c, the best corresponding position (X'i,
The candidate position (X ', Y') of Y'i) is set to the position (WX, WY) of the search range W. The candidate position (X ′, Y ′) and the size (SXi, SYi) of the partial image Qi are output to the correlation calculator 5b via the connection line 111.

In the correlation calculator 5b, the partial image Q'i having the size (SXi, SYi) at the candidate position (X ', Y') is the connecting line 110.
It is read from the partial image storage device 5a via. at the same time,
The partial image Qi is a partial image read line from the partial image storage unit 4.
Read through 104. Then, the correlation between the partial image Q'i and the partial image Qi is calculated, and the value C indicating the correlation is output to the best position selector 5c through the connection line 111.

To calculate the correlation value C, the brightness of the pixel at the position (X, Y) in the partial image Qi is set to f (X, Y), and the brightness of the pixel at the position (X, Y) in the partial image Q′i is calculated. If f '(X, Y), it is calculated as follows.

In response to the correlation value C calculated in this way, the best position selector 5c calculates the correlation value C and the candidate position (X ', Y') as
It is output to the best position memory 5d through the connection line 113. In the best position memory 5d, the correlation value C and the candidate position (X ',
Y ') is stored.

Next, in the best position selector 5c, the candidate position (X ',
By increasing the Y coordinate value of Y ′) by 1, the correlation calculator 5
With b, a new correlation value C'is calculated. If the correlation value C'is smaller than the correlation value C already stored in the best position memory 5d, the best position memory 5d stores the candidate position (X ', Y') and the newly calculated correlation value. C'and are written.

Further, in the best position selector 5c, the X and Y coordinate values of the candidate position (X ', Y') are changed by one to change all of WX≤X'≤WX + WSX-SXi WY≤Y'≤WY + WSY-SYi. Similar processing is performed for the candidate position (X ', Y').

By performing such processing, the candidate position (X ′, Y ′) when the correlation value C ′ finally becomes the smallest in the search range W is the corresponding position (X ′ of the partial image Qi on the face image P2). ′ I, Y′i) and stored in the best position memory 5d. It can be said that the partial image Q'i at the corresponding position (X'i, Y'i) is the image most similar to the partial image Qi in the search range W.

Since the face image P1 and the face image P2 that are cut out of the partial image Qi are images at different times, some deformation or movement occurs in the face in the image. However, the face image P1 and the face image P2 are
Since the moving images are images that are temporally continuous, only a slight change should occur between them. Therefore, the corresponding position (X′i, Y ′) having the highest correlation with the partial image Qi
Since the partial image Q′i in i) includes a part similar to the part of the face included in the partial image Qi, the corresponding position determining unit 5 determines that the face image You can keep track of specific parts of the.

Finally, from the corresponding position determining unit 5, the corresponding position (X'i, Y'i) obtained by the above process and the size (SXi, SYi) of the partial image Qi are obtained as the corresponding position information 105. Is output.

FIG. 5 is a configuration diagram showing details of the feature quantity extraction unit 6.
As shown in FIG. 5, using the corresponding position information 105 output from the corresponding position determination unit 5, the face image P2 stored in the moving image storage unit 2 is analyzed, and the feature amount 106 related to the face as a subject is analyzed. A feature amount extraction unit 6 for outputting is provided.

The feature amount extraction unit 6 is provided with an edge extraction unit 6a, an edge image analyzer 6b, and a partial information storage unit 6c.

FIG. 8 is an explanatory diagram showing the operation of the feature quantity extraction unit 6. Corresponding position information 105 output from corresponding position determining unit 5
In the feature quantity extraction unit 6 having received the data, the size at the corresponding position (X′i, Y′i) on the face image P2 is first passed through the image read line 103 in the edge extractor 6a, as shown in FIG. An image R centering on the partial image Q'i of (SXi, SYi) and having a size twice as large as the partial image Q'i is referred to.

In this way, the image R having a double size is referred to because the object included in the partial image Qi is
This is because, as shown in the figure, the partial image Q′i may sometimes run off. Even in the case of such an overflow, an object can be included inside by referring to the image R having a size twice as large.

The thus-referenced image R is subjected to the second-order differential operation processing in the edge extractor 6a, and the edge image 115 of these images is extracted. An example of such a second-order differential calculation process is described in the Institute of Electronics and Communication Engineers, Volume 56-D, Vol.
, Toshiyuki Sakai, Makoto Nagao, Takeo Kanade, "Analysis of facial photographs by computer".

Next, in the edge image analyzer 6b, this edge image 11
5 is analyzed. At that time, the partial information 116 about the face portion included in the partial image Qi is read from the partial information storage unit 6c and used. This Partial Information 116
Thus, the feature amount 106 to be extracted is determined.

For example, when the partial image Q'i is an image including eyes,
As the partial information 116, information regarding the shape of the eyes and the method of deformation is set. The analysis is performed according to the partial information 112, and as the feature amount 106, the size of the eyes (EWX, EWY), the center position (EX, EY), the direction of the eyes, the degree of opening, etc. as shown in FIG. Is extracted.

The processing performed by the feature amount extraction unit 6 is the same as that of the conventional face image recognition apparatus. An example of this conventional face image recognition apparatus is also described in the above-mentioned paper,
It is shown as a method of analyzing a facial photograph by a computer.

However, since the size of the image to be processed is limited to the partial image Q'i of the face image P2 and its surroundings, the processing can be performed faster than before. Further, since it is known which part of the face the partial image Q′i is, the knowledge about the part, that is, the description of the partial information 116 becomes easy.

As a result of such image processing, the feature amount extraction unit 6
106 is output. The extracted feature amount 106 includes parameters related to the shape, size, and position of each part of the face (eyes, nose, mouth, etc.) and parameters related to the face orientation.

By using the feature quantity 106 extracted in this way, a new partial image Qi from the face image P2 stored in the moving image storage unit 2 is obtained.
(I = 1,2, ..., n) is determined, and is stored in the partial image storage unit 4.
A partial image cutout unit 7 is provided for storing the partial image Qi and the cutout position information of the partial image Qi. The partial image cutout unit 7 and the partial image storage unit 4 are connected via a partial image writing line 107.

FIG. 9 is an explanatory diagram showing the operation of the partial image cutout unit 7. As shown in FIG. 9, in the partial image cutout unit 7, when a new partial image Qi is cut out from the face image P2, the size (SXi, Y′i) at the corresponding position (X′i, Y′i) is obtained. When the image Q′i of SYi) is cut out, the partial image Q′i may come into contact with an object forming an actual face, as shown in FIG.
As shown in FIG. 6B, the object may be much larger than the object in the face image.

Therefore, in the partial image cutout unit 7, the face portion is surrounded as shown in FIG. 9C from the position and size of the face portion indicated by the feature amount 106 input from the feature amount extraction unit 6. The position and size of such an image are obtained and cut out as a new partial image Q ″ i. For example, in the case of an eye as shown in FIG. EX, EY) and size (EWX, EWY) are used, that is, as the partial image Q ″ i to be cut out,
The center position is (EX, EY) and the size is (αX * SX, αY
The cutting position information is calculated so that * SY). These coefficients αX and αY are values preset in the partial image cutout unit 7.

Then, based on this cut-out position information, the partial images Q ″ i are read out via the image reading line 103. These partial images Q ″ i are parted together with the cut-out position information via the partial image writing line 107. It is written in the image storage unit 4.

In this way, by using the partial image Qi newly cut out from the face image P2 and written in the partial image storage unit 4, the third face image P3 stored in the moving image storage unit 2 is displayed. Processing is performed. Then, by sequentially repeating this processing, it is possible to continuously and easily extract the feature amount from the face image as the moving image. At this time, since the processing result of the image just before the processing is already used is used as the partial image Qi, the processing becomes accurate and the processing time is shortened.

(Effects of the Invention) As described in detail above, according to the moving image recognition apparatus of the present invention, the input moving image is recognized by fully utilizing the fact that only a slight change occurs between consecutive images. It is processing.

Therefore, the moving image recognition apparatus of the present invention has an advantage that the recognition processing time of the input moving image can be significantly shortened and also has an advantage that the recognition processing can be executed more accurately than before.

Further, the device of the present invention says that the recognition process is easy even when the whole subject to be processed moves or rotates in the moving image or various parts of the subject deform. There are also advantages.

[Brief description of drawings]

FIG. 1 is a configuration diagram showing an embodiment of the present invention, FIG. 2 is a diagram showing contents of a partial image storage unit in the embodiment, and FIG. 3 is a diagram showing an example of a cutout position and a size of a partial image. ,
FIG. 4 is a block diagram showing a concrete example of the corresponding position determining unit in the embodiment, FIG. 5 is a block diagram showing a concrete example of the feature amount extracting unit in the embodiment, and FIG. 6 is shown in FIG. FIG. 7 is an explanatory diagram showing an example of a moving image to be input, FIG. 7 is a diagram explaining an example of processing performed by the corresponding position determining unit in FIG. 4, and FIG. 8 is an operation of the feature amount extracting unit in FIG. 9A to 9C are diagrams for explaining the operation of the partial cutout portion in the embodiment of FIG. 2 ... moving image storage unit, 3 ... moving image analysis unit, 4 ... partial image storage unit, 4a ... partial image storage memory, 4b ... cutout position information storage memory, 5 ... corresponding position determination unit, 5a ......
Partial image memory, 5b ... correlation calculator, 5c ... best position selector, 5d ... best position memory, 5e ... search range setting device,
6 ... Feature amount extraction unit, 6a ... Edge extractor, 6b ... Edge image analyzer, 6c ... Partial information storage unit, 7 ... Partial image cutout unit, 102 ... Moving image signal, 103 ... Image Read line, 104 ... Partial image read line, 105 ... Corresponding position information, 106 ... Feature amount, 107 ... Partial image writing line, 108
…… Initial partial image writing line, 110 to 114 …… Connecting line, 11
5 …… Edge image, 116 …… Partial information.

Claims (1)

[Claims] 1. A moving image storage unit that stores an input moving image, and a moving image that analyzes the moving image and extracts feature quantities indicating positions and shapes in the moving image of a plurality of portions of the subject. In a moving image recognition device including an analysis unit, the moving image analysis unit, a plurality of first partial images cut out to include a plurality of portions of the subject from the already analyzed first image and The partial image storage unit that stores the cut-out position information of each of these partial images, and the plurality of first partial images in the second image stored in the moving image storage unit that is temporally continuous with the first image. Corresponding position determining unit that determines position information corresponding to, and extract a feature amount indicating the position and shape of the plurality of portions in the second image by analyzing the second image based on the corresponding position information. Feature extraction Parts and the plurality of second partial images of the plurality of second partial images based on the positions and shapes of the plurality of portions of the subject represented by the feature amount in the second image. A moving image characterized by being configured with a plurality of second partial images and a partial image cutout unit for writing cutout position information in the partial image storage unit by determining cutout position information from the second image Recognition device.