JPH03252780A - Feature quantity extracting method - Google Patents

Abstract

PURPOSE:To efficiently detect the rotating amount of a head part by detecting feature quantity from an image to which an address where effective feature quantity exists in every hierarchy from a low resolution image to a high resolution image sequentially is inputted by limiting. CONSTITUTION:An portrait image is fetched from a camera input part 1, and the low resolution image is generated sequentially based on the images in which the images with plural kinds of resolution are inputted at a plural resolution image generating part 2, and plural generated images are stored in a hierarchical image storage part 3. Processing is performed independently at every hierarchy i.e. a first hierarchy processing part 4-1, a second hierarchy processing part 4-2,... an (n)-th hierarchy processing part 4-n to perform the extraction of the effective feature quantity corresponding to the resolution. The locating position and the key of motion of the effective feature quantity extracted from every hierarchy in each image are unified at a general processing part 5, and since all the feature quantities obtained from each hierarchy are feature candidates, they are selected by using plural hierarchies. In such a way, it is possible to efficiently detect the rotating amount and direction of the head part from the effective feature quantity extracted from the general processing part 5 at a head part motion quantity detecting part 6.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to a feature amount extraction method for generating images with a plurality of resolutions to form a hierarchical structure and detecting feature amounts from the hierarchical images. It is something.

[Prior Art 1] In the field of intelligent encoded communications and technical fields that require man-machine interfaces, detection of the direction and amount of movement of a person's head is one of the important issues. In particular, there is a strong desire to perform detection without having to wear special equipment on the head.

There are non-contact detection methods that rely on image processing, but in order to detect head movements, one of the prerequisites is to first extract features from the head and face regions. It will be done.

FIG. 11 is an example of feature extraction using the conventional method.

This is an example in which various effective feature amounts are extracted from the facial region of the frontal image 101 through manual operations. That is, eyebrow 1
02. Eye 103. This is because finding the location of the nose 1041, mouth 105, etc. has not yet been solved. Processing such as edge detection and shape detection is performed near the center of gravity of each segment. A problem with such processing is that the process of extracting each segment is not automated.

FIG. 12 is an explanatory diagram of a conventional head orientation detection method. Since the various features on the facial region are all on the elastic body, they move along with facial expressions. Each feature amount, especially the outer corner of the eye 10
3a, the mouth 105a, etc., move relatively little (generally called fixed points).Assuming that these fixed points are detected, the accuracy of head orientation detection is high.However,
There is a problem that each point disappears due to rotation of the head or the like. Furthermore, it is a prerequisite that each feature quantity has been extracted.

FIGS. 13(a) and 13(b) are explanatory diagrams regarding the main problems of the conventional head orientation detection method. In the conventional method, when extracting various segments from a face part, it is assumed that the face is a normal face.

This is because, in the case of facial parts with expressions, many wrinkles 106 occur as shown in FIG. 13(a), making it difficult to detect various characteristic quantities. That is, since the various segments and wrinkles 106 are connected, their separation from each other becomes a problem. Furthermore, when facing the tank to a certain extent as shown in Fig. 13(b), for example, the eyes 103, eyebrows 102, etc. are missing or disappear, so extracting features from the frontal image is a major challenge, and practical has not yet reached that level.

[Problems to be Solved by the Invention] As described above, in the conventional feature extraction method, the extraction of each segment for extracting the feature amount is not automated, and especially in the case of the face, the extraction of each segment is not automated. There have been problems such as segments disappearing due to rotation, and it is difficult to extract special quantities when the face has expressions.

This invention was made to solve the above-mentioned conventional problems, and it is possible to relatively easily select effective feature amount candidates from the head region and efficiently detect the amount of rotation of the head. The purpose of this study is to obtain a method for extracting feature quantities that is possible.

[Means for Solving the Problems] In order to achieve the above object, the feature extraction method according to the present invention, in the invention set forth in claim (1), calculates the resolution of an image input by a camera from this image. Generate multiple images with different resolutions to form a hierarchical structure, extract effective feature quantities from each of these layers, and limit addresses where effective feature quantities exist in each layer from low resolution images to high resolution images. By doing so, the feature quantity is detected from the input image.

In addition, in the invention described in claim (2), a plurality of resolution images having different resolutions are generated from the image input by the camera to form a hierarchical structure, and each layer has an effective function according to the resolution. By extracting feature quantity candidates, selection is made using the addresses where effective feature quantities exist in images with one resolution lower than each other in each layer as clues, and feature quantities are detected from the input image.

[Effect]

The invention described in claim (1) of the invention forms a hierarchical structure of a plurality of images with different resolutions from a camera-input image of a person, for example, and creates an image of a person on the head and face. Feature quantity candidates are extracted from each layer, effective feature quantity candidates are selected sequentially from low-resolution images to high-resolution images, and finally, the orientation and rotation amount of the head are detected from the positional change of the selected effective features.

Furthermore, the invention described in claim (2) of this invention is:
After extracting the effective feature amount from each layer, the selection is made using the address where the effective feature amount exists in the image with one resolution lower than that of each layer. Since the effective feature amount is extracted from the partial region, it is more efficient than the conventional technology, which extracts the effective feature amount from a single image and detects the head orientation. .

[Embodiment] FIG. 1 is a block diagram of an apparatus for implementing the feature extraction method according to the present invention.

In this figure, 1 is a camera input section, 2 is a multi-resolution image generation section, 3 is a hierarchical image storage section, 4-1.4-2.・・・・・・
- 4-n is a first layer processing section and a second layer processing section.

. . . A third layer processing section, 5 is an integrated processing section, and 6 is a head movement amount detection section.

Next, the operation will be explained.

A human image is input from the camera input unit 1, and the multi-resolution image generation unit 2 sequentially generates low-resolution images based on the input images of multiple resolutions, and the generated images are stored in the hierarchical image storage unit. Please remember it in 3. In order to extract effective feature amounts according to the resolution, each layer, that is, the first layer processing section 4-1, the second layer processing section 4-2. . . . Processing is performed independently in the third layer processing section 4-n. Regarding the effective features extracted from each layer, the presence position and movement clues in each image are collectively processed by the integration processing unit 5. Since the feature amounts obtained from each layer are all feature candidates, selection is made using multiple layers. The head movement amount detection section 6 detects the rotation amount and direction of the head from the effective feature amount extracted by the integration processing section 5.

FIGS. 2, 3(a) and 3(b) are explanatory diagrams regarding the multi-resolution image generation section 2 and the hierarchical image storage section 3. As shown in FIG. 3(a), based on the original image 10, (mXn)
The average of the blocks is taken to generate a pixel M of an image with one resolution lower as shown in FIG. 3(b). Using this method, the first layer (original image) 10-1 to the nth layer 10 are
- Generate each n hierarchical image. The address between each image is
For example, the address (i, j) of a pixel M in a low resolution image
For an image with one higher resolution, (i−u~i+n,j
Pixels within the range m~j10-) 2 2 2 2 are candidates. In addition, since the averaging process is performed in this way, it is effective to remove wrinkles 106 on faces with facial expressions, which were a problem with conventional methods, and the low-resolution image has the effect of removing "strong" features in facial areas. Only a minute amount will be present in the image.Therefore, the rough “feature amount” in the low-resolution image will be “fine degree” in the high-resolution image.
The search time is greatly reduced by providing clues to the existence of effective feature quantities.In other words, the larger the number of rough features, the more the number of calculations can be reduced in almost inverse proportion to the number of calculations required.Image of each layer is placed in the hierarchical image storage section 3.

FIG. 4 is an example of processing at each layer. Top layer (low resolution image) (nth layer) image 1O-na.

At 1O-nb, the hair part is the strongest effective feature amount in the head region. Therefore, the direction of rotation of the head can be detected by detecting changes in the area of the hair and face in the head region.

Next, images 1O-(n-1)a, 10-(
In n-1)b, the shadows near the eyebrows 102 and the mouth 105 become hand-shaped. In this way, effective feature quantities are detected from each resolution.

Images 1O-1a and 1O-1b of the first layer 10-1 of the original image
There are many wrinkles 106 associated with facial expressions. However, since the effective position of the feature amount is detected from each upper layer, there is no need to search from the outside of the face area to the inside. That is, for example, in the eye 103, a clue 103X of its existence has been obtained as shown in FIG. It becomes easier to select 103Y.

FIGS. 6(a), 6(b), and 7 are explanatory diagrams of examples of "transferring" main information between layers. Original image 10-1 (front image 1O
-1a, landscape image 1o-1b), generate images with multiple resolutions as shown in FIG.
(shown only for O-1b). In the image 10A with the lowest resolution, the head existing region can be detected 11. Further, the rightward direction detection 12 of the head direction can be obtained from the change in the hair part. In the image 10B with one higher resolution, the orientation of the head and the region where the head exists are obtained as the first clues, so the feature amount search range can be narrowed in this layer.

Then, the head region within this range is extracted as a "skin color" region. Through this operation, the eyebrow and eye existing region detection 13 can be further narrowed. Using information regarding this eyebrow and eye existing region detection 13 as a clue, Then, the eyes and eyebrows are extracted as achromatic colors in the higher resolution image 10C, and the center of gravity region of these eyebrows and eyes is detected 14.Original image 1O
-1b edge image is generated, and the presence address ((ax, ay),
(bx, by), (ex, cy), (dx,
dyl) Selection 15 of feature quantity candidate points constituting the neighboring eyes 103 is performed. Note that in the address between each layer, each pixel in the upper layer corresponds to a pixel in an mXn block in the lower layer. Therefore, eyebrows 1o2. After the center of gravity region of the eye 103 is detected 14, the center of gravity is further determined within an m×n range on the original image and the center of gravity of the eyebrows 102. A process is performed in which each eye 103 is represented by one pixel. In this way, by processing head images according to their resolution, they can be processed without being affected by the occurrence of ineffective features such as "wrinkles"106 associated with facial expressions, and by processing head images according to their resolution. It has the advantage that rough "feature quantities" can be processed efficiently with a small calculation cost. FIG. 6(a), eyebrows 102 and eyes 103 are extracted by the integrated processing unit 5. This image includes many images associated with facial expressions There is a wrinkle (dotted line) 106, but since it was searched and detected from inside the feature quantity, the effective feature quantity and wrinkle 1 are different from those in the conventional method.
The connection problem in 06 was solved from the beginning. The center of gravity (triangle mark) of each segment extracted from the hierarchical image is calculated. Here, the center of gravity of the 102° eyebrow 103 is 21.22. The amount of rotation of the head is detected from the relative positional changes of these centers of gravity 21 and 22. This eyebrow 1o2. Virtual springs 31 and 32°3 that connect the centers of gravity 21 and 22 of the eye 103 to each other
Consider model 30 using 3.34. Front image 1O-
Remember the distances between the springs 30 to 34 in 1a,
Spring 31 ~ in landscape images 1O-1bR and 1O-1bL
The amount of rotation is calculated from the expansion and contraction of 34.

FIG. 9, FIG. 10 (a), and (b) show the results of head orientation detection. The center of gravity 21 of the eyebrows 102 and the center of gravity 22 of the eye 103 are connected by virtual springs 31 to 34. These four points are applied to orthogonal coordinates within the image. Initial registration of the positions of the four points in front is performed. FIG. 10(a) shows the results when the subject made head movements only to the left and right. Here, it is the relative displacement from the initial position of the springs 31 (○ mark) and 32 (△ mark). Next, FIG. 10(b) shows the positional displacement when only the vertical movement is performed. Here, spring 33 (○
mark), 34 (△ mark) from the initial position.

In both cases, head movements could be detected well. Also,
The maximum positional displacement width was larger in the case of left and right movements.

Capturing a person's image with a camera and detecting effective features,
We were able to perform consistent processing efficiently, which was not possible with conventional methods, up to the detection of the amount of movement.

[Effects of the Invention] As described above, the present invention generates a plurality of images with different resolutions from an image input by a camera to form a hierarchical structure, and extracts effective feature quantities from each of these layers. The feature quantity is detected from the input image by sequentially limiting the addresses where the effective feature quantity exists in each layer from the low-resolution image to the high-resolution image. By applying this method to detect facial movements, and extracting effective feature amounts according to the resolution in each layer, it is possible to efficiently detect the amount of movement from facial areas with facial expressions.

Furthermore, since the selection is made based on the address where the effective feature quantity of the image with one resolution lower than that of each layer is used as a clue,
It has the advantage of being able to reliably extract features and quantities.

[Brief explanation of drawings]

FIG. 1 is a block diagram showing an example of an apparatus for implementing the feature extraction method of the present invention, FIG. 2 is an explanatory diagram of a multi-resolution image generation section and a hierarchical image storage section of the present invention, and FIG. 3(a), (
b) is an explanatory diagram of the generation principle of images with different resolutions, FIG.
FIG. 5 is a diagram showing an example of processing in each layer of the present invention, and FIG. 6 (
a) and (b) are diagrams showing original images in front and sideways, FIG. 7 is a diagram showing an example of information utilization of this invention, and FIG. 8 is a diagram showing an example of spring modeling of effective feature quantities of this invention. , Fig. 9 and Fig. 10 are diagrams showing the spring model of this invention and the results of detecting the amount of head movement based on this, Fig. 11 is a diagram showing a feature extraction method using the conventional method, and Fig. FIG. 13 is a diagram showing a conventional method for detecting the amount of movement, and is a diagram for explaining problems with the conventional method. In the figure, 1 is a camera input section, 2 is a multi-resolution image generation section,
3 is a hierarchical image storage unit, 4-1.4-2゜-...4
-n is the first layer processing section, the second layer processing section,...the third layer processing section, etc.
Layer processing unit, 5 is an integrated processing unit, 6 is a head movement amount detection unit, 1
0 is the original image, 11 is the detection of the region where the head exists, 12 is the detection of the right direction, 13 is the detection of the region where the eyebrows and eyes are present, 14 is the detection of the centroid region of the eyebrows and the eyes, 15 is the feature quantity candidate point selection, 21. 22
is the center of gravity, 30 is the model, and 31, 32, 33, and 34 are springs. Figure (a) etc.) Figure 6 (a) (b) Figure Figure 0 Figure 11 Figure 2 Figure 3 (I3)

Claims (2)

[Claims] (1) For the image input by the camera, generate multiple images with different resolutions from this image to form a hierarchical structure, extract effective features from each of these layers, and
A feature amount extraction method characterized by detecting feature amounts from an input image by sequentially limiting addresses where effective feature amounts exist in each layer from a low-resolution image to a high-resolution image. (2) For the image input by the camera, multiple resolution images with different resolutions are generated from this image to form a hierarchical structure, and effective feature amount candidates are extracted according to the resolution for each layer. A feature quantity extraction method characterized by detecting feature quantities from an input image by making a selection based on an address where an effective feature quantity of images having one resolution lower than each other is used as a clue.