JP4762329B2 - Face image processing apparatus and face image processing method - Google Patents

Description

The present invention relates to an image processing apparatus, and more particularly to a face image processing apparatus and a face image processing method for handling a human face image.

Recently, digital image apparatuses such as electronic still cameras have been widely used, and are widely used in various fields.
For example, when shooting a person with an electronic still camera, videophone, or surveillance camera, if you want to shoot the face of one or more people when the face orientation, eyes, mouth, etc. are in the desired state, The user can adjust his / her face to the desired state, or a video camera can be used to continuously shoot with a video camera, etc. at all times while watching the optimal image later. The method of choosing is taken.

  However, when taking a picture of one or more people, it is necessary to inform the photographee in advance of the desired facial condition in order to obtain the desired image of the photographer. If there are people who are in a state that is not suitable for one person, it is necessary to take another picture. Therefore, it is very difficult to shoot if you do not want the other party to know that you are shooting, or if you have multiple people who always have disjointed faces. There is a problem that there is.

Therefore, an object of the present invention is to provide a face image processing apparatus and a face image processing method that can automatically determine facial expressions and acquire a desired image.

The face image processing apparatus of the present invention detects a face image, inputs an image of a plurality of persons including the face image, and from each of the plurality of images input by the image input means, The state in which the similarity between the image of the dictionary prepared in advance for the plurality of eye states and the obtained pupil image is the highest is the current eye state, and this is used for each of the plurality of persons. An image calculated by evaluating whether the facial expression is an expression desired by the photographer, and using the evaluation value of the facial expression for each person of the plurality of persons based on the evaluation result Image selection means for selecting and outputting one image having the highest evaluation value for each image .
The face image processing apparatus of the present invention includes an image input unit that inputs a plurality of images of a plurality of persons including a face image, and the plurality of human faces for each image input by the image input unit. Similarity between the face area extracting means for extracting an area and the evaluation value of facial expression of a plurality of persons for each image extracted by the face area extracting means with the dictionary corresponding to the part to be evaluated From the facial expression evaluation means obtained by the difference in similarity with the dictionary not corresponding to the part, and the evaluation values of a plurality of persons evaluated by the facial expression evaluation means are displayed in the image input from the image input means. Further, for each of the plurality of persons, facial expression determination means for determining whether the facial expression is an image having a facial expression desired by the photographer, and using the determination result of the facial expression determination means, Among the multiple images An image selection means for selecting an image in which the facial expressions of several persons are determined to be those desired by the photographer, and facial expressions of the plurality of persons selected by the image selection means are desired by the photographer. Image synthesizing means for synthesizing and outputting the image determined to be.

According to the present invention, it is possible to provide a face image processing apparatus and a face image processing method that can automatically determine facial expressions and acquire a desired image.

The block diagram which shows an example of the system which is embodiment of this invention. The block diagram along the process of the system which is embodiment of this invention. Explanatory drawing explaining the process of the face area extraction part of this invention. Explanatory drawing explaining the process of the circular separation degree filter of the pupil detection part of this invention. Explanatory drawing explaining the positional relationship of the pupil, a nostril, and a mouth in the pupil detection part and nostril detection part of this invention. Explanatory drawing explaining the detection process of the pupil detection part in this invention. Explanatory drawing explaining the detection process of the mouth detection part in this invention. Explanatory drawing explaining the determination process of the pupil state determination part in this invention. The flowchart which shows the determination process of the pupil state determination part in this invention. Explanatory drawing explaining the determination process of the pupil state determination part in this invention. The flowchart explaining the determination process of the mouth state determination part in this invention. Explanatory drawing explaining the determination process of the face state determination part in this invention. Explanatory drawing explaining the size correction process of the face size correction | amendment part in this invention. The figure which shows the picked-up image selection screen and interface in this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
First, this method is used to recognize the face state (facial expression) of one or more persons included in a continuous image input from a TV camera or an electronic still camera, and to select a face in a state desired by the photographer. 1 shows an embodiment of an apparatus for photographing.

(1) Description of overall processing overview of the embodiment
FIG. 1 is a configuration diagram illustrating an example of a system according to an embodiment of the present invention. In FIG. 1, the present embodiment is a calculation and storage device similar to a PC in a portable housing such as a TV camera and monitor 1, devices 2 and 3 consisting of a PC (or workstation), or an electronic still camera. Etc., and comprises a device 4 equipped with a small display such as liquid crystal or plasma.

FIG. 2 is a block diagram along the processing of the system according to the embodiment of the present invention. 2, the system according to the present invention includes an image input unit 11, an image storage unit 12, a face area extraction unit 13, a pupil detection unit 14, a nostril detection unit 15, a mouth detection unit 16, and a pupil state. Determination unit 17, mouth state determination unit 18, face state determination unit 19, attribute-based counting unit 20, optimal image photographing unit 21, optimal image composition unit 22, face size correction unit 23, and output unit 24 And have.

  In such a system, the image processing of the present invention is performed in the following procedure. That is, the digitized image is input from the image input unit 11 and the contents are continuously stored in the image storage unit 12. By applying the face area extraction unit 13 to the input image, one or more faces of the person existing in the input image are extracted, and in each of the extracted face areas, the pupil detection unit 14, the nostril detection unit 15, the mouth The detection unit 16 is used to detect the eye, nose, and mouth regions in the face. When each part of the face is detected, the pupil state determination unit 17 and the mouth state determination unit 18 obtain the open / closed state of the pupil, the line of sight, the open / closed state of the mouth, and the face state determination unit 19 uses the results to detect The state of each photographer's face is determined.

The attribute-specific counting unit 20 obtains the attributes of each person in the shooting area, such as sex and adult / children, and measures the number of persons for each attribute and in the shooting area. The optimum image photographing unit 21 determines whether or not the obtained image is in the state desired by the photographer, and outputs the image closest to the optimum state among a plurality of obtained images. In the case where a plurality of persons are photographed, the optimum image composition unit 22 stores an optimum image for each person to be photographed and composes the final output image.
The obtained results and candidate images are displayed by the output unit 24 while the size is corrected by the input image size or face size correcting unit 23 to notify the photographer of the results.

  Next, the operation will be described in detail along the respective processing units 11 to 23 with reference to the drawings.

(2) Processing description of the image input unit 11
An image is digitized and input in color or monochrome using a moving image input television camera and a still image input electronic still camera installed so that one or more persons can be photographed. The gradation and size of the input image are not particularly limited, and follow the input gradation and input resolution of the camera.

(3) Explanation of processing of image storage unit 12
The image captured from the image input unit 11 is stored in the memory as it is, and a plurality of immediately preceding images (up to N frames before) are stored in another area.

(4) Processing description of the face area extraction unit 13
Of the human face area, the upper and lower ends are located near the lips from the eyebrows, and the left and right ends are located outside both ends of the eyes as face search areas, and an average image or KL development is performed using a plurality of images in advance. Then, a face dictionary for face search is created by using upper component eigenvectors.

Also, various images are evaluated in advance using a face search dictionary, and if a non-face image is obtained in a region having a high similarity to the face dictionary, the image is collected as a non-face dictionary. In order to eliminate the influence of the face size on the input image, create enlarged / reduced images in multiple stages, and use the composite similarity method or template matching method for each image. Perform a search. The scanning procedure is shown in the explanatory diagram of FIG. Ideally, the face area should have high similarity with the face dictionary and low similarity with the non-face dictionary.
Evaluation value = similarity with face dictionary−similarity with non-face dictionary
The location having the highest evaluation value given by is obtained as the first face detection area. By setting a face detection area for an area that gives an evaluation value equal to or higher than a predetermined evaluation threshold at a position that is not more than a predetermined distance away from the area that has given the highest value, a plurality of persons can be included in the input image. It is also possible to detect all persons even when they are present and measure the number of persons in the imaged area.

(5) Processing description of the pupil detection unit 14
For each of the face regions extracted by the face region extraction unit 13, a circular separability filter (“Face recognition system using moving images”, Osamu Yamaguchi et al., IEICE Technical Report PRMU97-50, PP17-) with a plurality of radii. 23) are enumerated as pupil candidate points that are circular and darker than the surroundings. Since the pupil region is assumed to be in the upper region of the face, the search region does not need to be processed for the entire face.

Further, it is possible to increase the speed by calculating the circular separation degree for obtaining the ratio of the luminance dispersion in each of the outer region and the inner region shown in FIG. 4 only in the binarized and determined to be dark. It is. Obtained using the geometry conditions then according to the application with respect to the candidate points each narrow down the combinations of candidate points (set left and right). For example, the threshold value of the distance between both pupils is determined by the distance from the camera. Alternatively, when there is only a face in a stationary front state, the angle threshold is determined so that the line connecting both pupils is nearly horizontal. The following evaluation value calculation is performed for each of the eyes, and the left and right evaluation values are added to obtain an evaluation value of the combination.
Evaluation value = similarity with pupil dictionary−similarity with non-pupil dictionary It should be noted that each dictionary is created in advance in the same manner as the face area extraction unit 13 from data of a plurality of subjects in this case. The pupil dictionary has various types of pupils such as blinking eyes, blinking eyes, horizontal eyes, half eyes, etc. as separate multiple dictionaries. Can be detected.

In addition, the non-pupil dictionary is divided into classes such as the nostrils, the corners of the eyes, and the eyebrows that are likely to be mistaken for pupils, and has multiple dictionaries. To deal with various extraction failures. This is shown in FIG.
Moreover, the accuracy of pupil detection can be improved by determining geometric constraint conditions in combination with the nostril detection unit 15 as shown in FIG.

(6) Processing description of the nostril detection unit 15
The nose region is limited using the positional relationship between the face detection unit 13 and the pupil detection unit 14. The dark and round areas are listed as nostril candidate points by performing binarization and circular separation degree filter processing in the center of the face area and below both pupils in the same manner as the pupil detection unit 14. Similar to the face detection unit, the similarity is calculated with the nostril dictionary and the non-nasal dictionary, and the following evaluation values are obtained at each point.
Evaluation value = similarity with nostril dictionary−similarity with non-nasal dictionary In addition, among the combinations of all of the two candidate points, the above matches the geometrical arrangement condition with the pupil given in advance. A pair of points (two points on the left and right) with the highest evaluation value is obtained and detected as both nostril positions. As also shown in the pupil detection unit 14, the accuracy can be improved by performing four points of the pupil and the nostril in the geometric arrangement condition.

(7) Explanation of processing of mouth detection unit 16
Since the face region extraction unit 13, the pupil detection unit 14, and the nostril detection unit 15 determine the arrangement of the face and the eyes and nose, the center of both pupils and the center of both nostrils are obtained, and the mouth is obtained using an average geometric arrangement. Perform calculations that you think will be. FIG. 5 is an explanatory diagram for explaining the positional relationship between the pupil, the nostril, and the mouth in the pupil detection unit 14 and the nostril detection unit 15 of the present invention. Refer to FIG.

FIG. 7 shows an explanatory diagram of the processing of the mouth detection unit 16, which is an explanatory diagram for explaining the detection processing of the mouth detection unit 16 in the present invention.
In FIG. 7, binarization processing is performed in which a pixel having a luminance equal to or lower than a predetermined threshold value, in which only the darkest pixel appears in the region, is a black pixel, and other pixels are white pixels. An image. Since the region extracted even with this threshold value is a dark portion or a black portion, the region is a beard region or an open mouth region. From there, the threshold is gradually raised and binarized, and the difference image from the reference image is labeled, and when a horizontal area (label) appears and grows larger, the area becomes vertical and horizontal The area of the mouth is used when the size is exceeded. On the other hand, a black area such as a whisker or the like whose size is hardly different from the binarization result of the initial threshold value can be excluded by the difference processing and can be distinguished from the mouth area.

(8) Explanation of processing of pupil state determination unit 17
For each left and right pupil region obtained by the pupil detection unit 14, a dictionary is created according to various eye states such as “eye blink”, “half eye”, “horizontal eye”, “upper eye”, etc. The state having the highest similarity to the pupil image is determined as the current pupil state.

  Further, as described in the face state determination unit 19 described later, when the photographer side has previously selected which state is desired, the optimum image is selected by the following method.

  FIG. 9 is a flowchart showing the determination process of the pupil state determination unit 17. By this process, an optimal image can be selected even when the pupil state changes sequentially, such as blinking or eye movement, or even if the subject has a narrow eye and it is difficult to determine whether the pupil is opened or closed.

  The evaluation value is the difference between the similarity with the dictionary indicating the desired state and the highest similarity among the other dictionaries. If this value is high, it can be judged that it is close to the ideal state and can be clearly distinguished from other states. If this evaluation value is judged with a single image, it cannot be distinguished whether a person with narrow eyes is in an open state or a person with large eyes in a half-eye state. The image is continuously accumulated by the number N sufficient for photographing only, and the variance and average value of the evaluation values are calculated.

  In FIG. 9, when the variance of the evaluation values is small (S31), there is almost no change in the state of the eyes, and when the time higher than the average value is long (S32), the evaluation value is higher than the average. The state that gives the evaluation value closest to the average is set as the optimum image (S35), and when the time lower than the average value is long, the state that gives the evaluation value closest to the average among the evaluation values lower than the average is set as the optimum image. (S33). On the other hand, when the variance is large, it is considered that the state of the eye has fluctuated greatly, and an image that gives the highest evaluation value is set as the optimum image (S34).

  FIG. 10 is an explanatory diagram for explaining the determination process of the pupil state determination unit 17 in the present invention. Taking this as an example, (a) and (b) are times when there is little movement and little variance and is higher than the average. Therefore, an image that gives an evaluation value that is closest to the average value and is higher than the average value is selected. In (c), since the fluctuation is large and the variance becomes large, an image giving the highest value is selected. In (d), since the dispersion is small and the time lower than the average is long, an image closest to the average value is selected among the evaluation values lower than the average.

(9) Explanation of processing of mouth state determination unit 18
Next, a flowchart of the processing of the mouth state determination unit 16 is shown in FIG.
In FIG. 11, it is determined whether the mouth is open or closed by comparing the vertical width and horizontal width of the mouth, the vertical and horizontal widths, and the threshold values determined for each. If the vertical width of the mouth is equal to or greater than a predetermined threshold (S41), it is determined that the mouth is open (S44). If the width is equal to or smaller than the predetermined threshold and the lateral width is equal to or greater than the predetermined threshold (S42) Is determined to be closed (S45). Furthermore, if it does not belong to either of them, the vertical and horizontal widths of the mouth, and the vertical and horizontal widths of the normalized image so that it is a constant size, a dictionary of a plurality of states (normal mouth, pointed mouth, The mouth state is determined (S46, S47) by comparing with (creating a dictionary for each of squeaks, candy, etc.) (S43).

(10) Processing description of the face state determination unit 19
Using the outputs of the pupil state determination unit 17 and the mouth state determination unit 18, it is determined whether or not the face state is desired by the photographer. The desired state is, for example, the state in the case of an ID photo or the like, which means “the pupil is open facing the front and the mouth is closed”. The mouth state may be either, "or" the mouth laughs with the eyes open. "

  For the actual pupil state determination, a matrix is prepared in which the pupil and mouth states are shown on the vertical axis and the horizontal axis as shown in FIG. 12, and whether or not the desired state is entered is put in each cell. Become a shape.

(11) Explanation of processing of the attribute-specific counting unit 20
Each face area extracted by the face area extraction unit 13 has a dictionary consisting of average faces of men and women, a dictionary consisting of average faces of adult children, and an average face image dictionary for each nationality, etc., to calculate similarity. The number of persons is measured for each attribute depending on which is close, and attribute labeling is performed on the face area based on the obtained result. In addition, the number of persons can be measured by integrating all the numbers of persons existing in the non-photographing area regardless of the attribute.

(12) Explanation of processing of optimum image photographing unit 21
In a time-series continuous image accumulated within a predetermined time, a matrix as shown by the face state determination unit 19 is used to determine whether the photographer desires for each image, An evaluation value is obtained for each person and for each part by counting and integrating. The formula is as follows.
Evaluation value = (similarity with desired dictionary-highest similarity in non-desired dictionary)
Here, “face” indicates all faces included in the imaging region, and “part” indicates eyes and mouth in each face region. The image having the highest evaluation value is selected as the optimum image among the plurality of obtained images.

(13) Explanation of processing of optimum image composition unit 22
If you are shooting for multiple people and you want to take a photo in the desired state, such as when you want to take a picture of all the people in the shooting area laughing with your eyes open (open mouth) Among the images accumulated by repeating the process up to the face state determination unit 19 for a predetermined time, the most suitable image is stored for each person to be photographed and the peripheral image of the face area and the predetermined range of the face is stored, and the final output image is optimal By applying the images and combining them, the photographed person creates an optimal image without the need to adjust the shooting timing and surroundings. When compositing, it is assumed that the subject does not move as much as possible, but if it moves, it is unnatural by applying anti-aliasing along the periphery of the storage area that is larger than the face area. Processing is performed so that it is no longer a composite image.

(14) Processing description of face size correction unit 23
Although the image input when output to the output unit 24 can be output as it is, the size of the output image is enlarged or reduced according to the size of the extracted face area of one or more people. The face size can be obtained by using the size of the multi-resolution face dictionary used in the face area extracting unit 13, but the resolution is required by the resolution of the size, so another method is used here.

  Using only the luminance distribution in the region extracted as the face region, binarization is performed by a method such as a P-Tile method in which the white pixel black pixel ratio is constant, or a constant threshold value, discriminant analysis method, etc. Then, the area including the peripheral area of the face is binarized with the threshold value when the face area is binarized. By labeling the binarized image, a connected region including the center of the face is extracted, and the left and right ends of the region are used as the left and right ends of the face, and the width value is used as the face size. However, since the ears may be hidden or the ears may be hidden by the hair, classification is performed using the pupil positions obtained by the pupil detection unit 14 and the positions of the left and right ends of the face.

  FIG. 13 is an explanatory diagram of the processing, and the explanation will be made by taking the left side as an example with reference to the center D of both pupils. The threshold value is set in advance so that the left end of the face is in the position A when the ear is out and the AD length / CD length is equal to or greater than a predetermined threshold value. If the ear is hidden by the hair, the left end position is the B position (the length of BD / the length of CD) is smaller than when the ear is out. Judge whether or not Similarly, it is determined whether or not the ear on the other side is out.

  When the ear is not heard, the position extracted as the left and right ends as it is is the face area. When the ear is heard, the position is calculated in advance using data of a plurality of persons (AD) / (BD). The position of B is calculated using the average value of B without affecting the ear position. When the photographer inputs a desired size based on the face size obtained as described above, an image is output at the desired size by performing enlargement / reduction processing.

(15) Processing description of output unit 24
Finally, the processing of the output unit 24 will be described below.
In the case of a stationary apparatus such as a TV camera, the optimal image and the optimal candidate image are arranged and output on a monitor in the case of a portable type apparatus and in a built-in monitor. As shown in FIG. 14, an image determined to be the optimum image is output large, and images having high evaluation values are arranged along the time sequence. If the desired image is in the candidate row, the desired image can be selected with the up / down / left / right buttons so that the final output image can be changed, and a rectangular region H surrounded by a dotted square in FIG. It is also possible to mark the face area of each image and manually synthesize an optimal face from a plurality of images.

As described above in detail, according to the above-described embodiment, when one or a plurality of persons' faces are photographed with an electronic still camera, a videophone, or a surveillance camera, the desired photographing state or photographing is performed on the other party. Without informing you, you can determine whether the face is facing the front, whether the face is facing the front, the open / closed state of the mouth, the open / closed state of the mouth, etc. It is possible to automatically select an optimal one while confirming a suitable face state and perform shooting.
In addition, when a plurality of persons such as a group photo are taken, it is possible to easily obtain the optimum images of all the subjects by automatically synthesizing the images in the optimum state of each subject.

DESCRIPTION OF SYMBOLS 1 ... Camera, 2 ... Display, 3 ... Personal computer or workstation, 4 ... Digital camera including PC equivalent calculation / memory | storage device and internal display device, 11 ... Image input part, 12 ... Image storage part, 13 ... Face area Extraction unit, 14 ... pupil detection unit, 15 ... nostril detection unit, 16 ... mouth detection unit, 17 ... pupil state determination unit, 18 ... mouth state determination unit, 19 ... face state determination unit, 20 ... counting unit by attribute, 21 ... Optimum image photographing unit 22. Optimum image composition unit 23. Face size correction unit 24.

Claims (3)

Image input means for detecting a face image and inputting images of a plurality of persons including the face image;
The state in which the similarity between the image of the dictionary prepared in advance for the plurality of eye states and the obtained pupil image is highest for each of the plurality of images input by the image input means And using this, it is evaluated whether the facial expression of each of the plurality of persons is an expression desired by the photographer, and based on the evaluation result, the plurality of persons in the plurality of images Image selecting means for selecting and outputting one image having the highest evaluation value for each image calculated using the evaluation value of facial expression for each person;
A face image processing apparatus comprising: Image input means for inputting a plurality of images of a plurality of persons including face images;
Face area extracting means for extracting the face areas of the plurality of persons for each image input by the image input means;
The evaluation value of facial expression of a plurality of persons for each image extracted by the face area extraction means is similar to a dictionary that does not correspond to the part from the similarity of the part to be evaluated to the dictionary corresponding to the part. Facial expression evaluation means determined by the difference in degree;
The facial expression desired by the photographer for each of the plurality of persons displayed in the image input from the image input means using the evaluation values of the plurality of persons evaluated by the expression evaluation means. Facial expression determination means for determining whether each image is an image,
An image selection means for selecting an image in which the facial expression of the plurality of persons is determined to be a facial expression desired by a photographer among the plurality of images using the determination result of the expression determination means;
Image synthesizing means for synthesizing and outputting images in which facial expressions of the plurality of persons selected by the image selecting means are determined to be those desired by the photographer;
A face image processing apparatus comprising: An image input step of detecting a face image and inputting images of a plurality of persons including the face image;
The state in which the similarity between the image of the dictionary and the obtained pupil image prepared in advance for each of the plurality of images input by this image input process is the highest is currently And using this, it is evaluated whether the facial expression of each of the plurality of persons is an expression desired by the photographer, and based on the evaluation result, the plurality of persons in the plurality of images An image selection step of selecting and outputting one image having the highest evaluation value for each image calculated using the evaluation value of the facial expression for each person;
A face image processing method comprising:

Images