JP2011154717A - Facial image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a facial image processing device that automatically determines expression of a face and obtains a desired image. <P>SOLUTION: A plurality of images of a person including a facial image are input. The face area of the person is extracted for each of the input images. An evaluation value of the expression in the face area for each of the extracted images is calculated. Using the evaluated evaluation value, whether the facial expression of the person displayed in the input image satisfies a photographer is determined for each of the images. Using the determination result, from among the plurality of images, the image having the facial expression that satisfies the photographer is selected and output. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  Embodiments described herein relate generally to a face image processing apparatus.

Recently, digital image apparatuses such as electronic still cameras have been widely used, and are widely used in various fields.
For example, when shooting a person with an electronic still camera, videophone, or surveillance camera, if you want to shoot one or more people's faces when the face orientation, eyes, mouth, etc. are in the desired state, The photographer can adjust his / her face to the desired state, or a video camera can be used continuously to monitor the camera with a video camera at all times, and the optimal image can be seen later. The method of choosing while taking it.

  However, when taking a picture of one or more people, it is necessary to inform the photographee in advance of the desired facial condition in order to obtain the desired image of the photographer. If there are people who are in a state that is not suitable for one person, it is necessary to take another picture. Therefore, it is very difficult to shoot if you do not want the other party to know that you are shooting, or if you have multiple people who always have disjointed faces. There is a problem that there is.

  The problem to be solved by the present invention is to provide a face image processing apparatus capable of automatically determining facial expressions and acquiring a desired image.

  The face image processing apparatus according to the embodiment includes an image input means for inputting a plurality of images of a person including a face image, and a face for extracting the face area of the person for each image input by the image input means. Region extraction means, facial expression evaluation means for obtaining an evaluation value of facial expression for each image extracted by the face area extraction means, and the image input means using the evaluation value evaluated by the facial expression evaluation means Using facial expression determination means for determining for each image whether the facial expression of the person displayed in the input image is an image that the photographer desires, and using the determination result of the facial expression determination means Image selecting means for selecting and outputting an image in which the facial expression of the person is determined to be the facial expression desired by the photographer among the plurality of images.

The lineblock diagram showing an example of the system concerning an embodiment. The block diagram along the process of the system which concerns on embodiment. Explanatory drawing explaining the process of the face area extraction part which concerns on embodiment. Explanatory drawing explaining the process of the circular separability filter of the pupil detection part which concerns on embodiment. Explanatory drawing explaining the positional relationship of the pupil, a nostril, and a mouth in the pupil detection part and nostril detection part which concern on embodiment. Explanatory drawing explaining the detection process of the pupil detection part which concerns on embodiment. Explanatory drawing explaining the detection process of the mouth detection part which concerns on embodiment. Explanatory drawing explaining the determination process of the pupil state determination part which concerns on embodiment. The flowchart which shows the determination process of the pupil state determination part which concerns on embodiment. Explanatory drawing explaining the determination process of the pupil state determination part which concerns on embodiment. The flowchart explaining the determination process of the mouth state determination part which concerns on embodiment. Explanatory drawing explaining the determination process of the face state determination part which concerns on embodiment. Explanatory drawing explaining the size correction process of the face size correction | amendment part which concerns on embodiment. The figure which shows the picked-up image selection screen and interface which concern on embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
First, this method is used to recognize the face state (facial expression) of one or more persons included in a continuous image input from a TV camera or an electronic still camera, and to select a face in a state desired by the photographer. 1 shows an embodiment of an apparatus for photographing.

(1) Description of overall processing overview of the embodiment
FIG. 1 is a configuration diagram illustrating an example of a system according to an embodiment. In FIG. 1, the present embodiment is a calculation and storage device similar to a PC in a portable housing such as a TV camera and monitor 1, devices 2 and 3 consisting of a PC (or workstation), or an electronic still camera. Etc., and comprises a device 4 equipped with a small display such as liquid crystal or plasma.

  FIG. 2 is a block diagram along the processing of the system according to the embodiment. 2, the system according to the present embodiment includes an image input unit 11, an image storage unit 12, a face area extraction unit 13, a pupil detection unit 14, a nostril detection unit 15, a mouth detection unit 16, and a pupil. State determination unit 17, mouth state determination unit 18, face state determination unit 19, attribute-specific counting unit 20, optimal image photographing unit 21, optimal image composition unit 22, face size correction unit 23, and output unit 24.

  In such a system, the image processing of this embodiment is performed in the following procedure. That is, the digitized image is input from the image input unit 11 and the contents are continuously stored in the image storage unit 12. By applying the face area extraction unit 13 to the input image, one or more faces of the person existing in the input image are extracted, and in each of the extracted face areas, the pupil detection unit 14, the nostril detection unit 15, the mouth The detection unit 16 is used to detect the eye, nose, and mouth regions in the face. When each part of the face is detected, the pupil state determination unit 17 and the mouth state determination unit 18 obtain the open / closed state of the pupil, the line of sight, the open / closed state of the mouth, and the face state determination unit 19 uses the results to detect The state of each photographer's face is determined.

The attribute-specific counting unit 20 obtains the attributes of each person in the shooting area, such as sex and adult / children, and measures the number of persons for each attribute and in the shooting area. The optimum image photographing unit 21 determines whether or not the obtained image is in the state desired by the photographer, and outputs the image closest to the optimum state among a plurality of obtained images. In the case where a plurality of persons are photographed, the optimum image composition unit 22 stores an optimum image for each person to be photographed and composes the final output image.
The obtained results and candidate images are displayed by the output unit 24 while the size is corrected by the input image size or face size correcting unit 23 to notify the photographer of the results.

  Next, the operation will be described in detail along the respective processing units 11 to 23 with reference to the drawings.

(2) Processing description of the image input unit 11
An image is digitized and input in color or monochrome using a moving image input television camera and a still image input electronic still camera installed so that one or more persons can be photographed. The gradation and size of the input image are not particularly limited, and follow the input gradation and input resolution of the camera.

(3) Explanation of processing of image storage unit 12
The image captured from the image input unit 11 is stored in the memory as it is, and a plurality of immediately preceding images (up to N frames before) are stored in another area.

(4) Processing description of the face area extraction unit 13
Of the human face area, the upper and lower ends are located near the lips from the eyebrows, and the left and right ends are located outside both ends of the eyes as face search areas, and an average image or KL development is performed using a plurality of images in advance. Then, a face dictionary for face search is created by using upper component eigenvectors.

Also, various images are evaluated in advance using a face search dictionary, and if a non-face image is obtained in a region having a high similarity to the face dictionary, the image is collected as a non-face dictionary. In order to eliminate the influence of the face size on the input image, create enlarged / reduced images in multiple stages, and use the composite similarity method or template matching method for each image. Perform a search. The scanning procedure is shown in the explanatory diagram of FIG. Ideally, the face area should have high similarity with the face dictionary and low similarity with the non-face dictionary.
Evaluation value = similarity with face dictionary−similarity with non-face dictionary
The location having the highest evaluation value given by is obtained as the first face detection area. By setting a face detection area for an area that gives an evaluation value equal to or higher than a predetermined evaluation threshold at a position that is not more than a predetermined distance away from the area that has given the highest value, a plurality of persons can be included in the input image. It is also possible to detect all persons even when they are present and measure the number of persons in the imaged area.

(5) Processing description of the pupil detection unit 14
For each of the face regions extracted by the face region extraction unit 13, a circular separability filter (“Face recognition system using moving images”, Osamu Yamaguchi et al., IEICE Technical Report PRMU97-50, PP17-) with a plurality of radii. 23), the locations that are circular and darker than the surroundings are listed as pupil candidate points. Since the pupil region is assumed to be in the upper region of the face, the search region does not need to be processed for the entire face.

Further, it is possible to increase the speed by calculating the circular separation degree for obtaining the ratio of the luminance dispersion in each of the outer region and the inner region shown in FIG. 4 only in the binarized and determined to be dark. It is. Next, with respect to each of the obtained candidate points, a combination of candidate points (one set on the left and right) is narrowed down using a geometric arrangement condition according to the application. For example, the threshold value of the distance between both pupils is determined by the distance from the camera. Alternatively, when there is only a face in a stationary front state, the angle threshold is determined so that the line connecting both pupils is nearly horizontal. The following evaluation value calculation is performed for each of the eyes, and the left and right evaluation values are added to obtain an evaluation value of the combination.
Evaluation value = similarity with pupil dictionary−similarity with non-pupil dictionary It should be noted that each dictionary is created in advance in the same manner as the face area extraction unit 13 from data of a plurality of subjects in this case. The pupil dictionary has various types of pupils such as blinking eyes, blinking eyes, horizontal eyes, half eyes, etc. as separate multiple dictionaries. Can be detected.

In addition, the non-pupil dictionary is divided into classes such as the nostrils, the corners of the eyes, and the eyebrows that are likely to be mistaken for pupils, and has multiple dictionaries. To deal with various extraction failures. This is shown in FIG.
Moreover, the accuracy of pupil detection can be improved by determining geometric constraint conditions in combination with the nostril detection unit 15 as shown in FIG.

(6) Processing description of the nostril detection unit 15
The nose region is limited using the positional relationship between the face detection unit 13 and the pupil detection unit 14. The dark and round areas are listed as nostril candidate points by performing binarization and circular separation degree filter processing in the center of the face area and below both pupils in the same manner as the pupil detection unit 14. Similar to the face detection unit, the similarity is calculated with the nostril dictionary and the non-nasal dictionary, and the following evaluation values are obtained at each point.
Evaluation value = similarity with nostril dictionary−similarity with non-nasal dictionary In addition, among the combinations of all of the two candidate points, the above matches the geometrical arrangement condition with the pupil given in advance. A pair of points (two points on the left and right) with the highest evaluation value is obtained and detected as both nostril positions. As shown in the pupil detection unit 14, it is also possible to improve the accuracy by performing four points of the pupil and the nostril in the geometric arrangement condition.

(7) Explanation of processing of mouth detection unit 16
Since the face region extraction unit 13, the pupil detection unit 14, and the nostril detection unit 15 determine the arrangement of the face and the eyes and nose, the center of both pupils and the center of both nostrils are obtained, and the mouth is obtained using an average geometric arrangement. Perform calculations that you think will be. FIG. 5 is an explanatory diagram for explaining the positional relationship between the pupil, the nostril, and the mouth in the pupil detection unit 14 and the nostril detection unit 15 of the present embodiment. Refer to FIG.

Further, FIG. 7 shows an explanatory diagram of the processing of the mouth detection unit 16, which is an explanatory diagram for explaining the detection processing of the mouth detection unit 16 in the present embodiment.
In FIG. 7, binarization processing is performed in which a pixel having a luminance equal to or lower than a predetermined threshold value, in which only the darkest pixel appears in the region, is a black pixel, and other pixels are white pixels. An image. Since the region extracted even with this threshold value is a dark portion or a black portion, the region is a beard region or an open mouth region. From there, the threshold is gradually raised and binarized, and the difference image from the reference image is labeled, and when a horizontal area (label) appears and grows larger, the area becomes vertical and horizontal The area of the mouth is used when the size is exceeded. On the other hand, a black area such as a whisker whose size is almost the same as the binarization result of the initial threshold value can be excluded by difference processing and can be distinguished from the mouth area.

(8) Explanation of processing of pupil state determination unit 17
For each left and right pupil region obtained by the pupil detection unit 14, a dictionary is created according to various eye states such as “eye blink”, “half eye”, “horizontal eye”, “upper eye”, etc. The state having the highest similarity to the pupil image is determined as the current pupil state.

  Further, as described in the face state determination unit 19 described later, when the photographer side has previously selected which state is desired, the optimum image is selected by the following method.

  FIG. 9 is a flowchart showing the determination process of the pupil state determination unit 17. By this process, an optimal image can be selected even when the pupil state changes sequentially, such as blinking or eye movement, or even if the subject has a narrow eye and it is difficult to determine whether the pupil is opened or closed.

  The evaluation value is the difference between the similarity with the dictionary indicating the desired state and the highest similarity among the other dictionaries. If this value is high, it can be judged that it is close to the ideal state and can be clearly distinguished from other states. If this evaluation value is judged with a single image, it cannot be distinguished whether a person with narrow eyes is in an open state or a person with large eyes in a half-eye state. The image is continuously accumulated by the number N sufficient for photographing only, and the variance and average value of the evaluation values are calculated.

  In FIG. 9, when the variance of the evaluation values is small (S31), there is almost no change in the state of the eyes, and when the time higher than the average value is long (S32), the evaluation value is higher than the average. The state that gives the evaluation value closest to the average is set as the optimum image (S35), and when the time lower than the average value is long, the state that gives the evaluation value closest to the average among the evaluation values lower than the average is set as the optimum image. (S33). On the contrary, when the variance is large, it is considered that the state of the eye is greatly fluctuated, and the image that gives the highest evaluation value is set as the optimum image (S34).

  FIG. 10 is an explanatory diagram for explaining the determination process of the pupil state determination unit 17 in the present embodiment. When this is described as an example, (a) and (b) are less moving, less variance, and higher than the average. Since the time is long, an image that gives an evaluation value closest to the average value among the average values is selected. In (c), since the fluctuation is large and the variance becomes large, an image giving the highest value is selected. In (d), since the dispersion is small and the time lower than the average is long, an image closest to the average value is selected among the evaluation values lower than the average.

(9) Explanation of processing of mouth state determination unit 18
Next, a flowchart of the processing of the mouth state determination unit 16 is shown in FIG.
In FIG. 11, it is determined whether the mouth is open or closed by comparing the vertical width and horizontal width of the mouth, the vertical and horizontal widths, and the threshold values determined for each. If the vertical width of the mouth is equal to or greater than the predetermined threshold (S41), it is determined that the mouth is open (S44), and if the width is equal to or smaller than the predetermined threshold and the lateral width is equal to or greater than the predetermined threshold (S42). It is determined that the mouth is closed (S45). Furthermore, if it does not belong to either of them, the vertical and horizontal widths of the mouth, and the vertical and horizontal widths of the normalized image so that it is a constant size, a dictionary of a plurality of states (normal mouth, pointed mouth, The mouth state is determined (S46, S47) by comparing with (creating a dictionary according to each of squeaks, candy, etc.) (S43).

(10) Processing description of the face state determination unit 19
Using the outputs of the pupil state determination unit 17 and the mouth state determination unit 18, it is determined whether or not the face state is desired by the photographer. The desired state is, for example, the state in the case of an ID photo, etc., in which the pupil is open with the front facing and the mouth is closed. The state of the mouth may be either “state”, “the state where the mouth is laughed with the pupil open”, or the like.

  For the actual pupil state determination, a matrix is prepared in which the pupil and mouth states are shown on the vertical axis and the horizontal axis as shown in FIG. 12, and whether or not the desired state is entered is put in each cell. Become a shape.

(11) Explanation of processing of the attribute-specific counting unit 20
Each face area extracted by the face area extraction unit 13 has a dictionary consisting of average faces of men and women, a dictionary consisting of average faces of adult children, and an average face image dictionary for each nationality, etc., to calculate similarity. The number of persons is measured for each attribute depending on which is close, and attribute labeling is performed on the face area based on the obtained result. In addition, the number of persons can be measured by integrating all the numbers of persons existing in the non-photographing area regardless of the attribute.

(12) Explanation of processing of optimum image photographing unit 21
In a time-series continuous image accumulated within a predetermined time, a matrix as shown by the face state determination unit 19 is used to determine whether the photographer desires for each image, An evaluation value is obtained for each person and for each part by counting and integrating. The formula is as follows.
Evaluation value = (similarity with desired dictionary-highest similarity in non-desired dictionary)
Here, “face” indicates all faces included in the imaging region, and “part” indicates eyes and mouth in each face region. The image having the highest evaluation value is selected as the optimum image among the plurality of obtained images.

(13) Explanation of processing of optimum image composition unit 22
If you are shooting for multiple people and you want to take a photo in the desired state, such as when you want to take a picture of all the people in the shooting area laughing with your eyes open (open mouth) Among the images accumulated by repeating the process up to the face state determination unit 19 for a predetermined time, the most suitable image is stored for each person to be photographed and the peripheral image of the face area and the predetermined range of the face is stored, and the final output image is optimal By applying the images and combining them, the photographed person creates an optimal image without the need to adjust the shooting timing and surroundings. When compositing, it is assumed that the subject does not move as much as possible, but if it moves, it is unnatural by applying anti-aliasing along the periphery of the storage area that is larger than the face area. Processing is performed so that it is no longer a composite image.

(14) Processing description of face size correction unit 23
Although the image input when output to the output unit 24 can be output as it is, the size of the output image is enlarged or reduced according to the size of the extracted face area of one or more people. The face size can be obtained by using the size of the multi-resolution face dictionary used in the face area extracting unit 13, but the resolution is required by the resolution of the size, so another method is used here.

  Using only the luminance distribution in the region extracted as the face region, binarization is performed by a method such as a P-Tile method in which the white pixel black pixel ratio is constant, or a constant threshold value, discriminant analysis method, etc. Then, the area including the peripheral area of the face is binarized with the threshold value when the face area is binarized. By labeling the binarized image, a connected region including the center of the face is extracted, and the left and right ends of the region are used as the left and right ends of the face, and the width value is used as the face size. However, since the ears may appear or the ears may be hidden by the hair, classification is performed using the pupil positions obtained by the pupil detection unit 14 and the positions of the left and right ends of the face.

  FIG. 13 is an explanatory diagram of the processing, and the explanation will be made by taking the left side as an example with reference to the center D of both pupils. The threshold value is set in advance so that the left end of the face is in the position A when the ear is out and the AD length / CD length is equal to or greater than a predetermined threshold value. If the ear is hidden by the hair, the left end position is the position B, and the value of (BD length / CD length) is smaller than when the ear is out. Judge whether it is out. Similarly, it is determined whether or not the ear on the other side is out.

  When the ear is not heard, the position extracted as the left and right ends as it is is the face area. When the ear is heard, the position is calculated in advance using data of a plurality of persons (AD) / (BD). The position of B is calculated using the average value of B without affecting the ear position. When the photographer inputs a desired size based on the face size obtained as described above, an image is output at the desired size by performing enlargement / reduction processing.

(15) Processing description of output unit 24
Finally, the processing of the output unit 24 will be described below.
In the case of a stationary apparatus such as a TV camera, the optimal image and the optimal candidate image are arranged and output on a monitor in the case of a portable type apparatus and in a built-in monitor. As shown in FIG. 14, an image determined to be the optimum image is output large, and images having high evaluation values are arranged along the time sequence. If the desired image is in the candidate row, the final output image can be changed by selecting the desired image with the up / down / left / right buttons, and the rectangular area H surrounded by the dotted square in FIG. As described above, it is possible to mark the face area of each image and manually synthesize the optimum face from a plurality of images.

According to at least one embodiment described above, when photographing one or more people's faces by photographing with an electronic still camera, a videophone, or a surveillance camera, the desired photographing state or photographing is performed on the other party. Without being informed, it is possible to determine whether the face is facing the front, the open / closed state of the pupil, the open / closed state of the mouth, etc. While checking the face state, it is possible to automatically select the optimum one and take a picture.
In addition, when a plurality of persons such as a group photo are taken, it is possible to easily obtain the optimum images of all the subjects by automatically synthesizing the images in the optimum state of each subject.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 1 ... Camera, 2 ... Display, 3 ... Personal computer or workstation, 4 ... Digital camera including PC equivalent calculation / memory | storage device and internal display device, 11 ... Image input part, 12 ... Image storage part, 13 ... Face area Extraction unit, 14 ... pupil detection unit, 15 ... nostril detection unit, 16 ... mouth detection unit, 17 ... pupil state determination unit, 18 ... mouth state determination unit, 19 ... face state determination unit, 20 ... counting unit by attribute, 21 ... Optimum image photographing unit 22. Optimum image composition unit 23. Face size correction unit 24.

Claims (2)

Image input means for inputting a plurality of images of a person including a face image;
A face area extracting means for extracting the face area of the person for each image input by the image input means;
Facial expression evaluation means for obtaining an evaluation value of facial expression for each image extracted by the facial area extraction means;
For each image, whether the facial expression of the person displayed in the image input from the image input means using the evaluation value evaluated by the facial expression evaluation means is the image desired by the photographer. Facial expression determination means for determining
An image selection means for selecting and outputting an image in which the facial expression of the person is determined to be a facial expression desired by the photographer among the plurality of images using the determination result of the facial expression determination means;
A face image processing apparatus comprising:   2. The method for obtaining an evaluation value in the facial expression evaluation means is determined by a difference in similarity between a part to be evaluated and a dictionary not corresponding to the part from a similarity to the dictionary corresponding to the part. The face image processing apparatus described.

Images