JP2000259833A - Face image processor and processing method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a face image processor capable of obtaining a desired face image by automatically judging the state of the face of an object. SOLUTION: This face image processor is provided with an image input part 11 for continuously inputting the image of a human body including the face image and image selection parts 14-19 and 21 for judging the state of the face of the human body, e.g., the state of the pupil of the human body, from the image and automatically selecting the image including an optimum face image. Thus, for instance, by turning a camera to the object for a fixed time, an image including the optimum face image in which eyes are not closed is obtained.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, and more particularly, to a face image processing apparatus for handling a human face image and a method thereof.

[0002]

2. Description of the Related Art In recent years, digital imaging devices such as electronic still cameras have been remarkably popularized and widely used in various fields.

For example, when photographing a person with an electronic still camera, a TV phone, or a surveillance camera, when photographing the face of one or a plurality of persons when the face direction, eyes, mouth, etc. are in a desired state, Take the subject's face condition to the desired condition, or use a surveillance camera etc. to continuously shoot using videotape etc. at all times and watch the optimal image later. We take the method of selecting while watching.

[0004]

However, when one or more persons are being photographed, the photographer is informed in advance of his / her desired face state in order to obtain the image desired by the photographer. If it is necessary to have the user take that state, or if more than one person takes a picture, and there is a person who is not in a suitable state, it is necessary to take the picture again. Therefore, it is very difficult to shoot when you do not want the other party to know that the shooting is being performed like surveillance, or when the shooting target is such that there are multiple people and all of them are in disparate faces at all times. There is a problem that there is.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide a face image processing apparatus capable of automatically determining a face state and obtaining a desired image.

[0006]

According to the present invention, there is provided an image input means for continuously inputting an image of a person including a face image, and a plurality of continuous images input by the image input means. And an image selecting means for determining the state of the person's face and selecting an image including an optimal face image.

That is, according to the present invention, a face image of a person is detected from a continuous face image from a camera or the like by the above-described structure.
The purpose is to determine the state of the face image intended by the photographer, for example, the face image with the subject's eyes open, and select this. As a result, the face of the subject, such as not closing his / her eyes, merely pointing the camera or the like at the subject for a certain period of time without having to worry about whether or not the subject's eyes are closed as in the past, A face image processing apparatus that automatically selects an image can be provided.

Further, the present invention provides an image input means for continuously inputting a plurality of images of a person including a face image, and a plurality of images of the faces of the plurality of persons from the plurality of continuous images input by the image input means. Image selecting means for judging a state and selecting and outputting an image having an optimum face state of each of the plurality of persons.

The present invention evaluates each of a plurality of face images of a person using the above-described structure, and automatically selects an image having the best overall image from the stored images. Also provides a face image processing apparatus that automatically selects the most suitable one.

Further, according to the present invention, there is provided image input means for continuously inputting images of a plurality of persons including a face image, and a plurality of images of the faces of the plurality of persons are obtained from the plurality of continuous images input by the image input means. An image selection unit that determines the state of each of the plurality of persons, and selects an image that is an optimal state of the faces of all of the plurality of persons for each of the plurality of face images of the plurality of persons; A face image synthesizing means for synthesizing respective optimum face images and outputting the result as one image.

According to the present invention, the face image of each of a plurality of persons is evaluated by the above structure, and an optimum face image is detected for each person. Then, by synthesizing the optimum face image later, it is possible to provide a face image processing apparatus that automatically realizes a group photograph of the optimal expression for each person, which was not easily realized by the conventional camera.

The present invention also provides image input means for continuously inputting a plurality of images of a person including a face image, and converting the plurality of images of the faces of the plurality of persons from the plurality of continuous images input by the image input means. An attribute head count determining means for determining a head count configuration for each attribute of a plurality of persons in the photographing area by comparing the state with the average face value of each attribute such as gender, nationality, adult / child, etc. This is a facial image processing device that is a feature.

[0013] The present invention provides an image having a larger amount of information by automatically judging the gender and the attributes of subjects such as adults and children in a photograph in which a plurality of persons are photographed by the above structure. Can be.

According to the present invention, there is provided image input means for continuously inputting an image of a person including a face image, and detecting a pupil region of the face image of the person from a plurality of continuous images input by the image input means. Pupil region detection means for detecting an image of a pupil region that is continuous for a predetermined time of the pupil region detected by the pupil region detection device, and outputting a fluctuation state thereof; Based on the fluctuation state output from the state detection means, the image is compared with a previously stored pupil dictionary to select an image including a face image having a pupil region closest to the state desired by the photographer. And an image selecting unit for outputting the image.

According to the present invention, the pupil region is automatically detected by the above-described structure, and an image of this region for a predetermined time is examined.
The image of the eye state including the blink is recognized, and an image including the eye state most suitable for photographing is automatically selected.
By simply pointing the camera, etc. at the subject for a certain period of time, the eyes are not too open and not closed. Can be provided.

According to the present invention, there is provided image input means for continuously inputting an image of a person including a face image, and a position of a pupil and a nostril of the face of the person from a plurality of continuous images input by the image input means. And a binarized image using a threshold that extracts only dark regions of the mouth region candidates determined by the mouth region candidate determining unit that detects mouth region candidates based on the detected Is used as the reference image, binarization is performed while changing the threshold value in the direction in which the number of dark pixels gradually increases, a difference image from the reference image is obtained, and the obtained difference image is labeled to detect a label having a predetermined size. And a mouth region detecting means for detecting a mouth region by performing the operation.

According to the present invention, by automatically detecting the mouth area according to the above-described procedure, the mouth area can be accurately recognized even when a mustache or the like is present. It is possible to detect any mouth area such as

According to the present invention, there is provided image input means for continuously inputting an image of a person including a face image, and a face center including a pupil, a mouth and a nostril from a plurality of continuous images input by the image input means. Calculate an appropriate threshold value that can detect pupils and nostrils in the area, and use the threshold value to binarize the entire face area and perform labeling processing to extract one connected face area contained inside the hair A face region extracting unit, and a left and right end of the face region extracted by the face region extracting unit;
Face area measurement means for determining whether the ears are hidden by the hair using the ratio of the distances to the center coordinates of both eyes, and determining the size of the face area not including the ears based on the determination result,
A face image processing apparatus comprising: a face area correction unit configured to enlarge or reduce an image of the face area to an arbitrary size based on the size of the face area obtained by the face area measurement unit.

With the above structure, the present invention detects the original width of the face regardless of whether or not the ears protrude from the hair. For example, even when the face is only small, the optimum size is determined according to the setting. Then, the image is automatically enlarged and output. Thus, by automatically recognizing the size of the face image and automatically performing scaling according to the size, for example, a photograph of a person who was only small on the screen is automatically enlarged to an appropriate size. And a face image processing apparatus capable of automatically providing a good image.

According to the present invention, there is provided image input means for continuously inputting an image of a person including a face image, image storage means for storing a plurality of images input by the image input means, Means for extracting a face area in the image stored by the means, pupil detection means for detecting a pupil area from the face areas extracted by the face area extraction means, and face area extraction means. Mouth detection means for detecting a mouth area from the extracted face area, pupil state determination means for determining a state of the pupil area detected by the pupil detection means, and determination of a state of the mouth area detected by the mouth detection means Mouth state determining means to perform, a face state determining means for determining the state of the extracted face area based on the determinations made by the pupil state determining means and the mouth state determining means, and a determination made by the face state determining means Based on the results, Image selecting means for automatically selecting an image including a face image in the face state from the plurality of images stored by the image storing means, and detecting the size of the face image in the image selected by the image selecting means, And a face area correcting means for enlarging or reducing the size of the face image.

The present invention is a face image processing apparatus for automatically realizing an optimum pupil state, mouth state, and face size by the above structure.

It is needless to say that the same effect can be obtained with the same effect in the invention of the method.

[0023]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below in detail with reference to the drawings.

First, using this method, the state of one or more human faces included in a continuous image input from a television camera or an electronic still camera is recognized, and a face in a state desired by the photographer is photographed. 1 shows an embodiment of an apparatus.

(1) Description of Processing Overview of Overall Processing of Embodiment FIG. 1 is a configuration diagram showing an example of a system according to an embodiment of the present invention. In FIG. 1, this embodiment is a television camera and monitor 1, a PC (or workstation).
, Or a device 4 including a calculation and storage device similar to a PC in a portable housing such as an electronic still camera and equipped with a small display such as a liquid crystal or a plasma.

FIG. 2 is a block diagram showing the processing of the system according to the embodiment of the present invention. 2, the system according to the present invention includes an image input unit 11, an image storage unit 12, a face region extraction unit 13, a pupil detection unit 14,
A nostril detection unit 15, a mouth detection unit 16, a pupil state determination unit 17, a mouth state determination unit 18, a face state determination unit 19, an attribute-based counting unit 20, an optimal image photographing unit 21, It has an image synthesizing unit 22, a face size adjusting unit 23, and an output unit 24.

In such a system, the image processing of the present invention is performed in the following procedure. That is, a digitized image is input from the image input unit 11, and the contents are continuously stored in the image storage unit 12. The face region extracting unit 13 is applied to the input image to extract the faces of one or more persons present in the input image, and the pupil detecting unit 14, the nostril detecting unit 15, and the mouth are extracted in each of the extracted face regions. The detection unit 16 is used to detect the site of the eyes, nose, and mouth in the face. When each part of the face is detected, the pupil state determination unit 17 and the mouth state determination unit 18 determine the open / closed state of the pupil, the state of the line of sight, the open / closed state of the mouth, and the like. The state of the face of each photographer is determined.

The attribute-based counting section 20 determines the gender, attributes of adults / children, etc. of the persons in the photographing area, and measures the number of persons in each attribute and in the photographing area. The optimum image photographing unit 21 determines whether the obtained image is in a state desired by the photographer, one by one, and outputs an image closest to the optimum state among a plurality of obtained images. When a plurality of persons are photographed, the optimal image combining unit 22 stores the optimal image for each of the photographed persons and combines the images with the final output image.

The obtained result and the candidate image are displayed by the output unit 24 while correcting the size by the input image size or the face size correcting unit 23 to inform the photographer of the result.

Next, the operation of each of the processing units 11 to 23 will be described in detail with reference to the drawings.

(2) Processing Description of Processing Description of Image Input Unit 11 A television camera for inputting a moving image, an electronic still camera for inputting a still image, and the like, which are installed so that one or more persons appear therein, are used. And digitize the image in color or monochrome and input. The gradation and the size of the input image are not particularly limited, and are based on the input gradation and the input resolution of the camera.

(3) Description of Processing of Image Storage Unit 12 The image captured from the image input unit 11 is stored in the memory as it is, and a plurality of images immediately before (up to N frames before) are stored in another area. I do.

(4) Processing Description of Processing by Face Area Extraction Unit 13 Of the human face area, the upper and lower ends are defined as face search areas that are located near the lips from the eyebrows and the left and right ends are located outside both ends of both eyes. , An average image or K
A face dictionary for face search is created by performing L expansion and using the upper component eigenvector. In addition, various images are evaluated in advance using a face search dictionary, and if a non-face image is obtained in an area having a high degree of similarity to the face dictionary, the images are collected as a non-face dictionary. In order to eliminate the influence of the size of the face on the input image, an enlarged / reduced image is created in multiple stages, and the face area is determined for each image using the composite similarity method or template matching method. Perform a search. The scanning procedure is shown in the explanatory diagram of FIG. Ideally, the face area has a high degree of similarity to the face dictionary and a low degree of similarity to the non-face dictionary. The evaluation value = the degree of similarity to the face dictionary minus the degree of similarity given by the similarity to the non-face dictionary. Is determined as the first face detection area. By setting a face detection area to an area that gives an evaluation value equal to or higher than a predetermined evaluation threshold at a position that is not overlapped with the area where the highest value is obtained and that is at least a predetermined distance, a plurality of people can input images. Even in the case of entering, it is also possible to detect all the members and measure the number of persons in the photographed area.

(5) Processing Description of Processing of Pupil Detector 14 For each of the face regions extracted by the face region extractor 13, a circular separation degree filter with a plurality of radii (“Face recognition system using moving images”) Osamu Yamaguchi et al., IEICE Technical Report PRMU9
7-50, refer to PP17-23) and circumscribe the places that are circular and darker than the surroundings as pupil candidate points. Since the pupil region is assumed to be above the face, the search region need not be processed for the entire face. Further, it is possible to speed up the calculation by calculating the circular separation degree for calculating the ratio of the luminance variance in each of the outer region and the inner region shown in FIG. is there. Next, for each of the obtained candidate points, combinations of candidate points (one pair on the left and right sides) are narrowed down using a geometric arrangement condition according to the application. For example, the threshold value of the distance between the pupils is determined based on the distance from the camera. Alternatively, if there is only a face in a front stationary state, the threshold value of the angle is determined so that the line connecting both pupils is nearly horizontal. The following evaluation values are calculated for both eyes, and the sum of the left and right evaluation values is used as the evaluation value of the combination.

Evaluation value = similarity with pupil dictionary−similarity with non-pupil dictionary Note that each dictionary is prepared in advance from data of a plurality of subjects in the same manner as the face area extraction unit. The pupil dictionary in this case has various pupil states, such as eye-opening, blinking, side-eyed eyes, half-eyed eyes, etc., all in separate dictionaries, and it is stable even in various states such as eyes-closed and side-eyed eyes A pupil region can be detected. In addition, the non-pupil dictionary is divided into classes such as the nostrils, the nostrils, the outer corners of the eyes, the eyebrows, etc. The calculation will deal with various extraction failures. This is shown in FIG.

Further, by determining the geometric constraint conditions as shown in FIG. 5 in combination with the nostril detection section 15, it is possible to increase the accuracy of pupil detection.

(6) Description of Processing of Nose Detector 15 The nose area is limited using the positional relationship between the face detector 13 and the pupil detector 14. By performing binarization and circular separation filter processing in the center of the face area and below both pupils in the same manner as in the pupil detection unit 14, the area of the dark and round part is enumerated as nostril candidate points, and Similar to the face detection unit, similarity calculation is performed with the nostril dictionary and non-nostril dictionary, and the following evaluation values are obtained at each point.

Evaluation value = similarity with nostril dictionary-similarity with non-nostril dictionary Also, in the combination of two points of all candidate points, it matches the geometric arrangement condition with the pupil given in advance. Among them, a set of points (two points on the left and right) having the highest evaluation value is obtained and detected as the position of both nostrils. Although shown in the pupil detection unit 14, the accuracy can be improved by performing four points of the pupil and the nostril in the geometric arrangement condition.

(7) Description of Processing of Mouth Detection Unit 16 Since the arrangement of the face, eyes, and nose is obtained by the face area extraction unit 13, pupil detection unit 14, and nostril detection unit 15, the center of both pupils and the center of both nostrils are obtained. Uses the average geometry to make calculations that you think will have a mouth. FIG. 5 is an explanatory diagram illustrating the positional relationship between the pupil, the nostrils, and the mouth in the pupil detection unit and the nostril detection unit according to the present invention, and FIG.

FIG. 7 is an explanatory diagram of the processing of the mouth detecting section 16, which is an explanatory view for explaining the detecting processing of the mouth detecting section in the present invention. In FIG. 7, a binarization process is performed in which a pixel having a luminance equal to or lower than a predetermined threshold value, at which only the darkest pixel appears in the region, is set as a black pixel, and the other pixels are set as white pixels. Make an image. Since the area to be extracted even with this threshold value is a dark part or a black part, it is defined as a beard area or an open mouth area. From there, the threshold value is gradually increased to perform binarization, and a labeling process is performed on the difference image from the reference image. The area of the mouth is determined when the size of the mouth is exceeded. On the other hand, a black region such as a beard where the size hardly changes from the binarization result of the initial threshold value can be excluded by the difference processing, and can be distinguished from the mouth region.

(8) Description of the processing of the pupil state determination unit 17 For each of the left and right pupil regions obtained by the pupil detection unit 14,
A dictionary is created according to various states of the eyes such as "blind eyes", "half eyes", "side eyes", "upper eyes", etc., and the state in which the similarity with the obtained pupil image is the highest is determined for the current pupil. Judge as the state.

If the photographer has previously selected which state he or she wants, as described in the face state determination unit 19 described later, the optimum image is selected by the following method.

FIG. 9 is a flowchart showing the judgment processing of the pupil state judgment section. By this processing, an optimal image can be selected even when the state of the pupil such as blinking or movement of the line of sight changes sequentially, or even for a subject whose eyes are thin and it is difficult to determine whether the pupil is open or closed.

The evaluation value is the difference between the similarity with the dictionary indicating the desired state and the highest similarity among the other dictionaries. When this value is high, it can be determined that the state is close to the ideal state and can be clearly distinguished from other states. If this evaluation value is determined on a single image, it cannot be distinguished whether a person with a narrow eye is in an open state or a person with a large eye is in a half-eyed state, so it takes more time than the time from when blinking starts to when it ends. The image is accumulated continuously for the number N of images sufficient to capture only the images, and the variance and average value of the evaluation values are calculated.

In FIG. 9, when the variance of the evaluation value is small (S31), it is determined that there is almost no change in the state of the eyes, and when the time higher than the average value is longer (S32),
The state in which the evaluation value closest to the average among the evaluation values higher than the average is given as the optimal image (S35). If the time lower than the average value is long, the state closest to the average among the evaluation values lower than the average is used. The state in which the evaluation value is given is selected as the optimum image (S33). Conversely, if the variance is large, it is considered that the state of the eyes has fluctuated greatly, and the one giving the highest evaluation value is determined as the optimal image (S34).

FIG. 10 is an explanatory diagram for explaining the judgment processing of the pupil state judging section in the present invention. In this example, (a) and (b) show little movement and little variance,
An image that gives an evaluation value closest to the average value among those higher than the average is selected because the time longer than the average is longer. In (c), since the fluctuation is large and the variance is large, the image giving the highest value is selected. In (d), since the variance is small and the time lower than the average is long, the image closest to the average value is selected among the evaluation values lower than the average.

(9) Description of Processing of Mouth State Determining Section 18 Next, a flowchart of the processing of the mouth state determining section 16 will be described with reference to FIG.
It is shown in FIG.

In FIG. 11, it is determined whether the mouth is open or closed by comparing the upper and lower widths and the left and right widths of the mouth, the upper and lower widths and the left and right widths, and threshold values respectively defined.
If the vertical width of the mouth is equal to or greater than a predetermined threshold (S41), it is determined that the mouth is open (S44). If the width is equal to or less than the predetermined threshold and the width is equal to or greater than the predetermined threshold (S42). Is determined to be closed (S45). In addition, if it does not belong to either of them, the dictionary in multiple states (normal mouth, pointed mouth, clenched mouth) in the image in which the top, bottom, left, and right widths and the top, bottom, left, and right widths are normalized to have a fixed size (S43), and the state of the mouth is determined (S46,
S47).

(10) Description of Processing of Face State Determining Unit 19 The output of the pupil state determining unit 17 and the mouth state determining unit 18 is used to determine whether the face state is desired by the photographer. The desired state is, for example, the state in the case of an identification photograph or the like, which is "a state in which the eyes are open facing the front and the mouth is in a closed state". And the state of the mouth may be either. "The state of the mouth laughing with the eyes open."

For the actual state determination, a matrix is prepared in which the pupil and mouth states as shown in FIG. 12 are plotted on the vertical and horizontal axes, and whether or not the desired state is entered in each cell. It takes shape.

(11) Description of Processing by Attribute-Based Counting Unit 20 In each of the face regions extracted by the face region extracting unit 13, a dictionary composed of average faces of men and women, a dictionary composed of average faces of adults and children, nationality, etc. Each has an average face image dictionary, calculates the degree of similarity, measures the number of persons for each attribute based on which is closer, and labels the face area with the attribute based on the obtained result. Also, the number of persons can be measured by integrating all the numbers of persons existing in the non-photographing area regardless of the attribute.

(12) Description of Processing of Optimal Image Shooting Unit 21 In the time-series continuous images accumulated within a predetermined time,
Using a matrix such as that shown in the face state determination unit 19, whether or not the photographer desires the state is counted and integrated for each image, for each person, and for each region. The calculated value is obtained as an evaluation value. The formula is as follows.

Evaluation value = (similarity with desired dictionary−
(The highest similarity in the non-desired dictionary.) Here, “face” indicates all faces included in the photographing area, and “part” indicates eyes and mouth in each face area. The image having the highest evaluation value among the plurality of obtained images is selected as the optimum image.

(13) Description of Processing of Optimal Image Compositing Unit 22 A photograph is taken for a plurality of persons, and all the persons in the photographing area open their eyes and laugh (open their mouths). When it is desired to take a desired state such as a desired state, by repeating the processing up to the face state determination unit 19 for a predetermined period of time, an optimum image for each of the subjects can be obtained in the face area and the predetermined range. By storing the peripheral image of the face and applying the optimum image to the final output image and synthesizing the image, the subject can create an optimum image without having to adjust the shooting timing and surroundings. In the case of composition, it is assumed that the subject does not move as much as possible, but if it moves, unnatural composition is performed by applying anti-aliasing along the periphery of the storage area larger than the face area Processing is performed so that the image is no longer displayed.

(14) Description of the Processing of the Face Size Correction Unit 23 When the image is output to the output unit 24, the input image can be output as it is. To enlarge or reduce the size of the output image. The size of the face can be obtained by using the size of the face dictionary of a plurality of resolutions used in the face area extraction unit 13. However, since the resolution is required for the resolution of the size, another method is used here.

Using only the luminance distribution in the region extracted as the face region, the P-Tile method or the fixed threshold value, the discriminant analysis method, or the like, in which the ratio of white pixels to black pixels is constant, is used. The binarization is performed, and the region including the peripheral region of the face is binarized using the threshold value when the face region is binarized. By labeling the binarized image, a connected area including the face center is extracted, and the left and right ends of the area are defined as the left and right ends of the face, and the width value is used as the face size. However, because your ears may be out and your ears may be hidden by your hair,
Classification is performed using the pupil position obtained by the pupil detection unit 14 and the positions of the left and right ends of the face.

FIG. 13 is an explanatory diagram of the processing. The description will be made by taking the left side as an example with reference to the center D of both pupils. The left end of the face is located at position A when the ear is out, and the threshold value is set in advance so that the length of AD / the length of CD is equal to or greater than a predetermined threshold value. If the ear is hidden by the hair, the left end position is the position B (BD length / C
Since the value of (length of D) is smaller than when the ear is out, it is determined here whether or not the ear is out. Similarly, it is determined whether or not the ear on the opposite side is out.

If no ears are present, the positions extracted as the left and right ends are regarded as face areas. If ears are present, (AD) / (AD) / is calculated in advance using data of a plurality of persons.
The position of B is calculated and obtained using the average value of (BD) without affecting the ear position. If the photographer has input the desired size based on the face size obtained as described above, the image is output at the desired size by performing enlargement / reduction processing.

(15) Description of Processing of Output Unit 24 Finally, the processing of the output unit 24 will be described below.

In the case of a stationary device such as a television camera, an optimum image and an optimum candidate image are output side by side on a monitor in the case of a stationary type and a built-in monitor in the case of a portable type. As shown in FIG. 14, the image determined to be the optimum image is output in large size, and the images having the higher evaluation values are arranged beside the image in the time sequence. If the desired image is located in the candidate column, the final output image can be changed by selecting the desired image using the up, down, left, and right buttons, and a rectangular area H surrounded by a dotted rectangle in FIG. It is also possible to mark the face area of each image and manually synthesize the optimum face from a plurality of images.

[0061]

As described in detail above, according to the present invention,
When photographing the face of one or more people using an electronic still camera, TV phone, or surveillance camera, the effect of the fineness and movement of the eyes can be further improved without informing the other party of the desired photographing state or shooting. It is possible to determine whether the face is facing forward, whether the eyes are open or closed, whether the mouth is open or closed, etc. You can select an object and shoot.

When a plurality of persons, such as a group photograph, are photographed, by automatically synthesizing the images in the optimum state of each of the persons, it is possible to easily obtain the optimum images of all the persons. .

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing an example of a system according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating processing of the system according to the embodiment of the present invention.

FIG. 3 is an explanatory diagram illustrating a process of a face area extraction unit according to the present invention.

FIG. 4 is an explanatory diagram illustrating processing of a circular separation filter of a pupil detection unit according to the present invention.

FIG. 5 is an explanatory diagram illustrating a positional relationship between a pupil, a nostril, and a mouth in a pupil detection unit and a nostril detection unit according to the present invention.

FIG. 6 is an explanatory diagram illustrating a detection process of a pupil detection unit according to the present invention.

FIG. 7 is an explanatory diagram illustrating detection processing of a mouth detection unit according to the present invention.

FIG. 8 is an explanatory diagram illustrating a determination process of a pupil state determination unit according to the present invention.

FIG. 9 is a flowchart illustrating a determination process of a pupil state determination unit according to the present invention.

FIG. 10 is an explanatory diagram illustrating a determination process of a pupil state determination unit according to the present invention.

FIG. 11 is a flowchart illustrating a determination process of a mouth state determination unit according to the present invention.

FIG. 12 is an explanatory diagram illustrating a determination process of a face state determination unit according to the present invention.

FIG. 13 is an explanatory diagram illustrating a size correction process of a face size correction unit according to the present invention.

FIG. 14 is a diagram showing a captured image selection screen and an interface according to the present invention.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Camera 2 ... Display 3 ... Personal computer or workstation 4 ... Digital camera including a calculation / storage device and internal display device equivalent to a PC 11 ... Image input unit 12 ... Image storage unit 13 ... Face region extraction unit 14 ... Eye detection Unit 15: Nostril detection unit 16: Mouth detection unit 17: Pupil state detection unit 18: Mouth state detection unit 19: Face state determination unit 20: Attribute-based counting unit 21: Optimal image photographing unit 22: Optimal image synthesis unit 23: Face Size adjustment unit 24… Output unit

Claims (16)

[Claims] An image input means for continuously inputting an image of a person including a face image, and a plurality of continuous images input by the image input means,
An image selecting means for judging the state of the person's face and selecting an image including an optimal face image. 2. An image input means for continuously inputting a plurality of images of a person including a face image, and a plurality of continuous images input by the image input means.
A face image processing apparatus comprising: an image selecting unit that determines the state of the faces of the plurality of persons and selects and outputs an image in which the states of the faces of the plurality of persons are optimal. 3. An image input means for continuously inputting a plurality of images of a person including a face image; and a plurality of continuous images input by the image input means.
An image selection unit that determines the state of the faces of the plurality of persons, and selects an image that is the optimal state of the faces of all of the plurality of persons for each of the plurality of face images of the persons; A face image synthesizing means for synthesizing optimal face images of the plurality of selected persons and outputting the synthesized image as one image. 4. An image input means for continuously inputting images of a plurality of persons including a face image, and a plurality of continuous images input by the image input means,
The number of attributes to determine the composition of the number of persons for each attribute of the plurality of persons in the shooting area by comparing the state of the faces of the plurality of persons with the average face value of each attribute such as gender, nationality, adult / child, etc. A face image processing apparatus comprising: a determination unit. 5. An image input means for continuously inputting an image of a person including a face image, and a pupil area for detecting a pupil area of the face image of the person from a plurality of continuous images input by the image input means. Detection means; pupil area fluctuation state detection means for detecting an image of the pupil area continuous for a predetermined time of the pupil area detected by the pupil area detection means and outputting the fluctuation state thereof; Based on the output of the fluctuation state, the image is compared with a previously stored pupil dictionary to select and output an image including a face image having a pupil region closest to the state desired by the photographer. A face image processing apparatus, comprising: an image selection unit. 6. An image input means for continuously inputting an image of a person including a face image, and detecting a position of a pupil and a nostril of the face of the person from a plurality of continuous images input by the image input means. A mouth region candidate determining unit for determining a mouth region candidate based on the reference region, and a binarized image using a threshold from which only a dark region of the mouth region candidate determined by the mouth region candidate determining unit is extracted as a reference image. By changing the threshold value in the direction in which the number of dark pixels gradually increases, binarization is performed to obtain a difference image from the reference image, and the obtained difference image is labeled to detect a label having a predetermined size. A face image processing apparatus comprising: a mouth area detection unit that detects a mouth area. 7. An image input means for continuously inputting an image of a person including a face image, and a pupil in a face central region including a pupil, a mouth, and a nostril from a plurality of continuous images input by the image input means. Extraction of an appropriate threshold value that can detect the nostrils and nostrils, and using that threshold value to binarize the entire face area and perform labeling processing to extract a single connected face area contained inside the hair Means for determining whether or not the ears are hidden by the hair using the ratio of the distance to the center coordinates of the pupils, the left and right ends of the face area extracted by the face area extraction means, and the ratio of the distance to the center coordinates of both pupils. A face area measuring means for obtaining the size of the face area not including the ears based on the result; and a face for scaling the image of the face area to an arbitrary size based on the size of the face area obtained by the face area measuring means. Region correction means; And a facial image processing apparatus. 8. An image input means for continuously inputting an image of a person including a face image, an image storage means for storing a plurality of the images input by the image input means, and an image storage means for storing the plurality of images input by the image input means. Face area extracting means for extracting a face area in the extracted image, pupil detecting means for detecting a pupil area from the face areas extracted by the face area extracting means, and a face extracted by the face area extracting means. A mouth detecting means for detecting a mouth area from the area; a pupil state determining means for determining a state of the pupil area detected by the pupil detecting means; a mouth state for determining a state of the mouth area detected by the mouth detecting means Determining means, a face state determining means for determining the state of the extracted face area based on the determination made by the pupil state determining means and the mouth state determining means, and based on the determination result determined by the face state determining means The best face Image selecting means for automatically selecting an image including a face image in a state from a plurality of images stored by the image storing means, detecting a size of the face image in the image selected by the image selecting means, and selecting an arbitrary size A face image processing apparatus comprising: a face area correction unit that scales up and down. 9. An image inputting step of continuously inputting an image of a person including a face image; and a plurality of continuous images input in the image inputting step.
An image selecting step of determining the state of the face of the person and selecting an image including an optimal face image. 10. An image inputting step of continuously inputting images of a plurality of persons including a face image; and a plurality of continuous images input in the image inputting step.
An image selecting step of determining the state of the faces of the plurality of persons and selecting and outputting an image in which the states of the faces of the plurality of persons are optimal. 11. An image inputting step of continuously inputting images of a plurality of persons including a face image; and a plurality of continuous images input in the image inputting step.
The image selection step of determining the state of the faces of the plurality of persons, and selecting an image that is the optimum state of the faces of all of the plurality of persons for each of the plurality of face images of the persons, A face image synthesizing step of synthesizing optimal face images of the plurality of selected persons and outputting the synthesized image as one image. 12. An image inputting step of continuously inputting a plurality of images of a person including a face image; and a plurality of continuous images input in the image inputting step.
The number of attributes to determine the composition of the number of persons for each attribute of the plurality of persons in the shooting area by comparing the state of the faces of the plurality of persons with the average face value of each attribute such as gender, nationality, adult / child, etc. A face image processing method, comprising: a determining step. 13. An image inputting step of continuously inputting an image of a person including a face image, and a pupil area detecting a pupil area of the face image of the person from a plurality of continuous images input in the image inputting step. A detection step, a pupil area fluctuation state detection step of detecting an image of a continuous pupil area for a predetermined time of the pupil area detected in the pupil area detection step and outputting a fluctuation state thereof, and the pupil area fluctuation state detection step Based on the fluctuation state output in step (1), the image is compared with a previously stored pupil dictionary to select and output an image including a face image having a pupil region closest to the state desired by the photographer. A face image processing method, comprising: an image selecting step. 14. An image inputting step of continuously inputting an image of a person including a face image, and detecting a position of a pupil and a nostril of the face of the person from a plurality of continuous images input in the image inputting step. A mouth region candidate determining step of determining a mouth region candidate based on this, and a binarized image using a threshold from which only the dark region of the mouth region candidate determined in the mouth region candidate determining step is extracted as a reference image By changing the threshold value in the direction in which the number of dark pixels gradually increases, binarization is performed to obtain a difference image from the reference image, and the obtained difference image is labeled to detect a label having a predetermined size. A face image processing method comprising: a mouth area detection step of detecting a mouth area. 15. An image inputting step of continuously inputting an image of a person including a face image, and a pupil in a face central region including a pupil, a mouth, and a nostril from a plurality of continuous images input in the image inputting step. Extraction of an appropriate threshold value that can detect the nostrils and nostrils, and using that threshold value to binarize the entire face area and perform labeling processing to extract a single connected face area contained inside the hair And determining whether the ears are hidden by the hair by using the ratio of the distance to the center coordinates of both pupils and the left and right ends of the face area extracted in the face area extraction step. A face area measuring step of obtaining the size of the face area not including the ears based on the result; and a face for scaling the image of the face area to an arbitrary size based on the size of the face area obtained in the face area measuring step. And an area correction step. A face image processing method characterized by the above-mentioned. 16. An image inputting step of continuously inputting an image of a person including a face image, an image storing step of storing a plurality of the images input in the image inputting step, and storing by the image storing step A face area extracting step of extracting a face area in the image obtained, a pupil detecting step of detecting a pupil area from the face areas extracted in the face area extracting step, and a face extracted in the face area extracting step. A mouth detection step of detecting a mouth area from the area, a pupil state determination step of determining a state of the pupil area detected in the pupil detection step, and a mouth state of determining a state of the mouth area detected in the mouth detection step A determination step, a face state determination step of determining a state of the extracted face region based on the determination made in the pupil state determination step and the mouth state determination step, and a determination result determined in the face state determination step. The best An image selection step of automatically selecting an image including the face image in the state from the plurality of images stored in the image storage step; detecting a size of the face image in the image selected in the image selection step; A face image processing method, comprising: