JP2006133824A - Method and apparatus for image processing, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method, etc. capable of accurately detecting a chin position in a face photograph image. <P>SOLUTION: An eye detection portion 30 detects each position of both eyes in a face photograph image S0. A chin area estimation portion 70a estimates a chin area constituted of a chin portion and a neighboring area of the chin, based on each position of both eyes. Also, a chin position detection portion 90 detects the chin position in the chin area in which non-skin color pixels are eliminated by a non-skin color pixel elimination portion 80. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing method and apparatus for acquiring a chin position from a face photograph image, and a program therefor.

  When applying for a passport or a license, or creating a resume, it is often required to submit a photo of a predetermined output standard (hereinafter referred to as a certification photo) showing the person's face. For this reason, a photo room is provided for taking pictures of the user, and the ID photo sheet is created by photographing the user sitting on the chair in the photo room and recording the face photo image for the ID photo of the user on the sheet. ID photo automatic creation devices that automatically do this have been used. Since such an automatic creation device is large and has a limited installation location, it is necessary for the user to go to the location where the automatic creation device is installed in order to obtain an ID photo. Inconvenient.

  In order to solve this problem, for example, as described in Patent Document 1, a face photo image (an image showing a face) used for creating an ID photo is displayed on a display device such as a monitor. When the head position and the tip position of the chin (hereinafter referred to as the chin position) in the displayed face photograph image are designated, the computer enlarges the face based on the two designated positions and the output standard of the identification photograph. The image is enlarged and reduced by obtaining the reduction ratio and the position of the face, and the ID photo image is obtained by trimming the enlarged face photo image so that the face in the enlarged image is arranged at a predetermined position in the ID photo. A method of forming has been proposed. With this method, the user can request the creation of ID photos from DPE stores, etc., which exist more than the ID photo auto-creating device, and the photo of the photo on hand is good. By bringing a photographic film or recording medium on which such a favorite photograph is recorded to a DPE store or the like, it is possible to create an ID photo from the favorite photograph.

  However, in this technique, the operator needs to perform a complicated operation of designating the head position and the chin position with respect to the displayed face photograph image. For example, the burden on the operator is large. Also, since the color of the chin and the color of the neck part under the chin are the same skin color, especially when the area of the face area in the face photo image is small or the resolution of the face photo image is rough, etc. In this case, it is difficult for the operator to quickly and accurately indicate the position of the jaw, and there is a problem that it is impossible to create an ID photo appropriately and quickly.

  Compared to other parts such as eyes and mouth on the face, the color of the chin and the color of the neck under the chin are the same skin color, and the strength of the edge formed between the chin and neck is also compared. Therefore, it is difficult to detect using image processing technology.

  Patent Documents 2 and 3 are based on the assumption that the eyes and mouth are detected from the face, and the ratio between the distance between the jaw and mouth in the face and the distance between the eyes and mouth is substantially constant. A method for estimating the position of the jaw from the position of the eyes and mouth in the face photograph image has been proposed.

  Since it is more accurate to detect the position of the eyes and mouth from the face photograph image than the position of the jaw, the positions of the eyes and mouth are detected from the face photograph image by the methods proposed in Patent Document 2 and Patent Document 3, If the position of the jaw is estimated from the detected eye and mouth position, an ID photo can be created from the face photo image without requiring the operator to indicate the tip position of the jaw.

Similarly, specifying the eye position and mouth position is more appropriate and quicker than the jaw position, so let the operator specify the eye position and mouth position, and then estimate the jaw position from the specified position. Even so, the ID photo can be created from the face photo image without requiring the operator to indicate the position of the tip of the jaw.
JP-A-11-341272 JP 2004-5384 A JP2004-96486

  However, the positional relationship between the parts on the face is not always constant. In particular, the positional relationship between the jaw and other parts varies depending on the race, gender, age, etc., as there are individual differences such as those with long jaws and people with short jaws. Also, it is difficult to say that the positional relationship between the chin and other parts is constant, depending on factors such as whether you are fat or thin, expressionless or laughing.

  In order to avoid specifying the jaw position by the operator, the method proposed in Patent Document 2 and Patent Document 3 is based on the position of the jaw from the position of the eyes and mouth that is easier to detect and specify by the operator than the position of the jaw. Therefore, an average jaw position can be obtained, but an accurate jaw position cannot be obtained. As a result, there is a problem that, for example, trimming processing that requires the position of the jaw cannot be performed properly, and a good ID photo cannot be obtained.

  In view of the above circumstances, an object of the present invention is to provide an image processing method and apparatus capable of accurately detecting the position of a jaw in a face photographic image, and a program therefor.

The first image processing method of the present invention detects the approximate position and size of a face from a face photograph image,
Based on the detected position and size of the face, estimate a jaw range consisting of the jaw in the face and a region near the jaw;
In the estimated region, the position of the jaw is detected.

  In the present invention, the “jaw range” is a range composed of the jaw and a region near the jaw, and is a range where the possibility of the presence of the jaw is large and the region is as small as possible. In estimating the jaw range, it is desirable to estimate so as to eliminate as much as possible other parts of the face other than the jaws and clothes on the neck.

The second image processing method of the present invention obtains the positions of both eyes in a face photo image,
Based on the position of both eyes in the acquired face photo image, estimate a jaw range consisting of the jaw in the face photo image and a region near the jaw,
The position of the jaw is detected within the estimated range.

The first image processing apparatus of the present invention includes a face detection means for acquiring an approximate face position and size from a face photograph image;
Jaw range estimation means for estimating a jaw range composed of a jaw in the face and a region near the jaw based on the detected position and size of the face;
Jaw position detecting means for detecting the position of the jaw in the estimated range is provided.

The second image processing apparatus of the present invention includes eye position acquisition means for acquiring the positions of both eyes in a face photo image,
Jaw range estimation means for estimating a jaw range consisting of a jaw in the face photograph image and a region near the jaw based on the acquired positions of both eyes;
Jaw position detection means for detecting the position of the jaw within the estimated range is provided.

  Here, the eye position acquisition unit may be an input unit that allows the operator to specify the positions of both eyes. In order to reduce the burden on the operator, the eye position acquisition unit is an eye detection unit that detects the positions of both eyes from the face photograph image. Preferably there is.

The image processing apparatus of the present invention further includes mouth position acquisition means for acquiring the position of the mouth in the face,
It is preferable that the jaw range estimation means estimates the jaw range below the position of the mouth.

The image processing apparatus of the present invention further comprises non-skin color pixel removing means for detecting and removing pixels other than the skin color among the pixels in the jaw range estimated by the jaw range estimating means,
It is preferable that the jaw position detecting means detects the position of the jaw in the jaw range from which pixels other than the skin color are removed.

The jaw position detecting means in the image processing apparatus of the present invention projects the pixel value of each pixel in the jaw range with respect to an axis along the vertical direction of the face, for each height in the axial direction. A projection means for obtaining projection information of
Based on the projection information, an edge strength acquisition means for obtaining an edge strength for each height;
It is preferable that the image forming apparatus includes jaw position determining means for determining the position of the pixel corresponding to the height having the largest edge intensity as the position of the jaw.

  You may provide the image processing method of this invention as a program for making a computer perform.

  According to the first image processing method and apparatus of the present invention, the approximate position and size of the face are detected from the face photograph image, and the jaw and the vicinity of the jaw are based on the detected position and size of the face. Estimate the jaw range consisting of Then, the position of the jaw is detected in the estimated jaw range. Even if the approximate position and size of the face are known, the position of the jaw in the face varies from person to person, so the position of the jaw cannot be estimated accurately, but the approximate jaw range can be estimated. Then, by detecting the position of the jaw in the jaw range estimated in this way, the position of the jaw can be accurately obtained despite individual differences.

  According to the second image processing method and apparatus of the present invention, the positions of both eyes in the face photograph image are acquired, and the jaw range is estimated based on the acquired positions of both eyes. Since the position of the jaw is detected in the estimated range, the same effect as that of the first image processing method and apparatus can be obtained.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an image processing system A according to the first embodiment of the present invention. The image processing system A of the present invention detects the position of the jaw from the face photograph image (hereinafter referred to as a photographic image for short) S0, and the process for detecting the position of the jaw is read into the auxiliary storage device. This is realized by executing the processing program on a computer (for example, a personal computer). Further, this processing program is stored in an information storage medium such as a CD-ROM, or distributed via a network such as the Internet and installed in a computer.

  As shown in the figure, the image processing system A of the present embodiment detects an approximate position and size of a face in an image input unit 10 that inputs a photographic image S0 and a photographic image input by the image input unit 10, A face detection unit 20 that obtains a face image (hereinafter referred to as a face image) S1, an eye detection unit 30 that detects the positions of both eyes from the face image S1, and the face detection unit 20 and the eye detection unit 30 described later. A database 40a that stores reference data E1 and E2, a skin color pixel detection unit 50 that detects skin color pixels from the face image S1, and a mouth detection unit 60 that detects the position of the mouth based on the detection result of the skin color pixel detection unit 50; The jaw range estimation unit 70a for estimating the jaw range based on the positions of both eyes detected by the eye detection unit 30 and the mouth position detected by the mouth detection unit 60; Based on the result, the non-skin color pixel removing unit 80 that removes pixels that are not skin color pixels among the pixels of the jaw range estimated by the jaw range estimation unit 70a, and the jaw range from which the non-skin color pixels are removed, A jaw position detection unit 90 that detects the position of the jaw and an output unit 100 that outputs information indicating the jaw position obtained by the jaw position detection unit 90 and performs processing such as trimming are provided.

  The image input unit 10 inputs a photographic image S0 to be processed to the image processing system A of the present embodiment. For example, the image input unit 10 receives a photographic image S0 transmitted via a network, a CD- A reading unit that reads a photographic image S0 from a recording medium such as a ROM, a scanner that reads an image printed (including a print) on a printing medium from a printing medium such as paper or print paper by photoelectric conversion, and the like. It can be.

  FIG. 2 is a block diagram showing a configuration of the face detection unit 20 in the image processing system A shown in FIG. The face detection unit 20 detects an approximate position and size of the face in the photographic image S0, extracts an image of an area indicated by the position and size from the photographic image S0, and obtains a face image S1. As shown in FIG. 2, a face that performs face detection using the first feature quantity calculation unit 22 that calculates the feature quantity C0 from the photographic image S0, and the feature quantity C0 and the reference data E1 stored in the database 40a. And a detection execution unit 24. Here, the detail of each structure of the reference data E1 memorize | stored in the database 40a and the face detection part 20 is demonstrated.

  The first feature amount calculation unit 22 of the face detection unit 20 calculates a feature amount C0 used for face identification from the photographic image S0. Specifically, the gradient vector (that is, the direction in which the density of each pixel on the photographic image S0 changes and the magnitude of the change) is calculated as the feature amount C0. Hereinafter, calculation of the gradient vector will be described. First, the first feature amount calculation unit 22 performs a filtering process on the photographic image S0 using a horizontal edge detection filter shown in FIG. 5A to detect a horizontal edge in the photographic image S0. Further, the first feature amount calculation unit 22 performs filtering processing by the vertical edge detection filter shown in FIG. 5B on the photographic image S0 to detect the vertical edge in the photographic image S0. Then, a gradient vector K at each pixel is calculated from the horizontal edge size H and the vertical edge size V at each pixel on the photographic image S0, as shown in FIG.

  It should be noted that the gradient vector K calculated in this way is an eye in a dark part such as the eyes and mouth as shown in FIG. 7B in the case of a human face as shown in FIG. It faces the center of the mouth and faces outward from the position of the nose in a bright part like the nose. Further, since the change in density is larger in the eyes than in the mouth, the gradient vector K is larger in the eyes than in the mouth.

  The direction and magnitude of the gradient vector K are defined as a feature amount C0. The direction of the gradient vector K is a value from 0 to 359 degrees with reference to a predetermined direction of the gradient vector K (for example, the x direction in FIG. 6).

  Here, the magnitude of the gradient vector K is normalized. This normalization obtains a histogram of the magnitude of the gradient vector K in all the pixels of the photographic image S0, and the distribution of the magnitudes is a value that each pixel of the photographic image S0 can take (0 to 255 if 8 bits). The histogram is smoothed so as to be uniformly distributed, and the magnitude of the gradient vector K is corrected. For example, when the gradient vector K is small and the histogram is distributed with the gradient vector K biased toward the small side as shown in FIG. The magnitude of the gradient vector K is normalized so that it extends over the region so that the histogram is distributed as shown in FIG. In order to reduce the calculation amount, as shown in FIG. 8C, the distribution range in the histogram of the gradient vector K is divided into, for example, five, and the frequency distribution divided into five is shown in FIG. 8D. It is preferable to normalize so that the value of 0 to 255 is in a range divided into five.

  The reference data E1 stored in the database 40a identifies, for each of a plurality of types of pixel groups composed of a combination of a plurality of pixels selected from a sample image, which will be described later, with respect to a combination of feature amounts C0 in each pixel constituting each pixel group. The conditions are specified.

  In the reference data E1, the combination and identification condition of the feature amount C0 in each pixel constituting each pixel group are a plurality of sample images that are known to be faces and a plurality of sample images that are known not to be faces. It is predetermined by learning a sample image group consisting of

  In the present embodiment, when the reference data E1 is generated, the sample image that is known to be a face has a 30 × 30 pixel size, and as shown in FIG. The distance between the centers of both eyes of the image is 10 pixels, 9 pixels, and 11 pixels, and the face standing vertically at the distance between the centers of both eyes is rotated stepwise by 3 degrees within a range of ± 15 degrees on the plane. (That is, the rotation angle is -15 degrees, -12 degrees, -9 degrees, -6 degrees, -3 degrees, 0 degrees, 3 degrees, 6 degrees, 9 degrees, 12 degrees, 15 degrees) To do. Therefore, 3 × 11 = 33 sample images are prepared for one face image. In FIG. 9, only sample images rotated at −15 degrees, 0 degrees, and +15 degrees are shown. The center of rotation is the intersection of the diagonal lines of the sample image. Here, if the distance between the centers of both eyes is a 10-pixel sample image, the center positions of the eyes are all the same. The center position of this eye is set to (x1, y1) and (x2, y2) on the coordinates with the upper left corner of the sample image as the origin. In addition, the eye positions in the vertical direction in the drawing (ie, y1, y2) are the same in all sample images.

  As a sample image that is known not to be a face, an arbitrary image having a 30 × 30 pixel size is used.

  Here, as a sample image that is known to be a face, learning is performed using only a center image whose distance between the centers of both eyes is 10 pixels and the rotation angle on the plane is 0 degree (that is, the face is vertical). When performed, only the face which is identified as a face by referring to the reference data E1 is a face which is not rotated at all with a distance between the centers of both eyes of 10 pixels. Since the size of a face that may be included in the photographic image S0 is not constant, when identifying whether or not a face is included, the size of the sample image is enlarged by scaling the photographic image S0 as described later. It is possible to identify the position of a face of a size that fits. However, in order to accurately set the distance between the centers of both eyes to 10 pixels, the size of the photographic image S0 needs to be identified while being enlarged or reduced in steps of, for example, 1.1 units as an enlargement ratio. Will be enormous.

  Further, the face that may be included in the photographic image S0 is not only rotated at 0 degree on the plane as shown in FIG. 11A, but is rotated as shown in FIGS. 11B and 11C. Sometimes it is. However, when learning is performed using only a sample image in which the distance between the centers of both eyes is 10 pixels and the rotation angle of the face is 0 degrees, FIGS. As shown in (), the rotated face cannot be identified.

  Therefore, in this embodiment, as a sample image known to be a face, the distance between the centers of both eyes is 9, 10, 11 pixels as shown in FIG. 9, and ± 15 degrees on the plane at each distance. In this range, a sample image obtained by rotating the face step by step in units of 3 degrees is allowed to learn the reference data E1. As a result, when the face detection execution unit 24 to be described later performs identification, the photographic image S0 may be enlarged or reduced in steps of 11/9 as an enlargement rate. For example, the calculation time can be reduced as compared with the case where the enlargement / reduction is performed in steps of 1.1 units. In addition, as shown in FIGS. 11B and 11C, a rotating face can be identified.

  Hereinafter, an example of a learning method for the sample image group will be described with reference to the flowchart of FIG.

The creation of a classifier will be described with reference to FIG. As shown in the sample image on the left side of FIG. 13, each pixel constituting the pixel group for creating the discriminator is a pixel at the center of the right eye on a plurality of sample images that are known to be faces. P1, a pixel P2 on the right cheek, a pixel P3 on the forehead, and a pixel P4 on the left cheek. Then, combinations of feature amounts C0 in all the pixels P1 to P4 are obtained for all sample images that are known to be faces, and a histogram thereof is created. Here, the feature amount C0 represents the direction and magnitude of the gradient vector K. Since the gradient vector K has 360 directions from 0 to 359 and the gradient vector K has 256 sizes from 0 to 255, If it is used as it is, the number of combinations is 360 × 256 four pixels per pixel, that is, (360 × 256) ^four , and the number of samples, time and memory for learning and detection are large. Will be required. For this reason, in this embodiment, the gradient vector directions are 0 to 359, 0 to 44, 315 to 359 (right direction, value: 0), 45 to 134 (upward value: 1), and 135 to 224 (left). Direction, value: 2), 225-314 (downward, value 3), and quaternarization, and the gradient vector magnitude is ternarized (value: 0-2). And the value of a combination is computed using the following formula | equation.

Combination value = 0 (when gradient vector size = 0)
Combination value = ((gradient vector direction + 1) × gradient vector magnitude (gradient vector magnitude> 0)
Thus, since the number of combinations is nine patterns ^4, can reduce the number of data of the characteristic amounts C0.

  Similarly, histograms are created for a plurality of sample images that are known not to be faces. For the sample image that is known not to be a face, pixels corresponding to the positions of the pixels P1 to P4 on the sample image that is known to be a face are used. A histogram used as a discriminator shown on the right side of FIG. 13 is a histogram obtained by taking logarithmic values of ratios of frequency values indicated by these two histograms. The value of each vertical axis indicated by the histogram of the discriminator is hereinafter referred to as an identification point. According to this classifier, an image showing the distribution of the feature quantity C0 corresponding to the positive identification point is highly likely to be a face, and it can be said that the possibility increases as the absolute value of the identification point increases. Conversely, an image showing the distribution of the feature quantity C0 corresponding to the negative identification point is highly likely not to be a face, and the possibility increases as the absolute value of the identification point increases. In step S <b> 2, a plurality of classifiers in the above-described histogram format are created for combinations of feature amounts C <b> 0 in the respective pixels constituting a plurality of types of pixel groups that can be used for identification.

  Subsequently, the most effective classifier for identifying whether or not the image is a face is selected from the plurality of classifiers created in step S2. The most effective classifier is selected in consideration of the weight of each sample image. In this example, the weighted correct answer rate of each classifier is compared, and the classifier showing the highest weighted correct answer rate is selected (S3). That is, in the first step S3, since the weight of each sample image is equal to 1, the number of sample images in which the image is correctly identified by the classifier is simply the largest. Selected as a valid discriminator. On the other hand, in the second step S3 after the weight of each sample image is updated in step S5, which will be described later, a sample image with a weight of 1, a sample image with a weight greater than 1, and a sample image with a weight less than 1 The sample images having a weight greater than 1 are counted more in the evaluation of the correct answer rate because the weight is larger than the sample images having a weight of 1. Thereby, in step S3 after the second time, more emphasis is placed on correctly identifying a sample image having a large weight than a sample image having a small weight.

  Next, the correct answer rate of the classifiers selected so far, that is, the result of identifying whether each sample image is a face image using a combination of the classifiers selected so far, is actually It is ascertained whether or not the rate that matches the answer indicating whether the image is a face image exceeds a predetermined threshold (S4). Here, the sample image group to which the current weight is applied or the sample image group to which the weight is equal may be used for evaluating the correct answer rate of the combination. When the predetermined threshold value is exceeded, learning can be completed because it is possible to identify whether the image is a face with a sufficiently high probability by using the classifier selected so far. If it is less than or equal to the predetermined threshold, the process advances to step S6 to select an additional classifier to be used in combination with the classifier selected so far.

  In step S6, the discriminator selected in the most recent step S3 is excluded so as not to be selected again.

  Next, the weight of the sample image that could not be correctly identified as a face by the classifier selected in the most recent step S3 is increased, and the sample image that can be correctly identified as whether or not the image is a face is increased. The weight is reduced (S5). The reason for increasing or decreasing the weight in this way is that in selecting the next discriminator, an image that cannot be discriminated correctly by the already selected discriminator is regarded as important, and whether or not those images are faces can be discriminated correctly. This is to increase the effect of the combination of the discriminators by selecting the discriminators.

  Subsequently, the process returns to step S3, and the next valid classifier is selected based on the weighted correct answer rate as described above.

  By repeating the above steps S3 to S6, the classifier corresponding to the combination of the feature amount C0 in each pixel constituting the specific pixel group is selected as a classifier suitable for identifying whether or not a face is included. If the correct answer rate confirmed in step S4 exceeds the threshold value, the type of the discriminator used for discriminating whether or not a face is included and the discriminating condition are determined (S7), thereby the reference data E1. Finish learning.

  In the case of adopting the above learning method, the discriminator provides a reference for discriminating between a face image and a non-face image using a combination of feature amounts C0 in each pixel constituting a specific pixel group. As long as it is not limited to the above histogram format, it may be anything, for example, binary data, a threshold value, a function, or the like. Further, even with the same histogram format, a histogram or the like indicating the distribution of difference values between the two histograms shown in the center of FIG. 13 may be used.

  Further, the learning method is not limited to the above method, and other machine learning methods such as a neural network can be used.

  The face detection execution unit 24 refers to the identification conditions learned by the reference data E1 for all the combinations of the feature amounts C0 in the respective pixels constituting the plural types of pixel groups, and the features in the respective pixels constituting the respective pixel groups. An identification point for the combination of the quantity C0 is obtained, and a face is detected by combining all the identification points. At this time, the direction of the gradient vector K that is the feature amount C0 is quaternized and the magnitude is ternary. In the present embodiment, all the identification points are added, and a face is detected based on the positive / negative and magnitude of the added value. For example, if the total sum of the identification points is a positive value, it is determined that the face is present, and if the sum is negative, it is determined that the face is not a face.

  Here, unlike the sample image of 30 × 30 pixels, the size of the photographic image S0 may have various sizes. When a face is included, the rotation angle of the face on the plane is not always 0 degrees. For this reason, as shown in FIG. 14, the face detection execution unit 24 enlarges or reduces the photographic image S0 stepwise until the vertical or horizontal size becomes 30 pixels and rotates it 360 degrees stepwise on the plane. (In FIG. 14, a state of reduction is shown) A mask M having a size of 30 × 30 pixels is set on the photographic image S0 enlarged and reduced at each stage, and the mask M is set to one pixel on the photographic image S0 enlarged and reduced. While moving the images one by one, it is determined whether or not the image in the mask is a face image (that is, whether or not the added value of the identification points obtained for the image in the mask is positive or negative). Then, this identification is performed on the photographic image S0 at all stages of enlargement / reduction and rotation, and the mask identified from the photographic image S0 of the size and rotation angle at the stage where the positive value with the highest added value of the identification points is obtained. An area of 30 × 30 pixels corresponding to the position of M is detected as a face area, and an image of this area is extracted as a face image S1 from the photographic image S0.

  Since the sample images learned at the time of generating the reference data E1 have 9, 10, and 11 pixels at the center position of both eyes, the enlargement ratio when the photographic image S0 is enlarged / reduced is 11 / 9 is enough. In addition, since the sample image learned at the time of generating the reference data E1 uses a face rotated within a range of ± 15 degrees on the plane, the photographic image S0 can be rotated 360 degrees in units of 30 degrees. Good.

  Note that the first feature amount calculation unit 22 calculates the feature amount C0 at each stage of deformation such as enlargement / reduction and rotation of the photographic image S0.

  In this way, the face detection unit 20 detects the approximate position and size of the face from the photographic image S0, and obtains the face image S1.

  FIG. 3 is a block diagram illustrating a configuration of the eye detection unit 30. The eye detection unit 30 detects the positions of both eyes from the face image S1 obtained by the face detection unit 20, and as illustrated, a second feature amount calculation unit that calculates a feature amount C0 from the face image S1. 32, and an eye detection execution unit 34 that detects the position of the eye based on the feature amount C0 and the reference data E2 stored in the database 40a.

  In the present embodiment, the eye position identified by the eye detection execution unit 34 is the center position (indicated by x in FIG. 4) between the corners of the eyes and the eyes, as shown in FIG. 4 (a). In the case of an eye facing directly in front, it is the same as the center position of the pupil. However, in the case of an eye facing right as shown in FIG. 4B, not the center position of the pupil but a position deviating from the center of the pupil. Or located in the white eye area.

  The second feature quantity calculation unit 32 is the same as the first feature quantity calculation unit 22 in the face detection unit 20 shown in FIG. 2 except that the feature quantity C0 is calculated from the face image S1 instead of the photographic image S0. Therefore, detailed description thereof is omitted here.

  The second reference data E2 stored in the database 40b is similar to the first reference data E1, with respect to each of a plurality of types of pixel groups each including a combination of a plurality of pixels selected from a sample image to be described later. This specifies the identification condition for the combination of the feature values C0 in each pixel constituting the group.

  Here, in learning of the second reference data E2, as shown in FIG. 9, the distance between the centers of both eyes is 9.7, 10, 10.3 pixels, and each distance is within a range of ± 3 degrees on the plane. Sample images in which the face is rotated step by step by 1 degree. Therefore, the tolerance of learning is smaller than that of the first reference data E1, and the eye position can be accurately detected. Note that the learning for obtaining the second reference data E2 is the same as the learning for obtaining the first reference data E1 except that the sample image group used is different. Is omitted.

  The eye detection execution unit 34 performs identification in which the second reference data E2 has learned all the combinations of the feature amounts C0 in the respective pixels constituting the plurality of types of pixel groups on the face image S1 obtained by the face detection unit 20. With reference to the condition, an identification point for a combination of the feature amount C0 in each pixel constituting each pixel group is obtained, and the position of the eye included in the face is identified by combining all the identification points. At this time, the direction of the gradient vector K that is the feature amount C0 is quaternized and the magnitude is ternary.

  Here, the eye detection execution unit 34 enlarges / reduces the size of the face image S1 obtained by the face detection unit 20 in stages and rotates it 360 degrees on the plane in steps, and enlarges / reduces in each stage. A mask M having a size of 30 × 30 pixels is set on the face image, and the position of the eye in the image in the mask is detected while moving the mask M pixel by pixel on the enlarged / reduced face.

  Since the sample image learned at the time of generating the second reference data E2 has a number of pixels at the center position of both eyes of 9.07, 10, and 10.3 pixels, the face image S1 is enlarged or reduced. The enlargement ratio may be 10.3 / 9.7. Further, as the sample image learned at the time of generating the second reference data E2, a face image rotated in a range of ± 3 degrees on the plane is used, so the face image is rotated 360 degrees in units of 6 degrees. Just do it.

  Note that the second feature amount calculation unit 32 calculates the feature amount C0 at each stage of deformation of enlargement / reduction and rotation of the face image S1.

  In this embodiment, all the identification points are added at all stages of deformation of the face image S1, and the upper left corner is set as the origin in the image in the 30 × 30 pixel mask M at the stage of deformation having the largest added value. The coordinates corresponding to the coordinates (x1, y1) and (x2, y2) of the eye position in the sample image are obtained, and the position corresponding to this position in the face image S1 before deformation is set as the eye position. To detect.

In this way, the eye detection unit 30 detects the positions of both eyes from the face image S1 obtained by the face detection unit 20.

The skin color pixel detection unit 50 detects a skin color pixel from the face image S1 obtained by the face detection unit 20, and any existing method may be used as the detection method. For example, R, G, and B values are R, G, and B, respectively, and r and g expressed by “r = R / (R + G + B), g = G / (R + G + B)” are two coordinate axes 2 In the dimension plane, a skin color range is set in advance, and a pixel having a color included in the set skin color range may be detected as a skin color pixel.

The mouth detection unit 60 detects the position of the mouth in the vertical direction (a method perpendicular to the direction in which both eyes are aligned) from the face image S1 in the detection result of the skin color pixel detection unit 50. Any of these may be used. For example, first, the redness intensity of each pixel of the face image S1 that is not detected as a skin color pixel by the skin color pixel detection unit 50 (hereinafter referred to as a non-skin color pixel) is calculated. Then, the calculated redness intensity of each non-skin color pixel is accumulated in the horizontal direction of the face (direction perpendicular to the vertical direction) to obtain an accumulated value for each vertical height of the face. Then, the height corresponding to the highest cumulative value is detected as the position of the mouth in the vertical direction of the face.

  The jaw range estimation unit 70 a estimates the jaw range based on the positions of both eyes detected by the eye detection unit 30 and the position of the mouth detected by the mouth detection unit 60. The estimation of the jaw range is intended to accurately and quickly detect the position of the jaw, and the area where the jaw is present and other parts of the face other than the jaw and clothes on the neck are excluded as much as possible. It is desirable to estimate as a range. For example, an estimation method that determines only the upper limit of the jaw range (the jaw range is the uppermost position in the vertical direction of the face) so that the area below the center position of the face or the area below the eye position is estimated as the jaw range. The top part of the head and the eyes can be excluded, but in order to eliminate the clothes on the neck as much as possible, the upper part of the jaw area and the lower part of the jaw area (the jaw area is in the vertical direction of the face) It is preferable to determine the position of the lowermost end. Furthermore, since the error portions on the left and right sides of the lower part of the face form edges, it is preferable to exclude the error portions from the jaw range in order to detect the position of the jaw more accurately. Here, the jaw range estimation unit 70a will be described with reference to FIG.

  The jaw range estimation unit 70a in the image processing system A of the present embodiment estimates the jaw range as follows.

  1. First, a distance D between both eyes is obtained from the positions of both eyes (indicated by points A1 and A2 in FIG. 15), and the temporary jaw position is estimated based on the positions of both eyes and the distance D between the eyes.

  The size of a person's face varies, but except for special cases, the size of the face (width and height) has a corresponding relationship with the distance between the eyes and the distance from the eyes to the chin (both eyes). The vertical distance from the center position to the jaw) also has a corresponding relationship with the distance between the eyes. The jaw range estimation unit 70a in the present embodiment pays attention to this point, and first estimates the temporary position of the jaw by estimating the distance L from the eyes to the jaw according to the following equation (1).

L = u × D (1)
L: Estimated distance from eye to jaw
D: Distance between eyes
u: 2.170

Here, the coefficient u is obtained using a number of sample face photograph images.

  The jaw range estimation unit 70a is located below the eye position on the center line between the eyes, and the distance to the center position between the eyes (point A in the figure) is the distance L obtained according to the equation (1). (C in the figure as an example) is estimated as the temporary position of the jaw.

  2. Next, the jaw range estimation unit 70a estimates a frame indicating the jaw range centering on the temporary jaw position C. Specifically, first, the distance D between both eyes is set as the width of the frame. Then, on the center line between the eyes, the point B is below the position (point B in the figure) corresponding to the height of the mouth detected by the mouth detector 60, and between the point B and the temporary position C of the jaw. The center of the upper side of the frame is the position near the point (in this embodiment, the point (B1 in the figure) where the distance from the point B is 1/5 of the distance (d) between the point B and the temporary jaw position C). Let it be a point. That is, the jaw range estimation unit 70a estimates a frame (hatched portion in the figure) having a height of (2 × d) / 5 and a horizontal width D centered on the temporary position C of the jaw as the jaw range.

The jaw range estimation unit 70a estimates the jaw range in this way, and outputs information indicating the position and size of the jaw range to the non-skin color pixel removal unit 80.

  The non-skin color pixel removing unit 80 removes pixels that are not skin color pixels from the pixels in the jaw range obtained by the jaw range estimating unit 70a so that the position of the jaw can be detected more accurately. It is. Specifically, the non-skin color pixel removing unit 80 is a pixel other than the pixels detected as the skin color pixels by the skin color pixel detecting unit 50 among the pixels in the chin range obtained by the chin range estimating unit 70a. Remove skin color pixels.

  The jaw position detection unit 90 detects the position of the jaw within the jaw range from which the non-skin color pixel is removed by the non-skin color pixel removal unit 80. FIG. 16 is a block diagram illustrating a configuration of the jaw position detection unit 90. As illustrated, the jaw position detection unit 90 includes a projection processing unit 92, an edge strength detection unit 94, and a jaw position determination unit 96.

  Here, the jaw position detector 90 will be described with the axis in the direction in which both eyes are arranged in the face as the x axis and the axis in the direction perpendicular to the x direction as the y axis.

  First, the projection processing unit 92 sets the pixel value to y for the pixels in the jaw range estimated by the jaw range estimation unit 70a indicated by the hatched portion in FIG. 15 (non-skin color pixels have been removed). Project to the axis. That is, the pixel values (for example, luminance values) of pixels having the same height on the y axis are accumulated to obtain accumulated pixel values for each y axis height.

  The edge strength detection unit 94 detects the edge strength for each height on the y-axis using the accumulated pixel value. Specifically, for example, a differential filter as shown in FIG. 17 is applied to the accumulated pixel value in the direction along the y-axis to detect the edge intensity for each height on the y-axis.

  The chin position determination unit 96 sets the height of the edge strength among the edge strengths detected by the edge strength detection unit 94 as the height of the chin position, and this height on the center line between both eyes. The position corresponding to is detected as the jaw position.

  The output unit 100 outputs information indicating the position of the jaw detected by the jaw position detection unit 90, and uses the information for a trimming process that requires the position of the jaw.

  FIG. 18 is a flowchart showing processing of the image processing system A according to the embodiment shown in FIG. As shown in the figure, in the image processing system A according to the embodiment of the present invention, for the face photo image S0 input by the image input unit 10, first, the face detection unit 20 obtains the approximate position and size of the face. Face detection is performed (S10, S12). The position of both eyes is further detected by the eye detection unit 30 from the face image S1 obtained by the face detection unit 20 (S14). In parallel with the detection of the eye position, the skin color pixel detection unit 50 detects the skin color pixels in the face image S1 (S16). The mouth detection unit 60 detects the position of the mouth in the face image S1 using the detection result of the skin color pixel detection unit 50 (S18). Based on the position of both eyes obtained by the eye detection unit 30 and the position of the mouth obtained by the mouth detection unit 60, the jaw range is estimated by the jaw range estimation unit 70a (S20). The chin position detection unit 90 detects the position of the chin within the chin range from which the non-skin color pixels are removed by the non-skin color pixel removal unit 80 (S25, S30). Information indicating the position of the jaw detected by the jaw position detection unit 90 is output by the output unit 100 (S40), and the processing of the image processing system A ends.

  As described above, according to the image processing system A of the present embodiment, the positions of both eyes are detected from the face photograph image, the jaw range is estimated based on the detected positions of both eyes, and the estimated jaw range is within the estimated jaw range. Since the position of the jaw is detected, an accurate jaw position can be obtained by a conventional jaw position acquisition method that estimates the position of the jaw from the positions of both eyes. Furthermore, since the position of the mouth is also detected and the jaw range is set to a region below the mouth, it is possible to prevent a decrease in detection accuracy of the position of the jaw due to the influence of the mouth.

  Further, since the chin and the neck have substantially the same skin color, the edge where the chin and the neck are formed is weak, so it is difficult to detect the position of the chin by a normal edge detection method. The jaw position detection unit 90 projects the pixel values of the pixels in the jaw range onto the y-axis to obtain a cumulative pixel value, and obtains the edge strength for each height in the direction along the y-axis with the cumulative pixel value. Since the position of the jaw is determined based on the obtained edge strength, the position of the jaw can be reliably detected.

  FIG. 19 is a block diagram showing a configuration of an image processing system B according to the second embodiment of the present invention. As shown in the figure, the image processing system B of the present embodiment detects an approximate position and size of a face in an image input unit 10 that inputs a photographic image S0 and a photographic image input by the image input unit 10, A face detection unit 20 that obtains a face image S1, a database 40b that stores reference data E1 used in the face detection unit 20, a skin color pixel detection unit 50 that detects skin color pixels from the face image S1, and a skin color pixel detection unit 50 Mouth detection unit 60 that detects the position of the mouth based on the detection result, the face image S1 obtained by the face detection unit 20, and the jaw that estimates the jaw range based on the position of the mouth detected by the mouth detection unit 60 Based on the detection results of the range estimation unit 70b and the skin color pixel detection unit 50, non-skin color pixels that remove pixels that are not skin color pixels from pixels in the jaw range estimated by the jaw range estimation unit 70b In the jaw 80 from which the non-skin color pixels have been removed, the leaving part 80, the jaw position detection part 90 for detecting the position of the jaw, and the information indicating the jaw position obtained by the jaw position detection part 90 are output. And an output unit 100 used for processing such as trimming.

  In addition, about each structure of the image processing system B of this embodiment, while attaching | subjecting the same code | symbol about the same thing as the corresponding structure in the image processing system A shown in FIG. 1, the detailed description is abbreviate | omitted.

  The jaw range estimation unit 70b of the image processing system B estimates the jaw range based on the face image S1 obtained by the face detection unit 20 and the mouth position obtained by the mouth detection unit 60. Specifically, as shown in FIG. 20, for example, a predetermined range below the mouth has the same center line as the center line of the face image S1, and the position of the base of the face image S1 is set as the position of the base. A frame whose width is ½ of the width of the face image S1 and whose height is 9/10 of the distance d from the mouth position B to the bottom of the face image S1 is estimated as a jaw range.

  FIG. 21 is a flowchart showing processing in the image processing system B of the present embodiment. As shown in the figure, in the image processing system B according to the present embodiment, for the face photograph image S0 input by the image input unit 10, first, the face detection unit 20 obtains the approximate position and size of the face. Face detection is performed (S50, S52). Skin color pixels are detected by the skin color pixel detection unit 50 from the face image S1 obtained by the face detection unit 20 (S54). Then, the mouth detection unit 60 detects the position of the mouth in the face image S1 using the detection result of the skin color pixel detection unit 50 (S56). Based on the face image S1 obtained by the face detection unit 20 and the mouth position obtained by the mouth detection unit 60, the jaw range is estimated by the jaw range estimation unit 70b (S58). The chin position detection unit 90 detects the position of the chin within the chin range from which the non-skin color pixel removal unit 80 has removed the non-skin color pixels (S60, S65). When the information indicating the position of the jaw detected by the jaw position detection unit 90 is output by the output unit 100 (S70), the processing of the image processing system B ends.

  As described above, the image processing system B of the present embodiment uses the base of the face image S1 as the base of the frame of the jaw range, and thus the jaw estimated by the image processing system B of the embodiment shown in FIG. The height in the vertical direction is longer than the range, and there is a high possibility that clothes on the neck are included in the jaw range, but the jaw is detected by detection within the range where the jaw is present and other parts on the face are excluded. Since the position is acquired, the accurate jaw position can be obtained by the conventional jaw position acquisition method in which the position of the jaw is estimated from the positions of both eyes. Further, the non-skin color pixel removing unit 80 removes the non-skin color pixels from the pixels in the jaw range, thereby preventing the influence of clothes and the like and further improving the detection accuracy.

  The preferred embodiments of the image processing method and apparatus of the present invention and the program therefor have been described above, but the image processing method, apparatus and program of the present invention are not limited to the above-described embodiments, and do not depart from the gist of the present invention. As long as there are various changes, changes can be made.

  For example, in the image processing system A according to the embodiment shown in FIG. 1, the eye position is detected, but the user may designate it. Of course, the same applies to the position of the mouth.

  Further, the image processing system A according to the embodiment shown in FIG. 1 estimates the temporary position of the jaw according to the formula (1) based on the position of both eyes and the distance between both eyes. In addition to the position of both eyes, the temporary position of the jaw may be estimated based on the position of both eyes and the position of the mouth.

  In addition, the eye detection unit 30 in the image processing system A of the embodiment shown in FIG. 1 detects the center position of the eyes as the eye position, but the position of the pupil, the corner of the eye, the position of the eyes, etc. may be used. Of course.

  In addition, the detection of the face and the detection of the mouth is not limited to the method used in the above-described embodiment, and any method can be used as long as the approximate position and size of the face and the position of the mouth can be detected. It may be used.

1 is a block diagram showing a configuration of an image processing system A according to a first embodiment of the present invention. Block diagram showing the configuration of the face detection unit 20 The block diagram which shows the structure of the eye detection part 30 Diagram for explaining the center position of eyes (A) is a diagram showing a horizontal edge detection filter, (b) is a diagram showing a vertical edge detection filter Diagram for explaining calculation of gradient vector (A) is a figure which shows a person's face, (b) is a figure which shows the gradient vector of eyes and mouth vicinity of the person's face shown to (a). (A) is a diagram showing a histogram of magnitudes of gradient vectors before normalization, (b) is a diagram showing histograms of magnitudes of gradient vectors after normalization, and (c) is a magnitude of gradient vectors obtained by quinarization. The figure which shows the histogram of the length, (d) is a figure which shows the histogram of the magnitude | size of the quinary gradient vector after normalization The figure which shows the example of the sample image known to be a face used for learning of 1st reference data The figure which shows the example of the sample image known to be a face used for learning of 2nd reference data Illustration for explaining face rotation Flow chart showing learning method of reference data Diagram showing how to derive a classifier The figure for demonstrating the stepwise deformation | transformation of an identification object image The figure for demonstrating the jaw range estimation part 70a Block diagram showing the configuration of the jaw position detector 90 The figure which shows the example of the filter for calculating | requiring edge strength The flowchart which shows the process of the image processing system A of embodiment shown in FIG. The block diagram which shows the structure of the image processing system B used as the 2nd Embodiment of this invention. The figure for demonstrating the jaw range estimation part 70b The flowchart which shows the process of the image processing system B of embodiment shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image input part 20 Face detection part 22 1st feature-value calculation part 24 Face detection execution part 30 Eye detection part 32 2nd feature-value calculation part 34 Eye detection execution part 40a, 40b Database 50 Skin color pixel detection part 60 Mouth detection Unit 70a, 70b jaw range estimation unit 80 non-skin color pixel removal unit 90 jaw position detection unit 92 projection processing unit 94 edge strength detection unit 96 jaw position determination unit 100 output unit S0 face photograph image S1 face image E1, E2 reference data

Claims (10)

Detect approximate face position and size from face photo image,
Based on the detected position and size of the face, estimate a jaw range consisting of a jaw in the face and a region near the jaw;
An image processing method, wherein the position of the jaw is detected in the estimated region. Get the position of both eyes in the face photo image,
Based on the position of both eyes in the acquired face photo image, estimate a jaw range consisting of the jaw in the face photo image and a region near the jaw,
An image processing method comprising: detecting a position of the jaw within the estimated range. A face detection means for acquiring an approximate face position and size from a face photo image;
Jaw range estimation means for estimating a jaw range composed of a jaw in the face and a region near the jaw based on the detected position and size of the face;
An image processing apparatus comprising: a jaw position detecting means for detecting the position of the jaw in the estimated range. Eye position acquisition means for acquiring the position of each eye in the face photo image;
Jaw range estimation means for estimating a jaw range consisting of a jaw in the face photograph image and a region near the jaw based on the acquired positions of both eyes;
An image processing apparatus comprising: a jaw position detecting means for detecting the position of the jaw within the estimated range.   5. The image processing apparatus according to claim 4, wherein the eye position acquisition means is eye position detection means for detecting the positions of the eyes from the face photograph image. Mouth position obtaining means for obtaining the position of the mouth in the face,
The image processing apparatus according to claim 3, wherein the jaw range estimating unit estimates the jaw range below the position of the mouth. Non-skin color pixel removing means for detecting and removing pixels other than skin color among each pixel in the jaw range estimated by the jaw range estimating means,
The image processing according to any one of claims 3 to 6, wherein the jaw position detecting means detects the position of the jaw in the jaw range from which pixels other than the skin color are removed. apparatus. Projection means for obtaining projection information for each height in the axial direction, wherein the jaw position detection means projects the pixel value of each pixel in the jaw range to an axis along the vertical direction of the face. When,
Based on the projection information, an edge strength acquisition means for obtaining an edge strength for each height;
The image according to any one of claims 3 to 7, further comprising jaw position determining means for determining the position of the pixel corresponding to the height having the largest edge intensity as the position of the jaw. Processing equipment. A process for detecting the approximate position and size of a face from a face photo image;
Based on the detected position and size of the face, a process for estimating a jaw range consisting of a jaw in the face and a region near the jaw;
A program for causing a computer to execute processing for detecting the position of the jaw within the estimated range. Processing to obtain the positions of both eyes in the face photo image;
Based on the position of both eyes in the acquired face photograph image, a process of estimating a jaw range consisting of a jaw in the face photograph image and a region near the jaw;
A program for causing a computer to execute processing for detecting the position of the jaw within the estimated range.

Images