KR100950138B1 - A method for detecting the pupils in a face image - Google Patents

Abstract

PURPOSE: A method for detecting a pupil in a facial image is provided to promptly detect an eye through a mobile terminal. CONSTITUTION: An eye extraction range is set up in an inputted facial image, and the start position of pupil detection is determined in the facial image(102). A first template consisting of first and second masks is applied to an eye extraction range. Based on the difference between the brightness value in the region covered by a second mask and the brightness value of the region converted by a first mask, pupil candidate groups are detected(103). Among the candidate groups, the final pupil candidate group is determined based on the difference between the brightness values of inner and outer portions in a radius of pupil.

Description

A METHOD FOR DETECTING THE PUPILS IN A FACE IMAGE}

The present invention relates to a method of extracting a face shape from a face image file. More particularly, the present invention relates to a method of detecting a pupil from a face image, and more particularly, to a detection method capable of quickly and effectively detecting a pupil even in a mobile device. .

Background Art Conventionally, a technique for recognizing and detecting a face shape from a face image file has been widely known, and the face shape recognition technology has been applied in security-related fields such as identity verification, entertainment fields such as face synthesis, and second generation prediction. .

Particularly in the entertainment field, for example, each face shape detected from two face images is synthesized and displayed (face synthesis), or a makeup, hair, clothes, etc. are changed in a portrait photographed by a camera, and a different look is made (face). Decorating), or by combining photos of lovers and predicting the appearance of 2 years old (predicting 2 years old), etc., has become a new element of interest that induces expectations and curiosity for users.

In such a face shape extraction method, technically, the facial feature point is first detected first, and in the next step, application work such as face synthesis, face decoration, or second generation prediction is performed.

Facial feature point detection detects key features such as eyes, nose and mouth constituting the face, and is the most basic technique for subsequent operations such as face synthesis and face decoration. Fig. 1 simply shows an operation step for face feature point detection.

First, a face image file that is a face detection target is input to a device such as a personal computer or a face detection device (step 10). Thereafter, basic feature points are extracted from the input image (step 20). The basic feature point is a standard for all the feature points, and may be, for example, the center of the eyes and the lips. Once the basic feature point is extracted, the entire feature point is obtained based on this information (step 30). Global feature points can be, for example, eyebrows, eyelids, nose, lips, jawline and the like. In this way, the facial feature points are detected from the face image, and then application processes such as face synthesis, face decoration, and second generation prediction are performed using the detected feature points.

Recently, as mobile devices such as mobile phones are becoming more common, technology for taking, storing, and editing pictures with a camera embedded in a mobile phone is gradually developed. Therefore, face synthesis, transformation, and decoration can be performed on a mobile phone using face pictures. There is a technology out there that can do it right away.

However, since the performance of a processor embedded in a mobile device such as a cellular phone is remarkably different from a processor installed in a general face detection device or a desktop computer, the existing face detection method cannot be applied to a mobile device as it is. Furthermore, as mentioned above, it is important to perform basic feature point extraction quickly and accurately since the basic feature points must be detected quickly and accurately in face detection.

Therefore, there is a need for a novel face detection method capable of accurately detecting basic feature points such as, for example, pupils, and capable of quickly and smoothly operating on a mobile device.

In the present invention to achieve the above object, in the method for detecting the pupil in the face image,

Setting a range (eye extraction range) to detect pupils from the input face images and determining a start position of pupil detection on the face image;

By applying a first template consisting of a first mask and a second mask to the eye extraction range, the brightness value of the area covered by the first mask of the eye extraction range and the brightness value of the area covered by the second mask Detecting a pupil candidate group based on the difference; And

From the pupil candidate group, determining the final pupil candidate based on the difference between the brightness value inside the pupil radius and the outside brightness value; provides a pupil detection method comprising a.

By the above-described eye detection method, it is possible to accurately detect the position and size of the pupil in the face image and perform the eye detection quickly and efficiently on a mobile device, thereby detecting the entire feature point of the face and furthermore the face synthesis, It will have an effect that can perform a variety of applications such as face decorating more quickly.

In order to achieve the above object, the present invention provides a method for detecting a pupil in a face image, the method comprising: setting a range (eye extraction range) to detect a pupil from an input face image, and determining a start position of pupil detection on the face image; ; By applying a first template consisting of a first mask and a second mask to the eye extraction range, the brightness value of the area covered by the first mask of the eye extraction range and the brightness value of the area covered by the second mask Detecting a pupil candidate group based on the difference; And determining a final pupil candidate among the pupil candidate groups based on a difference between a brightness value inside a pupil radius and an outside brightness value.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

FIG. 2 is a flowchart corresponding to the basic feature point detection 20 of FIG. 1, and in particular, a flowchart illustrating an embodiment of detecting a pupil position among facial feature points.

In step 101, a face image is input and an initialization operation is performed to be suitable for detecting. Then, in order to determine at which position on the image the pupil detection is to be performed, a detection starting position is determined (102), and all pupils within a range are examined from the determined starting position to create a pupil candidate group (103). A final candidate is determined from the pupil candidate group (104), and the pupil detection is completed (105) through correction and redetection operations. However, according to an embodiment, the recalibration operation as shown in FIG. 9 may be additionally roughened, and thus, more precise eye detection may be performed.

3 shows an embodiment of the image preparation step 101 of FIG.

In order to detect pupils according to the present invention, a face image is input to a device (for example, a mobile phone, a desktop, or a dedicated detection device) on which a program implementing the pupil detection method of the present invention is installed, and an initialization operation is performed to be suitable for detecting. .

The input image may be an image file of a known file format such as JPG. Although the size or resolution of the input image may differ from image to image, the preferred embodiment of the present invention shown uses a 200 × 300 pixel image in width x height, so if the input image is different from this size, Convert to this size.

In order to detect the pupil, the pupil detection operation may be performed on only a part of the entire face image where the pupil is expected to be located. In the embodiment of the present invention, as shown in FIG. When (w, h) is set to (0,0), a rectangle with the corner where (w, h) = (0.1, 0.27) and (0.4, 0.6) becomes the eye extraction range for right eye detection The squares with the corners at the points (0.6, 0.27) and (0.9, 0.6) are set as the eye extraction range for left eye detection.

This area range is set based on the statistical basis that the pupil is located in the extraction range in almost all cases of the 200 × 300 pixel face photograph, and the range may be changed according to the embodiment.

Meanwhile, in the preferred embodiment, as shown in FIG. 3 (b), an image extracted by the R channel among the R, G, and B colors is used in the input color face image, and the white and the black color of the eyes are markedly different in skin color and R value. Because it shows. Additionally, an autocontrast filter may be applied one or more times to make the contrast more apparent (see FIG. 3 (c)).

After that, according to the coordinate values of the above-described automatic eye extraction range, two rectangles for pupil detection are cut out (see FIG. 3 (d)). Preferably, median filtering can be applied one or more times for each cropped extraction range, which is a 'highlight' that tends to appear on the eye blacks on the face image (the black part of the eyes in the picture taken under lighting). This is to remove noise such as the phenomenon of reflection in white by illumination (see Fig. 3 (e)).

Thereafter, the process proceeds to the detecting starting position search step 102 of FIG. 4A and 4B illustrate the step of finding the pupil detection starting position in the face image. This step is performed on the face image first input (i.e., the image in FIG. 3 (a)), and the probability of the pupil's position within the entire face image (by removing the unnecessary area such as the eyebrow part for eye detection). This is done for the purpose of detecting only this highest area.

First, the R channel is extracted from the color image as shown in FIG. Since this step is the same process as described with reference to FIG. 3 (b), the result performed in FIG. 3 (b) may be used as it is.

After the R channel extraction step, the extracted face image is cropped based on the topic point (the boundary between the skin and the hair). In general, the analysis of the shape of the face shows that the ratio of face length (not including both ears) to face length (from chin to forehead) is 1: 1.38 to 1: 1.4. Therefore, in one embodiment of the present invention, by using such statistics, the upper part of the face image, which is initially input so that the face chin is positioned at the lower end of the image, is removed, leaving only the vertical height of approximately 1.4 horizontal lengths. Second figure of FIG. 4A).

We then apply a 'mask' to this presented image to find the starting position for detection. In one embodiment, the mask uses the mask 11 shown on the left side of Fig. 4B. As shown in the drawing, the mask 11 is a rectangle composed of a white region 11-1 and two black regions 11-2 having a '凸' shape, and its size is preset. Also, in one embodiment, the length of the black region 11-2 is 1/2 of the length of the mask 11.

In one embodiment the mask 11 is applied as in the right figure of Figure 4b.

That is, the mask is positioned so that the point 11-4, which is 1/2 of the height of the mask 11, is located at the point 12-1, which is 1/3 of the height of the presented facial image. Next, the brightness average value of the portion of the image covered by the black area 11-2 of the mask and the area covered by the white area 11-1 of the mask are calculated, and the difference between the two average values is compared. do. Then, the mask 11 is lowered by one pixel (in the direction of the arrow 12-3) to calculate and compare the difference between the two average values as above, and in this way, calculate and compare the average value. The middle part 11-4 of the mask is gradually moved downward from the point 12-1 on the image in the direction of the arrow 12-3 to the point 12-2 of the image. do.

The difference between the average value of the image portion covered by the white region 11-1 of the mask 11 and the average value of the image portion covered by the black region 11-2 is large. Since the possibility of being located in the part covered by -2) is so large, the position of the mask is selected when the difference between the two average values is greatest. Next, when the position of the mask 11 is determined as mentioned above, the height below 1/4 of the mask height from the mask top 11-3 is set as the starting position y of the pupil detection, and the pixel at this time Remember the y position of.

The method then proceeds to step 103 of finding a pupil candidate group in FIG. In one embodiment, the pupil candidate group is searched for by applying the 'eye mask (horseshoe template)' shown in FIG. 6 to the extraction range shown in FIG. 3 (e).

In the present invention, the reason for using the shape of the horseshoe as shown in Fig. 6 is that although the actual human eye is circular, the upper eyelid slightly covers the upper part of the eye in the face usually taken in the picture (see Fig. 3, Fig. 4, etc.). For the purpose of eye detection, it is more effective to use a U-shaped mask instead of a circular mask.

As an embodiment of the present invention, the horseshoe template shown in FIG. 6 is composed of an inner mask and an outer mask, each of which has a size of 13 × 13 pixels and 31 × 31 pixels. The size of the inner mask and the outer mask is obtained by setting in advance how many pixels the pupil size is statistically when the size of the presented face of the person is 200 × 300 pixels, and the size may vary according to embodiments.

As shown in Fig. 5, the mask when the two masks are combined (hereinafter referred to as a horseshoe template) is applied to the extraction range of Fig. 3 (e) to find a pupil candidate group. For example, in the case of a left eye image, with the left side of the horseshoe template overlapping the left side of the left eye extraction range, the center of the horseshoe template is placed at the detection start position (y position) obtained in step 102 above. (See Figure 5 left), the average brightness of the image area covered by the inner mask of the horseshoe template ('average brightness by the internal mask') and the average brightness of the image area covered by the outer mask ('by the external mask'). Average brightness "),

'Average brightness by internal mask' <'Average brightness by external mask' * 0.57

It is determined whether the conditional expression is satisfied. At this time, the value of 0.57 is a value in a preferred embodiment selected by an experiment, and may be set differently according to the pixel size of the entire image, the contrast of the image, and the like.

Thereafter, the horseshoe template is moved to the right one pixel by one pixel until the right side of the horseshoe template reaches the right side of the left eye image, and the average value calculation and determination is continued, in this way the pixels below the y position of the eye extraction range. For, continue until the bottom side of the horseshoe template reaches the bottom of the eye extraction range. This operation is then performed for all cases where the exact center position of the horseshoe template is located inside the red rectangle shown in FIG. As described above, the average value calculation and determination are performed to store the x, y coordinate values of all pixels satisfying the conditional expression as the pupil candidate group, and in a preferred embodiment, the maximum number of pupil candidate groups is 1000.

 On the other hand, in one embodiment of the present invention, for the pupil candidate group that satisfies the above conditional expression, the pupil radius is also determined to distinguish between the black and white of the pupil. In one embodiment of obtaining a pupil radius, the brightness range of the image area covered by the inner mask is obtained by increasing pixels by 6 pixels within the range of 6 to 10 pixels within the radius range of the horseshoe template, and then the brightness of the area. The radius of the darkest and the largest radius is determined as the pupil radius.

In this way, the x, y coordinate values and the pupil radius at that time for each pixel of the pupil candidate group are stored in the memory through the above average value calculation, comparison, and pupil radius calculation.

The detection method according to the present invention then proceeds to the pupil final candidate determination step 104. In this step, the final pupil position is obtained for each pupil candidate based on the radius, the average radius inner brightness value and the radius outer brightness value average, and then select the candidate having the largest difference. If there is more than one candidate with the same difference, the candidate having the smallest internal brightness is selected. In addition, at this time, when the average value of the brightness value is calculated, it is calculated in the same form as the horseshoe template of FIG.

In addition, in an exemplary embodiment of the present invention, the average value of the inner highlight of the pupil is removed before the average of the internal brightness values is obtained (see FIG. 7). In one embodiment of removing highlights, for every pixel value inside the pupil. Pixels brighter than 50 (in case of dividing the brightness into 256 levels) above the first averaged brightness value are calculated by excluding them from the summation.

When the final candidate is determined from the pupil candidate group as described above, the correction and redetection operation for the pupil position determined in the next step 105 is performed. In this regard, Fig. 8 shows one preferred embodiment of the correction and redetection operations for the detected pupil positions.

In the above embodiment, template matching is performed based on the detected coordinates in order to determine whether the coordinates detected for each of the left and right pupils are correct. That is, when the eye extraction range of FIG. 3 (d) is folded and symmetrically overlapped in the center, theoretically, the detected right eye and left eye positions should coincide, but the actual detected position may be slightly different. In this case, when the distance difference between the candidate coordinates of both pupils is large (in one embodiment, when the distance difference is 20 pixels or more), the suitability of the left and right pupil candidate coordinates is respectively determined by using a template to determine whether the pupil is suitable as the pupil shape. By calculating, the one with the low suitability of the two is templated to the candidate groups and rediscovered with the high suitability. At this time, the template for the left eye can be created by creating a template for the left eye in advance, as shown in the left figure of FIG. 8, and a template for the right eye by mirroring the left eye template. have. In addition, since the template shape at this time applies to the already-detected and positioned pupil candidate, it does not need to be an exact eye shape, but may be preset to have a rough eye shape as shown in FIG.

On the other hand, if the distance difference between the candidate coordinates of both pupils is less than 20 pixels, the template is applied only for 5 × 5 pixel regions (ie 25 pixels) based on the two candidate coordinates to improve the pupil position coordinates more precisely. Can be.

As described above, according to the present invention, the final position of the pupil in the face image and the pupil radius at that time can be obtained by performing the above steps described with reference to FIGS. 3 to 8. The pupil position and radius determined by the method are then used as basic data to determine other basic feature points other than the pupil and the overall feature points of the eyebrows, nose, lips, jawline, etc. in subsequent steps.

However, although the detected pupil position and size (radius) may be used as it is, the correction and / or redetection of the pupil position may be further performed in order to detect more accurately.

In this regard, FIG. 9 is a flowchart illustrating a step of performing secondary correction and redetection on the detected pupil positions according to the embodiments of FIGS. 3 to 8. FIG. 10 is a flowchart illustrating the second correction and redetection. Illustrating an image initialization step, FIG. 11 is a diagram illustrating an embodiment of performing secondary correction and redetection.

The image preparation step 201 is a step of preparing an image necessary for performing finer coordinate detection based on the detected pupil position by performing up to FIG. 8. In one embodiment, an image of a certain size (eg, 60 × 40 pixels) is selected around each of the detected left and right pupils as shown in FIG. 10, and median filtering is performed twice, for example, to remove noise. The image is then enlarged 1.5 times and extracted into the R channel.

Thereafter, as shown in FIG. 11A, a step similar to the step of applying the horseshoe template of FIG. 6 is performed. In FIG. 11A, a second horseshoe-shaped template having a size of 8 to 19 pixels in radius is generated (or retrieved from the template previously stored in memory) and applied to the left and right eye images of FIG.

In a preferred embodiment the second horseshoe template has a size of 45 × 45 pixels. In addition, although the second horseshoe template may comprise the black region, the white region, and the calculation-free region as separate masks, in a preferred embodiment, as shown in FIG. 11A, the black masturbation region, the white region, and no calculation are required. The second horseshoe template is made up of one mask consisting of the areas.

As such, a second horseshoe template is created for each radius with a radius of 8 to 19 pixels, and the image covered by the white part of the second template is applied to pixels within 10 pixels of the top, bottom, left, and right positions of the detected eye center. The brightness average value of the area and the brightness average value of the image area covered by the black masturbation portion of the second template are calculated. Since the greater the difference in brightness average value means the closer to the center of the pupil, the pupil position when the difference in brightness average value is greatest is determined as the final pupil position.

As described above, since the second correction and redetection steps described with reference to FIGS. 9 to 11 are performed by applying a template to the results already detected through the steps up to FIG. 8, the second correction and redetection steps have more detailed correction effects. However, in the exemplary embodiment, the second correction and redetection step may be omitted since the pupil detection with the accuracy enough to be necessary may be performed by only the steps of FIGS. 3 to 8.

The present invention described above is not limited to the above-described embodiment and the accompanying drawings, and various modifications are possible. For example, the image preparation step 101 and the detection start position finding step 102 of FIG. 3 may be interchanged. In other words, step 102 may be performed first to determine a y value for applying a mask first, and then select an eye extraction range from the original face image to search for a pupil candidate group from the y value position. Also, the information about the mask 11, the horseshoe template of FIG. 6, the two eye templates of FIG. 8, and the second horseshoe template of FIG. 11A mentioned in an embodiment of the present invention may be stored in advance in the memory of the face detection device. Each time a specific template is required in each step, it may be generated and used according to a predetermined function for generating the template.

Applicable to security and entertainment industries.

1 is a flow chart briefly illustrating a step of detecting feature points of a face in a face image file;

2 is a flowchart showing a step of detecting a pupil position among facial feature points;

3 is a diagram illustrating an image preparation step for pupil detection;

4A and 4B illustrate a step of finding a pupil detecting starting position in a face image;

5 is a diagram illustrating a step of finding a pupil candidate group by pupil detection;

6 illustrates an embodiment of a mask used for pupil detection;

7 illustrates a step of determining a pupil final candidate;

8 illustrates a correction / redetection step of a pupil candidate;

9 is a flowchart showing a step of performing secondary correction / redetection on the pupil position detected by the steps of FIGS. 3 to 8;

10 is a diagram illustrating an image initialization step for the secondary correction / redetection;

Figure 11 illustrates one embodiment of performing secondary correction / redetection.

Claims (10)

In the method for detecting the pupil in the face image, Setting a range (eye extraction range) to detect pupils from the input face images and determining a start position of pupil detection on the face image; By applying a first template consisting of a first mask and a second mask to the eye extraction range, the brightness value of the area covered by the first mask of the eye extraction range and the brightness value of the area covered by the second mask Detecting a pupil candidate group based on the difference; And Determining a final pupil candidate among the pupil candidate groups based on a difference between a brightness value inside a pupil radius and an outside brightness value. The method of claim 1, wherein the determining of the pupil detection start position comprises: The third mask 11 including the first and second regions is applied to the face image to determine the brightness value of the portion covered by the first region and the brightness value of the portion covered by the second region. Determining the starting position of the pupil detection based on the difference. The method of claim 1, wherein the detecting of the pupil candidate group comprises obtaining a pupil radius for all detected pupil candidates. 4. The method according to any one of claims 1 to 3, In the detecting of the pupil candidate group, the first mask is a semicircular shape and an inner mask corresponding to the pupil black spot, and the second mask is an U-shaped outer mask corresponding to a portion other than the pupil black spot. Eye detection method in the face image. 4. The method according to any one of claims 1 to 3, In the step of determining the final pupil candidate, if the brightness value inside the pupil radius and the outside brightness value is the same, the pupil detection in the face image, characterized in that the candidate with the smallest internal brightness value is selected as the final pupil candidate Way. 6. The method of claim 5, further comprising the step of removing highlights in the pupil before obtaining the brightness value within the pupil radius. 4. The method according to any one of claims 1 to 3, And after the determining of the final pupil candidate, a first correction step of determining and correcting whether the determined coordinates of the final left and right pupil candidates coincide with the actual pupil coordinates. The method of claim 7, wherein the first correction step, Measuring a distance difference between final left and right pupil candidate coordinates when the eye extraction ranges for the left and right eyes are mirrored; And If the distance difference is larger than a predetermined value, the fitness for calculating whether the eye is suitable as the shape of the pupil is calculated, the coordinates having the highest suitability among the detected pupil candidate groups are re-detected for the lower-fitting eye among the left and right pupils, and the distance difference is If less than the predetermined value, calculating the goodness of fit of the pixels within a predetermined distance from the left and right pupil candidate coordinates to select the coordinates with the highest suitability; comprising a pupil detection method in the face image . The method of claim 7, wherein And after the first correction step, a second correction step for the position of the detected final pupil candidate. The method of claim 9, wherein the second correction step, Selecting an image having a predetermined size based on the detected final pupil candidate coordinates; Enlarging the selected image; A plurality of second templates consisting of a third U-shaped mask representing black and white of the pupil and having different sizes of the U-shaped radius, the coordinates of the final pupil candidate in the enlarged image are centered. Applying each to a pixel area of a predetermined size located; And Correcting the coordinates of the final pupil candidate based on the difference in brightness values between the black and white portions of the third mask, the image regions covered by the white and white portions of the third mask.

Images