JP4586963B2 - Automatic pupil correction method - Google Patents

Description

  The present invention relates to an automatic correction method for an image, particularly a pupil image.

  Conventionally, when a part of a digital image is corrected, image processing is performed on a specified area by manually specifying an area to be corrected and masking the outside of the specified area. In this type of technology, for example, when the subject is a person and the eyes, pupils, and the like are image-processed, a plurality of points are manually specified by the operator for the contours of the eyes and pupils. Some perform detection. In addition, there is a photo processing method that automatically detects a processing region only by specifying two points that sandwich the processing region (see, for example, Patent Document 1).

JP 2000-244727 A

  However, in the photographic processing method of Patent Document 1, the area can be easily specified, but the image processing is not performed. In a photographed image where the subject is a person, if the eyes of the person appear in the so-called three white eyes, the eyes of which the eyes are biased upward and the white eyes are on the left, right and bottom, the impression of the person is not good. In order to improve the image quality, it is required to perform image processing to correct the three white eyes. In the case of performing such correction, for example, the eye and pupil regions are designated using the photographic processing method of Patent Document 1, and image processing is manually performed on the designated regions. Manual image processing has a problem that it is not efficient and requires a lot of time for the work, and it requires skill, so that proper correction may not be performed.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide an automatic pupil correction method and apparatus that performs a correction of a pupil image automatically and satisfactorily without any sense of incongruity.

To achieve this object, the feature of the automatic pupil correction method according to the present invention includes a pupil detection step for detecting a pupil from captured image data, and white pixels distributed around the pupil detected in the pupil detection step. A white pixel recognition step for recognizing the image based on a predetermined white recognition condition, and a white region composed of the white pixels recognized in the white pixel recognition step is continuously distributed around the pupil by a predetermined angle or more. The determination step includes a determination step that is sometimes determined as a correction target, and a pupil correction step that expands the pupil relative to the white region when the correction step is determined as the correction target.
According to the above-described characteristic configuration, it is possible to automate image processing for an eye in which the extracted pupil is biased upward and a white region is present in the left, right, and lower sides, so-called three white eyes. Here, if it is a general Japanese, the black part in an eyeball will be called a pupil, and the part containing a pupil and the white area | region around it will be called an eye. In the determination step, when white pixels are continuously distributed over a predetermined angle, it is determined as a correction target. Therefore, even if the top / bottom / left / right of the captured image data does not match the top / bottom / left / right of the subject person, The correction target can be determined. Further, in the pupil correction step, correction is performed on the white area, so that the image of the eye composed of the corrected pupil and white area can be made satisfactory without a sense of incongruity. Therefore, by configuring as in this feature configuration, it is possible to reduce the burden of manual work for image correction, and it is possible to perform image correction satisfactorily even if the image correction is not skilled. .

Another characteristic configuration of the automatic pupil correction method according to the present invention is that the pupil correction step obtains the center of the pupil and enlarges the pupil in the radial direction from the center.
That is, according to this feature configuration, the center of the pupil is obtained and correction is performed, so that the corrected pupil image can be improved without feeling uncomfortable.

Still another characteristic configuration includes a skin color pixel recognition step for identifying a skin color pixel distributed around the pupil or the white pixel based on a predetermined skin color recognition condition, and the determination step includes the pupil and the skin color. This is in that it is determined that the white region distributed between the pixels is continuously distributed when the radial width of the pupil is wider than a predetermined range.
Eyes with a pupil biased upward and white areas on the left, right, and lower sides are defined as three white eyes. Especially when the resolution of the captured image data is very high, the distribution is continuously distributed below the pupil in the captured image data. Even if there is a white region to be displayed, when the width of the white region is very narrow with respect to the eyes and pupils, the white eye may not be visible to the eyes. In such a case, eye correction is unnecessary and image processing can be omitted. Therefore, as in this feature configuration, when the width of the white region is wider than the predetermined range, it is determined as the three white eyes, and unnecessary image processing is performed, for example, when it is not recognized as the three white eyes and there is no need for correction. Can be avoided, and the reliability of determination can be improved.

In order to achieve this object, the automatic pupil correction program according to the present invention includes a pupil detection function for detecting a pupil from captured image data, and white pixels distributed around the pupil detected by the pupil detection function. And a white region consisting of the white pixels certified by the white pixel certification function is continuously distributed around the pupil by a predetermined angle or more. In some cases, the computer executes a determination function for determining a correction target and a pupil correction function for enlarging the pupil with respect to the white region when the determination function determines a correction target.
The present invention also covers an automatic pupil correction program that causes a computer to execute the automatic pupil correction method described above. With such an automatic pupil correction program, it is possible to reduce the burden of manual work for image correction, as in the case of the automatic pupil correction method, and to perform image correction satisfactorily even if the image correction is not skilled. It can be done.

To achieve this object, the automatic pupil correction device according to the present invention is characterized by a pupil detection unit that detects a pupil from captured image data, and white pixels distributed around the pupil detected by the pupil detection unit. And a white area consisting of the white pixels certified in the white pixel certification unit is continuously distributed around the pupil by a predetermined angle or more. The determination unit is sometimes determined to be a correction target, and a pupil correction unit that expands the pupil with respect to the white region when the determination unit determines the correction target.
The present invention is also intended for the automatic pupil correction device that executes the above-described automatic pupil correction method, and all the effects of the automatic pupil correction method can be obtained. That is, according to the automatic pupil correction device of the present invention, it is possible to reduce the manual work load for image correction and to perform image correction satisfactorily.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a configuration diagram showing an automatic pupil correction device according to the automatic pupil correction technology of the present invention, and an image processing terminal 2 is connected to the automatic pupil correction device 1. The image processing terminal 2 includes a memory 21 that stores captured image data, a display unit 22 that displays captured images, and a graphic interface (hereinafter abbreviated as GUI) that generates control commands from the operation unit 24. A GUI unit 23 to be constructed is provided. The display unit 22 is configured to display an instruction relating to image processing of the photographed image together with the photographed image, and each instruction is executed by being selected by an operation of a keyboard or a mouse constituting the operation unit 24. . Here, FIG. 7 shows an example of photographed images before and after applying the pupil correction technique according to the present invention. In the present embodiment, the memory 21 has captured image data including the eyes shown in FIG. 7A, and the automatic pupil correction technique according to the present invention is applied to the captured image data.

  The automatic pupil correction device 1 includes a pupil detection unit 10 that detects a pupil from captured image data, a specific color pixel recognition unit 11 that identifies white pixels and skin color pixels, and whether or not the eyes of the captured image data are correction targets. A determination unit 12 that determines whether or not and a pupil correction unit 13 that corrects the pupil when the determination unit 12 determines that the correction is to be performed is provided, and includes a memory 14 that stores white recognition conditions 14a and the like.

  In this embodiment, the pupil detection unit 10 detects the pupil from the captured image data using the existing face / pupil detection means 3 installed on the network. The face / pupil detection means 3 has a function of extracting a face region from the captured image data, detecting the pupil, and outputting data such as the center position and contour of the pupil. Note that the pupil detection unit 10 causes the image processing terminal 2 to manually specify a certain region including the eyes of the captured image displayed on the display unit 22 by using a mouse, and the image data relating to this region is displayed on the face. -If it is set as the structure output to the pupil detection means 3, the load concerning a network and the time concerning a pupil detection can be reduced. Further, it is also desirable to adopt a configuration in which the entire face and the outline and center of the pupil are automatically extracted from the captured image data in the pupil detection unit 10 without using the existing face / pupil detection means 3. .

  The specific color pixel certification unit 11 includes a white pixel certification unit 11a and a skin color pixel certification unit 11b. The white pixel recognition unit 11a recognizes white pixels distributed around the pupil detected by the pupil detection unit 10 based on a predetermined white recognition condition 14a. Here, a luminance range is set as the white recognition condition 14a. In the present embodiment, the saturation of the skin color pixels around the white pixels is further used as a parameter. The skin color pixel recognition unit 11b recognizes the skin color pixels distributed around the pupil or the white pixel based on a predetermined skin color recognition condition 14b. As the skin color certification condition 14b, a luminance range is set.

  The determination unit 12 determines that a white region composed of white pixels certified by the white pixel certification unit 11a is a correction target when the white region is continuously distributed around the pupil by a predetermined angle or more. In the present embodiment, for example, the predetermined angle is set to 180 degrees in consideration of being recognized as a three white eye when there are white regions on the left and right and lower sides of the pupil. Further, when the width in the radial direction of the pupil of the white region distributed between the pupil and the skin color pixel is wider than a predetermined range, it is determined that the distribution is continuous. The range of the width of the white area is set in consideration of the size of the captured image data and the size of the eyes, and a range that is not recognized as three white eyes when an image is displayed.

The pupil correction unit 13 enlarges the pupil with respect to the white region when the determination unit 12 determines that the correction is to be performed. Further, in the present embodiment, the center of the pupil is obtained based on the data from the face / pupil detection means 3, and the pupil is enlarged outward from the center.
The memory 14 includes a white pixel map 14c that stores a determination condition including a white color recognition condition 14a and a skin color recognition condition 14b, a position of a white pixel recognized by the white pixel authentication unit 11a, and a skin color pixel authentication unit 11b. A skin color pixel map 14d or the like for storing the position of the skin color pixel thus set is stored.

Next, a processing procedure of photographed image data by the automatic pupil correction device 1 configured as described above will be described with reference to FIG.
First, the automatic pupil correction device 1 captures captured image data as shown in FIG. 7A from the image processing terminal 2 and detects the pupil from the captured image data using the face / pupil detection means 3 (# 100). Subsequently, skin color pixel authorization (# 200), white pixel authorization (# 300), and white area authorization (# 400) are performed, and it is determined whether or not the correction target is based on the white area. If so, the pupil is enlarged with respect to the white region (# 500). Then, the corrected image data is output to the image processing terminal 2 and the image is displayed on the display unit 22.

Details of the processing described above will be described with reference to FIGS.
The skin color pixel recognition (# 200) is executed by the skin color pixel authentication unit 11b based on the skin color pixel authentication subroutine shown in FIG. The skin color pixel recognition unit 11b first takes in the skin color recognition condition 14b from the memory 14 (# 201). The skin color recognition condition 14b is that when the R, G, and B luminance values of the target pixel are R, G, and B and (R + G + B) / 3 = I, [fs_gb_lo <GB <fs_gb_hi] and [fs_rg_lo <RG <fs_rg_hi ] And [fs_br_lo <BR <fs_br_hi] and [I> fs_I]. A pixel of interest that satisfies the skin color recognition condition 14b consisting of these four conditions is regarded as a skin color. In this embodiment, the R, G, and B luminance values are 8-bit color data (0 to 255), and the above constants are fs_gb_lo = -24, fs_gb_hi = 16, fs_rg_lo = -16, fs_rg_hi = 64. , Fs_br_lo = −88, fs_br_hi = 8, and fs_I = 104 (hereinafter, R, G, and B luminance values are assumed to be 8-bit color data).

  Subsequently, the target pixel to be certified is identified from the captured image data, the R, G and B luminance values of the target pixel are taken in (# 202), and whether or not the skin color certification condition 14b is satisfied is checked (# 203). When the skin color determination condition is satisfied, the pixel position of the target pixel is stored as the pixel position of the skin color pixel (# 204). The skin color certification in steps # 202 to # 204 is performed for all the pixels of the photographed image data, and when the skin color check for all the pixels is completed (# 205), the skin color pixel map 14d is based on the stored pixel positions of the skin color pixels. Is created (# 206). The skin color pixel map 14d may be created in step # 204 every time the skin color pixel is recognized.

  The white pixel recognition (# 300) is executed by the white pixel recognition unit 11a based on the white pixel recognition subroutine shown in FIG. In the present embodiment, white color recognition is performed using the skin color pixel map 14d created by the skin color recognition unit. First, the white pixel recognition unit 11a captures the skin color pixel map 14d created by the skin color pixel authentication subroutine (# 301), and SF = SF_rg × (RG) + SF_br × ( The skin color saturation coefficient SF defined in (BR) is calculated to create a skin color saturation map (# 302). Here, by setting values such as SF_rg = 0.7 and SF_br = −1.2, the higher the saturation of the skin color pixel, the larger the value of the skin color saturation coefficient SF. The value of the coefficient SF becomes small.

  Subsequently, the white recognition condition 14a is fetched from the memory 14 (# 303). Further, the R, G, B luminance values of the target pixel are sequentially taken from the image data (# 304). The white color recognition condition 14a is divided into two stages. First, it is checked whether the detection condition of the first stage is satisfied (# 35). The detection conditions of the first stage are represented by [abs (G-B) <ws_gb], [abs (R-G) <ws_rg], [abs (B-R) <ws_br], and [I <ws_I]. abs () is a function expression for obtaining the absolute value of the value in (). A pixel of interest that satisfies the first-stage determination condition composed of these four conditions is regarded as a white pixel candidate. Here, the constants are set to ws_gb = ws_rg = ws_br = 45 and ws_I = 112.

  When the target element is a candidate for a white pixel (Yes branch at # 305), the second detection condition is applied. This detection condition includes the average color of the skin color pixels in the vicinity of the target pixel for white pixel detection. Since the degree coefficient and the saturation coefficient of the white pixel candidate are included as one of the condition elements, first, with reference to the skin color saturation map created in step # 302, 40 × 40 centering on the target pixel. The average value Ave_SF of the saturation coefficient of the skin color pixels located in the vicinity region of about the pixel is calculated (# 306). The size of this neighborhood region depends on the output gradation. Further, the saturation coefficient SFw of the target pixel (white pixel candidate) is also calculated using the same expression as the expression for obtaining the skin color saturation coefficient SF (# 307). Through such pre-processing, it is checked whether the pixel of interest (white pixel candidate) satisfies the detection condition of the second stage (# 308). This second stage detection condition is represented by [SFw-Ave_SF <sh_SF], and the value of the constant sh_SF is set to -12. The target pixel that satisfies the second-stage detection condition (Yes in # 38), that is, a white pixel candidate, is regarded as a white pixel, and the pixel position is stored as the white pixel position (# 309). This white recognition is performed for all pixels of the photographed image data in two stages, and when the white check for all the pixels is completed (# 310), a white pixel map 14c is created based on the stored pixel positions of the white pixels. (# 311). The white pixel map 14c may be created in step # 309 every time a white pixel is detected.

  In the present embodiment, the average value Ave_SF of the saturation coefficient of the flesh color pixel is calculated from the flesh color pixel map 14d and used as a parameter for the white image qualification. -It is good also as a structure which recognizes a white pixel only using B luminance value.

  The white area recognition (# 400) is executed by the white pixel recognition unit 11a based on the white area recognition subroutine shown in FIG. Here, a white region consisting of continuously distributed white pixels is specified using the white pixel map 14c.

  First, the white pixel recognition unit 11a obtains a white pixel for specifying a region from the white pixel map 14c created by the white pixel recognition subroutine, and sets this as a target pixel (# 401). Here, since the photographed image data may include a white region other than the white region around the pupil, that is, the so-called white-eye portion, the white color around the pupil is excluded in order to exclude white pixels other than the white-eye portion. Let the pixel be the pixel of interest.

  Subsequently, white pixels that are recognized as white pixels and do not belong to any region are extracted from the pixels in the vicinity of 8 of the target pixel (# 402). The extracted white pixel is stored as a pixel belonging to the same region as the target pixel (# 403). Then, Steps # 402 and # 403 are executed by using each of the white pixels stored as pixels belonging to the same region in Step # 403 as the target pixel. Steps # 402 and # 403 are repeated until there are no white pixels that do not belong to any region in the surrounding area, and a continuously distributed white region is specified. The range of pixels determined to be in the same region need not be limited to pixels in the vicinity of 8, but is set in accordance with the size and resolution of the captured image data, such as 4 neighborhoods.

  Further, white pixels that are around the pupil and that do not belong to any region are obtained from the white pixel map 14c (# 404). If there is a white pixel that does not belong to any region around the pupil, the process proceeds to step # 401, and the white pixel is set as the target pixel. Then, as described above, steps # 402 and # 403 are executed to determine the white area. Steps # 402 to # 404 are repeatedly executed until there are no white pixels that do not belong to any region around the pupil, and to which white region all white pixels around the pupil of the white pixel map 14c belong. Confirm.

  The pupil correction (# 500) is executed by the determination unit 12 and the pupil correction unit 13 based on the pupil correction subroutine shown in FIG. The determination of the correction target here is performed based on whether the white area is continuously distributed around the pupil by a predetermined angle or more, or whether the width of the lower part of the pupil of the white area is a predetermined range or more.

  First, the determination unit 12 identifies a white region around the pupil, and determines whether the white region is continuously distributed over a predetermined angle (# 501). If the white region is not continuously distributed around the pupil by 180 degrees or more, it is determined that the white area is not a three-white eye and is not a correction target, and the process is terminated. Also, when the white area is divided around the pupil, that is, when there are a plurality of white areas around the pupil, it is determined that the white area is not a target for correction and the process is terminated. In other cases, the center of the pupil is obtained based on the data from the face / pupil detection means 3, and the width of the white region in the radially outward direction of the pupil is calculated (# 502). Here, the width of the narrowest portion is calculated near the approximate center of the continuous distribution of the white region. Subsequently, it is determined whether or not the calculated width is wider than a predetermined range (# 503). The width range is set by, for example, a ratio or a ratio with respect to these size values in consideration of the size of the eyes and pupils of the captured image data. When the width is wider than a predetermined range, it is determined that the eye has three white eyes. If it is not determined that the width is wider than the predetermined range, it is determined that the object is not a three-white eye and is not a correction target, and the process ends.

  If it is determined in step # 503 that the eyes are three white eyes, the pupil correction unit 13 enlarges the pupil by a predetermined amount in the radial direction from the center of the pupil, and corrects only the white region (#). 504). Thereafter, the process proceeds to step # 502. Then, Steps # 502 to # 504 are repeatedly executed, and when it is determined in Step # 503 that the width of the white area is within the predetermined range and it is determined that it is not the three white eyes, the process is terminated. In addition, it is also desirable that the pupil enlargement in step # 504 is configured to enlarge to a size that is not recognized as the three white eyes using the range used for the determination of the three white eyes used in step # 503.

  Here, FIG.7 (b) has shown the picked-up image after applying the pupil correction technique which concerns on this invention. By applying the pupil correction technique according to the present invention, the photographed image of the eye shown in FIG. 7A is not recognized as the white area below the pupil as shown in FIG. Not so, the eye impression is good.

  In the above-described embodiment, the automatic pupil correction device 1 is constructed. However, an automatic pupil correction program that executes each function of the automatic pupil correction device 1 is constructed by being executed on a computer, and is incorporated into the image processing terminal 2. It is good also as a structure.

Configuration diagram of automatic pupil correction device according to the present invention The flowchart which shows the processing routine of the automatic pupil correction which concerns on this invention Flowchart showing a skin color pixel recognition subroutine for automatic pupil correction according to the present invention A flowchart showing a white pixel recognition subroutine for automatic pupil correction according to the present invention. Flowchart showing a white area recognition subroutine for automatic pupil correction according to the present invention A flowchart showing a pupil correction subroutine for automatic pupil correction according to the present invention. Captured images before and after performing automatic pupil correction according to the present invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Automatic pupil correction apparatus 2 Image processing terminal 3 Face / pupil detection means 10 Pupil detection part 11 Specific color pixel recognition part 11a White pixel recognition part 11b Skin color pixel recognition part 12 Determination part 13 Pupil correction part 14, 21 Memory 22 Display part 23 GUI unit 24 Operation unit

Claims (5)

A pupil detection step for detecting a pupil from the captured image data;
A white pixel recognition step for recognizing white pixels distributed around the pupil detected in the pupil detection step based on a predetermined white recognition condition;
A determination step of determining a correction target when a white region composed of the white pixels certified in the white pixel certification step is continuously distributed over a predetermined angle around the pupil; and
An automatic pupil correction method comprising a pupil correction step of enlarging the pupil with respect to the white area when it is determined as a correction target in the determination step.   The automatic pupil correction method according to claim 1, wherein the pupil correction step obtains a center of the pupil and enlarges the pupil in a radially outward direction from the center. A skin color pixel certification step for authorizing skin color pixels distributed around the pupil or the white pixel based on predetermined skin color certification conditions;
The determination step determines that the white area distributed between the pupil and the skin color pixel is continuously distributed when the width in the radial direction of the pupil is wider than a predetermined range. Alternatively, the automatic pupil correction method described in 2. A pupil detection function for detecting pupils from captured image data;
A white pixel recognition function that recognizes white pixels distributed around the pupil detected by the pupil detection function based on a predetermined white recognition condition;
A determination function for determining a correction target when a white region composed of the white pixels certified by the white pixel certification function is continuously distributed over a predetermined angle around the pupil; and
An automatic pupil correction program for causing a computer to execute a pupil correction function for enlarging the pupil with respect to the white area when the determination function is determined as a correction target. A pupil detector that detects the pupil from the captured image data;
A white pixel certifying unit that certifies white pixels distributed around the pupil detected by the pupil detection unit based on a predetermined white qualification condition;
A determination unit that determines a correction target when a white region composed of the white pixels certified in the white pixel certification unit is continuously distributed over a predetermined angle around the pupil;
An automatic pupil correction device including a pupil correction unit that expands the pupil with respect to the white region when the determination unit determines that the correction target.