EP2143069A1 - Procédé pour une détection d'yeux rouges dans une image numérique - Google Patents

Procédé pour une détection d'yeux rouges dans une image numérique

Info

Publication number
EP2143069A1
EP2143069A1 EP07723937A EP07723937A EP2143069A1 EP 2143069 A1 EP2143069 A1 EP 2143069A1 EP 07723937 A EP07723937 A EP 07723937A EP 07723937 A EP07723937 A EP 07723937A EP 2143069 A1 EP2143069 A1 EP 2143069A1
Authority
EP
European Patent Office
Prior art keywords
regions
redness
face
skin
red
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP07723937A
Other languages
German (de)
English (en)
Inventor
Claudio Cusano
Francesca Gasparini
Raimondo Schettini
Paolo Gallina
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Universita degli Studi di Milano Bicocca
Original Assignee
Universita degli Studi di Milano
Telecom Italia SpA
Universita degli Studi di Milano Bicocca
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Universita degli Studi di Milano, Telecom Italia SpA, Universita degli Studi di Milano Bicocca filed Critical Universita degli Studi di Milano
Publication of EP2143069A1 publication Critical patent/EP2143069A1/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing
    • G06T2207/20012—Locally adaptive
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30216—Redeye defect

Definitions

  • the present invention relates to a method for red eye detection in a digital image, in particular a method in which the red eye detection is fully automatic and does not need user intervention.
  • the method of the invention maximises the detection rate, minimizing at the same time the "false positive" rate.
  • the red eye effect is a well known problem in photography. It is often seen in amateur shots taken with a built-in flash, but the problem is also well known to professional photographers.
  • Red eye is the red reflection of the blood vessels in the retina caused when a strong and sudden light strikes the eye, in particular when the pupil is expanded in dim light to allow more light to enter the eye.
  • red-eye removal solutions Most of them are semi-automatic or manual solutions. The user has to either click on the red-eye or draw a box containing it before the removal algorithm can find the red-eye pixels and correct them.
  • a method for removing a red eye from an image which includes 1) calculating a weighted red value for each pixel in the image from green, red and blue colour values and a luminance value of each pixel in the image, 2) selecting a plurality of pixels in the image having weighted red values greater than a threshold as red eye pixels, and 3) correcting some of the red eye pixels to remove the red eye from the image.
  • a definition of weighted red value is given and it represents the likelihood that a pixel is a purple-red pixel forming part of a red eye, is disclosed.
  • the software determines if each red eye region so identified is proximate to a facial region. If the facial region has a colour that is within the range of threshold skin colour values, then it is probably a pupil with red eye that is proximate to a facial region.
  • US patent application n. 2004/0213476 is relative to a system and method of detecting and correcting red-eye in a digital image.
  • measures of pixel redness in the digital image are computed.
  • a preliminary set of candidate red eye pixel areas is identified based on the computer pixel redness measures.
  • Each candidate red eye pixel area having a computed redness contrast relative to at least one respective neighbouring pixel area less then a prescribed redness contrast threshold is filtered from the preliminary set.
  • each candidate red eye pixel area located in an area of the digital image having a computed greyscale contrast relative to at least one respective neighbouring pixel area less than a prescribed greyscale contrast threshold is filtered from the preliminary set.
  • at least one candidate red eye pixel area in the image is detected.
  • Pixels in each candidate red eye pixel area are classified based on the redness and skin tone coloration. For each candidate red eye pixel area, an initial pixel mask identifying all pixels in the candidate red eye pixel area classified as red and non-skin tone is generated. A final pixel mask is generated based on each initial pixel mask. Red eye in the digital image is corrected by darkening and desaturating original colour values identified as red in the final pixel mask, when original colour values of pixels identified as red in the final pixel mask are desaturated by respective amounts that vary with pixel location in the final pixel mask.
  • the defective colour areas are recorded with a colour spectrum deviating form the actual colour spectrum of said area without colour defects, wherein basic areas in the image are identified on the basis of features which are common for these recorded defective areas, said basic areas supporting an increased likelihood to include defective areas, and the processing is then reduced to the basic areas to identifies borderlines and/or centres of the defective areas, and afterwards, it is identified whether the localized basic area or areas deemed to be defective are defective or not, and finally, if a location basic area has been identified to be defective, a correction mask is created to correct the visual appearance of the defective area.
  • Redeye is searched for only within these regions, seeking areas with high redness satisfying some geometric constraints.
  • a novel redeye removal algorithm is then applied automatically to the red eyes identified, and opportunely smoothed to avoid unnatural transitions between the corrected and original parts.
  • Experimental results on a set of over 450 images are reported.
  • a multi-resolution neural network approach is then exploited to create an analogous probability map for candidate faces. These two distributions are then combined to identify the most probable facial regions in the image. Redeye is then searched for within these regions, seeking areas with high redness and applying geometric constraints to limit the number of false hits. The redeye removal algorithm is then applied automatically to the red eyes identified. Candidate areas are opportunely smoothed to avoid unnatural transitions between the corrected and the original parts of the eyes. Experimental results of application of this procedure on a set of over 300 images are presented. Summary of the invention
  • the present invention is relative to a method to detect red eye(s) in a digital image, in particular a fully automatic detection method which involves no user's activity or interaction.
  • One of the main goals of the method of the invention is to obtain a high red-eye detection rate, so that most of the red eyes in the image are detected, and a very low "false positive" rate, i.e. only "real" red eyes are recognized and not other image details.
  • the method of the invention includes a face detection step. It is indeed desirable to determine whether or not there is a face in the image: once a face region is detected, the task of detecting red-eyes within the face region becomes simpler than processing the whole image.
  • an automatic face detector e.g., a computer-implemented algorithm, is preferably used.
  • any face detector that fulfills the requirement can be used in the invention.
  • the face detector operates in accordance to the method described in P. Viola and M.J.
  • the output of the face detection step comprises a first set of face regions, which are substantially delimited portions of the original input image, and for each identified face region in the input image the face detection module generates a bounding box enclosing the respective face.
  • the method of the invention comprises an additional step of skin detection in which the whole input image is processed. In this step, potential skin regions present in the image are analyzed in search for other potential face candidates that have not been detected by the face detector.
  • a skin detection algorithm based on the colour information of the pixels is applied to the input image.
  • the preferred colour space used to select pixel regions as skin regions is the luminance-chrominance space.
  • skin pixels are defined as those pixels having a chrominance within a given interval. More preferably a suitable algorithm used in this step of the invention can be found in F. Gasparini, R. Schettini "Skin segmentation using multiple thresholding" Proc. Internet imaging VII, Vol. SPIE 6061 (S. Santini, R. Schettini, T. Gevers eds.), pp. 60610F-1 , 60610F-8, 2006.
  • the output of this skin detection algorithm is a set of candidate skin regions.
  • a masking step is performed on the input image. Large “red” and “dark” regions (the meaning of “dark” and “red” will become clearer in the following detailed description) are removed form the input image to easier the task of skin detection (step detailed below).
  • the output of the skin detector algorithm comprises a relatively large set of candidate skin regions. Indeed, among the detected skin regions not only faces are included (which are the target regions because they contain eyes), but any other body part having a skin colour (i.e. arms, legs, necks, etc), background elements, clothes, etc. It is therefore preferred, that the skin detection process comprises a further step having the aim of limiting the number of skin regions to be further processed.
  • the skin regions number reduction is preferably performed using a first set of geometrical constraints.
  • a face has a typical geometrical shape: it is substantially elliptic and therefore its
  • roundness value can be measured, it has a possible range of ratios between its width and length, and preferably includes several other purely geometric characteristics (i.e., the presence of "holes” such eyes and mouth, etc) that can be used for identifying the skin regions which are probably also faces.
  • the face regions detected by the face detector are subtracted from the skin regions set. Those regions already identified by the face detector are considered good face candidates and therefore there is no need of processing them further.
  • a second aspect of the invention relates to a method of red eye detection.
  • the face regions obtained by applying a face detector and preferably a skin detector being limited using geometrical constraints are analyzed to identify red eye artifacts.
  • red eye artifacts have a dominant purple-brown component.
  • red eyes appear as bright spots both in a redness and in a purple-brown map.
  • Applicants have noticed that an evaluation of the magnitude of the purple-brown component within the image increases the probability of detecting red eye artifacts while decreasing the occurrence of false positive.
  • the analysis of the purple-brown component allows a good discrimination between hair and red eye artifacts within a face region.
  • R edness(x,y) LJ 2 x * » 0r ) - fo x,y) + B fr y j X .
  • PurpleBrown(;c, y) Max[ ⁇ , B(x, y) - G(x, y)] .
  • Two monochromatic maps are generated by this step of the method of the invention: a redness map and a purple-brown map, in which the original pixel colour data of the pixels of the face and skin regions under consideration is replaced by their redness value or the purple-brown value, respectively.
  • Many other red regions i.e. regions that appear "bright” in the redness map
  • the local maxima of the redness are first detected, in order to calculate the following quantities only on those regions.
  • a second set of geometric constraints is preferably created. Based on geometric considerations on the typical geometric shape of a red-eye, on the fact that eyes are located substantially at the centre of the face and not at its boundary, etc, the number of candidate red-eye regions is reduced.
  • a colour constraint is imposed on the candidate red-eye regions, preferably only on those regions satisfying the geometric constraints' second set.
  • the colour constraint requires that only those candidate red-eye regions whose pixels satisfy the following equations:
  • an additional step of the invention is performed, to detect the second eye.
  • the candidate red-eye regions already processed in the previous step to detect the first red eye are thus re-processed, loosening the geometric and colour conditions.
  • the red eye region selected as second red-eye is the one having the highest R ⁇ PB values.
  • the red eyes so detected are also corrected using a suitable algorithm. Any known algorithm can be used in this step of the method of the invention.
  • the algorithm to correct the red-eye region comprises a smoothing masking step (the resulting smoothed mask is referred in the following to as MASKsmooth) and a pixel color changing step of the red-eye region pixels.
  • the smoothing step is for example described in F. Gasparini, R. Schettini, "Automatic redeye removal for smart enhancement of photos of unknown origin", Proc. of the 8th International Conference on Visual Information Systems, Amsterdam, July 2005, Springer, Lecture Notes in Computer Sciences, Vol. 3736, 2005, pp. 226-233 and in F. Chazli, F. Gasparini, R. Schettini, "A modular procedure for automatic redeye correction in digital photos", Proc. Color Imaging IX: Processing, Hardcopy, and Applications IX, Proceedings of SPIE Vol. 5293 (R. Eschbach, G.G. Marcu eds.), pp.139- 147, 2004.
  • each pixel of coordinates (x,y) belonging to the red eye region(s) is then corrected.
  • a weighted average, based on a weighting factor w, of the B and G components is calculated.
  • the weighting factor w depends on the redness intensity of the pixel under consideration.
  • the original RGB colors of the pixel are then replaced with
  • R n ⁇ v (x,y) R(x, y) - (l - MASKsmooth) + MASKsmooth ⁇ (B(x, y) + G(x, y)) - — , W
  • B new (x, y) B(x, y) - (l - MASKsmooth) + MASKsmooth ⁇ (B(x, y) + G(x, y)) ⁇ — and
  • G new (x, y) G(x, y) (l - MASKsmooth) + MASKsmooth ⁇ (B(x, y) + G(x, y)) - — .
  • - fig. 1 is a schematic flow diagram of the method of the invention
  • - figs. 2 and 3 are exemplary input images and their corresponding processed results according to a step of face detection of the method of the invention
  • - figs. 4 and 5 are processed images according to an additional step of the invention of the input images of figs. 2 and 3, respectively;
  • - figs. 6 and 7 are processed images according to an additional step of the invention of the input images of figs. 4 and 5, and of the processed images of figs. 2 and 3, respectively;
  • - figs. 8 and 9 are processed images according to an additional step of the invention of the input images of figs. 6 and 7, respectively;
  • - fig. 10 represents the processed image of fig. 3 (top image), the same image filtered using a filter of the method of the invention (middle image), and a corresponding redness map (bottom image) according to an additional step of the method of the invention;
  • fig. 10a represents the redness mask of the region selected in fig. 9;
  • - fig. 11 represents the processed image of fig. 2 (top image), a redness map of the same image (middle image) and a purple-brown map (bottom image) of the same;
  • - figs. 12-13-14 represent the local redness maxima of the redness maps of figs. 10 -11 -
  • - figs. 15 and 16 represent the red eyes detected in the bounding box obtained in the image of fig. 3 and in the geometric skin-face region obtained for the same image, respectively;
  • - fig. 16a shows the additional red-eye detection in the geometric skin-face region of fig. 3; - figs. 17 and 18 shows the original and corrected images of fig. 2 and 3.
  • a flow chart representing the steps of the method of the invention according to a preferred embodiment of the same is depicted.
  • the method is applicable to any digital input image 100.
  • the digital colour input image 100 is processed according to the method of the invention which generates a digital colour output image 200 wherein the occurrences of red eyes present in the input image 100 have been detected and optionally corrected.
  • the input image 100 can be obtained for example using a digital camera, a mobile phone or the like (not shown). It can also be downloaded from the Internet, received by e-mail or stored in a removable memory storage device, such as a CD-Rom or diskette (also not shown).
  • the steps of the method of the invention of fig. 1 are implemented as one or more respective software modules that are for example executable on a computer (workstation).
  • a computer on which modules of the method may execute includes a processing unit, a system memory and a system bus that couples the processing unit to the various components of the computer.
  • the processing unit may have one or more processor.
  • the software modules of the method are incorporated in a printer and, once initiated, operate automatically.
  • the printer includes sufficient memory and processing resources to execute the programming instruction corresponding to the method of the invention.
  • the printer may for example include a "red eye correction button" that can be pressed in order to activate the method of the invention.
  • the modules may be incorporated in a photographic digital camera that takes and stores the image 100.
  • the input image 100 comprises a plurality of pixels that can be considered, at least logically, to be organized in an array or matrix having a plurality or rows and columns. Each pixel has an X- coordinate, which identifies the particular column within which the pixel is located, and a Y- coordinate, which identifies the particular row within which the pixel is located.
  • each pixel contains digital data which may be in the well known Red, Greed; Blue (RGB) colour space.
  • RGB Red, Greed
  • RGB Blue
  • the input image 100 can be in different formats other that RGB, such as, for example, Cyan, Magenta, Yellow, Black (CMYK); Hue, Saturation, Value (HSV), among others.
  • a length and a height of the image 100 are also defined, being the number of pixels along the X and Y direction, respectively. The longer between the length and the height is the maximum dimension (Max Dim) of the image 100.
  • the output image 200 has preferably the same number of pixels as the input image 100 organized in the same array. Those pixels identified, as detailed in the following, as corresponding to occurrences of red eyes are detected and preferably corrected, while all the other pixels remain unchanged.
  • the image size is re-scaled to a "standard size", so that the maximum dimension (Max Dim) of the rescaled image has always the same value (i.e., it is equal for example to 240 pixels, however other pixel numbers can be used).
  • a minimum and maximum face area is also preferably defined. As an example, it is supposed, from geometric consideration, that the maximum (minimum) possible area occupied by a face within the rescaled input image 100 is equal to:
  • the minimum and maximum area of a face are preferably to be modified, i.e. the two above written values are selected for faces having a height of at least 1/10 of the total height of the input image, which are the "most probable" face dimensions in pictures.
  • the subsequent step of the method of the invention is a face detection step.
  • the whole input image 100 (possibly rescaled in step 1) is processed by a suitable algorithm so that human faces are searched and located through the image 100.
  • a bounding box BB is generated in step 2a surrounding or encompassing each identified face.
  • the bounding box BB is preferably rectangular and each pixel within the bounding box is identified with its x and y coordinates. According to an embodiment of the invention, this step 2 of the method operates in accordance with the algorithm described in the Viola - Jones paper already identified above.
  • the algorithm used in the preferred embodiment of the invention is a face detector based on the combination of multiple "rectangle features", computed as the difference of the sum of pixel values in different rectangular areas of the input image.
  • the rectangle features are computed very quickly using an alternative image representation called the "Integral Image”.
  • AdaBoost® learning algorithm Using the AdaBoost® learning algorithm, a small number of critical visual features is selected from a very large set of potential features. The learning algorithm is executed several times to produce multiple classifiers which are combined in a "cascade". The cascade structure allows background regions of the image to be quickly discarded while spending more computation on promising face-like regions. Every square sub-window of the input image is processed by the trained cascade, and those sub-windows which are accepted by all the classifiers are classified as faces. The probability of a region to be a face depends on how many times the region has been considered a face in the sub-sampled images.
  • a binary mask 102 is thus created in step 2a, the dimensions of which are preferably equal to those of the rescaled input image 100. In mask 102, all pixels belonging to a region classified as a face region (the bounding box BB) have the value of "1" (which corresponds to white pixels), while all pixels outside the face regions have the value of "0" (which corresponds to black pixels).
  • n face bounding boxes BB are thus located included in mask 102.
  • fig. 2 and 3 The results of this method step 2 are illustrated in fig. 2 and 3.
  • the rescaled input image 100 at the far left is processed by the face detection algorithm and different face regions are located, which are represented as rectangles in the middle image 101.
  • the final binary mask 102 is represented at the far right: some of the rectangles previously selected as probable face regions have been disregarded as being too small compared to the average size face region.
  • Each rectangle represents the output bounding box BB containing a face. Both faces in the input image 100 are detected by the face detection algorithm and an additional bounding box is also represented which does not correspond to any "real" face in the input image 100.
  • the step 2 of face detection is applied to a different input image 100'.
  • the partially rotated man face has not been found by the algorithm, only the frontal child face is recognized and a single bounding box BB is created.
  • the resulting masked image 102' represents the original image 100 masked with the binary mask obtained using the face detection method: the regions corresponding to the "0" of the mask are deleted from the original image and only the region corresponding to the "1" of the mask are kept and shown.
  • the method of the invention preferably further comprises a skin detection step 3.
  • the whole input image 100, or preferably the rescaled image as detailed above is processed to detect regions that may contain "skin": this step in particular discriminates between skin pixels and non-skin pixels. Skin is preferably detected using a colour analysis of the image pixels, as better outlined below.
  • the skin detection step 3 is substantially an additional face detection step with the scope of detecting possible faces that have been "missed” by the face detector. According to the skin detection step, the detection of faces is based on the fact that a face contains "skin" and thus skin clusters are searched.
  • a first red and dark regions removal step 3a is performed on the input image 100.
  • a redness value for each pixel of the input image 100 (or the rescaled image) having coordinates (x,y) is calculated using the following equation:
  • RednessQc RednessQc,, (1) where R, G , B are red, green and blue component pixel values in the RGB colour space of the input image 100. Clearly, a similar redness calculation can be performed in different colour spaces depending on the input image colour format.
  • a redness mask is created in which all pixels of the original image whose computed redness is greater than a first redness threshold are set equal to "1", whereas the remaining pixels are set equal to "0".
  • the redness threshold is equal to 1.2.
  • the redness and darkness masks are combined in an AND operation so that the resulting mask is a binary matrix having values "1" corresponding to the pixels that have value "1" both in the redness and in darkness mask. All the other pixels (i.e.
  • pixel being "0" in both the redness and darkness masks or pixel being “0” either in the redness or in the darkness mask) are set equal to "0".
  • the resulting mask is then applied to the rescaled original input image 100 and the pixels corresponding to the "1" values are removed from the image 100.
  • a new masked image is therefore created, which is formed by applying the complementary mask of the AND combination of the redness and darkness masks. These remaining regions of the original rescaled input image 100 are called in the following darkness and redness masked image.
  • redness mask "small" red areas, i.e. smaller than 5/1000 of the rescaled input image 100 area, are removed because they can contain potential red eyes artifacts and thus need to be considered in the red eye detection phase. Therefore, in the masked image obtained after the application of the redness and darkness mask on the rescaled input image 100, these "small” red areas, i.e. areas the redness of which is above the redness threshold, are still present.
  • the skin detection step 3 takes place.
  • the skin detection algorithm which is preferably a recall oriented algorithm, based on the hypothesis that skin pixels exhibit similar colour coordinates in a properly chosen colour space, processes the rescaled input image 100, preferably masked using the redness and darkness masks.
  • a "skin cluster boundary" is defined, expressly defining the boundaries of the skin cluster in a certain colour space.
  • the rescaled redness and darkness masked input image 100 is transformed by the skin detection algorithm from the RGB colour space (or any other colour space in which the input image 100 is stored) to the YCbCr space, where Y is the luma component and Cb and Cr are the blue and red chroma components for each pixel.
  • the conversion matrix from one colour space to another is known in the art.
  • the rescaled and masked input image 100 is thus segmented in a plurality of m candidate skin regions SR, each region being a cluster of pixels satisfying eq. (3).
  • the areas of the candidate skin regions SR are evaluated: in case of too small overall skin regions, i.e., if the sum of the areas of all skin regions SR found by the skin detection algorithm is smaller than two times the minimum face area (see step 1 of the method of the invention where the minimum face area is calculated), a colour balancing method is applied to the input image 100, in a colour balancing step 5, such as Gray World.
  • This algorithm is based on the fact that the colors in a sufficiently complex scene average to gray.
  • the algorithm estimates the average colors of the image and shifts the average color so it becomes gray.
  • a different average colour may depend on the type of light used to illuminate the scenes or on defects of the camera used to take the input image 100.
  • a different colours balancing algorithm can be used, such as a white patch algorithm, as the one described in K. Barnard, V. Cardei, B. Funt, "A Comparison of Computational Color Constancy Algorithms-Part I: Methodology and Experiments with Synthesized Data” , IEEE Transactions on Image Processing 11 (9) (2002) 972-983.
  • For each pixel of the input image 100 preferably rescaled and masked using the redness and darkness masks, whose computed Cb and Cr values satisfy eq.
  • a binary skin mask 104 is then the output of the skin detection module 3.
  • the result of the skin detection step applied to the input image 100 of fig. 2 is shown.
  • the white regions represent the candidate skin regions SR.
  • the skin detection step 3 (and the redness and darkness masking 3a) is performed on input image 100' of fig. 3, as shown in fig. 5.
  • the result is displayed as masked image 104', which is obtained applying the skin binary mask to the original image 100'.
  • First the skin binary mask for input image 100' is calculated and then the regions of the binary mask set equal to 0 are removed from image 100', while the regions in which the corresponding mask is set equal to 1 are maintained. As visible, a single skin region SR is detected.
  • two binary masks are available: a mask 102 from the face detector step 2 including n bounding boxes BB, one for each candidate face, and a mask 104 from the skin detection step 3 containing m candidate skin regions SR.
  • the two masks are then preferably "subtracted" in step 6 of the method of the invention, i.e. regions identified as faces by the face detection algorithm (i.e. the bounding boxes BB) are set equal to 0 in the binary mask 104 obtained by the skin detection algorithm.
  • a new skin-face binary mask 105 is thus obtained, in which pixels are set equal to 1 if they belong to the regions of skin NOT identified as face regions by the face detector algorithm. In the skin-face mask 105 therefore all regions that are skin but not faces (in the sense of candidate face regions detected by the face detector) are included.
  • the corresponding masked image i.e.
  • the rescaled input image 100 masked with the skin-face mask 105 is thus supposed to include background elements having "skin colour", faces missed by the face detector algorithm and possible other body parts (such as arms, legs, etc.). These "skin-face” regions are referred to as SFR.
  • figs. 6 and 7 The result of the application of the subtraction step 6 is visualized in figs. 6 and 7.
  • fig. 6 the masks 102 and 104 of figs. 2 and 4, respectively, and the resulting skin-face mask 105 are shown
  • fig. 7 the masked image 105', which is the original (rescaled and masked) image 100 masked using the skin-face mask, is depicted showing the remaining skin regions SFR.
  • a geometric analysis step 7 is then preferably performed in the method of the invention on the skin-face mask 105.
  • the skin-face mask 105 is analyzed and at least one of the following geometric constrains is verified: all skin-face regions SFR having an area smaller than the minimum face area (defined in step 1 of the method of the invention) are ruled out. - Discontinuity (D) calculation. "Holes" within the skin-face region SFR are searched.
  • a second binary skin-face mask 106 is thus calculated.
  • This mask 106 contains as pixels having value set equal to 1 the skin-face regions SFR of mask 105 whose pixels satisfy the following equation
  • geometric threshold (5) wherein according to a preferred embodiment the geometric threshold has a value of 1.6. All the other skin-face regions SFR not satisfying eq. (5), are set equal to 0.
  • the geometric analysis step 7 thus reduces the number of skin-face regions which are candidates to contain faces.
  • the effect of the application of the geometric analysis step 7 is shown. In the original input image 100, in which two persons are shown, both faces have been detected by the face detector of step 2 (two correctly detected bounding boxes BB).
  • the image masked with the skin-face mask 106 is now rescaled again to return to its original size, i.e. to the original input image size, in order to process the image further at its full resolution.
  • the bounding boxes BB obtained via the face detector algorithm of step 2a are then analyzed in a subsequent step 20 of the invention. This step may be performed immediately after the step of face detection 2.
  • Each bounding box BB is processed using the skin detector of step 3. If at least 50% of the total area delimited by the bounding box is recognized as a skin region(s), the pixels within the box BB are filtered using the redness filter of equation (1) (i.e. only pixels having a redness less than 1.2 are maintained).
  • Each masked bounding box BB is then transformed to the YCbCr space and only the pixels satisfying eq. (3) are kept, thus generating a new filtered bounding box FBB for each bounding box BB of mask 102.
  • non- convex regions are closed using morphological operations.
  • the skin detector of step 3 reveals one or more skin regions within the bounding box BB whose area(s) is less than 50% of the total area of the bounding box BB, only the redness filter is applied, but not eq. (3), obtaining for each bounding box BB of mask 102 having "small" skin portions a masked bounding box MBB.
  • Masked bounding boxes MBB and filtered bounding boxes FBB form a mask 109.
  • the redness mask (see eq. (1)) is again applied to the skin-face regions which have been rescaled back to their full resolution.
  • this (sub)step corresponds to the box "definition of the single skin-face region" in the analysis step 7. Due to the rescaling, additional "extremely red” pixels may have been reintroduced and may create disturbance in the following step of red-eye detection.
  • step 8 of the method of the invention in which the red eye artifacts are detected, only the face regions of the input image identified by the face detector and additionally processed as explained in step 20 above (which are the masked bounding boxes and the filtered bounding boxes), and the geometric skin-face regions obtained masking the rescaled image 100 with mask 106 obtained at the end of the geometric analysis step 7 are further analyzed. Both those regions are the candidate face regions CFR in which the algorithm for red eye detection looks for red eyes.
  • the masks 109 and 106 are used to segment the original input image 100: only the regions of the original image 100 having pixels set to 1 in masks 109 or mask 106 are further analyzed in the following steps of the method of the invention. In other words, only the "1" regions of masks 106 and 109 are kept of the original image 100 and are the "candidate face regions" CFR, all the other regions (which are "0" both in mask 106 and in mask 109) are removed. Each candidate face region is processed as outlined below. At the end of the processing, all the following is repeated for the next candidate face region till all of them have been processed.
  • PurpleBrownO, y) Max[ ⁇ , B ⁇ x, y) - G ⁇ x, y)] (6) where B and G are the blue and green components in the RGB color space.
  • the redness values calculation creates for each candidate face region CFR a monochrome image 110, in gray scale, wherein the highly intense red regions appear as bright spots.
  • the purple-brown values calculation generates for each candidate face region CFR a second monochrome image 111 wherein bright regions appear, which are the regions in which the purple-brown component is dominant. Applicants have noted that the purple-brown component is an important component of red eyes' artifacts.
  • the bounding box BB originated applying the face detector to original image 100' of fig. 3 is shown (top image).
  • the redness filter and then the skin detector are applied, generating the filtered bounding box FBB (middle image).
  • the results of the redness calculation, i.e., the redness values as defined in eq. (1) calculated for each pixel of the filtered bounding box FBB and then plotted, are represented as image 110' (bottom image).
  • one of the bounding boxes BB obtained applying the face detector step to the image 100 is depicted (top image).
  • the respective corresponding redness map 110 and purple-brown map 111 are also shown (middle and bottom images, respectively).
  • the redness map is shown for the skin region GSFR (see masked image 106' of fig. 9) detected by the skin detector in the image 100' of fig. 3.
  • Figure 10 shows that red eyes are identified as bright spots both in the redness and purple- brown maps.
  • the purple-brown and redness maps are preferably smoothed using, for example, a bi- dimensional Gaussian filter, in order to remove small discontinuities and highlights local redness maxima.
  • the Gaussian filter is calculated from the following equation (7):
  • redness and purple-brown maps 110, 111 obtained using eq. (1) and (6) for each candidate face region CFR.
  • the local maxima of the redness are also calculated. This calculation selects a given number of "red regions" RLM which might be red eyes to be optionally corrected.
  • the imextendedmax function identifies all regional maxima that are greater than a specified threshold. It computes the extended-maxima transform, which is the regional maxima of the H-maxima transform. H is a nonnegative scalar.
  • a regional maximum is a connected set of pixels of constant intensity from which it is impossible to reach a point with higher intensity without first descending; that is, a connected component of pixels with the same intensity value, t, surrounded by pixels that all have a value less than t.
  • the local maxima with external boundary pixel having a value lower than 50%, 40%, 30%, 20%, 10% of the maximum value t are then calculated and summed.
  • PrMax is the redness smoothed using the Gaussian filter and then normalized.
  • each redness local maximum region RLM is processed according to the following sub-steps.
  • Too large local maxima regions are also discarded. Too large means for example larger than 0,01 of the total area of the candidate face region.
  • red eye region has the following characteristics:
  • the upper and lower limits depend, among other, on the colour space of the input image. If several regions satisfy equations (11) and (12), the two having higher PR are considered to be red eyes RE. If no region satisfies the above mentioned equations, no redeye RE is detected.
  • a single region satisfying eq. (11) and (12) is found (the number of red eyes found is calculated in step 8a of the method of the invention), then the region is considered to be a red eye RE and a subsequent step 10 of the method of the invention is performed to locate the second eye of the face.
  • the red-eye detection step is shown.
  • both red eyes RE of the bounding box of image 100' are immediately detected using the geometric and colour constraints above described.
  • a single red eye RE is detected and thus the following step of the invention is performed.
  • the second red eye is not immediately recognized due, most probably, to the presence of glasses.
  • the second eye locator step 10 it is assumed that frequently in an input image both eyes of a person are shown. Therefore, after the localization of the first red eye region RE, all the local redness maxima RLM considered in the sub-steps above outlined, with the exception of all regions RLM too close or too far from the first red eye region RE already detected, are again processed.
  • the distance between the first detected red eye region RE and the candidate second red eye region is preferably comprised between 3 and 10 times the major axis of an ellipsis encircling the first detected red eye region RE. Additionally, also redness local maxima regions having an area 25% smaller than, or larger than 2 times, the area of the first red eye region RE already located are discarded.
  • a new geometric and colour analysis of the redness local maxima regions RLM (already calculated using the redness map 110) is performed.
  • the constraints for the detection of the second red eye are less strict than for the first one.
  • the new geometric constraints to be satisfied by a local maxima to be considered the second red eye are the following: O > 17% e ⁇ 0.85 (13)
  • PB fifth lower purple-brown limit
  • ⁇ & R fifth lower redness limit
  • the third and fourth lower redness and purple-brown limits are smaller or equal to the first and second redness and purple-brown limits, respectively.
  • the above values are equal to:
  • PB 0 and R > 1.4.
  • the region having the highest PB R value is considered to be second red eye region RE2.
  • the detection of the second eye RE2 is shown in fig. 16a. Before the local maxima regions processing, the region already identified as the first red eye RE is removed from the image. The second red eye RE2 is thus immediately identified, as shown in the mask at the right. An optional additional step 11 of the method of the invention is thus performed in order to correct the red eyes RE, RE2 artefacts detected.
  • the algorithm used in this step is the colour correction of the red-eye(s). If a pixel has been detected as belonging to a red eye region RE, RE2 it is replaced with a substantially monochrome pixel, as detailed below. First of all, a smoothing mask is created having the size of the red-eye region to be corrected. This mask avoids unnatural transitions between corrected and original parts of the eye.
  • the weighting factor w is calculated as follows:
  • Equation (15) above the minimum value is taken between the lred value and 1.
  • the value of lred is normalized with respect to its maximum value reached in the red region under consideration.
  • G new (JC, y) G(x, y) - (l - MASKsmooth) + MASKsmooth ⁇ (B(x, y) + G(x, y)) — .
  • Each pixel of the red eye region is thus replaced with a pixel having the above calculated colour. It is however avoided to apply the mentioned correction to the possible white pixels (the flesh "glint”) and black (eyelashes) present in the red eye region.
  • a corrected digital image 200 is thus created.
  • Figs. 17 and 18 show the red eye correction on image 100 and 100' obtaining image 200 and 200' respectively. From the modified picture, it is visible the "natural" appearance of the corrected eyes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

La présente invention porte sur un procédé pour une détection d'yeux rouges dans une image d'entrée numérique (100, 100'), comprenant : une étape de détection de visage pour identifier une ou plusieurs régions de visage (SR, BB) à l'intérieur de l'image d'entrée (100, 100'), une étape de mappage de rougeur et de brun-pourpre, dans laquelle pour chaque pixel desdites régions de visage (SR, BB) ayant des coordonnées (x, y) dans l'image d'entrée (100, 100') ses valeurs de rougeur et de brun-pourpre sont calculées comme suit (formule (I)), f et f' étant des fonctions, α - (β + γ) = 0; α, μ, v > 0; β, γ ≥ 0, et une étape d'identification de régions d'yeux rouges candidates (LRM), lesdites régions d'yeux rouges candidates (LRM) étant des parties desdites régions de visage (SR, BB) ayant une moyenne (R) de rougeur calculée et une moyenne (PB) de brun-pourpre calculée qui satisfont : PB > première limite inférieure brun-pourpre et R > première limite inférieure de rougeur, et limite inférieure PR < PR mod (R.PB) < limite supérieure PR. Un autre aspect de l'invention porte sur un procédé d'identification d'au moins un visage dans une image numérique.
EP07723937A 2007-04-03 2007-04-03 Procédé pour une détection d'yeux rouges dans une image numérique Withdrawn EP2143069A1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2007/002996 WO2008119368A1 (fr) 2007-04-03 2007-04-03 Procédé pour une détection d'yeux rouges dans une image numérique

Publications (1)

Publication Number Publication Date
EP2143069A1 true EP2143069A1 (fr) 2010-01-13

Family

ID=38828753

Family Applications (1)

Application Number Title Priority Date Filing Date
EP07723937A Withdrawn EP2143069A1 (fr) 2007-04-03 2007-04-03 Procédé pour une détection d'yeux rouges dans une image numérique

Country Status (2)

Country Link
EP (1) EP2143069A1 (fr)
WO (1) WO2008119368A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8170332B2 (en) 2009-10-07 2012-05-01 Seiko Epson Corporation Automatic red-eye object classification in digital images using a boosting-based framework
WO2014203082A2 (fr) * 2013-05-28 2014-12-24 Marvell World Trade Ltd. Systèmes et procédés de correction des yeux rouges

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6016354A (en) * 1997-10-23 2000-01-18 Hewlett-Packard Company Apparatus and a method for reducing red-eye in a digital image
US7116820B2 (en) * 2003-04-28 2006-10-03 Hewlett-Packard Development Company, Lp. Detecting and correcting red-eye in a digital image
US7333653B2 (en) * 2003-08-29 2008-02-19 Hewlett-Packard Development Company, L.P. Detecting and correcting redeye in an image
JP4757559B2 (ja) * 2004-08-11 2011-08-24 富士フイルム株式会社 被写体の構成要素を検出する装置および方法
US8374403B2 (en) * 2005-05-16 2013-02-12 Cisco Technology, Inc. Methods and apparatus for efficient, automated red eye detection

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2008119368A1 *

Also Published As

Publication number Publication date
WO2008119368A1 (fr) 2008-10-09

Similar Documents

Publication Publication Date Title
US7953251B1 (en) Method and apparatus for detection and correction of flash-induced eye defects within digital images using preview or other reference images
US8254674B2 (en) Analyzing partial face regions for red-eye detection in acquired digital images
JP5547730B2 (ja) 顔検知を用いた顔及び肌の自動美化
EP1807805B1 (fr) Procede et dispositif de detection d&#39;yeux rouges dans une image numerique acquise
US8520089B2 (en) Eye beautification
US8331666B2 (en) Automatic red eye artifact reduction for images
US20060280363A1 (en) Image processing apparatus and method, computer program, and storage medium
US20170221186A1 (en) Device for and method of enhancing quality of an image
JP2007504719A (ja) 組み込みアプリケーションに適した、画像における赤目の検出と補正を行うシステム及び方法
JP2002183729A (ja) 画像肌色検出における金髪画素除去方法
EP2143069A1 (fr) Procédé pour une détection d&#39;yeux rouges dans une image numérique
Schettini et al. A modular procedure for automatic red eye correction in digital photos
CN117522802A (zh) 视网膜图像预处理方法、装置、电子设备及存储介质
Safonov et al. Automatic Red Eye Correction
Kong et al. Adaptive image backlight compensation for mobile phone users

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20091103

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: UNIVERSITA DEGLI STUDI DI MILANO - BICOCCA

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20130621

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20131101