Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/77—Retouching; Inpainting; Scratch removal
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/90—Dynamic range modification of images or parts thereof
  • G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/0002—Inspection of images, e.g. flaw detection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/11—Region-based segmentation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/10—Segmentation; Edge detection
  • G06T7/187—Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/90—Determination of colour characteristics
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/18—Eye characteristics, e.g. of the iris
  • G06V40/193—Preprocessing; Feature extraction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
  • H04N1/46—Colour picture communication systems
  • H04N1/56—Processing of colour picture signals
  • H04N1/60—Colour correction or control
  • H04N1/62—Retouching, i.e. modification of isolated colours only or in isolated picture areas only
  • H04N1/624—Red-eye correction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20112—Image segmentation details
  • G06T2207/20156—Automatic seed setting
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30216—Redeye defect
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
  • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
  • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
  • G06V40/178—Human faces, e.g. facial parts, sketches or expressions estimating age from face image; using age information for improving recognition

Definitions

• the invention relates to image processing and more particularly relates to correction of redeye defects in images of humans.
• BACKGROUND OF THE INVENTION When flash illumination is used to photograph humans, the pupils of the humans sometimes appear red. This phenomena is called “redeye” and reduces the quality of an image. Redeye is caused by red light reflecting off of the eye's retina and then exiting the eye through the pupil and returning to the camera.
• redeye is caused by red light reflecting off of the eye's retina and then exiting the eye through the pupil and returning to the camera.
• Many algorithms have been proposed to correct redeye, with the goal of generating an improved image where the pupils appear natural. In some cases, an operator manually paints the redeye portion of a photographic print with a black marker.
• the image capture device 10 is a conventional photographic film camera for capturing a scene on color negative or reversal film, and a film scanner device for scanning the developed image on the film and producing a digital image.
• the capture device can also be an electronic capture unit (not shown) having an electronic imager, such as a charge coupled device or CMOS imager.
• the electronic capture unit can have an analog-to-digital converter/amplifier that receives the signal from the electronic imager, amplifies and converts the signal to digital form, and transmits the image signal to the digital image processor 20.
• the digital image processor 20 provides the means for processing the digital images to produce pleasing looking images on the intended output device or media.
• redeye defect detector 110 skin regions of the digital image are identified based on color and shape and resized for analysis. Each skin region is searched for pairs of small red candidate defects. Various scores are analyzed (for example, symmetry, score with respect to matching an eye template, and so on) and a final classification is performed indicating the position of likely redeye defect pairs 112 in an image.
• the redeye defect detector 110 can internally scale the size of the digital image 102 by interpolation to normalize the analysis image size or to normalize the size of faces or skin regions in the image.
• Each defect pair 112 has two seed defects (also referred to herein as "defect seeds”), one for each of the left and right eyes of a human visage.
• the distance measurer 125 measures the distance between the members of the defect pair.
• This defect pair separation R D is used to prevent overcorrections by the grown defects.
• the defect pair separation can be calculated at different stages of the method. Pixels are convenient units of measurement. In a particular embodiment, having a defect pair with a pair of single pixel seed defects, the values are: left seed defect : (x ⁇ ,y ⁇ ) right seed defect: (x2, Y2)
• the distance measurer 125 measures the distance between the pair of single-pixel seed defects using the well-known distance formula:
• Ti is related to the maximum diameter of a pupil divided by the distance between human pupils and can be determined by measuring the pupil sizes and separation distances for many people and finding the maximum and an average or median distance between human pupils.
• T 2 is related to providing a sufficient "safety margin" so that the first size limit S L I will not damage the defects by trimming too aggressively, if the distance RD is erroneously low.
• a convenient value of Ti is
• SL 2 is the second size limit
• RD is the defect pair separation
• the distance of defective pixels affected by redeye can be greater than 0.072 * RD due to the fact the blurring or defocus in the optical system effectively spreads the redeye effect onto pixels in the vicinity of the pupil.
• the blurring amount is represented by constant
• Blur is conveniently expressed as a blurring radius for the imaging system expressed in units of pixels of a blurring radius. If the actual blurring radius is known for the imaging system that captured the image, it can be substituted for T4.
• FIG. 5 illustrates a partially adjusted grown defect 150, corresponding to the grown defect of FIG. 4 following the completion of the first stage.
• the partially adjusted grown defect 150 has a seed defect 152, pixels 154 to be removed, because Dc exceeds
• the order of the first and second adjustment stages can be switched, that is the adjustment based on the grown defect's centroid can precede the adjustment based on the grown defect's seed defect location; however, it has been determined empirically that the described order provides a higher quality correction.
• the defect pair separation R D may be disproportionate to the actual separation of the human subject's eyes due to rotation of the subject's head relative to the image plane of the camera.
• the defect pair separation RD is proportional to actual eye separation when the subject is front-on to the camera, that is, when a line connecting the subject's eyes is parallel to the image plane.
• the defect pair separation R D can be replaced in the above formulas with a modified separation distance R M that is invariant to head rotation about an internal vertical axis.
• R M is expressed by the formula:
• R M and R D are as above indicated and the rotation angle A is the smallest angle between an imaginary line connecting the human subject's eyes and the image plane.
• the rotation angle A is the angle of head rotation about a vertical axis passing through the top to the bottom of the head. For example, when the subject is looking straight on to the camera, the angle is 0. When the image is a profile of the left side of the face, the angle A is p/2 radians.
• Head pose (and angle A) can be derived through automatic analysis of an image. An example of such a procedure is disclosed in "Estimating Facial Pose Using the EM Algorithm", K. Choi et al., Ninth British Machine Vision Conference, [online], 1998 [retrieved on 2004-03-01]. Retrieved from the Internet: URL: lTtt ⁇ ://www.bmva.ac.uk/bmvc/1998/ ⁇ dfpl47. ⁇ df
• Choi et al. describe estimating 3D facial pose by using 2D projections of 3D templates fitted to the 2D feature positions.
• the process involves projecting control points of the 3D template to 2D with a set of Euler rotations and orthographic projection.
• the projected control points are examined in light of the 2D feature positions to arrive at an updated estimate for the Euler rotations.
• the process iterates to arrive at a final set of Euler rotations describing the 3D facial pose.
• the head rotation determiner 129 analyzes the image of the human subject's face corresponding to the defect pair, and outputs an angle A of in-plane rotation that is input to the size limiter 134 to be used to determine the value of the second size limit SL 2 -
• a rotation-independent second size limit Su can be calculated using the knowledge of the head rotation as: ⁇ _ ⁇ ° D i T ⁇ L2 - cos A 7 + l 4
• SL ' is the rotation-independent second size limit A is the angle of head rotation about a vertical axis passing from the top to the bottom of the head; and RD, T3, and T 4 have the meanings above discussed.
• the ratio of pupil radius to distance between the eyes is highly dependent on the age of the human subject. For babies and small children, the ratio (ignoring blur) can achieve a maximum of about 0.072 when the subject is photographed straight-on and has maximally dilated pupils. With adults, the ratio achieves a maximum value of about 0.05. Therefore, in order to further improve the second size limit, the age of the human subject is determined, possibly as a coarse category classification (e.g. baby, child, teen, adult).
• the correction is as follows. For each pixel in the modified defect, the color is modified by replacing the values of every color component with that of the blue color component value.
• the color modification of pixels included in the modified defect for digital images having red, green and blue color components can be described as:
• the blue color component does not need to be modified and the glint of the eye can be treated with the same correction technique as the redeye pixels, thus this correction has the advantage that it is very fast.
• the described method of using the blue color component to correct the red and green color components of the redeye pixels not only empirically provides a redeye defect correction that is preferred over other methods, but it can be justified through data analysis.
• the pixel values of a large number of images of human pupils were examined. The pupils were classified as either "redeye pupils" if the pupils appeared to be a redeye defect or as a "non-redeye pupils" when the pupils did not appear to be a redeye defect.
• the red color component of the redeye pixels received 95% more exposure than the red color component of the non- redeye pixels.
• T 5 is a constant useful for adjusting the desired lightness of corrected redeye defects.
• the value of T 5 can range from 0 to -20 depending on the desired lightness of the corrected redeye and on the color space representation of the digital image 102.
• the color modifier 136 can, alternatively, apply a correction based on the color characteristics of neighboring skin regions. It has been determined through study of images of human faces without the redeye defect (i.e. the image of the human pupils appear black) that there exists a relationship between skin lightness and pupil lightness.
• the color modifier 136 computes a target pupil value V based on the average flesh color of the skin region associated with the defect pair 112. The target pupil value V is then substituted for all color components of all pixels of the modified defect by the color modifier 136.
• the target pupil value can be determined with the following equation:
• F a is the average value of the green component for the skin region containing the defects.
• the target pupil value V is 21.
• a scene exposure value that renders to an sRGB code value of 21 is about 3.19 stops below the scene exposure value that renders to an sRGB code value of 150 in some digital cameras, such as the DX7630 digital camera, which is marketed by Eastman Kodak Company of Rochester, New York.
• the defect blender 138 which reduces the visibility of the border between the defect pixels and the border pixels.
• a spatial operator is computed.
• the spatial operator is an NxN filter. To preserve phase, N must be odd.
• An alternative spatial operator has a circular region of support, rather than a square region.
• the spatial operator is a symmetric lowpass filter.
• This defect blender 138 operates as follows. For each pixel, a local NxN neighborhood is examined. The number of pixels P belonging to the adjusted defect within the local neighborhood is tabulated. The number P can range between 0 and N 2 , inclusive. When the number P is either zero (no pixels belong to the adjusted defect) or N 2 (all pixels in the local neighborhood belong to the adjusted defect) the pixel is left unchanged.
• Ic(x,y) (1-W) pc (x,y) + W B c (x,y)
• W is from 0 to 1.
• W is a weight related to the aforementioned number of pixels P in the local NxN neighborhood belonging to the modified defect.
• a currently preferred weight W is:
• W is maximized when the local neighborhood is centered on the border between pixels belonging to modified defect and non-defect pixels in the image.
• the improved pixel values are substituted into the color modified image, producing the output improved digital image 120.
• the improved digital image 120 has been improved by modifying redeye affected pixels, producing an image with naturally appearing human pupils.
• the method of the present invention can be performed in a digital camera, a digital printer, or on a personal computer.
• the invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Multimedia (AREA)
• Human Computer Interaction (AREA)
• Health & Medical Sciences (AREA)
• Ophthalmology & Optometry (AREA)
• General Health & Medical Sciences (AREA)
• Quality & Reliability (AREA)
• Signal Processing (AREA)
• Image Processing (AREA)
• Facsimile Image Signal Circuits (AREA)
• Color Image Communication Systems (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (2)

Family

ID=34911769

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (35)

Family Cites Families (15)

• 2004
  • 2004-03-03 US US10/792,079 patent/US7684642B2/en not_active Expired - Fee Related
• 2005
  • 2005-02-16 EP EP05713667A patent/EP1721296B1/en not_active Ceased
  • 2005-02-16 DE DE602005021462T patent/DE602005021462D1/de not_active Expired - Lifetime
  • 2005-02-16 DE DE602005021489T patent/DE602005021489D1/de not_active Expired - Lifetime
  • 2005-02-16 JP JP2007501809A patent/JP4747163B2/ja not_active Expired - Fee Related
  • 2005-02-16 WO PCT/US2005/004930 patent/WO2005093660A2/en not_active Ceased
  • 2005-02-16 EP EP06026154A patent/EP1762981B1/en not_active Ceased

Also Published As

Similar Documents

Legal Events