US8797448B2 - Rapid auto-focus using classifier chains, MEMS and multiple object focusing - Google Patents

Rapid auto-focus using classifier chains, MEMS and multiple object focusing Download PDF

Info

Publication number
US8797448B2
US8797448B2 US14174821 US201414174821A US8797448B2 US 8797448 B2 US8797448 B2 US 8797448B2 US 14174821 US14174821 US 14174821 US 201414174821 A US201414174821 A US 201414174821A US 8797448 B2 US8797448 B2 US 8797448B2
Authority
US
Grant status
Grant
Patent type
Prior art keywords
focus
out
slightly
significantly
face
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US14174821
Other versions
US20140193088A1 (en )
Inventor
Adrian Capata
Stefan Petrescu
Petronel Bigioi
Peter Corcoran
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fotonation Ltd
Original Assignee
Fotonation Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Grant date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/62Methods or arrangements for recognition using electronic means
    • G06K9/6267Classification techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00221Acquiring or recognising human faces, facial parts, facial sketches, facial expressions
    • G06K9/00228Detection; Localisation; Normalisation
    • G06K9/00255Detection; Localisation; Normalisation using acquisition arrangements
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration, e.g. from bit-mapped to bit-mapped creating a similar image
    • G06T5/001Image restoration
    • G06T5/003Deblurring; Sharpening

Abstract

A smart-focusing technique includes identifying an object of interest, such as a face, in a digital image. A focus-generic classifier chain is applied that is trained to match both focused and unfocused faces and/or data from a face tracking module is accepted. Multiple focus-specific classifier chains are applied, including a first chain trained to match substantially out of focus faces, and a second chain trained to match slightly out of focus faces. Focus position is rapidly adjusted using a MEMS component.

Description

BENEFIT CLAIM

This application claims the benefit and priority under 35 U.S.C. §120 as a Continuation of U.S. patent application Ser. No. 12/944,701 (now U.S. Pat. No. 8,648,959), titled “Rapid Auto-Focus Using Classifier Chains, Mems and/or Multiple Object Focusing,” filed Nov. 11, 2010, which is related to a contemporaneously filed U.S. patent application Ser. No. 12/944,703 titled “Rapid Auto-Focus Using Classifier Chains, Mems and/or Multiple Object Focusing” by the same inventors and assignee. The entire contents of each of the foregoing documents are hereby incorporated by reference as if fully set forth herein. The applicants hereby rescind any disclaimer of claim scope in the parent application or the prosecution history thereof and advise the USPTO that the claims in this application may be broader than any claim in the parent applications.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to face detection and recognition, particularly under out of focus conditions

2. Description of the Related Art

Viola-Jones proposes a classifier chain consisting of a series of sequential feature detectors. The classifier chain rejects image patterns that do not represent faces and accepts image patterns that do represent faces.

A problem in face recognition processes arises when faces that are out of focus are distributed in a large area of face space making correct classification difficult. Faces with similar focus conditions tend to be clustered together and correct clustering of images of the same person is difficult. It is desired to be able to detect faces that are out of focus within images, or where another difficult characteristic of a face exists such as a face having a non-frontal pose or uneven illumination. It is also desired to have a method to normalize focus on faces, for example, for use in face recognition and/or other face-based applications.

Each of Mitra S et al “Gaussian Mixture Models Based on the Frequency Spectra for Human Identification and Illumination Classification”, 4th IEEE Workshop on Automatic Identification Advanced Technologies, 2005, Buffalo, N.Y., USA 17-18 Oct. 2005, pages 245250; and Kouzani A Z “Illumination-effects compensation in facial images”, IEEE International Conference on Systems, Man, and Cybernetics, 1999, IEEE SMS '99 Conference Proceedings, Tokyo, Japan 12-15 Oct. 1999, vol 6 pages 840-844 disclose conventional face illumination normalization methods.

Having objects at different distances to a digital camera, or a camera-phone, video camera, or other camera-enabled device or image acquisition device, in focus is a well-known problem in the digital photography industry. Solutions such as extended depth of field do tackle this problem, but only partially, ensuring that the close objects are still sharp when the camera focuses to infinity (deep focus). It is desired to have an efficient technique to handle digital images initially having out of focus objects.

BRIEF DESCRIPTION OF THE DRAWINGS

The present approach is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1 is a block diagram illustrating the principle components of an image processing apparatus according to a preferred embodiment of the present invention.

FIG. 2 is a flow diagram illustrating the operation of the image processing apparatus of FIG. 1.

FIGS. 3A-3D shows examples of images processed by the apparatus of the preferred embodiment.

FIG. 4 is a block diagram of an image processing system in accordance with certain embodiments.

FIG. 5 illustrates a main Image sorting/retrieval workflow m accordance with certain embodiments.

FIG. 6A illustrates an exemplary data storage structure for an image collection data set.

FIGS. 6B and 6D illustrate aspects of an image classifier where the feature vectors for individual patterns can be determined relative to an “averaged” pattern (mean face) and where feature vectors for individual patterns are determined in absolute terms (colour correlogram), respectively.

FIGS. 6C and 6E illustrate the calculation of respective sets of similarity measure distances from a selected classifier pattern to all other classifier patterns within images of the Image Collection.

FIG. 6F illustrates how multiple classifiers can be normalized and their similarity measures combined to provide a single, similarity measure.

FIG. 7 is a block diagram of an in-camera image processing system according to certain embodiments.

FIG. 8 illustrates a face-based auto focus method in accordance with certain embodiments.

FIG. 9A-9B illustrate face detection methods in accordance with certain embodiments.

FIGS. 10A-10B illustrate a further method in accordance with certain embodiments.

FIGS. 11A-11E illustrate a further technique in accordance with certain embodiments.

DETAILED DESCRIPTIONS OF THE EMBODIMENTS

A method of digital image processing using face or other object detection is provided. A scene is acquired with a digital image acquisition device including multiple features. A first group of pixels is identified that corresponds to a first feature within the scene. An optic is adjusted in real time using a MEMS component to a first focus position to focus the device on the first feature. A first digital image is captured at the first focus position. A second group of pixels is identified that corresponds to a second feature within approximately the same scene. The optic is adjusted in real time using the MEMS component to a second focus position to focus the device on the second feature. A second digital image is captured at the second focus position. The first and second digital images are registered including the first and second features. The first and second digital images are stored, transmitted, captured, combined and/or displayed together.

The method may include determining that the first feature appears blurry in the second digital image and/or that the second feature appears blurry in the first image. The storing, transmitting, combining, capturing and/or displaying of the first and second digital images together may involve generating a composite image including the first feature from the first digital image and the second feature from the second digital image, such that both of the first and second features appear to be sharp in the composite image. The generating a composite image may involve blending and/or morphing the first and second images.

The storing, capturing, combining, transmitting or displaying of the first and second digital images together may include providing a toggle feature to call the first and second digital images together and toggle between them. A selection of one of the first and second digital images may be received for further storing, transmitting, displaying and/or further processing. The toggle feature may permit a display to appear to demonstrate toggling between the first and second focus positions in a same image.

The registering may include aligning the first and second digital images by applying digital or mechanical image stabilization. The identifying of the first or second groups of pixels respectively may include applying face detection to the acquired scene and identifying the first or second features as corresponding to first or second faces. The method may include applying face tracking to the first and/or second group of pixels for respectively continuing to identify in subsequent images the first and/or second groups of pixels as corresponding to the first and/or second face.

A further method is provided for smart-focusing on a detected face region in a scene. The method includes acquiring a digital image using a camera-enabled device including a lens, an image sensor, a memory and a processor. A group of pixels is identified that contain a face, and a focus-generic classifier chain trained to match both focused and unfocused faces is applied and/or data is accepted from a face tracking module. Multiple focus-specific classifier chains are applied, including a first chain trained to match substantially out of focus faces, and a second chain trained to match slightly out of focus faces. When the face is determined not to be sharply focused as significantly matching the first and/or second focus specific classifier chains, the focus position is adjusted using a MEMS component one or more times based on a degree of focus determined at one or more preceding focus positions, such as to ultimately focus on the face using a series of one or more selected smart-focus adjustments based on application of focus-specific classifier chains.

When the face is determined to be substantially unfocused as best matching the first focus specific classifier chain, then the adjusting may include coarsely adjusting to a second focus position and repeating the applying the multiple focus-specific classifier chains.

When the face is determined to be substantially unfocused at the second focus position, then the method may further include coarsely adjusting focus to a third focus position oppositely as the adjusting to the second focus position in a reduced and/or increased amount as the adjusting to the second focus position.

The multiple focus-specific classifier chains may include a third focus-specific classifier chain trained to match sharply focused faces. When the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting may include finely adjusting to a second focus position and repeating the applying of the at least three focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain.

When the face is determined to be slightly unfocused as best matching the second focus specific classifier chain, then the adjusting may include finely adjusting to a second focus position and repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the face.

The multiple focus-specific classifier chains may include a third focus-specific classifier chain trained to match sharply focused faces. When the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting may include finely adjusting to a second focus position and repeating the applying of the multiple focus-specific classifier chains one or more times until a best match is achieved with the third focus-specific classifier chain.

When the face is determined to be slightly unfocused as best matching the second focus-specific classifier chain, then the adjusting may include finely adjusting to a second focus position and repeating the applying of the second focus-specific classifier chain until a focus position is achieved between two slightly unfocused positions in front of and behind the face.

The method may further include applying one or more specific facial feature classifier chains each trained to enable optimal focus on a feature of the detected face including one or both eyes, mouth, chin, nose, one or both ears, hairline, forehead, profile or other partial face region, or combinations thereof. The method may include adjusting focus position to obtain optimal focus on a selected specific facial feature of the detected face.

The applying of at least one of the multiple focus-specific classifier chains may be repeated at each focus position.

A focus normalization method is also provided. A digital image is acquired including data corresponding to a face that appears to be out of focus. One or more out-of-focus classifier programs are applied to the face data, and the face data is identified as corresponding to a face. An out of focus condition is also determined for the face as a result of the applying of the one or more out of focus classifier programs. The out of focus condition of the face is corrected based on the determining to thereby generate a corrected face image appearing to be sharply focused. The method also includes electronically storing, transmitting, capturing, applying a face recognition program to, editing, or displaying the corrected face image, or combinations thereof.

A face recognition program may be applied to the corrected face image. The detecting of the face and the determining of the focus condition of the face may be performed simultaneously. A set of feature detector programs may be applied to reject non-face data from being identified as face data.

A sharply focused classifier program may be also applied to the face data. A focus condition may be determined based on acceptance of the face data by one of the focus condition classifier programs. The digital image may be one of multiple images in a series that include the face, and the correcting may be applied to a different image in the series than the digital image within which the focus condition is determined.

The out of focus classifier programs may include one or more slightly out of focus classifiers and/or one or more significantly out of focus classifiers. A sharply focused classifier program may be applied to the face data. Two or more full classifier sets may be applied after determining that no single focus condition applies and that the face data is not rejected as a face.

A face detection method is also provided. The face detection method includes acquiring a digital image and extracting a sub-window from the image. Two or more shortened face detection classifier cascades are applied that are trained to be selectively sensitive to a characteristic of a face region. A probability is determined that a face with a certain form of the characteristic is present within the sub-window. An extended face detection classifier cascade is applied that is trained for sensitivity to the certain form of the characteristic. A final determination is provided that a face exists within the image sub-window. The method is repeated one or more times for one or more further sub-windows from the image and/or one or more further characteristics.

The characteristic or characteristics may include a focus condition such as sharply focused, slightly out of focus and/or significantly out of focus, directional illumination of the face region, an in-plane rotation of the face region, a 3D pose variation of the face region. a degree of smile, a degree of eye-blinking, a degree of eye-winking, a degree of mouth opening, facial blurring, eye-defect, facial shadowing, facial occlusion, facial color, or facial shape, or combinations thereof.

A focus condition of a face may be determined within a sub-window based on acceptance by one of the focus condition classifier cascades. The digital image may be one of multiple images in a series that include the face, and an out of focus condition of the face may be corrected within a different image in the series than the digital image within which the focus condition is determined. An out of focus classifier cascade may include one or more slightly out of focus and/or significantly out of focus classifiers.

A further face detection method is provided that includes acquiring a digital image and extracting a sub-window from said image. Two or more shortened face detection classifier cascades may be applied that are trained to be selectively sensitive to a focus condition of a face or other object of interest. A probability may be determined that a face having a certain focus condition is present within the sub-window. An extended face detection classifier cascade may be applied that is trained for sensitivity to the certain focus condition. A final determination is provided that a face exists within the image sub-window. The method may be repeated one or more times for one or more further sub-windows from the image and/or one or more further focus conditions.

The digital image may be one of multiple images in a series that include the face, and an out of focus condition of the face may be corrected within a different image in the series than the digital image within which the focus condition is determined.

The out of focus classifier cascades may include one or more slightly out of focus classifiers and/or one or more significantly out of focus classifiers. A sharply focused classifier cascade may also be applied. A focus condition of a face may be determined within a sub-window based on acceptance by one of the classifier cascades.

A digital image acquisition device is also provided including an optoelectronic system, e.g., including a lens and image sensor, for acquiring a digital image, and a digital memory having stored therein processor-readable code for programming the processor to perform any of the methods described herein.

One or more non-transitory processor-readable storage media having code embedded therein are also provided for programming one or more processors to perform any of the methods described herein.

FIG. 1 illustrates subsystems of a face detection and tracking system according to certain embodiments. The solid lines indicate the flow of image data; the dashed line indicates control inputs or information outputs (e.g. location(s) of detected faces) from a module. In this example an image processing apparatus can be a digital still camera (DSC), a video camera, a cell phone equipped with an image capturing mechanism or a hand held computer equipped with an internal or external camera.

A digital image is acquired in raw format from an image sensor (CCD or CMOS) [105] and an image subsampler [112] generates a smaller copy of the main image. A digital camera may contain dedicated hardware subsystems to perform image subsampling, for example, to provide preview images to a camera display and/or camera processing components. The subsampled image may be provided in bitmap format (ROB or YCC). In the meantime, the normal image acquisition chain performs post-processing on the raw image [110] which may include some luminance and color balancing. In certain digital imaging systems, subsampling may occur after post-processing, or after certain post-processing filters are applied, but before the entire post-processing filter chain is completed.

The subsampled image is next passed to an integral image generator [115] which creates an integral image from the subsampled image. This integral image is next passed to a fixed size face detector [120]. The face detector is applied to the full integral image, but as this is an integral image of a subsampled copy of the main image, the processing required by the face detector may be proportionately reduced. If the subsample is ¼ of the main image, then this implies that the processing time is only 25% of that for the full image.

This approach is particularly amenable to hardware embodiments where the subsampled image memory space can be scanned by a fixed size DMA window and digital logic to implement a Haar-feature classifier chain can be applied to this DMA window. However, certain embodiment may use one or more different sizes of classifier or several sizes of classifier (e.g., in a software embodiment), or multiple fixed-size classifiers may be used (e.g., in a hardware embodiment). An advantage is that a smaller integral image is calculated.

After application of the fast face detector [280], newly detected candidate face regions [141] may be passed onto a face tracking module [111] when it is desired to use face tracking, where one or more face regions confirmed from previous analysis [145] may be merged with the new candidate face regions prior to being provided [142] to a face tracker [290].

The face tracker [290] provides a set of confirmed candidate regions [143] back to the tracking module [111]. Additional image processing filters are applied by the tracking module [111] to confirm either that these confirmed regions [143] are face regions or to maintain regions as candidates if they have not been confirmed as such by the face tracker [290]. A final set of face regions [145] can be output by the module [111] for use elsewhere in the camera or to be stored within or in association with an acquired image for later processing either within the camera or offline; as well as to be used in the next iteration of face tracking.

After the main image acquisition chain is completed a full-size copy of the main image [130] will normally reside in the system memory [140] of the image acquisition system. This may be accessed by a candidate region extractor [125] component of the face tracker [290] which selects image patches based on candidate face region data [142] obtained from the face tracking module [111]. These image patches for each candidate region are passed to an integral image generator which passes the resulting integral images to a variable-sized detector [121], as one possible example a VJ detector, which then applies a classifier chain, preferably at least a 32 classifier chain, but fewer than 32 are used in some embodiments, to the integral image for each candidate region across a range of different scales.

The range of scales [144] employed by the face detector [121] is determined and supplied by the face tracking module [111] and is based partly on statistical information relating to the history of the current candidate face regions [142] and partly on external metadata determined from other subsystems within the image acquisition system.

As an example of the former, if a candidate face region has remained consistently at a particular size for a certain number of acquired image frames then the face detector [121] may be applied at this particular scale and perhaps at one scale higher (i.e. 1.25 time larger) and one scale lower (i.e. 1.25 times lower).

As an example of the latter, if the focus of the image acquisition system has moved to infinity, then the smallest scaling would be applied in the face detector [121]. Normally these scaling would not be employed because they are applied a greater number of times to the candidate face region in order to cover it completely. The candidate face region will have a minimum size beyond which it should not decrease, and this is in order to allow for localized movement of the camera by a user between frames. In some image acquisition systems which contain motion sensors it may be possible to track such localized movements and this information may be employed to further improve the selection of scales and the size of candidate regions.

The candidate region tracker [290] provides a set of confirmed face regions [143] based on full variable size face detection of the image patches to the face tracking module [111]. Clearly, some candidate regions will have been confirmed while others will have been rejected and these can be explicitly returned by the tracker [290] or can be calculated by the tracking module [111] by analyzing the difference between the confirmed regions [143] and the candidate regions [142]. In either case, the face tracking module [111] can then apply alternative tests to candidate regions rejected by the tracker [290] (as explained below) to determine whether these should be maintained as candidate regions [142] for the next cycle of tracking or whether these should indeed be removed from tracking.

Once the set of confirmed candidate regions [145] has been determined by the face tracking module [111], the module [111] communicates with the sub-sampler [112] to determine when the next acquired image is to be sub-sampled and so provided to the detector [280] and also to provide the resolution [146] at which the next acquired image is to be sub-sampled.

It will be seen that where the detector [280] does not run when the next image is acquired, the candidate regions [142] provided to the extractor [125] for the next acquired image will be the regions [145] confirmed by the tracking module [111] from the last acquired image. On the other hand, when the face detector [280] provides a new set of candidate regions [141] to the face tracking module [111], these candidate regions are merged with the previous set of confirmed regions [145] to provide the set of candidate regions [142] to the extractor [125] for the next acquired image.

FIG. 2 illustrates a exemplary workflow. The illustrated process is split into (i) a detection/initialization phase which finds new candidate face regions [141] using the fast face detector [280] which operates on a subsampled version of the full image; (ii) a secondary face detection process [290] which operates on extracted image patches for the candidate regions [142], which are determined based on the location of faces in one or more previously acquired image frames and (iii) a main tracking process which computes and stores a statistical history of confirmed face regions [143]. Although the application of the fast face detector [280] is illustrated as occurring prior to the application of the candidate region tracker [290], the order is not critical and the fast detection is not necessarily executed on every frame and in certain circumstances may be spread across multiple frames. Also, face detection may be used for various applications such as face recognition whether or not face tracking is also used.

In step 205, the main image is acquired and in step 210 primary image processing of that main image is performed as described in relation to FIG. 1. The sub-sampled image is generated by the subsampler [112] and an integral image is generated therefrom by the generator [115], step 211 as described previously. The integral image is passed to the fixed size face detector [120] and the fixed size window provides a set of candidate face regions [141] within the integral image to the face tracking module, step 220. The size of these regions is determined by the sub-sampling scale [146] specified by the face tracking module to the sub-sampler and this scale is based on the analysis of the previous sub-sampled/integral images by the detector [280] and patches from previous acquired images by the tracker [290] as well as other inputs such as camera focus and movement.

The set of candidate regions [141] is merged with the existing set of confirmed regions [145] to produce a merged set of candidate regions [142] to be provided for confirmation, step 242. For the candidate regions [142] specified by the face tracking module 111, the candidate region extractor [125] extracts the corresponding full resolution patches from an acquired image, step 225. An integral image is generated for each extracted patch, step 230 and variable-sized face detection is applied by the face detector 121 to each such integral image patch, for example, a full Viola-Jones analysis. These results [143] are in turn fed back to the face-tracking module [111], step 240.

The tracking module [111] processes these regions [143] further before a set of confirmed regions [145] is output. In this regard, additional filters can be applied by the module 111 either for regions [143] confirmed by the tracker [290] or for retaining candidate regions which may not have been confirmed by the tracker 290 or picked up by the detector [280], step 245.

For example, if a face region had been tracked over a sequence of acquired images and then lost, a skin prototype could be applied to the region by the module [111] to check if a subject facing the camera had just turned away. If so, this candidate region could be maintained for checking in the next acquired image to see if the subject turns back the face to the camera. Depending on the sizes of the confirmed regions being maintained at any given time and the history of their sizes, e.g. whether they are getting bigger or smaller, the module 111 determines the scale [146] for sub-sampling the next acquired image to be analyzed by the detector [280] and provides this to the sub-sampler [112], step 250.

The fast face detector [280] need not run on every acquired image. So for example, where only a single source of sub-sampled images is available, if a camera acquires 60 frames per second, 15-25 sub-sampled frames per second (fps) may be required to be provided to the camera display for user previewing. These images are sub-sampled at the same scale and at a high enough resolution for the display. Some or all of the remaining 35-45 fps can be sampled at the scale determined by the tracking module [111] for face detection and tracking purposes.

The decision on the periodicity in which images are being selected from the stream may be based on a fixed number or alternatively be a run-time variable. In such cases, the decision on the next sampled image may be determined on the processing time it took for the previous image, in order to maintain synchronicity between the captured real-time stream and the face tracking processing. Thus in a complex image environment the sample rate may decrease.

Alternatively, the decision on the next sample may also be performed based on processing of the content of selected images. If there is no significant change in the image stream, the full face tracking process might not be performed. In such cases, although the sampling rate may be constant, the images will undergo a simple image comparison and only if it is decided that there is justifiable differences, will the face tracking algorithms be launched.

It will also be noted that the face detector [280] may run at regular or irregular intervals. So for example, if the camera focus is changed significantly, then the face detector may be run more frequently and particularly with differing scales of sub-sampled image to try to detecting faces which should be changing in size. Alternatively, where focus is changing rapidly, the detector could be skipped for intervening frames, until focus has stabilised. However, it is generally only when focus goes to infinity that the highest resolution integral image is produced by the generator [115].

In this latter case, the detector in some embodiments may not be able to cover the entire area of the acquired, subsampled, image in a single frame. Accordingly the detector may be applied across only a portion of the acquired, subsampled, image on a first frame, and across the remaining portion(s) of the image on subsequent acquired image frames. In one embodiment, the detector is applied to the outer regions of the acquired image on a first acquired image frame in order to catch small faces entering the image from its periphery, and on subsequent frames to more central regions of the image.

An alternative way of limiting the areas of an image to which the face detector 120 is to be applied comprises identifying areas of the image which include skin tones. U.S. Pat. No. 6,661,907, discloses one such technique for detecting skin tones and subsequently only applying face detection in regions having a predominant skin color.

In one embodiment, skin segmentation 190 is preferably applied to the sub-sampled version of the acquired image. If the resolution of the sub-sampled version is not sufficient, then a previous image stored at image store 150 or a next sub-sampled image are preferably used when the two images are not too different in content from the current acquired image. Alternatively, skin segmentation 190 can be applied to the full size video image 130.

In any case, regions containing skin tones are identified by bounding rectangles and these bounding rectangles are provided to the integral image generator 115 which produces integral image patches corresponding to the rectangles in a manner similar to the tracker integral image generator 115.

Not alone does this approach reduce the processing overhead associated with producing the integral image and running face detection, but in certain embodiments, it also allows the face detector 120 to apply more relaxed face detection to the bounding rectangles, as there is a higher chance that these skin-tone regions do in fact contain a face. So for a VJ detector 120, a shorter classifier chain can be employed to more effectively provide similar quality results to running face detection over the whole image with longer VJ classifiers required to positively detect a face.

Further improvements to face detection are also possible. For example, it has been found that face detection is significantly dependent on focus conditions and on illumination conditions and so small variations in focus and/or illumination can cause face detection to fail, causing somewhat unstable detection behavior.

In one embodiment, confirmed face regions 145 are used to identify regions of a subsequently acquired subsampled image on which focus and/or luminance correction should be performed to bring the regions of interest of the image to be analyzed to the desired parameters. One example of such correction is to improve the focus and/or luminance contrast within the regions of the subsampled image confirmed by the confirmed face regions 145.

Contrast enhancement may be used to increase the local contrast of an image, especially when the usable data of the image is represented by close contrast values. Through this adjustment, the intensities for pixels of a region when represented on a histogram which would otherwise be closely distributed can be better distributed. This allows for areas of lower local contrast to gain a higher contrast without affecting the global contrast. Histogram equalization accomplishes this by effectively spreading out the most frequent intensity values.

The method is useful in images with backgrounds and foregrounds that are both bright or both dark. In particular, the method can lead to better detail in photographs that are over or under-exposed. Alternatively, focus and/or luminance correction could be included in the computation of an “adjusted” integral image in the generators 115.

In another improvement, when face detection is being used, the camera application is set to dynamically modify the exposure from the computed default to a higher values (from frame to frame, slightly overexposing the scene) until the face detection provides a lock onto a face. In a separate embodiment, the face detector 120 will be applied to the regions that are substantively different between images. Note that prior to comparing two sampled images for change in content, a stage of registration between the images may be needed to remove the variability of changes in camera, caused by camera movement such as zoom, pan and tilt.

It is possible to obtain zoom information from camera firmware and it is also possible using software techniques which analyze images in camera memory 140 or image store 150 to determine the degree of pan or tilt of the camera from one image to another.

In one embodiment, the acquisition device is provided with a motion sensor 180, as illustrated in FIG. 1, to determine the degree and direction of pan from one image to another so avoiding the processing requirement of determining camera movement III software. Motion sensors may be incorporated in digital cameras, e.g., based on accelerometers, but optionally based on gyroscopic principals, primarily for the purposes of warning or compensating for hand shake during main image capture. In this context, U.S. Pat. No. 4,448,510, Murakoshi, discloses such a system for a conventional camera, or U.S. Pat. No. 6,747,690, Molgaard, discloses accelerometer sensors applied within a modern digital camera.

Where a motion sensor is incorporated in a camera, it may be optimized for small movements around the optical axis. The accelerometer may incorporate a sensing module which generates a signal based on the acceleration experienced and an amplifier module which determines the range of accelerations which can effectively be measured. The accelerometers may allow software control of the amplifier stage which allows the sensitivity to be adjusted.

The motion sensor 180 could equally be implemented with MEMS sensors of the sort which will be incorporated in next generation consumer cameras and camera-phones. In any case, when the camera is operable in face tracking mode, i.e. constant video acquisition as distinct from acquiring a main image, shake compensation might not be used because image quality is lower. This provides the opportunity to configure the motion sensor 180, to sense large movements, by setting the motion sensor amplifier module to low gain. The size and direction of movement detected by the sensor 180 is provided to the face tracker Ill. The approximate size of faces being tracked is already known and this enables an estimate of the distance of each face from the camera. Accordingly, knowing the approximate size of the large movement from the sensor 180 allows the approximate displacement of each candidate face region to be determined, even if they are at differing distances from the camera.

Thus, when a large movement is detected, the face tracker III shifts the location of candidate regions as a function of the direction and size of the movement. Alternatively, the size of the region over which the tracking algorithms are applied may also be enlarged (and, if necessary, the sophistication of the tracker may be decreased to compensate for scanning a larger image area) as a function of the direction and size of the movement.

When the camera is actuated to capture a main image, or when it exits face tracking mode for any other reason, the amplifier gain of the motion sensor 180 is returned to normal, allowing the main image acquisition chain 105,110 for full-sized images to employ normal shake compensation algorithms based on information from the motion sensor 180. In alternative embodiments, sub-sampled preview images for the camera display can be fed through a separate pipe than the images being fed to and supplied from the image sub-sampler [112] and so every acquired image and its sub-sampled copies can be available both to the detector [280], as well as for camera display.

In addition to periodically acquiring samples from a video stream, the process may also be applied to a single still image acquired by a digital camera. In this case, the stream for the face tracking comprises a stream of preview images and the final image in the series is the full resolution acquired image. In such a case, the face tracking information can be verified for the final image in a similar fashion to that illustrated in FIG. 2. In addition, the information such as coordinates or mask of the face may be stored with the final image. Such data for example may fit as an entry in the saved image header, for future post processing, whether in the acquisition device or at a later stage by an external device.

FIGS. 3A-3D illustrate operations of certain embodiments through worked examples. FIG. 3A illustrates the result at the end of a detection & tracking cycle on a frame of video or a still within a series of stills, and two confirmed face regions [301, 302] of different scales are shown.

In this embodiment, for pragmatic reasons, each face region has a rectangular bounding box, as it is easier to make computations on rectangular regions. This information is recorded and output as [145] by the tracking module [111] of FIG. 1. Based on the history of the face regions [301,302], the tracking module [111] may decide to run fast face tracking with a classifier window of the size of face region [301] with an integral image being provided and analyzed accordingly.

FIG. 3B illustrates the situation after the next frame in a video sequence is captured and the fast face detector has been applied to the new image. Both faces have moved [311, 312] and are shown relative to previous face regions [301, 302]. A third face region [303] has appeared and has been detected by the fast face detector [303]. In addition the fast face detector has found the smaller of the two previously confirmed faces [304] because it is at the correct scale for the fast face detector. Regions [303] and [304] are supplied as candidate regions [141] to the tracking module [111]. The tracking module merges this new candidate region information [141], with the previous confirmed region information [145] comprising regions [301] [302] to provide a set of candidate regions comprising regions [303], [304] and [302] to the candidate region extractor [290]. The tracking module [111] knows that the region [302] has not been picked up by the detector [280]. This may be because the face has disappeared, remains at a size that could not have been detected by the detector [280] or has changed size to a size that could not have been detected by the detector [280]. Thus, for this region, the module [111] will specify a large patch [305].

The large patch 305 may be as illustrated at FIG. 3C around the region [302] to be checked by the tracker [290]. Only the region [303] bounding the newly detected face candidate needs to be checked by the tracker [290], whereas because the face [301] is moving a relatively large patch [306] surrounding this region is specified to the tracker [290].

FIG. 3C illustrates the situation after the candidate region extractor operates upon the image, candidate regions [306, 305] around both of the confirmed face regions [301, 302] from the previous video frame as well as new region [303] are extracted from the full resolution image [130]. The size of these candidate regions has been calculated by the face tracking module [111] based partly on partly on statistical information relating to the history of the current face candidate and partly on external metadata determined from other subsystems within the image acquisition system. These extracted candidate regions are now passed on to the variable sized face detector [121] which applies a VJ face detector to the candidate region over a range of scales. The locations of one or more confirmed face regions, if any, are then passed back to the face tracking module [111].

FIG. 3D illustrates the situation after the face tracking module [111] has merged the results from both the fast face detector [280] and the face tracker [290] and applied various confirmation filters to the confirmed face regions. Three confirmed face regions have been detected [307, 308, 309] within the patches [305, 306, 303]. The largest region [307] was known but had moved from the previous video frame and relevant data is added to the history of that face region. The other previously known region [308] which had moved was also detected by the fast face detector which serves as a double-confirmation and these data are added to its history. Finally, a new face region [303] was detected and confirmed and a new face region history must be initiated for this newly detected face. These three face regions are used to provide a set of confirmed face regions [145] for the next cycle.

There are many possible applications for the regions 145 supplied by the face tracking module. For example, the bounding boxes for each of the regions [145] can be superimposed on the camera display to indicate that the camera is automatically tracking detected face(s) in a scene. This can be used for improving various pre-capture parameters. One example is exposure, ensuring that the faces are well exposed. Another example is auto-focusing, by ensuring that focus is set on a detected face or indeed to adjust other capture settings for the optimal representation of the face in an image.

The corrections may be done as part of the pre-processing adjustments. The location of the face tracking may also be used for post processing and in particular selective post processing where the regions with the faces may be enhanced. Such examples include sharpening, enhancing saturation, brightening or increasing local contrast. The preprocessing using the location of faces may also be used on the regions without the face to reduce their visual importance, for example through selective blurring, de-saturation, or darkening.

Where several face regions are being tracked, then the longest lived or largest face can be used for focusing and can be highlighted as such. Also, the regions [145] can be used to limit the areas on which for example red-eye processing is performed when required. Other post-processing which can be used in conjunction with the light-weight face detection described above is face recognition. In particular, such an approach can be useful when combined with more robust face detection and recognition either running on the same or an off-line device that has sufficient resources to run more resource consuming algorithms.

In this case, the face tracking module [111] reports the location of any confirmed face regions [145] to the in-camera firmware, preferably together with a confidence factor. When the confidence factor is sufficiently high for a region, indicating that at least one face is in fact present in an image frame, the camera firmware runs a light-weight face recognition algorithm [160] at the location of the face, for example a DCT-based algorithm. The face recognition algorithm [160] uses a database [161] preferably stored on the camera comprising personal identifiers and their associated face parameters.

In operation, the module [160] collects identifiers over a series of frames. When the identifiers of a detected face tracked over a number of preview frames are predominantly of one particular person, that person is deemed by the recognition module to be present in the image. One or both of the identifier of the person and the last known location of the face are stored either in the image (in a header) or in a separate file stored on the camera storage [150]. This storing of the person's ID can occur even when the recognition module [160] has failed for the immediately previous number of frames but for which a face region was still detected and tracked by the module [111].

When an image is copied from camera storage to a display or permanent storage device such as a PC (not shown), the person ID's are copied along with the images. Such devices are generally more capable of running a more robust face detection and recognition algorithm and then combining the results with the recognition results from the camera, giving more weight to recognition results from the robust face recognition (if any). The combined identification results are presented to the user, or if identification was not possible, the user is asked to enter the name of the person that was found. When the user rejects identification or a new name is entered, the PC retrains its face print database and downloads the appropriate changes to the capture device for storage in the light-weight database [161]. When multiple confirmed face regions [145] are detected, the recognition module [160] can detect and recognize multiple persons in the image.

It is possible to introduce a mode in the camera that does not take a shot until persons are recognized or until it is clear that persons are not present in the face print database, or alternatively displays an appropriate indicator when the persons have been recognized. This allows reliable identification of persons in the image.

This feature solves the problem where algorithms using a single image for face detection and recognition may have lower probability of performing correctly. In one example, for recognition, if the face is not aligned within certain strict limits it is not possible to accurately recognize a person. This method uses a series of preview frames for this purpose as it can be expected that a reliable face recognition can be done when many more variations of slightly different samples are available.

Further improvements to the efficiency of systems described herein are possible. For example, a face detection algorithm may employ methods or use classifiers to detect faces in a picture at different orientations: 0, 90, 180 and 270 degrees. According to a further embodiment, the camera is equipped with an orientation sensor. This can comprise a hardware sensor for determining whether the camera is being held upright, inverted or tilted clockwise or anti-clockwise. Alternatively, the orientation sensor can comprise an image analysis module connected either to the image acquisition hardware 105, 110 or camera memory 140 or image store 150, each as illustrated in FIG. 1, for quickly determining whether images are being acquired in portrait or landscape mode and whether the camera is tilted clockwise or anti-clockwise.

Once this determination is made, the camera orientation can be fed to one or both of the face detectors 120, 121. The detectors need then only apply face detection according to the likely orientation of faces in an image acquired with the determined camera orientation. This feature significantly reduces face detection processing overhead, for example, by avoiding the employing of classifiers which are unlikely to detect faces or increase its accuracy by running classifiers more likely to detects faces in a given orientation more often.

According to another embodiment, there is provided a method for image recognition in a collection of digital images that includes training image classifiers and retrieving a sub-set of images from the collection. The training of the image classifiers preferably includes one, more than one or all of the following: For each image in the collection, any regions within the image that correspond to a face are identified. For each face region and any associated peripheral region, feature vectors are determined for each of the image classifiers. The feature vectors are stored in association with data relating to the associated face region.

The retrieval of the sub-set of images from the collection preferably includes one, more than one or all of the following: At least one reference region including a face to be recognized is/are selected from an image. At least one classifier on which said retrieval is to be based is/are selected from the image classifiers. A respective feature vector for each selected classifier is determined for the reference region. The sub-set of images is retrieved from within the image collection in accordance with the distance between the feature vectors determined for the reference region and the feature vectors for face regions of the image collection.

A component for image recognition III a collection of digital images is further provided including a training module for training image classifiers and a retrieval module for retrieving a sub-set of images from the collection.

The training module is preferably configured according to one, more than one or all of the following: For each image in the collection, any regions are identified that correspond to a face in the image. For each face region and any associated peripheral region, feature vectors are determined for each of the image classifiers. The feature vectors are stored in association with data relating to the associated face region.

The retrieval module is preferably configured according to one, more than one or all of the following: At least one reference region including a face to be recognized is/are selected from an image. At least one image classifier is/are selected on which the retrieval is to be based. A respective feature vector is determined for each selected classifier of the reference region. A sub-set of images is selected from within the image collection in accordance with the distance between the feature vectors determined for the reference region and the feature vectors for face regions of the image collection.

In a further aspect there is provided a corresponding component for image recognition. In this embodiment, the training process cycles automatically through each image in an image collection, employing a face detector to determine the location of face regions within an image. It then extracts and normalizes these regions and associated non-face peripheral regions which are indicative of, for example, the hair, clothing and/or pose of the person associated with the determined face region(s). Initial training data is used to determine a basis vector set for each face classifier.

A basis vector set comprises a selected set of attributes and reference values for these attributes for a particular classifier. For example, for a DCT classifier, a basis vector could comprise a selected set of frequencies by which selected image regions are best characterized for future matching and/or discrimination and a reference value for each frequency. For other classifiers, the reference value can simply be the origin (zero value) within a vector space.

Next, for each determined, extracted and normalized face region, at least one feature vector is generated for at least one face-region based classifier and where an associated non-face region is available, at least one further feature vector is generated for a respective non-face region based classifier. A feature vector can be thought of as an identified region's coordinates within the basis vector space relative to the reference value.

These data are then associated with the relevant image and face/peripheral region and are stored for future reference. In this embodiment, image retrieval may either employ a user selected face region or may automatically determine and select face regions in a newly acquired image for comparing with other face regions within the selected image collection. 5 Once at least one face region has been selected, the retrieval process determines (or if the image was previously “trained”, loads) feature vectors associated with at least one face-based classifier and at least one non-face based classifier. A comparison between the selected face region and all other face regions in the current image collection will next yield a set of distance measures for each classifier. Further, while calculating this set of distance measures, mean and variance values associated with the statistical distribution of the distance measures for each classifier are calculated. Finally these distance measures are preferably normalized using the mean and variance data for each classifier and are summed to provide a combined distance measure which is used to generate a final ranked similarity list.

In another embodiment, the classifiers include a combination of wavelet domain PCA (principle component analysis) classifier and 2D-DCT (discrete cosine transform) classifier for recognizing face regions. These classifiers do not require a training stage for each new image that is added to an image collection. For example, techniques such as ICA (independent component analysis) or the Fisher Face technique which employs LDA (linear discriminant analysis) are well known face recognition techniques which adjust the basis vectors during a training stage to cluster similar images and optimize the separation of these clusters.

The combination of these classifiers is robust to different changes in face poses, illumination, face expression and image quality and focus (sharpness). PCA (principle component analysis) is also known as the eigenface method. A summary of conventional techniques that utilize this method is found in Eigenfaces for Recognition, Journal of Cognitive Neuroscience, 3(1), 1991 to Turk et al. This method is sensitive to facial expression, small degrees of rotation and different focus conditions and/or illuminations. In the preferred embodiment, high frequency components from the image that are responsible for slight changes in face appearance are filtered. Features obtained from low pass filtered sub-bands from the wavelet decomposition are significantly more robust to facial expression, small degrees of rotation and different illuminations than conventional PCA.

In general, the steps involved in implementing the PCA/Wavelet technique include: (i) the extracted, normalized face region is transformed into gray scale; (ii) wavelet decomposition in applied using Daubechie wavelets; (iii) histogram equalization is performed on the grayscale LL sub-band representation; next, (iv) the mean LL sub-band is calculated and subtracted from all faces and (v) the 1st level LL sub-band is used for calculating the covariance matrix and the principal components (eigenvectors). The resulting eigenvectors (basis vector set) and the mean face are stored in a file after training so they can be used in determining the principal components for the feature vectors for detected face regions. Alternative embodiments may be discerned from the discussion in H. Lai, P. C. Yuen, and G. C. Feng, “Face recognition using holistic Fourier invariant features” Pattern Recognition, vol. 34, pp. 95-109, 2001.

In the 2D Discrete Cosine Transform classifier, the spectrum for the DCT transform of the face region can be further processed to obtain more robustness (see also, Application of the DCT Energy Histogram for Face Recognition, in Proceedings of the 2nd International Conference on Information Technology for Application (ICITA 2004) to Tjahyadi et al.

The steps involved in this technique are generally as follows: (i) the resized face IS transformed to an indexed image using a 256 color gif colormap; (ii) the 2D DCT transform is applied; (iii) the resulting spectrum is used for classification; (iv) for comparing similarity between DCT spectra the Euclidian distance was used. Examples of non-face based classifiers are based on color histogram, color moment, colour correlogram, banded colour correlogram, and wavelet texture analysis techniques. An implementaton of color histogram is described in “CBIR method based on color-spatial feature,” IEEE Region 10th Ann. Int. Conf 1999 (TENCON'99, Cheju, Korea, 1999). Use of the colour histogram is, however, typically restricted to classification based on the color information contained within one or more sub-regions of the image. Color moment may be used to avoid the quantization effects which are found when using the color histogram as a classifier (see also “Similarity of color images,” SPIE Proc. pp. 2420 (1995) to Stricker et al.). The first three moments (mean, standard deviation and skews) are extracted from the three color channels and therefore form a 9-dimensional feature vector.

The color auto-correlogram (see, U.S. Pat. No. 6,246,790 to Huang et al.) provides an image analysis technique that is based on a three-dimensional table indexed by color and distance between pixels which expresses how the spatial correlation of color changes with distance in a stored image. The color correlogram may be used to distinguish an image from other images in a database. It is effective in combining the color and texture features together in a single classifier (see also, “Image indexing using color correlograms,” In IEEE Coif. Computer Vision and Pattern Recognition, PP. 762 et seq (1997) to Huang et al.).

In certain embodiments, the color correlogram is implemented by transforming the image from ROB color space, and reducing the image colour map using dithering techniques based on minimum variance quantization. Variations and alternative embodiments may be discerned from Variance based color image quantization for frame buffer display, “Color Res. Applicat., vol. 15, no. 1, pp. 52-58, 1990 to by Wan et al. Reduced colour maps of 16, 64, 256 colors are achievable. For 16 colors the VOA colormap may be used and for 64 and 256 colors, a gif colormap may be used. A maximum distance set D=1; 3; 5; 7 may be used for computing auto-correlogram to build a N×D dimension feature vector where N is the number of colors and D is the maximum distance.

The color autocorrelogram and banded correlogram may be calculated using a fast algorithm (see, e.g., “Image Indexing Using Color Conelograms” from the Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97) to Huang et al.). Wavelet texture analysis techniques (see, e.g., “Texture analysis and classification with tree-structured wavelet transform,” IEEE Trans. Image Processing 2(4), 429 (1993) to Chang et al.) may also be advantageously used. In order to extract the wavelet based texture, the original image is decomposed into 10 de-correlated sub-bands through 3-level wavelet transform. In each sub-band, the standard deviation of the wavelet coefficients is extracted, resulting in a 10-dimensional feature vector.

Another embodiment is described in relation to FIG. 4. This takes the form of a set of software modules 1162 implemented on a desktop computer 1150. A second preferred embodiment provides an implementation within an embedded imaging appliance such as a digital camera.

In this embodiment, a program may be employed in a desktop computer environment and may either be run as a stand-alone program, or alternatively, may be integrated in existing applications or operating system (OS) system components to improve their functionality.

Image Analysis Module

An image analysis module 1156, such as that illustrated at FIG. 4, cycles through a set of images 1170-1 . . . 1180-2 and determines, extracts, normalizes and analyzes face regions and associated peripheral regions to determine feature vectors for a plurality of face and non-face classifiers. The module then records this extracted information in an image data set record. Components of this module are also used in both training and sorting/retrieval modes of the embodiment. The module is called from a higher level workflow and in its normal mode of usage is passed a set of images which, as illustrated at FIG. 7, are analyzed [2202]. The module loads/acquires the next image [2202] and detects any face regions in said image [2204]. If no face regions were found, then flags in the image data record for that image are updated to indicate that no face regions were found. If the current image is not the last image in the image set being analyzed [2208], upon image subsampling [2232], face and peripheral region extraction and region normalization [2207], the next image is loaded/acquired [2204]. If this was the last image, then the module will exit to a calling module. Where at least one face region is detected the module next extracts and normalizes each detected face region and, where possible, any associated peripheral regions.

Face region normalization techniques can range from a simple re-sizing of a face region to more sophisticated 2D rotational and affine transformation techniques and to highly sophisticated 3D face modeling methods.

Image Sorting/Retrieval Process

The workflow for an image sorting/retrieval process or module is illustrated at FIGS. 5 and 6A-6F and is initiated from an image selection or acquisition process (see US 2006/0140455) as the final process step [1140]. It is assumed that when the image sorting/retrieval module is activated [1140] it will also be provided with at least two input parameters providing access to (i) the image to be used for determining the search/sort/classification criteria, and (ii) the image collection data set against which the search is to be performed. If a data record is determined to be unavailable [1306] and undetermined for the search image, which proceeds to select persons and search criteria in the image [1308], then main image analysis module is next applied to it to generate the data record [1200]. The image is next displayed to a user who may be provided options to make certain selections of face regions to be used for searching and/or also of the classifiers to be used in the search [1308]. Alternatively, the search criteria may be predetermined or otherwise automated through a configuration file and step [1308] may thus be automatic. User interface aspects are described in detail at US 2006/0140455, incorporated by reference.

After a reference region comprising the face and/or peripheral regions to be used in the retrieval process is selected (or determined automatically) the main retrieval process is initiated [1310] either by user interaction or automatically in the case where search criteria are determined automatically from a configuration file. The main retrieval process is described in step [1312] and comprises three main sub-processes which are iteratively performed for each classifier to be used in the sorting/retrieval process:

  • (i) Distances are calculated in the current classifier space between the feature vector for the reference region and corresponding feature vector(s) for the face/peripheral regions for all images in the image collection to be searched [1312-1]. In the preferred embodiment, the Euclidean distance is used to calculate these distances which serve as a measure of similarity between the reference region and face/peripheral regions in the image collection.
  • (ii) The statistical mean and standard deviation of the distribution of these calculated distances is determined and stored temporarily [1312-2].
  • (iii) The determined distances between the reference region and the face/peripheral regions in the image collection are next normalized [1312-3] using the mean and standard deviation determined in step [1312-2].

These normalized data sets may now be combined in a decision fusion process [1314] which generates a ranked output list of images. These may then be displayed by a UI module [1316].

An additional perspective on the process steps [1312-1, 1312-2 and 1312-3] is given in US 2006/0140455. The classifier space [1500] for a classifier may be such as the Wavelet/PCA face recognition described at US 200610140455. The basis vector set, [λ1, λ2, . . . λn] may be used to determine feature vectors for this classifier. The average or mean face is calculated [1501] during the training phase and its vector position [1507] in classifier space [1500] is subtracted from the absolute position of all face regions. Thus, exemplary face regions [1504-1 a, 1504-2 a and 1504-3 a] have their positions [1504-1 b, 1504-2 b and 1504-3 b] in classifier space defined in vector terms relative to the mean face [1501].

After a particular face region [1504-2 a] is selected by the user [1308] the distances to all other face regions within a particular image collection are calculated. The face regions [1504-1 a] and [1504-3 a] are shown as illustrative examples. The associated distances (or non-normalized rankings) are given as [1504-1 c] and [1504-3 c].

An analogous case arises when the distances in classifier space are measured in absolute terms from the origin, rather than being measured relative to the position of an averaged, or mean face. For example, the color correlogram technique as used in certain embodiments is a classifier of this type which does not have the equivalent of a mean face.

The distances from the feature vector for the reference region [1504-2 a] and [1509-2 a] to the feature vectors for all other face regions may be calculated in a number of ways. In one embodiment, Euclidean distance is used, but other distance metrics may be advantageously employed for certain classifiers other than those described here.

Methods for Combining Classifier Similarity Measures Statistical Normalization Method

A technique is preferably used for normalizing and combining the multiple classifiers to reach a final similarity ranking. The process may involve a set of multiple classifiers, C1, C2 . . . CN and may be based on a statistical determination of the distribution of the distances of all patterns relevant to the current classifier (face or peripheral regions in our embodiment) from the selected reference region. For most classifiers, this statistical analysis typically yields a normal distribution with a mean value MCn and a variance VCn.

In-Camera Implementation

As imaging appliances continue to increase in computing power, memory and non-volatile storage, it will be evident to those skilled in the art of digital camera design that many advantages can be provided as an in-camera image sorting sub-system. An exemplary embodiment is illustrated in FIG. 7.

Following the main image acquisition process [2202] a copy of the acquired image is saved to the main image collection [2212] which will typically be stored on a removable compact-flash or multimedia data card [2214]. The acquired image may also be passed to an image subsampler [2232] which generates an optimized subsampled copy of the main image and stores it in a subsampled image collection [2216]. These subsampled Images may advantageously be employed in the analysis of the acquired image.

The acquired image (or a subsampled copy thereof) is also passed to a face detector module [2204] followed by a face and peripheral region extraction module [2206] and a region normalization module [2207]. The extracted, normalized regions are next passed to the main image analysis module [2208] which generates an image data record [1409] for the current image. The main image analysis module may also be called from the training module [2230] and the image sorting/retrieval module [2218].

A UI module [2220] facilitates the browsing & selection of images [2222], the selection of one or more face regions [2224] to use in the sorting/retrieval process [2218]. In addition classifiers may be selected and combined [2226] from the UI Module [2220].

Various combinations are possible where certain modules are implemented III a digital camera and others are implemented on a desktop computer.

Focus Condition Classifiers

A branched classifier chain may be used for simultaneous classification of faces and classification of an out of focus (or focused) condition. In certain embodiments, a classifier chain is constructed that, after an initial set of feature detectors that reject the large majority of objects within an image as non-faces, applies a set of, for example 3, 4, 5, 6, 7, 8 or 9, feature detectors. The feature detectors may tuned so that they accept faces that are sharply focused, slightly out of focus or significantly out of focus. Greater numbers of classifier cascades than three may be used, such as sharply focused, slightly out of focus, a little more out of focus, a little more out of focus, . . . , significantly out of focus, and two may be used, e.g., focused and out of focus. Other combinations are possible, and some may be excluded, e.g., after application of one classifier provides a determination that a face exists within the image or a sub-window of the image of a certain focus condition. When one of the classifier branches accepts the face, it can be said that the face and the focus condition of the face are detected. This detection can be used to process the image with greater attention to faces than non-faces, and/or to correct the out of focus condition, improving face detection and/or face recognition results.

Alternatively, the detected focus condition problems in one detection frame may be corrected in the next frame so the face detection algorithm has a better chance of finding the face. The focus condition detection comes essentially for free as the length of the classifier chain is no longer than in the previous design.

FIG. 8 illustrates a face focus normalization method in accordance with certain embodiments. A digital image is acquired at 602. One or more out of focus classifier sets are applied to the data at 604, beginning with one cascade at a time. The sets may be used to find faces and/or to determine an out of focus (or focused) condition within an already detected face image. Depending on the data retrieved in 604, a method according to different embodiments would next identify a face within the image at 606, or determine an out of focus (or focused) condition for a face at 608, or both 606 and 608 contemporaneously or one after the other in either order. For example, a face may be found and then a focus condition found for the face, or a focus condition for an object may be found followed by a determination whether the object is a face.

It may also be determined that no single focus condition exists at 618. If a face is determined to exist at 606 or the data is not (yet) rejected as not including a face at 606, then at 616, a set of feature detector programs may be applied to reject non-face data from being identified as a face (or accept face data as being identified as a face).

If an out of focus condition is determined at 608, then at 610 the out of focus condition may be corrected for the image and/or for another image in a series of images. For example, the original image may be a preview image, and a full resolution image may be corrected either during acquisition (e.g., by adjusting a flash condition or by providing suggestions to the camera-user to move before taking the picture, etc.) or after acquisition either in-camera before or after storing a permanent image, or on an external device later-on. Corrected face image data may be generated at 612 appearing to be in focus or more sharply focused, and the corrected face image may be stored, transmitted, captured, combined with other face data, applied to a face recognition program, edited and/or displayed at 614.

If it is determined at 618 that no single focus condition applies, then the face data may be rejected or not rejected as a face at 620. If the face data is not rejected as a face at 620, then at 622, combinations of two or more classifier sets may be applied to the data.

FIGS. 9A-9B illustrate face detection methods in accordance with certain further embodiments. A digital image is acquired at 702. A sub-window is extracted from the image at 704. Two or more shortened face detected classifier cascades are applied to the sub-window at 706. These cascades are trained to be selectively sensitive to a characteristic of a face region.

At 708, a probability is determined that a face with a certain form of the characteristic is present within the sub-window. The characteristic may include a focus condition, or an illumination condition, or a pose or direction of the face relative to the camera, or another characteristic such as resolution, size, location, motion, blurriness, facial expression, blink condition, red, gold or white eye condition, occlusion condition or an appearance, e.g., of a face within a collection having multiple appearances such as shaven or unshaven, a hair style, or wearing certain jewelry, among other features. An extended face detection classifier cascade is applied at 710 for sensitivity to the form of the characteristic. A final determination is provided at 712 whether a face exists within the sub-window. If so, then optionally at 714, an out of focus condition or an uneven illumination condition, etc., for the face image may be corrected within the image and/or within a different image in a series of images. In addition, the process may return to 704 to extract a further sub-window, if any, from the image.

At 742, a digital image may be acquired, and a sub-window extracted therefrom at 744. Two or more shortened face detection classifier cascades may be applied at 746 that are trained to be selectively sensitive to out of focus conditions. A probability is determined that a face having a certain out of focus condition is present within the sub-window at 748. An extended face detection classifier cascade is applied at 750 that is trained for sensitivity to the certain form of out of focus condition, e.g., slightly out of focus, or significantly out of focus, and a separate classifier cascade may be trained for sharply focused conditions. A final determination is provided at 752 whether a face exists within the image sub-window. A further sub-window, if any, may then be extracted by returning the process to 744 and/or an out of focus condition of the face may be corrected within the image and/or a different image 5 in a series of images at 754.

The “Chain Branching” idea for Luminance is fairly straight-forward to implement and to test since it requires no alterations to the training algorithm. The variations/“mutations” of a face are considered as distinct objects and each one receives a distinct detector/cascade of classifiers. The detectors are all the same, linear chains of full extent.

In detection the straightforward approach would be to exhaustively run all the detectors and see which ones accept the window and then choose the best score. This means that the correct detector is selected at the end. However, this is not what we tested, being very time consuming.
Chain1=cls11+cls12+ . . . +cis1M
ChainN=cls1+clsN2+clsN2+ . . . +clsNM

The detectors may be run in series or in parallel or some combination thereof, and an at least partial confidence may be accumulated, viz:
Partial1=cls11+cls12+ . . . +cls1P

PartialN=clsN1+clsN2+ . . . +clsNP, with P<M

The best detector is chosen at this point with maximum Partial confidence value. Only that detector continues execution with:
ChainMax=PartialMax+elsMax(P+1)+clsMax(P+2)+ . . . +clsMaxM
So an exemplary workflow is:
Partial1

    • . . . \
      PartialMax . . . (choose Max)→continue with the rest of Max
      . . . /
      PartialN . . . .

This approach may be applied for focus condition, face pose variation and/or an illumination condition or other characteristic. In the illumination case, one may use any combination of (i) frontally illuminated faces; (ii) faces illuminated from the top; (iii) faces illuminated from bottom; (iv) faces illuminated form the left and (v) faces illuminated from right. In the focus condition case, one may use for example (i) sharply focused faces, (ii) slightly out of focus faces, and (iii) significantly out of focus faces. The system may be trained to detect depths of various objects in the scene, and so may also be able to detect whether the out of focus condition is due to a focus plane being in front of or in back of the face or other object of interest. For example, if a background object is in focus, then the face or other object of interest is in front of the focal plane, whereas if an object known to be in front of the face or other object of interest is in focus, then the focal plane is in front of the face or other object of interest. In this case, there may be front and back slightly out of focus classifiers and/or front and back significantly out of focus classifiers. In the illumination case, because of the symmetric nature of faces, one could use just one of (iv) and (v) as there is symmetry between the classifiers obtained. The training images used for determining these classifier sets may be generated using an AAM model with one parameter trained to correspond to the level of top/bottom illumination and a second parameter trained to correspond to left/right illumination.

FIGS. 10A-10B illustrate an exemplary detailed workflow. At 802, a sub-window is tested with an in focus classifier set (e.g., using 3-5 classifiers). If a cumulative probability is determined at 804 to be above a first threshold, then the face is determined to be in focus at 806, and the process is continued with this full classifier chain. If the cumulative probability is determined to be below a second threshold (which is even lower than the first threshold), then at 812, it is determined that the sub-window does not contain a face, and the process is returned via 864 to 802. If the cumulative probability is determined at 808 to be above a second threshold, yet below the first threshold of 804, then the sub-window is deemed to still likely be a face at 810, but not an in focus one. Thus, a next out of focus specific partial classifier set is applied at 814.

The classifier can be applied in any order, although at step 814, the sub-window is tested with a slightly out of focus classifier set (e.g., using 3-5 classifiers). If the cumulative probability is determined to be above a first threshold at 816, then face is determined to be slightly out of focus at 818, and the process is continued with this full classifier chain. If the cumulative probability is deemed to be between the first threshold and a lower second threshold at 820, then at 822 the sub-window is determined to still likely contain a face, but not a slightly out of focus one, and so the process moves to 826 for applying a next out of focus specific partial classifier set. If the cumulative probability is deemed to be less than the second threshold, then at 824 the sub-window is determined to not contain a face, and the process moves back through 864 to the next sub-window and 802.

At 826, a test of the sub-window is performed with a illuminated significantly out of focus partial classifier set (e.g., using 3-5 classifiers). If the cumulative probability is determined at 828 to be above a first threshold, then the face is determined to be significantly out of focus and at 830 the process is continued with this full classifier chain. If cumulative probability is below the first threshold, but above a lower second threshold at 832, then the sub-window is determined to still likely contain a face at 834, although not a significantly out of focus one, and so the process moves to 838 and FIG. 10B to apply a next out of focus specific partial classifier set, if any. If at 832, the cumulative probability is deemed above a second threshold lower than the first indicated at 828, then the sub-window is still deemed to be likely to contain a face at 858, although neither a sharply focused one, nor a slightly out of focus one, nor a significantly out of focus one, and so now pairs of specific partial classifier sets are applied at 862. This is because at this point, the window has not passed any of the focus condition specific classifiers at their first threshold but neither has it been rejected as a face. Thus, a likely scenario is that the sub-window contains a face that is represented by a combination of focus condition types. So, the two highest probability thresholds may be first applied to determine whether it is in between sharply focused and slightly out of focus, or between slightly and significantly out of focus or perhaps more greatly focused than significantly out of focus or perhaps better focused than sharply focused, then multiple full classifier sets are applied to determine if it survives as a face region. If at 832 the cumulative probability was deemed to be below the second threshold, then at 860, the sub-window is deemed not to contain a face and the processes moves through 864 to 802 to the next image sub-window.

In an alternative embodiment split classifier chains can be used to perform rapid focusing onto individual face regions within an image.

In such an embodiment the classifier chain is split into a first component which serves as a generic face detector, using the higher-order, or simpler classifiers. These classifiers cannot match to finer features within the face region and as such they provide a good match to both focused and unfocused faces. In one preferred embodiment these classifiers are Haar classifiers.

Once a face region is confirmed—this may be either by application of this first component of a face detector, or by accepting data from a face tracking module or by combining information from both detector and tracker—then a set of additional classifier components are selectively applied to the confirmed face region.

In a preferred embodiment at least three distinct additional classifier components are provided and employed with a specialized MEMS lens system which enables rapid focusing of the image acquisition subsystem. All of these additional classifier components comprise of a chain of more complex classifiers.

In one preferred embodiment these classifiers are census classifiers. In alternative embodiments these may be combined with other complex classifiers.

The first additional classifier chain is selectively trained to match with sharply focused face regions to be neutral to face regions which are slightly out of focus and to actively reject faces which are significantly out of focus.

A second additional classifier chain is selectively trained to optimally match with face regions which are slightly out of focus, being neutral to faces which are sharply in focus and to reject face regions which are significantly out of focus.

A third additional classifier chain is selectively trained to optimally match to faces which are significantly out of focus and to actively reject sharply focused faces, being neutral to slightly unfocused faces.

In a preferred embodiment each of these components provides a score between 1 and 0 for the confirmed face region, said score being normalized against a large dataset to provide a substantially accurate indication of how well the confirmed face region matches to each of the criteria.

The operation of the invention is then as follows: (i) a confirmed face region has each of the applied additional components applied in turn to determine the degree of focus of the face; (ii) the MEMS lens is quickly adjusted and a second application of the three components is made; if the face was substantially unfocused and remains substantially unfocused then the movement of the lens was in the wrong direction and the lens is moved in the opposite direction, or the increment of the movement may be reduced; (iii) once a better match is achieved by the slightly unfocused component the increment of lens movement is further reduced and adjustments made until the score of the sharply focused component is greater than that of the slightly unfocused component then the face region is determined to be in-focus.

Because the MEMS lens assembly can rapidly change its focus it is possible to repeat this sequence on multiple faces within a single scene and to determine the optimal focus for each face region. In certain embodiments more than one image of the same scene may be captured and merged/blended into a single composite image to provide multiple, optimally focused faces. In alternative embodiments a global focus may be determined from the focus settings of all face regions within the image.

In other embodiments specific classifier chains may be employed for facial features. For example one set of classifiers may be trained to enable optimal focus on a person's eyes; a second set may be trained to enable optimal focus on their chin or mouth region; a third set may be trained for their hairline/forehead region; a fourth set may be trained for their ears and sides of the face. In combination such classifiers enable more accurate focus on a single face for portrait photography, thus enabling individual regions of the face to be kept in sharp focus, or slightly or substantially out of focus depending on the requirements of the portrait photographer.

The outline focus process described above may employ more complex and sophisticated algorithms. In addition more than the three level of focus described above may be employed although the three levels outlined above should be sufficient for most embodiments of this invention.

In certain embodiments it would also be possible to eliminate the sharply focused component and by rapidly moving the focal point of the lens between slightly unfocused positions in-front of and behind the face to determine the optimal focal distance.

Referring now to FIGS. 11A-11E, a system in accordance certain embodiments includes an embedded component in an environment where face (or other object) detection (and/or tracking) information is provided. Face detection particularly by training face classifiers have been widely researched and developed by the assignee of the present application, e.g., as described at U.S. Pat. Nos. 7,362,368, 7,616,233, 7,315,630, 7,269,292, 7,471,846, 7,574,016, 7,440,593, 7,317,815, 7,551,755, 7,558,408, 7,587,068, 7,555,148, 7,564,994, 7,565,030, 7,715,597, 7,606,417, 7,692,696, 7,680,342, 7,792,335, 7,551,754, 7,315,631, 7,469,071, 7,403,643, 7,460,695, 7,630,527, 7,469,055, 7,460,694, 7,515,740, 7,466,866, 7,693,311, 7,702,136, 7,620,218, 7,634,109, 7,684,630, 7,796,816 and 7,796,822, and U.S. published patent applications nos.

US 2006-0204034, US 2007-0201725, US 2007-0110305, US 2009-0273685, US 2008-0175481, US 2007-0160307, US 2008-0292193, US 2007-0269108, US 2008-0013798, US 2008-0013799, US 2009-0080713, US 2009-0196466, US 2008-0143854, US 2008-0220750, US 2008-0219517, US 2008-0205712, US 2009-0185753, US 2008-0266419, US 2009-0263022, US 2009-0244296, US 2009-0003708, US 2008-0316328, US 2008-0267461, US 2010-0054549, US 2010-0054533, US 2009-0179998, US 2009-0052750, US 2009-0052749, US 2009-0087042, US 2009-0040342, US 2009-0002514, US 2009-0003661, US 2009-0208056, US 2009-0190803, US 2009-0245693, US 2009-0303342, US 2009-0238419, US 2009-0238410, US 2010-0014721, US 2010-0066822, US 2010-0039525, US 2010-0165150, US 2010-0060727, US 2010-0141787, US 2010-0141786, US 2010-0220899, US 2010-0092039, US 2010-0188530, US 2010-0188525, US 2010-0182458, US 2010-0165140 and US 2010-0202707, which are all incorporated by reference.

The system in certain embodiments contains MEMS (or similar) technology which allows changing the point of focus at a very fast speed. The MEMs technology may be as set forth at any of U.S. Pat. Nos. 7,769,281, 7,747,155, 7,729,603, 7,729,601, 7,702,226, 7,697,834, 7,697,831, 7,697,829, 7,693,408, 7,663,817, 7,663,289, 7,660,056, 7,646,969, 7,640,803, 7,583,006, 7,565,070, 7,560,679, 7,555,210, 7,545,591, 7,515,362, 7,495,852, 7,477,842, 7,477,400, 7,403,344, 7,359,131, 7,359,130, 7,345,827, 7,266,272, 7,113,688 and/or 6,934,087, which are hereby incorporated by reference.

The system has the capability to capture and/or store a limited sequence of images captured in a short amount of time. The method combines mature solutions, such as face detection and tracking, and MEMS technologies, as described for example in the reference cited above and as may otherwise be understood by those skilled in the art. Image registration and enhancement may be provided in various embodiments to order to obtain a well-focused system. The method may be used for face and non-face objects, such as pets, cars, houses, personal property, natural scenery, avatars and other gaming objects, and many other identifiable objects, as long as automatic object detection and/or tracking is present in the system. A system in accordance with certain embodiments is capable of providing photos and digital images taken with any f-number where, together with a user-selected or automatically selected objects of interest such as faces, regardless of their distance to the camera, are kept in focus. The method involves MEMS capability to move the focus point quickly, rapidly, immediately or almost instantaneously, after an image has been captured. Object detection and tracking capability determines the position and the size of the objects in real time, while preview information is created. Image registration capability is provided in order to align the sequence of images to enable combining them on the spot, or later on after the images have been captured. Image blending and/or morphing capability may also be included, e.g., such that sharp faces from one image can be transposed in another image with a different point of focus.

The solution proposed solves the problem not only for deep focus but also for lenses with shallow focus for photos with groups of people.

With regard to selective focus, the viewer may be allowed to manually toggle between different focal points. A digital image may be acquired as usual, while a focal point is selected on an embedded device or computer afterward. With regard to multi-face focus, a single image may be created such that two or more or all of the faces (or other objects of interest) are sharply focused. The multi-face focus image may have a higher frequency than each of the individual images from the sequence, thereby providing best possible input data for security applications such as face recognition. These multi-face embodiments provide advanced, qualitative solutions which are suitable to real-time embedded implementation.

Referring to FIGS. 11A-11E, in certain embodiments, the system will capture a sequence of images, starting with an initial one, and following with a number of images equal with the number of objects (faces) of interest from the field of view. Initial reference, preview or postview image capturing may include an initial image being captured with default settings of a camera-enabled device. A focus on a main element (e.g. background, a certain object, a certain face) may be performed.

An object-of-interest image sequence capturing may be performed. For the following images the focus point is determined from the characteristics of the list of faces (or objects) detected. An example is illustrated. Multiple images captured at different focus positions may be saved altogether, providing the user or programmed device the possibility to choose or merge between them at a later stage, or further processed in the device, to provide a single, multi-focused image.

In an advanced use case, image alignment may involve optical, digital and/or mechanical stabilization to enable combining them together. Features described at any of U.S. Pat. Nos. 7,660,478, 7,639,889, 7,636,486, 7,639,888, 7,697,778, and/or U.S. Pat. Nos. 7,773,118, and 7,676,108 and/or U.S. patent application Ser. Nos. 121330,719, 111856,721, 12/485,316, 12/901,577, 12/820,002, 12/820,034, 12/820,086, 12/336,416, U.S. Ser. No. 111753,098, and/or U.S. Ser. No. 121137,113, which are hereby incorporated by reference, may be used in alternative embodiments.

Image blending and morphing is also possible such as illustrated at FIG. 11E for multi-focus capability. A number of images are blended together in such a way that areas of interest in focus take priority over the areas out of focus. In cases where object/faces of interest have moved from one image to another, morphing may be used in order to make the combination realistic and artifact free. While an exemplary drawings and specific embodiments of the present invention have been described and illustrated, it is to be understood that that the scope of the present invention is not to be limited to the particular embodiments discussed. Thus, the embodiments shall be regarded as illustrative rather than restrictive, and it should be understood that variations may be made in those embodiments by workers skilled in the arts without departing from the scope of the present invention.

In addition, in methods that may be performed according to preferred embodiments herein and that may have been described above, the operations have been described in selected typographical sequences. However, the sequences have been selected and so ordered for typographical convenience and are not intended to imply any particular order for performing the operations, except for those where a particular order may be expressly set forth or where those of ordinary skill in the art may deem a particular order to be necessary.

In addition, all references cited above and below herein, as well as the background, invention summary, abstract and brief description of the drawings, are all incorporated by reference into the detailed description of the preferred embodiments as disclosing alternative embodiments.

Claims (20)

What is claimed is:
1. A method comprising:
acquiring data of a digital image which depicts one or more objects;
determining a plurality of in-focus probabilities by applying a plurality of in-focus classifier sets to the data;
wherein each in-focus probability indicates a likelihood that any of the one or more objects is depicted in focus in the digital image;
determining an in-focus cumulative probability based on the plurality of in-focus probabilities;
determining whether the in-focus cumulative probability is below an in-focus threshold;
in response to determining that the in-focus cumulative probability is below the in-focus threshold, performing the steps of:
determining a plurality of slightly-out-of-focus probabilities by applying a plurality of slightly-out-of-focus classifier sets to the data;
wherein each slightly-out-of-focus probability indicates a likelihood that any of the one or more objects is depicted slightly-out-of focus in the digital image;
determining slightly-out-of-focus cumulative probability based on the plurality of slightly-out-of-focus probabilities; and
in response to determining that the slightly-out-of-focus cumulative probability is equal to, or above, a slightly-out-of-focus threshold, determining a particular object, of the one or more objects, that is depicted in the digital image slightly-out-of-focus; and
wherein the method is performed by one or more computing devices.
2. The method of claim 1,
wherein an in-focus classifier set of the plurality of in-focus classifier sets comprises an in-focus-program configured to determine an in-focus probability that any of the one or more objects is depicted in-focus in the digital image;
wherein a slightly-out-of-focus classifier set of the one or more slightly-out-of-focus classifier sets comprises a slightly-out-of-focus-program configured to determine a slightly-out-of-focus probability that any of the one or more objects is depicted slightly-out-of-focus in the digital image; and
wherein the slightly-out-of-focus cumulative probability is computed from one or more slightly-out-of-focus probabilities determined by the one or more slightly-out-of-focus classifier sets.
3. The method of claim 2, further comprising:
determining whether the slightly-out-of-focus cumulative probability is below a slightly-out-of-focus threshold; and
in response to determining that the slightly-out-of-focus cumulative probability is below, the slightly-out-of-focus threshold, applying one or more significantly-out-of-focus classifier sets to determine a significantly-out-of-focus cumulative probability.
4. The method of claim 3,
wherein a significantly-out-of-focus classifier set of the one or more significantly-out-of-focus classifier sets comprises a significantly-out-of-focus-program configured to determine a significantly-out-of-focus probability that any of the one or more objects is depicted significantly-out-of-focus in the digital image; and
wherein the significantly-out-of-focus cumulative probability is computed from one or more significantly-out-of-focus probabilities determined by the one or more significantly-out-of-focus classifier sets.
5. The method of claim 4, further comprising:
determining whether the significantly-out-of-focus cumulative probability is equal to, or above, a significantly-out-of-focus threshold; and
in response to determining that the significantly-out-of-focus cumulative probability is equal to, or above, a significantly-out-of-focus threshold, determining the particular object, of the one or more objects, that is depicted in the digital image significantly-out-of-focus.
6. The method of claim 5, further comprising:
determining whether the significantly-out-of-focus cumulative probability is below the significantly-out-of-focus threshold; and
in response to determining that the significantly-out-of-focus cumulative probability is below the significantly-out-of-focus threshold, determining that none of the one or more objects is depicted in the digital image.
7. The method of claim 6, further comprising:
in response to determining that the in-focus cumulative probability is above, or equal to, the in-focus threshold, determining the particular object, of the one or more objects, that is depicted in-focus face in the digital image.
8. A non-transitory computer-readable storage medium, storing one or more computer instructions which, when executed by one or more processors, cause the one or more processors to perform:
acquiring data of a digital image which depicts one or more objects;
determining a plurality of in-focus probabilities by applying a plurality of in-focus classifier sets to the data;
wherein each in-focus probability indicates a likelihood that any of the one or more objects is depicted in focus in the digital image;
determining an in-focus cumulative probability based on the plurality of in-focus probabilities;
determining whether the in-focus cumulative probability is below an in-focus threshold;
in response to determining that the in-focus cumulative probability is below the in-focus threshold, performing the steps of:
determining a plurality of slightly-out-of-focus probabilities by applying a plurality of slightly-out-of-focus classifier sets to the data;
wherein each slightly-out-of-focus probability indicates a likelihood that any of the one or more objects is depicted slightly-out-of focus in the digital image;
determining slightly-out-of-focus cumulative probability based on the plurality of slightly-out-of-focus probabilities; and
in response to determining that the slightly-out-of-focus cumulative probability is equal to, or above, a slightly-out-of-focus threshold, determining a particular object, of the one or more objects, that is depicted in the digital image slightly-out-of-focus.
9. The non-transitory computer-readable storage medium of claim 8,
wherein an in-focus classifier set of the plurality of in-focus classifier sets comprises an in-focus-program configured to determine an in-focus probability that any of the one or more objects is depicted in-focus in the digital image;
wherein a slightly-out-of-focus classifier set of the one or more slightly-out-of-focus classifier sets comprises a slightly-out-of-focus-program configured to determine a slightly-out-of-focus probability that any of the one or more objects is depicted slightly-out-of-focus in the digital image; and
wherein the slightly-out-of-focus cumulative probability is computed from one or more slightly-out-of-focus probabilities determined by the one or more slightly-out-of-focus classifier sets.
10. The non-transitory computer-readable storage medium of claim 9, storing additional instructions which, when executed, cause the one or more processors to perform:
determining whether the slightly-out-of-focus cumulative probability is below a slightly-out-of-focus threshold; and
in response to determining that the slightly-out-of-focus cumulative probability is below, the slightly-out-of-focus threshold, applying one or more significantly-out-of-focus classifier sets to determine a significantly-out-of-focus cumulative probability.
11. The non-transitory computer-readable storage medium of claim 10,
wherein a significantly-out-of-focus classifier set of the one or more significantly-out-of-focus classifier sets comprises a significantly-out-of-focus-program configured to determine a significantly-out-of-focus probability that any of the one or more objects is depicted significantly-out-of-focus in the digital image; and
wherein the significantly-out-of-focus cumulative probability is computed from one or more significantly-out-of-focus probabilities determined by the one or more significantly-out-of-focus classifier sets.
12. The non-transitory computer-readable storage medium of claim 11, storing additional instructions which, when executed, cause the one or more processors to perform:
determining whether the significantly-out-of-focus cumulative probability is equal to, or above, a significantly-out-of-focus threshold; and
in response to determining that the significantly-out-of-focus cumulative probability is equal to, or above, a significantly-out-of-focus threshold, determining the particular object, of the one or more objects, that is depicted in the digital image significantly-out-of-focus.
13. The non-transitory computer-readable storage medium of claim 12, storing additional instructions which, when executed, cause the one or more processors to perform:
determining whether the significantly-out-of-focus cumulative probability is below the significantly-out-of-focus threshold; and
in response to determining that the significantly-out-of-focus cumulative probability is below the significantly-out-of-focus threshold, determining that none of the one or more objects is depicted in the digital image.
14. The non-transitory computer-readable storage medium of claim 13, storing additional instructions which, when executed, cause the one or more processors to perform:
in response to determining that the in-focus cumulative probability is above, or equal to, the in-focus threshold, determining the particular object, of the one or more objects, that is depicted in-focus face in the digital image.
15. A device, comprising:
an image retrieval unit coupled to one or more memory units and configured to acquire data of a digital image which depicts one or more objects; and
an image analysis unit configured to:
determine a plurality of in-focus probabilities by applying a plurality of in-focus classifier sets to the data;
wherein each in-focus probability indicates a likelihood that any of the one or more objects is depicted in focus in the digital image;
determine an in-focus cumulative probability based on the plurality of in-focus probabilities;
determine whether the in-focus cumulative probability is below an in-focus threshold;
in response to determining that the in-focus cumulative probability is below the in-focus threshold, performing the steps of:
determine a plurality of slightly-out-of-focus probabilities by applying a plurality of slightly-out-of-focus classifier sets to the data;
wherein each slightly-out-of-focus probability indicates a likelihood that any of the one or more objects is depicted slightly-out-of focus in the digital image;
determine slightly-out-of-focus cumulative probability based on the plurality of slightly-out-of-focus probabilities; and
in response to determining that the slightly-out-of-focus cumulative probability is equal to, or above, a slightly-out-of-focus threshold, determine a particular object, of the one or more objects, that is depicted in the digital image slightly-out-of-focus.
16. The device of claim 15,
wherein an in-focus classifier set of the plurality of in-focus classifier sets comprises an in-focus-program configured to determine an in-focus probability that any of the one or more objects is depicted in-focus in the digital image;
wherein a slightly-out-of-focus classifier set of the one or more slightly-out-of-focus classifier sets comprises a slightly-out-of-focus-program configured to determine a slightly-out-of-focus probability that any of the one or more objects is depicted slightly-out-of-focus in the digital image; and
wherein the slightly-out-of-focus cumulative probability is computed from one or more slightly-out-of-focus probabilities determined by the one or more slightly-out-of-focus classifier sets.
17. The device of claim 16, wherein the image analysis unit is further configured to:
determine whether the slightly-out-of-focus cumulative probability is below a slightly-out-of-focus threshold; and
in response to determining that the slightly-out-of-focus cumulative probability is below, the slightly-out-of-focus threshold, apply one or more significantly-out-of-focus classifier sets to determine a significantly-out-of-focus cumulative probability.
18. The device of claim 17,
wherein a significantly-out-of-focus classifier set of the one or more significantly-out-of-focus classifier sets comprises a significantly-out-of-focus-program configured to determine a significantly-out-of-focus probability that any of the one or more objects is depicted significantly-out-of-focus in the digital image; and
wherein the significantly-out-of-focus cumulative probability is computed from one or snore significantly-out-of-focus probabilities determined by the one or more significantly-out-of-focus classifier sets.
19. The device of claim 18, wherein the image analysis unit is further configured to:
determine whether the significantly-out-of-focus cumulative probability is equal to, or above, a significantly-out-of-focus threshold; and
in response to determining that the significantly-out-of-focus cumulative probability is equal to, or above, a significantly-out-of-focus threshold, determine the particular object, of the one or more objects, that is depicted in the digital image significantly-out-of-focus.
20. The device of claim 19, wherein the image analysis unit is further configured to:
determine whether the significantly-out-of-focus cumulative probability is below the significantly-out-of-focus threshold; and
in response to determining that the significantly-out-of-focus cumulative probability is below the significantly-out-of-focus threshold, determine that none of the one or more objects is depicted in the digital image.
US14174821 2010-11-11 2014-02-06 Rapid auto-focus using classifier chains, MEMS and multiple object focusing Active US8797448B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US12944701 US8648959B2 (en) 2010-11-11 2010-11-11 Rapid auto-focus using classifier chains, MEMS and/or multiple object focusing
US14174821 US8797448B2 (en) 2010-11-11 2014-02-06 Rapid auto-focus using classifier chains, MEMS and multiple object focusing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US14174821 US8797448B2 (en) 2010-11-11 2014-02-06 Rapid auto-focus using classifier chains, MEMS and multiple object focusing

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US12944701 Continuation US8648959B2 (en) 2010-11-11 2010-11-11 Rapid auto-focus using classifier chains, MEMS and/or multiple object focusing

Publications (2)

Publication Number Publication Date
US20140193088A1 true US20140193088A1 (en) 2014-07-10
US8797448B2 true US8797448B2 (en) 2014-08-05

Family

ID=45065869

Family Applications (2)

Application Number Title Priority Date Filing Date
US12944701 Active 2031-08-30 US8648959B2 (en) 2010-11-11 2010-11-11 Rapid auto-focus using classifier chains, MEMS and/or multiple object focusing
US14174821 Active US8797448B2 (en) 2010-11-11 2014-02-06 Rapid auto-focus using classifier chains, MEMS and multiple object focusing

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US12944701 Active 2031-08-30 US8648959B2 (en) 2010-11-11 2010-11-11 Rapid auto-focus using classifier chains, MEMS and/or multiple object focusing

Country Status (6)

Country Link
US (2) US8648959B2 (en)
EP (3) EP3236391A1 (en)
JP (2) JP5530568B2 (en)
KR (1) KR101870853B1 (en)
CN (1) CN103052960B (en)
WO (1) WO2012062893A3 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130044923A1 (en) * 2008-12-05 2013-02-21 DigitalOptics Corporation Europe Limited Face Recognition Using Face Tracker Classifier Data
US20140139721A1 (en) * 2012-11-12 2014-05-22 Samsung Electronics Co., Ltd. Method and apparatus for shooting and storing multi-focused image in electronic device
US20140226858A1 (en) * 2013-02-14 2014-08-14 Samsung Electronics Co., Ltd. Method of tracking object using camera and camera system for object tracking

Families Citing this family (44)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8970770B2 (en) 2010-09-28 2015-03-03 Fotonation Limited Continuous autofocus based on face detection and tracking
US8995715B2 (en) * 2010-10-26 2015-03-31 Fotonation Limited Face or other object detection including template matching
US8648959B2 (en) 2010-11-11 2014-02-11 DigitalOptics Corporation Europe Limited Rapid auto-focus using classifier chains, MEMS and/or multiple object focusing
US8659697B2 (en) 2010-11-11 2014-02-25 DigitalOptics Corporation Europe Limited Rapid auto-focus using classifier chains, MEMS and/or multiple object focusing
US8308379B2 (en) 2010-12-01 2012-11-13 Digitaloptics Corporation Three-pole tilt control system for camera module
US8508652B2 (en) 2011-02-03 2013-08-13 DigitalOptics Corporation Europe Limited Autofocus method
US20120328160A1 (en) * 2011-06-27 2012-12-27 Office of Research Cooperation Foundation of Yeungnam University Method for detecting and recognizing objects of an image using haar-like features
US9501834B2 (en) * 2011-08-18 2016-11-22 Qualcomm Technologies, Inc. Image capture for later refocusing or focus-manipulation
US9294667B2 (en) 2012-03-10 2016-03-22 Digitaloptics Corporation MEMS auto focus miniature camera module with fixed and movable lens groups
US9001268B2 (en) 2012-08-10 2015-04-07 Nan Chang O-Film Optoelectronics Technology Ltd Auto-focus camera module with flexible printed circuit extension
US9007520B2 (en) 2012-08-10 2015-04-14 Nanchang O-Film Optoelectronics Technology Ltd Camera module with EMI shield
US9242602B2 (en) 2012-08-27 2016-01-26 Fotonation Limited Rearview imaging systems for vehicle
US9918017B2 (en) 2012-09-04 2018-03-13 Duelight Llc Image sensor apparatus and method for obtaining multiple exposures with zero interframe time
JP2014081420A (en) * 2012-10-15 2014-05-08 Olympus Imaging Corp Tracking device and method thereof
US9807322B2 (en) 2013-03-15 2017-10-31 Duelight Llc Systems and methods for a digital image sensor
US9350916B2 (en) 2013-05-28 2016-05-24 Apple Inc. Interleaving image processing and image capture operations
US9491360B2 (en) * 2013-06-06 2016-11-08 Apple Inc. Reference frame selection for still image stabilization
US9384552B2 (en) 2013-06-06 2016-07-05 Apple Inc. Image registration methods for still image stabilization
US9262684B2 (en) 2013-06-06 2016-02-16 Apple Inc. Methods of image fusion for image stabilization
CN104469127A (en) * 2013-09-22 2015-03-25 中兴通讯股份有限公司 Photographing method and photographing device
JP6184276B2 (en) * 2013-09-27 2017-08-23 オリンパス株式会社 Focusing device
US20150103190A1 (en) * 2013-10-16 2015-04-16 DigitalOptics Corporation Europe Limited Image acquisition method and apparatus with mems optical image stabilization (ois)
US9832359B2 (en) * 2013-10-16 2017-11-28 Novatek Microelectronics Corp. Focusing method
CN104601873A (en) * 2013-10-30 2015-05-06 联咏科技股份有限公司 Focusing method
US20150116552A1 (en) * 2013-10-30 2015-04-30 Jan kåhre Digital camera system and a method for controlling the digital camera system
US9091843B1 (en) 2014-03-16 2015-07-28 Hyperion Development, LLC Optical assembly for a wide field of view point action camera with low track length to focal length ratio
US9494772B1 (en) 2014-03-16 2016-11-15 Hyperion Development, LLC Optical assembly for a wide field of view point action camera with low field curvature
US9316808B1 (en) 2014-03-16 2016-04-19 Hyperion Development, LLC Optical assembly for a wide field of view point action camera with a low sag aspheric lens element
US9726859B1 (en) 2014-03-16 2017-08-08 Navitar Industries, Llc Optical assembly for a wide field of view camera with low TV distortion
US9995910B1 (en) 2014-03-16 2018-06-12 Navitar Industries, Llc Optical assembly for a compact wide field of view digital camera with high MTF
US9316820B1 (en) 2014-03-16 2016-04-19 Hyperion Development, LLC Optical assembly for a wide field of view point action camera with low astigmatism
US20150278631A1 (en) * 2014-03-28 2015-10-01 International Business Machines Corporation Filtering methods for visual object detection
CN104123570B (en) * 2014-07-22 2018-06-05 西安交通大学 Based on a combination of weak classifiers share manpower and classifier training and detection methods
CN104184942A (en) * 2014-07-28 2014-12-03 联想(北京)有限公司 Information processing method and electronic equipment
CN104200237A (en) * 2014-08-22 2014-12-10 浙江生辉照明有限公司 High speed automatic multi-target tracking method based on coring relevant filtering
US9589351B2 (en) * 2014-09-10 2017-03-07 VISAGE The Global Pet Recognition Company Inc. System and method for pet face detection
CN105488371A (en) * 2014-09-19 2016-04-13 中兴通讯股份有限公司 Face recognition method and device
US9589175B1 (en) 2014-09-30 2017-03-07 Amazon Technologies, Inc. Analyzing integral images with respect to Haar features
JP2016081249A (en) * 2014-10-15 2016-05-16 株式会社ソニー・コンピュータエンタテインメント Information processing device and information processing method
US9531961B2 (en) 2015-05-01 2016-12-27 Duelight Llc Systems and methods for generating a digital image using separate color and intensity data
CN105072339A (en) * 2015-08-03 2015-11-18 广东欧珀移动通信有限公司 Method, mobile terminal and system for regulating and controlling focusing of camera rapidly through selfie stick
US9819849B1 (en) 2016-07-01 2017-11-14 Duelight Llc Systems and methods for capturing digital images
CN106303221A (en) * 2016-07-29 2017-01-04 广东欧珀移动通信有限公司 Control method and device, and mobile terminal
US20180063409A1 (en) * 2016-09-01 2018-03-01 Duelight Llc Systems and methods for adjusting focus based on focus target information

Citations (118)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4448510A (en) 1981-10-23 1984-05-15 Fuji Photo Film Co., Ltd. Camera shake detection apparatus
US5926212A (en) 1995-08-30 1999-07-20 Sony Corporation Image signal processing apparatus and recording/reproducing apparatus
US6122004A (en) 1995-12-28 2000-09-19 Samsung Electronics Co., Ltd. Image stabilizing circuit for a camcorder
US6246790B1 (en) 1997-12-29 2001-06-12 Cornell Research Foundation, Inc. Image indexing using color correlograms
US20010026632A1 (en) 2000-03-24 2001-10-04 Seiichiro Tamai Apparatus for identity verification, a system for identity verification, a card for identity verification and a method for identity verification, based on identification by biometrics
US20040090551A1 (en) 2002-11-01 2004-05-13 Kunio Yata Auto focus system
US6747690B2 (en) 2000-07-11 2004-06-08 Phase One A/S Digital camera with integrated accelerometers
US6934087B1 (en) 2002-09-25 2005-08-23 Siimpel Corporation Fiber optic collimator and collimator array
US20050270410A1 (en) 2004-06-03 2005-12-08 Canon Kabushiki Kaisha Image pickup apparatus and image pickup method
US20060140455A1 (en) * 2004-12-29 2006-06-29 Gabriel Costache Method and component for image recognition
US20060204034A1 (en) 2003-06-26 2006-09-14 Eran Steinberg Modification of viewing parameters for digital images using face detection information
US7113688B2 (en) 2002-02-04 2006-09-26 Siimpel Corporation Base, payload and connecting structure and methods of making the same
US20060239579A1 (en) 2005-04-22 2006-10-26 Ritter Bradford A Non Uniform Blending of Exposure and/or Focus Bracketed Photographic Images
US20070030381A1 (en) 2005-01-18 2007-02-08 Nikon Corporation Digital camera
US20070110305A1 (en) 2003-06-26 2007-05-17 Fotonation Vision Limited Digital Image Processing Using Face Detection and Skin Tone Information
WO2007093199A2 (en) 2006-02-14 2007-08-23 Fotonation Vision Limited Foreground/background segmentation in digital images
US20070201725A1 (en) 2006-02-24 2007-08-30 Eran Steinberg Digital Image Acquisition Control and Correction Method and Apparatus
US7266727B2 (en) 2004-03-18 2007-09-04 International Business Machines Corporation Computer boot operation utilizing targeted boot diagnostics
US7269292B2 (en) 2003-06-26 2007-09-11 Fotonation Vision Limited Digital image adjustable compression and resolution using face detection information
US20070269108A1 (en) 2006-05-03 2007-11-22 Fotonation Vision Limited Foreground / Background Separation in Digital Images
US20070285528A1 (en) 2006-06-09 2007-12-13 Sony Corporation Imaging apparatus, control method of imaging apparatus, and computer program
US20070296833A1 (en) 2006-06-05 2007-12-27 Fotonation Vision Limited Image Acquisition Method and Apparatus
US7315630B2 (en) 2003-06-26 2008-01-01 Fotonation Vision Limited Perfecting of digital image rendering parameters within rendering devices using face detection
US7315631B1 (en) 2006-08-11 2008-01-01 Fotonation Vision Limited Real-time face tracking in a digital image acquisition device
US7317815B2 (en) 2003-06-26 2008-01-08 Fotonation Vision Limited Digital image processing composition using face detection information
US20080013798A1 (en) 2006-06-12 2008-01-17 Fotonation Vision Limited Advances in extending the aam techniques from grayscale to color images
US20080013799A1 (en) 2003-06-26 2008-01-17 Fotonation Vision Limited Method of Improving Orientation and Color Balance of Digital Images Using Face Detection Information
US7345827B1 (en) 2005-02-28 2008-03-18 Siimpel Corporation Lens barrel
US7359131B1 (en) 2005-02-28 2008-04-15 Siimpel Corporation Lens positioning systems and methods
US7359130B1 (en) 2002-02-04 2008-04-15 Siimpel Corporation Lens mount and alignment method
US7362368B2 (en) 2003-06-26 2008-04-22 Fotonation Vision Limited Perfecting the optics within a digital image acquisition device using face detection
US7403643B2 (en) 2006-08-11 2008-07-22 Fotonation Vision Limited Real-time face tracking in a digital image acquisition device
US7403444B2 (en) 2003-12-09 2008-07-22 Micron Technology, Inc. Selectable memory word line deactivation
US20080175481A1 (en) 2007-01-18 2008-07-24 Stefan Petrescu Color Segmentation
US20080205712A1 (en) 2007-02-28 2008-08-28 Fotonation Vision Limited Separating Directional Lighting Variability in Statistical Face Modelling Based on Texture Space Decomposition
US20080219581A1 (en) 2007-03-05 2008-09-11 Fotonation Vision Limited Image Processing Method and Apparatus
US20080218868A1 (en) * 2003-12-16 2008-09-11 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Image correction using individual manipulation of microlenses in a microlens array
US20080219517A1 (en) * 2007-03-05 2008-09-11 Fotonation Vision Limited Illumination Detection Using Classifier Chains
US20080220750A1 (en) 2007-03-05 2008-09-11 Fotonation Vision Limited Face Categorization and Annotation of a Mobile Phone Contact List
US20080252773A1 (en) 2007-04-11 2008-10-16 Fujifilm Corporation Image pickup apparatus, focusing control method and principal object detecting method
US20080259176A1 (en) 2007-04-20 2008-10-23 Fujifilm Corporation Image pickup apparatus, image processing apparatus, image pickup method, and image processing method
US20080266419A1 (en) 2007-04-30 2008-10-30 Fotonation Ireland Limited Method and apparatus for automatically controlling the decisive moment for an image acquisition device
US20080292193A1 (en) 2007-05-24 2008-11-27 Fotonation Vision Limited Image Processing Method and Apparatus
US20080309769A1 (en) 2007-06-14 2008-12-18 Fotonation Ireland Limited Fast Motion Estimation Method
US7469071B2 (en) 2006-02-14 2008-12-23 Fotonation Vision Limited Image blurring
US20080316328A1 (en) 2005-12-27 2008-12-25 Fotonation Ireland Limited Foreground/background separation using reference images
US7471846B2 (en) 2003-06-26 2008-12-30 Fotonation Vision Limited Perfecting the effect of flash within an image acquisition devices using face detection
US20090002514A1 (en) 2003-06-26 2009-01-01 Fotonation Vision Limited Digital Image Processing Using Face Detection Information
US20090003708A1 (en) 2003-06-26 2009-01-01 Fotonation Ireland Limited Modification of post-viewing parameters for digital images using image region or feature information
US7477400B2 (en) 2005-09-02 2009-01-13 Siimpel Corporation Range and speed finder
US7477842B2 (en) 2004-03-12 2009-01-13 Siimpel, Inc. Miniature camera
US7495852B2 (en) 2005-02-28 2009-02-24 Siimpel Corporation Zoom lens assembly
US20090059061A1 (en) * 2007-08-30 2009-03-05 Samsung Techwin Co., Ltd. Digital photographing apparatus and method using face recognition function
EP2037320A1 (en) 2007-09-14 2009-03-18 Sony Corporation Imaging apparatus, imaging apparatus control method, and computer program
US20090080713A1 (en) 2007-09-26 2009-03-26 Fotonation Vision Limited Face tracking in a camera processor
US7515740B2 (en) 2006-08-02 2009-04-07 Fotonation Vision Limited Face recognition with combined PCA-based datasets
US7518635B2 (en) 1998-02-24 2009-04-14 Canon Kabushiki Kaisha Method and apparatus for image sensing
US7545591B1 (en) 2006-10-17 2009-06-09 Siimpel Corporation Uniform wall thickness lens barrel
US20090153725A1 (en) 2007-12-18 2009-06-18 Canon Kabushiki Kaisha Image capturing apparatus, control method therefor, and program
US7551755B1 (en) 2004-01-22 2009-06-23 Fotonation Vision Limited Classification and organization of consumer digital images using workflow, and face detection and recognition
US7551754B2 (en) 2006-02-24 2009-06-23 Fotonation Vision Limited Method and apparatus for selective rejection of digital images
US7555210B2 (en) 2005-02-28 2009-06-30 Siimpel, Inc. Axial snubbers for camera
US20090167893A1 (en) 2007-03-05 2009-07-02 Fotonation Vision Limited RGBW Sensor Array
US7558408B1 (en) 2004-01-22 2009-07-07 Fotonation Vision Limited Classification system for consumer digital images using workflow and user interface modules, and face detection and recognition
US7560679B1 (en) 2005-05-10 2009-07-14 Siimpel, Inc. 3D camera
US20090179999A1 (en) 2007-09-18 2009-07-16 Fotonation Ireland Limited Image Processing Method and Apparatus
US7565070B1 (en) 2005-02-28 2009-07-21 Siimpel Corporation Telephone vibrator
US20090185753A1 (en) 2008-01-18 2009-07-23 Fotonation Ireland Limited Image processing method and apparatus
US20090190803A1 (en) 2008-01-29 2009-07-30 Fotonation Ireland Limited Detecting facial expressions in digital images
US20090196466A1 (en) 2008-02-05 2009-08-06 Fotonation Vision Limited Face Detection in Mid-Shot Digital Images
US7583006B2 (en) 2005-07-26 2009-09-01 Siimpel Corporation MEMS digital linear actuator
US7587068B1 (en) 2004-01-22 2009-09-08 Fotonation Vision Limited Classification database for consumer digital images
US20090238419A1 (en) 2007-03-05 2009-09-24 Fotonation Ireland Limited Face recognition training method and apparatus
US20090238487A1 (en) 2008-03-13 2009-09-24 Olympus Corporation Image processing apparatus and image processing method
US20090244296A1 (en) 2008-03-26 2009-10-01 Fotonation Ireland Limited Method of making a digital camera image of a scene including the camera user
US20090245693A1 (en) 2003-06-26 2009-10-01 Fotonation Ireland Limited Detecting orientation of digital images using face detection information
US20090268080A1 (en) 2008-04-25 2009-10-29 Samsung Techwin Co., Ltd. Bracketing apparatus and method for use in digital image processor
US7616233B2 (en) 2003-06-26 2009-11-10 Fotonation Vision Limited Perfecting of digital image capture parameters within acquisition devices using face detection
US7620218B2 (en) 2006-08-11 2009-11-17 Fotonation Ireland Limited Real-time face tracking with reference images
US20090303342A1 (en) * 2006-08-11 2009-12-10 Fotonation Ireland Limited Face tracking for controlling imaging parameters
US20090303343A1 (en) 2007-03-05 2009-12-10 Fotonation Ireland Limited Low-light video frame enhancement
US20090310885A1 (en) * 2008-06-13 2009-12-17 Fujifilm Corporation Image processing apparatus, imaging apparatus, image processing method and recording medium
US7636486B2 (en) 2004-11-10 2009-12-22 Fotonation Ireland Ltd. Method of determining PSF using multiple instances of a nominally similar scene
US7639888B2 (en) 2004-11-10 2009-12-29 Fotonation Ireland Ltd. Method and apparatus for initiating subsequent exposures based on determination of motion blurring artifacts
US7639889B2 (en) 2004-11-10 2009-12-29 Fotonation Ireland Ltd. Method of notifying users regarding motion artifacts based on image analysis
US7640803B1 (en) 2004-05-26 2010-01-05 Siimpel Corporation Micro-electromechanical system inertial sensor
US7646969B2 (en) 2005-02-28 2010-01-12 Siimpel Corporation Camera snubber assembly
US20100014721A1 (en) 2004-01-22 2010-01-21 Fotonation Ireland Limited Classification System for Consumer Digital Images using Automatic Workflow and Face Detection and Recognition
US7663817B1 (en) 2007-01-19 2010-02-16 Siimpel Corporation Optical system with plano convex lens
US7663289B1 (en) 2004-02-18 2010-02-16 Siimpel Corporation Centipede actuator motion stage
US20100054533A1 (en) 2003-06-26 2010-03-04 Fotonation Vision Limited Digital Image Processing Using Face Detection Information
US20100054549A1 (en) 2003-06-26 2010-03-04 Fotonation Vision Limited Digital Image Processing Using Face Detection Information
US20100066822A1 (en) 2004-01-22 2010-03-18 Fotonation Ireland Limited Classification and organization of consumer digital images using workflow, and face detection and recognition
US7693408B1 (en) * 2006-10-13 2010-04-06 Siimpel Corporation Camera with multiple focus captures
US20100085436A1 (en) 2007-03-23 2010-04-08 Kabushiki Kaisha Toshiba Hand movement correction apparatus, image pickup apparatus, hand movement correction program, image pickup program, hand movement correction method
US7697831B1 (en) 2007-02-20 2010-04-13 Siimpel Corporation Auto-focus with lens vibration
US7697834B1 (en) 2006-10-13 2010-04-13 Siimpel Corporation Hidden autofocus
US7697829B1 (en) 2005-09-02 2010-04-13 Siimpel Corporation Electronic damping for stage positioning
US7702222B2 (en) 2003-10-10 2010-04-20 Panasonic Corporation Playback apparatus program and playback method
US20100128163A1 (en) * 2008-11-25 2010-05-27 Sony Corporation Imaging device and imaging method
US7729601B1 (en) 2007-08-31 2010-06-01 Siimpel Corporation Shutter for autofocus
US7729603B2 (en) 2005-02-28 2010-06-01 Siimpel Corporation Resolution adjustment for miniature camera
US20100141787A1 (en) 2008-12-05 2010-06-10 Fotonation Ireland Limited Face recognition using face tracker classifier data
US20100182458A1 (en) 2005-12-27 2010-07-22 Fotonation Ireland Limited Digital image acquisition system with portrait mode
US20100188535A1 (en) 2009-01-23 2010-07-29 Sony Corporation Image processing apparatus, image processing method, and imaging apparatus
US7769281B1 (en) 2006-07-18 2010-08-03 Siimpel Corporation Stage with built-in damping
US20100194869A1 (en) 2009-01-30 2010-08-05 Olympus Corporation Scene-change detecting device, computer readable storage medium storing scene-change detection program, and scene-change detecting method
US7773118B2 (en) 2007-03-25 2010-08-10 Fotonation Vision Limited Handheld article with movement discrimination
US20100208091A1 (en) * 2009-02-19 2010-08-19 Apple Inc. Auto focus speed enhancement using object recognition and resolution
US7796822B2 (en) 2004-08-16 2010-09-14 Fotonation Vision Limited Foreground/background segmentation in digital images
US7796816B2 (en) 2004-08-16 2010-09-14 Fotonation Vision Limited Foreground/background segmentation in digital images with differential exposure calculations
US20100271494A1 (en) 2009-04-23 2010-10-28 Canon Kabushiki Kaisha Motion vector detection apparatus, motion vector detection method, and image capturing apparatus
US20100329582A1 (en) 2009-06-29 2010-12-30 Tessera Technologies Ireland Limited Adaptive PSF Estimation Technique Using a Sharp Preview and a Blurred Image
US20110135208A1 (en) * 2009-12-03 2011-06-09 Qualcomm Incorporated Digital image combining to produce optical effects
US20110249173A1 (en) 2010-04-13 2011-10-13 Sony Corporation Four-dimensional polynomial model for depth estimation based on two-picture matching
US20120075492A1 (en) 2010-09-28 2012-03-29 Tessera Technologies Ireland Limited Continuous Autofocus Based on Face Detection and Tracking
WO2012062893A3 (en) 2010-11-11 2012-07-12 DigitalOptics Corporation Europe Limited Object detection and recognition under out of focus conditions
US20120218461A1 (en) 2007-11-28 2012-08-30 Fujifilm Corporation Image pickup apparatus and image pickup method used for the same

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2801211B2 (en) * 1988-07-29 1998-09-21 キヤノン株式会社 Automatic focusing apparatus and a camera control apparatus and an optical apparatus and a camera apparatus
JP3610176B2 (en) * 1996-12-03 2005-01-12 キヤノン株式会社 Imaging apparatus, a control method, and a lens control device
US7099510B2 (en) 2000-11-29 2006-08-29 Hewlett-Packard Development Company, L.P. Method and system for object detection in digital images
JP3760846B2 (en) * 2001-11-22 2006-03-29 コニカミノルタフォトイメージング株式会社 Subject extracting device and imaging apparatus
CN100543574C (en) * 2004-10-22 2009-09-23 亚洲光学股份有限公司 Automatic focusing method and automatic focusing apparatus of electronic camera
US7403344B2 (en) 2005-02-28 2008-07-22 Siimpel Corporation Lens Assembly
US7668388B2 (en) * 2005-03-03 2010-02-23 Mitutoyo Corporation System and method for single image focus assessment
JP2007259004A (en) * 2006-03-23 2007-10-04 Nikon Corp Digital camera, image processor, and image processing program
US7702226B1 (en) 2006-07-20 2010-04-20 Siimpel Corporation Board mounted hermetically sealed optical device enclosure and manufacturing methods therefor
JP4218720B2 (en) * 2006-09-22 2009-02-04 ソニー株式会社 Imaging device, and imaging apparatus control method, and computer program
KR101488655B1 (en) * 2007-09-14 2015-02-03 삼성전자 주식회사 Method and apparatus for auto focusing
KR20100134085A (en) * 2008-04-01 2010-12-22 휴렛-팩커드 디벨롭먼트 컴퍼니, 엘.피. Systems and methods to increase speed of object detection in a digital image
JP5267149B2 (en) * 2009-01-19 2013-08-21 ソニー株式会社 Display control apparatus, a display control method, and program
JP5158720B2 (en) * 2009-03-11 2013-03-06 富士フイルム株式会社 Optical module and a method of manufacturing the same, and an imaging device

Patent Citations (158)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4448510A (en) 1981-10-23 1984-05-15 Fuji Photo Film Co., Ltd. Camera shake detection apparatus
US5926212A (en) 1995-08-30 1999-07-20 Sony Corporation Image signal processing apparatus and recording/reproducing apparatus
US6122004A (en) 1995-12-28 2000-09-19 Samsung Electronics Co., Ltd. Image stabilizing circuit for a camcorder
US6246790B1 (en) 1997-12-29 2001-06-12 Cornell Research Foundation, Inc. Image indexing using color correlograms
US7518635B2 (en) 1998-02-24 2009-04-14 Canon Kabushiki Kaisha Method and apparatus for image sensing
US20010026632A1 (en) 2000-03-24 2001-10-04 Seiichiro Tamai Apparatus for identity verification, a system for identity verification, a card for identity verification and a method for identity verification, based on identification by biometrics
US6747690B2 (en) 2000-07-11 2004-06-08 Phase One A/S Digital camera with integrated accelerometers
US7113688B2 (en) 2002-02-04 2006-09-26 Siimpel Corporation Base, payload and connecting structure and methods of making the same
US7359130B1 (en) 2002-02-04 2008-04-15 Siimpel Corporation Lens mount and alignment method
US6934087B1 (en) 2002-09-25 2005-08-23 Siimpel Corporation Fiber optic collimator and collimator array
US20040090551A1 (en) 2002-11-01 2004-05-13 Kunio Yata Auto focus system
US7471846B2 (en) 2003-06-26 2008-12-30 Fotonation Vision Limited Perfecting the effect of flash within an image acquisition devices using face detection
US20100188530A1 (en) 2003-06-26 2010-07-29 Fotonation Vision Limited Perfecting the Effect of Flash within an Image Acquisition Devices Using Face Detection
US20100165140A1 (en) 2003-06-26 2010-07-01 Fotonation Vision Limited Digital image adjustable compression and resolution using face detection information
US20070110305A1 (en) 2003-06-26 2007-05-17 Fotonation Vision Limited Digital Image Processing Using Face Detection and Skin Tone Information
US20070160307A1 (en) 2003-06-26 2007-07-12 Fotonation Vision Limited Modification of Viewing Parameters for Digital Images Using Face Detection Information
US20100165150A1 (en) 2003-06-26 2010-07-01 Fotonation Vision Limited Detecting orientation of digital images using face detection information
US7702136B2 (en) 2003-06-26 2010-04-20 Fotonation Vision Limited Perfecting the effect of flash within an image acquisition devices using face detection
US20090003708A1 (en) 2003-06-26 2009-01-01 Fotonation Ireland Limited Modification of post-viewing parameters for digital images using image region or feature information
US20060204034A1 (en) 2003-06-26 2006-09-14 Eran Steinberg Modification of viewing parameters for digital images using face detection information
US20100092039A1 (en) 2003-06-26 2010-04-15 Eran Steinberg Digital Image Processing Using Face Detection Information
US7693311B2 (en) 2003-06-26 2010-04-06 Fotonation Vision Limited Perfecting the effect of flash within an image acquisition devices using face detection
US7684630B2 (en) 2003-06-26 2010-03-23 Fotonation Vision Limited Digital image adjustable compression and resolution using face detection information
US7315630B2 (en) 2003-06-26 2008-01-01 Fotonation Vision Limited Perfecting of digital image rendering parameters within rendering devices using face detection
US20100054549A1 (en) 2003-06-26 2010-03-04 Fotonation Vision Limited Digital Image Processing Using Face Detection Information
US7317815B2 (en) 2003-06-26 2008-01-08 Fotonation Vision Limited Digital image processing composition using face detection information
US20100188525A1 (en) 2003-06-26 2010-07-29 Fotonation Vision Limited Perfecting the Effect of Flash within an Image Acquisition Devices Using Face Detection
US20080013799A1 (en) 2003-06-26 2008-01-17 Fotonation Vision Limited Method of Improving Orientation and Color Balance of Digital Images Using Face Detection Information
US20100039525A1 (en) 2003-06-26 2010-02-18 Fotonation Ireland Limited Perfecting of Digital Image Capture Parameters Within Acquisition Devices Using Face Detection
US20090179998A1 (en) 2003-06-26 2009-07-16 Fotonation Vision Limited Modification of Post-Viewing Parameters for Digital Images Using Image Region or Feature Information
US20100220899A1 (en) 2003-06-26 2010-09-02 Fotonation Vision Limited Method of improving orientation and color balance of digital images using face detection information
US7362368B2 (en) 2003-06-26 2008-04-22 Fotonation Vision Limited Perfecting the optics within a digital image acquisition device using face detection
US20080143854A1 (en) 2003-06-26 2008-06-19 Fotonation Vision Limited Perfecting the optics within a digital image acquisition device using face detection
US7634109B2 (en) 2003-06-26 2009-12-15 Fotonation Ireland Limited Digital image processing using face detection information
US7466866B2 (en) 2003-06-26 2008-12-16 Fotonation Vision Limited Digital image adjustable compression and resolution using face detection information
US7630527B2 (en) 2003-06-26 2009-12-08 Fotonation Ireland Limited Method of improving orientation and color balance of digital images using face detection information
US7616233B2 (en) 2003-06-26 2009-11-10 Fotonation Vision Limited Perfecting of digital image capture parameters within acquisition devices using face detection
US20090245693A1 (en) 2003-06-26 2009-10-01 Fotonation Ireland Limited Detecting orientation of digital images using face detection information
US20090052749A1 (en) 2003-06-26 2009-02-26 Fotonation Vision Limited Digital Image Processing Using Face Detection Information
US20090052750A1 (en) 2003-06-26 2009-02-26 Fotonation Vision Limited Digital Image Processing Using Face Detection Information
US7574016B2 (en) 2003-06-26 2009-08-11 Fotonation Vision Limited Digital image processing using face detection information
US7269292B2 (en) 2003-06-26 2007-09-11 Fotonation Vision Limited Digital image adjustable compression and resolution using face detection information
US7440593B1 (en) 2003-06-26 2008-10-21 Fotonation Vision Limited Method of improving orientation and color balance of digital images using face detection information
US20090087042A1 (en) 2003-06-26 2009-04-02 Fotonation Vision Limited Digital Image Processing Using Face Detection Information
US20100054533A1 (en) 2003-06-26 2010-03-04 Fotonation Vision Limited Digital Image Processing Using Face Detection Information
US20090002514A1 (en) 2003-06-26 2009-01-01 Fotonation Vision Limited Digital Image Processing Using Face Detection Information
US7702222B2 (en) 2003-10-10 2010-04-20 Panasonic Corporation Playback apparatus program and playback method
US7403444B2 (en) 2003-12-09 2008-07-22 Micron Technology, Inc. Selectable memory word line deactivation
US20080218868A1 (en) * 2003-12-16 2008-09-11 Searete Llc, A Limited Liability Corporation Of The State Of Delaware Image correction using individual manipulation of microlenses in a microlens array
US7558408B1 (en) 2004-01-22 2009-07-07 Fotonation Vision Limited Classification system for consumer digital images using workflow and user interface modules, and face detection and recognition
US7587068B1 (en) 2004-01-22 2009-09-08 Fotonation Vision Limited Classification database for consumer digital images
US20100014721A1 (en) 2004-01-22 2010-01-21 Fotonation Ireland Limited Classification System for Consumer Digital Images using Automatic Workflow and Face Detection and Recognition
US7551755B1 (en) 2004-01-22 2009-06-23 Fotonation Vision Limited Classification and organization of consumer digital images using workflow, and face detection and recognition
US20100066822A1 (en) 2004-01-22 2010-03-18 Fotonation Ireland Limited Classification and organization of consumer digital images using workflow, and face detection and recognition
US7663289B1 (en) 2004-02-18 2010-02-16 Siimpel Corporation Centipede actuator motion stage
US7477842B2 (en) 2004-03-12 2009-01-13 Siimpel, Inc. Miniature camera
US7266727B2 (en) 2004-03-18 2007-09-04 International Business Machines Corporation Computer boot operation utilizing targeted boot diagnostics
US7640803B1 (en) 2004-05-26 2010-01-05 Siimpel Corporation Micro-electromechanical system inertial sensor
US20050270410A1 (en) 2004-06-03 2005-12-08 Canon Kabushiki Kaisha Image pickup apparatus and image pickup method
US7796822B2 (en) 2004-08-16 2010-09-14 Fotonation Vision Limited Foreground/background segmentation in digital images
US7796816B2 (en) 2004-08-16 2010-09-14 Fotonation Vision Limited Foreground/background segmentation in digital images with differential exposure calculations
US7636486B2 (en) 2004-11-10 2009-12-22 Fotonation Ireland Ltd. Method of determining PSF using multiple instances of a nominally similar scene
US7639888B2 (en) 2004-11-10 2009-12-29 Fotonation Ireland Ltd. Method and apparatus for initiating subsequent exposures based on determination of motion blurring artifacts
US7676108B2 (en) 2004-11-10 2010-03-09 Fotonation Vision Limited Method and apparatus for initiating subsequent exposures based on determination of motion blurring artifacts
US7639889B2 (en) 2004-11-10 2009-12-29 Fotonation Ireland Ltd. Method of notifying users regarding motion artifacts based on image analysis
US7697778B2 (en) 2004-11-10 2010-04-13 Fotonation Vision Limited Method of notifying users regarding motion artifacts based on image analysis
US7660478B2 (en) 2004-11-10 2010-02-09 Fotonation Vision Ltd. Method of determining PSF using multiple instances of nominally scene
US20100202707A1 (en) 2004-12-29 2010-08-12 Fotonation Vision Limited Method and Component for Image Recognition
US20060140455A1 (en) * 2004-12-29 2006-06-29 Gabriel Costache Method and component for image recognition
US20070030381A1 (en) 2005-01-18 2007-02-08 Nikon Corporation Digital camera
US7359131B1 (en) 2005-02-28 2008-04-15 Siimpel Corporation Lens positioning systems and methods
US7646969B2 (en) 2005-02-28 2010-01-12 Siimpel Corporation Camera snubber assembly
US7345827B1 (en) 2005-02-28 2008-03-18 Siimpel Corporation Lens barrel
US7495852B2 (en) 2005-02-28 2009-02-24 Siimpel Corporation Zoom lens assembly
US7747155B1 (en) 2005-02-28 2010-06-29 Siimpel Corporation Zoom lens assembly
US7555210B2 (en) 2005-02-28 2009-06-30 Siimpel, Inc. Axial snubbers for camera
US7729603B2 (en) 2005-02-28 2010-06-01 Siimpel Corporation Resolution adjustment for miniature camera
US7515362B1 (en) 2005-02-28 2009-04-07 Siimpel Corporation Lens positioning systems and methods
US7565070B1 (en) 2005-02-28 2009-07-21 Siimpel Corporation Telephone vibrator
US7660056B1 (en) 2005-02-28 2010-02-09 Siimpel Corporation Lens barrel
US20060239579A1 (en) 2005-04-22 2006-10-26 Ritter Bradford A Non Uniform Blending of Exposure and/or Focus Bracketed Photographic Images
US7560679B1 (en) 2005-05-10 2009-07-14 Siimpel, Inc. 3D camera
US7583006B2 (en) 2005-07-26 2009-09-01 Siimpel Corporation MEMS digital linear actuator
US7477400B2 (en) 2005-09-02 2009-01-13 Siimpel Corporation Range and speed finder
US7697829B1 (en) 2005-09-02 2010-04-13 Siimpel Corporation Electronic damping for stage positioning
US20080316328A1 (en) 2005-12-27 2008-12-25 Fotonation Ireland Limited Foreground/background separation using reference images
US20100182458A1 (en) 2005-12-27 2010-07-22 Fotonation Ireland Limited Digital image acquisition system with portrait mode
US20090040342A1 (en) 2006-02-14 2009-02-12 Fotonation Vision Limited Image Blurring
US20090273685A1 (en) 2006-02-14 2009-11-05 Fotonation Vision Limited Foreground/Background Segmentation in Digital Images
US7469071B2 (en) 2006-02-14 2008-12-23 Fotonation Vision Limited Image blurring
WO2007093199A2 (en) 2006-02-14 2007-08-23 Fotonation Vision Limited Foreground/background segmentation in digital images
US7551754B2 (en) 2006-02-24 2009-06-23 Fotonation Vision Limited Method and apparatus for selective rejection of digital images
US20070201725A1 (en) 2006-02-24 2007-08-30 Eran Steinberg Digital Image Acquisition Control and Correction Method and Apparatus
US20070269108A1 (en) 2006-05-03 2007-11-22 Fotonation Vision Limited Foreground / Background Separation in Digital Images
US20070296833A1 (en) 2006-06-05 2007-12-27 Fotonation Vision Limited Image Acquisition Method and Apparatus
US20070285528A1 (en) 2006-06-09 2007-12-13 Sony Corporation Imaging apparatus, control method of imaging apparatus, and computer program
US20080013798A1 (en) 2006-06-12 2008-01-17 Fotonation Vision Limited Advances in extending the aam techniques from grayscale to color images
US7769281B1 (en) 2006-07-18 2010-08-03 Siimpel Corporation Stage with built-in damping
US20090238410A1 (en) 2006-08-02 2009-09-24 Fotonation Vision Limited Face recognition with combined pca-based datasets
US7515740B2 (en) 2006-08-02 2009-04-07 Fotonation Vision Limited Face recognition with combined PCA-based datasets
US7620218B2 (en) 2006-08-11 2009-11-17 Fotonation Ireland Limited Real-time face tracking with reference images
US7315631B1 (en) 2006-08-11 2008-01-01 Fotonation Vision Limited Real-time face tracking in a digital image acquisition device
US7403643B2 (en) 2006-08-11 2008-07-22 Fotonation Vision Limited Real-time face tracking in a digital image acquisition device
US20100060727A1 (en) * 2006-08-11 2010-03-11 Eran Steinberg Real-time face tracking with reference images
US20080267461A1 (en) 2006-08-11 2008-10-30 Fotonation Ireland Limited Real-time face tracking in a digital image acquisition device
US20090208056A1 (en) 2006-08-11 2009-08-20 Fotonation Vision Limited Real-time face tracking in a digital image acquisition device
US20090263022A1 (en) 2006-08-11 2009-10-22 Fotonation Vision Limited Real-Time Face Tracking in a Digital Image Acquisition Device
US7460694B2 (en) 2006-08-11 2008-12-02 Fotonation Vision Limited Real-time face tracking in a digital image acquisition device
US7469055B2 (en) 2006-08-11 2008-12-23 Fotonation Vision Limited Real-time face tracking in a digital image acquisition device
US20090303342A1 (en) * 2006-08-11 2009-12-10 Fotonation Ireland Limited Face tracking for controlling imaging parameters
US7460695B2 (en) 2006-08-11 2008-12-02 Fotonation Vision Limited Real-time face tracking in a digital image acquisition device
US7697834B1 (en) 2006-10-13 2010-04-13 Siimpel Corporation Hidden autofocus
US7693408B1 (en) * 2006-10-13 2010-04-06 Siimpel Corporation Camera with multiple focus captures
US7545591B1 (en) 2006-10-17 2009-06-09 Siimpel Corporation Uniform wall thickness lens barrel
US20080175481A1 (en) 2007-01-18 2008-07-24 Stefan Petrescu Color Segmentation
US7663817B1 (en) 2007-01-19 2010-02-16 Siimpel Corporation Optical system with plano convex lens
US7697831B1 (en) 2007-02-20 2010-04-13 Siimpel Corporation Auto-focus with lens vibration
US20080205712A1 (en) 2007-02-28 2008-08-28 Fotonation Vision Limited Separating Directional Lighting Variability in Statistical Face Modelling Based on Texture Space Decomposition
US20090003661A1 (en) 2007-02-28 2009-01-01 Fotonation Vision Limited Separating a Directional Lighting Variability In Statistical Face Modelling Based On Texture Space Decomposition
US20090303343A1 (en) 2007-03-05 2009-12-10 Fotonation Ireland Limited Low-light video frame enhancement
US20080219581A1 (en) 2007-03-05 2008-09-11 Fotonation Vision Limited Image Processing Method and Apparatus
US20080219517A1 (en) * 2007-03-05 2008-09-11 Fotonation Vision Limited Illumination Detection Using Classifier Chains
US20080220750A1 (en) 2007-03-05 2008-09-11 Fotonation Vision Limited Face Categorization and Annotation of a Mobile Phone Contact List
US20090238419A1 (en) 2007-03-05 2009-09-24 Fotonation Ireland Limited Face recognition training method and apparatus
US20090167893A1 (en) 2007-03-05 2009-07-02 Fotonation Vision Limited RGBW Sensor Array
US20100085436A1 (en) 2007-03-23 2010-04-08 Kabushiki Kaisha Toshiba Hand movement correction apparatus, image pickup apparatus, hand movement correction program, image pickup program, hand movement correction method
US7773118B2 (en) 2007-03-25 2010-08-10 Fotonation Vision Limited Handheld article with movement discrimination
US20080252773A1 (en) 2007-04-11 2008-10-16 Fujifilm Corporation Image pickup apparatus, focusing control method and principal object detecting method
US20080259176A1 (en) 2007-04-20 2008-10-23 Fujifilm Corporation Image pickup apparatus, image processing apparatus, image pickup method, and image processing method
US20080266419A1 (en) 2007-04-30 2008-10-30 Fotonation Ireland Limited Method and apparatus for automatically controlling the decisive moment for an image acquisition device
US20080292193A1 (en) 2007-05-24 2008-11-27 Fotonation Vision Limited Image Processing Method and Apparatus
US20080309769A1 (en) 2007-06-14 2008-12-18 Fotonation Ireland Limited Fast Motion Estimation Method
US20090059061A1 (en) * 2007-08-30 2009-03-05 Samsung Techwin Co., Ltd. Digital photographing apparatus and method using face recognition function
US7729601B1 (en) 2007-08-31 2010-06-01 Siimpel Corporation Shutter for autofocus
EP2037320A1 (en) 2007-09-14 2009-03-18 Sony Corporation Imaging apparatus, imaging apparatus control method, and computer program
US20090179999A1 (en) 2007-09-18 2009-07-16 Fotonation Ireland Limited Image Processing Method and Apparatus
US20090080713A1 (en) 2007-09-26 2009-03-26 Fotonation Vision Limited Face tracking in a camera processor
US20120218461A1 (en) 2007-11-28 2012-08-30 Fujifilm Corporation Image pickup apparatus and image pickup method used for the same
US20090153725A1 (en) 2007-12-18 2009-06-18 Canon Kabushiki Kaisha Image capturing apparatus, control method therefor, and program
US20090185753A1 (en) 2008-01-18 2009-07-23 Fotonation Ireland Limited Image processing method and apparatus
US20090190803A1 (en) 2008-01-29 2009-07-30 Fotonation Ireland Limited Detecting facial expressions in digital images
US20090196466A1 (en) 2008-02-05 2009-08-06 Fotonation Vision Limited Face Detection in Mid-Shot Digital Images
US20090238487A1 (en) 2008-03-13 2009-09-24 Olympus Corporation Image processing apparatus and image processing method
US20090244296A1 (en) 2008-03-26 2009-10-01 Fotonation Ireland Limited Method of making a digital camera image of a scene including the camera user
US20090268080A1 (en) 2008-04-25 2009-10-29 Samsung Techwin Co., Ltd. Bracketing apparatus and method for use in digital image processor
US20090310885A1 (en) * 2008-06-13 2009-12-17 Fujifilm Corporation Image processing apparatus, imaging apparatus, image processing method and recording medium
US20100128163A1 (en) * 2008-11-25 2010-05-27 Sony Corporation Imaging device and imaging method
US20100141786A1 (en) 2008-12-05 2010-06-10 Fotonation Ireland Limited Face recognition using face tracker classifier data
US20100141787A1 (en) 2008-12-05 2010-06-10 Fotonation Ireland Limited Face recognition using face tracker classifier data
US20100188535A1 (en) 2009-01-23 2010-07-29 Sony Corporation Image processing apparatus, image processing method, and imaging apparatus
US20100194869A1 (en) 2009-01-30 2010-08-05 Olympus Corporation Scene-change detecting device, computer readable storage medium storing scene-change detection program, and scene-change detecting method
US20100208091A1 (en) * 2009-02-19 2010-08-19 Apple Inc. Auto focus speed enhancement using object recognition and resolution
US20100271494A1 (en) 2009-04-23 2010-10-28 Canon Kabushiki Kaisha Motion vector detection apparatus, motion vector detection method, and image capturing apparatus
US20100329582A1 (en) 2009-06-29 2010-12-30 Tessera Technologies Ireland Limited Adaptive PSF Estimation Technique Using a Sharp Preview and a Blurred Image
US20110135208A1 (en) * 2009-12-03 2011-06-09 Qualcomm Incorporated Digital image combining to produce optical effects
US20110249173A1 (en) 2010-04-13 2011-10-13 Sony Corporation Four-dimensional polynomial model for depth estimation based on two-picture matching
US20120075492A1 (en) 2010-09-28 2012-03-29 Tessera Technologies Ireland Limited Continuous Autofocus Based on Face Detection and Tracking
WO2012062893A3 (en) 2010-11-11 2012-07-12 DigitalOptics Corporation Europe Limited Object detection and recognition under out of focus conditions

Non-Patent Citations (24)

* Cited by examiner, † Cited by third party
Title
H. Lai, P. C. Yuen, and G. C. Feng, Face recognition using holistic Fourier invariant features, Pattern Recognition, vol. 34, pp. 95-109, 2001.
Hamm P., Schulz J. and Englmeier K.-H.: "Content-Based Autofocusing in Automated Microscopy", Image Analysis Stereology, vol. 29, Nov. 1, 2010, pp. 173-180, XP002677721.
Jing Huang, S. Ravi Kumar, Mandar Mitra, Wei-Jing Zhu, Ramin Zabih, Image Indexing Using Color Correlograms, IEEE Conf. Computer Vision and Pattern Recognition, (CVPR '97) pp. 762-768, 1997.
K. Sauer and B. Schwartz, Efficient Block Motion Estimation Using Integral Projections, IEEE Trans. Circuits, Systems for video Tech., 1996, vol. 6, No. 5, October, pp. 513-518.
Karen P Hollingsworth, Kevin W. Bowyer, Patrick J. Flynn: "Image Averaging for Improved Iris Recognition", Advances in Biometrics, by M. Tistarelli, M.S. Nixon (Eds.): ICB 2009, LNCS 5558, Springer Berlin Heidelberg, Berlin Heidelberg, Jun. 2, 2009, pp. 1112-1121, XP019117900, ISBN: 978-3-642-01792-6.
Kouzani, A Z, Illumination-effects compensation in facial images, IEEE International Conference on Systems, Man, and Cybernetics, 1999, IEEE SMS '99 Conference Proceedings, Tokyo, Japan Oct. 12-15, 1999, vol. 6, pp. 840-844.
M. E. Moghaddam, Out of focus blur estimation using genetic algorithm, in Proc. 15th International Conf. On Systems, Signals and Image Processing, IWSSIP 2008, pp. 417-420.
Markus Stricker and Markus Orengo, Similarity of color images, SPIE Proc., vol. 2420, pp. 381-392, 1995.
Mitra S, et al., Gaussian Mixture Models Based on the Frequency Spectra for Human Identification and Illumination Classification, 4th IEEE Workshop on Automatic Identification Advanced Technologies, 2005, Buffalo, NY, USA Oct. 17-18, 2005, pp. 245-250.
Mohsen Ebrahimi Moghaddam, Out of Focus Blur Estimation Using Genetic Algorithm, Journal of Computer Science 4 (4): 298-304, 2008, Science Publications, ISSN 1549-3636.
Paul Viola and Michael J. Jones: Robust Real-Time Face Detection, International Journal of Computer Vision 57(2), 137-154, 2004 Kluwer Academic Publishers. Manufactured in the Netherlands, pp. 137-154.
Paul Viola and Michael Jones: Rapid Object Detection using a Boosted Cascade of Simple Features, 2001 IEEE, pp. I-511-I-518.
PCT Communication in Cases for Which No Other Form is Applicable, Form PCT/ISA/224, for PCT Application No. PCT/EP2011/069904, dated Jun. 22, 2012, 1 Page.
PCT Notification of Transmittal of International Preliminary Report on Patentability Chapter I, International Preliminary Report on Patentability Chapter I, for PCT Application No. PCT/EP2011/066835, report dated Apr. 11, 2013, 9, Pages.
PCT Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration, for PCT Application No. PCT/EP2011/069904, report dated Jun. 22, 2012, 21 Pages.
PCT Notification of Transmittal of the International Search Report and the Written Opinion of the International Searching Authority, or the Declaration, for PCT Application No. PCT/EP2011/069904, report dated May 16, 2012, 17 Pages.
S. J. Wan, P. Prusinkiewicz, S. L. M. Wong, Variance-based color image quantization for frame buffer display, Color Research and Application, vol. 15, No. 1, pp. 52-58, 1990.
Sang Ku Kim, Sang Rae Park and Joan Ki Paik: Simultaneous Out-Of-Focus Blur Estimation and Restoration for Digital Auto-Focusing System, 1998-08-01, vol. 44, No. 3, Aug. 1, 1998, pp. 1071-1075, XP011083715.
Tianhorng Chang and C.-C. Jay Kuo, Texture Analysis and Classification with Tree-Structured Wavelet Transform, IEEE Trans. Image Processing, vol. 2. No. 4, Oct. 1993.
Tjahyadi et al., Application of the DCT Energy Histogram for Face Recognition, in Proceedings of the 2nd International Conference on Information Technology for Application (ICITA 2004), pp. 305-310, 2004.
Turk, et al., Eigenfaces for Recognition, Journal of Cognitive Neuroscience, vol. 3, No. 1, 1991.
Wei Huang, Zhongliang Jing, Evaluation of focus measures in multi-focus image fusion, Pattern Recognition Letters, Mar. 2007, vol. 28, No. 4, p. 493-500.
Wikipedia reference: Autofocus, retrieved on Feb. 3, 2011, URL: http://en.wikipedia.org/wiki/Autofocus, 5 Pages.
Zhang Lei, Lin Fuzong, Zhang Bo, A CBIR method based on color-spatial feature. IEEE Region 10th Ann. Int. Conf. 1999, TENCON'99, Cheju, Korea, 1999.

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130044923A1 (en) * 2008-12-05 2013-02-21 DigitalOptics Corporation Europe Limited Face Recognition Using Face Tracker Classifier Data
US8977011B2 (en) * 2008-12-05 2015-03-10 Fotonation Limited Face recognition using face tracker classifier data
US20140139721A1 (en) * 2012-11-12 2014-05-22 Samsung Electronics Co., Ltd. Method and apparatus for shooting and storing multi-focused image in electronic device
US9319580B2 (en) * 2012-11-12 2016-04-19 Samsung Electronics Co., Ltd. Method and apparatus for shooting and storing multi-focused image in electronic device
US20140226858A1 (en) * 2013-02-14 2014-08-14 Samsung Electronics Co., Ltd. Method of tracking object using camera and camera system for object tracking

Also Published As

Publication number Publication date Type
JP2013242571A (en) 2013-12-05 application
EP3236391A1 (en) 2017-10-25 application
KR20130139243A (en) 2013-12-20 application
JP6081297B2 (en) 2017-02-15 grant
EP2548154B1 (en) 2015-10-21 grant
US20140193088A1 (en) 2014-07-10 application
CN103052960A (en) 2013-04-17 application
KR101870853B1 (en) 2018-06-25 grant
US8648959B2 (en) 2014-02-11 grant
EP3007104B1 (en) 2017-07-19 grant
JP5530568B2 (en) 2014-06-25 grant
WO2012062893A3 (en) 2012-07-12 application
JP2014505915A (en) 2014-03-06 application
EP2548154A2 (en) 2013-01-23 application
EP3007104A1 (en) 2016-04-13 application
WO2012062893A2 (en) 2012-05-18 application
US20120120283A1 (en) 2012-05-17 application
CN103052960B (en) 2017-10-24 grant

Similar Documents

Publication Publication Date Title
US7110575B2 (en) Method for locating faces in digital color images
US7062086B2 (en) Red-eye detection based on red region detection with eye confirmation
US6940545B1 (en) Face detecting camera and method
US20050129311A1 (en) Object detection
US20100026832A1 (en) Automatic face and skin beautification using face detection
US20060008173A1 (en) Device and method for correcting image including person area
US20110002506A1 (en) Eye Beautification
US20080166026A1 (en) Method and apparatus for generating face descriptor using extended local binary patterns, and method and apparatus for face recognition using extended local binary patterns
US20100054592A1 (en) Analyzing partial face regions for red-eye detection in acquired digital images
US20090109304A1 (en) Image processing device, image processing method, and computer program product
Wong et al. Saliency-enhanced image aesthetics class prediction
US7555148B1 (en) Classification system for consumer digital images using workflow, face detection, normalization, and face recognition
US7564994B1 (en) Classification system for consumer digital images using automatic workflow and face detection and recognition
US20110091113A1 (en) Image processing apparatus and method, and computer-readable storage medium
US7551755B1 (en) Classification and organization of consumer digital images using workflow, and face detection and recognition
US7558408B1 (en) Classification system for consumer digital images using workflow and user interface modules, and face detection and recognition
US7587068B1 (en) Classification database for consumer digital images
US8213737B2 (en) Digital image enhancement with reference images
US20080316328A1 (en) Foreground/background separation using reference images
US20050129331A1 (en) Pupil color estimating device
US20080317378A1 (en) Digital image enhancement with reference images
US20070263928A1 (en) Method, apparatus, and program for processing red eyes
US20080317357A1 (en) Method of gathering visual meta data using a reference image
US20080218603A1 (en) Imaging apparatus and control method thereof
US20100141786A1 (en) Face recognition using face tracker classifier data

Legal Events

Date Code Title Description
AS Assignment

Owner name: FOTONATION LIMITED, IRELAND

Free format text: CHANGE OF NAME;ASSIGNOR:DIGITALOPTICS CORPORATION EUROPE LIMITED;REEL/FRAME:034524/0882

Effective date: 20140609

MAFP

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551)

Year of fee payment: 4