US20130021490A1

US20130021490A1 - Facial Image Processing in an Image Capture Device

Info

Publication number: US20130021490A1
Application number: US13/281,521
Authority: US
Inventors: Geraint James; Efrat Swissa
Original assignee: Broadcom Corp
Current assignee: Avago Technologies International Sales Pte Ltd
Priority date: 2011-07-20
Filing date: 2011-10-26
Publication date: 2013-01-24
Also published as: US20130021489A1; US20130021488A1; US20130021483A1; US20130021491A1; US20130021484A1; US20130021504A1; US20130021512A1; US9092861B2; US20130022116A1

Abstract

Disclosed are various embodiments of applying image processing techniques in an image capture device to faces within an image. Faces and/or facial features can be detected within an image. A facial signature can be calculated that corresponds to the face within the image. A facial profile corresponding to the facial signature can reference one or more image processing techniques. The image processing techniques can be applied to the region of the image in which the face is located.

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to co-pending U.S. provisional application entitled, “Image Capture Device Systems and Methods,” having Ser. No. 61/509,747, filed Jul. 20, 2011, which is entirely incorporated herein by reference.

BACKGROUND

Image capture devices (e.g., still cameras, video cameras, etc.) can apply various image processing techniques. These techniques can be applied globally or, in other words, to an entire image. Images captured by an image capture device can often contain various objects and/or subjects such that application of a single image processing technique to the entirety of the image can result in a less than desirable result.

BRIEF DESCRIPTION OF THE DRAWINGS

Many aspects of the invention can be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present invention. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the several views.

FIGS. 1A-1B are drawings of a mobile device incorporating an image capture device according to various embodiments of the disclosure.

FIG. 2 is a drawing of an image capture device that can be incorporated into the mobile device of FIG. 1 according to various embodiments of the disclosure.

FIGS. 3-4 are drawings of an example image that can be captured and processed by an image capture device of FIG. 2 according to various embodiments of the disclosure.

FIGS. 5-7 are drawings of an example image processing techniques that can be performed on an image captured by an image capture device of FIG. 2 according to various embodiments of the disclosure.

FIG. 8 is a flowchart depicting one example of a process that can be executed in an image capture device according to various embodiments of the disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure relate to systems and methods that can be executed in an image capture device. More specifically, embodiments of the present disclosure further comprise systems and methods that can implement facial beautification techniques to faces and/or facial regions that can be detected in images and/or video captured by an image capture device. An image capture device can include a camera, video camera, a mobile device with an integrated image capture device, or other devices suitable to capturing imagery and/or video as can be appreciated. In some embodiments, an image capture device according to an embodiment of the disclosure can include a device such as a smartphone, tablet computing system, laptop computer, desktop computer, or any other computing device that has the capability to receive and/or capture imagery via image capture hardware.
Accordingly, image capture device hardware can include components such as lenses, image sensors (e.g., charge coupled devices, CMOS image sensor, etc.), processor(s), image signal processor(s), a main processor, memory, mass storage, or any other hardware or software components that can facilitate capture of imagery and/or video. In some embodiments, an image signal processor can be incorporated as a part of a main processor in an image capture device module that is in turn incorporated into a device having its own processor, memory and other components.
An image capture device according to an embodiment of the disclosure can provide a user interface via a display that is integrated into the image capture device. The display can be integrated with a mobile device, such as a smartphone and/or tablet computing device, and can include a touchscreen input device (e.g., a capacitive touchscreen, etc.) with which a user may interact with the user interface that is presented thereon. The image capture device hardware can also include one or more buttons, dials, toggles, switches, or other input devices with which the user can interact with software executed in the image capture device.
Referring now to the drawings, FIGS. 1A-1B show a mobile device 102 that can comprise and/or incorporate an image capture device according to various embodiments of the disclosure. The mobile device 102 may comprise, for example, a processor-based system, such as a computer system. Such a computer system may be embodied in the form of a desktop computer, a laptop computer, a personal digital assistant, a mobile device (e.g., cellular telephone, smart phone, etc.), tablet computing system, set-top box, music players, or other devices with like capability. The mobile device can include, for example, an image capture device 104, which can further include a lens system as well as other hardware components that can be integrated with the device to facilitate image capture. The mobile device 102 can also include a display device 141 upon which various content and other user interfaces may be rendered. The mobile device 102 can also include one or more input devices with which a user can interact with a user interface rendered on the display device 141. For example, the mobile device 102 can include or be in communication with a mouse, touch input device (e.g., capacitive and/or resistive touchscreen incorporated with the display device 141), keyboard, or other input devices.
The mobile device 102 may be configured to execute various applications, such as a camera application that can interact with an image capture module that includes various hardware and/or software components that facilitate capture and/or storage of images and/or video. In one embodiment, the camera application can interact with application programming interfaces (API's) and/or other software libraries and/or drivers that are provided for the purpose interacting with image capture hardware, such as the lens system and other image capture hardware. The camera application can be a special purpose application, a plug-in or executable library, one or more API's, image control algorithms, image capture device firmware, or other software that can facilitate communication with image capture hardware in communication with the mobile device 102. Accordingly, a camera application according to embodiments of the present disclosure can capture imagery and/or video via the various image capture hardware as well as facilitate storage of the captured imagery and/or video in memory and/or mass storage associated with the mobile device 102.
FIG. 2 illustrates an embodiment of the various image capture components, or one example of an image capture device 104, that can be incorporated in the mobile device 102 illustrated in FIGS. 1A-1B. Although one implementation is shown in FIG. 2 and described herein, an image capture device according to an embodiment of the disclosure more generally comprises an image capture device that can provide images in digital form.
The image capture device 104 includes a lens system 200 that conveys images of viewed scenes to an image sensor 202. By way of example, the image sensor 202 comprises a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor that is driven by one or more sensor drivers 204. The analog image signals captured by the sensor 202 are provided to an analog-to-digital (ND) converter 206 for conversion into binary code that can be processed by a processor 208. The processor can also execute a facial processing application 151 that can identify one or more faces appearing in an image and perform facial beautification image processing techniques on the face. In some embodiments, the facial processing application 151 can take the form of API's, firmware, or other software accessible to the image capture device 104 and/or a mobile device 102 or other system in which the image capture device 104 is integrated.
Operation of the sensor driver(s) 204 is controlled through a camera controller 210 that is in bi-directional communication with the processor 208. In some embodiments, the controller 210 can control one or more motors 212 that are used to drive the lens system 200 (e.g., to adjust focus, zoom, and/or aperture settings). The controller 210 can also communicate with a flash system, user input devices (e.g., buttons, dials, toggles, etc.) or other components associated with the image capture device 104. Operation of the camera controller 210 may be adjusted through manipulation of a user interface. A user interface comprises the various components used to enter selections and commands into the image capture device 104 and therefore can include various buttons as well as a menu system that, for example, is displayed to the user in, for example, a camera application executed on a mobile device 102 and/or on a back panel associated with a standalone digital camera.
The digital image signals are processed in accordance with instructions from an image signal processor 218 that can be implemented as a standalone processor within the image capture device as well as being a part of the processor 208. Processed (e.g., compressed) images may then be stored in storage memory, such as that contained within a removable solid-state memory card (e.g., Flash memory card). The embodiment shown in FIG. 2 further includes a device interface 224 through which the image capture device 104 can communicate with a mobile device or other computing system in which it may be integrated. For example, the device interface 224 can allow the image capture device to communicate with a main processor associated with a mobile device as well as memory, mass storage, or other resources associated with the mobile device. The device interface 224 can communicate with a mobile device in various communications protocols, and this communication can be facilitated, at a software level, by various device drivers, libraries, API's or other software associated with the image capture device 104 that is executed in the mobile device.
An image capture device according to an embodiment of the disclosure can apply various image processing techniques to various regions of an image that can be associated with a particular region type. The identification and characterization of regions within captured imagery as well as application of various image processing techniques can be accomplished via software executed by the processor 208, the ISP 218 as well as a processor associated with a device in communication with the image capture device 104. It should be appreciated that the specific implementation and/or embodiments disclosed herein are merely examples.
As noted above, the facial processing application 151 can perform various image processing techniques on faces that are identified in an image captured by the image capture device 104. A user can specify one or more image processing techniques that can be applied to the faces identified in the image. These image processing techniques can include scar removal algorithms, blemish removal algorithms, color correction techniques, or any other image processing technique that can be employed to modify the appearance of a face in an image captured by the image capture device 104. In one embodiment, the facial processing application 151 can analyze an image captured by the image capture device and determine whether there are objects that correspond to facial features. An image processing technique can also include an image processing operation that alters a dimension of the face and/or facial features within the face. For example, such a technique can include an alteration of the shape and/or size of a nose, forehead, chin, lip, cheeks, or other portion of the face. For example, the technique can include modifying the dimensions of the face, or altering the appearance of facial bone structures. Another example of an image processing technique can include a technique that applies, enhances, or alters color of a region of the face and/or skin. For example, the facial processing application 151 can enhance and/or modify colorization of a subject's cheeks, whiten teeth, or other portions of the face within the image. As yet another example, the image processing technique can include removal of grey hairs that may appear in an image by adjusting a color of a hair region. Therefore, an image processing technique can also include, but is not limited to, the application of one or more signal processing techniques, filters, or any other process that receives as an input image data associated with a region and outputs image data that is altered or modified in some form.
Accordingly, the facial processing application 151 can detect whether an image captured by the image capture device 104 contains a face. In some embodiments, the facial processing application 151 can employ object detection algorithms, edge detection algorithms, or other image recognition techniques on the captured image to identify whether one or more objects within an image corresponds a head and/or face. The facial processing application 151 can then employ one or more facial recognition algorithms to determine whether the face(s) detected in the image correspond to a facial profile that can specify one or more image processing techniques that can be applied to a region of the image that corresponds to the location of the face within the image. In some embodiments, the facial processing application 151 can calculate a facial signature, which can comprise a digital representation of a face depicted within an image that allows the face to be uniquely identified.
Accordingly, a facial profile can contain the facial signature as well as references to one or more image processing techniques that the facial processing application 151 can apply to the face. In this way, the facial processing application 151 can generate a resultant image that includes the processed face. To facilitate the above functionality, the facial processing application 151 can generate a user interface that allows the user to specify image processing techniques to apply to a given face depicted within an image. The facial processing application 151 can store a reference to a facial signature as well as the image processing techniques selected by the user in a memory accessible to the image capture device. The facial profile can also store a location within a facial at which a specified image processing technique should be applied. In some embodiments, facial profiles can be stored in memory of a mobile device 102 in which the image capture device 104 is integrated. Therefore, the facial processing application 151 can apply the selected image processing techniques to the face in subsequent images that contain the face when it can be identified by its facial signature. Additional examples and variations are provided in the discussion that accompanies the following drawings.
Accordingly, reference is now made to FIG. 3, which illustrates an example image that can be captured by the image capture device 104 (FIG. 2) according to various embodiments according to the disclosure. In the depicted non-limiting examples of FIGS. 3-4, the image capture device 104 is incorporated into a mobile device 102, which can execute a camera application that renders a user interface for display on a display device associated with the mobile device 102. It should be appreciated that this is only one non-limiting illustrative implementation.
Therefore, FIG. 3 illustrates an example of an image 300 that can be captured by the image capture device. As one example, the image 300 can be captured via a camera application executed on a mobile device where the camera application is configured to communicate with API's associated with the image capture device for the purposes of initiating capture of imagery, display of imagery on a display of the mobile device as well as storage of captured imagery in the form of still images and/or video in memory or mass storage associated with the mobile device.
FIG. 4 continues the example of FIG. 3 by illustrating an example of a face that can be detected by the facial processing application 151. In one embodiment, the facial processing application 151 can be invoked when an image is captured by the image capture device 104. Accordingly, the facial processing application 151 can calculate a facial signature to determine whether a facial profile associated with the face is stored in a memory accessible to the facial processing application 151. As noted above, a facial profile can specify one or more image processing techniques that can be applied to at least a portion of the region of the image. In this sense, a facial profile can represent a preconfigured set of one of more image processing techniques that can be applied to the face 302 when the image capture device 104 captures an image that contains the face 302 having a facial signature that matches a signature associated with a facial profile accessible to the facial processing application 151. Additionally, the facial processing application 151 can also process frames of a video in which a face corresponding to a facial signature associated with a facial profile by applying the image processing techniques that are specified by the profile. In this way, the facial processing application 151 can apply the image processing techniques specified by a facial profile to a face that appears in a video when the face can be identified as matching a facial signature associated with the profile.
Continuing the example of FIG. 4, reference is now made to FIGS. 5-6, which illustrate an example of a region 506 in the image 300 in which an image processing technique can be applied. In the depicted example, when the facial processing application 151 detects that the face corresponds to a facial profile by calculating a facial signature and determining that the signature matches a profile accessible to the facial processing application 151, it can apply the specified image processing techniques. In FIGS. 5-6, the facial profile can specify a blemish removal algorithm that can be applied to the region. Additionally, in subsequent images captured by the image capture device 104 that contain a face with a facial signature that corresponds to the facial profile, the facial processing application 151 can apply the same image processing techniques that are specified by the facial profile. FIG. 7 illustrates how additional image processing techniques can be specified by the facial profile, such as those that can alter the dimensions of a face and/or regions of the face.
Referring next to FIG. 8, shown is a flowchart that provides one example of the operation of a portion of a facial processing application 151 executed by an image capture device 104, a mobile device 102 or any other device in which an image capture device 104 is integrated according to various embodiments of the disclosure. It is understood that the flowchart of FIG. 8 provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the portion of logic employed by the image capture device as described herein. As an alternative, the flowchart of FIG. 8 may be viewed as depicting an example of steps of a method implemented in a computing device, processor, or other circuits according to one or more embodiments.
First, in box 801, image capture can be initiated in the image capture device so that one or more images are captured by the lens system, image sensor, and other image capture device hardware as discussed above. In box 803, the facial processing application 151 can determine whether a face is present within the captured image. If, in box 803, the facial processing application 151 determines that a face is present in the image, a facial signature associated with the face can be calculated in box 805. In box 807, the facial processing application 151 can determine whether the face corresponds to a facial profile accessible to the facial processing application 151. If so, then in box 815, the facial processing application 151 applies the image processing techniques that are referenced by the corresponding facial profile. If not, then in box 809, the facial processing application can facilitate generating a user interface that prompts a user to create a facial profile corresponding to the face detected in the captured image.
If, in box 811, the user opts to create such a profile, then in box 813 the facial processing application 151 can store the profile in a memory accessible to the image capture device 104. Subsequently, the facial processing application 151 can apply the image processing techniques specified by the facial profile. In some embodiments, the facial processing application 151 can prompt a user prior to applying the image processing techniques specified by the facial profile. Additionally, in some embodiments, the facial processing application 151 can apply the image processing techniques and modify the image data captured by the image capture device 104. In other embodiments, the facial processing application 151 can generate a copy of the image and apply the image processing techniques to the copy, retaining the originally captured image data. In yet another embodiment, the image processing application 151 can apply the image processing techniques by including a reference to the image processing technique and the location within the image to which it is applied in meta data associated and stored with the image data.
Embodiments of the present disclosure can be implemented in various devices, for example, having a processor, memory as well as image capture hardware that can be coupled to a local interface. The logic described herein can be executable by one or more processors integrated with a device. In one embodiment, an application executed in a computing device, such as a mobile device, can invoke API's that provide the logic described herein as well as facilitate interaction with image capture hardware. Where any component discussed herein is implemented in the form of software, any one of a number of programming languages may be employed such as, for example, processor specific assembler languages, C, C++, C#, Objective C, Java, Javascript, Perl, PHP, Visual Basic, Python, Ruby, Delphi, Flash, or other programming languages.
As such, these software components can be executable by one or more processors in various devices. In this respect, the term “executable” means a program file that is in a form that can ultimately be run by a processor. Examples of executable programs may be, for example, a compiled program that can be translated into machine code in a format that can be loaded into a random access portion of memory and run by a processor, source code that may be expressed in proper format such as object code that is capable of being loaded into a random access portion of the memory and executed by the processor, or source code that may be interpreted by another executable program to generate instructions in a random access portion of the memory to be executed by the processor, etc. An executable program may be stored in any portion or component of the memory including, for example, random access memory (RAM), read-only memory (ROM), hard drive, solid-state drive, USB flash drive, memory card, optical disc such as compact disc (CD) or digital versatile disc (DVD), floppy disk, magnetic tape, or other memory components.
Although various logic described herein may be embodied in software or code executed by general purpose hardware as discussed above, as an alternative the same may also be embodied in dedicated hardware or a combination of software/general purpose hardware and dedicated hardware. If embodied in dedicated hardware, each can be implemented as a circuit or state machine that employs any one of or a combination of a number of technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates for implementing various logic functions upon an application of one or more data signals, application specific integrated circuits having appropriate logic gates, or other components, etc. Such technologies are generally well known by those skilled in the art and, consequently, are not described in detail herein.
The flowchart of FIG. 8 shows the functionality and operation of an implementation of portions of an image capture device according to embodiments of the disclosure. If embodied in software, each block may represent a module, segment, or portion of code that comprises program instructions to implement the specified logical function(s). The program instructions may be embodied in the form of source code that comprises human-readable statements written in a programming language or machine code that comprises numerical instructions recognizable by a suitable execution system such as a processor in a computer system or other system. The machine code may be converted from the source code, etc. If embodied in hardware, each block may represent a circuit or a number of interconnected circuits to implement the specified logical function(s).
Although the flowchart of FIG. 8 shows a specific order of execution, it is understood that the order of execution may differ from that which is depicted. For example, the order of execution of two or more blocks may be scrambled relative to the order shown. Also, two or more blocks shown in succession in FIG. 8 may be executed concurrently or with partial concurrence. Further, in some embodiments, one or more of the blocks shown in FIG. 8 may be skipped or omitted. In addition, any number of counters, state variables, warning semaphores, or messages might be added to the logical flow described herein, for purposes of enhanced utility, accounting, performance measurement, or providing troubleshooting aids, etc. It is understood that all such variations are within the scope of the present disclosure.
Also, any logic or application described herein that comprises software or code can be embodied in any non-transitory computer-readable medium for use by or in connection with an instruction execution system such as, for example, a processor in a computer device or other system. In this sense, the logic may comprise, for example, statements including instructions and declarations that can be fetched from the computer-readable medium and executed by the instruction execution system. In the context of the present disclosure, a “computer-readable medium” can be any medium that can contain, store, or maintain the logic or application described herein for use by or in connection with the instruction execution system. The computer-readable medium can comprise any one of many physical media such as, for example, magnetic, optical, or semiconductor media. More specific examples of a suitable computer-readable medium would include, but are not limited to, magnetic tapes, magnetic floppy diskettes, magnetic hard drives, memory cards, solid-state drives, USB flash drives, or optical discs. Also, the computer-readable medium may be a random access memory (RAM) including, for example, static random access memory (SRAM) and dynamic random access memory (DRAM), or magnetic random access memory (MRAM). In addition, the computer-readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other type of memory device.
It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations set forth for a clear understanding of the principles of the disclosure. Many variations and modifications may be made to the above-described embodiment(s) without departing substantially from the spirit and principles of the disclosure. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.

Claims

1. An image capture device, comprising:

at least one image sensor; and

an application executed in the image capture device, the application comprising:

logic that initiates capture of at least one image via the at least one image sensor associated with the image capture device;

logic that detects a face within the image;

logic that calculates a facial signature associated with the face;

logic that identifies a facial profile accessible to the image capture device corresponding to the facial signature, the facial profile specifying at least one image processing technique; and

logic that applies the at least one image processing technique to at least a portion of the face within the image.

2. The image capture device of claim 1, wherein the image capture device is configured to capture a plurality of images associated with a plurality of frames of a video, and the application further comprises:

logic that tracks the face in subsequent frames of the video; and

logic that applies the image processing technique associated with the face in the subsequent frames.

3. The image capture device of claim 1, wherein the at least one image processing technique further comprises at least one of: a wrinkle reduction algorithm, a blemish removal algorithm, whitening of teeth appearing in the image, adjusting colorization of hair appearing in the image, altering dimensions of the face, and a skin tone correction algorithm.

4. The image capture device of claim 1, wherein the at least one image processing technique further comprises adjusting at least one dimension associated with the face.

5. The image capture device of claim 4, wherein the at least one dimension is associated with at least one sub-region of the face, the sub-region being one of: a nose, a forehead, a cheek, a lip, and a chin.

6. The image capture device of claim 1, wherein the at least one image processing technique further comprises increasing at least one color level associated with a portion of the face.

7. The image capture device of claim 6, wherein the application further comprises:

logic that identifies at least one cheek region associated with the face; and

the image processing technique further comprises increasing the at least one color level in the at least one cheek region.

8. The image capture device of claim 1, wherein the application further comprises:

logic that generates a user interface allowing a user to specify the at least one image processing technique associated with the face;

logic that associates the facial signature and a reference corresponding to the at least one image processing technique with the facial profile; and

logic that stores the facial profile in a memory accessible to the image capture device.

9. The image capture device of claim 8, wherein the logic that stores the facial profile further comprises logic that stores the facial profile in a mobile device memory associated with a mobile device in which the image capture device is integrated.

10. The image capture device of claim 1, wherein the logic that applies the at least one image processing technique to at least a portion of the face within the image further comprises logic that stores a reference associated with the at least one image processing technique in meta data associated with the image captured by the image capture device.

11. A method executed in an image capture device, comprising the steps of:

initiating capture of at least one image via at least one image sensor associated with the image capture device;

detecting an object corresponding to a face within the image captured by the image capture device;

calculating a facial signature associated with the face;

identifying a facial profile accessible to the image capture device corresponding to the facial signature, the facial profile specifying at least one image processing technique; and

applying the at least one image processing technique to at least a portion of the face within the image.

12. The method of claim 11, wherein the image capture device is configured to capture a plurality of images associated with a plurality of frames of a video, and the method further comprises the steps of:

tracking the face in subsequent frames of the video; and

applying the image processing technique associated with the face in the subsequent frames.

13. The method of claim 11, wherein the at least one image processing technique further comprises at least one of: a wrinkle reduction algorithm, a blemish removal algorithm and a skin tone correction algorithm.

14. The method of claim 11, wherein the at least one image processing technique further comprises adjusting at least one dimension associated with the face.

15. The method of claim 14, wherein the at least one dimension is associated with at least one sub-region of the face, the sub-region being one of: a nose, a forehead, a cheek, a lip, and a chin.

16. The method of claim 11, wherein the at least one image processing technique further comprises increasing at least one color level associated with a portion of the face.

17. The image capture device of claim 16, further comprising the step of identifying at least one cheek region associated with the face, wherein the image processing technique further comprises increasing the at least one color level in the at least one cheek region.

18. The method of claim 11, further comprising the steps of:

generating a user interface allowing a user to specify the at least one image processing technique associated with the face;

associating the facial signature and a reference corresponding to the at least one image processing technique with the facial profile; and

storing the facial profile in a memory accessible to the image capture device.

19. The method of claim 18, wherein the step of storing the facial profile further comprises the step of storing the facial profile in a mobile device memory associated with a mobile device in which the image capture device is integrated.

20. A system, comprising:

means for capturing at least one image via an image sensor means;

means for detecting an object corresponding to a face within the image;

means for calculating a facial signature associated with the face;

means for identifying a facial profile accessible to the image capture device corresponding to the facial signature, the facial profile specifying at least one image processing technique; and

means for applying the at least one image processing technique to at least a portion of the face within the image.