JP2012085083A - Image processing apparatus, image pickup device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus having a technology for detecting a desired face portion in an accurate and easy manner irrespective of the effect of an exposure condition or white balance processing.SOLUTION: An image processing apparatus includes: a face detection part that recognizes a face of an object from an object image having multiple color components and detects the face image; an extraction part that extracts the face portion of a processing object based on information about the multiple color components included in the face image; and a color correction part that corrects the color component value of an area corresponding to the face portion in the object image.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  Conventionally, various techniques have been developed to improve the appearance of a person who is a subject in an image captured by a digital camera or the like. For example, a white image region is detected together with a tooth image region from a face image, and a correction parameter for correcting the saturation of the tooth image region is calculated based on the saturation of the white image region. There is a technique for correcting the color of the image area of the tooth so as not to cause (see Patent Document 1).

JP 2008-160474 A

  By the way, in the conventional technique, an image region of a facial part such as a tooth or a white eye is detected from a face image based on an edge image generated by any one of luminance, saturation, and hue of the image. However, since the generated edge image is greatly influenced by the exposure condition at the time of imaging and the setting of white balance processing, a desired face part may not be detected accurately.

  In view of the above-described problems of the related art, an object of the present invention is to provide a technique that can easily detect a desired face part with high accuracy regardless of the influence of exposure conditions and white balance processing.

  In order to solve the above problems, an image processing apparatus of the present invention recognizes a face of a subject from a target image having a plurality of color components and detects a face image, and a plurality of colors included in the face image. An extraction unit that extracts a facial part to be processed based on component information, and a color correction unit that corrects a value of a color component in a region corresponding to the facial part in the target image.

  The extraction unit may extract a facial part based on information on hue and saturation.

  Further, when the facial part is a tooth, the extraction unit extracts a lip image region including the tooth based on the hue, and the saturation is calculated based on multivariate analysis among the lip image region. An image area that is less than or equal to the value may be extracted as a tooth image area.

  The extraction unit may extract a facial part based on information on luminance and saturation.

  In addition, when the facial part is a white eye, the extraction unit extracts an image region of the eye including the white eye based on a comparison between the luminance and the first threshold value that dynamically changes. Alternatively, an image area whose saturation is equal to or less than the second threshold value calculated based on multivariate analysis may be extracted as an image area for white eyes.

  Further, the color correction unit may correct the value of the color component in the region corresponding to the face part based on the information on the luminance and the saturation.

  An imaging apparatus of the present invention includes an imaging unit that captures a subject and acquires a target image having a plurality of color components, and the image processing apparatus of the present invention.

  An image processing program of the present invention includes an input procedure for reading a target image having a plurality of color components, a face detection procedure for recognizing a face of a subject from the target image and detecting a face image, and a plurality of color components included in the face image. Based on the information, the computer is caused to execute an extraction procedure for extracting the face part to be processed and a color correction procedure for correcting the value of the color component of the region corresponding to the face part in the target image.

  According to the present invention, a desired face part can be easily detected with high accuracy regardless of the influence of exposure conditions and white balance processing.

1 is a block diagram illustrating an example of a configuration of a digital camera 1 according to an embodiment of the present invention. The flowchart which shows the processing operation by the digital camera 1 The figure which shows an example of the target image 30 imaged with the digital camera 1. The figure which shows the process result with respect to the partial image 40 cut out including a mouth part The figure which shows the process result with respect to the partial image 70 cut out including the left eye

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating an example of a configuration of a digital camera 1 according to the present embodiment.

  The digital camera 1 according to the present embodiment includes an imaging optical system 11, an imaging element 12, a DFE 13, a CPU 14, a memory 15, an operation unit 16, a monitor 17, and a media I / F 18. Here, the DFE 13, the memory 15, the operation unit 16, the monitor 17, and the media I / F 18 are each connected to the CPU 14.

  The image sensor 12 is a device that captures a subject image formed by a light beam that has passed through the imaging optical system 11. The output of the image sensor 12 is connected to the DFE 13. Note that the image sensor 12 of the present embodiment may be a progressive scanning solid-state image sensor (CCD or the like) or an XY address type solid-state image sensor (CMOS or the like).

  A plurality of light receiving elements are arranged in a matrix on the light receiving surface of the imaging element 12. In each light receiving element of the image sensor 12, red (R), green (G), and blue (B) color filters are arranged according to a known Bayer array. Therefore, each light receiving element of the imaging element 12 outputs an image signal corresponding to each color by color separation in the color filter. Thereby, the image sensor 12 can acquire a color image at the time of imaging.

  The DFE 13 is a digital front-end circuit that performs signal processing such as A / D conversion of image signals input from the image sensor 12 and correction of defective pixels. In this embodiment, the DFE 13 constitutes an image pickup unit together with the image pickup device 12 and outputs an image signal input from the image pickup device 12 to the CPU 14 as image data.

  The CPU 14 is a processor that comprehensively controls each unit of the digital camera 1. For example, the CPU 14 reads and executes a control program from the memory 15, and performs autofocus (AF) control by known contrast detection, known automatic exposure (AE) calculation, and the like based on the output of the image sensor 12. Further, the CPU 14 performs image processing such as digital processing on the image data from the DEF 13 based on the image processing program. As an example, digital processing includes interpolation processing, white balance processing, gradation conversion processing, contour enhancement processing, color conversion processing, and the like. Furthermore, the CPU 14 of this embodiment operates as a face detection unit 20, an image extraction unit 21, a face part extraction unit 22, and a color correction unit 23 by executing the image processing program.

  The face detection unit 20 recognizes the face of the subject in the target image by using a known face recognition method such as edge detection for the captured image (target image), and detects the face image.

  The image extraction unit 21 performs edge detection on the face image detected by the face detection unit 20 as in the case of face recognition, thereby recognizing the face part to be processed and displaying a partial image including the part. cut.

  The face part extraction unit 22 calculates information on luminance, hue, and saturation from the pixel values of the color components R, G, and B of the partial image cut out by the image extraction unit 21, and based on the information, An image region of the facial part is extracted. In this embodiment, it is assumed that teeth and white eyes are used as face parts to be processed, and the face part extraction unit 22 extracts image areas using information on hue and saturation when extracting tooth image areas. When extracting an image area of a white eye, the image area is extracted using information on luminance and saturation. Note that the face part extraction unit 22 of this embodiment operates as an extraction unit together with the image extraction unit 21. In addition, as a method of calculating luminance, hue, and saturation information from the pixel values of the color components R, G, and B, a known conversion method such as HSV conversion is used.

  The color correction unit 23 is an image of the face part to be processed extracted by the face part extraction unit 22 in order to make the face part to be processed a desired color (for example, in the case of teeth or white eyes). Based on the region information, the pixel value of each color component in the corresponding region in the target image is corrected.

  The memory 15 is a non-volatile flash memory that stores a captured image and various programs such as an image processing program executed by the CPU 14.

  For example, the operation unit 16 receives an input of an imaging mode switching setting, an instruction to capture a still image or a moving image, and the like from the user. In addition, the operation unit 16 according to the present embodiment also receives from the user an instruction to set a facial part for performing color correction processing.

  The monitor 17 is a monitor such as a liquid crystal monitor and displays various images according to control instructions from the CPU 14. For example, after the image processing of the present embodiment, the monitor 17 displays the image that has been subjected to the image processing based on a control instruction from the CPU 14.

  A non-volatile storage medium 19 can be detachably connected to the media I / F 18. The media I / F 18 executes data writing / reading with respect to the storage medium 19. The storage medium 19 includes a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is illustrated as an example of the storage medium 19.

  Next, processing operations by the digital camera 1 according to the present embodiment will be described with reference to the flowchart of FIG.

  Note that, as described above, in this embodiment, it is assumed that face parts to be color-corrected are teeth and white eyes, and those face parts are color-corrected white. In addition, the digital camera 1 according to the present embodiment extracts the white image areas of the left and right eyes and corrects the colors. In the following, image processing for the white eye of the left eye will be described, but the same applies to the case of the right eye. In addition, the digital camera 1 according to the present embodiment receives and holds in advance setting instructions for setting teeth and white eyes as face parts for color correction from the user via the operation unit 16.

  When the CPU 14 receives an imaging instruction of a subject (for example, a full press operation of a release button included in the operation unit 16) from the user, the CPU 14 captures an image of the subject on the imaging device 12. The CPU 14 acquires the image signal output from the image sensor 12 via the DEF 13 and performs digital processing such as interpolation processing, white balance processing, gradation conversion processing, contour enhancement processing, and color conversion processing. Thereby, the target image 30 having pixel values of R, G, and B color components for each pixel is generated. FIG. 3 shows an example of the target image 30. The CPU 14 performs processes from step S101 to step S109 on the target image 30.

  Step S101: The face detection unit 20 of the CPU 14 recognizes the face of the subject in the target image 30 using a known face recognition method, and detects the face image 31 shown in FIG. The CPU 14 holds the detected face image 31 in an internal memory (not shown). Or CPU14 may hold | maintain the information regarding the face image 31 (an image size, a pixel position, etc.) in an internal memory (not shown).

  Step S102: The image extraction unit 21 of the CPU 14 performs edge detection on the face image 31 detected in step S101 in order to perform tooth color correction. The image extraction unit 21 recognizes the mouth portion including the teeth based on the detected edge amount, and cuts out the partial image 40 including the mouth portion. The CPU 14 holds the cut partial image 40 in an internal memory (not shown). FIG. 4A shows an example of the cut out partial image 40.

  Note that the size of the cut-out partial image 40 is determined according to the processing capability of the CPU 14 and the size of the mouth, such as 30% increase in the size of the mouth portion in the horizontal and vertical scanning directions in the target image 30. Is preferred.

  Step S103: The face part extracting unit 22 of the CPU 14 extracts a lip image area including teeth from the partial image 40 cut out in step S102. For this purpose, the face part extraction unit 22 calculates a hue value from the R, G, and B pixel values of each pixel of the partial image 40 in FIG. The hue image of is generated. The face part extraction unit 22 includes pixels having a red hue (for example, 290 ° to 400 ° (= 290 ° to 360 °, 0 ° to 40 °)) indicating the color of the lips in the generated hue image. Extract.

  The hue of 290 ° to 400 ° includes not only the lips but also the skin. Therefore, the face part extraction unit 22 of the present embodiment applies a multivariate analysis, for example, a discriminant analysis method, to the pixel value of the extracted pixel to obtain a value that maximizes the variance ratio of the interclass variance. The face part extraction unit 22 sets the maximum value as a threshold value, and extracts pixels having gradation values equal to or lower than the threshold value as pixels of the lip image. Thereby, it is possible to extract only the lip image excluding a part such as skin. FIG. 4B is a binarized image 50 of the partial image 40 showing the lip image extracted by the face part extracting unit 22 as a white region.

  Further, the face part extraction unit 22 applies a known convex hull or Snake method to the lip image of the binarized image 50 to extract a lip image area including a tooth image area. FIG. 4C is a binarized image 51 showing the image area of the lips extracted by the convex hull as a white area.

  In the present embodiment, the red hue is set to 290 ° to 400 °. However, it is preferably set according to the exposure condition, white balance processing, and the like. Further, when extracting a pixel having a gradation value equal to or smaller than a threshold value as a lip image, the face part extracting unit 22 removes a pixel that exists independently or an image region that is equal to or smaller than a predetermined area as noise. You may do it. For example, the predetermined area is preferably determined according to the size of the target image 30 or the partial image 40, the processing capability of the CPU 14, or the like, such as 2% of the number of pixels of the partial image 40.

  Step S104: The face part extraction unit 22 extracts a tooth image region using the partial image 40 cut out in step S102 and the binarized image 51 acquired in step S103. For this purpose, the face part extraction unit 22 generates a saturation image 60 from the partial image 40 using a known conversion method as in the case of hue (FIG. 4D). As shown in FIG. 4D, in the saturation image 60, the saturation of pixels such as highlight portions on the teeth and lips is lowered (black portion).

  The face part extraction unit 22 applies, for example, a discriminant analysis method to the saturation of the pixels of the saturation image 60 corresponding to the image area of the lips of the binarized image 51, and the variance ratio of the interclass variance is maximum. Find the value that The face part extraction unit 22 uses the maximum value as a threshold value, and extracts pixels having a saturation equal to or lower than the threshold value as pixels in the tooth image region. FIG. 4E is a binarized image 61 of the partial image 40 that shows the tooth image area extracted by the face part extraction unit 22 as a white area.

  Note that the facial part extraction unit 22 extracts a pixel having a saturation equal to or lower than a threshold value as a pixel in the tooth image region, and then performs a mask process using the binarized image 50 on the binarized image 61. By doing so, noise and highlights on the lips may be removed.

  Step S105: The color correction unit 23 of the CPU 14 corrects the teeth of the subject in the original target image 30 to white. For this purpose, for example, the color correction unit 23 of the present embodiment performs a smoothing process on the binarized image 61 in an area having a size of 3 pixels × 3 pixels, and generates a smoothed image 62 having a gradation of 0 to 255. (FIG. 4 (f)). The color correction unit 23 corrects the R, G, and B pixel values of the corresponding target image 30 by gamma conversion according to the size of the pixel value of the smooth image 62, and whites the subject's teeth. Correct the color. In this embodiment, the pixel values of the R, G, and B color components themselves are not gamma-converted, but a tone curve having a γ value larger than 1 such as 1.3 is applied to the luminance of the pixel. A tone curve having a γ value smaller than 1 such as 0.6 is applied to the saturation. The color correction unit 23 converts the gamma-converted luminance and saturation information into R, G, and B pixel values by a known conversion method and corrects them.

  Step S106: The image extraction unit 21 performs edge detection on the face image 31 detected in step S101 in order to perform white eye color correction. The image extraction unit 21 recognizes the left eye based on the detected edge amount, and cuts out the partial image 70 including the left eye. The CPU 14 holds the cut partial image 70 in an internal memory (not shown). FIG. 5A shows an example of the cut out partial image 70.

  Note that the size of the cut-out partial image 70 is determined according to the processing capability of the CPU 14 and the size of the eye, such as 30% increase in the size of the left eye in the horizontal and vertical scanning directions in the target image 30. preferable.

  Step S107: The face part extraction unit 22 extracts the image area of the left eye including the white from the partial image 70 cut out in step S106. For this purpose, the face part extraction unit 22 calculates the luminance from the R, G, and B pixel values of each pixel of the partial image 70 in FIG. Is generated. The face part extraction unit 22 binarizes the generated luminance image by applying a dynamic threshold method, for example, adaptive binarization processing, and narrows down the area of the left eye part (FIG. 5B). By using this adaptive binarization processing, a locally different threshold value (first threshold value) is provided for each part in the luminance image, and exposure conditions and white balance during imaging such as uneven illumination are provided. The luminance image can be binarized while avoiding the influence of processing and the like. FIG. 5B is a binarized image 80 of the partial image 70 showing the left eye portion narrowed down by the adaptive binarization process as a white region.

  However, the binarized image 80 includes image regions such as hair and eyebrows. Therefore, the face part extraction unit 22 removes, as noise, pixels in the binarized image 80, such as pixels that exist independently and pixels in an image region such as hair that is smaller than a predetermined area (FIG. 5C). The predetermined area in the present embodiment is preferably determined according to the size of the target image 30 and the partial image 70, the processing capability of the CPU 14, and the like, for example, 5% of the number of pixels of the partial image 70.

  Further, the face part extraction unit 22 performs binarized image 81 based on the ferret diameter representing the size of the left eye in the horizontal and vertical scanning directions, or the ratio of the ferret diameters (eg, ferret diameter ratio = 1). To obtain a binarized image 82 of only the left eye (FIG. 5D). The face part extraction unit 22 applies a known convex hull or Snake method to the binarized image 82 to extract the image area of the left eye including the image area of the white eye. FIG. 5E is a binarized image 83 that shows the image area of the left eye obtained by the convex hull as a white area.

  Note that when extracting an image area of an eye including a white eye using a known convex hull or the Snake method, the face part extraction unit 22 is likely to generate a strong edge at the boundary between the white eye and the pupil or the outline of the eye. Therefore, it is preferable to extract the image having the larger average value of the Sobel edges of luminance and saturation as the image area of the eye. Further, the ferret diameter and ferret diameter ratio of the left eye and the right eye are preferably set appropriately according to the subject such as an adult, a child, or a male and female, and are preferably stored in the memory 15.

  Step S108: The face part extraction unit 22 extracts a white eye image region using the partial image 70 cut out in step S106 and the binarized image 83 acquired in step S108. For this purpose, the face part extraction unit 22 generates a saturation image 90 from the partial image 70 using a known conversion method as in the case of teeth (FIG. 5 (f)). As shown in FIG. 5 (f), in the saturation image 90, the saturation of pixels such as white eye and highlight portion of the pupil is lowered (black portion).

  The face part extraction unit 22 applies, for example, a discriminant analysis method to the saturation of the pixels of the saturation image 90 corresponding to the image area of the left eye of the binarized image 83, and the variance ratio of the interclass variance is maximum. Find the value that The face part extraction unit 22 uses the maximum value as a threshold value (second threshold value), and extracts a pixel having a saturation equal to or lower than the threshold value as a pixel in the image region of the white of the left eye. FIG. 5G is a binarized image 91 of the partial image 70 showing the white eye image area extracted by the face part extracting unit 22 as a white area.

  Note that the CPU 14 also extracts the white eye image region of the right eye by performing the processing of step S106 to step S108 as in the case of the left eye.

  Step S109: The color correction unit 23 corrects the white color of the subject in the original target image 30 to white. For this purpose, for example, the color correction unit 23 of the present embodiment smoothes the binarized image 91 in an area having a size of 3 pixels × 3 pixels, and generates a smooth image 92 having a gradation of 0 to 255. (FIG. 5 (h)). The color correcting unit 23 corrects the R, G, and B pixel values of the corresponding target image 30 by gamma conversion according to the pixel value of the pixel of the smoothed image 92, and whites the subject's white eye. To. In the present embodiment, as in the case of the tooth, the pixel values of the R, G, and B color components themselves are not gamma-converted, but the luminance of the pixel is larger than 1 such as 1.3. Apply a tone curve with a value and apply a tone curve with a γ value less than 1 such as 0.6 for saturation. The color correction unit 23 converts the gamma-converted luminance and saturation information into R, G, and B pixel values by a known conversion method and corrects them. In addition, the γ value of the tone curve applied to each of luminance and saturation may be the same as or different from the value in step S105.

  As described above, in the present embodiment, when extracting a tooth image region, the determination is made based on the hue that is not easily affected by the difference in exposure conditions and the saturation that is not easily affected by the white balance process. The image area can be easily detected with high accuracy.

Also, when extracting the image area of the white eye, the image area of the white eye can be extracted by using a dynamic threshold method and a multivariate analysis in which the threshold setting can be adjusted according to the difference in the exposure condition and the white balance processing. Can be easily detected with high accuracy.
<< Additional items of embodiment >>
(1) In the above embodiment, an example has been described in which the CPU 14 implements each process of the face detection unit 20, the image extraction unit 21, the face part extraction unit 22, and the color correction unit 23 of the digital camera 1 by software. Of course, each of these processes may be realized by hardware using an ASIC.

  Further, each of the face detection unit 20, the image extraction unit 21, the face part extraction unit 22, and the color correction unit 23 has been processed on the assumption that the subject is a person, but the subject can also be applied to animals such as dogs and cats. It is.

  (2) The image processing apparatus of the present invention is not limited to the example of the digital camera 1 of the above embodiment. For example, the computer may function as the image processing apparatus of the present invention by causing the computer to read the captured image and causing the computer to perform color correction processing on teeth and white eyes.

  (3) In the above embodiment, the image extraction unit 21 recognizes the face part to be processed based on the edge detection and cuts out an image including the face part. However, the present invention is not limited to this. For example, according to the composition of the subject to be imaged, data in which each facial part is previously stored is stored in the memory 15, so that the image extraction unit 21 recognizes the composition of the subject in the target image 30 and creates the recognized composition. Accordingly, the position of the face part to be processed may be specified, and a partial image including the face part may be cut out.

  (4) In the above embodiment, the target image 30 is an image having R, G, and B pixel values in each pixel by digital color interpolation processing by the CPU 14, but the present invention is not limited to this. It may be a RAW image before color interpolation processing.

  (5) In the above embodiment, the color correction unit 23 smoothes the binarized image with 3 pixels × 3 pixels, and generates the smoothed image 62 and the smoothed image 92 with gradations of 0 to 255. The invention is not limited to this. For example, the color correction unit 23 may perform a smoothing process on an area having a size of 5 pixels × 5 pixels and generate a smooth image having an arbitrary gradation range.

  Further, the color correction unit 23 may perform gamma conversion on the pixel values of the respective color components of the pixels of the target image 30 corresponding to the tooth and white image regions without performing the smoothing process according to its own value. A predetermined image value indicating white (for example, R = G = B = 255) may be substituted for the pixel value of each color component of the pixel of the target image 30 corresponding to the tooth or white eye image region.

  (6) In the above embodiment, when extracting the image areas of the left and right eyes, the partial images 70 of the left and right eyes are cut out. However, the present invention is not limited to this, and includes, for example, both the left and right eyes. One partial image may be cut out to extract the left and right eye image regions.

  From the above detailed description, features and advantages of the embodiment will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and changes, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents included in the scope disclosed in.

DESCRIPTION OF SYMBOLS 1 Digital camera, 11 Imaging optical system, 12 Image sensor, 13 DFE, 14 CPU, 15 Memory, 16 Operation part, 17 Monitor, 18 Media I / F, 19 Storage medium, 20 Face detection part, 21 Image extraction part, 22 Face part extraction unit, 23 color correction unit

Claims (8)

A face detection unit for recognizing a face of a subject from a target image having a plurality of color components and detecting a face image;
An extraction unit that extracts a face part to be processed based on information of the plurality of color components included in the face image;
A color correction unit that corrects a value of the color component in a region corresponding to the face part of the target image;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The image processing apparatus, wherein the extraction unit extracts the face part based on information on hue and saturation. The image processing apparatus according to claim 2,
When the facial part is a tooth, the extraction unit extracts a lip image region including the tooth based on the hue, and the saturation is calculated based on multivariate analysis in the lip image region. An image processing apparatus that extracts an image area that is equal to or less than a threshold value as the tooth image area. The image processing apparatus according to claim 1.
The image processing apparatus, wherein the extraction unit extracts the face part based on information on luminance and saturation. The image processing apparatus according to claim 4.
When the facial part is a white eye, the extraction unit extracts an image region of the eye including the white eye based on a comparison between the luminance and a first threshold value that dynamically changes, and the image of the eye An image processing apparatus, wherein an image area in which the saturation is equal to or less than a second threshold value calculated based on multivariate analysis is extracted as the white-eye image area. The image processing apparatus according to any one of claims 1 to 5,
The image processing apparatus, wherein the color correction unit corrects the value of the color component in a region corresponding to the face part based on information on luminance and saturation. An imaging unit that captures a subject and obtains a target image having a plurality of color components;
An image processing apparatus according to any one of claims 1 to 6,
An imaging apparatus comprising: An input procedure for reading a target image having a plurality of color components;
A face detection procedure for recognizing the face of the subject from the target image and detecting the face image;
An extraction procedure for extracting a face part to be processed based on information of the plurality of color components included in the face image;
A color correction procedure for correcting a value of the color component in a region corresponding to the face part of the target image;
An image processing program for causing a computer to execute.

Images