JP2012157057A - Image processing apparatus and method, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform more stable white balance correction according to a face detection result.SOLUTION: An image processing apparatus for processing an image signal of an image comprises: first calculation means (103) that calculates a first white balance correction value by detecting white pixels from the image; face detection means (114) that detects a face area; determination means (103) that, when white balance correction is performed on an image signal of the face area based on the first white balance correction value, determines whether or not it is in a second color signal area around a first color signal area indicating a flesh color; second calculation means (103) that, when it is in the second color signal area, calculates a second white balance correction value for correcting the image signal so that it moves toward a direction of the first color signal area based on a relationship between the corrected image signal of the face area and the first color signal area, and when it is not in the second color signal area, does not calculate the second white balance correction value; and white balance correction means (103).

Description

  The present invention relates to an image processing apparatus and method, and an imaging apparatus, and more particularly to an image processing apparatus and method and an imaging apparatus that perform image processing using a face detection result.

The operation of a conventional white balance gain calculation circuit used in a digital camera or the like will be described. First, as shown in FIG. 8, the screen is divided in advance into a plurality of arbitrary blocks (m) by a plurality of pixels. Then, for each block (1 to m), pixel values are added and averaged for each color to calculate a color average value (R [i], G [i], B [i]). Then, for example, the color evaluation values (Cx [i], Cy [i]) are calculated using the following formula (1).
Cx [i] = (R [i]-B [i]) / Y [i] x 1024
Cy [i] = (R [i] + B [i])-2G [i] / Y [i] x 1024 (1)
Where Y [i] = R [i] + 2G [i] + B [i], [i] is the index number of each block

  Then, a white subject is photographed in advance under various light sources, and a color evaluation value is calculated. Thereby, when the color evaluation value calculated for each block is included in the preset white detection range 301 as shown in FIG. 9, it is determined that the block is white. Similarly, the pixel values of the blocks determined to be white are integrated. The white detection range 301 is obtained by photographing white under a different light source in advance and plotting the calculated color evaluation value. When the negative direction of the x coordinate (Cx) in FIG. The color evaluation value is a color evaluation value when the positive direction is a white image of a low color temperature subject. The y coordinate (Cy) means the degree of the green component of the light source, and the G component increases as it goes in the negative direction, that is, the light source is a fluorescent lamp.

Then, white balance coefficients (WBCo_R, WBCo_G, WBCo_B) are calculated from the integrated pixel values (sumR, sumG, sumB) using the following equation (2).
WBCo_R = sumY × 1024 / sumR
WBCo_G = sumY × 1024 / sumG (2)
WBCo_B = sumY × 1024 / sumB
However, sumY = (sumR + 2 × sumG + sumB) / 4

  However, the conventional white balance coefficient calculation method has the following problems. That is, under a light source such as sunlight, the white color evaluation value is distributed in the vicinity of the region A in FIG. 9 and the skin color distribution is distributed in the vicinity of the region B. The skin color evaluation value under the solar light source is distributed in a region substantially equivalent to the white point color evaluation value under the low color light source. Therefore, when the angle of view is small and a person is photographed up as shown in FIG. 10, the color evaluation value of the screen is distributed in the region B of FIG. 9, so the skin color is misunderstood as white under a low color temperature. Another problem is that human skin is corrected to white. Further, when a chromatic color distributed in the vicinity of a region showing human skin is on the entire surface of the field of view, the chromatic color may be misidentified as white and the skin may be corrected to white.

  As a conventional countermeasure against this problem, when the subject illuminance is bright, it is determined that the light is outside, and the white detection range is narrowed so that the skin color is not erroneously determined to be white.

  However, in general fluorescent lamps, there are also light sources in which white is near the region C and light sources distributed below the region C. In order to cope with such light sources, the white detection range must be expanded. . However, there are cases where the skin color distribution in low-light high-color temperature fluorescent lamps and medium-color temperature fluorescent lamps is distributed near the downward direction (region C) of the black body radiation axis. Has been mistaken for white, and the skin color has faded.

  Therefore, in the method described in Patent Document 1, it has been proposed that face detection is performed, and when a face is detected, the skin color of the face is extracted, compared with the reference skin, and white balance correction is performed based on the result. Yes.

US Pat. No. 6,975,759

  However, in the above-described conventional white balance correction value calculation method, the case where the face is erroneously recognized in the face detection circuit is not taken into consideration. Therefore, even if a non-face area is erroneously detected as a face in the face detection circuit, the non-face area operates so as to have an appropriate skin color, and as a result, a desired white balance correction value cannot be obtained. There was a drawback that there was.

  In addition, depending on the performance of the imaging apparatus such as processing speed, face detection may not be possible during shooting. In such an imaging apparatus, even if a face is detected in the shooting preparation stage, a case may occur in which the position of the subject is shifted during shooting, but such a case is not taken into consideration. That is, even when a face is correctly detected at the time of shooting preparation, there is a drawback that the white balance correction accuracy may be lowered as a result. FIG. 11 shows an example in which face detection is performed before SW1, and the position of the subject is shifted immediately before SW2 (main shooting). By using the face detection result obtained immediately before SW1, it becomes impossible to acquire a correct face color evaluation value at SW2 (main photographing), and the white balance correction accuracy is lowered.

The present invention has been made in view of the above problems, and an image processing apparatus according to the present invention that processes an image signal of an image obtained by performing imaging first detects a white pixel from the image. First calculation means for calculating a white balance correction value, a face detection means for detecting a face area from the image, and an image signal of the face area detected by the face detection means as the first white balance correction value. Determining means for determining whether the image signal of the corrected face area is in a second color signal area around the first color signal area representing the skin color when white balance correction is performed based on If the determination means determines that the second color signal area is present, the corrected face area image signal is based on the relationship between the corrected face area image signal and the first color signal area. Is the first color signal Calculating a second white balance correction value obtained by correcting the image signal of the image to move in the direction of the band, when it is determined that not in the second color signal region by the determining means, the second White that performs white balance correction on an image signal of an image obtained by performing imaging using a second calculation unit that does not calculate a white balance correction value and the first or second white balance correction value. And a balance correction unit .

In the image processing method of the present invention for processing an image signal of an image obtained by imaging, the first calculation means detects the white pixel from the image, thereby obtaining the first white balance correction value. A first calculating step for calculating; a face detecting step for detecting a face region from the image by a face detecting unit; and a determining unit for outputting an image signal of the face region detected by the face detecting step to the first white A determination step of determining whether or not the image signal of the corrected face area is in the second color signal area around the first color signal area representing the skin color when white balance correction is performed based on the balance correction value; When the second calculating means determines that the second color signal area is in the second color signal area by the determination step, based on the relationship between the corrected face area image signal and the first color signal area, Image signal of the corrected face area There calculating a second white balance correction value obtained by correcting the image signal of the image to move in the direction of the first color signal region, it is determined not to the second color signal region by the determining means The second white balance correction value is not calculated, and the image signal of the image obtained by imaging using the first white balance correction value or the second white balance correction value. A white balance correction step of performing white balance correction .

  The imaging apparatus of the present invention includes an imaging unit that outputs an image signal of an image obtained by imaging, and the image processing apparatus.

  According to the present invention, more stable white balance correction can be performed according to the face detection result.

It is a schematic block diagram which shows the function structure of the imaging device provided with the face detection function in embodiment of this invention. It is a flowchart explaining the pattern recognition process by template matching. It is a figure explaining the concept of template matching. It is a flowchart which shows the calculation method of the WB correction value in embodiment of this invention. It is a flowchart which shows the calculation process of the 1st WB correction value in embodiment of this invention. It is a figure explaining the judgment method of skin color correction in an embodiment of the invention. It is a conceptual diagram of the 2nd WB correction value calculation in embodiment of this invention. It is a figure which shows the example which divided | segmented the screen into arbitrary several blocks. It is a figure which shows a white detection range. It is a figure which shows the example which image | photographed the face-up as a to-be-photographed object. It is a figure explaining an example of the problem which arises in the conventional white balance correction method.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

  FIG. 1 is a schematic block diagram illustrating a functional configuration of an imaging apparatus having a face detection function according to an embodiment of the present invention as an example of an image processing apparatus.

  In FIG. 1, reference numeral 101 denotes a solid-state imaging device made of a CCD, a CMOS, or the like, and the surface thereof is covered with, for example, an RGB color filter having a Bayer arrangement so that color imaging can be performed. A memory 102 temporarily stores an image signal obtained from the image sensor 101.

  Reference numeral 114 denotes a face detection unit that detects a face area from the image signal stored in the memory 102. Various techniques have been proposed as techniques for detecting a face region, and any technique may be used as long as it can acquire face position and size information, and the present invention is limited by the face detection technique. It is not a thing. For example, a method using learning typified by a neutral network and a method of extracting a part having a physical shape such as eyes and nose from an image region by template matching are known. Other methods include detecting image feature quantities such as skin color and eye shape and analyzing them using statistical methods (for example, Japanese Patent Laid-Open Nos. 10-232934 and 2000-48184). reference). In addition, what is currently proposed as a product includes a method of detecting a face using wavelet transform and image feature amounts, a method of combining template matching, and the like.

  Here, template matching, which is one of pattern recognition, will be described as a face detection method. Pattern recognition is a process of matching an observed pattern with one of predetermined concepts (classes). FIG. 2 is a flowchart of the pattern recognition process, which is executed by the face detection unit 114.

  First, image data is read from the memory 102 and preprocessed (step S1), and a pattern of a characteristic portion is extracted from the preprocessed image data (step S2). Then, the extracted pattern is made to correspond to the template (standard pattern) (template matching). For example, when the characteristic part pattern 63 is extracted as shown in FIG. 3, the center point 62 of the template 61 is placed at a certain coordinate point (i, j) of the acquired pattern 63. Then, while shifting the position of the center point 62 in the pattern 63, the similarity of the overlapping portion between the template 61 and the pattern 63 is calculated, and the position where the similarity is maximized is determined. By matching the pattern 63 with, for example, a template 61 including shapes such as eyes and ears, position information of eyes and face areas (face coordinates) can be acquired.

  Thus, a recognition pattern is acquired (step S3), the acquired recognition pattern is output (step S4), and a pattern recognition process is complete | finished.

  Returning to the description of FIG. The CPU 115 calculates the shutter speed Tv and the aperture value Av so that the face becomes optimal brightness if the face is detected based on the signal sent from the face detection unit 114, and focuses on the face. The driving amount of the focus lens is calculated as follows. On the other hand, if the face is not detected, the CPU 115 calculates the shutter speed Tv and the aperture value Av so that the entire image has the optimum brightness, and focuses on the subject in the preset focus area. The driving amount of the focus lens is calculated. The exposure values (Tv, Av) calculated by the CPU 115 and the driving amount of the focus lens are sent to the control circuit 113, and a lens, a diaphragm, a shutter, and an image sensor 101 (not shown) are controlled based on each value.

  A white balance (WB) control unit 103 calculates a WB correction value based on the image signal stored in the memory 102 and the face information obtained from the face detection unit 114. Then, white balance correction (WB correction) is performed on the image signal stored in the memory 102 using the calculated WB correction value. In other words, the WB control unit 103 corresponds to white balance correction means, first calculation means, and second calculation means. The WB correction value calculation method used by the WB control unit 103 will be described in detail later.

  Reference numeral 104 denotes a color conversion matrix (MTX) circuit that converts the color difference signals RY and BY into color difference signals so that the image signal corrected by the WB control unit 103 is WB corrected with an optimal color. . Reference numeral 105 denotes a low-pass filter (LPF) circuit that limits the bands of the color difference signals RY and BY. Reference numeral 106 denotes a CSUP (Chroma) that suppresses a false color signal in a saturated portion of the image signal band-limited by the LPF circuit 105. Supress) circuit.

  On the other hand, the WB corrected image signal by the WB control unit 103 is also output to the luminance signal (Y) generation circuit 111 to generate the luminance signal Y, and the edge enhancement circuit 112 performs edge processing on the generated luminance signal Y. Emphasis processing is performed.

  The color difference signals RY and BY output from the CSUP circuit 106 and the luminance signal Y output from the edge emphasis circuit 112 are converted into RGB signals by the RGB conversion circuit 107, and the gamma correction circuit 108 converts the gradation signals Y to B and Y. Tone correction is applied. After that, it is converted into a YUV signal by the color luminance conversion circuit 109, and further compressed by, for example, JPEG by the compression circuit 110 and recorded as an image signal on an external recording medium or an internal recording medium.

  Next, a method for calculating the WB correction value in the present embodiment will be described with reference to FIG. The calculation method of the WB correction value in the present embodiment makes it possible to obtain a WB correction value that optimizes the face area according to the face detection result. The face detection process is performed by the face detection unit 114 by the method described above with reference to FIGS. 2 and 3, for example, based on the obtained image signal immediately before the process shown in FIG. 4 is performed. It is assumed that the result has already been obtained. In addition, the calculation of the WB correction value performed here may be performed by the CPU 115 or the WB control unit 103, and a dedicated configuration for calculating the WB correction value may be added. It does not matter either.

  First, in step S11, white pixels are detected from the image signal stored in the memory 102, and a first white balance correction value (first WB correction value) is calculated. Here, the calculation method of the first WB correction value will be described in detail with reference to FIG.

First, the image signal stored in the memory 102 is read, and the screen is divided into arbitrary m blocks as shown in FIG. 8 (step S101). Then, for each block (1 to m), the pixel values are added and averaged for each color to calculate a color average value (R [i], G [i], B [i]). The color evaluation values (Cx [i], Cy [i]) are calculated by using them (step S102).
Cx [i] = (R [i]-B [i]) / Y [i] x 1024
Cy [i] = (R [i] + B [i])-2G [i] / Y [i] x 1024 (1)
Where Y [i] = R [i] + 2G [i] + B [i], [i] is the index number of each block

  Next, it is determined whether or not the color evaluation value (Cx [i], Cy [i]) of the i-th block calculated in step S102 is included in the preset white detection range 301 shown in FIG. S103). The white detection range 301 is obtained by photographing white under different light sources in advance and plotting the calculated color evaluation values. The negative direction of the x coordinate (Cx) in FIG. 9 is a color evaluation value when white of a high color temperature subject is photographed, and the positive direction is a color evaluation value when white of a low color temperature subject is photographed. The y coordinate (Cy) means the degree of the green component of the light source, and the G component increases as it goes in the negative direction, that is, the light source is a fluorescent lamp.

  When the calculated color evaluation values (Cx [i], Cy [i]) are included in the white detection range 301 (YES in step S103), it is determined that the block is white. Then, the color average values (R [i], G [i], B [i]) of the block are accumulated (step S104), and if not included, the process proceeds to step S105 without adding. The processing of step S103 and step S104 can be expressed by equation (3).

Here, in Equation (3), Sw [i] is set to 1 when the color evaluation value (Cx [i], Cy [i]) is included in the white detection range 301, and Sw [i is set when it is not included. ] Is set to 0. In this way, the process of whether or not to add the color average values (R [i], G [i], B [i]) is substantially performed according to the determination in step S103.
In step S105, it is determined whether or not the above process has been performed for all blocks. If there is an unprocessed block, the process returns to step S102 to repeat the above process, and if all the blocks have been processed, the process proceeds to step S106. .

In step S106, the first WB correction values (WBCo1_R, WBCo1_G, WBCo1_B) are calculated from the integrated values (sumR, sumG, sumB) of the obtained color evaluation values using the following equation (4).
WBCo1_R = sumY × 1024 / sumR
WBCo1_G = sumY × 1024 / sumG (4)
WBCo1_B = sumY × 1024 / sumB
However, sumY = (sumR + 2 × sumG + sumB) / 4

  When the first WB correction value is calculated as described above, it is determined whether or not a face is detected in step S12. If no face is detected, the first WB correction value calculated in step S11 is determined as the WB correction value to be used for the WB processing by the WB control unit 103 (step S20), and the process is terminated.

  If a face has been detected, a block of the face area is acquired in step S13, and the color average value (R) obtained in step S102 for calculating the first WB correction value for one of the blocks is determined. [i], G [i], B [i]) are acquired (step S14).

  Next, the color average value acquired in step S14 is multiplied by the first WB correction value obtained in step S11, respectively, and the skin color average value (the value obtained by WB correction of the color average value of the face area with the first WB correction value) is obtained. , Corrected image signal). The skin color average value is any of the skin color area (area (A) in FIG. 6), the skin color correction target area (area (B) in FIG. 6), and the skin color correction non-target area (area (C) in FIG. 6). It is determined whether it exists (step S15). The skin color area (A) corresponds to the first color signal area, and the skin color correction target area (B) corresponds to the second color signal area which is a peripheral area of the first color signal area. In the case of the skin color area (A) (in the first color signal area) or the skin color correction target area (B) (in the second color signal area) of FIG. 6, the skin color average value has been calculated so far. It is added to the sum of the skin color average values (step S16). If it is in the skin color correction non-target region (C), the skin color average value of the block is not added, and the process directly proceeds to step S17. Note that the skin color region (A) and the skin color correction target region (B) shown in FIG. 6 can be set using a statistical method by photographing a plurality of skin colors in advance under white light such as sunlight. .

  For example, when a face is erroneously detected (as shown in FIG. 6B) or the subject is completely displaced, the skin evaluation value of the block is distributed in the non-skin color correction target region (C) in FIG. There are many cases to do. When the face area is shifted (as shown in FIG. 6C), the skin evaluation values of the face area are often distributed in the skin color area (A) or the skin color correction target area (B) in FIG. Thus, only the blocks in the skin color area (A) and the skin color correction target area (B) are extracted and reflected in the WB calculation process. Note that weighted addition by weighting may be performed instead of excluding the skin color evaluation value in the skin color correction non-target area (C) based on the determination in step S15. In that case, the weight applied to the skin color evaluation value in the skin color correction non-target area (C) is set lower than the weight applied to the skin color evaluation value in the skin color area (A) and the skin color correction target area (B). Keep it.

  After performing the above-described processing for the number of acquired blocks (until YES in step S17), the process proceeds to step S18. In step S18, it is determined whether the sum of the flesh color average values (total image signal) obtained in step S16 is within the flesh color correction target area (B).

  If it is in the skin color area (A) of FIG. 6 (NO in step S18), it can be determined that the skin color has been appropriately WB corrected by the first WB correction value, and therefore it is determined to use the first WB correction value. (Step S20). If it is within the skin color correction target area (B) (YES in step S18), it can be determined that the skin color corrected by the first WB correction value has not been properly WB corrected. Therefore, a second white balance correction value (second WB correction value) that moves in the skin color region direction is calculated (step S19). If it is outside the skin color correction target region (C) (NO in step S18), it is determined that the detected skin color evaluation value does not represent human skin, and the first WB correction value is used (step S18). S20).

  FIG. 7 shows the concept of the second WB correction value calculation performed in step S19. FIG. 7 assumes a case where the skin color is erroneously recognized as white under a certain light source and the first WB correction value is calculated. Since the first WB correction value is located in the lower color temperature direction than the appropriate WB correction value, the flesh color after the first WB correction is shifted in the cold color direction. In this case, since the skin color evaluation value is located in the cold color direction from the region showing the proper skin color, the white skin correction value is corrected to an appropriate skin color by applying an R gain larger than the first WB correction value. It becomes possible.

  As described above, according to the present embodiment, it is possible to reduce miscorrection and overcorrection caused by misrecognition of a face or subject deviation at the time of shooting. Therefore, more stable white balance can be achieved according to the face detection result. Correction can be performed.

<Other embodiments>
Note that the present invention can be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a camera head, etc.), or a device (for example, a digital still camera, a digital video camera, etc.) composed of a single device. You may apply to.

  The object of the present invention can also be achieved as follows. First, a storage medium (or recording medium) that records a program code of software that implements the functions of the above-described embodiments is supplied to a system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the following can be achieved. That is, when the operating system (OS) running on the computer performs part or all of the actual processing based on the instruction of the read program code, the functions of the above-described embodiments are realized by the processing. It is. Examples of the storage medium for storing the program code include a flexible disk, hard disk, ROM, RAM, magnetic tape, nonvolatile memory card, CD-ROM, CD-R, DVD, optical disk, magneto-optical disk, MO, and the like. Can be considered. Also, a computer network such as a LAN (Local Area Network) or a WAN (Wide Area Network) can be used to supply the program code.

Claims (7)

An image processing apparatus that processes an image signal of an image obtained by imaging,
First calculating means for calculating a first white balance correction value by detecting a white pixel from the image;
Face detection means for detecting a face region from the image;
A first color signal area in which the corrected face area image signal represents skin color when the face area image signal detected by the face detection means is white balance corrected based on the first white balance correction value. Determining means for determining whether or not the second color signal area in the vicinity of
When the determination means determines that the second color signal area is present, the corrected face area image is based on the relationship between the corrected face area image signal and the first color signal area. signal calculating a second white balance correction value obtained by correcting the image signal of the image to move in the direction of the first color signal region, determined not to the second color signal region by the determining means A second calculating unit that does not calculate the second white balance correction value ,
White balance correction means for performing white balance correction on an image signal of an image obtained by imaging using the first or second white balance correction value;
An image processing apparatus comprising:   2. The second white balance correction value according to claim 1, wherein the second calculation means calculates the second white balance correction value so that an image signal of the face region after correction falls within the first color signal region. An image processing apparatus according to 1.   The first calculation means detects white pixels of the image in a predetermined block unit, and the determination means performs the determination on the image signal of the face area detected in the predetermined block unit. The image processing apparatus according to claim 1. Imaging means for outputting an image signal of an image obtained by imaging;
Imaging apparatus characterized by comprising an image processing apparatus according to any one of claims 1 to 3. An image processing method for processing an image signal of an image obtained by imaging,
A first calculating unit that calculates a first white balance correction value by detecting a white pixel from the image;
A face detecting step in which a face detecting means detects a face region from the image;
When the determination unit performs white balance correction on the image signal of the face area detected by the face detection step based on the first white balance correction value, the corrected face area image signal represents a skin color. A determination step of determining whether or not the second color signal area is around the color signal area;
When the second calculating means determines that the second color signal area is in the second color signal area by the determination step, based on the relationship between the image signal of the corrected face area and the first color signal area, calculating a corrected second white balance correction value image signal of the face region is corrected image signal of the image to move in the direction of the first color signal region, the second by the determination unit A second calculation step that does not calculate the second white balance correction value when it is determined that the color signal area is not present ;
A white balance correction step of performing white balance correction on an image signal of an image obtained by imaging using the first or second white balance correction value;
An image processing method comprising: A program for causing a computer to execute each step of the image processing method according to claim 5 . A computer-readable storage medium storing the program according to claim 6 .

Images