JP2008301279A - White balance correction device and method, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To more accurately specify a light source that has illuminated an object, in order to perform white balance correction. <P>SOLUTION: A white balance correction device includes: an information detection section (131) for detecting information about the picture pattern of the object from an image signal of the object captured from an imaging element; a light source estimation section (111) for estimating a light source that has illuminated the object, using the information about the picture pattern; and white balance correction means (106, 110) which judges and extracts an image signal indicative of white from the image signal in accordance with the estimated light source, and performs white balance correction in such a way that the extracted image signal becomes a signal indicative of white. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a white balance correction apparatus and method, and an imaging apparatus equipped with the white balance correction apparatus. More specifically, the present invention relates to a white balance correction apparatus and method for specifying a light source that illuminates a subject at the time of shooting and performing white balance according to the specified light source, and an imaging apparatus equipped with the white balance correction apparatus.

  In recent imaging apparatuses such as video cameras, a TTL (Through The Lens) system that performs control based only on information of an output signal of an imaging element without using an external sensor during auto white balance is the mainstream. As an image pickup apparatus equipped with such a white balance correction apparatus that performs white balance correction, for example, Patent Document 1 discloses the following image pickup apparatus.

  First, color difference signals (RY, BY) and luminance signal (Y) are obtained from red (R), green (G), and blue (B) color signals, and image data is divided into fine blocks. The signal is averaged within each divided block. In addition, a signal indicating the aperture state of the diaphragm of the imaging device, a signal indicating the control state of the electronic shutter of the imaging device, and the gain state of the AGC amplifier are acquired. Then, the illuminance of the subject is detected from the luminance signal of each divided block, the aperture release state, the electronic shutter control state, and the AGC amplifier gain state, and the light source is estimated based on this illuminance information. Then, among the signal values of each divided block, a signal close to white is extracted based on the estimated light source information, and the white balance of the red (R) and blue (B) signals so that the extracted signal approaches the target white. Determine the gain.

  FIG. 5 is a diagram showing the concept of the white extraction range according to the type of light source. In FIG. 5, 600 indicates a white extraction range in the color difference plane. In the above-described extraction of a signal close to white, a signal whose luminance signal is larger than a predetermined threshold and is within the white extraction range 600 is extracted. When the light source is sunlight (outdoors), the white extraction range 600 is set to a range in which a blue color is determined as an achromatic color as shown in FIG. Further, when the light source is an artificial light (indoor), the white extraction range 600 has a red color corresponding to the halogen light and a green color corresponding to the fluorescent light as shown in FIG. 5B. The range should be judged.

  On the other hand, in the field of computer vision, many methods have been proposed in the past in which a light source is estimated from a single still image to determine outdoor / indoor (for example, see Non-Patent Document 1).

  These methods have a plurality of characteristic parameters for light source estimation, and have a high outdoor / indoor identification rate in general scenes. However, for subjects that are not good at white balance control devices of imaging devices such as video cameras (such as indoor fluorescent lamps and outdoor vegetation green scenes, indoor white walls and outdoor sky scenes) Identification has been difficult.

JP-A-5-64219 Martin Szummer and Rosalind W. picard, “Indoor-Outdoor Image Classification” IEEE Intl Workshop on Content-based Access of Image Video Database, Jan 1998.

  As described above, an apparatus for identifying outdoor / indoor based on illuminance information may erroneously determine that the scene is indoors when shooting in a dark place such as an outdoor shade. This is because if threshold processing is performed using only illuminance as a reference, there is a portion where the distribution of indoor light and outdoor light overlap, and the light source cannot be separated by the one-dimensional feature threshold.

  In general, the illuminance of an indoor light source has been considered to be lower than that of outdoor sunlight, but nowadays there are many indoor light source environments with high illuminance and color temperature. For example, among commercial fluorescent lamps, both color temperature and illuminance are higher than those used at home. That is, even in an environment limited to indoors, the above-described light source determination method may not be able to perform appropriate white balance correction.

  The present invention has been made in view of the above problems, and an object of the present invention is to make it possible to specify a light source that illuminates a subject more accurately in order to perform white balance correction.

  In order to achieve the above object, the white balance correction apparatus of the present invention uses information detection means for detecting information relating to the pattern of the subject from the image signal of the subject obtained from the image sensor, and information relating to the pattern, A light source estimation unit that estimates a light source that illuminates a subject, and an image signal indicating white is determined and extracted from the image signal according to the estimated light source, and the extracted image signal becomes a signal indicating white. And white balance correction means for performing white balance correction.

  The white balance correction method of the present invention includes an information detection step of detecting information about a pattern from an image signal of a subject obtained from an image sensor, and a light source that estimates a light source that illuminates the subject using the information about the pattern. A white balance that determines and extracts an image signal indicating white from the image signal according to the estimation step and the estimated light source, and performs white balance correction so that the extracted image signal becomes a signal indicating white And a correction step.

  According to another configuration, the white balance correction apparatus of the present invention includes an information detection unit that detects information on the pattern of the subject from the image signal of the subject obtained from the image sensor, and a predetermined hue from the image signal. The color information detecting means for detecting feature information of the image signal belonging to the image, the illuminance detecting means for detecting the illuminance of the subject, the information on the pattern, the feature information, and the illuminance are used to illuminate the subject. A light source estimating means for estimating a light source, and determining and extracting an image signal indicating white from the image signal according to the estimated light source, and white balance so that the extracted image signal becomes a signal indicating white White balance correction means for performing correction.

  The white balance correction method according to the present invention further includes an information detection step of detecting information about a pattern from an image signal of a subject obtained from an image sensor, and detects feature information of an image signal belonging to a predetermined hue from the image signal. A color information detection step, an illuminance detection step for detecting the illuminance of the subject, information on the pattern, the feature information, and a light source estimation step for estimating a light source that illuminates the subject using the illuminance, A white balance correction step of determining and extracting an image signal indicating white from the image signal in accordance with the estimated light source, and performing white balance correction so that the extracted image signal becomes a signal indicating white .

  In addition, an imaging apparatus according to the present invention includes the above-described white balance correction device.

  According to the present invention, in order to perform white balance correction, it is possible to specify the light source that illuminated the subject more accurately.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a block diagram illustrating an example of a configuration of an imaging apparatus including an auto white balance correction apparatus according to the first embodiment of the present invention.

  In FIG. 1, 101 is a lens that forms an image of a light beam of a subject, 102 is a stop, 103 is an image sensor that photoelectrically converts incident light, and 104 is an AGC amplifier that amplifies a signal from the image sensor 103 to an appropriate level. Reference numeral 120 denotes a camera signal processing unit that processes a signal output from the AGC amplifier 104, and 130 denotes a camera control microcomputer that controls the entire image pickup apparatus. The camera control microcomputer 130 includes a function for setting a white balance gain. It is.

  In the camera signal processing unit 120, a luminance / color signal generation unit 105 converts a signal generated by the image sensor 103 into a luminance signal (Y) and a color signal (R, G, B). Reference numeral 106 denotes a white balance control unit that controls the gain of the color signals (R, G, and B) output from the luminance / color signal generation unit 105 and controls white balance. Reference numeral 107 denotes a color difference signal (RY). , BY) generator. Furthermore, the camera signal processing unit 120 includes an information detection unit 113 that detects and outputs information on the pattern of the subject from the signal generated by the image sensor 103. Here, the information regarding the pattern is the amount of horizontal and vertical edge components included in the subject, and is extracted from the signal generated by the image sensor 103 by performing spatial frequency analysis (that is, frequency component analysis) of the subject.

  An encoder 108 converts a luminance signal (Y) output from the luminance / color signal generation unit 105 and a color difference signal (RY, BY) output from the color difference signal generation unit 107 into a standard television signal or the like. Part.

  A signal dividing unit 109 divides the luminance signal (Y) and the color difference signals (RY, BY) into predetermined regions, and 112 is an illuminance detecting unit that detects the illuminance of the subject. A light source estimation unit 111 estimates a light source that illuminates the subject based on the illuminance information from the illuminance detection unit and the amounts of vertical and horizontal edge components from the information detection unit 113. Reference numeral 110 denotes a white balance gain determination unit that determines a gain for correcting white balance based on the signal obtained from the signal dividing unit 109 and the light source information from the light source estimation unit 111. A white balance correction unit is configured by the white balance gain determination unit 110 that determines the gain of the white balance and the white balance control unit 106 that actually performs white balance correction using the determined gain.

  Next, the operation of the imaging apparatus having the above configuration will be described.

  Light incident on the image sensor 103 is photoelectrically converted, amplified to an appropriate level by the AGC amplifier 104, and then output to the luminance / color signal generation unit 105 and the information detection unit 113. The information detection unit 113 analyzes the spatial frequency of the subject from the input signal, extracts the amount of horizontal / vertical edge components included in the subject, and inputs the extracted amount to the light source estimation unit 111. On the other hand, the luminance / color signal generation unit 105 generates a luminance signal (Y) and color signals (R, G, B), and the color signal is output to the white balance control unit 106. The white balance control unit 106 adjusts the gain of the color signal based on the white balance gain control signal output from the white balance gain determination unit 110, which will be described later, and the color signal (R, G, B) that has been gain-adjusted is color-differenced. The signal is output to the signal generator 107. The color difference signal generation unit 107 generates color difference signals (RY, BY) from the color signals (R, G, B), and outputs the color difference signals to the encoder 108 and the signal division unit 109. The encoder 108 generates and outputs a standard television signal such as NTSC from the luminance signal (Y) and the color difference signals (RY, BY).

  The signal dividing unit 109 divides the image signal into blocks (for example, 8 × 8 areas), and calculates an average value of the luminance signal (Y) and the color difference signals (RY, BY) in each block. Then, an average luminance signal and an average color difference signal are output to the white balance gain determination unit 110, and an average luminance signal is output to the illuminance detection unit 112, respectively.

  The illuminance detection unit 112 receives a signal indicating the open state of the diaphragm 102, a signal indicating the control state of the electronic shutter of the image sensor 103, and a signal indicating the gain state of the AGC amplifier 104. The illuminance detection unit 112 detects the illuminance of the subject from these input signals and the luminance signal from the signal division unit 109 described above, and outputs the detected illuminance information to the light source estimation unit 111.

  The light source estimation unit 111 estimates a light source based on two types of feature amounts, the illuminance information input from the illuminance detection unit 112 and the amount of vertical / horizontal edge components output from the information detection unit 113.

  FIG. 2 is a diagram illustrating an example of a determination criterion when the above-described two types of feature quantities are used for identifying a light source and a subject performed in the light source estimation unit 111 in the first embodiment.

  In general, an image including many horizontal / vertical edge components is often an artificial object, and conversely, an image having few horizontal / vertical edge components is often a natural object.

  Therefore, in the first embodiment, when the horizontal / vertical edge component indicated by the pattern-related information is equal to or more than the preset first threshold, the image is taken indoors, particularly under a fluorescent lamp. Judge. Next, when the illuminance of the image is lower than the preset first illuminance, it is determined that photographing is being performed under a general household fluorescent lamp (first fluorescent lamp). In addition, when the illumination intensity is higher than the preset second illuminance, photographing is performed under a commercial fluorescent lamp (second fluorescent lamp) that has a higher color temperature and illuminance than a fluorescent lamp for home use. It is determined that

  When the horizontal and vertical edge components indicated by the pattern-related information are equal to or smaller than a preset second threshold value, it is determined that the image is taken outdoors, that is, under sunlight. Next, when the illuminance of the image is higher than the preset third illuminance, it is determined that the image is sunlight and taken in the sun, and the preset fourth illuminance is obtained. If it is low, it is determined that the same sunlight was taken in the shade (or under cloudy weather).

  When the horizontal / vertical edge component indicated by the pattern-related information is between the first threshold value and the second threshold value, the light source is estimated by a conventionally known method.

  Then, the light source estimation unit 111 sends the determination result of the light source obtained as described above to the white balance gain determination unit 110. The white balance gain determination unit 110 extracts a signal close to white from the luminance signal (Y) and the color difference signals (RY, BY) output from the signal division unit 109 based on the sent determination result. To do. Then, the white balance gain is determined so that the extracted signal close to white approximates the target white, and is sent to the white balance control unit 106.

  As described above, according to the first embodiment, by estimating the light source using the illuminance information of the subject obtained from the imaging device and the horizontal and vertical edge components indicated by the information related to the pattern of the image, Indoor light source estimation accuracy is improved. Specifically, 1) outdoor sunlight, 2) outdoor sunlight, 3) indoor household fluorescent lamp (first fluorescent lamp), 4) indoor commercial fluorescent lamp ( The four types of light sources (second fluorescent lamp) can be distinguished.

  As described above, since it is possible to estimate the light source more accurately than in the past, it is possible to improve the accuracy of white balance correction.

  In the first embodiment, the first and second threshold values and the first to fourth illuminances are set by a statistical method based on images obtained by shooting under various light sources. be able to.

  In the first embodiment, the illuminance is classified into two types. However, the present invention is not limited to this, and may be classified into three or more types. In that case, for outdoor shooting, it becomes possible to classify the light sources more finely, for example, to classify daytime shooting and morning / evening shooting.

  In the above-described example, the information detection unit 113 detects the horizontal / vertical edge component and outputs the amount thereof. However, the information regarding the pattern is not limited to the horizontal / vertical edge component. Other feature amounts may be detected. For example, it may be configured to detect a high-frequency component and specify the light source using the amount.

  Further, by using wavelet transform, in addition to the horizontal edge / vertical edge component / high frequency / low frequency component, it is possible to know in which part of the screen they exist (screen position information). For this reason, for example, since the lower part of the screen is a road, and the artificial object and the upper part of the screen are empty, it is possible to estimate a light source by estimating a natural object or the like.

<Second Embodiment>
Next, a second embodiment of the present invention will be described.

  In the second embodiment, in addition to the illuminance information and pattern information used in the first embodiment, feature information (average saturation / average luminance) obtained from an image signal belonging to a specific hue is added to the feature parameter. The case will be described.

  FIG. 3 is a block diagram illustrating an example of a configuration of an imaging apparatus including the auto white balance correction apparatus according to the second embodiment. In FIG. 3, the same reference numerals are given to the same components as those in FIG.

  In FIG. 3, reference numeral 330 denotes a camera control microcomputer, but the second embodiment has the following configuration. First, reference numeral 313 denotes a color information detection unit that detects feature information of an image signal belonging to a specific hue from the image signal. In the second embodiment, the green block detection unit detects green as the specific hue. 314, a green saturation detection unit 315, and a green luminance detection unit 316. Reference numeral 311 denotes a light source estimation unit that estimates a light source, and reference numeral 310 denotes a white balance determination unit 310 that determines a white balance gain based on a signal obtained from the signal division unit 109 and light source information from the light source estimation unit 311.

  Reference numeral 312 denotes an illuminance detector that detects the illuminance of the subject. Here, the configuration is independent of the camera control microcomputer 330, but the configuration may be of course included in the camera control microcomputer 330 as in the first embodiment.

  Next, light source estimation processing in the imaging apparatus having the above configuration will be described.

  First, among the average luminance signal (Y) and average color difference signals (RY, BY) output from the signal dividing unit 109 for each block, the white balance gain determining unit 110 supplies the average luminance signal and average color difference. A signal and a luminance signal are output to the illuminance detection unit 312. Further, a color difference signal is output to the green block detection unit 314, and a luminance signal is output to the green luminance detection unit 316.

  The green block detection unit 314 determines whether or not the input average color difference signal enters the green hue region, and the information on the blocks that have entered the green hue region is used as the green saturation detection unit 315 and the green luminance detection unit 316. Output to. The green saturation detection unit 315 acquires the saturation of the block that has entered the green hue area, calculates the average value of the saturation of all the blocks that have entered the green hue area, and estimates the light source as the green block average saturation To the unit 311.

  On the other hand, the green luminance detection unit 316 acquires the luminance value (Y) of the block that has entered the green hue area, calculates the average value of the luminance of all the blocks that have entered the green hue area, It outputs to the light source estimation part 311.

  The light source estimation unit 311 obtains illuminance information from the illuminance detection unit 312 and horizontal / vertical edge components from the information detection unit 113 in the same manner as in the first embodiment. Furthermore, in the second embodiment, from the four types of feature amounts obtained by adding the green block average saturation output from the green saturation detection unit 315 and the green block average luminance output from the green luminance detection unit 316, Estimate the light source.

  The green obtained by photographing the outdoors is often green of vegetation or lawn, and tends to have relatively high saturation and low brightness. On the other hand, green obtained by shooting indoors has a relatively high luminance, and the light from the fluorescent lamp may be reflected on a subject close to white to become green, so there is a tendency that the saturation is low and the luminance is high. is there. Therefore, it is possible to further improve the accuracy of the light source determination by adding the average luminance and average saturation of the green block as elements of the light source determination.

  FIG. 4 is a diagram illustrating an example of a light source determination result based on the feature values of the green block average saturation and the green block average luminance in addition to the illuminance and the amount of horizontal and vertical edge components.

  In the first embodiment described above, it has been determined that an object with a small amount of edge and low illuminance is outdoor sunlight in the shade, but by using information on the green block average luminance, in addition to the above conditions, green If the brightness is high, it is determined that the fluorescent lamp is for indoor use. For example, a scene such as a close-up view of a foliage plant under an indoor fluorescent lamp is classified under this condition (if the green block average saturation is used, it is further classified).

  Similarly, in the first embodiment, when the edge amount is large and the illuminance is low, it is determined that the fluorescent lamp is for home use indoors. A thing with low brightness | luminance is determined to be the sunlight of the outdoors shade. For example, a scene where a plant or the like is photographed in front of an artificial object such as a building in an outdoor shade is classified under this condition (if the green block average saturation is used, it is further classified).

  The light source estimation unit 311 sends the determination result of the light source obtained as described above to the white balance gain determination unit 110. The white balance gain determination unit 110 extracts a signal close to white from the luminance signal (Y) and the color difference signals (RY, BY) output from the signal division unit 109 based on the sent determination result. To do. Then, the white balance gain is determined so that the extracted signal close to white approximates the target white, and is sent to the white balance control unit 106.

  As described above, according to the second embodiment, by adding the feature information (average saturation / average luminance) of the image obtained from the green hue region to the feature parameter, a special shooting scene (for example, indoors) More detailed light source estimation including the case of photographing a foliage plant) is possible. As described above, since the light source can be estimated more accurately, the accuracy of white balance correction can be improved.

  Note that “small”, “many”, “high”, and “low” of each feature parameter in FIG. 4 are determined by comparing a threshold value set in advance for each feature parameter with each feature parameter. Each threshold value can be appropriately set by a statistical method based on images obtained by shooting under various light sources.

  In addition, although the case where the information of a green component was detected as a specific hue was demonstrated, the case of a red component and a blue component other than a green component can also be considered.

  In the case of the red component, there is a situation in which an outdoor sunset and an indoor light bulb (halogen) are misidentified. If you try to pull in the light bulb (halogen) completely, it will also pull in the sunset. Since they are distinguished using illuminance (the sunset has a high illuminance and the light bulb has a low illuminance), for example, under a light source such as a light bulb with a high illuminance, it is misclassified as a sunset (outdoor) and is not pulled in.

□ Here, by detecting the horizontal and vertical edge components as information about the pattern, the sunset has fewer vertical and horizontal edge components, and the bulb (halogen) is indoors, so there are many vertical and horizontal edge components. Can be prevented.
The case of a blue component is also conceivable. In the case of the blue component, there is a situation where the outdoor sky and the outdoor shade are misidentified. If you try to pull in the outdoor shade completely, you will also pull in the blue of the sky. In other words, the shade of the shade is allowed to some extent while leaving the sky blue.

□ Use high-frequency and low-frequency component information, such as wavelet transform, as pattern information. As a result, the sky is a low-frequency component, and the outdoor shade depends on the location, but it can be prevented from being erroneously identified as being generally a relatively high-frequency component such as a place with vegetation.
In the above embodiment, a digital video camera is assumed as the imaging device. However, the imaging apparatus of the present invention is not limited to a digital video camera, and can convert an optical image of a subject and output an electrical image signal such as a digital camera or a mobile terminal with a camera (including a mobile phone with a camera). As long as it is anything, it may be anything.

1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. It is a figure which shows the light source estimation result by the combination of the characteristic parameter in the 1st Embodiment of this invention. It is a block diagram which shows the structure of the imaging device in the 2nd Embodiment of this invention. It is a figure which shows the light source estimation result by the combination of the characteristic parameter in the 2nd Embodiment of this invention. It is a figure which shows the white extraction range according to the kind of conventional light source.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Lens 102 Diaphragm 103 Image pick-up element 104 AGC amplifier 105 Brightness and color signal generation part 106 White balance control part 107 Color difference signal generation part 108 Encoder 109 Signal separation part 110 White balance gain determination part 111, 311 Light source estimation part 112, 312 Illuminance detection Unit 113 Information detection unit 120 Camera signal processing unit 130, 330 Camera control microcomputer 313 Color information detection unit 314 Green block detection unit 315 Green saturation detection unit 316 Green luminance detection unit

Claims (9)

Information detecting means for detecting information related to the pattern of the subject from the image signal of the subject obtained from the image sensor;
Light source estimation means for estimating a light source that illuminates the subject using information on the pattern;
White balance correction means for determining and extracting an image signal indicating white from the image signal according to the estimated light source, and performing white balance correction so that the extracted image signal becomes a signal indicating white. A white balance correction apparatus comprising:   The white balance correction apparatus according to claim 1, wherein the information related to the pattern is information related to at least one of a frequency component and an edge component. The information related to the pattern is information indicating the amount of the edge component of the image signal of the subject,
The light source estimation means includes
When the amount of the edge component is larger than a preset first threshold and the illuminance is lower than the preset first illuminance, the light source is estimated as a first fluorescent lamp,
When the amount of the edge component is larger than the first threshold value and the illuminance is higher than the preset second illuminance, the light source has a color temperature and illuminance higher than those of the first fluorescent lamp. Presumed to be a high second fluorescent lamp,
When the amount of the edge component is less than a preset second threshold that is lower than the first threshold and the illuminance is higher than the preset third illuminance, Estimated sunlight,
When the amount of the edge component is smaller than the second threshold and the illuminance is lower than a preset fourth illuminance, the light source is estimated as shaded sunlight or cloudy weather. The white balance correction apparatus according to claim 1. Information detecting means for detecting information related to the pattern of the subject from the image signal of the subject obtained from the image sensor;
Color information detecting means for detecting feature information of an image signal belonging to a predetermined hue from the image signal;
Illuminance detection means for detecting the illuminance of the subject;
Light source estimation means for estimating a light source that illuminates the subject using information on the pattern, the feature information, and the illuminance;
White balance correction means for determining and extracting an image signal indicating white from the image signal according to the estimated light source, and performing white balance correction so that the extracted image signal becomes a signal indicating white. A white balance correction apparatus comprising: The color information detecting means includes
Determining means for determining whether an image signal belongs to the predetermined hue;
Saturation detecting means for calculating an average value of the saturation of the image signal belonging to the predetermined hue determined by the determining means;
Luminance detecting means for calculating an average value of luminances of image signals belonging to the predetermined hue determined by the determining means,
5. The white balance correction apparatus according to claim 4, wherein the feature information is an average value of saturation calculated by the saturation detection unit and an average value of luminance calculated by the luminance detection unit. .   6. The white balance correction apparatus according to claim 4, wherein the information on the pattern is information on at least one of a frequency component and an edge component.   An imaging apparatus comprising the white balance correction apparatus according to claim 1. An information detection step of detecting information about the pattern from the image signal of the subject obtained from the image sensor;
A light source estimation step of estimating a light source that illuminates the subject using information on the pattern;
A white balance correction step of judging and extracting an image signal indicating white from the image signal according to the estimated light source, and performing white balance correction so that the extracted image signal becomes a signal indicating white. A white balance correction method comprising: An information detection step of detecting information about the pattern from the image signal of the subject obtained from the image sensor;
A color information detection step of detecting feature information of an image signal belonging to a predetermined hue from the image signal;
An illuminance detection step for detecting the illuminance of the subject;
A light source estimation step of estimating a light source that illuminates the subject using information on the pattern, the feature information, and the illuminance;
A white balance correction step of judging and extracting an image signal indicating white from the image signal according to the estimated light source, and performing white balance correction so that the extracted image signal becomes a signal indicating white. A white balance correction method comprising:

Images