JP5740147B2 - Light source estimation apparatus and light source estimation method - Google Patents

Description

  The present invention relates to a light source estimation device and a light source estimation method.

  The human visual system can identify the color of an object to some extent even if the characteristics of the light source color are unknown. For example, when a white paper is viewed under sunlight, humans recognize it as white paper. On the other hand, when the paper is viewed under, for example, a halogen lamp, the halogen lamp is an orangeish light source, so the paper should look orange. However, even when viewed under a halogen lamp, humans perceive it as white paper as when viewed under sunlight. Thus, the function of recognizing the original color even when the light source color changes is called color constancy. Color constancy is synonymous with white balance adjustment in electronic imaging devices such as digital still cameras and digital video cameras. White balance is adjustment between RGB color channels so that an originally achromatic object looks achromatic. This adjustment is originally a process that cannot be performed unless the spectral characteristics of the light source and the object surface are known. That is, the spectral characteristic of light reaching the human eye is represented by the product of the spectral reflection characteristics of the light source and the object surface, but since both are unknown, this is a defect setting problem. For this reason, various algorithms have been proposed in which color constancy is modeled by making assumptions and restrictions on the light source, the object surface, and the scene being observed.

  One of the algorithms often used for solving the color constancy is the gray hypothesis (see, for example, Non-Patent Document 1). The gray hypothesis is based on the rule of thumb that the average color is close to gray because various colors are distributed in the field of view in a general scene. If the average color deviates from gray, it is determined that the deviation is due to the light source color, and white balance adjustment can be performed by adjusting between the RGB color channels. This method is advantageous in that the calculation cost is low and it can be easily mounted on an embedded device such as a digital still camera. On the other hand, since it is based on an assumption about the object surface that should be physically independent of the light source, the assumption does not hold depending on the scene, and the light source estimation result greatly depends. For example, suppose there is a gray logo on a part of a red wall. At this time, the average image color is red, and the gray hypothesis determines that the redness is due to the light source color. For this reason, cyan, which is the opposite color of red, is added to the entire screen in order to make the entire scene gray, so that the portion of the logo that was originally gray has a cyan color. This is called a color feria.

  For the gray hypothesis, a method of estimating using the spectral characteristics of the light source and the object surface has been proposed (for example, see Non-Patent Document 2). This method expresses the spectral characteristics of the light source and the object surface with low-order orthogonal bases, sets constraints based on assumptions about the light source / object surface / imaging system, and uses the probability distribution of known information about the light source / object surface / imaging system By integrating them, the light source estimation accuracy is statistically improved. The advantage of this method is that detailed light source estimation can be performed. However, on the other hand, since the amount of calculation increases, when used in image processing calculations such as white balance processing inside a digital camera, real-time processing may be hindered.

  Luminance-color correlation has been proposed as a method of estimating the type of light source in consideration of the light source and the spectral characteristics of the object surface while reducing the calculation cost (see Non-Patent Document 3, for example). This method uses the correlation between brightness and color that the red region becomes red when the light source color is red (same for green and blue), and determines whether the light source color is a white light source or not. Identify. The concept of luminance-color correlation will be described with reference to FIGS. 9 and 10 are schematic diagrams of spectral distributions for explaining the luminance-color correlation. The horizontal axis of the graphs shown in FIGS. 9 and 10 is the wavelength, and the vertical axis is the energy of each wavelength. FIG. 9A shows, from the left, an image of a general object X including various colors, a red light source Lr that irradiates the object X, and a captured image that is captured when the object X is irradiated with the light source Lr. It is a schematic diagram of the spectrum distribution. In the spectrum of the captured image in FIG. 9A, the intensity on the short wavelength side (blue side) decreases and the intensity on the long wavelength side (red side) increases. That is, an image captured by irradiating a general object with a red light source has a correlation such that redness increases in a bright region in the image. However, this correlation occurs when there are a plurality of colors in the screen. On the other hand, FIG. 9B shows an image of a red object Xr from the left, a white light source Lh (for example, a light source having a color temperature of 5,500 [K (Kelvin)]) irradiated on the object Xr, and the light source Lh as an object. It is a schematic diagram of each spectrum distribution of the captured image imaged when it irradiated to Xr. Since the spectral distribution of the white light source is flat, in this case, the spectral distribution of the imaging target is detected as it is. Therefore, in this case, there is no correlation between luminance and color. In addition, this correlation has not only redness but also blueness.

  FIG. 10 is a schematic diagram of a spectral distribution when a red light source is irradiated. FIG. 10A illustrates, from the left, the image of the red region Sr, the red light source Lr that irradiates the red region Sr, and the spectrum of each captured image that is captured when the red region Sr is irradiated with the light source Lr. It is a schematic diagram of distribution. FIG. 10B shows, from the left, the image of the green region Sg, the red light source Lr that irradiates the green region Sg, and the spectrum of each captured image that is captured when the light source Lr is irradiated to the green region Sg. It is a schematic diagram of distribution. FIG. 10C shows, from the left, the image of the blue-based region Sb, the red light source Lr that irradiates the blue-based region Sb, and the spectrum of each captured image captured when the light-source region Lb is irradiated to the blue-based region Sb. It is a schematic diagram of distribution. Since the blue-based region Sb has less energy on the long wavelength side, when the red light source Lr is irradiated to the blue-based region Sb, it becomes darker than when the red light source Lr is irradiated to the red-based region Sr. On the other hand, the energy on the short wavelength side included in the reflected light of the blue region Sb is attenuated by the spectral characteristics of red light, but the blue region Sb has a shorter wavelength than when red light strikes the red region Sr. The component remains (because the spectrum on the short wavelength side is large in the blue region). In other words, the negative correlation is such that the darker the color, the greater the blueness. The above is the principle called “luminance-color correlation”.

  The advantage of this method is that it is possible to estimate rough light source characteristics (white or non-white) with a small amount of computation. This is because it is not necessary to consider the detailed spectral characteristics of the light source and the object surface as compared with the method using the spectral reflection characteristics described above. Therefore, it has sufficient light source type estimation performance to reduce the color feria problem, and the calculation cost is low, so it is suitable for mounting embedded devices.

  Japanese Patent Application Laid-Open No. H10-228561 describes that a spectral characteristic indicating the color of a light source that irradiates a subject is estimated from response values of sensors having a plurality of different spectral sensitivity characteristics.

Japanese Patent No. 3767541

E. H. Land, “Recent Advance in Retinex Theory”, Vision Research, 26, 1986. D. H. Brainard & W. T. Freeman, “Baysian Color Constancy”, J. Opt. Soc. Am. A, Vol. 14, No. 7, 1997. J. Golz & D. I. A. Macleod, “Influence of scene statistics on color constancy”, NATURE, VOL. 415, 7, FEBRUARY, 2002.

However, there is a problem that the above-described luminance-color correlation is affected by the degree of image texture due to the camera position and the light source intensity. For example, if the color texture of the image is flat and the number of colors is small, the luminance-color correlation may become unstable because the luminance and color input data distribution does not spread. In order to ensure stable light source estimation performance in an actual environment, it is necessary to reduce the influence of this problem.
Also, since the luminance-color correlation assumes a normal distribution of input data, if the number of input data is small or there is a deviation value in the data, the normal distribution is not assumed and the correct luminance-color correlation cannot be calculated. There is. In addition, in a scene where some pixels are white-out due to specular reflection of an object and a highlight area exists, these pixels may become outliers and the luminance-color correlation may become unstable.
In addition, the technique described in Patent Document 1 has a problem that the amount of calculation increases because the spectral characteristic indicating the color of the light source is estimated based on the response value of the sensor. In addition, the imaging means must include a plurality of sensors having different spectral sensitivity characteristics.

  The present invention has been made in view of the above points, and an object thereof is to provide a light source capable of accurately estimating a light source with a small amount of calculation even when the color texture of the image is flat and the number of colors is small. An object of the present invention is to provide an estimation device and a light source estimation method.

  SUMMARY An advantage of some aspects of the invention is that a correlation coefficient between luminance and color is calculated from an image signal generated by an imaging element that images a subject. A luminance-color correlation calculating unit, a first color balance calculating unit that calculates a color balance of the entire image in the image signal, and a color balance of a neutral gray region in which the average color is a gray region in the image signal A second color balance calculation unit, a feature vector generation unit that generates a feature vector based on the correlation coefficient, a color balance of the entire image, and a color balance of the neutral gray region, and the feature vector generation unit And a discriminator for determining the type of light source in the image signal based on the feature vector generated by Is the source estimating apparatus.

  According to this invention, in addition to the correlation coefficient between brightness and color, the type of light source is determined based on the color balance of the entire image and the color balance of the neutral gray area. As a result, even when the color texture of the image is flat and the number of colors is small, the light source can be accurately estimated with a small amount of calculation.

  According to another aspect of the present invention, the light source estimation device includes a texture analysis unit that calculates a color texture feature indicating color diversity in the image signal, and the feature vector generation unit includes the color texture feature. Based on this, the feature vector is generated.

  According to this invention, in addition to the correlation coefficient between luminance and color, the color balance of the entire image, and the color balance of the neutral gray area, the type of light source is determined based on the color texture characteristics. Thereby, a light source can be estimated more accurately.

  According to another aspect of the present invention, the light source estimation device includes a luminance unevenness analysis unit that specifies a constant luminance region in which the luminance is constant in the image signal, and the luminance-color correlation calculation unit analyzes the luminance unevenness analysis. The correlation coefficient is calculated in a constant luminance region specified by the unit.

  According to the present invention, since the correlation coefficient is calculated in the constant luminance region, the correlation coefficient can be calculated without depending on the position and intensity of the light source. Thereby, color feria can be reduced.

  According to another aspect of the present invention, the light source estimation device includes a time-series data buffer that sequentially stores the image signals generated by the imaging element in time series, and the luminance-color correlation calculation unit includes the time series The correlation coefficient is calculated from a plurality of image signals stored in.

  According to the present invention, since the correlation coefficient is calculated using a plurality of image signals, the correlation coefficient can be calculated with high accuracy. In addition, since time series data is used, it is possible to cope with temporal changes in image information caused by camera shake, movement of a subject existing in a scene, changes in illumination conditions, and the like.

  According to another aspect of the present invention, in the above-described light source estimation device, the discriminator is configured based on learning data including feature vectors of a plurality of image signals and types of light sources corresponding to the image signals. A learning unit that learns a criterion for deriving a type of light source; and an identification unit that determines a type of light source corresponding to the feature vector generated by the feature vector generation unit based on a learning result by the learning unit. It is characterized by providing.

  According to the present invention, since the light source type is determined based on the learning result by the opportunity learning, it is not necessary to manually set a determination criterion for deriving the light source type from the feature vector.

  According to another aspect of the present invention, a luminance-color correlation calculation unit of the light source estimation device calculates a correlation coefficient between luminance and color from an image signal generated by an image sensor that images a subject. A step in which a first color balance calculation unit of the light source estimation device calculates a color balance of the entire image in the image signal; and a second color balance calculation unit of the light source estimation device calculates an average color in the image signal. Calculating a color balance of a neutral gray area in which the gray area is a gray area; and a feature vector generation unit of the light source estimation device includes the correlation coefficient, the color balance of the entire image, and the color balance of the neutral gray area. And a classifier of the light source estimation device is generated by the feature vector generation unit. Based on symptom vector a light source estimation method characterized by having a step of determining the type of the light source in the image signal.

  According to another aspect of the present invention, in the above light source estimation method, the texture analysis unit of the light source estimation device includes a step of calculating a color texture feature indicating color diversity in the image signal, and the feature vector The generation unit generates the feature vector based on the color texture feature.

  In addition, according to another aspect of the present invention, in the light source estimation method, the luminance unevenness analysis unit of the light source estimation device includes a step of identifying a constant luminance region where the luminance is constant in the image signal, and the luminance-color The correlation calculation unit calculates the correlation coefficient in a constant luminance region specified by the luminance unevenness analysis unit.

  According to another aspect of the present invention, in the above light source estimation method, the light source estimation device includes a time-series data buffer that sequentially stores the image signals generated by the imaging device in time series, and calculates a luminance-color correlation. The unit calculates the correlation coefficient from a plurality of image signals stored in the time series.

  According to another aspect of the present invention, in the light source estimation method, the classifier is based on learning data in which a learning unit of the classifier includes a plurality of image signals and types of light sources corresponding to the image signals. Learning a criterion for deriving the type of light source from the feature vector, and the discriminator of the discriminator corresponds to the feature vector generated by the feature vector generator based on the learning result by the learning unit Determining the type of light source to be used.

  According to the present invention, in addition to the correlation coefficient between luminance and color, the type of light source is determined using the color texture feature, the color balance of the entire image, and the color balance of the neutral gray area. As a result, even when the color texture of the image is flat and the number of colors is small, the light source can be accurately estimated with a small amount of calculation.

It is a block diagram which shows the function structure of the white balance adjustment apparatus by this embodiment. It is a block diagram which shows the function structure of the light source kind estimation part by this embodiment. It is a flowchart which shows the procedure of the light source kind estimation process by this embodiment. It is a flowchart which shows the procedure of the feature vector calculation process by this embodiment. It is a graph which shows the correlation of the brightness | luminance and color in the input image imaged by irradiating the red light source with respect to the general to-be-photographed object by this embodiment. It is a graph which shows the correlation of the brightness | luminance and color in the input image imaged by irradiating the general subject by this embodiment with a white light source. It is the two-dimensional scatter diagram which plotted the color-luminance correlation coefficient by this embodiment. It is a graph which shows the three-dimensional plot of the feature vector by this embodiment. It is a schematic diagram of the spectrum distribution for demonstrating luminance-color correlation. It is a schematic diagram of the spectrum distribution for demonstrating luminance-color correlation.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a functional configuration of the white balance adjusting apparatus 1 according to the present embodiment.
The white balance adjusting device 1 is a device that adjusts the white balance of an image signal in a device that converts a subject optical image and electrically outputs an image signal such as a digital video camera, a digital still camera, or a mobile terminal with a camera. is there. The white balance adjustment device 1 includes a lens 101, an iris 102, an image sensor 103, an AGC (Automatic Gain Control) unit 104, a preprocessing unit 105, an image information calculation unit 106, an illuminance detection unit 107, and a drive. Unit 108, light source type estimation unit 109, white balance gain calculation unit 111, white balance control unit 112, color correction control unit 113, gamma correction control unit 114, and output unit 115. .

  The lens 101 guides an optical image to the light receiving surface of the image sensor 103. The iris 102 adjusts the light output from the lens 101 to the image sensor 103. The image sensor 103 photoelectrically converts the optical image formed on the light receiving surface to generate an image signal, and outputs the generated image signal to the AGC unit 104. The AGC unit 104 makes the level of the input image signal constant based on the illuminance detected by the illuminance detection unit 107, and outputs the image signal with the constant level to the preprocessing unit 105. The preprocessing unit 105 performs preprocessing such as noise removal, defect correction, and shading correction on the input image signal, and outputs the preprocessed image signal to the image information calculation unit 106 and the white balance control unit 112. To do.

  The image information calculation unit 106 reduces the input image signal to generate a reduced image signal for white balance calculation, and uses the generated reduced image signal for the illuminance detection unit 107, the light source type estimation unit 109, and the white balance gain calculation unit. And output to 111. The illuminance detection unit 107 calculates the illuminance used for the exposure process based on the input reduced image signal, and outputs the calculated illuminance to the AGC unit 104, the drive unit 108, and the white balance gain calculation unit 111. The driving unit 108 drives the iris based on the input illuminance and adjusts the exposure.

  The light source type estimation unit (light source estimation device) 109 calculates an image feature amount (feature vector) based on the “luminance-color correlation” principle using the input reduced image signal, and the light source type (type of light source). Is white or non-white. Then, the light source type estimation unit 109 outputs the determination result to the white balance gain calculation unit 111. Details of the light source type estimation unit 109 will be described later.

  The white balance gain calculation unit 111 calculates a white balance gain based on the light source type determined by the light source type estimation unit 109 and the illuminance detected by the illuminance detection unit 107. Specifically, the white balance gain calculation unit 111 reads the white balance gain corresponding to the type of light source from the white balance tuning parameter storage unit 110 and outputs the read white balance gain to the white balance control unit 112. The white balance tuning parameter storage unit 110 stores a white balance tuning parameter indicating a white balance gain optimized for each light source type.

  In this embodiment, the white balance gain calculation unit 111 calculates the white balance gain based on the white balance tuning parameter. However, the white balance gain calculation unit 111 calculates the white balance gain using a general white balance calculation algorithm. Also good. For example, the white balance gain calculation unit 111 calculates the white balance gain so that the color balance is gray if the color of the entire reduced image is the light source type, and the color of the entire reduced image is an object other than the light source color. If it is a color, the white balance gain that remains without correcting the color is calculated. This method has the advantage of high system compatibility because there is no need to change the original white balance algorithm. Still another example will be described. If the type of light source can be estimated, the range of subject brightness (Brightness Value) when each light source is irradiated can be limited. For this reason, it is conceivable to use the subject luminance range as a parameter for white balance gain calculation. A general white balance calculation algorithm often includes logic for estimating the light source type from the subject brightness, but depending on the camera system, the subject brightness cannot be acquired because the lens 101 and the main body (after the image sensor 103) cannot communicate. The accuracy of white balance calculation may be affected. According to this example, since the subject luminance range is estimated to some extent by estimating the light source type, it is possible to prevent deterioration in accuracy of white balance calculation.

  The white balance control unit 112 performs white balance adjustment on the image signal input from the preprocessing unit 105 based on the input white balance gain. Then, the white balance control unit 112 outputs the white balance adjusted image signal to the color correction control unit 113. The color correction control unit 113 performs color correction on the input image signal, and outputs the color-corrected image signal to the gamma correction control unit 114. The gamma correction control unit 114 performs gamma correction on the input image signal and outputs the gamma corrected image signal to the output unit 115. The output unit 115 outputs the input image signal to an external storage medium, a display, a network, or the like.

FIG. 2 is a block diagram illustrating a functional configuration of the light source type estimation unit 109 according to the present embodiment.
The light source type estimation unit 109 includes a time series data buffer 202, a luminance unevenness analysis unit 203, a feature amount calculation unit 204, and a discriminator 211.
The time-series data buffer 202 receives the reduced image signal 201 and stores image data of the input reduced image signal (hereinafter referred to as reduced image data) in time series for each frame. The time-series data buffer 202 is a ring buffer secured on a flash memory such as a DRAM (Dynamic Random Access Memory), for example, and can store a predetermined number of frames of image data. The reduced image data stored in the time series data buffer 202 is sequentially input to the luminance unevenness analysis unit 203.

  The luminance unevenness analysis unit 203 specifies a constant luminance region and a highlight pixel in units of frames. The constant luminance region is a region where the luminance is constant when the position of the light source is assumed. The highlight pixel is a dichroic reflection model (for example, SAShafer, “Using color to separate reflection components,” in COLOR Research and Application, Vol. 10, No. 4, pp. 210-218, 1985. This is an extremely bright pixel obtained by (see Fig.). Then, the luminance unevenness analysis unit 203 calculates the feature amount of luminance unevenness information that is information (area, coordinates) of the specified constant luminance region, highlight pixel information that is information (coordinates) of highlight pixels, and reduced image data. Output to the unit 204. Details of processing in the luminance unevenness analysis unit 203 will be described later.

  The feature amount calculation unit 204 calculates a feature amount (hereinafter referred to as a feature vector) for determining the light source type in the reduced image data, and outputs the calculated feature vector to the classifier 211. The feature amount calculation unit 204 includes a luminance-color correlation calculation unit 205, a texture analysis unit 206, an average color calculation unit 207, a neutral gray area search unit 208, a color balance calculation unit 209, and a feature vector calculation unit 210. It is comprised including.

  The luminance-color correlation calculation unit 205 corresponds to reduced image data of K frames (K is an integer of 2 or more) arranged in time series, luminance unevenness information corresponding to each reduced image data, and each reduced image data. Based on the highlight pixel information, a luminance-color correlation coefficient indicating a correlation between luminance and color is calculated. This luminance-color correlation coefficient is a feature quantity that can accurately estimate the type of light source for an image with a large number of color textures (a large number of colors). Then, the luminance-color correlation calculating unit 205 outputs the calculated luminance-color correlation coefficient to the feature vector calculating unit 210. Details of the processing in the luminance-color correlation calculation unit 205 will be described later.

  The texture analysis unit 206 calculates color texture features from the reduced image data of K frames arranged in time series, and outputs the calculated color texture features to the feature vector calculation unit 210. The color texture feature is an index representing color diversity. Details of processing in the texture analysis unit 206 will be described later.

  The average color calculation unit 207 calculates the average color of each of the RGB channels in the reduced image data, and outputs the calculated average color and reduced image data to the color balance calculation unit 209. The neutral gray area search unit 208 searches for a neutral gray area in the reduced image data, and determines whether or not there is a neutral gray area in the reduced image data. The neutral gray area is an area where the average color is gray. Then, the neutral gray area search unit 208 outputs the presence / absence of the neutral gray area and the reduced image data to the color balance calculation unit 209.

  The color balance calculation unit 209 calculates the color balance of the entire image in the reduced image data from the average colors of the RGB channels calculated by the average color calculation unit 207. Further, the color balance calculation unit 209 calculates the color balance of the neutral gray area searched by the neutral gray area search unit 208. Then, the color balance calculation unit 209 outputs the calculated color balance of the entire image and the color balance of the neutral gray area to the feature vector calculation unit 210. Details of the color balance calculation method will be described later. The combination of the color balance of the entire image and the color balance of the neutral gray region is a feature quantity that can accurately estimate the type of light source for an image with little color texture.

  The feature vector calculation unit 210 converts the luminance-color correlation calculated by the luminance-color correlation calculation unit 205, the color texture feature extracted by the texture analysis unit 206, and the color balance calculated by the color balance calculation unit 209. Based on this, a feature vector (luminance-color correlation, color texture feature, color balance of the entire image, color balance of the neutral gray region) is generated and output to the learning unit 212 or the identification unit 215 of the classifier 211.

  The discriminator 211 performs machine learning on a criterion for deriving the light source type from the feature vector based on learning data including a feature vector of a plurality of image data and a light source type (teaching data) corresponding to each image data. At this time, the identification unit 11 performs machine learning using boosting (Boosting), support vector machine (SVM), decision tree, or Random Forest that is an extension of the decision tree. Then, the classifier 211 determines the light source type corresponding to the feature vector input by the feature amount calculation unit 204 based on the learned result. The classifier 211 includes a learning unit 212 and a classifying unit 215. The discriminator 211 is machine-learned in the present embodiment, but may determine the light source type based on a judgment criterion by manual operation, for example.

  The learning unit 212 performs machine learning on the criterion for deriving the light source type from the feature vector based on the learning data, and writes the learning result in the learning result storage unit 214. The learning result storage unit 214 is an external storage device that is external to the white balance adjustment device 1, and stores the learning result of the learning unit 212. Note that the learning unit 212 writes the learning result in a flash memory or the like provided in the white balance adjustment device 1 when performing the light source type estimation process following the learning.

  The identification unit 215 determines the light source type corresponding to the feature vector input by the feature vector calculation unit 210 based on the learning result from the learning unit 212, and outputs the determination result to the white balance gain calculation unit 111. For example, the identification unit 215 determines that the luminance vector is large when the color texture feature is large (the color texture is large) in the feature vector (luminance-color correlation, color texture feature, color balance of the entire image, color balance of the neutral gray region). The light source type is determined by increasing the weight for the color correlation, and if the color texture feature is small (the color texture is small), the light source is set by increasing the weight for the combination of the color balance of the entire image and the color balance of the neutral gray area. Determine the type.

Next, the light source type estimation process by the light source type estimation unit 109 will be described with reference to FIG. FIG. 3 is a flowchart showing the procedure of the light source type estimation process according to this embodiment.
First, the identification unit 215 reads a learning result from the learning result storage unit 214 (step S10). Next, the light source type estimation unit 109 writes the reduced image data in the time series data buffer 202 (step S20).

  Next, the light source type estimation unit 109 determines whether or not the number of reduced image data written in the time series data buffer 202 is equal to or greater than a predetermined threshold (step S30). If the number of reduced image data is greater than or equal to the threshold, the process proceeds to step S40. If the number of reduced image data is less than the threshold, the process returns to step S20. The purpose of this branching is to secure a sufficient number of data in order to stably calculate the luminance-color correlation coefficient described later.

Next, the luminance unevenness analysis unit 203 specifies a constant luminance region and a highlight pixel in units of frames by luminance unevenness analysis (step S40). Specifically, the luminance unevenness analysis unit 203 specifies a constant luminance region using the following region approximation method. The luminance unevenness depends on the light source position and the shape / arrangement of the object. For this reason, in this region approximation method, it is assumed that the light source position has eight directions (up, down, left, and right) with the front as the center, and further, the depth size of the object is assumed to be sufficiently small compared to the distance from the camera to the object. The constant luminance region is specified on the assumption that the illumination intensity is constant in any one of the horizontal, vertical, and diagonal directions on the image.
Note that the present invention is not limited to this method, and the constant luminance region may be specified using another region approximation method. For example, there is a region approximation method for dividing an image region. This region approximation method is a method in which a shadow region or a region that is whitened by specular reflection is divided into regions, and only a region having a constant luminance level is used. The shadow area can be obtained by color and luminance clustering. The specularly reflected region can be estimated using, for example, a dichroic reflection model. Note that these two region approximation methods may be used in combination.

Next, the feature amount calculation unit 204 calculates a feature vector for determining the light source type based on the luminance unevenness information, which is information of the constant luminance region, highlight information, and reduced image data (step S50). ). Details of this feature vector calculation process will be described later.
Next, the identification unit 215 calculates a score for the calculated feature vector for each light source type based on the read learning result (step S60). And the identification part 215 determines a light source type based on the score calculated for every light source type (step S70). Specifically, the identification unit 215 sets the light source type having a high score as the light source type corresponding to the calculated feature vector.

Next, the feature vector calculation process in step S50 described above will be described with reference to FIG. FIG. 4 is a flowchart showing the procedure of the feature vector calculation process according to this embodiment.
First, the feature amount calculation unit 204 receives reduced image data of K frames arranged in time series, luminance unevenness information corresponding to each reduced image data, and a high level corresponding to each reduced image data from the luminance unevenness analysis unit 203. Light pixel information is acquired (steps S100, S110, S120). Next, the feature amount calculation unit 204 performs a luminance-color correlation calculation process, a color texture analysis process, a color balance calculation process for the entire image, and a color balance calculation process for the neutral gray area in parallel. In the present embodiment, the luminance-color correlation calculation process, the color texture analysis process, the color balance calculation process for the entire image, and the color balance calculation process for the neutral gray area are performed in parallel. You may go.

[Luminance-Color Correlation Calculation Processing (Step S140)]
First, luminance-color correlation calculation processing will be described.
The luminance-color correlation calculating unit 205 is based on the reduced image data of K frames arranged in time series, luminance unevenness information corresponding to each reduced image data, and highlight pixel information corresponding to each reduced image data. The luminance-color correlation coefficient is calculated (step S140).
Here, the reduced image data of the K frame is used in order to stably calculate a correlation coefficient described later. The calculation of the correlation coefficient cannot be performed stably unless the data distribution is a normal distribution. However, if the number of data used for calculating the correlation coefficient is small, it is not known whether the data distribution is a normal distribution. For this reason, the number of data for calculating the correlation coefficient is secured using a plurality of frames. The reason for using time-series data is to cope with temporal changes in image information caused by camera shake, movement of subjects existing in the scene, changes in lighting conditions, and the like.

Specifically, first, the luminance-color correlation calculation unit 205 removes regions other than the luminance constant region and highlight pixels from each reduced image data. This is because when the light source position is biased or there is an extreme difference in the reflection characteristics of the object surface, there is a large difference in the luminance distribution of the captured image (shading), and the luminance and color data distribution is normal. This is because the calculation of the luminance-color correlation coefficient becomes unstable without being distributed.
Next, the luminance-color correlation calculation unit 205 divides each reduced image data from which regions other than the constant luminance region and highlight pixels are removed into N (N is an integer of 2 or more) vertically and horizontally. Hereinafter, the divided area is referred to as a divided block. Note that the value of N is arbitrary. For example, the pixel may be a one-division block. Then, the luminance - color correlation calculating unit 205, dividing the block i (i = 1,2, ..., N × K) for each and the average value _{R i} of R channels, an average value _{G i} of the G channel, and B channel An average value B _i is calculated. Then, the luminance - color correlation calculator 205 calculates a luminance Lum _i of each divided block i by the following equation (1).

Next, the luminance - color correlation calculation unit 205, by the following equation (2), to calculate the red redness _i of each divided block i.

Next, the luminance-color correlation calculation unit 205 calculates the blueness blue _i of each divided block i by the following equation (3).

Next, the luminance - color correlation calculator 205 calculates the correlation coefficient of Pearson represented a correlation coefficient _{F r} of luminance Lum and redness Redness by the following equation (4).

  Hereinafter, a character with “-” in the upper part of the character in the formula is indicated by (−) in front of the character in the sentence. n is the number of data, and n = N × K. (−) Lum is an average value of the luminance Lum. (−) Redness is an average value of redness Redness.

Then, the luminance - color correlation calculator 205 calculates the correlation coefficient of Pearson represented a correlation coefficient _{F b} of luminance Lum and bluish Blueness by the following equation (5).

  However, (-) Blueness is an average value of bluish blueness.

Finally, the luminance-color correlation calculation unit 205 sets the luminance-color correlation coefficient F = (F _r , F _b ). The calculation of the correlation coefficient uses the Pearson product-moment correlation coefficient expressed by the equations (4) and (5), which assumes a normal distribution of the data. For this reason, when the number of data is small or there is an outlier (Outlier), a correct correlation coefficient may not be calculated. For this reason, in the present embodiment, the correlation coefficient is calculated by securing the number of data by image storage in the time series buffer (step S20). Further, by removing the luminance variation and the highlight pixel due to the luminance unevenness analysis (step S40), the error factor of the correlation coefficient calculation is reduced.
In addition to these factors, there is a possibility that a deviation value is mixed. For example, this is a case where there is a region that emits light in the scene. In such a case, since the outlier value of the data cannot be specified by the luminance unevenness analysis, there is a possibility that the data distribution does not become a normal distribution. Therefore, in such a case, Spearman's rank correlation coefficient that does not assume normal distribution of data may be used.

FIG. 5 is a graph showing a correlation between luminance and color in an input image captured by irradiating a general subject according to the present embodiment with a red light source.
FIG. 5A is an input image captured by irradiating a general subject including various colors with a red light source (here, a halogen lamp). FIG. 5B is a graph plotting the change in redness Redness with respect to the luminance Lum in the input image shown in FIG. The vertical axis in this graph is redness Redness, and the horizontal axis is luminance Lum. As shown in the figure, the luminance Lum and the redness Redness have a positive correlation. FIG. 5C is a graph in which changes in bluish blueness with respect to the luminance Lum in the input image shown in FIG. 5A are plotted. In this graph, the vertical axis is bluish blueness, and the horizontal axis is luminance Lum. As shown in the figure, the luminance Lum and the bluish blueness have a negative correlation.

FIG. 6 is a graph showing a correlation between luminance and color in an input image captured by irradiating a general subject with a white light source according to the present embodiment.
The graph shown in this figure is an input imaged by irradiating a general object including various colors shown in FIG. 5A with a white light source (here, a white light source having a color temperature of 5,500 [K]). A change in color with respect to luminance in the image H is shown. FIG. 6A is a graph in which changes in redness Redness with respect to the luminance Lum in the input image H are plotted. The vertical axis in this graph is redness Redness, and the horizontal axis is luminance Lum. As shown in the figure, there is no correlation between the luminance Lum and the redness Redness. FIG. 6B is a graph in which changes in blueness with respect to luminance in the input image H are plotted. In this graph, the vertical axis is bluish blueness, and the horizontal axis is luminance Lum. As shown in the figure, there is no correlation between the luminance Lum and the bluish blueness.
From these facts, it can be confirmed that the luminance-color correlation occurs only when the red light source is irradiated.

FIG. 7 is a two-dimensional scatter diagram in which color-luminance correlation coefficients according to the present embodiment are plotted.
The horizontal axis in the graph of the 2-dimensional scatter diagram is the correlation coefficient F _r of the luminance and red and the vertical axis represents the correlation coefficient F _b of luminance and blue. There are five types of plot legends: three subjects imaged by irradiating a red light source (subjects including various colors, 24-color Macbeth color chart, gray chart), and two subjects imaged by irradiating a white light source ( A subject including various colors, a red subject (red object)). For each legend, ten images were taken at different imaging positions. From this figure, it can be seen that the legend imaged with a red light source has a large luminance-redness correlation and a small luminance-blueness correlation, so that the plot is concentrated in the lower right region of the graph. On the other hand, in the legend imaged with a white light source, it can be seen that the plot is scattered in an area other than the area at the lower right of the graph. Therefore, if the learning boundary 212 can specify an identification boundary for detecting the lower right region of the graph shown in FIG. 7, the plot that has entered the region can be determined as a red light source. However, in FIG. 7, the red light source plot and the white light source plot are partially mixed in the lower right region. As a factor of this phenomenon, the correlation coefficient calculation becomes unstable because the number of colors contained in the textureless subject is small, and the value of the luminance-redness correlation is increased accidentally even though the white light source is irradiated. It is possible. Therefore, in order to complement the luminance-color correlation feature, three feature amounts of a color texture feature, a color balance of the entire screen, and a color balance of the neutral gray region are introduced.

[Color Texture Analysis Processing (Step S150)]
Returning to FIG. 4, the color texture analysis processing will be described next.
The texture analysis unit 206 performs color texture analysis of the reduced image data of K frames arranged in time series (step S150). First, the texture analysis unit 206 converts the RGB signal of the reduced image data into an HSV signal. H represents Hue (hue), S represents Saturation, and V represents Vlightness (luminance). Next, the texture analysis unit 206 quantizes the H component range in the quantization step, and generates a histogram including L bins. Let the histogram be h (i) (i = 1, 2,..., L). Then, the texture analysis unit 206 calculates, from this histogram, a relative histogram p (i) in which the influence of pixels regarded as noise, here, extremely dark or bright pixels is canceled, from the following equation (6).

  Here, M is the number of pixels included in the reduced image data, and e is the number of pixels regarded as noise. Subsequently, the texture analysis unit 206 calculates entropy E represented by the following equation (7).

This entropy E is a color texture feature indicating color diversity. As shown in the following equation (8), this entropy E takes the maximum value E _max when the relative frequencies p (i) of the bins of the histogram are equal. That is, as the number of types of colors in the image increases, the histogram becomes flat. Therefore, the number of types of colors in the image increases, and the value of entropy E increases.

  The reason why the entropy E of the above formula (7) is used as the feature quantity for the light source type estimation will be described below. In an extreme example, if the entire screen is filled with a single color, the plot of brightness and redness (or blueness) will be concentrated at one point, and the correlation coefficient between brightness and redness (or blueness) can be calculated. Disappear. The correlation coefficient cannot be calculated unless the data distribution is wide to some extent, which indicates that there must be a plurality of colors in the screen. Also, from the spectral characteristics of the light source and the object surface, when the entire screen is a single color, it cannot be distinguished whether the color is due to the light source or the original object color. Therefore, an index representing the color diversity as shown in equation (7) is required. The index of color diversity is not limited to entropy, and for example, a variance value can be used.

  In this embodiment, the entropy E calculated by the color texture analysis is used as one element of the feature vector for estimating the light source type, but may be a weight for the correlation coefficient calculated by the luminance-color correlation calculation. That is, the correlation coefficient of a pixel (or region) having a large entropy can be regarded as highly reliable and given a large weight, and then sent to the identification unit 115 to estimate the light source type.

[Color Balance Calculation Processing for Entire Image (Steps S160 and S165)]
Next, the color balance calculation process for the entire image will be described.
First, the average color calculation unit 207 calculates the average value (−) R of the R signal, the average value (−) G of the G signal, and the average value (−) B of the B signal in the entire reduced image data of K frames. calculate. (Step S160). Next, the color balance calculation unit 209 calculates the color balance CB _a = ((−) R / (−) G, (−) B / (−) G) of the entire image (step S165).

  The color balance of the entire image is based on the gray hypothesis model (Non-Patent Document 2). This model is based on the hypothesis that there are various colors throughout the image, and their average value is neutral gray. If the color balance deviates from neutral gray, it is determined that the deviation is due to the light source color. However, although the gray hypothesis is a simple and easy-to-use model, if red bricks and carpets are reflected on the entire screen, the color balance of the entire screen will shift to red, so it may be mistaken for a color cast. is there. In order to reduce such a problem, the color balance of the neutral gray area described below is added to the feature amount.

[Neutral Gray Area Color Balance Calculation Processing (Steps S170 to S190)]
Next, the neutral gray area color balance calculation process will be described.
First, the neutral gray area search unit 208 determines whether or not there is a neutral gray area in each frame (step S170). Specifically, the neutral gray area searching unit 208 scans a rectangular window of a predetermined size in the image, and determines that there is a neutral gray area when there is a rectangular area that satisfies the following expression (9). When there is no rectangular area satisfying (9), it is determined that there is no neutral gray area.

i is the x coordinate value of the reduced image data, and j is the y coordinate value of the reduced image data. (−) R _{i, j} is the R channel average value in the rectangular area centered on the coordinate value (i, j). (−) G _{i, j} is an average value of the G channel in the rectangular area centered on the coordinate value (i, j). (−) B _{i, j} is the B channel average value in the rectangular area centered on the coordinate value (i, j).

  Next, the color balance calculation unit 209 calculates the color balance C of the frame determined by the neutral gray region search unit 208 as having a neutral gray region using the following equation (10) (step S180).

Here, D is the image area of the reduced image data. Then, the color balance calculation unit 209 extracts the minimum value from the color balance C of each frame, and uses the extracted minimum value as the color balance CB _{n of} the neutral gray area (step S190).

  The color balance of the neutral gray region is a feature quantity that supports the gray hypothesis and the luminance-color correlation principle. In the gray hypothesis, if a red brick or carpet is reflected on the entire screen, the color balance of the entire screen approaches red, so it is mistaken that a color cast has occurred. At this time, if there is an area where the color balance is neutral gray (for example, an area where the concrete surface can be seen even a little), the red color of the red brick or carpet can be expected to be the object color. On the contrary, if the color cast is caused by a light source, such a neutral gray region is also reddish. In order to identify this, the neutral gray area searching unit 208 searches for the neutral gray area by the above-described equation (9).

  This feature amount (color balance of the neutral gray region) is effective even when the color texture of the image is small. For this reason, for example, when the value of the entropy E obtained by the color texture analysis is small, that is, when the image is textureless, the identification unit 215 gives a large weight to the color balance of the neutral gray region, and conversely the luminance -Reduce the weight for the color correlation coefficient. This makes it possible to estimate the light source type independent of the scene texture.

  In the present embodiment, the neutral gray area search unit 208 searches for a neutral gray area by scanning a rectangular window in the image. At this time, the rectangular window to be scanned is changed to an area that is colored with the neutral gray area. In the case of crossing or when a neutral area does not exist in a certain frame in a moving image, a procedure for detecting a frame in which a rectangular area falls within a neutral gray area can be considered. This is the minimum value extraction in step S190 described above.

[Feature Vector Calculation Processing (Step S200)]
Finally, the feature vector calculation unit 210 calculates a feature vector V = (luminance-color correlation coefficient F, color texture feature E, overall image color balance CB _a , neutral gray color balance CB _n ), and performs processing. The process ends (step S200). In the present embodiment, four feature amounts have been all feature vectors, at least luminance - hue correlation coefficient F, the whole image and color balance CB _a, the color balance CB _n and the feature vector of the neutral gray area It only has to be included.

FIG. 8 is a graph showing a three-dimensional plot of feature vectors according to this embodiment.
The graph shown in this figure is the correlation coefficient F _r brightness and redness, and color balance CB _a of the entire image, and the axial and color balance CB _n neutral gray area. There are five types of plot legends: three subjects imaged by irradiating a red light source (subjects including various colors, 24-color Macbeth color chart, gray chart), and two subjects imaged by irradiating a white light source ( A subject including various colors, a red subject (red object)). For each legend, ten images were taken at different imaging positions. As shown in FIG. 8, by adding the color balance CB _{a of the} entire image and the color balance CB _n of the neutral gray area to the correlation coefficient F _r of luminance and red, the plot of the red light source and the plot of the white light source You can see how it becomes easier to separate.

  As described above, in the present embodiment, the feature amount calculation unit 204 collects four feature amounts (luminance-color correlation coefficient, color texture feature, entire screen color balance, neutral gray area color balance) as a feature vector, and an identification unit. At 115, the light source type is determined. For this reason, since the white balance gain calculation unit 111 can calculate an appropriate white balance for each type of light source, color failure can be reduced.

Further, by recording a program for realizing each step shown in FIGS. 3 and 4 on a computer-readable recording medium, causing the computer system to read and execute the program recorded on the recording medium, the light source type An estimation process or a feature vector calculation process may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices.
Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to
For example, in this embodiment, the explanation of the principle of luminance-color correlation and the application method are described assuming a red light source, but a light source containing a lot of green components such as a fluorescent light or an outdoor sunny day. The same applies to a light source having a high color temperature and containing a blue component.
The white adjustment device 1 according to the present embodiment can also be used for an image search in an electronic image storage device such as a hard disk recorder, an electronic video device that enables image quality adjustment according to external light, and the like.

  DESCRIPTION OF SYMBOLS 1 ... White balance adjustment apparatus 101 ... Lens 102 ... Iris 103 ... AGC part 105 ... Pre-processing part 106 ... Image information calculation part 107 ... Illuminance detection part 108 ... Drive part 109 ... Light source type estimation part 110 ... White balance tuning parameter memory | storage part DESCRIPTION OF SYMBOLS 111 ... White balance gain calculation part 112 ... White balance control part 113 ... Color correction control part 114 ... Gamma correction control part 115 ... Output part 202 ... Time series data buffer 203 ... Brightness nonuniformity analysis part 204 ... Feature-value calculation part 205 ... Luminance -Color correlation calculation unit 206 ... Texture analysis unit 207 ... Average color calculation unit 208 ... Neutral gray area search unit 209 ... Color balance calculation unit 210 ... Feature vector calculation unit 211 ... Classifier 212 ... Learning unit 215 ... Identification unit

Claims (8)

A luminance-color correlation calculating unit that calculates a correlation coefficient between luminance and color from an image signal generated by an image sensor that images a subject;
A first color balance calculator that calculates the color balance of the entire image in the image signal;
A second color balance calculation unit that calculates a color balance of a neutral gray area in which the average color is a gray area in the image signal;
A texture analysis unit for calculating a color texture characteristic indicating color diversity in the image signal;
A feature vector generation unit that generates a feature vector based on the correlation coefficient, the color balance of the entire image, the color balance of the neutral gray region, and the color texture feature ;
A discriminator for determining the type of light source in the image signal based on the feature vector generated by the feature vector generation unit;
A light source estimation device comprising: A luminance unevenness analysis unit that identifies a constant luminance region in which the luminance is constant in the image signal;
Luminance - color correlation calculation unit, the light source estimating apparatus according to claim 1, characterized in that to calculate the correlation coefficient in the luminance constant region specified by the luminance unevenness analysis unit. A time series data buffer for sequentially storing the image signals generated by the image sensor in time series,
Luminance - color correlation calculation unit, the light source estimating device according to the plurality of image signals stored in the time series, to claim 1 or 2, characterized in that to calculate the correlation coefficient. The identifier is
A learning unit that learns a criterion for deriving the type of light source from the feature vector based on learning data consisting of feature vectors of a plurality of image signals and the type of light source corresponding to each image signal;
An identification unit for determining a type of a light source corresponding to the feature vector generated by the feature vector generation unit based on a learning result by the learning unit;
The light source estimation apparatus according to any one of claims 1 to 3, further comprising: A luminance-color correlation calculation unit of the light source estimation device calculating a correlation coefficient between luminance and color from an image signal generated by an image sensor that images a subject;
A first color balance calculation unit of the light source estimation device calculating a color balance of the entire image in the image signal;
A second color balance calculating unit of the light source estimating device calculating a color balance of a neutral gray area in which the average color is a gray area in the image signal;
Calculating a color texture feature indicative of color diversity in the image signal;
A feature vector generation unit of the light source estimation device generates a feature vector based on the correlation coefficient, the color balance of the entire image, the color balance of the neutral gray region, and the color texture feature ;
A discriminator of the light source estimation device determining the type of light source in the image signal based on the feature vector generated by the feature vector generation unit;
A light source estimation method comprising: The luminance unevenness analysis unit of the light source estimation device includes a step of specifying a constant luminance region where the luminance is constant in the image signal,
The light source estimation method according to claim 5 , wherein the luminance-color correlation calculation unit calculates the correlation coefficient in a constant luminance region specified by the luminance unevenness analysis unit. The light source estimation device includes a time-series data buffer that sequentially stores the image signals generated by the imaging device in time series,
The light source estimation method according to claim 5 , wherein the luminance-color correlation calculation unit calculates the correlation coefficient from the plurality of image signals stored in the time series. The identifier is
The learning unit of the classifier learns a determination criterion for deriving a light source type from a feature vector based on learning data composed of a plurality of image signals and a light source type corresponding to each image signal;
A step of determining a type of light source corresponding to the feature vector generated by the feature vector generation unit based on a learning result by the learning unit;
Light source estimation method according to claims 5 7 any one characterized in that it comprises a.

Images