JPH06323908A - Illumination-color estimation apparatus - Google Patents

Abstract

PURPOSE:To estimate the color distribution of illumination light on the basis of a standard dichroic reflection model without assuming the subject region of a white color from an RGB color image. CONSTITUTION:By a clustering operation which features a color an a position, a region division part 101 divides an image into small regions. While it is assumed that a standard dichroic reflection model is established regarding the small regions divided into regions, a color plane approximation part 102 find a color-vector correlation matrix, and it finds a normal-line vector in a color plane as a third inherent vector in the correlation matrix. A normal-line vector selection part 103 selects a color-plane normal-line vector inside a small region whose reliability is high, and an illumination-color vector most-plausible estimation part 104 estimates the color distribution of illumination light as a color vector perpendicular to the normal-line vector in a sense that the square sum of inner products becomes minimum.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention estimates the color of illumination light from an image, so that it can be used for, for example, white correction of a color video camera at the time of photographing an image, and color correction processing of an image or a photograph after photographing. The present invention relates to a color estimation device.

[0002]

2. Description of the Related Art When a white subject is illuminated with daylight and then with a tungsten bulb having a color temperature of 2700K, the observed spectral distribution is different. However, in the human visual system, it is perceived as white in both cases due to chromatic adaptation. In capturing an image such as a video camera or a photograph, it is desired to reproduce a white subject as white even under different illumination conditions in response to this chromatic adaptation. For this purpose, it is necessary to estimate the spectral distribution of illumination light (hereinafter referred to as illumination color). Conventionally, there is a technique for performing color correction using a color distribution of a captured image in association with an illumination color estimation device for this purpose. In the technique disclosed in "Automatic White Balance Adjusting Device" described in Japanese Patent Application Laid-Open No. 61-260790, an image formed of three primary color signals of red, green and blue (hereinafter abbreviated as RGB signal) is used as a reference. From the white signal, an area that is considered to be white in the image is determined, and the illumination color is consequently represented by the color difference from the reference white signal in that area. Further, as a conventional technique using an external illumination light sensor, Japanese Patent Laid-Open No. 61-24 has been disclosed.
A technique for estimating the illumination color is shown in "Automatic White Balance Adjustment Device for Color Video Camera" described in Japanese Patent No. 86. In the case of this conventional technique, the color temperature of the ambient light of the scene to be photographed is measured by two silicon photodiodes for the spectral intensities of red and blue to obtain the color temperature of the illumination light.

[0003]

In an apparatus that estimates the illumination color by considering the area close to the white reference signal as a white area from the color distribution of the captured image as described above, even if the object is not originally white, When observed near the reference white color due to the relationship, the estimation of the illumination color will be incorrect. Further, as a theoretical problem, color correction cannot be performed unless a white subject exists in the image. In the method using the external illumination light sensor, the illumination light is obtained from the external sensor, not from the captured image. Therefore, 1) the external sensor information does not always reflect the information of the illumination light that illuminates the subject. , 2) There is a problem that color correction cannot be performed on an image in which information from an external sensor is lost, for example, an image after being photographed.

[0004]

SUMMARY OF THE INVENTION In order to solve the above problems, the present invention divides an image composed of a multi-dimensional color vector, which is an array of luminances of different frequency components, into regions based on color similarity. Area dividing means, color plane approximating means for approximating the color vector distribution in each area divided by the area dividing means to obtain a normal vector of the plane, and each area obtained by the color plane approximating means A color vector estimating unit that selects a color vector whose inner product is small with respect to a normal vector group that is a subset of the set of normal vectors, and uses the color vector as a frequency component of illumination light. Is an illumination color estimation device.

[0005]

The present invention is based on the standard dichroic reflection model (Tominaga, Ohashi: Color reflection model of objects, IPSJ journals, V
ol33, No. 1, pp. 37-45, 1992)
Are using. This standard dichroic reflection model is a model in which the reflected light brightness Y of the insulator can be expressed by (Equation 1).

[0006]

[Equation 1]

In (Equation 1), λ is a wavelength in the visible light region,
θ is a geometrical parameter determined by the line of sight, the incident angle of illumination light, etc., S _S and S _D are the reflection coefficients of the specular reflection component and the perfect diffuse reflection component (S _S (λ) is constant regardless of wavelength), E is a spectral distribution of illumination light, and c _S and c _D are weighting factors. In this model, the reflected light observed from the subject is considered to be separated into a specular reflection component and a perfect diffuse reflection component. In the present invention,
An illumination color is estimated from an image composed of a multidimensional color vector (that is, a luminance sequence obtained for discrete λ) that is a luminance sequence of different frequency components. According to Equation 1, the brightness of the insulator illuminated by a single illuminating light is observed as a linear combination of the specular reflection component and the perfect diffuse reflection component, and the brightness should be distributed on the two-dimensional plane in the multidimensional color vector space. become. If there are two independent normal vectors, this 2
The spectral distribution of illumination light can be obtained as a color vector orthogonal to one normal vector (FIG. 5 shows the case where a multidimensional color vector is an RGB signal). This requires dividing the image into different insulator regions. Therefore, in the present invention, the image is divided into regions by the region dividing means based on color similarity. It is considered that the expression 1 holds in many of the small areas obtained by the area division based on the color similarity. Next, the color plane approximating means approximates the color vector distribution in each area divided by the area dividing means to a plane to obtain a plane normal vector. There is no guarantee that all normal vectors on this plane are obtained accurately. Therefore, the color vector estimating unit selects a color vector whose inner product becomes small with respect to a normal vector group that is a subset of the normal vector set of each region obtained by the color plane approximating unit. , Estimate the illumination color vector.
Therefore, the illumination color can be effectively estimated from the color distribution of the captured image.

[0008]

Embodiments of the present invention will be described below with reference to FIGS.
This will be described with reference to FIGS. 4, 5, and 6. FIG. 1 is a functional block diagram of the illumination color estimation device in the embodiment. FIG. 2 is a functional block diagram of the area dividing unit in the embodiment. Figure 3
FIG. 3 is a functional block diagram of a color plane approximating unit in the embodiment. FIG. 4 is a functional block diagram of the illumination color vector maximum likelihood estimator in the embodiment. FIG. 5 is a conceptual diagram of a standard dichroic reflection model. FIG. 6 is an explanatory diagram of the assumption of the feature vector distribution in the embodiment. FIG. 7 is an operation explanatory diagram of the area dividing unit in the embodiment. In this embodiment, the image is assumed to be composed of color vectors of RGB signals. This is R
It is expressed as x, which is a GB column vector. Further, the pixel position is expressed as p, which is a column vector of the horizontal position ξ and the vertical position η. In FIG. 1, 101 is a region dividing unit, 102 is a color plane approximating unit, 103 is a normal vector selecting unit, and 104 is an illumination color vector maximum likelihood estimating unit. First, the area division part (1
The operation (01) will be described with reference to FIGS. 2, 6, and 7. The area dividing unit (101) divides the image into small areas having similar colors as shown in FIG. 7 by extracting a feature vector (x, p) from the image and clustering the pixels. The result of clustering is not always guaranteed to be an ellipse, and a connected region is not created, but since the position is a feature vector in addition to the color, pixels with similar colors and close distances have only one pixel. Form clusters (small areas). As shown in FIG. 6, this region division has an assumption that the feature vector in the cluster has the color vector and the position vector independently distributed with the average and covariance as parameters. Each cluster is assigned a distribution parameter consisting of a mean x-and a covariance X of the color vector x and a mean p-and a covariance P of the position vector p in the small area. I'll call it. Throughout the image, the reference vectors form a set. The color mean and covariance will then be used later for illumination color vector estimation.

FIG. 2 is a functional block diagram of the area dividing unit (101). 201 is a frame memory for storing an input image, 202 is a random address generating unit for randomly generating a position vector p of the image, and 203 is a random address. A feature vector synthesis unit that reads a color vector x of a pixel located at the position based on the position vector p obtained by the generation unit, synthesizes the feature vector (x, p), and outputs the synthesized feature vector (x, p).
Reference numeral 4 is a reference vector determining unit, 205 is a reference vector changing unit, and 206 is a reference vector memory bank. The reference vector memory bank (206) has 1 to N slots for storing N reference vectors. The reference vector set is represented by θ _i = (x− _i , X _i , p− _i , P _i ), 1 ≦ i ≦
I will call it N. i corresponds to the slot number. Here, in each slot of the reference vector memory,
A valid flag that takes a binary value of 1, 0 is attached. When 1 is valid, when 0 is invalid. The clustering procedure is shown below.

Step 1: First, the valid flags of m slots (m ≦ N) of N slots are set to 1 and the others are set to 0. Let θ _i (t) be the set of reference vectors at discrete time t
_{= (X¯ i (t),} X i (t), p¯ i (t), P
_i (t)), 1 ≦ i ≦ N. Initialize the memory contents of the reference vector slot which is 1 of the valid flag. Initialization is performed by covariance matrix X _i (0), P _i (0)
Is the identity matrix, and the mean x ￣ of the color vector x
_i (0) and the average P _i of the position vector p (0) is set at random. The discrete time t at the time of initialization is 0.

Step 2: The feature vector synthesizing unit reads the color vector x of the pixel at that position by the position vector p obtained by the random address generating unit (202), synthesizes the feature vector, and outputs it. The reference vector determination unit (203) calculates the distance from the feature vector for the reference vector in the slot whose valid flag is 1 among the N slots. This distance is evaluated by Equation 2 (t means transposition of matrix and vector).

[0012]

[Equation 2]

Here, | X | and | P | are determinants of each covariance matrix. Number 2, when the distribution of pixel is assumed to color vector x position vector p are each an average independently x¯ _i, p¯ _i and covariance X _i, to be normally distributed with P _i, the probability density function It is a statistical measure that can be obtained from the logarithm of (log likelihood). Of the N slots, the distance d _i is calculated for the reference vector whose valid flag is 1, and the reference vector determination unit refers to the number k and the sample of the feature vector, where k is the number of the slot having the shortest distance. Send to the vector change unit (205). Reference vector change unit (205)
Then, the reference vector of the slot number k is changed to the sample under the distance function of the equation 2 so as to be closer to the sample by the equations 3, 4, 5, and 6. Equation 7 is an experimentally determined gain coefficient.

[0014]

[Equation 3]

[0015]

[Equation 4]

[0016]

[Equation 5]

[0017]

[Equation 6]

[0018]

[Equation 7]

Then, the time t is incremented by 1 and the above step 2 is repeated a predetermined number of times (10,000 times, for example), whereby clustering is performed by self-organization of pixels.

That is, one cluster is formed by modifying the distribution parameter data so that the reference vector closest to the randomly selected feature vector is closer to the sample. Random address generation is
This is to prevent bias in parameter convergence caused by changing the successive reference vector. In the repetition of step 2, the reference vector changing unit (205) satisfies the condition 1: (total number of reference vector slots N) − (currently valid reference vector slot number m)> 0, and condition 2: ln | X _i | +
If ln | P _i | is greater than or equal to a predetermined threshold value, the cluster represented by the reference vector θ _i is divided. Therefore, the reference flag θ _i has the valid flag of 0 (invalid).
Duplicate into reference vector slot j to make θ _j = θ _i . Then, the average vector is adjusted by ε, respectively. The value of ε is experimentally determined. This is shown in Equations 8, 9, 10, and 11.

[0021]

[Equation 8]

[0022]

[Equation 9]

[0023]

[Equation 10]

[0024]

[Equation 11]

Then, the valid flag of the slot j is set to 1 and the number of valid reference vector slots m is increased by 1, and used for the repetition of self-organization thereafter. As a result, ln | X
_A cluster having a large _i | + ln | P _i | is divided.

In addition, during the repetition of step 2, the valid flag of the slot of the reference vector for which the allocation of the randomly selected sample vector has never been obtained, that is, the shortest distance has not been set, is set to 0 for a predetermined period. , Reduce the number of valid reference vector slots m by 1. This is necessary to cancel the parameters of the reference vector that have converged to the wrong value.

By the above processing, the input image is divided into small regions having similar colors as shown in FIG. 7, and a reference vector set consisting of N reference vectors including invalid reference vectors for each region. Is generated.

Next, the operation of the color plane approximation unit will be described with reference to FIG. In FIG. 3, reference numeral 301 denotes a correlation matrix generation unit that reads the average and covariance of color vectors from the reference vector storage memory (206) to generate a correlation matrix, and 302 a spectral decomposition unit that decomposes the generated correlation matrix into a spectrum.
303 is the eigenvalue of the spectrally decomposed correlation matrix and the third
An eigenvector, a color plane normal vector to be stored, and an eigenvalue storage memory. The operation of the color plane approximating unit (102) in the present embodiment configured as described above will be described below.
The correlation matrix generation unit (301) stores the reference vector set from the reference vector storage memory (206) as θ _i = (x− _i ,
X _i , p − _i , P _i ), x − _i , X _i whose valid flag is 1 is read out from 1 ≦ i ≦ N. Region division was performed using a model that has a three-dimensional normal distribution centered on the average vector x shown in FIG. 6, but in order to use the standard dichroic reflection model in which color vectors are distributed on a plane that passes through the origin, The correlation matrix R _i is generated from Equation 12.

[0029]

[Equation 12]

It can be considered that the first principal component and the second principal component of the correlation matrix R _{i form} a plane and the third principal component represents a normal vector of the plane. Therefore, the spectrum decomposition unit (30
2) decomposes the correlation matrix R _i . This is expressed by Equation 13.

[0031]

[Equation 13]

In Equation 13, λ1 (i), λ2 (i), λ3
(I) is an eigenvalue of the correlation matrix R _i , and a _i , b _i , and c _i are eigenvectors normalized to the size 1. Here, the third eigenvector is called a color plane normal vector. Then, the eigenvalue and the color plane normal vector c _i are stored in the storage memory (303). Next, the operation of the normal vector selection unit (103) will be described. Some of the reference vectors obtained by the area dividing unit (101) may not be data of the insulator area. In addition, there are areas where there is an error in area division, and there are areas that do not include specular reflection components and perfect diffuse reflection components that are sufficient for normal vector estimation. Therefore, not all the above-mentioned third eigenvectors are correct vectors according to the standard dichroic reflection model. Therefore, the normal vector selection unit (103) uses the threshold values T1 and T2 (0 <T1, T2 <1) to obtain the color plane normal vector c corresponding to the following Expressions 14 and 15.
Illumination color vector maximum likelihood estimator by selecting only _i (104)
Output to.

[0033]

[Equation 14]

[0034]

[Equation 15]

The color plane normal vector is a vector that should be orthogonal to the color vector of the illumination light. Expression 14 is a threshold value process for using only the normal vector c _i, although the absolute value of the inner product with the unit vector having the same RGB components (that is, the color vector that is white) is small. The value of Expression 15 represents the certainty of the color plane normal vector obtained by spectral decomposition. When the value of Expression 15 is large, the estimated value of the color plane normal vector is assumed to be certain, and only the normal vector c _i that exceeds the threshold value T2 is used. Next, the operation of the illumination color vector maximum likelihood estimator (104) will be described with reference to FIG. In FIG. 4, 401 is a color plane normal vector storage memory, and 402 is a singular value decomposition unit. The color plane normal vector selected by the normal vector selection unit (103) is stored in the color plane normal vector storage memory. here,
The number is set to m. Vectors orthogonal to the m vectors represent the spectral distribution of illumination. In reality,
Since there is no vector orthogonal to all color plane normal vectors due to error and area division error, a vector that reduces the sum of inner products is found. This can be expressed by Equation 16. In Expression 16, 1 is a color vector representing the spectral distribution of illumination. Equation 16 represents the square norm. This minimization can be easily obtained by arranging m color plane normal vectors (column vectors) and performing singular value decomposition on the matrix.

[0036]

[Equation 16]

In the present embodiment, as described above, the image is divided into areas by the clustering featuring the color and the position, and the standard dichroic reflection model is established for the divided small areas, so that the reliability is high. Select the color plane normal vector in the small area, and in the sense that the sum of squares of inner products is minimized,
The color distribution of the illumination light is estimated as a color vector orthogonal to the normal vector. The feature of the present embodiment is that the color vector covariance can be easily calculated at the time of region division to calculate the color vector correlation matrix and the color plane normal vector. In this embodiment, in order to increase the accuracy of area division, as shown in FIG. 6, clustering is performed by the sum of the normalized distance to the color vector average in the area and the normalized distance with respect to the position. Instead of the normalized distance to the color vector average, the projection component up to the color plane illustrated in FIG. 5 may be used. In this case, the correlation matrix is used as it is, instead of Equation 2, Equation 1
7 will be used instead of equation (4) (λ3
(I) and c _i are the third eigenvalue and the third eigenvector of the correlation matrix), and the procedure of obtaining the correlation matrix R from the covariance matrix X and the procedure of obtaining the color vector average are advantageous.

[0038]

[Equation 17]

[0039]

[Equation 18]

The present invention is not limited to the above embodiments, and various modifications can be made based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention.

[0041]

As described in detail above, according to the present invention, the following effects can be obtained. 1) The illumination light can be estimated directly from the subject in the image. 2) It is possible to estimate the spectral distribution of illumination light without a white subject. Therefore, it can be used in a more complicated environment as compared with the conventional illumination color estimation device, and can be used for color correction applications such as white adjustment of a color video camera and color image printing.

[Brief description of drawings]

FIG. 1 is a functional block diagram of an illumination color estimation device according to an embodiment.

FIG. 2 is a functional block diagram of a region dividing unit in the embodiment.

FIG. 3 is a functional block diagram of a color plane approximation unit in the embodiment.

FIG. 4 is a functional block diagram of an illumination color vector maximum likelihood estimator in the embodiment.

FIG. 5 is a conceptual diagram of a standard dichroic reflection model.

FIG. 6 is an explanatory diagram of the assumption of feature vector distribution in the embodiment.

FIG. 7 is an explanatory diagram of the operation of the area dividing unit in the embodiment.

[Explanation of symbols]

 101 area dividing unit 102 color plane approximating unit 103 normal vector selecting unit 104 illumination color vector maximum likelihood estimating unit 201 frame memory 202 random address generating unit 203 feature vector synthesizing unit 204 reference vector determining unit 205 reference vector changing unit 206 reference vector storage Memory 301 Correlation matrix generation unit 302 Spectral decomposition unit 303 Color plane normal vector and eigenvalue storage memory 401 Color plane normal vector storage memory 402 Singular value decomposition unit

Claims (1)

[Claims] 1. An area dividing means for dividing an image composed of a multi-dimensional color vector, which is an array of luminances of different frequency components, based on color similarity, and within each area divided by the area dividing means. A color plane approximation means for approximating the color vector distribution of the plane to obtain a plane normal vector, and a normal vector group which is a subset of the set of normal vectors of the respective areas obtained by the color plane approximating means. On the other hand, an illumination color estimation device having a color vector estimation means for selecting a color vector having a smaller inner product, and using the color vector as a frequency component of illumination light.