TW201830335A - Method and device for determining an illuminant of an image and for chromatic adaptation of an image - Google Patents

Abstract

A method and a device determining an illuminant of an image and for chromatic adaptation of an image. The method for determining an illuminant of an image, comprising: calculating an indicator value for each of a set of candidate illuminants of the image, wherein each candidate illuminant is described by a corresponding coordinate pair (p, q) in a chromaticity coordinate system; determining a threshold for indicator values of the candidate illuminants; identifying a subset of the candidate illuminants that have the indicator values not greater than the threshold; and for all candidate illuminants in the subset, calculating a weighted average of corresponding coordinate pairs to obtain an averaged coordinate pair to describe the illuminant of the image in the chromaticity coordinate system.

Description

   Method and device for determining light source of image and color vision adaptation of image   

The disclosed embodiments relate to the fields of color photography, digital cameras, color printing, and digital color image processing.

All users' color display devices are calibrated so that when the color channel's red (R) = green (G) = blue (B), the displayed color is the standard "white point" chromaticity, according to the International Commission on Illumination (International Commission on Illumination, CIE for short) standards, most of which are D65 or D50. Digital color cameras using complementary metal-oxide semiconductor (CMOS) or charge-coupled device (CCD) have different sensitivities to RGB channels, resulting in a certain color cast of the original image (E.g. greenish). In addition, the color of an object changes due to the color of the light source (such as a tungsten lamp or daylight) and the reflection of surrounding objects from each other. Therefore, it is often necessary to adjust the "white point" of the original image before processing and rendering the display image in appropriate colors. The adjustment of the white point is called white balance (WB), which is usually achieved by applying appropriate gains to the color channels so that neutral objects (such as black, gray, and white) in the image are rendered approximately equal R, G, B values. In digital cameras, the white point can be adjusted manually or automatically. Automatic white balance (AWB) is therefore an important operation in color imaging applications.

Most traditional AWB algorithms rely on some physical characteristics (such as color gamut) and statistical characteristics (such as average color distribution) of natural scenes. The traditional AWB algorithm is sensitive to the statistics of the scene content, and often encounters one or more of the following difficulties: 1) the main color color deviation affects the result, 2) when there is no neutral color in the image, the probability of error is high 3) Incorrect camera calibration may cause scene statistics to be different from those used by the camera, 4) A large number of reference standards are required as training samples to establish reliable statistics, and 5) the performance of the algorithm is affected by the components in the camera's mass production The effect of differences with components. Therefore, there is a great need to develop an AWB technology that is more robust and relatively insensitive to multi-scenario content.

According to an exemplary embodiment of the present invention, a feedthrough signal transmission device / method and a related feedthrough signal transmission circuit are provided to solve the above problems.

The invention provides a method and a device for determining a light source of an image and performing color vision adaptation on the image.

In one embodiment, a method of determining a light source of an image is provided. The method includes: calculating an indication value of each candidate light source in a group of candidate light sources of an image, wherein each candidate light source is described by a corresponding coordinate pair ( p, q ) in a chromaticity coordinate system; determining a candidate A threshold value of the indication value of the light source; identifying a subset of candidate light sources having an indication value that is not greater than the threshold value; and for all candidate light sources in the subset, calculating a weighted average of the corresponding coordinate pair to obtain a color coordinate for the The system describes the average coordinate pair of the light source of the image.

In another embodiment, a method for color vision adaptation of an image is provided. The method includes: calculating a light source of an image, wherein the light source is described by a coordinate pair ( p, q ) in a chromaticity coordinate system; identifying a brightness level of a scene in the image; using at least part of the adaptation requirements and the image One or more adaptation degrees derived from the brightness level of the middle scene to adjust the coordinate pair ( p, q ) of the light source to obtain an adapted light source; and adapt the color of the image to the adapted light source.

In yet another embodiment, an apparatus for determining a light source of an image is provided. The device includes: a memory for storing an image; and an image processing pipeline coupled to the memory. The image processing pipeline is used to calculate an indication value of each candidate light source in a group of candidate light sources of an image, where each candidate light source is described by a corresponding coordinate pair ( p, q ) in a chromaticity coordinate system; Determine the threshold value of the indication value of the candidate light source; identify a subset of candidate light sources having an indication value not greater than the threshold value; and calculate a weighted average of the corresponding coordinate pairs for all candidate light sources in the subset to obtain The average coordinate pair of the light source describing the image.

In yet another embodiment, a device for color vision adaptation of an image is provided. The device includes: a memory for storing an image; and an image processing pipeline coupled to the memory. The image processing pipeline is used to calculate the light source of the image, where the light source is described by the coordinate pair ( p, q ) in the chromaticity coordinate system; identify the brightness level of the scene in the image; use at least partially according to the adaptation requirements One or more fitting degrees derived from the brightness level of the scene in the image to adjust the coordinate pair ( p, q ) of the light source to obtain the adapted light source; and adapt the color of the image to the adapted light source .

Embodiments of the present invention improve the results of AWB calculations. In addition, color vision adaptation can be efficiently performed according to a given specification. The advantages of the embodiments will be explained in detail in the following description.

100‧‧‧Image Processing Pipeline

110‧‧‧AWB Module

120‧‧‧Color Correction Matrix Module

130‧‧‧Gray Correction Module

140‧‧‧ Display

150‧‧‧ Equipment

160‧‧‧Memory

300, 500, 700‧‧‧AWB modules

310, 510‧‧‧ pretreatment unit

320‧‧‧ Projection Plane Calculator

330‧‧‧ Projected Area Calculator

340, 730‧‧‧ Comparator

345‧‧‧offset

350‧‧‧Gain adjustment unit

380‧‧‧MPA calculator

515‧‧‧block division unit

540‧‧‧weighted average unit

600‧‧‧MPA method

610 ~ 640‧‧‧step

720‧‧‧ Difference calculator

780‧‧‧MTV calculator

800‧‧‧MPA method

810 ~ 880‧‧‧step

900‧‧‧ Method

910 ~ 940‧‧‧ steps

1181‧‧‧MPA calculator

1101‧‧‧Average Calculator

1182‧‧‧MTV Calculator

1200‧‧‧Method

1210 ~ 1240‧‧‧step

1310‧‧‧ Adaptation requirements

1320‧‧‧A _p curve

1410, 1420‧‧‧AWB Module

1450‧‧‧Color Vision Adaptation Module

1500‧‧‧Method

1510 ~ 1540‧‧‧‧step

FIG. 1A is an example of an image processing pipeline performing color correction in an embodiment provided by the present invention; FIG. 1B is a form of a device including the image processing pipeline shown in FIG. 1A in an embodiment of the present invention System; FIG. 2 is an example of projection of two color surfaces on a plane perpendicular to a light source vector in an embodiment of the present invention; and FIG. 3 is a diagram for performing a minimum projection area in an embodiment provided by the present invention (minimum projected area, MPA) method structure diagram of the automatic white balance module; Figures 4A, 4B and 4C are projection results using three different candidate light sources in an embodiment of the present invention; Figure 5 is an implementation of the present invention The schematic diagram of the structure of the automatic white balance module that executes the block MPA method in the example; FIG. 6 is a flowchart of the MPA method in an embodiment of the present invention; and FIG. 7 is the implementation of the minimum total change in an embodiment of the present invention The structure diagram of the automatic white balance module of the minimum total variation (MTV) method; FIG. 8 is a flowchart of the MTV method in another embodiment of the present invention; and FIG. 9 is a flowchart for the MTV method in an embodiment of the present invention. Square of Auto White Balance Figure 10 is an example of the G / R ratio in a certain range in the embodiment of the present invention, where the indicated value is less than the threshold; Figure 11A is an example of the MPA calculator in the embodiment of the present invention; FIG. 11B is an example of an MTV calculator in an embodiment of the present invention; FIG. 12 is a flowchart of a method for determining an image light source in an embodiment of the present invention; and FIG. 13 is a flowchart of an embodiment of the present invention. Schematic diagram of the adaptation requirements; Figure 14A is an example of an AWB module in one embodiment of the invention; Figure 14B is an example of an AWB module in another embodiment of the invention; Figure 15 is an implementation of the invention A flowchart of a method for performing color vision adaptation on an image based on a set of brightness level adaptation requirements in the example.

In the following description, many specific details are stated. It is understood, however, that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known circuits, structures, and techniques have not been described in detail, so as not to obscure the understanding of this specification. However, those skilled in the art will understand that the present invention can be practiced without these specific circumstances. Those skilled in the art will be able to implement appropriate functions without undue experimentation through the descriptions they have.

The present invention provides a system and method based on surface reflection decomposition to perform automatic white balance (AWB). When compared to systems and methods based on traditional AWB algorithms, the systems and methods are robust and relatively insensitive to scene content. The system and method do not rely on detailed scene statistics or large image databases for training. In the following, the minimum projected area (MPA) method and the minimum total variation (MTV) method are described, both of which are based on decomposing the surface reflection into specular and diffuse reflection components, and based on Elimination of specular reflection components.

As used herein, the term "tri-color value" or equivalent "RGB value" or "RGB channel" refers to the three color values (red, green, blue) of a color image. The terms "illuminant" and "light source" are used interchangeably. In addition, a chroma image refers to a color difference image, which can be calculated by obtaining a difference between one color channel and another color channel, or can be calculated by a difference between linear combinations of color channels.

FIG. 1A is an example of an image processing pipeline 100 that performs color correction in an embodiment of the present invention. The image processing pipeline 100 includes an AWB module 110 that receives an original RGB value as an input, and outputs an RGB value after white balance correction. Raw RGB values can be generated by image sensors, cameras, video recorders, etc. The operation of the AWB module 110 will be explained in detail below with reference to FIGS. 2-9. The image processing pipeline 100 further includes a color correction matrix (CCM) module 120. The color correction matrix 120 performs a 3 × 3 matrix calculation on the RGB values output from the AWB module 110. The CCM module 120 can reduce the difference between the spectral characteristics of the image sensor and the spectral response of a standard color device (eg, a standard RGB color display). The image processing pipeline 100 may further include an image grayscale correction module 130 that applies a non-linear function to the RGB values output from the CCM module 120 to compensate for the non-linear brightness effect of the display device. The output of the image processing pipeline 100 is a set of standard RGB (sRGB) values ready for display. In one embodiment, the image processing pipeline 100 includes a plurality of processing elements (for example, Arithmetic and Logic Units (ALU), general-purpose processors, special-purpose circuits, or any combination thereof) for implementing the AWB module. Functions of the group 110, the CCM module 120, and the image grayscale correction module 130.

FIG. 1B is a schematic diagram of a system structure of a device 150 including the image processing pipeline 100 shown in FIG. 1A according to an embodiment of the present invention. In addition to the image processing pipeline 100, the device 150 includes a memory 160 for storing images. Like data or intermediate image data to be processed by the image processing pipeline 100, the device 150 further includes a display 140 for displaying an image having a standard RGB value. It can be understood that the device 150 may further include other additional components, including but not limited to: an image sensor, one or more processors, a user interface, a network interface, and the like. In one embodiment, the device 150 may be a digital camera, or the device 150 may be part of a computing and / or communication device such as a computer, laptop, smart phone, smart watch, or the like.

Before describing the embodiment of the AWB module 110, it is helpful to first explain the principle on which the AWB module 110 operates.

Let f ( θ ; λ ) be a bidirectional spectral reflectance distribution function (BSRDF), where θ represents all angle-dependent factors, and λ represents the wavelength of light. The BSRDF of the surface of most colored objects can be described as a combination of two reflection components, which are the interface reflection (specular) component and the body reflection (diffuse reflection) component. Interface reflection is usually non-selective, that is, it reflects all light with the same wavelength as visible light. This model is called a neutral interface reflection (NIR) model. Based on the NIR model, BSRDF f ( θ ; λ ) can be expressed as: f ( θ ; λ ) = ρ ( λ ) h ( θ ) + ρ _s k ( θ ), (1) where ρ ( λ ) is the diffuse reflectance factor (diffuse reflectance factor), ρ _s specular reflectivity (specular reflectance factor), h ( θ) and k (θ) is the angle two reflectivity correlation function. The key feature of the NIR model is that the spectral and geometric factors in each reflection component are completely separated.

Assumed that L (λ) is the spectral power distribution of light _{source, S r (λ), S} g (λ) and S _b (λ) is the basis of the three sensors (i.e., the spectral response function). The RGB color space can be derived as: R = ʃ L ( λ ) f ( θ ; λ ) S _r ( λ ) dλ = h ( θ ) ʃ L ( λ ) ρ ( λ ) S _r ( λ ) dλ + ρ _s k ( θ ) ʃ L ( λ ) S _r ( λ ) dλ , G = h ( θ ) ʃ L ( λ ) ρ ( λ ) S _g ( λ ) dλ + ρ _s k ( θ ) ʃ L ( λ ) S _g ( λ ) dλ , B = h ( θ ) ʃ L ( λ ) ρ ( λ ) S _b ( λ ) dλ + ρ _s k ( θ ) ʃ L ( λ ) S _b ( λ ) dλ . (2)

Let L _r = ʃ L ( λ ) S _r ( λ ) dλ , L _g = ʃ L ( λ ) S _g ( λ ) dλ , L _b = ʃ L ( λ ) S _b ( λ ) dλ , , , .

Then, R = L _r [ ρ _r h ( θ ) + ρ _s k ( θ )], G = L _g [ ρ _g h ( θ ) + ρ _s k ( θ )], and B = L _b [ ρ _b h _{(θ) + ρ s k (} θ)], (3) where, L _r, L _g, and L _b are tristimulus values of light, RGB color space can be rewritten in matrix form as shown below:

Let v ₁ and v ₂ be two independent vectors in RGB space. If the RGB values are projected onto the plane V generated by v ₁ and v ₂ , the projected coordinates will be

Let L = [ L _r L _g L _b ] ^T be the light source vector. When [ v ₁ v ₂ ] ^T L = 0, the second term in equation (5) disappears. This means that when the plane V is perpendicular to the light source vector L , the specular reflection component is eliminated.

FIG. 2 is an example of projecting the colors of two surfaces onto a plane V in an embodiment of the present invention. The NIR model, at a given surface (e.g., ₁ S) on each of the color vector is the specular component (represented by the light source vector L) and diffuse reflection components (indicated by C ₁₎ is a linear combination. All colors of S ₁ are on the same plane as L and C ₁ . Similarly, all colors of the other surface (eg, S ₂ ) are on the same plane as L and C ₂ . Therefore, the common vector L is shared by the planes of all colors under the same light source. If all colors are projected along the light source vector L , their projections will form multiple lines, and these multiple lines intersect at a point, which is the projection point of the light source vector. If the direction of the projection is not along the direction of the light source vector L (ie, if V is not perpendicular to L ), the specular component is not eliminated. In this case, the color of the projected line is formed not on the plane V, but will expand on the two-dimensional area of the V plane. This two-dimensional area is called the projected area on the plane V. When v ₁ and v _{2 are} orthogonal, the area of the two-dimensional region can be calculated. When v ₁ and v ₂ change, the plane V changes accordingly. By changing v ₁ and v ₂ , when the plane V is perpendicular to the light source vector L , the projection area becomes the smallest. It doesn't matter which particular of v ₁ and v ₂ is used as the basis vector, because they both produce essentially the same result.

In the AWB calculation, the light source vector L of the real light source is unknown. The MPA method changes the plane V by selecting different candidate light sources. From the selected candidate light source vector _{L = (L r, L g} , L b), can be calculated orthogonal basis vectors v ₁ and v _2, and also given image can be calculated by v ₁ and v ₂ Projected area on the generated plane. When the selected light source vector L is closest to the real light source of the image, the projection area is the smallest.

In one embodiment, the orthogonal basis vectors can be parameterized as follows:

When α = L _g / L _r and β = L _g / L _b , the plane V ( α , β ) is perpendicular to L.

In one embodiment, because searching all possible planes V ( α, β ) is very time-consuming, the search range of the light source is narrowed to a subspace where the light source is more likely to occur. Narrowing the search also helps reduce the possibility of finding the wrong light source. In one embodiment, the search range may be set to a set of light sources that often occur in consumer images in the intended field of application. The term "user image" refers to a color image that can generally be seen on an image display device used by content users. Alternatively or in addition, a suitable mixture of daylight locus and blackbody radiator locus can be used. This blend can provide a light source trace that covers most of the light sources in the user's image. In order to search for the light source of the image, the MPA method is used to calculate the projected area of the image for each candidate light source in the set of candidate light sources along the light source trajectory. The candidate light source that produces the smallest projection area is the best estimate of the scene light source (ie, the real light source), and the image is white-balanced according to the best estimate of the scene light source. In one embodiment, the expression for the MPA method to find the minimum projection area is as follows: argmin _{α, β} w ( α, β ) Area ( α, β ), (8) where w ( α, β ) is an offset A function (bias function), Area ( α, β ) is the area projected on the plane V ( α, β ), and this area is generated by v ₁ ( α, β ) and v ₂ ( α, β ). This bias function can be used to modify the projection area, thereby improving the performance of the MPA method. The bias function depends on the overall scene light distribution, not the scene content. Therefore, the same offset function can be applied to any camera model after camera calibration. Details of the bias function w ( α, β ) will be provided later. In other embodiments, the bias function may be omitted (ie, set to 1).

FIG. 3 is a structural diagram of an AWB module 300 for performing an MPA method according to an embodiment of the present invention. The AWB module 300 is an example of the AWB module 110 shown in FIG. 1A. The AWB module 300 includes a pre-processing unit 310 for processing raw RGB data of an input image to remove over-exposed, under-exposed, and saturated pixels. Removing these pixels can speed up AWB calculations and reduce noise. In one embodiment, if the color channel of one or more of the R value, G value, and B value of a pixel is greater than the threshold, the pixel is considered to be overexposed, and the pixel is removed. After these pixels are removed, the pre-processing unit 310 may perform an operation on the input image by dividing the image into multiple groups of adjacent pixels and calculating a weighted average of the tri-color values of the adjacent pixels in each group. Group average. The weight of each group can be one or other values. In one embodiment, after calculating the group average, the pre-processing unit 310 may remove underexposed pixels from the image. If the sum of the R, G, and B values of a pixel is higher than the first threshold, the pixel is overexposed; if the sum of the R, G, and B values of a pixel is lower than the second threshold, the pixel Underexposure. The pre-processing unit 310 may also remove saturated pixels from the image. If one of the R value, G value, and B value of a pixel is lower than a predetermined threshold, the pixel is saturated.

In one embodiment, after removing over-exposed, under-exposed, and / or saturated pixel and group averaging operations, the pre-processing unit 310 may sub-sample the image to generate a pre-processed image. The pre-processed image is provided to the MPA calculator 380 in the AWB module 300 for MPA calculation.

In one embodiment, the MPA calculator 380 includes a projection plane calculator 320 and a projection area calculator 330. The projection plane calculator 320 calculates two orthogonal vectors v ₁ and v ₂ , and v ₁ and v ₂ generate a plane perpendicular to the light source vector L ( L _r , L _g , L _b ) of the candidate light source. In one embodiment, when the values of α and β have been given or calculated from candidate light sources, the projection plane calculator 320 may calculate v ₁ and v ₂ according to equations (6) and (7).

After determining the projection plane, the projection area calculator 330 projects the RGB value of each pixel in the preprocessed image to the projection plane. The result of the projection is a collection of points that fall on the projection plane. If each color is represented as a single point, the result of the projection will produce a scattered set of points on the projection plane, as shown in Figures 4A, 4B, and 4C, and each of Figures 4A, 4B, and 4C The images are the result of projection using different candidate light sources. When the projection is performed along the real light source vector, the density of local points is changed to be high. However, calculating the density of points requires a lot of calculations. In one embodiment, the projection plane is divided into a set of spatial bins (eg, squares). When one or more pixels are projected into a square, the square is counted. The number of counted squares can be used as an estimate of the projected area.

Referring to Figures 4A, 4B, and 4C, in each example, the "x" mark represents the projection points of all pixels in the image. The closer the candidate light source is to the real light source, the smaller the total area of the projection marked by "x". Each example uses different candidate light sources described by different orthogonal basis vectors v ₁ and v ₂ . The projection area 119 shown in FIG. 4B has the smallest area, so its corresponding candidate light source is closest to the real light source among the three candidate light sources.

Please refer to FIG. 3 again. After the projection area calculator 330 calculates the projection areas of a set of different candidate light sources, the comparator 340 compares the sizes of these projection areas and identifies the candidate light source that produces the smallest projection area. In one embodiment, before the comparison, the comparator 340 may multiply each projected area with the offset function described above as an option to improve the AWB result, which is shown as the offset value 345 in this embodiment ( That is, weights). The offset value 345 may be determined based on prior knowledge of the frequency of the light source along the light source trajectory in the user image. That is, the offset value 345 represents the prior knowledge of the scene light source distribution, and is not related to the scene content. In one embodiment, each candidate light source is associated with an offset value, which can be expressed as a function w ( α, β ), where α and β are the color ratios of the candidate light sources. From one camera model to another, the offset value is stable.

After the comparator 340 recognizes the candidate light source that produces the smallest projection area, the gain adjustment unit 350 adjusts the color gain of the input image according to the color ratios α and β of the candidate light source.

For images with multiple different colors, when projecting along the light source vector, the projection area is usually the smallest. However, for images with only a single main color, when the specular or diffuse reflection components of the main color are eliminated Only when the minimum projection area appears. In order to better process this kind of image with few color types, the search range is limited to the smallest projection area caused only by the specular reflection component being eliminated instead of the diffuse reflection component being eliminated. One method is to search for these candidate light sources in the chromaticity space near the location of potential light sources. Therefore, a minimum projection area is searched along a light source trajectory that passes through a population of known light sources.

In one embodiment, the chromaticity coordinate system ( p, q ) can parameterize the distribution of the light source trajectory in a less distorted chromaticity region. The coordinate system ( p, q ) is defined as: Where r = R / ( R + G + B ), g = G / ( R + G + B ), and b = B / ( R + G + B ).

For the candidate light source ( L _r , L _g , L _b ), its ( p, q ) coordinate system can be changed by replacing the R, G, and B values in equation (9) with L _r , L _g , and L _b values. Decide.

The light source trajectory can be obtained by fitting color data obtained by the reference camera under different light sources. For example, curve fitting of light from three light sources (shadow, daylight, and tungsten lamp) can provide very good light source trajectories. In one embodiment, a given light source trajectory can be represented by a second-order polynomial function in the ( p, q ) domain, which has the form: q = a _i p ² + a ₂ p + a _3. (10)

Given the value of ( p, q ), the equation shown below is used to calculate ( r, g, b ):

The color ratios α and β can be obtained by the equation shown below:

Therefore, given ( p, q ) along the trajectory of the light source, the color ratios α and β can be calculated. The orthogonal vectors v ₁ ( α, β ) and v ₂ ( α, β ) can be calculated using equations (6) and (7). And the area of the image projected onto the plane V developed by v ₁ ( α, β ) and v ₂ ( α, β ) can also be calculated.

When the scene is illuminated by a single main light source, the MPA method can accurately estimate the light source. However, some scenes have more than one light source. In one embodiment, the block MPA method is used to process such a multi-light source scene. Using the block MPA method is to divide the image into several blocks and apply the MPA method to each block.

FIG. 5 is an AWB module 500 that executes the block MPA method according to an embodiment of the present invention. The AWB module 500 is an example of the AWB module 110 shown in FIG. 1A. The AWB module 500 includes a pre-processing unit 510, and the pre-processing unit 510 further includes a block dividing unit 515. The block dividing unit 515 is configured to divide an input image into a plurality of blocks. The pre-processing unit 510 performs the same pixel removal operation on each block as the pre-processing unit 310 shown in FIG. 3. The pre-processing unit 510 removes pixels that are overexposed, underexposed, and saturated. The pre-processing unit 510 also determines whether each block has a sufficient number of pixels (for example, 10 pixels) to perform the MPA method after the pixel removal operation. If the number of blocks with sufficient pixels is less than the threshold number (for example, half of the total number of blocks), the pre-processing unit 510 re-divides the image into a smaller number of blocks such that the number of new blocks in the image is greater than the threshold Quantity.

In one embodiment, the AWB module 500 includes one or more MPA calculators 310. The MPA calculator 310 is used to perform MPA methods on each block. The results of each block are collected by the weighted average unit 540. The weighted average unit 540 first averages the chromaticity coordinates p , and then finds out based on the fitted curve of the given light source trajectory (such as the second-order polynomial function in (10)) Another chromaticity coordinate q . In one embodiment, the weighted average unit 540 applies a weight to each block. For example, a block with a main object may have a higher weight than other blocks. In other embodiments, the weighted average unit 540 may apply the same weight to all the blocks. The output of the weighted average unit 540 is the result candidate light source or its representative candidate light source. The gain adjustment unit 350 then adjusts the color gain of the input image according to the color ratios α and β of the result candidate light sources.

FIG. 6 is a flowchart of an MPA method 600 performed on a color image according to an embodiment of the present invention. MPA method 600 may be performed by a device such as device 150 of FIG. 1B; more specifically, MPA method 600 may be performed by AWB module 110 of FIG. 1A, AWB module 300 of FIG. 3, and / or FIG. The AWB module 500 is executed.

In the MPA method 600, the device first starts preprocessing the image to obtain preprocessed pixels. Each preprocessed pixel is represented by a three-color value. The three-color value includes red (red, R) values, and green (green, G). Value and blue (B, B) value (step 610). For each candidate light source in a group of candidate light sources, the device performs the following operations: calculates a projection plane perpendicular to the vector of the three color values of the candidate light sources (step 620), and projects the three color values of each preprocessed pixel to the calculated To obtain a projection area (step 630). The one with the smallest projection area among the candidate light sources is identified as a result illuminant (step 640). The device can use the color ratio of the resulting light source to adjust the color gain of the image.

According to another embodiment, the AWB method can be performed using the MTV method, which is also based on the same principle as the MPA method: by trying to eliminate the specular reflection component. According to the NIR model, a pair of chroma images ( αC ₁ - C ₂ ) and ( βC ₃ - C ₂ ) can be created from a given image by scaling one color channel and subtracting from another color channel. . ( C ₁ , C ₂ , C ₃ ) is a linear transformation of the three color values (R, G, B).

( αC ₁ - C ₂ ) and ( βC ₃ - C ₂ ) are both functions of the spatial position in the image. The two chroma images can be expressed as: ( αC ₁ - C ₂ ) = [( αa ₁₁ - a ₂₁ ) L _r ρ _r + ( αa ₁₂ - a ₂₂ ) L _g ρ _g + ( αa ₁₃ - a ₂₃ ) L _b ρ _b ] h ( θ ) + [( αa ₁₁ - a ₂₁ ) L _r + ( αa ₁₂ - a ₂₂ ) L _g + ( αa ₁₃ - a ₂₃ ) L _b ] ρ _s k ( θ ), ( βC ₃ - C ₂ ) = [( βa ₃₁ - a ₂₁ ) L _r ρ _r + ( βa ₃₂ - a ₂₂ ) L _g ρ _g + ( βa ₃₃ - a ₂₃ ) L _b ρ _b ] h ( θ ) + (( βa ₃₁ - a ₂₁ ) L _r + ( βa ₃₂ - a ₂₂ ) L _g + ( βa ₃₃ - a ₂₃ ) L _b ] ρ _s k ( θ ). (14)

When α = ( a ₂₁ L _r + a ₂₂ L _g + a ₂₃ L _b ) / ( a ₁₁ L _r + a ₁₂ L _g + a ₁₃ L _b ) and β = ( a ₂₁ L _r + a ₂₂ L _g + a ₂₃ L _b ) / ( a ₃₁ L _r + a ₃₂ L _g + a ₃₃ L _b ): ( αC ₁ - C ₂ ) = [( αa ₁₁ - a ₂₁ ) L _r ρ _r + ( αa ₁₂ - a ₂₂ ) L _g ρ _g + ( αa ₁₃ - a ₂₃ ) L _b ρ _b ] h ( θ ), ( βC ₃ - C ₂ ) = [( βa ₃₁ - a ₂₁ ) L _r ρ _r + ( βa ₃₂ - a ₂₂ ) L _g ρ _g + ( βa ₃₃ - a ₂₃ ) L _b ρ _b ] h ( θ ). (15)

For ( αC ₁ - C ₂ ) and ( βC ₃ - C ₂ ), the specular reflection components are eliminated. When cancellation occurs, the total variation of ( αC ₁ -C ₂ ) and ( βC ₃ -C ₂ ) is greatly reduced because the modulation caused by the specular reflection component disappears. Only the signal modulation of the diffuse reflection component remains.

By searching along a given light source trajectory, the MTV method can find candidate light sources, which are represented by the color ratios α and β , so that the total variation is minimized in the following representations. The color ratios α and β can be calculated using equations (11) and (12) from a given point ( p, q ) on a given light source locus. The total variation in this embodiment can be expressed as the sum of the magnitudes of the absolute gradients of the two chroma images in equation (14):

It should be noted that the gradient of a two-dimensional image is a vector with x and y components. To improve the calculation efficiency, a simplified one-dimensional approximation of total variation can be used:

In one embodiment, if any adjacent pixel has been removed due to overexposure, underexposure, or color saturation, the gradient of that pixel is excluded from the total variation calculation.

FIG. 7 is a schematic structural diagram of an automatic white balance module 700 performing an MTV method according to an embodiment of the present invention. The AWB module 700 is another example of the AWB module 110 in FIG. 1A. The AWB module 700 includes a pre-processing unit 310 for processing raw RGB data of an input image to remove over-exposed, under-exposed, and saturated pixels. The AWB module 700 also includes an MTV calculator 780, which searches for a solution of the smallest total variation in the set of candidate light sources. More specifically, the MTV calculator 780 further includes a difference calculator 720 and a comparator 730. The difference calculator 720 calculates the total variation of each candidate light source, and the comparator 730 compares the results calculated by the difference calculator 720 to identify the smallest total variation. In one embodiment, the comparator 730 may multiply each total variation by an offset value 345 (ie, a weight) before comparison. The offset value 345 may be determined based on prior knowledge of the frequency of the light source along the light source trajectory in the user image. That is, the offset value 345 represents the prior knowledge of the scene light source distribution and has nothing to do with the scene content. In one embodiment, each candidate light source is associated with an offset value, which can be expressed as a function w ( α, β ), where α and β are the color ratios of the candidate light sources, from a camera model to In another camera model, the offset value remains stable.

After the comparator 730 identifies the candidate light source that produces the smallest total variation, the gain adjustment unit 350 adjusts the color gain of the input image using the color ratios α and β of the candidate light source. The experimental results show that the MTV method performs well in the scenario of a single main light source and multiple light sources.

FIG. 8 is a flowchart of an MTV method 800 performed on a color image in another embodiment of the present invention. In this alternative embodiment, a linear transformation is used to calculate the three-color value in the process of calculating the total variation. MTV method 800 may be performed by a device such as device 150 shown in FIG. 1B. More specifically, MTV method 800 may be performed by AWB module 110 shown in FIG. 1A and / or shown in FIG. 7 AWB module 700.

In the MPA method 800, the device first starts preprocessing the image to obtain a plurality of preprocessed pixels, each of which is represented by a three-color value, and the three-color value includes a red (R) value, a green (G) value, and Blue (B) value (step 810). For each candidate light source in the set of candidate light sources, the device calculates the total variation of the tri-color values between adjacent pixels in the pre-processed pixels (step 820). The step of calculating the total variation value includes the following operations: calculating a linear transformation of the three-color values to obtain three transformation values (step 830); calculating a first scaling factor and a second scaling factor, the first scaling factor and the second scaling factor being used Representing two color ratios of candidate light sources (step 840); constructing a first chroma image by obtaining a difference between a first transform value and a second transform value scaled by a first scaling factor (step 850); construct a second chroma image by obtaining a difference between the third transform value and the second transform value scaled by the second scaling factor (step 860); and by converting the absolute value of the first chroma image The gradient magnitude and the absolute gradient magnitude of the second chrominance image are added to calculate a total variation (step 870). After calculating the total variation of all candidate light sources, the device selects the candidate light source corresponding to the total variation with the smallest median value of all total variations (step 880).

FIG. 9 is a schematic flowchart of a method for performing automatic white balance on an image according to an embodiment of the present invention. Method 900 may be performed by a device such as device 150 shown in FIG. 1B; more specifically, method 900 may be performed by AWB module 110 shown in FIG. 1A, AWB module 300 shown in FIG. 3, and The AWB module 500 shown in the figure and / or the AWB module 700 shown in FIG. 7 are executed.

In method 900, the device first begins to pre-process the image to obtain a plurality of pre-processed pixels, each of which is represented by a tri-color value, the tri-color value includes a red (R) value, a green (G) value, and blue (B) value (step 910). For each candidate light source in the set of candidate light sources, the device calculates an indication value including a specular reflection component and a diffuse reflection component (step 920). The device then identifies one candidate light source among the plurality of candidate light sources as the result light source, and its corresponding indication value is the smallest indication value among all candidate light sources, where the smallest indication value corresponds to the elimination of the specular reflection component (step 930). Based on the color ratio derived from the resulting light source, the device modulates the color gain of the image (step 940). In one embodiment, the indication value is the projection area as described in the MPA method 600 shown in FIG. 6. In other embodiments, the indication value is the description of the MTV method 800 as shown in FIG. 8. Total variation.

In the following description, improvements to the MPA and MTV methods will be demonstrated. The purpose of this improvement is to make the light source of the calculated image closer to the real light source, for example, when the projection area (or total variation) generated by multiple candidate light sources has similar sizes; that is, their numerical differences may be Within the error range. Therefore, the coordinate values of the candidate light sources in the PQ domain are averaged to generate an average light source, instead of using one of the candidate light sources as the light source of the image. In one implementation scenario, this method may be useful when a large number of pixels in an image are located on the light source trajectory used to obtain a candidate light source, or within a threshold distance of the light source trajectory. In this case, the minimum projection area calculated by the MPA method and the minimum total variation calculated by the MTV method may be unstable, and the average light source may produce the minimum projection area (or the minimum total variation) than in the calculation. The candidate light source is closer to the real light source. In the description herein, the term "average" may be used interchangeably with "weighted average", where the weight may be 1 or another value; for example, a probability value derived from prior knowledge about candidate light sources. For example, if the first candidate light source appears more frequently than the second candidate light source in the user image group, the probability value of the first candidate light source may be higher than the probability value of the second candidate light source.

To simplify the description, the term "indicative value" is used in the following description to refer to any of the "projected area" in the MPA method, the "total variation" in the MTV method, and their respective weighted values. Figure 10 is an example of the results of applying the MPA method to an image, where for a range of G / R ratios, the indicated value (for example, weighted area) remains substantially the same, where each G / R ratio represents a candidate light source . In this embodiment, the indication value of the real light source is not the minimum indication value, but both the indication value and the minimum indication value of the real light source are lower than the threshold (T). In one embodiment, all candidate light sources with an indication value less than or not greater than the threshold (T) are used to calculate the average light source, and the average light source may be obtained by averaging the coordinate values of p and q of the corresponding candidate light sources. The average light source can be used as the light source of the image.

In one embodiment, the threshold (T) may be a value between the maximum value and the minimum value of the indication value, for example, T = V_min +, where V_min is the smallest indication value among all indication values generated by the candidate light source, and is smaller than The difference between the maximum and minimum values. In some cases, it may be greater than zero, such as when the candidate light source generating the minimum indication value is not a boundary candidate light source. It should be noted that the term “boundary” here refers to the p- coordinate boundary, because using the q- coordinate boundary to calculate the threshold (T) may cause significant errors in the result of calculating the average light source.

In some cases, it may be zero, such as when one of the boundary candidate light sources generates the minimum indication value, and / or when the difference between the minimum indication value and the second minimum indication value is greater than the tolerance range. The boundary candidate light source is described by a coordinate pair ( p, q ), where the p- coordinate value is at the boundary of time or space including all candidate light sources in the chromaticity coordinate system. For example, a one-dimensional curve if all candidate light source is a p-coordinate value range of the parameter of the (p, Q) space, it has on the light source trajectory candidate illuminants minimum and maximum p-coordinate value is a boundary candidate illuminants. If all candidate light sources are located in a two-dimensional chromaticity space described by a p- coordinate value range and a q- coordinate value range, the candidate light sources with the smallest and largest p- coordinate values are boundary candidate light sources.

Therefore, the general formula of the threshold T can be: T = min [(V_min + k (V_max-V_min)), V_boundary], where k is a value between 0 and 1 (for example, k = 0.125), and V_min is the minimum indication Value, V_max is the maximum indication value, and V_boundary represents the boundary indication value generated by the boundary candidate light source as defined above.

FIG. 11A is an example of an MPA calculator 1181 in an embodiment of the present invention. The MPA calculator 1181 is an alternative example of the MPA calculator 380 in FIGS. 3 and 5. As described in the foregoing, the projection plane calculator 320 and the projection area calculator 330 calculate the projection area of each candidate light source. The comparator 340 compares the projection area or weighted projection area, determines a threshold (T), and identifies those candidate light sources whose projection area or weighted projection area is not greater than T. The MPA calculator 1181 further includes an average calculator 1101 for calculating an average value of the identified candidate light sources. Each of these candidate light sources is described by a corresponding coordinate pair ( p, q ) in the chromaticity coordinate system. The average calculator 1101 averages the corresponding p- coordinate values and the corresponding q- coordinate values, respectively, to determine an average p- coordinate value ( p _avg ) and an average q- coordinate value ( q _avg ). A result average light source identified or described by an average coordinate pair ( p _avg , q _avg ) in the chromaticity coordinate system can be used as a light source for an image for subsequent image processing (eg, AWB processing).

Similarly, FIG. 11B is an example of the MTV calculator 1182 in an embodiment of the present invention. The MTV calculator 1182 is an alternative example of the MTV calculator 780 in FIG. 7. As described earlier, the difference calculator 720 calculates the total variation of each candidate light source. The comparator 730 compares the total variation or weighted total variation, determines a threshold (T), and identifies candidate light sources whose total variation or weighted total variation is not greater than T. The MTV calculator 1182 also includes an average calculator 1102 for calculating an average value of the identified candidate light sources. Each of these candidate light sources is described by a corresponding coordinate pair ( p, q ) in the chromaticity coordinate system. The average calculator 1102 averages these corresponding p- coordinate values and corresponding q- coordinate values, respectively, to determine an average p- coordinate value ( p _avg ) and an average q- coordinate value ( q _avg ). The light source of the image is identified or described by the average coordinate pair ( p _avg , q _avg ) in the chromaticity coordinate system.

FIG. 12 is a flowchart of a method 1200 for determining a light source of an image according to an embodiment of the present invention. The method 1200 may be performed by a device such as the device 150 shown in FIG. 1B. More specifically, the method 1200 may be performed by the MPA calculator 1181 of FIG. 11A or the MTV calculator 1182 of FIG. 11B.

In method 1200, the device first starts to calculate an indication value for each of a group of candidate light sources in the image (step 1210), and each candidate light source is obtained by a corresponding coordinate pair ( p, q ) in the chromaticity coordinate system. description. The device determines a threshold value of the indication value of the candidate light source (step 1220); and identifies a subset of the candidate light sources having an indication value not greater than the threshold value (step 1230). For all candidate light sources in the subset, the device averages the corresponding coordinate pairs to obtain a weighted average coordinate pair of the light sources describing the image in the chromaticity coordinate system (step 1240). More specifically, the device may average these corresponding p- coordinate values and corresponding q- coordinate values, respectively, to determine an average p- coordinate value ( p _avg ) and an average q- coordinate value ( q _avg ). The light source of the image is identified or described by a weighted average coordinate pair ( p _avg , q _avg ) in the chromaticity coordinate system.

In some embodiments, the light source of the image as calculated above can be adjusted according to the user's requirements or specifications. The user may require the image to be reproduced on an imaging device (eg, a camera) so that the white balance of the resulting image is adjusted to suit the customer's color preference. In one embodiment, chromatic adaption is performed in a chromaticity coordinate system; that is, adaptation is performed in the PQ chromaticity domain. The color vision adaptation in the PQ chromaticity domain is much simpler than the computational complexity in the LMS color space. The LMS color space is also called the human photoreceptor response color space (where "L" stands for long wavelength sensitivity and "M" stands Medium wavelength sensitivity, "S" stands for short wavelength sensitivity, cone response). Color vision adaptation in the LMS color space usually requires multiple 3x3 matrix conversions.

The user can specify the target RGB color. For example, the user may provide several original images with a gray card, and corresponding targets with gray card images. The gray card can be any percentage gray or any color suitable as a reference point. According to the images given by the user, the values of ( p _origin , q _origin ) in the original image and ( p _adapt , q _adapt ) in the target image can be calculated for the brightness levels of the scenes in these images. Values, where ( p _origin , q _origin ) are the p- coordinate and q- coordinate values of the light source before color vision adaptation; ( p _adapt , q _adapt ) are the p- coordinate and q- coordinate values of the light source after color vision adaptation .

Based on ( p _origin , q _origin ) and ( p _adapt , q _adapt ) derived from the user specification, the following models represented by equation (18) can be used to determine the degrees of adaptation A _p and A _q .

Where p _D65 and q _D65 are the p- coordinate value and the q- coordinate value of the standard light source D65, respectively. In alternative embodiments, the p- coordinate and q- coordinate values of different default light sources such as D50, light source E, or any other preferred light source may be used instead of p _D65 and q _D65 . In one embodiment, the adaptation degree _Ap required by the user specification and derived from the user specification may include multiple ranges of _Ap values, each range corresponding to a brightness level. The same method is also applicable to the fitness degree A _q .

FIG. 13 is a schematic diagram of an adaptation requirement 1310 in an embodiment of the present invention. The horizontal axis in the graph represents the brightness level (L _A ), and the vertical axis is the degree of adaptation A _p of the p coordinate. The brightness level (L _A ) of the scene in the input image can be calculated from the camera's exposure time, focal length, aperture size, and signal gain when the image is taken. In this embodiment, the chart specifies adaptation requirements 1310 through three vertical stacks marked by x, where each stack indicates the range of A _p values for the corresponding brightness level. The range of A _p values can be generated by combining user-supplied images as described above with equation (18). Although in this embodiment, the adaptation requirement 1310 specifies the range of _Ap values for the three corresponding brightness levels, it can be understood that the optional adaptation requirements can specify the range of _Ap values for any number of brightness levels.

In one embodiment, a monotonically increasing _Ap curve 1320 as a function of L _A can be generated by curve fitting such that the _Ap curve 1320 passes through all three _Ap ranges at three given brightness levels . In this embodiment, the _Ap curve 1320 can be expressed in the following form:

The parameters x ₁ , x ₂ and x ₃ can be generated by known curve fitting methods. Similarly, a user request may include another chart that L _A as a function of degree of adaptation is q A _q coordinate adaptation requirements specified. For example, the A _q curve can be expressed as:

The parameters x ₄ , x ₅ and x ₆ can be generated by known curve fitting methods. In an alternative embodiment, equations (19) and (20) may include different numbers of parameters.

As described above, the light source in the chromaticity coordinate system can be described by the coordinate pair ( p, q ), and according to the brightness level (L _A ) of the scene in the image and the adaptation level ( _A ) of the brightness level _p ) or the degree of adaptation (A _p and A _q ) to adjust the light source. Using the adaptation curve 1320, the equation (18) can be used to find an adapted light source ( p _adapt , q _adapt ) of any brightness level in the graph according to a given ( p _origin , q _origin ).

In one embodiment, equation (18) can be used to find the corresponding p _origin T ₁ or p _origin In the case of T ₂ ( p _adapt , q _adapt ), where T ₁ and T ₂ mark a boundary point, beyond which the color of the image needs to be adjusted. For T ₁ < p _origin <T ₂ , A _p = A _q = 1; therefore p _adapt = p _origin and q _adapt = q _origin .

In another embodiment, the model of (18) can be simplified based on the observation that q _adapt is generally not sensitive to A _q . Therefore, the following model represented by equation (21) can be used in p _origin T ₁ or p _origin In the case of T ₂ , find the adaptive light source ( p _adapt , q _adapt ):

Where q _offset is the distance between q _origin and the light source trajectory, and a ₁ , a ₂ and a ₃ are the coefficients of the light source trajectory (see equation (10)). In conjunction with equation (18) described above, the p- coordinate and q- coordinate values of different default light sources such as D50, light source E, or any other preferred light source can be used instead of p _D65 and q _D65 . For T ₁ <p _origin <T ₂ , A _p = A _q = 1; therefore p _adapt = p _origin and q _adapt = q _origin .

In some scenarios, the user's specification can be converted to the PQ chromaticity domain of multiple regions including curve fitting, and a different monotonically increasing _Ap curve is generated for each region. For example, when the user's specification PQ convert chrominance domain, you can specify L _A = 10, A _p is the range [0,0.1]; L _A = 11, A _p is the range [0.5, 0.6]; When L _A = 40, the range of A _p is [0.3,0.4]. In this case, a single monotonically increasing curve cannot meet all the required ranges. Therefore, multiple regions can be used, where each region includes a range or data point of _Ap that can be satisfied by a single monotonically increasing curve.

For example, the boundary of a region can be defined by p _origin , such as region 1: p _origin <T ₁ ; region 2: T ₁ p _origin <T ₂ ; ...; area n: T _n-1 p _origin <T _n ; area (n + 1): p _origin T _n .

The color vision adaptation model for each region i can be expressed as:

among them , And where x _i1 , x _i2 and x _i3 are found by curve fitting based on user specifications. Combining equations (18) and (21) as described above, the p- coordinate and q- coordinate values of different default light sources such as D50, light source E or any other preferred light source can be used instead of p _D65 and q _D65 . The transitions between regions are continuous so that slight offsets do not cause visible color changes due to different adaptations.

FIG. 14A is an AWB module 1410 in an embodiment of the present invention. The AWB module 1410 includes an MPA calculator 1181 (or an MPA calculator 380 in FIG. 3) coupled to the color vision adaptation module 1450 in FIG. 11A, and the color vision adaptation module 1450 is further coupled to the combination The gain adjustment unit 350 described above in FIGS. 3, 5, and 7 adjusts the color gain of the input image. Although only one MPA calculator is shown in the figure, in some embodiments, in combination with Figure 5 above, the AWB module 1410 may include multiple MPA calculators 380 or 1181, and each MAP calculator is an input One block of the image calculates the light source (or average light source). The color vision adaptation module 1450 calculates p _adapt and q _{adapt in} order to adjust the ( p, q ) value of the light source.

FIG. 14B is an AWB module 1420 in another embodiment of the present invention. The AWB module 1420 includes an MTV calculator 1182 (or an MTV calculator 780 in FIG. 7) coupled to the color vision adaptation module 1450 in FIG. 11B. The color vision adaptation module 1450 is further coupled to the gain. Adjusting unit 350. The color vision adaptation module 1450 calculates p _adapt and q _{adapt in} order to adjust the ( p, q ) value of the light source.

FIG. 15 is a flowchart of a method 1500 for performing color vision adaptation on an image based on a set of brightness level adaptation requirements according to an embodiment of the present invention. The method 1500 may be performed by a device such as the device 150 in FIG. 1B; more specifically, the method 1500 may be performed by an AWB module 1410 in FIG. 14A or an AWB module 1420 in FIG. 14B.

In method 1500, the device first starts to calculate the light source of the image (step 1510), where the light source is described by a coordinate pair ( p, q ) in a chromaticity coordinate system. The device recognizes the brightness level of the scene in the image (step 1520); and adjusts the coordinate pair ( p, q ) of the light source using an adaptation degree derived at least in part based on the adaptation requirements and the brightness level of the scene in the image, To obtain a suitable light source (step 1530). The device adapts the color of the image to the adapted light source (step 1540).

Referring to Figs. 1A, 1B, 3, 5, 5, 7, 7, 11A, 11B, 14A, and 14B, the descriptions of Figs. 6 and 8 will be described. The operations in the flowcharts of FIGS. 9, 12 and 15. However, it should be understood that the operations in the flowchart can be referred to in the present invention in addition to FIG. 1A, FIG. 1B, FIG. 3, FIG. 5, FIG. 7, FIG. 11A, FIG. 11B, and FIG. 14A. And other embodiments than those discussed in FIG. 14B, and FIG. 1A, FIG. 1B, FIG. 3, FIG. 5, FIG. 7, FIG. 11A, FIG. 11B, FIG. 14A, and FIG. The embodiment discussed in FIG. 14B may perform operations different from those discussed with reference to the flowchart. Although the flowchart illustrates a particular order of operations performed by certain embodiments of the invention, it should be understood that such an order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combining certain operations , Overlapping certain operations, etc.).

Different from the prior art, the present invention calculates an indication value of each candidate light source in a group of candidate light sources, where each candidate light source is described in a chromaticity coordinate system by a corresponding coordinate pair ( p, q ); determining the candidate light source The threshold value of the indication value of; identifying a subset of candidate light sources having an indication value that is not greater than the threshold value; and for all candidate light sources in the subset, calculating a weighted average of the corresponding coordinate pair to obtain a color coordinate for the The average coordinate pair of the light source of the image is described in the system, so that the real light source closest to the image can be found, the sensitivity of the white balance to the content of the scene is reduced, and the white balance method is more stable. Then, the brightness level of the light source of the image is identified according to the calculated light source; one or more adaptation degrees derived from the adaptation requirements and the brightness level of the scene in the image are used to at least partially adjust the coordinate pair of the light source ( p, q ) obtaining a suitable light source; and adjusting the color of the image to fit the light source. This can effectively perform color vision adaptation according to a given specification.

Various functional components or blocks have been described here. As will be understood by those skilled in the art, functional blocks will preferably be implemented by circuits (dedicated circuits or general purpose circuits that operate under the control of one or more processors and coded instructions), which typically include transistors The transistor is configured to control the operation of the circuit in accordance with the functions and operations described herein.

Although the invention has been described in terms of several embodiments, those skilled in the art will recognize that the invention is not limited to the described embodiments and can be modified and modified within the spirit and scope of the appended claims. Change to implement. The description is therefore to be regarded as illustrative instead of restrictive.

Claims (22)

A method for determining a light source of an image, comprising: calculating an indication value of each of a group of candidate light sources of the image, wherein each of the candidate light sources is represented by a corresponding coordinate pair ( p, q in a chromaticity coordinate system) ) To determine; determine the threshold value of the indication value of the candidate light source; identify a subset of the candidate light source having the indication value not greater than the threshold value; and for all candidate light sources in the subset, calculate the Weighted average to obtain the average coordinate pair ( p _avg , q _avg ) of the light source describing the image in the chromaticity coordinate system. The method according to item 1 of the scope of patent application, wherein the step of calculating the weighted average of the corresponding coordinate pair further comprises: averaging the corresponding p- coordinate value and the corresponding q- coordinate value, respectively, to determine the average p- coordinate value ( p _avg ) and an average q coordinate value ( q _avg ), where the average coordinate pair ( p _avg , q _avg ) describes the light source of the image in the chromaticity coordinate system.     The method according to item 1 of the scope of patent application, wherein the indication value is a projection area generated by projecting the image on a projection plane perpendicular to a vector representing a three-color value of the candidate light source.         The method according to item 1 of the scope of patent application, wherein the indication value is a total variation of three-color values between adjacent pixels, and the step of calculating the total variation further includes: calculating a linearity of the three-color values Transform to obtain three transform values; calculate a first scaling factor and a second scaling factor, the first scaling factor and the second scaling factor being used to represent two color ratios of the candidate light source; Constructing a first chroma image by the difference between the scaled first transformation value and the second transformation value; obtaining the difference between the third transformation value scaled by the second scaling factor and the second transformation value Constructing a second chroma image by the difference; and calculating the total variation by adding the absolute gradient size of the first chroma image and the absolute gradient size of the second chroma image.     The method according to item 1 of the scope of patent application, wherein when the minimum value in the indication value is generated by a candidate light source that is not one of the boundary candidate light sources, the threshold value is set higher than the minimum value in the indication value, Each of the boundary candidate light sources has a p- coordinate value located at a boundary of an interval or space of all the candidate light sources including the image in the chromaticity coordinate system. The method according to item 1 of the patent application range, wherein when the minimum value of the indication value is generated by a candidate light source of one of the boundary candidate light sources, the threshold value is set to the minimum value of the indication value, wherein Each of the boundary candidate light sources has a p- coordinate value located at a boundary of an interval or space of all the candidate light sources including the image in the chromaticity coordinate system. A method for color vision adaptation of an image, comprising: calculating a light source of the image, wherein the light source is described by a coordinate pair ( p, q ) in a chromaticity coordinate system; identifying the brightness of a scene in the image Level; adjusting the coordinate pair ( p, q ) of the light source using one or more adaptation degrees derived at least in part based on the adaptation requirements and the brightness level of the scene in the image to obtain an adapted light source ; And adapting the color of the image to the adapted light source. The method according to item 7 of the scope of patent application, wherein the step of adjusting the coordinate pair further comprises: scaling the p- coordinate value of the light source according to the degree of adaptation for the brightness level; and scaling the p The coordinate value is added to the p- coordinate value of the default light source which is a multiple of scaling (1-the adaptation degree) to obtain the adapted p- coordinate value of the adapted light source. The application method of claim 8 patentable scope item, wherein by evaluating the adaptation function of coordinate values p, q is adapted to calculate the coordinate value of the adaptation of the light source. The method according to item 7 of the scope of patent application, wherein the method further comprises: scaling the q- coordinate value of the light source according to the degree of adaptation for the brightness level; and adding scaling to the q- coordinate value after scaling ( 1-the degree of adaptation) of the default q- coordinate value of the light source to obtain the adapted q- coordinate value of the adapted light source. The method according to item 7 of the scope of patent application, wherein the method further comprises: calculating different adaptation curves of different regions of the p- coordinate value based on the adaptation requirement; and after calculating the light source of the image, One of the region and one of the adaptation curves are identified according to the p- coordinate value of the light source, for calculating the adapted light source. An apparatus for determining a light source of an image, comprising: a memory for storing the image; and an image processing pipeline coupled to the memory for: calculating each of a set of candidate light sources for the image , Where each candidate light source is described by a corresponding coordinate pair ( p, q ) in the chromaticity coordinate system; determining the threshold value of the indication value of the candidate light source; identifying the indication having a value not greater than the threshold value A subset of the candidate light source values; and for all candidate light sources in the subset, calculating a weighted average of the corresponding coordinate pair to obtain an average coordinate pair of the light source describing the image in the chromaticity coordinate system ( p _avg , q _avg ). The device according to item 12 of the scope of patent application, wherein, when calculating the weighted average of the corresponding coordinate pair, the image processing pipeline is further used for: respectively performing a corresponding p- coordinate value and a corresponding q- coordinate value Average to determine an average p- coordinate value ( p _avg ) and an average q- coordinate value ( q _avg ), where the average coordinate pair ( p _avg , q _avg ) describes the light source of the image in the chromaticity coordinate system.     The device according to item 12 of the scope of patent application, wherein the indication value is a projection area generated by projecting the image onto a projection plane perpendicular to a vector representing a three-color value of the candidate light source.         The device according to item 12 of the scope of patent application, wherein the indication value is the total variation of the three-color values between adjacent pixels. When calculating the total variation, the image processing pipeline is further used for: calculating Linear conversion of the three color values to obtain three transform values; calculating a first scaling factor and a second scaling factor, the first scaling factor and the second scaling factor being used to represent two color ratios of the candidate light source; Constructing a first chroma image by a difference between the first transformed value and the second transformed value scaled by the first scaling factor; obtaining a third transformed value scaled by the second scale factor and the first transformed value Constructing a second chroma image by the difference between the two transform values; and calculating the total change by adding the absolute gradient size of the first chroma image and the absolute gradient size of the second chroma image difference.     The device according to item 12 of the scope of patent application, wherein when the minimum value in the indication value is generated by a candidate light source that is not one of the boundary candidate light sources, the threshold value is set higher than the minimum value in the indication value , Wherein each of the boundary candidate light sources has a p- coordinate value located at a boundary of an interval or space of all the candidate light sources including the image in the chromaticity coordinate system. The device according to item 12 of the scope of patent application, wherein when the minimum value in the indication value is generated by a candidate light source of one of the boundary candidate light sources, the threshold value is set to the minimum value in the indication value, wherein Each of the boundary candidate light sources has a p- coordinate value located at a boundary of an interval or space including all the candidate light sources of the image in the chromaticity coordinate system. A device for color vision adaptation of an image includes: a memory for storing the image; and an image processing pipeline coupled to the memory for: calculating a light source of the image, wherein the light source is composed of Describe the coordinate pair ( p, q ) in the chromaticity coordinate system; identify the brightness level of the scene in the image; use one or at least partially derived from the adaptation requirements and the brightness level of the scene in the image Multiple adaptation degrees to adjust the coordinate pair ( p, q ) of the light source to obtain an adapted light source; and adapt the color of the image to the adapted light source. The device according to item 18 of the scope of patent application, wherein when adjusting the coordinate pair, the image processing pipeline is further configured to: scale the p- coordinate value of the light source according to the degree of adaptation for the brightness level; and the coordinate values of the p-scaled coordinate values p added to the scaled multiple of the default light source (1 the adaptation degree), adapted to obtain the coordinate value of the p light source adapted. The device according to item 19 of the scope of patent application, wherein when adjusting the coordinate pair, the image processing pipeline is further configured to calculate an adaptation of the adapted light source by evaluating a function of the adapted p coordinate value. q coordinate value. The device according to item 18 of the scope of patent application, wherein when adjusting the coordinate pair, the image processing pipeline is further configured to: scale the q- coordinate value of the light source according to the adaptability for the brightness level; the default value of the source coordinate q q coordinate value of the added scaled the scaling (l the adaptation of) a multiple of q coordinate value for the adaptation of the adaptation of the light source. The device according to item 18 of the scope of patent application, wherein when adjusting the coordinate pair, the image processing pipeline is further used for: calculating different adaptation curves of different regions of the p- coordinate value based on the adaptation requirement; And after the light source of the image is calculated, one of the region and one of the adaptation curves are identified according to the p- coordinate value of the light source for calculating the adapted light source.