TWI543631B - Image processing system adaptable to a dual-mode image device - Google Patents

Description

Image processing system for dual mode imaging devices

The present invention relates to an image processing system, and more particularly to an adaptable color correction system suitable for use in a dual mode image device.

In the image system, when all the pixels of the pixel array are subjected to the same illumination, the pixels far from the center usually have a lower pixel signal, which is called a lens shading phenomenon. Therefore, the brightness of the image will decrease from the center to the corner. In other words, the maximum brightness is at or near the center and decreases along the radial direction of the pixel array. The cause of the lens shadow phenomenon may be the lens mechanism, the optical element, the sensor pixel, the light travel distance, the aperture effect, or the incident light angle.

Lens Shading Correction (LSC) uses different gains to compensate for the reduction in brightness, especially for pixels that are far from the center of the pixel array.

A complementary metal oxide semiconductor (CMOS) image sensor not only senses the visible light band, but also senses the near-infrared band. The IR-cut filter can pass the wavelength of the visible light band and block the wavelength of the infrared light band, thus avoiding discoloration caused by the wavelength of the infrared light band. However, in monitoring applications, artificial infrared light is used to improve the visibility of low brightness conditions. If a dual-band IR-cut filter is used, artificial infrared light can be used at low brightness to receive the infrared light band signal, and infrared light is not used at normal brightness to avoid discoloration.

Image processing systems typically use color correction to correct for color distortion. Therefore, it is necessary to propose a novel color correction mechanism for applications such as monitoring.

In view of the above, one of the objectives of the embodiments of the present invention is to provide an image processing system suitable for a dual mode image device, which is used for an image according to a chief ray angle (CRA) of a pixel and an infrared light signal of a light source. The pixels are corrected.

According to an embodiment of the invention, an image processing system suitable for a dual mode image device includes an adaptive color correction system that receives an image and uses the infrared light signal of the light source associated with the dual mode image device according to the chief ray angle of the pixel of the image. Correctly correct color distortion.

The first figure shows a block diagram of an image processing system 100 suitable for dual mode video device 200 in accordance with an embodiment of the present invention. The dual mode imaging device 200 primarily includes a lens 20 for passing (eg, transmitting and refracting) light. In this specification, the term "lens" may refer to a single lens or combination of lenses.

The dual mode imaging device 200 also includes a dual band infrared light cut filter 21 that is optically coupled to receive light passing through the lens 20 for passing visible light (of the first frequency band) and infrared light range (of the second frequency band). The dual mode imaging device 200 further includes a filter array 22 that includes color filters (eg, red (R), green (G), and blue (B) filters) and infrared (IR) filters. The filter array 22 is optically coupled to receive light passing through the dual band infrared light cut filter 21 for generating a color message. The second A diagram illustrates the color filter and the infrared light filter of the filter array 22 of the first figure. In this example, red light accounts for 25%, green light accounts for 25%, blue light accounts for 25%, and infrared light accounts for 25%. The second B diagram illustrates the spectral sensitivity of the dual band infrared light cut filter 21 and filter array 22 of the first figure.

The dual mode image device 200 further includes an image sensor 23, such as a complementary metal oxide semiconductor (CMOS) image sensor, disposed under the filter array 22 for converting optical color information into an electronic signal, which is subjected to image processing. Processing of system 100. In this specification, each component block of image processing system 100 may refer to a structure or function individual that may be executed by circuitry, such as a digital image processor.

In the present embodiment, the image processing system 100 may include a lens shading correction (LSC) unit 11 for correcting lens shading artifacts. The details of the lens shading correction can be found in U.S. Patent No. 8,228,406, issued to Kuo et al., entitled "Adaptive lens shading correction" or U.S. Patent No. 8,130,292, to the inventor Lee, entitled "Video Device "Scene illumination adaptive lens shading correction for imaging devices".

The image processing system 100 can also include a demosaicking unit 12 that receives the output of the lens shading correction unit 11 for reforming the non-complete color samples of the image display 23 covered with the filter array 22 into a full color. image. The demosaicing unit 12 can typically be executed by interpolation techniques.

The image processing system 100 of the present embodiment includes an adaptive color correction system 13 that receives the (full color) image reconstructed by the demosaicing unit 12 for adaptively correcting color distortion. According to one of the features of the present embodiment, the adaptive color correction system 13 performs the correction of the image pixels based on the chief ray angle (CRA) of the pixel and the infrared light signal of the light source (from the image display 23). In another embodiment, the adaptive color correction system 13 performs correction of the image pixels based on the chief ray angle (CRA) of the pixel, the color temperature of the light source, and the infrared light signal of the light source. In the present specification, the chief ray angle of the pixel far from the center of the lens 20 is larger than the chief ray angle of the pixel near the center of the lens 21.

Figure 3A shows a detailed block diagram of an adaptive color correction system 13 (first image) of a first particular embodiment of the present invention. In the present embodiment, four color correction matrices CCM1, CCM2, CCM3, and CCM4 are provided. The first color correction matrix CCM1 represents a color correction matrix of the high color temperature light source and the image center, the second color correction matrix CCM2 represents a color correction matrix of the low color temperature light source and the image center, and the third color correction matrix CCM3 represents a low brightness infrared light signal. The color correction matrix of the light source and the corner of the image, and the fourth color correction matrix CCM4 represents a color correction matrix of the high-intensity infrared light signal source and the image corner. In the present specification, "high" and "low" are relatively comparative. According to another definition, a high color temperature means that the color temperature is higher than a preset value, and a low color temperature means that the color temperature is lower than a preset value. In a similar situation, high brightness means that the brightness is higher than a predetermined value, and low brightness means that the brightness is lower than a predetermined value.

The tunable color correction system 13 can include a color temperature (CT) weight generator 131 that receives a color temperature index that can be provided by the color temperature estimator 14. The color temperature weight generator 131 generates color temperature weights for mixing the first color correction matrix CCM1 and the second color correction matrix CCM2 by the first mixing unit 132, thereby generating a first mixed color correction matrix (CCM).

The tunable color correction system 13 can also include an infrared light (IR) weight generator 133 that receives the infrared light ratio (i.e., the ratio of the infrared light signal to the color signal), which can be provided by the infrared light estimator 15. The infrared light weight generator 133 generates infrared light weights for mixing the third color correction matrix CCM3 and the fourth color correction matrix CCM4 by the second mixing unit 134, thereby generating a second mixed color correction matrix.

The adaptive color correction system 13 may further include a chief ray angle (CRA) weight generator 135 that generates a chief ray angle weight based on the pixel coordinates of the image. The chief ray angle weight is used by the third blending unit 136 to blend the first blend color correction matrix with the second blend color correction matrix, thereby producing a third (final) blend color correction matrix. The third mixed color correction matrix is used to color correct the color signals (eg, R, G, and B) by the color correction unit 137, thereby generating a corrected color signal. The correction color signal may be subjected to other processing such as gamma correction performed by the gamma unit 16 illustrated in the first figure.

Figure 3B shows a detailed block diagram of an adaptive color correction system 13 (first image) of a second particular embodiment of the present invention. The second specific embodiment is similar to the first specific embodiment, and the different places are explained below. As shown in the third B diagram, three color correction matrices CCM1, CCM2, and CCM3 are provided. The first color correction matrix CCM1 represents a color correction matrix of the high color temperature light source and the image center, the second color correction matrix CCM2 represents a color correction matrix of the low color temperature light source and the image center, and the third color correction matrix CCM3 represents the high brightness infrared light. The color correction matrix of the signal source and the image center. The color temperature weights are used by the first mixing unit 132 to mix the first color correction matrix CCM1 with the second color correction matrix CCM2, thus producing a first mixed color correction matrix. The infrared light weight is used by the second mixing unit 134 to mix the first mixed color correction matrix with the third color correction matrix CCM3, thereby producing a second mixed color correction matrix. The chief ray angle weight is used by the third blending unit 136 to blend the first blend color correction matrix with the second blend color correction matrix, thereby producing a third (final) blend color correction matrix.

Figure 3C shows a detailed block diagram of an adaptive color correction system 13 (first image) of a third particular embodiment of the present invention. The third specific embodiment is similar to the first specific embodiment, and the different places are explained below. As shown in the third C diagram, five color correction matrices CCM1, CCM2, CCM3, CCM4, and CCM5 are provided. The first color correction matrix CCM1 represents a color correction matrix of a high color temperature light source without an infrared light signal, and the second color correction matrix CCM2 represents a color correction matrix of a low color temperature light source without an infrared light signal, and the third color correction matrix CCM3 represents The color temperature correction matrix of the high color temperature light source of the infrared light signal and the image center, the fourth color correction matrix CCM4 represents the color correction matrix of the low color temperature light source with the infrared light signal and the image center, and the fifth color correction matrix CCM5 represents the high brightness infrared light. The color correction matrix of the signal and image corners. The color temperature weights are used by the first mixing unit 132 to mix the first color correction matrix CCM1 with the second color correction matrix CCM2, thus producing a first mixed color correction matrix. The color temperature weights are also used by the second mixing unit 134 to mix the third color correction matrix CCM3 with the fourth color correction matrix CCM4, thus producing a second mixed color correction matrix. The chief ray angle weight is used by the third mixing unit 136 to mix the second mixed color correction matrix with the fifth color correction matrix CCM5, thus producing a third mixed color correction matrix. The infrared light weight is used by the fourth mixing unit 138 to mix the first mixed color correction matrix with the third mixed color correction matrix, thereby producing a fourth (final) mixed color correction matrix.

Details of the third A/three B/three C diagram and the related blocks of the first diagram will be described below. The color temperature estimator 14 of the first figure is based on the gain (which may be provided by a white balance unit (not shown)) to estimate the color temperature of the light source. Figure 4A illustrates the relationship between red light gain and blue light gain, where the high color temperature source contains more power in the short wavelength band (i.e., blue light). Therefore, the red channel requires a higher gain such that the balanced white object has the same red, green, and blue values. Conversely, a low color temperature source contains more power in the long wavelength band (i.e., red light). Therefore, the blue channel requires a higher gain, so that the balanced white object has the same red, green, and blue values. Thereby, the color temperature estimator 14 can provide a color temperature index based on the gain (especially the red and blue gains) to represent the estimated color temperature of the light source. In this embodiment, the color temperature index is the difference in gain between blue light and red light.

Next, the color temperature weight generator 131 of the third A/three B/triple C map generates a color temperature weight based on the color temperature index. The fourth B graph shows the relationship between the color temperature weight and the gain difference between blue light and red light. As shown in the fourth B diagram, the color temperature weight at the low difference point Th_1 is 0, and the linear temperature weight is increased to 1 at the high critical point Th_h. When the gain difference is less than the low critical point Th_1, the color temperature weight is 0; when the gain difference is greater than the high critical point Th_h, the color temperature weight is 1.

Figure 5A shows a block diagram of the infrared light estimator 15 of the first figure. The original signals (from image sensor 23) are transmitted to the accumulators 152 of red, green, blue, and infrared pixels by demultiplexer 151, respectively. The red light pixel, the green light pixel, the blue light pixel, and the infrared light pixel (from the accumulator 152) are respectively divided by the divider 153 by an individual count value, thereby obtaining a red pixel, a green pixel, a blue pixel, and an infrared pixel. Average signal (ie, R_avg, G_avg, B_avg, and IR_avg). The adding unit 154 sums the average signals of the red, green, and blue pixels, and the obtained sum is divided by the dividing unit 155 to remove the average signal of the infrared pixels. The resulting quotient is limited by the clipping unit 156 to a magnitude between 0 and 1, thus producing an infrared light ratio.

Next, the infrared light weight generator 133 of the third A/triple B/triple C map generates an infrared light weight based on the infrared light ratio. Figure 5B shows the relationship between the infrared light weight and the infrared light ratio. As shown in FIG. 5B, the infrared light weight at the low-threshold point Th_1 is 0, and the linear weight is increased to 1 at the high critical point Th_h. When the infrared light ratio is less than the low critical point Th_1, the infrared light weight is 0; when the infrared light ratio is greater than the high critical point Th_h, the infrared light weight is 1.

Regarding the chief ray angle weight generator 135 of the third A/three B/tripty C diagram, the sixth graph shows the relationship between the chief ray angle weight and the (proportional adjustment) distance R of the pixel-image center. As shown in FIG. 6A, the chief ray angle weight of the distance R at the low critical point Th_1 is 0, and the chief ray angle weight at which the linear or nonlinear increase to the high critical point Th_h is 1. When the distance R is smaller than the low critical point Th_1, the chief ray angle weight is 0; when the distance R is greater than the high critical point Th_h, the chief ray angle weight is 1. Figure 6B shows the relationship between the chief ray angle weight and the (proportional adjustment) X coordinate, (proportional adjustment) Y coordinate.

The execution of the first mixing unit 132, the second mixing unit 134, the third mixing unit 136, and the fourth mixing unit 138 of the third A/three B/triple C diagram can be expressed as follows: Where alpha represents the color temperature weight, infrared light weight or chief ray angle weight, [c ₁ ... c ₉ ] and [d ₁ ... d ₉ ] represent the color correction matrix input of the mixing unit, and [e ₁ ... e ₉ ] represents the mixing unit Color correction matrix output.

In another embodiment, different principal ray angles may be used to weight the third mixing unit 136, expressed as follows:

The execution of the color correction unit 137 of the third A/three B/tripty C diagram can be expressed as follows: Where [ R _i G _i B _i ] represents the color (ie, R, G, and B) signals, [ R _o G _o B _o ] represents the corrected color (ie, R, G, and B) signals, and [ c ₁ ... c ₉ ] represents the final mixed color correction matrix.

The above description is only the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention; all other equivalent changes or modifications which are not departing from the spirit of the invention should be included in the following Within the scope of the patent application.

100 image processing system 11 lens shading correction unit 12 demosaicing unit 13 adaptive color correction system 131 color temperature weight generator 132 first mixing unit 133 infrared light weight generator 134 second mixing unit 135 main ray angle weight generator 136 third Mixing unit 137 Color correcting unit 138 Fourth mixing unit 14 Color temperature estimator 15 Infrared light estimator 151 Demultiplexer 152 Accumulator 153 Divider 154 Adding unit 155 Division unit 156 Amplitude unit 16 Gamma unit 200 Dual mode image device 20 Lens 21 Dual band infrared light cut filter 22 Filter array 23 Image sensor CCM1 First color correction matrix CCM2 Second color correction matrix CCM3 Third color correction matrix CCM4 Fourth Color correction matrix CCM5 Fifth color correction matrix Th_l Low critical point Th_h High critical point R_avg Average signal of red light pixels G_avg Average signal of green light pixels B_avg Average signal of blue light pixels IR_avg Average signal of infrared light pixels R Distance

The first figure shows a block diagram of an image processing system suitable for a dual mode image device in accordance with an embodiment of the present invention. The second A diagram illustrates the color filter and the infrared light filter of the filter array of the first figure. The second B diagram illustrates the spectral sensitivity of the dual-band infrared cut-off filter and the filter array of the first figure. Figure 3A shows a detailed block diagram of an adaptive color correction system (first image) of a first particular embodiment of the present invention. Figure 3B shows a detailed block diagram of an adaptive color correction system (first image) of a second particular embodiment of the present invention. Figure 3C shows a detailed block diagram of an adaptive color correction system (first image) of a third particular embodiment of the present invention. Figure 4A illustrates the relationship between color temperature, red light gain, and blue light gain. The fourth B graph shows the relationship between the color temperature weight and the gain difference between blue light and red light. Figure 5A shows a block diagram of the infrared estimator of the first figure. Figure 5B shows the relationship between the infrared light weight and the infrared light ratio. Figure 6A shows the relationship between the chief ray angle weight and the distance from the pixel-image center. Figure 6B shows another relationship between the chief ray angle weight and the X coordinate and the Y coordinate.

100 Image Processing System 11 Lens Shading Correction Unit 12 De-mosaic Unit 13 Adjustable Color Correction System 14 Color Temperature Estimator 15 Infrared Light Estimator 16 Gamma Unit 200 Dual Mode Image Device 20 Lens 21 Dual Band Infrared Optical Cut Filter 22 Filter Array 23 image sensor

Claims (15)

An image processing system suitable for a dual mode image device, comprising: an adaptable color correction system, receiving an image, according to a primary light angle of a pixel of the image and an infrared light signal of a light source associated with the dual mode image device; Adjusting color distortion adaptively; a lens shading correction unit receiving an electronic signal from the dual mode image device for correcting lens shadow artifacts; and a demosaicing unit receiving the output of the lens shading correction unit for The color samples output by the dual mode image device are reformed. An image processing system suitable for a dual mode image device according to the first aspect of the patent application, wherein the adjustable color correction system further performs correction according to a color temperature of the light source. An image processing system suitable for a dual mode image device as described in claim 2, wherein the adjustable color correction system performs correction based on the color temperature and an image center. An image processing system suitable for a dual mode image device according to claim 3, wherein the adjustable color correction system performs correction based on the infrared light signal and an image corner. An image processing system suitable for a dual mode image device according to the first aspect of the patent application, wherein the dual mode image device comprises: a lens for passing light; and a dual band infrared light cut filter for receiving the lens Light passing through the visible light and infrared light range; a filter array comprising a color filter and an infrared light filter, the filter array receiving the light passed by the dual-band infrared light cut filter to generate a color message; An image sensor is disposed under the filter array for converting a color message into an electronic signal. An image processing system suitable for a dual mode image device, comprising: An adjustable color correction system receives an image, and adjusts color distortion according to a primary light angle of a pixel of the image and an infrared light signal of a light source associated with the dual mode image device, and the adjustable color correction system is further based on The color temperature of the light source, an image center or an image corner to perform correction, wherein the adaptive color correction system comprises: a complex color correction matrix having different color temperature light sources, infrared light signals or chief ray angles; and a color temperature weight generator Receiving a color temperature index to generate a color temperature weight; an infrared light weight generator receiving an infrared light ratio to generate an infrared light weight; a principal light angle weight generator, generating a master according to the pixel coordinates of the image a color angle weighting unit; and a color correction unit that performs color correction on the color signal of the image to generate a corrected color signal; wherein the color correction matrices are mixed according to the color temperature weight, the infrared light weight, and the chief ray angle weight To generate a final mixed color correction matrix; and the color correction list The final mixing of the color correction matrix to perform correction. The image processing system applicable to the dual mode image device according to claim 6 of the patent application scope further includes a color temperature estimator for estimating the color temperature of the light source according to the gain, thereby generating the color temperature index. An image processing system suitable for a dual mode image device as described in claim 7 of the patent application, wherein the color temperature index is a difference in gain between blue light and red light. According to the image processing system of the dual mode image device described in claim 8 of the patent application, wherein the color difference weight of the gain difference at the low critical point is 0, and the linear temperature weight increase to a high critical point is 1. The image processing system applicable to the dual mode image device according to claim 6 of the patent application scope further includes an infrared light estimator that receives the original signal from the dual mode image device to generate the infrared light ratio. An image processing system suitable for a dual mode image device according to claim 10, wherein the infrared light estimator comprises: a demultiplexer for demultiplexing the original signal; a plurality of accumulators, receiving The multiplexed signal is used to separately add the color pixel and the infrared light pixel; the complex divider divides the accumulated color pixel and the infrared light pixel by the individual pixel count values, thereby obtaining an average signal of the color pixel and the infrared light pixel; An addition unit that sums the average signals of the color pixels to obtain a sum value; a division unit divides the average signal of the infrared pixels by the sum value to generate a quotient value; and a limiting unit, The quotient is limited to produce the infrared ratio. An image processing system suitable for a dual mode image device according to claim 10, wherein the infrared light weight at a low critical point is 0, and the linear weight is increased to 1 at a high critical point. . An image processing system suitable for a dual mode image device as described in claim 6 wherein: the color correction matrix comprises a first color correction matrix to represent a color correction matrix of the high color temperature source and the image center, and a second color correction The matrix represents a color correction matrix of the low color temperature light source and the image center, a third color correction matrix to represent a low brightness infrared light signal source and a color correction matrix of the image corner, and a fourth color correction matrix to represent the high brightness infrared light signal source and image. Color correction matrix of the corner; The first color correction matrix and the second color correction matrix are mixed according to the color temperature weight to generate a first mixed color correction matrix; the third color correction matrix and the fourth color correction matrix are mixed according to the infrared light weight To generate a second mixed color correction matrix; and the first mixed color correction matrix and the second mixed color correction matrix are blended according to the chief ray angle weight to generate the final mixed color correction matrix. An image processing system suitable for a dual mode image device as described in claim 6 wherein: the color correction matrix comprises a first color correction matrix to represent a color correction matrix of the high color temperature source and the image center, and a second color correction The matrix represents a color correction matrix of the low color temperature light source and the image center, and a third color correction matrix to represent a color correction matrix of the high brightness infrared light signal source and the image center; the first color correction matrix and the second color correction matrix are Color temperature weights, mixed to produce a first mixed color correction matrix; the first mixed color correction matrix and the third color correction matrix are blended to generate a second mixed color correction matrix according to the infrared light weight; and the first blend The color correction matrix and the second mixed color correction matrix are blended according to the chief ray angle weights to produce the final mixed color correction matrix. An image processing system suitable for a dual mode image device according to the scope of claim 6 wherein: the color correction matrix comprises a first color correction matrix to represent a color correction matrix of a high color temperature light source having no infrared light signal, and a second The color correction matrix represents a color correction matrix of a low color temperature light source that does not have an infrared light signal, and a third color correction matrix to represent a high color temperature light source and image with an infrared light signal a central color correction matrix, a fourth color correction matrix to represent a color temperature correction matrix of a low color temperature light source with an infrared light signal and an image center, and a fifth color correction matrix to represent a high brightness infrared light signal and a color correction matrix of the image corner; The first color correction matrix and the second color correction matrix are mixed according to the color temperature weight to generate a first mixed color correction matrix; the third color correction matrix and the fourth color correction matrix are mixed according to the color temperature weight to generate a second mixed color correction matrix; the second mixed color correction matrix and the fifth color correction matrix are mixed according to the chief ray angle weight to generate the third mixed color correction matrix; and the first mixed color correction matrix and the A third hybrid color correction matrix is blended to generate the final blend color correction matrix based on the infrared light weights.