KR100791397B1 - Method and apparatus for composing image signal having different exposure time - Google Patents

Abstract

The present invention relates to a method and apparatus for synthesizing image signals having different exposure times, wherein a minimum value M is determined from pixel values R, G, and B of a first image signal having a pixel value of (R, G, B). The color gains (g _R , g _G , g _B ) of the first image signal are determined using the M, and the pixel values (r, determining a minimum value m in g, b), wherein the exposure time of the second image signal is a low exposure image signal shorter than the exposure time of the first image signal; Determine the color gain (g _r , g _g , g _b ), and add the pixel values (r, g, b) of the second image signal to the pixel values (R, G, B) of the first image signal Obtain (R + r, G + g, B + b), color gain of the first image signal (g _R , g _G , g _B ) and color gain of the second image signal (g _r , g _g , g _b ) of the obtained (R + r, G + g, B + b) The color gains C _R , C _G , and C _B are determined according to the magnitudes, and a composite value of the first image signal and the second image signal is determined based on the color gains C _R , C _G , and C _B. Calculating.

Exposure, color image, compositing, image sensor, gain

Description

Method and apparatus for synthesizing image signals having different exposure times {Method and apparatus for composing image signal having different exposure time}

1 to 3 are photographs showing an example of synthesizing two images according to the prior art to widen the dynamic range.

4 and 5 are photographic exemplary diagrams for explaining a problem in synthesizing two images according to the prior art.

6 is a block diagram of an apparatus for synthesizing two image signals having different exposure times according to an exemplary embodiment of the present invention.

7 is a flowchart illustrating a procedure of synthesizing two image signals having different exposure times according to an exemplary embodiment of the present invention.

8 is a graph of color gain versus image signal for obtaining color gain in accordance with a preferred embodiment of the present invention.

9 is an exemplary view of a photograph outputted by a method of synthesizing two image signals having different exposure times according to an exemplary embodiment of the present invention.

10 is a configuration diagram of an image sensor to which a method of synthesizing two image signals having different exposure times according to an exemplary embodiment of the present invention is applied.

<Explanation of symbols for the main parts of the drawings>

603: minimum value determination unit

605: color gain determination unit

607 addition operation

609: color gain determination unit

611: compound calculation unit

The present invention relates to a method and apparatus for synthesizing image signals having different exposure times.

The image sensor refers to a semiconductor device that converts an optical image into an electrical signal. Device types of image sensors include CCD (Charge Coupled Device) devices and CMOS (Complementary Metal-Oxide-Silicon) devices. CCD-type device is a device in which charge carriers are stored and transferred to capacitors while individual metal-oxide-silicon (MOS) capacitors are located in close proximity to each other. It is a device that adopts the switching method of making the MOS transistor as many pixels as the number of pixels using the technique used and detecting the output sequentially using it.

CCD (charge coupled device) has many disadvantages such as complicated driving method and high power consumption, complicated process due to the large number of mask process steps, and difficulty in making one chip because signal processing circuit cannot be implemented in CCD chip. Recently, in order to overcome such drawbacks, the development of a CMOS image sensor using a sub-micron CMOS manufacturing technology has been studied a lot. The CMOS image sensor forms an image by forming a photodiode and a MOS transistor in a unit pixel (Pixel) and sequentially detects a signal in a switching method, and implements an image.As a CMOS manufacturing technique, the power consumption is low and the number of masks is about 30 to 40 with 20 masks. Compared to CCD process that requires two masks, the process is very simple, and it is possible to make various signal processing circuits and one chip, which is attracting attention as the next generation image sensor.

An image sensor for realizing a color image has an array of color filters on the light sensing portion that receives and receives light from the outside to generate and accumulate photocharges. The color filter array (CFA) consists of three colors: red, green, and blue, or three colors: yellow, magenta, and cyan. It is made of color.

On the other hand, one of the important criteria for indicating the quality of an image device is the dynamic range. Dynamic range generally represents the maximum range in which a signal can be processed without distorting the input signal. For image sensors, the wider the dynamic range, the better the image can be obtained regardless of the wide range of brightness changes. However, conventional color image sensors have a narrow dynamic range, such as red, green, and blue, for example, in color image sensors consisting of three colors: red, green, and blue. ) If one or more of the colors are saturated, there is a disadvantage that the original color of the image is not well represented. In order to overcome the shortcomings of such a dynamic range, a method of synthesizing images having different exposure times has been attempted. As a method of synthesizing images having different exposure times, a conventional method of combining two image signals and multiplying a predetermined gain is used. In this case, the gain may be determined by a specific exposure value of the image. 1 to 3 are photographs showing an example of synthesizing two images according to the prior art to widen the dynamic range. 1 to 3, a high exposure photograph 101 and a low exposure photograph 201 were synthesized to obtain a photograph 301 having a wide dynamic range. In this case, a method of adding a high-exposure pixel value and a low-exposure pixel value and multiplying a gain of 0.5 is used. In the above example, a good picture can be obtained by synthesizing two images without any problem, but such a synthesis method has a problem in that color information is lost in the case of saturated pixels at high exposure. That is, when the R, G, and B pixel values are (255, 255, 255), the original (255, 510, 765) should be saturated and become (255, 255, 255), and (255, 266 and 277 may be saturated to become (255, 255, 255). In this case, the color information may be lost in the corresponding pixel when saturated. In this case, when the compositing is performed in the above manner, there is a problem that the image restored by the compositing may not be properly restored. . For example, referring to FIGS. 4 and 5, there is a problem in that a portion that is saturated and not well represented in the high-exposure picture 401 is corrected to be close to gray in the synthesized picture 501.

An object of the present invention for solving the above problems is to provide a method and apparatus for synthesizing image signals having different exposure times, which can widen the dynamic range by synthesizing image signals having different exposure times.

Another object of the present invention is to provide a method and apparatus for synthesizing image signals having different exposure times so as not to distort or damage color images.

In order to achieve the above objects, according to an aspect of the present invention, in the method of synthesizing image signals having different exposure times, the pixel value R of the first image signal having a pixel value of (R, G, B) Determining the minimum value M in G, B); Determining a color gain (g _R , g _G , g _B ) of the first image signal using the M; Determining a minimum value m from pixel values (r, g, b) of the second image signal having a pixel value of (r, g, b), wherein an exposure time of the second image signal is determined by Low exposure image signal shorter than exposure time; Determining a color gain (g _r , g _g , g _b ) of the second image signal using the m; Obtaining (R + r, G + g, B + b) by adding the pixel values (r, g, b) of the second image signal to the pixel values (R, G, B) of the first image signal step; The obtained (R + r, G +) based on the color gains (g _R , g _G , g _B ) of the first image signal and the color gains (g _r , g _g , g _b ) of the second image signal. g, B + b) to determine the color gain (C _R , C _G , C _B ); And calculating a composite value of the first image signal and the second image signal based on the color gains C _R , C _G , and C _B. Can provide.
Here, the color gains g _R , g _G and g _B of the first image signal may be (R / M, G / M, B / M).
In addition, the color gain g _r g _g g _b of the second image signal may be (r / m, g / m, b / m).
In addition, the color gain C _R when R + r is less than a predetermined r1 and g _R, R + r is a greater than a predetermined r2 g _r, when R + r is large is smaller than r2 than the r1 g _R + ((g _r -g _R ) / (r2-r1)) (R + r-r1).
In addition, the r1 is 250, the r2 may be 300 to 350.
In addition, the combined value of the two image signals may be (S (M * C _R + r), S (M * C _G + g), S (M * C _B + b)).
In addition, S may be a gain greater than 0 and less than 1.
According to another aspect of the present invention, in an apparatus for synthesizing image signals having different exposure times, the minimum value M in pixel values R, G, and B of the first image signal having pixel values of (R, G, and B). And a minimum value determiner that determines a minimum value m from pixel values (r, g, b) of the second image signal having a pixel value of (r, g, b), wherein the exposure time of the second image signal is A low exposure image signal shorter than an exposure time of the first image signal; Color gain (g _R , g _G , g _B ) of the first image signal is determined using M, and color gain (g _r , g _g , g _b ) of the second image signal is determined using m. A color gain determination unit to determine a; Obtaining (R + r, G + g, B + b) by adding the pixel values (r, g, b) of the second image signal to the pixel values (R, G, B) of the first image signal An addition operation unit; Based on the color gains (g _R , g _G , g _B ) of the first image signal and the color gains (g _r , g _g , g _b ) of the second image signal, (R + r, G + g, A color gain determiner that determines the color gains C _R , C _G , C _B according to the size of B + b); And a synthesis calculator configured to calculate a combined value of the first image signal and the second image signal based on the color gains C _R , C _G , and C _B. A device can be provided.

delete

delete

Next, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

6 is a block diagram of a device for synthesizing image signals having different exposure times according to an exemplary embodiment of the present invention.

Referring to FIG. 6, in the method of synthesizing image signals having different exposure times according to the present invention, the color information of the region saturated in the photograph 101 due to the high exposure of FIG. It is based on the fact that the image of the photo 201 due to low exposure of the. That is, the photo 201 due to the low exposure of FIG. 2 has a low color level, but the color information is alive for a portion that is not represented in the photo 101 due to the high exposure of FIG. 1. Prior to the description of FIG. 6, the image signal due to high exposure is defined as (R, G, B), and the image signal due to low exposure is defined as (r, g, b).

The data synthesizing unit 600, which is a device for synthesizing two image signals having different exposure times according to the present invention, includes an image signal receiving unit 601, a minimum value determining unit 603, a color gain determining unit 605, and an addition calculating unit 607. ), A color gain determination unit 609, a synthesis calculation unit 611, and an output unit.

The image signal receiver 601 is a part that receives two image signals having different exposure times from the pixel portion of the image sensor. In this case, the image signal represents a color image and will be the high exposure image signals R, G, and B and the low exposure image signals r, g, and b defined above.

The minimum value determining unit 603 performs a function of determining a minimum value among pixel values for each of the two image signals received by the image signal receiving unit 601. That is, the minimum value determination unit 603 determines the minimum value M of the image signals R, G, and B due to high exposure, and determines the minimum value m of the image signals r, g and b due to the low exposure.

The color gain determination unit 605 calculates a color gain for each pixel using the minimum values M and m determined by the minimum value determination unit. At this time, if the color gain of the high exposure image signal is (g _R , g _G , g _B ) and the color gain of the low exposure image signal is (g _r , g _g , g _b ), they are respectively (R / M, G / M, B / M) and (r / m, g / m, b / m).

The addition operation unit 607 performs a function of adding two image signals R, G, B and (r, g, b) received by the image signal receiving unit 601. At this time, the image signal when the addition operation is performed in the addition operation unit 607 will be (R + r, G + g, B + b).

The color gain determiner 609 uses the color gain calculated by the color gain determiner 605 and the image gains R + r, G + g, and B + b added by the add operation unit 607. It performs the function of determining (C _R , C _G , C _B ). At this time, the color gain is determined differently according to the added image signals R + r, G + g, and B + b.

The synthesis operation unit 611 combines the two image signals received by the image signal receiving unit 601 through a predetermined synthesis equation using the color gain determined by the color gain determination unit 609.

The output unit 613 outputs the value synthesized by the synthesis operation unit 611 as an output value of the image sensor.

7 is a flowchart illustrating a procedure of synthesizing image signals having different exposure times according to an exemplary embodiment of the present invention.

Referring to FIG. 7, first, the image signal receiving unit has a relatively short exposure time compared to two image signals having different exposure times from the pixel portion, that is, a high exposure image signal R, G, and B and the high exposure image signal. The exposure image signals r, g and b are received (step 701). The minimum value determination unit then determines the minimum value M in the high exposure image signals R, G and B and the minimum value m in the low exposure image signals r, g and b (step 703).

The color gain determining unit calculates the color gains (g _R , g _G , g _B ) of the high-exposure image signal and the color gains (g _r , g _g , g _b ) of the low-exposure image signal using the minimum values M and m. (Step 705). At this time, the color gains (g _R , g _G , g _B ) of the high-exposure image signal are (R / M, G / M, B / M), and the color gains (g _r , g _g , g _b ) of the low-exposure image signal. ) Is (r / m, g / m, b / m).

Thereafter, the color gain determiner determines the color gains C _R , C _G , and C _B based on the color gain of the high exposure image signal and the color gain of the low exposure image signal (step 707). In this case, the method of determining the color gain will be described in detail with reference to FIG. 8.

Thereafter, the synthesis calculating unit calculates a value obtained by combining the high exposure image signal and the low exposure image signal using the color gain and the low exposure image signal determined in step 707 (step 709). In this case, the synthetic formula is as follows.

R '= S (M * C _R + r)

G '= S (M * C _G + g)

B '= S (M * C _B + b)

In the above formula, R ', G', and B 'are pixel values of the synthesized image signal, and S is a gain used in the conventional image synthesizing method, and has a value of 0 to 1 depending on the characteristics of the image sensor and is fixed. The value may or may not be.

Thereafter, the output unit outputs the image signal value synthesized by the synthesis calculator as an output value of the image sensor (step 711).

8 is a graph of color gain versus image signal for obtaining color gain in accordance with one preferred embodiment of the present invention.

Referring to Fig. 8, the horizontal axis represents the sum of the low exposure image signal and the high exposure image signal, and the vertical axis represents the color gain. In this case, the graph shows only the R value among the pixel values R, G, and B of the color image, and the color gain values for the G and B also show the same type of graph.

In other words, the graph shows a curve for calculating color gain C _R values for red (R) using R and r of two image signals. In the graph, r1 and r2 are boundary values for dividing the area of the graph. In the graphs, g _R and g _r denote color gain of the high exposure image signal and color gain of the low exposure image signal, respectively. The g _R is defined as a value obtained by dividing the red value R by the minimum value M among the pixel values of the high exposure image signal, and the g _r is defined as a value obtained by dividing the red value r among the pixel values of the low exposure image signal by the minimum value m. In the graph, g _r is shown to be greater than g _R , but g _r may be larger or smaller than g _R.

r1 and r2 divide a section into a region (region A) that depends on the high exposure color, a region (C region) that depends on the low exposure color, and an intermediate region (region B). In the case of an 8-bit system, since the pixel values have a value between 0 and 255, r1 may be set to a value of about 250. If R + r is about 250, it means that R is close to saturation but not yet saturated. In this case, since the r value is insufficient, if R + r is smaller than r1, that is, located in the A region, the color of the high-exposure image has high reliability. A value of 300 to 350 may be used for r2. When R + r is larger than r2, i.e., located in the C region, it means that R of the high-exposure picture signal has reached saturation, and that r value of the low-exposure picture signal has reached a certain degree of reliability. That is, in the C region, it is preferable to determine the color of the pixel by the r value. The middle region between R1 and R2 may be a straight line or a curve as a region belonging to the transition period. In the above example, it is represented by a straight line.

In summary, the A area is an area that depends on the high-exposure picture signal, and the color gain is g _R , and the C area is an area that depends on the low-exposure picture signal, and the color gain is g _r . The B region is an intermediate region, and the color gain C _R can be obtained by the following equation.

C _R = g _R + ((g _r -g _R ) / (r2-r1)) (R + r-r1)

9 is an exemplary view of a photograph output by a method of synthesizing two image signals having different exposure times according to an exemplary embodiment of the present invention.

Referring to FIG. 9, the synthesized picture 901 is a picture synthesized according to the synthesis equation described with reference to FIG. 7, and it can be seen that the high-exposure part is well restored to the saturated part. In the conventional synthesis method, the synthesized picture 501 of FIG. 5 restores the green part that is not well represented by high exposure to gray, but the synthesized picture 901 according to the present invention uses the original picture. The saturated green part is well restored.

10 is a configuration diagram of an image sensor to which a method of synthesizing image signals having different exposure times is applied according to an exemplary embodiment of the present invention.

Referring to FIG. 10, a CMOS image sensor to which a color image synthesis method according to the present invention is applied may include a pixel unit 1001, a first readout unit 1010, a second readout unit 1020, and data synthesis. The unit 600 is included.

The pixel unit 1001 has the same structure as the pixel unit of a conventional CMOS image sensor and transfers two image signals obtained by varying exposure time to an image of a subject to the readout units 1010 and 1020. Perform.

The lead-out units 1010 and 1020 are divided into a first lead-out unit 1010 and a second lead-out unit 1020, and each of the lead-out units 1010 and 1020 is a switch for reset data 1011 and 1021. ), Switches 1012 and 1022 for pixel data, capacitors 1013 and 1023 for storing reset data, capacitors 1014 and 1024 for storing pixel data, differential amplifiers 1015 and 1025, and automatic gain controller ( Auto Gain Controller (AGC) and Analog-to-Digital Converter (ADC).

When the reset data switches 1011 and 1021 are turned on in the reset state, voltage values are stored in the capacitors 1013 and 1023 in the reset state. In the state where the transfer transistor Tx is on, the pixel data switches 1012 and 1022 are turned on and the voltage values in the state are stored in the capacitors 1014 and 1024. The difference between the two values stored in the reset data capacitors 1013 and 1023 and the pixel data capacitors 1014 and 1024 is amplified by the differential amplifiers 1015 and 1025. Amplified by the Automatic Gain Controller (1016, 1026), and then converted into a digital signal through an analog-to-digital converter (ADC; 1017, 1027) and transmitted to the data synthesizing unit 600.

The conventional image sensor has one lead out portion, but the image sensor according to the present invention has two lead out portions. The two lead-out units 1010 and 1020 share a data line out of the pixel unit 1001, and alternately read two image signals having different exposure times out of the pixel unit 1001. That is, the first readout unit 1010 reads the INT1 data having the integration time, that is, the exposure time INT1, and transmits the INT1 data to the data synthesis unit 600, and the second readout unit 1020 performs the integration time. The INT2 data of the INT2 is read and transferred to the data synthesizing unit 600. In this case, INT1 refers to an exposure time operating in a conventional image sensor, and INT2 has a value smaller than INT1 and preferably 20 times smaller than INT1.

The data synthesizing unit 600 combines two data received from the readout units 1010 and 1020 and outputs the final output value of the image sensor. When double sampling is performed with two readout units as described above, when the INT1 data and the INT2 data are read out and transferred to the data synthesizing unit 600, there is a time difference between the two data. Therefore, the INT2 data read out first is temporarily stored in the memory, and the instantaneous data synthesis unit 600 combines the two data according to the above-described synthesis equation when the INT2 data is read out.

The present invention is not limited to the above embodiments, and many variations are possible by those skilled in the art within the spirit of the present invention.

According to the present invention, two color images having different exposure times may be synthesized to produce a composite image having a wide dynamic range without distorting colors.

Claims (8)

In a method of synthesizing image signals having different exposure times, Determining a minimum value M from pixel values R, G, and B of the first image signal having a pixel value of (R, G, and B); Determining a color gain (g _R , g _G , g _B ) of the first image signal using the M; Determining a minimum value m from pixel values (r, g, b) of the second image signal having a pixel value of (r, g, b), wherein an exposure time of the second image signal is determined by Low exposure image signal shorter than exposure time; Determining a color gain (g _r , g _g , g _b ) of the second image signal using the m; Obtaining (R + r, G + g, B + b) by adding the pixel values (r, g, b) of the second image signal to the pixel values (R, G, B) of the first image signal step; The obtained (R + r, G +) based on the color gains (g _R , g _G , g _B ) of the first image signal and the color gains (g _r , g _g , g _b ) of the second image signal. g, B + b) to determine the color gain (C _R , C _G , C _B ); And Calculating a composite value of the first image signal and the second image signal based on the color gains C _R , C _G , and C _B And synthesizing image signals having different exposure times. The method of claim 1, The color gain (g _R , g _G , g _B ) of the first image signal is (R / M, G / M, B / M) And synthesizing image signals having different exposure times. The method of claim 1, The color gain (g _r g _g g _b ) of the second image signal is (r / m, g / m, b / m) And synthesizing image signals having different exposure times. The method of claim 1, The color gain C _R is g _R when R + r is less than a predetermined _{r 1} , g _r when R + r is greater than a predetermined _{r 2} , and g _R + (when R + r is greater than _{r 1} and less than _{r 2} . (g _r -g _R ) / (r2-r1)) (R + r-r1) And synthesizing image signals having different exposure times. The method of claim 4, wherein The r1 is 250 and the r2 is from 300 to 350 And synthesizing image signals having different exposure times. The method of claim 1, The combined value of the two picture signals is (S (M * C _R + r), S (M * C _G + g), S (M * C _B + b))-where S means gain - And synthesizing image signals having different exposure times. The method of claim 6, S is greater than 0 and less than 1 And synthesizing image signals having different exposure times. An apparatus for synthesizing image signals having different exposure times, The minimum value M is determined from the pixel values R, G and B of the first image signal having the pixel value of (R, G, B), and the pixel value of the second image signal having the pixel value of (r, g, b). a minimum value determining unit that determines a minimum value m at (r, g, b), wherein the exposure time of the second image signal is a low exposure image signal shorter than the exposure time of the first image signal; Color gain (g _R , g _G , g _B ) of the first image signal is determined using M, and color gain (g _r , g _g , g _b ) of the second image signal is determined using m. A color gain determination unit to determine a; Obtaining (R + r, G + g, B + b) by adding the pixel values (r, g, b) of the second image signal to the pixel values (R, G, B) of the first image signal An addition operation unit; Based on the color gains (g _R , g _G , g _B ) of the first image signal and the color gains (g _r , g _g , g _b ) of the second image signal, (R + r, G + g, A color gain determiner that determines the color gains C _R , C _G , C _B according to the size of B + b); And A synthesis calculation unit calculating a composite value of the first image signal and the second image signal based on the color gains C _R , C _G , and C _B Apparatus for synthesizing image signals having different exposure times, including.

Images