KR20010013304A - Extended dynamic range imaging system and method - Google Patents

Abstract

In one embodiment, an imager (201) has an array of photodetectors, each of which accumulates a charge during an integration period as a result of light detected during the integration period, the array having a charge capacity that increases during the integration period. A charge capacity controller (202) coupled to the imager (201) adjusts how the imager (201) increases the array capacity charge based on the brightness distribution detected by the imager (201) during at least one previous integration period.

Description

System and method for extending the dynamic range of an imaging system {EXTENDED DYNAMIC RANGE IMAGING SYSTEM AND METHOD}

Various types of imagers or image sensors are in use today, including charge coupled device (CCD) image sensors and complementary metal oxide semiconductor CMOS image sensors. These devices are typically integrated into CCD and CMOS imaging systems, respectively. Such a system comprises a pixel array, each pixel array comprising a light sensing sensor element such as a CCD, or in the case of a CMOS image sensor a virtual gate buried n-channel photodetector, an N + -p-substrate photodiode or light A gate detector. Such photosensitive sensor elements will be referred to herein as photodetectors.

In such a device, the photodetector accumulates charges and voltages during the light integration period depending on the light intensity near the relevant sensing area of the photodetector. As the charge accumulates, the photodetector begins to charge. The charge stored in the photodetector is sometimes referred to as being stored in the "charge well" of the CCD photodetector. If the photodetector is fully charged by charge, excess charge is blocked to partially prevent blooming. Blooming is a phenomenon in which excess charge above pixel saturation diffuses toward adjacent pixels, causing blue rings and associated image artifacts. However, if the photodetector is fully charged before the end of the integration period and before any additional photons hit the photodetector, additional charge may not accumulate. Thus, for example, if very bright light is supplied to the photodetector, this causes the photodetector to be fully charged before the end of the integration period so that the information is saturated and lost.

US Patent No. 3,953,733 to Levine (“Levine”), issued April 27, 1976, discloses a method of operating a CCD imager to prevent such problems, which is incorporated herein by reference. . The voltage supplied to the electrodes of the CCD causes a depletion region to form below the electrode, which forms a "potential well" or charge well of a given maximum charge capacity. The largest electrode voltage causes a correspondingly large charge capacity well to form. The voltage that controls the maximum charge capacity of the photodetector, such as the CCD electrode voltage, is referred to as the charge capacity control voltage, and the maximum charge that can accumulate in the photodetector will be referred to as the charge capacity of the photodetector. The charge capacitance control voltage is sometimes referred to as the blooming barrier voltage. This is because the charge capacitance control voltage operates as a blooming drain that removes charge from the pixel photodiode to prevent charge from diffusing to adjacent pixels during light overload.

Typically, the supplied charge capacitance control voltage is constant throughout the integration period such that a given charge capacity is maintained throughout the integration period for each pixel of the imager array. In Levine's patent, the charge capacitance control voltage is changed during the integration period to increase the optical dynamic range of the CCD imager. For example, in one embodiment, the Levine patent discloses a method of increasing the charge capacity control voltage (charge capacity) in a non-linear fashion by increasing the charge capacity control voltage in discrete steps near the end of the integration period. The Levine patent uses multiple discrete steps to execute a continuously increasing charge capacitance control voltage, or by increasing the charge capacitance control voltage waveform and increasing the slope of the waveform to extend the operating range of the imaging system. Another method of increasing the charge capacity control voltage and charge capacity near the end of the integration period is disclosed.

However, although this method is used to extend the dynamic range of a given imager, the extended dynamic range cannot be used for a given frame. For example, a given scene is quite dark, which wastes dynamic range while losing contrast ratio and scene information content. This occurs for example in the case of surveillance imaging systems.

The present invention relates to an imaging system, in particular an imaging system that increases the charge capacity of a photodetector at the end of an integration period to extend the dynamic range of the imaging system.

1A-C are waveform diagrams illustrating a method of extending the dynamic range of an imager in accordance with an embodiment of the invention.

2 is a block diagram of a video processing system that dynamically determines a charge capacity control voltage function used to extend the dynamic range of the system in accordance with the present invention.

3 is a typical histogram illustrating a method of operation of the video processing system of FIG.

The present invention relates to an imaging system and method. In one embodiment of the invention, the imager has an array of photodetectors, each array accumulating charge during integration as a result of light detected during integration and having increasing charge capacity during integration. The charge capacity controller connected to the imager adjusts the manner in which the imager increases the charge capacity of the array based on the intensity distribution detected by the imager during at least one previous integration period.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

1A-C show wave diagrams describing a method for extending the dynamic range of an imager in accordance with one embodiment of the present invention. Each of FIGS. 1A-C shows three light levels in the graph of relative signal charge over time, where I3 is the strongest light level and I2 is the light level of positive intensity. The relative signal charge is the charge accumulated in the charge accumulation region or well of the photodiode in response to a given constant intensity. The time is shown as t = 0 to t = 100% of the integration period, which is typically 1/30 or 1/67 seconds. The relative signal charge is shown in the range 0.0 to 1.0, where 1.0 represents the maximum charge capacity of the photodetector indicating the saturation of the photodetector.

Figure 1A shows time versus relative signal charge during integration periods for light levels I1, I2 and I3, where a fixed charge capacitance control voltage is supplied for the entire integration period, as does a conventional photodetector. As described, both high light levels I2 and I3 saturate before the integration period is completed. This is because each causes the charge capacity of the photodetector to be exceeded. Thus, both light levels I2 and I3 occur in the same signal read from the photodetector, although they are different. Thus, since both light levels I2 and I3 are saturated before the end of the integration period, it is possible for the generated zero to display any light level in detail above these light levels. The array of photodetectors configured to operate at any constant charge capacitance control voltage during the entire integration period does not have sufficient dynamic range for the light levels I1, I2 and I3 to be mode measured.

This may be the case for saturation, such as for surveillance purposes where the dynamic range in the scene may saturate or bloom beyond the dynamic range, resulting in loss of information. If the imager lens is adjusted by adjusting the mechanical iris, this effectively lowers the slope of all three light levels I1, I2 and I3 so that they do not saturate and may also cause loss of information. The signal-to-noise ratio at is reduced.

Thus, as previously described in the Levine patent, it is useful to extend the dynamic range of an imaging system using a charge capacity control voltage function that measures the charge capacity control voltage in any manner during the integration period. In one embodiment of the invention, the charge capacitance control voltage function is such that the first charge capacitance control voltage V1 is supplied to the photodetector during the first portion of the integration period and the high charge capacitance control voltage V2 is integrated after the first portion of the integration period. It is a two-step nonlinear voltage function supplied to the photodetector up to the end of the period. In one embodiment of the invention, the voltage is the maximum charge capacity control voltage, that is, voltage V2 causes the photodetector to have the maximum charge capacity, and voltage V1 is part of the voltage V2 such that the photodetector is the corresponding portion of the maximum charge capacity. Have a phosphorus charge capacity.

Thus, the charge capacitance control voltage V1 is supplied during the first period which is a specific part of the integration period, and the charge capacity control voltage V2 is supplied during the second period equal to the remaining period of the integration period after the first period. By increasing the length of the first period in which the voltage V1 is supplied and decreasing the second time in which V2 is supplied, a high overall dynamic range can be obtained.

In the present invention, the charge capacity control voltage function is changed to provide sufficient dynamic range to prevent saturation while maximizing the contrast ratio and information content of the scene. In the two step nonlinear voltage function described above, the charge capacity control voltage function is changed by changing the way in which the integration period is divided between the first and second periods. A wider dynamic range can be obtained by lengthening the first period and shortening the second period while taking some loss of contrast information for high intensity light levels. When the maximum charge capacity control voltage is supplied for the entire integration period, it is saturated, and because they do not add additional charge during the second period, the unsaturated strong light level causes extra charge in the photodetector for a period shorter than the low intensity light level. It may be referred to as "compression" because it adds.

Thus, the charge capacitance control voltage V1 is supplied at times t = 0 to t1, and the charge capacitance control voltage V2 is supplied for the time Δt1, where Δt1 is the period between the time t1 and the end of the integration period. By increasing the period t1 in which v1 is supplied and decreasing the time Δt1 in which V2 is supplied, a higher overall dynamic range can be obtained. Thus, in one embodiment of the present invention, for a given first charge capacity control voltage V1 and a second charge capacity control voltage V2, the charge capacity control voltage function is changed by changing when time t1 occurs.

This charge capacity control voltage function is described in FIGS. 1B and 1C, respectively, with different times when voltage V1 changes to voltage V2. Referring to FIG. 1B, the charge capacity control voltage V1 is set to 80% of the maximum charge capacity control voltage V2. The voltage V1 is increased to the voltage V2 at time t1. Thus, by time t1, there is a saturation point because of the charge capacity which is only 80% of the maximum charge capacity present when the largest voltage V2 is supplied after time t1. This allows for a contrast difference between light levels I2 and I3 which is not possible using the embodiment shown in FIG. 1A where the contrast charge capacity control voltage is used throughout the entire integration period. This is because even though light level I3 continues to saturate, there is no light level I2 and the strong level I3 generates more photodetector charge than the light level I2 that can be used to divide the light level in the image.

However, since the light level I3 is saturated, the first period t1 does not allow the light levels of the intensity around the light level I3 to be distinguished from each other. In Fig. 1C, voltage V1 is switched to voltage V2 at time t2 later than time t1. In this embodiment, the light levels I2 and I3 are both not saturated so that all light levels up to I3 are distinguished from each other.

In the present invention, before each frame of video data is captured at the photodetector's array, the charge capacity control voltage function provides a saturation for the brightest portion of the captured scene or image while maximizing the contrast ratio and information content of the scene. The membrane is optimally adjusted to provide sufficient dynamic range. This is done by using dynamic feedback control from the previous image to set the first period so that the brightest object in the scene reaches the saturation capacity of the imager. Information from the previous frame or image is used based on the assumption that the brightness value distribution of the previous frame is a reliable predictor of the brightness value distribution of the next frame.

Changing the charge capacity control voltage effectively changes the pixel charge capacity, allowing a portion of the maximum charge capacity to be integrated for most of the frame time, thereby increasing the charge capacity to the maximum charge capacity near the end of the integration period. Can be extended. This can generate a detailed image in the city in the bright and dark areas of the scene. Dynamic regulation of the charge capacitance control voltage effectively generates a nonlinear photodetection response during the integration period.

2 is a block diagram of a video processing system 200 that dynamically determines a charge capacity control voltage function used to extend the dynamic range of the system in accordance with one embodiment of the present invention. System 200 includes an imager 201 that includes an array of pixels each having a photodetector. The system 200 further includes a charge capacity controller 202 connected to the imager 201. The charge capacity controller 202 includes a histogram 204, a feedback function generator 206, a contrast corrector 205, a histogram equalization / projection block 207, and a MUX 210, 211. System 200 includes an output monitor 220 connected to the output of MUX 211 of charge capacitance controller 202. The pixel array of imager 201 may be, for example, a CCD or CMOS array, and in one embodiment a 12-bit output pixel value 4096 for each pixel corresponding to the amount of charge accumulated by each pixel during the integration period. Level). The output monitor 220 needs 8 bit input pixel information. The histogram 204 and contrast corrector 205 are connected or connectable to the imager 201 via a line 212, and the feedback function generator 206 is connected to the imager 201 via a line 213. Connected or connectable.

The output pixel information for the captured frame of the pixel is supplied by the imager 201 to the histogram 204 and the contrast corrector 205 via the bus 212. The tab line 216 further provides output pixel information as an input of the MUXs 210 and 211. The output of histogram 204 is supplied via line 214 to the input of feedback function generator 206 and to histogram equalization / projection block 207. The feedback block provides function information to imager 201 via line 213 and provides inverse function information to the input of contrast corrector 205. The output of contrast corrector 205 is fed to histogram equalization / projection block 207 and MUX 211 via MUX 210 and line 217. The output of histogram equalization / projection block 207 is supplied to the input of monitor 220 via line 218 and MUX 211.

As will be apparent to those skilled in the art, some or all of the functions of the charge capacity controller 202 (ie, histogram 204, feedback function generator 206, contrast corrector 205, histogram equalization / projection block 207, and MUXs 210 and 211 may be executed in real time by dedicated digital signal processing hardware, by a specific application integrated circuit (ASIC), or by a suitably programmed general purpose computer processor.

The system 200 operates as follows to dynamically adjust the charge capacitance control voltage function to optimize the performance of the imager 201. First, the scene (frame 0) is captured by imager 201 and supplied to histogram 204. As will be apparent to those skilled in the art, the histogram aligns the received pixels to the gray level of the pixel, i. Thus, histogram 204 provides a histogram that includes information indicative of the global intensity distribution of a given frame. From this distribution, the charge capacity control voltage function can be adjusted to expand or compress the next image to the full 4096 level of the gray scale resolution of the imager 201.

The feedback function generator 206 generates function information that sets the first period for the imager 201 to capture the next frame so that light at the level of the brightest object in frame 0 is saturated in the imager. Try to reach the point. Only when the light reaches a given level, the saturation point of the imager means that at the end of the integration period the photodetector causes this level of light to reach almost the full charge level. Therefore, the function information generated by the function generator 206 for the two step nonlinear voltage function is a number representing the time during the integration period in which the first voltage V1 is changed to the second voltage V2.

For example, the histogram 300 may be used by the feedback function generator 206 to determine what the intensity or contrast value of the brightest portion of the image considered as the object (cluster of pixels). This analysis is done to avoid being determined based on defective (always white) pixels or slightly bright pixels rather than simply using the intensity levels of the brightest pixels in the frame. Thus, for example, such a determination is made by feedback function generator 206 to determine if the charge capacity control voltage function used for the next frame is based on the intensity level or gray scale of the brightest 100 pixels. Thus, for example, if the brightest object (not simply a separate pixel) of frame 0 has a roughness of approximately I2, the feedback function generator 206 is slightly later than t1 as shown in FIG. 1B. Function information for setting the first period is generated. However, if the brightest object of frame 0 has an intensity of approximately I3, the feedback function generator 206 generates function information that sets the first period to approximately t2 as shown in FIG. 1C.

Thus, by using the dynamic adjustment of the charge capacity control voltage function of the present invention, the next frame to be captured is efficiently compressed within the dynamic range of the imager, but to the minimum content necessary to maximize the contrast ratio and scene information content.

In one embodiment of the invention, the imager 201 controls the charge capacitance control voltage during the integration period in accordance with the function information provided by the function generator 206. In another embodiment of the present invention, function generator 206 provides a charge capacitance control voltage directly to imager 201 through line 213 during the integration period in accordance with the generated function. In yet another embodiment of the present invention, the feedback function generator 206 is configured so that only light at the level of the brightest object in frame 0 does not saturate the imager, but the light at this level is roughly or accurately imaged. Generate function information in order not to saturate the group.

System 200 essentially further provides signal processing for captured video frames. First, the inverse of this function is generated by the feedback function generator 206 and supplied to the contrast corrector 205 because the dynamic adjustment of the charge capacitance control voltage efficiently generates a nonlinear photodetection response during the integration period. This allows a non-linear manner in which charge is accumulated to suitably map to the corresponding light intensity level causing the charge directed by the pixel. This helps to keep the actual color and contrast representations that the pixel color is essential for the color image represented by the next image sensor element. Contrast correction is important when several sensor elements are combined to make color pixels. If one sensor element containing one color component (e.g. red) has a low brightness, the red is not compressed while the other element (e.g. blue) is compressed without correction for unbalanced compression. However, the detected color is not a real representation of the scene. By executing the inverse function for each pixel, the red pixel in this case is kept constant and the blue pixel is restored to recover the contrast between the luminance of the sensor elements.

This contrast correction is necessary for the contrast to be maintained and the image will extend from the dynamic range of the image pixels (eg 12 bits) to the high optical dynamic range, eg 16 bits. Thus, the output of contrast corrector 105 may have a 16-bit dynamic range, for example, although each pixel of imager 201 has only 12 bits of resolution.

In one embodiment of the invention, the histogram equalization / projection block 207 is an input image (e.g., image via line 216 and MUX 210) received on an output monitor 220 with 8-bit resolution. Is used to optimally display a wide dynamic range of a 12-bit image or contrast corrector 205 and 16-bit contrast corrected image through MUX 210). This is used to efficiently display the maximum amount of information on the output monitor 220 by optimizing the information content of each link in the video chain.

Histogram equalization and histogram projection are conventional methods of enhancing contrast using so-called "dot processing" techniques. Using this technique, images with uniform density of gray levels are generated to maximize the contrast seen to provide the most important information in the viewer. This technology maps high resolution data onto smaller resolution monitors without clipping for optimal display on limited resolution monitors. Thus, histogram equalization of histogram equalization / projection block 207 redistributes the gray scale so that the details of the bright and dark areas of the scene are appropriately enhanced. To perform this process, histogram equalization / projection block 207 needs only the histogram already available from the feedback function generator 206.

Line 216 is bypassed by setting multiplexer 211 to send uncorrected video via line 216 to output monitor 220 when contrast correction of contrast corrector 205 is needed. Line 217 causes the histogram equalization / projection of histogram equalization / projection block 207 to be bypassed by setting multiplexer 211 to send uncorrected video to output monitor 220 via line 217. Thus, by appropriately switching the MUXs 210 and 211, one of the four video signals is either displayed on the output monitor 220, (1) unprocessed (i.e., the contrast corrector 205 and the histogram equalization / projection block). 207), (2) contrast enhancement (i.e., skip histogram equalization / projection block 207), (3) histogram enhancement (skip histogram equalization / projection block 207), or (4) A) Histogram and contrast are enhanced.

The dynamic feedback of the present invention can be used with a charge capacity control voltage function other than the two-voltage nonlinear function shown in FIGS. 1B and 1C. For example, multiple discrete steps and voltages or continuously increasing charge capacity control voltages can be used as linearly increasing the charge capacity control voltage waveform. In these cases, global intensity distribution information provided by histogram 204 is used to appropriately adjust these functions. Thus, for example, if a continuously increasing charge capacitance control voltage is used, the contrast value of the brightest object in the previous frame is used to adjust the shape of the charge capacitance control voltage function so that the light level of this contrast value is saturated. do. In this case, the feedback function generator 206 generates function information instructing the imager 201 how to change the charge capacitance control voltage function.

Histograms from immediately preceding frames and other previous frames may be used. For example, a histogram of the running average of the last five frames can be used.

Those skilled in the art can modify the present invention without departing from the scope of the present invention. Accordingly, the invention is limited only by the spirit and scope of the claims.

Claims (13)

(a) an imager having a photodetector array each accumulating charge during the integration period as a result of light detected during the integration period, the photodetector array having an increased charge capacity during the integration period; (b) connected to the imager, the imager comprising a charge capacity controller that adjusts the manner of increasing the charge capacity of the array based on the intensity distribution detected by the imager during at least one previous integration period. Imaging system characterized by the above. 2. The array according to claim 1, wherein the imager provides a video frame during the integration period in accordance with the charge accumulated by the respective photodetectors, and the array during the integration period in accordance with function and function information specifying how to change the function. To increase the charge capacity of; And the charge capacitance controller comprises a function generator for generating the function information in accordance with a distribution of contrast values of at least one previous video frame provided by the imager. 3. The charge capacity controller of claim 2, wherein the charge capacity controller comprises a histogram connected to the imager and the function generator, the histogram receiving a video frame provided by the imager and providing the contrast value to the function generator. An imaging system comprising a histogram having a distribution. 4. The apparatus of claim 3, wherein the charge capacity controller comprises a histogram equalization / projection block connected to the histogram or imager, the histogram equalization / projection block to optimally display information content of the video frame on a monitor. And a histogram for the current video frame. 3. The charge capacitor of claim 2 further comprising a contrast corrector, The function generator generates inverse function information; The contrast corrector receives the video frame provided by the imager, receives the inverse information provided by the function generator, and eliminates the phenomenon of the video frame caused by the charge capacity of the array of photodetectors increased during the integration period. An inverse function information in order to perform 3. The imager of claim 2, wherein the imager supplies a first charge capacitance control voltage to the array of photodetectors during the first portion of the integration period and applies a second charge capacitance control voltage to the array of photodetectors after the first portion. Supplying increases charge capacity for the array of photodetectors during the integration period; And the length of the first portion is determined according to the function information provided by the function generator, and wherein the second charge capacitance control voltage is greater than the first charge capacitance control voltage. 3. The apparatus of claim 2, wherein the function generator generates the function information according to a distribution of contrast values of the most recent previous video frame provided by the imager; The function information is generated by the function generator so that the light that has the intensity level of the brightest object in the most recent previous video frame and maps onto the photodetector of the imager during the integration period is such that the charge And to prevent accumulation above the charge capacity of the photodetector at the end of the period. 8. The imaging system of claim 7, wherein the function information is generated by the function generator such that the light causes charge to reach but exceed the charge capacity of the photodetector at the end of the integration period. 2. The imager of claim 1, wherein the imager is a complementary metal oxide semiconductor imager having an array of virtual gate buried n-channel photodetectors, each photodetector proportional to the magnitude of the charge capacitance control voltage supplied to the photodetector. Imaging system having a charge capacity. 10. The method of claim 1, wherein the imager comprises an array charge coupling device (CCD) photodetector, wherein each CCD photodetector has a charge capacity proportional to the magnitude of the charge capacity control voltage supplied to the CCD photodetector. Imaging system characterized by the above. A charge capacity controller connected to the imager having photodetector arrays each accumulating charge during the integration period as a result of light detected during the integration period; The photodetector array has an increasing charge capacity during the integration period; And the charge capacity controller adjusts the manner in which the imager increases the charge capacity of the array based on the intensity distribution detected by the imager during at least one previous integration period. (a) increasing the charge capacity of the photodetector array of the imager during the integration period in which each photodetector accumulates charge as a result of light detected during the integration period; (b) adjusting the manner in which the charge capacity of the array is increased based on the intensity distribution detected by the imager during at least one previous integration period. (a) means for increasing the charge capacity of the photodetector array of the imager during the integration period in which each photodetector accumulates charge as a result of light detected during the integration period; and (b) means for adjusting the manner in which the charge capacity of the array is increased based on the intensity distribution detected by the imager during at least one previous integration period.