WO2013127338A1

WO2013127338A1 - Imaging method and imaging device

Info

Publication number: WO2013127338A1
Application number: PCT/CN2013/071940
Authority: WO
Inventors: 徐辰
Original assignee: Xu Chen
Priority date: 2012-02-27
Filing date: 2013-02-27
Publication date: 2013-09-06
Also published as: CN103297701B; CN103297701A

Abstract

The present invention relates to an imaging method and an imaging device. The imaging method in the present invention exposes the entire pixel array at different times, or divides the pixel array into a plurality of parts to expose each part at a different time, then combines the images obtained at different exposure times, thus improving the dynamic optical range of an imaging device. Correspondingly proposed are an imaging device and a method for HDR image combination.

Description

Imaging method and imaging device

The present invention relates to the field of imaging, and in particular to an imaging method and an imaging apparatus.

Background technique

The requirements for image quality have been constantly increasing. As the pixel integration of imaging devices becomes higher and higher, the resolution of images is no longer the most important issue in the imaging field, and the performance of images in other aspects has received more attention. In particular, obtaining high-quality images without relying on complicated hardware is the current direction of research and development in the imaging field. For example, a high quality photo is obtained on a portable imaging device such as a card camera.

Imaging devices typically have a pixel array. Each of the pixels in the pixel array includes a photosensitive device such as a photodiode, an optical switch, or the like. Each photosensor has a different ability to receive light. This difference in capabilities is reflected on the imaging device such that the imaging device has a different range of light dynamics, i.e., the range in which the imaging device can receive light. When the optical dynamic range of the imaging device is smaller than the change in external light intensity, the external scene cannot be completely reflected in the acquired image. There has been a desire in the art to have an easy way to solve this problem. Summary of the invention

In view of the problems in the prior art, according to an aspect of the present invention, an imaging method is provided, including: exposing pixels in a pixel array to obtain a first image in a first time; Exposing the pixels in the pixel array to obtain a second image, wherein the first time is different from the second time; and combining the first image and the second image.

According to another aspect of the present invention, an imaging method is provided, including: exposing pixels in a first pixel group in a pixel array to obtain a first image in a first time; Exposing pixels in the second pixel group in the pixel array to obtain a second image, wherein the first time is different from the second time; simultaneously reading the first image and the second image; And combining the first image and the second image.

According to another aspect of the present invention, an imaging apparatus is provided, comprising: a pixel array including a plurality of pixels arranged in rows and columns; a control circuit controlling the pixel array; wherein the pixel array includes a first pixel group a group, which is exposed in a first time to obtain a first image; the pixel array includes a second group of pixels that are exposed in a second time to obtain a second image, wherein the first time is different from the second a time; wherein the control circuit further reads the first image and the second image simultaneously; and an image processor that combines the first image and the second image.

DRAWINGS

Figure 1 is a schematic view showing the structure of an image forming apparatus;

Figure 2 is a schematic diagram showing a representative pixel structure;

3 is a flow chart of an imaging method in accordance with an embodiment of the present invention;

4 is a flow chart of an imaging method in accordance with another embodiment of the present invention;

Figure 5 is a schematic illustration of a pixel array in accordance with one embodiment of the present invention;

6 is a timing diagram of an image taken by a pixel array in accordance with an embodiment of the present invention;

Figure 7 is a schematic illustration of a pixel array in accordance with another embodiment of the present invention;

8 is a timing diagram of an image taken by a pixel array in accordance with another embodiment of the present invention;

Figure 9 is a schematic illustration of a pixel array in accordance with another embodiment of the present invention;

FIG. 10 is a timing diagram of an image taken by a pixel array according to another embodiment of the present invention; FIG.

11 is a schematic diagram of an HDR method of combining two-exposure images, in accordance with one embodiment of the present invention;

Figure 12 is a schematic illustration of an HDR method of combining images of four exposures, in accordance with one embodiment of the present invention;

Figure 13 is a schematic illustration of respective optical response curves for each exposure time of T1, Τ2, Τ3, and Τ4;

Figure 14 is a schematic diagram of the induction curve and its SNR curve after the synthesis algorithm for the four exposure times is completed; Figure 15 is a schematic illustration of the effect on the SNR curve using different mask coefficients;

Figure 16 shows the effect of four exposures and two exposures on SNR;

Figure 17 is a schematic illustration of a system in accordance with one embodiment of the present invention.

detailed description

In the following detailed description, reference should be made to the accompanying drawings of the claims In the drawings, like reference characters refer to the FIGS The specific embodiments of the present application are described in sufficient detail below to enable those of ordinary skill in the art to practice the invention. It is to be understood that other embodiments may be utilized or structural, logical or electrical changes may be made to the embodiments of the application.

The term "pixel" refers to an electronic component that contains a photosensitive device or other device for converting an electromagnetic signal into an electrical signal. For purposes of illustration, Figure 1 depicts a representative imaging device that includes an array of pixels. A representative pixel is depicted in Figure 2, and all pixels in the pixel array will typically be fabricated in a similar manner.

Fig. 1 is a schematic view showing the structure of an image forming apparatus. The imaging device 100 shown in Fig. 1, such as a CMOS imaging device, includes a pixel array 110. Pixel array 110 includes a plurality of pixels arranged in rows and columns. Each row of pixels in the pixel array 110 is simultaneously turned on by the row select lines, and each column of pixels is selectively outputted by a column select line. Each pixel has a row address and a column address. The row address of the pixel corresponds to the row select line driven by the row decode and drive circuit 120, and the column address of the pixel corresponds to the row select line driven by the column decode and drive circuit 130. Control circuit 140 controls row decode and drive circuit 120 and column decode and drive circuit 130 to selectively read pixel output signals corresponding to appropriate rows and columns in the pixel array.

The pixel output signal includes a pixel reset signal V _ret and a pixel image signal V _slg . The pixel reset signal V _ret represents a signal obtained from the floating diffusion region when the floating diffusion region of the photosensitive device (such as a photodiode) is reset. The pixel image signal V _slg represents a signal obtained after the charge of the representative image acquired by the photosensitive device is transferred to the floating diffusion region. Both the pixel reset signal V _ret and the pixel image signal V _slg are read by the column sample and hold circuit 150 and subtracted by the differential amplifier 160. The V _ret - V _slg signal outputted by the differential amplifier 160 represents an image signal acquired by the photosensitive device. The image signal is converted to a digital signal by an analog to digital converter ADC 170 and then further processed by image processor 180 to output a digitized image. Image processor 180 can be either a separate processor or a central processing unit or other processor of the system.

Figure 2 is a schematic diagram showing a representative pixel structure. The pixel 200 of FIG. 2 includes a photodiode 202, a transfer transistor

204, reset transistor 206, source follower transistor 208 and row select transistor 210. Photodiode 202 is coupled to the source of transfer transistor 204. Transfer transistor 204 is controlled by signal TX. When the TX controls the transfer transistor to the "on" state, the charge accumulated in the photodiode is transferred to the storage region 21. At the same time, the photodiode 202 is reset. The gate of the source follower transistor 208 is connected to the memory region 21. The source follower transistor 208 amplifies the signal received from the memory area 21. The source of the reset transistor 206 is also connected to the memory area 21. The reset transistor 206 is controlled by a signal RST for resetting the memory area 21. The pixel 200 still further includes a row selection transistor 210. The row selection transistor 210 is controlled by the signal RowSd, and the signal amplified by the source follower transistor 208 is output to the output line V. _Ut .

Two exposures with different exposure times for the same image can increase the optical dynamic range of the imaging device. If the exposure time is long enough, the darker part of the image can be fully reflected in the final image; however, if the light intensity of the image changes beyond the dynamic range of the imaging device, the brighter part of the image is reflected to the final image. The generals are all white. That is, this portion of the light intensity change information that exceeds the imaging capability of the imaging device will be lost. If the exposure time is short enough, the strongest light intensity in the image does not exceed the sensitivity of the imaging device, and the light intensity change information in the brighter part of the image will remain; however, due to the short exposure time, lack of sufficient sampling, Information in the darker parts of the image will be lost. The method of the present invention for increasing the optical dynamic range of an imaging device using different exposure times takes into account the above two cases. For the same image, two exposures are performed with different exposure times; then, in the subsequent processing of the image, the results of the two exposures are comprehensively considered to reflect the image information obtained by the two exposures in the finally obtained image. Because the resulting image retains both the information in the brighter part of the image and the information in the darker part of the image. The image reflects a greater range of light intensity changes. Thereby, the optical dynamic range of the imaging device is improved without adding any hardware cost.

There are two ways to accomplish two exposures. The first way is to use a short time exposure for the entire pixel array, then read the entire image; then use long exposure, then read the entire image; then combine the two images to get the final image. This approach is the easiest to implement, does not require complex hardware changes, and can even be completely controlled by software. The second method is to perform a partition length exposure for the entire pixel array. For example, a part of the pixels in the pixel array start to be exposed for a long time, and after a period of time, the short-time exposure to another part of the pixels in the pixel array is directly started, and the double-exposure imaging result is read out, and the imaging results are performed twice. Combine to get the final image. The method of using short-time exposure and then long-time exposure is similar, and will not be described again. The way in which the two exposure results are not read at the same time is similar, and is not described here.

The following embodiments of the present invention are described in detail in the two modes.

3 is a flow chart of an imaging method in accordance with an embodiment of the present invention. As shown in FIG. 3, imaging method 300 captures an image using an imaging device that includes a pixel array. The imaging device has a predetermined optical dynamic range. At step 310, it is judged whether the light intensity change of the image to be taken exceeds the optical dynamic range of the imaging device, and if it is exceeded, the highlight dynamic range mode is activated, otherwise the image is taken in the normal mode. Existing imaging devices, such as digital cameras, many have a display to show the user the target to which the lens of the imaging device is pointing in real time. The real-time image can distinguish whether the image is too bright or too dark, and whether it reflects the details that you want to pay attention to, so you can directly distinguish whether the highlight dynamic range mode should be enabled. It should be noted that the display screen of the imaging device is for illustrative purposes only. The imaging device or imaging method of the present invention is not required to include a display screen.

A variety of methods can also be employed to determine whether the light intensity of the image to be ingested exceeds the optical dynamic range of the imaging device. For example, it can be judged by calculating the average brightness, contrast, or brightness or contrast of the area of interest with other areas. For example, in general, an image will have a Region of Interest (ROI). The image taken should reflect the details of the region of interest as much as possible. In the case where the details of the region of interest are best handled, it is judged whether other regions of the image are too bright or too dark, so that it is possible to determine whether the change in light intensity exceeds the optical dynamic range of the imaging device.

At step 320, it is further determined whether the image to be ingested contains a scene in motion. The main reason for making this judgment is that if the image contains a moving scene, since all the pixels in the entire pixel array are read after a short exposure, the long exposure is started, and the time between the two may cause the moving scene to be The second exposure is already at different positions in the image, resulting in overlapping images around the moving scene in the final combined image, forming a "ghost". Since "ghosting" is difficult to eliminate in subsequent image combination and processing, it is necessary to judge in advance.

A variety of methods can be used to determine if the scene is moving and the speed of the movement. For example, the concept of a motion vector in video coding can be borrowed or the position of the same scene in an image at different times can be directly compared to make a judgment. Unlike video encoding, it is not necessary to compare successive frames of different frames or adjacent images to determine if the scene is moving. The position of the same scene in the two frames of images having a predetermined time interval can be compared to determine whether the image contains the moving scene and the corresponding speed. If the motion speed of the scene exceeds a predetermined threshold, it can be concluded that a "ghost" may appear in the image, prompting the user not to adopt such a highlight dynamic range mode or to prevent the user from operating the imaging device to take an image. Another way to interpret whether or not "ghosting" occurs is to determine whether a "ghost" has occurred by comparing the position of the feature area of the image generated after two exposures. The feature area is created by a manual designation or an automatic method such as smile recognition. It is compared whether the position of the feature area in the generated image after the double exposure is changed to determine whether the moving scene and the corresponding moving speed are included. If the image to be ingested does include a moving scene and the speed of motion is sufficient to produce a "ghost", then an "anti-ghost" highlight dynamic mode mode that does not produce "ghosting" must be used. This will be explained in detail in the subsequent embodiments.

At step 330, the entire pixel array of the imaging device is exposed for a first time. According to an embodiment of the invention, the exposure time of the first time is relatively short, for example: 10 milliseconds. At step 340, the entire pixel array is read line by line, resulting in a first image that is exposed during the first time. Since this process is not substantially different from the general image capturing process, it will not be described in detail herein. At step 350, the entire pixel array of the imaging device is exposed for a second time. According to an embodiment of the invention, the exposure time of the second time is relatively long, for example: 40 milliseconds. At step 360, the entire pixel array is read line by line, resulting in a second image that is exposed during the second time. Since this process is not substantially different from the general image capturing process, it will not be described in detail herein.

At step 370, the first image and the first image are combined to arrive at a final image. The final image includes both the information of the brighter portion of the image to be taken contained in the first image and the information of the darker portion of the image to be taken contained in the second image. Thereby, a larger optical dynamic range than the imaging device itself is obtained in the final image. In the combination of images, different methods can be used. For example, the simplest averaging of corresponding pixels is the result of that pixel in the final image. Other combinations can be used for better contrast, sharpness, or color expression.

4 is a flow chart of an imaging method in accordance with another embodiment of the present invention. As shown in FIG. 4, the imaging method 400 takes an image using an imaging device including a pixel array. The imaging device has a predetermined optical dynamic range. As described above, one of the problems faced by the embodiment shown in Fig. 3 is that "ghosting" occurs for a moving scene. The main reason for the appearance of "ghosting" is that the entire pixel array is read twice, and the time interval between two readings is equal to one frame time, for example, about 30 milliseconds. This time interval is sufficient to change the position of the moving scene in the image. The embodiment shown in Figure 4 addresses this problem by using a single read of the entire pixel array. The entire pixel array is divided into two sections, different exposure times are used for different sections, and the resulting results are combined to arrive at a final image. In the embodiment shown in Figure 4, a higher optical dynamic range is obtained at the expense of the resolution of the image.

In step 410, it is determined whether the light intensity of the image to be ingested exceeds the optical dynamic range of the imaging device, and if it is exceeded, the highlight dynamic range mode is activated, otherwise the image is taken in the normal mode. Step 410 is similar to step 310 in the embodiment of Figure 3 and will not be described in detail herein. It should be noted that whether for the embodiment of Figure 3 or Figure 4, determining whether to activate the highlight dynamic range mode is an optional step.

At step 420, the first group of pixels in the entire pixel array are exposed for a first time. The first group of pixels is a portion of the pixels in the entire pixel array. In accordance with an embodiment of the present invention, the first group of pixels are distributed as evenly as possible throughout the array of pixels to reflect the captured image as much as possible. According to an embodiment of the invention, the exposure time of the first time is relatively long, for example: 40 milliseconds. At step 430, the second group of pixels in the entire pixel array are exposed for a second time. The second group of pixels is a portion of the pixels in the entire pixel array. In accordance with an embodiment of the present invention, the second group of pixels are distributed as evenly as possible throughout the array of pixels to reflect as much of the captured image as possible. According to an embodiment of the invention, the exposure time of the second time is relatively short, for example: 10 milliseconds.

At step 440, all of the pixels in the entire pixel array are read. The images obtained by the first group of pixels and the second group of pixels are combined to arrive at a final image. The final image includes information of a darker portion of the image to be captured acquired by the first group of pixels, and information of a brighter portion of the image to be captured acquired by the second group of pixels. Thereby, a larger optical dynamic range than the imaging device itself is obtained in the final image. Moreover, the time interval between the exposure time of the first group of pixels and the second group of pixels is very short, approximately equal to the time of one line readout, for example, 10 microseconds, so no "ghosting" is produced in the final image.

In the second mode, the entire pixel array is divided into two or more sections, how to divide the pixel array and how to reduce the edge effect and reduce the signal-to-noise ratio due to the division of the pixel array are all problems to be considered.

Figure 5 is a schematic illustration of a pixel array in accordance with one embodiment of the present invention. As shown in Fig. 5, the pixel array 500 is a color pixel array, and R, G, Gb, and B represent different colors, respectively. White pixels, such as Rl, Gl, Gbl, and B1, represent pixels with an exposure time of T1, respectively; and diagonally-spaced pixels, such as R2, G2, Gb2, and B2, represent pixels with an exposure time of T2, respectively. T1 is different from Τ2. In general, a set of pixels 501, 502, 503, and 504 of different colors in the color pixel array of Figure 5 represent different color values for one pixel. Therefore, they should have the same exposure time. As can be seen from FIG. 5, the first group of pixels having different exposure times, such as R1, G1, Gbl, and B1, and the second group of pixels, such as R2, G2, Gb2, and B2, are spaced apart in two rows. In the entire pixel array, that is, the first group of pixels and the second group of pixels are alternately arranged in two rows.

Figure 6 is a timing diagram of an image taken by a pixel array, in accordance with one embodiment of the present invention. The timing diagram shown in Figure 6 can be applied to the diagram In the embodiment shown in 5. In pixel array 500, the Tx, RST, and RowSel signals are shared by the same row of pixels. Therefore, pixels of the same row accumulate charge in the same time.

For the R1/G1 line where pixels 501 and 502 are located, first a pulse is provided on the RowSel line to select the line. A pulse is provided on the RST line to reset the memory area, such as memory area 21 in FIG. Next, a pulse is provided on the SHR line to sample the reset memory region to generate a V _ret signal. A pulse signal is provided on the Tx line to transfer charge on the respective R1 and G1 pixel photosensitive devices (e.g., photodiode 202 in FIG. 2) on the R1/G1 line to their respective memory areas. A pulse signal is then provided on the SHS line to sample the charge stored on the storage areas of the respective R1 and G1 pixels on the R1/G1 line to produce the V _slg signal.

For the R2/G2 line where pixel 505 is located, similar to the R1/G1 line, first a pulse is provided on the RowSel line to select the line. A pulse is provided on the RST line to reset the memory area and generate a V _ret signal. A pulse signal is provided on the Tx line to transfer the charge to its respective memory area, and then a pulse signal is provided on the SHS line to sample the charge stored on the memory area of each pixel on the R2/G2 line to generate the V _slg signal. For the Gb2 B2 line where the pixel 506 is located, since it has the same exposure time as the R2/G2 line, the Gb2 B2 line can share the control signal of the R2/G2 line.

It can be seen that although the R1/G1 and Gbl Bl lines and the pixels on the R2/G2 and Gb2 B2 lines belong to a group of pixels having different exposure times, they are simultaneously sampled to generate an image signal.

Next, when the RST line is high, a pulse is provided on the Tx lines of the R1/G1 and Gbl Bl lines to reset the photosensitive devices of the respective pixels of the R1/G1 and Gbl Bl lines. At different times, when the RST line is high, another pulse is provided on the Tx lines of the R2/G2 and Gb2 B2 lines to reset the photosensitive devices of the respective pixels of the R2/G2 and Gb2 B2 lines. The photosensitive element begins to accumulate charge after resetting. Since the R1/G1 and Gbl Bl lines and the pixels on the R2/G2 and Gb2 B2 lines accumulate charge from different moments; and, as described above, they are almost simultaneously sampled, and therefore, the R1 belonging to the first group The /G1 and Gbl Bl rows have different charge accumulation times from the R2/G2 and Gb2 B2 rows belonging to the second group, thereby having different exposure times.

For the Gbl Bl line where pixels 503 and 504 are located, since it has the same exposure time as the R1/G1 line, the Gbl Bl line can use the same control signal as the R1/G1 line.

Figure 7 is a schematic illustration of a pixel array in accordance with another embodiment of the present invention. As shown in Fig. 7, the pixel array 700 is a color pixel array, and R, G, Gb, and B represent different colors, respectively. White pixels, such as Rl, Gl, Gbl, and B1, represent pixels with an exposure time of T1, respectively; and diagonally-spaced pixels, such as R2, G2, Gb2, and B2, represent pixels with an exposure time of T2, respectively. T1 is different from T2. In general, a set of pixels 701, 702, 703, and 704 of different colors in the color pixel array of Figure 7 represent different color values for one pixel in the final image. Therefore, they should have the same exposure time. As can be seen from Fig. 7, for the first group of pixels having different exposure times, and the second group of pixels, two rows are staggered in two directions. That is, if different color pixels belonging to the same group are taken as a whole, each group of pixels is adjacent to another group of pixels having different exposure times.

In the embodiment shown in Figure 7, since the same row of pixels needs to have different exposure times, the same row of pixels cannot share the Tx signal, but they can still share the RST and RowSel signals. Therefore, two sets of Tx signals must be provided for each row of pixels to transmit different signals. By controlling the Tx signal, pixels on the same line can be made to have different exposure times. The pixel array grouping of the embodiment shown in Fig. 7 has a significant advantage over the grouping method of Fig. 5 in that the edge sawtooth effect of the composite image can be reduced.

Figure 8 is a timing diagram of an image taken by a pixel array, in accordance with one embodiment of the present invention. The timing diagram shown in Figure 8 can be applied to the diagram

In the embodiment shown in 7.

Referring to the upper half of Figure 8, for the R1/G1 R2/G2 rows in which pixels 701, 702, 705, and 706 are located, a pulse is first provided on the RowSel line to select the row. A pulse is provided on the RST line to reset the memory area of each pixel on the Rl/Gl/R2/G2 line. Next, a pulse is provided on the SHR line to sample the memory area after each pixel reset to generate a V _ret signal.

Next, a pulse signal is provided on the TxA line to transfer a portion of the pixels on the R1/G1 R2/G2 line, for example, the charge on the photosensitive device including the white pixels R1/G1 of the pixels 701 and 702 to their respective storage areas. . At the same time, on the TxB line A pulse signal is used to transfer charge on another portion of the pixels on the R1/G1/R2/G2 line, such as the slanted pixels R2/G2 comprising pixels 703 and 704, onto their respective memory areas.

A pulse signal is provided on the SHS line to sample the charge stored on the memory area of each pixel on the R1/G1 R2/G2 line to generate a V _slg signal. It can be seen that although the pixels on the R1/G1/R2/G2 line belong to groups of pixels having different exposure times, they are simultaneously sampled to generate image signals.

Next, when the RST line is high, a pulse is supplied on the TxA line of the R1/G1 R2/G2 line to reset the photosensitive device including the white pixels R1/G1 of the pixels 701 and 702. At the same time, when the same RST line is high, another pulse is supplied on the TxB line of the R1/G1 R2/G2 line to reset the photosensitive device including the diagonal pixels R2/G2 of the pixels 703 and 704. The photosensitive element begins to accumulate charge after resetting. Since the white pixel R1/G1 and the oblique line pixel R2/G2 accumulate charges from different timings; and, as described above, they are almost simultaneously sampled, and therefore, the white pixels R1/G1 belonging to the first group belong to The second group of oblique line pixels R2/G2 have different charge accumulation times, thereby having different exposure times.

For the Gbl Bl/Gb2 B2 row where pixels 703, 704, 707, and 708 are located, since they have the same exposure time as the R1/G1 R2/G2 rows, both can use the same control signal.

For the third row in the pixel array shown in Figure 7, that is, the R2/G2 /R1/G1 row, and the fourth row of the pixel array, that is, the Gb2 B2/Gbl Bl row, the signal timing is as follows in Figure 8. Half part. As can be seen from the figure, the signal timing of the R2/G2 /R1/G1 line and the Gb2 B2/Gbl Bl line is very similar to the signal timing of the R1/G1 R2/G2 line and the Gbl Bl/ Gb2 B2 line described above. The difference between the two is that for the R2/G2 /R1/G1 line and the Gb2 B2/Gbl Bl line, the TxA signal of R2/G2 is reset after resetting the TxB signal of R1/G1. Thus, R2/G2 has a shorter charge accumulation time as Gb2 B2, and R1/G1 and Gbl Bl have longer charge accumulation times. The other parts are the same as the upper part of Fig. 8, and will not be described here.

Due to the multiple exposure method, how to reduce the influence of the exposure conversion on the signal-to-noise ratio, and improving the high dynamic image quality is a problem to be considered. The present invention proposes to solve this problem by increasing the number of exposures, such as 4 exposures, and a specific HDR (ffigh Dynamic Range) algorithm.

Figure 9 is a schematic illustration of a pixel array in accordance with another embodiment of the present invention. As shown in Fig. 9, the pixel array 900 is a color pixel array, and R, G, Gb, and B represent different colors, respectively. White pixels, such as Rl, Gl, Gbl, and B1, represent pixels with an exposure time of T1, respectively; slanted pixels, such as R2, G2, Gb2, and B2, represent pixels with an exposure time of T2; For example, R3, G3, Gb3, and B3 represent pixels whose exposure time is T3, respectively; and pixels of vertical lines, such as R4, G4, Gb4, and B4, respectively represent pixels whose exposure time is T4. Tl, Τ2, Τ3, and Τ4 are different. As can be seen from Fig. 9, for the first group of pixels, the second group of pixels, the third group of pixels, and the fourth group of pixels having different exposure times, two rows are staggered.

In the embodiment shown in Figure 9, since the same row of pixels needs to have different exposure times, the same row of pixels cannot share the Τχ signal, but they can still share the RST and RowSel signals. Therefore, two sets of Tx signals must be provided for each row of pixels to transmit different signals. By controlling the Tx signal, pixels on the same line can be made to have different exposure times.

Figure 10 is a timing diagram of an image taken by a pixel array, in accordance with one embodiment of the present invention. The timing chart shown in Fig. 10 can be applied to the embodiment shown in Fig. 9. The upper half of Figure 10 is the signal timing employed by the first and second rows; the lower half of Figure 10 is the signal timing used for the third and fourth rows. Among them, the signals on TxA and TxB corresponding to reset 1, reset 2, reset 3, and reset 4 are different, thereby making the pixels of the four groups have different charge accumulation start times. Because these pixels are being sampled at the same time, the four groups of pixels have different exposure times.

In addition to the two exposure modes and the four exposure modes and the division of the pixel array employed in the above embodiments, the present invention may also employ multiple exposure modes of more than two times, or other pixel array division methods. For example, the present invention can use nine different exposure times. This is completely feasible for pixel arrays with high resolution. In addition, for four exposures, T1, T2, Τ3, Τ4 can be arranged in the same row of pixels, and TxA, TxB, TxC, TxD are used to control different exposure times; and the pixel array can also be divided into 3x3. The arrangement of pixels. 11 is an HDR method of combining two-exposure images, in which a first pixel and a second pixel have different exposure times, and reading a first pixel to obtain a first output voltage, read, in accordance with an embodiment of the present invention. The second pixel derives a second output voltage. In this embodiment, the first and second output voltages derived from the first pixel and the second pixel are combined to derive a final output voltage. As shown in FIG. 11, in step 1120, the first output voltage VI of the first pixel is first read. The read first output voltage VI can be held in the memory 1. In step 1140, the first output voltage VI is amplified by a predetermined multiple. This predetermined multiple is the ratio of the exposure time of the second pixel to the first pixel. For example, if the exposure time of the second pixel is twice the exposure time of the first pixel, this magnification is 2. The magnification can also be less than one. At step 1150, it is determined if the amplified first output voltage VI exceeds a predetermined threshold. The predetermined threshold is less than or equal to the saturation voltage. It is usually determined by multiplying the saturation voltage by a mask. The mask factor is a fraction of less than or equal to 1, such as 1/2, 3/4, or 1. At step 1160, if the amplified first output voltage VI is greater than the threshold, the first output voltage VI is discarded and the second output voltage V2 of the second pixel is read and retained. At step 1170, if the amplified first output voltage VI is less than the threshold, the second output voltage V2 of the second pixel is discarded while the first output voltage VI of the first pixel is retained. At step 1180, the retained voltage is output as the combined final voltage.

12 is an HDR method of combining images of four exposures, wherein the first pixel, the second pixel, the third pixel, and the fourth pixel have different exposure times, and the first pixel is read, in accordance with an embodiment of the present invention. A first output voltage is obtained, a second pixel is read to obtain a second output voltage, a third pixel is read to obtain a third output voltage, and a fourth pixel is read to obtain a fourth output voltage. In this embodiment, the first pixel and the second pixel are first combined, and the third pixel and the fourth pixel are combined at the same time, and then the combined result of the first and second pixels is combined with the third and fourth pixels. The results are combined to arrive at the final output voltage. The manner of each combination is similar to that described in the embodiment of Fig. 11.

As shown in FIG. 12, in step 1202, the first output voltage VI of the first pixel is first read. The read first output voltage VI can be held in the memory 1. In step 1204, the first output voltage VI is amplified by a predetermined multiple. This predetermined multiple is the ratio of the exposure time of the second pixel to the first pixel. At step 1205, it is determined if the amplified first output voltage VI exceeds a predetermined threshold. The predetermined threshold is less than or equal to the saturation voltage. It is usually determined by multiplying the saturation voltage by a mask. The mask factor is a fraction of less than or equal to 1, such as 1/2, 3/4, or 1. At step 1206, if the amplified first output voltage VI is greater than the threshold, the first output voltage VI is discarded and the second output voltage V2 of the second pixel is read and retained. At step 1207, if the amplified first output voltage VI is less than the threshold, the second output voltage V2 of the second pixel is discarded while the first output voltage VI of the first pixel is retained. At step 1208, the retained voltage is output as a result of the combination, i.e., the first resulting voltage.

The third output voltage V3 of the third pixel is read in step 1220. The read first output voltage V3 can be held in the memory 2. In step 1240, the first output voltage V3 is amplified by a predetermined multiple. This predetermined multiple is the ratio of the exposure time of the fourth pixel to the third pixel. At step 1250, it is determined whether the amplified third output voltage V3 exceeds a predetermined threshold. The predetermined threshold is less than or equal to the saturation voltage. It is usually determined by multiplying the saturation voltage by a mask. The mask factor is a fraction of less than or equal to 1, such as 1/2, 3/4, or 1. At step 1260, if the amplified third output voltage V3 is greater than the threshold, the third output voltage V3 is discarded and the fourth output voltage V4 of the fourth pixel is read and retained. At step 1270, if the amplified third output voltage V3 is less than the threshold, the fourth output voltage V4 of the fourth pixel is discarded while the third output voltage V3 of the third pixel is retained. At step 1280, the retained voltage is output as a result of the combination, i.e., the second resulting voltage.

Next, the first resulting voltage and the second resulting voltage are combined. In step 1290, the first resulting voltage is amplified by a predetermined multiple. This predetermined multiple is the product of the ratio of the exposure time of the second pixel to the first pixel and the ratio of the exposure times of the fourth pixel and the third pixel. At step 1291, it is determined whether the amplified first resulting voltage exceeds a predetermined threshold. The predetermined threshold is typically determined by multiplying the saturation voltage by a mask and multiplying the ratio of the ratio of the second pixel to the first pixel exposure time to the ratio of the fourth pixel to the third pixel exposure time. The mask factor is a fraction of less than or equal to 1, such as 1/2, 3/4, or 1. At step 1292, if the amplified first voltage is greater than the threshold, the first voltage is discarded and the second output voltage is read and retained. At step 1293, if the amplified first voltage is less than the threshold, the second output voltage is discarded while the first output voltage is retained. At step 1280, the retained voltage is output as a combined result output. Usually, the ratio of the exposure time of the second pixel to the first pixel is the same as the ratio of the exposure time of the fourth pixel and the third pixel, for example, a positive integer n, n = 2, 4, 6, 8 and so on. Thus, when combining the first resulting voltage and the second resulting voltage, the predetermined magnification is n ² and the threshold is n times the saturation voltage and multiplied by the mask coefficient.

The following is a specific example to illustrate the calculation and SNR of the dynamic range of the HDR synthesis. (Signal Noise Ratio This embodiment is an example of applying the four-time exposure synthesis algorithm of the present invention to a 1.4 um pixel image sensor. These four different The ratio between exposure times can be a multiple of 2, for example 1:2:4:8. That is, the relationship between the four exposure times is:

T1: T2: T3: T4 = 1: 2 : 4 : 8

The ratio between exposure times can also vary, depending on the requirements for the dynamic range of the synthesized image. The larger the ratio, the larger the dynamic range.

For the sake of simplicity, the ratio of exposure time in this example is η = 2.

The other indicators of the pixel are shown in the following table:

Sensitivity 800mv/lux*s

T1 5 ms potential well 8000e

Τ 2 10ms CG 200uV/e

Τ3 20ms saturated output voltage 1.6V

Τ4 40ms 3/4 saturated output voltage 1.2V

Figure 13 shows the respective optical response curves for each exposure time of Tl, Τ2, Τ3, and Τ4. The slope of the sensing curve of a pixel with a short exposure time is small (such as Tl). The slope of the pixel sensing curve with a long exposure time is larger (eg Τ4). Figure 14 shows the induced curve and its SNR curve after the synthesis algorithm for the four exposure times is completed. As can be seen from Figure 14, the final composite curve is still a straight line. Finally, the saturation voltage of the entire response curve is equivalent to the previous 1.6V to 12.8V. The dynamic range of the curve after synthesis can be calculated by the following formula compared to the increase in only one exposure time:

Delta DR = 201og(T4/Tl)

For the present embodiment, the amount of increase in the dynamic range is 201 og (8: 1), i.e., 18 dB.

It can also be seen from Fig. 14 that the SNR (Signal to Noise Ratio) curve is an up-and-down interleaved conversion of multi-segment sensing points into turning points. There will be a valley of SNR at the turning point, which means that the image is relatively noisy near this point. The digital processing of the image using the HDR algorithm of the present invention allows the curve near the turning point to be smoother to reduce noise and avoid unevenness in the saturation region. In the above example, the mask coefficient used in the combination is 3/4.

Figure 15 shows the effect on the SNR curve with different mask coefficients. As shown in Figure 15, when the mask coefficients are 1, 3/4, 1/2, respectively. The closer the mask coefficient is to the saturation voltage, the higher the S R of the induced curve. Therefore, the choice of mask coefficients needs to be optimized after balancing SNR and saturation voltage non-uniformity. However, if the mask coefficient is too high, although the signal-to-noise ratio (SNR) is high, some unfavorable factors are likely to cause unevenness in the pixel saturation region, which in turn affects the response of the pixel in the saturation region. Therefore, the saturation point is generally not used as the decision point for curve synthesis. Otherwise, a huge FPN (fixed pattern noise) is generated at the inflection point of TO and T1, which affects the image quality. Therefore, the choice of mask factor will not be too close to 1. A preferred mask factor is 3/4.

Figure 16 shows the effect of four exposures and two exposures on SNR. As shown in Figure 16, the combined dynamics of the four exposures (Tl, Τ2, Τ3, Τ4) and the two exposures (Τ0, Τ3) have the same dynamic range, but their SNR is different. of. The SNR of the double exposure at the curve turning point is much lower than that of the four exposures. This will have a big impact on the quality of the image. Therefore, the four exposures are better than the two exposures, although this will further reduce the resolution of the image.

Figure 17 is a schematic illustration of a system in accordance with one embodiment of the present invention. Figure 17 illustrates where the image sensor 1710 is included Processor system 1700. Among them, the image sensor 1710 is an image sensor as described in the present invention. The processor system 1700 exemplifies a system having digital circuitry that can include image sensor devices. Without limitation, the system may include computer systems, camera systems, scanners, machine vision, vehicle navigation, video telephony, surveillance systems, autofocus systems, astronomical tracker systems, motion detection systems, image stabilization systems And data compression system.

Processor system 1700 (e.g., camera system) typically includes a central processing unit (CPU) 1740 (e.g., a microprocessor) that communicates with input/output 0/0) device 1720 via bus 1701. Image sensor 1710 is also in communication with CPU 1740 via bus 1701. The processor-based system 1700 also includes random access memory (RAM) 1730 and may include a removable memory 1750 (e.g., flash memory) that also communicates with the CPU 1740 via bus 1701. Image sensor 1710 can be combined with a processor (e.g., a CPU, digital signal processor, or microprocessor) with or without a memory storage device on a single integrated circuit or on a different chip than the processor. The calculation of image combination and processing can be performed by image sensor 1710 or by CPU 1740.

The technical contents and technical features of the present invention have been disclosed as above, and those skilled in the art can make various alternatives and modifications without departing from the spirit and scope of the invention. Therefore, the scope of the present invention is not to be construed as being limited by the scope of the invention, and

Claims

Claim

1. An imaging method comprising:

Exposing pixels in the pixel array to obtain a first image in a first time;

Exposing the pixels in the pixel array to a second image, wherein the first time is different from the second time;

The first image and the second image are combined.

2. The method of claim 1 further comprising:

Determining whether the change in light intensity of the image to be ingested exceeds the optical dynamic range of the pixel in the array of pixels.

3. The method of claim 1 or 2, further comprising:

Determine if the image to be ingested is in motion.

4. An imaging method comprising:

Exposing pixels in the first group of pixels in the pixel array to obtain a first image;

Exposing pixels in the second pixel group in the pixel array to obtain a second image, wherein the first time is different from the second time;

Reading the first image and the second image simultaneously;

The first image and the second image are combined.

5. The method of claim 4, wherein:

The pixels in the first group of pixels and the pixels in the second group of pixels are staggered in two rows.

6. The method of claim 4, wherein:

The pixels in the first group of pixels are spaced apart from the pixels in the second group of pixels by two rows and are staggered in two directions.

7. The method of claim 4, further comprising:

And exposing the pixels in the third pixel group in the pixel array to obtain a third image; and in the fourth time, in the fourth pixel group in the pixel array The pixel is exposed to obtain a fourth image, where the first time, the second time, the third time, and the fourth time are different;

Reading the third image and the fourth image while reading the first image and the second image; and combining the first image, the second image, the third image, and the fourth image.

8. The method of claim 4, wherein combining the first image and the second image comprises:

For the first pixel in the first image and the second pixel in the second image,

Amplifying the first output voltage of the first pixel by a predetermined multiple;

Determining whether the amplified first output voltage exceeds a threshold;

Rejecting the first output voltage in response to exceeding a threshold, retaining a second output voltage of the second pixel;

In response to the threshold being not exceeded, the first output voltage is retained and the second output voltage of the second pixel is discarded.

9. The method of claim 8 wherein:

The predetermined multiple is a ratio of an exposure time of the second pixel to an exposure time of the first pixel.

10. The method of claim 8 or 9, wherein:

The threshold is a product of a saturation voltage and a mask coefficient, wherein the mask coefficient is less than one.

11. The method of claim 10, wherein:

The mask factor is 3/4.

12. The method according to claim 7, wherein combining the third image and the fourth image comprises:

For the third pixel in the third image and the fourth pixel in the fourth image, Amplifying the third output voltage of the third pixel by another predetermined multiple;

Determining whether the amplified third output voltage exceeds another threshold;

Rejecting the third output voltage in response to exceeding another threshold, retaining a fourth output voltage of the fourth pixel; retaining the third output voltage in response to not exceeding a threshold, discarding a fourth of the fourth pixel The output voltage.

13. The method of claim 12, wherein:

The combining the first image, the second image, the third image, and the fourth image includes:

Combining the first image and the second image to obtain a first result voltage;

Combining the third image and the fourth image to obtain a second result voltage;

Encoding the first result voltage by the predetermined multiple of times by the another predetermined multiple;

Determining whether the amplified third output voltage exceeds a product of a saturation voltage and an average of the threshold and the another threshold; in response to exceeding, discarding the first resulting voltage, retaining the second resulting voltage;

In response to not exceeding, the second resulting voltage is retained and the first resulting voltage is discarded.

14. The method of claim 13 wherein:

The predetermined multiple is the same as the other predetermined multiple; and the average of the threshold and the other threshold is the same.

15. An imaging device comprising:

a pixel array comprising a plurality of pixels arranged in rows and columns;

a control circuit that controls the pixel array; wherein

The pixel array includes a first group of pixels that are exposed in a first time to obtain a first image;

The pixel array includes a second pixel group that is exposed in a second time to obtain a second image, wherein the first time is different from the second time;

Wherein the control circuit further reads the first image and the second image simultaneously;

An image processor that combines the first image and the second image.

16. The image forming apparatus according to claim 15, wherein:

In the pixel array, the pixels in the first group of pixels and the pixels in the second group of pixels are staggered in two rows.

17. The image forming apparatus according to claim 15, wherein:

In the pixel array, the pixels in the first group of pixels are spaced apart from the pixels in the second group of pixels by two rows and staggered in two directions.

18. The image forming apparatus according to claim 15, further comprising:

The pixel array includes a third pixel group that is exposed for a third time to obtain a third image;

The pixel array includes a fourth pixel group that is exposed for a fourth time to obtain a fourth image, wherein the first time, the second time, the third time, and the fourth time Different from each other;

Wherein the control circuit further reads the third image and the fourth image while reading the first image and the second image;

The image processor combines the first image, the second image, the third image, and the fourth image.

19. The imaging apparatus according to claim 18, wherein the image processor is further

Multiplying the first resulting voltage by a predetermined multiple of the predetermined multiple;

Determining whether the amplified third output voltage exceeds a product of a saturation voltage and an average of the threshold and the another threshold; in response to exceeding, discarding the first resulting voltage, retaining the second resulting voltage; In response to not exceeding, the second resulting voltage is retained and the first resulting voltage is discarded.

20. The image forming apparatus according to claim 15, wherein the image processor further

Amplifying the first output voltage of the first pixel by a predetermined multiple, the predetermined multiple being a ratio of an exposure time of the second pixel to an exposure time of the first pixel;

Determining whether the amplified first output voltage exceeds a threshold, the threshold being a product of a saturation voltage and a mask coefficient, wherein the mask coefficient is less than 1;