EP2957097A2

EP2957097A2 - High dynamic range cmos image sensor having anti-blooming properties and associated methods

Info

Publication number: EP2957097A2
Application number: EP14752193.4A
Authority: EP
Inventors: Jiang JUTAO; Matt BORG
Original assignee: SiOnyx LLC
Current assignee: SiOnyx LLC
Priority date: 2013-02-15
Filing date: 2014-02-18
Publication date: 2015-12-23
Also published as: EP2957097A4

Abstract

A method of providing blooming protection to a CMOS imager having a pixel array of a plurality of pixels arranged in rows and columns, where the CMOS imager is operable to capture high dynamic range images using a rolling shutter, is provided. Such a method can include reading out charge accumulated by the pixels in a readout row of a first integration time, applying a reset to the readout row for a reset time sufficient to allow readout and reset to occur in at least one subsequent row, and starting a second integration time of the pixels in the readout row, wherein the second integration time is shorter than the first integration time, and wherein the at least one subsequent row is a sufficient number of rows to have a combined reset to preclude blooming effects from the pixel array during the second integration time.

Description

HIGH DYNAMIC RANGE CMOS IMAGE SENSOR

HAVING ANTI-BLOOMING PROPERTIES AND ASSOCIATED METHODS

BACKGROUND

In general, CMOS image sensors tend to have a limited dynamic range due to well capacity limitations. Various approaches have been attempted to improve dynamic range using such sensors, such as, for example, multiple exposure, lateral overflow, logarithmic pixel, in-pixel delta-sigma ADC, pixel with multiple-size- photodiodes, pixel arrays with different neutral density filters, dual conversion gain, etc. The multiple exposure approach is one common technique for high dynamic range (HDR) imaging.

In lateral overflow HDR schemes, pixel integration time is divided into two or more segments. In each segment, the effective pixel well capacity is varied. At the end of integration, the total accumulated charge is readout. However, this approach normally has fixed pattern noise at each knee point (the time at which an intermediary reset voltage is applied) on the signal response curve. Lateral overflow approaches also tend to be flexible regarding dynamic range extension.

Delta-sigma ADC (Analog to Digital Converter) is a method for encoding analog signals into digital signals or for encoding high-resolution signals to lower- resolution signals. In the in-pixel delta-sigma ADC scheme, each pixel's integration time is individually controlled. This approach in theory can provide the best HDR scene capture and is the most flexible to achieve the highest possible dynamic range. However, the pixel size is large, complex and not appealing for most consumer applications.

In multiple-diode-pixel systems, each pixel has multiple photodiodes. The effective sensitivity of two photodiodes can thus be made different by design.

Therefore, the HDR scene multiple-exposure can be achieved simultaneously.

However, the enhanced dynamic rage is fixed in design and it is not flexible.

BRIEF DESCRIPTION OF THE DRAWINGS

For a fuller understanding of the nature and advantage of the present invention, reference is being made to the following detailed description of preferred embodiments and in connection with the accompanying drawings, in which: FIG. 1 is a schematic diagram of a four transistor CMOS image sensor pixel in accordance with one aspect of the present disclosure.

FIG. 2 is a depiction of a method of providing blooming protection in accordance with another aspect of the present disclosure.

FIG. 3 is a depiction of a method of providing blooming protection in accordance with another aspect of the present disclosure.

FIG. 4 is a graphical representation of a 2-pointer HDR image sensor readout and shutter timing scheme, in accordance with another aspect of the present disclosure.

FIG. 5 is a graphical representation of a 2-pointer FfDR image sensor readout and shutter timing scheme, in accordance with another aspect of the present disclosure.

FIG. 6 is a graphical representation of a timing diagram for a rolling shutter readout scheme, in accordance with another aspect of the present disclosure.

DETAILED DESCRIPTION

Before the present disclosure is described herein, it is to be understood that this disclosure is not limited to the particular structures, process steps, or materials disclosed herein, but is extended to equivalents thereof as would be recognized by those ordinarily skilled in the relevant arts. It should also be understood that terminology employed herein is used for the purpose of describing particular embodiments only and is not intended to be limiting.

Definitions

The following terminology will be used in accordance with the definitions set forth below.

It should be noted that, as used in this specification and the appended claims, the singular forms "a," and, "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a pixel" includes one or more of such pixels and reference to "the shutter" includes reference to one or more of such shutters.

In this application, "comprises," "comprising," "containing" and "having" and the like can have the meaning ascribed to them in U.S. Patent law and can mean "includes," "including," and the like, and are generally interpreted to be open ended terms. The terms "consisting of or "consists of are closed terms, and include only the components, structures, steps, or the like specifically listed in conjunction with such terms, as well as that which is in accordance with U.S. Patent law. "Consisting essentially of or "consists essentially of have the meaning generally ascribed to them by U.S. Patent law. In particular, such terms are generally closed terms, with the exception of allowing inclusion of additional items, materials, components, steps, or elements, that do not materially affect the basic and novel characteristics or function of the item(s) used in connection therewith. For example, trace elements present in a composition, but not affecting the composition's nature or characteristics would be permissible if present under the "consisting essentially of language, even though not expressly recited in a list of items following such terminology. When using an open ended term, like "comprising" or "including," it is understood that direct support should be afforded also to "consisting essentially of language as well as "consisting of language as if stated explicitly, and vice versa. Further, it is to be understood that the listing of components, species, or the like in a group is done for the sake of convenience and that such groups should be interpreted not only in their entirety, but also as though each individual member of the group has been articulated separately and individually without the other members of the group unless the context dictates otherwise. This is true of groups contained both in the specification and claims of this application. Additionally, no individual member of a group should be construed as a de facto equivalent of any other member of the same group solely based on their presentation in a common group without indications to the contrary.

As used herein, the term "substantially" refers to the complete or nearly complete extent or degree of an action, characteristic, property, state, structure, item, or result. For example, an object that is "substantially" enclosed would mean that the object is either completely enclosed or nearly completely enclosed. The

exact allowable degree of deviation from absolute completeness may in some cases depend on the specific context. However, generally speaking the nearness of completion will be so as to have the same overall result as if absolute and total completion were obtained. The use of "substantially" is equally applicable when used in a negative connotation to refer to the complete or near complete lack of an action, characteristic, property, state, structure, item, or result. For example, a composition that is "substantially free of particles would either completely lack particles, or so nearly completely lack particles that the effect would be the same as if it completely lacked particles. In other words, a composition that is "substantially free of an ingredient or element may still actually contain such item as long as there is no measurable effect thereof.

As used herein, the term "about" is used to provide flexibility to a numerical range endpoint by providing that a given value may be "a little above" or "a little below" the endpoint.

Concentrations, amounts, and other numerical data may be expressed or presented herein in a range format. It is to be understood that such a range format is used merely for convenience and brevity and thus should be interpreted flexibly to include not only the numerical values explicitly recited as the limits of the range, but also to include all the individual numerical values or sub-ranges encompassed within that range as if each numerical value and sub-range is explicitly recited. As an illustration, a numerical range of "about 1 to about 5" should be interpreted to include not only the explicitly recited values of about 1 to about 5, but also include individual values and sub-ranges within the indicated range. Thus, included in this numerical range are individual values such as 2, 3, and 4 and sub-ranges such as from 1-3, from 2-4, and from 3-5, etc., as well as 1, 2, 3, 4, and 5, individually.

This same principle applies to ranges reciting only one numerical value as a minimum or a maximum. Furthermore, such an interpretation should apply regardless of the breadth of the range or the characteristics being described.

The Disclosure

The presently disclosed technology provides systems, methods, and devices for use in capturing high dynamic range (HDR) images using CMOS imagers. More specifically, blooming protection is provided through the strategic use of shuttering, particularly between images captured at different integration times. These novel techniques can reduce blooming issues that apply to any row-wise HDR CMOS imager such as, for example, rolling shutter CMOS imagers.

CMOS image sensors generally include an array of pixels arranged in rows and columns. Each pixel can generally include a photodiode and a transfer gate, which is used in image readout to control the transfer of collected charge from the photodiode. Imagers can further include a reset gate used to reset or shutter a plurality of pixels (i.e. one or more rows of pixels in some aspects), a row select transistor, among others. It is noted that CMOS imagers are well known in the art, and as such any known component or design incorporated into an imager using a rowwise readout scheme should be considered to be within the present scope. As examples, CMOS imagers having four transistor (4T), 5T, or more can be utilized. One non-limiting example of a 4T CMOS imager is shown in FIG. 1. Such a device can include a photodiode 102, a transfer transistor 104, a floating diffusion region 106, a reset 108, source follower 110, a row select 112, a power supply voltage for the pixel array 114, and the voltage out 116.

Row-wise readout, also known as rolling shutter, is often used as a mechanism in CMOS imaging to control exposure time of the pixels to incident light. The rolling shutter is an electronic shutter that operates at the level of the pixel, and rolls along the pixel array row-by-row. When a row is selected to be shuttered, a reset voltage is applied to the pixel via a reset transistor and a voltage is applied to the transfer gate, thus clearing the contents of the photodiode of the pixel and placing the pixel in a reset state. While in the reset state, the pixel does not accumulate charge from incoming photons. Once the reset voltage and the transfer gate voltage has been removed, the pixel is capable of absorbing incoming photons and accumulating the resulting charge. In one non-limiting example of a typical rolling shutter process, rows of pixels in the imager can be reset in sequence, starting, for example, at the top of the array and proceeding row-by-row to the bottom of the array. It is noted, however, that rolling shutter processes can move from any point in the array to another, and that such is merely exemplary. Once the reset operation has moved past a row of pixels, charge integration begins. Following a given integration time, set by the device or the user, the readout operation begins. Rows can be read out of the pixel array in the order of the rolling shutter to maintain a constant integration time for each row. Thus, the exposure time for the pixel array is controlled by the timing difference between the passing of the rolling shutter and readout of the rows of the array. This exposure time is often referred to as the integration time. Increasing the integration time increases the duration that the pixels are accumulating charge, while decreasing the integration time decreases charge accumulation.

One common problem that can arise when capturing an image, particularly with a long integration time, is referred to as "blooming." A pixel has a limited charge-well capacity, and thus can only accumulate a fixed amount of charge. When this well is full, and particularly then the pixel continues to convert photons to charge, excess charge can spill over into neighboring pixels, thus corrupting the associated charge concentration in those pixels. This negative effect can also spill over into pixels multiple rows away, depending on the relative charge between the source of the blooming and the affected rows, the pixel size, the epitaxial layer thickness, etc. This can be particularly problematic for imaging scenes with bright light on a dark background, such as, for example, car headlights in the dark. Such blooming thus spreads, causing unwanted noise throughout at least portions of the image.

As has been described, multiple exposure imaging is one technique for generating high dynamic range (HDR) images with CMOS technology. HDR images captures a greater dynamic range between the lightest and darkest regions of a given scene as compared to normal imaging technologies. Common HDR scenes occur when there is a large contrast of light intensity between objects in the scene, such as illumination of a car's headlights at night. In multiple exposure HDR schemes, multiple images are taken at different integration times for each frame for the same scene. Afterwards, the multiple images are combined to reconstruct the final image. Typically, the multiple exposure HDR schemes can be frame-wise or row-wise. In a frame-wise HDR scheme, two (or more) frames are readout sequentially, meaning the whole focal plane array is read out at least twice and combined to create the HDR image. The first frame will have a predetermined integration time, also known as exposure time, and the second frame will have a predetermined integration time. Typically, the frame for capturing the low lighting intensities in a scene will have a longer exposure or integration time than the frame capturing regular lighting intensities or high intensities of a given scene. Once the raw image data from the two frames has been read out, they are combined to create a HDR image. In row-wise HDR imaging, on the other hand, every row is read out multiple times in a single frame image, in which each readout has an integration time, which can, in some cases, be different integration times. For a three exposure HDR readout scheme, for example, each row is read out three times at different integration times. To reconstruct the final HDR image, the raw image data from a row-wise HDR is separated into individual frame data first. Both frame-wise and row-wise HDR approaches are flexible regarding extended dynamic range and can be applied to most CMOS imager with minimal design changes. However, both designs can utilize on- chip or off-chip memory to store the image data before final image re-construction. One implementation of a row-wise HDR imaging scheme initiates a first image process having a longer integration time exposure using a rolling shutter, with a second image process having a shorter integration time being initiated following the readout of the first image. In situations where blooming occurs in the longer integration time image, such blooming charge can cross the reset pixels into the second image having the shorter integration time, thus causing undesirable noise. As one example, assume a rolling shutter one row wide rolls across a pixel array, initiating integration. At the end of a 100 row integration time, each row is read out and reset with a second rolling shutter, again 1 row wide, that initiates integration of the pixels for a shorter 3 row integration time. In this case, rows that have yet to be read out for the long integration time may have wells with sufficient charge to bloom, and as such can cross the second rolling shutter prior to readout of the row, causing corruption of the charge in the pixels undergoing short integration.

The present technology reduces or eliminates such undesirable blooming through the strategic use of shuttering to limit the blooming across the rows that are being reset or shuttered following readout of the long integration time image. Such strategic shuttering can be accomplished in a variety of ways, such as, for example, variation in the number of rows shuttered, the pattern of the rows shuttered, temporal variations in the shuttering voltage, and the like. In one aspect, the number of shuttered rows can be increased that are directly adjacent to one another. In this case, readout of the long integration time is followed up by a shutter that is 2, 3, 4, or more rows wide. Following this extended shutter, the reset is removed and the pixels are allowed to integrate for the short integration time. Charge from pixels having full charge wells that have not yet been read out in the long integration time region is thus precluded or limited from crossing the multiple row shutter, or in other words, the multiple rows that are in the reset state. In the case of a 4 row shutter, for example, a given row is read out for the long integration time image, followed by a reset that is present for the next 3 row readouts. The reset of the given row is then release, and the short integration time is initiated. The last row read out is shuttered and the 4-row rolling shutter moves across the pixel array in a row-wise manner, thus providing bloom protection to the shorter integration time region. Note that the beginning of the short integration time may be delayed compared to a traditional rolling shutter due to the width of the reset. Furthermore, it is noted that the present technology is not limited to HDR imaging having only two integrations times, but the same or similar imaging process can be applied for a third, fourth, or more images having different integration times. Thus, in the above example, when the given row is read out for the second integration time, the row can be reset with a shutter having characteristics to provide bloom protection to a third integration image, and so on.

A variety of shutter configurations and shutter behaviors are contemplated, and can include any shutter scheme that provides blooming protection to HDR images. In general the shutter has at least two rows of pixels that are simultaneously shuttered to provide the blooming protection. In one aspect, as has been described, the shutter can be applied across 2, 3, 4, 5, or more directly adjacent rows. In another aspect, the shutter can be applied across 2, 3, 4, 5, or more rows that are not directly adjacent, or that contain directly adjacent rows and non-directly adjacent rows in the same shutter. For example, a 4 row shutter can be applied to rows 10, 11, 12, and 13, or a 4 row shutter can be applied to rows 10, 12, 14, and 16, or to rows 10, 13, 14, and 17, or any other combination that facilitates blooming protection.

As has been described, the chance of blooming can vary depending on the relative charge between the source of the blooming and the affected rows, the pixel size, the epitaxial layer thickness, etc. As such, the shutter configurations can be designed to account for such factors. In one aspect, the present technology can utilized epitaxial layer thicknesses of from about 1 to about 10 microns, and can have pixel sizes of from about 0.9 microns to about 30 microns, or from about 0.9 microns to about 6 microns, or from about 0.9 microns to about 3 microns, to name a few non- limiting examples. As the size of the pixel is decreased, an increase in the number of rows shuttered to prevent or reduce blooming may need to be increased. For example, in a 0.9 micron pixel, the shutter can be up to 30 rows or more in some cases. It is, however, within the skill of those in the art to readily calculate the necessary shutter durations to account for these variations, once in possession of the present disclosure.

Furthermore, various shutter behaviors are contemplated, and can vary depending on the design and/or complexity of the device. In one aspect, a device can have a fixed shutter scheme. For example, a shutter may be fixed at a 4 row width in a device no matter what the imaging conditions. In another aspect, the shutter in a device can be manually set by a user for a given image or image conditions. In other aspects, the device can include an automatic mode, where the shutter characteristics are automatically set to match lighting conditions. Additionally, hybrid approaches are contemplated whereby a user sets the device to behave in a desired manner, and the device automatically sets or varies the shutter characteristics to optimal or near optimal conditions within the parameters set by the user.

The shutter can additionally be varied within a given frame or image process.

For example, in one aspect the device can increase the duration of the shutter between a long integration time process and a medium integration time process, but decrease the duration of the shutter between the medium integration time process and a shorter integration time process. Thus, the duration of the shutter can be increased during HDR image capture in between the image sections that may have blooming issues, but decreased in between image sections that are not as likely to experience blooming. While in some devices such a behavior may be fixed, in other aspects the device can contain logic to dynamically increase the shutter duration or to apply a given shutter pattern when blooming is detected or when a scene is detected where blooming may occur. So, the logic can dynamically adjust the shutter in between integration time images, or in some aspects during a given rolling shutter operation for a given integration time. If an image is being read out for a longer integration time and a potential for blooming is detected, for example, the logic can adjust the shutter to limit or preclude blooming from occurring.

Additionally, in some aspects the strength (i.e., the voltages applied) of the reset can be constant throughout the duration of a given shutter, while in other aspects the shutter strength can be variable. For example, one energy saving technique may be to hard reset a row as the shutter is applied, lower the reset voltage during the middle of the shutter, and hard reset the row again just prior to the initiation of the next integration cycle. Variations in the shutter strength can be fixed in the device, set by the user, and/or dynamically controlled by the device. In the dynamically controlled case, the logic of the device can be utilized to detect blooming or the potential for blooming, and can thus increase the strength of the reset during such blooming periods and decrease the strength when blooming potential is lower.

As is shown in FIG. 2, one exemplary method provides blooming to a CMOS imager having a pixel array of a plurality of pixels arranged in rows and columns, where the CMOS imager is operable to capture high dynamic range images using a rolling shutter. The method can include 202 selecting a readout row of pixels, 204 starting a first integration time of the pixels in the readout row, and 206 reading out charge accumulated by the pixels in the readout row to obtain a first readout. The method also includes 208 applying a reset to the readout row for a reset time sufficient to allow readout and reset to occur in at least one subsequent row, 210 removing the reset and starting a second integration time of the pixels in the readout row, wherein the second integration time is shorter than the first integration time, and wherein the at least one subsequent row is a sufficient number of rows to have a combined reset to at least substantially preclude blooming effects from the pixel array in the readout row during the second integration time, and 212 reading out charge accumulated by the pixels in the readout row to obtain a second readout. It is noted that a subsequent row is a row that is read out or otherwise processed following the readout row. As such, applying a reset to the readout row for a reset time sufficient to allow readout and reset to occur in at least one subsequent row includes allowing the reset to continue on the reset row to at least give sufficient time for at least one subsequent row to be read out and reset. Thus, by allowing the reset of the readout row to be maintained for sufficient time to read out and reset two subsequent rows, a three row shutter has been implemented. Accordingly, in one aspect the reset can be applied for the reset time sufficient such that at least two subsequent rows and the readout row are simultaneously reset or in a reset state. In another aspect, the reset can be applied for the reset time sufficient such that at least three subsequent rows and the readout row are simultaneously reset or are in a reset state.

While it is convenient to describe reset and integration times in terms of row timing, it can also be useful to describe actual timing ranges. For example, in one aspect the reset time sufficient to allow readout and reset to occur in at least one subsequent row can be from about 10 nanoseconds to about 50 microseconds. In another aspect, the reset time sufficient to allow readout and reset to occur in at least one subsequent row can be from about 0.5 microseconds to about 2 microseconds. Understandably, these reset times can vary depending on the duration of the integration and the degree of potential blooming in the array. In another aspect, the first integration time can be from about 1 millisecond to about 1 second and the second integration time can be from about 10 nanoseconds to about 100 milliseconds. In yet another aspect, the first integration time can be from about 20 microseconds to about 33 milliseconds and the second integration time can be from about 1 microsecond to about 16 milliseconds. As has been described, the present scope should also include shuttering or reset schemes that are sequential, non-sequential, adjacent, and non-adjacent. For example, in one specific aspect, the readout row and the at least three subsequent rows can be sequentially adjacent. In another aspect, the readout row and the at least three subsequent rows can be sequentially non-adjacent. The same would thus also apply to any number of subsequent rows greater than one (i.e., greater than two rows, the readout row and the at least one subsequent row).

Additionally, the method can be repeated for subsequent rows. In one aspect, for example, the method can be repeated on at least one subsequent row, at least two subsequent rows, at least three subsequent rows, and so on. In some aspects the at least one subsequent row is at least substantially all pixel rows in the pixel array. Furthermore, depending on the design of the device, the rows can be read out in various orders. For example, the method can be repeated on at least substantially all pixel rows in the pixel array in a sequential order. Such orders can include directly adjacent rows as a sequence, alternating rows as a sequence, and the like.

Alternatively, the method can be repeated on at least substantially all pixel rows in the pixel array in a non-sequential order.

In addition to a first and second integration time, it is additionally

contemplated that the present technology encompasses a third, fourth, or more integration times utilized to produce an HDR image. As such, in one aspect the method can further include applying the reset to the readout row following the second readout for a reset time sufficient to allow readout and reset to occur in at least one subsequent row, removing the reset and starting a third integration time of the pixels in the readout row, wherein the third integration time is shorter than the second integration time, and wherein the at least one subsequent row is a sufficient number of rows to have a combined reset to at least substantially preclude blooming effects from the pixel array in the readout row during the third integration time, and reading out charge accumulated by the pixels in the readout row to obtain a third readout. Similar steps can be accomplished for a fourth or more integration time.

It should be understood, however, that the terms "first," "second," "third," etc., should not be construed as meaning the first integration, second integration, and so on of a HDR image taken, but rather are intended to merely describe an order of execution of the integration times. Thus, in the method described above, for example, the terms "first" and "second" are merely describing two integration times that are temporally timed in the order of first and second. So in a HDR imaging process, the first integration time and the second integration time may represent the first and second integrations of the image, or they may represent the third and fourth, or the second and third, or any other pair of integration operations to which blooming protection applies. Thus, in cases where the first integration time corresponds to the first integration of a HDR image, starting the first integration time can include specifically applying the reset to the readout row and releasing the reset from the readout row. In cases where the first integration time described in the method does not correspond to the first integration of the HDR image, integration can be initiated by the readout and reset of the readout row from the previous integration.

As has been described, in one aspect the method can also include applying the reset at a continuous voltage level throughout the reset time duration. In another aspect, the reset can be applied at a variable voltage level throughout the reset time duration. Furthermore, in describing a multiple row shutter reset, it should be understood that for each subsequent readout and reset the reset voltage on each row of the shutter can be maintained or the reset voltage can be reapplied for each processing cycle. As a non-limiting example, for a shutter across rows five, six, seven, and eight, as row nine is read out the shutter moves to rows six, seven, eight, and nine, and is released from row five to begin integration of the next image part. In such a case, the reset can be maintained on rows six, seven, eight, and then applied to nine, or the reset can be reapplied to rows six, seven, eight, and then applied to nine. It is thus non- limiting as to how the shutter is applied and/or maintained, provided sufficient shutter is in place to prevent or limit blooming from corrupting across the shutter.

The method can additionally further include, in some aspects, the formation of a HDR image from the individual images derived from the different integrations, in some cases at different integration times. For example, in one aspect the first readout and the second readout can be combined to form a high dynamic range image. Thus, at a row level, the various integration times for each row can be combined to form the resulting image. At the image level, the row data from the various integration times can be formed into discrete images or image data sets and then the images or image data sets from each integration can be combined to form the HDR image. Thus, two integration images, three integration images, four integration images or more can be combined into the HDR image. In some cases, all integration images are combined, while in other cases only a subset of integration images are combined. In another aspect of the present disclosure, as is shown in FIG. 3, a method of using a rolling shutter to provide blooming protection in a CMOS imager in high dynamic range mode and having a pixel array of a plurality of pixels arranged in rows and columns is provided. Such a method can include 302 capturing a first image in the pixel array having a first integration time, 304 reading the first image out of the pixel array sequentially by rows using at least one readout row of the first image, and 306 hard resetting a plurality of rows in proximity to the at least one readout row of the first image. The method also includes 308 capturing a second image in the pixel array having a second integration time, the second integration time being shorter than the first integration time, wherein the second image is protected from blooming caused by the first image by the hard resetting of the plurality of rows in proximity to the readout row of the first image, and 310 reading the second image out of the pixel array sequentially by rows using at least one readout row of the second image.

As with the previous method aspects, the present method is not limited to two images, nor should the terms "first" and "second" be construed as necessarily the first and second images taken, but rather describe an order the images are captured. As such, in some aspects the method can further include hard resetting a plurality of rows in proximity to the at least one readout row of the second image, capturing a third image in the pixel array having a third integration time, the third integration time being shorter than the second integration time, wherein the third image is protected from blooming caused by the first or the second image by the hard resetting of the plurality of rows in proximity to the readout row of the second image, and reading the third image out of the pixel array sequentially by rows using at least one readout row of the third image. In yet other aspects similar processing can be utilized to generate a fourth, fifth, sixth, or more images for HDR purposes.

Additionally, in some aspects the first image and the second image can be combined to form a HDR image. In other aspects, the first image, the second image, and the third image can be combined to form a HDR image. Thus, any number of images taken can be combined, either all together or a particular subset thereof, to form a HDR image.

The following is a more detailed description of the CMOS processing, components, and methodologies that may be useful in the practice and/or

implementation of the present technology. It should be understood that wide variations in devices, device architectures, device design, and method implementation are contemplated that may depend, at least in part, on the particular CMOS technology used, the desired HDR capabilities, the specific types of lighting scenes in which HDR is utilized, various design preferences, device cost, and the like. As such, the present scope should not be accordingly limited. It should also be understood that those skilled in the art can readily design and produce systems and devices incorporating the present technology once in possession of the present disclosure. In some cases, this technology can be implemented into current HDR imagers using minimal row driver digital design modifications.

Furthermore, it is also contemplated that other beneficial technologies can also be implemented to further improve the HDR imaging process. One example of such a technology that can be utilized during a readout process is correlated double sampling (CDS). CDS is a technique for measuring a signal that allows for removal of an undesired offset and low frequency temporal noise (e.g. kTC noise for CMOS pixel). The output signal is thus measured twice: once in a known condition and once in an unknown condition. The value measured from the known condition is then subtracted from the unknown condition to generate a value with a known relation to the physical quantity being measured. In one aspect, a CDS operation can be performed on a row that is being read out from the imager.

In one example of a rolling shutter readout operation is shown, as a timing diagram, in FIG. 6. In this example, the integration time is 1 row. To read out the signal, the floating diffusion (FD) is reset with the reset signal (RST), and the signal from the FD will be sampled with the SHR signal. Next, TX will activate for charge transfer, and the signal will then be sampled with the SHS pulse. The difference between the SHR and SHS signals will be used for the final signal, which will be CDS mode. Following CDS, a shutter operation can be performed in which both RST and TX will be pulsed for the shutter row, or rows in some cases. This readout operation can then be repeated for other rows in the array.

As another example, a CDS operation can include setting the reset select transistor of the row being read to a high state and setting the power supply to a high power supply setting to set the floating diffusion regions to the high power supply setting in the row being read. As a non-limiting example, the power supply can be increased from 2.8 V to a high power supply setting of 3.1 V. The electrical charge in the floating diffusion regions can then be read to obtain a first electrical charge value. The transfer transistor of the row being read can then be set to a high state to transfer electrical charge in the photodiode to the floating diffusion regions of the row being read. Subsequently, the electrical charge in the floating diffusion regions can be read to obtain a second electrical charge value. The second charge value can then be subtracted from the first electrical charge value to obtain the CDS sampled output.

The following equations and techniques represent various methodologies to provide blooming protection using a rolling shutter in HDR imaging. Such are meant to be exemplary, and should not be seen as limiting. As such, various mathematical equations can be utilized in the understanding of the overall anti-blooming process, and such may be useful in the practice of the present technology.

In rolling shutter mode, a CMOS pixel array is read out essentially a row at a time, although in some aspects more than one row may be read out simultaneously or in an overlapping fashion. In the following equations, r_c represents the readout row, or in other words, the pointer to the readout row. The readout can be accomplished by any readout technique. As has been described above, one beneficial readout technique is CDS mode readout, which allows for removal of an undesired offset and/or temporal noise associated with reset of floating diffusion (FD). CDS readout mode can be signified in the following equations by the term r_CDS, which represents the readout row being read out in CDS mode. Thus in the case of Τς_Β5, as is shown in Equation (1), the physical row index is set to row 1 which is being read out, and the readout data has a CDS operation is applied to it. r_c = rcos (!)

The term "shutter" refers to a shutter operation or a resetting operation of the pixels. As such, the terms "shutter" and "reset" can be used interchangeably, unless the context clearly indicates otherwise. A reset can thus occur in a selected pixel or pixel row or rows to produce a "reset state," or in other words a state in which the pixel is not in a charge accumulation or integration state. Thus a reset state is a state in which the pixel is not accumulating charge for the purpose of readout for imaging. In general, the reset is a reset voltage that is applied to the pixel to remove accumulated charge and to prevent or reduce further accumulation. It is additionally contemplated that the reset voltage can be applied and/or maintained at different levels, depending on the voltage applied and/or maintained in the pixel. So, for example, a complete reset can be applied to the pixel to at least substantially erase the accumulated charge and, if maintained in the pixel, to prevent or significantly prevent further charge accumulation. This initial strong or sharp voltage application can be referred to herein as a "hard reset." In other cases, particularly for power saving operations, a lower voltage can be applied to and/or maintained in the pixel if desired. It is noted that in some cases a pixel can accumulate charge when in a reset state; however such charge accumulation is not a purposeful charge accumulation for readout of an image.

Accordingly, when a shutter is removed from a pixel, the pixel can begin accumulating charge from incoming photons. From the beginning of such charge accumulation to the readout of the pixel is generally referred to as the integration time of the pixel. Thus, in a rolling shutter mode, a shutter or reset voltage can be applied to a row of pixel to place the row in a reset state. When the reset voltage is removed, the row of pixels can then begin accumulating charge from incoming light and is thus in an integration state. For the purposes of the present disclosure, r _hutter is used to refer to the physical row index of a pixel row that is being shuttered for the i_th pointer operation. Also, t}_nt refers to the first pointer's integration time in units of row(s) and "^£int" refers to i_th pointer's integration time in units of row(s). For example, a given row is reset and allowed to integrate charge over a given integration time. Thus the readout row r_c is shuttered or reset and read out following an integration time in numbers of rows (t-_nt), as is described by Equation (2): rshutter ^{= r}c + (2) In some aspects in rolling shutter mode, as a physical row r_c is being read out, row ^rc + ^rshutter ^can t>^e hard reset, where r_shutter is at least one row, but can also include multiple rows in the imager array.

As has been described, blooming protection can be provided to HDR capture through the strategic use of a shutter in between at least two of the images used in the HDR imaging process. Such blooming protection thus effectively minimizes or prevents the crossover or blooming of charge from one region of the array to another. In the following equations, the term "p" is used to describe the blooming protection row count (or purge row count) in numbers of rows that is provided by this strategic shutter. It is noted that the rows may or may not be sequential, that various numbers of rows or patterns of rows can be used as blooming protection, and that such variations are intended to be within the present scope. Those of ordinary skill in the art would recognize appropriate variations in the equations and methodologies disclosed herein to account for such variations in design. In one aspect, however, p can be greater than one row, or more. Any number of rows or patterns of rows that can reduce or eliminate blooming are contemplated and included in the present scope.

As is shown in Equation (3), a shutter is applied to a row or rows of the array to provide protection from blooming. rhs = r&l - ti_t - v (for i > l) (3)

While the terms of this equation indicate that the row is read out in CDS mode, it is also to be understood that any readout mode is additionally contemplated, and that CDS terminology is used merely for convenience. The same is true for the other equations shown herein.

Accordingly, a larger-than-l-row purge row count can enable the HDR imager device to have reduced blooming effects. This blooming purge count p provides additional shifts between different pointer operation regarding readout and shutter row index. The shift can be adjusted to achieve the proper anti-blooming properties given a particular scene. In Equation (4), the total shutter count of p times is allowed, and in this case provides the blooming protection. In one aspect, for example on a CMOS imager design, a (p+1) shutter can be executed sequentially after a readout such as CDS. ^rshutter ^{= r}CDS ~~ V> ( shutter ^{= r}CDS ~ V + 1<—> ^rshutter ^{= r}CDs)(f^{or ί} > 1, P≥ 0)

(4)

As one example, FIG. 4 graphically shows an exemplary embodiment having a first and second pointer row-wise HDR CMOS image sensor timing scheme without any blooming protection (e.g. p = 0). In this timing scheme t}_nt = 100 rows and t _nt = 3 rows. In this aspect it is assumed that the valid row address starts at row 1 and will end at row N. As such, the physical row readout for the first pointer will start on row 1 (CDS: 1). Since the first integration time is 100 row units, row 101 is shuttered (S: 101) as row 1 is read out. It is noted in the timing scheme another row (CDS: -2) is also read out. Since the physical row -2 is non-existent no row data will be readout. Once read out, row 1 is shuttered (S: l) in preparation for the second integration.

In the next time frame, the 2^nd physical row (CDS: 2) is read out in CDS mode. Since the integration time is 100 rows, row 102 is shuttered (S: 102) and row 101 begins integration. Row 2 is shuttered (S:2) in preparation for the second integration of row 2, and the shutter is removed from row 1, which begins the second integration (tf_nt). Because the shutter is only one row wide, blooming can cross into the second integration time.

A similar timing scheme to FIG. 4 is shown in FIG. 5, but having blooming protection or blooming purge count of 3 rows (p = 3). For this example, the integration times remain at 3 row. It is noted that these integration times are merely exemplary, and any desired integration time can be used, in some cases depending on light intensity. The readout sequence for the first integration or the 1^st pointer for row 1 is: CDS: 1 ; Shutter: 101, CDS: -5; Shutter: -2, - 1, 0, 1. Since physical row index starts at 1, the CDS of row -5 will not produce any data since -5 is not a valid physical row address. For the 2^nd pointer shutter operation, a total of 4 rows are being reset: -2, -1, 0, 1. If row 7 is read out for the 1^st pointer, its CDS and shutter sequence will be derived by setting p = 3 for Equations (1) - (4), for example. Thus, the resultant terms will be CDS: 7; Shutter: 107, 108, 109, 110; CDS: 1, Shutter: 4, 5, 6, 7. For the 2^nd pointer shutter operation, a total of 4 rows are being reset sequentially: 4, 5, 6, and 7. Among those 4 sequential shutter operations, the 1^st one (for row 4) is used to start integration of row 4 for the 2^nd pointer. Additional bloom purge shutter 5, 6, 7 are used to provide blooming protection as has been described. If a high intensity light impinges on physical rows 5, 6, and 7, the intensity of the light spot can cause the pixel to reach saturation after 5 rows of integration time. Once rows 5, 6, and 7 are shuttered (reset), they change from a saturation state to an unsaturation state. For the same light intensity as used in the example of FIG. 4, rows 5, 6, and 7 will not reach saturation and blooming during row 4's integration for 2^nd pointer or second integration time. Thus, the integration for row 4 will not be corrupted by blooming. Since the blooming purge count is set to 3 (p=3), the additional sequential shutter count is not more than 3. For example, sequential shutters can include 4, 5, or 4, 5, 6 in this case. Notably, depending on the scene, additional blooming purge shutters can provide better blooming protection. However, one caveat of using longer shutter durations can be, in some cases, a negative impact on frame rate depending on the imager readout architecture and how the shutter operation is applied. If the shutter operation is applied in parallel during ADC conversion and data readout, additional blooming purge shutter may not affect the frame rate. Since every row will be reset multiple times on average, signal levels of extremely hot pixels will also be reduced by the additional bloom purge shutters.

It is noted that in CMOS imager design, a register can be used for additional bloom purge shutter count. The value of this blooming purge count can be controlled by the user directly or other by image processing algorithms that detect blooming issues on the image. The row driver control logic can also be designed based on Equations (l)-(4) as shown above.

Furthermore, in one aspect the additional bloom purge shutter can be applied sequentially to each pointer's shutter operation. However, the application of the additional bloom purge shutter should apply only to a particular pointer operation. This can be beneficial if the additional blooming purge shutter has a negative impact on frame rate. For example, for a 2-pointer row-wise HDR image sensor, if 100 row, and 3 row, adding additional bloom purge shutter for the 1^st pointer operation might not be necessary since the integration time is already 100 rows. The additional purge shutter can be applied to the 2^nd pointer in order to correctly expose the brighter portions of a scene. For a 3-pointer row-wise HDR image sensor, for example, if 2 rows, and an intense light spot saturates the pixels with a 5 row integration time, the additional blooming purge shutter count of 3 can be applied only to the 3^rd pointer shutter operation.

As examples, 3-pointer row-wise image sensor readout and shutter algorithms for a third integration time are shown in Equations (5)-(10). Similar equations can readily be derived therefrom for fourth, fifth, and greater integration times.

-1 1

'shutter ~ ' ~ ^Liirnt (6) _ i

shutter = τ^ι - ρ, rj_h^_tter = r^l - p + 1, ... , r_s 2J5-1

shutter (8)

As another example and moving to the timing scheme where row 4 is read out for the 1^st pointer, using the equations above, the CDS and shutter sequence can be calculated by setting p = 0. The resultant parameters are the following: (CDS: 4; Shutter: 104; CDS: 1; Shutter: 4). The raw data from row 4 is read out; then a shutter is applied to rows 104, 105, 106, and 107; data from row 1 is read out, noting that the data readout of row 1 is after a 3 row integration time; and subsequently a shutter operation is applied to rows 4, 5, 6 and 7. It should be pointed out that row 4 will not be read out again until row 7 is read out. Equations (1) - (4) can be applied to the above example as follows; after row 4 is read out for first pointer operation, shutter is applied to row 104 (i.e. r_c (4) + t^_nt(100)) for first point operation. Subsequently, row 1 is readout for 2^nd pointer operation and a shutter is applied to row 4 (i.e. r_s ² _¾utter = rCDS ~ V = 4— 0 = 4) for the 2^nd pointer operation. Since row 1 is a valid row address, its data will be output from the imager. It should be noted that, after readout of row 4 for the 1^st pointer, the image data will be in a row-wise interlaced pattern in which an odd row will be associated with one integration time and even rows will be associated with a different integration time. After the above operation, row 4 starts integration for its second integration time. If a high intensity light spot is being integrated by rows 5, 6, and 7, the intensity of this light spot and can cause the pixel to reach saturation after about five (5) rows of integration time. Since rows 5, 6, and 7 are still in their first pointer operation (which has integration time of 100 rows), all those rows are saturated (since those rows previously integrated 99, 98, 97 rows already for physical rows 5, 6, 7, respectively) and start blooming into their neighboring rows. Therefore, the integrated signal of row 4 for the 2^nd pointer will likely be corrupted from blooming from row 5, 6, and 7 (depending on the pixel size and doping conditions). Notably, pixels of the present technology can bloom into neighboring rows since row 5, 6, and 7 are not idle even if the image sensor is running a true "anti-blooming" timing. In a typical so-called "anti-blooming" timing, an idle row's reset (RST) and transfer (TX) gate are set to high and remain high during the entire idle period to provide blooming protection. An idle row refers to a physical row in the sensor array that is not in integration or in shutter operation. Additionally, idle rows could include all those rows that are outside the addressed image output windows. For example, in a sensor array with 100 rows, and the image output window is the center 50 rows, then the top 25 rows and the bottom 25 rows can be considered in an idle row operation or mode. To implement "anti-blooming" timing, a latch-based row decoder design is typically required. Therefore, an image sensor running such "anti-blooming" timing cannot protect the rows in a second pointer from blooming corruption.

Various benefits of the present technology include the reduction in die size due to not needing a latch-based row driver, flexible to handle 2-pointer, 3 -pointer, or more pointer row-wise HDR operation, the reduction in hot pixel impact on final images since the pixels will reset multiple times in average, among other things.

The present scope additionally includes systems and devices utilizing the present technology. Such should include any CMOS device, system, architecture, and/or design that allows HDR imaging using a rolling shutter. As has been described, those skilled in the art can readily design and produce systems and devices incorporating the present technology once in possession of the present disclosure. In some cases, this technology can be implemented into current HDR imagers using minimal row driver digital design modifications.

The following are exemplary of various aspects of the present disclosure. In example 1, a method of providing blooming protection to a CMOS imager having a pixel array of a plurality of pixels arranged in rows and columns, where the CMOS imager is operable to capture high dynamic range images using a rolling shutter, is provided. Such a method can include selecting a readout row of pixels, starting a first integration time of the pixels in the readout row, reading out charge accumulated by the pixels in the readout row to obtain a first readout, applying a reset to the readout row for a reset time sufficient to allow readout and reset to occur in at least one subsequent row, removing the reset and starting a second integration time of the pixels in the readout row, wherein the second integration time is shorter than the first integration time, and wherein the at least one subsequent row is a sufficient number of rows to have a combined reset to at least substantially preclude blooming effects from the pixel array in the readout row during the second integration time, and reading out charge accumulated by the pixels in the readout row to obtain a second readout.

In another example, the method of example 1 can be repeated on at least one subsequent row.

In another example, the at least one subsequent row is at least substantially all pixel rows in the pixel array.

In another example, the method of example 1 can be repeated on at least substantially all pixel rows in a sequential order.

In another example, the method of example 1 can be repeated on at least substantially all pixel rows in a non-sequential order.

In another example, the first readout and the second readout are combined to form a high dynamic range image.

In another example, the method of example 1 can further include applying the reset to the readout row following the second readout for a reset time sufficient to allow readout and reset to occur in at least one subsequent row, removing the reset and starting a third integration time of the pixels in the readout row, wherein the third integration time is shorter than the second integration time, and wherein the at least one subsequent row is a sufficient number of rows to have a combined reset to at least substantially preclude blooming effects from the pixel array in the readout row during the third integration time, and reading out charge accumulated by the pixels in the readout row to obtain a third readout.

In another example, the method can be repeated for a fourth or more integration time.

In another example, starting the first integration time further includes applying the reset to the readout row and releasing the reset from the readout row.

In another example, applying the reset to the readout row for the reset time sufficient to allow readout and reset to occur in at least one subsequent row includes applying the reset at a continuous voltage level throughout the reset time duration.

In another example, applying the reset to the readout row for the reset time sufficient to allow readout and reset to occur in at least one subsequent row includes applying the reset at a variable voltage level throughout the reset time duration. In another example, the reset is applied for the reset time sufficient such that at least two subsequent rows and the readout row are simultaneously reset.

In another example, the reset is applied for the reset time sufficient such that at least three subsequent rows and the readout row are simultaneously reset.

In another example, the readout row and the at least three subsequent rows are sequentially adjacent.

In another example, the readout row and the at least three subsequent rows are sequentially non-adjacent.

In another example, the reset time sufficient to allow readout and reset to occur in at least one subsequent row is from about 10 nanoseconds to about 50 microseconds.

In another example, the first integration time is from about 1 millisecond to about 1 second and the second integration time is from about 10 nanoseconds to about 100 milliseconds.

In another example, a method of using a rolling shutter to provide blooming protection in a CMOS imager in high dynamic range mode and having a pixel array of a plurality of pixels arranged in rows and columns can include capturing a first image in the pixel array having a first integration time, reading the first image out of the pixel array sequentially by rows using at least one readout row of the first image, hard resetting a plurality of rows in proximity to the at least one readout row of the first image, capturing a second image in the pixel array having a second integration time, the second integration time being shorter than the first integration time, wherein the second image is protected from blooming caused by the first image by the hard resetting of the plurality of rows in proximity to the readout row of the first image, and reading the second image out of the pixel array sequentially by rows using at least one readout row of the second image.

In another example, the first image and the second image are combined to form a high dynamic range image.

In another example, the method can further include hard resetting a plurality of rows in proximity to the at least one readout row of the second image, capturing a third image in the pixel array having a third integration time, the third integration time being shorter than the second integration time, wherein the third image is protected from blooming caused by the first or the second image by the hard resetting of the plurality of rows in proximity to the readout row of the second image, and reading the third image out of the pixel array sequentially by rows using at least one readout row of the third image.

In another example, the first image, the second image, and the third image are combined to form a high dynamic range image.

Of course, it is to be understood that the above-described arrangements are only illustrative of the application of the principles of the present disclosure.

Numerous modifications and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of the present disclosure and the appended claims are intended to cover such modifications and arrangements. Thus, while the present disclosure has been described above with particularity and detail in connection with what is presently deemed to be the most practical embodiments of the disclosure, it will be apparent to those of ordinary skill in the art that numerous modifications, including, but not limited to, variations in size, materials, shape, form, function and manner of operation, assembly and use may be made without departing from the principles and concepts set forth herein.

Claims

What is claimed is: 1. A method of providing blooming protection to a CMOS imager having a pixel array of a plurality of pixels arranged in rows and columns, where the CMOS imager is operable to capture high dynamic range images using a rolling shutter, comprising: selecting a readout row of pixels;

starting a first integration time of the pixels in the readout row;

reading out charge accumulated by the pixels in the readout row to obtain a first readout;

applying a reset to the readout row for a reset time sufficient to allow readout and reset to occur in at least one subsequent row;

removing the reset and starting a second integration time of the pixels in the readout row, wherein the second integration time is shorter than the first integration time, and wherein the at least one subsequent row is a sufficient number of rows to have a combined reset to at least substantially preclude blooming effects from the pixel array in the readout row during the second integration time; and

reading out charge accumulated by the pixels in the readout row to obtain a second readout.

2. The method of claim 1, further comprising repeating the method of claim 1 on at least one subsequent row.

3. The method of claim 2, wherein the at least one subsequent row is at least substantially all pixel rows in the pixel array.

4. The method of claim 3, wherein the method is repeated on at least substantially all pixel rows in a sequential order.

5. The method of claim 3, wherein the method is repeated on at least substantially all pixel rows in a non-sequential order.

6. The method of claim 1, wherein the first readout and the second readout are combined to form a high dynamic range image.

7. The method of claim 1, further comprising:

applying the reset to the readout row following the second readout for a reset time sufficient to allow readout and reset to occur in at least one subsequent row; removing the reset and starting a third integration time of the pixels in the readout row, wherein the third integration time is shorter than the second integration time, and wherein the at least one subsequent row is a sufficient number of rows to have a combined reset to at least substantially preclude blooming effects from the pixel array in the readout row during the third integration time; and

reading out charge accumulated by the pixels in the readout row to obtain a third readout.

8. The method of claim 7, repeated for a fourth or more integration time.

9. The method of claim 1, wherein starting the first integration time further includes applying the reset to the readout row and releasing the reset from the readout row.

10. The method of claim 1, wherein applying the reset to the readout row for the reset time sufficient to allow readout and reset to occur in at least one subsequent row includes applying the reset at a continuous voltage level throughout the reset time duration.

11. The method of claim 1, wherein applying the reset to the readout row for the reset time sufficient to allow readout and reset to occur in at least one subsequent row includes applying the reset at a variable voltage level throughout the reset time duration.

12. The method of claim 1, wherein the reset is applied for the reset time sufficient such that at least two subsequent rows and the readout row are simultaneously reset.

13. The method of claim 1, wherein the reset is applied for the reset time sufficient such that at least three subsequent rows and the readout row are simultaneously reset.

14. The method of claim 13, wherein the readout row and the at least three subsequent rows are sequentially adjacent.

15. The method of claim 13, wherein the readout row and the at least three subsequent rows are sequentially non-adjacent.

16. The method of claim 1, wherein the reset time sufficient to allow readout and reset to occur in at least one subsequent row is from about 10 nanoseconds to about 50 microseconds.

17. The method of claim 1, wherein the first integration time is from about 1 millisecond to about 1 second and the second integration time is from about 10 nanoseconds to about 100 milliseconds.

18. A method of using a rolling shutter to provide blooming protection in a CMOS imager in high dynamic range mode and having a pixel array of a plurality of pixels arranged in rows and columns, comprising:

capturing a first image in the pixel array having a first integration time;

reading the first image out of the pixel array sequentially by rows using at least one readout row of the first image;

hard resetting a plurality of rows in proximity to the at least one readout row of the first image;

capturing a second image in the pixel array having a second integration time, the second integration time being shorter than the first integration time, wherein the second image is protected from blooming caused by the first image by the hard resetting of the plurality of rows in proximity to the readout row of the first image; and

reading the second image out of the pixel array sequentially by rows using at least one readout row of the second image.

19. The method of claim 18, wherein the first image and the second image are combined to form a high dynamic range image.

20. The method of claim 18, further comprising:

hard resetting a plurality of rows in proximity to the at least one readout row of the second image;

capturing a third image in the pixel array having a third integration time, the third integration time being shorter than the second integration time, wherein the third image is protected from blooming caused by the first or the second image by the hard resetting of the plurality of rows in proximity to the readout row of the second image; and

reading the third image out of the pixel array sequentially by rows using at least one readout row of the third image.

21. The method of claim 18, wherein the first image, the second image, and the third image are combined to form a high dynamic range image.