Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/50—Control of the SSIS exposure
  • H04N25/57—Control of the dynamic range
  • H04N25/58—Control of the dynamic range involving two or more exposures
  • H04N25/587—Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields
  • H04N25/589—Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields with different integration times, e.g. short and long exposures
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/50—Control of the SSIS exposure
  • H04N25/57—Control of the dynamic range
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/70—SSIS architectures; Circuits associated therewith
  • H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
  • H04N25/77—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components

Definitions

• the disclosure relates to image sensors and, in particular, techniques for producing high dynamic range images.
• the dynamic range of a scene is determined by a ratio of the highest and lowest light levels of the scene.
• the dynamic range of a sensor is determined by a ratio of the highest light level in a scene that does not saturate the sensor to the lowest light level, i.e., above a noise floor, detectable by the sensor.
• the dynamic range of a scene may be, for example, 100,000:1, while the dynamic range of a sensor may be, for example, 100: 1.
• Exposure may be adjusted in a manner such that the dynamic range of a sensor covers as much useful light level as possible. Exposure cannot increase the dynamic range of the sensor. However, combining two or more different exposures of a scene may permit the capture of a higher dynamic range of the scene then would otherwise be possible by a single exposure.
• this disclosure describes techniques for producing high dynamic range images by applying a variable weighting factor to a sample prior to combining the sample with another sample.
• An image sensor may sample a pixel cell signal at two or more different exposure times in order to produce two or more samples that may be combined to produce a high dynamic range image.
• a long exposure time may cause a photosensitive semiconductor of the pixel cell to become saturated.
• a variable weighting factor may be applied to the signal produced by the long exposure time in order to reduce or eliminate the contribution of the saturated photosensitive semiconductor.
• the disclosure is directed to a method comprising sampling a first pixel cell signal at a first time to produce a first sample, sampling a second pixel cell signal at a second time to produce a second sample, applying a variable weighting factor to the second sample, wherein the variable weighting factor is defined based on a function, and combining the first sample and the weighted second sample.
• the disclosure is directed to a system comprising an image sensor comprising a plurality of pixel cells and a controller that samples, from at least one of the plurality of pixel cells, a first pixel cell signal at a first time to produce a first sample and samples, from the at least one of the plurality of cells, a second pixel cell signal at a second time to produce a second sample.
• the system further comprises a processor that applies a variable weighting factor to the second sample, wherein the variable weighting factor is defined based on a function and combines the first sample and the weighted second sample.
• the disclosure is directed to a system comprising means for sampling a first pixel cell signal at a first time to produce a first sample, means for sampling a second pixel cell signal at a second time to produce a second sample, means for applying a variable weighting factor to the second sample, wherein the variable weighting factor is defined based on a function, and means for combining the first sample and the weighted second sample.
• the disclosure is directed to a computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to sample a first pixel cell signal at a first time to produce a first sample, sample a second pixel cell signal at a second time to produce a second sample, apply a variable weighting factor to the second sample, wherein the variable weighting factor is defined based on a function, and combine the first sample and the weighted second sample.
• FIG. 1 is a schematic diagram of an example pixel cell that may be used to implement the techniques of this disclosure.
• FIG. 2 is an example graph of a combing curve that may be used in a combining scheme in accordance with the techniques of this disclosure.
• FIG. 3 is a block diagram of an image sensor that may be used to implement the techniques of this disclosure.
• FIG. 4 is a block diagram of another image sensor that may be used to implement the techniques of this disclosure.
• FIG. 5 is a block diagram of another image sensor that may be used to implement the techniques of this disclosure.
• FIG. 6 depicts a conceptual diagram illustrating the combination of short and long exposure time samples from even and odd-numbered lines in accordance with the techniques of this disclosure.
• FIG. 7 is a flow diagram illustrating an example method that may be used to implement the techniques of this disclosure.
• FIG. 8 is a flow diagram illustrating another example method that may be used to implement the techniques of this disclosure.
• FIG. 9 is a flow diagram illustrating another example method that may be used to implement the techniques of this disclosure.
• FIG. 10 is a flow diagram illustrating another example method that may be used to implement the techniques of this disclosure.
• This disclosure describes techniques for producing high dynamic range (HDR) images, e.g., still images and video images, by applying a variable weighting factor to a sample prior to combining the sample with another sample.
• a HDR image refers to an image that has a dynamic range, e.g., a ratio of the highest intensity light level relative to the lowest intensity light level in the image, greater than that of image produced by standard imaging techniques.
• One example HDR imaging technique combines images captured with different exposure times to create a combined image with a greater dynamic range then that of an image produced from either of the exposure times.
• This disclosure provides techniques that apply a variable weighting factor to one of the images captured with different exposure times prior to combining the images in order to produce a HDR image.
• FIG. 1 is a schematic diagram of an example picture element ("pixel") cell that may be used to implement the techniques of this disclosure.
• example pixel cell 10 comprises photosensitive semiconductor 12, switches 14, 16, and 18, capacitor 20, and transistors 22, 24. Switches 14, 16, and 18 may be transistors, for example.
• a capacitance is associated with photosensitive semiconductor 12, as indicated by equivalent capacitor 28 (shown by dashed lines).
• the cathode of photosensitive semiconductor 12 is connected to voltage source VDD through switch 14, and the anode of photosensitive semiconductor 12 is connected to reference voltage Vss, which may be ground or other potential less than voltage source VDD-
• Vss reference voltage
• photosensitive semiconductor 12 is reverse-biased because its cathode at node 30 is at a higher potential than its anode.
• the gate of transistor 22 is connected to the cathode of photosensitive semiconductor 12 through switch 16.
• the gate of transistor 22 is also connected to capacitor 20 through switch 18.
• the drain of transistor 22 is connector to voltage source VDD, and the source of transistor 22 is connected to the drain of transistor 24.
• Gate 26 of transistor 24 is connected to row select driver circuitry, as described in more detail below.
• the source of transistor 24 is connector to pixel cell output line 32.
• transistors 22 and 24 are depicted as enhancement-mode metal-oxide-semiconductor field-effect transistors (MOSFETs) without their respective bulks attached, numerous other configurations are possible.
• transistors 22 and 24 are described below as n-MOS type transistors, in other example configurations, transistors 22 and 24 may be p-MOS type transistors.
• Photosensitive semiconductor 12 e.g., a photodiode
• Photosensitive semiconductor 12 is a device that converts photons from incident light into electrical charge.
• photons from the incident light cause the photosensitive semiconductor to generate charge carriers within a region of the photosensitive semiconductor.
• the amount of charge carriers generated is proportional to the amount of incident light. As light intensity increases, the amount of charge carriers generated increases.
• the photosensitive semiconductor may become "saturated" if the amount of charge carriers can no longer increase as the amount of incident light increases. It should be appreciated that once a photosensitive semiconductor becomes saturated, it is no longer an accurate indicator of the amount of incident light. As such, the saturated photosensitive semiconductor has little information of value.
• photosensitive semiconductor 12 e.g., a photodiode
• pixel cell 10 Prior to exposure to light, pixel cell 10 may be reset to an initial voltage.
• photosensitive semiconductor 12 and its equivalent capacitance 28 may be reset to an initial voltage of V DD , for example.
• reset switch 14 may be closed in order to reset pixel cell 10 of FIG. 1. Closing reset switch 14 connects node 30 and, in particular, photosensitive semiconductor 12 and its equivalent capacitor 28, to supply voltage V DD - Although FIG.
• pixel cell 10 may include a second voltage, similar to or different from supply voltage V DD that acts as a dedicated reset voltage for pixel cell 10.
• Closing reset switch 14 drains charge out of photosensitive semiconductor 12 and charges equivalent capacitor 28 to known voltage V DD -
• switches 16 and 18 are closed at reset, which can cause capacitor 20 to become charged to known voltage VDD-
• photosensitive semiconductor 12 begins converting photons to charge upon exposure to light,.
• photosensitive semiconductor 12 Upon exposure to light, i.e., at the beginning of its integration time, photosensitive semiconductor 12 begins generating charge carriers. Because photosensitive semiconductor 12 is reverse-biased, photocurrent will begin to flow from the cathode to the anode of photosensitive semiconductor 12. The photocurrent is proportional to the amount of incident light. As light intensity increases, the amount of charge carriers generated by photosensitive semiconductor 12 increases, thereby increasing the photocurrent. The photocurrent begins to discharge equivalent capacitor 28.
• Pixel cell 10 may be used to sample a first pixel cell signal at a first time, e.g., a short exposure time, and a second pixel cell signal at a second time, e.g., a long exposure time.
• pixel cell 10 may sample a first signal from photosensitive semiconductor 12 by closing switches 16 and 18.
• a controller e.g., controller 54 of FIG. 3 closes switches 16 and 18 to transfer the short exposure time charge from photosensitive semiconductor 12 and, in particular, its equivalent capacitor 28, to capacitor 20.
• a controller e.g., controller 54 of FIG. 3 opens switches 16 and 18, thus capturing via capacitor 20 a voltage representing the amount of light incident upon photosensitive semiconductor 12 during time T S hort- In this manner, capacitor 20 collects a short exposure time sample.
• a long exposure time sample is represented by the voltage remaining on equivalent capacitor 28.
• the long exposure time sample may be read out by closing switch 16 and applying a row select signal to gate 26 of transistor 24 (described in more detail below).
• a controller e.g., controller 54 of FIG. 3 closes switch 16 and applies a row select signal to gate 26 of transistor 24 to turn on transistors 22 and 24.
• a voltage representing the long exposure time sample is then read out from pixel cell 10 and, in particular, equivalent capacitor 28, via line 32 through transistors 22 and 24.
• the short exposure time sample stored in capacitor 20 may be read out.
• the short exposure time sample may be read out by opening switch 16, closing switch 18, and applying a row select signal to gate 26 of transistor 24. Closing switch 18 and applying a row select signal to gate 26 of transistor 24 turns on transistors 22 and 24.
• the voltage stored on capacitor 30, which represents the short exposure time sample, is then read out from pixel cell 10 via line 32. The voltage that is read out is approximately equal to the voltage across capacitor 30 minus the threshold voltages of transistors 22 and 24.
• CMOS complimentary MOS
• FPN fixed pattern noise
• a controller for example, controller 54 of FIG. 3 on the image sensor may perform correlated double sampling by closing switch 14, thereby resetting pixel cell 10 to reference voltage VDD, and then sampling the output level of the pixel cell after the pixel cell has been reset.
• the controller may reset, sample, and read out the reset value of a pixel cell, e.g. pixel cell 10. Then, the controller may subtract the pixel cell's reset value from the value of the signal sample in order to provide a more accurate representation of the signal.
• the short exposure time sample and the long exposure time sample may be combined by a processor after applying a variable weighting factor to the long exposure time sample.
• a variable weighting factor that is defined by a function at the value of the long exposure time sample, the dynamic range provided by the two different exposure times may be improved.
• the contribution of the long exposure time sample may be reduced or eliminated by the weighting factor during the combination of the short exposure time sample and the long exposure time sample.
• a photosensitive semiconductor becomes saturated, it is no longer an accurate indicator of the amount of incident light. As such, the saturated photosensitive semiconductor has little information of value.
• a pixel cell is unlikely to saturate. Hence, the short exposure time sample likely has more valuable information than the saturated, or overexposed, long exposure time sample.
• M(i, j) C(S LO NG(i, j))
• S C oMBiNED(i, j) S L oN G (i, j)*M(i, j) + n*S SHO RT(i + H(i, j), j + V(i, j))*(l- M(i, j))
• SCOMBINED is the combination of the short exposure time sample SSHORT(I, j) and the long exposure time sample SLONG(I, j) for the pixel cell located at (i, j)
• C( ) is a combining curve that begins at a value of one and decreases to a value of zero as long exposure time sample SLONG(I, j) becomes saturated
• M(i, j) is a scalar value of the long exposure time sample SLONG(I, j) weighted by C( )
• n is scalar value that is dependent on the difference in the time of exposure between the short exposure time (TSHORT) and the long exposure time (TLONG
• ScoMBiNED(i, j) S LO N G (i, j)*M(i, j) + n*S S HOR T (i, j)*(l- M(i, j))
• the long exposure time sample SLONG(I, j) in equations (2) and (3) may be modified by the scalar value n.
• the scalar value n may comprise the ratio of the short exposure time and the long exposure time.
• the scalar value n may be used in order to normalize the long exposure time sample SLONG(I, j) and the short exposure time sample SSHORT(I, j) by bringing the two samples to a common base.
• the combining scheme described by equations (2) and (3) adjusts the value n so that objects appear at the same level in both images prior to their combination.
• a processor may multiply the short exposure time sample SSHORT by the ratio of exposure times, e.g., TLONG / TSHORT, to normalize the short exposure time sample SSHORT in order to account for the difference in exposure time.
• the "noise floor" of the pixel cell i.e., the low- level signal produced by pixel cells in the absence of incident light, is a factor in reducing dynamic range.
• the short exposure time sample SSHORT(I, j) and long exposure time sample SLONG(I, j) may be combined in a signal-to-noise (SNR) efficient manner. That is, if a sample has a value below the SNR of the pixel cell itself, the sample may be discarded.
• SNR signal-to-noise
• FIG. 2 is an example graph of a combining curve, namely C ( ), that may be used in a combining scheme in accordance with the techniques of this disclosure.
• the x-axis of the graph depicted in FIG. 2 represents the luma value of the pixel cell located at a spatial location (i, j) on the pixel cell array after a long exposure time.
• the y-axis of the graph depicted in FIG. 2 represents the weighted value, M(i, j), of SLONG(I, j) as defined by combining curve C( ), shown generally at 40.
• the value of SLONG(I, j) is zero. If the pixel cell located at (i, j) is saturated after a long exposure time, then the value of SLONG(I, j) is at a maximum, e.g., 255 for an eight-bit implementation. As seen in FIG. 2, as the value of SLONG(I, j) increases, the value of M(i, j) remains at a value of "1" until an SLONG(I, j) value of about 210.
• the combining curve C( ) begins to decrease sharply between the SLONG(I, j) values of about 210 and about 230 until the weighted value M(i, j) reaches a value of "0" at an SLONG(I, j) value of about 230.
• the weighted value M(i, j) may be a value between "0" and "1.” For example, at an SLONG(I, j) value of about 220, the weighted value M(i, j) is about 0.5.
• the example combining curve C ( ), depicted at 40 in FIG. 2 includes three segments, 42, 44 and 46. In other examples, combining curve C ( ) may have more, or fewer segments. As seen in FIG. 2, combining curve C ( ) may generally be considered a decaying function as the value of the long exposure time sample SLONG(I, j) increases. In other examples, combining curve C ( ) may decay less sharply.
• the dynamic range of the sensor can be made variable and programmable. Every pixel cell may have a different weighted value M(i, j) that varies according to the value of the long exposure time sample value SLONG(I, j) for that particular pixel cell. Unlike other image sensors in which the dynamic range of the sensor is fixed based upon values of capacitors used to construct the sensor, using the techniques of this disclosure the dynamic range is adaptable, e.g., for a particular scene.
• f-stops are a measure of dynamic range for digital cameras
• the exposure times may be adjusted to provide four f-stops, and then the two samples from the two exposure times may be combined.
• a combining curve may be generated according to one or more of the following three guidelines.
• the combining curve may be image level dependent. Any of the two images may be used with comparable results but the two curves will differ.
• the combining curve may weight less noisy images more heavily than more noisy images.
• the long exposure time sample, SLONG which likely has less noise than short exposure time sample SSHORT(I, j), may be weighted more heavily than SSHORT- In some examples, however, equal gain combining may be used, rather than weighted one image more heavily than another.
• the combining curve may be monotonically decreasing.
• prior knowledge may be used to model noise statistics based on the image level and utilize maximum ratio combining where each image is scaled inversely proportional to its noise levels.
• an enhanced version of the first example methodology may be used for which the transition from 1 to 0 is made more gradual.
• One combining curve C(L) may be generated using the following:
• L is the luma value associated with SLONG(I, j), e.g., a combination of R, G, B components like (R+G+B)/2, max(R,G,B), or the like and having a value of 0-255 in an 8-bit implementation
• the dynamic range of the imaging sensor may be reduced to improve the noise characteristics of the sensor. Collecting and combining two samples of approximately equal exposure times, i.e., TLONG is approximately equal to TSHORT, may reduce noise. When combined, the noise can be averaged between the two samples, and thus may be significantly reduced.
• FIG. 3 is a block diagram of an example image sensor that may be used to implement the techniques of this disclosure.
• image sensor 50 of FIG. 3 is a complimentary MOS (CMOS) image sensor
• CMOS complimentary MOS
• the signal combining techniques described in this disclosure are applicable to other image sensors as well, including, for example, charge-coupled device (CCD) image sensors.
• CCD charge-coupled device
• image sensor 50 includes pixel cell array 52 comprising numerous pixel cells 10.
• Pixel cell array 52 may be arranged in M rows and N columns of pixel cells 10, as shown in FIG. 3.
• Each pixel cell 10 of pixel cell array 52 may be addressed by a row and column address.
• controller 54 provides row address decoder 56 with a digital address of the row of the desired pixel cell, e.g., row M.
• Row address decoder 56 decodes the received row address and outputs a "row select" signal to row driver 58 to apply a voltage to row M, thereby selecting each pixel cell 10 in row M, e.g., by applying a voltage on gate 26 of transistor 24 in FIG. 1.
• Controller 54 also provides column address decoder 60 with a digital address of the column of the desired pixel cell, e.g., column N.
• Column address decoder 60 decodes the received column address and outputs a "column select” signal to column diver 62 to apply a voltage to column N, thereby selecting each pixel cell 10 in column N. In this manner, pixel cell 10 at row M, column N may be selected and its voltage read out.
• Controller 54 may include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, or other processor arrangements suitable for processing computer code to facilitate the execution of the functions described in this disclosure.
• DSPs digital signal processors
• ASICs application specific integrated circuits
• FPGAs field programmable gate arrays
• discrete logic or other processor arrangements suitable for processing computer code to facilitate the execution of the functions described in this disclosure.
• the voltage of pixel cell 10 at row M, column N is sampled and held by sample and hold (S/H) circuitry 70.
• S/H sample and hold
• Processing such as black level compensation, noise reduction, e.g., temporal noise and FPN, and amplification is performed on analog signals by processing circuitry 72.
• the analog signals from pixel cells 10 are quantized to digital values via analog-to-digital converter (ADC) 74.
• ADC analog-to-digital converter
• the digital values are then input into processor 76.
• Processor 76 may combine, for each pixel cell, the short exposure time sample SSHORT and the long exposure time sample SLONG, in accordance with the techniques described in this disclosure, with particular reference to equations (l)-(3) above.
• processor 76 may execute instructions in order to perform the functions described in this disclosure.
• the instructions may be stored in memory 78.
• Processor 76 may include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, or other processor arrangements suitable for processing computer code to facilitate the execution of the functions described in this disclosure.
• DSPs digital signal processors
• ASICs application specific integrated circuits
• FPGAs field programmable gate arrays
• discrete logic discrete logic
• the executable instructions stored in memory 78 may comprise computer-readable instructions encoded in a computer-readable medium such as random access memory (RAM), synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like.
• RAM random access memory
• SDRAM synchronous dynamic random access memory
• ROM read-only memory
• NVRAM non-volatile random access memory
• EEPROM electrically erasable programmable read-only memory
• FLASH memory magnetic or optical data storage media, and the like.
• processor 76 receives the digitized pixel cell values for pixel cell 10 at row M, column N.
• processor 76 receives a digital value for long exposure time sample SLONo(m, n) and a digital value for short exposure time sample SsHORT(m, n) from pixel cell (m, n).
• processor 76 may combine long exposure time sample SLONo(m, n) and a digital value for short exposure time sample SsHORT(m, n) from pixel cell (m, n).
• Processor 76 may calculate, obtain, or otherwise acquire variable weighting factor M(m, n) for long exposure time sample SLONo(m, n), where M(m,n) is determined by evaluating C( ) at SLONo(m, n), or C(SLONo(ni, n)).
• combining curve C( ) may be defined by one or more equations. Each equation defining a combining curve C ( ) may, for example, be dependent upon the desired dynamic range, e.g., f-stop value. For example, a user may have selected a certain f-stop value to achieve a desired dynamic range, which in turn corresponds to a specific combining curve C ( ) that may be retrieved from memory 78.
• processor 76 may determine the value of variable weighting factor M(m, n) by evaluating a particular equation that defines combining curve C ( ), e.g., retrieved from memory 78, for the desired dynamic range at the value of SLONo(m, n). For example, assume that the pixel cell 10 was saturated at long exposure time sample SLONo(m, n). Regardless of the number of bits in the implementation, if the combining curve depicted in FIG. 2 is used by processor 76, then the resulting variable weighting factor M(m, n) is zero.
• variable weighting factor M(m, n) is zero, as determined by evaluating combining curve C ( ), or an equation defining combining curve C ( ), at long exposure time sample, S ( , n).
• the combined value S is simply the value of the short exposure time sample, S , adjusted by the ratio of the two exposure times (T T T
• S C oMBiNED(m, n) S , n)*l + n*S (i, j)*0.
• the combined value S is the value of the long exposure time sample, S , which likely has less noise than short exposure time sample S ( , j)- [0059]
• the pixel cell 10 was not saturated at long exposure time sample S ( , n) and, instead, long exposure time sample S ( , n) had a luma value of 220 for an 8-bit implementation. If the combining curve depicted in FIG.
• variable weighting factor M(m, n) is 0.5, as seen by the fact that a luma value of 220 for S ( 3 ⁇ 4 n) intersects combining curve C ( ) in segment 44 at a variable weighting factor M(m, n) value of 0.5.
• the variable weighting factor M(m, n) may be 0.5, as determined by evaluating combining curve C ( ), or an equation defining combining curve C ( ), at long exposure time sample, S , n).
• each combining curve C ( ) may be stored in memory, e.g., memory 78, in the form of one or more lookup tables.
• processor 76 may access a particular lookup table corresponding to a specific combining curve C ( ) stored in memory 78 and, for a given value of S ( , j), retrieve the value of variable weighting factor M(i, j).
• FIG. 4 is a block diagram of another image sensor that may be used to implement the techniques of this disclosure. Some aspects of the example image sensor 100 depicted in FIG. 4 are similar to the image sensor depicted in FIG. 3 and will not be described in detail again. In contrast to FIG. 3, FIG. 4 depicts two ADCs 74A, 74B. For simplicity, ADC 74 of FIG. 3 is represented in FIG. 4 as 74 A and 74B.
• ADC 74 A is a dedicated ADC for converting short exposure time sample S ( , j) and ADC 74B is a dedicated ADC for converting long exposure time sample S ( , j)- Following the conversion of short exposure time sample S ( , j) and long exposure time sample S ( , j) by ADC 76A and ADC 76B, respectively, the outputs of ADC 74A and ADC 74B are input into processor 76.
• processor 76 Upon receiving the converted samples, processor 76 combines the samples in accordance with the techniques described above with particular reference to equations (l)-(3).
• An example image sensor configuration that uses two ADCs, as shown in FIG. 4, may be useful for implementing the techniques of this disclosure because, as described in detail above, two samples, namely short exposure time sample S ( , j) and long exposure time sample S ( , j) are read out for each pixel cell.
• An image sensor configuration with parallel ADCs, one for converting short exposure time samples and another for converting long exposure time samples, may improve throughput by allowing the short exposure time sample and the long exposure time sample for each pixel cell to be converted substantially simultaneously.
• the example image sensor configuration of FIG. 4 may require additional circuitry as compared to the image sensor configuration depicted in FIG.
• such additional circuitry may, in some implementations, e.g., CMOS backside illumination configurations, be added behind the pixel cell without interfering with the light capturing surface area of the pixel cell. In other words, the additional circuitry will not reduce the pixel cell's fill factor, i.e., the area of the photosensitive semiconductor relative to the area of the pixel cell.
• FIG. 5 is a block diagram of another image sensor that may be used to implement the techniques of this disclosure. Some aspects of the example image sensor 120 depicted in FIG. 5 are similar to the image sensor depicted in FIG. 3 and will not be described in detail again.
• the image sensor in FIG. 5 is similar to the image sensor in FIG. 4 in that two ADCs, one for converting short exposure time sample S ( , j) and another for converting long exposure time sample S ( , j), are used in parallel, which may improve the throughput to processor 76.
• parallel ADCs 74A and 74B the example configuration depicted in FIG.
• FIG. 5 further includes parallel sample and hold circuitry (70A and 70B) and parallel black level compensation, noise reduction, and amplification circuitry (72A and 72B), which may, in some instances, further improve signal throughput to processor 76.
• the configuration depicted in FIG. 5 includes switch 80 for directing short exposure time sample S ( , j) for processing through the circuitry generally indicated at 82, and directing long exposure time sample S ( , j) for processing through the circuitry generally indicated at 84. Following the conversion of short exposure time sample S ( , j) and long exposure time sample S , j) by ADC 74 A and ADC 74B, respectively, the outputs of ADC 74A and ADC 74B are input into processor 76. Upon receiving the converted samples, processor 76 combines the samples in accordance with the techniques described above with particular reference to equations (l)-(3).
• the techniques of this disclosure have been described above with respect to combining only two samples, a short exposure time sample S ( , j) and a long exposure time sample S ( , j)-
• the techniques of this disclosure are not so limited. Rather, the combination scheme described in this disclosure may be used to combine more than two, e.g., five, samples from different exposure times. For example, in order to take a picture of a very bright object, e.g., the sun, a wide dynamic range would be required. As such, more than two images may be necessary in order to achieve that dynamic range.
• multiple samples may be collected and combined using the variable weighting techniques described above.
• the combining scheme described throughout this disclosure may be implemented using a variety of imaging techniques to produce a HDR image.
• the even- numbered lines of a pixel cell array e.g., pixel cell array 52 of FIG. 3
• the two different exposures times may be aligned so that they finish at the same time, e.g., substantially simultaneously.
• the two exposures may be combined to produce an image that is two vertically downs ampled images.
• the odd and even lines of the pixel cell array may be constructed and arranged such that they may be independently controlled.
• the image may be downsampled or upsampled in order to generate a value for each spatial location on the pixel cell array for the two different exposures times.
• first exposure time samples may be generated for each of the photosensitive semiconductors on the odd-numbered lines by downsampling or upsampling the samples collected for the even-numbered lines.
• second exposure time samples are generated for each of the photosensitive semiconductors on the even-numbered lines by downsampling or upsampling the samples collected for the odd-numbered lines.
• the first exposure time sample for each of the plurality of spatial locations in each of the plurality of odd- numbered lines are generated by one of downsampling and upsampling the samples collected from the even-numbered lines over the first exposure time
• the second exposure time sample for each of the plurality of spatial locations in each of the plurality of even-numbered lines are generated by one of downsampling and upsampling the samples collected from the odd-numbered lines over the second exposure time.
• the short exposure time samples and long exposure time samples are combined, e.g., using equations (l)-(3). The combination of these samples may produce either a half resolution image or full resolution image that has been upsampled.
• FIG. 6 depicts a conceptual diagram illustrating the combination of short and long exposure time samples from even and odd-numbered lines using the techniques described above.
• the image depicted at 150 in FIG. 6 illustrates the sensor frame plane with even and odd-numbered line and short and long exposure time control.
• Long exposure time e.g., second exposure time
• samples are collected for the odd-numbered lines and are depicted at 152.
• the long exposure time has caused pixel cells at 153 to become saturated.
• Short exposure time e.g., first exposure time
• samples are collected for the even numbers and are depicted at 154.
• the samples collected for the odd and even-numbered lines are upsampled.
• second exposure time image 156 and first exposure time image are combined to produce a full resolution image, shown at 160.
• the pixel cells at 161 are no longer saturated.
• the two images may be combined on-sensor or off-sensor via an external processor.
• the main difference between combining on-sensor or off-sensor is the amount of additional memory that is needed.
• Combining on the sensor may allow for faster read out of the samples, thereby reducing latency.
• combining on the sensor may allow for reduced data transfers, e.g., reduced bandwidth.
• two frames may need to be read out from the sensor, e.g., a short exposure time frame and a long exposure time frame.
• the data rate must be doubled in order to accommodate the read outs of the two frames.
• the combination is performed on the sensor, the long exposure time frame and short exposure time frame are combined and read out within one frame duration.
• the combination of the long exposure time sample and the short exposure time sample may be performed on-sensor.
• extra memory e.g., a line buffer
• an entire even line may be read out at the short exposure time, converted via an A/D converter, and stored in memory. Then, the entire odd line may be read out at the long exposure time and converted via an A/D converter.
• a processor may retrieve the value from memory for the even line, combine with the converted values for the odd line using the combining techniques described in this disclosure, and then output a single, combined image. Again, using such an on-sensor scheme may reduce the bandwidth needed for the imaging processing pipeline, at the expense of additional memory requirements.
• pixel cell 10 of FIG. 1 may also be used in a sensor, e.g., a CMOS sensor, that utilizes an off-sensor combining scheme.
• a sensor e.g., a CMOS sensor
• the sensor may be reset in the manner described previously, and then the sensor may begin acquiring photons immediately.
• charge may be transferred to capacitor 20.
• correlated double sampling may be used, as described above. Then, samples may be read using a rolling shutter technique.
• correlated double sampling may be performed again.
• the values from the long exposure time may be read out.
• the signal combining techniques described in this disclosure are applicable to image sensors other than CMOS sensors, such as CCD image sensors.
• a CCD image sensor may be used in another example imaging technique to produce an HDR image.
• a mechanical shutter may be added to the CCD sensor.
• the mechanical shutter may be opened and the sensor may begin acquiring photons immediately.
• charge, or first sample is transferred to temporary storage region, e.g., a shielded region that may store a copy of the pixel cell.
• the mechanical shutter is closed.
• FIG. 7 is a flow diagram illustrating an example method that may be used to implement the techniques of this disclosure.
• Controller 54 of FIG. 3 samples a first pixel cell signal at a first time, e.g., a short exposure time, to produce a first sample, e.g., SSHORT(I, j), (200).
• Controller 54 also samples a second pixel cell signal at a second time, e.g., a long exposure time, to produce a second sample, e.g., SLONG(I, j) (200).
• the second time is temporally subsequent to the first time.
• the first pixel cell signal and the second pixel cell signal are produced by one of a complimentary MOS (CMOS) image sensor and a charge- coupled device (CCD) image sensor.
• CMOS complimentary MOS
• CCD charge- coupled device
• the value of M(i, j) is between zero and one, inclusive.
• the variable weighting factor M(i, j) C(L) and is defined by the following:
• Processor 76 then combines the first sample and the weighted second sample to produce an HDR image (220).
• combining the first sample and the weighted second sample comprises combining the first sample and the weighted second sample on the image sensor, e.g., image sensor 50.
• FIG. 8 is a flow diagram illustrating another example method that may be used to implement the techniques of this disclosure.
• the method of FIG. 8 may be used, for example, with a CCD image sensor.
• a controller opens a mechanical shutter on a CCD image sensor, exposing the image sensor to light (300).
• the controller transfers a first sample developed based on the light to a storage region at a first time (310), and then reads out the first sample from the storage region (320).
• the controller closes the mechanical shutter (330), and a second sample may be transferred to the storage region (340).
• the controller may then read out the second sample from the storage region (350).
• a processor applies a variable weighting factor, e.g., M(i, j), to the second sample (360).
• the processor then combines the first sample and the second weighted sample (370), e.g., by processor 76 to produce a HDR image.
• FIG. 9 is a flow diagram illustrating another example method that may be used to implement the techniques of this disclosure.
• the method of FIG. 9 may be used, for example, with a CMOS image sensor.
• controller 54 of FIG. 3 may align a first exposure time and a second exposure time so that the first exposure time finishes substantially simultaneously with the second exposure time (400).
• Controller 54 may expose a plurality of even-numbered lines of the CMOS image sensor at the first time to produce a first sample for each of a plurality of spatial locations in each of the plurality of even-numbered lines, and may expose a plurality of odd-numbered lines of the CMOS image sensor at the second time to produce the second sample for each of a plurality of spatial locations in each of the plurality of odd-numbered lines (410).
• Processor 76 for example, generates the first sample for each of the plurality of spatial locations in each of the plurality of odd-numbered lines by one of downsampling and upsampling the first sample for each of a plurality of spatial locations in each of the plurality of even- numbered lines (420).
• Processor 76 for example, generates the second sample for each of the plurality of spatial locations in each of the plurality of even-numbered lines by one of downsampling and upsampling the second sample for each of a plurality of spatial locations in each of the plurality of odd-numbered lines (430).
• Processor 76 applies a variable weighting factor, e.g., M(i, j), to the second sample (440).
• Processor 76 then combines the first sample and the second weighted sample to produce a HDR image (450).
• FIG. 10 is a flow diagram illustrating another example method that may be used to implement the techniques of this disclosure.
• the method of FIG. 10 may be used, for example, with a CMOS image sensor.
• controller 54 of FIG. 3 may reset a photosensitive semiconductor, e.g., photosensitive semiconductor 12, and expose the photosensitive semiconductor to light 54 (500).
• controller 54 may transfer a first sample developed based on the light to a capacitor (510), e.g., capacitor 20 of FIG. 1.
• controller 54 may read out a second sample developed based on the light from the photosensitive semiconductor (520).
• Controller 54 may read out the first sample after controller 54 reads out the second sample (530).
• Processor 76 may apply a variable weighting factor, e.g., M(i, j), to the second sample (540).
• Processor 76 may combine the first sample and the second weighted sample to produce a HDR image (550).
• a variable weighting factor e.g., M(i, j
• processors including one or more microprocessors, digital signal processors (DSPs), network processors (NPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuitry, as well as any combinations of such components.
• DSPs digital signal processors
• NPs network processors
• ASICs application specific integrated circuits
• FPGAs field programmable gate arrays
• processors or “processing circuitry” may generally refer to any of the foregoing logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry.
• a control unit comprising hardware may also perform one or more of the techniques of this disclosure.
• Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various operations and functions described in this disclosure.
• any of the described units, modules or components may be implemented together or separately as discrete but interoperable logic devices. Depiction of different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be realized by separate computing hardware or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware or software components, or integrated within common or separate hardware or software components.
• Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer readable media.
• RAM random access memory
• ROM read only memory
• PROM programmable read only memory
• EPROM erasable programmable read only memory
• EEPROM electronically erasable programmable read only memory
• flash memory a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer readable media.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Transforming Light Signals Into Electric Signals (AREA)
• Studio Devices (AREA)
• Exposure Control For Cameras (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (2)

Family

ID=44675825

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (30)

Family Cites Families (31)

• 2010
  • 2010-09-01 US US12/873,981 patent/US8994843B2/en active Active
• 2011
  • 2011-08-31 EP EP11760911.5A patent/EP2612492B1/en not_active Not-in-force
  • 2011-08-31 CN CN201180046572.8A patent/CN103141081B/zh active Active
  • 2011-08-31 JP JP2013527289A patent/JP2013537020A/ja not_active Withdrawn
  • 2011-08-31 WO PCT/US2011/050068 patent/WO2012031026A2/en not_active Ceased
  • 2011-08-31 KR KR1020137008243A patent/KR101473597B1/ko active Active
• 2015
  • 2015-12-11 JP JP2015242442A patent/JP2016096558A/ja active Pending

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events