WO2024168432A1 - Capteur d'image et procédé de conversion flux-numérique et de codage d'exposition par pixel - Google Patents

Capteur d'image et procédé de conversion flux-numérique et de codage d'exposition par pixel Download PDF

Info

Publication number
WO2024168432A1
WO2024168432A1 PCT/CA2024/050187 CA2024050187W WO2024168432A1 WO 2024168432 A1 WO2024168432 A1 WO 2024168432A1 CA 2024050187 W CA2024050187 W CA 2024050187W WO 2024168432 A1 WO2024168432 A1 WO 2024168432A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel
exposure
flux
coded
image sensor
Prior art date
Application number
PCT/CA2024/050187
Other languages
English (en)
Inventor
Rahul GULVE
Kiriakos Neoklis Kutulakos
Roman GENOV
Original Assignee
The Governing Council Of The University Of Toronto
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by The Governing Council Of The University Of Toronto filed Critical The Governing Council Of The University Of Toronto
Publication of WO2024168432A1 publication Critical patent/WO2024168432A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/53—Control of the integration time
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
    • H04N25/77—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
    • H04N25/771—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising storage means other than floating diffusion

Definitions

  • the following relates generally to imaging, and more specifically, to image sensor and method for flux-to-digital conversion and pixel-wise exposure coding.
  • Consumer cameras typically use computational imaging techniques to digitally enhance images through the use of software post-processing to yield both high fidelity and low cost.
  • One particular technique for example, is to combine multiple shots with different camera settings into one enhanced image that has a high dynamic range (HDR).
  • HDR high dynamic range
  • this post-processing- based approach can encounter a number of problems; for example, failing when there is fast motion and/or rapidly-changing illumination. Such conditions are often present in autonomous driving, drone imaging, and action-camera applications, or when the illumination itself is actively controlled (e.g., for depth sensing).
  • a method for flux-to-digital conversion for an image sensor comprising: receiving light for a pixel in the image sensor while the pixel is being exposed to the light for an exposure period, the pixel provides an output that is a function of the light received; performing line fitting on transients of the pixel output, the transients determined using a periodic reference voltage; and outputting the line fitting value as the digital representative of the received light flux.
  • the pixel output is determined by comparing the transients with the periodic reference voltage using a binary comparator.
  • the line fitting comprises linear regression.
  • a timestamp of the binary comparator output transient is used to estimate an instantaneous incident flux, where the instantaneous incident flux is equal to a slope of the fitted line.
  • the binary comparator outputs comparisons over multiple periods of the periodic reference voltage.
  • the periodic reference voltage is a sinusoidal reference voltage.
  • the sinusoidal voltage waveform has no more than one sinusoidal period for the duration of the exposure time.
  • the sinusoidal reference voltage is generated using a resonance-based circuit.
  • a plurality of transients are generated within a single exposure period, and wherein the plurality of transients in the single exposure period are used to determine the line in the line fitting.
  • the number of cycles in the periodic reference voltage is selected to detect multiple flux values in the exposure period.
  • a coded-exposure-pixel image sensor comprising an array of coded-exposure pixels that each receive light and a set of readout circuits that convert light received at the pixels to a digital representative value that is a function of the light received, the image sensor further comprising: a first port for receiving an exposure code that configures an exposure of the pixels in the array; and a second port for readout of the digital representative value of the pixel, where line fitting is performed on transients of the readout of the pixels, the transients determined using a periodic reference voltage, and where the line fitting value is the digital representative value.
  • the exposure code is received concurrently with a subexposure readout of the pixel.
  • the image sensor comprising a stacked-wafer fabrication, wherein the digital representative value is generated on a lower wafer for each pixel or group of pixels.
  • the image sensor further comprising a pixel exposure circuit to generate different pixel exposure codes or to perform decompression of received pixel exposure codes.
  • a method for high-speed imaging over an exposure period using a coded-exposure-pixel image sensor the exposure period is subdivided into a plurality of subexposures, the image sensor comprising an array of coded-exposure pixels, each coded-exposure pixel comprises one or more taps, the method comprising: receiving, at each coded-exposure pixel, an exposure code for each of the subexposures; receiving light at each coded-exposure pixel and converting the light into a photogenerated charge; for each subexposure, selectively integrating the photogenerated charge into one of the taps of the coded-exposure pixel based on the exposure code for such pixel; and for each of the coded- exposure pixels in sequence, exposing the photogenerated charge from one of the taps of the coded-exposure
  • each coded-exposure pixel comprises two taps, and wherein the photogenerated charge is exposed from a first of the two taps.
  • the method further comprising generating a frame of a high-speed video, the frame generated from the parallel readout of the collective outputted photogenerated charges after the exposure period by spatially demultiplexing the frame into a plurality of images, with the number of such images equal to the number of subexposures.
  • the array of coded-exposure pixels is arranged in a programmable-sized subsets of coded-exposure pixels, a size of the subsets varies based on a speed of local motion or illumination.
  • a speed of the readout scales based on the motion or changing illumination present in a captured scene.
  • FIGS. 1A to 1 D illustrate Single-slope-ADC (SS-ADC) pixel output processing in comparison to the image sensor of the present embodiments for single and multiple comparisons;
  • FIGS. 2A and 2B illustrate a block diagram of an example of the image sensor of the present embodiments and various coding strategies that are implemented on the image sensor;
  • FIG. 3 illustrates examples of circuit diagrams of the codded-exposure pixel and readout path, and the corresponding timing diagrams
  • FIG. 4 illustrates experimentally measured signal-to-noise ratio (SNR) and the pixel charge-transfer contrast for the image sensor of the present embodiments
  • FIG. 5 shows raw experimentally measured output and reconstructed HDR image that was mapped by log-compression to a low-dynamic-range (LDR) space
  • FIG. 6 illustrates a comparative analysis table for the image sensor of the present embodiments showing a comparison to other image sensors with wide-dynamic-range readout and/or coded exposure
  • FIG. 7 illustrates an example of a micrograph of the image sensor of the present embodiments fabricated in a 110nm CIS process
  • FIG. 8 illustrates simulation results comparing the SNR of the image sensor of the present embodiments with various VREF waveforms
  • FIG. 9A illustrates comparison of quantization using different reference voltage waveforms
  • FIG. 9B illustrates a prototype test PCB implementing the present embodiments
  • FIG. 10 illustrates direct/indirect-light intensity imaging application of the image sensor of the present embodiments
  • FIG. 11 illustrates simplified multispectral imaging where a custom spectrum of a scene, beyond RGB, is obtained
  • FIG. 12 illustrates depth-gated imaging application of the image sensor of the present embodiments
  • FIG. 13 is a flowchart of a method for flux-to-digital conversion for a pixel of an image sensor, in accordance with an embodiment
  • FIG. 14 shows single-port architecture equivalent block diagram of (a) non-coded, (b) iToF, and (c) CEP image sensors; in addition to (d) a Dual-port architecture equivalent block diagram of the CEP image sensor of the present embodiments;
  • FIG. 15 illustrates a block diagram of a dual-port Coded Exposure Pixel (CEP) image sensor, in accordance with the present embodiments;
  • CEP Coded Exposure Pixel
  • FIG. 16A illustrates a readout path, incorporating the coded-exposure pixel and the RFDC
  • FIG. 16B shows a timing diagram for concurrent operation of both ports of the CEP sensor
  • FIG. 17A illustrates the high-speed imaging configuration for TPM using the CEP sensor of the present embodiments
  • FIG. 17B shows coded exposure, demultiplexed images, and an upscaled image for the CEP sensor
  • FIG. 18 illustrates an example of a super-pixel schematic, and principle of operation
  • FIG. 19 illustrates a sensor block diagram, and spatially varying exposure speed programming strategy
  • FIG. 20 shows experimental results of the sensor of the present embodiments capturing high-speed motion using spatially varying exposure rate
  • FIG. 21A shows a conventional pixel readout path estimating light intensity using conventional ADCs, such as the single-slope ADC;
  • FIG. 21 B shows a pixel voltage waveform in CIS during the exposure and readout phase of a frame where the maximum digitized flux value is determined by the pixel’s full well capacity and exposure time;
  • FIG. 22A illustrates a chart showing energy-efficient high-dynamic-range flux estimation in conventional pixels using a single-comparison flux-to-digital converter (FDC);
  • FDC flux-to-digital converter
  • FIG. 22B illustrates a chart showing multiple-comparisons regression-based flux-to- digital converter (RFDC);
  • FIG. 23A shows a signal path for both SS-ADC and FDC
  • FIGS. 23B and 23C depict graphical representations of the pixel output and the reference voltage waveforms during the SS-ADC and the FDC operation, respectively;
  • FIGS. 24A and 24B illustrate a noisy pixel voltage can be sampled multiple times during a crossing;
  • FIG. 25A shows sinusoidal reference voltage with up to 4 and with up to 20 crossing during an exposure;
  • FIG. 25B shows SNR comparison between RFDC with increasing number of sinusoidal crossings ranging from 1 to 32 during exposure over a wide flux range.
  • FIG. 25C shows SNR plots comparing single-crossing FDC and RFDC with up to 4 crossings
  • FIG. 25D shows SNR plots comparing single-crossing FDC and RFDC with up to 20 crossings
  • FIG. 26A illustrates a CIS block diagram of a VLSI implementation and FIG. 26B illustrates a dual-tap coded-exposure pixel schematic;
  • FIG. 27A shows an FDC/RFDC readout path
  • FIG. 27B shows a strong-arm latch comparator schematic
  • FIG. 27C shows a timing diagram for exposure and readout using FDC
  • FIG. 28 shows a simplified basic example of a sinusoidal reference voltage generation concept for FDC using a digital pulse, PWM INPUT, controlled by an FPGA.
  • Any module, unit, component, server, computer, terminal, engine or device exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape.
  • Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data.
  • Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD- ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information, and which can be accessed by an application, module, or both. Any such computer storage media may be part of the device or accessible or connectable thereto. Further, unless the context clearly indicates otherwise, any processor or controller set out herein may be implemented as a singular processor or as a plurality of processors.
  • the plurality of processors may be arrayed or distributed, and any processing function referred to herein may be carried out by one or by a plurality of processors, even though a single processor may be exemplified. Any method, application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media and executed by the one or more processors.
  • the following relates generally to imaging, and more specifically, to image sensor and method for flux-to-digital conversion and pixel-wise exposure coding.
  • Pixel-wise generally refers to operations or functions on a per-pixel or pixel-by-pixel basis; however, it is understood that in some cases, pixel-wise can include operations or functions on a small-group-of-pixels by small-group-of-pixels basis.
  • Some of these sensors can offer spatial exposure control for single-shot HDR imaging but may require multiple analog-to-digital (ADC) types and/or a number of pre-processing and post-processing steps (e.g., adaptive pixel-wise exposure control, HDR reconstruction, etc.).
  • ADC analog-to-digital
  • Other coded sensors may support a variety of computational imaging techniques (e.g., robust depth imaging, compressed sensing), but their conventional ADCs do not offer HDR readout.
  • Other HDR sensors can digitize the pixel output during exposure, before it saturates, but generally offer no coding.
  • HDR high-dynamic- range
  • HDR scene-adaptive high-dynamic-range
  • DMP dual-tap coded-exposure data-memory pixels
  • CMOS image sensors capable of performing the following functions either jointly (e.g., concurrently or at different times), or independently (e.g., separately, each in its own right):
  • CEP coded-exposure-pixel
  • ROI region-of-interest
  • FDC flux-to-digital conversion
  • RFDC regression flux-to-digital conversion
  • CEP imaging such imaging is able to be performed rapidly by fast pixelwise exposure coding of regions of interest (ROIs).
  • ROIs regions of interest
  • a provided sensor architecture permits updating of codes only in pixels or rows of interest; so the entire subexposure duration can be very short (e.g., as short as 80ns in a particular experimental prototype when only one row of pixel codes is updated per subframe).
  • the present embodiments allow for implementation of high-speed imaging using the sensor architecture.
  • the present embodiments also allow for spatially-varying-speed imaging capability by the sensor architecture.
  • the sensor of the present embodiments can include an array of flux-to-digital converters (FDCs) which can be column-shared, column-parallel, or in-pixel; the latter being more practical in stacked-wafer image sensors implementations.
  • FDCs flux-to-digital converters
  • CEP the present embodiments can use regression-based flux-to-digital conversion (RFDC), which is a special case of FDC where regression or another approach is used to estimate the output.
  • RFDC regression-based flux-to-digital conversion
  • the present embodiments can allow for HDR readout and/or low- resolution readout (such as binary readout).
  • the present embodiments can allow for exposure-concurrent intraframe readout.
  • the image sensor of the present embodiments provides a dual-port that allows for not only separate engagement of the two functionalities above, but also for simultaneous synergistic combination of coded-exposure-pixel (CEP) imaging; which receives its input through a port which can be referred to as port 1, and an FDC readout which provides its output through a port which can be referred to as port 2.
  • CEP coded-exposure-pixel
  • the sensor architecture can, for example, include a bank of conventional single-slope (SS) or successive-approximation-register (SAR) analog-to-digital conversion (ADC) units, which can be used for conventional intensity readout.
  • SS single-slope
  • SAR successive-approximation-register
  • ADC analog-to-digital conversion
  • the sensor CEP pixel can be configured like a regular intensity pixel, or any other pixel type can be included and combined with the FDC readout.
  • the present embodiments provide an image sensor with an ADC-free flux-readout scheme that can output one or more digital HDR flux samples per frame. Such outputs can be based on periodic-reference-waveform (such as sinusoidal-reference) comparator binary outputs that are read out at, for example, 26kHz during exposure.
  • a readout scheme of the present embodiments which can be referred to as Regression-based Flux-to-Digital Conversion (RFDC), offers wider dynamic range compared to other ADC architectures.
  • RFDC Regression-based Flux-to-Digital Conversion
  • the readout scheme is generally applicable to any pixel design and can advantageously reduce digital processing to a simple pixel-wise regression.
  • the image sensor of the present embodiments can support 80-nanosecond (ns) rapid-update pixel-wise exposure coding at rates more than two orders of magnitude faster than other approaches. This functionality enables capture of light with fast-changing intensity, and is independent and complementary to the RFDC scheme.
  • SS-ADC single-slope
  • CIS CMOS image sensors
  • the SS-ADC consists of a comparator and a shared global ramp generator. As shown in FIG. 1A, a voltage corresponding to an integrated photogenerated charge is sampled at the end of an exposure period and compared by the SS-ADC with a ramped reference voltage. The timestamp of the comparator output transient is then converted to a digital number corresponding to the pixel intensity.
  • FWC full-well-capacity
  • the sensor’s dynamic range for high flux and its output is only available at the end of the exposure.
  • the SS-ADC is generally not particularly fast or energy efficient, with the ramp generator often limiting the ADC speed and dominating its power requirements.
  • FIGS. 1A to 1D illustrate SS-ADC versus RFDC with single and multiple comparisons.
  • RFDC uses only a comparator with a sinusoidal reference voltage or any other periodic-waveform reference voltage.
  • the timestamp of the comparator output transient is used to estimate the instantaneous incident flux (flux is equal to the line’s slope).
  • High flux values, above a predetermined threshold trigger the comparator earlier in the exposure period, before the pixel saturates. This allows for capturing scenes with a wider dynamic range than other approaches.
  • the sensor can advantageously operate with the sinusoidal voltage waveform having no more than one sinusoidal period for the duration of the exposure time.
  • a sinusoidal waveform serves as a good approximation for the theoretically derived HDR-optimum shape of the reference voltage.
  • a low-noise sinusoidal voltage can be generated using negligibly small power; for example, by a resonance-based circuit such as by a resonantly clocked LC tank (inductor and capacitor resonator circuit) with small resistive losses or by a resonantly-generated square waveform, which may or may not be filtered by a low-pass filter.
  • FIG. 10 shows that such an approach yields multiple comparator output transients within one exposure period, which can then be used to improve the flux estimations by line fitting.
  • different numbers of voltage crossings can occur; where lower-flux pixels cross the sinusoidal reference voltage a maximum number of times (four times in the example shown), and higher-flux pixels cross at least once.
  • FIGS. 2A and 2B illustrate a block diagram of an example of the image sensor and various coding strategies that are implemented on the chip.
  • FIG. 2A depicts a block diagram of the example image sensor, which has two respective ports: (1) fast per-pixel-exposure binary code inputs at a maximum rate of 12.5M pixel rows per second, and (2) comparator outputs at a maximum effective rate of 26k binary pixel array readouts per second, which is used to estimate intra-frame HDR flux.
  • Each port can be addressable by an independent random-access row decoder.
  • pixel-wise exposure coding can be implemented using a two-tap pixel design based on an indirect time-of-flight (iToF) pixel, but with two substantial differentiators:
  • iToF pixel performs fixed temporal coding only (such as in FIG. 2B); whereas, in some cases, the present embodiments include per-pixel charge-sorting circuitry between taps 1 and 2 which enables pixel-wise spatial exposure programmability, and
  • a moderate charge transfer contrast of 60-70% is generally sufficient for iToF pixel as it operates on phase measurements; whereas, in some cases, the present embodiments enable applications that have much stricter contrast requirements, above 90-95%, which limits the coded subexposure rate to around 12.5MHz versus higher operating iToF frequencies.
  • many spatially coded computational image sensors offer coarse-resolution coding on pixel clusters (such as in FIG. 2B, above the middle); for example, on a 16x16 pixel grid. This effectively reduces spatial resolution by a large factor (e.g., by 256 times).
  • the pixel coding in the present embodiments can be performed at the native spatial resolution (such as in FIG. 2B, below the middle) and scales well to stacked-wafer technologies by means of local temporal multiplexing on small pixel groups; without the need for large 3D interconnect in each pixel or large temporal lag.
  • the coded subexposure rate in the present embodiments is a factor of up to 320x higher than the fastest previous pixel-wise exposure coding implementation, owing to the random-access nature of the code update port (such as in FIG. 2B, bottom).
  • many coded imaging techniques require sparse temporal code updates (e.g., only in a single row or a few rows at a time).
  • a custom code generator can locally generate three types of common codes without having to spend the power to ship them from off the chip. It also has a capability to decompress codes received by it from another location to reduce the incoming code data rate.
  • an exposure code electrical circuit can be included in the image sensor that can locally generate different types of pixel exposure codes and/or perform decompression of received codes.
  • FIG. 3 depicts examples of circuit diagrams of the codded-exposure pixel and readout path, and the corresponding timing diagrams.
  • the processes of pixel coding and RFDC can be temporally staggered or interleaved, offering an extra degree of freedom for various computational imaging techniques.
  • FIG. 4 illustrates experimentally measured RFDC signal-to-noise ratio (SNR) and the pixel charge-transfer contrast. A total dynamic range of 95dB has been measured, with SNR ranging between approximately 10 and 42dB, for four VREF-VIN crossings per RFDC output.
  • SNR signal-to-noise ratio
  • FIG. 5 shows raw experimentally measured output and reconstructed HDR image that was mapped by log-compression to a low-dynamic-range (LDR) space using a four-comparison RFDC.
  • the RFDC output was split into four images to show the outputs at the four different VREF-VIN crossings, one per VREF half-period during the exposure.
  • the insets in FIG. 5 show the dark and bright regions of the scene mapped to different 8-bit LDR spaces.
  • FIG. 6 depicts a comparative analysis table for the example experiments showing a comparison to other image sensors with wide-dynamic-range readout and/or coded exposure.
  • FIG. 7 shows the micrograph of the image sensor fabricated in a 110nm CIS process.
  • the 3.67mW flux-to-digital converter bank has a very small foot-print and was used to obtain all experimental results (a SAR ADC is included for comparison).
  • FIG. 8 illustrates simulation results comparing the SNR of RFDC with various VREF waveform examples, including but not limited to a sinusoidal waveform.
  • FIG. 9A illustrates comparison of quantization using different reference voltage waveform examples
  • FIG. 9B illustrates a prototype test PCB implementing the present embodiments.
  • FIGS. 10, 11 and 12 illustrate example applications of the single-shot coded-exposure image sensor (experimentally measured using a test chip with same coded-pixel design).
  • FIG. 10 illustrates direct/indirect-light intensity imaging application of the sensor, which enables imaging in highly scattering, reflective or refractive media; and, depending on the application, can avoid and/or utilize multipath light propagation.
  • the charges generated by direct (i.e. , reflected once) and indirect incident light are sorted into two taps based on ray optics. Unlike the arbitrary path of indirect light, direct light travels through the “epipolar plane”, from the projector to the camera. Using the pixel codes as shown, the charges generated by direct and indirect incident light are sorted into two taps
  • FIG. 11 illustrates simplified multispectral imaging where a custom spectrum of a scene, beyond RGB, is obtained.
  • 5 LEDs of different wavelengths, A1 to A5 are individually turned on and 5 code matrices, time-multiplexed in 2x2-pixel tiles, are synchronously submitted to the sensor. From 8 taps of each 2x2-pixel tile, 5 images at the 5 wavelengths are then extracted by demultiplexing.
  • FIG. 12 illustrates depth-gated imaging application of the sensor.
  • the pixel- programmable sensor allows for capturing of two modalities, intensity and depth information, with depth-range-selective imaging capabilities demonstrated.
  • the image sensor example of the present embodiments provides an ADC-free flux-readout scheme that uses regression on sinusoidal- reference binary comparator outputs read out at 26KHz during exposure to produce one or more digital HDR flux samples per frame. Additionally, an 80ns sparse-update pixel-wise exposure coding scheme is provided for imaging applications with fast-changing intensity. The sensor is demonstrated for HDR imaging and exhibits a greater than 300 times faster pixel-wise exposure coding rate than other approaches.
  • FIG. 13 illustrates a method for flux-to-digital conversion for a pixel of an image sensor, in accordance with an embodiment.
  • the image sensor receives light flux for the pixel while the pixel is being exposed to the light.
  • a processor of the image sensor performs line fitting on transients of the light flux to a multi-period sinusoidal reference voltage using linear regression.
  • the processor outputs the line fitting value as the digital representative of the received light flux to storage or to an interface with another device.
  • a plurality of transients are generated within a single exposure period.
  • the plurality of transients in the single exposure period can then be used to determine the line in the line fitting.
  • Image sensor pixel arrays resemble random access memory (RAM) in that they both serve as storage mediums, holding data to be accessed as needed. This similarity in purpose leads to a considerable overlap in the principles of their operation and design.
  • RAM random access memory
  • CMOS image sensors CISs
  • each pixel stores and converts light into an electrical charge, which is then sequentially read out, which is analogous to the operation of single-port RAM.
  • the rows of pixels in CMOS image sensors can be analogously considered.
  • FIG.14 shows single-port architecture equivalent block diagram of (a) non-coded, (b) iToF, and (c) CEP image sensors; in addition to (d) a Dual-port architecture equivalent block diagram of the CEP image sensor of the present embodiments.
  • Coded-exposure pixel (CEP) sensors receive additional inputs in the form of exposure codes.
  • Indirect Time-of-Flight (iToF) sensors are generally the simplest form of CEP sensors. In such an arrangement, each pixel within the array accepts a globally shared modulation signal as its exposure code, as indicated in FIG. 14(b), and no row-address control is required for the coded exposure.
  • iToF sensors are capable of operating within the framework of a single-port equivalent architecture.
  • a dual-port architecture of a CEP image sensor is provided. Within this sensor, the row decoders for exposure-code delivery and readout are distinct, facilitating concurrent coded exposure and readout. This design eliminates the issues associated with the blank period observed in existing CEP sensors and enhances light efficiency.
  • Pixel-wise exposure coding leverages a two-tap pixel design inspired by indirect time-of- flight (iToF) pixels.
  • Other approaches for iToF pixels reach temporal coding frequencies, also commonly known as modulation frequencies, exceeding 300 MHz.
  • temporal coding frequencies also commonly known as modulation frequencies, exceeding 300 MHz.
  • PPD pinned photodiode
  • these pixels do not generally employ standard pinned photodiode (PPD)-based pixels. Instead, they rely on specialized pixel technology optimized for ToF, which results in high fabrication costs and compromises conventional intensity imaging performance.
  • PPD-based iToF pixels achieving temporal coding/modulation frequencies up to 75 MHz, the fastest reported pixel-wise exposure-coding frequency stands at merely 39 kHz.
  • iToF pixel performs fixed temporal coding only, as depicted in FIG 2B.
  • per-pixel charge-sorting circuitry between taps 1 and 2 can be used, facilitating pixel-wise spatial exposure programmability.
  • a moderate charge transfer contrast of 60-70% suffices for an iToF pixel, due to its operation on phase measurements; in contrast, the CEP can support applications demanding stringent contrast requirements, exceeding 90-95%.
  • FIG. 15 illustrates a block diagram of a dual-port Coded Exposure Pixel (CEP) image sensor 1500, in accordance with the present embodiments.
  • This sensor 1500 incorporates two ports: Port 1 serves as an input port for exposure codes, and Port 2 functions as an output for pixel readout.
  • the sensor comprises a 640 x 480 array of dual-tap coded- exposure pixels.
  • the exposure codes for the array can be distributed through a bank of twenty 1:32 deserializers.
  • Each deserializer functioning in a dual-data rate (DDR) mode, can receive codes at up to 400 Mbps with a 200 MHz clock.
  • this sensor 1500 can incorporate a row decoder to permit random-row access for rapid-update exposure coding. Consequently, a 9-bit row address should be accompanied by the mask data in such cases.
  • the exposure codes for the array can either be provided externally or generated on-chip using a custom code generator block. This block can produce three distinct types of masks: pseudo-random, repeated tiles, and sliding patterns.
  • Port 2 in an example, such port can function as an output port employed to read out pixel values.
  • the sensor 1500 incorporates two data converter banks: (1) traditional successive- approximation (SAR) analog-to-digital converter (ADC), and (2) the RFDC.
  • SAR successive- approximation
  • ADC analog-to-digital converter
  • RFDC the RFDC.
  • Each ADC/RFDC in the bank can be shared with two neighboring columns.
  • Each ADC can, for example, operate at a clock frequency of 10 MHz, yielding a sampling rate of 769,230 samples per second. This sampling rate allows the ADC bank to reach a maximum frame rate of 400 FPS while digitizing both taps of the entire pixel array.
  • the SAR ADC delivers a signal-to-quantization-noise ratio of 68.6 dB, corresponding to 10.6 effective number of bits (ENOB).
  • the sensor 1500 can also incorporate an alternative HDR digitization methodology utilizing the RFDC described herein.
  • the RFDC has a notably small footprint, comprising only a comparator in some cases. It can operate, for example, at 30 FPS while conducting 870 comparisons per frame and attains a total dynamic range of 95 dB.
  • the reference voltage for each RFDC is globally shared and supplied from an external source.
  • FIG. 16A illustrates a readout path, incorporating the coded-exposure pixel and the RFDC.
  • the timing diagram for operation shown in FIG. 16B illustrates the concurrent operation of both ports.
  • the sensor 1500 only selects the rows necessitating an update of the exposure code. For instance, FIG. 16B, at the top, illustrates that in subexposure, SUBEXP[2], a set of r rows R[0] to R[r- 1] are accessed, which do not have to follow a sequential order.
  • High Dynamic Range (HDR) readout employing RFDC occurs; as shown in the lower part of the timing diagram in FIG. 16B.
  • the RFDC can produce M 1-bit frame outputs by scanning all rows from 0 to V- 1 (V equals 480 in this example).
  • the dual-port CEP image sensor 1500 provides an array of dual-tap coded- exposure pixels (e.g., 640x480).
  • This sensor 1500 addresses the limitations of single-port architecture in CEP, notably eliminating the inefficient blank period and enabling simultaneous exposure coding and readout. Furthermore, it enables rapid-update sparse exposure coding, significantly improving the subexposure speed to, for example, 12.5 MHz.
  • the sensor 1500 can include two readout modes using SAR ADC and RFDC.
  • the SAR ADC can achieve a frame rate of 400 FPS with SQNR of 68.6 dB and RFDC can capture HDR images of up to 95 dB in single-shot.
  • the spatial and temporal (i.e., frame rate) resolution of conventional image sensors has been fixed. This results in a scene with a faster motion than the frame rate appearing blurred in the final image.
  • high-speed cameras and burst imaging sensors have been utilized.
  • these high-speed cameras consume significant power due to the increased data rate, and existing burst-imaging sensors face challenges such as high pixel pitch and low fill factor, owing to the large number of taps per pixel.
  • the high subexposure rate of the CEP image sensor of the present embodiments can introduce a temporal pixel multiplexing (TPM) paradigm for imaging that enables the simultaneous capture of high-speed and high-resolution images. This is done by exposing several pixels in a rapid sequence, effectively implementing high-speed sequential-phase exposure, with a subsequent parallel readout of multiple such pixels at a lower output rate.
  • TPM temporal pixel multiplexing
  • FIG. 17A illustrates the high-speed imaging configuration for TPM using the CEP sensor of the present embodiments.
  • the pixel array is partitioned into m x n-pixel tiles, referred to as super-pixels, and the exposure duration is subdivided into m x n subexposures.
  • m x n subexposures During each subexposure, a single pixel from the super-pixel is configured to collect light in TAP1, while the remaining pixels accumulate the light in TAP2. Consequently, each pixel in the tile captures a different l/(m x n) th fraction of the exposure time.
  • This configuration does not consume any extra power when compared to a conventional camera, as all pixels are digitized at the same readout rate, e.g., 60 FPS or 360 fps.
  • a digitized TAP1 image (320 x 320-pixels) following 25 subexposures at 4.7 kHz is illustrated in FIG. 17B, left side.
  • the inset in the figure highlights a 20 x 20-pixel neighborhood, and the smaller rectangle indicates a 5 x 5-pixel tile, with each pixel capturing a unique l/25 th fraction of the exposure time.
  • Image-15 one of the demultiplexed images, is upscaled and exhibited in FIG. 17B, right side.
  • CEP sensors can have two significant limitations. First, they would be confined to high-speed burst imaging rather than continuous high-speed video due to the exposure dead time during ADC1 readout. Photons incident on the pixel during the readout phase are discarded, leading to the dead time.
  • the dual-port sensor of the present embodiments overcomes this disadvantage using simultaneous coded exposure and readout capability, which can eliminate dead time and enable continuous high-speed video acquisition through TPM Secondly, any other taps such as the second tap of a dual-tap CEP would generally not be used - any photogenerated charge collected in TAP2 would be discarded.
  • TAP2 reaches saturation by the end of the frame because it is exposed for (mn - 1) x longer than TAPI.
  • FDC readout can be used together with the dual-tap CEP sensor of the present embodiments, for example to obtain additional high-spatial-resolution information about the scene and thus increasing fidelity of the scene capture leading to improved noise robustness compared to a single-tap implementation.
  • High-speed CEP imaging through its flexible coded-exposure schemes, not only facilitates the capture of rapid motion but also mitigates power consumption through its low-rate readout feature, as compared to high-frame-rate cameras.
  • Such image sensors enabled by the integration of pixel-wise coded exposure on-chip, present a cost-effective, compact and scalable solution for achieving a high exposure rate of sensors, while offering output compatible to the conventional standard-video-rate imaging pipeline.
  • coded-exposure-pixel (CEP) imaging can be used to adapt to fast changes in the scene, both locally and globally, by taking advantage of programmable- size/speed super-pixels and conventional scalable-rate ADCs (not necessarily the FDC/RFDC, as described herein), respectively.
  • each super-pixel includes up to 8x8 pixels exposed in a rapid sequence within one frame period, yielding a 64x maximum boost of the exposure rate over the output rate, without the corresponding increase in the ADC power. At 360fps readout, this corresponds to over 23,000 exposures/second while using only 24.5mW, and comes at the cost of moderately lower local resolution, only in the regions with the fast changes in the scene.
  • the scalable-rate ADC allows for scaling down to the 30fps ‘slow’ mode, yielding an additional factor of 12x savings in the output data and camera digital power, and 30% savings in the ADC power.
  • Modern applications may demand image sensors that can capture high-speed high- quality images at a low cost, while accommodating rapidly changing motion dynamics and illumination. Sensors with enhanced reconfigurability, featuring per-pixel programmable exposure and motion-adaptive frame rates generally suffer either from uniformly-low spatial resolution at high speed or from low speed at high spatial resolution, respectively.
  • the image sensor of the present embodiments can be used to bridge this gap and offer the flexibility to be programmed using software: (1) to selectively trade spatial resolution of parts of the sensor array where fast motion is present for high exposure rate, without compromising spatial resolution or output data rate elsewhere in the scene, and additionally, (2) to dynamically adapt the readout rate based on global motion information.
  • the sensor can include programmable nonuniform-exposure-speed super-pixels and programmable adaptive-rate ADCs, respectively.
  • Low-cost image sensors such as those in most mobile phones, have relatively slow exposure and are intolerant to fast motion or rapidly changing illumination, leading to degraded image quality.
  • High-speed image sensors avoid motion artifacts, but lead to a high camera cost, due to the hardware resources needed to process and store the high-speed video output.
  • the software-defined image sensor of the present embodiments combines the best of both worlds - it is both motion-tolerant and low-cost, with as many as 23,040 exposures per second at the maximum readout rate of only 360fps, and without the corresponding increase in the ADC/digital power.
  • FIG. 18 illustrates an example of a super-pixel schematic, and principle of operation.
  • Conventional image sensors typically employ HxV-pixel uniform exposure and fixed-rate ADCs, resulting in limited software programmability which constraints the imaging speed, as illustrated in the top of FIG. 18.
  • the software-defined sensor of the present embodiments (as illustrated in FIG. 18, middle) employs programmable-exposure-speed super-pixels, each comprising an individually sized cluster of NxN exposure-phase-programmable dual-tap pixels (as illustrated in FIG. 18, bottom), where N can be, for example, 1, 2, 4 or 8.
  • the super-pixel deserializes incoming light by: (1) sequentially exposing tap 1 of each of its constituent pixels over N 2 rapid subexposure intervals, and (2) holding that photogenerated charge until the end of the exposure time when all tap-1 values are digitized.
  • the size and location of each super-pixel can be programmed based on information on local motion or changing illumination in the scene, trading local spatial resolution for local exposure speed without unnecessarily decreasing spatial resolution in other regions of the image that are static or slow-moving.
  • local motion/illumi nation can be estimated by comparing two subsequent subexposure results, accumulated on taps 1 and 2, and computing an optical flow map. To save power and reduce computation, these can be sparsely sampled, for example, using only one pixel in each 4x4 pixel subarray (as illustrated in FIG. 18, middle).
  • FIG. 19 illustrates a sensor block diagram and spatially varying exposure speed programming strategy.
  • a 640x480-pixel 110nm CIS image sensor (FIG. 19, left) includes a dynamically configured mosaic of spatially-varying-size/speed super-pixels.
  • FIG. 19 (right) shows the sensor’s principle of operation in the example of a front-facing car camera. Regions near the edge of the sensor observe the fastest motion, and, accordingly, the largest, 8x8-pixel super-pixels are used in that region, boosting the local exposure rate by a factor of 64x from the 360fps readout rate to 23,040 exposures/second.
  • the super-pixel size becomes progressively smaller towards the center of the scene, where single unit pixels are exposed at the lowest rate of 360 exp/s.
  • Power-of-2 sized super-pixels are optimally mosaicked together using quadtree tiling.
  • the ADC readout rate is determined by externally-supplied global motion information (e.g., camera motion from an accelerometer/speedometer) and can be scaled down to as low as 30fps reducing the output data and camera digital power by a factor of 12x when motion is slow.
  • FIG. 20 shows experimental results of the sensor of the present embodiments capturing high-speed motion using spatially varying exposure rate.
  • the original scene is depicted in FIG. 20 (top, left) where the two high-speed fans are turned off so that the fan blades are clearly visible.
  • the image in FIG. 20 (top, right) captured by the sensor when using simultaneous uniform exposure for all pixels at 360 exposures/second shows degradation due to motion blur from both fans.
  • FIG. 20 (bottom, left) illustrates how the sensor is then adaptively reconfigured to capture the fast fan ( ⁇ 2200 RPM) with super-pixels of size 2x2 (the left highlighted box), and the faster fan (over 6600 RPM) - with super-pixels of size 8x8 (the right highlighted box) with exposure rate speedups of 4x and 64x, respectively.
  • the corresponding locally boosted exposure rates are 1,440 exposures/second and 23,040 exposures/second, respectively. While the approach proportionally decreases the spatial resolution in these two local regions (by 2 and 8, respectively, in both H and V dimensions), it does not negatively affect the spatial resolution elsewhere in the scene, so those areas are captured at the maximum spatial resolution (640x480) and the nominal speed (360 FPS).
  • HDR high-dynamic-range
  • CEP coded-exposure pixel
  • an intrinsically HDR quantization technique can be used that uses the flux-to-digital converter (FDC) and, particularly, one of its special cases, the regression-based FDC (RFDC).
  • FDC flux-to-digital converter
  • RFDC regression-based FDC
  • the FDC/RFDC constitutes one of the ports in the dual-port image sensor that enables simultaneous exposure coding and HDR readout. This architecture then frees up the coded-exposure capability of the sensor for computational imaging tasks beyond HDR imaging.
  • ADC Analog-to-digital converters
  • ADCs play a crucial role in various data acquisition applications, such as imaging, audio capture, instrumentation, biomedical sensing, industrial automation, and environmental monitoring, among others.
  • ADCs quantize an input analog voltage into a digital number, without having detailed knowledge of the nature of the input voltage waveform. By constraining the input signal through bandwidth and amplitude requirements, one can improve the performance of ADCs by considering trade-offs of area, power, noise, and speed.
  • high-frequency signals in high-speed communication are digitized using very fast but relatively less precise ADCs, with a lower effective number of bits (ENOB), while slower but more precise (higher ENOB) ADCs are used to digitize low- frequency analog signals such as in temperature sensing.
  • ENOB effective number of bits
  • higher ENOB higher ENOB
  • ADCs are integral part of CMOS image sensors (CIS), converting the flux information translated to an analog voltage by pixels into digital numbers.
  • CIS CMOS image sensors
  • a bank of ADCs located at the periphery of the pixel array is used in conventional CIS.
  • stacked-wafer CIS with per-pixel or per-group of pixels ADCs can be used. This approach enables higher pixel density, lower power consumption, and better signal-to-noise ratio, as well as enabling high dynamic range (HDR) imaging.
  • HDR high dynamic range
  • the image sensor can be arranged such that where a stacked-wafer is used for its fabrication, the digital representative values for the incoming light flux can be generated on a lower wafer for each pixel, or for a group of pixels (ideally small group of pixels).
  • CIS mostly conventional ADC architectures are utilized, with the physical layout size and sampling speed being constrained by the pixel pitch and frame readout rate, respectively, with a goal to minimize quantization noise in comparison to the thermal noise from the transistors in the pixel’s analog readout path.
  • FIG. 21A shows a conventional pixel readout path estimating light intensity using conventional ADCs, such as the single-slope ADC.
  • FIG. 21 B shows a pixel voltage waveform in CIS during the exposure and readout phase of a frame where the maximum digitized flux value is determined by the pixel’s full well capacity and exposure time.
  • the signal path in FIG. 21A comprises a photodiode that generates electrons, a tap that stores the photogenerated charge and a source follower, which buffers the readout lines connecting to either column-parallel or per-pixel ADCs.
  • the single-slope (SS) ADC has been the most popular choice in CIS, as shown in FIG. 21A (bottom-left), mainly due to its compactness and linearity, but it typically offers limited dynamic range, speed, and energy efficiency.
  • the SS-ADC comprises a comparator and a shared global ramp generator.
  • FIG. 21 B illustrates the operating principle of a SS-ADC as it depicts the pixel voltage during a frame period.
  • the voltage corresponding to the integrated photogenerated charge is sampled at the end of the exposure period and is compared by the SS-ADC with a ramp reference voltage.
  • the timestamp of the comparator output transient is then converted to a digital number corresponding to the pixel intensity.
  • the full well capacity of the pixel severely limits the dynamic range of the sensor for high flux levels.
  • the output of the SS- ADC is only available at the end of the exposure period, as shown in FIG. 21 B.
  • the SS-ADC is not particularly fast or energy efficient, with the ramp generator often being the limiting factor in terms of ADC speed and power consumption.
  • the present embodiments provide energy-efficient high-dynamic-range flux estimation in conventional pixels using a single-comparison flux-to-digital converter (FDC), as illustrated in FIG. 22A, and a multiple-comparisons regression-based flux-to-digital converter (RFDC), as illustrated in FIG. 22B.
  • FDC flux-to-digital conversion
  • RFDC multiple-comparisons regression-based flux-to-digital converter
  • the approaches can be referred to as flux-to-digital conversion (FDC), as they directly measure the flux instead of the light intensity as is done in traditional ADCs.
  • FDCs the timing of when the photogenerated charge is sampled after the start of an exposure directly depends on the incident flux value, e.g., pixels capturing higher flux values are digitized earlier than pixels capturing lower flux value.
  • Some approaches offer high dynamic range readout but rely on novel pixels which need extra technology modifications and, therefore, can be very expensive to manufacture. Additionally, none of such approaches provide a unified solution of small pixel pitch,
  • One particular FDC approach introduces a subexposure-rate scene-adaptive HDR imaging technique that combines coded-exposure pixel with FDC for high flux values and with conventional ADCs for low flux values. This is a powerful HDR imaging technique, but it does use up the coded-exposure ability of the pixels to perform the HDR FDC. Hence, the coded- exposure feature cannot be used for computational imaging applications other than HDR imaging (or vice versa).
  • Embodiments of the present invention provide a low-power HDR ADC-free regressionbased flux-to-digital converter (RFDC) implemented as a part of the dual-port CIS; where both CEP (Port 1) and FDC/RFDC (Port 2) can operate independently, freeing up the coded- exposure capability for other computational imaging tasks while performing HDR readout.
  • RFDC flux-to-digital converter
  • SPAD Single-photon avalanche diode
  • In-pixel counters can be used to measure the frequency of photon arrival over a wide flux range, where higher pulse frequency corresponds to higher flux values and lower pulse frequency corresponds to lower flux values.
  • the dynamic range of this approach is only limited by the regeneration time of the quenching circuit and the dark current of SPAD, resulting in a very wide dynamic range.
  • SPAD pixels suffer from large pixel pitch, power-hungry peripheral circuits and use expensive fabrication technology compared to conventional CMOS imagers.
  • CMOS image sensors have also been used that utilize flux readout. Some CIS implement flux readout using only FDC, while others combine FDC with ADC or introduce other imaging modalities such as coded-exposure pixels. For instance, utilizing digital pixel arrays to generate a digital pulse after the start of an exposure, which depends on the incident flux value. The digital pixels enable this sensor to quantize a wide dynamic range of flux with improved energy efficiency. However, the in-pixel PMOS of this CIS limits the choice of photodetectors and increases the pixel pitch.
  • CIS features an active-pixel-sensor array with dual-gain pixels and a triple quantization scheme, which is used for HDR imaging.
  • This scheme includes per-pixel FDC to quantize high flux values and per-pixel ADCs to linearly quantize two taps of different conversion gains in each pixel.
  • this architecture benefits from the energy efficiency of a stacked technology, it requires both FDC and ADCs to digitize the entire intensity range, and it can only start readout at the end of exposure.
  • CIS can be used to utilize a PMOS-free coded-exposure pixel array and introduces an HDR imaging technique that combines coded-exposure pixels with FDCs for high flux values and with conventional ADCs for low flux values.
  • the sensor achieves a relatively small pixel-pitch with pinned-photodiode, but relies on the coded-exposure capability of the pixels. Additionally, the readout process requires both FDCs and ADCs, and therefore the output is only available at the end of the exposure, increasing power consumption and latency.
  • Embodiments of the present disclosure can use an ADC-free flux-readout scheme that can output one or more digital HDR flux samples per frame, based on sinusoidal-reference comparator binary outputs.
  • Such readout scheme can be referred to as a regression-based flux- to-digital conversion (RFDC).
  • RFDC regression-based flux- to-digital conversion
  • Such approach offers a much wider dynamic range compared to conventional ADC architectures, and is generally applicable to any (non-SPAD) pixel design and reduces digital processing to a simple pixel-wise linear regression.
  • Such approach has been verified on a CMOS image sensor (CIS).
  • the CIS has a 640 x 480-pixel resolution array that samples light with a global shutter to avoid rolling-shutter artifacts.
  • the FDC techniques presented herein address the challenges encountered by existing FDC approaches and SS-ADC approaches in CIS, while using only simple circuit building blocks such as a comparator and an oscillator.
  • a first technique uses a comparator with a half-period sinusoidal waveform as a reference voltage changing from the pixel saturation level to the reset level. It guarantees that each pixel must cross the reference voltage and, therefore, can be digitized before saturation. While it has advantages in increasing the inherent dynamic range and having low implementation complexity, it can be susceptible to reset noise present in the pixel-tap voltage at the start of an exposure.
  • RFDC regression-based FDC
  • the 22B utilizes a comparator with multiple periods (2 in this example) of a sinusoidal waveform with the voltage ranging from the pixel saturation level to the reset level. This technique also has the potential to enable intra-frame motion estimation.
  • the multiple periods allow the FDC to sample the pixel output corresponding to the same flux value multiple times. These samples are then used to estimate the slope of the pixel voltage, which is the flux value of the light collected by the pixel.
  • the regression-based approach eliminates the reset noise from the estimated flux, as the slope calculation does not rely on the reset level of the pixel voltage, thereby improving the SNR performance.
  • both of these FDC techniques estimate the light flux while the pixel is being exposed, allowing pixel digitization before saturation. Since these FDC techniques operate during exposure, no extra readout time is required, reducing latency.
  • FIG. 23A illustrates the signal path from the photodiode to the ADC comparator and the various noise sources along the path.
  • a photogenerated electron-hole pair is generated, and the probability of this occurrence is determined by the pixel’s quantum efficiency, q.
  • the photogenerated holes are discarded, and the electrons are collected in a charge collection site called a tap.
  • the amount of the photogenerated charge is delivered as the input to the ADC comparator. This approach enables the conversion of the incident photon count into a digital representation.
  • the photogenerated charge is collected in a pixel-tap, CFD.
  • the charge is buffered using an in-pixel source follower, SF, prior to being sampled by an ADC at the periphery of the pixel array.
  • a comparator in the data-converter then compares the sampled value with a known reference voltage waveform, VREF, to generate a stream of 1 -bit comparisons at every clock cycle, which is then processed to estimate the analog value.
  • FIG. 23A The signal path in FIG. 23A is similar for both the SS-ADC and the presented FDC, with the a key difference being the comparator’s reference voltage waveform, VREF.
  • FIGS. 23B and 23C depict graphical representations of the pixel output and the reference voltage waveforms during the SS-ADC and the FDC operation, respectively, assuming for now a noise-free system.
  • the SS-ADC operates after the pixel has been exposed for a certain amount of time, whereas the FDC readout happens in parallel with the exposure.
  • the exposure is divided into /V subexposures, and a pixel voltage, V PIX[n], is compared to V REF[n] at the end of each subexposure, n.
  • the following equations show the number of photogenerated electrons measured by the SS-ADC, ADC[n], and the FDC, FDC[n], corresponding to a /og2(/V)-bit digital number, n:
  • n represents the comparison index when the comparator output toggles (for example, SS-ADC 1->0 and FDC 0— >1)
  • N is the total number of comparisons
  • FWC is the full well capacity in terms of the number of electrons
  • V REF[n] is the reference voltage during the n th comparison
  • V SAT is the saturation voltage
  • V RST is the reset voltage of the pixel tap.
  • motion artifacts depend on the pixel brightness and can be compensated for during post-processing by a digital processor, if needed, similarly to how it is done for motion blur compensation.
  • any electronic signal path in reality is not without noise.
  • five primary sources of noise in the pixel-readout path can be taken into account: (1)
  • the input of CIS-ADC is also subject to the thermal noise as it is buffered from the pixel tap to the ADC input through analog amplifiers.
  • the resistors and the transistor channels in the signal path add thermal noise to the signal.
  • This noise is herein referred to as the read noise, V n re ad, and its main contributors are the noise from the in-pixel source follower, V nsp, and the input-referred noise of the comparator including the clock jitter, V n C om P .
  • V A denotes the supply voltage noise and thermal noise due to reset transistor that is sampled on the pixel tap when it is reset between two exposures.
  • V n re f The noise from the reference voltage generator of the comparator is denoted by V n re f.
  • the pixel voltage has an additional noise due to the probabilistic nature of photons, known as the photon shot noise, denoted by ln S hot- The photon shot noise follows the Poisson distribution, and the noise power is equal to the square root of the signal power.
  • CIS-ADCs are designed to keep the quantization noise below or equal to the readout noise across the entire pixel voltage range.
  • the SNR of the digitized pixel output is dictated by the photon shot noise. Understanding and mitigating these noise sources can be used to achieve high-quality HDR imaging.
  • Conventional ADCs are generally over-designed when considering photon shot noise at higher intensities leading to a sub-optimal use of resources.
  • F[n] denotes the flux during subframe n in normalized units, and N represents the total number of subexposures in an exposure.
  • the term ED[n] represents the total number of electrons accumulated in the tap due to leakage and dark current Id during subexposure n.
  • the Poisson probability distribution with mean represented by Pois(p) is employed to introduce shot noise to the ideal photon flux and dark current in the pixel tap.
  • EC[n] signifies the number of electrons collected in the tap by the end of subexposure n.
  • the reset noise is sampled at the start of the exposure as EC[0].
  • the function Norm( ,a) generates a random number based on a normal distribution which simulates noise with a mean and a standard deviation o.
  • FWC is the full well capacity of a pixel tap represented by the number of electrons.
  • V PIX[n] represents the normalized pixel voltage as seen by the input of the comparator.
  • variables ENOBRO, ENOBVREF and ENOBVRST establish the standard deviation of noisy signals, which corresponds to the noise power required to achieve the required effective number of bits (ENOB) of resolution in the pixel voltage signal readout path, V PIX, reference voltage, V REF, and reset power supply, V RST
  • the read noise in the signal path is the primary contributor to the output noise power, with the quantization error of the SS-ADC being less than the read noise.
  • the shot noise from the incident photons becomes the dominant noise source. This transition leads to an increased headroom between the total noise and the quantization error, which may be perceived as inefficiency with respect to ADC performance parameters such as power, speed, and dynamic range.
  • the FDC method of the present embodiments addresses these data converter inefficiencies and enhances the inherent dynamic range by non-uniform quantization process.
  • the simulation results for the FDC determined its capability to digitize a considerably broader flux range whilst employing the same number of digital codes as the SS-ADC.
  • the FDC facilitates the sampling of flux prior to the saturation of the pixel tap and digitizes flux exceeding the pixel’s saturation limit, which in turn increases the dynamic range.
  • intensities 20 dB above the pixel’s full well capacity the SNR performance of the pixel is constrained by the quantization error.
  • the SNR performance of the FDC is limited by photon shot noise, similar to the performance of SS-ADCs.
  • FIGS. 24A and 24B illustrates a noisy pixel voltage can be sampled multiple times during a crossing.
  • FIG. 24A shows example of noisy readout signals from two pixel taps sampled by a noisy VREF signal.
  • FIG. 24B shows average number of comparator samples in the singlecrossing FDC with a sinusoidal reference voltage, in a practical case of a noisy signal and a noisy reference voltage.
  • the pixel-tap voltage which is largely affected by the read noise, ends up being sampled multiple times and generates multiple measurements for the same flux within an exposure. These FDC outputs are subsequently averaged, resulting in a reduced read noise contribution to the total noise power.
  • FIG. 24A shows the output voltage of two pixels each receive varying flux along with the sinusoidal reference voltage.
  • the output voltage of PIXELS changes at a slower pace than that of PIXEL2, thereby generating three FDC samples from PIXELS but only a single FDC sample from PIXEL2.
  • the multiple FDC samples are averaged, therefore suppressing the read noise contribution in the output of PIXELS.
  • FIG. 24B demonstrates the average number of samples (and their standard deviation) generated over the entire flux range in the simulation presented.
  • the quantity of samples steadily declines as the flux increases until, at high flux values, only a single FDC sample is procured in a frame.
  • the averaging of multiple noisy FDC samples can markedly diminish the contribution of readout and reference voltage noise at lower flux values.
  • Such averaging is not feasible with conventional CIS-ADCs, as they digitize a single sampled value at the conclusion of the exposure.
  • the single-crossing FDC method for digitizing CMOS pixel voltage outputs presents several advantages over conventional ADC methods, such as the ability to capture a wider dynamic range and reduced frame latency.
  • the use of non-uniform quantization and averaging multiple samples results in a reduced read noise at lower flux levels. This has been possible due to the pixel voltage-specific signal constraints, for example, linearly increasing voltage and the SNR of the signal limited by the inherent photon shot noise.
  • the simulation results demonstrate the effectiveness of the FDC method in addressing the limitations of conventional ADC designs for CIS.
  • the single-crossing FDC in these simulations uses a half cycle of the sinusoidal signal as the reference voltage.
  • n denotes the comparison number at the end of the subexposure
  • /V represents the total number of subexposures.
  • V REFCONST remains constant;
  • V REFRAMP linearly transitions from saturation to reset value;
  • V REF SINE changes from saturation to reset value in a sinusoidal manner wherein half of the period equates to the total exposure time;
  • V REFANGLE uniformly samples the flux in the radial (angular) domain; and
  • V REF QUAD samples the flux such that the quantization error is always proportional to the photon shot noise.
  • V REFCONST The simplest reference voltage waveform for FDC is V REFCONST, which is a constant voltage throughout the exposure period. This waveform, despite being easy to generate, allows the FDC to digitize only when the flux surpasses a certain threshold. As a result, it needs additional techniques to quantize lower flux levels, such as the coded-exposure pixels or dualgain pixels with additional help from conventional ADCs. Moreover, at higher flux values, its signal-to-noise ratio (SNR) falls short compared to the FDC with V REFSINE.
  • SNR signal-to-noise ratio
  • V REFRAMP transitions linearly from the saturation level, V SAT, to the reset level, V RST, over a full exposure period. Consequently, every flux value gets sampled once within the exposure period.
  • the SNR performance of the FDC with V REFRAMP 'IS nearly identical to that of the FDC with V REFSINE. Therefore, expending additional power to generate a precise V REFRA p svn ramp-generator circuits may be deemed unnecessary.
  • a low-noise sinusoidal voltage can be generated using minimal power. For instance, this can be achieved by a resonantly clocked LC tank with small resistive losses.
  • the FDC with reference voltage V REF ANGLE can divide the flux into uniform radial steps of TT/2N radians each.
  • the SNR performance of FDC with V REF ANGLE matches that of V REFSINE at lower flux levels.
  • V REFSINE consistently outperforms V REF ANGLE- While the immediate applications for such quantization may not be apparent, it highlights FDC’s flexibility with the reference voltage, enabling users to quantize flux as per their specific requirements. For instance, a quantization strategy can be used that leverages inherent photon shot noise to capture HDR scenes, ensuring the quantization error is always proportional to the photon shot noise.
  • a multi-crossing regression-based FDC in accordance with the present embodiments, can be sued that has a sinusoidal reference voltage.
  • RFDC improves the SNR of the measured flux using multiple samples (beyond those due to noise) and reduces the reset noise contribution.
  • the multiple samples from the RFDC also enable motion estimation within a single exposure and can be used to further reduce motion artifacts.
  • the RFDC method of the present embodiments can utilize a multi-period oscillating sinusoidal reference voltage. This choice offers benefits in terms of both area and SNR performance, without a significant power penalty.
  • the generation of higher-frequency sinusoidal signals for more robust and quicker flux estimation demands only a minor additional energy cost; for example, it simply requires smaller inductor or capacitor values in a resonator, thereby reducing the area.
  • FIGS. 25A to 25D show simulated performance of regression-based flux-to-digital conversion (RFDC) with a sinusoidal reference voltage.
  • FIG. 25A shows sinusoidal reference voltage with up to 4 and with up to 20 crossing during an exposure.
  • 25B shows SNR comparison between RFDC with increasing number of sinusoidal crossings ranging from 1 to 32 during exposure over a wide flux range.
  • FIG. 25A depicts two examples of the sinusoidal reference voltage used in the RFDC, which permits up to 4 and up to 20 crossings of the reference voltage and the pixel-tap voltage.
  • This approach yields multiple comparator output transients spread out over one exposure period, which can then be used to improve the flux estimations.
  • different numbers of voltage crossings can occur: lower-flux pixels cross the sinusoidal reference voltage the maximum number of times (four times in the example shown), and higher-flux pixels cross at least once.
  • linear regression we use linear regression as it also reduces the effect of the reset noise (corresponding to the offset of the fitted line) and acts as a viable alternative to the conventional correlated-double-sampling (CDS) technique.
  • CDS correlated-double-sampling
  • FIG. 25B maps the SNR of the RFDC method with a sinusoidal reference voltage and varying numbers of crossings, ranging from 1 to 32, within a single exposure period.
  • FIGS. 25C and 25D display the SNR plot for the RFDC with a sinusoidal reference voltage, allowing for up to 4 and up to 20 crossings, respectively. Both of these figures also depict the SNR plot for the single-crossing FDC with a sinusoidal reference voltage, which is limited by reset noise. It is evident from the figures that the RFDC method consistently outperforms the single-crossing FDC, particularly at lower flux values, where the RFDC mitigates the effect of reset noise by fitting the line through multiple crossings of the same flux value.
  • a higher frequency sinusoidal reference voltage waveform may be used by applications where the motion estimation is of higher importance than the SNR.
  • Linear regression was selected for the RFDC as it is best suited for application-specific integrated circuit (ASIC) designs. Compared to other types of regression, such as polynomial regression or logistic regression, linear regression requires less computational resources and is more power-efficient. It is implemented using simple arithmetic operations, such as multiply and accumulate (MAC), which can be performed efficiently using hardware circuits. In contrast, other types of regression, such as polynomial regression, require more complex mathematical operations, such as exponentiation, which are more computationally expensive and power- hungry. Logistic regression requires additional operations, such as the sigmoid function, which adds to the computational cost.
  • MAC multiply and accumulate
  • linear regression is more easily scalable than other types of regression, making it suitable for large-scale image sensors.
  • the simplicity of its implementation allows for more parallel processing, which can significantly improve processing speed and reduce power consumption. Its simplicity, scalability, and low computational requirements make it a suitable choice for many applications that require high processing speeds and low power consumption.
  • FIG. 26A illustrates a CIS block diagram of a VLSI implementation and FIG. 26B illustrates a dual-tap coded-exposure pixel schematic.
  • FIG. 26A presents the block diagram of the image sensor.
  • the sensor in this example includes a front-side illuminated array of 640 x 480 pixels.
  • a bank of 340 comparators designated for the RFDC.
  • the outputs from these comparators are serialized and subsequently transmitted to an FPGA by way of a bank of 17 sets of 20:1 serializers, that can operate at a frequency of 200 MHz.
  • the sensor’s pixel array is composed of a dual-tap coded-exposure pixel (CEP).
  • CEP coded-exposure pixel
  • PPD pinned photodiode
  • the intermediate storage diode which functions as a global buffer when the signal TG _GLOB is triggered for all pixels simultaneously, thereby enabling global-shutter operation.
  • ROW SEL Once a row is selected using ROW SEL signal, the charge is transferred from the storage diode to one of the taps depending on the digital CODE signal.
  • the in-pixel source follower buffers this charge on column-parallel readout lines that extend to the comparator at the periphery of the pixel array.
  • the pixel Due to its NMOS-only structure, the pixel achieves a small coded-exposure pixel pitch of 7 pm and a fill-factor of 38.5%.
  • the signals DRAIN and RST are utilized to reset the photodiode and the taps at the start of each frame, respectively.
  • the RFDC can be utilized with any CMOS pixel, not necessarily with a CEP.
  • the resulting effective single-tap pixel structure is shown in FIG. 27A. This configuration ensures that the photo-generated charge is always collected in TAP , while disabling the other tap, TAP2.
  • FIG. 27A shows a FDC/RFDC readout path
  • FIG. 27B shows a strong-arm latch comparator schematic
  • FIG, 27C shows a timing diagram for exposure and readout using FDC.
  • FIG. 27A illustrates the implemented signal path for pixel readout using the FDC/RFDC.
  • This signal path includes a comparator, as shown in FIG. 27B, with one terminal connected to a pair of neighboring readout lines from odd and even columns via an analog multiplexer. The other terminal is connected to the reference voltage.
  • the sensor has a bank of 340 comparators at its periphery, with 320 comparators connected to the 640 columns of the pixel array, while the additional 20 comparators are used for general testing and debugging. All comparators’ reference voltage terminals are connected to one of the analog IO-pads of the chip, giving users the flexibility to supply the reference voltage according to their application’s needs.
  • FIG. 27C presents the timing diagram of the sensor operation. Both exposure and readout take place simultaneously within a frame.
  • the exposure time is divided into N subexposures, from O to N - 1.
  • the RFDC comparison for the flux captured in subexposure / is carried out during subexposure / + 1.
  • the RFDC operates on the photogenerated charge stored in the pixel tap by the end of subexposure /.
  • the COL SEL signal sequentially links the comparator bank to odd and even columns of the array within a given subexposure. For each COL SEL state, the row selector cycles through all row addresses. This scanning method of the array is chosen over switching between odd and even columns for each row, as it allows equal settling time for both odd and even columns, thus reducing column fixed pattern noise.
  • the comparator uses the strong-arm latch architecture and can function at a clock frequency exceeding 32 MHz, sufficient for performing up to 1000 full-frame 1-bit comparisons at 30 FPS with RFDC readout, consuming a mere 8.56 mW.
  • the energy efficiency is expected to be even higher if pixels and comparators are 3D-stacked, with low interconnect capacitance.
  • a one-bit binary frame buffer can be used to detect and send out only the comparator transient timestamps, which reduces the output data rate to or below that of most conventional image sensors.
  • the linear regression adds a very small power overhead, requiring the computation of an inner product of two low-dimensional vectors.
  • FIG. 28 shows a simplified basic example of the sinusoidal reference voltage generation concept for FDC using a digital pulse, PWM INPUT, controlled by an FPGA. Simulations confirmed that a sinusoidal signal serves as an optimal choice for the RFDC reference voltage waveform.
  • This reference voltage signal can be effortlessly generated with digital control signals and passive components.
  • the reference voltage signal for the FDC is bundled together and directly connected to the pre-amplifier gate in the strong-arm latch. This signal operates at a very low frequency and consumes negligible amounts of current. For a frame rate of 30 frames per second (FPS) and the sinusoidal reference voltage with up to 4 crossings, the FDC requires a sine wave of 60 Hz frequency.
  • FPS frames per second
  • the generated reference voltage waveform won’t be a single-tone sinusoidal due to non-idealities and component mismatches, the FDC performance remains almost unaffected, as demonstrated by the comparison between V REFRAMP and V REFSINE.
  • V REF[n] it is generally important to accurately measure V REF[n] at each comparison. Therefore, after finalizing the parameters for the LC tank, RPOT, and duty cycle, the generated reference voltage can be precisely measured using an oscilloscope just once and then utilized for all subsequent FDC calculations.
  • the on-chip bandwidth of the reference voltage IO pin is significantly higher than the reference voltage frequency, as it is constrained by routing-metal resistance and parasitic/gate capacitance which have relatively low values. This results in minimal reference voltage signal degradation due to on-chip parasitics. Therefore, a low-power reference voltage generation method is suitable for the RFDC.
  • the sensor of the present embodiments was compared with other CISs and a SPAD imager, all of which offer some form of flux-to-digital conversion.
  • the sensor of the present embodiments uses pinned-photodiode and can be fabricated using a standard 110 nm CMOS image sensor process. It achieves the smallest pixel pitch among non-stacked sensors and has highest pixel resolution compared to all other CIS, while offering global shutter exposure.
  • the FDC method described herein can be utilized alongside any conventional global- shutter or rolling-shutter pixel architectures that offer even the smallest pixel pitch. Further scaling can be realized by integrating the FDC architecture with stacked technologies. The per- pixel stacking reduces the readout lines’ capacitance, thus together enhancing the power efficiency of the sensor. Additionally, the stacked architecture eliminates the necessity of scanning all rows for every FDC comparison. This reduction in subexposure time can help further decrease quantization noise, increase dynamic range, and enhance the SNR at higher flux values.
  • the performance of FDC is demonstrated at 30 FPS while the power consumption stands at 8.58 mW, achieving a dynamic range of 95 dB.
  • the low power consumption of the FDC of the present embodiments can be attributed to power-efficient sinusoidal reference voltage generation, reduced sampling speed, and negligible static power consumption from the strong-arm latch comparator.
  • the linear regression for the RFDC could be implemented on-chip, as this regression relies on simple low-dimensional vector MAC operations.
  • the FDC demonstrates the capability to digitize very high flux values.
  • the FDC architecture relies on constant integration of the photogenerated charge in the pixel tap, i.e., floating diffusion, which is affected by charge leakage and dark current over time.
  • MIM metal-insulator-metal
  • the multi-crossing RFDC improves SNR by capturing multiple samples of the same flux value
  • the present approach could be further extended to enable motion estimation. Changes in the scene’s motion result in a shift of flux at a pixel.
  • the measurement would contain motion blur.
  • a multiple piece-wise linear fits on subgroups of samples can be performed to estimate this flux change.
  • the regression-based flux-to-digital converter (RFDC) of the present embodiments offers ADC-free HDR quantization in CISs.
  • RFDC flux-to-digital converter

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)

Abstract

L'invention concerne un procédé et un capteur d'image pour une conversion flux-numérique pour un pixel d'un capteur d'image. Le procédé consiste à : recevoir de la lumière pour un pixel dans le capteur d'image tandis que le pixel est exposé à la lumière pendant une période d'exposition, le pixel fournissant une sortie qui est une fonction de la lumière reçue ; effectuer un ajustement de ligne sur les transitoires de la sortie de pixel, les transitoires étant déterminés à l'aide d'une tension de référence périodique ; et générer la valeur d'ajustement de ligne en tant que représentant numérique du flux de lumière reçu. Le capteur d'image comprend un premier port permettant de recevoir un code d'exposition qui configure une exposition des pixels dans le réseau ; et un second port permettant de lire la valeur représentative numérique, un ajustement de ligne étant effectué sur des transitoires de la lecture, les transitoires étant déterminés à l'aide d'une tension de référence périodique, et la valeur d'ajustement de ligne étant la valeur représentative numérique.
PCT/CA2024/050187 2023-02-14 2024-02-14 Capteur d'image et procédé de conversion flux-numérique et de codage d'exposition par pixel WO2024168432A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US202363484845P 2023-02-14 2023-02-14
US63/484,845 2023-02-14
US202363485628P 2023-02-17 2023-02-17
US63/485,628 2023-02-17

Publications (1)

Publication Number Publication Date
WO2024168432A1 true WO2024168432A1 (fr) 2024-08-22

Family

ID=92421394

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CA2024/050187 WO2024168432A1 (fr) 2023-02-14 2024-02-14 Capteur d'image et procédé de conversion flux-numérique et de codage d'exposition par pixel

Country Status (1)

Country Link
WO (1) WO2024168432A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018195669A1 (fr) * 2017-04-28 2018-11-01 The Governing Council Of The University Of Toronto Procédé et système d'imagerie par pixels
WO2020252592A1 (fr) * 2019-06-21 2020-12-24 The Governing Council Of The University Of Toronto Procédé et système pour étendre une plage dynamique d'image à l'aide d'un codage par pixel de paramètres de pixel

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018195669A1 (fr) * 2017-04-28 2018-11-01 The Governing Council Of The University Of Toronto Procédé et système d'imagerie par pixels
WO2020252592A1 (fr) * 2019-06-21 2020-12-24 The Governing Council Of The University Of Toronto Procédé et système pour étendre une plage dynamique d'image à l'aide d'un codage par pixel de paramètres de pixel

Similar Documents

Publication Publication Date Title
US10277843B2 (en) Oversampled image sensor with conditional pixel readout
CN110753192B (zh) 一种集成电路图像传感器
CN105794203B (zh) 高动态范围图像传感器
US7859447B2 (en) Image processing method, semiconductor device for detecting physical quantity distribution, and electronic apparatus
US9030581B2 (en) Solid-state imaging device, imaging apparatus, electronic appliance, and method of driving the solid-state imaging device
US8031241B2 (en) Solid-state imaging device and imaging apparatus
US8547460B2 (en) Solid-state imaging element, driving method therefor, and camera system with a pixel drive control unit concurrently accessing a window region and the adjacent outer row
Sasaki et al. A wide-dynamic-range CMOS image sensor based on multiple short exposure-time readout with multiple-resolution column-parallel ADC
JP2010520707A (ja) 画像センサのダイナミックレンジを向上及び制御する方法及び装置
KR101927326B1 (ko) 모바일 어플리케이션용의 ｃｍｏｓ 이미지 센서에 대한 다이나믹 레인지 확장
EP2832090A1 (fr) Capteurs d'image cmos mettant en oeuvre un échantillonnage double corrélé numérique plein cadre avec un obturateur global
JP2016526817A (ja) スプリットゲート条件付きリセットイメージセンサ
KR20080067963A (ko) 고체 상태 촬상 장치 및 촬상 장치
KR20090017444A (ko) 고체 촬상 소자 및 카메라 시스템
KR20120088605A (ko) 모바일 어플리케이션용의 ｃｍｏｓ 이미지 센서에 대한 다이나믹 레인지 확장
KR20220022481A (ko) 픽셀 파라미터의 픽셀 단위 코딩을 사용하여 이미지 다이나믹 레인지를 확장하기 위한 방법 및 시스템
JP4157862B2 (ja) アナログ‐デジタル変換器及びその動作方法、撮像装置並びにランプ発生器の動作方法
JP2010259109A (ja) 半導体装置の制御方法
WO2024168432A1 (fr) Capteur d'image et procédé de conversion flux-numérique et de codage d'exposition par pixel
JP2010273385A (ja) 半導体装置の制御方法
Skorka et al. Bio-inspired design: Nonlinear digital pixels for multiple-tier processes
Hong et al. High-speed, high sensitivity 25 mega pixel cmos image sensor with column parallel 12 bit hybrid adc architecture
JP4521050B2 (ja) アナログ‐デジタル変換器及びその動作方法、ランプ発生器及びその動作方法並びに撮像装置
Okada et al. A 4K2K 60-fps Image Sensor Based on Stagger-laced Dual-exposure Technique
Bae et al. A high dynamic range CMOS image sensor with a digital configurable logarithmic counter

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24755800

Country of ref document: EP

Kind code of ref document: A1