WO2009073054A1 - Dispositif d'imagerie perfectionné - Google Patents

Dispositif d'imagerie perfectionné Download PDF

Info

Publication number
WO2009073054A1
WO2009073054A1 PCT/US2008/009739 US2008009739W WO2009073054A1 WO 2009073054 A1 WO2009073054 A1 WO 2009073054A1 US 2008009739 W US2008009739 W US 2008009739W WO 2009073054 A1 WO2009073054 A1 WO 2009073054A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel
integration period
reset
row
column
Prior art date
Application number
PCT/US2008/009739
Other languages
English (en)
Other versions
WO2009073054A8 (fr
Inventor
Jon H. Bechtel
Original Assignee
Gentex Corporation
Andrus, Jeremy, C.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US11/999,623 external-priority patent/US8289430B2/en
Priority claimed from US12/082,215 external-priority patent/US8629927B2/en
Priority claimed from US12/150,234 external-priority patent/US8305471B2/en
Priority claimed from US12/157,476 external-priority patent/US8587706B2/en
Application filed by Gentex Corporation, Andrus, Jeremy, C. filed Critical Gentex Corporation
Publication of WO2009073054A1 publication Critical patent/WO2009073054A1/fr
Publication of WO2009073054A8 publication Critical patent/WO2009073054A8/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/76—Circuitry for compensating brightness variation in the scene by influencing the image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/53—Control of the integration time
    • H04N25/531—Control of the integration time by controlling rolling shutters in CMOS SSIS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/53—Control of the integration time
    • H04N25/533—Control of the integration time by using differing integration times for different sensor regions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/71—Charge-coupled device [CCD] sensors; Charge-transfer registers specially adapted for CCD sensors
    • H04N25/74—Circuitry for scanning or addressing the pixel array

Definitions

  • Imaging devices including CMOS and CCD versions, are becoming popular in a number of contexts.
  • the present invention provides an improved imaging device having high dynamic range and various apparatuses incorporating these high dynamic range imaging devices.
  • FIGs. 1 and 2 depict a pattern for a reset and row read schedule
  • Fig. 3 depicts a timing sequence
  • Fig. 4 depicts a block diagram of an exemplary design
  • FIGs. 5 and 5a depict simplified diagrams of circuits associated with one column of pixels in an imaging array
  • FIGS. 6 and 6a depict simplified schematic representations of a 3T pixel
  • FIGS. 7 and 7a depict simplified schematic representations of a 4T pixel
  • Fig. 8 depicts a dual slope pixel response; )10] Fig. 9 depicts a detailed outline of timing for making a determination of what integration period to select for a pixel;
  • FIG. 10 and 10a depict simplified diagrams of column logic
  • Fig. 11 depicts a normalization circuitry block diagram
  • Figs. 12 and 12a illustrate pictorially organization and repetition of components which are described verbally in connection with Figs. 5 for 3T based designs and Fig. 10; )14]
  • Fig. 13 depicts various mathematical conversion routines;
  • Fig. 14 depicts an automatic vehicle equipment control system
  • Fig. 15 depicts a controlled vehicle with various equipment
  • Figs. 16a and 16b depict a vehicular interior rearview mirror assembly
  • Fig. 17 depicts an exploded, perspective view of an accessory and rearview mirror mount assembly for a vehicle; and 119] Fig. 18 depicts a profile view of a digital camera.
  • Consumer electronics devices such as, digital cameras, digital video recorders, video telephones and the like are becoming commonplace. These devices along with a host of residential, commercial, industrial and vehicular vision systems have popularized CCD and CMOS based image sensors.
  • CMOS based image sensors have popularized CCD and CMOS based image sensors.
  • Known systems designed to automatically control vehicle exterior lights for example, utilize a forward looking digital imaging system to acquire images of the scene generally in front of and/or behind the controlled vehicle and analyze images to detect headlights of oncoming vehicles and taillights of leading vehicles.
  • Security cameras are prevalent throughout residential, commercial and industrial facilities, as well as, associated vehicle parking areas.
  • light sources within the scene comprise such drastically varying relative brightness that a given pixel exposure may not allow for detection of dim light sources while not saturating due to bright light sources.
  • objects in the scene are primarily illuminated by the sun rather than the objects themselves being sources of light (i.e. headlights, taillights, road signs, etc.).
  • the present invention provides a high dynamic range image sensor capable of accurately detecting and digitally representing dim light sources, bright light sources and objects illuminated by other light sources in the same scene.
  • a number of imaging techniques to provide high dynamic range images are known in prior art. These include selective reset of pixels in various combinations.
  • One embodiment in US patent 6,175,383 includes choice of available integration periods for which the second available integration period is half the first, the third is half the second and so on. In other words, for a successive list of available integration periods, each available integration period is shorter than its immediate predecessor by a factor of two.
  • the patent also notes the possibility of using nondestructive readout to determine the optimum exposure period for a region of interest.
  • the patent depicts monitoring of pixel values by using a conventional readout of a row of pixels followed by serialized pixel by pixel processing of the pixels of the row prior to sending them to an evaluation circuit.
  • the patent does not, however, detail a fast responding, automated method to determine what integration period to select for each pixel of the image while a readout sequence for a given image is in progress.
  • Another prior art device incorporates a periodic sample period for which every pixel in the imager is scanned during each sample period with a mechanism for conditional reset of each pixel on each successive scan of the array.
  • the device has a memory to record reset tasks and a single analog output to which each of the analog pixel values from the entire imaging array are presented in sequential fashion and routed to a single comparator.
  • the accumulation of enough charge to reasonably indicate if the current integration period is appropriate followed by the decision to reset to begin the shortest integration period followed by conditional reset of the pixel needs to be accomplished just 6 microseconds after the preceding reset.
  • row parallel processing is incorporated to read a signal indicative of the light induced charge for each pixel of the row; to compare the indicated level of the light induced charge against a threshold level for each pixel of the row; to, for each pixel of the row, interrogate the memory record of the reset condition; to determine whether or not to reset each pixel of the row based on the result of the comparison, the memory indication, and the stage in the readout sequence; to store an updated record of the partial reset history, as required, for each pixel of the row; and finally to conditionally reset each pixel of the row based on the above information for the respective pixel.
  • the set of pixels constituting a given spectrally filtered group i.e. red filtered pixels within a row or blue pixels within a row or green pixels within a row.
  • the pixel value associated with the accumulated charge is preferably read and preferably converted to digital form in a conventional manner.
  • the readout of analog values from the row of pixels is done in a row parallel manner and the values read and temporarily stored for the row are then preferably converted to digital form.
  • the digital record of the integration period selected for each pixel is preferably combined with the pixel readout value for the respective pixel to establish a value indicative of the received light level for the pixel.
  • the value is preferably in a form which is directly usable so that it is not necessary to retain the separate record of the integration period used to integrate the light level to obtain the pixel reading.
  • the encoding of this value may assume any of a large number of formats and the value may, for example, be in linear or in logarithmic form.
  • the processed pixel value is preferably made available for readout at this point.
  • one or more of the steps may be accomplished by a processor which is not part of the imaging array.
  • Pixels which are successively reset begin a shorter integration period with each successive reset giving the pixel an effective optical sensitivity which is successively reduced in approximate proportion to the reduction in integration period for the pixel for each reset task of the sequence.
  • conditional reset logic it is preferable to configure the conditional reset logic so that once a conditional reset for a given pixel is skipped; the pixel will not be reset again until it is read.
  • the pixel signal indicative of the accumulated light induced charge is accumulated for the pixel during a partial, not a full, integration time. It is preferable to select the compare threshold against which to compare the indicated pixel charge after partial integration so that the threshold is high enough that sufficient charge will be accumulated at the end of the integration time to reasonably utilize the readout range of the pixel.
  • the threshold is preferably not set so high that the pixel is likely to saturate before the end of the integration period.
  • the circuit is preferably arranged so that to establish the integration for readout of each pixel, a prescribed sequence of row parallel resets is performed for each row.
  • the first row parallel reset task for a row read sequence is preferably unconditional and the memory associated with each pixel is preferably updated to reflect the initial reset state.
  • successive resets of the row are preferably conditional as outlined above.
  • the optical gain for the integration period initiated by each successive row parallel conditional reset task be ratiometrically less than the optical gain for the integration period initiated by the immediately preceding row parallel reset task.
  • the optical gain is normally proportional to integration period so in the exemplary design, the desired ratios will be applied directly to the integration periods.
  • the integration period is actually set for each pixel during the exposure sequence in which the pixel reading is taken.
  • the exposure is rather closely bracketed for each pixel and is, for example, adjusted in 4 to 1 increments in the exemplary embodiment so if a 10 bit readout is used, the readings may nominally extend from one quarter of full scale to full scale nominally giving readout resolution of 250 to 1 or better over a very large range of light level. This is ideal and some extra margin may be provided to minimize the likelihood of saturation of the pixels.
  • the full accuracy of the A/D is still realized over the selected integration period which is known and precisely determined so that the dramatically increased measuring range is realized without compromising the readout accuracy of the A/D.
  • the noise, banding, and totally blacked out areas in shadowed regions should be dramatically reduced as should saturated highlights. There are some compromises to be made but many possibilities for great improvement for still picture or video camera applications.
  • Many of the techniques for nondestructive readout do not provide full accuracy for the readout. For example, a pixel is not normally resetable for a nondestructive readout so with these imagers reference readings taken after reset cannot be used to improve readout accuracy for intermediate readings used to determine conditional reset.
  • the exemplary system also has inherent advantages over wide dynamic range sensors which use multiple slopes in the response characteristics to increase effective range.
  • the nonlinearity for each pixel is normally determined at least in part by components in each pixel so in addition to the difficulty in processing readings taken with a piecewise linear readout, the slopes and breakpoints are not likely to match well because of mismatches in pixel level characteristics of the breakpoint and slope determining mechanism.
  • the slope changes are normally applied in the analog domain, the entire value must be read by the A/D with further loss in readout accuracy for an A/D of a given resolution.
  • the piecewise linear scaling of data is not friendly to processing because in addition to the problems with readout accuracy caused by the factors above, the piecewise linear format requires somewhat calculation intensive processing to convert it to a more calculation friendly form such as linear or perhaps logarithmic.
  • peak dynamic range may often be lowered in exchange for the extended dynamic range provided by the multiple integration period acquisition taking place within individual images.
  • the row parallel reset process to select the integration period for each pixel in the sensing array is applied in a sequence which for any given row is begun by an unconditional parallel reset of pixels in the row to begin the longest attainable integration period for reading (given the imager settings) each pixel of the row. Then just before 3/4 of the integration period has elapsed, a conditional reset of each pixel in the row as described above is done to conditionally reset pixels which are saturated or too likely to saturate with the current integration period setting. This process is repeated six more times with each succeeding integration period to which pixels are conditionally set sized to make the readout sensitivity one quarter of what it was for the immediately previous range.
  • the integration period may range in 4 to 1 increments from approximately 32,768 microseconds for the longest selectable integration period to 2 microseconds for the shortest selectable integration yielding a ratio of 16,384 to 1 between the longest and shortest of the integration periods which are automatically and individually set for each pixel in the frame for the exemplary design.
  • the timing cadence for the reset and read process is preferably based on the column by column incremental scan through the column scan counts corresponding to each pixel of a scan row.
  • the completion of an incremental scan through the column count or column address values for a given row count then preferably results in the incremental advance of the row scan count to the next scan row until all of the rows in the image scan frame have been covered at which point the image acquisition may halt or advance to another column and row scan sequence.
  • the time interval between two pixel locations in the pixel/row scan frame is expressed as the number of incremental pixel times to advance from the (column, row) location of the first pixel to the (column, row) location of the second pixel.
  • the incremental pixel times between the starting and ending pixel locations may be expressed in terms of an incremental column offset which may be defined as the column address of the second pixel minus the column address of the starting pixel.
  • the row offset may be defined as the incremental row offset required to increment from the row containing the starting pixel to the row which contains the ending pixel.
  • the row read is done at a prescribed column time and each of the eight respective integration periods are initiated at a column time associated with the respective integration period such that the time interval between the point at which the reset is just completed to the point at which the row is read is nearly equal to the intended integration time.
  • the number of columns and rows in the scan frame were selected with 626 columns by 420 rows and with a pixel scan rate of 8 pixels per microsecond.
  • integration times and available number of integration rows may be, and are chosen so that the row read task and each of the 8 pixel reset tasks for the 8 available integration time periods occur over ranges of row column time intervals which are non-overlapping.
  • the row select circuit, the array read-out column lines, the row of column parallel reset logic, and reset history memory access may be shared without conflict between the 8 reset tasks or the row read task.
  • Each of the eight integration time specific row reset or conditional row reset tasks and the row read are preferably initiated at specified column counts. This will be covered in more depth elsewhere herein. In general, in the exemplary embodiment, the row length etc.
  • a specified column count value is preferably used to initiate each of the sequences and is preferably pre-selected so that adherence to the timing provides the proper column offset to create the proper interval for each of the integration periods.
  • readout is preferably performed on the row pointed to by the row counter, possibly modified by a row offset value preferably used for providing an origin for the readout window.
  • an additional integration period specific row offset value which corresponds to the integration period for which the reset or conditional reset is being made is added to the row count, preferably using modulo arithmetic, to select the row to which to apply the reset task.
  • This added integration period specific row offset is used to set the row time interval between the reset task and the row read task for the row to which the current reset task is applied.
  • the scan is a repeating sequence which cycles from column 0 through column 625 for each row and at the completion of each column cycle it advances to the next row from row 0 through row 419 and back to row 0 etc. for as many image frames as are to be acquired in the readout sequence.
  • the first scan, or possibly partial scan through the image frame is normally used to initialize integration periods with readout of rows being suppressed until an integration period has been provided for the row. This implies an initial frame (or partial frame) is needed to perform the first reset tasks so that to readout n frames, n frames plus the portion of the frame needed for the initial reset are needed for the readout sequence.
  • interest may be primarily to read images in a continuous mode in which case suppression of readout of the startup frame may not be necessary.
  • integration is interleaved with readout so that the frame repetition time needs to be only slightly longer than the longest integration time provided for a pixel.
  • synchronization may be established by counting pixels.
  • a readout format which includes signaling to establish row and frame synchronization is desirable.
  • the reset sequence for reset of pixels in row 0 is illustrated for a high dynamic range image taken by an imager of the exemplary design using 420 scan rows by 626 scan columns.
  • the term scan is used here to refer to the rows and columns sequenced through in the scan sequence. Not every value sequenced through needs to be used to access an active row or column and in general it is only necessary to have enough rows and columns in the scan sequence to access rows and columns which are actually read in an appropriate sequence and perhaps enough additional scan rows and/or columns to provide for the multiple integration periods of the high dynamic range sensor with a non-overlapping or non-conflicting schedule of tasks for the read and the reset tasks.
  • the actual imaging array may have fewer rows and/or columns (or in some cases more rows and/or columns) than are cycled through in the scan sequence.
  • Figs. 1 through 3 use of the scan sequence to generate the time base for the reset and read sequence is illustrated.
  • Fig. 1 only the initiation of reset or conditional reset tasks which apply to pixels in row 0 are illustrated.
  • the exemplary design includes a high dynamic range scan sequence for which the column and row count progresses through successive column counts for each row with 626 columns (column 0 through 625) provided and through rows of a frame having 420 rows (row 0 through 419).
  • the sequence is to index from column 0 through 625 for each row, to index to the next row when cycling from column 625 to column 0 and to cycle from row 419 to row 0.
  • the sequence is repeated until the desired number of frames has been read.
  • the column count is indexed at a rate of 8 million counts per second so the frame of 262,920 pixels is sequenced through in 32.865 milliseconds providing a frame rate of just over 30 frames per second.
  • the row read sequence for row 0 (101) is initiated at column 1 of row 0 (121) and the effective readout takes place at column 14.
  • This readout is preferably suppressed during the scan through the initial frame so that the reset sequence may be initiated during this frame and is preferably performed for all of the succeeding frames included in the readout sequence. For example, when only one frame is read, two scan frames are provided, the first to begin initiation of the integration periods and the second to commence readout and continue handling of resets and conditional resets to provide the proper reset sequence for each pixel of each row which is read.
  • the row to be reset is calculated by adding the row offset of 419 for the 32.768 millisecond integration period to row 1 in which the reset is performed yielding 420 modulo 420 which is 0 so row 0 is reset in the sequence (102).
  • the reset is released and the integration period begins at column 164 (112) so the column offset from the start of integration, column 164, to the row read, column 14, is:
  • the time from the release of the reset to the read is the row offset multiplied by the number of pixels per scan row plus the signed column offset between pixels at the start and end of the integration period this quantity divided by the scan rate.
  • One option to provide a standard dynamic range mode is to inhibit some or all of the succeeding conditional resets after the initial one which was just described and to provide flexibility to program the row and column offsets and the integration time with the associated row and column offsets to give wide ranging options in setting of the integration period.
  • the image control may provide flexibility to adjust the frame sizes for both the scan frame and the sub-frame over which image data is read.
  • the 8.192 ms pixel by pixel conditional reset sequence (103) for row 0 (121) is initiated at column 176 of row 315 (123).
  • the row to be reset is calculated by adding the row offset of 105 for the 8.192 millisecond integration period to row 315 in which the reset is performed yielding 420 modulo 420 which is 0 so row 0 is reset in the sequence 103.
  • the time from the release of the reset to the read is the row offset multiplied by the number of pixels per scan row plus the signed column offset between pixels at the start and end of the integration period this quantity divided by the scan rate.
  • the 2.048 ms pixel by pixel conditional reset sequence (104) for row 0 (121) is initiated at column 500 of row 393 (124).
  • the row to be reset is calculated by adding the row offset of 27 for the 2.048 millisecond integration period to row 393 in which the reset is performed yielding 420 modulo 420 which is 0 so row 0 is reset in the sequence 104.
  • the 512 ⁇ s pixel by pixel conditional reset sequence (105) for row 0 (121) is initiated at column 268 of row 413 (125).
  • the row to be reset is calculated by adding the row offset of 7 for the 512 microsecond integration period to row 413 in which the reset is performed yielding 420 modulo 420 which is 0 so row 0 is reset in the sequence 105.
  • the 128 ⁇ s pixel by pixel conditional reset sequence (106) for row 0 (121) is initiated at column 210 of row 418 (126).
  • the row to be reset is calculated by adding the row offset of 2 for the 128 microsecond integration period to row 418 in which the reset is performed yielding 420 modulo 420 which is 0 so row 0 is reset in the sequence 106.
  • the 32 ⁇ s pixel by pixel conditional reset sequence (107) for row 0 (121) is initiated at column 352 of row 419 (127).
  • the row to be reset is calculated by adding the row offset of 1 for the 32 microsecond integration period to row 419 in which the reset is performed yielding 420 modulo 420 which is 0 so row 0 is reset in the sequence 107.
  • the reset is released and the integration period begins at column 384 (116) so the column offset from the start of integration, column 384, to the row read, column 14, is:
  • the 8 ⁇ s pixel by pixel conditional reset sequence (108) for row 0 (121) is initiated at column 544 of row 419 (127).
  • the row to be reset is calculated by adding the row offset of 1 for the 8 microsecond integration period to row 419 in which the reset is performed yielding 420 modulo 420 which is 0 so row 0 is reset in the sequence 108.
  • the reset is released and the integration period begins at column 576 (118) so the column offset from the start of integration, column 576, to the row read, column 14, is:
  • the 2 ⁇ s pixel by pixel conditional reset sequence (109) for row 0 (121) is initiated at column 592 of row 419 (127).
  • the row to be reset is calculated by adding the row offset of 1 for the 2 microsecond integration period to row 419 in which the reset is performed yielding 420 modulo 420 which is 0 so row 0 is reset in the sequence 109.
  • the row read sequence for row 50 (201) is initiated at column 1 of row 50 (221) and the effective readout takes place at column 14.
  • This readout is preferably suppressed during the scan through the initial frame so that the reset sequence may be initiated during this frame and is preferably performed for all of the succeeding frames included in the readout sequence. For example, when only one frame is read, two scan frames are provided, the first to begin initiation of the integration periods and the second to commence readout and continue handling of resets and conditional resets to provide the proper reset sequence for each pixel of each row which is read.
  • the 32.768 ms unconditional reset sequence (202) for row 50 (221) is initiated at column 132 of row 51 (222).
  • the row to be reset is calculated by adding the row offset of 419 for the 32.768 millisecond integration period to row 51 in which the reset is performed yielding 470 modulo 420 which is 50 so row 50 is reset in the sequence (202).
  • the time from the release of the reset to the read is the row offset multiplied by the number of pixels per scan row plus the signed column offset between pixels at the start and end of the integration period this quantity divided by the scan rate.
  • the 8.192 ms pixel by pixel conditional reset sequence (203) for row 50 (221) is initiated at column 176 of row 365 (223).
  • the row to be reset is calculated by adding the row offset of 105 for the 8.192 millisecond integration period to row 365 in which the reset is performed yielding 470 modulo 420 which is 50 so row 50 is reset in the sequence 203.
  • the time from the release of the reset to the read is the row offset multiplied by the number of pixels per scan row plus the signed column offset between pixels at the start and end of the integration period this quantity divided by the scan rate.
  • the 2.048 ms pixel by pixel conditional reset sequence (204) for row 50 (221) is initiated at column 500 of row 23 (224).
  • the row to be reset is calculated by adding the row offset of 27 for the 2.048 millisecond integration period to row 50 in which the reset is performed yielding 50 modulo 420 which is 50 so row 50 is reset in the sequence 204.
  • the 512 ⁇ s pixel by pixel conditional reset sequence (205) for row 50 (221) is initiated at column 268 of row 43 (225).
  • the row to be reset is calculated by adding the row offset of 7 for the 512 microsecond integration period to row 43 in which the reset is performed yielding 50 modulo 420 which is 50 so row 50 is reset in the sequence 205.
  • the reset is released and the integration period begins at column 300 (215) so the column offset from the start of integration, column 300, to the row read, column 14, is:
  • the 128 ⁇ s pixel by pixel conditional reset sequence (206) for row 50 (221) is initiated at column 210 of row 48 (226).
  • the row to be reset is calculated by adding the row offset of 2 for the 128 microsecond integration period to row 48 in which the reset is performed yielding 50 modulo 420 which is 50 so row 50 is reset in the sequence 206.
  • the 32 ⁇ s pixel by pixel conditional reset sequence (207) for row 50 (221) is initiated at column 352 of row 49 (227).
  • the row to be reset is calculated by adding the row offset of 1 for the 32 microsecond integration period to row 49 in which the reset is performed yielding 50 modulo 420 which is 50 so row 50 is reset in the sequence 207.
  • the 8 ⁇ s pixel by pixel conditional reset sequence (208) for row 50 (221) is initiated at column 544 of row 49 (227).
  • the row to be reset is calculated by adding the row offset of 1 for the 8 microsecond integration period to row 49 in which the reset is performed yielding 50 modulo 420 which is 50 so row 50 is reset in the sequence 208.
  • the reset is released and the integration period begins at column 576 (218) so the column offset from the start of integration, column 576, to the row read, column 14, is:
  • Figs. 1 and 2 in combination illustrate the pattern for the reset and row read schedule which is chosen.
  • Row 50 comes 50 rows after row 0.
  • the reset and row read tasks in Fig. 2 all occur 50 rows after the corresponding row read or reset tasks for row 0.
  • This same correspondence holds for any pair of rows in the imaging array. That is, for any pair of rows in the imaging array, there is a constant row offset (using modulo arithmetic) between intervals over which corresponding tasks are performed for the individual rows of the pair.
  • this property is satisfied by the stronger restriction for a particular high dynamic range reading sequence that the processing intervals for each row read and integration time related reset task are related to column time intervals which are preferably pre-selected for the reading sequence and which are arranged to occur over non-overlapping ranges of column timing values.
  • the column intervals are preferably applied uniformly to the sequencing of the row read and reset tasks for each row of the imager which is read in the particular image reading sequence.
  • the column range over which the row read is done for each row is preferably pre-selected for a particular read task. Then, the column count range for which a reset task associated with a particular integration period is performed is preferably selected to provide the desired column offset which is associated with the particular integration period and the row offset is preferably selected to provide the desired row offset which is associated with the particular integration period.
  • the column, and row offsets in combination with the timing cadence of the scan sequence then preferably provide the desired integration period.
  • the column count serves to provide row by row timing and also serves as a time base for processing which is synchronized with row sequencing tasks.
  • this timing does not have to be directly related to the number or columns of pixels in a row. Relating the column count to actual columns of pixels is only a convenience since the time base may then be conveniently used to sequence pixel A/D conversion, and pixel format processing and readout tasks which are usually directly related to pixel by pixel sequencing through the image frame.
  • the scan frame extends from column 0 at 303 through column
  • the 32.768 millisecond row reset for each row is applied to the row whose address is computed by adding 419 (the row offset for the 32.768 millisecond integration period) to the row address modulo 420 (the number of rows in the imager scan frame).
  • the reset takes place over the column interval extending from column 132 through column 165 represented by bar 306.
  • the reset is released so that the integration period begins at the beginning of column period 164 (314). This provides the desired column offset of -150 columns to the row read task at column 14 for the 32.768 millisecond integration period.
  • the 8.192 millisecond pixel level conditional row reset for each row is applied to the row whose address is computed by adding 105 (the row offset for the 8.192 millisecond integration period) to the row address modulo 420 (the number of rows in the imager scan frame).
  • the reset takes place over the column interval extending from column 176 through column 209 represented by bar 307.
  • the reset is released so that the integration period begins at the beginning of column period 208 (315). This provides the desired column offset of -194 columns to the row read task at column 14 for the 8.192 millisecond integration period.
  • the 2.048 millisecond pixel level conditional row reset for each row is applied to the row whose address is computed by adding 27 (the row offset for the 2.048 millisecond integration period) to the row address modulo 420 (the number of rows in the imager scan frame).
  • the reset takes place over the column interval extending from column 500 through column 533 represented by bar 311.
  • the reset is released so that the integration period begins at the beginning of column period 532 (319). This provides the desired column offset of -518 columns to the row read task at column 14 for the 2.048 millisecond integration period.
  • the 512 microsecond pixel level conditional row reset for each row is applied to the row whose address is computed by adding 7 (the row offset for the 512 microsecond integration period) to the row address modulo 420 (the number of rows in the imager scan frame).
  • the reset takes place over the column interval extending from column 268 through column 301 represented by bar 309.
  • the reset is released so that the integration period begins at the beginning of column period 300 (317). This provides the desired column offset of -286 columns to the row read task at column 14 for the 512 microsecond integration period.
  • the 128 microsecond pixel level conditional row reset for each row is applied to the row whose address is computed by adding 2 (the row offset for the 128 microsecond integration period) to the row address modulo 420 (the number of rows in the imager scan frame).
  • the reset takes place over the column interval extending from column 210 through column 243 represented by bar 308.
  • the reset is released so that the integration period begins at the beginning of column period 242 (316). This provides the desired column offset of -228 columns to the row read task at column 14 for the 128 microsecond integration period.
  • the 32 microsecond pixel level conditional row reset for each row is applied to the row whose address is computed by adding 1 (the row offset for the 32 microsecond integration period) to the row address modulo 420 (the number of rows in the imager scan frame).
  • the reset takes place over the column interval extending from column 352 through column 385 represented by bar 310.
  • the reset is released so that the integration period begins at the beginning of column period 384 (318). This provides the desired column offset of -370 columns to the row read task at column 14 for the 32 microsecond integration period.
  • the 8 microsecond pixel level conditional row reset for each row is applied to the row whose address is computed by adding 1 (the row offset for the 8 microsecond integration period) to the row address modulo 420 (the number of rows in the imager scan frame).
  • the reset takes place over the column interval extending from column 544 through column 577 represented by bar 312.
  • the reset is released so that the integration period begins at the beginning of column period 576 (320). This provides the desired column offset of -562 columns to the row read task at column 14 for the 8 microsecond integration period.
  • exposure control processing has been distributed so that at least partially row parallel tasks related to exposure control and image acquisition share row parallel processing components, row select and column readout select logic, row column cross-point pixel selectable reset logic, and integration period setting memory access without conflict and the tasks are active approximately 50% of the time during image acquisition.
  • the same row offset is applied for all of the resets for the same integration period which are made in a given range of column times so that each reset task still has its own row with its own non- conflicting time slot in the dedicated column range within the row in which it may be performed.
  • Fig. 4 depicts a block diagram of an exemplary design of a high dynamic range imaging device.
  • the imaging device is preferably based on a conventional rolling shutter CMOS design which is enhanced to include the dynamic range improvements capability of the present invention.
  • the diagram is simplified, particularly with respect to features which carry over from the conventional rolling shutter design.
  • Diagram blocks 410 through 415 all operate in one way or another on signals related to columns of pixels in the imager. Thus, it is assumed that the column related signals may be routed through some of the blocks without necessarily being used in each block and that each block has access to the appropriate column related signals.
  • Integration and row read timing is based on a scan column counter 403 which in the exemplary design counts incrementally from zero through its highest value (625 in the exemplary design), goes to zero on the next count, again counts incrementally from zero to highest value and so on. Each time that the scan column count cycles back to zero the scan row counter is incremented to its next value.
  • the scan row count 402 begins at count zero, counts up incrementally through its highest value (419 in the exemplary design), goes back to zero on the next count, counts up incrementally through its highest value, and so on.
  • the pixels of a row are preferably all reset at the beginning of the longest of the integration periods and then each pixel and its associated reset record are interrogated just prior to the beginning of each of the succeeding shorter integration periods and each pixel of the row is conditionally reset based on the comparison of the current pixel value against a threshold value and the reset history for the pixel. Whenever a pixel is not reset, it is preferably not reset again prior to readout but is allowed to integrate to complete the integration period initiated by its most recent prior reset.
  • the memory associated with the pixel 3 bits per pixel are all that are required to identify the one of 8 possible selectable reset states) is updated to indicate the integration period for which it is being reset.
  • the memory value for a pixel is not changed during a conditional reset task for which the pixel is not reset.
  • the value associated with the pixel indicates which of the integration periods was used for the reading. This information is preferably read out in a synchronized manner with the readout and A/D conversion of the pixels in the row and is preferably used to properly scale the final readout value.
  • the integration periods in a row are preferably initiated so that they all end simultaneously, that is so that the row read which ends the integration period may be done simultaneously for all pixels of the row regardless of the actual integration period for each pixel.
  • the row is read, preferably by a sequence which transfers pixel values in the selected row to a row of sampling capacitors and preferably also transfers the reset voltages to a second row of sampling capacitors and digitizes the values and adjusts them to reflect the record of the integration period used. This value is typically output from the imager.
  • the scan row counter 402, the scan column counter 403, the general column offset adder 404, the general row offset adder 405, the row select 409, the row read sample and hold 411, the amplifier 416, and the analog to digital conversion 417 may be similar to those used in conventional prior art imaging arrays. With all of these features retained from prior art imagers, it is reasonably straight forward to provide the flexibility in an implementation of the imager of this invention to provide an option to operate in a standard dynamic range mode in addition to being able to operate in one or more high dynamic range modes.
  • a control circuit 400 receives instructions communicated to the imager and controls the instruction sequence described below.
  • the scan column counter 403 cycles through the column range for each row and the scan row counter 402 is incremented for each new row as indicated by completion of a cycle through the column counts by scan column counter 403.
  • the pixel reset and row read function select circuit 407 decodes column count ranges and generates signals to initiate and control the row read and the various reset functions.
  • the pixel reset threshold generator 408 generates a threshold value which is optionally dependent on the particular integration period initiated by the active reset task. This threshold value is compared against signals which indicate the integrated charge level for each pixel of the selected row to make partial determination on whether or not to reset each pixel of the row. This determination is preferably made on a pixel by pixel basis.
  • Pixel reset row offset generator 406 receives a signal from the pixel reset row read function select circuit 407 which indicates which of the row read or reset tasks, if any, is active and, for the row reset tasks, generates a row offset appropriate to the duration of the integration period to be initiated by the reset.
  • the reset row offset modulo adder 401 adds the offset value appropriate to the duration of the integration period associated with the reset being processed to the row address, preferably using modulo arithmetic to keep the generated row address in a range to appropriately cycle from the bottom back to the top of the scan frame.
  • General row and column offsets are optionally added to the row and column values in blocks 404 and 405 to provide added flexibility to control the origin of the image frame in the imaging array 410.
  • the row select circuit 409 decodes the row select address and gates control signals used for row processing to the selected row.
  • the imaging array 410 includes an individual pixel reset capability which is preferably implemented by providing a feature within each pixel whereby reset is accomplished by the coincident assertion of a row and column signal for each pixel which is to be reset. In use, to selectively reset pixels of a row, the row reset for the selected row is asserted and column reset lines are asserted only for pixels of the row selected for reset.
  • the row read sample and hold circuit 411 preferably includes sampling capacitors to sample analog values for each column of pixels in the imager. This circuit is similar to ones normally employed and preferably includes sequential sampling of both the readout values and of the reset reference levels on each of the column readout lines.
  • the circuit also has provision to respond to signals from the column select circuit 412 and to gate analog values for the selected column to output amplifier 416.
  • the amplified and preferably offset corrected value from amplifier 416 is converted to digital form by analog to digital converter 417.
  • a pixel processing circuit 418 conditions the digital value from analog to digital converter 417 combining it with the memory indication of the integration period used to take the reading obtained from column addressable pixel row buffer memory 413 so that the output value is scaled to properly indicate the light level received by the pixel taking into account the integration period used to take the reading as indicated by the value stored in memory.
  • the pixel read column select circuit 412 preferably selects corresponding values from the row read sample and hold circuit 411 and the column addressable pixel row buffer memory 413.
  • the column addressable pixel row buffer memory preferably accesses pixel reset memory 415 during the row read task and buffers the reset information for the row being read in memory register 413.
  • the pixel reset memory 415 preferably contains three bits per pixel.
  • the memory is preferably organized so that the same signal used for the row select, optionally before application of the general row offset, is used to address the memory.
  • the three bits per pixel are available in parallel for each pixel, or pixel related column, of the selected row.
  • the pixel threshold compare and conditional reset block 414 has access to the pixel reset threshold from the pixel reset threshold generator output from block 408, the column lines which are driven with signals which indicate the level of integrated charge on each pixel of the selected row of the array from imaging array 410, read and write access to the addressed column related word of the pixel reset memory 415, the column reset lines to imaging array 410, and controlling signals from the pixel reset and row read function select block 407.
  • this block preferably contains logic to perform parallel tasks for significant portions of the row being processed and which in response to control signals operates on the indication of integrated light induced charge in combination with the pixel reset memory entry for each pixel of the selected row (or a significant portion of the pixels in the row) to make a determination as to whether or not to reset each or a significant number of the active pixels of the row.
  • the circuit selectively outputs the column reset signals to selectively reset pixels of the row and updates the reset memory for pixels of the row as required. This update preferably includes writing an indication of the duration of the new integration period which is established for pixels which are selectively reset.
  • Fig. 5 depicts a simplified diagram of the circuit associated with one column of pixels in an imaging array.
  • one pixel of the column which, for example, contains 420 pixels in the exemplary design is depicted as pixel 516 within block 501.
  • pixel processing is normally suppressed during the noise critical row read task and suppression of pixel processing during this period also allows the parallel portion of the row read task to be completed before pixel by pixel processing of pixels in the row is begun eliminating timing conflicts between the two tasks.
  • the first 36 or so numerical columns of the imager are preferably not implemented providing about 590 readable columns.
  • the imager pixel array is preferably restricted to rows which are actually read with the optional addition of some guard rows and possible rows and columns of dark reference pixels which are shielded from light around the periphery of the active imaging area.
  • Column processing circuits need to be implemented only for columns which are read.
  • the array would have 590 columns each having 420 pixels with one pixel for each of the 420 rows of the exemplary imager.
  • the remaining portion of the circuit would be replicated 590 times once for each of the 590 active pixels in the 626 pixel scan row.
  • the exemplary imager is simplified to best illustrate the reset process which provides the exceptionally high dynamic range.
  • Implementations of the imager which are also considered to be within the scope of the patent may include nearly any feature provided with prior art imagers.
  • features may be included in the readout structure to suppress readout of larger portions of the frame covered by the scanning process than just those which potentially conflict with the parallel portion of the row read task.
  • the imager instruction set may include instructions which do not use the high dynamic range capability and which may accordingly provide added capabilities such as much higher frame repetition rates.
  • image based control applications it is frequently desirable to supplement reading of a high dynamic range image with images which may be taken in faster sequence, using smaller scan frames which cover smaller portions of the overall field of view. Exposure for these images may, for example, be at least partially based on light levels observed in the high dynamic range image.
  • the scan frame size, the choice of row and column offsets for each of the resets, and even the number of conditional resets per reading may optionally be programmable.
  • the ratio of integration period from one reset to the next does not need to be 4 as in the exemplary design and does not need to be a constant for each successive integration period. For example, an imager might have some reset periods where the ratio of the period from one to the next is 4 and where the ratio of another to the next is 8 or even some non integral number or some value which is not an integral power of 2.
  • the count ranges may extend between low and high values where the low value may be nonzero and may even be negative.
  • the count direction may be reversed to go from high to low.
  • the concept of modulo arithmetic must be extended so that the count values are sequenced to stay within the bounds of the intended frame and cover the intended ranges. Readout does not even need to be in a strict monotonic sequence, but application of nonstandard scan sequences to features such as the row offset applied to establish the integration times must be carefully planned to provide proper operation. Formulas to calculate scan time must also be extended to properly indicate integrations times for the modified frame scan structures.
  • the second integration period might be chosen as 16.384 milliseconds for a 2 to 1 ratio instead of at 8.192 milliseconds for the 4 to 1 ratio.
  • succeeding integration periods might be lengthened by 2 to 1 or one period might be changed to 8 to 1 or extra bits might be used to encode the tag for the integration period and more than 8 integration periods could be used.
  • the balance between row offsets and row lengths and possible adjustments in integration periods to provide for non-conflicting column period intervals for each of the integration periods needs to be evaluated and appropriately chosen for each set of integration periods used in the high dynamic range mode of operation.
  • the row reset line 515 (one per each row), the row readout enable line 520 (one per each row), the column address bus 521, the pixel readout bus 523, the pixel reset reference value 524, the row buffer memory write enable 526, the integration time output bus 506, the unconditional pixel reset enable 529, the reset indication flip-flop clock 530, the integration period tag bus 513, the tag memory update enable 531 , and the row address bus 534 are preferably routed to and shared by corresponding circuits for each of the active columns.
  • the column reset line 503 and the column readout line 519 is preferably shared by each of the active pixels in the column.
  • Representative pixel 501 is selected by asserting a select signal on row select line 520.
  • Reset circuit 502 causes the pixel to reset when signals are simultaneously asserted on row reset line 515 and column reset line 503.
  • Block 504 shown in greatly simplified form, contains circuit 522 which includes column sampling capacitors and a column select circuit to respond to signals on column select address and control bus 521 and to sample signals on column readout line 519 in response to control signals on bus 521 and to gate the sampled pixel readout signals to readout bus 523 in response to assertion of the matching column select address along with appropriate control signals on bus 521.
  • a signal is asserted on select line 535 when the column is selected by the column address asserted on bus 521.
  • select signal 535 is used to gate the memory record which indicates the integration period used to read the selected pixel to bus 506 in synchronization with readout of the analog pixel value which is derived from signals and analog values gated to bus 523.
  • Compare circuit 525 compares the signal level on column readout line 519 which is indicative of the light induced charge on the selected pixel against a threshold reference level on line 524.
  • a logic true signal is asserted on output 536 when the signal 519 indicates that the light induced charge on the pixel exceeds the reference level indicated by signal 524.
  • Bus 513 is used to communicate the numerical integration period identifying memory tag for the integration periods.
  • This numerical memory tag may be encoded in a number of ways. In the exemplary design, the longest through the shortest integration periods are assigned identifying numerical memory tags of 0 through 7, respectively.
  • a pixel will be reset unconditionally for the longest integration period in a reading sequence and for successive shorter integration periods in the sequence, the pixel will be reset if and only if the compare output 536 is asserted for the pixel indicating that the light induced integrated charge on the pixel exceeds the threshold value on 524 and the pixel has also been reset for every previous longer integration period in the reading sequence.
  • it is sufficient to check the integration period identifying memory tag entry for the pixel to see if the pixel was reset for the immediately preceding integration period. If not, the current reset is inhibited for the pixel regardless of the state of the compare output 536.
  • the memory block 533 preferably has an entry for each pixel of the column, the current pixel being selected for read and write tasks by the row address of the selected row which is communicated on bus 534 and serves as the address input to memory block 533.
  • the compare circuit 528 performs a bit by bit matching comparison of the integration period identifying numerical tag saved for the pixel against the tag value asserted on bus 513.
  • the "and” gate 510 asserts a true output when and only when all three of the bitwise match comparisons match indicating that the memory tag value for the pixel matches the tag value broadcast on bus 513 and the compare output 536 is asserted indicating that the light induced integrated charge on the pixel of the selected row in the column exceeds the threshold value.
  • "D" type flip-flop 512 registers a reset indication for the pixel when it is clocked by a clock signal on line 530. Line 529 is asserted at the appropriate time to force a reset of the pixel for the initial longest integration period setting, "or" gate 511 combines this unconditional reset command with the conditional reset indication just described from “and” gate 510.
  • the output of flip-flop 512 drives the column reset line 503 for the selected column which, when asserted, causes reset of pixel 516 when row reset line 515 is also asserted during an overlapping time period.
  • Reset line 515 is asserted at the proper time in the row processing sequence to selectively reset pixels for which associated column reset lines 503 are also set.
  • additional gating logic may be added to assert vertical reset line 503 only when the reset task is to be asserted.
  • the functions of the row select line 520 and the row reset line 515 may optionally be merged into a single line so that simultaneous assertion of select line 520 and column reset line 503 cause reset of the pixel.
  • Memory update enable line 531 is asserted to update the integration period identifying tag for the pixel being processed in memory 533.
  • the update is performed to the memory location selected by the row address asserted on address bus 534 when the memory update line 531 is asserted, and the pixel is reset as indicated by assertion of column reset line 503.
  • the output 538 of "and" gate 514 which ands the update line 531 and the pixel column reset line 503 serves as the write enable to memory block 533.
  • write enable 538 When write enable 538 is asserted, the numerical integration period identifying memory tag value for the integration period which is communicated on bus 513 is written into the entry for the pixel in memory block 533.
  • bus 513 is used to communicate two separate integration period identifying tags, each at a separate time interval within the sequence.
  • the tag for the immediately preceding integration period is communicated on bus 513 for use by compare circuit 528.
  • the tag value for the current integration period is communicated on bus 513 to update the memory indication as just described for pixels which are reset to initiate a new and shorter integration period.
  • the first of which is preferably a row parallel task which is preferably performed during a column count range which does not overlap the column count ranges used for any of the reset tasks.
  • the second phase is preferably a pixel by pixel or a small pixel group by small pixel group processing stage and is preferably configured so that it does not require access to row select, row readout, column readout, or column reset lines needed for the ongoing reset tasks.
  • the pixel processing in the second phase may overlap ongoing reset tasks and may optionally be performed using any one of a number of organizational approaches. For example, processing may be a pixel by pixel operation or, perhaps, a sequential operation using pipelining or parallel processing on small subgroups of the pixels in the row.
  • the readout select line 520 is asserted for the row of pixels being processed gating the pixel value to column readout line 519.
  • a readout sequence which may be very similar to that used in prior art imagers is performed whereby the pixel value asserted on column line 519 is sampled by capacitors or other circuitry in pixel readout processing block 522. The sequence may include unconditional reset of the pixel and additional sampling of the reset value to be used as a zero reference for the pixel in the final readout task.
  • the row address of the row being read is asserted on address bus 534 so that the integration period identifying tag for the pixel being read is presented on output lines 532 of memory block 533.
  • write enable line 526 is asserted to store the integration period identifying tag for the pixel being read in tag buffer memory 507.
  • the task is preferably performed in parallel for all active pixels of the row so the integration period of each pixel of the row is available in the associated buffer memory 507 of the replicated column logic blocks.
  • the column address is communicated to pixel readout 522 on column address bus 521. This sequentially incremented column address is decoded in block 522 and serves to sequentially select column sampling capacitors for analog to digital conversion of the sampled pixel value and processing of the pixel and also to gate the integration period used for the pixel on bus 506.
  • the final readout value for the pixel is adjusted to indicate the proper light level reading given the integration period used to acquire the reading.
  • the record of the integration period which was used for the pixel reading is buffered in memory block 507.
  • the column address decode logic in block 522 asserts output line 535 when the column is selected to enable bus drivers 527 to gate the integration period identifying tag stored in memory 507 onto bus 506.
  • the pixel readout information which was sampled and held in block 522 is gated to readout bus 523, preferably using a select output from the same column select decoding circuitry used to gate the integration period identifying tag for the pixel.
  • the pixel values which are adjusted to reflect the integration period are presented for readout sequentially just after completion of the analog to digital conversion and the adjustment to reflect the integration period used.
  • a range determination period may be added before integration for readout begins and a sequence of preferably row parallel compare tasks (perhaps including multiple threshold compares to handle determination of more than one reset period option in a compare task) similar to those described for the combined conditional reset range may be used to determine the range to use.
  • the integration period determined may be saved in the integration period memory associated with each pixel. This memory value may then be used during the readout phase to set the proper integration period for each pixel and also at readout to indicate the integration period used to read the pixel to properly scale the readout value.
  • an imaging device is configured to, during the normal image acquisition period for a single readout frame; dynamically select an integration period individually for each desired pixel, the integration period being automatically chosen from a set of selectable integration periods which includes a limited number of selectable periods each having an identifying index; set the integration period for the given pixel to the selected integration period and record in memory associated with the given pixel an indication of the identifying index of the integration period selected for the pixel; perform integration; read and digitize the pixel reading at a predetermined time; and output an indication of the digitized reading along with an indication of the selected integration period.
  • the imaging device includes an imaging array of pixels arranged in groups of pixels which are addressed in parallel and includes circuits to perform various parallel tasks on pixels in the, then, currently addressed parallel addressable group. All of the pixel read and reset tasks, including those which perform conditional, selective reset tasks, are intended to operate on all of the active pixels of the parallel addressable group of pixels that is being accessed.
  • the groups of pixels which are addressed in parallel are arranged in rows and readout is performed a row at a time with a pixel clock supplied to control the pixel conversion rate and to supply finer timing increments to clock and sequence logic functions.
  • row is replaced with the more broad term, "parallel addressable group”.
  • row time is replaced by "task sequencing interval” which is used to indicate the interval used to schedule and perform readout of pixels in a parallel addressable group and to also schedule and perform more diverse reset and selective reset tasks.
  • pixel clock is replaced by the more functionally descriptive term “sequencing clock”. It should be understood that principles of this invention are not limited to devices where readout is performed a row at a time or where clocking is directly tied to pixel rate but extends to the more general context implied by "parallel addressable group", “associated task sequencing intervals” and “sequencing clock”.
  • Each pixel of a parallel addressable group is set to an integration period based on its response to the light to which it is exposed and each pixel in the parallel addressable group is read in a partially parallel read task during the associated task sequencing interval.
  • Pixels of the array include a selective reset capability whereby pixels of each parallel addressable group may be selected for reset on a pixel by pixel basis.
  • the pixels preferably also include provision to read an indication, preferably analog, of the integrated charge, preferably nondestructively, after partial integration.
  • the imaging device is preferably configured so that, for each active pixel, successively shorter integration periods may be selected from a preferably predetermined set of available integration periods through use of the selective reset capability.
  • Each member of the set of available integration periods is preferably identified by an associated integration period identifying index.
  • Each active pixel of the array preferably has an associated memory element which is used to store an indication of the integration period identifying index of the integration period which is in progress for the associated pixel.
  • each pixel is unconditionally reset to begin its longest available integration period and its associated memory element is initialized to indicate the integration period identifying index of the longest available integration period.
  • the pixel is subjected to conditional selective reset tasks to conditionally initiate a shorter one of the available integration periods.
  • each of the decisions to selectively reset each pixel is partially based on the comparison of an indication of the integrated charge read (preferably in analog form) after partial integration (i.e. after partial completion of the integration period which is currently active for the pixel) to a pre-selected threshold level (The pre-selected threshold level may be a constant or may be dependent on the integration period for which it is applied and perhaps on its position within the array or some other variable).
  • Each of the selective reset tasks is used to conditionally initiate a successively shorter integration period from the set of selectable integration periods. Whenever a pixel is reset to begin a shorter integration period, the record in memory which is associated with the pixel of the identifying index of the currently active integration period for the associated pixel is updated to indicate the identifying index of the integration period which is being initiated. The final reset of the pixel establishes the integration period used to read the pixel. In the preferred sequence of successive reset tasks for a given pixel for a given readout period, once a conditional reset for a pixel is bypassed due to a decision partially based on the result of the compare task, no further resets are performed on the given pixel prior to readout of the pixel.
  • conditional reset task if the pixel was reset in the immediately preceding reset task for the pixel, it is preferably reset when and only when the result of the compare task for the pixel indicates that the level of the integrated, light induced charge on the pixel exceeds the threshold value against which it is compared as part of the conditional reset task for the associated selectable integration period.
  • the indication of the identifying index of the integration period selected for the pixel is read from the memory element associated with the pixel and the light induced integrated charge stored on the pixel during the selected integration period is read from the pixel and preferably digitized as a digitized pixel charge level reading.
  • the reading of the preferably digitized pixel charge level is paired with the indication from the memory associated with the pixel of the integration period used for the pixel.
  • the digitized pixel charge level reading and the integration period selected for the pixel indicate the light level to which the pixel was exposed.
  • This information is normally assembled in a convenient, correlated form and communicated from the imaging device to an external device or optionally to another memory or image handling function within the imaging device itself.
  • the selectable integration periods are chosen such that each successively shorter integration period is shorter than the next longer integration period by a prescribed ratio.
  • each succeeding selectable integration period is 4 times shorter than the next longer selectable integration period so that there is a nominal 4 to 1 ratio between any two successive selectable integration periods. This provides a nominal 16,384 to 1 ratio between the longest and shortest of the eight selectable integration periods. Ratios other than 4 may be chosen and not all successive selectable integration periods have to differ by the same ratio from their immediate predecessor or successor and more or fewer selectable integration periods may be provided.
  • the range of approximately 16,384 to 1 from the longest to the shortest integration period adds 84dB to the dynamic range of typically more than 5OdB provided by digitizing the light induced integrated charge on the pixel.
  • This provides a dynamic range of more than 134dB with a nearly linear output independently for each pixel for each image acquired.
  • the minimum resolution may be better than a part in 100 or, ideally as great as a part in 250 over a large part of the total light level range.
  • a 3 bit memory item is used to store the identifying index of the integration period which is currently active for its associated pixel.
  • the 3 bit item requires less than one fourth of the memory which would be required to store the full 13 bit value saving substantial memory space in comparison with high dynamic range designs which require storage of a complete or nearly complete image frame.
  • the memory block used to identify the integration period for each pixel is preferably fabricated on the same silicon substrate as the imaging array and may optionally be constructed of dynamic memory elements.
  • the value for each pixel needs to be stored only through the duration of the full integration and readout period for the associated pixel.
  • the limited storage time needed during the actual exposure sequence limit the time period that values need to be held in storage and the repeated memory access and write tasks during the conditional reset tasks provide additional, natural refresh tasks limiting or even eliminating additional refresh tasks which may be needed when dynamic memory elements are employed.
  • the 24ms period between the initial unconditional reset and the first conditional reset is the longest interval between successive memory read/write tasks which provide refresh for a particular location in the memory. For some dynamic memory designs, more frequent refresh cycles may not be needed.
  • added memory access cycles may be inserted during idle periods between execution of the parallel addressable group read task and/or parallel addressable group reset tasks within each task sequencing interval to provide additional refresh cycles as needed.
  • These refresh tasks may be structured and scheduled as a refresh task or tasks during each task sequencing interval much like an added conditional reset task by assigning the number of task sequencing interval time periods appropriate to the timing requirement and by assigning a sequencing clock count value (Task sequencing intervals and sequencing clock counts are defined below.) to initiate the refresh task during which the memory location is accessed and rewritten (as necessary) to perform the added refresh cycle.
  • the change in timing caused by various choices of the sequencing clock count value is very small relative to the time period provided by multiple task sequencing intervals needed for the refresh intervals so available intervals of sequencing clock count time which do not interfere with the parallel addressable group read, or one of the reset intervals already scheduled during the task sequencing interval may be selected for providing the added memory refresh cycles.
  • ratiometrically proportioned integration periods from one to the next shorter, allows the ratio between any two adjacent integration periods to be relatively small (4 in at least one embodiment) and to still provide for a very large range of integration time with a reasonably small number of selectable integration periods (A range 16,384 to 1 in integration time is provided with 8 selectable integration periods in the above referenced exemplary design.)- There is a tendency to look at the eight available integration periods and the nominal 32ms of the longest integration period and conclude that 32ms/8 or 4ms is nominally available to handle each integration range.
  • a rolling shutter design is used as a starting point and a novel combination of features is added to provide the necessary device functions to perform necessary tasks during the various available time periods, including the very short ones, to obtain the exceptionally wide pixel by pixel dynamic range.
  • Pixels are partitioned into parallel addressable groups for readout and reset tasks and readout of the parallel addressable groups of pixels is distributed, preferably task sequencing interval and associated parallel addressable group by task sequencing interval and associated parallel addressable group, at the uniform time increments provided by the task sequencing intervals within the time period provided to read all of the parallel addressable groups of pixels of the image frame. Tasks to select and establish the appropriate integration period for each pixel are also scheduled and organized within the task sequencing intervals.
  • readout of pixels in a parallel addressable group is done at least partially in parallel, a parallel addressable group at a time during successive task sequencing intervals, by transferring pixel values of the selected parallel addressable group into a group of sampling circuits.
  • Pixel related values retained in the group of sampling circuits are then preferably conditioned and digitized, optionally in sequential order, to render a digitized reading indicative of the light induced integrated charge for each pixel of the sampled parallel addressable group.
  • the digitized reading indicative of the light induced integrated charge is then paired with an indication, read from a memory item associated with the pixel, of the integration period used for the pixel. Together the pixel reading and the integration period used for the reading indicate the light level sensed by the pixel and this indication of the light level read by the pixel is output from the imaging device.
  • nine tasks are scheduled for pixels of each addressable group to complete integration and read the pixels of the addressable group.
  • the tasks include, unconditional reset, and multiple conditional reset, one conditional reset for each selectable integration period after the first, to conditionally initiate the selectable integration periods at proper times by scheduling and performing each of the required tasks in the appropriate task sequencing interval at the appropriate time, based on the sequence count, within the task sequencing interval. Finally, at the end of the integration periods for pixels of the given parallel addressable group, readout of the pixels in the addressable group is scheduled.
  • each parallel addressable set of pixels must be scheduled for eight separate tasks (one task for each selectable integration period) to initiate or conditionally initiate each of the selectable integration periods for each active pixel of the parallel addressable group.
  • initiation of the various selectable integration periods for pixels in the given parallel addressable group do not in general occur in the same task sequencing interval.
  • the three shortest integration periods for a given parallel addressable group are initiated in the same task sequencing interval at different sequencing clock counts and the readout and initiation or conditional initiation of the remaining five selectable integration periods for pixels in the given parallel addressable group are all initiated in different task sequencing intervals so that, for the cited exemplary design, various tasks for a given parallel addressable group are sequenced from a total of seven different task sequencing intervals.
  • the readout task and a separate task to initiate or conditionally initiate each of the selectable integration periods is performed for some addressable group of pixels (for each addressed addressable group of pixels which is active) during each task sequencing interval.
  • the read task plus eight reset or conditional reset tasks are scheduled during each task sequencing interval with seven different addressable groups normally being selected by the nine tasks performed in a given task sequencing interval, the seven different parallel addressable groups including one parallel addressable group which is scheduled for conditional initiation of each of the three shortest integration periods during the associated task sequencing interval.
  • the hardware is configured so that the tasks are harmless or so that they are inhibited.
  • tasks scheduled in a given task sequencing period include tasks which each operate on different parallel addressable groups.
  • the task sequencing time interval is broken into task sequencing time interval sub- increments which are called sequencing clock periods and a count of sequencing clock periods is normally initialized at the start of each task sequencing interval and incrementally advanced to provide a time base within the task sequencing time interval.
  • the sequencing clock period is conveniently chosen to correlate with pixel processing rate.
  • a row of pixels corresponds to a parallel addressable group
  • a row processing time interval corresponds to a task sequencing time interval
  • a column or pixel clock count time interval corresponds to a sequencing clock count time interval.
  • Readout of pixels in a parallel addressable group is preferably performed at least partially in parallel allocating an associated task sequencing intervals for readout and for scheduling and performance of tasks to select and establish the correct one of the selectable integration periods for each pixel of addressable groups scheduled for an unconditional reset or a conditional reset task during the task sequencing interval.
  • Each of the unconditional and conditional reset tasks to initiate a given one of the selectable integration periods for the pixels of a given addressable group of pixels are scheduled and performed in a task sequencing interval and over a sequencing clock count range within the task sequencing interval which provide the proper number of intervening task sequencing intervals and, fractional task sequencing interval, sequencing clock offset counts to provide the correct duration for the particular selectable integration period.
  • the number of integral task sequencing intervals in a particular integration period is used as the course timing increment and the additional sequencing clock count offset value is used to provide the finer timing increment used to determine the duration of a particular selectable integration period.
  • the preferably parallel portion of the total row read task takes a relatively small part of the total task sequencing interval time.
  • various ones of the selectable integration periods for various ones of the parallel addressable groups of pixels are subjected to scheduled reset or conditional reset tasks to initiate or conditionally initiate selectable integration periods at times scheduled to properly establish the integration times for the pixels of parallel addressable groups which are subjected to the tasks and which will be read during succeeding task sequencing tasks.
  • the sequencing clock count is used to sequence the task sequencing interval identifying index to reference the next task sequencing interval and associated parallel addressable groups and begin a new task sequencing time interval.
  • Image frames with task sequencing intervals with their associated parallel addressable groups and with their sequencing clock time units are normally sequenced in a repetitive frame by frame, task sequencing interval and associated parallel addressable groups by task sequencing interval and associated parallel addressable groups, and sequencing clock count by sequencing clock count cadence.
  • Each task sequencing interval is assigned an identifying index to identify its position within the image frame capture sequence.
  • Sequencing clock count values (or corresponding pixel positions) within the task sequencing interval are assigned numerical index values so that coordinates within the image capture sequence for that frame may be expressed in terms of a coordinate value pair consisting of a task sequencing interval identifying index and sequencing clock count value or number.
  • These task sequencing interval identifying index and sequencing clock count are stored in some form in hardware registers in the circuitry in the imaging device to provide a time base to sequence events in the image acquisition sequence including integration period time interval control, selection of parallel addressable groups, selection of pixels within a group, selection of hardware components and initiation of events in the image acquisition sequence.
  • the orderly progression through imaging frames, through successive task sequencing intervals of the frame and through a sequencing clock count range for a task sequencing interval is somewhat analogous to a clock where an hour could be compared to an image frame scan time with 60 minutes where each minute could be compared with a task sequencing interval and with 60 seconds where each second could be compared with sequencing clock count times.
  • Time intervals such as integration times are conveniently established by counting task sequencing intervals for coarse timing increments and additional sequence clock count offsets that occur to sequence between starting and ending points in task sequencing cadence for finer timing increments.
  • a task may begin at a sequencing interval with its identifying index within the scan sequence in combination with a starting sequencing clock count in the starting task sequencing interval and end at a task sequencing interval with its identifying index within the scan sequence in combination with an ending sequencing clock count in the ending task sequencing interval.
  • time intervals may be expressed using the number of task sequencing intervals to express the coarse portion of the timing increment and a sequencing clock count offset to express the fine portion of the timing increment.
  • the fine portion of the timing increment is preferably expressed as a signed difference of the sequencing clock count at which the interval ends in the ending task sequencing interval minus the clock count at which the interval begins in the beginning task sequencing interval.
  • the readout typically occurs, ending integration, at a low sequencing clock count early in the task sequencing interval period and integration periods are started, typically later in their respective task sequencing intervals, at higher sequencing clock counts and the sequencing clock count offset for the integration period, is calculated as the sequencing clock count (within the task sequencing interval) at which the integration period ends minus the sequencing clock count (within the task sequencing interval) at which the integration period begins.
  • This result is signed and may be negative.
  • the identifying index for a given task sequencing interval will be assigned so that it is equal to the address of the parallel addressable group which is read out during the given task sequencing interval.
  • an offset to origin the readout frame may be added to calculate physical pixel locations.
  • Further successive sequencing task period identifying index values are assigned as consecutive integers which range from the value corresponding to the beginning of the frame scan sequence and increase incrementally to the value corresponding to the value corresponding to the end of a frame scan sequence.
  • a task sequencing interval identifying index may be used to directly select a parallel addressable group for readout and the "look ahead" function to select a parallel addressable group to reset or selectively reset pixels within the parallel addressable group to initiate an integration period is effected by adding the number of task sequencing intervals for the integration period, using appropriate modulo arithmetic, to the parallel addressable group address used to select the parallel addressable group to read. Then, after a reset or conditional reset task, as the scan cadence continues, the parallel addressable group for which the reset task was applied will be read after progressing through the number of task sequencing intervals determined by the value expressing the integration period in terms of the number of task sequencing intervals which were added.
  • the reset will be performed with the correct number of task sequencing intervals and the correct sequencing clock count offset interval to initiate the integration period at the instant which will yield the desired integration period.
  • Integration intervals for groups of pixels which are read in parallel are preferably reset or conditionally reset in parallel using the same grouping for these tasks as for readout.
  • appropriate parallel compare, logic, and memory access functions are provided to implement the parallel conditional, pixel selective reset tasks.
  • a number of components including parallel addressable group select, task sequencing interval offset adder, column readout lines, pixel associated memory access, and the parallel set of conditional reset compare and logic circuits are shared by various integration period specific unconditional or selective reset tasks and by the parallel addressable group read function and possibly by added memory refresh functions.
  • the parallel addressable group read, each of the reset functions to initiate a specific selectable integration period, and added memory refresh functions are performed in pre-assigned substantially non-overlapping time intervals (or during time intervals which are assigned so that shared resources are available as required) characterized to prevent contention for shared resources during each task sequencing interval.
  • this objective is satisfied by providing hardware based task scheduling functions which sequence the parallel addressable group read, the unconditional reset, and each of the conditional reset tasks, one for each of the conditionally selectable integration periods so that they are performed during substantially non-overlapping ranges of sequencing clock counts during a task sequencing interval.
  • memory refresh tasks are also added to the list of tasks performed over substantially non-overlapping ranges of sequencing clock counts.
  • each specific task is preferably initiated in response to reaching a predetermined sequencing clock count during each task sequencing interval, the predetermined sequencing clock counts being chosen such that there are no overlapping tasks which would conflict. It is preferable to perform the sequence just described for each task sequencing interval in the progression of parallel addressable group addresses through successive frames. Whenever, after application of appropriate offsets, a parallel addressable group with pixels in the active area is selected, the sequenced task is performed but is preferably suppressed or done in a way that there are no harmful effects when an addressed parallel addressable group is not in the active imaging area.
  • the parallel addressable group read and each of the reset and conditional reset tasks By performing the parallel addressable group read and each of the reset and conditional reset tasks whenever a parallel addressable group with pixels in the active image area is addressed, all of the tasks are performed for every parallel addressable group with active pixels. )106] There are two sets of possibly conflicting criteria for selection of sequencing clock count ranges for the parallel addressable group read and the reset tasks to initiate the various integration periods.
  • the first criterion is to provide sequencing clock count ranges for task assignments within a task sequencing interval for execution of the parallel addressable group read and reset tasks which are non-overlapping or at least partially non-overlapping in a way that shared resources are available as needed.
  • a potentially conflicting second criterion is to provide sequencing clock count ranges for task assignments within a task sequencing interval for execution of the parallel addressable group read and reset tasks which provide respective sequencing clock count offsets to establish the desired integration period for each of the selectable integration periods.
  • conflicts may be found by choosing sequencing clock count ranges within a task sequencing interval to properly set integration time periods and inspecting for overlapping sequencing clock count ranges for the task assignments and/or conflicts in use of resources. Conflicts which occur for the shorter integration periods are often resolved by small adjustments in the design such as scaling all integration periods by a small factor or small changes in the number of sequencing clock periods per task sequencing interval.
  • sequencing clock offset may have a very small effect on the integration period relative to the time in the larger number of task sequencing intervals in these integration periods so that sequencing clock ranges for the tasks may often be adjusted to avoid contention in the use of resources with minimal effect on the resulting integration period.
  • Integration periods may be chosen (as in an exemplary design) so that successively shorter integration periods provide a ratio of the effective sensitivity of the pixel for each integration period to the effective sensitivity of the pixel for the next successively shorter integration period which is equal to the exponential base of a number system used to represent the reading.
  • each successively shorter integration period is chosen so that it is 4 times shorter than the immediately longer integration period yielding corresponding decreases in sensitivity which is nominally 4 to 1 for each successively shorter integration period.
  • the identifying indices assigned to the integration periods (call them i) are chosen in an exemplary design so that 0 is assigned to the longest integration period and so that the assigned index increases incrementally to 7 for the seven successively shorter integration periods.
  • a represent the A/D reading of the light induced integrated charge on the pixel.
  • b represent the exponential base for the numerical representation of the reading which preferably represents the light level in approximately linear form.
  • a is the A/D reading for the pixel
  • i is the index of the integration period used to obtain the reading
  • 4 is the base
  • 2-i may be obtained by assigning 0 to bit 0 of the binary number and assigning bits 0 through 2 of i to bits 1 through 3, respectively, of the binary number.
  • the number v may be readily mapped to a floating point format using various procedures including the exemplary procedure just outlined.
  • a above is preferably mapped directly or optionally scaled and mapped to the mantissa and i or 2-i, depending on the base, is mapped with an optional offset and/or scale factor, as appropriate, to the exponent.
  • the exponential representation of the pixel value above may be exploited to facilitate conversion to logarithmic form.
  • values are preferably first normalized to place the binary or radix point in a preferred location (analogous to adjusting the exponent and mantissa to provide a mantissa with a value greater than or equal to 1 and less than 10 for nonzero values for scientific notation).
  • the mantissa is preferably then converted to its logarithmic equivalent using a hardware and/or software based circuit.
  • the exponent after appropriate scaling, may be added to the logarithm of the mantissa to provide the logarithmic value for the pixel.
  • the scaling factor may be an integral power of 2 allowing direct insertion of the exponent into the logarithm of the mantissa to provide the logarithmic value for the pixel reading.
  • the numerical representation of the integration period may be directly mapped to the exponent of a binary floating point representation of the pixel value and the digitized pixel reading may be directly mapped to the mantissa of a binary floating point representation of the pixel value. This is true but there may be some leading zeros in the mantissas which are directly mapped in this way so that further processing may be required to adjust or "normalize" the value to a form which is directly compatible with floating point representations such as those specified in IEEE Standard 754 and its proposed revisions.
  • revisions now under consideration for IEEE Standard 754 include a 16 bit floating point format which is of particular interest for imaging devices in accordance with the present invention.
  • An electronic circuit which implements features depicted in the block diagram of Fig. 11 is described herein. The function of the circuit is to convert the floating point format as it is received from circuits in the high dynamic range imaging device to a floating point format which is readable (preferably directly) by another device. Specifically, for example, imager pixel values may be converted to the 16 bit format included in the proposed revision of IEEE Standard 754.
  • circuits to perform the functions indicated in Fig. 11 are preferably included as part of the same integrated circuit which includes the array of pixels and captures the image.
  • exponent value 1101 is preferably input directly as the index value which indicates the integration period selected to acquire the pixel reading.
  • mantissa value 1102 is preferably input directly from the analog to digital readout of the pixel value acquired with the selected integration period.
  • the pixel value may optionally be scaled or translated and/or an offset value may be applied preferably to establish the appropriate exponent value range for the converted value. It is convenient but not required to scale the output so that the output value for the highest light level reading is very close to one.
  • a full scale mantissa value and an exponent of 14 yields a value which is slightly less than one.
  • Imager designs given in examples have 8 integration periods with index values which range from 0 for the longest integration period through 7 for the shortest integration period and for which each successive integration period is four times shorter (or is sized to produce one fourth of the sensitivity) than the next longer integration period. For these imager designs, by multiplying or scaling the integration period identifying index by a factor of two, the scaled values are 0, 2, 4, 6, 8, 10, 12, and 14 which are the appropriate exponent values for the converted floating point value prior to the normalization operations which will now be explained.
  • the values for all ranges with a non-zero exponent must be normalized and only the fractional portion of the normalized mantissa without the leading one is stored to conform to the IEEE 754 specification.
  • the leading one is assumed or formally restored when the value is used.
  • Mantissa values with an exponent of zero are a special case and are not normalized and the leading one is not suppressed so the mantissa is effectively shifted right by one binary bit position when it is stored.
  • the mantissa may be scaled and an offset may be applied. It is preferable to provide an output which is an approximately linear representation of the light level which is sensed by the pixel. The value is also preferably established so that it is approximately zero for zero light level.
  • leading zeros are counted for use in the determination of the adjustment required to normalize the value.
  • the number of leading zeros from block 1105 is compared with the exponent from block 1103 and also balanced against available ranges in the exponent value to determine an adjustment in alignment of the mantissa and a corresponding adjustment in the exponent to normalize the value in conformance with the IEEE 754 standard.
  • the primary function of block 1 108 is to perform the necessary left shift operation to align the value.
  • the leading one may be shifted to an overflow position and not stored for nonzero exponents.
  • the primary function of block 1107 is to adjust the exponent, preferably by subtracting a value to reduce the exponent value by the number of places by which the mantissa is shifted to the left.
  • the mantissa is stored in de-normalized form and shifted right by one position to provide space to always include the most significant one in the mantissa value for this special case.
  • block 1110 is shown separately from block 1108 primarily to emphasize the special case.
  • blocks 1106, 1107, and 1108 may optionally be reconfigured to include and combine the function of block 1 110. This brings up the point that Fig. 11 represents one example and is simplified. Major inter-relations between blocks are depicted, it is intended that others may be included.
  • the option to flag readings is preferably made selectable, particularly if results are communicated with the image data in a way that causes abnormal pixel values in the image.
  • Such information is useful as an indicator on which to base the choice of compare thresholds to use in selection of the proper selectable integration range so that the full scale range of each selectable integration period may be adequately utilized but properly balanced against creating too many saturated readings.
  • a second reason for interest in readings which are greater or less than expected for pixels for which a particular integration period has been selected is that rapid changes in light level during a pixel reading are likely causes of lower or higher than expected pixel readings. Such rapid changes may, for example, be due to a high contrast object that is moving in the scene or a light source which has a rapidly time varying intensity.
  • the pixels which have values that are higher and/or lower than expected for the assigned integration range it is desirable to flag the pixels which have values that are higher and/or lower than expected for the assigned integration range to aid in locating features such as those just described in the image. Since, after normalization, a ten bit analog to digital value maps into nine bits in the fractional part of the mantissa, the least significant bit of a ten bit mantissa value is not utilized for converted data in this case. As one option, this bit may for example be set to one to flag a reading that is lower than expected for the selected integration range and may be set to zero otherwise. Such encoding may, as an option, be used to add information about selected pixel values without significantly changing the actual value of the pixel and without adding separate or additional signaling paths to the pixel data communication structure.
  • a circuit block 1113 may be provided to generate a histogram similar to that described, for example, in commonly assigned U. S. Patent Application Publication 2003/0103141 Al which is incorporated herein in its entirety by reference.
  • the histogram preferably includes counts of pixels whose values are unusually low for the selected integration period as well as pixels whose values are unusually high for the selected integration period.
  • Instructions to set up and read histogram data are preferably communicated separately from the primary image data stream.
  • a communication path which is at least partially separate from the path used to read the image data and to use this path to read and write instruction data. It is preferable to also include setup instructions and returned data about the image such as contained in the histograms just described in this instruction and control data communication path.
  • An imaging array has pixels organized in multiple parallel addressable groups of pixels (rows of pixels in an exemplary design). Pixels of each parallel addressable group may be addressed to select them as a group to perform hardware based tasks, preferably at least partially in parallel, on them as part of the image acquisition sequence.
  • the individual separately scheduled tasks to be performed for each pixel of the imaging array for each image frame which is acquired include a task to read, digitize and output or optionally to perform further tasks on the pixel, and also at the time that each of the selectable integration periods is initiated for the pixel to perform the task to initiate integration or conditionally initiate a new integration period. Eight selectable integration periods are included in an exemplary design.
  • a reset which is part of the immediately preceding read task for a given pixel may be used to initiate the longest of the selectable integration periods for the pixel and, as a second option, the shortest of the integration periods might be combined as part of a lengthened read task for the pixel. Either option is considered part of this invention but not preferred because each limits flexibility.
  • for each pixel there are seven to nine tasks for readout and pixel reset plus possible additional tasks for items such as memory refresh to separately schedule and perform to establish the proper selectable integration period and read the pixel.
  • parallel hardware based conditional reset tasks are performed; the integration period is established, preferably individually, for each pixel for each image acquired; the selection of a shorter selectable integration period for each pixel is partially based on comparison of the indicated pixel value (preferably in analog form) against a threshold value after partial integration; new and shorter integration periods are initiated by performing a selective reset on the pixel at the correct instant to complete integration on the original predetermined schedule; the selectable integration period to use to read each pixel is established during time allotted for the longest selectable integration period for the pixel during the image acquisition period and before completion of the integration period and final readout and digitization of the pixel value; pixel values are digitized only once for each image acquired; an indication of the integration period currently in progress for each pixel is recorded with initiation of each new conditionally selectable integration period which is initiated for the pixel.
  • the imaging device preferably has the flexibility to adjust many parameters such as the "comparison thresholds" for selective reset tasks, active "image frame size" including the number of task sequencing intervals in the frame, the number of sequencing clock periods in a task sequencing interval and the number duration of the sequencing clock period.
  • the use of and the number of selectable integration periods is also preferably programmable.
  • Pixels are organized in parallel addressable groups, with pixels in a group being accessible in an at least partially parallel manner for performance of tasks by issuing the address of the parallel addressable group.
  • the parallel addressable groups with active pixels preferably contain equal numbers of active pixels in each parallel addressable group.
  • the image acquisition interval will be considered to be the interval between capture of one image and capture of a succeeding image when operating in a continuous capture mode.
  • the image acquisition interval is divided into task sequencing intervals which are preferably of equal duration and which preferably fill the image capture interval. There are preferably at least as many sequencing intervals as parallel addressable groups with active pixels. Task sequencing intervals are divided into a preferably fixed number of numbered sequencing clock periods. There are preferably at least as many sequencing clock periods in a task sequencing interval as there are active pixels in an addressable group.
  • the readout task preferably contains a portion which is performed substantially in parallel for active pixels of the parallel addressable group. During this substantially parallel portion of the readout, the integration period ends at a predetermined sequencing clock count and analog information from each pixel is transferred to an associated sampling circuit.
  • the substantially parallel portion of the readout task is preferably sequenced over a predetermined range of sequencing clock periods.
  • the parallel portion of the readout takes a relative small portion of the task sequencing interval and the remainder of the interval is available for other tasks including reset and selective reset tasks. For scheduling the reset tasks, there are a number of possible vantage points from which to view the scheduling arrangement. In Figs.
  • FIG. 1 and 2 relating to an exemplary design, rows and columns or pixel values in these Figs, and in the related description may be compared: a row time and related row, to the more generalized task sequencing interval with its related parallel addressable group and a pixel clock or count to the more generalized sequencing clock count.
  • Figs. 1 and 2 each depict a row read task for only one row of the array and indicate the row and column offsets of the unconditional row reset and each of the seven conditional resets performed on pixels in the row indicated for readout to determine and select the appropriate selectable integration period of each pixel of the row to prepare the row for readout. In this view, we start with the row which was read and look back to the reset and conditional reset tasks performed to prepare the row for readout. Active pixels of each parallel addressable group are subject to conditional reset tasks during more than one task sequencing interval.
  • a task is scheduled to read out active pixels of an associated parallel addressable group and reset tasks, one associated with each of the selectable integration periods, are scheduled.
  • reset tasks one associated with each of the selectable integration periods
  • an unconditional reset task is scheduled to unconditionally reset pixels of the parallel addressable group for which the longest integration period is initiated.
  • Each of the remaining reset tasks is associated with a particular one of the shorter selectable integration periods and is configured to selectively reset pixels, on a pixel by pixel basis, to selectively initiate the associated shorter integration period for pixels which are selectively reset.
  • a reset task associated with each of the selectable integration periods is provided.
  • the parallel tasks associated with readout of active pixels of a parallel addressable group are scheduled over a range of sequencing clock times within the task sequencing interval and portions of the row read which do not require use of resources shared with the reset tasks, such as digitizing pixel values sampled in the parallel portion of the row read task, are scheduled through the remainder of the task sequencing interval.
  • the reset tasks, one for each selectable integration period are scheduled, preferably following the parallel tasks for the read task, through the remaining portions of the task sequencing interval, the scheduling being arranged to prevent contention for shared resources and to initiate integration periods, unconditionally or selectively for individual pixels as required, at a sequencing clock count within the task sequencing interval which provides initiation of the selectable integration period of the desired duration.
  • the base address used to select the parallel addressable group of pixels on which to perform the task is modified, based on the number of task sequencing intervals in the selectable integration period associated with the reset, to select the parallel addressable group, (a "row" in an exemplary design) which will be read after this number of task sequencing intervals thereby providing the desired integration period.
  • Scheduling and arrangement of the selectable integration periods is such that readout occurs on a predetermined schedule which does not depend on individual integration periods selected for various pixels.
  • digitized pixel values are acquired exactly once per active pixel per image and presented in a known, predetermined order for direct output or further processing without necessity for buffering major portions of the image to re-order pixels or to provide for selection of values when multiple readings are acquired for a pixel in the image acquisition process or when acquisition of digitized pixel values is staged and effectively interspersed for pixels over a large part of the array.
  • a selection of the integration period for the pixel is made from a limited set of selectable integration periods; an indication of the selected integration period is recorded; scheduling is such that the selected integration period ends and readout and digitization of the pixel value occurs at a predetermined time in the image capture sequence.
  • the 3T pixel structure typically includes a light sensing photodiode, a first transistor which is configured to reset the pixel to a specified potential (normally its highest operating potential), a second transistor which is configured in a source follower configuration to serve as a buffer amplifier to provide a lower impedance output which tracks the voltage on the photodiode, and a third transistor to connect the buffered pixel output to an output sensing line.
  • each pixel is reset and arranged to accumulate light induced charge during an integration period which is selected for all pixels of the array.
  • the third transistor is turned on to gate the buffered pixel voltage to the output line where it is sampled.
  • the readout normally includes a second phase in which the first reset transistor is turned on briefly to reset the pixel and the reset level is also gated to the output line and sampled as a non-correlated double sampling reference level.
  • the pixel voltage is sampled nondestructively at prescribed times during the image acquisition sequence to determine when to selectively reset the pixel to initiate a shorter integration period and an indication of the integration period which is selected is recorded for each pixel in memory to be output as part of the pixel reading.
  • Devices of this invention are not limited to three transistor or four transistor pixel structures (i.e. "3T” and "4T" pixels) as pixel designs within the scope of the present invention may share transistors and additional transistors may be used for variants in pixel design and for functions such as blooming control.
  • the current invention utilizes a capability to begin integration periods for individual pixels on a selective basis and this feature is best implemented by the addition of a transistor to the respective pixel structure.
  • a transistor is preferably added so that coincident row and column (coincident parallel addressable group and pixel within the parallel addressable group) selections are required to initiate reset of a pixel.
  • the 4T pixel the same selective coincidence is preferred, but selective operation of the transfer gate for individually selected pixels is preferred to direct selective operation of the reset transistor or transistors for individually selected pixels.
  • the 4T pixel typically includes a pinned photodiode or other light sensitive device such as a photo-gate structure and with either structure, the light sensitive device is configured to accumulate charge in response to light which strikes it. This accumulated charge will be referred to as light induced charge.
  • a second site often referred to as a floating diffusion is also provided.
  • the pixel structure also normally includes a first transistor to reset the floating diffusion to a specified potential (normally its highest operating potential), a second transistor which is connected in a source follower configuration to serve as a buffer amplifier to provide a lower impedance output which tracks the voltage on the floating diffusion, a third transistor to connect and thereby multiplex the buffered pixel output to an output sensing line, and a fourth transistor normally referred to as the transfer gate which is switched to its conductive state to transfer light induced charge from the pinned photodiode (or other light sensing structure) to the floating diffusion.
  • the reset transistor With the transfer gate in its non-conductive state, the reset transistor may be used to reset only the floating diffusion and with the transfer gate in the conducting state, operation of the reset transistor resets both the floating diffusion and the photodiode (or other light sensing structure).
  • the pinned photodiode structure used in many 4T pixel designs exhibits very low dark current as long as the transfer gate is maintained in the non-conducting state to confine the light induced charge to the junction of the pinned photodiode.
  • the transfer gate provides the option to first reset the floating diffusion and sample the pixel output to obtain a zero reference level and thereafter (preferably very shortly thereafter) to switch the transfer gate in the pixel being read to its conducting state to transfer the light induced charge from the pinned photodiode (or other light sensitive device) to the floating diffusion and then to sample the pixel output.
  • the sampled reset level may be subtracted from the sampled pixel reading to correct for offset errors in the readout chain including errors in the reset level on the floating diffusion.
  • This is known as correlated double sampling.
  • Correlated double sampling is not available in a conveniently usable form for imagers having conventional 3T pixels.
  • the low dark current of the pinned photodiode and the low noise level provided by correlated double sampling are particularly beneficial in providing enhanced performance at low light levels.
  • Nondestructive readout of the pinned photodiode (or other light sensitive device) may be provided by continued or repeated switching of the transfer gate to its conductive state to transfer additional light induced charge to the floating diffusion, but in general this results in higher effective dark current and may also prevent practical use of correlated double sampling to reduce readout noise.
  • non-digitized sample readings of the light level are preferably taken and compared to threshold values to determine the integration period to use for each pixel prior to initiation of the selected integration for the pixel.
  • a selectable integration period is preferably chosen for each pixel just prior to initiation of the selectable integration period for the pixel and once initiated, it is preferable that the selected integration period is carried to completion and that normal correlated double sampling is provided at least in reading of the pixel value which is used to provide the digitized output value.
  • all selectable integration periods for a pixel are initiated so that they end at the same time so that selective initiation of longer integration periods is scheduled before selective initiation of shorter integration periods for a pixel.
  • time is provided to take a sample reading to determine whether or not to select the longest of the available integration periods and the time required to take this sample reading plus the time for the longest integration period are sized to fit in the longest time interval which is available to acquire the image.
  • a single sample reading may optionally be used to determine whether to select more than one of the selectable integration periods; but, for a given design it may be preferable to provide a separate sample reading to be used in conditional selection of each selectable integration period for a given pixel of the array beginning with conditional selection of the longest integration period and continuing for conditional selection of each successively shorter selectable integration period until an integration period is selected for the particular pixel and the particular image. After the integration period is selected, an indication of the integration period selected for the pixel for acquisition of the particular image is recorded and the selected integration period is initiated and allowed to continue to completion.
  • the pixel value is read, preferably using correlated double sampling and preferably digitized and output along with an indication of the integration period used to acquire the reading.
  • the imager is preferably configured to perform the stated sequence of operations on every active pixel of the array for acquisition of an image and the entire sequence is preferably performed for every active pixel for every image which is acquired.
  • a sample reading is taken with the pixel and the decision to initiate a particular integration period is preferably at least partially based on a comparison of the sample reading from the pixel against a threshold value.
  • the shortest selectable integration period will be selected and initiated by default.
  • the indication of the shortest integration period for the pixel is preferably overwritten with the indication of the selected integration period at any point in the sequence at which a longer integration period is selected and initiated for the pixel.
  • the pixel is preferably initialized by beginning the longest integration period and by setting the associated memory to indicate that the longest integration is in progress and then using a reset to interrupt the longer integration period and also updating the associated memory indication of the integration in progress to start a shorter selectable integration period.
  • the reset sequence when a pixel which is not reset for the next longer integration period, the pixel is preferably allowed to integrate without further resets until readout.
  • the pixel level selection is preferably provided for switching of the transfer gate to it conductive state within the pixel.
  • the pixel is then selectively reset as noted above by switching the transfer gate to its conductive state (In this case, on a pixel level selective basis) and also operating the reset node.
  • Selective operation of the pixel transfer gates to selectively switch the transfer gates to their conductive states permits pixels for which integrations are in progress to continue undisturbed by selectively maintaining their transfer gates in the non-conductive state while the integrations are in progress.
  • the 4T pixel may be used in a way which is very similar to the 3T pixel by using continued or repeated switching of the transfer gate to its conductive state to transfer charge from the pixel light sensing structure to the readout node to enable nondestructive readout to sample the pixel value and to enable the use of selective reset to operate the pixel in a manner that may be directly used in the design which is detailed as an example elsewhere herein.
  • the transfer gate is preferably set to its non-conducting state during the integration period so that light induced charge accumulates in the capacitive well which is inherent to the pinned photodiode. Then, at the end of the integration period and preferably just after resetting the floating diffusion and reading the reset level of the reference node for the correlated double sampling, the transfer gate is switched to its conductive state to transfer light induced charge from the photodiode to the floating diffusion and this value is read.
  • the floating diffusion is designed to have a lower potential than the storage region of the pinned photodiode structure so that when the transfer gate in the pixel is switched to its conducting state the first increments of charge will flow from the photodiode to the floating diffusion and the voltage increment created by the charge will change in inverse proportion to the capacitance of the floating diffusion relative to the capacitance of the photodiode.
  • this excess charge will flow to equalize the voltage on the photodiode and the floating diffusion so that increased increments of charge will fill the combined capacitance of the floating diffusion and the photodiode resulting in a knee in the response curve of floating diffusion voltage vs.
  • nondestructive readout compromises operation of the 4T pixel so it is preferable to separately monitor or to pre-sample the light level to which each pixel is exposed during each image acquisition period in order to select the integration period appropriate for the pixel.
  • the embodiment described here takes a sample reading or readings from the pixel itself prior to initiation of the integration period for the pixel reading to provide one or more pre-samples of the pixel reading on which to base the selection of the integration period for the pixel.
  • the challenge is to create a structure which provides time to initiate and read these pre-samples and to schedule these sample readings so they are timely and convenient to act upon.
  • the invention is not limited to implementation of a specific choice of the relative sizes of the integration periods, for the sake of clarity, it will be assumed that there are eight available integration periods starting with the longest and with each successive additional integration period being one fourth of the length or preferably of a length which provides one fourth of the sensitivity of the next longer integration period.
  • an imaging device is configured to utilize characteristics of the 4T pixel or other pixel structure which has similarities to the 4T pixel in its operating characteristics, hi the present example, a sample reading is taken just prior to conditional initiation of the longest selectable integration period and an additional sample reading will be taken prior to each successively shorter selectable integration period until an integration period is selected and initiated for the pixel or none of the integration periods longer than the shortest have been initiated in which case the shortest selectable integration period is initiated by default and no sample reading is needed.
  • integration periods for additional pre-samples after the first are initiated only when an integration period has not been selected for the pixel for capture of the current image and, additionally, a selectable integration period other than the shortest remains as a candidate for selection as the integration period for the pixel for the current image acquisition sequence.
  • Initiation of an integration period for a pixel is performed by resetting the pixel and preferably placing the transfer gate in the pixel in its high impedance state after the reset while an integration period for the pixel, pre-sample or otherwise, is in progress.
  • Pre-sample readings are preferably taken at the beginning of intervals scheduled to make a determination of whether or not to initiate a selectable integration period for the pixel.
  • Each such reading is preferably compared against a threshold and the result of the comparison is used in part to determine whether or not to select the associated integration period for the pixel.
  • the same general organization including the use of at least partially row parallel processing for the comparisons and for selective initiation of selectable integration periods are applicable for this design as they were described elsewhere herein.
  • the comparison thresholds are preferably adjusted to take into account the integration period used for the pre-sample for each of the selectable integration periods, systematic nonlinearities in the reading, and the available readout range for the pixel.
  • the thresholds may be adjusted to take into account expected inaccuracies in readout and time variation in the light sources, these factors being balanced against the desire to avoid saturated pixel readings and the desire to utilize the full scale range of the pixel readout circuits.
  • the integration period for each of the pre-sample readings after the first is nominally three times longer than the integration period which is conditionally selected based on the reading.
  • the readout range for the final pixel value is preferably restricted generally to the linear portion of the output response of the pixel and additional gain may additionally be applied to the pixel value before it is digitized.
  • the comparison threshold on which selection of the integration period for the pixel is based may extend into the nonlinear reduced slope region of the pixel readout as outlined above. Any combination of the above may be used to more fully utilize the preferably linear readout range for the pixel while still allowing initiation of the integration periods to be grouped with tasks to initiate a selectable integration period.
  • the memory used to record an indication of the integration period for a pixel may be used to store an indication of an integration period which is selected and the integration period for the pixel may be initiated in a subsequent task based on the indication stored for the pixel.
  • Fig. 6 is a simplified schematic representation of a 3T pixel.
  • Reset transistor 601 is switched to its conducting state to charge light sensing photodiode 605 to a reset potential level and is then switched to its non-conducting state to enable a light induced change in the charge stored on photodiode 605 to occur during the integration period.
  • the reset potential is normally set to reverse bias the light sensing photodiode.
  • the light induced change in charge level results in a change in the voltage at the cathode of photodiode 605.
  • Transistor 602 serves as a source follower to buffer and track the voltage on the cathode of photodiode 605.
  • Transistor 603 is switched to its conducting state to connect the buffered pixel photodiode voltage to column line 607 for readout.
  • This pixel structure is well adapted for incorporation in the pixel 516 of Fig. 5.
  • logic is preferably provided at the individual pixel level to provide the pixel level selective reset provided in Fig. 5 by block 502 which responds to coincident assertion of prescribed signal levels on lines 503 and 515. )135]
  • the term 3T pixel it is intended to indicate a base pixel structure that does not include a transfer gate that is switched during normal pixel integration and /or read operations to isolate the light sensing area of the pixel from the readout node in the pixel.
  • the classic 3T pixel is typically implemented with three transistors in its simplest configuration and, thus, the name 3T.
  • the terminology as used herein is not intended to exclude addition of more transistors to perform functions such as anti-blooming control or to exclude a possible reduction in the number of transistors per pixel by employing a technique whereby components may be shared between pixels or to exclude an additional transistor or transistors to provide functions such as pixel level selective reset.
  • the pixel structure of Fig. 6a is based on the 3T structure of Fig. 6 and includes additional selective reset logic function transistor 608a to implement the selective reset function provided in block 502 of Fig. 5.
  • row (or parallel addressable group) reset line 515 is preferably connected to drain 610a of selective reset logic function transistor 608a
  • column reset line 503a is preferably connected to gate 609a of selective reset logic function transistor 608a
  • row (or parallel addressable group) select line 520 is preferably connected to the gate of row select transistor 603a and column readout line 607a is preferably provided as column readout line 519 in Fig. 5.
  • the gate input 611a of reset transistor 601a is pulled high to perform the pixel reset only when both the row (or parallel addressable group) reset signal asserted at drain 610a and the column reset signal asserted at gate 609a are high simultaneously.
  • This provides the logic function requiring assertion of both the row (or parallel addressable group) and the column reset signals to turn on transistor 601a to perform the reset function.
  • gate signal 609a is low, 61 Ia may be floating and the capacitance of the node maintains the voltage at least for a short period of time.
  • a preferred symmetrical drain and source structure for selective reset logic function transistor 608a will function with node 611a serving as the drain and 610a serving as the source to pull the gate of transistor 601a low to turn off the reset function. It is preferable to provide signal timing which at relatively frequent intervals pulls the gate of reset transistor 601a low to assure that the reset is de-asserted promptly to begin integration and that leakage does not elevate the potential at the gate of the reset transistor 601a to a level that current through the reset transistor will interfere with a reading which is in progress.
  • nodes 609a and 610a to use for the row (or parallel addressable group) reset input and which to use for the column reset input can help to facilitate this requirement.
  • the invention may be practiced with the row and column reset connections interchanged from those shown in the illustrative design. It is preferable to take into account the gate threshold voltages of transistors used and to establish high and low signal levels for assertion and de-assertion of the row (or parallel addressable group) and column signals accordingly. The levels may be different for the signals with gate connections than for signals with drain connection and also may differ based on function and electrical component properties.
  • selective reset logic function transistor 608a is designed to require a low gate to source voltage to turn it on.
  • Patent 6,175,383 indicates utilization of a normal sequential row by row pixel by pixel readout path through lines 74 and 76 to communicate pixel values to processor 78 which is utilized to make a determination of selective reset operations and features for highly parallelized organization and performance of tasks to choose appropriate integration periods for each pixel are not specified or generally supported by the control structure which is shown.
  • drain 610a low for a brief time before pulling 609a low in order to momentarily reverse the roles of the source and drain of transistor 608a thereby pulling gate 611a of reset transistor 601a low and turning it off to provide a definitive starting point for the integration period which starts when transistor 601a is turned off.
  • reset transistor gate signal 61 Ia is floating when the gate signal 609a of logic transistor 608a is low and it is preferable to provide refresh cycles to maintain the low level on gate signal 61 Ia to assure that leakage currents do not significantly increase the voltage on gate signal 611a and cause the reset transistor to conduct charge at the wrong time.
  • the gate 609a of transistor 608a may be pulled high, preferably momentarily, while drain signal 610a remains low to reverse the roles of the source and drain of transistor 608a to discharge the gate of reset transistor 601a to maintain it in its nonconducting state.
  • Typical row read rates may be 400 or more rows per frame times 30 frames per second yielding representative rates of 12,000 or more positive pulses per second on the gate 609a providing 12,000 refresh pulses per second in holding gate 611a low to prevent inadvertent reset or partial reset of pixel photodiode 605a.
  • the pixel of Fig. 6a is representative and similar refresh pulses and pulse rates are applied to all of the active pixels in the imager. This is a representative rate and may be much faster or slower depending on the specific embodiment and this operation is preferably performed for every pixel in the imager.
  • row reset signals rather than the column reset signals are connected to gate 609a of transistor 608a, a row reset select signal is provided at each row read and each conditional reset of the pixel thereby providing refresh of the reset gate voltage but at a much lower rate.
  • Fig. 6b is similar to Fig. 6a except that transistor 612b is added and receives a gate bias voltage at 613b to serve as a preferably weak current mirror to conduct current from the base 611b of the reset transistor 601b and conduct it to a lower potential such as ground at 614b.
  • the voltage at gate 613b may be variable and, for example, may be lowered to reduce the conduction or even turn off transistor 612b when one or more of the transistors 608b in the array are turned on.
  • transistor 714b is added to the circuit of Fig. 7a and functions in a manner similar to transistor 612b in Fig. 6b to sink leakage current which might otherwise cause transfer gate 704b to conduct when it is supposed to be in its nonconducting state.
  • Fig. 7 is a simplified schematic representation of a 4T pixel which may be adapted for application in the embodiments of this invention described for applications which include use of the 4T pixel structure.
  • Reset transistor 701 is switched to its conducting state to charge floating diffusion 706 to a reset potential level and is then preferably switched to its non-conducting state prior to taking a reference reading.
  • the transfer gate 704 is switched to its conducting state to transfer light induced charge stored on photodiode 705 to the floating diffusion causing a change in the voltage of the floating diffusion which may then be read.
  • the photodiode 705 is preferably reset by switching both transfer gate 704 and reset transistor 701 to their conducting states at the same time and this operation is preferably performed to reset the pixel to begin a new integration period.
  • transistor 702 serves as a source follower to buffer and track the voltage on the floating diffusion 706.
  • Transistor 703 is switched to its conducting state to connect the buffered pixel floating diffusion voltage to column line 707 for readout.
  • leakage currents at the light collecting photodiode site 705 are normally minimized when the transfer gate 704 is in its non-conducting state.
  • a reading of the pixel is normally taken by first resetting the pixel and floating diffusion as described above and providing an integration period to collect light induced charge on the photodiode.
  • readout is preferably accomplished by first resetting the floating diffusion node 706 by switching reset transistor 701 , preferably momentarily, to its conducting state and back to its non-conducting state and then sampling the reset level of the floating diffusion 706 through the readout chain provided by transistors 702 and 703.
  • the transfer gate 704 is switched to its conducting state, preferably momentarily, to transfer light induced charge from photodiode 705 to floating diffusion 706 and this value is sampled through the same readout chain used to sample the reset level.
  • the reset level sample reading is preferably used as a zero reference for the sample reading taken after transfer of light induced charge from the photodiode to the floating diffusion.
  • the gain or sensitivity at the pixel level may be increased, possibly at the expense of dynamic range, by making the capacitance of the floating diffusion 706 smaller relative to the capacitance of pixel photodiode 705.
  • a circuit to provide selective reset is preferred.
  • the transfer gate 704 which is selectable at the pixel level.
  • a logic circuit which may include one or more transistors may be added to the pixel structure of Fig. 7 to control application of an electrical signal to the gate of transfer gate transistor 704 in such a way that operation of the transfer gates may be selectively controlled for individual pixels within an addressable group of pixels.
  • 4T pixel 7 and elsewhere in this document where the term 4T pixel is used, it is intended to indicate a base pixel structure that does include a transfer gate that is switched during normal pixel integration and /or read operations to isolate the light sensing area of the pixel from the readout node in the pixel.
  • the classic 4T pixel is typically implemented with four transistors in its simplest configuration and, thus, the name 4T.
  • the terminology as used herein is not intended to exclude addition of more transistors to perform functions such as anti-blooming control or to exclude a possible reduction in the number of transistors per pixel by employing a technique whereby components may be shared between pixels or to exclude an additional transistor or transistors to provide functions such as pixel level selective reset.
  • FIG. 7a is based on the 4T structure of Fig. 7 and includes additional selective transfer gate operation logic function transistor 708a to implement the selective transfer gate switching function provided in block 1002 of Fig. 10.
  • row (or parallel addressable group) transfer gate control line 1021 is preferably connected to gate 709a of selective transfer gate operation logic function transistor 708a
  • column transfer gate control line 1003a is preferably connected to drain 710a of selective transfer gate operation logic function transistor 708a
  • row (or parallel addressable group) select line 1026 is preferably connected to the gate of row select transistor 703a
  • row reset line 1027 is preferably connected to the gate of reset transistor 701a
  • column readout line 707a is preferably provided as column readout line 1025 in Fig.
  • the gate input 709a of transfer gate transistor 708a is pulled high to switch the transfer gate to its conductive state only when both the row (or parallel addressable group) transfer gate control signal asserted at gate 709a and the column transfer gate control signal asserted at drain 710a are high simultaneously.
  • This provides the logic function requiring assertion of both the row (or parallel addressable group) and the column transfer gate control signals to turn on transistor 708a to switch the transfer gate in the pixel to its conductive state.
  • gate signal 709a is low, 711a may be floating and the capacitance of the node maintains the voltage at least for a short period of time.
  • a preferred symmetrical drain and source structure for selective transfer gate operation logic function transistor 708a will function with node 711a serving as the drain and 710a serving as the source to pull the gate of transfer gate transistor 708a low to turn off the transfer gate. It is preferable to provide signal timing which at relatively frequent intervals pulls the gate of transfer gate transistor 708a low to assure that the transfer gate closure is de-asserted promptly to begin integration and that leakage does not elevate the potential at the gate of the transfer gate transistor 708a to a level that current through the transfer gate transistor will interfere with a reading which is in progress.
  • nodes 709a and 710a to use for the row (or parallel addressable group) transfer gate control input and which to use for the column transfer gate control input can help to facilitate this requirement.
  • the invention may be practiced with the row and column transfer gate control connections interchanged from those shown in the illustrative design. It is preferable to take into account the gate threshold voltages of transistors used and to establish high and low signal levels for assertion and de-assertion of the row (or parallel addressable group) and column signals accordingly. The levels may be different for the respective signals.
  • selective transfer gate control logic function transistor 708a is designed to require a low gate to source voltage to turn it on.
  • drain 710a low for a brief time before pulling 709a low in order to momentarily reverse the roles of the source and drain of transistor 708a thereby pulling gate 711a of transfer gate transistor 704a low and turning it off to provide a definitive starting point for the integration period which starts when transistor 704a is turned off.
  • transfer gate transistor gate signal 71 Ia is floating when the gate signal 709a of logic transistor 708a is low and it is preferable to provide refresh cycles to maintain the low level on gate signal 71 Ia to assure that leakage currents do not significantly increase the voltage on gate signal 71 Ia and cause the transfer gate transistor to conduct charge at the wrong time.
  • the gate 709a of transistor 708a may be pulled high, preferably momentarily, while drain signal 710a remains low to reverse the roles of the source and drain of transistor 708a to discharge the gate of transfer gate transistor 704a to maintain it in its nonconducting state.
  • all of the column transfer gate select lines are pulled high during the unconditional operation of the transfer gates of all of the pixels in the row being read to transfer charge from the pixel photodiodes to the floating diffusion readout nodes.
  • Such a transfer gate operation is performed on each pixel of each row as it is read during the rolling shutter row readout sequence. This operation on pixels of the selected row as it is read pulls all of the column transfer gate operation select lines high.
  • the gate 709a is pulled high during the assertion of the column transfer gate select lines to transfer charge to the readout node in pixels in the row being read while the drain 710a remains low because the row is not selected and as explained above, this combination of voltage levels refreshes the gate voltage of the transfer gate transistors for the pixel in Fig. 7a and likewise for all of the pixels in the remaining rows except those in the row selected for the readout.
  • the transfer gate operation performed as described above pulls the gate voltage of the transfer gate transistor low at the end of the pixel transfer gate closure sequence.
  • Typical row read rates may be 400 or more rows per frame times 30 frames per second yielding representative rates of 12,000 or more positive pulses per second on the gate 709a providing 12,000 refresh pulses per second in holding gate 71 Ia low to prevent inadvertent switching of the transfer gate in the pixel to its conducting or partially conducting state.
  • the pixel of Fig. 7a is representative and similar refresh pulses and pulse rates are applied to all of the active pixels in the imager.
  • Fig. 8 indicates a dual slope pixel response which may result, dependent on pixel design parameters, in the response of a pixel as depicted in Fig. 7 as integrated pixel charge level for the reading increases. In the illustrations based on Fig.
  • Curve 800 depicts pixel output response as integrated charge as readout increases. For low levels of integrated charge, charge transfer from the pixel to the floating diffusion is essentially complete resulting in the high slope of portion 801 of curve 800. For higher levels of integrated charge (beyond point 804), the potential on the floating diffusion is high enough (Here the term higher is intended to indicate potential of greater magnitude with a sign chosen so that increasing light induced integrated charge increases the potential on the photo-sensor.
  • curve 801 would remain linear as indicated by 802A and charge level 816 would result in response level 815A and readout level 817A. Instead, it results in pixel response 815 and output level 817.
  • the illustrative levels have been chosen so that light induced integrated charge level 810 with response 809 and readout value 811 is from an integration period that is nominally one eighth of the integration period which creates light induced integrated charge level 813 with response 812 and readout value 814 and light induced integrated charge level 813 is from an integration period that is nominally one third of the integration period which creates light induced integrated charge level 816 with response 815 and readout value 817.
  • integrated charge level 813 may correspond to the level targeted for the full scale range selection threshold level for an integration period selected for a pixel reading. Then output level 81 1 resulting from integrated charge level 810 may represent a corresponding threshold level for an initial pre-sample for which the integration period of the pre-sample is set to one eighth of the selectable integration period (This is an example and other pre-sample integration periods may be selected.) which is selected based on the pre-sample reading.
  • the output level 817 resulting from integrated charge level 816 may represent the corresponding threshold level of pre-samples for a subsequent integration period which may, for example, be nominally three times as long as the integration period which is conditionally selected based on the pixel reading for the pre-sample.
  • the selection criteria may be to select and initiate the integration period if the pre-sample reading is less than the value indicated at 811 and to proceed to select a shorter integration period if the pre-sample reading is greater than the value indicated at 811.
  • the threshold value 811 for the pre- sample period corresponds to the level 814 for the conditionally selected integration period as illustrated above.
  • the selection criteria may be to select and initiate the integration period if the pre-sample reading is less than the value indicated at 817 and to proceed to select a shorter integration period if the pre-sample reading is greater than the value indicated at 817.
  • the threshold value 817 for the pre-sample period corresponds to the level 814 for the conditionally selected integration period as illustrated above.
  • This example illustrates the preferable choice of the threshold level 817 based on the nonlinear response of the pixel as opposed to use of level 817A which would be chosen for a pixel with an extended linear response. It also illustrates choice of a response threshold which is scaled to account for the relative lengths of the integration time for the pre-sample relative to the integration time for the integration period selected based on the sample reading. In each case, the range selection threshold level in the pre-sample is selected so that it corresponds to the desired full scale range selection threshold level for the integration period whose selection is based on the pre-sample.
  • either the pixel value itself (or a related value which is indicative of the light induced charge accumulated on the pixel) may be read nondestructively.
  • Pixel reset is typically used to begin a shorter integration period and for pixels for which readout is nondestructive, a longer integration period which is in progress may be permitted to continue by simply not performing the reset operation.
  • the indication of what integration period is in progress is initialized to a value which indicates that the associated pixel is not yet committed to a particular integration period at the beginning of the image acquisition period for the pixel or at least prior to conditional initiation of pre-samples for integration periods which are shorter than the longest.
  • This indication is preferably used to determine when to continue operations on the pixel prior to selection and initiation of an integration period for the pixel to select an integration period for the pixel.
  • the integration is preferably initiated and permitted to continue, uninterrupted, to completion; the updated indication of the integration period selected being used to inhibit further operations on the pixel which would interfere with the integration which is in progress for the pixel.
  • the mechanism for selective control at a point which is close to the light sensing element in the pixel.
  • a pinned photodiode is the light sensing device and the transfer gate is the device which provides the primary control mechanism to selectively isolate or connect the photodiode to the readout and reset mechanisms in the pixel.
  • transfer gate functionality which is selectable on a pixel by pixel basis and to use this capability to reset pixels selectively at the appropriate time to initiate the selected integration period and to provide the additional capability to isolate the pixel from operations such as reset of the floating diffusion in the pixel which would otherwise be disruptive to a readout which is in progress.
  • Both the pixel level selection of the photodiode reset by selectively closing the transfer gate and the pixel level selection of the isolation from other operations in the pixel by selectively inhibiting closure of the transfer gate are utilized in the preferred structure in order to initiate integration at the pixel selective level and to selectively shield the pixel from operations which include other pixels in the same parallel addressable group. If the reset of the floating diffusion was provided at the pixel level, operation of the transfer gate while integration is in progress would continue to be disruptive for modes of operation including correlated double sampling and provision of selective operation at the pixel level for both the reset and the transfer gate functions would add complexity to the pixel.
  • Fig. 9 a detailed outline of timing for making a determination of what integration period to select for a pixel is given as an example.
  • a procedure which is preferred but not required to practice the invention is followed whereby a pre-sample reading is taken prior to each selectable integration period except the shortest until an integration period is selected for the pixel.
  • Integration periods are initiated both to acquire pre-sample readings and to acquire a pixel reading using the selected integration period.
  • Each integration period for a pre- sample is preferably initiated in a task sequencing period during which at least one other task is performed for the pixel to reduce the number of tasks which must be separately scheduled.
  • Performance of these tasks normally includes parallel addressable group selection and access to the memory associated with each pixel being operated upon so grouping the reset to begin the integration period for the pre-sample with other tasks for the pixel is not only a convenience but also reduces the need to repeat required parallel addressable group selection and memory access operations. This may not be practical for all pre-sample periods but saves processing overhead for each pre-sample integration period which is initiated in a task sequencing period during which other tasks are performed.
  • Some or all of the following tasks are preferably accomplished within a single task sequencing period: These tasks include determination if a readout is in progress for the pixel, and if not, readout of the pre-sample value, and determination of whether to select the integration period and initiate integration for pixel readout or to initiate integration for another pre-sample; initiation, as needed, of an integration period for the newly initiated readout operation or for the next pre-sample; and update of the record of the integration period selected for the pixel when a new integration period for pixel readout is initiated. Similar tasks are preferably performed, at least partially in parallel for other active pixels in the addressed parallel addressable group. )152] Refer to Fig.
  • the pixel is preferably a member of a group of pixels which are preferably organized in parallel addressable groups with at least partially parallel operations being utilized to perform similar operations, preferably on each active pixel of the currently selected parallel addressable group.
  • each operation is preferably performed on each active pixel in each parallel addressable group which contains active pixels to preferably perform the set of operations on every active pixel in the imager for each image acquired.
  • a rolling shutter timing cadence is preferably employed to acquire the image.
  • 900 depicts the full 32.768 millisecond image acquisition period for the pixel; 907 details the last 7.168 milliseconds of the period 900; 912 details the last 1.792 milliseconds of the period 907; 917 details the last 448 microseconds of the period 912; 922 details the last 112 microseconds of the period 917; 927 details the last 28 microseconds of the period 922; 932 details the last 7 microseconds of the period 927; and 936 details the last 1.75 microseconds of the period 932.
  • the pixel reading acquisition sequence for the individual pixel begins with initialization of the memory item indicative of the integration period used for the pixel preferably to the index associated with the default shortest integration period to indicate that, at this point, an integration period has not been selected for the pixel, and the pixel is reset to initiate a 4.094 millisecond pre-sample integration period which extends to 905 at which point, the pixel value is read, preferably in analog form and the pixel reading is compared with a value which is preselected for conditional selection of the longest 28.762 millisecond integration period based on the pre-sample whose integration period is only one eighth as long as the integration period to be conditionally selected based on the comparison.
  • the 28.672 millisecond integration period is selected for the pixel reading and the memory associated with the pixel is updated to indicate selection of the longest, 28.672 millisecond integration period. Then the pixel is unconditionally reset to begin either the 28.672 millisecond integration period for the pixel reading or the integration period for the next 21.5 millisecond pre-sample otherwise. At 904, nominally 2 microseconds after reading of the pre- sample value, the reset is released to begin the integration period.
  • the memory indication of the integration period selected for the pixel is read and if it is set to a value other than that which indicates that no integration period has been selected, reading of the pixel and operations which affect the integrated charge on the pixel are suppressed, preferably by holding the transfer gate in the pixel in its nonconductive state so that other actions performed on the pixel during the task period do not have a substantial effect on the reading which is in progress.
  • the pre-sample value is read and compared against a threshold pre-selected for conditional selection of an integration period for pixel readout which is nominally one third of the duration of the integration period used for the pre-sample.
  • the associated memory is updated to indicate selection of the 7.168 millisecond integration period and the pixel is reset to begin the 7.168 millisecond integration period for the pixel reading. If the pixel pre-sample reading is taken and the pixel reading is greater than or equal to the threshold value, the pixel is reset to begin the integration period for the next 5.374 millisecond pre-sample.
  • the same reset operation may be used for either case above when the pre-sample value for the pixel has been read and the new integration, in either case, begins at 902, nominally 2 microseconds after reading of the pre-sample value.
  • the memory indication of the integration period selected for the pixel is read and if it is set to a value other than that which indicates that no integration period has been selected, reading of the pixel and operations which affect the integrated charge on the pixel are suppressed, preferably by holding the transfer gate in the pixel in its nonconductive state so that other actions performed on the pixel during the task period do not have a substantial effect on the reading which is in progress.
  • the pre-sample value is read and compared against a threshold pre-selected for conditional selection of an integration period for pixel readout which is nominally one third of the duration of the integration period used for the pre-sample.
  • the associated memory is updated to indicate selection of the 1.792 millisecond integration period and the pixel is reset to begin the 1.792 millisecond integration period for the pixel reading. If the pixel pre-sample reading is taken and the pixel reading is greater than or equal to the threshold value, the pixel is reset to begin the integration period for the next 1.347 millisecond pre-sample. The same reset operation may be used for either case above when the pre-sample value for the pixel has been read and the new integration, in either case, begins at 909, nominally 2 microseconds after reading of the pre-sample value.
  • the memory indication of the integration period selected for the pixel is read and if it is set to a value other than that which indicates that no integration period has been selected, reading of the pixel and operations which affect the integrated charge on the pixel are suppressed, preferably by holding the transfer gate in the pixel in its nonconductive state so that other actions performed on the pixel during the task period do not have a substantial effect on the reading which is in progress.
  • the pre-sample value is read and compared against a threshold pre-selected for conditional selection of an integration period for pixel readout which is nominally one third of the duration of the integration period used for the pre-sample.
  • the associated memory is updated to indicate selection of the 448 microsecond integration period and the pixel is reset to begin the 448 microsecond integration period for the pixel reading. If the pixel pre-sample reading is taken and the pixel reading is greater than or equal to the threshold value, the pixel is reset to begin the integration period for the next 334 microsecond pre-sample. The same reset operation may be used for either case above when the pre-sample value for the pixel has been read and the new integration, in either case, begins at 914, nominally 2 microseconds after reading of the pre-sample value.
  • the memory indication of the integration period selected for the pixel is read and if it is set to a value other than that which indicates that no integration period has been selected, reading of the pixel and operations which affect the integrated charge on the pixel are suppressed, preferably by holding the transfer gate in the pixel in its nonconductive state so that other actions performed on the pixel during the task period do not have a substantial effect on the reading which is in progress.
  • the pre-sample value is read and compared against a threshold pre-selected for conditional selection of an integration period for pixel readout which is nominally one third of the duration of the integration period used for the pre-sample.
  • the associated memory is updated to indicate selection of the 112 microsecond integration period and the pixel is reset to begin the 112 microsecond integration period for the pixel reading. If the pixel pre-sample reading is taken and the pixel reading is greater than or equal to the threshold value, the pixel is reset to begin the integration period for the next 82 microsecond pre-sample.
  • the same reset operation may be used for either case above when the pre-sample value for the pixel has been read and the new integration, in either case, begins at 919, nominally 2 microseconds after reading of the pre-sample value.
  • the memory indication of the integration period selected for the pixel is read and if it is set to a value other than that which indicates that no integration period has been selected, reading of the pixel and operations which affect the integrated charge on the pixel are suppressed, preferably by holding the transfer gate in the pixel in its nonconductive state so that other actions performed on the pixel during the task period do not have a substantial effect on the reading which is in progress.
  • the pre-sample value is read and compared against a threshold pre-selected for conditional selection of an integration period for pixel readout which is a little more than one third of the duration of the integration period used for the pre-sample.
  • the associated memory is updated to indicate selection of the 28 microsecond integration period and the pixel is reset to begin the 28 microsecond integration period for the pixel reading. If the pixel pre-sample reading is taken and the pixel reading is greater than or equal to the threshold value, the pixel is reset to begin the integration period for the next 19 microsecond pre-sample.
  • the same reset operation may be used for either case above when the pre-sample value for the pixel has been read and the new integration, in either case, begins at 924, nominally 2 microseconds after reading of the pre-sample value.
  • the memory indication of the integration period selected for the pixel is read and if it is set to a value other than that which indicates that no integration period has been selected, reading of the pixel and operations which affect the integrated charge on the pixel are suppressed, preferably by holding the transfer gate in the pixel in its nonconductive state so that other actions performed on the pixel during the task period do not have a substantial effect on the reading which is in progress.
  • the pre-sample value is read and compared against a threshold pre-selected for conditional selection of an integration period for pixel readout which is a little less than three eighths of the duration of the integration period used for the pre-sample.
  • the associated memory is updated to indicate selection of the 7 microsecond integration period and the pixel is reset to begin the 7 microsecond integration period for the pixel reading. If the pixel pre-sample reading is taken and the pixel reading is greater than or equal to the threshold value, the pixel will be set to the shortest 1.75 microsecond integration period. Following the initial, preferably shorter integration period used for the pre- sample on which to base conditional selection of the longest of the selectable integration periods, tasks to initiate the pre-sample integration periods and to read the results and to conditionally initiate the associated selectable integration period followed a repetitive pattern up to this point.
  • the task sequencing interval which extends from 930 to 929, if no longer integration period has been selected and the 7 microsecond integration period is not selected, only the 1.75 microsecond integration period remains and when no shorter selectable integration periods remain, it is preferably initiated by default, so a pre-sample integration period does not need to be initiated in the interval from 930 to 929 but performance of the reset should not have harmful consequences and may be beneficial to drain light induced charge from the pixel and to eliminate the need to add logic to inhibit the reset.
  • the memory indication of the integration period selected for the pixel is read and if it is set to a value other than that which indicates that no longer integration period has been selected, reading of the pixel and operations which affect the integrated charge on the pixel are suppressed, preferably by holding the transfer gate in the pixel in its nonconductive state so that other actions performed on the pixel during the task period do not have a substantial effect on the reading which is in progress. If no longer integration period has been selected, the following actions are taken to initiate the shortest 1.75 microsecond integration period.
  • the indication of the integration period which is set, if not already initialized to indicate the 1.75 microsecond integration period is set to indicate this value and the pixel is reset and the reset released to begin the 1.75 microsecond integration period at 934.
  • the pixel value is read along with the indication from memory of the integration period used for the reading.
  • the pixel read operation at 901 is preferably part of a parallel operation performed on pixels of the row or parallel addressable group.
  • the pixel value is preferably digitized, preferably for the first time in the pixel reading image acquisition sequence, and preferably combined with the indication of the integration period used to give an indication of the light level at the pixel.
  • the processing time in the task sequencing interval during which no integration is taking place reduces the duration of the integration periods for the pre-samples having progressively more effect on the duration of the pre-sample integration period relative to the integration period whose selection is based on the pre-sample as the pre-sample integration period becomes shorter. It is preferable to take this and other integration period related effects into account to set the compare threshold value used for each of the pre-sample integration periods.
  • Each selectable integration period is initiated to end at time 901 so that the pixel reading is scheduled to take place at 901 without regard for which integration period is selected and when the integrated pixel value is read at 901, the indication of the integration period used for the pixel is also read in a synchronized manner and communicated as part of the pixel readout value.
  • the same considerations presented earlier with regard to options to use floating point or various encoding options for pixel value readout apply to this embodiment also.
  • Fig. 10 is a simplified diagram of the column logic which is preferably replicated for each column which has active pixels in the imaging device so that many of the operations described may preferably be performed in parallel for active pixels in a row or parallel addressable group.
  • the structure and many of the features are similar to those of Fig. 5 except that the structure of Fig. 10 is intended for use with a pixel such as a 4T pixel and is configured to acquire pre-sample readings from each pixel just prior to conditional selection of an integration period for the pixel, with the conditional selection of the integration period preferably being based at least in part on the value of the pre-sample reading for the pixel.
  • an integration period for a shorter additional pre-sample is preferably initiated and the sequence above iterated for conditional selection of remaining shorter selectable integration periods until an integration period is selected or until only the shortest of the selectable integration periods remains, in which case, it is preferably initiated by default.
  • Initiation of an integration period preferably includes reset of the pixel to begin the integration period and recording of an indication of the integration period initiated in a memory item associated with the pixel.
  • pixel values need to be read only after completion of intended integration periods enabling application of the configuration to pixel structures for which readout of the pixel value before completion of the integration period is problematic.
  • the logic block of Fig. 10 associated with a column of pixels is preferably laid out to fit in a space equal to the width of the associated column of pixels so that similar logic blocks may be provided for each column of pixels and aligned with their respective associated columns of pixels.
  • Pixel 1001 is one representative pixel from the column of pixels which is served by the logic depicted in blocks 1006 and 1007 in Fig. 10.
  • the logic blocks 1006 and 1007 in Fig. 10 is associated with the large number of pixels in the associated column of pixels, preferably including one pixel for each row or parallel addressable group of pixels in the array.
  • Signal lines 1021, 1026, and 1027 are associated with the row or parallel addressable group to which pixel 1001 belongs and signals on these lines are used to selectively manipulate pixel 1001.
  • Other pixels in the column are each selectively manipulated by signal lines dedicated to the particular row or parallel addressable group to which the pixel belongs.
  • the signal lines provided separately for each pixel in the column include row transfer gate control line 1021 which is used in selective control of the transfer gate, row select line 1026 which is used to control connection of the pixel output to the column line 1025 for readout, and row reset line 1027 which is used to control reset of the floating diffusion (and pixel when the transfer gate is conducting).
  • row transfer gate control line 1021 which is used in selective control of the transfer gate
  • row select line 1026 which is used to control connection of the pixel output to the column line 1025 for readout
  • row reset line 1027 which is used to control reset of the floating diffusion (and pixel when the transfer gate is conducting).
  • Different select, reset, and row or parallel addressable group oriented transfer gate control lines are provided for each pixel in the column and each of these signals preferably extends to other pixels in the associated row or parallel addressable group. Further, it is preferred that logic similar to that depicted in Fig. 10 is provided for every column of pixels in the imaging array.
  • the pixel 1001 and others like it in the array may be based on a conventional 4T pixel with the functionality of block 1002 being added to provide switching of the transfer gate to its conductive state which is selective on a pixel by pixel basis.
  • the transfer gate in each pixel is switched to and from its conductive state simultaneously for every pixel in a selected row and the capability to selectively operate the transfer gate for individual pixels within a row is not provided.
  • Block 1023 represents the light sensing element, preferably a pinned photodiode;
  • block 1004 represents a transfer gate;
  • block 1005 represents a floating diffusion and reset transistor; and
  • block 1024 represents a buffer amplifier and analog switch transistor to connect the buffered pixel output to column line 1025.
  • the pixel structure is preferably similar to that described in Fig. 7 with block 1002 added to provide selective operation of the transfer gate.
  • column transfer gate control line 1003 and column readout line 1025 serve all of the pixels in the column.
  • Signals 1021, 1026, and 1027 and similar lines for each additional pixel in the column preferably serve all the pixels in a row or parallel addressable group.
  • Lines or buses 1010, 1017, 1028, 1030, 1031, 1034, 1036, 1037, 1040, 1041, 1042, and 1045 preferably serve all of the column blocks similar to 1006 and 1007 in the array.
  • This memory is used to store an indication of the integration period selected for each pixel in the column and provides just enough memory to identify one of eight selectable integration periods for each pixel.
  • the longest selectable integration period is identified by the lowest numbered identifying index and successively shorter selectable integration periods are identified by successively greater index values with the shortest selectable integration period preferably identified by an index value of all ones.
  • the identifying index is initialized to all ones to identify the shortest integration period at the beginning of the image acquisition sequence for the pixel and when a longer integration period is selected and initiated for a pixel, the integration period identifying index value for the pixel is changed to indicate the integration period just initiated.
  • the index value to be written or conditionally written is communicated on bus 1017 and the row or parallel addressable group address which identifies the selected parallel addressable group (also the selected row in this example) to which the pixel belongs is communicated on bus 1045.
  • Flip-flop 1016 is set either conditionally as will be described or unconditionally by asserting a true signal on line 1040 and clocking line 1041 to register the memory update enable indication in the flip-flop.
  • the memory update enable flip-flop 1016 is set, the value asserted on bus 1017 is written into the addressed 3 bit memory location associated with the selected pixel in the column. A new value is written into the 3 bit memory location associated with the pixel when the update indication from flip-flop 1016 is true and the memory write pulse is asserted on line 1042.
  • Gate 1019 "ands" the two write conditions together to provide the memory write strobe 1020.
  • the addressed 3 bit indication associated with the selected pixel is output from memory 1044 on local bus 1043 and communicated to 3 bit buffer memory 1011 where it is stored temporarily in response to a pulse on line 1034 during the parallel phase of the row readout and output on lines 1012.
  • a signal is asserted on select line 1009 to enable buffers 1035 to output the identifying index of the integration period used for the reading on bus 1010 so that it is synchronized with digitization of the pixel value and included in the readout of the pixel value.
  • Block 1029 includes column logic similar to that provided in prior art devices to sample pixel readout values and zero reference values during a preferably row or parallel addressable group parallel read operation and hold and communicate them at the appropriate time for conversion and readout which may be provided sequentially a pixel at a time and which may overlap operations performed by other column logic depicted in Fig. 10 which is used to select and initiate the appropriate one of the selectable integration periods for each pixel.
  • the column or pixel address is communicated on bus 1028 to indicate the column selected for readout processing.
  • This signal is decoded by logic in block 1029 and used to generate a signal on 1009 to gate the indication of the integration period used for the pixel to bus 1010 to synchronize its presentation for convenient inclusion as part of the readout value.
  • the analog pixel values sampled in the parallel row read operation may optionally be digitized in block 1029 but are preferably communicated in analog form to bus 1030 for final signal processing and digitization and combining with the indication of the selected integration period communicated on bus 1010.
  • This preferably digitized pixel reading in combination with the indication of the integration period used to acquire the reading provide an indication of the light level to which the pixel was exposed for a very wide range of light levels.
  • Gate 1038 has the integration period identifying index stored for the pixel as its input and outputs a high level on line 1047 when the index value is still equal to the all one value to which it was initialized at the beginning of the acquisition period indicating that a longer integration period is not in progress for the pixel.
  • Block 1033 is an optional zero offset control function which may be provided, preferably using double correlated sampling, to improve accuracy of pre-sample readings.
  • Compare circuit 1032 functions to compare (preferably in analog form) the pixel reading against a threshold value transmitted on line 1031 and to provide a low output on output line 1008 to indicate that the pixel reading is lower than the threshold value.
  • Flip-flop 1014 is set and its output asserted on line 1003 to enable switching the transfer gate in the selected pixel in the column to its conducting state. As described elsewhere, the transfer gate is switched to its conductive state as part of the reset operation to begin integration and also to transfer light induced charge to the floating diffusion during the pixel readout operation.
  • the output 1046 of gate 1039 is high when a longer integration is not in progress as indicated by a high on line 1047 and the reading is less than the threshold setting as indicated by a low level on line 1031. This is the condition used to conditionally initiate a new, longer integration period by writing the index of the integration period asserted on bus 1017 into memory 1044 to begin the new integration period.
  • Flip-flop 1016 is set to enable writing a new index value to the memory unconditionally when line 1040 is high or when logic output 1046 is high and 1041 is clocked.
  • Flip-flop 1014 is set and its high output asserted on line 1003 to enable switching the transfer gate in the selected pixel in the column to its conducting state.
  • Operation of the transfer gate is provided unconditionally by asserting signal 1036 when 1037 is clocked.
  • This unconditional setting is used for readout of the pixel value and for the initial reset to begin integration for the first pre-sample in the pixel reading sequence and for optional modes of operation where the high dynamic range feature is modified or not used.
  • the output 1047 indicative of the unselected state of the integration period is used by not asserting 1036 when 1037 is clocked to conditionally enable the transfer gate to be switched to its conductive state for operations performed prior to writing a new index to begin a longer integration period for the pixel.
  • the line 1047 which indicates whether or not an integration period is selected for the pixel is used to enable transfer gate transitions to the conductive state to continue until an integration period is selected for the pixel and to then selectively inhibit transitions of the transfer gate in the pixel to its conductive state during the integration period for the pixel until it is read.
  • flip-flop 1016 is first set in response to selection of an integration period and setting of flip-flop 1014 is timed to permit operation of the transfer gate as required to perform the reset operation to initiate the selected integration period.
  • the signal on 1047 is not asserted because of the new index stored in memory for the pixel. This prevents further updates in the selected integration period because flip-flop 1016 is not set and further switching of the transfer gate to its conductive state prior to readout of the pixel is inhibited because flip-flop 1014 is not set.
  • pixel structure 1001 may be derived from a conventional 4T design with the addition of block 1002 to provide selective operation of the transfer gate in the pixel.
  • element 1002 is added to a pixel structure which in other respects preferably conforms to normal 4T pixel design practices.
  • Block 1002 responds to a combination of signal levels asserted on a control line associated with the row or parallel addressable group to which the pixel belongs (1021) and on a control line (1003) associated with the column of pixels which contains pixel 1001.
  • the function of signals asserted on 1003 include control of the pixel to select between a mode to select an integration period for the pixel and a mode to allow an integration period which has been selected and initiated for the pixel to continue to completion at which point, the light induced, integrated charge on the pixel is read.
  • the functions of signals asserted on 1021 include selection of the pixel in the column on which operations are to be performed. This selection function may be shared, replaced, or augmented by pixel selective functions performed by operation of signals such as the reset signal controlled by 1027 or the row select function controlled by 1026.
  • logic 1002 provides a logical "and" function to switch the transfer gate to its conductive state when signals on 1021 and 1003 both indicate that the transfer gate should be in its conductive state and to maintain the transfer gate in its non-conductive state otherwise.
  • Options to implement the transfer gate function include use of two analog switching devices in series which provide conduction only when both of the series transfer gate devices are conductive.
  • a second and preferable option is to provide a logic "and” function to appropriately drive the transfer gate control input (transfer gate transistor gate) to switch the transfer gate to its conducting or nonconducting states selectively in response to the control signals.
  • transfer gate control input transfer gate transistor gate
  • special level shifting of signals may be provided to extend the usable dynamic range of the pixel. This may be a normal extension of current practice to provide extended ranges for driving the transfer gate control input or the reset transistor control input to enhance pixel performance and dynamic range.
  • Figs. 9 and 10 The logic sequence indicated in Figs. 9 and 10 with use of the pre-sample readings on which to base conditional selection of the integration period may also be applied to pixel structures which do not have or do not use a transfer gate. In these cases other selective control options may be used to initiate integration and to isolate the pixel so that integration may progress to completion once an integration period has been selected for a pixel. For example, for a 3T pixel, use of selective reset may be substituted for use of selective operation of the transfer gate, and remaining features of the circuits and operations described generally in relation to Figs. 9 and 10 may be used. This and similar cases are considered to be part of this invention. )171] Fig.
  • Fig. 12 is included to illustrate pictorially organization and repetition of components which are described verbally in connection with Figs. 5 for 3T based designs and Fig. 10 for 4T based designs.
  • Fig. 12 depicts an imager with an array 1200 of pixels organized in rows (or parallel addressable groups) with first row 1203, second row 1204, etc. through last row 1205. Each pixel of each row is also a member of a column with first column 1206, second column 1207, etc. through last column 1208.
  • Row select circuits 1217 enable control signals to single out a selected row or parallel addressable group in order to perform prescribed tasks on pixels of the selected row or parallel addressable group. Most operations such as initiation of integration periods, readout of pixel values, etc.
  • CMOS imaging devices that employ what is known as a rolling shutter image acquisition and readout sequence. It should be noted here that use of pixel by pixel sequential readout is not necessary to practice this invention.
  • Pixel level selection circuits 1207 with row input 1210 and column input 121 1 and column selection lines 1215 are added to the imaging array 1200 as part of the invention.
  • Parallelized column task execution units 1222 and pixel integration period memory circuits 1225 are provided as part of column logic block 1202. These added circuits are provided as part of this invention to implement the multiple selectable integration periods which are preferably selectable independently for each pixel, and for each image acquired as detailed elsewhere herein.
  • the diagram of Fig. 12 is applicable for either the 3T based design with column logic detailed in Fig. 5 or the 4T based design with column logic detailed in Fig. 10.
  • pixel 1228 corresponds to pixel 516 of Fig. 5
  • pixel level selection block 1207 corresponds to pixel reset logic function block 502
  • row or parallel addressable group logic select line 1214 corresponds to row reset line 520
  • column logic select line 1215 corresponds to column reset line 503
  • column logic block 1222 and associated memory 1225 and interconnect 1223 corresponds to column logic and memory block 505
  • pixel integration period output bus 1227 corresponds to output bus 506.
  • pixel 1228 corresponds to pixel 1022 of Fig. 10
  • pixel level selection block 1207 corresponds to pixel transfer gate logic function block 1002
  • row or parallel addressable group logic select line 1214 corresponds to row transfer gate operation line 1021
  • column logic select line 1215 corresponds to column transfer gate operation line 1003
  • column logic block 1222 and associated memory 1225 and interconnect 1223 corresponds to column logic and memory block 1007
  • pixel integration period output bus 1227 corresponds to output bus 1010.
  • Design attributes for a preferred floating point output circuit for the imager are illustrated in Fig. 13.
  • the depicted design in Fig. 13 provides features which may be implemented by a circuit presented in block diagram form in Fig. 11. Since the device depicted in Fig.
  • the circuit implementation of the circuit of Fig. 11 to conform to the objectives set forth in Fig. 13 preferably includes capability to shift a "1" from the least significant bit position to the most significant bit position in order to normalize A/D readings which are full zero except for a "1" in the least significant bit position of the reading.
  • a primary feature of the circuit is choice of a scale factor for the floating point values for which all (or substantially all) of the nonzero light level readings for the imager have output values which have exponent values greater than the lowest value so that the values are normalized to conform to standard floating point formats.
  • the output format is preferably chosen so that the values may be mapped directly into selected bit positions of a targeted industry standard floating point number such as those conforming to the IEEE 754 standard or proposed extensions thereof and so that such mappings are not restricted to a specific size for the targeted floating point number format.
  • a targeted industry standard floating point number such as those conforming to the IEEE 754 standard or proposed extensions thereof
  • the number of bits needed to convey the pixel values may be significantly less than the number of bits of the floating point number formats into which the pixel values are mapped.
  • An imaging device using the circuit of Figs. 11 and 13 will communicate with devices which accept certain floating point number sizes using direct input to selected pixels and with no additional or at most simple, additional interface circuits. Furthermore, the exponent of the normalized floating point numbers may be mapped directly into the integral portion of a binary representation of a logarithm of the value leaving only the fractional portion which spans one octave to be converted from linear to logarithmic form. With a limited resolution (10 bits, for example) of the AID used to digitize the pixel values, a lookup table or a decoding circuit may be used to perform the linear to logarithmic conversion.
  • Incorporation of a circuit to provide a direct logarithmic output in addition to or in place of a floating point output circuit is an option for the imaging devices described herein.
  • the floating point or logarithmic conversion circuits are incorporated into an imaging device, it is preferable to included programmable options to select or bypass use of the floating point and/or logarithmic output formats and to provide an output format which directly indicates the AID reading and the integration period used for each pixel reading.
  • conversion to a format which may be flexibly input to any one of several floating point formats is particularly desirable for the imaging devices of this invention with their extraordinarily wide dynamic range and selectable integration periods.
  • Choice of integration periods for which indication of the integration period maps naturally into the exponent of a floating point number and the digitized pixel reading maps naturally into a mantissa of a floating point number facilitates conversion to a floating point format which provides a compact format for transmission and storage of the corresponding pixel data and which also provides an effective way to manipulate the data using computational devices with floating point processing capabilities.
  • Use of floating point representation also facilitates conversion of imaging device related data to, and from, a logarithmic representation.
  • a color filter array such as one using a Bayer pattern may be used on the imaging device.
  • a preferable early step in image processing to produce a displayable image is to apply an interpolation algorithm to supply missing color components to each pixel.
  • the high dynamic range image is preferably processed to compress the range to render an image that is displayable on a display using a much lower dynamic range than present in the high dynamic range image from the imaging device.
  • this step there are two general options, the first being to use the same mapping for all pixels in an image and the second being to apply a locally variable mapping algorithm which is preferably based at least in part on neighboring pixel values, particularly those in a somewhat extended neighborhood of the pixel in question.
  • FDDR High-Dynamic-Range
  • An interpolation algorithm preferably utilizes values, in a linear, preferably floating point format.
  • a number of the high dynamic range compression techniques which have proven to be effective apply an algorithm to extract a luminance value from each pixel and next to convert the luminance value to a logarithmic form which is also preferably expressed in a floating point format. Then algorithms are applied to perform calculations to determine a new luminance value or a scaling factor for the initial pixel value.
  • This luminance value is preferably converted back to a linear range and combined with chroma information originally extracted from the pixel to yield a pixel color and luminance value for display.
  • This new pixel value is converted, if necessary, to a form in which it is transmitted to a display device as part of an image to display.
  • Advantages of the present invention include the synergies and circuit economies provided in direct mapping of a numerical indication of an integration period for a pixel to an exponent of the pixel value and retention of an A/D portion of the pixel reading as the binary value for the pixel reading; conversion of a pixel reading to a compact, normalized floating point representation, characterization of this floating point pixel value output from the imaging device so that it may be directly input to image processing devices which handle at least one of a variety of floating point formats, and finally utilization in the image processing device of the economies of conversion of the appropriately scaled floating point, pixel related values to, and from logarithmic, form.
  • Fig. 13 provides examples of the values at stages in the conversion; an example of mapping into a 16 bit float, and two examples for mapping into 32 bit float for each of four representative pixel values are included.
  • the output values in these examples conform to an output format which includes features referenced above.
  • 1801 depicts output of a pixel reading with a full scale value along with insertion into representative floating point formats.
  • 1802 depicts output of a pixel reading with a zero value along with insertion into representative floating point formats.
  • 1803 depicts output of a pixel reading with a minimum nonzero value along with insertion into representative floating point formats.
  • 1804 depicts output of a pixel reading with a value which is normalized along with insertion into representative floating point formats.
  • a pixel value input to the circuit includes a 10 bit A/D reading of a pixel value in linear binary format which is preferably used as a starting binary value for a pixel reading and a 3 bit indication of an integration period used for the pixel.
  • Integration periods for the device in the example are numbered, in binary form, from 0 for the longest to 7 for the shortest and sized so that the sensitivity for each higher numbered integration period is one fourth of the sensitivity for the integration period with the next lower number in the numbering sequence.
  • the integration period is multiplied by 2 and 16 is added to it to provide a 5 bit binary exponent.
  • the value at this stage in the circuit is scaled and offset but non-normalized.
  • a circuit detects the number of leading zeros in the binary value. If the binary value is full zero, the exponent and any other bits such as the range utilization indicator or indicators which are intended to be a part of a floating point output or to be mapped into a target floating point inputs are preferably set to zero to satisfy a usual floating point standard that the floating point zero value is represented by an all zero value. For non-zero values, the binary value is shifted left by a number of bit positions equal to the number of leading zeros in the non- normalized value and this number is subtracted from the binary exponent.
  • mapping For this mapping, a "0" is routed to the sign bit input and preferably a zero or optional range utilization indication is input to the least significant bit of the binary value.
  • the 14 bit floating value output from the imaging device is connected to the remaining 14 bits of the 16 bit floating point input as indicated by correspondence of values in vertical columns in Fig. 13.
  • the value labeled "Mapped to 32 Bit Float (Direct)” is an example of mapping the 14 bit encoding into an input which accepts a 32 bit floating point value in the IEEE 754 standard format.
  • a "0" is routed to the sign bit, the five most significant bits of the exponent and 15 of the least significant bits of the binary value.
  • a sixteenth zero or optional range utilization indication is input to the least significant bit of the binary value.
  • the 14 bit floating value output from the imager is connected to the remaining 14 bits of the 36 bit floating point input as indicated by correspondence of values in vertical columns in Fig. 13.
  • the value labeled "Mapped to 32 Bit Float (0-1)" is a second example of mapping the 14 bit encoding into an input which accepts a 32 bit floating point value in the IEEE 754 standard format.
  • the input for the 32 bit float for the (0-1) scaling is unchanged from the direct input to the 32 bit float.
  • the added leading ones with the zero in the most significant bit of the exponent result in a pixel value range between 0 and 1 being input in the floating point format.
  • a subset of the features just enumerated may be satisfied by encoding pixel values so that some of the values fall in the lowest non-normalized range.
  • many restrictions are added.
  • All non-normalized numbers must be mapped to the range with the zero binary exponent in IEEE 754 compatible encodings. Thus, the offset used for the exponent in the examples will not work.
  • the non-normalized values include the most significant 1 so a bit is not saved in the length of the binary value so 15 rather than 14 bits are required for lossless encoding of the range of pixel values present in the example.
  • the exponents of the non-normalized range do not map directly into the integral part of the exponent of a logarithm to the base two of the respective pixel values and more than one octave is included in the range of binary values for the non-normalized range complicating and extending the range needed for conversion of the binary value to or from logarithmic form.
  • This last consideration also applies generally to conversions to and from logarithmic form which are made in a processor used to operate on image data. In this case, it is preferable to scale values so that substantially all of the values in the range of interest for the conversion to or from logarithmic form are represented as normalized numbers in the floating point format used in a processor.
  • the integration period actually used to acquire a pixel reading is lost in the conversion to floating point. As indicated elsewhere herein, this information may be of interest to indicate how well selected integration ranges are utilized and other uses include application of adjustments of image data based on the integration range used to acquire the individual pixel data in order to compensate for residual errors due to integration range dependent differences in reading values. Uses of indications of the integration period used for a pixel reading may also include detection of features in the image such as images of moving, high contrast edges, and time varying light sources which may cause higher or lower pixel readings for given pixels than would be encountered if the light level to which the pixel was exposed was not varying rapidly during the pixel integration period.
  • the criteria suitable for the indication will vary and is preferably programmable and for some purposes, it is desirable to have more than one indication or to have multiple bit indications to convey more than a single condition.
  • the condition may be communicated as an output which is separate from the pixel data but which is still preferably correlated with individual pixels on a one to one basis.
  • the indication may be input separately from the pixel but may optionally and often conveniently be input and encoded in the pixel data for the associated pixel. This is the option which is illustrated in Fig. 13 where the bit values represented by "R" are determined as indicated in the column with the heading "Range Utilization Ind" and as an example are optionally substituted in the low order bit positions labeled with an R in the floating point input words.
  • the indication is inserted in known positions in low order bits of the binary value where they have minimal effect on the pixel value but where they are in known positions in the floating point numbers so that their state may be interrogated by the data processing device to which they are input.
  • they may be input in the sign bit of the floating point numbers which are normally otherwise held in the zero or positive state. Interrogation is particularly easy and may be performed by testing the sign of the input number. The effect of the sign setting may be ignored in computations involving the pixel values by using their absolute values which ignores the sign value. 1182] In the example of Fig.
  • the range utilization indication bit is set to 1 when an integration period other than the longest was used to acquire the pixel value and when the pixel A/D reading had two or more leading zeros before normalization. Such readings could have been acquired using the next longer integration period. Under favorable conditions, most pixel readings will be acquired using the longest or next to longest integration period so this information along with the pixel value is enough to effectively infer what integration period was used and will be accurate for most of the pixel readings.
  • programmable threshold values for the number of leading zeros or of an A/D reading threshold value to at least in part determine the "Range Utilization Indication" output is a desirable option for some designs. It is also desirable to output a zero indication whenever the A/D reading is zero so that the bit may be directly inserted into a floating point value without resulting in an invalid reading.
  • an automatic vehicle equipment control system 1406 is shown to be installed within a controlled vehicle 1405.
  • the control system 1406 is depicted to be integral with the interior rearview mirror assembly, it should be understood that the control system, or any of the individual components thereof, may be mounted in any suitable location within the interior, or on the exterior, of the controlled vehicle 1405.
  • the term "controlled vehicle” is used herein with reference to a vehicle comprising an automatic vehicle exterior light control system.
  • Suitable locations for mounting the associated image sensor are those locations that provide an unobstructed view of the scene generally forward of the controlled vehicle 1405 and allow for detection of headlights 1416 of oncoming vehicles 1415 and taillights 141 1 of leading vehicles 1410 within the glare zone 1408 associated with the controlled vehicle.
  • Fig. 15 depicts a controlled vehicle 1505 comprising an interior rearview mirror assembly 1506 incorporating an automatic vehicle exterior light control system.
  • the processing and control system functions to send configuration data to the imager, receive image data from the imager, to process the images and to generate exterior light control signals.
  • Detailed descriptions of such automatic vehicle exterior light control systems are contained in commonly assigned U.S.
  • the controlled vehicle is also depicted to include a driver's side outside rearview mirror assembly 1510a, a passenger's side outside rearview mirror assembly 1510b, a center high mounted stop light (CHMSL) 1545, A-pillars 1550a, 1550b, B-pillars 1555a, 1555b and C- pillars 1560a, 1560b; it should be understood that any of these locations may provide alternate locations for an image sensor, image sensors or related processing and, or, control components. It should be understood that any, or all, of the rearview mirrors may be automatic dimming electro- optic mirrors.
  • CHMSL center high mounted stop light
  • the controlled vehicle is depicted to include a host of exterior lights including headlights 1520a, 1520b, foil weather lights 1530a, 1530b, front turn indicator/hazard lights 1535a, 1535b, tail lights 1525a, 1525b, rear turn indicator lights 1526a, 1526b, rear hazard lights 1527a, 1527b and backup lights 1540a, 1340b.
  • additional exterior lights may be provided, such as, separate low beam and high beam headlights, integrated lights that comprise multipurpose lighting, etc.
  • any of the exterior lights may be provided with positioners (not shown) to adjust the associated primary optical axis of the given exterior light.
  • the controlled vehicle of Fig. 15 is generally for illustrative purposes and that suitable automatic vehicle exterior light control systems, such as those disclosed in the patents and patent applications incorporated herein by reference, may be employed along with other features described herein and within disclosures incorporated herein by reference.
  • FIGs. 16a and 16b an embodiment of an interior rearview mirror assembly 1600a, 1600b is shown.
  • the mirror assembly includes a stationary accessory assembly enclosed within a front housing 1685a, 1685b and a rear housing 1690a, 1690b.
  • the front housing comprises an aperture 1686b defining an image sensor visual opening.
  • the stationary accessory assembly along with a rearview mirror are carried by an attachment member 1655a, 1655b.
  • the rearview mirror comprises a mirror housing 1660a, 1660b, a bezel 1661a, 1661b and a mirror element 1662a.
  • a wire cover 1694a, 1694b is included to conceal related wiring 1615b.
  • the rearview mirror assembly 1600a, 1600b also incorporates an ambient light sensor 1665b, at least one microphone 1666b, a glare light sensor 1665a, operator interfaces 1663a, indicators 1664a and at least one information display 1670.
  • the accessory and rearview mirror mount assembly provides a rigid structure for mounting a repositionably mounted interior rearview mirror along with a precisely aligned image sensor either stationarily mounted as described in more detail within commonly assigned U.S. Patent application serial number 10/783,273 (7606) or automatically repositioning as described in commonly assigned U.S. Patent application serial number 10/645,801, both of which are hereby incorporated in their entireties herein by reference.
  • a preferred accessory and rearview mirror mount assembly facilitates ease of assembly as well as provides for repeatable, reliable and precise alignment of the related components.
  • Imager board 1710 is provided with an image sensor with lens 1711.
  • the imager board will also include an image sensor control logic and timing circuit, communication line drivers and wire harness receptacle 1713.
  • the imager board may comprise a processor for receiving and, at least partially, processing images obtained from the image sensor.
  • the image sensor and at least one other device selected from the group comprising; 1) an image sensor control logic; 2) an A/D converter; 3) a low voltage differential signal line driver; 4) a temperature sensor; 5) a control output; 6) a voltage regulator; 7) a second image sensor; 8) a microprocessor; 9) a moisture sensor and 10) a compass are integrated in a common ASIC, most preferably on a common silicon wafer.
  • the image sensor with lens 1711 includes lens cover snap portions 1712 for engaging a lens cover 1720 snap clips 1721.
  • the lens cover has an aperture 1722 for alignment with the optical axis of the image sensor and lens.
  • optical systems such as those depicted and described in commonly assigned U.S. Patents 5,990,469; 6,008,486; 6,130,421; 6,130,448; 6,049,171; and 6,403,942 and U.S. Patent Application serial number 60/495,906 (2880); the disclosures of which are incorporated herein in their entireties by reference; may be employed.
  • optics in accordance with the present invention may obviate the need for a lens cover 1720 as described in detail herein.
  • the lens cover snap portions, the lens optical cover and snap clips may be eliminated with use of optical elements in accordance with the present invention.
  • the "lens cover” is formed on a molded organic material optics element using a laser as described in detail herein.
  • An imager board wiring harness (not shown) is preferably provided with plugs on either end thereof.
  • the imager board is preferably provided with a male receptacle 1713 for receiving one of the plugs of the imager board wiring harness (not shown).
  • Fig. 18 depicts a profile view of a digital camera 1806 in accordance with the present invention having an imager with lens 1811. It should be understood that optics in accordance with the present invention may be incorporated into a host of assemblies included, but not limited to, light sensing, image acquisition, moisture sensing, rear-vision systems, lane departure detection systems, adaptive cruse control systems, occupancy detection systems, security systems, vision systems, color measurement systems, head lamp control systems, variable reflectance rearview mirror control systems, digital video recorders and digital cameras.

Abstract

La présente invention porte sur un dispositif d'imagerie perfectionné présentant une plage dynamique élevée et sur divers appareils incorporant ces dispositifs d'imagerie à plage dynamique élevée. Dans un mode de réalisation de conception préférée, un traitement en parallèle de rangée est incorporé pour lire un signal indicatif de la charge induite par la lumière pour chaque pixel de la rangée, pour comparer le niveau indiqué de la charge induite par la lumière par rapport à un niveau de seuil pour chaque pixel de la rangée pour, pour chaque pixel de la rangée, interroger l'enregistrement de mémoire de l'état de réinitialisation, pour déterminer s'il faut ou non réinitialiser chaque pixel de la rangée sur la base du résultat de la comparaison, l'indication de mémoire et l'étape dans la séquence de lecture, pour stocker un enregistrement mis à jour de l'historique réinitialisé partiel, selon ce qui est requis, pour chaque pixel de la rangée, et finalement pour réinitialiser de manière conditionnelle chaque pixel de la rangée sur la base des informations ci-dessus pour le pixel respectif.
PCT/US2008/009739 2007-12-06 2008-08-13 Dispositif d'imagerie perfectionné WO2009073054A1 (fr)

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US11/999,623 US8289430B2 (en) 2007-02-09 2007-12-06 High dynamic range imaging device
US11/999,623 2007-12-06
US12/082,215 2008-04-09
US12/082,215 US8629927B2 (en) 2008-04-09 2008-04-09 Imaging device
US12/150,234 2008-04-25
US12/150,234 US8305471B2 (en) 2007-02-09 2008-04-25 High dynamic range imaging device
US12/157,476 US8587706B2 (en) 2008-01-30 2008-06-11 Imaging device
US12/157,476 2008-06-11

Publications (2)

Publication Number Publication Date
WO2009073054A1 true WO2009073054A1 (fr) 2009-06-11
WO2009073054A8 WO2009073054A8 (fr) 2009-11-12

Family

ID=40718023

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2008/009739 WO2009073054A1 (fr) 2007-12-06 2008-08-13 Dispositif d'imagerie perfectionné

Country Status (1)

Country Link
WO (1) WO2009073054A1 (fr)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8070332B2 (en) 2007-07-12 2011-12-06 Magna Electronics Inc. Automatic lighting system with adaptive function
EP2439930A2 (fr) 2010-10-06 2012-04-11 Imec Améliorations concernant l'augmentation de la plage dynamique des capteurs d'images.
US8189871B2 (en) 2004-09-30 2012-05-29 Donnelly Corporation Vision system for vehicle
US8376595B2 (en) 2009-05-15 2013-02-19 Magna Electronics, Inc. Automatic headlamp control
US8842176B2 (en) 1996-05-22 2014-09-23 Donnelly Corporation Automatic vehicle exterior light control
US8890955B2 (en) 2010-02-10 2014-11-18 Magna Mirrors Of America, Inc. Adaptable wireless vehicle vision system based on wireless communication error
US9117123B2 (en) 2010-07-05 2015-08-25 Magna Electronics Inc. Vehicular rear view camera display system with lifecheck function
US9126525B2 (en) 2009-02-27 2015-09-08 Magna Electronics Inc. Alert system for vehicle
US9194943B2 (en) 2011-04-12 2015-11-24 Magna Electronics Inc. Step filter for estimating distance in a time-of-flight ranging system
CN103716554B (zh) * 2012-10-02 2017-04-12 佳能株式会社 光电转换装置及其测试方法和成像系统及其制造方法
IT201900003933A1 (it) * 2019-03-19 2020-09-19 Fond Bruno Kessler Un metodo per aumentare l’intervallo dinamico di un sensore di immagine e architettura circuitale di un sensore di immagine configurata per attuare il suddetto metodo

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060215882A1 (en) * 2005-03-22 2006-09-28 Omron Corporation Image processing apparatus and method, recording medium, and program
US20070257998A1 (en) * 2004-12-16 2007-11-08 Fujitsu Limited Imaging apparatus, imaging element, and image processing method

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070257998A1 (en) * 2004-12-16 2007-11-08 Fujitsu Limited Imaging apparatus, imaging element, and image processing method
US20060215882A1 (en) * 2005-03-22 2006-09-28 Omron Corporation Image processing apparatus and method, recording medium, and program

Cited By (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8842176B2 (en) 1996-05-22 2014-09-23 Donnelly Corporation Automatic vehicle exterior light control
US8483439B2 (en) 2004-09-30 2013-07-09 Donnelly Corporation Vision system for vehicle
US10623704B2 (en) 2004-09-30 2020-04-14 Donnelly Corporation Driver assistance system for vehicle
US8977008B2 (en) 2004-09-30 2015-03-10 Donnelly Corporation Driver assistance system for vehicle
US8189871B2 (en) 2004-09-30 2012-05-29 Donnelly Corporation Vision system for vehicle
US10807515B2 (en) 2007-07-12 2020-10-20 Magna Electronics Inc. Vehicular adaptive headlighting system
US8814401B2 (en) 2007-07-12 2014-08-26 Magna Electronics Inc. Vehicular vision system
US8070332B2 (en) 2007-07-12 2011-12-06 Magna Electronics Inc. Automatic lighting system with adaptive function
US8142059B2 (en) 2007-07-12 2012-03-27 Magna Electronics Inc. Automatic lighting system
US10086747B2 (en) 2007-07-12 2018-10-02 Magna Electronics Inc. Driver assistance system for vehicle
US10839233B2 (en) 2009-02-27 2020-11-17 Magna Electronics Inc. Vehicular control system
US11288888B2 (en) 2009-02-27 2022-03-29 Magna Electronics Inc. Vehicular control system
US9911050B2 (en) 2009-02-27 2018-03-06 Magna Electronics Inc. Driver active safety control system for vehicle
US11763573B2 (en) 2009-02-27 2023-09-19 Magna Electronics Inc. Vehicular control system
US9126525B2 (en) 2009-02-27 2015-09-08 Magna Electronics Inc. Alert system for vehicle
US10744940B2 (en) 2009-05-15 2020-08-18 Magna Electronics Inc. Vehicular control system with temperature input
US10005394B2 (en) 2009-05-15 2018-06-26 Magna Electronics Inc. Driver assistance system for vehicle
US9187028B2 (en) 2009-05-15 2015-11-17 Magna Electronics Inc. Driver assistance system for vehicle
US8376595B2 (en) 2009-05-15 2013-02-19 Magna Electronics, Inc. Automatic headlamp control
US11511668B2 (en) 2009-05-15 2022-11-29 Magna Electronics Inc. Vehicular driver assistance system with construction zone recognition
US8890955B2 (en) 2010-02-10 2014-11-18 Magna Mirrors Of America, Inc. Adaptable wireless vehicle vision system based on wireless communication error
US9117123B2 (en) 2010-07-05 2015-08-25 Magna Electronics Inc. Vehicular rear view camera display system with lifecheck function
US8792032B2 (en) 2010-10-06 2014-07-29 Imec Dynamic range enhancement
EP2439930A2 (fr) 2010-10-06 2012-04-11 Imec Améliorations concernant l'augmentation de la plage dynamique des capteurs d'images.
US9194943B2 (en) 2011-04-12 2015-11-24 Magna Electronics Inc. Step filter for estimating distance in a time-of-flight ranging system
US10288724B2 (en) 2011-04-12 2019-05-14 Magna Electronics Inc. System and method for estimating distance between a mobile unit and a vehicle using a TOF system
CN103716554B (zh) * 2012-10-02 2017-04-12 佳能株式会社 光电转换装置及其测试方法和成像系统及其制造方法
IT201900003933A1 (it) * 2019-03-19 2020-09-19 Fond Bruno Kessler Un metodo per aumentare l’intervallo dinamico di un sensore di immagine e architettura circuitale di un sensore di immagine configurata per attuare il suddetto metodo

Also Published As

Publication number Publication date
WO2009073054A8 (fr) 2009-11-12

Similar Documents

Publication Publication Date Title
US8890985B2 (en) Imaging device
US8305471B2 (en) High dynamic range imaging device
US9641773B2 (en) High dynamic range imaging device
WO2009073054A1 (fr) Dispositif d'imagerie perfectionné
US8289430B2 (en) High dynamic range imaging device
US8619168B2 (en) Image sensor with high dynamic range imaging and integrated motion detection
US8547476B2 (en) Image sensor including real-time automatic exposure control and swallowable pill including the same
US8174593B2 (en) Method and apparatus for detecting image darkening due to reset droop
US20150358571A1 (en) Advanced region of interest function for image sensors
WO2010084493A1 (fr) Pixel optique et capteur d'image
US9531959B2 (en) Imaging systems and methods for capturing image data at high scan rates
US7397503B2 (en) Systems and methods for reducing artifacts caused by illuminant flicker
EP2760199A1 (fr) Région automatique de fonction d'intérêt pour capteurs d'image
JP2019176467A (ja) ルックアップテーブル
Breidenassel et al. A flexible scheme for adaptive integration time control [CMOS image sensor]
KR101078648B1 (ko) 개량된 이미징 장치
US9200951B2 (en) Circuit and method for measuring an ambient light level
JP2000059699A (ja) 動き検出用撮像装置
KR20240008121A (ko) 다단계 바이너리 이미징 판독 방법 및 이를 실행하는 시스템
Fischer et al. The Triangulation Method of Contact-less Dimension Measurement Using an Area CMOS Imaging Sensor

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 08795333

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 08795333

Country of ref document: EP

Kind code of ref document: A1