IL91095A - Video imaging apparatus - Google Patents

Description

091095/2 VIDEO IMAGING APPARATUS SIGHT, INC.

C:08786 Inventors: Ran Ginosar Yehoshua Y. Zeevi Oliver Hilsenrath Ofra Zinaty Daniel Kligler 08786tec.int 1-497 23 - 7.1989 FIELD OF THE INVENTION The present invention relates to video imagery and more particularly to apparatus and techniques for providing enhanced video images.

BACKGROUND OF THE INVENTION Various types of video image enhancement apparatus and techniques have been proposed. AGC techniques have been employed, inter alia, in video signals for reducing the dynamic range of the video signal by subtraction of its DC level. Such a technique is described in U.S. Patent 4 ,7 .350. wherein linear AGC is applied to the video image on a pixel by pixel basis.

U.S. Patent 4 , 601 ,055 describes an image processor which provides, inter alia, neighborhood transforms.

U.S. Patent 4 ,471 , 228 describes a solid state image sensor with exposure controller. The exposure control appears to be applied to the entire frame and is not addressed to individual pixels.

U.S. Patent 4 ,556 ,908 describes a solid state image sensor including a photodiode array which is pixel addressable in what appears to be a random-accessible manner. A photometric output from a selected group of pixels to a light meter is provided, but pixel addressable sensitivity control is not suggested or provided.

U.S. Patent 4 ,472 , 638 describes a two-dimensional solid state image sensor which is able to directly output image information subjected to two dimensional image processing.

U.S. Patent 4 ,734 ,776 describes a system for extending the dynamic range of a CID array. The CID array is designed to provide two integration times, one short and one long, for every pixel in the array, and to read out both of the signals thus produced. The appropriate signal is then selected for each pixel, and, if it is the signal produced from the short integration time, it is amplified to compensate for the difference in sensitivity between the long and short integration times. Pixel-addressable sensitivity is not suggested or provided.

U.S. Patent 4 ,473.836 teaches a sensor or sensor array achieving wide dynamic range by means of a log amplifier integrated with a detector. It does not offer any variation of integration time.

U.S. Patent 4 , 636 ,865 discloses a detector design which allows discharge of excess photocarriers . This is an improved anti-blooming scheme.

U.S. Patent 4 , 656, 361 discloses an nMOS photodiode array with improved sensitivity and dynamic range achieved through the use of an integrated scheme for sensing photo-generated current.

U.S. Patent 4 , 609 ,825 teaches a line-transfer type CCD array providing variable integration time on a row-by-row basis. The variable integration time is achieved through variation of the time between the reset and the signal transfer pulses.

U.S. Patent 4 , 583.002 discloses a solid-state image sensor which provides sensitivity adjustment in response to the overall image intensity. . . a ent , , eac es a array w t an individual amplifier for each row in order to reduce circuit capacitance and to improve the signal/noise ratio.

U.S. Patent 4, 775.799 improves the dynamic range of an image scanner through the techniques of digital and analog signal averaging in a manner which adapts to the condition of the input signal.

U.S. Patent 4,706 , 123 discloses an apparatus for changing the exposure times of individual photodetectors in relation to the amount of incident light on only the individual photodetector. The exposure times are limited to a small number of distinct integration times, typically 2 , and is subject to restrictions as to which pixels can be controlled. Each line of detectors must be accompanied by at least two additional shift registers which is a prohibitive amount of cicuitry for two-dimensional arrays.

S. P. DeWeerth and C. A. Mead, in an article entitled "A Two-Dimensional Visual Tracking Array", presented at the Fifth MIT Conference on Advanced Research in VLSI, March 1988, MIT Cambridge, Massachusetts, U.S.A. and published in the proceedings thereof, disclose a semiconductor photosensitive device which computes, per pixel, the difference between the incident light intensity and the average intensity over some neighborhood. This device is limited by the dynamic range of the individual photodetector and by the complexity of circuitry required per pixel .

E.P. Patent 233.464 appears to show a photodiode array providing selectable feedback amplification for each photodiode The present invention seeks to provide improved video image enhancement apparatus which overcomes limitations of the prior art apparatus and techniques discussed above.

There is thus provided in accordance with a preferred embodiment of the present invention, video imaging apparatus comprising a multiplicity of photosensitive detectors arranged in a staring array for receiving input radiation and detector control apparatus for controlling the response of the photosensitive detectors such that they operate in different operative ranges within the total inherent dynamic range thereof, whereby the output of the staring array at any given time provides image information in respect of input radiation extending over an input radiation dynamic range which is greater than the operative input radiation dynamic range of any given detector.

In accordance with a preferred embodiment of the invention, the apparatus for controlling the response of the photosensitive detectors is addressable simultaneously to a multiplicity of individual pixels or a multiplicity of designated groups of pixels in a field.

In accordance with a preferred embodiment of the invention, the staring array is a generally two-dimensional array. Alternatively, the staring array may comprise a one-dimensional array.

Further in accordance with a preferred embodiment of the invention, the detector control apparatus includes means for se ec a y an var a y e erm n ng e n egra on me o eac of the multiplicity of detectors.

Additionally in accordance with a preferred embodiment of the invention, the staring array may be embodied on a unitary integrated chip, such as in the form of an array of diffusion diodes on a MOS chip.

In accordance with one embodiment of the invention, the detector control apparatus comprises apparatus for providing two control signals to each of the multiplicity of detectors. The detector is actuated in response to simultaneous receipt or absence of a predetermined combination of control signals.

According to an alternative embodiment of the invention, a control circuit is provided for each detector of the multiplicity of detectors.

Additionally in accordance with a preferred embodiment of the invention, the outputs of the detectors are supplied to apparatus for applying thereto a neighborhood transform in order to preserve image contents notwithstanding the reduction in overall dynamic range produced by operation of the detectors with variable integration times.

Further in accordance with a preferred embodiment of the present invention, the detector control apparatus comprises apparatus responsive to radiation incident on the multiplicity of photosensitive detectors for governing the response of the individual ones of the photosensitive detectors.

Still further, in accordance with a preferred embodiment of the present invention, the apparatus responsive to radiation is operative to govern the response of at least one individual one of the photosensitive detectors in response to radiation incident thereon and in response to radiation incident thereon and in the vicinity thereof.

Moreover, in accordance with a preferred embodiment of the present invention, the detector control apparatus includes apparatus for storing a value representing the response of each of the individual ones of the photosensitive detectors and apparatus for updating the stored values. The detector control apparatus is also operative to cause the multiplicity of detectors to provide an output having a dynamic range less than the dynamic range of the input radiation or having a selectable dynamic range.

There is provided in accordance with a preferred embodiment of the present invention, video imaging apparatus also comprising image processing apparatus receiving the outputs of the multiplicity of photosensitive detectors and control signals supplied to the multiplicity of photosensitive detectors and providing in response thereto modified outputs of the detectors such as would be obtained were the array composed of sensors having a uniform sensitivity and an extended dynamic range. The image processing apparatus is operative to cause the multiplicity of detectors to provide an output having a dynamic range generally equal to the dynamic range of the input radiation or having a selectable dynamic range.

Further, in accordance with a preferred embodiment of the present invention, the image processing apparatus receives control signals representing integration times of the mu p c y o e ec ors an prov es n response t ereto modified outputs of the detectors corresponding to a uniform sensitivity.

Additionally, in accordance with a preferred embodiment of the present invention, the multiplicity of photosensitive detectors includes a plurality of detectors, each sensitive to a different color. The detector control apparatus is operative to control the response of the plurality of detectors to a plurality of colors.

Moreover, in accordance with a preferred embodiment of the present invention, the color detector control apparatus includes apparatus responsive to achromatic intensity at a given image location for determining the response and for providing a single control output governing it for the plurality of detectors at that image location.

Further, in accordance with a preferred embodiment of the present invention, the video imaging apparatus comprises a multiplicity of detectors, column and row control apparatus together determining the sample timing of the plurality of photosensitive detectors and apparatus for reading out a video signal from the plurality of photosensitive detectors. Additionally, each of the multiplicity of detectors comprises column and row control apparatus together determining the precharge timing of the plurality of photosensitive detectors.

Still further, in accordance with a preferred embodiment of the present invention, the column and row control apparatus include apparatus for providing multiple precharges of at least some of the plurality of photosensitive detectors in order to prevent accumulation of excessive charge thereat.

There is provided in accordance with a preferred embodiment of the present invention, image processing apparatus comprising apparatus for receiving a video signal and for providing an output video signal having a dynamic range narrower than the input dynamic range, while preserving edge information and removing some relatively spatially unvarying information.

Additionally, in accordance with a preferred embodiment of the present invention, the apparatus for receiving and providing video signals comprises at least two neighborhood operator apparati and summation apparatus for combining the outputs of the at least two neighborhood operator apparati.

Further in accordance with a preferred embodiment of the present invention, the apparatus for receiving and providing comprises at least one neighborhood operator and at least one look up table which together receive the input video signal and provide a multiplication factor which is multiplied by the input video signal to provide the output video signal.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings in which: Fig. 1 is a block diagram illustration of a video imaging system constructed and operative in accordance with a preferred embodiment of the present invention; Fig. 2A is a simplified block diagram of a detector array and associated detector control apparatus in accordance with one embodiment of the invention; Figs. 2B and 2C are simplified circuit diagrams of two alternative embodiments of detector elements useful in the present invention; Fig. 2D is an illustration useful in the understanding of the operation of the detector array and control apparatus of Fig. 2A; Fig. 2E is a series of timing diagrams illustrating the operation of the apparatus of Fig. 2A; Figs. 3A and 3B are simplified block diagrams of a detector array and associated detector control apparatus in accordance with an alternative embodiment of the invention; Fig. 4 is an electrical diagram of detector control apparatus for a given detector in the embodiment of Figs. 3A and 3B; Fig. 5 is a pictorial illustration of the relationship between pixels processed in a typical neighborhood transform used in an embodiment of the invention; Fig. 6 is an electrical diagram of circuitry useful in carrying out a neighborhood transform in accordance with an embodiment of the invention; Fig. 7 is a simplified block diagram of control circuitry suitable for controlling the apparatus of Fig. 2A;.

Figs. 8A and 8B are simplified block diagrams of two alternative embodiments of control circuitry suitable for controlling the apparatus of Fig. 2A; Fig. 8C is a block diagram of circuitry useful in the circuitry of Figs. 8A and 8B; Figs. 9A and 9B are simplified block diagrams of two alternative embodiments of control circuitry suitable for controlling apparatus similar to that illustrated in Fig. 2A for full color operation; and Figs. IGA, 1GB and 1GC are simplified block diagrams of three alternative embodiments of image processing circuitry useful in the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Reference is now made to Fig. 1, which is a schematic block diagram illustration of a video imaging system constructed and operative in accordance with one preferred embodiment of the present invention. The system comprises a generally two-dimensional solid state detector array 10, which is interconnected with adaptive sensitivity control apparatus 12. The output of the detector array 10 is supplied via a suitable bus l4 to video readout control circuitry 16, which may comprise conventional raster scan circuitry or alternatively intelligent scan circuitry as described and claimed in Israel Patent Application 83213.

Fig. 2A illustrates the interconnection between the adaptive sensitivity control apparatus and the detector array 10 in accordance with one preferred embodiment of the invention. It is seen that each detector element 20, corresponding to an individual pixel, is provided with connections to both a row sample control circuit 22 and a column sample control circuit 24, both of which may employ shift registers.

Fig. 2B illustrates the specific interconnections of each detector element 20. Here it is seen that first and second AND gates 30 and 32 each have a first input connected to column sample control circuit 2 (Fig. 2A) via respective precharge (P) leads 3^ and sample (S) leads 36 and have a second input connected to row sample control circuit 22 (Fig. 2A) via a lead 38. The number of leads for each detector element 20 may be reduced to two if a local delay circuit for determining the time between sample and precharge is provided.

The photodetector element such as a photodiode 40 corresponding to each pixel is coupled to a precharging voltage V via a P switch 42 , governed by the output of AND gate 30 , and P is coupled via an S switch 44 , governed by the output of AND gate 32 , to a buffer 46 and thus to a capacitor 48. Alternatively, it is possible to reverse the location of switch 44 and buffer 46. It is also possible to reverse the location of buffer 46 and capacitor 48. It is to be appreciated that each pixel ceil may include additional circuitry not shown in Fig. 2B . The circuitry of fig. 2B , is however, all that is required for an understanding of the present invention.

It may thus be appreciated that the integration time of each photodetector element corresponding to a pixel is governed by the provision of appropriate electrical signals along leads 34 , 3 and 38. Initialization of integration is provided when AND gate 30 simultaneously receives inputs along leads 34 and 38 , thus precharging the photodiode 40. Termination of integration is provided when AND gate 3 simultaneously receives inputs along leads 3 and 38 , thus closing switch 44 and coupling the photodiode via buffer 46 to capacitor 48.

An alternative embodiment of the detector element 20 eliminates AND gate 32 and its connection, via lead 38 , to the row control circuit 22. The S lead 36 , connected to column control circuit 24 , is directly connected to S switch 44 . Thus, all pixels in a given column are simultaneously sampled and their exposure times are controlled by the precharge signals alone. w e apprec ate t at in the embodiment of Figs. 2A and 2B, the selection of the integration time for each pixel is governed externally of the detector array 10 by the row and column sample control circuits 22 and 24. Circuits 22 and 24 are preferably embodied as binary data registers, such as shift registers.

It will be appreciated that the embodiment of Figs. 2A and 2B may be applied to groups of pixels as well as to individual pixels.

The embodiment of Figs. 1, 2A and 2B may be applied also to modified CCD devices. The overall chip architecture remains as is shown in Fig. 2A but the detector element 20 is as shown schematically in Fig. 2C. In this alternative embodiment, the photodetector element is a MOS depletion capacitor 100, commonly used in CCD detector arrays. As in the previous embodiment, the P switch 42 releases a precharging voltage V P when AND gate 30 simultaneously receives inputs along leads 3 and 38· Precharging voltage V typically removes any accumulated P photocharges from the capacitor 100.

Unlike the previous embodiment, the sampling circuit comprises a conventional CCD transfer gate 102. A complete line (row or column) of such photo-sites is triggered simultaneously to transfer their charges to a parallel line of a CCD shift register 104. In due course, the CCD shift register 104 transfers the charge out of the photo-site. Thus, each detector element 20 receives its own individual precharge but all detector elements 20 along a line are sampled together.

This embodiment of the detector element 20 is similar to that of conventional CCD arrays with the exception that each detector element 20 is independently precharged by its own precharging circuit controlled by AND gate 30· It is a feature of the present invention that each pixel is independently precharged or sampled through switching of the signals on the column and row control leads 3^ 36 and 38. The switching operation is illustrated in Fig. 2D. The signals are typically determined by control words externally loaded into registers 110 and 112 of row and column control circuits 22 and 2k (Fig. 2k), respectively. The control words typically are comprised of a sequence of 1-bit values, typically of either 1 or 0. The control word of column register 112 typically comprises a 1 at a column address j to be precharged and O's at the remaining column addresses. The row control word loaded into register 110 simultaneously with the control word for column j comprises a 1 in each row address i at which a pixel (i,j) of detector array 10 is to be precharged. In Fig. 2D the letter P is placed at those pixels which would be precharged given the column and row control words shown.

The column control words are typically designed such that the 1 is shifted sequentially through register 112, thereby sequentially activating the columns of the detector array 10. The loaded row control word corresponds to the pixels to be precharged in the activated column. This cycle of the column precharge signal 1 through the column control register 112, while a new row word is loaded for each column, will be known hereinafter as a "column cycle" .

In order to produce a plurality of integration times, the column cycle is repeated a plurality of times where each successive column cycle corresponds to a shorter integration or exposure time. Since the pixels of a given column have varied integration times, the row control word loaded for a given column changes with each column cycle. The repetitions end once all of the pixels have completed their reset cycle. At that point, the pixel photocharge values are sampled and their photocharge is transferred out of the detector array 1G.

The sampling is performed on a column-by-column basis in order to maintain a constant integration time, for a given precharge cycle, over all the columns. Thus, the column cycle is repeated once more to produce the sampling. A separate column sample register is typically included in column control circuitry 24 for this purpose. In the first embodiment, shown in Fig. 2B, a row word of all l's is loaded into the row control register before sampling commences. For the other embodiments, whose S switch kk is not connected to row lead 38 & row word generally is not loaded.

Fig. 2E illustrates the precharging process described hereinabove and is an example schematic timing diagram for the pixels (i,j) through (i+2, j+1) of detector array 10. Fig. 2E presents the precharge signals for the two columns, j and j+1, and for the three rows, i, i+1 and i+2, as well as the sample signals for the two columns.

A pixel is precharged when its column and row addresses simultaneously receive the falling edge of their respective precharge signals. Thus, integration of the photon flux for each pixel begins with a precharge signal which is common to both the row and the column in which the pixel resides. Integration ends on the rising edge of the sample signal for the column in which the pixel resides.

Fig. 2E illustrates that each column receives its column precharge signal once per column cycle and that, for a given choice of integration time, column j is always precharged before column j+1. The timing of the row precharge signals is not as regular as that for the columns and depends on the integration time desired for each pixel. As shown in Fig. 2E, there are three example integration times, Tl, T2 and T3 where T3 is the longest and corresponds to the first column cycle.

Although not illustrated in Fig. 2E, it is possible for a pixel to be precharged a multiplicity of times. Each time the pixel is precharged, its accumulated photocharge is discharged. Thus, charge integration for sampling begins at the final precharge. It will be appreciated by those skilled in the art that this mechanism is operative to prevent blooming in pixels subjected to high photon flux, corresponding generally to bright areas of the optical image, which would tend to cause saturation at those pixels in conventional sensor arrays.

Since video readout is independent of the precharge/sample process described hereinabove, integration times are typically of arbitrary length. In the preferred embodiment of the present invention, useful for standard video applications, integration time is equal to or less than the video frame rate of typically 30 frame/sec. e integration t me reso ution of detector array 10 depends on the length of time required for a single column cycle. For example, detector array 10 could be a 512 X 512 pixel detector array operating at a 10 MHz clock rate with a 16-bit data bus for loading in the control words. The detector elements 20 are the CCD type illustrated in Fig. 2C. The example detector array 10 will take 3·2 usee to load a single 512-bit row word and 1.6 msec to step through all 5 columns of the array once. To increase the speed of the detector array 10, a faster clock rate and/or wider data busses may be utilized.

According to an embodiment of the invention, selection of the integration time for each pixel is governed internally of the detector array 10 by adaptive sensitivity control circuitry associated with the photodetector elements corresponding to each pixel. This embodiment is illustrated in Figs. 3A and 3B.

Fig. 3A illustrates a detector array 0 wherein each pixel module 1 communicates via a bus 5 with a video readout control circuit , which may be similar to circuit 16 (Fig. 1) .

Fig. 3B illustrates the structure of each pixel module. It is seen that each module 5 comprises a photodetector element such as a photodiode which is coupled via a P switch 58 to a pre-charging voltage source 60 and via an S switch 62 to a pixel buffer 64, which in turn communicates via bus 52 with the video readout control circuitry 54.

The output V of pixel buffer 64 is also supplied to ij a neighborhood transform processor 66 which is operative to compute a signal N , which is the neighborhood transform of V and of the output of the pixel buffer 64 for the 8 neighbors of pixel ij, as illustrated in Fig. 5. Alternatively another type of neighborhood transform may be employed, such as one which includes only the two vertical and two horizontal neighbors only or as a further alternative, a neighborhood larger than 3 x 3 · In the case of a linear scan, a neighborhood incorporating only the neighbors along the scan line may be employed.

The signal N is supplied to an adaptive sensitivity ij control circuit 68 , which provides control outputs to switches 58 and 62.

A preferred embodiment of neighborhood transform processor 66 is shown in Fig. 6. The circuitry of Fig. 6 indicates that the light intensity values from the eight pixels which abut on a given pixel are averaged by an averaging circuit 90, such as an 8-input differential amplifier or a resistor network as described in the article entitled "A Two-Dimensional Visual Tracking Array" by S. P. DeWeerth and C.A. Mead. The output of averaging circuitry 0 is supplied to a differential amplifier 92 , which also receives the video output signal V ij from the nominal pixel (i,j), via a 1 + epsilon amplifier 9^ · The output of differential amplifier 92 is the neighborhood signal N for pixel ( i , j ) . ij According to an alternative embodiment of the invention, any suitable neighborhood transform may be applied to obtain a neighborhood signal N ij A preferred embodiment of circuit 68 is illustrated in Fig. k. As seen in Fig. k , a differential amplifier 70 receives signal N from neighborhood transform processor 66 and a ij 1 differential amplifier 70 is proportional to the difference between N and V and governs the operation of a transistor 72 , ij 1 which in turn governs charging of a capacitor 74 . A switch 76 resets capacitor 7 when provided with a P signal.

The capacitor 74 is coupled to one input of a comparator 80 which also receives a reference voltage V . The r output of comparator 80 is supplied to a first one shot multivibrator 82 which provides a control signal to switch 62 (Fig. 3B) . The output of one shot 82 is also supplied to a second one shot multivibrator 84, which provides a control signal to switch 58 (Fig. 3B) which is also operative to reset switch 76.

The circuitry of Fig. 4 defines a first order negative feedback loop having a gain less than unity in order to avoid oscillations. Since the neighborhood signal N is monotonic in ij V , the circuitry of Fig. 4 is operative to compress the dynamic ij range of the image. Thus for a given cycle, the higher the image video signal is, the shorter the integration time for the next cycle will be, producing a lower V in that subsequent cycle. ij Higher order feedback loops may also be used for this purpose.

Instead of a photodetector which operates cyclically, it is possible to employ a photosensitive transistor, as described in the article entitled "A Two-Dimensional Visual Tracking Array" as referenced hereinabove.

Reference is now made to Fig. 7 which illustrates a second embodiment of the control circuitry. In this embodiment of the present invention, the adaptive sensitivity control circuitry 12 is partially internal to a sensor chip 119 and partially external to it. The internal circuitry of sensor chip 119 comprises the detector array 10 and the elements directly controlling it, inter alia, the row and column control circuitry 22 and 24, respectively, and video readout control circuitry 16. The adaptive control circuitry 12 external to the chip are those elements which determine the integration time for each pixel on the detector array 1G and which determine the wide dynamic range video signal which is typically the output of the video imaging system of the present invention.

The sensor chip 119 typically operates as described hereinabove for the individual elements. A timing control 120 defines several, typically 3. integration times for the array pixels such that a reasonably wide range of integration times is defined. An example set of integration times of 0.1, 1.6 and 25.6 msec yields a 256:1 range within the constraint of a 30 msec frame rate. To achieve these integration times typically requires a 25 MHz clock and a 64-bit data bus.

Row control words corresponding to the set of integration times are defined as follows. A row control memory 122 stores the row control word for each of the M columns of the M x N detector array 10 for each of r repetitions, where r is also the number of integration times. For the example set of integration times, each pixel has associated with it one of three entries, either [0,0,1] for the shortest integration time, [0,1,0] for the intermediate length integration time and [1,0,0] for the longest integration time. Row control memory 122 additionally stores the chosen integration time for use in .

Timing control 12G causes the column control 22 to cycle through r column cycles while indicating to row control memory 122 the appropriate row control word R to load into row J control register 110. It will be appreciated that it may be necessary to have a wait state between column cycles to achieve the requested integration times. Once the repetitions have finished, timing control 12Q sends the appropriate sample signal to the column control 24 to cause the array to be sampled.

It will be appreciated that all of the pixels of a given column are simultaneously sampled and that a gap between the time a given column is sampled and the time its lefthand neighbor is sampled is maintained in order to maintain uniformity of pixel integration times across detector array 10. For the example given above, the gap is 3 · 2 usee.

After sampling, the pixel intensities are read by the video readout control 16. The video output of readout 16 is amplified, in an amplifier 126, and digitized, in A/D converter 128, to produce V for use in an integration time update and ij extended dynamic range video output calculation operation. The algorithm utilized in the operation is disclosed in Israel Patent Application 87306, commonly owned by the common owners of the present application. Briefly, the algorithm defines the value for each pixel as that which was received when the pixel detector element 20 was operating in the optimal, linear range. The algorithm then multiplies (or shifts) each value by the inverse of the integration time which produced the value within the linear range of the pixel detector element 20. The pixel values are then combined into a single, expanded dynamic range picture.

Accordingly, a comparator 130 determines whether or not V is within the linear range of the detector element 20. If it ij is, the entry for pixel (i,j) stored in row control memory 122 remains unchanged. If V is out of the linear range, T is ij ij incremented or decremented to the next higher or lower integration time, depending on whether it was respectively below or above the linear range, by a Change T block 132 and the ij entry for pixel (i,j) is changed accordingly. If V is very far ij out of the linear range, the change in T may not be sufficient ij and therefore, T will need to be adjusted again at the next i frame. This will continue until V is within the linear range or ij until T has assumed an extreme value and cannot be adjusted any i further. Thus, there may be a lag of r-1 frames in adjusting the sensitivity to sudden temporal changes in scene brightness.

V is also input to a multiplier 13^ which is operative to adjust the video output signal for variations in pixel sensitivities, or integration times, over the detector array 10. Multiplier 13^ transforms V into V /T where T is ij ij ij ij calculated from the appropriate pixel entry in row control memory 122. For integration times which are multiples of 2, as in the example above, multiplier 13^ performs a shift operation by the appropriate number of bits. In the current embodiment, multiplier 13Ί typically produces a 16-bit digital video signal from the 8-bit video signal produced by detector array 10. Standard video display equipment is generally capable of displaying only analog signals of no more than 6-7 bits dynamic range. , invention also includes a dynamic range compressor 136. Compressor I36 typically comprises a FIFO serpentine buffer 138 typically of size M*(m-1) + m)*l6, three mxm convolvers l40, 142 and 1*14 , an adder 146 and a D/A converter 148 for producing the output analog video signal.

For typical 3x3 neighborhoods, FIFO buffer 138 typically stores 2*M + 3 pixels at any time and is tapped to read out a given 3x3 neighborhood around a given pixel until the entire array has passed through and been processed. Each of the 16-bit values of the given neighborhood is 'sliced' into three 8-bit values known as the high byte, the low byte and the middle byte. The middle byte is that which is produced by removing the four highest and four lowest bits.

The high bytes of the neighborhood are convolved together in convolver l40, the middle bytes are convolved by convolver 142 and the low bytes are convolved by convolver 144. he kernel for the three convolvers l40, 142 , 144 is as follows: -1 -1 -1 -1 8+eps -1 -1 -1 -1 where eps is a small-valued constant stored in block 150.

The outputs of the convolvers l40, 142 and 144 are summed and truncated by adder 146 to produce the 8-bit value which is passed to D/A 148 which, in turn, produces the analog value of the 8-bit signal.

It will be appreciated by those skilled in the art that the result of the operation described hereinabove is to stretch the inherent input dynamic range of the detector elements 20 in detector array 10 from a range of 7-8 bits to fully 16 bits while maintaining an output dynamic range that is compatible with common display and processing means. In an alternative embodiment, the 16-bit digital value read out of multiplier 13^ is furnished directly to a digital computer or other image processing means capable of processing the extended dynamic range signal. As mentioned hereinabove, the stretch is accomplished by choosing one of r possible integration times for each pixel.

It will further be appreciated that, with the exception of the detector array 10 and its column and row controllers 24 and 22, the necessary components are currently available.

Reference is now made to Fig. 8A which illustrates an alternative embodiment of the adaptive sensitivity control 12 of Fig. 7· In this embodiment, the analog video output is directly produced, thus eliminating the need to produce the 16-bit intensity values and to subsequently compress them. The integration time calculation is modified to increase the number of integration time intervals such that the integration time can be any value which is an integral multiple of the time required for a column cycle. For a minimum column cycle time of 0.16 msec and a frame rate of 30 msec, integration time may be varied over a 200:1 range. By restricting sampling time in 2x2 pixel neighborhoods to be identical over the neighborhood, a range of integration time of 800:1 can be attained.

Sensor chip 119 produces analog video output which is amplified by an amplifier 172 before output to display equipment. , elements of the adaptive sensitivity control apparatus 12 are external to sensor chip 119 and are comprised of generally digital circuits. The external integration time processing elements are typically comprised of an A/D converter 174 for converting the analog video output to digital intensities V for ij processing, an update integration time block 175 which produces optimal integration times T' given both the output intensity ij V and the integration time T used to produce that output, and ij ij a frame timing memory I78 of size M x N x 8 bits for storing the selected integration times as 8-bit values.

The update block 175 comprises a lookup table 176 , a multi-tap delay line 180 and an accumulator 182. Lookup table 176 chooses a preliminary new value, T new, for the integration time ij T . Many different functions of V and T can be used to ij ij ij generate the values, stored in lookup table 176 , which determine T new. In the preferred embodiment, the following logic is used: ij 1 . If V is at the saturation level, which is ij typically a value of 255 , then T new is set to T /2 ; otherwise, ij ij 2. T new = int( k *V *exp(-V )) ( 1 ) ij 1 ij ij where k is a normalization constant chosen to generate a smooth 1 spread of values of T over the range desired and the function ij int produces an integer value from a non-integer value.

In accordance with an alternative embodiment, T new is ij defined according to equation 2.

T new = k *(l-exp(lnE /E tau)) (2 ) ij ij ij where k is a normalization constant chosen to produce an E ij which is in a range which is convenient for computations, such as the range between 0 and 65,355. useful for 16-bit computations, and where E is defined in accordance with equation 3· ij E = k *V /T (3) ij 3 ij ij and where k and tau are normalization constants chosen to 3 produce an approximately logarithmic curve which stretches the low values and condenses the high values of T new. ij In order to locally enhance edges and to suppress artifacts, T new is modified by the average of the new ij integration times in its neighborhood. Thus, it is typically stored in multi-tap delay line 180 to wait for the neighborhood values to be produced. The neighborhood is then averaged by accumulator 182 to produce T' , the new value of the integration ij time for the next pass.

T' is stored as an 8-bit number in frame timing memory ij 178 for use in acquiring the next image. The appropriate row control words which will produce an integration time of T' at ij pixel (i,j) are generated by row control logic 184 as follows and are loaded into row control circuitry 22 as described hereinabove.

Row control logic 184 is shown generally in Fig. 8C. It generates the row word corresponding to column j at a current integration time T by reading the T values for that column n ij from memory 178 and comparing each of them, in a comparator 300. to T . A counter 302 provides the value T in units of the basic n n timing gap, which is 0.l6 msec in the example. If T is less ij than or equal to T , then address i in the current row word n receives a value of 1, indicating that pixel (i.j) will be . , receives a 0 value, indicating that pixel (i,j) will continue integrating charge until it is sampled. It will be appreciated that counter 302 is typically decremented each column cycle until a sampling cycle occurs; thus, T begins the integration cycle at n its largest value and ends at its smallest value, decrementing by one column cycle each time. Additionally, it will be appreciated that each pixel may be precharged at every column cycle until T n reaches its integration time value.

Timing control 12G causes the column control circuitry 24 to precharge each column once per column cycle.

It will be appreciated that the pixel-by-pixel update of the integration times T occurs independently of the ij precharge and sample operations of the sensor chip 119· The off-chip generation of row words is typically very fast; for example, to achieve a 0.16 msec minimal integration time, a row word of 512 bits must be generated every 300 nsec. A 64-bit bus operating at 2 MHz is capable of producing row words at such a speed.

An alternative embodiment of the adaptive sensitivity control 12 which directly produces analog video output is shown in Fig. 8B and differs from the embodiment illustrated in Fig. 8A in the update block 175 which produces the optimal new integration time value, T' . ij In accordance with this alternative embodiment, update block 175 comprises a first and a second lookup table, 190 and 192, respectively, and a multiplexer 194.

First lookup table 190 receives the output intensity V and the integration time T and chooses a value E which ij ij ij corresponds to the signals it receives. The values E are ij typically generated according to equation 2.

As in the previous embodiment, local edge enhancement and artifact suppression is provided by delay line 180 and accumulator 182 which together produce the neighborhood average E avg. ij Second lookup table 19 receives E avg and chooses a value for T new which corresponds to the input E avg. The ij ij values stored in second lookup table 192 are typically generated according to equation 4.

T new = k *(1 - exp(-lnE avg/E avg*tau)) (4) ij 4 ij ij where k and tau are normalization constants as described with 4 respect to equation 3 hereinabove.

If V is at the saturation level, then the value of ij T new computed in equation 4 is invalid and multiplexer 194 sets ij T* to T 12. Otherwise, multiplexer 1 4 sets T' to T new. ij ij ij ij It will be appreciated that the embodiment of Fig. 8A first determines T new and then performs the local edge ij enhancement and artifact suppression wheras the embodiment of Fig. 8B first determines the neighborhood average I avg and then ij uses it to compute the integration time T' . ij Reference is now made to Figs. 9A and 9B which illustrate two embodiments of the adaptive sensitivity control 12 for a system which produces color images. Three sensor arrays 200, 202 and 204, respectively, separately but synchronously generate the red (R), green (G) and blue (B) components of an image. The R, G and B components are individually digitized by , n respec ve y an e g a output is converted into the hue (H) , saturation (S) and intensity (I) color representation by color converter 212. It will be appreciated that the operations described above can also be performed on output from a single RGB color detector array.

In accordance with a first embodiment of the adaptive sensitivity control 12 for color, the intensity values I are ij treated exactly as the video output V is treated in the ij monochrome adaptive sensitivity embodiment of Fig. 7· I is ij multiplied (or shifted) by a multiplier 214, compressed by compressor 136 and then recombined, in a combiner 216, with the corresponding hue and saturation values, H and S ij ij respectively, to produce the corrected RGB signal for display.

The digital corrected RGB signal is subsequently converted to an analog signal by D/A converters 218, 220 and 222.

The intensity value I is also utilized for ij computation of the integration time T , as V was utilized in ij ij the monochrome embodiment depicted in Fig. 7 and described hereinabove. The operation is performed in update integration time block 22k which comprises comparator 130 (Fig. 7) and Change T block 132 (Fig. 7). The new integration time is identical for ij all three sensor arrays 200, 202 and 204 and is stored in row control memory 122. Thus, a single set of row control words is read out of row control memory 122 to be used for controlling the three sensor arrays 200, 202 and 204.

Alternatively, three separate control circuits can be employed, for each of the three sensor arrays 200, 202 and 20 , each similar to one described in Figs. 7 and 8.

A second embodiment of the adaptive sensitivity control for color, shown in Fig. 9B, is similar to that of the first color embodiment in that the intensity I is used to calculate ij the new integration times. However, update integration time block 22 of Fig. 9A is replaced by either of the update integration time blocks 175 of Figs. 8A and Fig. 8B and row control memory 122 is replaced by frame timing memory 178. Additionally, there is no 16-bit intensity value calculation nor the attendent compressor 136 since the analog video output is directly produced from sensor arrays 200, 202 and 204.

Reference is now made to Fig. 10 which illustrates three alternative image processing means useful for reducing the dynamic range of a wide range input video signal, V, without losing any details of the image. Fig. 10A comprises a small number, typically 3. of convolvers 310» 312 and 31^ each receiving a different consecutive portion of the bits of the original pixel word, and together covering all of the bits of the many bit pixel word. Three selectors 31 » 313 and 315 each produce one of the consecutive portions of the original many bit word. The convolvers typically use a 3*3 kernel, as described hereinabove in reference to Fig. 7· The results of the convolvers are added together in an adder 316 to produce a smaller dynamic range signal, v.

An alternative embodiment, shown in Fig. 10B, employs a circuit similar to that used in Fig. 8B to compute T new and ij comprises an averaging circuit 32 and a lookup table 324. The averaging circuit 322 is typically comprised of a delay line 320 ver g ne g intensity. Lookup table 3 4 implements any of the functions described hereinabove. The output of the lookup table 324 is multiplied, in a multiplier 326, with the extended dynamic range input signal V to produce the reduced dynamic range output signal v.

A further embodiment of the dynamic range compressor is illustrated in Fig. 1GC. The circuitry is similar to that shown in Fig. 10B with the exception that the lookup table 324 is placed before the averaging circuit 322.

It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove . Rather the scope of the present invention is defined only by the claims which follow:

Claims (28)

91095/2 C L A I M S

1. Video imaging apparatus comprising: a multiplicity of photosensitive detectors arranged in a staring array for receiving input radiation; and detector control means for controlling the response of individual ones of the photosensitive detectors such that they operate in different operative ranges within the total inherent dynamic range thereof, whereby the output of the staring array at any given time provides image information in respect of input radiation extending over an input radiation dynamic range-which is greater than the operative input radiation dynamic range of any given detector and wherein said detector control means comprises means responsive to radiation incident on said multiplicity of photosensitive detectors for governing the response of said individual ones of the photosensitive detectors.

2. Apparatus according to claim 1 and wherein said means for controlling the response of the photosensitive detectors is addressable simultaneously to a multiplicity of individual pixels or a multiplicity of designated groups of pixels in a field.

3. Apparatus according to claim 1 or 2 and wherein said staring array is a generally two-dimensional array.

4. Apparatus according to any of the preceding claims and wherein said detector control means includes means for selectably and variably determining the integration time of each of the multiplicity of detectors. 91095/2

5. Apparatus according to any of the preceding claims and wherein said staring array is embodied on a unitary integrated chip.

6. Apparatus according to any of the preceding claims and wherein said detector control means comprises means for providing two control signals to each of the multiplicity of detectors whereby each detector is actuated in response to simultaneous receipt or absence of a predetermined combination of control signals.

7. Apparatus according to any of the preceding claims - Jr ' - 5 and wherein said detector control means comprises a control'Circuit for each detector of the multiplicity of detectors.

8. Apparatus according to any of the preceding claims and wherein the outputs of the detectors are supplied to apparatus for applying thereto a neighborhood transform in order to preserve image contents notwithstanding the reduction in overall dynamic range produced by operation of the detectors with variable integration times.

9. Apparatus according to claim 1 and wherein said means responsive to radiation is operative to govern the response of at least one individual one of the photosensitive detectors in response to radiation incident thereon.

10. Apparatus according to claim 1 and wherein said means responsive to radiation is operative to govern the response of at least one individual one of the photosensitive detectors in response to radiation incident thereon and in the vicinity thereof. 3Ί 91095/2

11. Apparatus according to any of the preceding claims and wherein said detector control means includes means for storing a value representing the response of each of said individual ones of said photosensitive detectors and means for updating the stored values.

12. Apparatus according to any of the preceding claims 1 - 11 and wherein said detector control means is operative to cause said multiplicity of detectors to provide an output having a dynamic range less than the dynamic range of said input, radiation.

13. Apparatus according to any of the preceding claims 1 - 11 and wherein said detector control means is operative to cause said multiplicity of detectors to provide an output having a selectable dynamic range.

14. Apparatus according to any of the preceding claims and also comprising image processing means receiving the outputs of the multiplicity of photosensitive detectors and control signals supplied to said multiplicity of photosensitive detectors and providing in response thereto modified outputs of the detectors corresponding to a uniform sensitivity.

15. Apparatus according to claim 14 and wherein said image processing means is operative to cause said multiplicity of detectors to provide an output having a dynamic range generally equal to the dynamic range of said input radiation. 91095/2

16. Apparatus according to claim 14 and wherein said image processing means is operative to cause said multiplicity of detectors to provide an output having a selectable dynamic range.

17. Apparatus according to any of the preceding claims 14 - 16 and wherein said image processing means receives control signals representing integration times of said multiplicity of detectors and provides in response thereto modified^ outputs of the ' detectors corresponding to a uniform sensitivity.

18. Apparatus according to any of the preceding claims and wherein said multiplicity of photosensitive detectors includes a plurality of detectors, each sensitive to a different color and wherein said detector control means is operative to control the response of said plurality of detectors to a plurality of colors.

19. Apparatus according to claim 18 and wherein said detector control means includes means responsive to achromatic intensity at a given image location for determining the response of said plurality of detectors at said image location.

20. Apparatus according to claim 19 and wherein said detector control means includes means responsive to achromatic intensity at a given image location for providing a single control output governing the response of said plurality of detectors at said image location. 91095/2

21. Apparatus according to any of the preceding claims and wherein each of said multiplicity of detectors comprises: a plurality of photosensitive detectors; column and row control means together determining the sample timing of said plurality of photosensitive detectors; and means for reading out a video signal from said plurality of photosensitive detectors.

22. Apparatus according to any of the preceding claims and wherein each of said multiplicity of detectors comprises: a plurality of photosensitive detectors; column and row control means together determining the precharge timing of said plurality of photosensitive detectors; and means for reading out a video signal from said plurality of photosensitive detectors.

23. Apparatus according to claim 22 and wherein said column and row control means include means for providing multiple precharges of at least some of said plurality of photosensitive detectors in order to prevent accumulation of excessive charge thereat. 91095/2

24. Image processing apparatus comprising: means for receiving, a video signal having a first dynamic range and for providing an output video signal having a second dynamic range narrower than said first dynamic range, while preserving edge information and removing some relatively spatially unvarying information.

25. Apparatus according to claim 24 and wherein said means for receiving and providing comprises at least two neighborhood Operator means and summation means ' for combining the outputs of said at least two neighborhood operator means.

26. Apparatus according to claim 24 and wherein said means for receiving and providing comprises at least one neighborhood operator and at least one look up table which together receive said input video signal and provide a multiplication factor which is multiplied by the input video signal to provide said output video signal.

27. Apparatus substantially as shown and described hereinabove.

28. Apparatus substantially as illustrated in any of the drawings. For the Applicant,