WO1984003018A1

WO1984003018A1 - Method and apparatus for processing signals from a solid-state image sensor

Info

Publication number: WO1984003018A1
Application number: PCT/US1984/000072
Authority: WO
Inventors: Lee Nevil Davy
Original assignee: Eastman Kodak Co
Priority date: 1983-01-24
Filing date: 1984-01-20
Publication date: 1984-08-02
Also published as: JPS60500236A; EP0132277A1; US4473845A; EP0132277A4

Abstract

The output signals from a solid state image sensor (10) having defective photosites causing corresponding defects in the output signals are processed by correcting the defect-related portions of the signals, and image-enhancing (16, 18, 20, 38) only those portions of the signals not in the neighbourhood (as defined herein) of the defect-related portions. Those portions in the neighbourhood of defect-related portions are not subjected to image-enhancement.

Description

METHOD AND APPARATUS FOR PROCESSING SIGNALS ' FROM A SOLID-STATE IMAGE SENSOR

TECHNICAL FIELD The present invention relates to signal processing methods and apparatus for processing the output signals from a solid state image sensor, for example, a charge coupled device (CCD) image sensor. More particularly, it relates to a signal processing method and apparatus for performing defect correction and image-enhancement on such output signals.

BACKGROUND ART The use of solid state image sensors in television cameras, electronic still cameras, and film scanners for producing TV signals from pictures on film is well known. It is also known to be desirable to image-enhance the output signals produced by such image sensors. Well known tech¬ niques for image-enhancement include (a) crispening, sometimes called "unsharp masking", to accentuate the appearance of fine detail, and (b) noise reduction, for example "coring", to suppress the appearance of film grain in film scanners.

In the manufacture of solid state image sensors, such as CCD image sensors, many devices are produced having relatively few defects, but rela¬ tively few devices are produced having no defects. A "defect," in this sense, refers to a defective light-sensitive element (photosite) in the image sensor, i.e., one which exhibits a response to incident light which differs from the nominal response of the other elements in the sensor. It is well known that if a relatively few defects in such devices can be accepted, the manufacturing yield of useful devices increases dramatically. It has been found, however, that when the output signal of a sensor having a few tolerable defects is subjected to image-enhancement, such as crispening, the appearance of the defect becomes enhanced, thereby rendering the image unacceptable.

Several approaches have been suggested for correcting defects in the output signals of image sensors as produced by defective photosites. In one such approach, the output signal of the image sensor is sampled by a samp e-and-hold circuit. When a defect is present in the output signal, the sample- and-hold circuit is not clocked, thereby replacing the defective portion of the signal with an adjacent non-defective portion. For an example of this type of defect correction, see U.S. Patent No. 4,167,754 issued September 11, 1979 to Nagumo et al. It is also known to correct defects in the output signals of image sensors by interpolating between adjacent non-defective portions of the signal to compute a replacement value for the defective portion. (Note, it will be understood by those skilled in the art that no matter what means are employed to "correct" the signal produced by a defective photosite, such "defect-corrected" signal is always somewhat inferior or different from the nominal signal produced by the non-defective photosites. It would appear to be desirable to implement defect correction prior to image-enhancement of the output signal of a solid state image sensor to avoid enhancement of the defects in the signal. Unexpectedly, however, it has be^'en found that under some .circumstances, for example in the scanning of color negative film to produce a TV signal, the appearance of the image-enhanced, "defect-corrected" signal is even more objectionable than the appearance of an enhanced signal to which no defect correction has been applied. DISCLOSURE OF INVENTION The technical problem addressed by this invention is to provide a method and apparatus for processing the output of a solid-state image sensor having defective photosites in such a manner that such defective photosites are virtually non- detectable in an image-enhanced image produced by the output of such apparatus.

According to the invention, it has been found that solid-state image sensors having defective photosites can be used to produce acceptable image- enhanced images if the sensor output signal is processed in a special way, viz., (a) the defect- related portions of the sensor output signal are modified to correct for the defect (e.g. by inter¬ polation) ; (b) those portions of the sensor output signal in the "neighborhood" of the defect-related portions of the sensor output signal are not sub¬ jected to image-enhancement; and (c) the remaining portions of the sensor output signal are image- enhanced. As used herein, the "neighborhood of a signal portion" refers to the signal portion itself and those other portions of a signal which ordinarily would be employed in a particular procedure to image- enhance such signal portion.

The signal processing apparatus of the invention, in addition to comprising means for modifying the output signal of a defective photosite to produce a defect-corrected signal, and means for selectively enhancing (or leaving non-enhanced) the output signal of any photosite of the sensor to produce either enhanced or non-enhanced signals, also includes means for selecting, for each portion of the sensor output signal corresponding to a discrete photosite, either (a) a defect-corrected signal, (b) an image-enhanced signal, or (c) a nominal (non- enhanced) signal. Such selection means is effective to select a defect-corrected signal for each defect- related signal portion, a non-enhanced signal for all signal portions in the neighborhood of a defect- related signal portion, and an enhanced signal for all other signal portions.

In one embodiment of the invention, the sensor output signal is supplied to a finite impulse response (FIR) filter that produces a defect cor- rected signal, an image-enhanced signal, and an appropriately delayed non-enhanced output signal that matches the time base of the other signals. A programmed memory, containing information regarding which of the output signals from the FIR filter to select at any given time, is addressed in synchronism with the readout of the sensor. The information from the memory controls an array of switches that select the proper signal from the finite impulse response filter. In an alternative embodiment of the sensor invention, the output signal is sampled by a sample- and-hold circuit. Defects are corrected by inhibit¬ ing the sampling of the sample-and-hold circuit for the portion of the signal containing the defect, thereby replacing the defective portion with an adjacent non-defective portion of the signal. The image is enhanced by forming a "blurred" signal comprising a running average of the signal in a two dimensiona-1 neighborhood surrounding the portion of the signal being processed. The "blurred" signal is subtracted from an appropriately delayed version of the output signal to yield a detail signal. The detail signal is "cored" to remove high^'frequency, low amplitude information therefrom. The "cored" detail signal is boosted and added back to the

"blurred" signal to form the image-enhanced signal. In the neighborhood of a defect, the addition of the "cored" detail signal is inhibited, and only the "blurred" signal is output.

BRIEF DESCRIPTION OF THE DRAWINGS The invention will be described with reference to the drawings, wherein:

FIG. 1 is a schematic diagram of an exemplary embodiment of signal processing apparatus according to the present invention; and FIG. 2 is a schematic diagram of an alternative embodiment of a signal processing apparatus according to the present invention. MODES FOR CARRYING OUT THE INVENTION

Referring now to FIG. 1, an exemplary signal processing circuit for implementing the signal processing method according to the present invention is shown. A solid state image sensor 10, such as a frame transfer or interline transfer type CCD image sensor, is supplied with clocking signals from a clock generator 12 to produce an output signal on line 14. Such clocking signals serve to sequentially electrically address the individual photosites of the image sensor, each photosite corresponding to a picture element (pixel) of an image projected on the sensor. The photosites of image sensor 10 are commonly arranged in an rectangular array of hori¬ zontal rows and vertical columns, and for the sake of simplifying the explanation of this embodiment, it is assumed that one or more entire columns of photosites are defective. A finite impulse response (FIR) filter 15 comprising a pair of one-picture element

(pixel) delays 16 and 18, and three amplifiers 20, 22, and 24, receives the output signal and produces one of three different output signals for each photosite of the sensor on lines 26, 28 and 30, as discussed below. An image-enhanced signal from each photosite is produced on line 26 by a differential amplifier 20 which subtracts a weighted sum of the undelayed, and two pixel delayed, versions of the output signal from an amplified version of the one pixel delayed output signal. This arrangement represents one example of a family of well-known "crispening" filters employed to enhance the appearance of fine detail. For the sake of simplicity, a filter is shown which accentuates detail only in one dimension, e.g., vertical detail. The non-enhanced (i.e. nominal) output signal of each photosite is produced on line 28, delayed by one pixel to match the time base of the other output signals. A defect-corrected signal is produced by amplifier 24 by taking a weighted sum of the undelayed and the two-pixeled delayed signals, to produce an interpolated pixel value on line 30. The weighting coefficients used for the undelayed, one pixel delayed, and two pixel delayed signals to produce the image-enhanced, defect- corrected, and non-enhanced signals are shown in Table 1 below.

T A B L E 1 Weighting Coefficients

1 Pixel 2 Pi Lxxeell

Undelayed Delayed Delayed Output Signal

-.2 1.4 -.2 image-enhanced

.5 0 .5 defect-corrected 0 0 1 1..00 0 0 non-enhanced

The three signals on lines 26, 28, and 30 are supplied to an output amplifier 32 through FET switches 34, 36, and 38, respectively. The respec¬ tive gates of the FET switches are controlled to open or close the switches by a programable memory 40, programmed to contain information regarding the desired state of the gates of the FET transistor switches for each pixel (i.e. photosite) in a line of output. The programable memory 40 is addressed by a counter 42 that is clocked in synchronism with the clocking of the image sensor. The counter is recycled for each line of output from the image sensor. Memory 40 contains as many storage locations as there are vertical columns in the image sensor 10, e.g. approximately 500. The method of programming the memory based on the detection of defective locations in the image sensor will be apparent to one skilled in the art. Two bits per column are suffi¬ cient for encoding the required information regarding the state of FET switches 34, 36, and 38. The operation of the signal processing apparatus will now be described. Suppose for example that the tenth column of the image sensor is defec¬ tive. This information is stored, for example, in the tenth memory location of memory 40. The memory 40 contains in memory locations 1-8 (which correspond to columns 1-8) , the information to close switch 38 while switches 34 and 36 remain open, thereby supply¬ ing the image-enhanced signal of columns 1-8 to the output amplifier 32. Memory locations 9 and 11 contain information to close switch 36 while switches 34 and 38 remain open, thereby preventing image-enhancement in the neighborhood of a defect; i.e., providing the nominal signals of columns 9 and 11. Memory location 10 contains information to close switch 34, leaving open switches 36 and 38, thereby providing the non-enhanced, defect-corrected signal of column 10 to amplifier 32. Pictures produced by the apparatus of the present invention show enhance¬ ment of the image while suppressing the appearance of defects. The present invention is easily extended to filters having longer delays, for crispening in both the horizontal and vertical direction. Also, the invention is easily extended to the use of alterna- tive defect correction schemes, and further image-enhancement processes. FIG. 2 shows another version of the signal processing apparatus of the present invention. This apparatus is particularly adapted to process the output signals of an image sensor employed to produce a positive image from a negative film image. The signal processing apparatus employs an alternative scheme for defect correction, and image-enhancement.

Again, this example will be described with reference to the correction of column defects in the sensor, but it will be obvious to one of skill in the art, that the signal processing apparatus of the invention is easily extended to the correction of individual photosite defects. An image sensor 100, for example of the frame transfer or interline-transfer type, receives clock signals from a clock generator 102. The output signal from the image sensor is sampled and held by a sample-and-hold (S/H) circuit 104, which likewise receives sample signals from the clock generator 102 through an FET switch 103. The sample-and-hold signal is supplied to an averaging filter 106 that averages the signal over a 3 x 3 pixel area. This averaging is accomplished in a well-known manner by providing a pair of one-line delays and a plurality of single pixel delays configured to simultaneously deliver the output signals representing a 3 x 3 pixel area. A weighted average of the signals is then taken to produce a "blurred" signal on line 108. The "blurred" signal is subtracted in a differential amplifier 110 from an appropriately delayed version (provided via delay 111) of the output signal to produce a "detail" signal on line 112. The detail signal is cored in a conventional coring circuit 114 to remove the high-frequency low-amplitude informa- tion therefrom, thereby suppressing the appearance of film grain without totally eliminating fine picture detail. The cored detail signal is boosted in an amplifier 116, and supplied, through an FET switch 118 to one input of a summing amplifier 120. The blurred signal is supplied to another input of the summing amplifier. The output of summing amplifier 120 is the processed signal.

The gates of FET switches 103 and 118 are controlled by the outputs of a programable read-only memory 122. The control signal to FET switch 118 is shown delayed by a delay element 123 to compensate for the delay caused by averaging filter 106. It will be apparent to one skilled in the art that alternatively, the delay can be accounted for when programming memory 122, thereby obviating the need for delay element 123. The memory is addressed by a counter 124 that is clocked in synchronism with the output of the image sensor, as in the previously described example. The operation of the signal processing circuit will now be described. Assuming, again, a defect in the 10th column of the image sensor, the first eight locations in the memory contain informa¬ tion to close both switches 103 and 118, thereby causing the processing circuit to produce an image- enhanced output signal. The 9th and 11th locations in the memory contain information for opening FET switch 118 while leaving FET switch 103 closed, thereby providing a non-enhanced signal (the "blurred" signal) in the neighborhood of a defect. The 10th location in the memory contains information

fijREAi OMPI

^ to open both FET switches 103 and 118, thereby providing a defect corrected, non-enhanced signal. Note, when switch 103 is open, the signal from the previous column (i.e. column 9) is applied to the averaging circuit by the sample-and-hold circuit; thus, the defect associated with the 10th column is corrected by substituting the 9th column signal for it. This substituted signal is then averaged with the photosites in the neighborhood of the defective photosite in the 10th column.

The signal processing apparatus of the present invention has been described with reference to a monochrome image sensor; however, it will be apparent to those skilled in the art that the apparatus is readily extended to color image sensors, either by providing a full signal processing circuit for each color channel, or by providing the signal processing circuit for a luminance or green channel only. INDUSTRIAL APPLICABILITY

The invention described above is particu¬ larly useful in producing high quality images from an imagewise exposed solid-state image sensor having defective photosites.

IPO

Claims

CLAIMS ;

1. Apparatus for processing the output signal from a solid-state image sensor to produce an image-enhanced signal, said apparatus comprising means (12) for sequentially clocking-out the output of each of a plurality of photosites which comprise the image sensor to produce an output signal having nominal signal portions representing the output of discrete, non-defective photosites, and at least one defect-related portion representing the output of a defective photosite, said apparatus comprising means (16, 18, 24; 103, 104, 106, 122) for modifying said defect-related portion to produce a defect-corrected signal, and means (16, 18, 20; 106, 110, 111, 114, 116, 120) for selectively enhancing the nominal signal produced by any non-defective photosite to produce an image-enhanced signal, characterized in that said apparatus comprises control means (12, 42, 40; 102, 124, 122) for selecting (a) a defect- corrected signal for each defect-related signal portion, (b) a nominal signal for all signal portions in the neighborhood of a defect-related signal portion, and (c) an enhanced signal portion for all other signal portions.

2. The invention claimed in Claim 1 wherein said modifying means comprises a finite impulse response filter (15) for effective interpolation between portions of the image-enhanced output signal.

3. The invention claimed in Claim 1, wherein said modifying means for producing a defect- corrected signal comprises means (102, 103, 104) for replacing a defective signal portion with a nominal signal portion produced by a non-defective photosite which has been previously clocked-out.

4. The invention claimed in Claim 1, wherein said means for producing an image-enhanced

O PI_ signal comprises crispening filter means (106) for producing a blurred signal representing a running average over a neighborhood of signal portions, means (110) for subtracting the blurred signal from the output signal to produce a detail signal, and means (116, 120) for boosting the detail signal and adding it to the blurred signal to produce an image-enhanced signal.

5. The invention claimed in Claim 4, further comprising a coring filter (114) responsive to the detail signal to remove high frequency low amplitude information therefrom to produce a cored detail signal, said cored detail signal being added back to said blurred signal to produce said image-enhanced signal.

6. The invention claimed in Claim 4, wherein said nominal signal comprises said blurred signal.

7. The invention claimed in Claim 1, wherein said control means comprises a programmed memory containing information representing the signal processing required for each picture element of the sensor, means for addressing said memory in synchro¬ nism with the readout of the image sensor, and switch means responsive to the output of said memory for selecting the appropriate output signal for each picture element.

8. A method of processing the output signal from a solid state image sensor having defec- tive photosites causing corresponding defects in the output signal thereof, said method comprising the step of correcting those portions of the signal having defects; and being characterized by the step of image-enhancing only those portions of the signal outside the neighborhood of such corrected defects,

?*,r leaving the signal non-enhanced in the neighborhood of corrected defects.

9. The invention claimed in Claim 8, wherein said correcting step comprises replacing a defective portion of the signal with a neighboring non-defective portion.

10. The invention claimed in Claim 8, wherein said correcting step comprises replacing a defective portion of the signal with an interpolated average of neighboring non-defective portions.

11. The invention claimed in Claim 10, wherein said image-enhancing step comprises generat¬ ing a blurred signal representing a running average over a neighborhood of portions of said signal, subtracting said running average from said signal to yield a detail signal, amplifying said detail signal, and adding said detail signal to said blurred signal to produce an image-enhanced signal.

12. The invention claimed in Claim 11, wherein said enhancing step further comprises the step of coring said detail signal to remove high frequency, low amplitude information therefrom prior to adding said detail signal to said blurred signal.

13. The invention claimed in Claim 11, wherein said non-enhanced signal comprises said output signal.

14. The invention claimed in Claim 11, wherein said non-enhanced signal comprises said blurred signal.