Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/14—Picture signal circuitry for video frequency region
  • H04N5/21—Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
  • H04N25/67—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response
  • H04N25/671—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response for non-uniformity detection or correction
  • H04N25/677—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response for non-uniformity detection or correction for reducing the column or line fixed pattern noise
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/70—Denoising; Smoothing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
  • H04N25/63—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to dark current
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20004—Adaptive image processing
  • G06T2207/20008—Globally adaptive
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20076—Probabilistic image processing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
  • H04N25/67—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to fixed-pattern noise, e.g. non-uniformity of response

Definitions

• This description relates to reducing noise in digital images.
• dark current noise represents random noise levels that are produced by respective pixels of the CMOS sensor array whether or not light is being received by the sensor.
• dark current noise represents random noise levels that are produced by respective pixels of the CMOS sensor array whether or not light is being received by the sensor.
• digital image to refer to the array of pixel values that make up the image.
• the pixel's dark current level does, however, depend on the temperature of the pixel.
• the pixel's dark current causes charge to build up over time, so that the effect of dark current on a pixel value depends on the duration of exposure of the pixel.
• vertical patterns also called fixed pattern noise
• offset also called fixed pattern noise
• shot noise Other artifacts in the images include vertical patterns (also called fixed pattern noise), offset, and shot noise.
• Vertical patterns are due to unintended differences in the operations of the respective readout circuits of different columns of the array and generally do not change over time.
• Offset represents differences in overall signal level (brightness) from image to image that result from variations in certain electrical properties of the readout circuitry.
• Each pixel value generated by the sensor array includes random shot noise with variance proportional to the signal value.
• a target digital image is received from an image sensor.
• the image is contaminated by noise of unknown magnitude that is represented by a reference digital image.
• a process is applied that uses statistical analysis of the target digital image and of the reference digital image to estimate a magnitude of the noise for at least some pixels of the target digital image.
• the sensor is a CMOS sensor or a CCD sensor.
• the noise comprises dark current noise.
• the process estimates the dark current magnitude for every pixel of the target digital image.
• the process comprises program instructions.
• the statistical analysis includes a de- correlation analysis with respect to the target digital image and the reference digital image.
• the dark current magnitude estimates are produced without requiring information about a temperature of the sensor or a duration of exposure.
• the reference digital image is based on a dark current digital image that is substantially free of vertical patterns and has been generated from a grey digital image and a black digital image acquired respectively using different exposure periods.
• the reference digital image is based on a corrected dark current digital image that has been processed to reduce the effect of low- frequency spatial trends across the pixels of the CMOS sensor.
• the reference digital image is based on a de-correlation of the black digital image with the de-trended dark current image.
• the process subtracts vertical patterns, pixel by pixel, from the target digital image to produce a vertical pattern corrected digital image.
• the process applies a dark current removal function to the vertical pattern corrected digital image to produce a dark current corrected digital image.
• the noise in every pixel of the target digital image is reduced using the estimated dark current levels.
• the process applies an offset estimation and subtraction function to the dark current corrected digital image to remove offset.
• the noise reduced target digital image is provided to a processor for use in analyzing features of an image captured by the CMOS sensor.
• the target digital image includes possibly malignant lesions.
• Figure 1 is an image of dark current.
• Figure 2 is an image of vertical patterns.
• FIGS 3, 4 A, and 4B are schematic flow diagrams.
• Figures 5, 6, and 7 are images at stages of the calibration process.
• Figures 8 and 9 are graphs.
• an input digital image 10 (which we call T, for target) generated by a CMOS sensor array 12 in response to light 14 received through optics 20 from a target scene 16 (for example, skin with a pigmented lesion 18) can be processed (after temporary storage in storage 38) by noise reduction software 22 (run by a processor 24) to produce an output digital image 26 (which we call O) for use in quantitative analysis 28 (for example, to determine 30 whether the lesion is a malignant melanoma, using the MelaFind® melanoma detection product of Electro-Optical Sciences, Inc., of Irvington, NY).
• the noise reduction process is applicable broadly to any digital image produced by a any image sensor array for any purpose and in any context, including any in which the noise-reduced digital image may be subjected to later analysis for which noise in the digital image would be problematic.
• processing may also be required with respect to digital images produced by sensors in various applications, including processing to correct optical effects associated with a specific lens and illuminator.
• the noise reduction techniques described here thus have applications not limited by any optical correction or optical correction of a particular kind.
• reference information is acquired and processed.
• multiple independent sets of a grey digital image 34 are acquired in the dark at a known or unknown sensor temperature and stored in storage 38.
• the exposure time is in an intermediate range to avoid saturation from long exposures and yet have a reliable measurement of dark current.
• Figure 5 shows an example of a grey image acquired in the dark (intensities multiplied by 15 for display purposes.)
• the multiple grey digital images are averaged (39) to produce an average grey digital image 40 (which we call G) in which the level of shot noise at the individual pixels is reduced.
• multiple independent sets of a black digital image are acquired in the dark, if fixed pattern noise such as vertical patterns is present.
• the exposure time for the black sets is shorter than for grey sets (for example, as short an exposure time as the hardware permits) to have a reliable measurement of the fixed pattern noise and to minimize the dark current level in the black sets.
• the multiple black images are averaged (39) to produce an average black digital image 44 (which we call B) in which the effect of shot noise at the individual pixels is reduced, as it was for the average grey digital image.
• the average black digital image B is subtracted (46) from the average gray digital image G to produce a dark current digital image 48 (D).
• the subtraction of the black digital image from the gray digital image produces an image of pure dark current (free of vertical patterns).
• Figure 6 shows the image of figure 5 after subtraction of the vertical patterns (intensities multiplied by 15 for purposes of display).
• a de-trending function is applied (50) to the dark current digital image 48 to remove low-frequency spatial trends from the pixels of the data 48, because the trend in the dark current digital image can be correlated with the target digital image.
• a pure vertical pattern digital image V 56 is generated by first applying a dark current removal function (54) to B.
• the removal function in general returns a digital image that represents the difference, pixel by pixel, between (i) an input digital image B and (ii) a product of the dark current digital image D times a factor Al
• V B - A1* D
• the de-correlation function determines a factor Al that de-correlates the digital image B from the digital image S over some region of the image.
• the de-correlation determines the magnitude (Al) of dark current D in the black image B.
• the de-correlation function is an example of a statistical analysis that enables the dark current noise to be determined from target pixels and from reference pixels without the need to know the temperature of the sensor or the period of exposure. Other statistical approaches could also be used, such as a variance minimization analysis.
• the result of step 54 in the figure is the pure vertical pattern digital image V 56.
• Figure 7 shows the image of figure 6 after subtraction of dark current noise (intensities multiplied by 15 and 100 levels were added to each pixel for display purposes).
• Figure 8 provides a graphical illustration of cross sections of the images of figures 5, 6, and 7 without intensity adjustments.
• VRem is the intensity after subtraction of fixed pattern noise
• DCRem is the intensity after subtraction of fixed pattern noise and of dark current
• OffRem is the intensity after subtraction of fixed pattern noise, of dark current, and of offset.
• shot noise is about 7-8 levels, and it is not removed by the calibration process. Reduction of the shot noise would require either spatial or temporal averaging of images.
• the steps described above need only be performed once, e.g., during factory calibration, and the resulting calibration digital images can be stored and used for a large number of target images over a long period of time. It is not necessary to develop the calibration image information again each time a target image is captured.
• the pure vertical pattern digital image V is subtracted (58), pixel by pixel, from T to yield a vertical pattern corrected digital image Tl 60:
• the magnitude A2 of dark current D in that region of the image Tl is determined from
• T2 T1 - A2 * D .
• the CMOS sensor can be arranged to have a black region (for example, in a corner or along one of the edges) of the array which is screened from light (including any light from a target).
• the digital image from the black region can be used to correct for offset in the image.
• the black region digital image is first processed by the vertical pattern removal and dark current removal steps described earlier and the resulting processed black region data are averaged 66 to produce an average black region value NB 68.
• the average black region value is subtracted 70 from every pixel of the dark current corrected digital image T2 of an image to eliminate offset from the target image.
• the resulting offset corrected digital image T3 72 can be subjected to additional processing depending on the circumstances.
• a calibration digital image W may be acquired by imaging a uniform white target with a known diffuse reflectance. This white target digital image is then subjected to a series of operations that include vertical pattern removal (subtracting V, if applicable), determination of the magnitude of dark current by applying de-correlation to W - V and S, dark current removal, and offset removal, to produce Wl.
• a reflectance calibration 78 may be applied to the digital image T3 to produce a reflectance digital image T4 80 by the following computation (performed pixel by pixel):
• T4 (T3/W) * (E(W)/E(T3)) * p
• E is the exposure time
• p is reflectance of a white calibration target.
• the reflectance calibrations 78 removes from the digital images T3 non-uniformities imparted by the imaging system.
• the process described above can be applied to monochromatic digital images provided by a sensor.
• the process can be applied to multiple digital images in different spectral ranges that are produced simultaneously by the sensor (e.g., red, green, and blue — RGB).
• the digital images in different spectral ranges may be processed independently as described above.
• the process may take advantage of a statistical analysis to reduce the need, for some sensors, to control the temperature or duration of exposure as a way to reduce the effects of dark current noise.
• the techniques described here may be useful not only for sensors operating in the visible and infrared ranges but also for x-rays and possibly ultrasound, that is, for any sensors for which removal of dark current noise or effects similar to dark current noise would be useful.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Transforming Light Signals Into Electric Signals (AREA)
• Studio Devices (AREA)
• Image Processing (AREA)
• Measuring And Recording Apparatus For Diagnosis (AREA)
• Ultra Sonic Daignosis Equipment (AREA)
• Image Input (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (2)

Family

ID=39029242

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (44)

Family Cites Families (73)

• 2006
  • 2006-08-07 US US11/500,197 patent/US7813586B2/en active Active
• 2007
  • 2007-08-07 JP JP2009523947A patent/JP5198449B2/ja not_active Expired - Fee Related
  • 2007-08-07 CA CA2660337A patent/CA2660337C/en not_active Expired - Fee Related
  • 2007-08-07 EP EP07840735A patent/EP2054843A4/en not_active Withdrawn
  • 2007-08-07 WO PCT/US2007/075333 patent/WO2008019358A2/en not_active Ceased
  • 2007-08-07 AU AU2007281748A patent/AU2007281748B2/en not_active Ceased
• 2010
  • 2010-10-06 US US12/898,802 patent/US8160386B2/en active Active - Reinstated
• 2012
  • 2012-03-01 US US13/409,397 patent/US8630508B2/en not_active Expired - Fee Related

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events