WO2012116862A1

WO2012116862A1 - Automatic hot pixel recognition and correction for digital cameras and other pixel detectors

Info

Publication number: WO2012116862A1
Application number: PCT/EP2012/051295
Authority: WO
Inventors: Bernard DELLEY
Original assignee: Paul Scherrer Institut
Priority date: 2011-03-03
Filing date: 2012-01-27
Publication date: 2012-09-07

Abstract

A method and system for automatic bad pixel recognition and correction in image pixel sensors is provided. The process of identifying hot pixels requires few resources. The required resources can be met even by today's low price consumer point-and-shoot cameras and camera's built into phones. Hot pixels are identified via a complementary cumulative distribution function (CDF) for the raw pixel digital numbers (DN) of a black frame. This black frame hot pixel recording step is done once in a while like a sensor cleaning. Simple analysis of the tail of the CDF allow to define a sharply discriminating threshold for hot pixels. Subsequent analysis of the black frame for pixels with above threshold DN yields a table or map with hot pixel positions. In normal imaging mode, the table is used to replace the hot pixel DN by a value from a pixel nearby, thereby repairing the bright pixel blemish which is present in the unprocessed image data.

Description

Automatic hot pixel recognition and correction for digital cameras and other pixel detectors

This invention relates to image processing, and more specifically to automatic hot pixel recognition and correction by a procedure completely implemented in the camera.

Background Hot pixels record high brightness irrespective of the true image content. In an unprocessed image, hot pixels show up as bright spots where there should be no bright spots. Typically, this occurs in single, isolated pixels. It may also occur in clusters of pixels. Usually, the apparent high brightness is generated by an unwanted high value of the dark current for such a pixel.

Because of the dark current issue, long exposure, high sensitivity and higher temperature are all favouring the appearance of hot pixels in the image data. A hot pixel can saturate its DN at moderate exposure times, depending on the level of dark current. Sometimes hot pixels are also called 'white flaws' or in a less specific meaning 'bad pixels'. Pixels which show high DN at short exposure, irrespective of image content, can be called 'stuck pixels' . Hot pixels can result from manufacturing faults. Hot pixels can be arise after manufacture during shipment, travel, storage and normal use of the camera. One ubiquitous cause for the appearance of new hot pixels are high energy cosmic rays leading to radiation damage in the pixel sensor. As the damage can occur deep in the sensor, such a hot pixel tends to stay there for the remaining life span of the sensor. Cosmic ray intensity is elevated by about two orders of magnitude at the altitude of intercontinental flights, resulting in a correspondingly higher damage rate.

Therefore, damage of the pixel sensor by cosmic rays leading to some hot pixels is to be expected over the lifetime of the sensor, especially with air travel. The appearance of hot pixels cannot be avoided completely by improved design and manufacture. There is great variety of hot pixel behaviour thermally and over time, depending on the details and location of the sub-microscopic physical defect. Some hot pixels may get annealed away at

operating temperature after some time. As the physical damage affects particular pixels, the bright spots appear always in the same places. The annoying hot pixels cannot be prevented

completely by improved manufacturing or improved design.

The persistent hot pixels can be removed by a post-processing of the image data, preferably while the pixel data still has the full resolution of the sensor and before muddling the data in . jpg finalization for example.

Some basic methods with a bearing on hot pixels exist which may also be used in combinations:

Ml) An existing method (prior art) involves recording or mapping of the hot pixels beforehand and replacement of the hot pixel data by plausible data in normal operation. This existing method is deficient at the hot pixel recognition stage. A predefined

threshold for the digital number (DN) of pixels can be used in a controlled black frame (BF) exposure at the service laboratory to identify and map hot pixels. Automatic hot pixel correction for each image is available in many current camera models but only with hot pixels identified in the service lab. Mainly because of the temperature dependence of noise and hot pixels, identification and mapping of hot pixels is deemed not robust enough for all in camera use. However, several Olympus camera models have a built in 'pixel mapping' procedure. It is said to remove dead and stuck pixels. Unfortunately, information on hot pixels removal is inconsistent .

M2 ) Another existing method comprises an inspection of the image data and replacing pixel outlier values by plausible data from nearby pixels. A typical implementation in a 2d pixel array involves clipping the pixel DN to the maximum of its 8 neighbours (of the same colour in a colour sensor) . An improved version of this method uses in addition the 8 nearest neighbours of different colour, in a Bayer colour sensor, for the clipping. The clipping method is ineffective for clusters of hot pixels in that the brightness of the brightest pixel is only cut down to the second brightest value of the cluster. Further, the clipping method is a corruptive noise reduction method which destroys local information at a significant percentage of the pixels. Because of the

sometimes significant alteration of image content method M2 is highly to be objected when it is not under user control. M3) A BF, or synonymously a dark frame, is subtracted from the image data (prior art) . This compensates pixels individually for their dark current. Because of the temperature dependence, such a BF is typically taken for each image, in which case a pause equaling the exposure time results from BF exposure.

M4 ) Noise reduction methods involving averaging or binning nearby pixels are not specifically directed, nor really suitable, for hot pixel elimination. FIG 1 depicts a generic pixel detector layout with associated devices as used in the present invention. Image data 100 is passed to the photo sensitive cells 102 under the control of a shutter 101. In radiographic imaging, the shutter may be implemented via the power control of the radiation source. The pixel sensor cells 102 convert the light into analogue electrical values that are digitized in the A/D converter 103 into a digital number (DN) associated with each pixel. This data set is further treated as an original raw digital image 110. The A/D conversion involves a multiplier proportional to the ISO speed number, which is defined by the International Standards Organization. An image processing unit 105 passes the image date on to the output unit 106, either unchanged or modified by procedures Ml, M2, M3 or M4. For some operations, it uses a memory 104. A control unit 107 coordinates the workings of the components 101 to 106.

The invention relates to a method for finding a sharp threshold criterion for identifying hot pixels while in field use. The threshold is automatically found by probing the statistics of the normal good pixels occurring in a BF taken in the field at the user's initiative. Thus, a sharp threshold can be precisely

defined for the actual sensor temperature. The BF provides up to date information about hot pixel locations. For a camera under typical operation conditions, a list of hot pixel locations is automatically set up. The expected shortness of the list leads to efficient and computationally expedient corrections for each image.

The invention includes a fast and simple generation of a

complementary cumulative distribution function (CDF) histogram.

The procedure involves a simple method to calculate the DN

threshold beyond which the probability of falsely classifying a normal good pixel as a hot pixel falls below one across the entire sensor. The hot pixels are identified by their DN which is above that threshold. Since the hot pixel identification does not depend on the specific neighbourhood of the pixel, also clusters of hot pixels can be sharply identified. The hot pixel corrections are done in a way which is insensitive to clusters of hot pixels involving up to 4 neighbour pixels.

In another aspect, the invention provides a digital camera system with a capacity of being freed from hot pixel blemishes. When new hot pixels appear, perhaps as a consequence from cosmic rays, the user can invoke the hot pixel recording (or mapping) procedure in the field in loose analogy to the sensor cleaning available from the menus of many camera models.

Figure 1 is a block diagram of an exemplary digital camera system according to the present invention.

Figure 2 is a schematic graph of a complementary cumulative

distribution function histogram (CDF) .

Figure 3 is a flow chart illustrating the process according to the present invention.

TABLE 1 shows some values of the GCDF function used for the

definition of the hot pixel threshold. The foregoing description of Fig. 1 contains a hardware overview of the preferred embodiment. The implementation of the new process can possibly be emulated in the image processing unit 105 and its associated memory 104. Alternatively, the additional hardware can be implemented as ASIC and use memory for the sensor related hot pixel data. Intermediate, short time storage for black frames may be alternatively be provided by the flash card usually present in digital camera systems. Before going into a discussion of

procedure and flow chart, it is useful to discuss the statistics in DN and the CDF, which is central to the present invention. The image data of a BF consisting of a DN for each pixel can be

summarized in a histogram. The histogram shows the number of pixels that occur for each possible DN in a particular image. In a BF image, there are many pixels with very low DN and a decreasing number of pixels with higher DN. A few hot pixels may show very high DN. Many cameras can show a histogram on the display, albeit this histogram is usually originating downstream in the process flow from the compression and finalization unit. The complementary cumulative histogram function shows the number of pixels with a DN exceeding the strike DN plotted on the abscissa.

Fig. 2 shows a log-log plot of an exemplary CDF histogram for illustration. It should be recalled that the process, subject of this invention, does not require evaluating logarithms or

exponentials. The CDF of Fig. 2 arises from a hypothetical 16M pixel sensor with most pixels belonging to a Gaussian distribution of DN with RMS=7. A 1% sub-population of more noisy pixels was assumed with RMS=18. Finally, a small subpopulation of 100 hot pixels was assumed, with a probability distribution involving a symptomatic algebraic tail. The formally generated CDF is rounded to nearest integer pixel numbers as is the result of calculating the CDF from the histogram of raw DN. This is the full line in Fig. 2 with the characteristic stair step shape at the hot pixel end. Estimates for the variability of the CDF over a sequence of BF which is based on Poisson statistics, are indicated by the short dashed lines. Notably, the variability quickly becomes small, once the CDF exceeds a value in the tens. The two black dots have precalculated CDF strike values to be chosen depending on the sensor model. The DN values where the CDF falls below these strike values are expedient for finding the end of the Gaussian derived CDF (GCDF) . The GCDF through the two dots is indicated by the dotted line. This is systematically an accurate fit to the tail of the sub-population of the more noisy Gaussian pixels.

Most of the low noise pixels do not contribute at such

intermediately high DN. The fit treats them as if they were centered around a lower black level DN. The dotted line hits

GCDF=1 at a DN below 100 in the example of Fig. 2. The likelihood that any good pixel from the Gaussian populations produces a DN beyond that threshold value falls below one. This is the natural threshold which is indicated in Fig. 2 with the dashed vertical line. Pixels with greater DN are identified as hot pixels. The number of the discernible hot pixels is about 10 in this example. For the hypothetical conditions here, most of the hot pixels did not surface above the noise. For BF with longer exposure time, the DN of the Gaussian pixels scales in proportion to the square root of time while the dark current type hot pixels have DN proportional to exposure time. Temperature dependence is more complex. General expectation is a doubling of DN for hot pixels for 10 degrees Celsius (or Kelvin) increase in temperature. Two consequences for hot pixel recording are apparent. An exposure time near the upper end of application should be used for best recognition of the relevant hot pixels, preferably with a hot pixel recording under possibly relatively high temperature conditions in the field, it is again possible to recognise and correct hot pixels of relevance.

The process of hot pixel treatment involves two modes. In the first mode of hot pixel recording, a black frame is taken. As controlled by 107, the shutter remains closed. Fig. 3 shows a flow chart of the process in a preferred embodiment. The hot pixel recording mode starts with the analog image of the BF captured 201 and A/D converted 202. The branch for HP recording is taken 203 and the BF is stored 204. The histogram h(i) is calculated 205 by counting the number of occurrences for each DN across all pixels of the sensor.

Formally one may write h(i) = ∑_krl 5{i,DN_krl} , where k, 1 denote the pixel coordinates and the Kronecker 5{i,DN_k,i} symbol takes the value of 1 if i equals DN_k,i (the DN of pixel k,l) and 0 otherwise. The next step 206 is the calculation of the CDF. The CDF is calculated from the histogram starting at the maximum DN where CDF is set equal to the histogram value there which is possibly zero. The other values of the CDF c(i) are calculated by downward recursion adding the current histogram value to the upper previous CDF value to get current CDF. Formally this recursion is c (i) = c (i+1) + h(i)

For finding a sharp threshold 207, two fixed strike values for the ordinate of the CDF are used to probe the good pixels belonging to a Gaussian distribution. The strike values would be set

permanently for a sensor model, perhaps subject to BIOS revisions. The strike values sa and sb of Fig. 2 are marked by asterix in Table 1. The corresponding DN da and db are the DN values i where the CDF c(i) first falls below the strike value. This search across the CDF is done in step 207 and followed by an

extrapolation to the threshold point marked t in Table 1.

The threshold DN dt is calculated as dt = da + (db - da) * f where the factor f is f = (xt - xa) / (xb - xa) with xa = 3.25 , xb =3.75 and xt = 5.50 in the example. Note that the factor f is fixed once the strike values, here sa =9232 and sb =1415 have been chosen. With the threshold determined for the actual circumstances in the field, the search locations of DN exceeding the threshold in step 208 for the BF stored at step 204 and the corresponding list stored in step 209. In step 208, for the BF stored in step 204, the pixels are located which have an DN exceeding dt and its x,y coordinates are stored in step 209.

This completes the steps in HP recording or mapping procedure. A note: Fig. 2 was plotted implying a black level equal to zero, a convenient choice for the log-log figure. However, the method just described works just the same with any other black level.

In standard image capture mode, the shutter 101 is open and the branch of no BF is taken at step 203. Usually, no BF subtraction is done so step 210 branches to skip the BF subtraction in step 211. The step 212 involves replacing the DN of the listed hot pixels by plausible values extracted from the 8 neighbouring pixels of the same colour in a Bayer pixel array. In a monochrome sensor, simply the 8 direct neighbour pixels are considered. A preferred implementation may do a ranking of the 8 DN numbers and choose the 4th lowest to substitute for the hot pixel value (the 4th lowest is the highest of the lower half of all values, a variant of the usual definition of the median value) . This choice of median is insensitive to the presence of up to 4 neighbouring hot pixels. Thus, up to clusters of 5 neighbouring pixels are removed. For photographic cameras a relatively small number of hot pixels is expected, maybe none, perhaps up to hundreds as compared to several M pixels total. As this number is small, the procedure 212 is fast. The procedure 212 may be applied down to such short exposure times that even the hottest pixels do not significantly exceed the general image noise. If the procedure 212 is used for even the shortest exposure times also stuck on high DN pixels are consistently removed. Use of procedure 212 may not be put under user control, thus hiding all except newly created hot pixels. The absence of user control is in line with current service center managed bad pixel mapping. It is accepted by users as only a few wrong DN values are corrected, less than 0.01% or all pixels. Noise clipping 214, a powerful noise reduction method, should be made user controllable at step 213 as it alters image content in about 1 in 9 pixels (1 in 17 for the enhanced method) . The final step 215 may involve formatting of the image date and compression.

Variants

1) Double BF in hot pixel recording step 204. This can help to better define the Gaussian distributed good pixels by applying a BF - BF subtraction in step 211 before continuing with step 205. This is of interest if hot pixel recording is done with long exposures of the BF, for example exposure time greater than 1 s.

2) Double BF in hot pixel recording step 204. Given two BF, step 208 can be made to only admit coincident hot pixels. This is hot pixel locations where a DN above threshold is found in both BF. This is of interest if hot pixel recording is done with long exposures of the BF. With long exposures the probability of recording a mini flash arising from a non-damaging cosmic ray is proportional to exposure time. Such recorded flashes will be rejected by the coincidence procedure.

3) A bad pixel map may be used if a proportion of hot pixels in the percent range is expected for example in radiographic image capture. The bit map with a one to one correspondence of map bits and all sensor pixels becomes more efficient at a few percent of hot pixel incidence. For camera and sensor applications with less than thousands of hot pixels in multi M pixels,

a list recording of x,y coordinates of hot pixels is more

efficient.

4) Color separated evaluation of CDF and thresholds. This variant is suggested when color pixels differ by sensor geometry and not only by the color filter in the optical path.

5) The hot pixel recording and correction may be applied to binned pixels which may be relevant in video mode for potentially reduced computations in the video stream. However, hot pixel correction before binning tends to yield more accurate image information.

6) Optional BF subtraction in steps 210/211 in long exposure noise reduction can be placed in the work flow after the hot pixel correction step 212. The original and this variant help to avoid the appearance of dark pixels as a result from BF subtraction of saturated hot pixels.

7) Several hot pixel lists may be used: a short list determined with a short BF exposure time for hand held and video application. A longer list for long exposures, which is determined by a long BF exposure, to eliminate all the hot pixels relevant for long exposures .

8) Contrary to hot pixels, dead (or dark) pixels, register too low DN. However, dead pixels appear to be much less of an issue than hot pixels. Detection can be done with a variant of the present method using a uniformly illuminated frame instead of the BF. The complementary cumulative distribution function histogram is to be replaced by the direct cumulative distribution function histogram for dead pixels.

9) The method can be used in a computer analyzing the un-altered raw BF and image data in post-processing. Instead of un-altered, also compressed, but pixel resolved raw images and BF data are admissible .

Claims

1. A method for hot pixel recognition in image sensors,

identifying hot pixels based on the statistics of the vast

majority of good normal pixels.

2. The method according to claim 1 wherein a correction of clustered hot pixels is achieved.

3. The method according to claim 1 or 2 wherein an automated hot pixel recognition under variable field conditions is achieved.

4. The method according to any of the preceding claims wherein for a fast correction of hot pixels the very short list of hot pixels to be corrected for video exposure times is be compatible with the video data flow.

5. The method according to any of the preceding claims wherein healed hot pixels are allowed to come back to use again.