KR101559724B1 - Method and Apparatus for Detecting the Bad Pixels in Sensor Array and Concealing the Error - Google Patents

Abstract

The present invention relates to an image sensor device comprising such a sensor array adapted for correction and detection of defective pixels and detection and correction of defective pixels in a sensor array. Importantly, image sensors provided to convert optical images to electrical signals will provide better quality output with the help of such possible pixel error detection and correction methods. The method and apparatus enable the provision of a better quality camera in certain digital cameras including CCD image sensors and CMOS sensors. It monitors the status of the sensor array and extends its lifetime without additional hardware costs. The present invention targets all kinds of defective pixels, including dead pixels, stuck pixels, hot pixels, cold pixels, and the like, and is therefore adapted to provide a complete solution to the technique of dealing with image damage to a large number of bad pixels.

Description

[0001] The present invention relates to a method and an apparatus for detecting defective pixels in a sensor array and concealing an error,

The present invention relates to a method of detecting defective pixels in a sensor array, a sensor array and correcting the detected defective pixels by inpainting and also includes such a sensor array adapted for identification and correction of defective pixels To an image sensor device. Importantly, image sensors provided to convert optical images to electrical signals will provide better quality output with the help of detection and correction of possible pixel errors. Preferably, the present invention will make it possible to provide better quality cameras in certain digital cameras, including CCD image sensors or CMOS sensors that perform the task of capturing and converting light into electrical signals. The present invention aims at eliminating defective pixels generated in an image sensor as in a digital camera. The method of the present invention has a significant economic advantage in the production and use of sensors since it assists in monitoring the status of the sensor array without additional hardware cost and also increases its usable life. Importantly, the present invention aims at all possible various bad pixels, including dead pixels, stuck pixels, hot pixels, and cold pixels, It is also possible to provide a complete solution to the problem of dealing with image degradation for a large number of defective pixels including various defective pixels, thereby improving the accurate detection of the defective sensor, confirming how the image sensor has been damaged, Will enhance the possible improvement of the image obtained by painting.

Furthermore, when the concealment of errors due to in-painting is not applied after the detection of a bad pixel, the generated error pixel map can be used as a signature of the particular camera to confirm its authenticity with any image captured using the camera .

It is well known that sensors must have a dynamic range as a whole, but almost always have pixels that are not. Most of these pixels can also be corrected with flat filed correction parameters implemented in the camera. However, pixels that are not properly corrected by flat field correction are considered bad pixels. These dead pixels or hot pixels, normally fixed dark or bright, are corrected by a defective pixel correction routine. Such routines correct defective pixels with interpolated values based on neighboring pixels.

A typical calibration method involves the average of neighbors. If multiple pixels in a row are bad, correction may be made over multiple pixels. Boundary conditions also exist to handle bad pixels at the edges of the image array. Bad pixels occur in the image sensor of a digital camera. These pixels do not properly sense light. These defective pixels have many variations, such as dead pixels, stuck pixels, hot pixels, cold pixels, etc. (Philip Hurq, Robust Detection of Stars in CCD Images, Journal of Experimental Astronomy, Springer Holland, 9 (4), 251 -259, December 199). Dead pixels always read zero for all exposures. The stuck pixel always reads the maximum value for all exposures. The dead pixel and the stuck pixel are complementary to each other. Hot pixels read higher at longer exposures. Obviously the stuck pixel is an extreme case of a hot pixel. Similarly, some pixels are close to zero, but not exactly zero. Such pixels are referred to herein as cold pixels. In all cases, the pixels report these values regardless of the original scene. The dead pixel carries charge across the liquid crystal material so no light can pass through it. However, the stuck pixel allows all light to pass because the transistor does not work properly. (See Ube Ebert, Ubetsu Serpel, Klaus Barden Dick, Strategy for Replacing Radiation Films - A Comparison of Film and Digital Detectors, IEEE Transactions on Instrumentation and Measurement, 47 (1), February 1998, 17th World Conference on, Shanghai, China, October 25-28, 2008)

Poor or defective pixels are often transistors that do not work permanently and appear as black dots or as white or colored pixels attached. Occasionally, it is possible to repair a bad pixel by rubbing the screen with a soft cloth or by running a pixel repair computer program in that area for a few hours, the latter repeatedly blinking to remove the stuck pixels. Dead pixels are much less likely to correct themselves or repair in some way over time. There are several ways to fix a stuck pixel. These effects usually occur with exposure and, above all, with the temperature of the camera.

Usually defective pixels look like small random points throughout the image. Some cameras have noise reduction settings that can remove them.

A common mistake in testing for bad pixels is to cover the lens with a lens cap and set the camera to auto. When set to Auto, the camera will slow down the shutter speed, resulting in longer exposure. This creates red, green, and sometimes white pixels. This is normal and is called Christmas tree artifact. To test for bad pixels, use a photo editor such as Photoshop or Nikon Editor to compare two shots so that you can take several shots in a normal indoor environment, such as a portrait, and compare two images side-by-side. It is common to do. When bright pixels occur in one image, it is important to compare them with other images, both of which must have bright pixels at the same location. If it is in the same position it can be regarded as a hot pixel. In addition, the dead pixel is always dead and the pixel is black, so this test does not appear. Stuck pixels and dead pixels are generally not considered problematic for photographers.

Modern DSLRs such as the Canon 5D MK2 have built-in bad pixel remapping capabilities. Most RAW processing software (such as Lightroom or VibraWrap) has the ability to automatically fix bad pixels. However, when a photo is converted to JPEG, the strong discontinuity of the hot pixel becomes a bit more complicated because it causes JPEG compression artifacts in the surrounding pixels.

U.S. Patent No. 5,047,863 discloses a non-operational pixel detection system employing a frame buffer that is responsively coupled to an imaging device for storing oligochromatic data of dark pixel data from image locations of the imaging device. Correction device. A register, responsive to the imaging device and having an output coupled to the buffer, sequentially clocks the digital values of the image pixel data of the imaging device into the frame buffer. A comparator operatively coupled to the register and responsively coupled to the dark pixel data of the frame buffer produces an enable output and an inhibit output. If any selected dark pixel data is less than a threshold, then the data stored in the register corresponding to that pixel is stored in the frame buffer in response to the enable signal indicating a pixel element operating in the imaging device. When the value of the dark pixel data in the frame buffer is greater than the threshold, the comparator generates an inverse signal. The value of the image data of the previous active pixel element enters the register and frame buffer accordingly.

U.S. Patent No. 5,499,114 is a digital image scanning device that complements pixel data for bad photosites where pixel data from a bad photosite of a linear imaging device is excluded and the next available good photo site Is replaced by the pixel data from the pixel data. The loss of the block of pixel data at the beginning of the scan line is supplemented at the end of the scan line by causing the memory to continue writing the end pixel data the number of times necessary to complete the entire scan line data.

U.S. Patent No. 6,618,084 relates to a pixel correction system and method for a CMOS image imager, which provides a fault tolerant radiation imager, such as a CMOS imager. Such image sensors include circuitry for masking and / or correcting defective pixels during image generation.

U.S. Patent No. 7,522,200 relates to an on-chip dead pixel correction in a CMOS image sensor and discloses a dead pixel correction method for a CMOS image sensor.

However, from the above discussion all such pixel correction methods are complex and require accurate and simple detection and correction of defective pixels due to in-painting required for techniques for on-line monitoring of the state of the sensor, It does not work.

The basic objective of the present invention is therefore to detect defective pixels and carry out inpainting in a simple manner, which is necessary to improve on-line monitoring of image quality and sensor status by image sensors of a digital camera and to increase the usable life of the sensors And to develop a faulty pixel detection and error correction method for a sensor array to be provided.

It is a further object of the present invention to provide a method and apparatus for reducing the number of dead pixels, stuck pixels, etc., to be adapted to enhance utilization of image sensors or similar devices provided for converting optical images, such as those used in digital cameras and other imaging devices, A hot pixel, and a cold pixel.

It is a further object of the present invention to develop advanced techniques for dealing with image defects due to defective pixels in sensor arrays or a large number of defective pixels and similar devices.

Still another object of the present invention is to develop defective pixel detection and correction techniques such that the probability of defective pixel detection errors can be reduced to zero by using a plurality of images according to the providing technique of the present invention.

It is still another object of the present invention to provide an inpainting method for pixel error correction that enables application of an independent inpainting process to a defective pixel detection method and a defective pixel detection method.

Another object of the present invention is to provide a method for detecting defective pixels on-line or off-line according to the needs of an application, wherein on-line detection will reduce the correction delay and off-line detection will be adapted to reduce complexity and power cost.

It is a further object of the present invention to provide defective pixel detection and inpainting to reduce the impact of defective pixels and thereby reduce the reject rate at the time of production and increase the effective lifetime of the sensor array.

Still another object of the present invention is directed to a method of detecting defective pixels that is adapted for reduced false and omnidirectional detection, including independent acquisition counts, and further reduced by ROI selection in the image.

It is a further object of the present invention to provide a sensor array that will facilitate accurate detection of defective pixels for the better output and performance of sensor arrays and similar imaging devices and the required inpainting thereof.

It is another object of the present invention to provide a method for determining how much an image sensor is damaged and the possibility of improving the quality of an image through inpainting.

It is yet another object of the present invention to develop a defective pixel calibration method that includes a robust design to achieve a high PSNR.

It is still another object of the present invention to authenticate images from a camera having a bad pixel, wherein when an inpainting is not applied to a bad pixel, an error pixel map generated using the detected bad pixel is compared with the signature of the particular camera can be used as a signature.

A basic aspect of the present invention is therefore a method for detecting defective pixels in a sensor array and concealing the errors, the method comprising the step of using local statistical statistics of a digital image including a value indicating the location of a defective pixel and a normal pixel, And generating an error map or a noise map that includes detection of bad pixel position information of the sensor based on the sensor.

Another aspect of the invention relates to a method for detecting defective pixels in a sensor array, the method comprising: providing a noise map including detection of defective pixel position information of an intensity-based sensor using local statistics of a digital image, To determine whether the defect is bad or not, and thereby further evaluate the position of the defective pixel so determined to produce a final noise map having a value of 1 or 0 indicating the location of the defective pixel and the normal pixel, respectively, And generating the error map or the noise map.

Another aspect of the present invention is directed to a method for detecting defective pixels in a sensor array comprising the step of authenticating an image captured by a camera by comparing the error map or noise map with an image from the camera.

Preferably, the method for defective pixel detection of a sensor array according to the present invention comprises the generation of an error map for several subsequent images and a combination thereof for further confirming error detection.

Yet another aspect of the present invention relates to the above method for detecting defective pixels in a sensor array and concealing the errors

(i) generating an error map or a noise map including detection of bad pixel position information of the intensity-based sensor using local statistics of the digital image, each including a value indicating the location of bad pixels and normal pixels;

(ii) replacing the bad pixels with a median value (intensity) of surrounding normal pixels within a specific range of the detected bad pixel, wherein the noise map Performing an adaptive inpainting process to correct only the defective pixels detected in the defective pixels;

.

Yet another aspect of the present invention relates to the above method for detecting defective pixels in a sensor array and concealing the errors

(i) providing a noise map comprising detection of bad pixel position information of the intensity-based sensor using local statistics of the digital image, verifying whether the pixel is bad or not, Generating a noise map comprising further evaluating a location of a bad pixel so determined for its local statistics to produce a final noise map having a value of 1 or 0 indicating a location;

(ii) replacing defective pixels with median values (intensity) of neighboring normal pixels within a specific range of the detected defective pixels, wherein the final noise And a fitting process of correcting only the defective pixels detected in the map.

According to one aspect of the present invention, in the above method for detecting a defective pixel in a sensor array and concealing the error, the defective pixel includes a dead pixel, a stuck pixel, a hot pixel and a cold pixel.

According to another aspect of the present invention, the step of performing the detection of the dead pixel includes the following steps (I) and (II).

(I) centering the same pixel on the current pixel to impose a W ₁ x W ₂ window on each of the noise-free images and to determine a maximum value (S _max ) and a minimum value (S _min ) in the window, Similarly, a first-stage noise map r is generated.

Where S _{i , j} is the intensity value of the pixel at position (i, j) and S _min is the global minimum intensity value of the image and r (i, j) = 1 in this first stage noise map This means that the corresponding pixel in the noise image is probably the dead pixel, and r (i, j) = 0 means that the corresponding pixel in the noise image is normal.

(II) Generate a final noise map r that includes

Level 1) In the image with noise, around the dead pixel w ₁ x w ₂ We impose a window, where w ₁ , w ₂ are much smaller than the previous window size.

If the number of normal pixels such as having a value of 0 in the first stage noise map is found more than w _min , the distance d between the normal pixels and the central dead pixel is calculated and goes to the level 3. If the number of normal pixels is not greater than w _min , go to level 2. For simulation, d is assumed to be MAD (mean of absolute distance).

Level 2) w _i = w _i + p _i (i = 1, 2, pi is a small positive integer) and goes to level 1 to increase the window to a predetermined fixed size.

Level 3) If the corresponding d is less than the threshold δ (δ is a small positive value), replace the corresponding pixel in the temporary noise map r with 0, otherwise leave it as the next dead pixel and go to level 1.

According to another aspect of the present invention directed to the above method, the performance of the stuck pixel detection step includes:

(I) centering the same pixel on the current pixel to impose a W ₁ x W ₂ window on each of the noise-free images and to determine a maximum value (S _max ) and a minimum value (S _min ) in the window, Similarly, a first-stage noise map r is generated.

Where S _{i , j} is the intensity value of the pixel at position (i, j) and S _max is the global maximum intensity value of the image and r (i, j) = 1 in this first stage noise map This means that the corresponding pixel in a noisy image is probably a stuck pixel, and r (i, j) = 0 means that the corresponding pixel in a noisey image is normal.

(II) Generate a final noise map r that includes

Level 1) In the image with noise, around the stuck pixel w ₁ x w ₂ We impose a window, where w ₁ , w ₂ are much smaller than the previous window size. If the number of normal pixels such as having a value of 0 in the first stage noise map is found more than w _min , the distance d between the normal pixels and the central stuck pixel is calculated and goes to the level 3. If the number of normal pixels is not greater than w _min , go to level 2. Here d is assumed to be MAD.

Level 2) w _i = w _i + p _i (i = 1, 2, pi is a small positive integer) and goes to level 1 to increase the window to a predetermined fixed size.

Level 3) If the corresponding d is less than the threshold δ (δ is a small positive value), replace the corresponding pixel in the temporary noise map r with 0, otherwise leave it as the next stuck pixel and go to level 1. The noise map r by this is the final noise map.

Yet another aspect of the invention directed to the above method is the execution of the hot pixel and cold pixel detection steps, including:

(I) centering the same pixel on the current pixel to impose a W ₁ x W ₂ window on each of the noise-free images and to determine a maximum value (S _max ) and a minimum value (S _min ) in the window, Similarly, a first-stage noise map r is generated.

Where S _{i , j} is the intensity value of the pixel at position (i, j) and S _max and S _min are the global maximum and minimum intensity values, respectively. In this first stage noise map, r (i, j) = 1 means that the corresponding pixel in a noise image is probably a hot pixel or a cold pixel and r (i, j) = 0 means that the noise- The corresponding pixels in the image are normal.

(II) Generate a final noise map r that includes

Level 1) around the hot (cold) pixels in the image with noise w ₁ x w ₂ We impose a window, where w ₁ , w ₂ are much smaller than the previous window size. If the number of normal pixels is found more than w _min , the distance d between the normal pixels and the central hot (cold) pixel is calculated and goes to level 3. If the number of normal pixels is not greater than w _min , go to level 2. Here d is assumed to be MAD.

Level 2) w _i = w _i + p _i (i = 1, 2, pi is a small positive integer) and goes to level 1 to increase the window to a predetermined fixed size.

Level 3) Preferably, the minimum distance between the center hot (cold) pixels from both ends is obtained as T ₃ = min (S _ij , 255-S _ij ) if the image is 8 bits. If the corresponding d is less than the threshold qT3, then the corresponding pixel in the temporary noise map r is replaced by 0, otherwise it is moved to the next hot (cold) pixel and goes to level 1. Where q is a constant for generating the final noise map r (0 < q < 1).

Preferably, the method of the present invention is adapted to be applicable to any bit depth.

Yet another aspect of the present invention relates to correcting pixel errors by inpainting in the method and includes the following steps.

(1) impose a w3 x w4 window around a bad pixel in a noisy map to find a set S with all such image pixels with noise under the current window with a corresponding value of 0 in a noisy map, If it is not empty, go to step 2, otherwise go to step 3.

(2) Replace the current pixel with the median of the set S, go to the next defective pixel, and go to step 1.

(3) Let w _i = w _i + Δw _i (i = 3,4, Δw i is a small positive integer) and go to step 1 to increase the window to a predetermined maximum size.

Importantly, in the above method according to the present invention, the defective pixel detection procedure and consequently the defective pixel painting is adapted so as to obtain a high PSNR for an image which has been corrupted by a large number of defective pixels.

In the above method, the defective pixel detection is performed on-line or off-line, and once detected, in-painting can be applied on-line or off-line according to application requirements.

According to another aspect of the present invention, there is provided a sensor array adapted for detecting defective pixels and performing inpainting of defective pixels, including the following means.

(i) intensity-based sensor bad pixel location information detection using local statistics of the digital image with the values indicating the locations of bad pixels and normal pixels, respectively, as means adapted for noise map generation.

(ii) replacing defective pixels with median (intensity) values of normal pixels within a particular window around the detected defective pixel by means of performing a painting that is a fit that corrects only the defective pixels detected in the noise map .

Importantly, in the image sensor system the sensor array is adapted to detect various bad pixels selected from dead pixels, stuck pixels, cold pixels and hot pixels according to any of the methods mentioned above.

Another aspect of the present invention directed to the image sensor system is that it is adapted to convert signals including optical / thermal, thermal, microwave, ultrasonic, X-ray, etc. into electrical signals.

In the following, the details, objects and advantages of the invention are set forth with reference to the following figures in connection with non-limiting examples.

Figure 1 is a block diagram of a sensor based imaging system in accordance with the present invention, showing poor pixel detection and inpainting steps.
In Figure 2, the first row shows a damaged &quot; chemical plant &quot; image. The image was damaged by random dead lines and / or random dead pixels. The second row shows the corresponding image restored from above by the proposed formula.
In Figure 3, the first row shows the damaged &quot; Utrecht &quot; image. The image has been damaged by random deadlines and / or random dead pixels. The second row shows the corresponding image restored from above by the proposed formula.
In Figure 4, the first row shows a damaged &quot; outdoor &quot; image. The image has been damaged by random deadlines and / or random dead pixels. The second row shows the corresponding image restored from above by the proposed formula.
In Figure 5, the first row shows a damaged &quot; bridge &quot; image. The image has been damaged by random deadlines and / or random dead pixels. The second row shows the corresponding image restored from above by the proposed formula.
In Figure 6, the first row shows a damaged &quot; chemical plant &quot; image. The image was damaged by random stuck lines and / or random stuck pixels. The second row shows the corresponding image restored from above by the proposed formula.
In Figure 7, the first row shows the corrupted &quot; Utrecht &quot; image. The image was damaged by random stuck lines and / or random stuck pixels. The second row shows the corresponding image restored from above by the proposed formula.
In Figure 8, the first row shows a damaged &quot; outdoor &quot; image. The image was damaged by random stuck lines and / or random stuck pixels. The second row shows the corresponding image restored from above by the proposed formula.
In Figure 9, the first row shows the corrupted &quot; bridge &quot; image. The image was damaged by random stuck lines and / or random stuck pixels. The second row shows the corresponding image restored from above by the proposed formula.
In Figure 10, the first row shows the damaged images. The image was damaged by random hot pixels and cold pixels. The second row shows the corresponding image restored from above by the proposed formula.
11 (a) is an original 16-bit image with a hot pixel. Two crop regions with hot pixels appear in the red square region. (b) is an enlarged view of the cropping region 1 and (c) is an enlarged view of the cropping region 2.
In FIG. 12 (a), the cropping area corresponding to the output of the image shown in FIG. 11 is represented by red squares. (b) is an enlarged view of the corresponding crop area 1, and (c) is an enlarged view of the crop area 2. Fig.

As discussed above, the present invention includes fully automated dead pixels, stuck pixels, hot pixels, and cold pixel detection and inpainting methods. This method will help to extend the operating life of the sensors in the sensor array. Basically, the proposed detection method includes two steps. In the first step of the detection method, the local characteristics of the image with noises are used. The second stage is done in a different way for better performance. The proposed detection formula can deal with image damage up to a high level of noise. In addition, after the detection of bad pixels, a painting protocol (Bad Pixel Fit) follows for the recovery of the damaged image. The proposed method has real-time adaptability for real-time performance, which greatly reduces hardware costs.

Importantly, the proposed detection method for bad pixel detection involves two steps. In the first step, all the pixels are checked to prepare a first-stage noise map, which holds the location information of the defective pixels. This step uses the local statistics of the image. In the second step, only the pixels marked bad in the first step are checked using local statistics to ascertain whether they are bad or not. In the second step, the first-stage noise map is modified to obtain the final noise map accordingly. In the final noise map, the values of 1 and 0 indicate the positions of the defective pixel and the normal pixel, respectively.

1 schematically illustrating a block diagram of a sensor imaging system according to the present invention shows the steps of detecting and painting inferior pixels in a sensor array.

The steps of the proposed dead pixel detection method are as follows .

For illustrative purposes, images are assumed to be of 8-bit depth, but this is by no means a limitation of this method and may be applied to any bit depth.

Stage 1. For each pixel in the image with noise, centered on the current pixel W ₁ x W ₂ We impose a window and find the maximum value (S _max ) and the minimum value (S _min ) in the window. At this stage, the pixels are marked as very conservatively normal. The first-stage noise map r is prepared using equation (1).

(1)

(One)

Where S _{i , j} is the intensity value of the pixel at position (i, j) and S _min is the global minimum intensity value of the image. In the first stage noise map, r (i, j) = 1 means that the corresponding pixel in the noise image is probably the dead pixel and r (i, j) = 0 means that the corresponding pixel This is normal.

Step 2. The following operation is performed for each dead pixel of the image (having a corresponding 1 value in the first stage noise map).

Level 1) We apply w ₁ x w ₂ windows around the dead pixels in the noise image, where w ₁ , w ₂ are much smaller than the previous window size.

If more pixels than the normal pixel (w _min ) are found in the first stage noise map when the corresponding value is 0, the distance d between the normal pixels and the central dead pixel is calculated and goes to the level 3. If the number of normal pixels is not greater than w _min , go to level 2. For simulation, d is assumed to be MAD (mean of absolute distance).

Level 2) w _i = w _i + p _i (i = 1, 2, pi is a small positive integer). The window is increased to a predetermined fixed size.

Level 3) If the corresponding d is less than the threshold δ (δ is a small positive value), replace the corresponding pixel in the temporary noise map r with 0, otherwise leave it as the next dead pixel and go to level 1.

This noise map is now called the final noise map. This final noise map has values of 0 and 1, and 0 and 1 represent the corresponding normal pixel and dead pixel, respectively.

The proposed staple pixel detection procedure is as follows.

Step 1. For each pixel of the noisy image, we apply the W ₁ x W ₂ window centered on the current pixel and find the maximum (S _max ) and minimum (S _min ) values in that window. Use Equation (2) to prepare the first stage noise map r.

(2)

(2)

Where S _{i , j} is the intensity value of the pixel at position (i, j) and S _max is the global maximum intensity value of the image. In the first stage noise map, r (i, j) = 1 means that the corresponding pixel in the noise image is probably the stuck pixel and r (i, j) = 0 means that the corresponding pixel This is normal.

Step 2. Perform the following operations on each stuck pixel of the image.

Level 1) In the image with noise, around the stuck pixel w ₁ x w ₂ We impose a window, where w ₁ , w ₂ are much smaller than the previous window size. If the number of normal pixels is found more than w _min , the distance d between the normal pixels and the center stuck pixel is calculated and goes to level 3. If the number of normal pixels is not greater than w _min , go to level 2. Here d is assumed to be MAD.

Level 2) w _i = w _i + p _i (i = 1, 2, pi is a small positive integer). The window is increased to a predetermined fixed size.

Level 3) If the corresponding d is less than the threshold δ (δ is a small positive value), replace the corresponding pixel in the temporary noise map r with 0, otherwise leave it as the next stuck pixel and go to level 1. Now this noise map r is the final noise map.

The proposed hot pixel and cold pixel detection methods are as follows.

(I) Prepare a first-stage noise map r using equation (3) for each pixel of the image with noise. We apply the W ₁ WW ₂ window to each and center the same on the current pixel to determine the maximum value (S _max ) and the minimum value (S _min ) in the window, and calculate the first-stage noise map r .

(3)

(3)

Where S _{i , j} is the intensity value of the pixel at position (i, j) and S _max and S _min are the global maximum and minimum intensity values, respectively. In the first stage noise map, r (i, j) = 1 means that the corresponding pixel in a noise image is probably a hot pixel or a cold pixel and r (i, j) = 0 means that in a noisey image This means that the corresponding pixel is normal.

(II) Performs the following operation for each hot (cold) pixel in the image.

Level 1) around the hot (cold) pixels in the image with noise w ₁ x w ₂ We impose a window, where w ₁ , w ₂ are much smaller than the previous window size. If the number of normal pixels is found more than w _min , the distance d between the normal pixels and the central hot (cold) pixel is calculated and goes to level 3. If the number of normal pixels is not greater than w _min , go to level 2. Here, for the simulation, MAD is assumed to be a distance d value.

Level 2) w _i = w _i + p _i (i = 1, 2, pi is a small positive integer). The window is increased to a predetermined fixed size.

Level 3) The minimum distance between the center hot (cold) pixels from both ends is taken as T ₃ . If the image is 8 bits then T ₃ = min (S _ij , 255-S _ij ). If the corresponding d is less than the threshold qT3, replace the corresponding pixel in the temporary noise map r with 0, otherwise leave it as the next hot (cold) pixel and go to level 1. Where q is a constant (0 <q <1). Now this noise map r is the final noise map.

An accurate noise map (a noise map without detection missing or false detection) can be obtained by running a test on a plurality of different images acquired by the sensor. The noise map of these noisy images can be calculated by the proposed detection procedure. The exact noise map is detected by the best voting in all other images. The next time the image is tested, it is not necessary to calculate the noise map since the spatial locations of the defective lines and / or defective pixels are already known. So the inpainting procedure is applied directly.

The above detection procedure gives values of 0 and 1 to the normal pixel and the defective pixel (dead pixel, stuck pixel, hot pixel and cold pixel), respectively, in the noise map r. Here, an adaptive bad pixel inpainting procedure is proposed. It is an adaptive switching median filter (ASMF) modification according to the nature of sensor noise. Adaptive filtering involves calibrating only pixels with a value of 1 in the noise map r in a noisy map. These defective pixels are replaced by the median value of such pixels in the noisy image in a particular window w ₃ xw ₄ where the corresponding value of the noise map is zero. If there are no such pixels in the current window, the window size W _f is increased to a predetermined maximum size W _fmax which holds the original pixels intact for further processing.

The proposed inpainting method for 8 bit images is as follows.

Step 1. In the image with noise, we apply w ₃ × w ₄ window around the bad pixel. Finds a set S with all such image pixels with noise under the current window with a corresponding value of zero in the noise map. If S is not empty, go to step 2, otherwise go to step 3.

Step 2. Replace the current pixel with the median of the set S. Now go to the next bad pixel and go to step 1.

Step 3. Go to step 1 with w _i = w _i + Δw _i (i = 3,4, Δwi is a small positive integer). Increases the window to a predetermined maximum size.

In order to verify the performance of the proposed method, various 8 bit gray scale images were simulated. These images were damaged with dead (stuck) pixels and / or two dead (stuck) lines. Dead (stuck) pixels were created by replacing randomly selected (uniformly distributed) image pixels with zero (white). Dead (stuck) lines were created by replacing two or three consecutive rows or columns of images with zero (extreme values, 255 for 8 bit images). Where one bad line covers the entire width of the image and the other line covers half the image. The direction (row or column) and position of the defective line were uniformly distributed. Images damaged by hot (cold) pixels were created by replacing arbitrarily selected (uniformly distributed) image pixels with values close to extreme values (zero). The performance of the proposed detection procedure was analyzed as mean detection or false detection. Missing detection means that defective pixels are detected as normal pixels. An erroneous detection means that a normal pixel is detected as a bad pixel. Therefore, the detection performance deteriorates as the value of the detection missing or erroneous detection increases. The results show that the proposed bad pixel detection procedure certainly has no missing detection and very low false positives.

Experimental results have shown for accurate noise map detection by various combinations of multiple images. The results show that the three images are sufficient to obtain the exact position of the defective pixel. It is necessary to monitor the sensor periodically. Regular monitoring of these sensors provides the precise location of the defective sensor array. Once the number of bad sensor arrays is known, it is easy to see how the image sensor is damaged and how the acquired image quality can be improved by in-painting until the sensor array is replaced.

The quantitative performance of the restoration process is estimated as the average peak signal to noise ratio (PSNR). If MAX is the maximum intensity value of the image, then for a recovered image Z of size M × N, PSNR is

(4)

(4)

to be. In this case, the mean square error (MSE)

(5)

(5)

to be.

The PSNR average is calculated for 100 different ensembles of the same image that were randomly damaged. The above method clearly shows that new, accurate and simple bad pixel detection and inpainting can be achieved by the present invention. Importantly, the proposed method provides on-line monitoring of sensor status and helps increase the usable life of the sensor. This method uses lightweight real-time software without additional hardware costs to reduce rejection rates at production time. The proposed detection method consists of two steps. The first step uses local image statistics. The next step is to help eliminate false positives and bring their performance closer to the ideal detector. Extensive simulation results further confirm that the proposed method can achieve detection omission zero and very low erroneous detection. Accurate detection can be obtained by applying a defective pixel detection procedure to several different images. Thus, a painting method that is preferably a bad pixel fit can be applied directly to any image. The quantitative results for dead pixels and dead lines can be seen in Figures 2-5. The quantitative results for stuck pixels and stuck lines can be seen in Figures 6-9. The results for the hot pixel and the cold pixel can be seen in FIG. Simulations were also performed on 16-bit original medical images. The results for the medical images can be seen in FIGS. 11 and 12. FIG. The proposed bad (dead, stuck, hot, cold) pixel detection and inpainting procedures can be a powerful tool because of the high PSNR acquisition possible in spite of the high level of noise in the sensor array / imaging device.

Therefore, the present invention can provide a method for detection of defective pixels and correction by in-painting of a sensor array of an imaging device such as a digital camera. This method is adapted to ensure detection false negatives and very low false positives and thus ensure an improvement in image quality even at high defect pixel rates in the sensor array. The method and apparatus of the present invention therefore have significant economic advantages by enabling on-line monitoring of sensor status and improving the usable life of the sensor array by correcting defective pixels in such devices.

Claims (17)

Intensity-based detection of bad pixel position information in a sensor using local statistics of a digital image involving a value indicative of various locations of each of the bad pixels and bad pixels including one or more of dead pixels, stuck pixels, hot pixels, Generating an error / noise map involving &lt; RTI ID = 0.0 &gt;
Applying the location of the identified bad pixel to the evaluation of the local statistics to identify whether the pixel is bad and thereby generate a final noise map having a value of 1 or 0 indicating the location of each of the bad pixel and the normal pixel; And
(Ⅰ) to in order to find a set S with all the noise image pixels under the current window and the value corresponding to the noise map 0, impose w ₃ ⅹw ₄ window around a bad pixel in a noise image, and if S is non-empty, (Iii), (ii) replacing the current pixel with the median of the set S and proceeding to the next bad pixel to go to step (i) , And (iii) w _i = w _i + Δw _i (i = 3,4, Δw _i is a positive integer), and then proceeding to step (i) to increase the window to a predetermined maximum size Correcting the defective pixel by in-painting;
And detecting a defective pixel of the sensor array and correcting the detected defective pixel.
delete The method of claim 1,
Evaluate the number of bad pixels and their subgroups to determine the state of the sensor array; And
Generate an error image for a plurality of subsequent images;
Associating a combination of said subsequent images for further error detection confirmation;
Further comprising the step of: detecting a defective pixel of the sensor array and correcting the detected defective pixel.
The method of claim 1,
Further comprising the step of authenticating an image captured by the camera by comparing the error / noise map and the images obtained from the camera with the error / noise map. A method for correcting a defective pixel.
delete delete delete The method of claim 1, wherein detecting the dead pixel comprises:
(I) imposing w ₁ ⅹw ₂ window for each of the noise image, the maximum value (Smax) and the minimum value centering (centering) the window to the current pixel image to determine the (Smin) in the window, and to thereafter Generating a first stage noise map 'r' such as; And

(I, j) is the global minimum intensity value of the image, and S i, j is the intensity value of the pixel at the position (i, j) = 1 'means that the corresponding pixel in the noise image may be a dead pixel, and' r (i, j) = 0 'means that the corresponding pixel in the noise image is normal)
(II) generating a final noise map 'r';
Wherein the step (II)
Level 1) Impose a w ₁ x w ₂ window around the dead pixel in the noise image such that w ₁ , w ₂ is much smaller than the previous window size; If more pixels than the normal pixel (w _min ) are found in the first stage noise map when the corresponding value is 0, then the distance d between the normal pixels and the central dead pixel is calculated and level 3 Go to; Go to level 2 if the number of normal pixels is not greater than w _min ; In the above, d, which is assumed to be the absolute distance MAD for simulation, is calculated by the following equation;

(Where ei is the distance from the center pixel to the ith pixel)
Level 2) set w _i = w _i + p _i (i = 1, 2, p _i is a positive integer), go to level 1 to increase the window to a predetermined fixed size;
Level 3) If the corresponding d is less than the threshold δ (δ is a small positive value), replace the corresponding pixel in the temporary noise map 'r' with 0, otherwise, leave it as the next dead pixel, Going to;
And detecting a defective pixel of the sensor array and correcting the detected defective pixel.
2. The method of claim 1, wherein detecting the stuck pixels comprises:
(I) of claim 1, such as for since impose w ₁ ⅹw ₂ window for each noise image and centering the window on the current pixel in order to determine the maximum value (Smax) and the minimum value (Smin) in the window, and that Generating a step noise map 'r'; And

(I, j) = (i, j), where S _{i, j} is the intensity value of the pixel at position (i, j) and Smax is the global maximum intensity value of the image, 1 'means that the corresponding pixel in the noise image can be a stuck pixel, and' r (i, j) = 0 'means that the corresponding pixel in the noise image is normal)
(II) generating a final noise map 'r';
, Wherein step (II) comprises:
Level 1) The noise image imposes w _{1 2} ⅹw window around a pixel stuck in the w _1, w ₂ is much smaller than the previous window size; And, if more pixels than the normal pixel (w _min ) are found in the first stage noise map when the corresponding value is 0, the distance d (d is assumed as MAD) between the normal pixels and the central stuck pixel is Calculate and go to level 3; Go to level 2 if the number of normal pixels is not greater than w _min ;
Level 2) set w _i = w _i + p _i (i = 1, 2, p _i is a positive integer), go to level 1 to increase the window to a predetermined fixed size;
Level 3) If the corresponding d is less than the threshold δ (δ is a small positive value), replace the corresponding pixel in the temporary noise map 'r' with 0, otherwise, move to the next stuck pixel, So that the noise map 'r' becomes the final noise map;
And detecting a defective pixel of the sensor array and correcting the detected defective pixel.
2. The method of claim 1, wherein detecting the hot pixels and cold pixels comprises:
(I) applying a w ₁ x w ₂ window for each noise image, and centering the window on the current pixel, and then generating a first stage noise map 'r'; And

(I, j) = (i, j) _, where S _{i, j} is the intensity value of the pixel at the position (i, j), Smax and Smin are the global maximum and minimum intensity values, 1 'means that the corresponding pixel in the noise image is a hot pixel or a cold pixel, and' r (i, j) = 0 'means that the corresponding pixel in the noise image is normal)
(II) generating a final noise map 'r';
Wherein the step (II)
Level 1), the noise around the hot (cold) of pixels in the image w ₁ ⅹw imposed by the _second window w _1, w ₂ is much smaller than the previous window size; If more pixels than the normal pixel (w _min ) are found in the first stage noise map when the corresponding value is 0, the distance d between the normal pixels and the central hot (cold) ) And go to level 3; Go to level 2 if the number of normal pixels is not greater than w _min ;
Level 2) set w _i = w _i + p _i (i = 1, 2, p _i is a positive integer), go to level 1, and increase the window to a predetermined fixed size;
Level 3) Taking the minimum distance of the center hot (cold) pixel from both ends and taking T ₃ = min (S _ij , 255-S _ij ) if the image is 8 bits;
Step,
If the corresponding d is less than the threshold qT ₃ (q is a constant, 0 <q <1), replace the corresponding pixel in the temporary noise map 'r' with 0, Pixel to go to level 1, thereby producing a final noise map 'r'. A method of detecting a defective pixel in a sensor array and correcting said detected defective pixel.
The method of claim 1, wherein the method is applied at any bit depth. A method of detecting defective pixels in a sensor array and correcting the detected defective pixels.
delete 2. The method of claim 1, wherein the defective pixel detection procedure preceding the defective pixel painting obtains a high PSNR with image distortion for a plurality of defective pixels, and detecting a defective pixel in the sensor array, A method for correcting defective pixels.
The method of claim 1, wherein the defective pixel detection is performed on-line or off-line, and once detected, in-painting can be applied on-line / off-line according to an application request, And correcting the detected defective pixel.
(I) means for generating a noise map involving intensity-based detection of bad pixel position information of a sensor using local statistics of a digital image involving a value indicative of the location of each of a bad pixel and a normal pixel; And
(II) means for performing an adaptive inpainting procedure that corrects only the defective pixels detected in the noise map in relation to the noise image;
Wherein the conforming painting procedure comprises:
(I) impose a w ₃ x w ₄ window around the bad pixel in the noise map to find a set S having all the noise image pixels under the current window with a corresponding value of 0 in the noise map, and if S is not empty, Go to step (ii), otherwise go to step (iii);
(Ii) replacing the current pixel with the median of the set S and proceeding to the next bad pixel to go to step (i);
(Iii) set w _i = w _i + Δw _i (i = 3,4, Δw _i is a positive integer) and go to step (i) to increase the window to a predetermined maximum size;
And a sensor array configured to detect defective pixels and perform inpainting of defective pixels.
16. The system of claim 15, wherein the sensor array is configured to detect a variety of bad pixels selected from dead pixels, stuck pixels, cold pixels and hot pixels according to the method of any one of claims 8-11 and 13-14. Wherein the sensor array is configured to detect defective pixels and perform inpainting of defective pixels.
16. The system of claim 15, wherein the system is configured to detect defective pixels and to perform inpainting of defective pixels, characterized by converting optical and thermal signals including thermal, microwave, ultrasonic, x-rays, An image sensor system comprising a sensor array.

Images