JP2006100913A - Image processor and processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To correct a defective pixel properly by preventing deterioration in image quality due to incorrect detection of a high frequency component, or the like. <P>SOLUTION: The image processor comprises a first detection means for detecting first detection results, i.e. the level difference between a signal output from a remarked pixel and a signal output from a pixel other than the remarked pixel, a second detection means for detecting second detection results, i.e. the level of a signal output from a pixel other than the remarked pixel, and a means for correcting the signal output from the remarked pixel, based on the first detection results and the second detection results. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to correction of pixel defects in an image sensor.

  In an image pickup device such as a CCD, it is known that local insensitivity of a semiconductor occurs in a manufacturing process or after manufacturing among two-dimensionally arranged pixels. When these phenomena occur, a charge output corresponding to the amount of incident light cannot be obtained, so white spots and black spots can be seen on the captured image regardless of the subject and appear as so-called defective pixels.

  In order to correct the image quality degradation caused by the imaging output of such defective pixels by signal processing, conventionally, defective data about defective pixels included in the imaging device is detected at the manufactured semiconductor factory and stored in the nonvolatile memory in advance. Alternatively, on the image processing device side using the image sensor, when the shutter of the image sensor is in a light-shielded state, the pixel (white point defect pixel) exceeding a predetermined output level and the amount of light incident on the image processing device The position data of pixels (black spot defective pixels) that do not reach the predetermined output level is stored in the non-volatile memory, and the position data stored in the non-volatile memory at the time of normal imaging The defective pixel is corrected by substituting the imaging output of the previous pixel in place of the imaging output of the defective pixel. You have me.

  By the way, the number of pixels required for an image sensor in recent years has increased from about several hundred thousand pixels to millions of pixels, and defective pixels appearing in the image sensor in spite of progress in manufacturing technology of the image sensor. The probability of occurrence tends to increase. In particular, in an image sensor used in a consumer device that requires low cost, in order to increase the manufacturing yield, the number of defective pixels has to be allowed to be much larger than that in the past.

  Accordingly, it has become difficult to store the position data of defective pixels in an expensive non-volatile memory as in the prior art.

For such a situation, for example, in Reference 1, in normal imaging, a defective pixel is detected by detecting a level difference between each pixel signal of a certain pixel and the same color pixel adjacent thereto and performing a predetermined threshold determination. Detected and corrected. As a result, it is possible to omit a non-volatile memory for preliminarily storing position information of defective pixels. Similarly, in Document 2, for example, defective pixels included in a predetermined area of the image sensor are stored in a nonvolatile memory, and defective pixels included outside the predetermined area are corrected for defective pixels stored in the nonvolatile memory. After that, by detecting and correcting defective pixels simultaneously during normal imaging, even a limited non-volatile memory can handle a large number of defective pixels.
Japanese Patent Laid-Open No. 06-030425 JP 2002-027323 A

  However, in the above-described conventional technique, during normal imaging, defective pixel correction is performed by detecting a level difference between each pixel signal of a certain pixel and an adjacent same color pixel and performing a predetermined threshold determination. Of course, high frequency components such as subject edges are also erroneously detected and corrected. This hinders the image quality of the output image of normal pixels, and it cannot be said that this problem has been sufficiently studied. In addition, due to the influence of noise or the like, the determination result is finely switched near the threshold value, and there is a problem that the output image flickers during moving image shooting. Therefore, an object of the image processing apparatus of the present invention is to enable appropriate defective pixel correction without causing image quality degradation due to erroneous detection of defective pixels, such as erroneous detection of high-frequency components such as subject edges.

  In order to achieve the above-described problem, the image processing apparatus of the present invention detects a first detection result that is a level difference between a signal output from a target pixel and a signal output from a pixel other than the target pixel. First detection means; second detection means for detecting a second detection result that is a level of a signal output from a pixel other than the target pixel; the first detection result and the second detection result; And correcting means for correcting the signal output from the pixel of interest.

  The image processing method of the present invention is an image processing method of an image processing apparatus having an image pickup device having a plurality of pixels for converting a subject image formed by an optical system into an electric signal, and is output from a target pixel. A first detection result that is a level difference between the received signal and a signal output from a pixel other than the pixel of interest, and a second detection result that is a level of a signal output from a pixel other than the pixel of interest And detecting and correcting the signal output from the pixel of interest based on the first detection result and the second detection result.

  According to the present invention, it is possible to prevent erroneous detection and correction of defective pixels and to obtain an image with good image quality with little influence of noise.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 9 is a block diagram illustrating a configuration of the image processing apparatus 300 according to the first embodiment of the present invention. This configuration is, for example, a single-plate image processing apparatus (digital color camera or the like) using an image pickup device such as a CCD or CMOS sensor, and the image pickup device is continuously or once driven to generate a moving image or a still image. When applied to an image processing apparatus that obtains an image signal representing The light beam that has passed through the optical block 202 including a photographic lens is guided to the light receiving surface of the image sensor 100. Thereby, an image signal corresponding to the subject image is generated in the photodiode constituting the light receiving surface of the image sensor 100. The image sensor 100 is driven by a TG (timing generator) 209. The image signal output from the image sensor 100 is input to a CDS (correlated double sampling and hold) circuit 203. The image signal is subjected to predetermined signal processing such as reset noise removal in the CDS circuit 203 in accordance with the timing signal determined by the TG 209, and is sampled and held. A gain amplifier 212 amplifies the analog signal output of the image sensor 100 to set the sensitivity of the camera, and 204 is an A / D converter 204 that converts the analog signal output of the image sensor 100 into a digital signal. The A / D converter 204 performs A / D conversion according to the clock signal supplied from the TG 209. The pixel signal converted into the digital signal by the A / D converter 204 is corrected and output by the correction circuit 205 as correction means. The operation of the correction circuit 205 will be described later.

  The output of the correction circuit 205 is input to the camera signal processing circuit 206, and undergoes processing such as color conversion, white balance, and gamma correction, and is converted into YCrCb luminance and color difference signals. The output of the camera signal processing circuit 206 is subjected to processing such as compression in the recording circuit 207, converted into a predetermined format, and recorded on a removable recording medium 208. Further, the output of the camera signal processing circuit 206 is subjected to processing such as reduction / enlargement processing and superimpose in the display circuit 210, and then converted into, for example, an NTSC analog signal and displayed on the display 211. .

  Next, the operation of the correction circuit 205 will be described with reference to FIG. By inputting the A / D converted image signal to the defective pixel detection means 101 by the A / D converter, the difference value D between the target pixel and the average value of the surrounding pixels is obtained as shown in Equation (1). Find and output.

Target pixel-average value of peripheral pixels of target pixel = difference value D (1)
Further, by inputting an image signal to the peripheral level information detection means 102, an average value of peripheral pixels of a certain target pixel is detected and output. An example of the peripheral same color pixel corresponding to the target pixel is shown in FIG. When the target pixel is G, an average value of the surrounding G values is obtained and output as peripheral level information. Similarly, when the target pixel is R or B, the average value of the peripheral same color pixels is obtained and output as peripheral level information. The peripheral level information detection unit 102 can detect brightness information about pixels around the target pixel.

  The gain setting means 103 receives the output signal from the peripheral level information detection means 102 and outputs a gain g corresponding to the peripheral level information. FIG. 3A shows an example of correspondence between peripheral level information and gain output when white point defective pixels (white scratches) are corrected, with the horizontal axis indicating peripheral level information and the vertical axis indicating gain. g. When white point defect pixel correction is performed, there is a predetermined threshold Th1, and when the peripheral level information is smaller than the threshold Th1, the gain g is increased, and when the peripheral level information is larger than the threshold Th1, the gain g is decreased. . FIG. 3B shows an example of the correspondence between the peripheral level information and the output of the gain when correcting the black spot defective pixel (black defect). The horizontal axis represents the peripheral level information, and the vertical axis represents Gain g. When performing black point defect pixel correction, there is a predetermined threshold Th1, and when the peripheral level information is smaller than the threshold Th1, the gain g is decreased, and when the peripheral level information is larger than the threshold Th1, the gain g is increased. This correspondence includes a method of creating a ROM table and storing values at the time of shipment from the factory, and a method of calculating and obtaining the gain g using an arithmetic expression based on peripheral level information.

  The correspondence table between the white point defective pixel correction and the black point defective pixel correction is switched according to the characteristic of the difference value D that is an output signal of the defective pixel detection unit 101. Switching can be appropriately set by the image processing apparatus. As an example of this switching setting, there is a method in which black defect pixel correction is performed when the difference value D is smaller than a certain threshold, and white defect pixel correction is performed when the difference value D is larger. For example, when the threshold value is set to 0, when the difference value D is positive, it is switched to white defect pixel correction, and when the difference value D is negative, it is switched to black defect pixel correction.

  Then, a difference value D ′ is obtained by multiplying the difference value D, which is the output signal of the defective pixel detection means 101, and the gain g, which is the output signal of the gain setting means 103.

  An example of the output of the correction coefficient output means 104 is shown in FIG. The horizontal axis is the difference value D ′ that is the input signal, and the vertical axis is the mixing ratio k that is the output signal. The defective pixel correction means 105 is a value obtained by correcting the defective pixel using the input target pixel as a defective pixel, and an image signal A / D converted by the A / D converter, and performing the defective pixel correction. This is a coefficient that indicates the ratio of mixing non-image signals. The output image signal of the target pixel can be obtained by multiplying the signal after the defective pixel correction by the mixing ratio k and the signal before the defective pixel correction by (1-k) and adding the multiplication results. This calculation process is performed by the correction value determining means 106.

  First, as shown in FIG. 4, the correction coefficient output means 104 uses two threshold values Th2 and Th3, and outputs 0 when the difference value D ′ is smaller than Th2, and when Th2 <difference value D ′ <Th3. 1 is output, and the difference value D ′ is smaller than Th3. This correspondence includes a method of storing values using a ROM table at the time of shipment from the factory, a method of obtaining by calculation using an arithmetic expression, and the like. Thus, the correction coefficient k (mixing ratio k) corresponding to each pixel of interest is determined based on the value of the difference value D ′.

  As can be seen from FIG. 4, when the difference value D 'is larger than Th3, the mixture ratio k is set to 0 in order to reduce the mixture ratio k or to output a value that is not corrected for defective pixels. In this way, by setting the ratio of the values output without correcting defective pixels to be large, it is possible to prevent erroneous detection and correction of high-frequency components such as subject edges.

  The defective pixel correction means 105 outputs a value obtained by correcting the defective pixel with the input target pixel as a defective pixel. This defective pixel correction can be performed, for example, by obtaining an average value of signals from adjacent pixels of the same color around the target pixel and performing interpolation. The correction value determination means 106 uses the image signal (referred to as a pre-correction signal) A / D converted by the A / D converter and the output signal (referred to as a corrected signal) from the defective pixel correction means 105 as input signals. Then, the mixing ratio k, which is an output signal from the correction coefficient output means 104, is used as a control signal, and is linearly synthesized by Expression (2) to obtain an output signal.

Signal after correction × mixing ratio k + signal before correction × (1-k) = output signal (2)
By generating an output signal for each pixel of interest in this way, in the case of a white point defective pixel, a value that is not subjected to defective pixel correction is output at a location where the peripheral level information is large, that is, a bright location. Since the defective pixel correction value is output in small areas, that is, in dark areas, the adverse effects of defective pixel correction (influences of random noise and overcorrection) are reduced in bright areas, and defective pixels are corrected in dark areas. As a result, image quality deterioration can be reduced and a good output signal can be obtained. Further, in the case of a black spot defective pixel, a value obtained by performing defective pixel correction is output at a portion where peripheral level information is large, that is, a bright portion, and defective pixel correction is not performed at a portion where peripheral level information is small, that is, a dark portion. Since a value is output, image quality degradation can be reduced by correcting defective pixels in bright places, and adverse effects (effects caused by random noise or overcorrection) due to defective pixel correction can be reduced in dark places.

  With the above-described configuration, it is possible to adjust the rate at which the defective pixel-corrected value is output depending on the condition of the subject or whether the image is white or black, and obtain a high-quality image suitable for each shooting condition. be able to.

  The second embodiment of the present invention is omitted because the overall configuration is the same as that of the first embodiment, and the peripheral level information detecting means 102 having a different operation will be described. In the first embodiment, for example, when the target pixel is G, the peripheral G signals are averaged to detect the peripheral level information, so that the peripheral level is detected by using the same color signals for the R, G, and B pixels. Although information has been detected, in Example 2, the average value of the four pixels adjacent to the left and right is obtained as the peripheral level information for any color of the target pixel of R, G, and B.

  An example of the peripheral same color pixel corresponding to the target pixel is shown in FIG. The A / D converter inputs the A / D converted image signal to the peripheral level information detection means 102, and obtains and outputs the average value of four pixels adjacent to the left and right of a certain target pixel. When the target pixel is G, the average value of the four pixels adjacent to the left and right is obtained and output as peripheral level information. In the case of R and B as well, an average value is obtained in the same manner and output as peripheral level information. The peripheral level information detection unit 102 can detect brightness information about pixels around the target pixel.

  The gain setting means 103 receives the output signal from the peripheral level information detection means 102 and outputs a gain g corresponding to the peripheral level information. Then, a difference value D ′ is obtained by multiplying the difference value D, which is the output signal of the defective pixel detection means 101, and the gain g, which is the output signal of the gain setting means 103.

  Thereafter, the mixing ratio k is obtained based on the difference value D ′. Since the method for generating the correction value is the same as that in the first embodiment, the description thereof is omitted.

  The third embodiment of the present invention is omitted because the overall configuration is the same as that of the first embodiment, and the peripheral level information detecting means 102 having a different operation will be described.

  The A / D converter inputs the A / D converted image signal to the peripheral level information detecting means 102, thereby obtaining an average value for each color of the peripheral pixels of a certain target pixel. An example of this is shown in FIG. 6A shows a case where the target pixel is G, and FIG. 6B shows a case where the target pixel is R. FIG. 6C shows a case where the target pixel is B. For example, in the case where the target pixel is G, the average value of 6 pixels around G, the average value of 2 pixels around R, and the average value of 2 pixels around B are obtained. Then, the average value for each color of the pixels around the pixel of interest is multiplied by a fixed coefficient (a ROM table is created in advance and the value is stored) to obtain a sum, which is output as peripheral level information.

  The peripheral level information detection unit 102 can detect brightness information about pixels around the target pixel.

  The gain setting means 103 receives the output signal from the peripheral level information detection means 102 and outputs a gain g corresponding to the peripheral level information. Then, a difference value D ′ is obtained by multiplying the difference value D, which is the output signal of the defective pixel detection means 101, and the gain g, which is the output signal of the gain setting means 103.

  Thereafter, the mixing ratio k is obtained based on the difference value D ′. Since the method for generating the correction value is the same as that in the first embodiment, the description thereof is omitted.

  An overall view of the image processing apparatus according to the fourth embodiment of the present invention has the same configuration as that of FIG. FIG. 7 shows an application diagram of a correction circuit which is a correction means for correcting defective pixels in the fourth embodiment. By inputting the A / D converted image signal to the defective pixel detection means 101 by the A / D converter, a difference value D between the target pixel and the average value of the surrounding pixels is obtained and output.

  The correction coefficient output means 104 outputs the mixture ratio k in accordance with the difference value D that is the output signal of the defective pixel detection means 101 that has been input. An example of the correspondence between the difference value D and the mixing ratio k is shown in FIG. The horizontal axis is the difference value D that is the input signal, and the vertical axis is the mixing ratio k that is the output signal. The correction coefficient output means 104 outputs 0 <k ≦ 1 as the mixing ratio k when the difference value D is Th4 <D <Th5, and outputs 0 as the mixing ratio k when the difference value D is smaller than Th4. . Further, when Th5 <D, k becomes closer to 0 as the value of D increases. Correspondence between these difference values and mixing ratios includes a method of storing values in a ROM table at the time of factory shipment, a method of calculating by substituting D into an arithmetic expression, and the like. Thus, each correction coefficient k is determined by the value of the difference value D.

  The defective pixel correction unit 105 outputs a value obtained by correcting the input image signal of the target pixel with defective pixels. As a correction method, for example, there is a method of obtaining an average value of signals of a plurality of peripheral pixels of the target pixel and using the corrected signal of the target pixel.

  Using the mixture ratio k, which is an output signal of the correction coefficient output means 104, as a control signal, a value obtained by correcting the defective pixel, which is an output signal of the defective pixel correction means 105, and a value, which is not corrected by the defective pixel, which is an input signal. Perform linear synthesis. Since this linear synthesis method is the same as the linear synthesis method using the formula (2) in the first embodiment, a description thereof will be omitted. By generating the output signal of the correction value determining means 106 in this way, the value that has been subjected to defective pixel correction and the value that has not been corrected are finely switched and output for each frame due to the influence of noise or the like in the vicinity of the threshold value. The occurrence of flickering can be reduced.

  An application diagram of the fifth embodiment of the present invention is shown in FIG. FIG. 8 shows a specific configuration diagram of the correction circuit 205 and the camera signal processing circuit 206 in the block diagram of the configuration of the image processing apparatus in FIG. 9. Since the defective pixel detection unit 101, the defective pixel correction unit 105, and the correction value determination unit 106 are the same as those in the fourth embodiment, description thereof will be omitted, and only new elements of the present invention will be described.

  The camera signal processing circuit 206 performs camera signal processing (signal processing of an imaging system such as aperture correction, gamma correction, and white balance). Among these, a circuit gain or an image sensor that is a coefficient for amplifying a signal from a pixel An adjustment is made to the shutter speed at which the image is exposed. In the correction coefficient output unit 104, when the circuit gain (ISO sensitivity) adjusted by the gain amplifier 212 increases, the S / N of the image is prevented from deteriorating by making the slope of the correspondence table of FIG. 11 gentle. To do. For example, when the circuit gain is increased, Th6 and Th8 in FIG. 11 are fixed, and Th6 is decreased and Th9 is increased. When the circuit gain is decreased, Th7 and Th8 in FIG. 11 are fixed. If Th6 is increased and Th9 is decreased, the inclination of the slope can be adjusted. Note that information relating to shooting conditions (circuit gain, shutter speed, etc.) is transmitted to the correction coefficient output means 104 through the CPU 213.

When the configuration of the image processing apparatus is as described above, noise increases due to an increase in circuit gain, and this influence reduces the adverse effect that the correction value changes greatly for each shooting frame and the screen flickers. can do.
Further, in the correction coefficient output means 104, when the shutter speed becomes slow, the values of Th6 and Th7 are decreased and the values of Th8 and Th9 are increased while the inclination of the graph of FIG. 11 is fixed. When the shutter speed increases, the values of Th6 and Th7 are increased and the values of Th8 and Th9 are decreased with the inclination of the graph of FIG. 11 fixed. As the shutter speed decreases, the level of defective pixels increases. Therefore, by increasing the number of locations where the value of the mixing ratio k in FIG. 11 corresponds to 1, it is possible to easily output a value corrected for defective pixels, and to reduce adverse effects due to defective pixels. Also, as the shutter speed increases, the level of defective pixels decreases, so that the values of Th6 and Th7 are increased, the values of Th8 and Th9 are decreased, and the mixing ratio k is decreased with the inclination of the graph of FIG. 11 fixed. By reducing the number of locations where the value corresponds to 1, it is possible to easily output values that are not corrected for defective pixels, and to reduce the adverse effect of erroneously correcting pixels that are not defective pixels.

  As described above, by adjusting the ratio at which defective pixel corrected values are output in accordance with imaging conditions such as circuit gain and shutter speed, it is possible to obtain images with good image quality suitable for each imaging condition. Can do.

  According to the defective pixel correction method described in the first to fifth embodiments, white point defective pixels have a feature that they are not conspicuous in a bright place and are conspicuous in a dark place. Adverse effects can be reduced, and image quality deterioration due to defective pixels can be reduced by correcting defective pixels in dark places. Since black spot defective pixels are conspicuous in bright places and inconspicuous in dark places, image quality degradation due to defective pixels can be reduced by correcting defective pixels in bright places, and erroneously performed in dark places. An adverse effect due to defective pixel correction can be reduced. In addition, as in the fourth and fifth embodiments, if the threshold value is changed according to the imaging condition, the value near the threshold value for switching the correction is finely changed due to the influence of noise or the like to solve the problem that the screen flickers. Can do.

It is a figure explaining the detail of the correction circuit in Examples 1-3. FIG. 6 is a schematic diagram of peripheral pixels in the peripheral level information detection unit in the first embodiment. It is a related figure of the peripheral level information in the gain setting means in Examples 1-3, and a gain. It is a relationship diagram of the difference value and the mixture ratio in the correction coefficient output means in Examples 1 to 3. FIG. 6 is a schematic diagram of peripheral pixels in a peripheral level information detecting unit in Embodiment 2. It is a schematic diagram of the surrounding pixel in the surrounding level information detection means in Example 3. It is a figure explaining the detail of the correction circuit in Example 4. FIG. FIG. 10 is a diagram illustrating details of a correction circuit and a camera signal processing circuit according to a fifth embodiment. 1 is a block diagram of an image processing apparatus according to first to fifth embodiments of the present invention. FIG. 10 is a relationship diagram between a difference value and a mixture ratio in a correction coefficient output unit according to a fourth embodiment. FIG. 10 is a relationship diagram between a difference value and a mixture ratio in a correction coefficient output unit of Embodiment 5.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Defective pixel detection means 102 Peripheral level information detection means 103 Gain setting means 104 Correction coefficient output means 105 Defective pixel correction means 106 Correction value determination means 202 Optical block 100 Image sensor 203 CDS circuit 204 A / D converter 205 Correction circuit 206 Camera Signal processing 207 Recording circuit 208 Recording medium 209 TG
210 Display Circuit 211 Display 212 Gain Amplifier 213 CPU
300 Image processing device

Claims (6)

First detection means for detecting a first detection result which is a level difference between a signal output from a target pixel and a signal output from a pixel other than the target pixel;
Second detection means for detecting a second detection result which is a level of a signal output from a pixel other than the target pixel;
An image processing apparatus comprising: a correction unit that corrects a signal output from the target pixel based on the first detection result and the second detection result.   The correction unit outputs a signal obtained by mixing a signal output from a target pixel and a signal output from a pixel other than the target pixel based on the first detection result and the second detection result. The image processing apparatus according to claim 1, wherein:   The correction means includes control means for controlling the ratio of mixing the signal output from the target pixel and the signal output from a pixel other than the target pixel so as to be changed according to an imaging condition. The image processing apparatus according to claim 2.   The image processing apparatus according to claim 3, wherein the imaging condition is at least one of a circuit gain and a shutter speed. An image sensor comprising a plurality of pixels for converting an object image formed by an optical system into an electrical signal;
An A / D converter for A / D converting the electrical signal output from the image sensor and outputting an image signal;
The image processing apparatus according to claim 1, wherein the correction unit corrects the image signal output from the A / D converter. An image processing method of an image processing apparatus having an imaging device having a plurality of pixels for converting an object image formed by an optical system into an electrical signal,
A first detection result that is a level difference between a signal output from a target pixel and a signal output from a pixel other than the target pixel;
Detecting a second detection result that is a level of a signal output from a pixel other than the target pixel;
An image processing method, comprising: correcting a signal output from the target pixel based on the first detection result and the second detection result.

Images