JP2007201530A - Pixel defect correction apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a pixel defect correction apparatus capable of more accurately discriminating a defective pixel without the need for storing positional information of the defective pixel in advance. <P>SOLUTION: A memory section 141 receives image data and stores a pixel value by 3×3 pixels in a matrix form around a target pixel. The pixel value of the nine pixels is outputted to a comparison section 142, wherein whether or not a difference between the pixel value of the target pixel and the pixel value of the eight surrounding pixels is greater than a prescribed value is discriminated. A pattern discrimination section 143 discriminates whether the nine pixels around the target pixel correspond to a basic pattern of defective pixels or a basic pattern of normal pixels on the basis of a result of the discrimination by the comparison section 142. When the target pixel is not a defective pixel, a defect correction section 144 outputs the pixel value of the target pixel received from the memory section 141 as it is. When the target pixel is a defective pixel, a median (binary) calculation section 145 calualtes a median from two medians obtained from the pixel values of the eight surrounding pixels, and the defect correction section 144 corrects the pixel value of the target pixel by using the median. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an apparatus for correcting pixel data corresponding to a defective pixel. In particular, the present invention relates to a pixel defect correction apparatus that corrects pixel data output from an image sensor in real time.

  A defective pixel is generally present in a solid-state imaging device. Such a tendency becomes remarkable especially in a small and high density image sensor. For defective pixels, pixel data is corrected by pixel defect correction processing, and pixel defect correction processing is performed using pixel data of normal pixels around the defective pixel.

  In the pixel defect correction process, it is first determined whether the pixel data being processed is of a defective pixel. When the pixel data corresponds to a defective pixel, for example, correction (interpolation) processing is performed using an average value of surrounding pixel data. Whether or not a pixel is a defective pixel is determined with reference to a pattern stored in advance in a memory provided in an imaging device such as a camera on which a solid-state imaging device is mounted. Further, the pattern information related to the defective pixel determination held in the memory is obtained by inspecting a defective pixel generated uniquely in each image sensor at the time of manufacturing the solid-state image sensor (Patent Document 1).

In addition, when a pixel of interest (a pixel currently subject to pixel defect correction processing) and its surrounding pixels are compared in real time during shooting, and the pixel is extremely different from the surrounding pixel value level, the defective pixel Therefore, a method of performing correction using the average ground of surrounding pixels has been proposed (Patent Document 2).
JP-A-10-322603 JP 2002-320145 A

  However, in the method of Patent Document 1, a memory having a capacity corresponding to the number of defective pixels is required, and the inspection of defective pixels at the time of manufacturing causes an increase in manufacturing cost. Also, with this method, it is not possible to correct a pixel that has newly become a defective pixel due to aging or temperature change.

  In addition, the method of Patent Document 2 can deal with a newly defective pixel to some extent, but if two or more defective pixels continue in any of the vertical, horizontal, and diagonal directions, the pixel is defective. The pixel cannot be determined and correction is not performed.

  Furthermore, in the conventional method, the defective pixel data is interpolated using the average value of the surrounding pixel data, but if the surrounding pixels used for the interpolation include defective pixels, the interpolated pixel data is converted into the interpolated pixel data. Affects and image degradation occurs.

  An object of the present invention is to provide a pixel defect correction device that can accurately determine a defective pixel without needing to store position information of the defective pixel in advance.

  A pixel defect correction apparatus according to the present invention includes a defective pixel determination unit that determines whether or not a target pixel is a defective pixel, and a pixel that is a target pixel when the defective pixel determination unit determines that the target pixel is a defective pixel. Pixel value correcting means for correcting the value, and the defective pixel determining means determines whether the target pixel is a defective pixel based on at least one of a basic pattern of defective pixels or a basic pattern of normal pixels. It is characterized by performing the determination.

  The defective pixel determination unit determines whether the target pixel corresponds to a basic pattern based on a comparison of pixel values between the target pixel and surrounding pixels. The defective pixel determination means determines the target pixel based on both the basic pattern of defective pixels and the basic pattern of normal pixels.

  The corrected pixel value used as the pixel value of the target pixel determined to be a defective pixel is calculated based on, for example, the pixel values of pixels around the target pixel. For example, the correction pixel value is a median value of the pixel values of the surrounding pixels. For example, the surrounding pixels are 8 pixels around the target pixel.

  For example, the defective pixel determination unit and the pixel value correction unit are sequentially executed on the pixel data output from the image sensor, and the correction of the defective pixel is performed simultaneously with the imaging operation by the image sensor.

  In addition, a digital camera according to the present invention includes the above-described pixel defect correction device.

  As described above, according to the present invention, it is possible to provide a pixel defect correction apparatus that can determine defective pixels more accurately without having to previously store position information of defective pixels.

Embodiments of the present invention will be described below with reference to the accompanying drawings.
FIG. 1 is a block diagram schematically showing the overall configuration of an image processing system including a pixel defect correction processing system according to an embodiment of the present invention. Any image processing system may be used as long as it can perform pixel defect correction processing. However, in the following, a case where pixel defect correction is performed simultaneously with photographing using the digital camera 10 as an example (dynamic pixel defect correction) will be described. Give an explanation.

  In the digital camera 10, a subject image is formed on an image sensor (for example, CCD) 12 via a lens 11, and the image signal is input to an image processing unit 13. In the present embodiment, the image processing unit 13 is broadly divided into a dynamic defect correction device 14 as a pixel defect correction processing system and a circuit unit 15 that performs various other known processes in the digital camera 10. The image signal obtained by the imaging device 12 is subjected to pixel defect correction processing in the dynamic defect correction device 14 and other known image processing in the circuit unit 15. Thereafter, the image signal is output to the user interface 16 and displayed as a still image or a moving image (through image) on a display device (not shown), for example. In FIG. 1, a configuration that is not directly related to the pixel defect correction process, such as a configuration related to driving of the image sensor 12, is omitted.

  FIG. 2 schematically shows a part of the pixel array of the image sensor 12. In the example of FIG. 2, the image sensor 12 is an image sensor provided with RGB on-chip color filters. As shown in the figure, the pixel arrangement pattern is repeated with 4 pixels (2 × 2 pixels) composed of 2 pixels in the vertical and horizontal directions as a pattern unit. In the present embodiment, these four pixels are composed of one pixel corresponding to the B color filter, one pixel corresponding to the R color filter, and two pixels corresponding to the G color filter, and B and R are on one diagonal line. Two Gs are arranged on the other diagonal. This filter arrangement is an example, and the present invention is not limited to this. For example, the image pickup device 12 may use a complementary color filter, an image pickup device equipped with a monochromatic filter, or an image pickup device not equipped with a color filter.

  FIG. 2 is an excerpt of an arbitrary area of the image sensor 12. In the following description, the illustrated coordinate value is assigned to each pixel in this area. That is, in the illustrated area, numbers 1, 2, 3,... Are assigned to the pixels from left to right (vertical line) in the horizontal direction, and pixels from the bottom to top (horizontal line) in the vertical direction. Are assigned alphabets a, b, c,.

  FIG. 3 shows only the B pixel extracted from the array of FIG. In the following description, the pixel defect correction process performed on the B pixel will be described as an example, but the same applies to other colors.

  In the present embodiment, it is determined whether or not the pixel is a defective pixel based on the characteristics of a general pattern when a defective pixel occurs. That is, a pattern that a defective pixel can generally take is determined in advance as a defect detection pattern, and the value of the pixel of interest (the pixel that is currently subject to determination of the defective pixel) and its surrounding pixels are compared with the defect detection pattern. Thus, the defective pixel is determined.

  When a plurality of defective pixels are adjacent to each other, the defective pixels are generally often arranged vertically (vertical line direction) or horizontally (horizontal line direction). Therefore, in this embodiment, a pattern corresponding to the above feature is adopted as the first defect detection pattern, and in comparison between the pixel value of the target pixel and the pixel values of pixels around the same color as the target pixel, When this corresponds to the first defect detection pattern, the target pixel is extracted as a defective pixel or as a defective pixel candidate.

  The specific operation of the pixel defect correction processing according to the present embodiment will be described below with reference to FIGS. 4 to 10 by taking as an example the case where the pixel 5c in FIG. 2 and FIG. In the present embodiment, pattern determination is performed on the target pixel 5c and the eight pixels 3a, 5a, 7a, 3c, 7c, 3e, 5e, and 7e of the same color surrounding the target pixel 5c (that is, the target pixel in FIG. 3 × 3 pixels as the center).

  FIG. 4 is a block diagram schematically showing the configuration of the dynamic defect correction apparatus 14. The dynamic defect correction apparatus 14 mainly includes a memory unit 141, a comparison unit 142, a pattern determination unit 143, a defect correction unit 144, and an intermediate value (binary) calculation unit (correction value calculation unit) 145.

  The memory unit 141 includes, for example, a memory (not shown) that holds pixel values for five horizontal lines (a to e) around the target pixel 5c, and an image input from the image sensor 12 via the circuit unit 15, for example. Data is stored in the order of input. The memory unit 141 includes, for example, a plurality of flip-flops (not shown), and is configured to be able to simultaneously compare pixel values in the horizontal line direction using these. That is, the pixel data of 3 rows × 3 columns of the same color centered on the target pixel is held in the memory unit 141 (step S101 in FIG. 10).

  From the memory unit 141, pixel data of the target pixel 5c and pixel data of the surrounding eight pixels 3a, 5a, 7a, 3c, 7c, 3e, 5e, and 7e are output to the comparison unit 142. The pixel data of the target pixel 5c is also output to the defect correction unit 144, and the pixel data of the surrounding eight pixels 3a, 5a, 7a, 3c, 7c, 3e, 5e, and 7e around the target pixel 5c is an intermediate value calculation unit. Also output to 145.

  The comparison unit 142 compares the pixel value of the target pixel 5c with the pixel values of the surrounding eight pixels 3a, 5a, 7a, 3c, 7c, 3e, 5e, and 7e, and outputs the result to the pattern determination unit 143. The The pattern determination unit 143 determines whether or not the target pixel 5c is a defective pixel based on the result from the comparison unit 142, and outputs the result to the defect correction unit 144 (steps S102 to S105 in FIG. 10). .

  On the other hand, the intermediate value calculation unit calculates two intermediate values for determining the median value of the pixel values of the surrounding eight pixels 3a, 5a, 7a, 3c, 7c, 3e, 5e, and 7e around the target pixel 5c. That is, in this embodiment, since the pixel data is an even number, two pixel values for calculating the median value are selected and output to the defect correction unit 144, and the average value is obtained as the median value as will be described later. (Step S107 in FIG. 10).

  When it is determined that the target pixel 5c is not a defective pixel based on the determination result from the pattern determination unit 143, the defect correction unit 144 outputs the pixel data of the input target pixel 5c as it is, and the defective pixel Is determined, the pixel data is replaced with the average value of the two pixel values input from the intermediate value calculation unit 145 and output. As described above, the pixel value of the defective pixel is corrected, and the corrected image data is output to a user interface such as a display.

  In the detection of defective pixels in the present embodiment, the first defect detection pattern and the second defect detection pattern are used. The first defect detection pattern corresponds to a pattern whose pixel value distribution is estimated to be a defective pixel, and the second defect detection pattern corresponds to a pattern whose pixel value distribution is estimated to be a normal pixel.

  FIG. 5 shows a first defect detection pattern of the present embodiment. In the present embodiment, eight patterns presumed to be defective pixels are prepared in advance. In the pattern shown in the upper left of FIG. 5, the difference (absolute value) in pixel value between the target pixel 5c and the pixel 3c on the left side (in FIGS. 3 and 5) is less than (or less than) a predetermined threshold value. And the difference (absolute value) of the pixel value between the target pixel 5c and the other surrounding pixels 3a, 5a, 7a, 7c, 3e, 5e, and 7e excluding the pixel 3c is larger (or more) than the threshold value. Corresponds to the case. In other words, this corresponds to a case where the defective pixels are only 3c and 5c, and two pixels are expected to continue in the horizontal line direction. The threshold value is determined based on, for example, an average value of pixel data of eight surrounding pixels (for example, 50% of the average value).

  Similarly, in the second to fourth patterns from the left in the upper part of FIG. 5, the defective pixels are expected to be continuous for two pixels vertically or horizontally only 5c, 5e only, 5c, 7c only, 5a, 5c, respectively. It corresponds to. In other words, this corresponds to the case where the pixels whose difference from the pixel value of the target pixel 5c is equal to or less than the threshold are only 5e, 7c, and 5a, respectively.

  On the other hand, in the two patterns on the left side of the lower part of FIG. 5, when only three consecutive pixels 3c, 5c, and 7c on the horizontal line are expected to be defective pixels, the three consecutive pixels 5a and 5c on the vertical line This corresponds to the case where only 5e is expected to be a defective pixel. That is, this corresponds to the case where only 3c and 7c, and only 5a and 5e, respectively, are pixels whose difference from the pixel value of the target pixel 5c is equal to or less than the threshold value.

  The third pattern from the left in the lower part of FIG. 5 corresponds to a case where a defective pixel is expected to exist in a vertical and horizontal cross shape with the target pixel 5c as the center, and the difference from the pixel value of the target pixel 5c is equal to or less than the threshold value. This corresponds to the case where the number of pixels is only 3c, 7c, 5a, 5e. Further, the lower right pattern in FIG. 5 corresponds to a case where only the target pixel 5c is expected to be a defective pixel. That is, there are no pixels whose difference from the pixel value of the target pixel 5c is equal to or smaller than the threshold value in the eight pixels around the target pixel 5c.

  Next, an example of the second defect detection pattern is shown in FIG. The second defect detection pattern corresponds to a case where 2 × 2 pixels (4 pixels) of the same color including the target pixel 5c exist as a lump within a predetermined threshold. FIG. 6 illustrates a case where the target pixel 5c is the upper left pixel of the four pixels, and the absolute value of the difference between the pixel values of the three pixels 7c, 7a, and 5a and the pixel value of the target pixel 5c is within a threshold. In addition, when the pixel of interest 5c is an upper right pixel, when it is a lower right pixel, there may be a pattern of a lower left pixel (four patterns in total).

  In most cases, lines, dots, etc. appear in the captured image with vertical and horizontal 4 (2 × 2) pixels as the minimum unit. Therefore, when the target pixel is one of such 4 (2 × 2) pixels, there is a high possibility that it is a normal pixel. On the other hand, if the target pixel is not one of such four pixels, the target pixel is likely to be a defective pixel. Therefore, in the present embodiment, when the target pixel 5c does not correspond to any of the second defect detection patterns, it is regarded as a defective pixel.

  FIG. 7 schematically illustrates the configuration of the comparison unit 142 that performs comparison processing between the pixel value of the target pixel 5c and the pixel values of the surrounding eight pixels 3a, 5a, 7a, 3c, 7c, 3e, 5e, and 7e. As shown in FIG. 7, the pixel values of nine pixels 5 a, 5 c, 5 e, 7 a, 7 c, 7 e, 3 a, 3 c, and 3 e used for pattern determination and a predetermined threshold value are input to the comparison unit 142. Is done.

  A plurality of comparators are provided in the comparison unit 142, and a difference in pixel value between the target pixel 5c and the surrounding eight pixels 3a, 5a, 7a, 3c, 7c, 3e, 5e, and 7e is calculated. It is determined whether or not the absolute value is greater than the threshold value. The respective determination results are output to the pattern determination unit 143 as pixel value determination signals 5e_sig, 5a_sig, 7c_sig, 3c_sig, 7e_sig, 7a_sig, 3e_sig, 3a_sig. The pixel value determination signal is a logical value signal, for example, “1” when the absolute value of the difference between the pixel values is larger than the threshold value, and “0” when the absolute value is less than the threshold value.

  FIG. 8 is a block diagram schematically illustrating the configuration of the pattern determination unit 143 to which the pixel value determination signal from the comparison unit 142 is input. As illustrated, the pattern determination unit 143 includes a first pattern detection unit 143A that determines a first defect detection pattern and a second pattern detection unit 143B that determines a second defect detection pattern.

  In FIG. 8, a tilde symbol (˜) is a NOT operation, and a signal with a tilde symbol represents an inverted signal. For example, the logical expression shown at the top of the first pattern detection unit 143A in FIG. This is performed, and among the eight pixels around the target pixel 5c, only “1” is obtained when only the pixel value of the pixel 3c is within the threshold in relation to the pixel value of the target pixel 5c. “0”. That is, it is determined whether or not it corresponds to the upper left pattern in FIG.

  Similarly, the second to eighth logical expressions from the top determine whether or not they correspond to the second to fourth patterns from the upper left of FIG. 5 and the first to fourth patterns from the lower left, respectively. is there. The operation result of each logical expression is input to the OR circuit. If any one of the eight logical expressions is “1”, “1” is output as the defect signal 1 to the selector 143C, and “0” otherwise. Is output (step S102 in FIG. 10).

  On the other hand, the four logical expressions shown in the second pattern detection unit 143B are the pixel values between the three pixels other than the target pixel 5c and the target pixel 5c in 4 (2 × 2) pixels each including the target pixel 5c. It is “1” only when the absolute value of the difference is equal to or less than the threshold value. In order from the top, the target pixel 5c corresponds to the case of 4 pixels in the lower right, the lower left, the upper left, and the upper right. To do. The calculation result of each logical expression is input to the OR circuit, and the defect signal 2 is output from the OR circuit to the selector 143C via the inverter. That is, “1” is output to the selector 143C as the defect signal 2 only when the target pixel 5c is not included in any of the above four patterns, and “0” is output in other cases (step S103 in FIG. 10). .

  The selector 143C sequentially selects one of the defect signal 1 and the defect signal 2, and sets “1” as a defect detection signal when either the defect signal 1 or the defect signal 2 is at least “1”. 144 (steps S104 and S106 in FIG. 10). On the other hand, when both the defect signal 1 and the defect signal 2 are “0”, the defect detection signal is output as “0” to the defect correction unit 144 (step S105 in FIG. 10).

  FIG. 9 is a block diagram schematically showing the configuration of the defect correction unit 144. The defect correction unit 144 includes a selector 144A and an arithmetic unit 144B that calculates the average (median value) of the two pixel values output from the intermediate value (binary) calculation unit 145. As described above, the arithmetic unit 144B receives the intermediate value (binary value) of the pixel data of the surrounding eight pixels, obtains the average (median value), and inputs the result to the selector 144A (FIG. 10). Step S107).

  In addition, as described with reference to FIG. 4, the pixel data of the target pixel is also directly input from the memory unit 141 to the selector 144 </ b> A. The selector 144A can selectively output any one of the pixel data of the target pixel input from the memory unit 141 and the median data from the arithmetic unit 144B. The selection is performed by the pattern determination unit 143. It is controlled based on the defect detection signal from

  That is, when the pattern determination unit 143 determines that the target pixel is a defective pixel (when the defect detection signal is “1”, for example), the median data is selected and the center obtained from the eight pixels around the target pixel. The value is output as corrected pixel data of the target pixel (step S108 in FIG. 10). On the other hand, when it is determined that the target pixel is a normal pixel (when the defect detection signal is “0”, for example), pixel data of the target pixel input from the memory unit 141 is selected, and the selector 144A selects the target pixel. Pixel data is output as it is (step S109 in FIG. 10).

  As described above, the pixel defect correction processing operation of the present embodiment for one target pixel is completed, and the same processing is sequentially repeated for the newly input target pixel. Thus, dynamic pixel defect correction processing is performed on all pixel data of the photographed image.

  As described above, according to the present embodiment, it is not necessary to inspect the presence or absence of defects at the time of manufacturing, and it is not necessary to previously store the position of the defective pixel in the memory, so that the defective pixel can be corrected. In addition, since the pixel defect correction process is dynamically performed, it is possible to correct even a new defect that has been caused by aging or temperature change.

  In the present embodiment, since the defective pixel is determined based on the local distribution pattern of the defective pixel and the normal pixel, the defective pixel is detected even when the defective pixel continues in the vertical direction or the horizontal direction. This makes it possible to determine a defective pixel more accurately. Furthermore, in this embodiment, since the median value is used as the correction pixel data, even when a white defect or a black defect exists around the target pixel, the influence of these defective pixels can be given to the correction pixel data. Can be reduced.

  Further, the pattern used for determining the defective pixel is not limited to the one exemplified in the present embodiment, and a pattern that is not a defective pixel when a pixel within a threshold is present on the right diagonal or left diagonal, When pixels within the threshold value exist only in the right diagonal direction or the left diagonal direction, it is possible to adopt a pattern as a defective pixel. These patterns can be appropriately combined or used alone.

  In this embodiment, 3 × 3 pixels centered on the target pixel are targeted for dynamic pixel defect correction processing. However, by adding a line memory or flip-flop, a defective pixel is determined for a wider area. It is also possible to do this.

  In this embodiment, the dynamic pixel defect correction processing has been described as an example. However, for example, a captured image is temporarily stored in an image memory, and then the digital camera or a personal computer is used as necessary. It is also possible to correct the image data by applying the pixel defect correction processing of the embodiment. In such a case, it is possible to arbitrarily set an area used for determining a defective pixel.

It is a block diagram which shows the structure of the digital camera which is one Embodiment of this invention. It is a figure which shows typically a part of pixel arrangement | sequence of an image pick-up element. FIG. 3 is a diagram showing only a B pixel extracted from the array of FIG. 2. It is a block diagram which shows roughly the structure of a dynamic defect correction apparatus. It is a figure which shows the 1st defect detection pattern of this embodiment. It is a figure which shows the 2nd defect detection pattern of this embodiment. It is a block diagram which shows typically the structure of the comparison part which performs the comparison process between the pixel value of an attention pixel, and the pixel value of surrounding 8 pixels. It is a block diagram which shows typically the structure of the pattern determination part into which the pixel value determination signal from a comparison part is input. It is a block diagram which shows typically the structure of a defect correction part. It is a flowchart explaining the flow of a pixel defect correction process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 12 Image pick-up element 13 Image processing part 14 Dynamic defect correction apparatus 141 Memory part 142 Comparison part 143 Pattern determination part 144 Defect correction part 145 Intermediate value (binary) calculation part 143A 1st pattern detection part 143B 2nd pattern detection part

Claims (8)

Defective pixel determination means for determining whether or not the target pixel is a defective pixel;
Pixel value correction means for correcting the pixel value of the target pixel when the defective pixel determination means determines that the target pixel is a defective pixel;
The pixel in which the defective pixel determination unit determines whether or not the target pixel is a defective pixel based on at least one of a basic pattern of defective pixels and a basic pattern of normal pixels Defect correction device.   The defective pixel determining means determines whether or not the target pixel corresponds to the basic pattern based on a comparison of pixel values between the target pixel and surrounding pixels. Item 2. A pixel defect correction apparatus according to Item 1.   The pixel defect correction according to claim 1, wherein the defective pixel determination unit determines the target pixel based on both a basic pattern of the defective pixel and a basic pattern of the normal pixel. apparatus.   The pixel defect correction apparatus according to claim 1, wherein a correction pixel value used as a pixel value of a target pixel determined to be a defective pixel is calculated based on pixel values of pixels around the target pixel. .   The pixel defect correction apparatus according to claim 4, wherein the correction pixel value is a median value of pixel values of the surrounding pixels.   6. The pixel defect correction apparatus according to claim 5, wherein the surrounding pixels are eight pixels around the pixel of interest.   The defective pixel determination unit and the pixel value correction unit are sequentially executed on pixel data output from an image sensor, and correction of a defective pixel is performed simultaneously with an imaging operation by the image sensor. The pixel defect correction apparatus according to claim 1. A digital camera comprising the pixel defect correction device according to any one of claims 1 to 7.

Images