JP2012134783A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor for highly accurately detecting a defective pixel of an imaging device by utilizing a pupil-divided luminous flux.SOLUTION: The image processor includes: focus degree evaluation means for processing images obtained from the imaging device including a plurality of first pixels and a plurality of second pixels for respectively photoelectrically converting pupil-divided first luminous flux and second luminous flux and evaluating a focus degree using the output of the first pixels and the second pixels; defect detection means for detecting the defective pixel of the imaging device; interpolation means for interpolating the defective pixel; and signal processing means for generating a photographed image using the output of the first pixels, the second pixels and the pixels obtained by interpolating the defective pixel. The defect detection means determines whether or not a first specified pixel or a second specified pixel is a defective pixel by comparing the first ratio of pixel values of the first specified pixel of the plurality of first pixels and the second specified pixel of the plurality of second pixels with the second ratio of the pixel values of a first vicinity pixel positioned near the first specified pixel and a second vicinity pixel positioned near the second specified pixel.

Description

  The present invention relates to an image processing apparatus that performs image processing by detecting defective pixels of an image sensor.

  Conventionally, as a method for correcting defective pixels in an image sensor, there is a method in which a defect is detected by photographing a detection chart in a manufacturing process, the detected defect position is stored in a product ROM, and defect correction is performed. On the other hand, pixels that were not defective in the manufacturing process of the image sensor may become defective pixels during actual use. In this case, it is necessary to detect defective pixels while photographing the subject. Conventionally, there has been a focus detection method in which a plurality of photodiodes are arranged for each microlens of the image sensor to divide the pupil of the light beam, calculate the defocus amount based on the phase difference, and perform focus detection.

  In Patent Document 1, a defective pixel candidate is detected using a difference between image values of a specific pixel and a neighboring pixel, and whether or not the defective pixel candidate is erroneously detected is determined based on whether it is an edge portion of the image. A configuration for preventing detection is disclosed. Patent Document 2 discloses a configuration in which a plurality of photodiodes are arranged for each microlens of the image sensor to perform pupil division. The focus detection is performed by calculating the defocus amount and the defocus direction from the shift amount and direction of the projection position of the subject projected on each pixel group divided by the pupil.

Japanese Patent No. 3984936 JP 2001-083407 A

  However, in the method disclosed in Patent Document 1, erroneous detection occurs with respect to a defective portion that is not an edge of an image. In this method, if the threshold value for determining whether or not the pixel is a defective pixel is increased, noise is increased due to defect detection failure. Conversely, if the threshold value is decreased, the undulation of the luminance that should originally be present in the image is caused by erroneous detection. Corrects and disappears. On the other hand, in a configuration in which a plurality of photodiodes are arranged per microlens of an image sensor as in Patent Document 2 and pupil division of a light beam is performed, a defective pixel detection method using characteristics of the configuration has not been proposed.

  Therefore, the present invention provides an image processing apparatus and an image processing method for detecting defective pixels of an image sensor with high accuracy using a pupil-divided light beam.

  An image processing apparatus according to one aspect of the present invention includes a plurality of first pixels that photoelectrically convert a first light beam and a second light beam by subjecting a light beam from an imaging optical system to pupil division and the first pixel. An image processing apparatus that processes an image obtained using an imaging device including a plurality of second pixels that photoelectrically convert two light beams, and using outputs of the plurality of first pixels and the plurality of second pixels A degree-of-focus evaluation means for evaluating the degree of focus; and a defect detection means for detecting a defective pixel of the image sensor when the degree of focus evaluated by the degree-of-focus evaluation means is a predetermined value or more; Interpolation means for interpolating the defective pixels detected by the defect detection means, the plurality of first pixels, the plurality of second pixels, and pixels obtained by interpolating the defective pixels by the interpolation means Signal processing means for generating a captured image using the output The defect detection means includes a first ratio of pixel values of a first specific pixel of the plurality of first pixels and a second specific pixel of the plurality of second pixels, and a first ratio located in the vicinity of the first specific pixel. The first specific pixel or the second specific pixel is the defective pixel by comparing a second ratio of pixel values of one neighboring pixel and a second neighboring pixel located in the vicinity of the second specific pixel It is determined whether or not.

  An image processing method according to another aspect of the present invention includes a plurality of first pixels that photoelectrically convert the first light flux by obtaining a first light flux and a second light flux by dividing a light flux from an imaging optical system into a pupil. An image processing method for processing an image obtained by using an imaging device including a plurality of second pixels that photoelectrically convert a second light beam, using outputs of the plurality of first pixels and the plurality of second pixels. A focus degree evaluation step for evaluating a focus degree, a defect detection step for detecting a defective pixel of the image sensor when the focus degree is a predetermined value or more, and an interpolation step for interpolating the defective pixel; A signal processing step of generating a photographed image using an output of the pixels obtained by interpolating the plurality of first pixels, the plurality of second pixels, and the defective pixels, and the defect detecting step A first specific pixel of the plurality of first pixels and the plurality of first pixels. The first ratio of the pixel value of the second pixel to the second specific pixel, and the first neighboring pixel located in the vicinity of the first specific pixel and the second neighboring pixel located in the vicinity of the second specific pixel By comparing with a second ratio of pixel values, it is determined whether or not the first specific pixel or the second specific pixel is the defective pixel.

  Other objects and features of the present invention are illustrated in the following examples.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus and image processing method which detect the defective pixel of an image pick-up element with high precision using the pupil-divided light beam can be provided.

1 is a block diagram of an image processing apparatus in Embodiment 1. FIG. FIG. 2 is a diagram illustrating a pixel structure of an image sensor in Example 1. FIG. 2 is a diagram illustrating a pixel array of an image sensor in Embodiment 1. 4 is a sub-block diagram of a second defect detection circuit in Embodiment 1. FIG. 2 is a circuit diagram of a first defect detection circuit in Embodiment 1. FIG. 4 is a sub-block diagram of a second defect detection circuit in Embodiment 1. FIG. It is explanatory drawing of the seeding principle in Example 1. FIG. In Example 1, it is explanatory drawing of the ratio of A pixel and B pixel by shading. 3 is a circuit diagram of a focus degree evaluation unit in Embodiment 1. FIG. In Example 1, it is a related figure of the focused pixel and reference pixel by a 1st defect detection circuit. In Example 1, it is a related figure of the focused pixel and reference pixel by a 2nd defect detection circuit. In Example 1, it is a graph which shows the relationship between the focus degree (defocus amount) and the look-up table of the threshold value by the 2nd defect detection circuit. In Example 1, it is a relationship figure of the position of a screen, and a vignetting shape. FIG. 3 is a diagram illustrating a relationship between a second defect detection circuit and pixel positions in Example 1. 6 is a structural diagram of a pixel of an image sensor in Example 3. FIG. 10 is a flowchart of an image processing method in Embodiment 2. FIG. 6 is an optical structural diagram of a system in Example 4. FIG. 6 is an explanatory diagram for distinguishing between defective pixels and normal pixels in the first embodiment. FIG. 10 is an array diagram of a color filter of four-divided pixels in Example 3. 4 is a block diagram of a second defect detection circuit in Embodiment 1. FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

  First, the configuration of the image processing apparatus according to the first embodiment of the present invention will be described with reference to FIG. The image processing apparatus of the present embodiment pupil-divides a light beam from the imaging optical system to obtain a first light beam and a second light beam, and photoelectrically converts a plurality of first pixels and second light beams that photoelectrically convert the first light beam. An image obtained using an imaging device including a plurality of second pixels is processed.

  FIG. 1 is a block diagram of an image processing apparatus 100 in the present embodiment. In FIG. 1, 101 is an imaging lens (imaging optical system), and 102 is an imaging device. The image sensor 102 includes a plurality of first pixels (A pixels) and a plurality of second pixels (B pixels) that perform photoelectric conversion of the first light beam and the second light beam obtained by pupil division of the light beam from the imaging lens 101. ). Reference numeral 103 denotes an AD converter, 105 denotes a defect position storage ROM, 104 denotes a defect mark insertion unit, 106 denotes a first defect detection circuit, and 107 denotes a defect interpolation circuit.

  Reference numeral 108 denotes a focus degree evaluation circuit (focus degree evaluation means) that evaluates the focus degree (focus state) using the outputs of the plurality of first pixels and the plurality of second pixels. Reference numeral 109 denotes a second defect detection circuit (defect detection means), which detects a defective pixel of the image sensor 102 when the focus degree evaluated by the focus degree evaluation unit 108 is equal to or greater than a predetermined value. Reference numeral 110 denotes a defect interpolation circuit (interpolation means) that interpolates the defective pixels detected by the second defect detection circuit 109. Reference numeral 111 denotes an addition Bayer circuit that adds pupil division pixels (first pixel and second pixel) to form a Bayer array. Reference numeral 112 denotes a video signal processing circuit (signal processing means), which uses a plurality of first pixels, a plurality of second pixels, and a pixel image obtained by interpolating defective pixels by the defect interpolation circuit 110. Is generated. Reference numeral 113 denotes a microcomputer that controls the entire image processing apparatus 100. In the image processing apparatus 100, light that has passed through the imaging lens 101 forms an image on the imaging element 102.

  Next, the pixel structure of the image sensor 102 will be described with reference to FIG. 2A shows a front view of the pixel of the image sensor 102, and FIG. 2B shows a cross-sectional view of the pixel. In FIG. 2, 201 is a microlens, and 202 and 203 are photodiodes, respectively. By reading out an image using two photodiodes for one microlens, pupil division is performed on the left and right in the figure. Hereinafter, the left and right pixels are referred to as A pixel and B pixel, and the images formed by collecting the left and right pixels (A pixel and B pixel) are referred to as A image and B image, respectively. FIG. 3 is a diagram showing a pixel array (color filter array) of the image sensor in the present embodiment. As shown in FIG. 3, the color filter array is added to each microlens 201 to be a normal Bayer array.

  As shown in FIG. 1, the output signal from the image sensor 102 is digitally converted by the AD converter 103 and input to the defect mark insertion unit 104. A signal indicating the defect position is output from the defect position storage ROM 105 (memory) to the defect mark insertion unit 104 in synchronization with the image. The defect position storage ROM 105 is constituted by a one-time ROM or a flash ROM, and stores a defect position of a pixel detected in the manufacturing process. The defect mark insertion unit 104 receives a signal from the defect position storage ROM 105 and sets a specific value indicating the signal if it is a defective pixel. In this embodiment, the value 0 (zero) is used as a mark indicating a defective pixel, but the present invention is not limited to this. When a value 0 indicating a defect mark is set for a specific pixel by a signal from the defect position storage ROM 105, the pixel is treated as a defective pixel in all subsequent processes. For a normal pixel that is not defective, it is replaced with 1 when 0 is set so that the value does not become 0.

  However, the signal from the image sensor 102 includes an unexposed portion called optical black, and an offset for absorbing variations in the optical black portion is included as a signal component. For this reason, normally, data near 0 is not input. As described above, the defective pixels detected in the manufacturing process are replaced with defect marks in the defect mark insertion unit 104. The output of the defect mark insertion unit 104 is input to the first defect detection circuit 106.

  Next, the circuit configuration of the first defect detection circuit 106 will be described with reference to FIG. FIG. 5 is a circuit diagram of the first defect detection circuit 106. In FIG. 5, reference numeral 501 denotes a signal input terminal, and reference numerals 505 and 506 denote two-line delay circuits. The output of the two-line delay circuit 505 is synchronized so that the signal input to the terminal 501 and the output of the two-line delay circuit 506 are the pixels on the two lines and the pixels on the two lines, respectively. Reference numerals 502, 511, 503, and 504 denote delay elements. The outputs of the delay elements 502, 503, and 504 and the output of the two-line delay circuit 505 are all added by the adders 507, 508, and 509 and are reduced to a quarter by the shift circuit 510. As a result, centering on the output timing of the delay element 511, the average value of the same color and same pupil in the vertical and horizontal directions is calculated as the output for the input signal from the terminal 501.

  Next, the positional relationship of the pixels will be described with reference to FIG. FIG. 10 is a diagram illustrating a reference pixel by the first defect detection circuit 106. Assuming that the center defect inspection pixel is the output of the delay element 511, the average of the same pupil-shaped pixels of the same color filter in the upper, lower, left and right directions is the output of the shift circuit 510. The subtractor 512 takes the difference between the output of the shift circuit 510 and the output of the delay element 511 and converts it to the absolute value of the difference by the absolute value circuit 516. That is, the absolute value circuit 516 outputs the absolute value of the difference between the average value of the same color and same pupil on the top, bottom, left and right and the pixel of interest. The output of the absolute value circuit 516 is input as one input of the comparator 513, and the threshold value is input from the outside to the terminal 515 as the other input.

  As described above, a detection signal indicating whether or not a specific pixel is a defective pixel is generated as a comparison result of the absolute value of the difference between the average value of the same color and the same pupil in the vicinity, the difference between the target pixel value, and the threshold value. Is output. By outputting a detection signal from the terminal 514, a defect mark is inserted in the subsequent stage of the first defect detection circuit 106 in the same manner as the defect mark insertion unit 104. Note that a value input as a threshold value to the terminal 515 is given from the microcomputer 113.

  The output of the first defect detection circuit 106 is input to the second defect detection circuit 109 and the defect interpolation circuit 107. Pixels determined to be defective pixels by the defect mark insertion unit 104 and the first defect detection circuit 106 are treated as pixels having no defect by being interpolated by the defect interpolation circuit 107. Note that the interpolation method by the defect interpolation circuit 107 is not deeply related to the essence of the present invention, and thus the description thereof is omitted here. The signal subjected to the interpolation processing by the defect interpolation circuit 107 is evaluated by the focus degree evaluation unit 108.

  Next, the principle of this embodiment will be described before the details of the in-focus level evaluation unit 108 and the second defect detection circuit 109 are described. Defective pixels include pixels that output fixed values such as white and black scratches, and pixels with different black levels and sensitivities compared to other pixels, making it difficult to detect pixels with different sensitivities and black levels. . The second defect detection circuit 109 in this embodiment facilitates detection of defective pixels that are difficult to detect, such as those with different sensitivities and black levels.

  FIG. 18 is an explanatory diagram for distinguishing between a defective pixel and a normal pixel, and only the A image signal and the B pixel (second pixel) signal generated only by the A pixel (first pixel) signal at the time of focusing. The B image signal generated in FIG. At the time of focusing, since the projection positions of the pupil-divided light beams (the first light beam and the second light beam) on the imaging surface coincide with each other, the A image and the B image are the same image. As shown in FIG. 18, there is a slight level difference between the A image and the B image because the shading characteristics are different between the A pupil and the B pupil. Here, each of the portions 1801 and 1802 is at a pinpoint higher level than the previous and next levels. The portion 1801 is considered to have a highlight portion in the subject itself because there is a pinpoint level difference in both the A and B images. On the other hand, in the portion 1802, only the B image is at a high level pinpoint, and it is impossible in principle that the same portion is not in the A image. Therefore, it can be determined that the portion 1802 is due to a defective pixel. In the present embodiment, the portion 1801 is not erroneously detected as a defect, and only the portion 1802 is detected as a defect.

  Next, the principle of the detection method by the second defect detection circuit 109 will be described in detail. The images obtained by the A image and the B image divided into pupils on the right and left sides have a property that they substantially coincide in the in-focus state and the projected positions of the images shift to the left and right in the out-of-focus state. A method of performing focus detection using this property is known. Even in the in-focus state, the A and B images have different shading characteristics, so that a slight level difference occurs. The level difference increases as the distance from the center of the lens, that is, as the image height increases.

  Next, the principle of shading will be described in detail with reference to FIG. FIG. 7A is a cross-sectional view of the lens, and a point x located at the center of the optical axis on the image sensor 704 and a point y at a high image height are indicated by arrows. FIG. 7B shows the positional relationship of the frames projected when the lens is viewed from the positions of the points x and y. In FIG. 7, reference numerals 701, 702, and 703 denote frames formed by a subject-side lens, a diaphragm, and a lens on the image sensor 704 side, respectively. When viewed from the point x on the image sensor 704, the frames 701, 702, and 703 are provided in concentric circles, and the shape through which light can pass is determined by the diaphragm 702. When viewed from the point y on the image sensor 704, the upper side is cut by the frame 703, the lower side is cut by the frame 701, and the side surface is cut by the frame 702. FIG. 7B shows the shape of the place where the light cut by the three frames can pass. This shape is generally called vignetting or vignetting.

  FIG. 8 is an explanatory diagram of the ratio of the A pixel and the B pixel by shading, and is a diagram in which photodiodes divided into two pupils are superimposed on the vignetting shape. Light reaches the photodiode only in the shaded area in FIG. In the state of FIG. 8, light reaches the photodiode 202 over a larger area than the photodiode 203. For this reason, even if the A image and the B image coincide with each other in a focused state, a level difference occurs due to the influence of vignetting (vignetting shape).

  FIG. 13 is a relationship diagram (characteristic diagram) between a location on the image sensor (screen position) and a vignetting shape. As the image height increases, the vignetting shape also becomes irregular, and the shape varies depending on the location. However, the vignetting shapes of neighboring pixels are very similar at any location. Thus, as shown in FIG. 8, a level difference occurs between the photodiodes 202 and 203, but a level difference occurs with the same characteristics in the neighboring pixels.

The second defect detection circuit 109 in the present embodiment uses the following two principles.
(1) At the time of focusing, the A image and the B image are overlapped to form the same image. Therefore, the level difference between the A pixel and the B pixel is dominant due to the vignetting shape.
(2) The vignetting shapes of neighboring pixels are very similar.
By utilizing the above two principles, pixel defects are detected in the present embodiment as follows. That is, the level ratio of the A pixel and the B pixel of the neighboring pixels is multiplied by the pixel value on the facing side of the same microlens as the target pixel, and this is used as the reference value. When the difference between the reference value and the signal value of the target pixel exceeds a predetermined threshold value, it is determined that the target pixel is a defective pixel. In addition, the threshold value is set to be lower as the degree of focus is higher, and the threshold value is set to be higher as the degree of focus is lower. In this embodiment, such an algorithm can be realized by the in-focus degree evaluation unit 108 and the second defect detection circuit 109.

  Next, the circuit configuration of the in-focus level evaluation unit 108 will be described in detail with reference to FIG. FIG. 9 is a circuit diagram of the in-focus level evaluation unit 108, which is illustrated with a circuit scale smaller than the actual size for the sake of simplicity. In FIG. 9, a signal input to the terminal 901 is an output signal of the defect interpolation circuit 107. Reference numeral 902 denotes a delay element group for referring to pixels at different positions in the same cycle. In this embodiment, six adjacent pixels are taken out simultaneously. The clock supplied to the delay elements 903 to 907 with respect to the delay element group 902 is gated by the EVEN / ODD signal input from the terminal 915 and becomes half, apparently two pixels of A pixel and B pixel. Each time is input, one cycle is processed. The delay elements 903 to 907 have the same circuit configuration as each other, and only two input pixel positions are different. From the delay elements 903 to 907, a signal obtained by squaring the difference between two pixel values and adding them for seven cycles is output.

  This circuit calculates a correlation amount by an SSD (Sum of Squared Difference) calculation method. The input of the delay element 905 is such that the A pixel and the B pixel correspond to the same microlens. On the other hand, the input of the delay element 904 is the B pixel of the micro lens adjacent to the A pixel, the delay element 903 is the B pixel of the further adjacent micro lens, and the delay elements 906 and 907 are the same B pixel as the delay element 905. In this case, a pixel shifted from the A pixel is input. The delay elements 903 to 907 output a correlation amount indicating the degree of coincidence in the range of the shift amount of ± 2 between the A image and the B image. In the SSD calculation method, the evaluation value decreases as the image matching degree increases. Reference numeral 909 denotes a circuit in which only one bit corresponding to a shift having a low evaluation value and a high image matching degree is set to 1 and the other bits are set to 0. As a result of comparison by a two-input comparator, 908 outputs 1 if the expected magnitude relationship and 0 otherwise. Only the latter stage logic circuit whose combination of output results matches the set value outputs 1. If the in-focus portion, the output indicating the shift amount 0 (terminal 912) becomes 1, and if the in-focus level decreases, the output corresponding to the place where the shift amount is large (output of the terminals 910, 911, 913, 914) Becomes 1.

  In FIG. 9, the delay amount in the delay element group 902 is set to 6 and the integration width in the delay elements 903 to 907 is set to 7 to simplify the explanation. It corresponds. The shift amount needs to be wide enough to cover the maximum shift amount depending on the base line length and the maximum defocus amount of the A and B images determined by the optical characteristics of the lens. The result of the focusing degree evaluation unit 108 is read by the microcomputer 113 and is also used for autofocus. Further, the output of the focus degree evaluation unit 108 is also input to the second defect detection circuit 109.

  Next, the second defect detection circuit 109 will be described in detail with reference to FIGS. 20, 4 and 6. FIG. 20 is a block diagram of the second defect detection circuit 109. In FIG. 20, 2004 to 2007 are circuits for generating reference values, and 2008 is a circuit that generates a mask signal by comparing an input signal input from a terminal 2001 with a predetermined threshold signal input from a terminal 2002. Circuit. 2009 is a delay circuit for synchronizing the input signal from the terminal 2001 with the circuit 2010. In 2010, a signal regarding the degree of focus (defocus amount) input from the terminal 2003 is used to determine whether or not the pixel is finally defective, and an output signal corresponding to the determination result is output to the terminal 2011. Circuit.

  FIG. 4 is a sub-block diagram of the second defect detection circuit 109, and is a circuit diagram of a circuit 2005 that generates a left reference value. In FIG. 4, 401 is a terminal for inputting a signal from the first defect detection circuit 106, and 402 to 405 are delay elements for delaying the input signal. Reference numeral 411 denotes a terminal for inputting a signal for distinguishing between the cycle of the A pixel and the cycle of the B pixel. The signal input from the terminal 411 operates at each cycle toggle. A terminal 408 outputs a signal as a calculation result.

  The signal output to the terminal 408 is an interpolation signal having a delay corresponding to the output timing delayed by the delay element 404. Here, if the output timing of the delay element 404 is x [n], the output timing of the delay element 405 can be considered as x [n−1], and the output timing of the delay element 403 can be considered as x [n + 1]. The output timing of the delay element 402 is x [n + 2], and the timing of the input signal from the terminal 401 is x [n + 3]. This circuit outputs a reference value to terminal 408 in each cycle. Switches 406 and 409 are switched by an input signal from the terminal 411, and inputs to the divider 410 and the multiplier 407 are switched. As a result, the arithmetic expression of the signal output to the terminal 408 is switched between the cycle in which the input signal from the terminal 411 is 1 and 0. If the cycle of the input signal from the terminal 411 is ODD and the cycle of 0 is EVEN, the arithmetic expression is expressed as follows.

EVEN: reference value x [n] = (x [n + 2] / x [n + 3]) * x [n + 1]
ODD: reference value x [n] = (x [n + 1] / x [n + 3]) * x [n−1]
With reference to FIG. 14, the correspondence between the above arithmetic expression and the pixel position will be described. At the EVEN timing, x [n] is B pixel, the corresponding A pixel is x [n + 1], the adjacent A pixel is x [n + 3], and the B pixel is x [n + 2]. On the other hand, in the ODD cycle, x [n] is an A pixel and the corresponding B pixel is x [n−1]. The adjacent A pixel is x [n + 2], and the B pixel is x [n + 1]. As described above, the circuit 2005 is configured such that the arithmetic expression changes every cycle because the A pixel and the B pixel are alternately arranged. Only the circuit 2005 shown in FIG. 4 functions as the present embodiment. However, when the adjacent pixel has low luminance or there is a lot of noise, it may be affected. For this reason, in this embodiment, an average value of values calculated from the top, bottom, left, and right in the subsequent circuit is used.

  FIG. 6 is a block diagram of the circuit 2010 in the second defect detection circuit 109. The signals input to the four terminals 601 to 604 are reference values calculated using the ratio of the upper, lower, left, and right pixels, respectively. While the circuit 2005 of FIG. 4 refers to the left pixel as the reference pixel of the ratio, the circuits connected to the inputs 602 to 604 are different in that they refer to the upper and lower right.

  FIG. 11 shows the positional relationship between the pixel of interest and the reference pixel in the second defect detection circuit 109. As shown in FIG. 11, when the target pixel is the pixel B0 (second specific pixel of the second pixel), the ratio of the pixel values of the pixel A0 (first specific pixel of the first pixel) and the pixel B0 is set to It is defined as 1 ratio. The pixels A1 to A4 and B1 to B4 adjacent to the pixels A0 and B0 are used as reference pixels, and the ratio of the average value of the pixel values of the pixels A1 to A4 and the pixels B1 to B4 is defined as the second ratio. In this embodiment, the reference pixel is not limited to the pixel adjacent to the pixels A0 and B0, and may be any pixel located in the vicinity.

  In FIG. 6, signals input from terminals 601 to 604 are added by adders 605 to 607 and multiplied by a quarter by a shift circuit 608 to obtain an average value (reference value) thereof. The signal input from the terminal 610 has a signal value synchronized with the reference value. The subtractor 609 calculates the difference between the reference value and the signal value, and the absolute value circuit 611 obtains the absolute value of the difference. On the other hand, the degree of focus is input to the terminal 612 from the degree-of-focus evaluation unit 108 in FIG. 9 and converted into a threshold value corresponding to the degree of focus by the lookup table 613.

  FIG. 12 is a graph showing the characteristics of the look-up table 613, where the vertical axis represents the degree of focus (defocus amount) as an input value, and the horizontal axis represents a threshold value as an output value. The input value (degree of focus) indicates that the closer to 0, the more focused, and the larger the value, the less focused. On the other hand, the output value (threshold value) is low when the focus is in focus, and high when the focus is not achieved. The threshold value thus generated and the output of the absolute value circuit 611 (the absolute value of the difference between the reference value and the signal value) are compared by the comparator 614, and the absolute value of the difference between the reference value and the signal value is obtained. When the threshold value is exceeded, the pixel is determined as a defective pixel.

  Using the reference value obtained by multiplying the ratio of the A image and the B image by the facing pixels, it can be recognized that there is a defective pixel in the portion 1802 in FIG. 18, but either or both of the A pixel and the B pixel are It cannot be distinguished whether it is a defect. As a result, both the A pixel and the B pixel are detected as defects as the output of the comparator 614.

  In FIG. 20, reference numeral 2008 denotes a mask generation circuit for solving the above problem. The mask generation circuit 2008 has the same configuration as the first defect detection circuit 106 in FIG. 5, and the input threshold value is stricter than the first defect detection circuit 106. Specifically, for the B image in FIG. 18, the threshold is so small that both portions 1801 and 1802 are detected as defects. However, no defect is detected in the portion 1802 for the A image. The output of the mask generation circuit 2008 is input from the terminal 616 to the circuit 2010, masked by the AND element 617, and finally output to the terminal 615 as a defect detection result.

  The comparator 614 does not detect both the A pixel and the B pixel as defects in the portion 1801 in FIG. 18, but detects both the A pixel and the B pixel as defects in the portion 1802. On the other hand, the mask generation circuit 2008 detects both the A pixel and the B pixel as defective pixels in the portion 1801, and detects only the B pixel as a defective pixel in the portion 1802. As a result, the defect detection signal that passes through the AND element 617 is only the B pixel in the portion 1802.

  As described above, the second defect detection circuit 109 includes the first specific pixel (A0) of the plurality of first pixels (A pixels) and the second specific pixel (B0) of the plurality of second pixels (B pixels). A first ratio of pixel values is calculated. The second defect detection circuit 109 includes a first neighboring pixel (A1, A2, A3, A4) located in the vicinity of the first specific pixel and a second neighboring pixel (B1, B2) located in the vicinity of the second specific pixel. A second ratio of pixel values to B2, B3, B4) is calculated. Then, the second defect detection circuit 109 compares the first ratio and the second ratio, and determines whether or not the first specific pixel or the second specific pixel as the target pixel is a defective pixel. Preferably, each of the first neighboring pixel and the second neighboring pixel includes a plurality of neighboring pixels (A1 to A4, B1 to B4), and the second ratio is calculated using an average value of pixel values of the plurality of neighboring pixels. The Note that the number of each of the first neighboring pixels and the second neighboring pixels is not limited to four pixels. For example, 8 pixels or 16 pixels may be used as the neighboring pixels in order to suppress the influence of noise.

  The second defect detection circuit 109 determines that the pixel of interest (first specific pixel or second specific pixel) is a defective pixel when the ratio between the first ratio and the second ratio exceeds a predetermined threshold value. Is determined. At this time, preferably, the threshold value is changed gently so that the threshold value decreases as the in-focus degree evaluated by the in-focus degree evaluation unit 108 increases.

  As described above, according to the present embodiment, it is possible to provide an image processing apparatus and an image processing method that can effectively detect defective pixels having a small difference from a normal signal level.

  Next, Embodiment 2 of the present invention will be described with reference to FIG. In recent digital cameras, there is a method in which an image sensor output is recorded as a file as it is at the time of shooting, and signal processing (image processing) is performed later using software of a personal computer. Software that performs such image processing is called RAW development software.

  In the first embodiment, the above-described image processing is performed by a circuit incorporated in the imaging apparatus. In the present embodiment, image processing is performed by RAW development software. FIG. 16 is a flowchart of the image processing method in the present embodiment. Each step of the flowchart of FIG. 16 is executed based on an instruction of a CPU (microcomputer) of a personal computer, for example.

  First, image processing is started in step S1601, and a RAW file is taken into the memory in step S1602. In step S1603, first defective pixel detection is performed. The first defective pixel detection is performed by the same method as the first defect detection circuit 106 in the first embodiment. In step S1604, it is determined whether or not the first defective pixel detection in step S1603 has been completed for all pixels. If the first defective pixel detection has not been completed for all the pixels, the process returns to step S1603, and the first defective pixel detection is performed for the incomplete pixels. On the other hand, when the first defective pixel detection is completed for all the pixels, the process proceeds to step S1605, and defective pixel interpolation processing is performed. The defective pixel interpolation process is performed by the same method as the defect interpolation circuit 107 in the first embodiment.

  Subsequently, in step S1606, focus degree determination is performed (focus degree evaluation step). The in-focus degree determination is performed by the same method as the in-focus degree evaluation unit 108 in the first embodiment. If the in-focus degree is equal to or greater than the predetermined value (H), second defective pixel detection is performed in step S1607 (defect detection step). The second defective pixel detection is performed by the same method as the second defect detection circuit 109 in the first embodiment. On the other hand, if the focus level is less than the predetermined value (L), the process proceeds to step S1608 without performing the second defective pixel detection.

  In the present embodiment, it has been described that the second defective pixel detection is performed in step S1607 only when the focus degree is equal to or greater than the predetermined value in step S1606. However, the present embodiment is not limited to this. It is not something. Depending on the degree of focus obtained in step S1606, the threshold value for detection as a defective pixel may be changed when the second defective pixel is detected in step S1607. At this time, the threshold value is gently changed in accordance with the degree of focus so that the threshold value is lowered as the degree of focusing increases (the threshold value is increased as the degree of focusing decreases). .

  Subsequently, in step S1607, it is determined whether or not the focus degree determination in step S1606 has been performed for all pixels. If the focus level determination has not been completed for all the pixels, the process returns to step S1606, and the focus level determination is performed for uncompleted pixels. On the other hand, if the in-focus degree determination has been completed for all the pixels, the process advances to step S1609 to perform defective pixel interpolation processing (interpolation step). The defective pixel interpolation process is performed by the same method as the defect interpolation circuit 110 in the first embodiment. Thereafter, video signal processing is performed in step S1610 (signal processing step).

  According to this embodiment, it is possible to provide an image processing method for effectively detecting defective pixels having a small difference from a normal signal level using RAW development software.

  Next, Embodiment 3 of the present invention will be described with reference to FIG. FIG. 15 is a structural diagram of the image sensor in the present embodiment. FIG. 15A is a front view of a pixel of the image sensor, and FIG. 15B is a cross-sectional view of the pixel. In FIG. 15, 201 is a microlens, and 202, 203, 204, and 205 are photodiodes, respectively. In this embodiment, as shown in FIG. 15, four pixels (photodiodes 202 to 205) are arranged for one microlens 201 to perform pupil division.

  FIG. 19 is an arrangement diagram of color filters (four-divided pixels) in the present embodiment. FIG. 19A shows the structure of the sensor, and the same color filter is attached to the same microlens 201. FIG. 19B shows a state where the values of the pixels divided into four are added for each microlens 201, and is the same as a normal Bayer array. Therefore, it can be processed as it is by a conventional image processing circuit. FIG. 19C shows a state in which the upper and lower pixels are added, which is equivalent to dividing the pupil left and right. FIG. 19D shows a state in which the left and right pixels are added, which is equivalent to dividing the pupil vertically. In this way, when the pupil is divided in the vertical direction or the horizontal direction by changing the pixel addition direction, the degree of focus can be accurately adjusted by one method even when the subject edge is only one of the vertical and horizontal sides. Can be determined.

  In the first embodiment, two ratios of the A pixel (first pixel) and the B pixel (second pixel) are used, and when any one pixel is a defective pixel, any pixel is a defective pixel. Cannot determine if there is. Therefore, it is necessary to use the difference from the neighborhood average. On the other hand, since there are four types of pixels in this embodiment, the ratio can be used in various combinations. For example, if the photodiodes 202, 203, 204, and 205 are defined as A, B, C, and D pixels (first, second, third, and fourth pixels), A: B, B: C , C: D, D: A, B: D, and C: A. Assuming that the D pixel is a defective pixel, the ratio of C: D, D: A, and D: C is different from that of neighboring pixels, but the ratio of other combinations is the same as that of neighboring pixels. Become. In this way, by performing pupil separation in four divisions, it becomes possible to specify defective pixels using only the ratio. In this embodiment, a defective pixel is detected by using only the degree of focus and the ratio of the pixel value by pupil type of the neighboring pixel and the pixel of interest.

  Next, Embodiment 4 of the present invention will be described with reference to FIG. FIG. 17 shows a parallel light region in the relay lens (a region where light spreading from the point light source at the object-side focal position is parallel), in which incident light is simultaneously separated into left and right images, and A image and B image are separated into different image sensors. FIG. Reference numeral 1701 denotes a mirror, 1702 denotes an aperture, and 1703 denotes a relay lens. The light that has become a parallel region by the relay lens 1703 is divided into left and right by the mirror 1701. The light divided by the mirror 1701 is reflected by the mirrors 1705 and 1704, respectively. The reflected light is guided to imaging lenses 1706 and 1707 and forms images on imaging elements 1708 and 1709, respectively. Thereby, an A image and a B image can be obtained simultaneously by two image sensors. In this embodiment, the A image and the B image may be exposed in a time division manner. It is possible to apply the defective pixel detection described in the above embodiments to the above configuration.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  For example, in each of the above embodiments, an example is shown in which pupil division is performed on one microlens with a plurality of photodiodes. However, if the signal is divided into pupils, light is shielded between the microlens and the photodiode. The same effect can be obtained by dividing the pupil signal. Moreover, in each Example, the image by a single plate image pick-up element is assumed, However, A 2 plate or 3 plate image pick-up element has an effect similarly. The pupil division may not be between the microlens and the photodiode. Further, in each of the above embodiments, the calculation is based on the coincidence of the pupil-divided images as a method for calculating the degree of focus. However, the degree of focus may be calculated by a contrast method or a hybrid with the contrast method. .

  According to each of the above-described embodiments, it is possible to provide an image processing apparatus and an image processing method that detect defective pixels of an image sensor with high accuracy by using pupil-divided light beams.

108: In-focus degree evaluation unit 109 ... Second defect detection circuit 110 ... Defect interpolation circuit 112 ... Video signal processing circuit

Claims (4)

The pupil beam of the imaging optical system is divided into pupils to obtain a first beam and a second beam, and a plurality of first pixels that photoelectrically convert the first beam and a plurality of second pixels that photoelectrically convert the second beam. An image processing apparatus that processes an image obtained using an imaging element including:
A focus degree evaluation means for evaluating a focus degree using outputs of the plurality of first pixels and the plurality of second pixels;
A defect detection means for detecting a defective pixel of the image sensor when the focus degree evaluated by the focus degree evaluation means is a predetermined value or more;
Interpolation means for interpolating the defective pixels detected by the defect detection means;
Signal processing means for generating a captured image using the plurality of first pixels, the plurality of second pixels, and the output of the pixel obtained by interpolating the defective pixel by the interpolation means;
The defect detection means includes a first ratio of pixel values of a first specific pixel of the plurality of first pixels and a second specific pixel of the plurality of second pixels, and a first ratio located in the vicinity of the first specific pixel. The first specific pixel or the second specific pixel is the defective pixel by comparing a second ratio of pixel values of one neighboring pixel and a second neighboring pixel located in the vicinity of the second specific pixel Whether or not
An image processing apparatus. Each of the first neighboring pixel and the second neighboring pixel includes a plurality of neighboring pixels,
The image processing apparatus according to claim 1, wherein the second ratio is calculated using an average value of pixel values of the plurality of neighboring pixels. The defect detection means includes
When the ratio between the first ratio and the second ratio exceeds a predetermined threshold, it is determined that the first specific pixel or the second specific pixel is the defective pixel,
The image processing apparatus according to claim 1, wherein the threshold value is changed so that the threshold value decreases as the focus degree evaluated by the focus degree evaluation unit increases. The pupil beam of the imaging optical system is divided into pupils to obtain a first beam and a second beam, and a plurality of first pixels that photoelectrically convert the first beam and a plurality of second pixels that photoelectrically convert the second beam. An image processing method for processing an image obtained using an imaging element including:
A focus degree evaluation step for evaluating a focus degree using outputs of the plurality of first pixels and the plurality of second pixels;
A defect detection step of detecting a defective pixel of the image sensor when the in-focus degree is a predetermined value or more;
An interpolation step of interpolating the defective pixels;
A signal processing step of generating a captured image using an output of the pixels obtained by interpolating the plurality of first pixels, the plurality of second pixels, and the defective pixels,
The defect detection step includes a first ratio of pixel values of a first specific pixel of the plurality of first pixels and a second specific pixel of the plurality of second pixels, and a first ratio located in the vicinity of the first specific pixel. The first specific pixel or the second specific pixel is the defective pixel by comparing a second ratio of pixel values of one neighboring pixel and a second neighboring pixel located in the vicinity of the second specific pixel Whether or not
An image processing method.