JP2017169012A - Image processing device, control method of image processing device, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct a pixel signal of a defective pixel due to a parallax image.SOLUTION: An imaging apparatus performs an image signal processing of an imaging image acquired by an imaging element 107 and a parallax image. A sensor part of the imaging element 107 comprises a plurality of photoelectric conversion parts, and receives a light beam passed through an eye part having a different imaging optical system. A defective pixel detection part 121a detects a defective pixel of the parallax image. An isolation defective pixel correction part 121b corrects an isolation defective pixel in which the defective pixel is not adjacent to a detected object pixel. The adjacent defective pixel correction part 121c corrects an adjacent defective pixel by using the pixel signal of the imaging image and the parallax image in a case where the defective pixel is adjacent to the detected object signal. In accordance with a detection result of the defective pixel detection part 121a, the isolation defective pixel correction and the adjacent defective pixel correction are executed.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing technique for an image signal having parallax.

  There is an imaging apparatus that can divide an exit pupil of a photographing lens into a plurality of pupil partial areas and simultaneously capture a plurality of parallax images corresponding to the divided pupil partial areas. Patent Document 1 discloses an imaging apparatus including a two-dimensional imaging element in which one microlens and a plurality of divided photoelectric conversion units are formed for one pixel. The divided photoelectric conversion units receive light passing through different pupil partial regions of the photographing lens through one microlens. A plurality of parallax images corresponding to each pupil partial region can be generated from the signal obtained by each photoelectric conversion unit. Patent Document 2 discloses that a captured image is generated by adding all the signals received by the divided photoelectric conversion units.

US Pat. No. 4,410,804 JP 2001-083407 A

  However, when a defective pixel is generated in a small part of the captured parallax image, there is a possibility that the image quality is deteriorated. An object of the present invention is to correct a pixel signal of a defective pixel related to a parallax image.

  An image processing apparatus according to an embodiment of the present invention includes an acquisition unit that acquires pixel signals of a captured image and a parallax image, a detection unit that detects a defective pixel of the parallax image, and a pixel detected by the detection unit. A first correction that corrects a signal using a signal of a pixel that is adjacent to the pixel that is not a defective pixel, and a pixel signal detected by the detection unit is a signal of the pixel of the captured image corresponding to the pixel The correction means which performs the 2nd correction | amendment corrected using is provided.

  According to the image processing apparatus of the present invention, it is possible to correct the pixel signal of the defective pixel related to the parallax image.

1 is a schematic configuration diagram illustrating an imaging apparatus according to a first embodiment of the present invention. It is the schematic of the sensor arrangement | sequence in 1st Embodiment. It is the schematic plan view and schematic sectional drawing of the pixel in 1st Embodiment. It is a schematic explanatory drawing of the pixel and pupil division in 1st Embodiment. FIG. 6 is a schematic explanatory diagram of pupil division, defocus amount, and image shift amount in the first embodiment. It is the schematic which illustrates the detection pixel and surrounding pixel in 1st Embodiment. It is the schematic which illustrates the parallax image arrangement | sequence and captured image arrangement | sequence in 1st Embodiment. It is the schematic which shows another example of the parallax image arrangement | sequence in 1st Embodiment, and a captured image arrangement | sequence. It is a main flowchart explaining the process in 1st Embodiment. It is a subflow chart explaining defective pixel correction processing in a 1st embodiment. It is the schematic of the sensor arrangement | sequence and pixel structure in 2nd Embodiment of this invention.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In each embodiment, an example in which the image processing apparatus according to the present invention is applied to an imaging apparatus such as a digital camera will be described. However, the present invention can be widely applied to information processing apparatuses and electronic devices that execute the following image processing.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration example of an imaging apparatus having an imaging element according to the first embodiment of the present invention. The first lens group 101 disposed at the tip of the imaging optical system (imaging optical system) is held by a lens barrel so as to be able to advance and retract in the optical axis direction. The aperture / shutter 102 adjusts the aperture diameter to adjust the amount of light at the time of shooting, and also has a function as an exposure time adjustment shutter at the time of still image shooting. The second lens group 103 moves forward and backward in the direction of the optical axis integrally with the diaphragm / shutter 102 and has a zooming function in conjunction with the forward / backward movement of the first lens group 101. The third lens group 105 is a focus lens that performs focus adjustment by advancing and retreating in the optical axis direction. The optical low-pass filter 106 is an optical element for reducing false colors and moire in the captured image. The image pickup element 107 includes, for example, a two-dimensional CMOS (complementary metal oxide semiconductor) photosensor and a peripheral circuit, and is arranged on the image forming plane of the image pickup optical system.

  The zoom actuator 111 performs a zooming operation by moving the first lens group 101 and the second lens group 103 in the optical axis direction by rotating a cam cylinder (not shown). The aperture shutter actuator 112 controls the aperture diameter of the aperture / shutter 102 to adjust the amount of photographing light, and controls the exposure time during still image photographing. The focus actuator 114 performs the focus adjustment operation by moving the third lens group 105 in the optical axis direction.

  The electronic flash 115 for illuminating the subject is used at the time of photographing, and a flash illumination device using a xenon tube or an illumination device including a continuous light emitting LED (light emitting diode) is used. An AF (autofocus) auxiliary light source 116 projects an image of a mask having a predetermined opening pattern onto a subject field via a light projection lens. This improves the focus detection capability for low-luminance subjects or low-contrast subjects.

  A CPU (Central Processing Unit) 121 that constitutes a control unit of the camera body has a control center function that controls various controls. The CPU 121 includes a calculation unit, ROM (read only memory), RAM (random access memory), A (analog) / D (digital) converter, D / A converter, communication interface circuit, and the like. The CPU 121 drives various circuits in the camera according to a predetermined program stored in the ROM, and executes a series of operations such as AF control, imaging processing, image processing, and recording processing.

  The defective pixel detection unit 121a detects whether there is a defective pixel in the captured image. The isolated defective pixel correction unit 121b corrects isolated defective pixels that are not adjacent to each other. The adjacent defective pixel correction unit 121c corrects adjacent defective pixels. Details of the defective pixel detection and correction process will be described later. The defective pixel detection unit 121a, the isolated defective pixel correction unit 121b, and the adjacent defective pixel correction unit 121c are illustrated as functional blocks realized by the processing of the CPU 121. In the present embodiment, a case will be described in which the processing of the CPU 121 is performed as a part of the imaging apparatus. However, the present invention is not limited to this, and the image processing and defective pixel detection unit 121a, the isolated defective pixel correction unit 121b, and the adjacent defective pixel correction unit 121c may be provided in an image processing device different from the imaging device, among the functions of the CPU 121. .

  The electronic flash control circuit 122 controls lighting of the electronic flash 115 in synchronization with the photographing operation in accordance with a control command from the CPU 121. The auxiliary light source driving circuit 123 controls the lighting of the AF auxiliary light source 116 in synchronization with the focus detection operation according to the control command of the CPU 121. The image sensor driving circuit 124 controls the image capturing operation of the image sensor 107, A / D converts the acquired image signal, and transmits it to the CPU 121. The image processing circuit 125 performs processing such as gamma conversion, color interpolation, JPEG (Joint Photographic Experts Group) compression of an image acquired by the image sensor 107 in accordance with a control command of the CPU 121.

  The focus drive circuit 126 drives the focus actuator 114 based on the focus detection result in accordance with the control command of the CPU 121, and moves the third lens group 105 in the optical axis direction to perform focus adjustment. The aperture shutter drive circuit 128 drives the aperture shutter actuator 112 in accordance with a control command from the CPU 121 to control the aperture diameter of the aperture / shutter 102. The zoom drive circuit 129 drives the zoom actuator 111 according to the zoom operation instruction of the photographer according to the control command of the CPU 121.

  The display unit 131 has a display device such as an LCD (Liquid Crystal Display), and displays information related to the shooting mode of the camera, a preview image before shooting and a confirmation image after shooting, a focus state display image at the time of focus detection, and the like. To do. The operation unit 132 includes a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, and the like as operation switches, and outputs an operation instruction signal to the CPU 121. The flash memory 133 is a recording medium that can be attached to and detached from the camera body, and records captured image data and the like.

  Next, the arrangement of the sensor units (photoelectric conversion units) of the image sensor according to the present embodiment will be described with reference to FIG. FIG. 2 is a schematic diagram illustrating the arrangement of the sensor units of the image sensor according to the present embodiment. The left-right direction in FIG. 2 is defined as the x-axis direction, the up-down direction is defined as the y-axis direction, and the direction perpendicular to the x-axis direction and the y-axis direction (the direction perpendicular to the paper surface) is defined as the z-axis direction. An imaging sensor array of a two-dimensional CMOS sensor (imaging device) is illustrated in a range of 4 columns × 4 rows, and a focus detection sensor array is illustrated in a range of 8 columns × 4 rows. An image sensor unit that outputs an image signal is composed of a plurality of photoelectric conversion units. In the present embodiment, a photoelectric conversion unit divided into two in a predetermined direction is illustrated.

The sensor group 200 of 2 columns × 2 rows includes a pair of sensor units 200R, 200G, and 200B. The sensor unit 200R (see the upper left position) has an R (red) spectral sensitivity, and the sensor unit 200G (upper right and lower left positions) has a G (green) spectral sensitivity. The sensor unit 200B (see the lower right position) has a spectral sensitivity of B (blue). Further, each sensor unit includes a first photoelectric conversion unit 201 and a second photoelectric conversion unit 202 arranged in 2 columns × 1 row. Each of the photoelectric conversion units has a function as a focus detection sensor that outputs a focus detection signal. In the example illustrated in FIG. 2, a large number of sensor units of 4 columns × 4 rows (8 columns × 4 rows of photoelectric conversion units) are arranged on a plane, so that captured image signals and focus detection signals can be acquired. In the imaging device, the period P of the sensor unit is 4 μm (micrometers), and the number of pixels N is 5575 columns × 3725 rows (approximately 20.75 million pixels). Further, the divided arrangement direction period _{P SUB} of the photoelectric conversion unit and 2 [mu] m, the corresponding number of pixels _{N S} horizontal 11150 rows × vertical 3725 lines (about 41.5 million pixels).

FIG. 3A shows a plan view when one sensor unit 200G in the image sensor shown in FIG. 2 is viewed from the light receiving surface side (+ z side) of the image sensor. The z axis is set in a direction perpendicular to the paper surface of FIG. 3A, and the near side is defined as the positive direction of the z axis. Also, the y-axis is set in the vertical direction perpendicular to the z-axis, the upper direction is the positive direction of the y-axis, the x-axis is set in the horizontal direction perpendicular to the z-axis and the y-axis, and the right direction is the positive direction of the x-axis It is defined as FIG. 3B shows a cross-sectional view when viewed from the −y side along the line aa in FIG. The sensor unit 200G has a microlens 305 for condensing incident light on the light receiving surface side (+ z direction) of each pixel, and includes a plurality of divided photoelectric conversion units. For example, the number of division in the x direction and N _H, the division number in y-direction and N _V. FIG. 3 illustrates an example in which the pupil region is divided into two in the horizontal direction, that is, a case where N _H = 2 and N _V = 1, and a photoelectric conversion unit 301 and a photoelectric conversion unit 302 are formed. The photoelectric conversion unit 301 corresponds to the first focus detection sensor, and the photoelectric conversion unit 302 corresponds to the second focus detection sensor.

  The photoelectric conversion unit 301 and the photoelectric conversion unit 302 are formed as, for example, a pin structure photodiode in which an intrinsic layer is sandwiched between a p-type layer 300 and an n-type layer. Alternatively, if necessary, the intrinsic layer may be omitted and formed as a pn junction photodiode. In each pixel, a color filter 306 is formed between the microlens 305 and the photoelectric conversion unit 301 and the photoelectric conversion unit 302. If necessary, the spectral transmittance of the color filter 306 may be changed for each photoelectric conversion unit, or the color filter may be omitted.

  The light incident on the sensor unit 200G is collected by the microlens 305 and further dispersed by the color filter 306, and then received by the photoelectric conversion unit 301 and the photoelectric conversion unit 302, respectively. In the photoelectric conversion unit 301 and the photoelectric conversion unit 302, a pair of electrons and holes (holes) are generated according to the amount of received light, separated by a depletion layer, and then electrons having a negative charge are transferred to an n-type layer (not shown). Accumulated. On the other hand, the holes are discharged to the outside of the image sensor through a p-type layer connected to a constant voltage source (not shown). Electrons accumulated in the n-type layer (not shown) of the photoelectric conversion unit 301 and the photoelectric conversion unit 302 are transferred to the capacitance unit (FD) via the transfer gate and converted into a voltage signal.

  FIG. 4 is a schematic explanatory diagram showing the correspondence between the pixel structure and pupil division. 4A and 4B are a cross-sectional view of the cross section taken along the line aa of the pixel structure shown in FIG. 3A when viewed from the + y direction, and an exit pupil plane of the imaging optical system (see the exit pupil 400). ) Is viewed from the −z direction. In FIG. 4, in order to correspond to the coordinate axis of the exit pupil plane, the x axis and the y axis in the cross-sectional view are shown reversed from the state shown in FIG. 3.

  The first pupil partial region 501 is substantially conjugated by the microlens 305 with respect to the light receiving surface of the photoelectric conversion unit 301 whose center of gravity is deviated in the −x direction. That is, the first pupil partial region 501 represents a pupil region that can be received by the first photoelectric conversion unit 301, and the center of gravity is biased in the + X direction on the pupil plane. In addition, the second pupil partial region 502 is substantially conjugated by the microlens 305 with respect to the light receiving surface of the photoelectric conversion unit 302 whose center of gravity is deviated in the + x direction. The second pupil partial region 502 represents a pupil region that can be received by the second photoelectric conversion unit 302, and the center of gravity is biased in the −X direction on the pupil plane. A pupil region 500 illustrated in FIG. 4 is a pupil region that can be received by the entire sensor unit 200G when the photoelectric conversion unit 301 and the photoelectric conversion unit 302 are all combined.

  Incident light is condensed at the focal position by the microlens. However, due to the influence of diffraction due to the wave nature of light, the diameter of the focused spot cannot be made smaller than the diffraction limit Δ, and has a finite size. The size of the light receiving surface of the photoelectric conversion unit is about 1 to 2 μm, whereas the condensing spot of the microlens is about 1 μm. Therefore, the first pupil partial region 501 and the second pupil partial region 502 in FIG. 4 that are in a conjugate relationship with the light receiving surface of the photoelectric conversion unit via the microlens are not clearly divided into pupils due to diffraction loss. , The light reception rate distribution (pupil intensity distribution).

The correspondence between the image sensor and pupil division is shown in the schematic diagram of FIG. The light fluxes that have passed through different pupil partial areas, ie, the first pupil partial area 501 and the second pupil partial area 502, enter each pixel of the image sensor at different angles. Incident light is received and photoelectrically converted by the photoelectric conversion unit 301 and the photoelectric conversion unit 302 that are divided into N _H (= 2) × N _V (= 1). In this embodiment, an example in which the pupil region is divided into two in the horizontal direction is shown, but pupil division may be performed in the vertical direction as necessary.

  As described above, the imaging device of the present embodiment has a structure in which a plurality of sensor units each provided with a plurality of photoelectric conversion units that respectively receive light beams passing through different pupil partial regions of the imaging optical system are arranged. For example, by adding and reading out signals from a plurality of photoelectric conversion units for each pixel of the image sensor, a captured image having a resolution of the effective number of pixels is generated. In this case, the captured image is generated by combining light reception signals of a plurality of photoelectric conversion units for each pixel. In another method, the first parallax image is generated by collecting the light reception signals of the photoelectric conversion unit 201 of the image sensor. A second parallax image is generated by subtracting the first parallax image from the captured image. If necessary, the light reception signals of the photoelectric conversion unit 201 of the image sensor may be collected to generate a first parallax image, and the light reception signals of the photoelectric conversion unit 202 may be collected to generate a second parallax image. For each different pupil partial region, one or more parallax images can be generated from the light reception signal of the photoelectric conversion unit.

  In the present embodiment, each of the captured image, the first parallax image, and the second parallax image is a Bayer array image. If necessary, the demosaicing process may be performed on the captured image of the Bayer array, the first parallax image, and the second parallax image.

Next, the relationship between the defocus amounts of the first parallax image and the second parallax image acquired by the image sensor and the image shift amount will be described.
FIG. 5B is a relationship diagram schematically illustrating the defocus amounts of the first parallax image and the second parallax image, and the image shift amount between the first parallax image and the second parallax image. is there. An imaging element (not shown) is arranged on the imaging surface 800, and the exit pupil of the imaging optical system is divided into the first pupil partial region 501 and the second pupil partial region as in the case of FIGS. 4 and 5A. Divided into 502.

  The size | d | of the defocus amount d represents the distance from the imaging position of the subject image to the imaging surface 800. The orientation is defined as a negative sign (d <0) in the front pin state where the imaging position of the subject image is closer to the subject side than the imaging surface 800, and a positive sign (d> 0) in the rear pin state opposite to this. To do. In an in-focus state where the image formation position of the subject image is on the imaging surface (in-focus position), d = 0. The position of the subject 801 shown in FIG. 5B indicates a position corresponding to the in-focus state (d = 0), and the position of the subject 802 exemplifies a position corresponding to the front pin state (d <0). . Hereinafter, the front pin state (d <0) and the rear pin state (d> 0) are collectively referred to as a defocus state (| d |> 0).

  In the front pin state (d <0), the luminous flux that has passed through the first pupil partial area 501 (or the second pupil partial area 502) out of the luminous flux from the subject 802 is once condensed and then the gravity center position G1 of the luminous flux. (Or G2) is spread around the width Γ1 (or Γ2). In this case, the image is blurred on the imaging surface 800. The blurred image is received by the first photoelectric conversion unit 301 (or the second photoelectric conversion unit 302) arranged in the imaging device, and a first parallax image signal (or a second parallax image signal) is generated. Therefore, the first parallax image (or the second parallax image) is an image of a subject image (blurred image) having a width Γ1 (or Γ2) at the center of gravity G1 (or G2) on the imaging surface 800. Stored in memory as data. The width Γ1 (or Γ2) of the subject image increases approximately proportionally as the magnitude | d | of the defocus amount d increases. Similarly, when the image shift amount of the subject image between the first parallax image and the second parallax image is denoted by “p”, the magnitude | p | is the magnitude of the defocus amount d | d | It increases with the increase. For example, the image shift amount p is defined as a difference “G1−G2” in the center of gravity position of the light beam, and its magnitude | p | increases approximately proportionally as | d | increases. In the rear pin state (d> 0), the image shift direction of the subject image between the first parallax image and the second parallax image is opposite to that in the front pin state, but there is a similar tendency.

  Therefore, in the case of this embodiment, the magnitude of the defocus amount of the imaging signal obtained by adding the first parallax image and the second parallax image or the first parallax image and the second parallax image increases. As a result, the magnitude of the image shift amount between the first parallax image and the second parallax image increases.

Hereinafter, the parallax image correction processing according to the present embodiment will be described.
From each of the pixel units of the image sensor 107, that is, a plurality of photoelectric conversion units that receive light beams that pass through different pupil partial regions of the imaging optical system, a captured image and one or more parallax images (for example, a first parallax image) Parallax image) is acquired. The acquired captured image and the first parallax image are input to the image processing circuit 125, and correction processing is performed based on the captured image. If necessary, the acquired captured image and parallax image may be stored in a recording medium of the recording apparatus. In this case, the stored captured image data and parallax image data are read from the recording medium and input to the image processing circuit 125.

  Depending on the circuit configuration and driving method of the image sensor, the captured image is normal due to a short circuit of the transfer gate, but a scratch signal may be generated in the first parallax image, for example, which may be a point scratch or a line scratch. As a countermeasure, point scratch information and line scratch information inspected in a mass production process or the like are recorded in advance in the image processing circuit 125 or the like. A parallax image correction process is performed using the recorded point flaw information and line flaw information. Further, if necessary, the point defect determination or the line scratch determination may be performed by inspecting the parallax image in real time.

  The defective pixel detection will be described with reference to FIG. FIG. 6 illustrates evaluation by calculating a difference value between an output value (first output value) of a detected pixel and an output value (second output value) of a neighboring pixel close to the detected pixel when performing defective pixel detection. It is explanatory drawing of the method to do. FIG. 6A illustrates a case where defective pixel detection is performed using an adjacent 5 × 5 pixel region. FIG. 6B illustrates an example in which defective pixel detection is performed using adjacent ± 3 row regions (regions of 7 × 7 pixels). The position of each pixel is expressed using integer variables i and j. In FIG. 6, the vertical pixel position is represented by a variable i, and the horizontal pixel position is represented by a variable j.

As one of the general methods for detecting defective pixels, a representative value obtained by selecting a neighboring pixel close to the pixel to be detected, or a difference value between a representative value calculated using the neighboring peripheral pixel and the defective detected pixel There is a method of using. This method is a method for evaluating the degree of divergence by normalizing the output values of a pixel to be detected and peripheral pixels adjacent to the pixel with a representative value. When the normalized value exceeds a predetermined threshold (denoted as Error), the detection target pixel is detected as a defective pixel. A portion indicated by (i, j) in FIG. 6A indicates a target pixel on which defective pixel detection is performed. The output value is expressed as S (i, j). Representative value in the region shown by FIG. 6 (A), the words 5 × 5 If the representative value a median value of the output values of the pixels, denoted this S _{typ (i,} j) and. Instead of the median value, an average value or the like may be used, and the representative value setting method is arbitrary. A general evaluation value (first evaluation value) for defective pixel detection is expressed as E, and the following conditional expression (1) is used by using a predetermined threshold Error.

  Since the left side of Equation (1) is normalized by the luminance, defective pixels can be detected with high accuracy if the change in luminance is within a range of several percent. However, the difference in transmittance of color filters such as R, G, and B and the difference in shading do not fall within the order of several percent. For this reason, it is difficult to guarantee the detection accuracy using Equation (1) in the state where there is an influence of shading in addition to the color filter. Normally, when performing defective pixel detection using Equation (1), set an evaluation threshold for each transmittance of the color filter, devise an optical system to reduce shading as much as possible, and reduce noise due to differences in the amount of received light To improve the flaw detection accuracy.

  By the way, when photographing at various exit pupil distances with an interchangeable lens camera or the like, defective pixel detection of an image in a state where shading has occurred must be performed in real time. In this case, the inspection must be performed in a state where the amount of received light is different for each image region, and the detection accuracy is different for each image region. As an effective method for solving this, there is a method of further normalizing the evaluation value of Expression (1) in consideration of the amount of noise change due to the change in the output value of the detection pixel.

It is known that pixel noise includes fixed noise and random noise, and the random noise changes with the square root of the output value. In the defective pixel detection, the reference output value is expressed as S _std, and the entire evaluation value (second evaluation value) is expressed as E _total . Of the evaluation value E _total , a random component that changes according to the output value is expressed as E _random, and a fixed component that does not change according to the output value is expressed as E _fixed . When detecting a defective pixel in consideration of a change in output value, the following conditional expression (2) is used using a predetermined threshold value Error.

となる。式（３）を、Ｅ_{ｒａｎｄｏｍ}＝Ｅ−Ｅ_{ｆｉｘｅｄ}として式（２）に代入し、更に式を整理すると、下記（４）式となる。

By a formula (2), it can be seen that has been normalized with the reference output value _{S std} representative value _{S typ.} Therefore, even if the representative value S _typ changes always can be inspected at a constant standard, even the light amount is changed for each area comparable defective pixel detection accuracy can be obtained. Since the evaluation value E calculated by the equation (1) is composed of a random component E _random and a fixed component E _fixed ,

It becomes. Substituting equation (3) into equation (2) as E _random = EE _fixed and further organizing the equation, the following equation (4) is obtained.

In Expression (4), the second evaluation value E _total is composed of the sum of the first term and the second term. The first term, the first evaluation value E, which is a term for dividing the larger value as the representative value S _typ decreases. The second term is a term whose value increases as the representative value S _typ decreases. For E _fixed , an allowable value may be set when setting the predetermined threshold value Error for defective pixel detection. Further, in view of the balance between the required detection accuracy of defective pixels and the calculation scale, only the terms normalized by the rate of change of the output value may be used as the evaluation value E of Equation (4), or all terms may be used. It may be used.

Up to now, the defective pixel detection focused on one pixel has been described, but the present invention can also be applied in the case of the linear defective pixel detection shown in FIG. 6B, and the applicable range is shown in FIG. Not sure. The representative value S _typ used when calculating a first evaluation value E, in order to improve the accuracy and defective pixel detection accuracy of the normalization, it is set according to the processing conditions as defect detection pixels. The processing conditions include, for example, a color filter disposed on the pixel, a pupil partial region through which a light beam received by the pixel passes, pixel addition, and the like.

  The isolated defective pixel correction unit 121b performs defective pixel correction when the pixels detected by the defective pixel detection are not adjacent to each other. The defective pixel correction at the isolated point is a defective pixel correction performed by a bilinear method or a bicubic method using pixel information adjacent to the defective pixel. By correcting the isolated defective pixel signal, a high-quality image can be generated.

  The defective pixel correction at the isolated point is effective when the pixel adjacent to the defective pixel is not a defective pixel. However, when defective pixels are adjacent, the adjacent defective pixel correction unit 121c performs defective pixel correction. A specific example will be described with reference to FIG. The row in FIG. 7 is represented by an integer variable j, and the column is represented by an integer variable i. The position of the j-th row and the i-th column is expressed as (j, i), the first parallax image at the position is expressed as A (j, i), and the captured image is expressed as I (j, i). . FIG. 7A is a schematic diagram showing the first parallax image array of the Bayer array in the range of j−1 to j + 1 and i−1 to i + 1. FIG. 7B is a schematic diagram showing the captured image array of the Bayer array in the range of j−1 to j + 1 and i−1 to i + 1.

Assume that all the images in the j-th row of the first parallax image are defective pixels, and the captured image is normal. In this case, a correction process for calculating a signal of the first parallax image A (j, i) at the correction position (j, i) is performed. The following equation (5) is a correction equation when the signal of the first parallax image A (j, i) is a pixel signal of a pixel having a spectral sensitivity of G (green).

Next, a correction formula when the signal of the first parallax image A (j, i) is a pixel signal of a pixel having a spectral sensitivity of R (red) or B (blue) is shown in the following formula (6).

  FIG. 8 illustrates a case where the signal of the first parallax image A (j, i) is a pixel signal of a pixel having a spectral sensitivity of B (blue). The definitions of variables j and i are the same as in FIG. FIG. 8A shows the first parallax image array of the Bayer array in the range of j−2 to j + 2 and i−2 to i + 2. FIG. 8B shows the captured image array of the Bayer array in the range of j−2 to j + 2 and i−2 to i + 2. If necessary, correction processing is similarly performed on the pixel signal of the second parallax image.

  When defective pixels are adjacent to each other as described above, defective pixel correction can be performed using the above equation using the pixel signals of the first parallax image and the captured image. However, when the defective pixels are not adjacent, defective pixel correction is performed by using a single image by a bilinear method, a bicubic method, or the like from the viewpoint of reduction of calculation load and correction accuracy. The defective pixel correction can be performed by a predetermined calculation method without using information of the imaging optical system. In this embodiment, hardware processing is performed in the image processing apparatus. This is because defective pixel correction can be performed at a higher speed than when software processing is performed by an external device (such as a PC). After extraction of defective pixels, defective pixel correction processing is executed in the imaging apparatus. However, the present invention is not limited to this, and at least a part of the defective pixel extraction processing and defective pixel correction processing may be realized by software processing.

  Next, the defect pixel correction of the parallax image in the present embodiment will be described with reference to the flowchart of FIG. Processing is started in S100, and imaging processing by the image sensor 107 is performed in S101. In the next step S102, the CPU 121 performs control to acquire image data of a captured image and a parallax image, and then proceeds to step S103. In S103, the defective pixel detection unit 121a detects defective pixels in each image, and the process proceeds to S104. In S104, defective pixel correction processing is executed. The defective pixel correction process will be described later with reference to FIG. After step S104, the process advances to step S105, and the CPU 121 executes a recording process for the image data corrected in step S104. Then, the process proceeds to S106, and a series of processing ends.

Next, defective pixel correction (FIG. 9: S104) will be described with reference to the sub-flowchart of FIG.
In S200, defective pixel correction processing is started, and the process proceeds to S201. In step S201, the CPU 121 controls the capture of image data, and the process advances to step S202. In S202, the reference of each pixel of the image data acquired in S201 is started. In the next S203, the defective pixel detection unit 121a performs defective pixel detection. As a result of the defective pixel detection, a determination process is performed as to whether or not the defective pixel is adjacent. If it is determined in S203 that the defective pixel is not adjacent, the process proceeds to S204, and if it is determined that the defective pixel is adjacent, the process proceeds to S205.

  In S204, the isolated defective pixel correction unit 121b executes an isolated defective pixel correction process, and proceeds to S205. In step S <b> 205, the CPU 121 determines whether or not all the pixel references in the image data have been completed. If it is determined that all pixel references have been completed, the process proceeds to S206. If it is determined that all pixel references have not been completed, the process returns to S202. The next pixel reference is performed, and the processing from S203 to S205 is repeated.

  In step S <b> 206, the CPU 121 determines whether isolated defect pixel correction has been completed for both the captured image and the parallax image. When it is determined that the isolated defect pixel correction is completed for both the captured image and the parallax image, the process proceeds to S207. If the isolated defective pixel correction of the captured image or the parallax image has not been completed, the process returns to S201 and continues.

In step S207, the CPU 121 performs control for capturing a captured image and a parallax image. In step S208, the CPU 121 starts referencing pixels of the parallax image, and proceeds to step S209. In S209, the adjacent defective pixel correction unit 121c executes an adjacent defective pixel correction process, and the process proceeds to S210. In S210, the CPU 121 determines whether or not all the pixel references of the image data have been completed. If all the reference of the pixels of the image data has been completed, the process proceeds to the return process of S211 to complete the defective pixel correction process and return to the main flowchart. If all the pixel references in the image data are not completed in S210, the process returns to S208. The next pixel reference is performed, and the processes of S209 and S210 are repeated.
According to the present embodiment, it is possible to correct a defective pixel signal of a parallax image when a defective pixel occurs in a part of the captured parallax image.

[Second Embodiment]
With reference to the schematic diagram of FIG. 11, an arrangement of the image sensor unit and the focus detection sensor unit of the image sensor according to the second embodiment of the present invention will be described. In addition, about the component similar to the case of 1st Embodiment, the code | symbol already used is used, those detailed description is abbreviate | omitted, and it demonstrates centering around difference.

FIG. 11A shows an imaging sensor array of a two-dimensional CMOS sensor (imaging device) in a range of 4 columns × 4 rows, and shows an array of photoelectric conversion units divided into 4 in a range of 8 columns × 8 rows. FIG.
In the sensor group 200 of 2 columns × 2 rows, a sensor unit 200R having a spectral sensitivity of R (red) is arranged at the upper left position. A sensor unit 200G having a spectral sensitivity of G (green) is arranged in the upper right and lower left, and a sensor unit 200B having a spectral sensitivity of B (blue) is arranged in the lower right. Further, each sensor unit includes a first photoelectric conversion unit 201, a second photoelectric conversion unit 202, a third photoelectric conversion unit 203, and a fourth photoelectric conversion unit 204 arranged in 2 columns × 2 rows.

By arranging a large number of 4 × 4 imaging sensor units (8 × 8 focus detection sensor units) shown in FIG. 11A in an array on a plane, a captured image signal (a plurality of photoelectric conversions) is arranged. Part addition signal) can be acquired. In the imaging device described in the present embodiment, the pixel period P is 4 μm, and the number N of pixels is 5575 columns × 3725 rows (approximately 20.75 million pixels). The period P _SUB of the divided photoelectric conversion units is 2 μm, and the corresponding number of pixels N _SUB is 11150 columns wide × 7450 rows long (about 83 million pixels).

  FIG. 11B is a plan view when one sensor unit 200G in the image sensor is viewed from the light receiving surface side (+ z side) of the image sensor. A first photoelectric conversion unit 301 to a fourth photoelectric conversion unit 304 that are divided into two in the x direction and two in the y direction are formed. 11B, the photoelectric conversion unit 301 is replaced with the third photoelectric conversion unit 303 and the photoelectric conversion unit 302 is replaced with the fourth photoelectric conversion unit 304 in FIG. 3B. Since it is the same as FIG.

  In the present embodiment, for each sensor unit of the image sensor, the signals of the photoelectric conversion units 201 to 204 are added and read out to generate a captured image having a resolution of N effective pixels. The first, second, and third parallax images are generated by individually collecting the received light signals of the first to third photoelectric conversion units 201, 202, and 203 of the image sensor. Alternatively, the fourth parallax image is generated by subtracting the image signals of the first parallax image, the second parallax image, and the third parallax image from the captured image signal. If necessary, the received light signals of the fourth photoelectric conversion unit 204 of the sensor unit of the image sensor may be collected to generate a fourth parallax image. For each different pupil partial region, one or more parallax images can be generated from the light reception signal of the corresponding photoelectric conversion unit.

In the present embodiment, each of the captured image, the first parallax image, the second parallax image, and the third parallax image is a Bayer array image. If necessary, the demosaicing process may be performed on the captured image of the Bayer array, the first parallax image, the second parallax image, and the third parallax image.
According to this embodiment, it is possible to correct defective pixel signals related to a plurality of parallax images.

[Third Embodiment]
Next, a third embodiment of the present invention will be described.
In the present embodiment, pixel signals of a captured image and a parallax image are acquired, and first and second defective pixel corrections are performed using two types of defective pixel information. The defective pixel information is detected at the time of product shipment or at a predetermined timing and stored in a memory, and is used as defective pixel information corresponding to a parallax image separately from the captured image. For example, when one of a plurality of parallax images is an A image, the memory holds defective pixel information of a point for the A image and defective pixel information of a line. Hereinafter, the defective pixel correction related to the A image will be described.

For example, as the first defective pixel correction, the CPU 121 refers to the defective pixel information of the line and performs line defective pixel correction on the linear region. In this correction, pre-interpolation using the immediately preceding line that is not defective is performed on each line. Next, the CPU 121 detects a defective pixel at each point (each pixel) in real time, and performs second defective pixel correction using the captured image with reference to the defective pixel information stored at the memory. Note that line defect pixel correction and point defect pixel correction are examples, and defective pixel correction can be performed on a region having a predetermined shape as necessary.
A signal of another parallax image (B image) is generated by subtracting the signal of the A image subjected to the defective pixel correction from the signal of the captured image (for example, the signal of the A + B image obtained by adding the A image and the B image) can do.

  In the present embodiment, the first defective pixel information is acquired, the first correction that corrects the signal of the defective pixel using the signal of the pixel that is not the defective pixel, and the second defective pixel information is acquired and the imaging is performed. A second correction is performed to correct the defective pixel signal using the pixel signal of the image.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

107 Image sensor 121 CPU
121a defective pixel detection unit 121b isolated defective pixel correction unit 121c adjacent defective pixel correction unit

Claims (9)

Acquisition means for acquiring pixel signals of a captured image and a parallax image;
Detecting means for detecting a defective pixel in the parallax image;
A first correction that corrects a pixel signal detected by the detection means using a signal of a pixel that is adjacent to the pixel and is not a defective pixel, and a pixel signal detected by the detection means is applied to the pixel. An image processing apparatus comprising: a correction unit that performs a second correction using a pixel signal of the corresponding captured image.   The correction unit performs the first correction when a defective pixel is not adjacent to the pixel detected by the detection unit, and the correction unit performs the first correction when the defective pixel is adjacent to the pixel detected by the detection unit. The image processing apparatus according to claim 1, wherein the second correction is performed. The acquisition means acquires a pixel signal of the first parallax image from the first photoelectric conversion unit by using an imaging device having a plurality of microlenses and a plurality of photoelectric conversion units corresponding to each microlens, Obtain a pixel signal of the second parallax image from the photoelectric conversion unit,
The correction unit uses the pixel signal of the captured image obtained by adding the pixel signal of the first or second parallax image, the pixel signal of the first parallax image, and the pixel signal of the second parallax image. The image processing apparatus according to claim 1, wherein the second correction is performed. The acquisition unit acquires the pixel signal of the parallax image from the first photoelectric conversion unit or the second photoelectric conversion unit by using an imaging device having a plurality of microlenses and a plurality of photoelectric conversion units corresponding to each microlens. And obtaining a pixel signal of the captured image from the first photoelectric conversion unit and the second photoelectric conversion unit,
The image processing apparatus according to claim 1, wherein the correction unit performs the second correction using a pixel signal of the parallax image and the captured image.   The image processing apparatus according to claim 1, wherein the correction unit performs the second correction after performing the first correction. Acquisition means for acquiring pixel signals of a captured image and a parallax image;
Storage means for storing first and second defective pixel information relating to the parallax image;
Acquiring the first defective pixel information from the storage means, and correcting a signal of the defective pixel using a signal of a pixel adjacent to the pixel that is not a defective pixel; An image characterized by comprising second correcting means for acquiring second defective pixel information and performing second correction for correcting a signal of the defective pixel using a signal of the pixel of the captured image corresponding to the pixel. Processing equipment.   An imaging apparatus comprising the image processing apparatus according to claim 1. A control method executed by an image processing apparatus that processes a captured image and a parallax image,
An acquisition step of acquiring pixel signals of the captured image and the parallax image;
A detection step of detecting defective pixels in the parallax image;
A first correction step of correcting a signal of a pixel detected in the detection step using a signal of a pixel that is not a defective pixel adjacent to the pixel;
A control method for an image processing apparatus, comprising: a second correction step of correcting a pixel signal detected in the detection step using a pixel signal of the captured image corresponding to the pixel. A control method executed by an image processing apparatus that processes a captured image and a parallax image,
An acquisition step of acquiring pixel signals of the captured image and the parallax image;
A first correction step of acquiring first defective pixel information related to the parallax image from a storage unit, and correcting a signal of the defective pixel using a signal of a pixel adjacent to the pixel and not a defective pixel;
A second correction step of acquiring second defective pixel information related to the parallax image from the storage unit and correcting a signal of the defective pixel using a pixel signal of the captured image corresponding to the pixel; And a control method for the image processing apparatus.

Images