JP2018014625A - Image processing apparatus and control method thereof, imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve accuracy of defect detection of multiple image signals acquired by an image pickup device including multiple photoelectric conversion parts.SOLUTION: An image pickup device 102 includes multiple photoelectric conversion parts, and can read a first signal (A image signal) and a second signal (A+B image signal). An image signal processing unit 103 generates a third signal (B image signal) from the first and second signals. A first continuous defect detector 105 acquires first and third signals and detects continuous defects for each predetermined range. A second continuous defect detector 111 acquires first and third signals, and performs defect detection in a detection period corresponding to a focus detection region. A focus detector 106 performs focus detection by acquiring first and third signals, when the first continuous defect detector 105 or second continuous defect detector 111 does not detect a defect, and does not use the first and third signals for focus detection, when the first continuous defect detector 105 and second continuous defect detector 111 detected a defect.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a signal processing technique for a plurality of image signals acquired by an imaging device including a plurality of photoelectric conversion units.

  There is an imaging surface phase difference detection method for focus detection of an imaging device. In an imaging device having a pixel configuration including divided photodiodes as a plurality of photoelectric conversion units corresponding to one microlens, each photodiode receives light that has passed through different pupil partial regions of the imaging optical system. According to this configuration, the focus detection signal can be acquired simultaneously with the captured image. In Patent Document 1, in a configuration in which a signal obtained by adding the charges of each divided pixel is read out after reading a signal from one pixel among a plurality of divided pixels (divided pixels), a defect occurs in the focus detection pixel. A case detection and correction method is disclosed.

JP 2014-146023 A

In the technique disclosed in Patent Document 1, a row in which the level of one of the paired image signals is equal to or less than a threshold is detected as a defective row. For this reason, the defect of a divided pixel can be detected only when the entire row is a defective row.
An object of the present invention is to improve the accuracy of defect detection related to a plurality of image signals acquired by an image sensor including a plurality of photoelectric conversion units.

  An apparatus according to an embodiment of the present invention is an image processing apparatus that processes a plurality of image signals acquired by an imaging device having a plurality of pixels including a plurality of photoelectric conversion units, and each of the plurality of pixels Obtaining a first signal based on charges accumulated in a part of the plurality of photoelectric conversion units, and a second signal based on charges accumulated in the plurality of photoelectric conversion units, First signal processing means for performing signal processing to generate a third signal based on a difference between the first and second signals, and a first signal processing unit for detecting a defect in the first or third signal in a first detection period. A first detection means; a second detection means for detecting a defect of the first or third signal in a second detection period; and a second for obtaining the first and third signals and performing signal processing. Signal processing means. The second signal processing means outputs the result of signal processing by acquiring the first and third signals when the first or second detection means does not detect a defect, and outputs the first and third signals. When the second detection means detects a defect, the first and third signals are acquired and the result of signal processing is not output.

  ADVANTAGE OF THE INVENTION According to this invention, the precision of the defect detection which concerns on several image signals acquired by an image pick-up element provided with several photoelectric conversion parts can be improved.

It is a block diagram which shows the structure of Example 1 of this invention. FIG. 2 is a cross-sectional view and an array diagram illustrating a pixel portion of the image sensor according to the first embodiment. 3 is a timing chart showing signal waveforms of Example 1. FIG. 3 is a diagram illustrating a luminance signal waveform according to the first embodiment. FIG. 3 is a circuit diagram illustrating a configuration of a continuous defect detection circuit according to the first embodiment. FIG. 3 is a diagram illustrating a circuit configuration of a focus detection unit according to the first embodiment. 3 is a flowchart of focus detection and focus adjustment processing according to the first exemplary embodiment. 3 is a flowchart of focus detection calculation according to the first embodiment. It is a circuit diagram which shows the structure of the continuous defect detection circuit of Example 2 of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In each embodiment, an application example to an imaging apparatus is shown as an image processing apparatus that processes a plurality of image signals. For example, the imaging device includes a plurality of photoelectric conversion units that can receive light that has passed through different pupil partial areas of the imaging optical system, and the signal processing unit uses a plurality of image signals to detect the focus of the imaging optical system. Process.

[Example 1]
With reference to FIG. 1, an imaging apparatus 100 according to Embodiment 1 of the present invention will be described. The imaging lens unit 101 constitutes an imaging optical system and includes a plurality of lenses and optical members such as a diaphragm. For example, the angle of view is changed by driving the zoom lens, and the focus is adjusted by moving the focus lens.

  The image sensor 102 receives light from a subject imaged through the imaging lens unit 101, performs photoelectric conversion, and outputs an imaging signal. The image signal processing unit 103 acquires an image pickup signal from the image pickup element 102, and performs A image separation and B image generation processing described later. Hereinafter, among the pair of images acquired by the image sensor 102, the first image is referred to as an A image and the second image is referred to as a B image. An image obtained by combining the A image and the B image is referred to as an A + B image.

  The signal processing circuit 104 acquires the A + B image signal that is the output 108 of the image signal processing unit 103, and performs predetermined image processing. The predetermined image processing is gradation correction processing, conversion processing, or the like executed to generate a video signal for display or recording.

  The present embodiment includes a first continuous defect detection unit 105 and a second continuous defect detection unit 111, and acquires and detects the A image signal and the B image signal that are the outputs 109 and 110 of the image signal processing unit 103, respectively. Process. The focus detection unit 106 acquires the A image signal and the B image signal from the image signal processing unit 103 and performs focus detection by the imaging surface phase difference detection method. The control unit 200 calculates the defocus amount of the imaging lens unit 101 based on the focus detection result from the focus detection unit 106, and calculates the drive amount of the focus lens. Drive control of the focus lens is performed according to the drive amount calculated by the control unit 200, and a focus adjustment operation is performed.

  The pixel structure of the image sensor 102 will be described with reference to FIG. FIG. 2A is a cross-sectional view illustrating a pixel structure of the image sensor 102. FIG. 2B is a schematic diagram showing a pixel array. One pixel unit includes a microlens 201, a color filter 202, and a plurality of photoelectric conversion units 203 and 204. The first photoelectric conversion unit 203 is a photoelectric conversion unit corresponding to the A image, and an A image signal is acquired based on an output corresponding to the amount of charge accumulated in the photoelectric conversion unit in each pixel unit. The second photoelectric conversion unit 204 is a photoelectric conversion unit corresponding to the B image, and a B image signal is acquired based on an output corresponding to the amount of charge accumulated in the photoelectric conversion unit in each pixel unit.

  Since the photoelectric conversion units 203 and 204 are arranged for one microlens 201, parallax occurs between the A image generated from the output of the photoelectric conversion unit 203 and the B image generated from the output of the photoelectric conversion unit 204. . Therefore, focus detection is performed by detecting a shift in the projection position of an image having a parallax (parallax image).

  FIG. 2B shows an example in which the pixel portion including the photoelectric conversion portions 203 and 204 is an R, G, G, and B Bayer array. R represents the color of the red filter, G represents the color of the green filter, and B represents the color of the blue filter. In the image sensor 102, each pixel portion is regularly arranged on a two-dimensional array with four pixels as one set.

  The image sensor 102 according to the present exemplary embodiment reads the A + B image signal obtained by adding the charges of the A image and the B image after reading the pixel signal of the A image. The image signal processing unit 103 generates an A image signal, a B image signal, and an A + B image signal based on the signal read from the image sensor 102. For example, the B image signal can be generated by subtracting the A image signal from the A + B image signal.

  The image signal processing unit 103 has output terminals for outputting an A image signal, a B image signal, and an A + B image signal, respectively. The first output terminal corresponds to the output 108 of the A + B image. Since the output 108 is a state in which the pixel signals of the A image and the B image are added, it is equivalent to the image signal of the Bayer array. The output 108 is processed into a video signal by the signal processing circuit 104 and then sent to a subsequent circuit (not shown) for compression and recording processing. Since the circuit after the signal processing circuit 104 is not directly related to the contents of the present invention, the description thereof is omitted.

  The second output terminal of the image signal processing unit 103 corresponds to the output 109 of the A image. The third output terminal of the image signal processing unit 103 corresponds to the output 110 of the B image. The outputs 109 and 110 are subjected to detection processing by the focus detection unit 106 and the continuous defect detection units 105 and 111, respectively. The detection signals of the continuous defect detection units 105 and 111 are output to the AND operation unit 112. The AND operation unit 112 calculates a signal indicating the AND operation result (hereinafter referred to as a comprehensive defect detection signal) from each detection signal and outputs the signal to the focus detection unit 106. Details of the detection process will be described later.

  With reference to FIG. 3, the signal processing of the present embodiment will be described. FIG. 3 is a timing chart showing signals of each part. In the output signal 107 of the image sensor 102, a waveform 301 indicating the readout timing of the A image signal and a waveform 302 indicating the readout timing of the A + B image signal following the waveform are shown. The periods of the waveforms 301 and 302 correspond to one horizontal period (time interval of the horizontal synchronization signal) of the imaging signal, and pixel signals for one frame are transferred by repeating the period.

  The output 108 of the A + B image is obtained by passing the signal in the period of the waveform 302 out of the output signal 107 of the image sensor 102 and is output as the waveform 303. The A + B image signal is a Bayer array signal obtained by adding the pixel signals of the A image and the B image, and becomes a video signal when the signal processing circuit 104 performs signal processing.

  The output 109 of the A image is delayed by a delay circuit in the image signal processing unit 103 and output during the period of the waveform 304. The B image output 110 is generated by subtracting the delayed A image signal (see waveform 304) from the A + B image signal (see waveform 303) in the image signal processing unit 103, and is output during the period of the waveform 305. Is done. The periods of the waveforms 306 and 313 correspond to one horizontal period. Based on the output signal 107 of the image sensor 102, an A + B image signal, an A image signal, and a B image signal are each output within one horizontal period.

  Next, continuous defect detection will be described with reference to FIG. 4A and 4B, the horizontal axis represents the horizontal position of the pixel, and the vertical axis represents the signal level. FIG. 4A shows a signal for one horizontal period when there is no defect as the A + B image and a continuous defect occurs when the A + B image is separated. For example, a defect that an A + B image signal can be read normally but an A image signal is not correct may occur. This is because the A + B image signal is substantially output as the A image output 109 by short-circuiting the A image transfer circuit and the B image transfer circuit inside the image sensor 102. In this case, the signal obtained by adding the signals of the two photoelectric conversion units is normally output, but the signals of the individual photoelectric conversion units may be defective. 4A shows an amplitude 401 indicating the signal level of each pixel of the A + B image signal, an amplitude 402 indicating the signal level of each pixel of the A image signal, and an amplitude 403 indicating the signal level of each pixel of the B image signal. Show.

  A region 405 is a region in which the amplitude 401 of the A + B image signal and the amplitude 402 of the A image signal have the same signal level due to a continuous defect in which the A image transfer circuit and the B image transfer circuit are short-circuited. As a result, the level of the amplitude 403 of the B image signal is lowered to near zero. A region 406 is a region in a transition period that occurs in the process of gradually recovering the signal of the amplitude 402 of the A image signal to a correct value. Such a defect may occur when the A image signal readout circuit becomes electrically unstable. When returning to the normal state, the characteristics are gradually recovered. When the A image signal readout circuit is electrically restored to the original state, an amplitude that is not attributed to the subject is generated in the amplitude 403 of the B image signal.

  When performing the phase difference detection, the focus detection unit 106 extracts edge components in a specific frequency band using a bandpass filter, and calculates a phase with a high degree of coincidence by correlation calculation. At this time, if an edge component not derived from the subject is mixed into the A image and the B image, there arises a problem that an error in the calculation result becomes large. In this case, whether the level difference between the A image signal and the B image signal is caused by the defocused state relating to the subject or the instability of the A image signal readout circuit inside the image sensor as shown in FIG. It is difficult to distinguish what is caused by.

  In the region 407 in FIG. 4A, the focus detection is possible because the A image signal readout circuit is restored to a normal state. Here, FIG. 4B shows an example where no defect has occurred, that is, the levels of the A image signal and the B image signal are greatly different depending on the defocus state relating to the subject. In this case, it is necessary to handle it normally without detecting it as a defect.

  FIG. 4B shows an amplitude 408 of the A + B image signal, an amplitude 409 of the A image signal, and an amplitude 410 of the B image signal. The horizontal axis represents the horizontal position of the pixel and corresponds to the time axis of signal readout. The defect detection period 1 is shown as the first detection period 411, and the defect detection period 2 is shown as the second detection period 412. The defect detection period 2 is shorter than the defect detection period 1. A region 413 corresponds to the focus detection frame.

  In the region 413, the amplitude 409 of the A image signal is relatively large, and the amplitude 410 of the B image signal is relatively small. In the region 413 corresponding to the focus detection frame, it is difficult to distinguish from the defect state in the region 405 or 406 shown in FIG. On the other hand, if the defect state is evaluated for a wide area corresponding to the first detection period 411 that is sufficiently longer than the second detection period 412, the A image signal and the B image generated in the defocus state Less susceptible to signal imbalance.

  This embodiment aims to prevent the occurrence of false detection by detecting the defect state in the areas 405 and 406 shown in FIG. 4A and not using the image signal at that time for focus detection. That is, the purpose is to use the signal in the region 407 shown in FIG. 4A as an effective focus detection signal. Therefore, not only the evaluation in the local area, but also the defect detection is performed by evaluating the entire A line where the balance between the A image signal and the B image signal generated in the defocus state is less likely to occur. To do. Thereby, it becomes possible to prevent a detection error in a local detection area that is easily affected by defocusing.

  In this embodiment, one horizontal period is used as the first detection period 411 in FIG. 4B, and a period corresponding to the focus detection frame region 413 is used as the second detection period 412. In the first detection period 411 in FIG. 4B, the amplitude 409 of the A image signal and the amplitude 410 of the B image signal are substantially equal. On the other hand, in the case of FIG. 4A, since a continuous defect region 405 is included, there is a significant difference between the amplitude 402 of the A image signal and the amplitude 403 of the B image signal in the first detection period 411. There is a difference. Such a state is detected as a defect state.

  FIG. 5 is a circuit diagram illustrating a configuration example of the first continuous defect detection unit 105 and the second continuous defect detection unit 111. The first continuous defect detection unit 105 and the second continuous defect detection unit 111 have the same circuit configuration, but the detection period given from the terminal 503 and the set value of the detection threshold given from the terminal 504 are different. Therefore, in the following description, the circuit configuration is common to the two continuous defect detection units.

  The A image signal from the image signal processing unit 103 is input to the A image signal input terminal 501. A B image signal from the image signal processing unit 103 is input to the B image signal input terminal 502. The integrator 505 integrates the A image input signal, and the integrator 506 integrates the B image input signal. Integrators 505 and 506 include adders, switch elements, and flip-flops as delay circuits. The integrators 505 and 506 integrate the input signal when the input signal from the terminal 503 (hereinafter referred to as a detection period signal) becomes high (H) level, and integrate the input signal. L) When the level is reached, it is reset to zero.

  The difference absolute value calculation unit 507 calculates the absolute value of the difference between the output of the integrator 505 and the output of the integrator 506. The difference absolute value calculation unit 507 includes a subtracter and an absolute value circuit. The difference absolute value calculation unit 507 outputs a signal indicating the magnitude of the difference to the comparator 508. The comparator 508 compares the output of the difference absolute value calculation unit 507 with a signal input from the terminal 504 (hereinafter referred to as a threshold signal). The magnitude of the difference between the integral value of the A image and the integral value of the B image is compared with a set threshold value, and a signal of the comparison determination result is output to the holding circuit 509.

  The holding circuit 509 includes a switch element and a flip-flop. The holding circuit 509 outputs a signal obtained by delaying the input signal by one cycle to the terminal 510 during a period in which the detection period signal from the terminal 503 is at a high level. When the detection period signal becomes a low level, the holding circuit 509 holds the signal of the previous cycle and continues to output it to the terminal 510.

  The first continuous defect detection unit 105 and the second continuous defect detection unit 111 have different detection periods that are set. That is, the detection period signal and the threshold signal are signals having different values depending on the first detection period 411 and the second detection period 412. Since the first continuous defect detection unit 105 has a longer detection time than the second continuous defect detection unit 111 and targets a wide range, the influence of defocusing on the subject is also light. For this reason, the threshold value signal corresponding to the first detection period 411 can be a set value (threshold value) that increases the detection sensitivity.

  As described above, the continuous defect detection units 105 and 111 according to the present embodiment detect a luminance difference between the A image signal and the B image signal in each focus period (detection period). Therefore, according to the setting of the second detection period 412, the circuit of FIG. 5 operates in the period corresponding to the region 413 corresponding to the focus detection frame in FIG. 4B and the regions 405 and 406 in FIG. Detect defect status. On the other hand, in the case of setting the first detection period, the circuit of FIG. 5 detects a defect state in FIG. 4 (A), but in FIG. 4 (B), a significant difference appears between the integrated values of the A and B images. It is not detected as a defect state because it In the defect determination of this embodiment, it is assumed that a defect has occurred only when a signal that satisfies the determination condition is detected in both the detection result in the first detection period 411 and the detection result in the second detection period 412. To be judged.

  With reference to the timing chart of FIG. 3, each output of the 1st continuous defect detection part 105 and the 2nd continuous defect detection part 111 is demonstrated. The horizontal defect detection signal corresponds to the output of the first continuous defect detection unit 105, and the AF (autofocus) frame defect detection signal corresponds to the output of the second continuous defect detection unit 111. A period of the waveform 306 indicates a period in which the detection period signal input from the terminal 503 of the first continuous defect detection unit 105 is at a high level. A high level period 308 of the horizontal defect detection signal indicates that the output of the first continuous defect detection unit 105 is held at a high level at the falling edge of the signal in the period of the waveform 306 and a defect is detected. Since no defect is detected at the falling edge of the next waveform 313 in the horizontal period, the horizontal defect detection signal is at a low level.

  A high level period 307 of the AF frame period signal indicates a period in which the detection period signal input from the terminal 503 of the second continuous defect detection unit 111 is at a high level. A high level period 309 of the AF frame defect detection signal indicates that the output of the second continuous defect detection unit 111 is held at a high level in the period 307 and a defect is detected. In the next AF frame period, a defect is detected, and even in the period 310, the AF frame defect detection signal is at a high level.

  The total defect detection signal is a logical product signal of the horizontal defect detection signal and the AF frame defect detection signal calculated by the logical product calculation unit 112. Therefore, the total defect detection signal is at a high level only during the period 311 and is at a low (L) level during the period 312. The total defect detection signal is input to the focus detection unit 106.

  FIG. 6 is a block diagram illustrating a configuration example of the focus detection unit 106. The input terminal 601 is an A image signal input terminal, and the input terminal 602 is a B image signal input terminal. The input terminal 603 is an input terminal for a comprehensive defect detection signal, and a 1-bit signal indicating a defect is input to the microcomputer 607 of the control unit 200. The capture circuit 605 captures the A image signal and the B image signal and writes them into the memory 606. Each image signal is captured in accordance with the input of the AF frame period signal from the input terminal 604 and stored in the memory 606.

  A signal corresponding to the horizontal period is input to the input terminal 608 and is acquired by the microcomputer 607. When the processing in a certain horizontal period is completed, the microcomputer 607 processes the A image signal and the B image signal taken in the memory 606, and then waits for the next horizontal period. Thereafter, the same processing is repeated a predetermined number of times.

  The focus detection and focus adjustment processing will be described with reference to the flowchart of FIG. The following processing is realized according to a control program that the microcomputer 607 reads from the memory and executes. When focus detection starts in S701, a focus detection area (AF area) setting process in the imaging screen is performed in S702. In the next S703, reading from the image sensor 102 is started. In S704, focus detection is performed according to the A image signal, the B image signal, and the defect information. The defect information is information of a comprehensive defect detection signal, for example, information indicating a logical value “1” when a defect is detected and a logical value “0” when a non-defect is detected.

  In step S705, the microcomputer 607 calculates the driving amount of the focus lens based on the focus detection result, and performs drive control of the focus lens. In step S706, the microcomputer 607 determines whether to continue the process. If there is no change in operation such as transition to shooting operation or interruption of shooting (determination result: Yes), the process returns to S702 again to continue focus detection and focus adjustment processing. If it is determined in S706 that the shooting operation is to be switched or the shooting is interrupted (determination result: No), the process proceeds to S707 and the next operation is performed.

  FIG. 8 is a flowchart for explaining in detail the processing of S704 of FIG. In step S801, the focus detection calculation starts. In step S802, the microcomputer 607 refers to the total defect detection signal and determines whether there is a defect in the target row area. When it is determined that there is a defect in the area of the target row, the process proceeds to S805. If it is determined that there is no defect in the area of the target row, the process proceeds to S803. In step S803, the microcomputer 607 performs correlation calculation. For example, in the case of calculation by SAD (Sum of Absolute Difference), an array of absolute values of differences corresponding to the phases of the A image signal and the B image signal is calculated. In step S804, the microcomputer 607 adds the SAD array calculated in step S803 to the previous calculation result.

  In step S805, it is determined whether or not a predetermined number of rows have been processed for the A and B images. If the predetermined number of lines has not been processed, the process returns to S802, and the next line is processed. By repeating the processes from S802 to S805 a predetermined number of times, an array in which the arrays of SAD operations are added is created. In S805, when all the lines have been processed, the process proceeds to S806.

  If the total defect detection signal becomes high level in S802, that is, if it is determined that there is a defect, the correlation calculation in S803 and the correlation calculation result in S804 are not added. Therefore, an array of SAD operation results that does not include defective rows is created.

  In S806, the microcomputer 607 obtains the position of the minimum value of SAD and performs sub-pixel matching by parabolic fitting. The defocus amount is calculated by multiplying the obtained shift position by a magnification determined by optical conditions. When the calculation of the defocus amount is completed, the process returns in S807, and the process proceeds to S705 in FIG.

  In this embodiment, focus detection is performed using an effective A image signal and B image signal excluding a region having a high possibility of a defect. According to the present embodiment, it is possible to detect a defect of a divided pixel that occurs continuously in a part of one row, and thus it is possible to realize focus detection and focus adjustment with high accuracy.

  In this embodiment, SAD is used for correlation calculation, but the same effect can be obtained by using correlation calculation by Sum of Squared Difference (SSD) or Normalized Cross-Correlation (NCC). Moreover, although the example which performs the defect detection by comparing the absolute value of the difference between the integral value of the A image signal and the integral value of the B image signal with the threshold value has been described, the present invention is not limited to this example. For example, the same effect can be obtained even when a defect is detected by comparing the ratio between the integral value of the A image signal and the integral value of the B image signal with a threshold value. Although focus detection has been described as an example in which the above technique is applied to an imaging apparatus, this technique is also effective when a distance map is created based on acquired data. The distance map is composed of distance distribution data indicating the depth in the depth direction of each subject in the captured image, and is used in three-dimensional image processing, refocus processing, and the like.

[Example 2]
Next, Embodiment 2 of the present invention will be described with reference to FIG. The same components as those in the first embodiment are denoted by the same reference numerals as those used in the first embodiment, and detailed description thereof is omitted. Differences are mainly described.

  FIG. 9 is a circuit diagram illustrating a configuration example of the continuous defect detection units 105 and 111 according to the present embodiment. The difference from the first embodiment is a size ratio calculation circuit 901. In the first embodiment, the difference absolute value calculation unit 507 is used, but in this embodiment, the ratio between the larger signal value and the smaller signal value is calculated from two input signals using the magnitude ratio calculation circuit 901. calculate. That is, the magnitude ratio calculation circuit 901 includes outputs from the integrators 505 and 506, and includes a MAX circuit 901a that selects a larger signal value and a MIN circuit 901b that selects a smaller signal value. The division circuit 901c divides the output of the MAX circuit 901a by the output of the MIN circuit 901b and calculates the ratio between the two. The output of the magnitude ratio calculation circuit 901 is sent to the comparator 508, and compared with the threshold value from the terminal 504, so that the determination result is output to the holding circuit 509.

  According to this embodiment, when a defect is detected by comparing the ratio between the integral value of the A image signal and the integral value of the B image signal with a threshold value, the defect is continuously generated in a part of one row. A defect of a divided pixel can be detected.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 102 ... Image pick-up element 103 ... Image signal processing part 105 ... 1st continuous defect detection part 111 ... 2nd continuous defect detection part 112 ... AND operation part

Claims (10)

An image processing apparatus that processes a plurality of image signals acquired by an imaging device having a plurality of pixels including a plurality of photoelectric conversion units,
In each of the plurality of pixels, a first signal based on charges accumulated in some photoelectric conversion units of the plurality of photoelectric conversion units, and a second signal based on charges accumulated in the plurality of photoelectric conversion units. First signal processing means for performing signal processing for obtaining a third signal based on a difference between the first and second signals,
First detection means for detecting a defect in the first or third signal in a first detection period;
Second detection means for detecting a defect in the first or third signal in a second detection period;
Second signal processing means for acquiring the first and third signals and performing signal processing;
The second signal processing means outputs the result of signal processing by acquiring the first and third signals when the first or second detection means does not detect a defect, and outputs the first and third signals. An image processing apparatus characterized in that when the second detection means detects a defect, the first and third signals are acquired and the result of signal processing is not output.   The image processing apparatus according to claim 1, wherein the second detection period is shorter than the first detection period.   The first detection unit performs defect detection using a first threshold value, and the second detection unit performs defect detection using a second threshold value different from the first threshold value. The image processing apparatus according to claim 1 or 2.   4. The image processing apparatus according to claim 1, wherein the second signal processing unit performs calculation of phase difference detection between the first signal and the third signal. 5. . An image processing apparatus according to any one of claims 1 to 4,
An image pickup apparatus comprising the image pickup device for picking up an image of a subject through an image pickup optical system.   The imaging apparatus according to claim 5, wherein the second signal processing unit acquires the first and third signals and performs focus detection of the imaging optical system. The imaging device includes a plurality of microlenses and the plurality of photoelectric conversion units corresponding to the respective microlenses, and outputs the second signal after outputting the first signal,
The imaging apparatus according to claim 6, wherein the first signal processing unit outputs the first and third signals, which are image signals having parallax, to the second signal processing unit. The first detection means detects a defect for each horizontal period of an image signal by the image sensor,
The imaging apparatus according to claim 6, wherein the second detection unit detects a defect in the second detection period corresponding to a focus detection region. The first or second detection means includes
Integrating the first signal to calculate a first signal value and integrating the third signal to calculate a third signal value;
Defect detection is performed by calculating a magnitude of a difference between the first signal value and the third signal value, or a ratio between the first signal value and the third signal value and comparing it with a threshold value. The imaging device according to claim 6, wherein the imaging device is performed. A control method executed by an image processing apparatus that processes a plurality of image signals acquired by an image sensor having a plurality of pixels including a plurality of photoelectric conversion units,
In each of the plurality of pixels, a first signal processing unit includes a first signal based on charges accumulated in some photoelectric conversion units of the plurality of photoelectric conversion units, and the plurality of photoelectric conversion units. A first signal processing step of performing a signal processing for obtaining a second signal based on the accumulated charge and generating a third signal based on a difference between the first and second signals;
A first detection step of detecting defects in the first or third signal in a first detection period by a first detection means;
A second detection step of detecting defects in the first or third signal in a second detection period by a second detection means;
A second signal processing step in which a second signal processing means acquires the first and third signals and performs signal processing;
In the second signal processing step, when the first or second detection unit does not detect a defect, the second signal processing unit obtains the first and third signals and performs signal processing. And when the first and second detection means detect a defect, the second signal processing means does not output the signal processing result obtained by acquiring the first and third signals. For controlling an image processing apparatus.

Images