JP2017022528A - Image processing apparatus, control method therefor, program and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of correcting influences of saturation of a photoelectric conversion part upon a pupil divided image.SOLUTION: An imaging apparatus comprises: input means for inputting an image signal generated by receiving light fluxes passing different pupil regions of an imaging optical system by a plurality of photoelectric conversion parts forming a unit pixel and includes information about an angle and an intensity of the light flux incident to an imaging plane; combination determination means which exchanges a combination of signals received by the plurality of photoelectric conversion parts forming the unit pixel while using the information about the angle and the intensity of the incident light flux, thereby determining a combination of signals when focusing a specific subject; and correction means for correcting influences of saturation upon the plurality of photoelectric conversion parts caused by saturating any one of the plurality of photoelectric conversion parts forming the unit pixel during imaging on the basis of the determined combination of signals.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, a control method therefor, a program, and an imaging apparatus.

  2. Description of the Related Art Conventionally, a technique for acquiring a pupil division image (also referred to as a parallax image) using an image sensor in which a plurality of photoelectric conversion units are arranged for one microlens is known. When a plurality of photoelectric conversion units are arranged with respect to one microlens, the capacity that can be converted by each photoelectric conversion unit becomes small, and the signal of each photoelectric conversion unit may be easily saturated.

  When processing an image signal output from such an image sensor, it is known that a display image without parallax can be obtained by adding the outputs of photoelectric conversion units corresponding to the same microlens. However, when one of a plurality of arranged photoelectric conversion units reaches saturation, the addition result does not increase linearly, resulting in degradation of image quality. In response to such a problem, Patent Document 1 proposes a pixel structure in which charges leak to another photoelectric conversion unit corresponding to the same microlens when a certain photoelectric conversion unit reaches saturation. According to the proposed pixel structure, the amount of charge lost is reduced even when the pixel is saturated, so that the allowable saturation level of the image is improved when the outputs of the respective photoelectric conversion units are added.

JP 2013-84742 A

  However, since the allowable saturation level of the photoelectric conversion unit itself does not change, if the charge leaks to both photoelectric conversion units paired with the saturated photoelectric conversion unit, the original light intensity in the pupil division image is Become different strength. Therefore, when signal processing is performed on each pupil division image, a desired result may not be obtained due to the influence of saturation.

  The present invention has been made in view of the above-described problems of the prior art, and is an image processing apparatus capable of correcting the influence of saturation of a photoelectric conversion unit on a pupil divided image, a control method thereof, a program, and an imaging apparatus The purpose is to provide.

  In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is, an image signal generated by receiving a light beam passing through different pupil regions of the imaging optical system by each of a plurality of photoelectric conversion units constituting a unit pixel, and an angle and intensity of the light beam incident on the imaging surface By combining the input means for inputting the image signal including the information of the signal and the signals received by the plurality of photoelectric conversion units constituting the unit pixel by using the information on the angle and intensity of the incident light beam, a specific subject is obtained. In the case of focusing on a plurality of photoelectric conversion units, a combination determination unit that determines a combination of signals and a plurality of photoelectric conversion units based on the determined combination of signals due to saturation of any of a plurality of photoelectric conversion units that constitute a unit pixel at the time of imaging. Correction means for correcting the influence of saturation on the conversion unit.

  According to the present invention, it is possible to correct the influence of the saturation of the photoelectric conversion unit on the pupil divided image.

1 is a block diagram illustrating a functional configuration example of a digital camera as an example of an image processing apparatus according to an embodiment of the present invention. 2A and 2B are diagrams illustrating the configuration of an image sensor and the configuration of pixels according to the present embodiment. The figure explaining the refocus process which concerns on this embodiment The figure explaining an example of the saturation characteristic of a division | segmentation pixel A figure explaining an example of image collapse A figure explaining an example of vignetting The figure explaining the pixel correction process which concerns on this embodiment 7 is a flowchart showing a series of operations of refocus processing according to the first embodiment. 7 is a flowchart showing a series of operations of saturation correction processing according to the first embodiment. 7 is a flowchart showing a series of operations of pixel correction processing according to the first embodiment. FIG. 3 is a block diagram illustrating an example of a functional configuration of a digital camera according to a second embodiment 7 is a flowchart showing a series of operations of correction processing according to the second embodiment. 8 is a flowchart showing a series of operations of correction end determination processing according to the second embodiment.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, as an example of the image processing apparatus, an example in which the present invention is applied to an arbitrary digital camera capable of acquiring an output from an imaging element in which a plurality of photoelectric conversion units are associated with one microlens will be described. However, the present invention is not limited to a digital camera, and can be applied to any electronic device that can acquire an output from an image sensor in which a plurality of photoelectric conversion units are associated with one microlens. These devices may include, for example, electronic devices such as mobile phones, game machines, personal computers, clock-type or glasses-type information terminals, in-vehicle devices, medical devices, and the like.

(Configuration of digital camera 100)
FIG. 1 is a block diagram illustrating a functional configuration example of a digital camera 100 as an example of an image processing apparatus according to the present embodiment. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. May be. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject.

  The imaging optical system 101 includes an optical system such as a focus lens and a zoom lens, and forms a subject optical image by forming a light beam incident from the subject on the imaging surface of the imaging element 102.

  The imaging element 102 has a configuration in which a plurality of pixels each having a photoelectric conversion element are arranged two-dimensionally, and each pixel has a plurality of photoelectric conversion units. The imaging element 102 includes information on the intensity and angle of the incident light beam (also referred to as light space information) by separating and receiving the light beam passing through different pupil regions of the imaging optical system 101 by a plurality of photoelectric conversion units. Can output a signal. The image sensor 102 converts the output of the photoelectric conversion unit from analog to digital by an A / D conversion circuit, and outputs a digital signal (image signal) for each photoelectric conversion unit. Each pixel includes any color filter of R (red), G (green), and B (blue), and is arranged in a Bayer shape. The image sensor 102 outputs an image signal in the RGB format separated for each of these color components. Details of the image sensor 102 and the pixels will be described later.

  The control unit 103 includes, for example, a CPU or MPU, and develops a program stored in a ROM (not shown) in a work area of a RAM (not shown) and executes it, for example, according to the internal state of the digital camera 100. Control operations such as recording.

  The saturation detection unit 104 includes a detection circuit or a functional block, and detects the position of a saturated photoelectric conversion unit (also referred to as a saturation pixel) in the image signal based on the image signal output from the image sensor 102. Details of the operation of the saturation detection unit 104 will be described later.

  The correction amount calculation unit 105 includes an arithmetic circuit or a functional block, and calculates a correction value used for the correction process by a pixel correction process described later. Details of the operation of the correction amount calculation unit 105 will be described later.

  The saturation correction unit 106 includes an arithmetic circuit or a functional block, and corrects the image signal based on the correction value calculated by the correction amount calculation unit 105. The detailed operation of the saturation correction unit 106 will be described later.

  The combination determination unit 107 includes an arithmetic circuit or a functional block, and determines a combination of divided pixels to be focused on a predetermined subject. Details of the operation of the combination determination unit 107 will be described later.

  The refocus image generation unit 108 includes an arithmetic circuit or a functional block, and reconstructs (also referred to as refocus) image data acquired at an arbitrary focal plane by rearranging and adding the divided pixels. Details of the operation of the refocus image generation unit 108 will be described later.

(Configuration of the image sensor 102)
Next, the configuration of the image sensor 102 according to the present embodiment will be described with reference to FIGS.

  First, a part of the pixel array of the image sensor 102 is shown in FIG. As described above, in the image sensor 102, pixels having any of RGB color filters are arranged in a Bayer shape. The configuration of each pixel on the image sensor 102 is shown in FIGS. 2B and 2C. FIG. 2B shows a configuration in which one pixel on the image sensor 102 is observed from the front (the same direction as FIG. 2A). The micro lens 201 is a lens disposed in front of the pixel, and condenses the light beam that has passed through the imaging optical system 101. Each of the photoelectric conversion unit 202 and the photoelectric conversion unit 203 includes a photoelectric conversion region, and can photoelectrically convert each received light beam and output a signal individually. FIG. 2C shows a cross section of the pixel along the line P-P ′ shown in FIG. The color filter 204 transmits light of a specific wavelength among R, G, and B out of the light flux that has passed through the microlens 201.

  In the configuration of the image sensor 102 described above, a pixel is a set unit of elements 201 to 204 configured under one microlens on the image sensor 102 and may be referred to as a unit pixel. In addition, the photoelectric conversion unit corresponding to the position of the photoelectric conversion unit 202 in the pixel may be referred to as a divided pixel A, and the photoelectric conversion unit corresponding to the position of the photoelectric conversion unit 203 may be referred to as a divided pixel B.

  In an image configured by collecting only pixels of divided pixels A and an image configured by collecting pixels of only divided pixels B (also referred to as pupil-divided images), light beams that have passed through different areas on the exit pupil are independently received. Therefore, parallax occurs. Since the pupil division image has parallax, it can be used for focus detection processing, stereo matching processing, refocus processing, or the like. Further, if the outputs (pixel values) of the divided pixel A and the divided pixel B are added, the pixel value is the same as the pixel value obtained by a pixel having no pupil division. Therefore, normal signal processing can be performed on the pixel value obtained by adding the outputs of the divided pixels A and B, and a normal display image can be obtained. In addition, although the form which has two division | segmentation pixels with respect to one pixel was illustrated in FIG. 2, the number of the division | segmentation pixels with respect to one pixel is not limited to this. It is only necessary that the plurality of photoelectric conversion units for one pixel can independently receive light beams that have passed through different regions on the exit pupil. Further, in the present embodiment, the configuration based on the RGB Bayer array is exemplified, but a configuration in which photoelectric conversion units are stacked and RGB is output for each wavelength of light may be used.

(Refocus processing)
Next, the refocus processing performed by the refocus image generation unit 108 will be described with reference to FIG.

  FIG. 3A shows a relationship between incident light in a predetermined region of the image sensor 102 and a focal plane at the time of imaging. FIG. 3A shows a state in which a light beam enters each of the divided pixels 301 to 310 when the focal plane is at the position of the microlens 201. In this case, the light beams incident on different microlenses are indicated by different types of lines such as a solid line, a one-dot chain line, and a dotted line, and each of the light beams incident on the two divided pixels corresponding to the microlens is of the same type. Shown with lines. In this case, a focused image is formed on the imaging surface of the image sensor at the time of imaging.

  FIG. 3B schematically shows light beams received by the respective divided pixels in FIG. When the position of one microlens is used as a unit, the divided pixels A and the divided pixels B that are aligned are arranged vertically. That is, when the focal plane is at the position of the micro lens 201, for example, the positions of the divided pixel 301 and the divided pixel 302, the divided pixel 303 and the divided pixel 304, the divided pixel 305 and the divided pixel 306, and the like are the same micro lens position. It's all there. In this case, by adding the signals of the divided pixels A and the divided pixels B, it is possible to obtain an image (that is, an in-focus image) in which light beams indicated by the same type of line are imaged.

  FIG. 3C shows the correspondence between the incident light and the divided pixels on the focal plane (referred to as the virtual focal plane 1) when the focal plane position is changed hypothetically. Each of the divided pixels 301 to 310 actually receives the light beam in the state shown in FIG. In FIG. 3D, the position of the signal received in FIG. 3A is shifted and added, thereby reconstructing the state of the light beam incident on the virtual focal plane 1, and refocusing on the virtual focal plane 1. An example of generating an image is shown. In this example, the position of each pixel of one pupil division image (for example, an image composed of division pixels 301, 303, 305,...) Obtained in FIG. Are added to the pupil divided images (images formed from the divided pixels 302, 304,...). Thereby, image data in which light beams of different types of lines are imaged can be obtained. This is a refocus image equivalent to the image data that can be acquired on the virtual focal plane 1 shown in FIG.

  FIGS. 3E and 3F show an example in which addition is performed by shifting the position of the signal in a direction different from that in FIGS. 3C and 3D. The relationship between incident light and divided pixels on this focal plane (referred to as virtual focal plane 2) is shown. Thus, by shifting the position of the signal, the state of the light beam incident on the virtual focal plane 2 is reconstructed. Therefore, a refocus image focused on an arbitrary subject region can be generated by controlling the amount of signal shift and adding the shifted signals. In other words, an image focused on a specific subject can be generated by controlling the shift amount so that the degree of coincidence of the correlation image is the highest for the specific subject.

(Correction processing for saturated pixels)
Next, correction processing for saturated pixels according to the present embodiment will be described. Therefore, pixel saturation in the pixel structure of the image sensor 102 will be described with reference to FIG. FIG. 4 shows the relationship between incident light and output. The horizontal axis indicates the amount of light incident on the microlens, and the vertical axis indicates the pixel value output from the pixel. An output characteristic 401 indicates an output characteristic when the pixel is not divided. In this example, the output increases linearly until the saturation level is reached. When the light beam that has passed through the microlens is uniformly incident on each divided pixel, the output characteristics of the divided pixel are added as 401 when the outputs of the divided pixels are added.

  However, except for a state where the position of the microlens is the center of the image height and in focus, the light beam that has passed through the microlens does not enter the divided pixels equally. In pixels around the screen (that is, with a large image height), the light flux is incident on the optical axis of the microlens so that the light flux does not enter most of one divided pixel (for example, divided pixel B), and the other The luminous flux is incident on the divided pixels (divided pixels A). For this reason, the divided pixel (divided pixel A) on which the light beam is incident reaches saturation first. The output characteristics for the divided pixels are, for example, the output characteristics 403 of the divided pixels A and the output characteristics 404 of the divided pixels B. When the output characteristic 403 and the output characteristic 404 are compared, the divided pixel A first reaches the saturation level. For this reason, the characteristic of the pixel obtained by adding the divided pixel A and the divided pixel B becomes an output characteristic 402, and the saturation effect appears earlier than the output characteristic 401. In order to reduce the influence of such saturation, a known pixel structure shown in Patent Document 1 described above can be employed. That is, if a structure in which the charge that has reached saturation in the divided pixel A leaks into the divided pixel B, the sum of the outputs of the divided pixel A and the divided pixel B does not change. Becomes the output characteristic 401. Note that the characteristics of the divided pixel B in which charges leak from the divided pixels A are as shown by the output characteristics 405 in FIG.

  The above-described saturation affects, for example, the case of calculating a correlation between pupil division images. 5A to 5C show the waveforms of the image signals output from the respective divided pixels when the horizontal axis is the pixel position. In the figure, reference numeral 501 denotes an image by the divided pixel A, and 502 denotes an image signal by the divided pixel B.

  FIG. 5A shows an ideal image signal when saturation does not occur. Reference numerals 501 and 502 each have a pixel that outputs an output exceeding the saturation level. FIG. 5B shows a case where saturation occurs, and the output of each divided pixel is changed by the saturated and leaked charge, and the respective images are deformed (images are displaced). Is shown. FIG. 5C shows the difference in image between the divided image A and the divided image B when both the images shown in FIG. That is, the circled portion in FIG. 5C is a portion in which the waveform of the image signal is deformed due to image displacement, and an error occurs when performing correlation calculation by superimposing the images of the divided image A and the divided image B. appear. For this reason, it can cause image quality degradation of image data generated by refocus processing or stereo processing for performing correlation calculation.

  Next, correction processing for saturated pixels according to the present embodiment will be described with reference to FIGS. When superimposing the images of the divided pixel A and the divided pixel B, in principle, there is no parallax between the images of the divided pixel A and the divided pixel B in the focused state, and the divided pixel A and the divided pixel are divided. It also matches the added image of pixel B.

  However, when the pixel position is not the center of the image height, the balance of the light beams incident on the divided pixel A and the divided pixel B is not uniform due to the difference in the incident angle of the light beam and the vignetting, and saturation may occur in one of the divided pixels. . The influence of vignetting increases as the pixel position is further away from the center of the image height. Note that vignetting is a phenomenon in which the luminous flux is limited by the object side or the body side edge of the imaging optical system when the aperture is close to the full open, and is also referred to as vignetting.

  As described above, the incident angle and vignetting with respect to the pixel change according to the height of the image height. However, in the present embodiment, attention is paid to the fact that the incident angle and vignetting with respect to the pixel change gently as the image height increases from the center of the image height. That is, regarding a pixel having a certain image height, pixels in the vicinity thereof are similarly affected by the incident angle and vignetting, and therefore, it is utilized that they exhibit substantially the same characteristics. In other words, the light amount ratios (output ratios) of the divided pixel A and the divided pixel B in the in-focus region are the same if the divided pixel A and the divided pixel B are neighboring pixels. . For example, as shown in FIG. 6B, the neighboring pixels on the image sensor 102 shown in FIG. 6A have the same vignetting effect. If such a feature is used, in the saturation pixel correction process in the in-focus region, the light quantity ratio of the divided pixel A and the divided pixel B of the pixel to be corrected is set to a neighboring pixel where saturation is not generated. It may be corrected so as to be equal to the light quantity ratio.

The correction process for saturated pixels will be described more specifically. FIG. 7A shows a correction process for a region in a focused state. In this example, the fact that the divided pixel 704 is saturated and the charge leaks to the divided pixel 703 is indicated by an arrow in the drawing. Using the fact that the light quantity ratio between the divided pixel 703 and the divided pixel 704 is equal to the light quantity ratio between the divided pixel 701 and the divided pixel 702, the amount of charge leaked (or moved) from the divided pixel 704 to the divided pixel 703 Can be estimated.

704 + α: 703−α = 702: 701 (1)

In this way, the correction amount α that satisfies Equation 1 is obtained, and the outputs of the divided pixels 703 and 704 can be corrected.

  On the other hand, in a region that is not in focus, a pair of divided pixels does not refer to the same subject. Therefore, the combination of the divided pixels is changed to a combination of the divided pixels by changing the combination of the divided pixels using the principle of the refocus processing described above.

FIG. 7B shows the positional relationship when the image signal output from the divided pixel B is shifted by three pixels in the right direction on the paper and the combination of the divided pixels is rearranged. The positional relationship when a focused state is obtained for a saturated pixel and a pixel affected by the saturation pixel is shown. In the example of FIG. 7B, the saturation pixel (for example, the divided pixel 704), which is a combination of the physical divided pixels, and the pixel (for example, the divided pixel 703) affected by the shift are logically shifted by the number of shifts. Exists at a certain distance. Therefore, the divided pixel 704 and the divided pixel 703 can be corrected by obtaining a correction amount α that satisfies the following expression.

704 + α: 706 = 702: 705 (2)

In this way, by shifting the divided pixels (that is, changing the combination of the divided pixels) to create a focused state, the influence of saturation in the pupil divided image can be corrected for each output of each divided pixel. .

(A series of operations related to correction processing for saturated pixels)
Next, a series of operations for correction processing for saturated pixels will be described with reference to FIGS. This process is started when the control unit detects that the user has pressed a shutter button (not shown), for example. Further, this processing is realized by the control unit 103 developing and executing a program stored in a recording unit (not shown) in a work area of a storage unit (not shown).

  In step S <b> 801, the control unit 103 starts shooting processing and acquires an image signal output from the image sensor 102. The control unit 103 sets the imaging optical system 101 to a predetermined imaging condition by a user setting or automatically, and captures an object optical image formed on the imaging element 102 through the imaging optical system 101. The control unit 103 receives the image signal output from the image sensor 102 and outputs the image signal to each unit described later. Alternatively, the control unit 103 may temporarily store the input image signal in a storage unit (not illustrated), and each unit described later may input the image signal from the storage unit.

  In step S <b> 802, the control unit 103 controls a plurality of functional blocks such as the saturation detection unit 104 and executes correction processing. A series of operations related to the correction process will be described later.

  In step S <b> 803, the control unit 103 generates a refocus image using the refocus image generation unit 108. Specifically, the refocus image generation unit 108 inputs the image signal that has been subjected to the correction process in S802, performs the above-described refocus process, and obtains an image focused on a specific subject, that is, a refocus image. Generate. When executing the refocus processing, the refocus image generation unit 108 may perform the refocus processing after inputting a user operation for designating a specific subject. When the refocus image is output from the refocus image generation unit 108, the control unit 103 displays the output refocus image on a display unit (not illustrated), and ends the series of processes.

(A series of operations related to correction processing)
Next, a series of correction processing in S802 described above will be described with reference to FIG.

In step S <b> 901, the control unit 103 uses the saturation detection unit 104 to execute saturation position detection processing on the input image signal. More specifically, the control unit 103 inputs the coordinates indicating the pixel position of interest and the pixel value of the divided pixel A and the pixel value of the divided pixel B at the position of the target pixel to the saturation detection unit 104. . The saturation detection unit 104 compares the pixel value of the divided pixel A and the pixel value of the divided pixel B with a predetermined threshold value H_LIMIT for determining saturation.
H_LIMIT <pixel value of divided pixel A (3)
H_LIMIT <pixel value of divided pixel B (4)

The saturation detection unit 104 determines that the pixel of interest is a saturated pixel when either or both of Expression 3 and Expression 4 are satisfied. Then, flag information for saturation determination is assigned to each of the corresponding divided pixel A and divided pixel B, and the saturated pixel position is stored in the storage unit of the control unit 103.

  In step S902, the control unit 103 initializes a counter i indicating the number of pixel correction processing repetitions. In step S <b> 903, the control unit 103 performs pixel correction processing using each of the correction amount calculation unit 105, the saturation correction unit 106, and the combination determination unit 107. The pixel correction process according to this step will be described later with reference to the flowchart shown in FIG.

  In step S904, when the pixel correction process in step S903 ends, the control unit 103 increments a counter i indicating the number of repetitions of the pixel correction process (increments the counter by one).

  In step S905, the control unit 103 determines whether the counter i is smaller than a predetermined constant (for example, 2) indicating the number of repetitions. When the counter i is less than 2, the control unit 103 returns the process to S903 and executes the pixel correction process again in S903. On the other hand, when the counter i is 2 or more, the control unit 103 ends the correction process and returns the process to the caller.

  In addition, as shown in S903 to S905, the pixel correction process is repeated. In this way, every time the number of repetitions is repeated, pixel correction processing is performed using an image signal in which the influence of image corruption is reduced, and the accuracy of the correction processing can be improved. Note that the number of repetitions is not limited to the two illustrated in the present embodiment, and a larger number of repetitions may be repeated.

(A series of operations related to pixel correction processing)
Further, a series of operations related to pixel correction processing will be described with reference to a flowchart shown in FIG.

  In step S1001, the control unit 103 determines a saturated pixel to be corrected. The control unit 103 selects a saturated pixel to be corrected based on the coordinates stored in the storage unit of the control unit 103 in S901, and further in S1002, the control unit 103 centers on the saturated pixel to be corrected. The image signal of the adjacent area is read out.

  In step S1003, the control unit 103 causes the combination determination unit 107 to determine a combination of divided pixels in which the image signal area is focused using the image signal read in step S1002. More specifically, the combination determining unit 107 inputs a predetermined area of the image signal of the divided pixel A and the divided pixel B (here, referred to as image signals A and B, respectively) and calculates a correlation amount COR between the two image signals. To do. For example, the image signal A is shifted pixel by pixel, and the sum of the absolute values of the differences between the shifted image signal A and image signal B is calculated. Here, the shift amount is represented by i. Further, x is the start coordinate of the image signal area, and y is the end coordinate of the image signal area. Using these, the correlation amount COR can be calculated by the following equation.

  In Equation 5, the smaller the correlation amount COR, the higher the degree of coincidence between the image signal A and the image signal B (that is, the correlation becomes higher). Accordingly, when the divided pixel A and the divided pixel B that are associated with the shift amount with the smallest correlation amount COR are added, the contrast becomes the highest. In other words, a combination of divided pixels in the calculated shift amount (also referred to as an image shift amount) can be handled as a combination closest to the in-focus state.

  The control unit 103 temporarily stores the shift amount calculated by the combination determination unit 107 and having the highest correlation in the storage unit.

  Although the combination of the divided pixels is determined by calculating the correlation amount, a method other than the correlation amount can be used. For example, the shift amount may be determined in advance using the distance information of the object field obtained by an external measurement sensor (not shown) or the like, and the combination of divided pixels may be determined.

  In step S <b> 1004, the control unit 103 calculates a correction amount using the correction amount calculation unit 105. More specifically, the correction amount calculation unit 105 inputs the image signals of the divided pixel A and the divided pixel B and the shift amount obtained in S1003. Then, the correction amount calculation unit 105 shifts the image signal of the divided pixel A and the image signal of the divided pixel B by the shift amount, and generates a pair of image signals in the correspondence relationship having the highest correlation. As described above, the image obtained by adding the pair of image signals can be handled as being in focus on the subject, so that the divided pixels A and B in the vicinity (within a predetermined range). The influence of the vignetting and the incident angle is dominant in the light quantity ratio.

Therefore, the correction amount calculation unit 105 determines the amount of charge leaked due to saturation based on the light amount ratio between the divided pixel A (NEI_A) and the divided pixel B (NEI_B) located in the vicinity of the saturation pixel of interest. Estimate the amount of leaked charge α. That is, α satisfying Expression 6 can be obtained by using the light amount ratio of the saturation pixel (SAT_A) of interest and the divided pixel (SAT_B) that is a pair of saturation pixels.

NEI_A: NEI_B = SAT_A + α: SAT_B (6)

Since the calculated α is a difference amount from the original signal strength, in the present embodiment, this α is set as a correction amount for saturation.

In step S1005, the control unit 103 causes the saturation correction unit 106 to correct the saturation pixel of interest using the correction amount α calculated by the correction amount calculation process. The saturation correction unit 106 adds the correction amount α to the pixel value of the divided pixel A of interest, and subtracts the correction amount α from the pixel value of the corresponding divided pixel B.

Saturated divided pixel A + α (7)
Saturated division pixel B-α (8)

In this way, it is possible to correct the influence of image corruption due to saturation.

  In step S1006, the control unit 103 determines whether correction processing for all saturated pixels has been completed. If there is a saturated pixel that has not been subjected to the correction process, the control unit 103 returns the process to S1001 again and executes the next saturated pixel correction process. On the other hand, when the correction of all saturated pixels has been completed, the control unit 103 ends a series of operations of the pixel correction process and returns the process to the caller. As described above, the control unit 103 ends the series of operations related to the correction process when the repetition process is ended in S905.

  In this series of processing, the processing is started when the user gives a shooting instruction by pressing the shutter button. For example, when the user specifies recorded data and issues a playback instruction. The series of processes described above may be performed. In this way, it is possible to apply correction processing to saturated pixels when displaying an already captured image on a display unit (not shown).

  In the present embodiment, an example in which the corrected image signal is used for the refocus processing has been described. However, the present correction method may be applied when the stereo image is used for the distance information calculation processing or used for focus detection. .

  As described above, in this embodiment, the pixel configuration includes a plurality of photoelectric conversion units corresponding to one microlens, and is acquired in a pixel configuration in which charge leaks to an adjacent photoelectric conversion unit when saturation occurs. Correction processing for saturated pixels is performed on the processed image signal. In the correction processing according to the present embodiment, the output of the divided pixels is rearranged to be in a focused state, and the correction amount for the saturated pixel is calculated based on the light amount ratio in the photoelectric conversion unit after the rearrangement. . By doing so, it becomes possible to correct the influence of the saturation of the photoelectric conversion unit on the pupil divided image. Furthermore, by calculating the correction amount a plurality of times, it is possible to reduce the influence of image shift and obtain a good image signal for each divided pixel.

(Embodiment 2)
Next, Embodiment 2 will be described. In the first embodiment, the example in which the correction process is repeated a predetermined number of times has been described, but in the second embodiment, the number of repetitions of the correction process is dynamically controlled. For this reason, the digital camera 1100 of the present embodiment has a configuration including a saturation correction end determination unit 1109 unlike the first embodiment, and the other configurations are the same as those of the first embodiment. For this reason, in the drawings to which reference is made to the description of the present embodiment, the same components as those of the first embodiment are denoted by the same reference numerals, and redundant description is omitted, and differences are mainly described.

(Configuration of digital camera 1100)
FIG. 11 shows a functional configuration example of the digital camera 1100 according to the second embodiment. As described above, the digital camera 100 shown in the first embodiment further includes a saturation correction end determination unit 1109.

  The saturation correction end determination unit 1109 includes an arithmetic circuit or a functional block, and determines whether or not the correction process has ended based on a predetermined condition. The detailed operation of the saturation correction end determination unit 1109 will be described later.

(A series of operations related to correction processing for saturated pixels)
With reference to FIGS. 12 and 13, correction processing for saturated pixels according to the present embodiment will be described. Since the operation (S801 to S803) shown in FIG. 8 and the pixel correction processing (S1001 to S1006) shown in FIG. 10 are common, the correction processing will be described below.

  FIG. 12 shows a series of operations of the correction processing according to this embodiment. First, the control unit 103 performs the pixel correction process by performing the processes of S901 and S903 as in the first embodiment. When the pixel correction process ends, the control unit 103 advances the process to S1201.

  In step S <b> 1201, the control unit 103 executes a correction end determination process using the saturation correction end determination unit 1109. A series of operations of the correction end determination process performed by the saturation correction end determination unit 1109 will be described later with reference to FIG. The saturation correction end determination unit 1109 sets the value of the determination flag to 0 when the shift amount has converged to such an extent that the correction process is ended, and sets the determination flag when the correction process is repeated without repeating the correction process. Set to a value other than 0.

  In step S1202, the control unit 103 determines whether or not the value of the determination count counted in the correction end determination process is zero. When the determination count value is 0, the control unit 103 completes the correction process. On the other hand, if the determination count is not 0, the process returns to S903, and the pixel correction process is executed again using the corrected result.

  Next, a series of operations of the correction end determination process by the saturation correction end determination unit 1109 will be described with reference to FIG.

  In step S1301, the saturation correction end determination unit 1109 initializes a determination count for the control unit 103 to determine the end of the correction processing in step S1202.

  In step S1302, the saturation correction end determination unit 1109 reads the shift amount of the saturated pixel calculated in step S1003 in the previous pixel correction process from a storage unit (not illustrated). Since the first pixel correction process is executed only once, 0 is set as the read shift amount.

  In step S1303, the saturation correction end determination unit 1109 reads the saturation pixel shift amount calculated in step S1003 in the current pixel correction process from the storage unit. Note that the shift amounts read in S1302 and S1303 are shift amounts for saturated pixels located at the same coordinates.

  In step S1304, the saturation correction end determination unit 1109 compares the two read shift amounts, and determines whether or not they match. When the shift amounts match, that is, when the shift amounts in the (N−1) th and Nth pixel correction processes are the same, it can be determined that the influence of the image shift due to the correction has converged. Accordingly, the saturation correction end determination unit 1109 advances the processing to S1306 while leaving the determination count value as it is.

  On the other hand, if the shift amounts do not match, 1 is added to the determination count in S1305, and the process proceeds to S1306.

  In step S1306, the saturation correction end determination unit 1109 determines whether or not the shift amount comparison determination has been completed for all saturated pixel positions. If the determination is not completed for all the saturated pixel positions, the process returns to S1302 to read the shift amount of the next coordinate. On the other hand, when the determination for all saturated pixel positions is completed, the saturation correction end determination unit 1109 ends the correction end determination process and returns the process to the caller.

  By determining whether or not the shift amount calculated by the correction process converges in this way, the number of correction repetitions can be dynamically controlled. In the present embodiment, an example in which the correction processing end determination is performed using the shift amount variation has been described. However, in addition to the shift amount, the corrected pixel value variation (that is, the correction amount variation) is used. You may determine completion | finish of a correction process. In this way, it is possible to end the iterative process using the fact that the correction amount is stable, and it is possible to calculate a highly reliable correction amount.

  As described above, in the present embodiment, when the correction process is repeatedly performed, the process is repeatedly performed until the result of the correction process converges before and after the repetition. By doing so, it is possible to dynamically control the number of repetitions of the correction process and improve the reliability of the correction process.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 101 ... Imaging optical system, 102 ... Image sensor, 104 ... Saturation detection part, 105 ... Correction amount calculation part, 106 ... Saturation correction part, 107 ... Combination determination part, 1109 ... Saturation correction end determination part

Claims (14)

An image signal generated by receiving a light beam passing through different pupil regions of the imaging optical system by each of a plurality of photoelectric conversion units constituting a unit pixel, and information on the angle and intensity of the light beam incident on the imaging surface Input means for inputting an image signal including:
The combination of signals when a combination of signals received by a plurality of photoelectric conversion units constituting the unit pixel is focused on a specific subject by recombination using information on the angle and intensity of the incident light beam. A combination determining means for determining
Correcting means for correcting the influence of saturation on the plurality of photoelectric conversion units due to saturation of any of the plurality of photoelectric conversion units constituting the unit pixel based on the determined combination of signals; An image processing apparatus comprising: The correcting means uses a combination of signals received by a plurality of photoelectric conversion units constituting a unit pixel within a predetermined range from the unit pixel in the determined combination of signals, so that the plurality of photoelectric conversion units The image processing apparatus according to claim 1, wherein an influence of saturation received is corrected. The correction means includes, in the determined combination of signals, a combination of signals received by a plurality of photoelectric conversion units constituting the unit pixel, and a plurality of photoelectric elements constituting a unit pixel within a predetermined range from the unit pixel. The image processing apparatus according to claim 2, wherein an influence of saturation received by the plurality of photoelectric conversion units is corrected so that a combination of signals received by the conversion units has an equivalent output ratio. . The combination determining unit is configured to select the unit pixel when focusing on the specific subject based on a correlation of pupil-divided images corresponding to signals received by a plurality of photoelectric conversion units constituting the unit pixel. The image processing apparatus according to any one of claims 1 to 3, wherein a combination of signals received by a plurality of photoelectric conversion units is determined. The correction unit corrects a signal received by another photoelectric conversion unit configuring the unit pixel by using a correction amount with respect to a signal received by a predetermined photoelectric conversion unit configuring the unit pixel. The image processing apparatus according to claim 1. The correcting unit specifies an amount of charge leaked from the predetermined photoelectric conversion unit due to saturation of the predetermined photoelectric conversion unit constituting the unit pixel, and the predetermined photoelectric conversion unit is based on the specified charge amount. 6. The image processing apparatus according to claim 1, wherein a signal received by a conversion unit and a signal received by another photoelectric conversion unit configuring the unit pixel are corrected. The image processing apparatus repeats determination of a signal combination by the combination determination unit and correction of a signal received by each of the photoelectric conversion units constituting the unit pixel by the correction unit. The image processing apparatus according to 5 or 6. The image processing apparatus repeats determination of a signal combination by the combination determination unit and correction of a signal received by each of the photoelectric conversion units constituting the unit pixel by the correction unit until the correction amount converges. The image processing apparatus according to claim 5. Refocus image generation means for generating a refocus image focused on a predetermined subject using signals received by each of a plurality of photoelectric conversion units constituting the unit pixel, corrected by the correction means; The image processing apparatus according to claim 5, wherein the image processing apparatus is provided. An image signal generated by receiving a light beam passing through different pupil regions of the imaging optical system by each of a plurality of photoelectric conversion units constituting a unit pixel, and information on the angle and intensity of the light beam incident on the imaging surface Input means for inputting an image signal including:
The combination of signals when a combination of signals received by a plurality of photoelectric conversion units constituting the unit pixel is focused on a specific subject by recombination using information on the angle and intensity of the incident light beam. A combination determining means for determining
Based on the determined combination of signals, the amount of electric charge leaked from the predetermined photoelectric conversion unit due to saturation of the predetermined photoelectric conversion unit constituting the unit pixel at the time of imaging is specified, and the amount of leaked electric charge An image processing apparatus comprising: correction means for correcting a signal received by the predetermined photoelectric conversion unit and a signal received by another photoelectric conversion unit constituting the unit pixel based on An imaging unit in which unit pixels configured by a plurality of photoelectric conversion units are two-dimensionally arranged, and the imaging unit generates and outputs the image signal;
An image processing device according to any one of claims 1 to 10,
The plurality of photoelectric conversion units constituting the unit pixel are configured such that, when a light beam exceeding a predetermined amount of light is received by a predetermined photoelectric conversion unit, a part of charges output based on the light beam is included in the unit pixel. Configured to flow into other photoelectric conversion units,
An imaging apparatus characterized by that. The input means is an image signal generated by receiving light beams passing through different pupil regions of the imaging optical system by each of a plurality of photoelectric conversion units constituting a unit pixel, and an angle of the light beam incident on the imaging surface And an input process for inputting an image signal including intensity information;
When the determining unit refocuses a combination of signals received by a plurality of photoelectric conversion units constituting the unit pixel using information on the angle and intensity of the incident light beam, and focuses on a specific subject, A combination determining step for determining a combination of the signals;
Correction means for correcting the influence of saturation received by the plurality of photoelectric conversion units due to saturation of any of the plurality of photoelectric conversion units constituting the unit pixel at the time of imaging based on the determined combination of signals And a process for controlling the image processing apparatus. The input means is an image signal generated by receiving light beams passing through different pupil regions of the imaging optical system by each of a plurality of photoelectric conversion units constituting a unit pixel, and an angle of the light beam incident on the imaging surface And an input process for inputting an image signal including intensity information;
When the determining unit refocuses a combination of signals received by a plurality of photoelectric conversion units constituting the unit pixel using information on the angle and intensity of the incident light beam, and focuses on a specific subject, A combination determining step for determining a combination of the signals;
Based on the determined combination of signals, the correction unit specifies the amount of charge leaked from the predetermined photoelectric conversion unit due to saturation of the predetermined photoelectric conversion unit constituting the unit pixel during imaging, and the leakage And a correction step of correcting a signal received by the predetermined photoelectric conversion unit and a signal received by another photoelectric conversion unit constituting the unit pixel based on the amount of the generated charge. Control method of the device.   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 10.