JP2014175777A - Imaging apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus and an image processing method capable of easily reducing influence of crosstalk in a phase difference measure even in various systems.SOLUTION: The imaging apparatus includes an imaging device 21 in which phase difference detection pixels for detecting a focus are included in a part of the pixels, a correction pixel determination part 53 for determining target pixels to be corrected by using arrangement density of the phase difference detection pixels in the imaging device 21 and a pixel correction part 55 for correcting pixels in accordance with an output result from the correction pixel determination part 53.

Description

  The present invention relates to an image pickup apparatus having an image pickup element provided with a phase difference pixel for performing focus detection on a part of the pixel, and an image processing method for performing image processing of image pickup data from the image pickup element.

  In an imaging apparatus having an imaging device that converts an optical image formed by an optical system into an electrical signal, an imaging device (phase difference imager) in which phase detection pixels for focus detection by a phase difference detection method are arranged on a part of the imaging surface. (Also referred to as Patent Document 1).

  If strong incident light is present in the phase difference pixel based on the phase difference detection method as described above, a level difference due to crosstalk occurs. This point will be described with reference to FIGS. FIG. 13A shows an example of a normal pixel, and FIG. 13B shows an example of a phase difference pixel. In FIG. 13A, when a strong incident light LA is incident on the photodiode PD1a via the incident portion LI1a, a photoelectric charge PCa is generated. If almost no light is incident on the photodiode PD1a via the incident portion LI2a, almost no photocharge is generated here. However, the photocharge generated in the photodiode PD1a flows into the photodiode PD2a and generates crosstalk CTa.

  In FIG. 13B, when a strong incident light LA is incident on the photodiode PD1b via the incident portion LI1b, a photoelectric charge PCb is generated. However, in the phase difference pixel, since the light shielding plate S is provided, the photoelectric charge PCb is generated only by the light incident through the opening portion where the incident light LA is not blocked by the light shielding plate. Is less than the photocharge PCa in the case of a normal pixel. As a result, the crosstalk CTb generated near the photodiode PD2b is smaller than CTa.

  When crosstalk occurs in this way, the center position of the output of the phase difference pixel is shifted as shown in FIG. That is, in FIG. 14, the dotted line shows the phase difference pixel output tendency according to the image height when there is no phase difference pixel. On the other hand, the solid line indicates the phase difference pixel output tendency according to the image height when there are phase difference pixels. The R opening means a phase difference pixel output having an opening on the right side, and the L opening means a phase difference pixel output having an opening on the left side.

  As described above, since the phase difference pixel for focus detection by the phase difference detection method is provided with a light shielding member in a part of the pixel, the amount of crosstalk generated in the normal pixel is different from that of the normal pixel. Will occur. In order to solve this, a method of estimating the amount of crosstalk from the output value of the phase difference pixel has been proposed (see Patent Document 2).

Japanese Patent No. 3592147 JP 2009-124573 A

  According to Patent Document 2 described above, the influence of crosstalk caused by the phase difference pixels can be removed to some extent. However, since the output of the phase difference pixel varies depending on the incident characteristics of the optical system and the direction of the aperture, it is necessary to estimate the crosstalk amount in consideration of the characteristics of each optical system, which complicates the system design. In addition, it is necessary to acquire information in the optical system in real time such as during moving image recording, which may result in an incompatible system.

  The present invention has been made in view of such circumstances, and provides an imaging apparatus and an image processing method capable of easily reducing the influence of crosstalk in various systems in a phase difference measure. With the goal.

  In order to achieve the above object, an image pickup apparatus according to a first aspect of the present invention includes an image pickup element provided with a phase difference detection pixel for performing focus detection on a part of the pixels, and phase difference detection pixels among the image pickup elements. A correction pixel determination unit that determines a target pixel to be corrected using the arrangement density of the pixel, and a pixel correction unit that corrects the pixel according to the output result of the correction pixel determination unit.

  In the imaging device according to a second invention, in the first invention, the correction pixel determination unit sets an adjacent pixel adjacent to the phase difference detection pixel or a non-adjacent pixel not adjacent to the phase difference detection pixel as a correction target pixel. A correction reference pixel determining unit that determines whether or not to perform the correction.

  In the imaging device according to a third aspect, in the first or second aspect, the correction pixel determination unit performs the correction based on the number of phase difference detection pixels and non-phase difference detection pixels around the correction target pixel. A level difference determination unit that determines a target pixel to be processed.

  According to a fourth aspect of the present invention, in the image pickup apparatus according to the first aspect, the correction pixel determination unit determines whether or not to correct the output of the pixel according to the determination result in the correction reference pixel determination unit. A determination unit is provided.

  The imaging device according to a fifth invention is the imaging device according to the second invention, wherein the pixel correction unit is a correction target when the correction pixel determination unit sets the adjacent pixel adjacent to the phase difference detection pixel as a correction target. When the pixel is corrected based on the non-adjacent pixel that is not adjacent to another phase difference detection pixel, and the correction pixel determination unit sets the non-adjacent pixel that is not adjacent to the phase difference detection pixel as a correction target, The pixel is corrected based on the adjacent pixel adjacent to another phase difference detection pixel.

  In the imaging device according to a sixth aspect of the present invention, in the first or second aspect, the correction pixel determination unit determines the target pixel to be corrected based on an imaging drive mode.

  An image processing method according to a seventh aspect of the present invention acquires image data from a multi-purpose pixel other than image generation for a part of the pixel, and corrects based on the arrangement density of the multi-purpose pixel in the image data. The target pixel is determined, and the pixel is corrected based on the determination result of the target pixel to be corrected.

  According to the present invention, it is possible to provide an imaging apparatus and an image processing method capable of easily reducing the influence of crosstalk in various systems in a phase difference measure.

1 is a block diagram mainly showing an electrical configuration of a camera according to an embodiment of the present invention. It is a block diagram which shows the detail of the image process part of the camera which concerns on one Embodiment of this invention. It is a block diagram which shows the detail of the correction pixel judgment part of the camera which concerns on one Embodiment of this invention. It is a top view which shows arrangement | positioning of the pixel of the image pick-up element of the camera which concerns on one Embodiment of this invention. It is a figure explaining the judgment in the correction pixel part in the case of correct | amending the non-adjacent pixel of a phase difference pixel in the camera in one Embodiment of this invention. It is a figure explaining the judgment in the correction pixel part in the case of correct | amending the adjacent pixel of a phase difference pixel in the camera in one Embodiment of this invention. It is a figure explaining the judgment in the level | step difference judgment part in the case of correct | amending the non-adjacent pixel of a phase difference pixel in the camera in one Embodiment of this invention. It is a figure explaining correction | amendment of the pixel correction | amendment part which correct | amends using the adjacent pixel of a phase difference pixel, or a non-adjacent pixel in the camera in one Embodiment of this invention. 5 is a flowchart showing the overall processing operation in the camera of one embodiment of the present invention. 6 is a flowchart showing operations up to determination of a correction target in the camera of one embodiment of the present invention. 6 is a flowchart showing operations up to determination of a correction target in the camera of one embodiment of the present invention. 6 is a flowchart showing operations up to determination of a correction target in the camera of one embodiment of the present invention. It is a figure explaining the structure and crosstalk of the conventional image pick-up element. It is a graph which shows the image height and the output of a phase difference pixel in the conventional image sensor.

  Hereinafter, a preferred embodiment will be described using a camera to which the present invention is applied according to the drawings. A camera according to a preferred embodiment of the present invention is a digital camera, and an image pickup unit having an image pickup element having a phase difference pixel for focus detection by a phase difference detection method is provided in part, and an object image is provided by the image pickup unit. Is converted into imaging data, the defocus direction and the defocus amount are calculated by a known phase difference detection method based on the output from the phase difference pixel, and the photographing lens is focused.

  In addition, live view display is performed based on the imaging data generated by the imaging unit, still image shooting is performed, and moving image shooting is performed. At this time, in order to reduce the influence of crosstalk caused by the phase difference pixel, the adjacent pixel or non-adjacent pixel of the phase difference pixel is corrected using the imaging data of the adjacent pixel or non-adjacent pixel of the other phase difference pixel. To do.

  FIG. 1 is a block diagram mainly showing an electrical configuration of a camera according to an embodiment of the present invention. In FIG. 1, a solid line with an arrow indicates a data flow, and a broken line with an arrow indicates a control signal flow. In the camera, there are a photographing lens 11, a diaphragm 13, a mechanical shutter 15, a drive unit 17, an operation unit 19, an image sensor 21, an A-AMP 23, an analog-digital converter (ADC) 25, an image control circuit 26, a CPU (Central Processing Unit). 27, an image processing unit 29, a focus detection circuit 31, a video encoder 33, a bus 35, a DRAM (Dynamic Random Access Memory) 37, a ROM (Read Only Memory) 39, and a flash memory 41.

  The photographing lens 11 is composed of a plurality of optical lenses for forming a subject image, and is a single focus lens or a zoom lens. A diaphragm 13 is disposed behind the optical axis of the photographic lens 11. The diaphragm 13 has a variable aperture diameter and limits the amount of light of the subject light beam that has passed through the photographic lens 11. A mechanical shutter 15 is disposed behind the diaphragm 13 and controls the passage time of the subject luminous flux that has passed through the photographing lens. As the mechanical shutter 15, a known focal plane shutter, lens shutter, or the like is employed.

  Based on a control signal output from the CPU 27, the drive unit 17 performs focus adjustment of the photographing lens 11, aperture diameter control of the diaphragm 13, and opening / closing control (shutter control) of the mechanical shutter 15.

  The operation unit 19 includes operation members such as various input buttons and various input keys such as a power button, a release button, a playback button, and a menu button, detects operation states of these operation members, and outputs the detection results to the CPU 27. . A still image shooting mode and a moving image shooting mode can be selected on a shooting mode dial, a menu screen, or the like. When the release button is operated while the movie shooting mode is selected, movie shooting starts, and when the release button is operated again, movie shooting ends. It should be noted that a moving image button may be provided in the operation unit 19 and the shooting selection unit may be replaced with another method such as starting moving image shooting when the moving image button is operated.

  An image sensor 21 is disposed on the optical axis of the photographic lens 11, behind the mechanical shutter 15 and in the vicinity of the position where the subject image is formed by the photographic lens 11. The image sensor 21 is provided with a phase difference pixel for focus detection on a part of the pixel. In the image pickup device 21, photodiodes constituting each pixel are two-dimensionally arranged in a matrix, and each photodiode generates a photoelectric conversion current corresponding to the amount of received light. Charges are accumulated by a capacitor connected to the diode.

  A Bayer color filter is arranged in front of each pixel. The Bayer array has lines in which R pixels and G pixels are alternately arranged in the horizontal direction and lines in which G pixels and B pixels are alternately arranged. A part of the G pixel is replaced with a phase detection pixel for focus detection. In the present embodiment, a Bayer array is assumed, but the present invention is not limited to this. The arrangement period of the honeycomb array imaging device and the color filter may be irregular. Further, the phase difference pixel is not a part of the G pixel, and the arrangement period, the arrangement place, the type of the color filter, and the presence or absence of the color filter are not limited to this. A detailed configuration of the image sensor 21 will be described later with reference to FIG. In this specification, as long as the signal is based on an analog image signal output from the pixel of the image sensor 21, image data including not only the signal A / D converted by the ADC 25 described later but also the image signal ( Image data).

  The output of the image sensor 21 is connected to the A-AMP 23. The A-AMP 23 performs analog gain adjustment of the image signal output from the image sensor 21. The output of the A-AMP 23 is connected to the ADC 25.

  The ADC 25 is an analog-digital converter, and converts the image signal whose analog gain has been adjusted by the A-AMP 23 into imaging data (image data) in a digital format. The imaging data includes both data from a normal pixel for non-focus detection and data from a phase difference pixel for focus detection. The output of the ADC 25 is output to the bus 35, and the imaging data is temporarily stored in a DRAM 37 described later.

  The imaging control circuit 26 controls the start of exposure and reading of the imaging element 21 in accordance with a control command from the CPU 27. The imaging control circuit 26 changes the readout control according to the drive mode during still image shooting, live view display, moving image shooting, and the like. For example, during live view display or moving image shooting, the number of pixels is less than that of a still image, but a large number of frame images are acquired per second, so that pixel addition is performed.

  A CPU 27 connected to the bus 35 performs overall control of the camera according to a program stored in a ROM 39 described later.

  The image processing unit 29 inputs imaging data from the DRAM 37 via the bus 35, performs various image processing, generates still image or moving image recording image data, and the generated recording image data is temporarily Temporarily stored in the DRAM 37. Further, display image data is generated using the moving image data read from the DRAM 37 and temporarily stored in the DRAM 37. Further, the phase difference pixel data at the focus detection pixel position in the image sensor 21 is subjected to interpolation processing or the like using surrounding pixel data, and the effect of the step due to crosstalk is removed. A detailed configuration of the image processing unit 29 will be described later with reference to FIG.

  The focus detection circuit 31 acquires data from the phase detection pixels temporarily stored in the DRAM 37, and calculates a defocus direction and a defocus amount by a known phase difference AF based on this data. Note that, based on the defocus direction and the defocus amount calculated by the focus detection circuit 31, the CPU 27 focuses the photographing lens 11 by the driving unit 17.

  The video encoder 33 reads the display image data generated by the image processing unit 29 and temporarily stored in the DRAM 37 and outputs it to the LCD / TV 43. The LCD is a liquid crystal display unit, and is used for live view display, recorded image reproduction display, and the like on the back of the camera. The TV is an external television device and is used for reproducing and displaying recorded images.

  The DRAM 37 is an electrically rewritable memory, and temporarily stores image data, recording image data, display image data, and the like as described above. The CPU 27 also temporarily stores various data when the camera is controlled. An SDRAM (Synchronous Dynamic Random Access Memory) may be used for temporary storage of image data.

  The ROM 39 is a nonvolatile memory such as a mask ROM or a flash memory. In addition to the program used by the CPU 27 described above, the ROM 39 also stores various data such as camera adjustment values. The flash memory 41 can be built in or loaded in the camera, and is a storage medium for recording image data.

  Next, the image processing unit 29 will be described with reference to FIGS. 2 and 3. The image processing unit 29 includes a white balance (WB) correction processing unit 51, a correction pixel determination unit 53, a pixel correction unit 55, a synchronization processing unit 57, a luminance specifying conversion unit 59, an edge enhancement processing unit 61, and noise (NR) processing. A unit 63, a color reproduction processing unit 65, a distortion correction unit 67, and a chromatic aberration correction unit 69. Among these, the components other than the correction pixel determination unit 53 and the pixel correction unit 55 are processing units for performing known image processing, and thus detailed description thereof is omitted.

  As shown in FIG. 3, the correction pixel determination unit 53 includes a correction reference pixel determination unit 53a, a correction availability determination unit 53b, and a step level determination unit 53c. The correction reference pixel determination unit 53a detects the ratio between the phase difference pixel and the normal pixel (R pixel or B pixel), and determines whether the correction target is an adjacent pixel or a non-adjacent pixel based on this ratio. To do. The ratio of the non-phase difference pixel adjacent pixels means the number of adjacent pixels of the non-phase difference pixel within a certain interval and the number / the total number of pixels within the interval. An example of a determination method in the correction reference pixel determination unit 53a will be described later with reference to FIGS.

  The correctability determination unit 53b determines whether to correct according to the determination result in the correction reference pixel determination unit 53a. That is, when any of the adjacent pixels and non-adjacent pixels of the phase difference pixel is not biased to an extreme ratio, correction is performed, but when it is biased to an extreme ratio, it is determined that correction is not performed. An example of a determination method in the correction availability determination unit 53b will be described later in S45 and S49 in FIG. 10, S61 and S65 in FIG.

  The step level determination unit 53c determines the step level based on the determination result in the correction reference pixel determination unit 53a and the number of phase difference pixels and non-phase difference pixels around the correction target pixel, and a pixel in which a step occurs is added as a correction pixel. To do. That is, when there are a plurality of phase difference pixels in the up / down / left / right directions, the influence of crosstalk becomes strong, and therefore, it is a correction target. An example of the determination method of the level difference determination unit 53c will be described later with reference to FIG.

  The pixel correction unit 55 corrects the non-adjacent pixel of the phase difference pixel or corrects the correction target pixel from the adjacent pixel of the phase difference pixel according to the determination result of the correction pixel determination unit 53. To correct. An example of a correction method in the pixel correction unit 55 will be described later with reference to FIG.

  Next, the arrangement of normal elements (R pixels, B pixels, G pixels) for obtaining an image on the image sensor 21 and phase detection pixels for focus detection will be described with reference to FIG. FIG. 4A shows a part of the image sensor 21, and as described above, a Bayer array color filter is arranged in front of each pixel of the image sensor 21. In the Bayer array, R pixels in which a red color filter is arranged in the horizontal direction and G pixels in which a green color filter is arranged are alternately arranged, and B pixels in which a G pixel and a blue color filter are arranged are arranged. It has alternating lines.

  A part of the G pixel is replaced with a phase detection pixel for focus detection. In the example shown in FIG. 4A, an R phase difference pixel having an opening on the right side by a light shielding plate (in FIG. 4A, (x1, y1), (x5, y1), (x13, y1), etc. ), L phase difference pixel provided with an opening on the left side by a light shielding plate (in FIG. 4A, (x1, y9), (x5, y9), (x13, y9), etc.), and an opening on the upper side by the light shielding plate T phase difference pixel (in FIG. 4A, (x1, y3), (x9, y3), etc.) provided with a portion, and B phase difference pixel (FIG. ), (X5, y3), (x13, y3), etc.) are provided. FIG. 4B shows an enlarged view of the phase difference pixel located at (x3, y5).

  In focus detection by the phase difference detection method, a pair of R phase difference pixels and L phase difference pixels (in the example shown in FIG. 4A, a combination of (x1, y1) and (x1, y9), etc.) is used in the X direction ( The defocus direction and defocus amount of the subject along the horizontal direction of the screen are detected. In addition, a pair of T phase difference pixels and B phase difference pixels (in the example shown in FIG. 4A, a combination of (x1, y3) and (x5, y3), etc.) along the Y direction (vertical direction of the screen). The defocus direction and the defocus amount of the subject are detected.

  Next, an example of determination in the correction reference pixel determination unit 53a and the correction possibility determination unit 53b in the correction pixel determination unit 53 will be described with reference to FIGS.

  FIG. 5 shows a case where non-adjacent pixels of phase difference pixels are corrected. FIG. 5A shows an image sensor having a pixel arrangement similar to that shown in FIG. The top, bottom, left, and right pixels of the R, L, T, and B phase difference pixels are pixels that are affected by crosstalk. In the example shown in FIG. 5B, many pixels are affected by crosstalk. Of the R and B pixels, the only B pixels that are not affected by crosstalk are (x3, y2), (x7, y2), (x1, y6), and (x5, y6). The other B pixels and R pixels are all affected by crosstalk. Therefore, in such a case, the imaging data of the pixels not affected by the crosstalk is corrected using the imaging data of the same color pixels affected by the crosstalk.

  FIG. 6 shows a case where a pixel adjacent to the phase difference pixel is corrected. 6A has fewer phase difference pixels than the pixel arrangement shown in FIG. FIG. 6B shows the pixels affected by the crosstalk. Compared with FIG. 5B, the number of pixels affected by the crosstalk is small. Therefore, in such a case, the imaging data of the pixel affected by the crosstalk is corrected using the imaging data of the same color pixel not affected by the crosstalk.

  Therefore, the correction reference pixel determination unit 53a in the correction pixel determination unit 53 obtains the ratio between the adjacent pixels and the non-adjacent pixels of the phase difference pixel, and sets the correction target as the adjacent pixel of the phase difference pixel according to this ratio. Or a non-adjacent pixel. Note that the ratio of adjacent pixels to non-adjacent pixels of the phase difference pixel varies depending on how to add the normal pixels depending on whether a still image is captured, a live view is displayed, or a moving image is captured. For example, in a still image, even when the arrangement period of phase difference pixels is an interval of 4 pixels, a case is considered in which data is output by averaging the same color pixels of 4 horizontal / vertical pixels during moving image shooting. In that case, the cycle of the pixels in which the phase difference pixels are mixed is one pixel interval. In that case, the arrangement of the phase difference pixels and the normal pixels of the image sensor 21 may be stored in the ROM 39 and counted based on the drive mode and the stored arrangement. Further, the ratio of the adjacent pixels and non-adjacent pixels of the phase difference pixel corresponding to the driving mode (still image, moving image, live view display, etc.) is stored in the ROM 39 in advance, and the correction reference pixel determining unit 53a The ratio may be read and determined according to the above.

  Next, an example of determination in the step level determination unit 53c in the correction pixel determination unit 53 will be described with reference to FIG. FIG. 7A shows a pixel whose correction target is determined by the correction reference pixel determination unit 53a and the correction availability determination unit 53b, and is the same as FIG. 5B. In the determination results in the correction reference pixel determination unit 53a and the correction availability determination unit 53b, the pixels (x1, y2), (x5, y2), (x3, y6), (x7, y6) are affected by crosstalk. Is not subject to correction. However, these pixels are vertically adjacent to the phase difference pixels R and T or the phase difference pixels R and B, and are affected by other crosstalk that is adjacent to the phase difference pixel only at one place in the vertical and horizontal directions. Compared to pixels, the influence of crosstalk is large, and correction is necessary.

  Accordingly, the step level determination unit 53c is set as a correction target because the influence level of crosstalk is large when adjacent to a plurality of phase difference pixels. In the example shown in FIG. 7B, the pixels (x1, y2), (x5, y2), (x3, y6), and (x7, y6) adjacent to the phase difference pixels in the vertical direction are determined as correction targets.

  Next, an example of the correction operation in the pixel correction unit 55 will be described with reference to FIG. FIG. 8A illustrates an example in which correction is performed using pixels adjacent to the phase difference pixel, and FIG. 8B illustrates an example in which correction is performed using non-adjacent pixels of the phase difference pixel.

  The example shown in FIG. 5 is a case where the arrangement density of the phase difference pixels is relatively high. In such a case, since there are few non-adjacent pixels not adjacent to the phase difference pixels, Selected as a correction target pixel. In this case, pixel correction is performed by interpolation calculation using imaging data of pixels of the same color that are adjacent to the phase difference pixels and are affected by crosstalk.

  In the example shown in FIG. 8A, the imaging data of the blue pixel in (x5, y6) that is not affected by crosstalk is converted into (x3, y6) and (x7, Pixel correction by interpolation calculation is performed using the imaging data of the blue pixel of y6). Although not shown in the drawing, pixel correction is similarly performed on imaging data of blue pixels (x3, y2), (x7, y2), and (x1, y6).

  The example shown in FIG. 6 is a case where the arrangement density of the phase difference pixels is relatively low. In such a case, since there are few adjacent pixels adjacent to the phase difference pixels, the adjacent pixels are corrected. Selected as a pixel. In this case, pixel correction is performed by interpolation calculation using imaging data of pixels of the same color that are not adjacent to the phase difference pixel and are not affected by crosstalk.

  In the example shown in FIG. 8B, the imaging data of the blue pixel in (x5, y2) that is affected by crosstalk is converted into (x3, y2) and (x7, Pixel correction by interpolation calculation is performed using the imaging data of the blue pixel of y2). In addition to the pixel of (x5, y2), the pixel correction is similarly performed on the imaging data of the blue pixel and the red pixel which are affected by the crosstalk.

  Next, pixel correction according to the present embodiment will be described with reference to flowcharts shown in FIGS. The pixel correction in the present embodiment is executed by the CPU 27 controlling each part in accordance with a program stored in the ROM 39, and the image processing part 29 performs control in the image processing part 29. Run in hardware.

  When the entire processing flow shown in FIG. 9 is entered, first, moving image recording is started (S1). Here, recording of moving image data in the flash memory 41 is started using the imaging data output from the imaging element 21. In addition to moving image recording, still image recording, live view display, and external device display such as a television may be used.

  When moving image recording is started, next, image capturing is performed (S3). Here, the imaging data output from the imaging device 21 is AD converted by the ADC 25, and the digitized imaging data is temporarily stored in the DRAM 37.

  When image capturing is performed, focus detection processing is performed (S5). Here, the imaging data of the phase difference pixels is extracted from the imaging data temporarily stored in the DRAM, and focus detection processing by the phase difference detection method is performed using the imaging data.

  Once the focus detection process has been performed, next, phase difference pixel arrangement information is acquired (S7). As described above, the pixel addition is performed according to the drive mode such as a still image, live view display, and moving image, so that the arrangement of the phase difference pixels changes. Therefore, in this step, information regarding pixel arrangement is acquired for each drive mode.

  When the arrangement information of the phase difference pixels is acquired, a correction reference pixel determination is performed (S9). Here, as described with reference to FIGS. 5 and 6, the correction reference pixel determination unit 53 a determines the correction target from the ratio of the adjacent pixels adjacent to the phase difference pixels and the non-adjacent pixels that are not adjacent. This correction reference pixel determination will be described later using the flowcharts shown in FIGS.

  Once the correction reference pixel determination process is performed, a correction propriety determination process is then performed (S11). Here, the correction possibility determination unit 53b determines that the correction is not performed when the adjacent pixel adjacent to the phase difference pixel and the non-adjacent non-adjacent pixel are biased to either one. This determination as to whether or not correction is possible corresponds to S39 Yes and S41 Yes in FIG. 10 described later and S39 Yes and S41 Yes in FIG.

  Once the correction availability determination process is performed, a level difference determination process is then performed (S13). Here, as described with reference to FIG. 7, the step level determination processing unit 53 c performs determination processing of pixels having a large influence of crosstalk among adjacent pixels adjacent to the phase difference pixels when performing pixel correction. . This step level determination process will be described later with reference to the flowchart of FIG.

  Once the level difference determination process is performed, a pixel correction process is then performed (S15). Here, as described with reference to FIG. 8, the pixel correction processing unit 55 performs pixel correction processing by performing interpolation processing on the correction target pixel using imaging data of a predetermined pixel.

  Once the pixel correction processing is performed, next, image processing for recording the imaged data on the DRAM is performed (S17). Here, the image processing unit 29 performs image processing for recording the image data captured in step S3 and temporarily stored in the DRAM 37.

  Once image processing is performed for recording, the imaged data is then recorded in the flash memory (S19). Here, the imaging data subjected to image processing in step S17 is recorded in the flash memory 41.

  Once the imaging data is recorded in the flash memory, it is next determined whether or not to stop moving image recording (S21). Here, it is determined whether or not the user has operated the operation unit 19 to give an instruction to stop moving image recording.

  If the result of determination in step S <b> 21 is that there has been no instruction to stop moving image recording, processing returns to step S <b> 3 and the above processing is repeated. On the other hand, when an instruction to stop moving image recording has been given, the moving image recording ends.

  Next, detailed processing for determining the correction target in steps S9 and S11 of FIG. 9 will be described using the flowcharts shown in FIGS.

  When the flow shown in FIG. 10 starts, first, the number of R pixels and the number of B pixels adjacent to the phase pixel in a certain section are counted (S31). Here, the number of pixels adjacent to the phase difference pixel is counted for each same color within a certain interval. As the fixed section, a predetermined range such as an area unit for focusing or a period unit of an interval at which the focus detection elements are arranged may be determined. As a method for counting adjacent pixels, the ROM 39 stores the arrangement of phase difference pixels and normal pixels (R pixels and B pixels) of the image sensor 21, and counts them based on the drive mode and the stored arrangement. Also good. In addition, the ROM 39 may store the number of adjacent pixels of the phase difference pixel corresponding to the drive mode (still image, moving image, live view display, etc.) of the image sensor 21 and read the stored value.

  Subsequently, the number of R pixels and the number of B pixels in a certain section are counted (S33). Here, R pixels and B pixels in the fixed interval are counted regardless of whether or not they are adjacent to the phase difference pixel in the fixed interval. As a counting method, the ROM 39 may count based on the arrangement of normal pixels (R pixels and B pixels) stored in the ROM 39, and the number of R pixels and B pixels corresponding to the drive mode is stored. The stored value may be read out.

  Subsequently, the phase difference pixel adjacent B pixel / B pixel total number is obtained (S35). Here, the number of adjacent B pixels adjacent to the phase difference pixel obtained in step S31 is divided by the total number of B pixels in the fixed section obtained in step S33.

  Next, it is determined whether the ratio is 50% or more (S37). Here, it is determined whether the value of the number of adjacent B pixels / the total number of B pixels obtained in step S35 is 50% or more.

  If the result of determination in step S37 is 50% or higher, it is next determined whether or not the ratio is 90% or higher (S39). Here, it is determined whether or not the ratio obtained in step S35 is 90% or more.

  As a result of the determination in step S39, if the ratio is not 90% or more, the phase difference pixel non-adjacent pixel B pixel is set as a correction target (S43). In this case, the phase difference pixel adjacent B pixel / B pixel total number is 50% or more and less than 90%. In this state, there are many B pixels adjacent to the phase difference pixels, and as described with reference to FIG. 5B, non-adjacent pixels that are not adjacent to the phase difference pixels are targeted for correction. That is, as described above with reference to FIG. 13 and the like, adjacent pixels adjacent to the phase difference pixel are easily affected by crosstalk, so even if the subject brightness is the same, there is a step difference in output between the adjacent pixel and the non-adjacent pixel. Will occur. In this step, since there are fewer non-adjacent pixels than adjacent pixels, the non-adjacent pixels are corrected.

  If the result of determination in step S39 is 90% or more, no correction target pixel is provided (S45). In this case, most of the B pixels are adjacent to the phase difference pixels, and the adjacent pixels are overwhelmingly large. Therefore, the step between the adjacent pixels and the non-adjacent pixels is not conspicuous. Even if correction for non-adjacent pixels is not performed, the image does not feel uncomfortable, so correction is not performed.

  If the result of determination in step S37 is that the ratio is not 50% or more, it is next determined whether or not the ratio is 5% or less (S41). Here, it is determined whether or not the ratio obtained in step S35 is 5% or less.

  If the result of determination in step S41 is that the ratio is not less than 5%, the phase difference pixel adjacent pixel B pixel is set as the correction target (S47). In this case, the total number of adjacent B pixels / B pixels of the phase difference pixels is 5% or more and less than 50%. In this state, there are many B pixels that are not adjacent to the phase difference pixels, and as described with reference to FIG. 6B, the adjacent pixels adjacent to the phase difference pixels are targeted for correction. That is, since adjacent pixels adjacent to the phase difference pixel are easily affected by crosstalk, a step is generated between the adjacent pixel and the non-adjacent pixel even if the subject brightness is the same. In this step, since there are fewer adjacent pixels than non-adjacent pixels, correction is performed on adjacent pixels.

  If the result of determination in step S41 is 5% or less, no correction target pixel is provided (S49). In this case, most of the B pixels are not adjacent to the phase difference pixels, and the non-adjacent pixels are overwhelmingly large. Therefore, the step between the adjacent pixels and the non-adjacent pixels is not conspicuous. Even if no correction is performed on the adjacent pixels, the image does not feel uncomfortable, so no correction is made.

  When the process is determined for the correction target pixel in steps S43 to S49, the same process as steps S35 to S49 is performed for the R pixel in steps S51 to S65 shown in FIG. Since only the B pixel is replaced with the R pixel, detailed description is omitted.

  First, the total number of adjacent pixels / R pixels adjacent to the phase difference pixels is obtained (S51). Subsequently, it is determined whether or not the ratio obtained in step S51 is 50% or more (S53). As a result of this determination, if it is 50% or more, it is further determined whether or not the ratio is 90% or more (S55). If the ratio is not 90% or more as a result of this determination, non-adjacent R pixels that are not adjacent to the phase difference pixels are set as correction targets (S59). On the other hand, if the result of determination in step S55 is 90% or more, no correction target is provided (S61).

  If the result of determination in step S53 is that the ratio is not 50% or more, it is further determined whether or not the ratio is 5% or less (S57). As a result of this determination, if the ratio is not 5% or more, the adjacent R pixel adjacent to the phase difference pixel is set as a correction target (S63). On the other hand, if the result of determination in step S57 is 5% or less, no correction target is provided (S65). When the process is determined for the correction target pixel in steps S43 to S49, the flow for determining the correction target is terminated, and the process returns to step S13 of the flow shown in FIG.

  As described above, in the flowcharts shown in FIGS. 10 and 11, it is determined whether to correct adjacent pixels or non-adjacent pixels according to the arrangement density of the phase difference pixels (for example, (See S37, S39, S41, etc.). Therefore, the influence of crosstalk can be reduced according to the arrangement density of the phase difference pixels.

  Further, when the number of pixels to be corrected is extremely small or large, correction is not performed (for example, see S45, S49, etc.). Therefore, the influence of crosstalk can be reduced efficiently.

  In this flowchart, it is determined whether or not it is 50% or more in steps S37 and S53, and whether or not it is 90% or more or 5% or less in steps S39, S41, S55, and S57. The numerical value of this percentage (%) is an example, and other numerical values may of course be used.

  Next, detailed processing of the level difference determination processing in step S13 of FIG. 9 will be described using the flowchart shown in FIG. In this flow, the level difference determination performed in the level difference determination unit 53c shown in FIG. 7 is executed. As described above, even when a non-adjacent pixel is not subject to correction, if it is adjacent to a plurality of phase difference pixels vertically and horizontally, it is strongly affected by crosstalk, and the level difference from other pixels of the same color is It gets bigger. Therefore, in such a case, it is a correction target. In this flow, a plurality of phase difference pixels and adjacent pixels are subject to correction in the vertical and horizontal directions.

  In the flow shown in FIG. 12, first, the adjacent pixel of the next phase difference pixel is set as a determination target (S71). For example, in the example illustrated in FIG. 7, assuming that the phase difference R pixel (x1, y1) is a starting point, (x2, y1), (x4, y1), (x6, y1), (x1, y2), (x5 , Y2)..., Adjacent pixels of the phase difference pixel are selected in order of the order. Although the selection is sequentially performed in the x direction, the present invention is not limited to this, and the selection may be performed sequentially in the y direction, or the periphery of the phase difference pixels may be sequentially selected.

  If the adjacent pixel of the phase difference pixel is selected, it is next determined whether or not two or more of the upper, lower, left and right adjacent pixels are phase difference pixels (S73). For example, in the example shown in FIG. 7, the pixel (x1, y2) has a phase difference R pixel in the upper adjacent pixel and a phase difference T pixel in the lower adjacent pixel. Since the pixel is adjacent to the phase difference pixel, the determination result is Yes.

  As a result of the determination in step S73, if two or more pixels among the upper, lower, left and right adjacent pixels are phase difference pixels, they are subject to correction (S75).

  If it is determined in step S75 that correction is to be performed, or if the result of determination in step S73 is that two or more pixels are not phase difference pixels, the adjacent pixels of the next phase difference pixel are determined. The target pixel is selected (S77), and the process returns to step S71.

  As described above, in the level difference determination shown in FIG. 12, when there are two or more phase difference pixels in the vertical and horizontal directions, the correction target image is set. In this case, since the influence of crosstalk is strong, the level difference level with other pixels of the same color is increased, and the image is uncomfortable.

  As described above, in one embodiment of the present invention, a phase difference pixel for performing focus detection is provided in a part of the pixels of the image sensor 21. Further, the correction pixel determination unit 53 determines a target pixel to be corrected using the arrangement density of the phase difference pixels in the image sensor 21 (for example, S7 to S7 in FIGS. 5, 6, and 7, FIG. 9). (See S13). In addition, the pixel correction unit 55 corrects the pixels according to the output result of the correction pixel determination unit 53 (see, for example, S15 in FIGS. 8 and 9). For this reason, even in various systems, it is possible to easily reduce the influence of crosstalk. For example, even when the drive mode of the image sensor 21 is switched such as still image shooting, live view display, and moving image shooting, and the arrangement density of the phase difference pixels changes, the pixel correction is appropriately performed and the influence of crosstalk Can be reduced.

  In one embodiment of the present invention, the correction pixel determination unit 53 determines whether or not the adjacent pixel adjacent to the phase difference pixel or the non-adjacent pixel not adjacent to the phase difference pixel is set as the correction target pixel. The pixel determination unit 53a is included (see, for example, S9 in FIGS. 5, 6, and 9). Although adjacent pixels of the phase difference pixel are affected by crosstalk, according to the present embodiment, it is determined whether to correct for adjacent pixels or non-adjacent pixels. The impact can be effectively reduced.

  In one embodiment of the present invention, the correction pixel determination unit 53 includes a step level determination unit 53c that makes a determination based on the number of phase difference pixels and non-phase difference pixels around the correction target pixel (for example, , See S13 of FIG. 7 and FIG. 9, FIG. 12, etc.). When a plurality of phase difference pixels are adjacent to each other in the vertical and horizontal directions, the influence of crosstalk becomes strong, and a difference in level from other pixels of the same color is conspicuous. The level difference is inconspicuous, and an image without a sense of discomfort can be obtained.

  In one embodiment of the present invention, the correction pixel determination unit 53 includes a correction availability determination unit 53b that determines not to correct the pixel output (for example, S11 in FIG. 9 and S45 in FIG. 10). , S49, S61, S65 etc.). Even when the correction target is an adjacent pixel or a non-adjacent pixel, if the correction target is overwhelmingly small, the level difference from other pixels of the same color is not concerned, so no correction is performed. Thereby, the influence of crosstalk can be efficiently removed.

  In one embodiment of the present invention, when the correction pixel determination unit 53 sets an adjacent pixel adjacent to the phase difference pixel as a correction target, the pixel correction unit 55 sets the correction target pixel adjacent to another phase difference pixel. Correction is performed based on the non-adjacent pixels that are not to be corrected (see, for example, FIG. 8B), and when the correction pixel determination unit 53 sets the non-adjacent pixels that are not adjacent to the phase difference pixels as correction targets, Correction is performed based on adjacent pixels adjacent to the phase difference pixels (see, for example, FIG. 8A). As described above, since the correction target pixel is corrected by the adjacent pixel or the non-adjacent pixel, a step due to the influence of crosstalk can be reduced.

  In one embodiment of the present invention, the correction pixel determination unit switches determination according to the imaging drive mode. The drive mode of the image sensor 21, such as still image shooting, live view display, and moving image shooting, is switched, the phase pixel arrangement density changes, and the influence of crosstalk changes. According to the present embodiment, the influence of crosstalk can be reduced according to the imaging drive mode.

  In the embodiment of the present invention, four pixels on the top, bottom, left, and right of the phase difference pixel are targeted as the adjacent pixels of the phase difference pixel, but the upper right, the lower right, the upper left, and the lower left of the phase difference pixel. May be added to the outside. What is necessary is just to determine suitably according to the influence of the crosstalk which the image pick-up element 21 receives.

  In one embodiment of the present invention, the ratio of the total number of B pixels and the phase difference pixel adjacent B pixels and the ratio of the total number of R pixels and the phase difference pixel adjacent R pixels are used as the arrangement density of the phase difference pixels (FIG. 10). S35, S51 of FIG. 11, etc.). However, the present invention is not limited to this, and other evaluation values such as an average distance between the phase difference pixel and the B pixel or the R pixel may be used.

  In the embodiment of the present invention, the digital camera is used as the photographing device. However, the camera may be a digital single lens reflex camera or a compact digital camera, such as a video camera or a movie camera. It may be a camera for moving images, or may be a camera built in a mobile phone, a smart phone, a personal digital assistant (PDA), a game machine, or the like. In any case, the present invention can be applied to any device that has focus detection pixels on the imaging surface and performs focus detection by the phase difference method.

  In addition, regarding the operation flow in the claims, the specification, and the drawings, even if it is described using words expressing the order such as “first”, “next”, etc. It does not mean that it is essential to implement in this order.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 10 ... Subject, 11 ... Shooting lens, 13 ... Aperture, 15 ... Mechanical shutter, 17 ... Drive part, 19 ... Operation part, 21 ... Imaging element, 23 ... A-AMP, 25 ... analog-digital converter (ADC), 26 ... imaging control circuit, 27 ... CPU, 29 ... image processing unit, 31 ... focus detection circuit, 33 ... Video encoder 35 ... Bus 37 ... DRAM 39 ... ROM 41 ... Flash memory 43 ... LCD / TV 51 ... White balance (WB) processing unit 53 ..Correction pixel determination unit, 53a... Correction reference pixel determination unit, 53b... Correction possibility determination unit, 53c .. step level determination unit, 55... Pixel correction unit, 57. Unit, 59... Luminance characteristic conversion unit, 61. Di enhancement processing unit, 63 ... Noise Reduction (NR) processing unit, 65 ... color reproduction processing unit, 67 ... distortion correction unit, 69 ... color aberration corrector

Claims (7)

An image sensor provided with a phase difference detection pixel for performing focus detection on a part of the pixel;
A correction pixel determination unit that determines a target pixel to be corrected using the arrangement density of the phase difference detection pixels in the image sensor;
A pixel correction unit that corrects a pixel in accordance with an output result of the correction pixel determination unit;
An imaging device comprising:   The correction pixel determination unit includes a correction reference pixel determination unit that determines whether an adjacent pixel adjacent to the phase difference detection pixel or a non-adjacent pixel not adjacent to the phase difference detection pixel is a correction target pixel. The imaging apparatus according to claim 1.   2. The correction pixel determination unit includes a step level determination unit that determines the correction target pixel based on the number of phase difference detection pixels and non-phase difference detection pixels around the correction target pixel. Or the imaging device of 2.   The imaging according to claim 2, wherein the correction pixel determination unit includes a correction availability determination unit that determines whether or not to correct the output of the pixel according to a determination result in the correction reference pixel determination unit. apparatus. The pixel correction unit
When the correction pixel determination unit sets the adjacent pixel adjacent to the phase difference detection pixel as a correction target, the correction target pixel is corrected based on the non-adjacent pixel that is not adjacent to another phase difference detection pixel.
When the correction pixel determination unit sets the non-adjacent pixel that is not adjacent to the phase difference detection pixel as a correction target, the correction target pixel is corrected based on the adjacent pixel adjacent to another phase difference detection pixel.
The imaging apparatus according to claim 2.   The imaging apparatus according to claim 1, wherein the correction pixel determination unit determines the target pixel to be corrected based on an imaging drive mode. Obtain image data from a multi-purpose pixel for non-image generation in a part of the pixel,
Based on the arrangement density of the multipurpose pixels in the image data, determine the target pixel to be corrected,
Correcting the pixel based on the determination result of the target pixel to be corrected;
An image processing method.

Images