JP2023075146A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an image with little blur even in an unfocused area.

SOLUTION: An imaging apparatus includes: an imaging unit which has a plurality of pixels each having a first photoelectric conversion unit configured to perform photoelectric conversion of light passing through a first area of an optical system to generate an electric charge and a second photoelectric conversion unit configured to perform photoelectric conversion of light passing through a second area of the optical system to generate the electric charge; a section unit which sections a focus area and a non-focus area from each other or extracts at least one of the focus area and the non-focus area on an imaging surface of the imaging unit on the basis of a first signal based on the electric charge generated by the first photoelectric conversion unit and a second signal based on the electric charge generated by the second photoelectric conversion unit; and a first generation unit which generates image data by selecting at least one of the first signal and the second signal output from at least one pixel included in the non-focus area of the plurality of pixels.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to an imaging device.

2. Description of the Related Art An imaging device capable of focus detection by an image plane phase difference method is known (see Patent Document 1). Pixels corresponding to out-of-focus areas are not suitable for some applications because the image is blurred.

JP-A-2001-83407

The imaging apparatus according to claim 1 comprises a first photoelectric conversion unit that photoelectrically converts light that has passed through a first region of the optical system to generate electric charges, and a photoelectric conversion unit that photoelectrically converts light that has passed through the second region of the optical system. a first signal based on the charges generated by the first photoelectric conversion unit; Based on a second signal based on the generated charge, the imaging surface of the imaging unit is divided into an in-focus area and an out-of-focus area, or at least one of the in-focus area and the out-of-focus area is divided. Image data is generated by selecting one of the first signal and the second signal output from the section to be extracted and at least one pixel included in the out-of-focus area among the plurality of pixels. and a first generator.

1 is a diagram showing the configuration of a digital camera according to one embodiment of the present invention; FIG. It is a figure which shows the structure of a body drive control apparatus. It is a figure which shows the detailed structure of an image pick-up element. It is a figure which shows the detailed structure of an image pick-up element. 4 is a diagram showing the configuration of a pixel; FIG. FIG. 2 is a diagram showing the configuration of a focus detection optical system of a split-pupil phase difference detection method; 4 is a schematic optical path diagram in a focused state; FIG. 4 is a schematic optical path diagram in an out-of-focus state; FIG. FIG. 5 is a diagram illustrating an arrangement of image signals of pixels corresponding to a substantially focused area; It is a figure explaining interpolation processing of G component. It is a figure explaining the interpolation process of R component. FIG. 10 is a diagram illustrating an arrangement of output signals of a photoelectric conversion unit in pixels corresponding to an out-of-focus area; It is a figure explaining interpolation processing of G component. It is a figure explaining the interpolation process of R component. It is a figure explaining the relationship between the position of the exit pupil of an interchangeable lens, and vignetting.

EMBODIMENT OF THE INVENTION Hereinafter, the form for implementing this invention is demonstrated with reference to drawings. FIG. 1 is a diagram showing the configuration of a lens-interchangeable digital camera, which is an example of an imaging apparatus according to this embodiment. A digital camera 201 is composed of an interchangeable lens 202 and a camera body 203 . Various interchangeable lenses 202 are attached to a camera body 203 via a mount section 204 .

The interchangeable lens 202 includes a lens 209, a zooming lens 208, a focusing lens 210, an aperture 211, a lens drive control device 206, and the like. The lens drive control device 206 is composed of a microcomputer, a memory, a drive control circuit and the like (not shown). The lens drive control device 206 performs drive control for focus adjustment of the focusing lens 210 and adjustment of the aperture diameter of the diaphragm 211, state detection of the zooming lens 208, focusing lens 210 and diaphragm 211, and the like. Also, the lens drive control device 206 transmits lens information and receives camera information through communication with a body drive control device 214, which will be described later. A diaphragm 211 forms an aperture with a variable aperture diameter at the center of the optical axis for adjusting the amount of light and the amount of blurring.

The camera body 203 includes an imaging device 212, a body drive control device 214, a liquid crystal display device drive circuit 215, a liquid crystal display device 216, an eyepiece lens 217, and the like. A memory card 219 is attached to the camera body 203 .

The body drive control device 214 is composed of a microcomputer, a memory, a drive control circuit, and the like. The body drive control device 214 repeatedly performs drive control of the image sensor 212, reading of an output signal from the image sensor 212, focus detection calculation based on the output signal, and focus adjustment of the interchangeable lens 202, and based on the output signal. Performs image processing, recording, and camera operation control. Also, the body drive control device 214 communicates with the lens drive control device 206 via the electrical contact 213 to receive lens information and transmit camera information (defocus amount, aperture value, etc.).

The liquid crystal display element 216 functions as an electronic view finder (EVF). A liquid crystal display element drive circuit 215 displays a through image captured by the imaging element 212 on a liquid crystal display element 216 . A photographer can observe a through image through an eyepiece lens 217 . A memory card 219 is an image storage that stores images captured by the image sensor 212 .

A subject image is formed on the light receiving surface of the imaging element 212 by the light flux that has passed through the interchangeable lens 202 . This subject image is photoelectrically converted by each pixel of the image sensor 212 , and the output signal of each pixel is sent to the body drive control device 214 .

The body drive control device 214 calculates the defocus amount based on the output signal of the imaging device 212 and sends this defocus amount to the lens drive control device 206 . In addition, the body drive control device 214 processes the output signal of the image sensor 212 to generate image data, stores the image data in the memory card 219, and sends the through image signal from the image sensor 212 to the liquid crystal display element drive circuit 215. , to display a through image on the liquid crystal display element 216 . Further, the body drive control device 214 sends aperture control information to the lens drive control device 206 to control the opening of the aperture 211 .

The lens drive control device 206 updates the lens information according to the focusing state, zooming state, aperture setting state, aperture open F number, and the like. Specifically, the positions of the zooming lens 208 and the focusing lens 210 and the aperture value of the aperture 211 are detected, and lens information is calculated according to these lens positions and aperture values, or a previously prepared lookup function is detected. Select the lens information corresponding to the lens position and aperture value from the table.

The lens drive control device 206 calculates the lens drive amount based on the received defocus amount, and drives the focusing lens 210 to the in-focus position according to the lens drive amount. Also, the lens drive control device 206 drives the aperture 211 according to the received aperture value.

FIG. 2 is a block diagram showing the configuration of the body drive control device 214. As shown in FIG. The body drive control device 214 includes a focus detection unit 230, a division unit 231, a first image signal generation unit 232, a second image signal generation unit 233, an image analysis unit 234, a display control unit 235, a recording control unit 236, and A vignetting determination unit 237 is functionally provided.

The focus detection unit 230 performs focus detection by performing processing described later based on the output signal of the image sensor 212 . The division unit 231 divides the photographing screen into a substantially focused region and a non-focused region, or extracts at least one of the substantially focused region and the non-focused region. The first image signal generation unit 232 performs processing described later based on the output signal of the imaging element 212 to generate an image signal for image analysis. Note that the image signal for image analysis is not limited to a plurality of pixel signals generated for the purpose of being output as an image. The image signal for image analysis is a plurality of pixel signals obtained as raw data of a plurality of output signals output from a plurality of photoelectric conversion units in a plurality of pixels of the image sensor 212, and an image is generated based on the raw data. It may be a plurality of pixel signals generated as data for image analysis without being output as . The second image signal generation unit 233 performs processing described later based on the output signal of the image pickup device 212, and generates an image signal composed of a plurality of pixel signals constituting a display image (through image) and a recording image. do. The image analysis unit 234 analyzes the subject image based on the image signal generated by the first image signal generation unit 232 . The display control section 235 displays a through image on the liquid crystal display element 216 based on the image signal generated by the second image signal generation section 233 . The recording control section 236 records image data in the memory card 219 based on the image signal generated by the second image signal generating section 233 . The vignetting determination unit 237 determines the vignetting state based on the lens information (information on the exit pupil position) of the interchangeable lens 202 . Details of these units will be described later.

3 and 4 are front views showing the detailed configuration of the imaging device 212, and are views showing a part of the imaging device 212 in an enlarged manner. FIG. 3 is a diagram showing the layout of the pixels 311. As shown in FIG. The plurality of pixels 311 are two-dimensionally arranged in the row direction (horizontal direction) and the column direction (vertical direction). Each pixel 311 has a microlens (not shown) and a pair of photoelectric conversion units 13 and 14 . In FIG. 3, a pair of photoelectric conversion units 13 and 14 are arranged horizontally. FIG. 4 is a diagram showing the arrangement of color filters in the arrangement of the pixels 311 shown in FIG. Color filters (R: red filter, G: green filter, B: blue filter) are arranged in the pixel 311 according to the Bayer arrangement rule. That is, as the pixels 311, an R pixel having a spectral sensitivity characteristic for a red component (that is, a red filter is arranged) and a G pixel having a spectral sensitivity characteristic for a green component (that is, a green filter is arranged) and a spectrum for a blue component A B pixel having a sensitivity characteristic (that is, a blue filter is arranged) is provided. The pixels 311 serve as both imaging pixels and focus detection pixels, and the pixels 311 are arranged over the entire surface of the image sensor 212 . Therefore, it is possible to perform focus detection at any position on the photographing screen.

FIG. 5 is a cross-sectional view showing the configuration of the pixel 311. As shown in FIG. In the pixel 311 , the microlens 10 is arranged in front of the pair of photoelectric conversion units 13 and 14 . A pair of photoelectric conversion units 13 and 14 are formed on a semiconductor circuit board 29 . A color filter (not shown) is arranged between the microlens 10 and the pair of photoelectric conversion units 13 and 14 . With such a configuration, the pair of photoelectric conversion units 13 and 14 respectively receive a pair of light beams passing through a pair of ranging pupils of the exit pupil of the interchangeable lens 202 .

FIG. 6 shows the configuration of a split-pupil phase difference detection type focus detection optical system using microlenses. In FIG. 6, the exit pupil 90 is set at a position a distance d in front of the microlens 10 arranged on the intended imaging plane of the interchangeable lens 202 (see FIG. 1). FIG. 6 also shows the optical axis 91 of the interchangeable lens, the microlens 10, the photoelectric conversion units 13 and 14, the pixel 311, and the light beams 73 and 74. FIG.

The distance measuring pupils 93 and 94 are different partial regions of the exit pupil 90 , are arranged in the horizontal direction, and have a line-symmetrical shape with respect to a vertical line passing through the optical axis 91 . The photoelectric conversion unit 13 generates and outputs a signal corresponding to the intensity of the image formed on the microlens 10 by the light flux 73 passing through the distance measuring pupil 93 and directed to the microlens 10 of the pixel 311 . Further, the photoelectric conversion unit 14 generates and outputs a signal corresponding to the intensity of the image formed on the microlens 10 by the light flux 74 passing through the distance measuring pupil 94 and directed to the microlens 10 of the pixel 311 .

6 schematically illustrates five pixels 311 adjacent to each other near the optical axis 91. Even in the pixels 311 arranged around the screen, the respective photoelectric conversion units 13 and 14 perform distance measurement corresponding to each other. It is configured to receive light beams arriving at each microlens from the pupils 93 and 94 . Due to the microlens 10, the pair of photoelectric conversion units 13 and 14 and the different partial areas described above, that is, the pair of distance measuring pupils 93 and 94 are in a conjugate relationship with each other.

A row of output signals from the photoelectric conversion unit 13 and a row of output signals from the photoelectric conversion unit 14 of a plurality of pixels 311 horizontally arranged in a predetermined focus detection area pass through the distance measurement pupil 93 and the distance measurement pupil 94, respectively. Information about the intensity distribution of a pair of images formed on the array of pixels 311 by the light beams 73 and 74 is obtained. The focus detection unit 230 detects the amount of deviation between the output signal row of the photoelectric conversion unit 13 and the output signal row of the photoelectric conversion unit 14 by known image displacement detection arithmetic processing (correlation arithmetic processing, phase difference detection processing). Thus, the image shift amount between the pair of images is detected by a so-called split-pupil phase difference detection method. The focus detection unit 230 calculates the deviation (defocus amount) of the current imaging plane from the planned imaging plane based on this image shift amount.

FIG. 7 is a schematic optical path diagram in a state in which the subject is in focus, that is, in a focused state. FIG. 7 schematically illustrates five adjacent pixels 311 ₁ to 311 ₅ near the optical axis 91 . Light 75 emitted from a point P on the object in _FIG . On the other hand, the light 76 emitted from the point P on the subject and passing through the point 96 on the exit pupil 90 enters the photoelectric conversion portion 14 of the pixel ₃₁₁₃ . Thus, in the focused state, the lights 75 and 76 that are emitted from the same point P and have passed through different locations on the exit pupil 90 enter the photoelectric conversion units 13 and 14 of the same pixel ₃₁₁₃ , respectively.

FIG. 8 is a schematic optical path diagram in a state in which the object is out of focus, that is, in an out-of-focus state. FIG. 8 shows a so-called front focus state in which the front side of the subject is in focus. Also, FIG. 8 schematically illustrates five adjacent pixels 311 ₁ to 311 ₅ near the optical axis 91 . Light 75 emitted from a point P on the object in _FIG . On the other hand, the light 76 emitted from the point P on the object and passing through the point 96 on the exit pupil 90 enters the photoelectric conversion portion 14 of the pixel ₃₁₁₄ . Thus, in the out-of-focus state, the lights 75 and 76 that are emitted from the same point P and have passed through different locations on the exit pupil 90 are incident on the photoelectric conversion units 13 and 14 of the different pixels 311 ₂ and 311 _{4 ,} respectively. be. Considering this point, in the out-of-focus state, light emitted from different points on the subject is incident on the photoelectric conversion units 13 and 14 in the same pixel 311 . It should be noted that the same applies to a so-called rear focus state, in which the subject is in focus on the rear side.

When an image signal is generated by adding the output signal of the photoelectric conversion unit 13 and the output signal of the photoelectric conversion unit 14 in each pixel 311, in the pixels corresponding to the out-of-focus area, the light emitted from different points on the subject Since the output signals are added, the image becomes blurred. When performing image analysis processing, an image with as little blur as possible is preferable. Therefore, in this embodiment, when generating an image for analysis, the output signals of the photoelectric conversion units 13 and 14 are added in the pixel 311 corresponding to the substantially focused area to generate an image signal, and the image signal is generated in the out-of-focus area. The corresponding pixel 311 uses one of the output signals of the photoelectric conversion units 13 and 14 to generate an image signal. Note that when generating a through image or an image for recording, in order to obtain an image that makes use of blurring, the output signals of the photoelectric conversion units 13 and 14 are added for the pixels 311 corresponding to the entire photographing screen, and the image signal is obtained as an image signal. to generate Such image signal generation processing will be described in detail below.

(Generation of image signals for analysis)
A process of generating an image signal for analysis will be described. First, the focus detection unit 230 divides the photographic screen into a large number of areas and detects defocus amounts corresponding to these areas by the above-described processing, thereby detecting defocus amounts at various positions on the photographic screen. Then, a defocus amount map showing the distribution of the in-focus state is created. Based on this defocus amount map, the dividing unit 231 divides the photographing screen into a substantially focused region and a non-focused region, or divides at least one of the substantially focused region and the non-focused region into Extract. Note that the substantially in-focus area is an area where the defocus amount is smaller than a predetermined threshold (small defocus area), and the out-of-focus area is an area where the defocus amount is larger than the predetermined threshold (large defocus area).

<Generation of image signal in substantially in-focus area>
The first image signal generation unit 232 converts the output signal a of the photoelectric conversion unit 13 and the output signal b of the photoelectric conversion unit 14 to the pixel 311 corresponding to the substantially focused area divided or extracted by the division unit 231. (a+b)/2 obtained by adding and averaging is generated as an image signal. FIG. 9 is a diagram illustrating an arrangement of image signals of the pixels 311 corresponding to the substantially focused area generated in this way. The image signal of the pixel 311 has one of R, G, and B color components according to the Bayer array rule corresponding to each pixel position, and is an image with a color component different from the color component of the arranged color filter. Insufficient signal. The first image signal generation unit 232 generates an image signal of a missing color component using the image signals of the peripheral pixel positions for the image signal generated by adding the output signals of the photoelectric conversion units 13 and 14. Interpolation processing (demosaicing) is performed. During this color interpolation processing, the orientation of image structures, such as textures such as edges and linear structures, is determined, and the interpolation method is changed depending on whether the structure is in the vertical direction or the horizontal direction.

In FIG. 9, a pixel position to be interpolated (hereinafter referred to as a position of interest) (I, J) is indicated by oblique lines. The first image signal generator 232 determines the direction of the image structure with respect to the G component, which is the missing color component, at the position of interest (I, J). The first image signal generation unit 232 generates four pixel positions, (I−1, J), (I+1, J), (I, J−1), (I+1, J), and (I, J−1), which are vertically and horizontally adjacent to the target position (I, J). Using the image signal (G signal) of (I, J+1), direction determination is performed by the following equations (1) to (4). In addition, in FIG. 9, the above four positions are indicated by circles.

|G(I-1,J)-G(I+1,J)|>th_1 and |G(I,J-1)-G(I,J+1)|<th_2 (1)
|G(I-1,J)-G(I+1,J)|<th_1 and |G(I,J-1)-G(I,J+1)|>th_2 (2)
|G(I-1,J)-G(I+1,J)|>th_1 and |G(I,J-1)-G(I,J+1)|>th_2 (3)
|G(I-1,J)-G(I+1,J)|<th_1 and |G(I,J-1)-G(I,J+1)|<th_2 (4)
However, the thresholds th_1 and th_2 are predetermined values.

The first image signal generator 232 determines that there is an edge in the horizontal (row) direction with respect to the position of interest (I, J) when the above formula (1) holds. The first image signal generator 232 determines that there is an edge in the vertical (column) direction with respect to the position of interest (I, J) when the above formula (2) holds. The first image signal generation unit 232 determines that there is an edge angle with respect to the position of interest (I, J) when the above formula (3) holds. The first image signal generation unit 232 determines that the position of interest (I, J) is not on the edge when the above formula (4) holds.

FIG. 10 is a diagram for explaining interpolation processing for the G component. When the first image signal generation unit 232 determines that there is an edge in the horizontal (row) direction in the direction determination, as shown in FIG. Based on the image signals (G signals) at two adjacent pixel positions (I, J−1) and (I, J+1), the G signal at the position of interest (I, J) is obtained using the following equation (5). .
G(I,J)={G(I,J-1)+G(I,J+1)}/2 (5)

Further, when the first image signal generation unit 232 determines that there is an edge in the vertical (column) direction in the direction determination, as shown in FIG. Based on the image signals (G component signals) of two pixel positions (I−1, J) and (I+1, J) adjacent to J) in the horizontal direction, the position of interest (I , J) is obtained.
G(I,J)={G(I-1,J)+G(I+1,J)}/2 (6)

Further, when the first image signal generation unit 232 determines that the edge has a corner or does not have an edge in the direction determination, as shown in FIG. The following equation ( 7) is used to obtain the G component signal at the position of interest (I, J).
G(I,J)={G(I-1,J)+G(I+1,J)+G(I,J-1)+G(I,J+1)}/4 ・・・（ 7)

The first image signal generation unit 232 performs the process of interpolating the G component signal at the B component position and the R component position, respectively, as described above, so that the position of each pixel 311 corresponding to the substantially in-focus region is A signal of the G component can be obtained at .

FIG. 11 is a diagram for explaining interpolation processing for the R component. FIG. 11(a) is a diagram obtained by extracting the R component signal from FIG. The first image signal generation unit 232 sequentially sets the positions of the B color component and the G color component in FIG. signal is generated by interpolation. In the interpolation processing of the R component signal, first, the first image signal generation unit 232 generates the signal of the color difference component Cr as shown in FIG. 11B based on the signal of the G component and the signal of the R color component. Calculate

Then, the first image signal generation unit 232 generates four pixel positions (I, J), ( (I+2, J), (I, J+2), and (I+2, J+2), the signal of the color difference component Cr at the target position (I+1, J+1) is calculated by the following equation (8).
Cr(I+1,J+1)={Cr(I,J)+Cr(I+2,J)+Cr(I,J+2)+Cr(I+2,J+2)}/4 ・・・・(8)

In addition, the first image signal generation unit 232 generates four pixel positions (I+1, J+1), ( I, J+2), (I+1, J+3), and (I+2, J+2), the signal of the color difference component Cr at the position of interest (I+1, J+2) is calculated by the following equation (9).
Cr(I+1,J+2)={Cr(I+1,J+1)+Cr(I,J+2)+Cr(I+1,J+3)+Cr(I+2,J+2 )}/4 (9)

After obtaining the signal of the color difference component Cr at the position of each pixel 311 in this manner, the first image signal generation unit 232 adds the signal of the G component in correspondence with the position of each pixel 311 to obtain each An R component signal can be obtained at the position of the pixel 311 .

Further, the first image signal generation unit 232 sequentially sets the positions of the R color component and the G color component in FIG. A B component signal is generated by interpolation processing. The interpolation processing of the B component signal is performed by obtaining the signal of the color difference Cb at the position of each pixel 311 and then adding the signal of the G component. Since this can be performed in the same manner as the interpolation processing for the R component signal, detailed description is omitted.

In this way, the first image signal generator 232 can obtain image signals of RGB components for the pixels 311 corresponding to the substantially focused area.

<Generation of image signal in out-of-focus area>
The first image signal generation unit 232 generates either the output signal of the photoelectric conversion unit 13 or the output signal of the photoelectric conversion unit 14 for the pixel 311 corresponding to the out-of-focus area divided or extracted by the division unit 231. generates an image signal based on the output signal of Note that either one of the photoelectric conversion units 13 and 14 may be selected as the photoelectric conversion unit to be used for generating the image signal, but the photoelectric conversion units to be selected are uniform within the continuous out-of-focus area. An example of generating an image signal based on the output signal of the photoelectric conversion unit 13 will be described below.

FIG. 12 is a diagram illustrating the arrangement of the output signals of the photoelectric conversion units 13 and 14 in the pixel 311 corresponding to the out-of-focus area. The output signals of the photoelectric conversion units 13 and 14 have one of R, G, and B color components according to the Bayer arrangement rule corresponding to each pixel position, and are different from the color components of the arranged color filters. Insufficient color component signals. Here, the first image signal generation unit 232 uses the output signal of the photoelectric conversion unit 13 as the image signal for the pixel 311 corresponding to the out-of-focus area. Therefore, the first image signal generation unit 232 performs color interpolation processing to generate signals of missing color components using the output signals of the photoelectric conversion units 13 of the peripheral pixels 311 . Also in this color interpolation process, the direction of the image structure is determined, and the interpolation method is changed depending on whether the structure is in the vertical direction or the horizontal direction.

In FIG. 12, a position (i, j) corresponding to the photoelectric conversion unit 13 in a pixel position to be interpolated (hereinafter referred to as a target position) is hatched. The first image signal generator 232 determines the direction of the image structure with respect to the G component, which is the missing color component, at the position of interest (i, j). The first image signal generation unit 232 generates four positions (i−2, j), (i+2, j), (i−2, j), (i+2, j), ( Using the photoelectric conversion signals (G signals) of (i, j−1) and (i, j+1), direction determination is performed by the following equations (10) to (13). In addition, in FIG. 12, the above four positions are indicated by circles.

|G(i-2,j)-G(i+2,j)|>th_1 and |G(i,j-1)-G(i,j+1)|<th_2 (10)
|G(i-2,j)-G(i+2,j)|<th_1 and |G(i,j-1)-G(i,j+1)|>th_2 (11)
|G(i-2,j)-G(i+2,j)|>th_1 and |G(i,j-1)-G(i,j+1)|>th_2 (12)
|G(i-2,j)-G(i+2,j)|<th_1 and |G(i,j-1)-G(i,j+1)|<th_2 (13)
However, the thresholds th_1 and th_2 are predetermined values.

The first image signal generation unit 232 determines that there is an edge in the horizontal (row) direction with respect to the position of interest (i, j) when the above formula (10) holds. The first image signal generator 232 determines that there is an edge in the vertical (column) direction with respect to the position of interest (i, j) when the above equation (11) holds. The first image signal generation unit 232 determines that there is an edge angle with respect to the position of interest (i, j) when the above formula (12) holds. The first image signal generator 232 determines that the position of interest (i, j) is not on the edge when the above equation (13) holds.

FIG. 13 is a diagram for explaining interpolation processing for the G component. FIG. 13A is a diagram for explaining a case where it is determined that there is an edge in the horizontal (row) direction in the direction determination. In this case, the first image signal generation unit 232 generates two pixels corresponding to the photoelectric conversion units 13 of the pixels vertically adjacent to the pixel corresponding to the position of interest (i, j) indicated by the thick frame in FIG. 13(a). By simply averaging the output signals (G component signals) of two positions (i, j−1) and (i, j+1) as shown in the following equation (14), the G component of the target position (i, j) Ask for a signal.
G(i,j)={G(i,j-1)+G(i,j+1)}/2 (14)

FIG. 13B is a diagram for explaining a case where it is determined that there is an edge in the vertical (column) direction in the direction determination. In this case, the first image signal generation unit 232 generates two pixels corresponding to the photoelectric conversion units 13 of the pixels laterally adjacent to the pixel corresponding to the position of interest (i, j) indicated by the thick frame in FIG. 13(b). By simply averaging the output signals (G component signals) at two positions (i-2, j) and (i+2, j) as shown in the following equation (15), the G component at the position of interest (i, j) Ask for a signal.
G(i,j)={G(i-2,j)+G(i+2,j)}/2 (15)

FIG. 13(c) is a diagram for explaining a case where it is determined that there is a corner of the edge or that there is no edge in the direction determination. In this case, the first image signal generation unit 232 generates 4 pixels corresponding to the photoelectric conversion units 13 of the pixels vertically and horizontally adjacent to the pixel corresponding to the position of interest (i, j) indicated by the thick frame in FIG. 13(c). Output signals (G component signals) at three positions (i-2, j), (i+2, j), (i, j-1), and (i, j+1) are weighted as shown in the following equation (16). By averaging, the G component signal at the position of interest (i, j) is obtained.
G(i,j)={G(i-2,j)+G(i+2,j)+2*G(i,j-1)+2*G(i,j+1)}/6 (16)

In equation (16), the weighting coefficients are changed according to the distances between the position of interest and the four positions. Position (i, j −1) and (i, j+1) are closer to the position of interest (i, j), so G(i, j−1) is closer than G(i−2, j) and G(i+2, j). ) and (i, j+1) are increased.

The first image signal generation unit 232 performs the process of interpolating the G component signal as described above at the B component position and the R component position, respectively, so that the position of each pixel 311 corresponding to the out-of-focus area is A signal of the G component can be obtained at .

FIG. 14 is a diagram for explaining interpolation processing for the R component. FIG. 14(a) is a diagram obtained by extracting the R component signal from FIG. The first image signal generation unit 232 sequentially sets the positions of the B color component and the G color component in FIG. 12 as the position of interest, and uses four R component signals positioned around the position of interest to generate the R component at the position of interest. signal is generated by interpolation. In the interpolation processing of the R component signal, first, the first image signal generation unit 232 generates the signal of the color difference component Cr based on the signal of the G component and the signal of the R color component as shown in FIG. Calculate

Then, the first image signal generation unit 232 generates four signals corresponding to the photoelectric conversion units 13 of the pixels diagonally adjacent to the pixel corresponding to the position of interest (i+2, j+1) indicated by the thick frame in FIG. 14(b). Based on the signals of the color difference components Cr at positions (i, j), (i, j+2), (i+4, j), and (i+4, j+2), the color difference components at the target position (i+2, j+1) are obtained by the following equation (17). Calculate the Cr signal.
Cr(i+2,j+1)={Cr(i,j)+Cr(i,j+2)+Cr(i+4,j)+Cr(i+4,j+2)}/4 (17)

Also, the first image signal generation unit 232 generates four signals corresponding to the photoelectric conversion units 13 of the pixels vertically and horizontally adjacent to the pixel corresponding to the position of interest (i+2, j+2) indicated by the thick frame in FIG. 14(c). Based on the signals of the color difference components Cr at the positions (i+2, j+1), (i, j+2), (i+2, j+3), and (i+4, j+2), the color difference components at the target position (i+2, j+2) are obtained by the following equation (18). Calculate the Cr signal.
Cr(i+2,j+2)={Cr(i+2,j+1)+Cr(i,j+2)+Cr(i+2,j+3)+Cr(i+4,J+2 )}/4 (18)

After obtaining the signal of the color difference component Cr at the position of each pixel 311 in this manner, the first image signal generation unit 232 adds the signal of the G component in correspondence with the position of each pixel 311 to obtain each An R component signal can be obtained at the position of the pixel 311 .

Further, the first image signal generation unit 232 sequentially sets the positions of the R color component and the G color component in FIG. A B component signal is generated by interpolation processing. The interpolation processing of the B component signal is performed by obtaining the signal of the color difference Cb at the position of each pixel 311 and then adding the signal of the G component. Since this can be performed in the same manner as the interpolation processing for the R component signal, detailed description is omitted.

In this manner, the first image signal generator 232 can obtain image signals of RGB components also for the pixels 311 corresponding to the out-of-focus area.

As described above, the first image signal generation unit 232 generates an image signal based on the addition signal obtained by adding the output signals of the photoelectric conversion units 13 and 14 for the pixel 311 corresponding to the substantially focused area, and For the pixel 311 corresponding to the focus area, an image signal is generated based on the output signal of one of the photoelectric conversion units 13 and 14 .

<Determination of vignetting>
In the above description, the case where an image signal is generated using the output signal of the photoelectric conversion unit 13 in the pixel 311 corresponding to the out-of-focus area has been described. Alternatively, one of them may be selected to generate the image signal.

FIG. 15 is a diagram for explaining the relationship between the position of the exit pupil of the interchangeable lens 202 and vignetting. A pair of images are formed in the pixel areas 101 and 102 on the image sensor 212 by light beams passing through the pair of distance measuring pupils 93 and 94, and signals corresponding to the pair of images are arranged in the pixel areas 101 and 102. Therefore, the pixel 311 that is connected will output. A pair of ranging pupils 93 and 94 are located on the ranging pupil plane of the exit pupil 90 of the interchangeable lens 202, and the ranging pupil plane is in front of the microlens 10 arranged on the intended imaging plane of the interchangeable lens 202. is set at the position of the distance d. The distance d is the distance from the exit pupil 90 to the microlens 10 in the interchangeable lens 202 set as standard, and is hereinafter referred to as the standard distance. Various interchangeable lenses 202 can be attached to the camera body 203, and depending on the interchangeable lens 202 to be attached, the exit pupil 90 may be positioned at a position other than the standard distance d from the microlens 10. FIG.

A luminous flux 272 passing through the ranging pupil 93 and a luminous flux 273 passing through the ranging pupil 94 form a pair of images in the pixel area 101 . A luminous flux 282 passing through the ranging pupil 93 and a luminous flux 283 passing through the ranging pupil 94 form a pair of images in the pixel area 102 . When the exit pupil of the interchangeable lens 202 is positioned at the standard distance d, the light beams 272 and 273 received by the photoelectric conversion units 13 and 14 in the pixel area 101 and the photoelectric conversion units 13 and 14 in the pixel area 102 receive light. The light beams 282, 283 are unrestricted (ie no vignetting).

For the pixel 311 belonging to the pixel area 101 on the optical axis 91, even if the position of the exit pupil of the interchangeable lens 202 is at a position other than the standard distance d, the light beams 272 and 273 received by the photoelectric conversion units 13 and 14 are light beams. Since they are restricted to be symmetrical with respect to the axis 91, either one of the photoelectric conversion units 13 and 14 may be selected when generating an image signal.

On the other hand, for a pixel 311 belonging to the pixel area 102 away from the optical axis 91 (located in the periphery of the imaging screen), the light beams 282 and 283 received by the photoelectric conversion units 13 and 14 are different depending on the position of the exit pupil of the interchangeable lens 202. Since the methods of restriction are different, the photoelectric conversion unit with less luminous flux restriction (less vignetting, that is, greater output at uniform luminance) is selected to generate an image signal.

For example, if the exit pupil of the interchangeable lens 202 is on the pupil plane 105 at a distance d1 shorter than the standard distance d, the luminous flux 282 is less vignetting than the luminous flux 283, so the photoelectric conversion unit 13 that receives the luminous flux 282 is selected. Generate an image signal.

If the exit pupil of the interchangeable lens 202 is on the pupil plane 110 at a distance d2 longer than the standard distance d, the luminous flux 283 is less vignetting than the luminous flux 282. Generate an image signal.

Regarding the pixel area on the opposite side of the optical axis 91 from the pixel area 102, the vignetting of the pair of light beams is opposite to that of the pixel area 102. FIG. That is, when the exit pupil of the interchangeable lens 202 is located on the pupil plane 105 at the distance d1 shorter than the standard distance d, the photoelectric conversion unit 14 is selected, and the exit pupil of the interchangeable lens 202 is located on the pupil plane 110 at the distance d2 longer than the standard distance d. , the photoelectric conversion unit 13 is selected.

In this way, the vignetting determination unit 237 determines the vignetting situation, that is, which of the pair of light beams is more vignetting at the position of the pixel 311 corresponding to the out-of-focus area, based on the positional information of the exit pupil of the interchangeable lens 202. is determined, and the photoelectric conversion unit that receives the light flux with less vignetting is selected from among the photoelectric conversion units 13 and 14 . The first image signal generation unit 232 generates an image signal based on the output signal of the photoelectric conversion unit selected by the vignetting determination unit 237 at the position of the pixel 311 corresponding to the out-of-focus area. Note that if the pair of light beams received by the photoelectric conversion units 13 and 14 have the same degree of vignetting, or if neither vignetting occurs, it does not matter which of the photoelectric conversion units 13 and 14 is selected. good.

Also, the positional information of the exit pupil of the interchangeable lens 202 may be acquired from the lens drive control device 206 of the interchangeable lens 202 through communication via the electrical contact 213 . Further, the vignetting determination unit 237 may determine the above vignetting state using information on the aperture value (F number) of the interchangeable lens 202 in addition to information on the position of the exit pupil of the interchangeable lens 202 .

<Image analysis>
The image analysis unit 234 analyzes the subject image based on the image signal generated by the first image signal generation unit 232 . As described above, the image signal uses the output signal of either one of the photoelectric conversion units 13 and 14 for the out-of-focus area. analysis processing can be performed with high accuracy.

The image analysis unit 234 performs at least one of, for example, edge detection processing for detecting edges from the subject image and face recognition processing for recognizing faces in the subject image, as subject image analysis processing. As the face recognition processing, for example, face authentication processing for detecting a face in the subject image and identifying whether or not the detected face is the face of a predetermined person is performed. In the face authentication processing, the face feature amount included in the subject image is compared with the face feature amount of the registered user registered in advance in a storage unit (not shown) in the camera body 203, and the face included in the subject image is compared. identifies whether is the face of a registered user. Since edge detection processing and face recognition processing are well-known techniques, detailed description thereof will be omitted.

(Generation of through image and image signal for recording)
The second image signal generation unit 233 adds and averages the output signal a of the photoelectric conversion unit 13 and the output signal b of the photoelectric conversion unit 14 for the pixels 311 corresponding to the entire shooting screen, resulting in (a+b)/2. is generated as a through image or an image signal for recording. This image signal generation processing is performed in the same manner as the image signal generation processing of the pixels corresponding to the substantially focused area in the above-described first image signal generation unit 232, so detailed description thereof will be omitted.

The display control unit 235 causes the liquid crystal display element 216 to display a through image based on the image signal generated by the second image signal generation unit 233 . Also, when the shutter button (not shown) is fully pressed, the recording control unit 236 records an image on the memory card 219 based on the image signal generated by the second image signal generating unit 233 .

When a through image or an image for recording is generated in this way, the output signals of the photoelectric conversion units 13 and 14 are added even in the out-of-focus area, unlike the image for analysis, in order to obtain an image that makes the most of the blur. to generate an image signal.

According to the embodiment described above, the following effects are obtained.
(1) In the digital camera 201, the imaging device 212 receives a pair of light beams that have passed through a pair of pupil regions of the microlens 10 and the interchangeable lens 202 via the microlens 10, respectively, and outputs a pair of output signals. A pixel 311 having a pair of photoelectric conversion units 13 and 14 to be generated is arranged two-dimensionally. The focus detection unit 230 detects the defocus amount at various positions on the shooting screen by detecting the amount of deviation between the columns of the pair of output signals output from the plurality of pixels 311 arranged in the horizontal direction. The division unit 231 divides the photographing screen into a substantially focused area and a non-focused area based on the defocus amount. The first image signal generation unit 232 generates an image signal by adding the output signal of the photoelectric conversion unit 13 and the output signal of the photoelectric conversion unit 14 to generate an image signal for the pixel 311 in the substantially in-focus area. For the pixel 311 in the focal region, an image signal is generated from one of the output signal of the photoelectric conversion unit 13 and the output signal of the photoelectric conversion unit 14 . With such a configuration, it is possible to obtain an image with little blur even in the out-of-focus area, that is, an image with a clear image of the entire photographing screen.

(2) In the digital camera 201, the second image signal generation unit 233 generates an image signal by adding the output signal of the photoelectric conversion unit 13 and the output signal of the photoelectric conversion unit 14 for the pixels 311 of the entire shooting screen. to generate With such a configuration, it is possible to obtain an image that makes the most of the out-of-focus area blur.

(3) In the digital camera 201, the image analysis unit 234 analyzes the subject image based on the image signal generated by the first image signal generation unit 232, and the display control unit 235 analyzes the second image signal. A through image is displayed on the liquid crystal display element 216 based on the image signal generated by the generation unit 233 . With such a configuration, since the analysis processing of the subject image is performed using the image with little blur in the out-of-focus area, the analysis processing can be performed with high accuracy, and the blur in the out-of-focus area is utilized as the through image. Images can be displayed.

(4) In the digital camera 201, the vignetting determination unit 237 determines which of the pair of light beams causes more vignetting based on information on the pupil position of the interchangeable lens 202. The photoelectric conversion unit that receives the smaller luminous flux is selected. The first image signal generation unit 232 generates an image signal from the output signal of the photoelectric conversion unit selected by the vignetting determination unit 237 for the pixel 311 in the out-of-focus area. With such a configuration, an image signal can be generated using the photoelectric conversion unit with less vignetting, that is, with a larger output at uniform luminance.

(5) In the digital camera 201, the first image signal generation unit 232 generates an image signal from the output signal of the photoelectric conversion unit 13 for the pixel 311 in the out-of-focus area. When color interpolation processing is performed based on the output signal of the conversion unit 13 and an image signal is generated using the output signal of the photoelectric conversion unit 14, color interpolation processing is performed based on the output signal of the photoelectric conversion unit 14 in the peripheral pixels 311. . With such a configuration, it is possible to accurately perform color interpolation processing.

(6) The digital camera 201 has an imaging device 212 and a first image signal generator 232 . The imaging device 212 receives a pair of first and second light beams 73 and 74 that have passed through a pair of ranging pupils 93 and 94 of the exit pupil 90 of the interchangeable lens 202, respectively, and outputs a first output signal. and a second photoelectric conversion unit 14 for generating a second output signal. The first image signal generator 232 generates signals of the pixels 311 used for image signal generation from the first output signals. With such a configuration, it is possible to obtain a clear image with little blur.

(7) The digital camera 201 has an imaging device 212 and a first image signal generator 232 . The imaging device 212 has a plurality of first pixels 311 and a plurality of second pixels 311 around the first pixels 311 . The first pixel 311 receives a pair of first and second light fluxes, which are a pair of light fluxes that have passed through a pair of ranging pupils 93 and 94 of the exit pupil 90 of the interchangeable lens 202, and outputs the first and second light fluxes of the first color component. It has a first photoelectric conversion unit 13 and a second photoelectric conversion unit 14 that generate a first signal and a second signal, respectively. The second pixel 311 receives a pair of first and second luminous fluxes that have passed through a pair of ranging pupils 93 and 94 of the exit pupil 90 of the interchangeable lens 202, respectively, and converts the first and second luminous fluxes of the second color component. It has a third photoelectric conversion unit 13 and a fourth photoelectric conversion unit 14 that generate the No. 3 signal and the fourth signal, respectively. The first image signal generation unit 232 generates an image signal generated by adding the third signal and the fourth signal in the substantially in-focus area, and the second color component at the position of the first pixel 311. Interpolate the signal. The first image signal generator 232 interpolates the signal of the second color component at the position of the first pixel 311 using the third signal in the out-of-focus area. With such a configuration, it is possible to selectively use color interpolation processing. In addition, it is possible to appropriately create an image with blur and an image with little blur.

The following modifications are also within the scope of the present invention, and it is also possible to combine one or more of the modifications with the above-described embodiments.
(Modification 1)
The body drive control device 214 can also use the first image signal generator 232 as an image signal generator for generating a through image or recording image. In this case, the through image or the image for recording becomes an image with little blur even in the out-of-focus area.

In this case, body drive control device 214 switches between a mode in which an image is generated by first image signal generation section 232 and a mode in which image is generated by second image signal generation section 233, for example, according to a user operation. , may generate suitable through images or images for recording. Accordingly, it is possible for the user to select whether to shoot an image with little blur even in an out-of-focus area or to shoot an image with blurring as desired.

In addition, the body drive control device 214 generates a through image or an image for recording using an image signal generated by the image signal generated by the first image signal generating section 232, and generates a second image signal. By generating a through image or an image for recording using the image signal generated by the unit 233, two types of images may be generated. As a result, two types of images, that is, an image with little blur even in an out-of-focus area and an image with blurring can be obtained in one shot. The user can appropriately use these two types of images as desired.

(Modification 2)
The first image signal generation unit 232 generates either the output signal of the photoelectric conversion unit 13 or the output signal of the photoelectric conversion unit 14 for both the pixels 311 corresponding to the substantially focused area and the pixels 311 corresponding to the non-focused area. An image signal may be generated based on one of the output signals. The output signals of the photoelectric conversion units 13 and 14 have one of R, G, and B color components according to the Bayer arrangement rule corresponding to each pixel position, and are different from the color components of the arranged color filters. Insufficient color component signals. Therefore, when the output signal of the photoelectric conversion unit 13 is used as the image signal, the first image signal generation unit 232 uses the output signal of the photoelectric conversion unit 13 of the peripheral pixel 311 to generate the signal of the missing color component. However, when the output signal of the photoelectric conversion unit 14 is used as an image signal, color interpolation processing is performed using the output signals of the photoelectric conversion unit 14 of the peripheral pixels 311 to generate the missing color component signals.

Based on the image signal generation processing and the color interpolation processing by the first image signal generation section 232, an image signal for analysis is generated. Furthermore, based on the image signal generation processing and color interpolation processing by the first image signal generation unit 232, a through image and an image signal for recording may be generated. In this case, the through image or the image for recording becomes an image with little blur even in the out-of-focus area.

(Modification 3)
The dividing section 231 for dividing the photographing screen into the substantially focused area and the non-focused area may not be provided. In this case, the first image signal generation unit 232 adds the output signals of the photoelectric conversion units 13 and 14 of the pixels 311 corresponding to the area within the photographing screen to generate an image signal. At that time, the first image signal generation unit 232 performs a first interpolation process of adding the output signals of the photoelectric conversion units 13 and 14 of the surrounding pixels 311 to generate the missing color component signal. The image signal generated by the first image signal generator 232 in this way may be used as an image signal for analysis, or may be used as an image signal for through-the-lens image and recording.

The second image signal generation unit 233 generates an image signal based on one of the output signal of the photoelectric conversion unit 13 and the output signal of the photoelectric conversion unit 14 of the pixel 311 corresponding to the area within the shooting screen. . At that time, when the output signal of the photoelectric conversion unit 13 is used as the image signal, the second image signal generation unit 233 uses the output signal of the photoelectric conversion unit 13 of the peripheral pixel 311 to generate the missing color component signal. When the output signal of the photoelectric conversion unit 14 is used as an image signal, the output signal of the photoelectric conversion unit 14 of the peripheral pixels 311 is used to perform a second interpolation process of generating the missing color component signal. The image signal generated by the second image signal generator 233 in this way may be used as an image signal for analysis, or may be used as an image signal for a through image and recording.

(Modification 4)
The method of color interpolation processing is not limited to the example described above. For example, in the G component interpolation processing shown in FIG. 13C, output signals (G component signals) of four positions corresponding to the photoelectric conversion units 13 of pixels vertically and horizontally adjacent to the pixel corresponding to the target position may be obtained by simply averaging the G component signal at the position of interest.

Further, for example, even when it is determined that there is an edge in the horizontal direction or the vertical direction in the interpolation processing of the G component, the weighted average of the output signals of the four pixels vertically and horizontally adjacent to the pixel corresponding to the position of interest can be obtained. may be used to obtain the G component signal at the position of interest. In this case, if it is determined that there is an edge in the horizontal direction, the weighting factor to be multiplied by the output signals of the two pixels adjacent in the vertical direction is increased, and the output signals of the two pixels adjacent in the horizontal direction are multiplied. Decrease the weighting factor for On the other hand, when it is determined that there is an edge in the vertical direction, the output signals of the two pixels adjacent in the horizontal direction are multiplied by a large weighting coefficient, and the output signals of the two pixels adjacent in the vertical direction are multiplied. Decrease the weighting factor.

(Modification 5)
In the above-described embodiment, as an example of the pixel 311, a 2PD configuration having two photoelectric conversion units per pixel has been described as an example, but the number of the plurality of photoelectric conversion units is not limited to two. I do not care.

(Modification 6)
In the above-described embodiment, as an example of the image sensor 212, an example in which pixels for focus detection (pixels having a pair of photoelectric conversion units) are provided over the entire imaging surface has been described. However, in the image sensor 212, pixels for focus detection may be arranged only at positions corresponding to the focus detection areas, and ordinary pixels (pixels having one photoelectric conversion unit) may be arranged at other positions. .

Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

DESCRIPTION OF SYMBOLS 10... Microlens 13, 14... Photoelectric conversion part 201... Digital camera 202... Interchangeable lens 203... Camera body 212... Imaging element 214... Body drive control apparatus 216... Liquid crystal display element 219... Memory card , 230... focus detection unit, 231... division unit, 232... first image signal generation unit, 233... second image signal generation unit, 234... image analysis unit, 235... display control unit, 236... recording control unit, 237... vignetting determination unit, 311... pixel

Claims (8)

A first photoelectric conversion unit that photoelectrically converts light passing through a first region of an optical system to generate charges, and a second photoelectric conversion unit that photoelectrically converts light passing through a second region of the optical system to generate charges. an imaging unit having a plurality of pixels each having a part;
Based on a first signal based on the charges generated by the first photoelectric conversion unit and a second signal based on the charges generated by the second photoelectric conversion unit, an in-focus area on the imaging surface of the imaging unit a division unit that divides an out-of-focus area or extracts at least one of the in-focus area and the out-of-focus area;
a first generating unit that selects either one of the first signal and the second signal output from at least one pixel included in the out-of-focus area among the plurality of pixels to generate image data;
An imaging device comprising: The imaging device according to claim 1,
The division unit divides the focused area and the out-of-focus area based on a defocus amount based on the first signal and the second signal, or divides the in-focus area and the out-of-focus area. An imaging device that extracts at least one of a region and a region. In the imaging device according to claim 1 or claim 2,
An imaging device comprising a second generator that generates image data based on the first signal and the second signal output from pixels included in the focus area. In the imaging device according to any one of claims 1 to 3,
An imaging device comprising an analysis unit that analyzes the image data generated by the first generation unit. In the imaging device according to claim 4,
The image pickup device, wherein the analysis unit analyzes the image data of the focus area based on the first signal and the second signal. In the imaging device according to any one of claims 1 to 5,
An imaging device comprising a display unit that displays an image based on the first signal and the second signal output from pixels included in the in-focus area and the out-of-focus area among the plurality of pixels. In the imaging device according to any one of claims 1 to 6,
An imaging device comprising a recording unit that records image data based on the first signal and the second signal output from pixels included in the focused area and the out-of-focus area among the plurality of pixels. In the imaging device according to any one of claims 1 to 7,
The first generator selects one of the first signal and the second signal output from pixels included in the out-of-focus area among the plurality of pixels based on information of the optical system. imaging device.

Images