JP7022534B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image selection technique.

Digital cameras are easier to shoot than film cameras, and even low-priced cameras can shoot at high speeds. Therefore, it is easy that the number of shots is larger than expected and many similar images are shot. From such a background, a technique for easily selecting an image intended by a user from a plurality of captured images has been proposed.

In Patent Document 1, a representative image is presented to a user from a plurality of images obtained by bracket photography, and an AF evaluation value or an AE evaluation in a region of a predetermined size centered on a position specified by the user is provided. An image pickup device that selects one image based on a value is disclosed. According to the technique disclosed in Patent Document 1, it is possible to easily select an image focused on a desired region or an image in which the desired region is properly exposed from a plurality of images obtained by bracket photography.

Japanese Unexamined Patent Publication No. 2009-11165

Patent Document 1 targets a plurality of images taken of the same scene while changing the focusing distance and the exposure state. Therefore, when the scene changes due to a change in the subject position or the like, it is not possible to select an image corresponding to the scene selected by the user from the plurality of captured images.

An object of the present invention is to provide an image processing device and an image processing method that enable a user to select a more intended image even when the scene changes due to a change in the subject position or the like.

The above-mentioned object is a display means for displaying a moving image including a plurality of frames acquired at a predetermined time interval, and a predetermined operation for a first frame of the moving image displayed by the display means is detected. An extraction means for extracting candidate images from a plurality of still images recorded in parallel with a moving image, a detection means for detecting a designated subject according to a predetermined operation for each of the candidate images, and a detection means for detecting the subject. Based on the setting means for setting the focus detection area corresponding to the position where the subject is detected, the calculation means for calculating the degree of focus of the focus detection area set by the setting means, and the degree of focus for the candidate image. This is achieved by an image processing apparatus comprising:, a selection means for selecting at least one of the candidate images in which a subject is detected.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide an image processing device and an image processing method that enable a user to select a more intended image even when the scene changes due to a change in the subject position or the like.

The figure which shows the functional structure example of the digital camera which concerns on embodiment The figure regarding the image pickup element which concerns on embodiment The figure explaining the sensor pupil distance in embodiment The figure explaining the relative position of the exit pupil and the focus detection pupil in an embodiment. The figure regarding the shading characteristic of the image sensor in an embodiment. The figure regarding the focal point detection in an embodiment. The figure regarding the shading correction information and the defocus amount conversion information in an embodiment. Flow chart regarding the shooting operation of the digital camera according to the embodiment Flow chart regarding the shooting operation of the digital camera according to the embodiment Flow chart regarding the shooting operation of the digital camera according to the embodiment Flowchart regarding still image selection processing in the embodiment Flowchart regarding still image selection processing in the embodiment The figure regarding the display example in the still image selection process in an embodiment. Flowchart regarding still image selection processing in the embodiment

Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. Hereinafter, an embodiment in which the present invention is applied to an image pickup device (digital camera) as an example of an image processing device will be described. However, the imaging function is not essential in the present invention. The present invention can be implemented in any electronic device capable of handling moving images and still images recorded in parallel. Such electronic devices include, but are not limited to, smartphones, personal computers, tablet terminals, game machines, and the like, as well as image pickup devices.

● (First embodiment)
<Overall outline configuration of the image pickup device>
FIG. 1 is a schematic diagram of a digital camera according to the present embodiment. Here, it is assumed that the digital camera is composed of a main body 100 and an image pickup lens 500 that can be attached to and detached from the main body 100, but the image pickup lens and the main body may be integrated.

The luminous flux incident on the main body 100 from the image pickup lens 500 is incident on the beam splitter 103. The beam splitter 103 may be, for example, a half mirror fixed to the main body 100, but the beam splitter 103 is not limited thereto. The beam splitter 103 reflects a part of the incident light flux and transmits the part of the incident light beam to split the incident light beam into two light fluxes.

The digital camera of this embodiment has two image pickup devices 101 and 102. The image sensor 101 receives the light flux transmitted through the beam splitter 103, and the image sensor 102 receives the light flux reflected from the beam splitter 103. The image pickup surfaces of the image pickup elements 101 and 102 are optically equivalent positions when viewed from the image pickup lens 500. In other words, the image pickup devices 101 and 102 are located on the image plane that is optically conjugate to the subject via the image pickup lens 500.

The brightness of the subject image formed on the image pickup surface of the image pickup elements 101 and 102 depends on the transmittance and the reflectance of the beam splitter 103. Since the flatness and the refractive index of the half mirrors constituting the beam splitter 103 are not uniform, the image quality of the reflected image and the transmitted image may be deteriorated. When the half mirror is made of thin glass, the deterioration of the image quality due to the physical characteristics of the half mirror appears more in the reflected image than in the transmitted image.

Therefore, in the present embodiment, an image sensor 101 that receives a transmitted image is used for shooting an image in which image quality is important, such as a high-resolution still image, and the image sensor 102 that receives a reflected image is a moving image or a low-resolution still image. Used for taking pictures. However, the types of images captured by the image pickup devices 101 and 102 may be reversed depending on the physical characteristics of the beam splitter 103 and the like.

For example, the image pickup elements 101 and 102, which are CMOS area sensors, convert a subject image into an electric signal (image signal) by pixels in which photoelectric conversion elements are arranged in a matrix.

The camera CPU 104 realizes the function of a digital camera by reading the program stored in the ROM 114 into the RAM 115 and executing the program to control the operations of the main body 100 and the image pickup lens 500. For example, the camera CPU 104 controls the operation of the image pickup devices 101 and 102 to generate an image signal. Further, the camera CPU 104 reads an image signal from the image pickup elements 101 and 102, executes image processing such as A / D conversion, white balance adjustment, and color interpolation to generate recording image data and display image data. The camera CPU 104 further executes automatic exposure control (AE) and automatic focus detection (AF) based on the image data. The camera CPU 104 may perform subject detection processing such as face detection, and execute AE or AF based on the detected subject area. It should be noted that at least a part of the operations exemplified here may be executed by using a dedicated hardware circuit such as ASIC or ASSP.

The operation unit 105 is a general term for input devices such as buttons, switches, and touch panels for the user to give instructions to the main body 100. The operation unit 105 includes a power switch, a shutter switch, a menu button, a direction key, an enter button, and the like. When the camera CPU 104 detects the operation of the operation unit 105, the camera CPU 104 executes an operation according to the detected operation.

The camera CPU 104 stores the generated still image data and moving image data in a data file of a predetermined type and records them on the recording medium 106. The recording medium 106 is, for example, a semiconductor memory card. The recording destination may be an external device.

The display unit 110 is, for example, an organic EL display or a liquid crystal display. When the display unit 110 is a touch display, the user can give an instruction to the main body 100 by touching the display unit 110.

The display unit 110 is used for setting the main body 100, displaying a live view image, a photographed image, a menu screen, and the like. The setting value of the main body 100 and the GUI data such as the menu screen are stored in the ROM 114.

The display unit 107 in the finder is a display device smaller than the display unit. The camera CPU 104 can make the display unit 107 in the finder function as an electronic viewfinder.
The eyepiece 109 is provided for observing the display of the display unit 107 in the finder from the outside.

The shutter 111 is a mechanical shutter arranged on the front surface of the image pickup device 101, and is driven by a shutter drive unit 112 (for example, a motor).
The communication terminals 113 and 508 are provided on the mount portions of the main body 100 and the image pickup lens 500, respectively, and are electrically connected while the image pickup lens 500 is attached to the main body 100. The camera CPU 104 and the lens CPU 507 in the image pickup lens 500 can communicate with each other through the communication terminals 113 and 508.

ROM 114 is a rewritable non-volatile memory. The ROM 114 stores a program for execution by the camera CPU 104, setting values of the main body 100, GUI data such as a menu screen, and the like.

The RAM 115 is a system memory used by the camera CPU 104 to read a program and temporarily store image data.

<Structure of image sensor>
FIG. 2 is a diagram schematically showing a configuration example of the image pickup device 101. In the present embodiment, the image pickup devices 101 and 102 have the same structure and the same number of pixels except for the amount of eccentricity of the microlens described later. Therefore, here, the structure of the image pickup device 101 will be typically described.

FIG. 2A is a plan view of a part of the pixels of the image pickup device 101 in the vicinity of the center of the image pickup surface (near the image height 0) as viewed from the image pickup lens 500 side. Each of the plurality of pixels constituting the pixel is a square pixel having a size of 4 μm in both the horizontal direction (x) and the vertical direction (y) on the imaging surface, and has substantially the same configuration. It is assumed that the image pickup element 101 has 6000 pixels in the horizontal direction and 4000 pixels in the vertical direction in a matrix of 24 million effective pixels. The size of the pixel, that is, the imaging region can be obtained by multiplying the pixel size (pixel pitch) = 4 μm by the number of pixels, and in this example, it is 24 mm in the horizontal direction and 16 mm in the vertical direction. Each pixel is provided with one red (R), green (G), or blue (B) color filter, and the color arrangement is predetermined. A typical color arrangement is called a Bayer arrangement.

FIG. 2B is a vertical cross-sectional view of one pixel. A first photoelectric conversion unit 101a and a second photoelectric conversion unit 101b are formed in the silicon substrate 101d forming the substrate of the CMOS image sensor. In the silicon substrate 101d, the photoelectrons generated by the photoelectric conversion units 101a and 101b are converted into a voltage through a switching transistor or the like and read out from each of the wiring layers 101e. The wiring layer 101e is insulated by a transparent interlayer film 101f.

The pixel is provided with one on-chip microlens 101c and one color filter 101g for color separation. The focal position of the on-chip microlens 101c substantially coincides with the upper surfaces of the photoelectric conversion units 101a and 101b. Therefore, the photoelectric conversion units 101a and 101b receive the emitted light in different partial regions of the exit pupil of the image pickup lens 500 via the on-chip microlens 101c. Therefore, in the image sensor 101, the image based on the pixel signal obtained by the photoelectric conversion unit 101a and the image based on the image signal obtained by the photoelectric conversion unit 101b form a parallax image.

For a plurality of pixels included in a certain pixel region, the position of the image signal (A image) based on the image signal obtained from the photoelectric conversion unit 101a and the image signal (B image) based on the image signal obtained from the photoelectric conversion unit 101b. The phase difference represents the degree of focusing of the pixel region. Therefore, the camera CPU 104 can perform the focus detection of the phase difference detection method by calculating the phase difference between the image signals by the correlation calculation between the A image and the B image and converting the phase difference into the defocus amount. ..

As will be described later, the camera CPU 104 can selectively read the image signals from the photoelectric conversion units 101a and 101b for each pixel, and can read the addition signal of the image signals of the photoelectric conversion units 101a and 101b. The camera CPU 104 can generate focus detection signals (A image and B image) from the image signals separately read from the photoelectric conversion units 101a and 101b. Further, the camera CPU 104 can obtain an image signal to be used for recording or display by reading the addition signal (or adding the separately read signal).

<Structure of the readout circuit of the image sensor>
FIG. 2C is a diagram schematically showing the circuit configuration of the image pickup device 101, and the same reference figures are attached to the same configuration as that of FIG. The image pickup device 101 has a horizontal scanning circuit 121 and a vertical scanning circuit 123. Further, horizontal scanning lines 122a and 122b and vertical scanning lines 124a and 124b are wired at the boundary portion of the pixel. Image signals having a voltage corresponding to the amount of electric charge stored in the photoelectric conversion units 101a and 101b are read out to the outside via these scanning lines. The image pickup device 102 also has a circuit configuration similar to that of the image pickup device 101.

The image pickup devices 101 and 102 have first and second read modes.
The first read mode is an all-pixel read mode in which image signals are read from all pixels. It is used to generate still images and moving images for recording.
The second read mode is a thinned read mode in which the image signal is read from fewer pixels than in the first read mode. It is used to generate an image having a resolution lower than the resolution of the image sensor, such as a live view image. The live view image is an image for causing the display unit 110 to function as a viewfinder. Since the resolution of the display unit 110 is lower than the resolution of the image pickup elements 101 and 102, it is possible to shorten the time from imaging to display, reduce the processing load, and reduce the power consumption by thinning out some pixels and reading them out. In the first and second read modes, the image signal is read out for each photoelectric conversion unit. Therefore, the camera CPU 104 can generate a focus detection signal regardless of the read mode.

Although the image sensor 101 is used for still image shooting in this embodiment, it may be used for moving image shooting. For example, when the image pickup device 102 is taking a moving image in the first readout mode, the image pickup element 101 may take a moving picture in the second readout mode. Similarly, although the image sensor 102 is used for moving image shooting, it may be used for still image shooting. For example, it is possible to perform still image shooting during movie shooting, or to record a frame of the shot movie as a still image. The moving image contains a plurality of frames acquired at predetermined time intervals.

<Relationship between the exit pupil of the image sensor and the focus detection pupil of the two image sensors>
FIG. 3 is a diagram illustrating the relationship between the exit pupil of the image pickup lens and the focus detection pupil of the two image pickup elements. FIG. 3A is a diagram relating to the first image sensor 101, and FIG. 3B is a diagram relating to the second image sensor 102.

In order to realize highly accurate phase-difference AF, the base line length of the focus detection pupil for phase difference detection is long, and the exit pupil and focus detection pupil are subject to changes in the aperture value (F number) of the image pickup lens and the exit pupil distance. It is preferable that there is little misalignment. A pixel structure for satisfying these performances will be described with reference to FIG. 3A.

FIG. 3A shows an optical element of the image pickup lens 500 and two pixels 1011 and 1012 included in the image pickup element 101. The pixel 1011 is a pixel arranged in the center of the image pickup surface, that is, the image height x = 0, and the pixel 1012 is a pixel arranged in a place near the edge of the image pickup surface, for example, an image height x = 10 mm. First, the exit pupil of the image pickup lens will be described.

The image pickup lens 500 is a zoom lens having a variable focal length, and the focal length, the open F-number, and the exit pupil distance change according to the zoom operation. In FIG. 3, it is assumed that the focal length of the image pickup lens 500 is the wide-angle end. The frontmost surface of the first lens group 501 constituting the image pickup lens 500 is held by the lens barrel with the front frame 501r, and the rearmost surface of the third lens group 503 is held by the lens barrel with the rear frame 503r. A diaphragm 505 is arranged between the first lens group 501 and the third lens group 503. In FIG. 3, the diaphragm 505 shows the state of the open F value, and at this time, the outermost light rays L11a and L11b among the light fluxes reaching the pixel 1011 are the light rays regulated by the opening of the diaphragm 505. The angle θ11 formed by the light rays L11a and L11b is the angle of the light flux corresponding to the open F number at the wide-angle end.

On the other hand, the light beam in the region sandwiched between the light rays L12a and L12b reaches the pixel 1012. The light ray L12a (underlined) is a light ray regulated by the front frame 501r. Further, the light ray L12b (upper line) is a light ray regulated by the rear frame 503r. The angle θ12 formed by the upper line L12a and the underline L12b is smaller than θ11 due to vignetting.

The ray L12c between the upper line L12b and the underline L12a is the main ray, and the optical axis L11c and the main ray L12c form an angle β1. Then, the plane perpendicular to the optical axis L11c passes through the point CL1 where the optical axis L11c and the main ray L12c intersect, and is the exit pupil plane EPL1. In the exit pupil surface EPL1, the region through which the light flux reaching the pixel 1012 passes is the exit pupil seen from the pixel 1012, and the distance PL1 between the image pickup surface and the exit pupil is the exit pupil distance. Here, the exit pupil distance PL1 of the image pickup lens 500 is referred to as a lens pupil distance.

Since the vignetting changes according to the size of the aperture of the aperture (F number), the exit pupil distance of the image pickup lens 500 also changes strictly according to the F number. However, the fluctuation of the exit pupil distance depending on the F-number is often small enough to be ignored. Therefore, also in this embodiment, the exit pupil distance changes depending on the zoom state and the focus state, but is unchanged for the F-number.

Next, the pixel structure and the focus detection pupil will be described. The photoelectric conversion units 1011a and 1011b of the pixel 1011 are arranged adjacent to each other, and a dead zone having an extremely small width exists at the boundary portion thereof. The center of the boundary portion coincides with the pixel center, and the pair of photoelectric conversion units 1011a and 1011b have a shape symmetrical with respect to the pixel center in the x direction. Therefore, the distance between the position of the center of gravity of the photoelectric conversion units 1011a and 1011b and the center of the pixel in the x direction is equal.

The plane that passes through the principal point 1011p of the microlens 1011c included in the pixel and is orthogonal to the optical axis is the planned image plane (imaging plane IP1) of the image sensor 101. That is, the image obtained in a state where the focus position of the subject image formed by the image pickup lens 500 coincides with the planned image plane IP1 including the principal point 1011p group of the microlens 1011c of each pixel is focused on the subject. It is an image. Since the thickness of the microlens 1011c is about 1 μm and the distance between the apex of the microlens 1011c and the principal point 1011p is a smaller value, the principal point 1011p may be regarded as the apex of the microlens.

The optical axis of the microlens 1011c coincides with the center of the pixel (the boundary center of the pair of photoelectric conversion units) of the pixel 1011. On the other hand, in the pixel 1012 located at the end of the image pickup surface IP1, the optical axis of the microlens 1012c and the center of the pixel do not match, and the pixel 1012 is eccentric by a predetermined amount dx1 toward the optical axis of the image pickup lens 500.

Here, if the line connecting the center of the pixel and the principal point of the microlens is defined as the principal ray of the pixel, the principal ray of the pixel is inclined by a predetermined angle ω1 with respect to the optical axis and has a predetermined distance from the imaging surface IP1. It intersects the optical axis at point CS1. The virtual plane that passes through this intersection CS1 and is orthogonal to the optical axis is called the sensor pupil plane SPL1. Further, the distance PS1 between the sensor pupil surface SPL1 and the imaging surface IP1 is referred to as a sensor pupil distance. The focus detection pupils of all the pixels are substantially the same on the sensor pupil surface SPL1, and the reason will be described later.

In the pixel 1011 the height h1 between the principal point 1011p of the microlens 1011c and the upper surface of the pair of photoelectric conversion units is the optical height of the pixel. The exact optical height is a value obtained by multiplying the height h1, which is a mechanical dimension, by the refractive index of the optical path portion in the pixel, but here, for the sake of simplicity, the illustrated height h1 is used as the pixel. The height. The shape (optical power) of the microlens 1011c is set so that the focal point of the microlens 1011c substantially coincides with the upper surfaces of the photoelectric conversion unit 1011a and the photoelectric conversion unit 1011b.

In FIG. 3, for convenience, the lens pupil distance PL1 and the sensor pupil distance PS1 are illustrated to be several times the pixel height h1. However, in reality, the pixel height h1 is on the order of micrometers, while the lens pupil distance PL1 and the sensor pupil distance PS1 are on the order of several tens of millimeters, and there is a difference of about four digits in their sizes. That is, when considering the light-collecting action of the microlens 1011c, it can be considered that the exit pupil of the image pickup lens 500 and the sensor pupil surface SPL1 seen from the pixel 1011 are very far away. When the focal position of the microlens 1011c substantially coincides with the upper surface of the photoelectric conversion unit 1011a and 1011b, the upper surface of the photoelectric conversion unit 1011a and 1011b is projected on a distant plane, but the size of the projected image is large. It is proportional to the distance to the projection plane. Here, assuming that the projection surface is the sensor pupil surface SPL1, a pair of back projection images AP1a and AP1b corresponding to the pair of photoelectric conversion units 1011a and 1011b are formed on the sensor pupil surface SPL1, and this causes the luminous flux at the time of focus detection. It becomes the specified focus detection pupil.

Next, the structure of the pixel 1012 and the sensor pupil distance will be described. In FIG. 3A, assuming that the x-coordinate of the pixel 1012 is X1, the similarity relationship between the two triangles having the principal point of the microlens 1012c and the point CS1 on the sensor pupil surface as vertices
X1 / (PS1 + h1) = dx1 / h1 (Equation 1)
Will be. Here, since PS1 >> h1, the denominator PS1 + h1 in the left term can be approximated to PS1, and when Equation 4 is modified,
PS1 = X1 × (h1 / dx1) (Equation 2)
dx1 = h1 × (X1 / PS1) (Equation 3)
Is obtained.

Further, the pixel structure of the pixel 1012 and the dimensions of each part are the same as those of the pixel 1011 except for the amount of eccentricity of the microlens 1012c. Therefore, a pair of back projection images corresponding to the pair of photoelectric conversion units 1012a and 1012b are formed on the sensor pupil surface SPL1, and the back projection images are the back projection images AP1a and 1011b of the photoelectric conversion units 1011a and 1011b of the pixel 1011. It is substantially the same as AP1b. That is, if the eccentricity of the microlens is set so that Equation 3 holds for all the pixels of the image sensor 101, the sensor pupil distance is PS1 for all the pixels, and the focus detection pupil of all the pixels is on the sensor pupil surface. Is common to AP1a and AP1b. That is, the photoelectric conversion unit included in each pixel and the sensor pupil surface are in a conjugate relationship via the microlens array, and the focus detection pupils of all the pixels are substantially aligned on the sensor pupil surface.

Next, the baseline length of the focus detection pupil will be described. The pair of photoelectric conversion units 1011a and 1011b in the pixel 1011 have a symmetrical shape in the x direction with respect to the pixel center, and the distance in the x direction between the sensitivity center of gravity positions of the photoelectric conversion units 1011a and 1011b is GS1. Then, the line extending in the exit pupil direction of the imaging lens by connecting the sensitivity center of gravity position of the photoelectric conversion unit and the principal point 1011p of the microlens 1011c is defined as the main ray of the photoelectric conversion unit.

In this case, the similarity relationship between the two triangles formed by the pair of main rays S11a and S11b with the principal point 1011p of the microlens 1011c as the apex is determined.
GP1 / PS1 = GS1 / h1 (Equation 4)
Is derived, and when this is deformed, the baseline length GP1 of the focal detection pupil becomes
GP1 = GS1 × (PS1 / h1) (Equation 5)
Will be. Further, in the region where the angle formed by the pair of main rays S11a and S11b in the pair of focal detection light beams is the baseline angle α11 and the approximation of sinα≈tanα≈α is established.
α11 = GS1 / h1 = GP1 / PS1 (Equation 6)
Will be.

Even in the pixel 1012 located at the end of the image pickup surface, the distance in the x direction between the sensitivity center of gravity positions of the photoelectric conversion units 1012a and 1012b is GS1. Then, the main rays S12a and S12b of the pair of photoelectric conversion units can be defined, and the above-mentioned equations 1 and 2 hold. That is, in the pixel 1011 and the pixel 1012, the baseline lengths of the focal detection pupils have the same value GP1. On the other hand, the baseline angles of the two are strictly different, the baseline angle of the pixel 1011 is α11, the baseline angle of the pixel 1012 is α12, and the relationship is α11> α12.

The term baseline length refers to the distance between the entrance pupils of a pair of optical systems in a twin-lens stereo camera or an external measurement type phase difference detection module, but in the imaging surface phase difference detection method in the present embodiment, the sensor pupil surface SPL1 The distance GP1 between the above pair of light beams is referred to as the baseline length. In this case, if the sensor pupil distance PS1 which is the distance between the sensor pupil surface SPL1 and the imaging surface IP1 changes, the baseline length GP1 also changes. Therefore, it is necessary to compare the magnitude of the baseline length with the value at the same sensor pupil distance PS1. be. Further, the baseline lengths GP1 and GP2 shown in FIG. 3 are the baseline lengths of a single pixel, but since the focus detection pupil is eclipsed by the exit pupil of the image pickup lens at the time of focus detection, the baseline length is also shortened depending on the eclipsed situation. Since the baseline angle α11 does not depend on the lens pupil distance or the sensor pupil distance and is determined only by the pixel structure and dimensions, it is appropriate to compare the baseline angle α when comparing the focal detection capability of the image sensor alone. Is. Since the relative lateral displacement amount of the one-to-two image at the time of phase difference detection is proportional to the baseline length GP1 or the baseline angle α11, the larger these values are, the higher the focus detection resolution is.

The focus detection principle of the first image sensor 101 will be described as follows. One photoelectric conversion unit (1011a to 1012a) in the pixel having an arbitrary image height arranged in the image sensor receives the light flux passing through the focus detection pupil AP1a via the microlens array and outputs the photoelectric conversion signal Sa. .. Similarly, the other photoelectric conversion unit (1011b to 1012b) in the pixel receives the light flux passing through the focus detection pupil AP1b via the microlens array and outputs the photoelectric conversion signal Sb. Therefore, the signals Sa that are output by a plurality of pixels continuously arranged in the x direction and the signals that are connected to the signals Sb are defined as an A image signal and a B image signal. Then, the A image signal and the B image signal are laterally displaced in the x direction according to the out-of-focus state of the subject image. This amount of lateral displacement is proportional to the amount of focus shift, that is, the amount of defocus of the subject image, and is also proportional to the baseline length GP1 or the baseline angle α11 of the focus detection pupil.

Next, the pixel structure and the focus detection pupil of the second image pickup device 102 shown in FIG. 3B will be described. In the present embodiment, the two image pickup elements 101 and 102 share one image pickup lens 500, but the luminous flux from the image pickup lens 500 toward the image pickup element 102 is bent 90 degrees via the beam splitter 103. Therefore, the subject image formed on the image pickup device 102 is a mirror image, but here, a state in which the luminous flux is linearly developed and the mirror image is returned to the original normal image will be described.

FIG. 3B shows two pixels 1021 and 1022 as in FIG. 3A, the pixels 1021 are arranged in the center of the imaging surface, that is, at the image height x = 0 mm, and the pixels 1022 have the image height x =. It is arranged at 10 mm.
Pixel 1021 has the same configuration as pixel 1011 in FIG. 3A. That is, the pixel height h2 of the pixel 1021 is the same as the pixel height h1 of the pixel 1011 and the surface orthogonal to the optical axis S21c through the principal point 102p of the microlens 1021c is the image pickup surface IP2. Further, the optical axis of the microlens 1021c coincides with the center of the boundary portion of the pair of photoelectric conversion portions, and the focal position of the microlens 1021c substantially coincides with the upper surface of the pair of photoelectric conversion portions.

On the other hand, in the pixel 1022 located near the end of the image pickup surface IP2, the optical axis of the microlens 1022c is eccentric with respect to the center of the pixel, as in the pixel 1012 of the image pickup element 101, but the amount of eccentricity is It is set to a predetermined amount dx2 different from the predetermined amount dx1. Therefore, the main ray S22c (the line connecting the boundary of the pair of photoelectric conversion portions and the main point of the microlens) of the pixel 1022 is inclined by a predetermined angle ω2 with respect to the optical axis, and at the point CS2 having a predetermined distance from the imaging surface IP1. It intersects the optical axis. The virtual surface passing through the intersection CS2 and orthogonal to the optical axis is referred to as the sensor pupil surface SPL2, and the distance PS2 between the sensor pupil surface SPL2 and the imaging surface IP2 is referred to as the sensor pupil distance.

Therefore, the equations 1 to 3 relating to the sensor pupil distance described in the image pickup element 101 are expressed in the image pickup element 102.
X1 / (PS2 + h2) = dx2 / h2 (Equation 7)
PS2 = X1 × (h2 / dx2) (Equation 8)
dx2 = h2 × (X1 / PS2) (Equation 9)
Will be. The pixel structure of the pixel 1022 and the dimensions of each part are the same as those of the pixel 1021 except for the amount of eccentricity of the microlens. Therefore, a pair of back projection images corresponding to the pair of photoelectric conversion units 1022a and 1022b are formed on the sensor pupil surface SPL2, and the back projection images are substantially the same as the back projection images AP2a and AP2b of the photoelectric conversion units of the pixel 1021. Is the same.

That is, if the eccentricity of the microlens is set so that the equation 9 holds for all the pixels of the image sensor 102, the sensor pupil distance of all the pixels becomes PS2, and the focus detection pupil of all the pixels is on the sensor pupil surface. It is common to AP2a and AP2b.

Here, the eccentricity dx2 of the microlens 1022c of the pixel 1012 is smaller than that of the first image sensor 101 shown in FIG. 3A is smaller than the eccentricity dx1 of the microlens 1012c of the pixel 1011. Therefore, the magnitude relationship between the main ray angle ω at the edge of the imaging surface and the sensor pupil distance PS is
ω1> ω2 (Equation 10)
PS1 <PS2 (Equation 11)
Will be. In the manufacturing process of the image sensor, the pixel height has a predetermined manufacturing variation, so that the main ray angle ω and the sensor pupil distance PS also vary. Therefore, it is preferable to set the eccentricities dx1 and dx2 of the microlens so that the magnitude relationship between the equations 10 and 11 does not reverse even if the manufacturing variation of the pixel height occurs.

Next, the baseline length of the focus detection pupil in the image sensor 102 will be described. Similar to the pixel 1011 of the image sensor 101, the pair of photoelectric conversion units 1021a and 1012b in the pixel 1021 have a symmetrical shape in the x direction with respect to the pixel center, and the x of the sensitivity center of gravity position of the respective photoelectric conversion units 1012a and 1012b. The direction spacing is GS2. The pixel height is h2. Then, the above-mentioned equations 4 to 6 in the image pickup device 101 are replaced with the above-mentioned equations 4 to 6 in the image pickup element 102.
GP2 / PS2 = GS2 / h2 (Equation 12)
GP2 = GS2 × (PS2 / h2) (Equation 13)
α21 = GS2 / h2 = GP2 / PS2 (Equation 14)
Will be. Since h1 = h2 and GS1 = GS2, the baseline angles are the same, and α11 = α21. On the other hand, the baseline length of the focus detection pupil is GP2> GP1, and both have different values. This is due to the difference in the distance between the surface on which the focus detection pupil is projected and the imaging surface. Since the focus detection accuracy depends on the baseline angle α, the focus detection performance of a single pixel in the center of the image pickup surface is the same in the image pickup element 101 and the image pickup element 102. Since the equations 12 and 13 also hold for the pixels 1022 at the end of the image pickup surface, the focus detection performances of the image pickup element 101 and the image pickup element 102 are the same for the image pickup element alone.

As described above, the first image sensor 101 and the second image sensor 102 have the same basic characteristics of each pixel, but have different sensor pupil distances. Therefore, both have the same focus detection characteristics at the center of the image pickup surface, but the positional relationship between the exit pupil of the image pickup lens and the focus detection pupil of the image pickup element is different at the end of the image pickup surface, and the focus detection characteristics are also different. The situation will be described with reference to FIG.

<Relative positional relationship between the focus detection pupil of each pixel and the exit pupil of the imaging lens>
FIG. 4 is a diagram showing the relative positional relationship between the focus detection pupil of each pixel and the exit pupil of the image pickup lens in the sensor pupil planes SPL1 and SPL2 shown in FIG. FIG. 4A is a diagram relating to the pixel 1011 of the image pickup device 101, and FIG. 4B is a diagram relating to the pixel 1012 of the image pickup device 101. Further, FIG. 4C is a diagram relating to the pixel 1021 of the image pickup device 102, and FIG. 4D is a diagram relating to the pixel 1022 of the image pickup element 102.

Earlier, it was explained that the size of the focus detection pupil is proportional to the projection plane distance of the focus detection pupil. Since the sensor pupil distances of FIGS. 4 (a) and 4 (b) and FIGS. 4 (c) and 4 (d) are different from each other, the scale of the drawing is adjusted so that the projection distances are the same. Further, in each figure, the optical axis position on the pupil surface of the sensor is used as the origin. First, FIG. 4A will be described.

FIG. 4A shows the focus detection pupil and the exit pupil of the image pickup lens when the sensor pupil surface SPL1 is viewed from the pixel 1011 located at the center of the image pickup surface of the first image pickup element 101. The origin is the intersection of the sensor pupil surface SPL1 and the optical axis, and is the point CS1 in FIG. 3A. The pair of focus detection pupils AP1a and AP1b are optically conjugated with the pair of photoelectric conversion units 1011a and 1011b of the pixel 1011 via the microlens 1011c. However, the contours of the pair of focus detection pupils AP1a and AP1b are blurred due to the optical aberration of the microlens 1011c and the diffraction of light waves due to the minute pixel size. The efficiency of each focus detection pupil is higher toward the central part indicated by cross-hatching, in other words, the light receiving intensity is higher toward the central part. Further, the area indicated by the halftone dots has lower efficiency toward the peripheral portion, in other words, the light receiving intensity becomes lower. And in the peripheral part, a part of each other's area overlaps.

EP11 shows the exit pupil when the aperture 503 of the image pickup lens 500 is open (F2), and the circle 505 shows the exit pupil when the aperture 503 is F5.6, for example. At the center of the imaging surface, the focus detection pupil and the exit pupil are symmetrical with respect to the optical axis, which is the origin, in the pupil division direction (horizontal axis u direction). .. Therefore, regardless of the F-number at the time of focus detection, the eclipse of the pair of focus detection pupils is symmetrical in the pupil division direction, and the light receiving signal intensities of the pair of photoelectric conversion units 1011a and 1011b possessed by the pixel 1011 are equal.

FIG. 4B shows the focus detection pupil and the exit pupil of the image pickup lens when the sensor pupil surface SPL1 is viewed from the pixel 1012 located at the end of the image pickup surface of the first image sensor 101. The origin is the point CS1 in FIG. 3 (a) as in FIG. 4 (a). The shapes and positions of the pair of focus detection pupils AP1a and AP1b are the same as those of the pixel 1011 shown in FIG. 3A, and are symmetrical with respect to the origin CS1 in the pupil division direction.

The exit pupil EP12 when the aperture 503 is opened, as seen from the pixel 1012, has a shape surrounded by two arcs due to vignetting, and the width in the u-axis direction is narrowed. As described with reference to FIG. 3A, the lens pupil distance PL1 and the sensor pupil distance PS1 of the first image pickup device 101 are slightly different. Therefore, the center CD1 of the exit pupils EP12 and 505 when the aperture 503 is open and F5.6 is slightly eccentric to the left with respect to the boundary center CS1 of the focus detection pupil. However, since the amount of eccentricity is small, the eccentricity of the pair of focus detection pupils is substantially symmetric in the pupil division direction, and the light receiving signal intensities of the pair of photoelectric conversion units 1012a and 1012b of the pixel 1012 are substantially the same.

FIG. 4C shows the focus detection pupil and the exit pupil of the image pickup lens when the sensor pupil surface SPL2 is viewed from the pixel 1021 located at the center of the image pickup surface of the second image pickup element 102. The origin is the intersection of the sensor pupil surface SPL2 and the optical axis, and is the point CS2 in FIG. 3 (b). The pair of focus detection pupils AP2a and AP2b are substantially the same as those of pixel 1011 in FIG. 4 (a).

Further, the exit pupils EP21 and 505 when the aperture 503 is opened and at F5.6 are also the same as the exit pupils EP11 and 505 shown in FIG. 4 (a). Therefore, even in the pixel 1021, the eclipse of the pair of focus detection pupils is symmetrical in the pupil division direction regardless of the F number at the time of focus detection, and the light receiving signal intensities of the pair of photoelectric conversion units 1021a and 1021b are equal.

FIG. 4D shows the focus detection pupil and the exit pupil of the image pickup lens when the sensor pupil surface SPL2 is viewed from the pixel 1022 located at the end of the image pickup surface of the second image pickup element 102. The origin CS2 in FIG. 4 (d) is the point CS2 in FIG. 3 (b), as in FIG. 4 (c). The pair of focus detection pupils AP2a and AP2b are symmetrical with respect to the origin CS2 in the pupil division direction as in FIG. 4 (c).

The exit pupil EP22 when the aperture 503 is open has a shape surrounded by two arcs due to vignetting, as in FIG. 4 (b). As described with reference to FIG. 3B, the lens pupil distance PL1 and the sensor pupil distance PS2 of the second image pickup device 102 are considerably different. Therefore, the center CD2 of the exit pupils EP22 and 505 when the aperture 503 is open and F5.6 is largely deviated to the left with respect to the boundary center CS2 of the focus detection pupil. As a result, the eclipse of the pair of focus detection pupils becomes asymmetric in the pupil division direction, and the light receiving signal intensities of the pair of photoelectric conversion units 1022a and 1022b of the pixel 1022 are significantly different. This light amount difference (strictly speaking, the light amount ratio) becomes larger as the image height becomes larger and the aperture becomes smaller (the F number becomes larger).

<Shading characteristics>
5 (a) and 5 (b) show the image height and the amount of received light of the image sensor when a subject having a uniform luminance distribution is imaged when the aperture 503 of the image pickup lens 500 is open (F2) and F5.6. It is a figure explaining the relationship with, that is, the shading characteristic. 5 (a) is a characteristic diagram of the first image sensor 101, and FIG. 5 (b) is a characteristic diagram of the second image sensor 102. First, FIG. 5A will be described.

The horizontal axis of FIG. 5A is the image height in the x direction on the image pickup surface IP1 of the first image sensor 101, and the vertical axis is the output signal of the photoelectric conversion unit that receives the light flux corresponding to the pair of focus detection pupils. It is a value proportional to the amount of light received by the photoelectric conversion unit included in the pixel. S1a (F2) is an output signal of a plurality of photoelectric conversion units corresponding to one focus detection pupil AP1a when the aperture is opened (F2), and this is called an A image signal. S1b (F2) is an output signal of a plurality of photoelectric conversion units corresponding to the other focal detection pupil AP1b to be paired, and this is called a B image signal.

At the time of focus detection, the amount of defocus is detected by performing a correlation calculation between the A image signal and the B image signal obtained by photographing a subject whose brightness is not constant. FIG. 5A shows a plain uniform brightness surface. It represents the shading characteristics of the signal, that is, the focus detection system. The values of both signals decrease with increasing absolute value of image height x, which is due to vignetting of the image pickup lens. Further, in the first image sensor 101, the lens pupillary distance PL1 and the sensor pupillary distance PS1 are substantially the same, but strictly speaking, the lens pupillary distance PL1 is slightly smaller as described with reference to FIG. 3A. Therefore, as described with reference to FIGS. 4 (a) and 4 (b), the neutral point between the exit pupil center of the image pickup lens and the focus detection pupil slightly deviates as the image height increases, and the A image signal S1a in FIG. 5 (a). The values of (F2) and the B image signal S1b (F2) are slightly different depending on the image height. Further, in the region where the image height x is positive, the image A> the image B, and in the region where the image height x is negative, the image A <the image B, and the magnitude relationship between the two signals is reversed according to the positive and negative of the image height.

S1a (F5.6) and S1b (F5.6) in FIG. 5A are A-image and B-image signals when the aperture 503 is F5.6. The values of both signals become smaller as the absolute value of the image height increases, but the rate of decrease is lower than when the aperture is open. The decrease in value is due to the Cosine 4th power rule. Further, the difference in signal values at the same image height also increases as the image height increases, and the reason is the same as in the case of opening the aperture.

FIG. 5B is a characteristic diagram of the second image pickup device 102, and the meaning of the diagram is the same as that of FIG. 5A. S2a (F2) and S2b (F2) have an A image signal and a B image signal when the aperture 503 is open (F2), and S2a (F5.6) and S2b (F5.6) have an aperture 503 of F5.6. The A image signal and the B image signal at that time.

As described with reference to FIG. 3B, the second image sensor 102 has a large discrepancy between the lens pupillary distance PL1 and the sensor pupillary distance PS2. Therefore, as shown in FIG. 4D, in the region where the image height is large, the deviation between the center of the exit pupil of the image pickup lens and the neutral point of the focus detection pupil is large, and the light amount ratio of the pair of focus detection pupils becomes unbalanced. Therefore, the shading characteristic of the second image sensor 102 is also the deviation between the A image signal and the B image signal due to the positional deviation between the exit pupil and the focus detection pupil in addition to the decrease in the amount of light due to vignetting and the cosine fourth power law. Has become prominent.

At the time of focus detection, it is common to perform shading correction on the A image signal and the B image signal using the information stored in the main body 100, and then perform the interphase calculation. However, if the level difference of the original image signal is large, the shading correction error tends to be large. Further, if the original signal level is lower than the originally assumed level, the signal amplification ratio by shading correction also becomes large, and as a result, noise and the like are amplified and the focus detection error increases.

That is, when the shading characteristics of FIGS. 5 (a) and 5 (b) are compared, it can be determined that the figure (a) has a smaller level difference between the A image signal and the B image signal and has better focus detection accuracy. Therefore, when the phase difference type focus detection signal can be generated from either of the two image pickup elements 101 and 102, the discrepancy between the lens pupil distance and the sensor pupil distance is small, and the shading characteristics of the A image signal and the B image signal are good. The focus detection result of the matching image sensor is preferentially used.

The degree of matching of shading characteristics is defined and judged as follows. At a predetermined image height near the edge of the imaging surface, the larger one of the A image signal and the B image signal is Max (A, B), the smaller one is Min (A, B), and the ratio of both is the shading ratio SH. , The following equation SH = Max (A, B) / Min (A, B) (Equation 14)
Defined in. The closer the shading ratio SH is to 1, the more the shading characteristics match.

FIG. 5C is a diagram showing how the shading ratio SH of the two image pickup devices 101 and 102 changes with respect to the lens pupil distance. The horizontal axis is the lens pupil distance PL of the image pickup lens, and the vertical axis is the shading ratio SH. In the image pickup lens 500 of the present embodiment, the lens pupil distance changes from PLmin to PLmax due to changes in the zoom and focus states. SH1 (F2) and SH1 (F5.6) are shading ratios of the first image sensor 101 when the aperture 503 is open (F2) and F5.6. When the lens pupil distance PL matches the sensor pupil distance PS1 of the image sensor 101, the shading waveforms of the A image signal and the B image signal are equal, and the shading ratio SH1 becomes 1 which is the minimum value. Then, as the lens pupil distance PL deviates from the sensor pupil distance PS1, the shading ratio SH1 increases, and the smaller the aperture (the larger the F-number), the more remarkable the increase in the shading ratio.

SH2 (F2) and SH2 (F5.6) are shading ratios of the second image sensor 102 when the aperture 503 of the image pickup lens 500 is open and F5.6. When the lens pupil distance PL matches the sensor pupil distance PS2 of the image sensor 102, the shading waveforms of the A image signal and the B image signal are equal, and the shading ratio SH1 becomes 1 which is the minimum value. Then, as the lens pupil distance PL deviates from the sensor pupil distance PS2, the shading ratio SH2 increases, and the smaller the aperture (the larger the F-number), the more remarkable the increase in the shading ratio.

PSmid is a lens pupil distance in which the magnitude relationship between the shading ratios SH1 and SH2 is reversed, and this is called a boundary pupil distance. The smaller the shading ratio SH, the smaller the level difference between the A image signal and the B image signal. Therefore, it is preferable to use the image sensor having the smaller shading ratio SH when detecting the focus. Therefore, when the lens pupil distance PL at the time of focus detection is smaller than the boundary pupil distance PSmid, the focus detection result of the first image sensor 101 is prioritized, and when the lens pupil distance PL is larger than the boundary pupil distance PSmid, the second It is preferable to give priority to the focus detection result of the image pickup device 102.

<Waveform example of focus detection signal>
6 (a) and 6 (b) are waveform examples of focus detection signals (A image and B image signal) when detecting the focus lens position in focus on the subject in the focus detection region, and FIGS. 6 (a) and 6 (a) are. ) Is the waveform before shading correction, and FIG. 6B is the waveform after shading correction. In both figures, the horizontal axis is the x-coordinate of the image pickup surface, the vertical axis is the output intensity of the focus detection signal, and Sa and Sb are the A image signal and B obtained from the first image sensor 101 or the second image sensor 102. It is an image signal.

Before shading correction (FIG. 6A), there is a level difference between the A image signal and the B image signal. That is, the average value of the A image signal Sa is higher than that of the B image signal Sb, and the level difference increases as the image height x increases. On the other hand, after shading correction (FIG. 6B), the levels of the A image signal Sa and the B image signal Sb are the same. The correlation calculation for obtaining the phase difference φ of the A image and the B image signal is executed for the A image and the B image signal after shading correction.

Various interphase arithmetic expressions have been proposed, and for example, the following expression 15 is used.
C (φ) = Σ | A (i) -B (i) | (Equation 15)
A (i) and B (i) are A image signals and B image signals output from a predetermined focus detection region, and i represents a pixel number in the x-axis direction. For example, if the number of focus detection pixels existing in the x-axis direction of the focus detection region is 100, i takes a value from 1 to 100. Therefore, the right side of the equation 15 is the sum of the absolute values of the differences between the A image signal and the B image signal in the focus detection region. C (φ) is a correlation value, and φ is a relative shift amount between the A image signal and the B image signal when the above integration calculation is performed. That is, in the interphase calculation, Equation 15 is calculated while relatively shifting the A image signal and the B image signal, and φ when the correlation value C (φ) takes a minimum value is the phase difference between the A image signal and the B image signal. I reckon.

<Relationship between phase difference and correlation value of one-to-two images>
FIG. 6C is a diagram illustrating the relationship between the phase difference of the one-to-two image and the correlation value. The horizontal axis is the relative shift amount between the A image signal and the B image signal, and the vertical axis is the correlation value C (φ) of the equation 15. C1 is a correlation value of the focus detection signal of the first image sensor 101, and the correlation value shows a minimum value C1min in the shift amount φ1. C2 is a correlation value of the focus detection signal of the second image sensor 102, and the correlation value shows a minimum value C2min in the shift amount φ2. In the present embodiment, the degree of coincidence between the lens pupil distance and the sensor pupil distance is better in the first image sensor 101, so that the focus detection signal obtained from the first image sensor 101 is more reliable. In many cases. Therefore, it is estimated that the minimum value of the correlation value C (φ) is smaller than that of the first image sensor 101, and the obtained phase difference is more reliable in φ1 than in φ2.

Next, the phase difference φ obtained by the equation 15 is converted into the defocus amount DEF. Here, the phase difference φ and the defocus amount DEF have the following relationship.
φ = DEF × α (Equation 16)
DEF = φ / α = φ × K (Equation 17)
α is the baseline angle of the pair of focal detection light fluxes, and is α11 or α12 in FIG. 3A, or α21 or α22 in FIG. 3B. However, although the baseline angle shown in FIG. 3 is a value for the image sensor alone, the focal detection light beam is eclipsed by the exit pupil of the image pickup lens, so that the baseline angle α in equations 16 and 17 is the optical state of the image pickup lens. That is, it also changes depending on the F-number of the image pickup lens and the pupil distance of the lens. Therefore, in order to calculate the defocus amount DEF, information on the baseline angle α is required, and this information will be described later.

The camera CPU 104 uses Equation 17 to convert the phase difference φ into a defocus amount DEF, further converts the defocus amount DEF into the drive amount and drive direction of the focus lens 503, and transmits the defocus amount DEF to the lens CPU 507 of the image pickup lens. The lens CPU 507 drives the focus drive unit 504 based on the received lens drive amount and drive direction. In this way, the image pickup lens 500 can be focused on the subject in the focus detection region.

<Shading correction information>
7 (a) and 7 (b) show an example of shading correction information stored in advance in the ROM 115 by the main body 100. The shading correction information is, for example, a look-up table type and is stored for each image sensor. 7 (a) is information for the first image sensor 101, and FIG. 7 (b) is information for the second image sensor 102.

In FIG. 7A, in the column direction (horizontal direction) of the look-up table, the F number of the image pickup lens 500 is assigned as eight kinds of values for each stop stop from F1.4 to F16. In the row direction (longitudinal direction), the lens pupil distance of the image pickup lens 500 ranges from the minimum value lens pupil distance 1 (for example, 50 mm) to the maximum value lens pupil distance 8 (for example, 200 mm) for a plurality of discrete focal lengths. , Arithmetic progression or geometric progression. The shading correction information Fs111 to Fs188 corresponding to the image sensor 101 are stored in the locations corresponding to each F number and the lens pupil distance. Here, Fs111 to Fs188 are not a single constant, but are composed of a plurality of coefficients for defining a predetermined function.

As described in FIGS. 5A and 5B, the shading generated in the focus detection signal continuously changes according to the image height on the image pickup surface. Therefore, the shading correction function is defined by a polynomial function with the image height x and the image height y as variables, and the coefficients at each order when the shading waveforms of FIGS. 5A and 5B are approximated by the shading correction function are obtained. It may be stored as shading correction information. FIG. 7B is shading correction information for the second image sensor 102, and the configuration and the content indicated by the correction information are the same as the shading correction information for the first image sensor 101, and thus the description thereof will be omitted. ..

<Conversion information for converting the phase difference φ to the defocus amount>
7 (c) and 7 (d) are conversion information for converting the phase difference φ of the two images shown in FIG. 6 (b) into the defocus amount DEF, which corresponds to K in Equation 17 and shading correction information. It is stored in ROM 115 in advance as a look-up table similar to the above. FIG. 7 (c) is conversion information for the first image sensor 101, and FIG. 7 (d) is conversion information for the second image sensor 102.

In FIG. 7C, conversion information Fk111 to Fk188 corresponding to the image pickup device 101 is stored in a position corresponding to the F number and the lens pupil distance of the look-up table. Here, Fk111 to Fk188 are not a single constant, but are composed of a plurality of coefficients for defining a predetermined function. Since the focal detection luminous flux received by each pixel on the imaging surface during phase difference detection changes depending on the x and y coordinates on the imaging surface, the baseline length of the focal detection pupil is also continuous according to the image height. Changes to. Therefore, the conversion coefficient distribution obtained by optical calculation or actual measurement is approximated by a polynomial function with image height x and image height y as variables, and the coefficient at each order of the approximated function is converted from the phase difference φ to the defocus amount DEF. It may be stored as a coefficient to be used. FIG. 7D shows conversion information for the second image sensor 102, and since the configuration and the content indicated by the correction information are the same as the conversion information for the first image sensor 101, the description thereof will be omitted.

<Shooting process>
FIG. 8 is a flowchart showing a procedure of shooting processing in the digital camera of the present embodiment. The process shown in this flowchart is started, for example, when the camera CPU 104 detects that the user has turned on the power switch included in the operation unit 105. When the on operation of the power switch is detected, the camera CPU 104 executes the operation check of each component of the main body 100, the initialization process of the RAM 114, the program, and the like as the startup process.

When the activation process is completed, the camera CPU 104 communicates with the lens CPU 507 in S102 to acquire the information of the image pickup lens 500. The information to be acquired may include, but is not limited to, an open F-number, a focal length, an exit pupil distance PL, a focus sensitivity which is a proportional constant between a focus lens extension amount and a focus change amount, and the like.

In S103, the camera CPU 104 determines which of the still image mode and the moving image mode is set, and proceeds to S111 if it is determined to be the still image mode and to S131 if it is determined to be the moving image mode.

In S111, the camera CPU 104 starts moving image shooting with the image pickup element 101 in order to generate a live view image to be displayed in real time in order to make the display unit 110 function as an electronic viewfinder. Further, the camera CPU 104 reads an image signal from the image sensor 101 in the second read mode. When generating the focus detection signal, the pixels in the focus detection region may be read out without thinning out.

In S112, the camera CPU 104 controls the aperture during operation shooting for live view display according to the brightness of the image signal (one frame of the moving image) read in S111. The camera CPU 104 can determine the brightness of the image signal based on, for example, the average luminance value of the entire frame or the focus detection region.

S111 to S115 are steps for performing an operation related to moving image shooting and focus detection for live view display in the still image mode. In still image shooting, there is no major problem even if the aperture value at the time of moving image shooting for live view display and the aperture value at the time of shooting still image for recording are different. On the other hand, since the still image taken for recording generally has a high resolution, the allowable value of the focusing error is small. Therefore, in S112, the camera CPU 104 performs aperture control with priority given to reducing the F-number (large aperture diameter) of the aperture 503 of the image pickup lens 500.

When the F number of the aperture 503 is reduced to cause overexposure, the camera CPU 104 lowers the gain of the amplifier that amplifies the signal of the image sensor 101, or shortens the exposure time (electronic shutter speed) during operation shooting. To ensure that the captured image has an appropriate exposure.

In S113, the camera CPU 104 generates a display image from the image signal read from the image sensor 101. Then, the camera CPU 104 converts the display image into a display signal and displays it on the display unit 107 or the display unit 110 in the finder. As a result, the live view display on the display unit 107 or the display unit 110 in the finder starts.

In S114, the camera CPU 104 executes a subroutine of focus detection 1 suitable for still image shooting. The details of the subroutine will be described with reference to FIG. 9A.
In S115, the camera CPU 104 transmits the focus lens drive amount and drive direction obtained in the focus detection process of S114 to the lens CPU 507 via the communication terminals 113 and 508. The lens CPU 507 drives the focus drive unit 504 according to the received drive amount and drive direction.

In S116, the camera CPU 104 determines whether or not a still image shooting start instruction such as an operation of the shutter switch of the operation unit 105 has been input, and if it is determined that the input has been input, the process proceeds to S121. On the other hand, if it is not determined that the still image shooting start instruction has been input, the camera CPU 104 returns the process to S111 and executes the process for the next frame.

In S121, the camera CPU 104 executes the subroutine of still image shooting 1. The details of the subroutine will be described with reference to FIG. 9 (b).
In S122, the camera CPU 104 generates still image data for recording from the image signal acquired in S121. Then, the camera CPU 104 stores the still image data for recording in a data file of a predetermined type and records it on the recording medium 106.

As described above, in the present embodiment, an image signal is read out from each pixel of the image sensor 101 for each photoelectric conversion unit. Therefore, in order to generate a display image or a recording image, it is first necessary to add the image signals read from the two photoelectric conversion units for each pixel. The addition may be performed by the camera CPU 104 or by another circuit.

The camera CPU 104 performs processing such as A / D conversion, white balance adjustment, color interpolation, gamma correction, aberration correction, signal type conversion, and coding for an image signal that has become one signal for each pixel by addition processing. Apply to generate image signal data for display or recording.

In the present embodiment, the camera CPU 104 also records the image signals read out for each of the photoelectric conversion units 101a and 101b on the recording medium 106.
The process of generating image data for display or recording from the image signal may be performed by providing a dedicated circuit.
Then, the camera CPU 104 ends the still image shooting process.

Next, the operation at the time of moving image shooting will be described.
In S131, the camera CPU 104 starts shooting a moving image with the image sensor 102 in order to generate a live view image to be displayed in real time in order to make the display unit 110 (or the display unit 107 in the finder) function as an electronic viewfinder. Further, the camera CPU 104 reads an image signal from the image sensor 102 in the second read mode. When generating the focus detection signal, the pixels in the focus detection region may be read out without thinning out.

In S132, the camera CPU 104 controls the aperture during operation shooting for live view display according to the brightness of the image signal (one frame of the moving image) read in S131. The camera CPU 104 can determine the brightness of the image signal based on, for example, the average luminance value of the entire frame or the focus detection region.

In the moving image mode, the same aperture value is used for the moving image shooting for live view display and the moving image shooting for recording. Therefore, in S132, the camera CPU 104 selects an aperture value suitable for moving image shooting. If the electronic shutter speed (exposure time) is set too short during movie shooting, the still image in stop motion will be frame-by-frame at high speed, resulting in an unnatural movie. Therefore, the camera CPU 104 first selects a predetermined electronic shutter speed so as to avoid such a phenomenon. Then, the camera CPU 104 controls the aperture value (F number) and the amplification gain of the image pickup signal so that a moving image frame having an appropriate exposure can be photographed. The camera CPU 104 transmits the determined F-number to the lens CPU 507 via the communication terminals 113 and 508. The lens CPU 507 controls the aperture drive unit 506, and makes the aperture diameter of the aperture 503 a size corresponding to the designated F-number.

In S133, the camera CPU 104 generates an image for displaying an image signal read from the image sensor 102. Then, the camera CPU 104 converts the display image into a display signal and displays it on the display unit 107 or the display unit 110 in the finder. As a result, the live view display on the display unit 107 or the display unit 110 in the finder starts.

In S134, the camera CPU 104 executes a subroutine of focus detection 2 suitable for moving image shooting. The details of the subroutine will be described with reference to FIG. 10 (a).
In S135, the camera CPU 104 transmits the focus lens drive amount and drive direction obtained in the focus detection process of S134 to the lens CPU 507 via the communication terminals 113 and 508. The lens CPU 507 drives the focus drive unit 504 according to the received drive amount and drive direction.

In S141, the camera CPU 104 determines whether or not an instruction to start video recording for recording, such as an operation of the operation shooting button of the operation unit 105, has been input, and if it is determined that the input has been input, the process proceeds to S142. On the other hand, if it is not determined that the instruction to start the moving image recording for recording has been input, the camera CPU 104 advances the process to S143.

In S142, the camera CPU 104 generates moving image data (for one frame) for recording and records it on the recording medium 106. If the resolution of the moving image frame read out in S131 is less than the resolution of the moving image for recording, the camera CPU 104 may re-execute the moving image shooting for recording in S142. If the resolution of the moving image frame read by S131 is equal to or higher than the resolution of the moving image for recording, the camera CPU 104 generates moving image data for recording from the image signal read by S131 and temporarily stored in the RAM 115. be able to. The moving image data for recording may be the same as the processing for a still image except that the coding processing at the time of recording is different.

In S143, the camera CPU 104 determines whether or not a still image shooting start instruction such as an operation of the shutter switch of the operation unit 105 has been input, and if it is determined that the input has been input, the process proceeds to S144. On the other hand, if it is not determined that the still image shooting start instruction has been input, the camera CPU 104 advances the process to S146. In the present embodiment, when a still image shooting is instructed during the live view display or the moving image recording in the moving image mode, the still image can be recorded by the image sensor 101.

In S144, the camera CPU 104 executes the subroutine of still image shooting 2. The subroutine is different from the subroutine of still image shooting 1 in S121, and is a subroutine for still image shooting in parallel with moving image shooting in a state where the moving image mode is selected. Details will be described with reference to FIG. 10 (b).

In S145, the camera CPU 104 generates still image data for recording from the image signal acquired in S144. Then, the camera CPU 104 stores the still image data for recording in a data file of a predetermined type and records it on the recording medium 106. Here, too, the camera CPU 104 records the image signals read out for each of the photoelectric conversion units 101a and 101b on the recording medium 106. The specific processing content in S145 is the same as that of S122 described above.

In S146, the camera CPU 104 determines whether or not an instruction to end recording moving image shooting, such as an operation of the operation shooting button of the operation unit 105, has been input. If it is not determined that the end instruction has been input, the camera CPU 104 repeatedly executes the steps S131 to S145. On the other hand, if it is determined that the end instruction has been input, the camera CPU 104 ends the moving image shooting.

Although it has been described here that the shooting process is terminated when the recording of both the still image and the moving image is completed, the shooting process may be terminated from S122 and S146 (YES) to S103 and wait for the next shooting start instruction.

<Focus detection 1>
FIG. 9A is a flowchart showing the details of “focus detection 1” executed in S114 of FIG.
In S151, the camera CPU 104 determines whether or not the lens pupil distance PL of the image pickup lens 500 acquired from the lens CPU 507 is smaller than the boundary pupil distance PSmid described with reference to FIG. 5 (c). Then, the camera CPU 104 proceeds to S152 if it is determined that the lens pupil distance PL is smaller than the boundary pupil distance PSmid, and to S153 if it is not determined.

In S152, the camera CPU 104 executes focus detection using the image sensor 101 for still image shooting. Specifically, the camera CPU 104 generates a pair of focus detection signals (A image signal and B image signal) from the image signals read from the pixels in the focus detection region of the image sensor 101. Then, the camera CPU 104 reads out the shading correction information corresponding to the combination of the F number of the current aperture 503 and the lens pupil distance PL from the lookup table (FIG. 7A) in the ROM 115, and reads the A image signal and the B image. Apply shading correction to the signal.

Next, the camera CPU 104 calculates the phase difference φ1 between the A image signal and the B image signal after shading correction by the interphase calculation shown in Equation 15. Then, the camera CPU 104 acquires the conversion information corresponding to the combination of the F number and the lens pupil distance PL from the look-up table shown in FIG. 7 (c), and calculates the defocus amount DEF 1 using the equation 17.

In S155, the camera CPU 104 converts the defocus amount DEF1 into the focus lens drive amount and the drive direction, and ends the focus detection 1.

On the other hand, when it is determined in S151 that the current lens pupil distance PL is not smaller than the boundary pupil distance PSmid (PL ≧ PSmid), the camera CPU 104 proceeds to S153.
In S153, the camera CPU 104 starts driving the image pickup device 102 and advances the processing to S154.
In S154, the camera CPU 104 calculates the defocus amount DEF2 from the image signal acquired by the image sensor 102 in the same manner as in S152. That is, the camera CPU 104 generates a pair of focus detection signals (A image signal and B image signal) from the image signal obtained by the image sensor 102. Then, the camera CPU 104 acquires shading correction information (FIG. 7B) for the image pickup device 102 corresponding to the combination of the current F-number and the lens pupil distance PL, and shades the A image signal and the B image signal. Apply the correction.

Next, the camera CPU 104 calculates the phase difference φ2 between the A image signal and the B image signal after shading correction by the interphase calculation shown in Equation 15. Then, the camera CPU 104 acquires the conversion information corresponding to the combination of the F number and the lens pupil distance PL from the look-up table shown in FIG. 7D, and calculates the defocus amount DEF2 using the equation 17.
Then, the camera CPU 104 advances the process to S155.

<Still image shooting 1>
FIG. 9B is a flowchart of the “still image shooting 1” subroutine executed in S121 of FIG.
In S161, the camera CPU 104 transmits, for example, an F number for still image shooting determined according to the brightness of the image shot immediately before and the program diagram stored in advance in the ROM 115 to the lens CPU 507 via the communication terminals 113 and 508. .. The lens CPU 507 drives the aperture drive unit 506 so that the aperture of the aperture 503 has a size corresponding to the received F-number.

In S162, the camera CPU 104 closes the shutter 111 that has been opened for live view shooting, and shields the image sensor 101 from light.
In S163, the camera CPU 104 starts a charge storage operation for taking a still image with the image sensor 101.
In S164, the camera CPU 104 drives the front curtain and the rear curtain of the shutter 111 based on the shutter speed for still image shooting determined together with the F number, for example, and exposes the image sensor 101.

When the operation of the shutter 111 is completed, the camera CPU 104 ends the storage operation of the image sensor 101 in S165, and transfers the charge stored in the photoelectric conversion unit of each pixel to the charge storage unit (floating diffusion) in the pixel. .. The above is the process of still image shooting 1.

In this way, when shooting a still image in the still image mode, the exposure amount of the image sensor 101 is controlled by the (mechanical) shutter 111, and the image sensor 101 is shielded from light when the stored charge is transferred. As a result, the occurrence of smear and blooming can be avoided, and a high-quality still image can be obtained.

<Focus detection 2>
FIG. 10A is a flowchart showing the details of “focus detection 2” executed in S134 of FIG.
In S171, the camera CPU 104 determines whether or not the lens pupil distance PL of the image pickup lens 500 acquired from the lens CPU 507 is larger than the boundary pupil distance PSmid described with reference to FIG. 5 (c). Then, the camera CPU 104 proceeds to S172 if it is determined that the lens pupil distance PL is larger than the boundary pupil distance PSmid, and to S173 if it is not determined.

In S172, the camera CPU 104 executes focus detection using the image sensor 102 that is performing moving image shooting, and calculates the defocus amount DEF2. Since the operation at the time of focus detection is substantially the same as S154 in FIG. 9 (a), the description thereof will be omitted.

When the defocus amount DEF2 is calculated in S172, the camera CPU 104 converts the defocus amount DEF2 into the focus lens drive amount and the drive direction in S175 in the same manner as in S155, and ends the focus detection 2.

On the other hand, when it is determined in S171 that the current lens pupil distance PL is not larger than the boundary pupil distance PSmid (PL ≦ PSmid), the camera CPU 104 proceeds to S173.
In S173, the camera CPU 104 starts driving the image pickup device 101 and advances the processing to S174.
In S174, the camera CPU 104 calculates the defocus amount DEF1 in the same manner as in S152 of FIG. 9A.
Then, in S175, the camera CPU 104 converts the defocus amount DEF1 into the focus lens drive amount and the drive direction in the same manner as in S155, and ends the focus detection 2.

<Still image shooting 2>
FIG. 10B is a flowchart showing the details of “still image shooting 2” executed in S144 of FIG. During movie shooting, the shutter 111 is in the open state, and the aperture of the aperture 503 of the image pickup lens 500 is controlled to a size corresponding to the F number for movie shooting.
Therefore, in S181, the camera CPU 104 determines the exposure time when taking a still image with the image sensor 101 based on the current F-number, the set sensitivity of the image sensor 101, and the program diagram.

In S182, the camera CPU 104 starts charge accumulation in the image pickup device 101 while the shutter 111 is in the open state (exposure start by the electronic shutter).
When the determined exposure time elapses in S183, the camera CPU 104 ends the charge storage operation (exposure by the electronic shutter ends).
In S184, the camera CPU 104 transfers the accumulated charge to the charge storage unit, and ends the still image shooting 2.

As described above, in the moving image mode, when the still image shooting is performed in parallel with the moving image shooting, the still image shooting is performed while maintaining the F number controlled for the moving image shooting. Further, the exposure time is controlled by the electronic shutter while the shutter 111 remains open. Therefore, the aperture diameter of the aperture 503 does not change due to the still image shooting during the moving image recording, and the operation sound of the shutter 111 is not recorded. Therefore, it is possible to shoot a still image without affecting the moving image being recorded.

In the above description, it is assumed that the still image shooting is performed on the condition that the start instruction for still image shooting is input during the moving image recording. However, it may be configured to perform still image shooting unconditionally during moving image recording. In this case, the determination process of S143 becomes unnecessary. Alternatively, it may be a process of determining whether or not a shooting mode for constantly executing still image shooting is set while S143 is recording a moving image.

<Still image selection process>
Next, an operation of automatically selecting a still image corresponding to a scene selected by the user from the moving images recorded on the recording medium 106 from the still images recorded on the recording medium 106 will be described. The selection process is started, for example, when an execution instruction for the selection process is input from the operation unit 105, but when the playback of the moving image recorded on the recording medium 106 is temporarily stopped, etc. It may be started according to the conditions of.

In S202, the camera CPU 104 determines whether or not the selection process can be executed, and proceeds to S203 if it is determined to be possible, and to S207 if it is not determined to be possible. Specifically, the camera CPU 104 determines that the selection process can be executed unless the video recording or the still image recording is being executed for the recording medium 106.

In S203, the camera CPU 104 executes the subroutine of the still image selection 1. In the still image selection 1, the subject is detected from the still image presumed to correspond to the scene selected by the user, and the still image recorded on the recording medium 106 is automatically selected based on the subject detection result. Details will be described later with reference to FIG.
In S204, the camera CPU 104 determines whether or not the non-selectable flag is ON. The non-selectable flag is a flag set in the process of still image selection 1 (S203) or still image selection 2 (S205). ON of the non-selectable flag indicates that the still image could not be selected in the process of still image selection 1 or still image selection 2. The camera CPU 104 proceeds to S205 if the unselectable flag is determined to be ON, and proceeds to S208 if the unselectable flag is not determined to be ON.

In S205, the camera CPU 104 executes the subroutine of the still image selection 2. In the still image selection 2, a subject is detected from a moving image that is presumed to correspond to a scene selected by the user, and a still image recorded on the recording medium 106 is automatically selected based on the subject detection result. Details will be described later with reference to FIG.
In S206, the camera CPU 104 determines whether or not the non-selectable flag is ON, and proceeds to S207 if it is determined to be ON, and to S208 if it is not determined.

In S207, the camera CPU 104 displays an error on the display unit 110. The error display is a display for notifying the user that the automatic selection of the still image could not be performed, and may be, for example, a message display. When the error display confirmation operation is detected through the operation unit 105, the camera CPU 104 ends the selection process.

In S208, the camera CPU 104 displays the automatically selected still image on the display unit 110. FIG. 13B shows a display example of an automatically selected still image. When there are a plurality of selected still images, the camera CPU 104 displays a predetermined plurality of images at once. When there are a plurality of selected still images, the camera CPU 104 may display one of the selected still images and a display in which the selected image is present in the other. Further, instead of displaying on the display unit 110, or further, the camera CPU 104 may create a folder on the recording medium 106 and save the selected still image data file in the folder.

Further, in S208, the camera CPU 104 can save the selected still image as a separate file on the recording medium 106 or switch the image to be displayed by a user instruction through the operation unit 105. Further, when there are a plurality of selected still images, the camera CPU 104 allows the user to select one still image and add a check mark, transfer it to an external device, save it under another name, or print it. You may do it. For example, when the instruction to restart the reproduction of the moving image is detected or the instruction to end the selection process is detected, the camera CPU 104 ends the execution of the selection process.

<Still image selection 1>
FIG. 12 is a flowchart showing the details of “still image selection 1” executed in S203 of FIG.
In S215, the camera CPU 104 reproducibly displays the moving image recorded on the recording medium 106 on the display unit 110. Specifically, the camera CPU 104 displays the moving image together with the user interface (UI) for playback control in a state where the playback is paused at a predetermined frame (for example, the first frame). When a plurality of moving images are recorded on the recording medium 106, the camera CPU 104 may allow the user to select the moving image to be displayed, or may select the latest moving image. In addition, moving images in which still images are not recorded in parallel are excluded from the playback target.

The user interface for playback control is a GUI for instructing the start and primary stop of moving image playback and changing the playback position. Such a user interface is generally used when displaying a moving image on a web browser, for example. Since the display unit 110 is a touch display, the user can play a video, pause the playback, or move the playback position of the video by touching the GUI displayed on the display unit 110. be able to.

The purpose of the process of still image selection 1 is to finally select a still image, but first, a moving image is displayed. There are two reasons for this. The first reason is that the number of recorded pixels is smaller in one frame of a moving image than in a still image, so that the display of one frame of a moving image takes less time than the display of one still image. .. The second reason is that if it is a moving image, the play / pause function can be used, and the user can easily search for a desired scene rather than viewing the still images one by one. ..

In S216, the camera CPU 104 determines whether or not a touch operation instructing the display unit 110 to play a moving image has been performed, and if it is determined that the touch operation has been performed, the process proceeds to S219, and if not, the process proceeds to S217.

In S217, the camera CPU 104 determines whether or not a touch operation for moving the playback start position has been performed on the display unit 110, and if it is determined that the touch operation has been performed, the process proceeds to S218, and if not, the process proceeds to S216. Return.

In S218, the camera CPU 104 moves the reproduction start position to a position corresponding to the operation, displays the frame corresponding to the reproduction start position on the display unit 110, and returns the process to S216.

In S219, the camera CPU 104 starts playing the moving image from the display start position, and advances the process to S220. The camera CPU 104 reads the moving image data from the recording medium 106 into the predetermined amount RAM 115, decodes the moving image data, and displays the moving image data on the display unit 110.

In S220, the camera CPU 104 determines whether or not a touch operation instructing the display unit 110 to pause playback has been performed, and if it is determined that the touch operation has been performed, the process proceeds to S222, and if not, the process proceeds to S219. Return to and continue playing the video.

In S221, the camera CPU 104 pauses the reproduction of the moving image, displays the frame at the stop position, and proceeds to the process in S222.

In S222, the camera CPU 104 determines whether or not a touch operation for instructing the subject to the display unit 110 has been performed. The touch operation for instructing the subject may be a predetermined touch operation for the moving image frame currently being displayed.

FIG. 13A is a diagram showing an example of displaying a moving image and a selected subject.
In the substantially center of the display unit 110, one frame D1 of the moving image is displayed in a paused state. At the upper part of the display unit 110, several frames before and after including the frame D1 are reduced and displayed. The frame D1m is a reduced display of the frame D1. The play button 255, the pause button 256, and the display position change slider 257 are touch-operable GUIs.

The frame D1 includes the subject H1. It can be seen from the frame display reduced at the top that the subject H1 is approaching the camera over time.

The user can specify the subject H1 by touching, for example, the face portion of the subject H1 in the frame D1 displayed on the display unit 110. Alternatively, it may be an operation of designating diagonal vertices by a drag operation from the upper left to the lower right of the face portion. The camera CPU 104 detects the position or area specified by these operations. Then, the camera CPU 104 determines the subject area HS based on the position or area designated in the frame D1 being displayed. For example, when the face area is detected in the frame D1 being displayed and the designated position or area includes the coordinates in the face area, the camera CPU 104 can determine the face area as the subject area HS.

It is not always necessary to determine a feature area such as a face area as a subject area HS. When one point in the frame D1 is specified, the subject area includes the specified point and consists of pixels having a color similar to the specified point, or a predetermined rectangular range centered on the specified point. It may be determined as HS. Further, when a range is specified, the specified range may be determined as it is as the subject area HS.

The display example of FIG. 13A is merely an example, and for example, the display position of the frame to be reduced may be set to the left, right, top, or bottom of the frame D1. Further, the front and rear frames temporally adjacent to the frame D1 may be displayed above and below or to the left and right of the frame D1 to be smaller than the frame D1 and larger than the frame D1m.

Returning to FIG. 12, in S223, the camera CPU 104 extracts a plurality of candidate images from the still images taken in parallel with the moving image being displayed and recorded on the recording medium 106. The candidate image is a still image for which subject detection is executed in S224. It is assumed that the moving images and still images recorded in parallel are related by a predetermined method, for example, they are recorded in the same folder or another folder having a related folder name. Alternatively, the still image data file generated during the recording start and end times of the moving image may be searched from the recording medium 106.

As described above, the camera of the present embodiment has a shooting mode for continuously and in parallel recording a still image at the time of moving image recording. In this shooting mode, the still image can be continuously shot and recorded by continuously inputting the start instruction for still image shooting during the moving image recording. Therefore, it is conceivable that the number of still images recorded on the recording medium 106 in parallel with the moving image will be very large. Therefore, candidate images are extracted in order to limit the number of images for which subject detection is performed to a certain number.

In the present embodiment, the camera CPU 104 includes one still image (referred to as a still image A) whose shooting time is closest to the currently displayed frame, and a still image shot within a predetermined time before and after the shooting time of the still image A. Is extracted as a candidate image. It should be noted that the still image A, the n images taken in the immediate vicinity of the still image A, and the n still images taken immediately after the still image A may be extracted using other conditions.

In S224, the camera CPU 104 executes a subject detection process for searching the subject area HS in each of the candidate images extracted in S223. The method of subject detection is not limited, but for example, a region considered to be the subject region HS can be detected in the candidate image by pattern matching using the subject region HS as a template image. In addition, subject detection processing may be performed using one or more of the subject region HS information such as color information and edge information. In addition, when multiple candidate images whose shooting times are close to each other have a mixture of successful and unsuccessful subject detection, the image in which the subject detection has failed based on the position information of the subject area in the image in which the subject detection has succeeded. You may predict the position of the subject area of. Whether or not the subject detection is successful can be determined according to a predetermined standard according to the subject detection method. For example, when pattern matching is used, it can be determined that the subject detection is successful if the degree of correlation between the region having the highest similarity and the subject region HS is equal to or higher than the threshold value.

In S225, the camera CPU 104 sets the focus detection area for the candidate image that has succeeded in detecting the subject. For example, the camera CPU 104 can set a rectangular area having a predetermined size including at least a part of the detected subject area as the focus detection area.

In S226, the camera CPU 104 determines whether or not the focus detection area can be set for at least one candidate image, and if it is determined that the focus detection area can be set, the process proceeds to S227, and if not, the process proceeds to S230. If the ratio of the candidate images for which the focus detection area could not be set exceeds a predetermined value (for example, 80%), the camera CPU 104 determines that the focus detection area could not be set for all the candidate images. You may.

In S227, the camera CPU 104 executes the focus detection 3. Details will be described with reference to FIG. 10 (c). The focus detection 3 is performed for each of the candidate images for which the focus detection region is set in S225.

In S228, the camera CPU 104 determines whether or not the processing of the focus detection 3 has been completed for all the candidate images for which the focus detection area has been set, and if it is determined that the processing has been completed, the process proceeds to S229, and the determination must be made. If so, the execution of the focus detection 3 of S227 is continued. Since the processing of the focus detection 3 is performed for each of the candidate images for which the focus detection area is set, it may take time. Therefore, in S228, it is determined whether or not the processing of the focus detection 3 is completed.

In S229, the camera CPU 104 selects one or more still images from the candidate images based on the result of the focus detection 3, that is, the degree of focusing on the subject specified by the user, and ends the process of the focus detection 1. ..

For example, the camera CPU 104 selects one or more still images including an image having the smallest absolute value of the defocus amount calculated in the focus detection 3 (the focus is most on the focus detection region). When selecting multiple still images, for example, select all candidate images whose defocus amount absolute value is below the threshold value (focus degree is above the threshold value), or select all candidate images whose defocus amount absolute value is small (focus degree). Can be selected in order of increasing number of candidate images. The selection may be performed in combination with conditions other than the defocus amount. The camera CPU 104 stores, for example, the identification information (for example, a file name) of the selected still image in the RAM 115.

If the defocus amount calculated by the focus detection 3 exceeds a predetermined threshold value, the camera CPU 104 may proceed to S230 without selecting a still image.

In S230, the camera CPU 104 sets the non-selectable flag, which is a variable stored in the RAM 115, to ON (for example, a value 1), and ends the process of focus detection 1.

<Still image selection 2>
FIG. 14 is a flowchart showing the details of “still image selection 2” executed in S205 of FIG. The still image selection 2 is executed when the still image cannot be selected in the still image selection 1.

In S244, the camera CPU 104 extracts a plurality of candidate frames from the moving image currently being displayed. The candidate frame is a frame for executing subject detection in S245. Specifically, the camera CPU 104 extracts candidate frames based on the timing instructed by the user corresponding to the candidate images extracted in the still image selection 1. When the timing is the shooting time, the moving image frame shot in the time range including the shooting time range of the candidate image extracted in the still image selection 1 is extracted as the candidate frame. For example, it is possible to extract from the moving image frame corresponding to the earliest shooting time of the candidate image to the moving image frame corresponding to the latest shooting time of the candidate image as candidate frames. It should be noted that several frames before and after may be added. Alternatively, a frame corresponding to the shooting time of each candidate image and a predetermined number of frames before and after may be extracted as candidate frames.

In S245, the camera CPU 104 executes a subject detection process for searching the subject area HS specified in S222 of the still image selection 1 for each of the candidate frames extracted in S244. Since the details of the subject detection process may be the same as those of S224, the description thereof will be omitted.

In S246, the camera CPU 104 sets a focus detection area for the candidate image (still image) extracted in S223 of the still image selection 1 by using the subject detection result in the candidate frame (moving image frame) in S245. The still image selection 2 is executed when the focus detection area cannot be set for the candidate image in the still image selection 1. One of the reasons why the focus detection area cannot be set in the still image selection 1 is that the number of still images taken per unit time is small. On the other hand, since the number of recorded pixels of a moving image is smaller than that of a still image, the number of shots taken per unit time of the moving image is larger than the number of shots taken per unit time of the still image. Therefore, by performing subject detection on the moving image frame in S245, the probability that the subject can be detected can be increased.

In S246, the camera CPU 104 sets a focus detection area for the candidate image, for example, based on the subject detection position of the candidate frame whose shooting time is the same as or close to that of the candidate image. If the subject detection is successful in the candidate frames having the same shooting time, the camera CPU 104 sets the focus detection area for the candidate image by using the position of the detected subject area as it is. Further, if the subject detection fails in the candidate frames having the same shooting time, the camera CPU 104 uses the position of the subject area detected in the candidate frames having the closest shooting time and the difference in shooting time being less than the threshold value. Set the focus detection area for the candidate image. At this time, the focus detection area may be set by using the position of the subject area detected in the candidate frame as it is, or the focus detection area may be set by using the position predicted based on the time course of the detection position of the subject area. You may. The focus detection area is not set for the candidate images for which the subject detection has failed in any of the candidate frames whose shooting time difference is less than the predetermined time.

Subsequent steps S247 to S252 are the same processes as S226 to S231 in FIG. 12, and thus the description thereof will be omitted.

<Focus detection 3>
FIG. 14 is a flowchart showing the details of the “focus detection 3” executed in S227 of FIG. 11 and S248 of FIG.
In S301, the camera CPU 104 executes the focus detection on the candidate image in which the focus detection area is set.
First, the camera CPU 104 identifies the image signals for each of the photoelectric conversion units 101a and 101b, which correspond to the candidate images recorded on the recording medium 106. Then, the camera CPU 104 outputs a pair of focus detection signals (A image signal and B image signal) from the image signals in the focus detection region set in the candidate image among the image signals for each of the photoelectric conversion units 101a and 101b. Generate.

Then, the camera CPU 104 reads out the shading correction information corresponding to the combination of the F number of the current aperture 503 and the lens pupil distance PL from the lookup table (FIG. 7A) in the ROM 115, and reads the A image signal and the B image. Apply shading correction to the signal.

Next, the camera CPU 104 calculates the phase difference φ3 between the A image signal and the B image signal after shading correction by the interphase calculation shown in Equation 15. Then, the camera CPU 104 acquires the conversion information corresponding to the combination of the F number and the lens pupil distance PL from the look-up table shown in FIG. 7 (c), and calculates the defocus amount DEF 3 using the equation 17.

In S302, the camera CPU 104 stores the defocus amount DEF3 in association with the identification information of the candidate image, for example, in the RAM 115, and ends the focus detection 3. As described above, the focus detection 3 is executed for each candidate image.

As described above, in the present embodiment, when a moving image and a still image are recorded in parallel, the corresponding still image is automatically selected using the recorded moving image. More specifically, among the still images extracted based on the timing (for example, shooting time) specified by the user corresponding to the frame specified in the moving image, the degree of focusing in the area specified in the frame is determined in advance. A still image that is equal to or higher than the set threshold value is automatically selected. Therefore, even if the number of still images recorded in parallel with the moving image is enormous, it is possible to easily find a still image that is sufficiently focused on an arbitrary subject in an arbitrary scene. Further, by roughly designating a desired scene based on the moving image, it is possible to easily find a still image in which the main subject of the scene has a better in-focus state.

(Other embodiments)
In the above-described embodiment, the sensor pupil distance is used as a criterion for selecting the image sensor used for focus detection, but the present invention is not limited to this. For example, the shading correction information shown in FIGS. 7 (a) and 7 (b) can be used as a determination criterion.
In FIGS. 7A and 7B, when the F number at the time of focus detection is F2 and the lens pupil distance is the lens pupil distance 2, the shading correction information of the image pickup element 101 is Fs122 and the image pickup element 102 from FIG. 7A. The shading correction information of FIG. 7 (b) is Fs222 from FIG. 7 (b). Then, by comparing the values of Fs122 and Fs222, it is possible to estimate the level difference between the A image signal and the B image signal. That is, it is also possible to compare the information stored for the correction of the focus detection signal and select the image sensor to be used for the focus detection based on the result.

In addition, the present invention can be carried out in any electronic device capable of carrying out the above-mentioned selection process. In this case, the configuration related to photography and recording described in the above-described embodiment is not essential. Further, the recording destination of the moving image or the still image is not particularly limited, and may be any kind of recording medium in an arbitrary place accessible from the device performing the selection process. Further, although the case where the focus detection process (focus detection 3) for the candidate image is performed by the phase difference detection method has been described, the contrast method may be used. In this case, for the pixels in the focus detection region, known contrast evaluation values such as the size of the high frequency component may be obtained, and the degree of focusing in the focus detection region may be determined based on the contrast evaluation value instead of the defocus amount. ..

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101, 102 ... Image sensor, 103 ... Beam splitter, 104 ... Camera CPU, 106 ... Recording medium, 110 ... Display unit

Claims (9)

A display means for displaying a video containing a plurality of frames acquired at predetermined time intervals, and
An extraction means for extracting a candidate image from a plurality of still images recorded in parallel with the moving image in response to the detection of a predetermined operation on the first frame of the moving image displayed by the displaying means.
For each of the candidate images, a detection means that performs detection processing of a designated subject according to the predetermined operation, and
For the candidate image in which the subject is detected, a setting means for setting a focus detection area corresponding to the position where the subject is detected, and a setting means.
A calculation means for calculating the degree of focusing of the focus detection region set by the setting means, and a calculation means.
A selection means for selecting at least one of the candidate images in which the subject is detected based on the degree of focusing, and
An image processing device characterized by having. When the subject is not detected from the candidate image, the detection means performs the subject detection process for a plurality of frames of the moving image based on the shooting time of the candidate image.
The image processing apparatus according to claim 1, wherein the setting means sets the focus detection region on the candidate image based on the position where the subject is detected in a plurality of frames of the moving image. The image processing apparatus according to claim 1 or 2, wherein the selection means selects a candidate image having a degree of focusing equal to or higher than a threshold value among the candidate images in which the subject is detected . The extraction means is characterized in that , among the plurality of still images, a predetermined number of still images including the still images whose shooting time is closest to the shooting time of the first frame are extracted as the candidate images. The image processing apparatus according to any one of claims 1 to 3. Further, it has a determination means for determining a subject area from the first frame according to the position where the predetermined operation is performed.
The detection means performs detection processing for the subject by searching for a region similar to the subject region for each of the candidate images.
The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is characterized in that. The image processing apparatus according to any one of claims 1 to 5, wherein the selection means displays the selected candidate image together with the moving image on the display means. The image processing apparatus according to any one of claims 1 to 6, wherein the moving image is recorded on the same recording medium as the plurality of still images. It is an image processing method executed by the device.
A display process that displays a moving image containing multiple frames acquired at predetermined time intervals, and
An extraction step of extracting a candidate image from a plurality of still images recorded in parallel with the moving image in response to the detection of a predetermined operation on the first frame of the moving image displayed in the displaying step.
For each of the candidate images, a detection step of performing detection processing of a designated subject according to the predetermined operation, and a detection step.
For the candidate image in which the subject is detected, a setting step of setting a focus detection area corresponding to the position where the subject is detected, and a setting step.
A calculation step for calculating the degree of focusing of the focus detection region set in the setting step, and a calculation step.
A selection step of selecting at least one of the candidate images in which the subject is detected based on the degree of focusing, and
An image processing method characterized by having. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 7 .

Images