JP6071748B2 - Imaging apparatus and control method thereof - Google Patents

Description

  The present invention relates to an imaging apparatus capable of capturing a stereoscopic image and a control method thereof, and more particularly to autofocus (automatic focus adjustment) control during live view display.

  In a stereoscopic image capturing apparatus capable of observing a stereoscopic image using parallax, image sensors are arranged at different positions, and images that have passed through a plurality of optical paths are captured. As a result, image data with different parallax can be acquired for the same subject and displayed as a left-eye image and a right-eye image. Patent Document 1 discloses an apparatus that acquires a stereo image using two optical means and two image sensors. Patent Document 2 discloses a method for acquiring a stereo image using one optical means.

  Incidentally, the imaging apparatus has a so-called live view display function in which a user takes a picture while observing a subject on a monitor unit on the back of the apparatus. Conventionally, a contrast AF (autofocus) method is known as a method of performing a focusing operation of an imaging optical system while performing live view display. Patent Document 3 discloses a method for controlling the focus lens position so that the contrast value is maximized with respect to the focusing of the two optical systems related to the left-eye image and the right-eye image. Patent Document 4 discloses an imaging apparatus in which a plurality of pairs of focus detection dedicated pixels that are pupil-divided into a part of the pixels constituting the imaging element are arranged. Patent Document 5 discloses a method of controlling the focus lens position by calculating an image shift amount by performing a horizontal correlation operation on the left eye image and the right eye image, calculating a defocus amount using parallax, and controlling the focus lens position. Has been.

JP-A-01-202985 JP-A-58-24105 JP 2010-204385 A JP 2013-54136 A JP2013-41103A

  In the contrast AF method, since it is necessary to sequentially search for a position where the contrast value becomes maximum by moving the focus lens in the optical axis direction, there is a problem that it takes time to focus. Furthermore, when the contrast of the subject is low, there is a problem that focusing control cannot be performed unless a position where the contrast value is maximized is found.

In addition, in the case of an imaging apparatus that performs phase difference detection by arranging focus detection dedicated pixels, pixel information at the position of the focus detection dedicated pixels cannot be obtained, so interpolation is performed from pixels around the pixel to avoid pixel loss. Need to predict. For this reason, it is difficult to obtain a high-quality image. The method of calculating the defocus amount by the correlation calculation between the left-eye image and the right-eye image cannot be focused unless the image correlation calculation is performed when the contrast of the subject is low. In addition, when the subject has a periodic pattern, there is a problem in that focusing cannot be performed unless a plurality of candidates can be narrowed down to one because positions with high correlation of images periodically exist.
An object of the present invention is to provide an imaging apparatus capable of shortening a focusing time required for focus adjustment at the time of displaying an image being shot and a control method therefor.

In order to solve the above-described problems, an apparatus according to the present invention includes a plurality of photoelectric conversion units that receive and photoelectrically convert light beams that have passed through different regions of an exit pupil of an imaging optical system with respect to each microlens. An image pickup apparatus that performs focus adjustment of an image pickup optical system using an image signal output from the image pickup device, and includes image data including a left-eye image and a right-eye image from a plurality of image signals acquired from the image pickup device Image generating means for generating image, display means for acquiring the image data and displaying the image, subject specifying means for acquiring the image data and specifying a subject image, and specifying by the image data and the subject specifying means Information indicating the imaged subject is obtained, and an image shift amount between the images for the subject images included in the left-eye image and the right-eye image is calculated. A subject distance calculating unit that calculates a distance to a subject, and a focus adjustment lens that constitutes the imaging optical system when the display unit displays a captured image, thereby moving the subject distance A lens driving unit that focuses on the distance calculated by the calculating unit, and when the subject distance calculating unit calculates a plurality of distance information for the distance to the subject specified by the subject specifying unit, It calculates the difference between the maximum value and the minimum value of the distance is compared with a threshold, when the difference is equal to or larger than the threshold, you notify the fact of focusing impossible.

  According to the present invention, it is possible to reduce the focusing time required for focus adjustment when displaying an image during shooting.

It is a functional block diagram of a 1st embodiment of the present invention. It is a figure which shows the structural example of the imaging device common to embodiment of this invention. It is a figure explaining the structural example of the image pick-up element of FIG. It is a figure which shows typically pixel arrangement | positioning of the image pick-up element of FIG. It is a schematic diagram explaining the mode of light reception of the image pick-up element of FIG. It is a flowchart explaining the live view imaging | photography process in embodiment of this invention. It is a flowchart explaining the focus adjustment operation | movement in 1st Embodiment of this invention. It is a functional block diagram of a 2nd embodiment of the present invention. It is a flowchart explaining the focus adjustment operation | movement in 2nd Embodiment of this invention. It is a schematic diagram which shows the example of a screen in order to demonstrate the function of a to-be-photographed instruction | indication part.

Embodiments of the present invention will be described below with reference to the accompanying drawings.
First, an apparatus configuration common to the embodiments will be described with reference to FIGS. 2 to 4. FIG. 2 is a block diagram schematically illustrating a configuration example of the stereoscopic image capturing apparatus. The stereoscopic image capturing apparatus is, for example, a digital camera that can perform stereoscopic shooting.
The apparatus main body 100 shown in FIG. 2 includes a lens unit or a detachable lens apparatus. A CPU (Central Processing Unit) 109 controls the entire imaging apparatus. The power supply 110 supplies power to each circuit in the apparatus main body 100. A memory card 121 that is a removable recording medium can be inserted into the card slot 120. With the memory card 121 inserted into the card slot 120, the memory card 121 is electrically connected to the card input / output unit 119. In this embodiment, the memory card 121 is used as a recording medium. However, other recording media such as a hard disk, an optical disk, a magneto-optical disk, a magnetic disk, and other solid-state memories may be used.

  Of the lens group constituting the lens unit, only the lenses 101 and 102 are shown in FIG. The lens 102 is a focus adjustment lens (focus lens), and performs focus adjustment by moving back and forth in the optical axis direction. The diaphragm 103 adjusts the amount of light to the image sensor 105 with respect to light incident through the lens group. The lenses 101 and 102 and the diaphragm 103 constitute an imaging optical system. The image sensor 105, the image signal processing unit 107, and the frame memory 108 constitute a photoelectric conversion system. This photoelectric conversion system converts the subject image formed by the imaging optical system into a digital image signal or image data. Note that illustration and description of the driving mechanism units such as the lens 102 and the diaphragm 103 of the imaging optical system are omitted.

  The image sensor 105 functions as a photoelectric conversion unit that photoelectrically converts a subject image formed by the imaging optical system and outputs an image signal. As the image sensor 105, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) is used. The image sensor 105 has an electronic shutter function and can adjust the exposure time. Alternatively, the exposure time may be adjusted with a mechanical shutter instead of the electronic shutter function. The imaging element 105 has a configuration in which a plurality of photodiodes (hereinafter abbreviated as PD) are arranged for each microlens. The first PD selection / synthesis unit 106 performs pixel data processing of the image sensor 105 and outputs the processing result to the image signal processing unit 107. The first PD selection / combination unit 106 has a function of selecting a necessary image signal from the outputs of a plurality of PDs and a function of combining and outputting image signals from the selected PDs. In FIG. 2, the first PD selection / combination unit 106 is provided in the image sensor 105, but may be provided outside the image sensor 105.

FIG. 3 is a diagram schematically illustrating a configuration example of an imaging element applied to the imaging apparatus of the present embodiment. FIG. 3A shows the overall configuration of the image sensor 105. The image sensor 105 includes a pixel array 301, a vertical selection circuit 302 that selects a row in the pixel array 301, and a horizontal selection circuit 304 that selects a column in the pixel array 301. The readout circuit 303 reads out the signal of the pixel portion selected by the vertical selection circuit 302 among the pixel portions in the pixel array 301. The reading circuit 303 includes a memory for storing signals, a gain amplifier, an A (Analog) / D (Digital) converter, and the like for each column.
A serial interface (SI) unit 305 determines an operation mode of each circuit according to an instruction from the CPU 109. The vertical selection circuit 302 sequentially selects a plurality of rows of the pixel array 301 and extracts pixel signals to the readout circuit 303. The horizontal selection circuit 304 sequentially selects a plurality of pixel signals read by the reading circuit 303 for each column. In addition to the components shown in FIG. 3A, the imaging element 105 includes, for example, a timing generator that provides a timing signal to the vertical selection circuit 302, the horizontal selection circuit 304, the readout circuit 303, a control circuit, and the like. Although they exist, detailed description thereof will be omitted.

  FIG. 3B illustrates a configuration example of the pixel portion of the image sensor 105. The pixel unit 400 includes a microlens 401 as an optical element and a plurality of PDs 402a to 402i as light receiving elements. These PDs function as a photoelectric conversion unit that photoelectrically converts a received light beam to generate an image signal. Note that in the example illustrated in FIG. 3B, the number of PDs included in one pixel portion is nine, but the number of PDs may be an arbitrary number of two or more. In addition to the illustrated components, the pixel unit 400 includes, for example, a pixel amplification amplifier for reading a PD signal to the reading circuit 303, a selection switch for selecting a row, a reset switch for resetting the PD signal, and the like.

  FIG. 4 illustrates the arrangement of the pixel array. The pixel array 301 is configured by arranging N pixel units 400 in the horizontal direction and M pixel units 400 in the vertical direction in order to output two-dimensional image data. Each pixel unit 400 includes a color filter. For example, odd rows are repetitions of red (R) and green (G) color filters, and even rows are repetitions of green (G) and blue (B) color filters. That is, the plurality of pixel units 400 included in the pixel array 301 are arranged according to a predetermined pixel arrangement (in this example, a Bayer arrangement).

Next, light reception by the image sensor having the pixel configuration shown in FIG. 4 will be described. FIG. 5 is a conceptual diagram showing a state in which the light beam emitted from the exit pupil of the photographing lens enters the image sensor. Each pixel portion in the cross section 501 of the pixel array includes a microlens 502, a color filter 503, PDs 504 and 505. The PD 504 corresponds to the PD 402a in FIG. The PD 505 corresponds to the PD 402c in FIG.
In FIG. 5, the optical axis 509 indicates the center of the light beam emitted from the exit pupil 506 with respect to the pixel portion having the exit pupil 506 of the photographing lens and the microlens 502. Light emitted from the exit pupil 506 is incident on the image sensor with the optical axis 509 as the center. Partial areas 507 and 508 in the exit pupil 506 are different divided areas. Light rays 510 and 511 are outermost light rays among the light passing through the partial region 507. Light rays 512 and 513 are the outermost peripheral light rays among the light beams passing through the partial region 508. Of the light flux from the exit pupil 506, the light flux shown on the upper side in FIG. 5 enters the PD 505 with the optical axis 509 as a boundary, and the lower light flux enters the PD 504. That is, the PD 504 and the PD 505 have a characteristic of receiving light in different regions with respect to the exit pupil 506 of the photographing lens. With this characteristic, the imaging apparatus can acquire at least two images with parallax. For example, the imaging device may acquire two images in a region in the pixel unit using data obtained from a plurality of left PDs as a first line and data obtained from the plurality of right PDs as a second line. it can. The imaging device can perform a stereo image data generation process using two images with parallax as a left-eye image and a right-eye image, and display a stereoscopic image on the stereo display device. Hereinafter, the left-eye image is also referred to as “left image”, and the right-eye image is also referred to as “right image”.

  Returning to FIG. 2 and continuing the description, the first PD selecting / composing unit 106 includes the vertical selection circuit 302, the reading circuit 303, the horizontal selection circuit 304, and the SI unit 305 described with reference to FIG. The first PD selection / combination unit 106 operates according to an operation mode set by the CPU 109. This operation mode is set according to the type of shooting whose start is detected in the imaging apparatus, and includes a still image mode, a normal live view mode, and a stereo live view mode. In the still image mode, the first PD selecting / composing unit 106 outputs all PD outputs to the image signal processing unit 107 as image data. Further, in the normal live view mode, the first PD selection / combination unit 106 performs an averaging operation on the outputs of all the PDs in the pixel (see FIG. 4A), and uses the calculation result as the output value of the pixel. . The stereo live view mode includes a live view right eye mode and a live view left eye mode. In the live view right-eye mode, the first PD selecting / composing unit 106 calculates an addition average value corresponding to each output of the PDs 402a, 402d, and 402g shown in FIG. 3B and uses this as an output value of the pixel. This is executed to generate image data for the right eye for live view. In the live view left-eye mode, the first PD selection / combination unit 106 calculates an addition average value corresponding to each output of the PDs 402c, 402f, and 402i shown in FIG. 3B and uses this as an output value of the pixel. Is executed, and image data for the left eye for live view is generated. Thus, the amount of data can be reduced by adding and synthesizing the outputs of a plurality of PDs and using these calculation results as the output of one pixel.

In the live view mode, a reduction in the amount of data output from the image sensor 105 is further required to increase the frame rate. For this reason, the first PD selection / combination unit 106 performs an averaging operation on the pixels of the same color filter at the closest position to obtain an output value. In the present embodiment, during live view, a process of averaging three pixel outputs in the horizontal direction and reading one pixel every three pixels in the vertical direction is executed. Thereby, the amount of data can be reduced to 1/9. In that case, the pixel to be used will be specifically described with reference to FIG. Here, in FIG. 4, the pixel portion in the m-th row and the n-th column is referred to as “mn pixel portion”. The ranges of the natural number variables m and n are M ≧ m ≧ 1, N ≧ n ≧ 1, respectively.
What is used for the arithmetic mean calculation related to the output of three pixels in the horizontal direction is the first red pixel portion in the first row, that is, the 1-1 pixel portion, the 1-3 pixel portion, and the 1-5 pixel portion. Each output. An arithmetic operation is performed on the output of each pixel unit selected according to the above-described mode, and a value for the corresponding pixel is calculated. The arithmetic processing is similarly executed for the next red pixel portion, which is the 1-7 pixel portion, the 1-9 pixel portion, and the 1-11 pixel portion. The arithmetic processing is similarly performed on the first green pixel portion in the first row, that is, the 1-2 pixel portion, the 1-4 pixel portion, and the 1-6 pixel portion. When the process of thinning out two pixels in the vertical direction is performed, the next row is the fourth row, and the first green pixel portion, that is, the 4-1 pixel portion, the 4-3 pixel portion, and the 4-5 pixel portion In the same manner, arithmetic processing is executed. The first blue pixel portion in the fourth row is a 4-2 pixel portion, a 4-4 pixel portion, and a 4-6 pixel portion, and arithmetic processing is similarly performed on these pixels.

  The image signal processing unit 107 in FIG. 2 performs image signal processing such as shading correction on the digital image signal. The image signal processing unit 107 corrects image density non-linearity due to the characteristics of the image sensor 105 and image color bias due to the light source. The frame memory 108 functions as a buffer memory that temporarily stores image data generated by the image sensor 105 and the image signal processing unit 107. The image data stored in the frame memory 108 has been subjected to correction processing and the like, and corresponds to the digital data of the charge energy accumulated in each pixel portion of the image sensor 105. Hereinafter, the image data stored in the frame memory 108 is referred to as RAW data. In particular, the RAW data acquired by the first PD selection / combination unit 106 in the still image mode is referred to as all PD-RAW data. The RAW data acquired by the first PD selection / combination unit 106 in the case of the stereo live view mode is referred to as RAW data for both eyes. For both-eye RAW data, an averaging operation is performed on the output of the PD in each pixel unit in order to increase the frame rate. On the other hand, all PD-RAW data is intended to obtain as much detailed information as image data in order to handle the data in various image processing in a later process. Therefore, selection processing and synthesis processing are not performed for all PD-RAW data, and data based on output information of each PD in each pixel unit is individually stored in the frame memory 108. A parameter relating to the image quality of the RAW data image is called an imaging parameter, and includes a setting value Av of the aperture 103, a shutter speed Tv, ISO sensitivity, and the like.

  The first PD selecting / composing unit 106 performs data processing for generating right-eye image data and left-eye image data for stereoscopic display, and data processing for generating image data for two-dimensional display. Hereinafter, the first mode related to data generation for the right image is referred to as “right eye selection synthesis mode”, and the second mode related to data generation for the left image is referred to as “left eye selection synthesis mode”. In addition, the third mode that is not directly involved in the data selection / combination processing for the right image and the left image is referred to as a “non-selection combining mode”.

  Connected to the bus 150 are a CPU 109, a power source 110, a nonvolatile memory 111, a development processing unit 112, a RAM (Random Access Memory) 113, a display control unit 114, and a main switch 116. The CPU 109 controls the image signal readout of the image sensor 105 and controls the operation timing of each unit from the image sensor 105 to the frame memory 108. Also, release switches 117 and 118 that are operated by a user with a release button are connected to the bus 150. Hereinafter, the first release switch 117 is abbreviated as a first switch, and the second release switch 118 is abbreviated as a second switch. The shooting preparation operation is started by operating the first switch 117, and the shooting operation is performed by operating the second switch 118. The left selection button 140, the right selection button 141, the setting button 142, the development parameter change button 143, and the live view start / end button 144 are illustrated as operation members of the key input means constituting the operation unit connected to the bus 150. Further, a card input / output unit 119, a lens driving unit 122, and a second PD selection / combination unit 151 are connected to the bus 150. The lens driving unit 122 moves the focus adjustment lens 102 in the optical axis direction in accordance with a control instruction from the CPU 109.

Furthermore, a USB (Universal Serial Bus) control device 127 and a LAN (Local Area Network) control device 129 are connected to the bus 150. The USB control device 127 controls communication between the imaging device and the external device via the USB terminal 128. The LAN control device 129 controls communication between the imaging device and the external device via the wired LAN terminal 130 or the wireless LAN interface unit 131.
The nonvolatile memory 111 stores an initial camera setting value to be set in the camera when the power supply 110 is turned on by operating the main switch 116. The non-volatile memory 111 is configured by an EEPROM (Electrically Erasable Programmable Read-Only Memory) or the like so that recorded data is not lost even when the power supply 110 is shut off.

  The second PD selection / synthesis unit 151 performs a selection / synthesis process on the RAW data on the frame memory 108 or the RAM 113 based on the setting of the pixel selection / synthesis parameter. Each image data subjected to the selection / combination processing is stored in the RAM 113. The pixel selection combining parameter is a parameter indicating which PD output is combined with still image RAW data to form one pixel data. That is, RAW data synthesized for each pixel is obtained by the processing of the second PD selection / synthesis unit 151. The PD selection process and the synthesis process for determining the pixels for generating the right image data and the left image data are the same as those in the first PD selection / synthesis unit 106. In the present embodiment, the RAW data is input to the second PD selection / synthesis unit 151 via the frame memory 108 or the RAM 113, but the RAW data is directly input from the image signal processing unit 107 to the second PD selection / synthesis unit 151. Form may be sufficient.

  The development processing unit 112 performs image processing on the RAW data generated for each pixel on the frame memory 108 or the RAM 113 read by the CPU 109 based on the development parameter setting. The development parameter is a parameter relating to the image quality of the digital image data, and includes parameters relating to white balance, color interpolation, color correction, γ conversion, edge enhancement, resolution, image compression, and the like of the digital image data. Hereinafter, processing for adjusting or changing the image quality of digital image data using one or more development parameters is referred to as development processing. Each image data subjected to the development processing is stored in the RAM 113 as image data in the YUV422 format or YUV411 format, for example. The image data that has been subjected to development processing including compression processing is stored in the RAM 113 as image data in JPEG format, for example. JPEG is an abbreviation for “Joint Photographic Experts Group”. The RAM 113 temporarily stores data used when the CPU 109 performs various processes in addition to the image data that is the development processing result.

  The display control unit 114 performs drive control of the display unit 115 capable of stereoscopic display. The display unit 115 is a TFT display having a liquid crystal display element, for example. The display control unit 114 outputs the image data on the RAM 113 arranged according to the display image format to the display unit 115. Further, the display control unit 114 outputs the image data arranged in the display image format in the RAM 113 to an external device via the video output terminal 132, the D terminal 133, the HDMI (registered trademark) terminal 134, and the like. HDMI (registered trademark) is an abbreviation for “High Definition Multimedia Interface”. Hereinafter, the RAM 113 in which the image format for display is arranged is referred to as VRAM. In order to perform stereo display, the VRAM includes a right-eye VRAM and a left-eye VRAM, which store right-eye image data and left-eye image data, respectively. By reading out these image data, a stereo image can be displayed on the screen of the display unit 115.

Next, the operation of the imaging apparatus will be described.
When the user turns on the main switch 116, the CPU 109 reads a predetermined program from the memory and executes it. When the main switch 116 is turned off, the imaging apparatus enters a standby mode according to a predetermined program. The first switch 117 is turned on by an operation of the first stroke (half press) of the release button, and the second switch 118 is turned on by an operation of the second stroke (full press) of the release button. The CPU 109 controls each unit according to an instruction signal by each operation of the left selection button 140, the right selection button 141, the setting button 142, and the like illustrated in FIG. 2 and the operation state of the imaging apparatus. For example, if the operation state of the imaging apparatus is the playback state, control is performed such that the previous image data is displayed with the left selection button 140 and the next image data is displayed with the right selection button 141. The development parameters can be confirmed and set with a graphical user interface by a menu operation with the development parameter change button 143. In addition, when the user operates the live view start / end button 144, processing is performed on a regular basis (for example, 30 times per second) by fetching RAW data from the image sensor 105, developing it, and placing it in the VRAM. As a result, image data can be captured from the image sensor 105 and displayed in real time. When the user operates the live view start / end button 144 during the live view operation, the live view ends.

[First Embodiment]
Next, a first embodiment of the present invention will be described. FIG. 1 is a functional block diagram according to the first embodiment. The processing of each block is realized by controlling each component shown in FIG. 2 according to a program read from the memory by the CPU 109 and executed.
The image generation unit R100 receives the left image data and the right image data processed by the first PD selection / synthesis unit 106 and the second PD selection / synthesis unit 151 under the control of the CPU 109. The image generation unit R100 includes a display image generation unit R101, generates display image data, and outputs the display image data to the live view display unit R200. The image generation unit R100 includes a contrast calculation image generation unit R113, and outputs the contrast calculation image data to a contrast AF control unit R500 described later. Further, the image generation unit R100 outputs each data of the left image and the right image to the subject distance calculation unit R400.

  The live view display unit R200 is a functional block that acquires display image data from the image generation unit R100 and displays an image during shooting. The live view display unit R200 includes a subject instruction unit R201, and the display control unit 114 controls the display unit 115. The live view display unit R200 displays a two-dimensional image or a three-dimensional image on the screen of the display unit 115 according to a user operation instruction signal, a shooting mode, or the like. For example, when a two-dimensional image is displayed, display image data is generated from one of left-eye image data and right-eye image data, or display image data is generated by adding and synthesizing both. In the case of stereo display, a left-eye image and a right-eye image are displayed and projected respectively to the user's left eye and right eye. The apparent depth amount and pop-out amount of the subject image are determined by the parallax amount of the subject image in the left-eye image and the right-eye image, and a stereoscopic effect is obtained. The subject instruction unit R201 includes a user interface for an operator to instruct the camera of a subject. The subject instruction unit R201 outputs subject instruction data to the subject specifying unit R300. The subject instruction data includes data for specifying the subject designated by the operator. Details thereof will be described later.

The subject identifying unit R300 acquires subject instruction data from the subject instruction unit R201, identifies the subject, and outputs the subject data to the subject distance calculation unit R400. The specification of the subject is to specify the region of the subject image in the display image. Details of the subject data will be described later.
The subject distance calculation unit R400 acquires the left image data and the right image data from the image generation unit R100, and calculates distance information to the subject position specified by the subject data. Hereinafter, the distance from the camera to the subject position is referred to as “subject distance”, and the subject distance information is referred to as “subject distance information”. The subject distance information is output to the lens driving unit 122 via the control method switching unit R600. The subject distance calculation unit R400 calculates an image shift amount between the images of the subject images included in the captured left-eye image and right-eye image by stereo matching processing. The subject distance can be calculated based on the parallax of the subject.

When the lens driving unit 122 acquires the subject distance information from the subject distance calculation unit R400, the lens driving unit 122 drives the focus lens according to the subject distance to perform a focus adjustment operation. In other words, the lens driving unit 122 calculates the distance between the focus lens and the image sensor using the subject distance information, and drives the focus lens so that the subject is in focus. Since the subject can be focused with a single lens movement, sequential control is unnecessary, and the time to focus can be reduced.
The contrast AF control unit R500 performs focus control of the contrast AF method. The contrast AF control unit R500 acquires the contrast calculation image data from the contrast calculation image generation unit R113, and outputs information on the lens control amount. The control method switching unit R600 selects subject distance information from the subject distance calculation unit R400 or lens control amount information from the contrast AF control unit R500 and outputs the selected information to the lens driving unit 122.

  Next, the operation of the imaging apparatus according to the present embodiment will be described with reference to the flowcharts of FIGS. 6 and 7. FIG. 6 is a flowchart for explaining live view shooting processing of the digital camera. When the process starts in S100, the CPU 109 determines in step S101 whether the second switch 118 is on or off. When the second switch 118 is in the ON state, the process proceeds to S102, and when it is in the OFF state, the process proceeds to S110. In S102, the CPU 109 sets the mode of the first PD selection / combination unit 106 to the non-selection / combination mode, and the process proceeds to S103. In S103, imaging starts with the imaging parameters at that time, and processing for acquiring all PD-RAW data on the frame memory 108 is executed. In step S <b> 104, the development processing unit 112 performs development processing on all PD-RAW data on the frame memory 108 using development parameters for RAW images, and places the developed image data in the RAM 113.

  In S105, the CPU 109 sets the mode of the second PD selection / synthesis unit 151 to the right-eye selection / synthesis mode, and the process proceeds to S106. In S <b> 106, all PD-RAW data on the frame memory 108 is input to the second PD selection / synthesis unit 151, and RAW data for the right eye is output to the RAM 113. Further, the development processing unit 112 performs development processing on the RAW data for the right eye on the RAM 113 using the development parameters for the JPEG image, and arranges the developed image data in the RAM 113.

  In S107, the CPU 109 sets the mode of the second PD selection / synthesis unit 151 to the left-eye selection / synthesis mode, and the process proceeds to S108. In S <b> 108, all PD-RAW data on the frame memory 108 is input to the second PD selection / synthesis unit 151, and the left-eye RAW data is output to the RAM 113. Further, the development processing unit 112 performs development processing on the RAW data for the left eye on the RAM 113 using the development parameters for JPEG images, and places the developed image data in the RAM 113. In S109, the RAW image data generated in S104, the JPEG image data for the right eye generated in S106, and the JPEG image data for the left eye generated in S108 are generated as one Exif standard file. Exif is an abbreviation for “Exchangeable Image File Format”. The generated file is stored in the memory card 121 via the card input / output unit 119.

  In S110, the CPU 109 determines a live view setting operation. When the live view start / end button 144 is turned on in a state where the live view is not started, the process proceeds to S111. If the live view start / end button 144 is not turned on, the process returns to S101. If the live view start / end button 144 is turned on while the live view is started, the process returns to S101, and if the live view is not turned on, the process proceeds to S111.

  In S111, the CPU 109 sets the mode of the first PD selection / synthesis unit 106 to the right-eye selection / synthesis mode, and the process proceeds to S112. In S112, shooting starts with shooting parameters for live view, and processing for acquiring RAW data for the right eye on the frame memory 108 is executed. The development processing unit 112 performs development processing using the display development parameters, places the developed image data in the right-eye VRAM, and proceeds to S113. In S113, the CPU 109 sets the mode of the first PD selection / synthesis unit 106 to the left-eye selection / synthesis mode, and the process proceeds to S114. In S <b> 114, processing for acquiring left-eye RAW data on the frame memory 108 is executed. The development processing unit 112 performs development processing using the development parameters for display, places the developed image data in the left-eye VRAM, and then returns the processing to S110.

  Next, AF control will be described with reference to FIGS. 1 and 7. In FIG. 7, the processing start is the time when the subject instruction unit R201 in FIG. 1 outputs subject instruction data. The function of the subject instruction unit R201 will be described with reference to FIG. A screen H100 schematically shows a screen example of the display unit 115 of FIG. 2 displayed when a certain landscape is captured. This shows a state in which object images are captured in the order of H101, H102, and H103 in order from the camera. H200 indicated by a dotted line frame is a marker (display index) for the operator to instruct a subject image, and is displayed as an overlay. When the operator gives an instruction on the screen of the display unit 115 by a touch operation, the touch panel detects a contact position of a finger or the like. The display control unit 114 displays the marker on the screen with the contact position as the center. In addition, an operation button for instructing a left / right / up / down movement instruction is provided on the camera body, and the marker H200 is moved in a desired direction when the operator operates the button. H101 shown in FIG. 10A is a subject image, and the case where the marker H200 is positioned on the face of the subject image is illustrated. In AF control, focus adjustment is performed by driving the focus lens so that the subject is in focus. The subject instruction data output from the subject instruction unit R201 in FIG. 1 includes display images and marker coordinate data.

  The live view display image is an image generated by the image generation unit R100. As illustrated in FIG. 1, the image generation unit R100 includes a display image generation unit R101 and a contrast calculation image generation unit R113. The input of the image generation unit R100 is left image and right image data, and the output is display image, contrast calculation image, and left image and right image data. The display image generation unit R101 generates display image data by using the left image data and the right image data as inputs. In the present embodiment, as described in the flowchart of FIG. 6, in the case of stereoscopic image display, the display images are the left image and the right image. Therefore, the display image generation unit R101 does not perform any particular processing. However, when the stereo image data including the left-eye image and the right-eye image is acquired using one optical unit as in the present embodiment, the display image generation unit R101 adds the left image and the right image. May be generated and display image data may be output. In this case, the live view display unit R200 displays a two-dimensional image instead of a stereoscopic image.

  The contrast calculation image generation unit R113 generates contrast calculation image data using the left image data and the right image data. The contrast calculation image is an image used when the contrast AF control unit R500 calculates a contrast value. In the present embodiment, the left image and the right image output from the image generation unit R100 are the same as the left image and the right image input to the image generation unit R100. However, the present invention is not limited to this, and display correction processing may be performed on the data of each image.

  In S212 of FIG. 7, the subject identifying unit R300 processes the subject instruction data acquired from the subject instruction unit R201 to identify the subject, and outputs the subject data. For example, when the marker is small, the subject specifying unit R300 determines a region expanded outside the outline of the marker as the region of the subject image. The subject data includes the region coordinates of the subject image and the display image.

  Next, in S220, the subject distance calculation unit R400 calculates the distance to the subject included in the subject data using the left image data and the right image data. In the subject distance calculation process, the corresponding points of the left image and the right image are obtained by stereo matching, and the distance is calculated based on the relationship between the parallax, the base line length, and the focal length. This calculation method is conventionally known in the fields of robot vision and image depth measurement. Stereo matching methods can be broadly divided into two methods: region-based methods and feature point-based methods. In the present embodiment, a region-based method is adopted. In this method, a position having the highest degree of matching is searched. Corresponding points are determined based on the degree of coincidence of local image regions. A subject distance calculation unit R400 generates a local image to be subjected to stereo matching from subject data. A local image is generated by cutting out an image of a region corresponding to the region coordinates of the subject from the display image included in the subject data. Note that the target area for stereo matching is in the vicinity of the subject area coordinates included in the subject data. In general stereo matching, the entire image is processed and the distances of all objects in the image are obtained, so that the calculation time is long. However, in this embodiment, the target is only one subject, and the target area for stereo matching is narrowed down. For this reason, the calculation time is short and the processing amount of stereo matching is small, so the calculation load is small. For the evaluation value of the degree of coincidence, there is a method of using a normalized cross-correlation of luminance values, a sum of square residuals, and a sum of absolute values. In this embodiment, normalized cross-correlation of luminance values is used.

  In step S230, the control method switching unit R600 determines whether the subject distance calculation unit R400 has calculated the subject distance. If the subject distance calculation unit R400 can calculate the subject distance, the process proceeds to S300. On the other hand, if the subject distance calculation unit R400 cannot calculate the subject distance, the process proceeds to S400. As an example in which the subject distance cannot be calculated by stereo matching, there is a case where the matching degree of the image region is low because the contrast of the subject is low, and the matching degree is almost the same everywhere in the region. Therefore, there is a possibility that the position with the highest degree of coincidence cannot be specified. In addition, when the subject includes a periodic pattern, the degree of coincidence of the image regions is high, but there are a plurality of positions where the degree of coincidence is high. Therefore, there is a possibility that the position with the highest degree of coincidence cannot be specified. However, even if the subject cannot be stereo-matched due to low contrast, the contrast may increase when the focus lens is moved back and forth for focus adjustment. In such a case, contrast AF is effective. Even if the subject includes a periodic pattern, it does not affect the contrast AF, so it is effective to switch to the contrast AF method.

  In S400, a focus adjustment operation is performed by lens control in the contrast AF method. The contrast value calculation unit R501 receives image data for contrast calculation in order to calculate a contrast value. The contrast value calculation unit R501 calculates the contrast value as an evaluation value. The contrast AF control unit R500 sequentially obtains the lens position where the contrast value is maximized by a so-called hill-climbing method. The control method switching unit R600 selects the lens control amount output from the contrast AF control unit R500 and outputs it to the lens driving unit 122. The lens driving unit 122 moves the focus lens forward (subject side) or backward along the optical axis based on the acquired lens control amount.

  In S300, the control method switching unit R600 selects the subject distance information output from the subject distance calculation unit R400 and outputs the selected subject distance information to the lens driving unit 122. The lens driving unit 122 moves the focus lens so that the subject is in focus according to the subject distance information. The process ends after S300 and S400.

In the present embodiment, when the subject distance calculation unit R400 cannot calculate the subject distance, a contrast AF focus detection and focus adjustment operation is performed. When the subject distance calculation unit R400 can calculate the subject distance, it is only necessary to move the focus lens once so that the subject is in focus. Therefore, even when the contrast AF method is used together, the average focusing time is shortened. it can. Even if focus detection cannot be performed using the contrast AF method, there is an effect that the possibility of focusing can be increased if the subject distance can be calculated.
In the present embodiment, in order to reduce the time until focusing, when the subject distance calculation unit R400 can calculate the subject distance in S230, the contrast AF is not executed. Further, when a plurality of regions having a high degree of coincidence are detected, the subject distance calculation unit R400 determines that the stereo matching has failed even during the execution of the normalized cross-correlation process. In general, since the time required for the normalized cross-correlation processing is long, such switching of the control method leads to a reduction in processing time.

When the distance between the photographing lens and the subject is known, the digital camera can calculate the distance between the lens and the imaging sensor for focusing on the subject. For example, in the case of a single lens, the distance between the lens and the imaging sensor for focusing on the subject is calculated by calculation using the Gaussian imaging formula. Alternatively, a subject distance B that is in focus is checked in advance corresponding to the distance A between the lens and the image sensor, and the digital camera associates the distance A and the distance B and holds them as a database. In this case, the distance between the lens and the image sensor corresponding to the subject distance information can be calculated. That is, in this embodiment, focus control is performed by calculating the distance from the imaging apparatus to the subject, not focus control using the defocus amount.
As described above, according to the present embodiment, the focus lens drive is executed so that the subject to be photographed is focused. Since it is not necessary to sequentially move the focus lens back and forth along the optical axis, it is possible to shorten the focusing time of the imaging apparatus during live view stereoscopic image display. In addition, since no focus detection dedicated pixels are arranged, pixel defects do not occur. Therefore, it is not necessary to interpolate and predict information on the subject at the focus detection dedicated pixel position from surrounding pixels, and a high-quality image can be obtained. Furthermore, even when the subject has a periodic pattern, the focusing time during live view display can be shortened.

  Further, in the present embodiment, when live view stereoscopic image display is performed, pixel data for the right eye and left eye is generated by the first PD selection / combination unit 106 in the image sensor 105, so that a high frame rate can be realized. On the other hand, in still image shooting, the first PD selection / combination unit 106 in the image sensor 105 outputs all PD information, thereby increasing the degree of freedom in post-processing. However, in order to display RAW data, development processing is required, and development processing takes time. Therefore, in live view display, control is performed to prepare and display a developed image for display in advance. In order to generate a developed image for display, there is an advantage that image data for stereo display can be generated by using the second PD selection / synthesis unit 151 outside the image sensor 105.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In addition, about the part similar to 1st Embodiment, by using the code | symbol already used, those detailed description is abbreviate | omitted and a difference is mainly demonstrated.
FIG. 8 is a functional block diagram showing main components according to the present embodiment. Differences from FIG. 1 are as follows.
The distance calculation unit R401 acquires the left image data and the right image data from the image generation unit R110, generates a distance map, and outputs the distance map to the live view display unit R200 and the subject distance calculation unit R402.
The subject specifying unit R302 acquires subject instruction data from the subject instruction unit R201 and subject extraction image data from the subject extraction image generation unit R112, and outputs the subject data to the subject distance calculation unit R402.
The subject distance calculation unit R402 acquires the distance map from the distance calculation unit R401 and the subject data from the subject specifying unit R302, and outputs the subject distance information to the control method switching unit R600.

The focus adjustment operation will be described below with reference to the flowchart of FIG. The live view display unit R200 starts a process triggered by the display image being displayed, and proceeds to S201.
In S201, the distance calculation unit R401 in FIG. 8 generates a distance map from the left image data and the right image data. The distance map is data in which the distance from the camera to the object is associated with the display image, and the distance information associated with the captured image is presented to the user. The distance map information may be a distance for each pixel or a distance for each block obtained by dividing the shooting screen into a plurality of blocks. In the present embodiment, the distance map includes distance information for each block, and the block is referred to as a distance block. The distance map is calculated by stereo matching as in the first embodiment. Next, the live view display unit R200 displays the distance as an overlay on the display image based on the distance map. An overlay display is illustrated in FIG. In the example of FIG. 10B, distances H401 (2.1 m), H402 (2.1 m), H403 (1.5 m), and H404 (2.1) are provided at the top, bottom, left, right, and center of the marker H200 for indicating the subject in accordance with the operation instructions. m), H405 (1.5 m) is displayed. The live view display unit R200 obtains a distance block corresponding to a position coordinate that is a predetermined number of pixels away from the marker H200, and displays the distance corresponding to the distance block. The display of the distance information may be performed by two-dimensional image display or stereoscopic image display, and the display form can be changed by a selection operation or automatic selection by the user.

  When the camera operator looks at the display screen of the live view display unit R200 and selects and instructs the subject image, or the first switch 117 is turned on, the process proceeds to S209 in FIG. In S209, the live view display unit R200 determines whether or not a subject is specified. If the operator has not performed an operation of selecting a subject, it is determined that the subject has not been specified, and the process proceeds to S212. If the operator performs an operation of selecting a subject in S209, the process proceeds to S213.

  In step S212, the subject specifying unit R302 extracts a subject image by image recognition using the subject extraction image acquired from the image generation unit R110. The image recognition target is, for example, a face when the subject is a person, but another object may be the recognition target. For example, in an application example to a fundus camera, a macular portion of a fundus image is a recognition target. In the first embodiment, the subject is specified by an operation in which the camera operator explicitly instructs the subject. On the other hand, in S212, the camera automatically identifies the subject. The advantage of this embodiment is that the trouble of operation is reduced because the operator of the camera can save the trouble of pointing the subject. However, depending on the accuracy of image recognition or the like, a method in which the operator explicitly indicates the subject when the subject is difficult to identify may be used in combination.

  The image generation unit R110 generates a subject extraction image used for image recognition. The image generation unit R110 includes a display image generation unit R101, a subject extraction image generation unit R112, and a contrast calculation image generation unit R113. The input of the image generation unit R110 is left image and right image data, and the output is display image, subject extraction image, contrast calculation image, and left image and right image data. The subject extraction image generation unit R112 generates subject extraction image data using the left image data and the right image data. The subject extraction image is an image that has been subjected to enhancement processing or size change processing so as to be suitable for image recognition. As for the subject extraction image, one image may be generated from the left image, or one image may be generated from the right image. Alternatively, as the subject extraction image, two images may be generated from the left image and the right image, or one image may be generated from the left image and the right image. The contrast calculation image generation unit R113 generates contrast calculation image data using the left image data and the right image data. The contrast calculation image is an image that has been subjected to enhancement processing or size change processing so as to be suitable for calculation of a contrast value.

  In step S213, the subject specifying unit R302 outputs subject data using the subject instruction data output by the subject instruction unit R201. The operation of the subject specifying unit R302 is the same as the operation of the subject specifying unit R300 of the first embodiment. Next, the subject distance calculation unit R402 calculates the subject distance using the distance map output from the distance calculation unit R401 and the region coordinates of the subject image included in the subject data. Since the distance map is used in the present embodiment, the subject distance calculation unit R402 does not need a stereo matching process. However, if the distance map cannot be calculated, stereo matching processing is executed as in the first embodiment.

In the subject distance calculation process in the subject distance calculation unit R402, it is determined whether or not a plurality of different distances are included in the subject image area, and the difference between the maximum value and the minimum value of the subject distance is compared with a threshold value. Is determined. When a plurality of different distances are included in the area of the subject image and the difference between the maximum value and the minimum value of the distance is greater than or equal to a predetermined threshold value, error processing is executed. The subject distance calculation unit R402 outputs an error display instruction to the live view display unit R200. For example, when a plurality of distances of 1.5 m and 5.0 m are included in the subject image area, it is impossible to focus on both. However, for example, when the distance is 1.5 m and 1.6 m, there is almost no difference in the captured images between the focus on 1.5 m and the focus on 1.6 m. If the threshold value Th is set to 0.5 m, for example, if the difference between the maximum value and the minimum value is 0.5 m or more, the subject distance calculation unit R402 determines that focusing cannot be performed, and outputs an error display instruction. The threshold Th is not a fixed value, but is changed according to the depth of field. In this case, the threshold value Th is set to a larger value as the depth of field is larger. If the subject distance calculation unit R402 does not determine that the subject cannot be focused, the subject distance calculation unit R402 has the following options.
A first process for outputting an average distance value.
A second process for outputting a weighted average value of distances according to the area in the region.
A third process for outputting a distance corresponding to the maximum area in the region.
The subject distance calculation unit R402 selects one of the first to third processes according to the setting mode, the shooting state, etc., and calculates the subject distance.

  In S214, in response to the error display instruction output in S213, the live view display unit R200 performs display processing indicating that focusing cannot be performed. A display example is shown in FIG. The marker H200 is located in the upper left of FIG. 10B, and information of distances 2.1 m and 1.5 m is mixed in the marker. On the other hand, in FIG. 10B, only the information of distance 1.5 m is included in the marker, and in this case, focusing is possible. In FIG. 10C, the frame line of the marker H200 is displayed thicker than in FIG. The thick frame display allows the camera operator to visually grasp that focusing is impossible. By displaying the subject distance, it becomes easier for the operator to guess which position the marker H200 can be brought into focus so that the operability is improved.

  In S216 following S212, the subject distance calculation unit R402 calculates the subject distance using the distance map output by the distance calculation unit R401 and the region coordinates of the subject image included in the subject data. In this case, stereo matching processing is not necessary. In step S <b> 300, the control method switching unit R <b> 600 selects the subject distance information output from the subject distance calculation unit R <b> 402 and outputs it to the lens driving unit 122. The lens driving unit 122 drives the focus lens so that the subject is in focus according to the subject distance information. After driving, the process proceeds to S400, and the contrast AF is performed with the time when the focusing operation on the subject is almost completed by the execution of S300 as an initial time. The operations of the contrast AF control unit R500, the control method switching unit R600, and the lens driving unit 122 are the same as those in the first embodiment. The contrast value calculation unit R501 calculates the in-focus position where the contrast value is maximized. The lens control amount output from the contrast AF control unit R500 is selected by the control method switching unit R600 and is output to the lens driving unit 122, and the process ends.

In the present embodiment, the focus adjustment operation is first performed based on the subject distance information output from the subject distance calculation unit R402, and then the focus lens position is finely adjusted by the contrast AF method. This eliminates the need to sequentially search for the focus lens position (in-focus position) at which the contrast value is maximized over the entire movement range of the lens, thereby speeding up the focusing operation. This is because there is a high possibility that the in-focus position is before and after the focus lens position corresponding to the subject distance output from the subject distance calculation unit R402, and it is only necessary to search for a range in the vicinity thereof. Further, when the calculation accuracy of the subject distance calculation unit R402 is low, there is an effect that the focusing accuracy can be complemented by the contrast AF control method.
Note that, when the subject distance calculation unit R402 cannot calculate the subject distance, the contrast AF may be performed in the previous stage without being limited to the configuration that supports the focus adjustment by the contrast AF method. For example, when focusing by the contrast AF method cannot be performed, the subject specifying unit R302 specifies the subject, and the subject distance calculating unit R402 calculates the subject distance and performs lens drive control for focusing on the distance.

DESCRIPTION OF SYMBOLS 102 Focus adjustment lens 105 Image pick-up element 122 Lens drive part R100 Image generation part R200 Live view display part R201 Subject instruction part R300, R302 Subject specification part R400, R402 Subject distance calculation part R401 Distance calculation part R500 Contrast AF control part R600 Control method Switching part

Claims (10)

The image sensor has a plurality of photoelectric conversion units that receive and photoelectrically convert light beams that have passed through different areas of the exit pupil of the imaging optical system for each microlens, and perform imaging using image signals output from the image sensor An imaging device for adjusting the focus of an optical system,
Image generating means for generating image data including a left-eye image and a right-eye image from a plurality of image signals acquired from the image sensor;
Display means for acquiring the image data and displaying the image;
Subject specifying means for acquiring the image data and specifying a subject image;
Obtaining the image data and information indicating the subject specified by the subject specifying means, calculating an image shift amount between the images for the subject images included in the left-eye image and the right-eye image, and using the parallax of the subject Subject distance calculating means for calculating the distance to the subject;
Lens driving means for focusing on the distance calculated by the subject distance calculation means by moving a focus adjustment lens that constitutes the imaging optical system when the display means is displaying an imaged image. equipped with a,
For the distance to the subject specified by the subject specifying means, when the subject distance calculation means calculates a plurality of distance information, the difference between the maximum value and the minimum value of the distance to the subject is calculated and compared with a threshold value. If the difference is equal to or larger than the threshold, the image pickup apparatus characterized that you notify the fact of focusing impossible. A focus control means for calculating a contrast value using the image data acquired from the image generation means and outputting a lens control amount;
The imaging apparatus according to claim 1, further comprising a switching unit that switches to the distance calculated by the subject distance calculation unit or the lens control amount output from the focusing control unit and outputs the lens control unit to the lens driving unit.   The switching means, when the subject distance calculation means cannot calculate the distance to the subject, switches to the lens control amount output by the focus control means and outputs to the lens driving means. 2. The imaging device according to 2. Display control means for performing control for acquiring display image data generated by the image generation means and displaying an image on the display means;
Subject indicating means for indicating a subject image in the image displayed by the display means;
The imaging apparatus according to claim 1, wherein the subject specifying unit specifies a subject image instructed by the subject instructing unit. The image generation means generates image data for subject extraction from the plurality of image signals,
The imaging apparatus according to claim 1, wherein the subject specifying unit specifies a subject image from the subject extraction image by image recognition.   Image data including the image for the left eye and the image for the right eye is acquired from the image generation unit, a distance map in which the distance from the imaging device to the object is associated with the display image is generated, and output to the subject distance calculation unit The imaging apparatus according to claim 1, further comprising a distance calculation unit that performs the calculation.   The imaging apparatus according to claim 6, wherein the display unit acquires a distance map generated by the distance calculation unit and displays distance information. The object distance calculating means, the image pickup apparatus according to any one of claims 1 to 7, characterized in that for changing the threshold depending on the depth of field. The display means, the image pickup apparatus according to any one of claims 1 to 8, wherein the displaying a stereoscopic image by acquiring the image data. The image sensor has a plurality of photoelectric conversion units that receive and photoelectrically convert light beams that have passed through different areas of the exit pupil of the imaging optical system for each microlens, and perform imaging using image signals output from the image sensor A control method executed by an imaging apparatus that performs focus adjustment of an optical system,
An image generation step of generating image data including a left-eye image and a right-eye image from a plurality of image signals acquired from the image sensor;
A subject specifying step of acquiring the image data and specifying a subject image;
Obtaining the image data and information indicating the subject specified in the subject specifying step, calculating an image shift amount between the images for the subject images included in the left-eye image and the right-eye image, and using the parallax of the subject A subject distance calculating step for calculating a distance to the subject;
A lens driving step of focusing on the distance calculated in the subject distance calculation step by moving a focus adjustment lens that constitutes the imaging optical system when the captured image is displayed on the display unit; Yes, and
For the distance to the subject specified in the subject specifying step, when a plurality of distance information is calculated in the subject distance calculating step, the difference between the maximum value and the minimum value of the distance to the subject is calculated and compared with a threshold value and, when the difference is equal to or greater than the threshold, the control method of an image pickup apparatus which is characterized that you notify the fact of focusing impossible.

Images