JP2014232137A - Imaging device and method of controlling the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device comprising a plurality of imaging elements capable of high-speed and accurate focusing operation without reducing throughput when reading pixel data on a captured image.SOLUTION: An imaging device 1 comprises: a first imaging element 100 used for capturing a still image; and a second imaging element 103 used for capturing moving images. While moving images are being captured with the second imaging element 103, an AF calculation unit 122 performs focus detection by acquiring pixel data output by the first imaging element 100, and a CPU 124 controls focusing operation on the basis of the result of focus detection.

Description

  The present invention relates to focus adjustment control of an imaging apparatus, and more particularly to an imaging apparatus including a plurality of imaging elements and a control method thereof.

  Conventionally, regarding an autofocus (AF) control method of an image pickup apparatus having two image pickup devices, there is a technique in which one image pickup device includes a phase difference detection pixel and performs focus determination from the detected phase difference. Patent Document 1 discloses a technique for efficiently performing processing related to data interpolation of phase difference detection pixels when an image is generated by an image sensor having phase difference detection pixels. Patent Document 1 describes that live view display and moving image shooting are performed by an image sensor having a phase difference detection pixel.

JP 2012-42544 A

In the prior art disclosed in Patent Document 1, it is necessary to read out moving image pixel data and phase difference detection pixel data from an image sensor having phase difference detection pixels during moving image shooting. As the phase difference detection pixel data is read, the throughput of reading the moving image pixel data may decrease. From the viewpoint of AF calculation speed and accuracy, it is desired to read out the phase difference detection pixel data a plurality of times at high speed. However, since the phase difference detection pixel data is read at the same time as the moving image pixel data is read, the processing speed of the AF calculation is limited by the frame rate of the moving image.
In addition, since the exposure accumulation time is the same for the moving image pixel and the phase difference detection pixel, the exposure accumulation time suitable for live view shooting and moving image shooting is not necessarily the optimum time for AF processing. In some cases, there is a concern that AF accuracy may be reduced.

  An object of the present invention is to realize a high-speed and high-precision focus adjustment operation without reducing throughput when reading out pixel data relating to a captured image in an imaging apparatus including a plurality of imaging elements.

  In order to solve the above-described problem, an apparatus according to the present invention is an imaging apparatus including a first imaging element and a second imaging element, and divides light incident through an imaging optical system to provide the first imaging element and the first imaging element. A light splitting unit that causes each of the two image sensors to receive primary imaging light; a focus detection unit that acquires pixel data output from the first image sensor and performs focus detection of the imaging optical system; and the focus detection unit And a control means for controlling the focus adjustment of the imaging optical system. The control means acquires detection information obtained by the focus detection means performing focus detection from pixel data output from the first image sensor during photographing by the second image sensor, and performs focus adjustment of the image pickup optical system. Control.

  According to the present invention, it is possible to realize a high-speed and accurate focus adjustment operation without reducing the throughput when reading out pixel data relating to a captured image.

It is a figure which shows the structural example of the imaging device which concerns on embodiment of this invention. It is a schematic diagram which shows the structure of the imaging part which concerns on embodiment of this invention. It is a figure showing an example of composition of an image sensor concerning an embodiment of the present invention. It is a figure showing an example of composition of an image sensor concerning a 1st embodiment of the present invention. It is explanatory drawing of the focus detection in this invention. 5 is a flowchart for explaining the operation of the imaging apparatus according to the first embodiment of the present invention. It is a figure which shows the read-out area | region of the image pick-up element which concerns on 1st Embodiment of this invention. It is a figure explaining each image which concerns on the focus detection in this invention. It is a figure explaining selection of a focus detection area concerning a 1st embodiment of the present invention. It is a figure explaining the read-out timing of the image sensor concerning a 1st embodiment of the present invention. It is a figure which shows the structural example of the image pick-up element which concerns on the modification of 1st Embodiment of this invention. It is a figure which shows the structural example of the image pick-up element which concerns on 2nd Embodiment of this invention. It is a flowchart explaining operation | movement of the imaging device which concerns on 2nd Embodiment of this invention, and shows the first half part of a process. It is a flowchart which shows the process following FIG. It is a figure which shows the example of an imaging | photography setting screen which concerns on 2nd Embodiment of this invention.

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus 1 according to the first embodiment of the present invention. The imaging device 1 includes an imaging optical system (imaging optical system) and a plurality of imaging elements 100 and 103.
The first lens group 111 is disposed on the front end side (subject side) of the photographing optical system, and is supported by the lens barrel so as to be able to advance and retract in the optical axis direction. The diaphragm 110 adjusts the light amount at the time of photographing by adjusting the aperture diameter. The second lens group 109 moves forward and backward in the optical axis direction together with the stop 110. The second lens group 109 has a zooming function (zoom function) in conjunction with the advance / retreat operation of the first lens group 111. The third lens group 108 is a focus lens group that performs focus adjustment by moving forward and backward in the optical axis direction.

  As shown in FIG. 2, the half mirror 107 is a light splitting unit that splits an incident light beam from a subject into reflected light and transmitted light. The light transmitted through the half mirror 107 enters the first image sensor 100, and the reflected light enters the second image sensor 103. The focal plane shutter 106 adjusts the exposure time during still image shooting. In the present embodiment, the exposure time of the first image sensor 100 is adjusted by the focal plane shutter 106. However, the present invention is not limited to this, and the first image sensor 100 may have an electronic shutter function and adjust the exposure time with a control pulse.

  The first image sensor 100 that converts an optical image into an electrical signal is mainly used for capturing a still image. A first AFE (analog front end) 101 performs gain adjustment and conversion to a digital signal corresponding to a predetermined quantization bit for an analog image signal output from the first image sensor 100. A first TG (timing generator) 102 controls the drive timing of the first image sensor 100 and the first AFE 101.

  The second image sensor 103 that converts an optical image into an electrical signal is mainly used for capturing a moving image. The second AFE 104 performs gain adjustment and conversion into a digital signal corresponding to a predetermined quantization bit for the analog image signal output from the second image sensor 103. The second TG 105 controls the drive timing of the second image sensor 103 and the second AFE 104. Image data output from each of the first AFE 101 and the second AFE 104 is sent to a CPU (Central Processing Unit) 124. The first TG 102 and the second TG 105 generate drive signals according to control signals from the CPU 124 and output them to the first image sensor 100 and the second image sensor 103, respectively. In the present embodiment, the first AFE 101 and the first TG 102 related to the first image sensor 100 and the second AFE 104 and the second TG 105 related to the second image sensor 103 are used, but the configuration in which the AFE and TG are built in the image sensor. But you can.

  The CPU 124 controls the imaging device in an integrated manner. The CPU 124 controls the focus drive circuit 112 and the aperture drive circuit 113. For example, the CPU 124 drives and controls the focus actuator 114 by the focus drive circuit 112 based on the focus detection result (detection information) of the AF calculation unit 122. As a result, the third lens group 108 advances and retreats in the optical axis direction, and the focus adjustment operation is performed. The CPU 124 controls the aperture actuator 115 by the aperture drive circuit 113 to control the aperture diameter of the aperture 110.

  Each unit (see 116 to 123) is connected to the CPU. The operation unit 116 is used when the user performs setting operations such as shooting instructions and shooting conditions on the CPU 124. A display unit 117 displays captured still images, moving images, menus, and the like. The display unit 117 includes a TFT (thin film transistor) liquid crystal display or a viewfinder on the back of the camera body. A RAM (random access memory) 118 stores image data digitally converted by the first AFE 101 and the second AFE 104 and data processed by the first image processing unit 120. The RAM 118 has both a function as an image data storage unit for storing image data processed by the second image processing unit 121 and a function as a work memory of the CPU 124. In this embodiment, the RAM 118 having these functions is mounted. However, other memories can be used as long as the access speed is a sufficient level. A ROM (Read Only Memory) 119 stores a program that is interpreted and executed by the CPU 124. A memory device such as a flash ROM is used.

The first image processing unit 120 performs processing such as correction and compression of a captured still image. The second image processing unit 121 performs processing such as correction and compression of the captured moving image. The second image processing unit 121 has a function of adding A image data and B image data, which will be described later.
The AF calculation unit 122 performs focus detection based on the pixel signal output from the first image sensor 100. The flash memory 123 is a detachable memory device for recording still image data and moving image data. In this embodiment, a flash memory is used as a recording medium. However, a non-volatile memory capable of writing data, a hard disk, or the like may be used. Further, a form in which these recording media are built in a case may be used.

  FIG. 2 is a schematic diagram for explaining the positional relationship between the first image sensor 100, the second image sensor 103, and the half mirror 107. The reflected light of the half mirror 107 enters the second image sensor 103, and the transmitted light enters the first image sensor 100. The distance a from the center of the half mirror 107 to the first image sensor 100 is equal to the distance b from the center of the half mirror 107 to the second image sensor 103 (a = b). That is, primary imaging light that is a subject image with the same magnification is incident on the first image sensor 100 and the second image sensor 103. Therefore, even when the AF control is performed using the pixel signal output from the first image sensor 100, the subject imaged on the second image sensor 103 can be focused.

  FIG. 3 shows the configuration of the first image sensor 100. The first image sensor 100 includes a pixel array 100a, a vertical selection circuit 100d that selects a row in the pixel array 100a, and a horizontal selection circuit 100c that selects a column in the pixel array 100a. The readout circuit 100b reads out signals of pixels selected by the vertical selection circuit 100d and the horizontal selection circuit 100c among the pixels in the pixel array 100a.

  The vertical selection circuit 100d selects the row of the pixel array 100a, and validates the readout pulse output from the first TG 102 in the selected row based on the horizontal synchronization signal output from the CPU 124. The readout circuit 100b includes an amplifier and a memory for each column, and the pixel signal of the selected row is stored in the memory via the amplifier. The pixel signals for one row stored in the memory are sequentially selected in the column direction by the horizontal selection circuit 100c and output to the outside through the output amplifier 100e. This operation is repeated for the number of rows, and all pixel signals are output to the outside. Since the configuration of the second image sensor 103 is the same, detailed description thereof is omitted.

  FIG. 4 shows a configuration example of the pixel array 100a. The pixel array 100a has a configuration in which a plurality of pixel portions are arranged in a two-dimensional array in order to output two-dimensional image data. FIG. 4A illustrates a configuration of a pixel array that can detect a phase difference, and FIG. 4B illustrates a configuration of a pixel array that cannot detect a phase difference. In the present embodiment, the pixel array of the first image sensor 100 is configured as shown in FIG. On the other hand, the pixel array of the second image sensor 103 is configured as shown in FIG.

  A plurality of photodiodes (hereinafter referred to as PDs) 100g and 100h indicated by a rectangular frame are provided for the microlens 100f indicated by a circular frame in FIG. The PDs 100g and 100h constitute a plurality of photoelectric conversion units. That is, one microlens is arranged on the subject side for the two PDs constituting each pixel unit. When the area sharing the microlens 100f is one pixel, h1 pixels in the horizontal direction and i1 pixels in the vertical direction are arranged side by side. The signals stored in the PD 100g and PD 100h are separately output to the outside by a read operation. Lights of different images having a phase difference are incident on the PD 100g and the PD 100h, as will be described later. Hereinafter, the PD 100g is referred to as an A image pixel, and the PD 100h is referred to as a B image pixel. In the present embodiment, a configuration example in which two PDs are arranged for one microlens is shown, but a configuration in which three or more PDs (for example, four, nine, etc.) are arranged for one microlens is also possible. Good. That is, the present invention can be applied to a configuration in which a plurality of PDs are arranged in the vertical direction or the horizontal direction with respect to one microlens.

In the configuration of FIG. 4B, only one PD 103g is provided for the micro lens 103f. That is, one microlens is arranged on the subject side for one PD indicated by a square frame. H2 pixels in the horizontal direction and i2 pixels in the vertical direction are arranged side by side.
The first image sensor 100 is used for still image shooting, and the second image sensor 103 is used for moving image shooting. For this reason, the number of pixels of the first image sensor 100 (h1 × i1 pixels) is larger than the number of pixels of the second image sensor 103 (h2 × i2 pixels). Further, since the second image sensor 103 has a smaller number of pixels than the first image sensor 100, the area of one PD is increased, so that the sensitivity is high. The light beam split by the half mirror 107 is split at a ratio of M: N between transmitted light and reflected light. The ratio of incident light to the second image sensor 103 that is set to “N <M” and has high sensitivity is smaller than the ratio of incident light to the first image sensor 100.

Next, pixel data output from the A image pixel and the B image pixel in the first image sensor 100 having the pixel array configuration shown in FIG. 4A will be described. 5A and 5B are schematic diagrams illustrating the relationship between the focus state and the phase difference in the first image sensor 100. FIG.
In the cross section 100 a of the pixel array shown in FIG. 5, the A image pixel 129 and the B image pixel 130 are arranged for one microlens 128. The taking lens 125 is an image pickup optical system in which the first lens group 111, the second lens group 109, and the third lens group 108 shown in FIG. The light from the subject 126 passes through each area of the photographing lens 125 with the optical axis 127 as the center, and forms an image on the image sensor. Here, the exit pupil and the center of the photographic lens are the same. Light passes through each region of the photographing lens 125 from different directions. In other words, this configuration is equivalent to dividing the pupil of the imaging optical system symmetrically when the imaging optical system is viewed from the A image pixel and from the B image pixel. In other words, the light beam from the imaging optical system is divided into so-called pupils as two light beams. The divided light beams (first light beam ΦLa and second light beam ΦLb) are incident on the A image pixel and the B image pixel, respectively.

  The first light flux from a specific point on the subject 126 is a light flux ΦLa that enters the A image pixel 129 through the divided pupil corresponding to the A image pixel 129. The second light flux from a specific point on the subject 126 is a light flux ΦLb incident on the B image pixel 130 through the divided pupil corresponding to the B image pixel 130. The two pupil-divided light beams are incident from the same point on the subject 126 through the imaging optical system. For this reason, in a state where the subject 126 is in focus, as shown in FIG. 5A, the light beams ΦLa and ΦLb pass through the same microlens and reach one point on the image sensor. Therefore, the phases of the image signals obtained from the A image pixel 129 and the B image pixel 130 are in agreement with each other.

  On the other hand, as shown in FIG. 5B, in the state where the focus is shifted by the distance Y in the optical axis direction, the arrival positions of the two light beams by the change in the incident angle at which the light beams ΦLa and ΦLb are incident on the microlens. Are shifted from each other. Therefore, there is a phase difference between the image signals obtained from the A image pixel 129 and the B image pixel 130, respectively. The PD subjects each of the two subject images (A image and B image) detected by the A image pixel 129 and the B image pixel 130 and having a phase difference to each other to undergo photoelectric conversion. Each signal of the A image and B image after photoelectric conversion is separately output to the outside and used in an AF operation described later.

Next, the operation of the imaging apparatus according to the present embodiment will be described with reference to the flowchart of FIG. The following processing is realized by the CPU 124 reading and executing a program from the memory.
First, a standby state is maintained until the moving image shooting switch included in the operation unit 116 is pressed in S101. When the moving image shooting switch is pressed by a user operation, the CPU 124 starts a moving image shooting operation in S102. The second image sensor 103, the second AFE 104, and the second TG 105 are powered on, and the CPU 124 performs moving image shooting settings. After this setting, the second TG 105 outputs a read pulse to the second image sensor 103 based on the synchronization signal output from the CPU 124, and the second image sensor 103 starts a read operation at a predetermined frame rate. In this embodiment, the charge accumulation and readout operations for moving images are performed using an electronic shutter function based on a slit rolling operation, but this is not restrictive.

  Image data output from the second image sensor 103 is transferred to the RAM 118 by the CPU 124. Thereafter, the image data is transferred to the second image processing unit 121, and correction, compression processing, and the like are performed to create moving image frame data. The display unit 117 displays an image according to the created moving image frame data (live view display). When the moving image recording mode is selected by the user using the operation unit 116 while viewing the menu displayed on the display unit 117 before shooting, moving image data is sequentially recorded in the flash memory 123. Is done.

  In S103, the CPU 124 determines whether the moving image shooting switch has been operated again. If the moving image shooting switch has not been operated, the moving image shooting operation continues, and the process proceeds to S104. If the moving image shooting switch is operated in S103, the moving image shooting operation is terminated. In step S <b> 104, the CPU 124 determines whether an AF switch included in the operation unit 116 has been pressed. If it is determined that the AF switch has been pressed, the process proceeds to S105, and AF calculation is performed during moving image shooting. If it is determined in S104 that the AF switch has not been pressed, the process proceeds to S108. In S <b> 105, the CPU 124 performs settings for reading pixel data from the phase difference detection pixels of the first image sensor 100. In this case, a process of reading pixel data from a partial area is executed instead of reading data of the entire screen. Data is partially read from the pixel portion in the region 100i shown in FIG. Thereby, high-speed AF calculation can be executed. Furthermore, since the operation time can be shortened, power consumption can be suppressed. In addition, in the area | region 100i of FIG. 7, although the data reading for 3 pixels is illustrated, the number of the pixels to read can be set arbitrarily. In addition, a partial area for reading out phase difference detection pixel data from the first image sensor 100 is appropriately changed according to a change in imaging conditions, an operation instruction, or the like, and reading out pixel data from the changed area. Processing is executed. The A image pixel data and the B image pixel data output from the first image sensor 100 are transferred to the RAM 118 by the CPU 124. The CPU 124 uses the image data (A image data corresponding to the A image) using the A image pixel data stored in the RAM 118 and the image data (B image data corresponding to the B image) using the B image pixel data. Transfer to the AF calculation unit 122.

  FIG. 8A illustrates A image data and B image data when the subject is in focus as described with reference to FIG. The horizontal axis represents the pixel position, and the vertical axis represents the output level. As shown in FIG. 8A, when the subject is in focus, the A image data and the B image data match. FIG. 8B illustrates A image data and B image data when the subject is not in focus as described with reference to FIG. In this case, the A image data and the B image data have a phase difference, and a shift amount X occurs in the pixel position. The AF calculation unit 122 calculates the shift amount X for each moving image frame, thereby calculating the focus shift amount, that is, the Y value in FIG. The AF calculation unit 122 transmits the calculated Y value to the focus driving circuit 112 via the CPU 124.

  In S <b> 106, the focus drive circuit 112 calculates the drive amount of the third lens group 108 based on the Y value acquired from the AF calculation unit 122, and outputs a drive command to the focus actuator 114. By the focus actuator 114, the third lens group 108 moves to the in-focus position where the subject is in focus, and the focus is positioned on the light receiving surface of the first image sensor 100. At this time, primary imaging light having the same image plane magnification is incident on the first image sensor 100 and the second image sensor 103, and the depth of field and the like are also the same. For this reason, when the subject is focused on the first image sensor 100, the subject is also focused on the second image sensor 103. In the next S107, the CPU 124 determines whether or not the focus state is in the focused state as a result of the focus drive (focus determination). Therefore, AF calculation is executed again in S107. Since the processing content of the AF calculation is the same as that in S105, the description is omitted. If it is determined that the imaging optical system is in focus, the process proceeds to S108. If it is determined that the imaging optical system is not in focus, the process returns to S104. In S <b> 108, the CPU 124 determines whether or not the still image shooting switch included in the operation unit 116 has been pressed. If the still image shooting switch is pressed, the process proceeds to S109. If the still image shooting switch is not pressed, the process returns to S103.

When the still image shooting operation starts in S109, the CPU 124 controls the focal plane shutter 106 to perform an exposure operation on the first image sensor 100. Thereafter, the first TG 102 outputs a read pulse to the first image sensor 100 based on the synchronization signal output from the CPU 124. As a result, the first image sensor 100 starts a reading operation. The image data output from the first image sensor 100 is stored in the RAM 118 after the first AFE 101 converts it into digital data. The CPU 124 transfers the image data stored in the RAM 118 to the first image processing unit 120. The first image processing unit 120 performs image data correction, compression processing, and the like, and the processed image data is recorded in the flash memory 123. After S109, the process returns to S103 and the above process is repeated.
If the AF switch is not pressed in S104, the process proceeds to S108. The same applies to the case where the AF operation is set to the OFF state by the operation instruction in the menu display using the display unit 117 and the operation unit 116.

  FIG. 9 is a diagram illustrating an AF frame frame (a frame for displaying a focus detection area) on the shooting screen. FIG. 9A shows an example of the AF frame selection process. The AF frame frame is displayed on the screen of the display unit 117 or the like, and the user can select the AF frame frame using the operation unit 116. The AF frame frame 132 in FIG. 9A indicates a selectable area, and the AF frame frame 131 indicates the currently selected AF frame frame. In S105 of FIG. 6, based on the position of the currently selected AF frame frame 132, a process for determining a corresponding area (see an area 100i shown in FIG. 7) is performed. In this area, the pixel data of the first image sensor 100 is read and AF calculation processing is executed. Note that the number and area of the AF frame shown in FIG. 9A are examples, and can be arbitrarily designed.

  FIG. 9B shows an example of processing for detecting the face area of the subject and automatically selecting the AF frame frame. FIG. 9B illustrates an AF frame frame 132 and a subject image 133. The CPU 124 determines a subject by image analysis of a moving image frame (which may be a live view image) captured by the second image sensor 103, and detects a face area of the subject. The AF frame frame 134 is an AF frame frame corresponding to the face area of the subject. An AF calculation process is executed based on pixel data corresponding to the position of the AF frame frame 134. The subject face detection method is not particularly limited.

FIG. 10 is a diagram for explaining the readout timing of the phase difference detection pixel in the present embodiment, in contrast with the conventional example. FIG. 10A illustrates the timing of reading data from the phase difference detection pixels of the moving image capturing image sensor in the conventional system. FIG. 10B illustrates timing for reading moving image pixel data from the moving image capturing second imaging element 103 according to the present embodiment. FIG. 10C illustrates the timing of reading phase difference detection pixel data from the still image first image sensor 100 according to this embodiment. In addition, the vertical axis | shaft of FIG. 10 shows the position of the vertical direction of each image sensor, and a horizontal axis shows a time axis.
The length of the period 140 illustrated in FIG. 10A indicates the accumulation time of the image sensor. This accumulation time is determined by moving image shooting settings. In the period 140, signal charge accumulation for the A image pixel and the B image pixel is performed. Pixel data indicated by the signal accumulated in the period 140 is read at the timing indicated by the oblique line segment 141 and stored in the RAM 118 by the CPU 124. Thereafter, the data is transferred to the second image processing unit 121, and processing for adding each data of the A image pixel and the B image pixel under the same microlens for each pixel is executed. As a result, moving image frame data is created.

The length of the period 142 indicated by the right-pointing arrow indicates the processing time of the AF calculation. As described with reference to FIG. 6, the AF calculation processing is executed using each data of the A image pixel and the B image pixel. The length of the period 143 indicates the focus drive time. As described with reference to FIG. 6, the focus drive process is executed based on the AF calculation result.
In the system of the conventional example shown in FIG. 10A, the pixel data for A image and the pixel data for B image are stored in order to perform moving image frame data creation and AF calculation processing at the readout timing shown at the time of the line segment 141. Need to read. For this reason, the read throughput (in this case, the processing capability corresponding to the frame rate of the moving image) cannot be fully exhibited. In order to solve this problem, a pixel data reading method according to the present embodiment will be described below with reference to FIGS. 10B and 10C.

  FIG. 10B illustrates a process of reading moving image pixel data of the second image sensor 103. The length of the period 140 indicates the accumulation time as in FIG. A time point indicated by an oblique line segment 144 indicates a pixel data read timing, and moving image pixel data is read. That is, only the moving image pixel data reading process is performed at the time of the line segment 144, so that the moving image frame rate can be maximized.

The length of the period 145 shown in FIG. 10C indicates the accumulation time of the phase difference detection pixels in the first image sensor 100. The accumulation time does not have to be the same as the accumulation time of moving image pixels (see period 140), and an accumulation time suitable for AF calculation processing can be set. In the example of FIG. 10C, the accumulation time is set longer than that in the case of FIG. 10B, but the present invention is not limited to this. In the present embodiment, since an accumulation time suitable for AF calculation processing can be set, a more accurate AF operation can be performed.
A time point indicated by a line segment 146 indicates the read timing of the phase difference detection pixel data. As described for the region 100i in FIG. 7, the reading time (required time) can be greatly reduced by performing the data reading process only on the partial region in the pixel group of the first image sensor 100. As a result, a faster AF operation can be realized. Note that the length of the period 147 represents the AF calculation processing time, which is the same as the period 142 in FIG. Further, the length of the period 148 represents the focus driving time, which is the same as the period 143 in FIG.

According to the present embodiment, it is possible to realize a moving image shooting operation that maximizes the reading throughput, and it is possible to perform AF processing at the time of moving image shooting with high accuracy and high speed.
In the present embodiment, all the pixel portions of the first image sensor 100 include focus detection pixels and have a configuration capable of performing phase difference detection AF processing, but this is not restrictive. For example, the first image sensor 100 may include a focus detection pixel that is discretely arranged, obtain a pair of image signals, and perform phase difference AF processing. In this case, the focus detection pixel unit has, for example, one PD for each microlens and performs pupil division type focus detection with a configuration in which the left and right or upper and lower portions are shielded by the light shielding layer.
Further, the first image sensor 100 may have a pixel configuration similar to that of the second image sensor 103, and may be configured to apply a contrast AF method in which the contrast of image data read from each pixel unit is detected and an AF operation is performed. is there. In this case, the first image sensor 100 may be configured to have one PD for one microlens.

  In the present embodiment, the image processing unit adds A image data and B image data acquired by the first image sensor 100 to generate a moving image. However, the present invention is not limited to this, and when A image data and B image data are not required, a configuration may be adopted in which A image data and B image data are added and output in the image sensor for some or all of the pixel portions.

[Modification of First Embodiment]
FIG. 11 schematically shows a modification of the pixel array of the image sensor. In general, the first image sensor 100 for taking a still image has a higher number of pixels than the second image sensor 103 for taking a moving image. On the other hand, the second image sensor 103 has a configuration in which the area per pixel is increased and the sensitivity is increased as compared with the first image sensor 100.
FIG. 11A illustrates a pixel array of the first image sensor 100, and FIG. 11B illustrates a pixel array of the second image sensor 103. The pixel portion of the first image sensor 100 shown in FIG. 11A corresponds to four pixels (see 100i) with respect to one pixel (see 103i) of the second image sensor 103 shown in FIG.

For convenience of explanation, the ratio of the area of one pixel of the first image sensor 100 to the area of one pixel of the second image sensor 103 is about 4: 1, but this ratio can be changed according to the specification. It is. In addition, for the A image pixel data and the B image pixel data to be paired for use in phase difference detection, after the addition processing is performed on the data of a plurality of pixels in the first image sensor 100, the addition is performed. By reading the data, it is possible to increase the processing speed. For example, when the ratio of the area of one pixel of the first image sensor 100 to the area of one pixel of the second image sensor 103 is N: 1, data of N pixels corresponding to the ratio is added. A process of reading data from the first image sensor 100 is performed.
In the modified example, a phase difference detection signal is acquired from a pixel portion in a specific area (selected area for focus detection) in the first image sensor 100 that can output a pixel with higher definition than the second image sensor 103. Thus, the defocus amount can be calculated.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. Note that, except for the first image sensor 200 and the second image sensor 203 in the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed descriptions thereof are omitted. The difference will be mainly described.
The basic configuration of the imaging apparatus according to the second embodiment is the same as that of FIG. 1, but the configuration of the pixel arrays of the first imaging element 200 and the second imaging element 203 is different. 12A shows a configuration example of the pixel array of the first image sensor 200, and FIG. 12B shows a configuration example of the pixel array of the second image sensor 203.

  Two PDs 200g and 200h are arranged for one microlens 200f shown in FIG. When the area sharing the microlens 200f is one pixel, h1 pixels in the horizontal direction and i1 pixels in the vertical direction are arranged side by side. Compared with FIG. 4A, it can be seen that the number of pixels per unit area is large and densely distributed. Further, two PDs 203g and 203h are arranged for one microlens 203f shown in FIG. H2 pixels in the horizontal direction and i2 pixels in the vertical direction are arranged side by side. Since the second image sensor 203 is used for moving image shooting, the number of pixels is smaller than that of the first image sensor 200 for still images, and the area per pixel is large. In the present embodiment, each of the first image sensor 200 and the second image sensor 203 includes a phase difference detection pixel.

Next, the imaging operation of this embodiment will be described with reference to the flowcharts shown in FIGS.
The shooting operation starts in S201 of FIG. 13, and the CPU 124 determines the shooting mode in S202. When the shooting mode is the still image shooting mode, the process proceeds to S205 and a preparation operation for still image shooting is performed. When the shooting mode is the moving image shooting mode, the process proceeds from S203 to S211 in FIG. Note that the shooting mode can be selected using the operation unit 116 or selected by operating the touch panel while viewing the menu screen displayed on the display unit 117.

First, processing in the still image shooting mode will be described.
In S <b> 205 of FIG. 13, the CPU 124 determines the operation state of the AF switch included in the operation unit 116. If the AF switch is pressed, the process proceeds to S206. If the AF switch is not pressed, the process enters a waiting state and the determination process of S205 is repeated.
In S206, AF calculation processing is executed during still image shooting. That is, power is turned on to the second image sensor 203, the second AFE 104, and the second TG 105, and the CPU 124 performs settings for reading data from the phase difference detection pixels of the second image sensor 203. The A image pixel data and the B image pixel data output from the second image sensor 203 are transferred to the RAM 118 by the CPU 124. The CPU 124 transfers the A image data corresponding to the A image using the A image pixel data stored in the RAM 118 and the B image data corresponding to the B image using the B image pixel data to the AF calculation unit 122. For convenience of explanation, FIGS. 13 and 14 are not described as flowcharts for performing still image shooting and moving image shooting simultaneously. However, when live view shooting is designated, the CPU 124 performs live view shooting in S206. Perform the setting process. If the moving image shooting is designated in S206, the CPU 124 performs moving image shooting setting processing.

Next, in S <b> 207, the focus drive circuit 112 calculates the drive amount of the third lens group 108 based on the defocus amount acquired from the AF calculation unit 122, and outputs a drive command to the focus actuator 114. The third lens group 108 is moved to the in-focus position by the focus actuator 114, and the focal point is located on the light receiving surface of the second image sensor 203. At this time, primary imaging light having the same image plane magnification is incident on the first imaging element 200 and the second imaging element 203, and the depth of field and the like are also the same. For this reason, when the second image sensor 203 is in focus with respect to the same subject, the first image sensor 200 is also in focus.
In the next S208, the CPU 124 determines whether or not it is in focus. For this purpose, the AF calculation is executed again. The processing content of the AF calculation is the same as S206. If the in-focus state is determined in S208, the process proceeds to S209. If the in-focus state is determined, the process returns to S205.

  In step S <b> 209, the CPU 124 determines whether the still image shooting switch included in the operation unit 116 has been operated. If the still image shooting switch has been pressed, the process proceeds to S210 and a still image shooting operation is performed. Thereafter, the process returns to S202 via S204. If the still image shooting switch is not pressed in S209, the process returns to S205.

Next, the operation in the moving image shooting mode will be described with reference to FIG.
In S <b> 211, the CPU 124 determines an operation state of the moving image shooting switch included in the operation unit 116. When the moving image shooting switch is pressed, the process proceeds to S212 and the moving image shooting operation starts. If the moving image shooting switch is not pressed, the process returns from S204 to S202 in FIG.
When moving image shooting is started in S212, the second image sensor 203, the second AFE 104, and the second TG 105 are powered on, and the CPU 124 performs moving image shooting setting processing. After this setting, the second TG 105 outputs a read pulse to the second image sensor 203 based on the synchronization signal output from the CPU 124. The second image sensor 203 starts a reading operation at a predetermined frame rate. In this embodiment, charge accumulation and readout operations for moving images are performed using an electronic shutter function based on a slit rolling operation. Pixel data output from the second image sensor 203 is transferred to the RAM 118 by the CPU 124. Thereafter, the data is transferred to the second image processing unit 121, subjected to correction and compression processing, and the like, and moving image frame data is generated. The display unit 117 displays a moving image on the screen according to the generated moving image frame data (live view). If the user has already instructed to select moving image recording using the menu screen displayed on the display unit 117 and the operation unit 116 before shooting, moving image data is sequentially recorded in the flash memory 123.

In S213, the CPU determines whether the moving image shooting switch has been pressed again. If the moving image shooting switch is not pressed, moving image shooting is continued, and the process proceeds to S214. If the moving image shooting switch is pressed, moving image shooting is terminated, and the process returns from S204 to S202 in FIG. In S214, the CPU 124 determines the operation state of the AF switch. If the AF switch is pressed, the process proceeds to S215. If the AF switch is not pressed, the process returns to S213.
In S215, AF calculation processing is executed during moving image shooting. That is, the first image sensor 200, the first AFE 101, and the first TG 102 are powered on, and the CPU 124 performs settings for reading data from the phase difference detection pixels of the first image sensor 200. When reading data from the phase difference detection pixels of the first image sensor 200, AF calculation is performed at high speed by partially reading pixel data in a limited target area instead of the entire area. As a result, the operation time can be shortened, so that power consumption can be suppressed.

  In S <b> 216, the focus drive circuit 112 calculates the drive amount of the third lens group 108 based on the defocus amount acquired from the AF calculation unit 122 and outputs a drive command to the focus actuator 114. The third lens group 108 moves to the in-focus position. In the next S217, after the AF calculation is executed again, the focus determination process is performed. If the determination of the in-focus state is made, the process proceeds to S213. If the determination of the out-of-focus state is made, the process returns to S214.

  With the above operation, the image sensor for detecting the phase difference is changed according to the photographing mode. That is, in the still image shooting mode, a data read operation is performed from the phase difference detection pixel of the second image sensor 203. In the moving image shooting mode, a data read operation is performed from the phase difference detection pixels of the first image sensor 200. By switching the image sensor according to the shooting mode, the reading throughput during moving image shooting can be maximized. Furthermore, the AF operation during moving image shooting can be performed with high accuracy and high speed.

Next, the setting operation will be described with reference to FIG. FIG. 15 shows a moving image recording size setting screen 230 and a subject tracking characteristic setting screen 240 as an example of the display unit 117. In this embodiment, the operation unit 116, the display unit 117, and the CPU 124 perform a recording size setting process for setting a moving image recording size and a frame rate setting process for setting a moving image frame rate.
FIG. 15A shows an example of a screen for setting the moving image recording size 231, the moving image frame rate 232, and the moving image compression format 233. For the moving picture recording size, for example, the following image quality can be selected (abbreviated as 1920, 1280, 640 in the figure).
-Full high-definition (Full HD) image quality of 1920 x 1080 pixels.
-1280 x 720 pixel high-definition (HD) image quality.
-Standard image quality of 640 x 480 pixels.

As for the frame rate of the moving image, 30 frames or 60 frames can be selected when the television broadcast video system is NTSC (National Television System Committee). When the television broadcast video system is PAL (Phase Alternation by Line), 25 frames or 50 frames can be selected. In addition, 24 frames can be selected as a movie-related usage example. FIG. 15A shows a display example in the case of the NTSC system.
As the moving image compression format, an IPB method for efficiently compressing and recording in units of a plurality of frames, an ALL-I method for compressing and recording in units of one frame, or the like can be selected.
When the user operates the operation unit 116 or the like to select an intended setting item, the content of the selected setting item is displayed in the display frame 235 of the screen.

Depending on the set moving image recording size and moving image frame rate, it may not be necessary to maximize the reading throughput of the second image sensor 203. For example, when the standard image quality is sufficient for the moving image recording size, it is possible to reduce the necessary reading throughput by thinning out and reading out the pixel data of the second image sensor 203. In the present embodiment, when the recording size of the moving image set by the recording size setting process is equal to or smaller than the threshold value, the AF calculation unit 122 acquires the pixel data output from the second image sensor 203 and performs focus detection.
Even when the moving image frame rate is 24 to 30 frames, the readout throughput of the image sensor may not be required so much. In the present embodiment, when the frame rate set by the frame rate setting process is equal to or less than the threshold value, the AF calculation unit 122 acquires pixel data output from the second image sensor 203 and performs focus detection. By performing phase difference detection with the second image sensor 203, the power consumption of the entire system can be significantly reduced.

FIG. 15B illustrates the subject tracking characteristic setting screen 240. “Servo AF” is a function of changing the focal position of the imaging optical system while following the subject during focus detection, and is useful for tracking moving objects. The operation unit 116, the display unit 117, and the CPU 124 perform a function setting process for setting whether to enable or disable this function.
The user operates the selection cursor 241 left and right using the operation unit 116, and determines the setting position using the SET button 243 of the operation unit 116. The set value is displayed in the display frame 242 of the screen. When the speed of a moving object changes greatly instantaneously, such as when the subject suddenly starts moving or stops suddenly, the followability to the subject can be set.

Depending on the setting of the subject tracking, even if the second image sensor 203 detects the phase difference, it may not be necessary to improve the subject tracking. When followability is not required, that is, when the function of servo AF is disabled (OFF in the figure), the AF calculation unit 122 acquires pixel data output from the second image sensor 203 and detects the focus. I do. By detecting the phase difference with the second image sensor 203, the power consumption of the entire system can be significantly reduced. Further, when the moving speed of the subject is substantially constant or when the followability is set for a low-speed moving body, high speed cannot be obtained. Therefore, when the set value is less than or equal to the threshold value, the power consumption of the entire system can be significantly reduced by detecting the phase difference using the second image sensor 203.
On the other hand, when shooting a subject that suddenly starts moving, suddenly accelerates, suddenly decelerates, or suddenly stops, high-speed tracking is required for AF tracking. When the set value exceeds the threshold value, the second image sensor 203 is exclusively used for moving image shooting, and the phase difference is detected by the first image sensor 200, whereby a high-speed and high-precision AF operation can be realized.

According to the present embodiment, the entire system can be changed by changing the image sensor that detects the phase difference in accordance with the shooting setting (including the recording size and frame rate) and the AF setting (including the tracking of the subject). Power consumption can be reduced.
As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

100, 200 First imaging device 103, 203 Second imaging device 107 Half mirror 120 First image processing unit 121 Second image processing unit 122 AF operation unit 124 CPU

Claims (14)

An imaging apparatus including a first imaging element and a second imaging element,
Light splitting means for splitting light incident through the imaging optical system and causing the first and second imaging elements to receive primary imaging light, respectively;
Focus detection means for acquiring pixel data output from the first image sensor and performing focus detection of the imaging optical system;
Control means for obtaining detection information of the focus detection means and controlling focus adjustment of the imaging optical system;
The control means acquires detection information obtained by the focus detection means performing focus detection from pixel data output from the first image sensor during photographing by the second image sensor, and performs focus adjustment of the image pickup optical system. An imaging device characterized by controlling.   The focus detection unit performs focus detection by acquiring pixel data read from phase difference detection pixels in a region set for the first image sensor while capturing a moving image by the second image sensor. The imaging apparatus according to claim 1.   The imaging apparatus according to claim 2, wherein the focus detection unit acquires pixel data read from the phase difference detection pixels in the region that are changed according to the imaging condition. Provided with operation means for selecting the focus detection area on the shooting screen,
The imaging apparatus according to claim 2, wherein the focus detection unit acquires pixel data read from the phase difference detection pixels in the focus detection region selected by the operation unit. Face detection means for detecting a face area of a subject image imaged by the second image sensor;
The imaging apparatus according to claim 2, wherein the focus detection unit acquires pixel data read from the phase difference detection pixels in a region corresponding to a face region detected by the face detection unit. . The first image sensor and the second image sensor have a plurality of photoelectric conversion units arranged in a two-dimensional array,
The imaging apparatus according to claim 1, wherein the first imaging element has a larger number of pixels than the second imaging element.   The focus detection unit obtains data obtained by adding a plurality of pixel data in the first image sensor when acquiring pixel data output from the first image sensor during capturing of a moving image by the second image sensor. The imaging device according to claim 6, wherein the imaging device is acquired.   The first image sensor outputs data obtained by adding data for pixels corresponding to a ratio of an area for one pixel of the first image sensor to an area for one pixel of the second image sensor. The imaging device according to claim 7. The focus detection unit obtains pixel data output from the first image sensor or the second image sensor and performs focus detection of the image pickup optical system,
The control means acquires detection information obtained by the focus detection means performing focus detection from pixel data output from the second image sensor during photographing by the first image sensor, and performs focus adjustment of the image pickup optical system. The imaging apparatus according to claim 1, wherein the imaging apparatus is controlled.   The focus detection unit obtains pixel data output by the phase difference detection pixels of the second image sensor when the still image is captured by the first image sensor, performs focus detection, and uses the second image sensor. The imaging apparatus according to claim 9, wherein when capturing a moving image, focus detection is performed by acquiring pixel data output from a phase difference detection pixel of the first imaging element. A recording size setting means for setting the recording size of the moving image;
The focus detection unit acquires the pixel data output from the second image sensor and performs focus detection when the recording size of the moving image set by the recording size setting unit is equal to or less than a threshold value. The imaging device according to claim 9 or 10. A frame rate setting means for setting a frame rate of the moving image;
The focus detection unit performs focus detection by acquiring pixel data output from the second image sensor when the frame rate set by the frame rate setting unit is equal to or less than a threshold value. The imaging device of any one of thru | or 11. A function setting means for setting valid or invalid for the function of changing the focal position of the imaging optical system while following the subject at the time of focus detection;
13. The focus detection unit according to claim 9, wherein when the function is disabled by the function setting unit, the focus detection unit acquires the pixel data output from the second image sensor and performs focus detection. The imaging device according to any one of the above. A control method executed by an imaging apparatus including a first imaging element and a second imaging element,
Splitting light incident through the imaging optical system by a light splitting means and causing the first imaging element and the second imaging element to receive primary imaging light, respectively;
A focus detection step of acquiring pixel data output from the first image sensor and performing focus detection of the imaging optical system;
A control step of obtaining detection information in the focus detection step and controlling focus adjustment of the imaging optical system;
In the control step, detection information obtained by performing focus detection in the focus detection step using pixel data output from the first image sensor during photographing by the second image sensor is acquired, and the imaging optical system A method for controlling an imaging apparatus, characterized by controlling focus adjustment.

Images