JP7058971B2 - Image pickup device and its control method - Google Patents

Description

The present invention relates to an image pickup apparatus and a control method for the image pickup apparatus. In particular, an image sensor having a plurality of photoelectric conversion units for one microlens is used to acquire a pupil division image, and distance information is generated from the phase difference of the obtained pupil division image, and the obtained distance is obtained. The present invention relates to an image pickup device that performs focus detection and image processing from information.

Conventionally, as an AF (autofocus) method of an image pickup device, a pupil division image is acquired by an image sensor having a plurality of photoelectric conversion units for one microlens, and the phase difference of the obtained pupil division image is obtained. The technique of performing focus detection is known. For example, in Patent Document 1, the phase difference is obtained from the pupil-divided image and the focus is detected, and at the same time, all the pixels corresponding to the same microlens are added and treated as one pixel, which is the same as the conventional image sensor arrangement. To be. Then, a technique for creating an ornamental image by a conventional signal processing technique is disclosed. Further, Patent Document 2 discloses a technique for removing local light distribution unevenness by the strobe (flash means) in a photographed image taken by strobe photography using a so-called strobe which is a flash means. .. Further, in Patent Document 2, in order to remove local light distribution unevenness due to the strobe, the distance information to the subject is calculated by obtaining the phase difference from the pupil-divided image, and the distance information and the light distribution characteristics of the strobe means are used. A technique for correcting light distribution unevenness based on the light distribution characteristic data is disclosed.

Japanese Unexamined Patent Publication No. 2008-134389 Japanese Unexamined Patent Publication No. 2013-59082

However, in the techniques disclosed in Patent Documents 1 and 2, focus detection and correction of light distribution unevenness by a strobe are common until phase difference detection is performed from a pupil-divided image, but the required processing time and in the image are the same. Since the resolution (number of divisions) is different, separate processing circuits are required. That is, for example, focus detection processes several hundred frames in a short time (several msec), while correction of light distribution unevenness by a strobe processes tens of thousands of frames in a long time (several hundred msec). Therefore, a separate processing circuit is required.

In view of the above problems, it is an object of the present invention to provide an image pickup apparatus capable of improving usability without increasing the circuit scale by adaptively switching the generation parameters when generating distance information.

In order to solve the above problems, the image pickup apparatus of the present invention includes a calculation circuit for calculating distance information of a subject corresponding to the plurality of pixel signals based on a plurality of pixel signals having different viewpoints, and the calculation circuit. Control means for controlling, a focus detection circuit that detects the focus of an image signal based on the plurality of pixel signals based on the distance information, and image processing on the image signal based on the plurality of pixel signals based on the distance information. The control means controls the number of pixels of the pixel signal used by the calculation circuit and the resolution of the generated distance information, and the calculation circuit has the number of pixels by the control means. It is characterized by outputting the distance information calculated to any one of the focus detection circuit and the image processing circuit based on the setting of the resolution. ..

According to the present invention, it is possible to provide an image pickup apparatus capable of improving usability without increasing the circuit scale by adaptively switching the generation parameters when generating distance information. Therefore, AF and image processing can be exclusively realized without increasing the circuit scale by adaptively switching the parameters given to the common processing circuit, and usability can be further improved.

It is a block diagram which shows the structure of an image pickup apparatus. It is a detailed figure of the distance map information generated by a distance map generation circuit. It is a still image single shooting sequence. It is a flowchart of operation branch determination. It is a still image continuous shooting sequence of 2nd Example. It is a flowchart of operation branch determination of 2nd Embodiment. It is a flowchart of operation branch determination of 3rd Embodiment.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings and the like.

(First Embodiment)
Hereinafter, the image pickup apparatus in this embodiment will be described.

The image sensor (image sensor) 101 is a photoelectric conversion unit that receives a light beam from an image pickup optical system via a plurality of microlenses and divides a pixel portion into two in the horizontal direction under at least a part of the microlenses. It is an image sensor. At this time, the pixel signals obtained from the photoelectric conversion unit in the same microlens have different viewpoints and have a phase difference (parallax). In the present embodiment, as will be described later, the minimum required two divisions are for calculating the distance information based on the phase difference (parallax, image shift amount) of the output from the image sensor 101, but this is the division method. Not limited to. There may be three or more divisions under the microlens in the horizontal direction, or a plurality of divisions in other directions such as vertical may be made. The user instruction unit 109 is an instruction unit for instructing a shooting mode or the like. The shutter button 110 is a shutter button operated by the user at the time of shooting. The auxiliary light light emitting unit 111 is a light emitting unit that emits auxiliary light such as a strobe. The posture detection unit 112 is a detection unit that detects the posture of a gyro sensor or the like. The system control unit 113 is a control unit for controlling a system such as a microcomputer.

In addition, the data that goes in and out of each component will be described. The data are RAW image data S101 for development, RAW image data S102 for distance measurement, developed image data S103, face detection data S104, corrected image data S105, distance map information S106, and lens control signal S107. Further, there are a shooting mode flag S108, a shutter state flag S109, an auxiliary light emission instruction signal S110, a light emission charge remaining amount signal S111, a camera attitude information S112, and a distance map generation parameter S113. Further, the AF control signal S114, the distance map application image processing control signal S115, and the buffered RAW image data S116 for development and distance measurement.

First, the outline of the operation of the image pickup apparatus according to the present embodiment will be described with reference to FIG. 1, and then the main components and their operations will be described. Since the image sensor 101 divides the pixel portion into two in the horizontal direction under the same microlens, by adding the two pixel outputs (pixel signals), the RAW for development used in the normal development process is used. Image data S101 can be generated. Further, the RAW image data S102 for distance measurement including the parallax information can be generated by reading out the two pixel outputs respectively or by using the calculation result of the pixel outputs. It should be noted that such an image sensor is already known, and since the image sensor itself is not the essence required by the present invention, detailed description thereof will be omitted.

The RAW image data S101 for development is sent to the development processing circuit 102 and the face detection circuit 103, and the developed image data S103 and the face detection data S104 are generated. The developed image data S103 may be, for example, Y / Cr / Cb data to which various image processes such as defect pixel correction, white balance adjustment, and demosaic processing have been performed. The face detection data S104 includes, for example, information indicating the position, size, and detection reliability of the human face included in the developed image data S103. The face detection data S104 is sent to the system control unit 113 and is used for controlling the image pickup apparatus according to the present embodiment, which will be described later.

The RAW image data S102 for distance measurement is sent to the distance map generation circuit (calculation circuit) 104, and the amount of image shift is calculated by a correlation calculation using a known technique such as SAD (Sum Of Absolute Difference). Then, the distance information (distance map information S106) is generated from the calculated image shift amount. The form of the distance information obtained from the image shift amount includes the defocus amount and the distance information to the subject, and the generation of these distance information by the correlation calculation is known, and it is not the essence required by the present invention. Omit. Here, the distance map information S106 is generated as information in units of small blocks in which the imaging screen is set horizontally and vertically at predetermined intervals, and the arrangement density (hereinafter referred to as spatial resolution of the distance map) is arbitrarily set. It is possible.

Further, the RAW image data S102 for distance measurement can be temporarily held in the image memory 114. Then, the buffered RAW image data S116 for development and distance measurement can be sent to the development processing circuit 102 and the distance map generation circuit 104, respectively, and the past RAW image data can be developed and the distance map generated. Can be done. The distance map information S106 is sent to the distance map application image processing circuit (image processing circuit) 105 and the AF control circuit (focus detection circuit) 107, and is used in each processing. In the distance map application image processing circuit 105 and the AF control circuit 107, processing is performed on the image signal based on the pixel signal based on the distance map information S106.

As disclosed in Patent Document 2, the distance map application image processing circuit 105 obtains the shape information of the subject based on the distance information. Then, the image taken based on the obtained shape information and distance information of the subject and the position of the virtual light source is processed, and a corrected image in which the subject is irradiated with light by the virtual light source is generated. The corrected image data S105 is the data obtained as a result of this processing, and it is known that the corrected image data S105 can be made into an image in which the partial shine of a human face is relaxed, which is taken with an auxiliary light such as a strobe. There is. The corrected image data S105 can be confirmed by the photographer by being output to the rear liquid crystal panel 106 and displayed. The corrected image data S105 is recorded on a recording medium such as an SD card as a recorded image in a recording system (not shown).

On the other hand, the AF control circuit 107 selects the distance from the distance map information S106 to the subject position where the lens should be focused, and automatically adjusts the focus (hereinafter referred to as AF). The AF control circuit 107 realizes AF control (focus detection) by outputting the lens control signal S107 to the focus lens unit 108.

Next, the components and their operations according to the present embodiment will be described in detail. FIG. 3 is a diagram showing a shooting sequence of the imaging device according to the present embodiment. The timings shown in FIG. 3 indicate a shooting sequence start t0, a shutter half-press t1, a shutter full-press t2, a preview image output t3 on the rear panel, and a next shooting sequence start t4. Further, a first shooting framing period T0 indicating an operation such as the photographer holding the camera with respect to the subject, and a second shooting framing period T1 indicating a state in which the photographer intentionally enters the AF operation are shown. Further, the distance map generation mode switching determination period T2 for determining whether the photographer intends to press the shutter with the intention of shooting and then prepares for the next shooting or has the intention of confirming the result of shooting is shown. Then, the distance map application image processing period T3 and the transition period T4 to the next shooting mode are shown, in which the confirmation image of the result taken immediately before by the photographer is corrected.

Here, the distance map generation mode for switching to the distance map generation mode switching determination period T2 includes the distance map application image processing mode and the AF mode. The distance map application image processing mode is a mode in which the photographer confirms the shooting result by operating the distance map application image processing circuit 105. The AF mode is a mode in which the AF control circuit 107 is operated so that the photographer performs the AF operation during shooting.

For example, a typical shooting operation when a photographer shoots a person with an auxiliary light such as a strobe will be described. In this shooting operation, the shooting operation is started at the shooting sequence start t0, the person to be the subject is framed during the first shooting framing period T0, and the AF operation is started by pressing the shutter halfway t1. Then, after the second shooting framing period T1, the shutter is fully pressed t2, and the system control unit 113 transmits the auxiliary light emission instruction signal S110 to the auxiliary light emission unit 111, so that the auxiliary light is emitted and one shot is taken. Still image shooting is completed.

Here, the period from the start to the completion of the above-mentioned shooting is defined as the first period. In the first period, the user instruction unit 109 for switching the shooting mode of FIG. 1 issues the shooting mode flag S108 as the single shooting still image shooting mode with auxiliary light at the shooting sequence start t0. Then, at the shutter half-press t1, the shutter button 110 is half-pressed by the photographer, so that the shutter state flag S109 is issued as a half-pressed state. Further, at the shutter fully pressed t2, the shutter state flag S109 is issued as a fully pressed state. Since the subject is clearly being photographed during the first period, the distance map generation mode is set to the AF mode.

Next, the period from the first period to the next shooting is defined as the second period. In the second period, the actions that the photographer can take after the shooting is completed are divided into the following two. One is to prepare for the next shooting while framing the image pickup device with respect to the subject. The other is to stop framing the subject and check the shooting result by looking at the back panel. The distance map generation mode switching determination period T2 is also a period for determining the above two operation branches. In the present embodiment, at least one of the face detection data S104 from the face detection circuit 103, the light emission charge remaining amount signal S111 from the auxiliary light light emitting unit 111, and the camera attitude information from the attitude detection unit 112 such as a gyro sensor. To determine the operation branch using.

FIG. 4 is a flowchart of operation branch determination in the single-shooting still image shooting mode with auxiliary light. When the operation branch determination process is started in step S410, the process proceeds to step S402. Then, in step S402, it is determined whether or not the auxiliary light function such as the strobe is enabled by the user. In the present embodiment, the auxiliary light emitting unit 111 is a component, but whether or not the function is enabled can be switched by the user's setting. When the auxiliary light function such as a strobe is enabled by the user (Yes), the process proceeds to step S403, and as the first determination of the operation branch determination, the determination is made as to whether or not the auxiliary light such as the strobe can emit the next light. .. On the other hand, when the auxiliary light function such as the strobe is not enabled by the user (No), it cannot be determined whether the next light emission is possible or not, which is the first determination of the operation branch determination. Therefore, the next step is step S404. Proceed to the determination of whether or not the face detection function of is enabled.

Next, in step S403, it is determined whether or not the next light emission is possible based on the light emission charge remaining amount signal S111. If, as a result of the determination, it is determined that the next light emission is impossible (No), it is not possible to prepare for the next shooting. Therefore, as the distance map generation mode, the process proceeds to step S408, and the distance map application image processing mode is performed. Set to. On the other hand, if it is determined in step S403 that the next light emission is possible (Yes), it is possible to prepare for the next shooting, but in order to make the operation branch determination more reliable, step S404 is the next step. Proceed to the determination of whether or not the face detection function of is enabled.

Then, when the face detection function is enabled by the user (Yes), the process proceeds to step S405, and as the second determination of the motion branch determination, the determination of whether or not the face of the subject is detected proceeds. On the other hand, when the face detection function is not enabled by the user (No), there is no determination as to whether or not the face of the subject is detected, which is the second determination of the motion branch determination, so that the motion branch determination is the second. As the determination of 3, the process proceeds to step S406. Then, in step S406, it is determined whether or not the entire image pickup apparatus has transitioned downward based on the camera posture information S112.

Next, in step S405, it is determined whether or not the face is continuously detected based on the face detection data S104. If the face continues to be detected (Yes), it is determined that the photographer continues framing the subject person and intends to prepare for the next shooting, and the distance map generation mode proceeds to step S407. , Set to AF mode. On the other hand, when the face is not detected in step S405 (No), it can be determined that the photographer intends to stop framing the subject person and confirm the shooting result, but in order to make the operation branch determination even more reliable. , Step S406. Then, as a third determination of the motion branch determination, it is determined in step S406 whether or not the entire image pickup apparatus has transitioned downward (the direction in which the image pickup device is facing the ground) based on the camera posture information S112. do. In the operation of the photographer checking the shooting result by looking at the rear panel attached to the image pickup device, the lens surface of the image pickup device faces diagonally downward, that is, the back panel faces the photographer's face. There are many.

Next, in step S406, when it is determined that the image pickup device has changed its posture downward (Yes), the photographer certainly has the intention of confirming the shooting result, and proceeds to step S408, and the distance map generation mode is set. , Set to distance map application image processing mode. On the other hand, if the posture of the image pickup apparatus has not changed downward in step S406 (No), the process proceeds to step S407. That is, although the photographer has stopped framing the subject person for another intention, the process proceeds to step S407 and the AF mode is set, assuming that there is no intention of confirming the shooting result and the intention is to continue shooting.

As described above, in the second period, by performing the first, second, and third determinations as the operation branch determination, the distance map generation mode is the AF mode or the distance map application image processing mode. Set to either. Then, when the above operation is completed, the operation branch determination process is terminated in step S409.

The flow chart of the process shown in FIG. 4 is only an example, and the essence required by the present invention is the same even if the combination of flows is changed depending on how the photographer's intention is determined. Further, the operation branch determination described in the flowchart of FIG. 4 is a determination that captures the intention of the photographer, and may be set to the distance map application image processing mode in the second period. .. Further, in selecting the distance map generation mode to either the AF mode or the distance map application image processing mode, the determination method is not limited to the method disclosed in the present embodiment. That is, at least one of the face detection data S104 from the face detection circuit 103, the light emission charge remaining amount signal S111 from the auxiliary light light emitting unit 111, and the camera attitude information S112 from the attitude detection unit 112 such as the gyro sensor is used. It is not limited as long as it is a judgment method.

FIG. 2 is a detail of the distance map information S106 generated by the distance map generation circuit 104. The map information shown in FIG. 2 shows distance map information having spatial resolution of 7 horizontal points and 7 vertical points, and shows the entire captured image 201 and the block division 202 for the range in which the distance information of one point is generated. The spatial resolution of the distance map has an upper limit depending on the number of pixels that can be used for the correlation calculation, but can be arbitrarily set as long as it is within the limit. However, the distance map generation circuit 104 requires a predetermined time to calculate one point of distance map information. Therefore, if the spatial resolution of the distance map is increased, it takes time to generate the distance map, and if the spatial resolution is decreased, the distance map generation time is shortened.

In the process shown in FIG. 4, when the distance map generation mode is selected as the distance map application image processing mode, the spatial resolution is set to the resolution required for the distance map application image processing (for example, 100x100) as the distance map generation parameter S113. Set. As a result, the distance map information S106 has a resolution required for the distance map application image processing, and a predetermined image processing is executed in the distance map application image processing circuit 105.

On the other hand, in the flowchart shown in FIG. 4, when the distance map generation mode is selected as the AF mode, the spatial resolution is set to the resolution required for AF control (for example, 10x10) as the distance map generation parameter S113. As a result, the distance map information S106 has a resolution required for AF control, and a predetermined AF control is executed by the AF control circuit 107.

Further, when the distance map generation mode is selected as the distance map application image processing mode, the development and distance measurement RAW image data S116 buffered from the image memory 114 are generated in the distance map application image processing period T3 of FIG. Read out. Then, the distance map generation circuit 104 executes a distance map generation with a spatial resolution of 100x100, the development processing circuit 102 and the distance map application image processing circuit 105 perform development and distance map application image processing, and the rear liquid crystal panel 106 Is displayed in.

As described above, the distance map generation mode is set to the AF mode during the first period. Then, in the second period that follows, the distance map generation mode is set to either the AF mode or the distance map application image processing mode using at least one of the face detection data, the charge remaining amount signal for light emission, and the camera attitude information. do. This makes it possible to exclusively use the common distance map generation circuit and control the image pickup device so as not to give a sense of discomfort to the photographer.

(Second Embodiment)
FIG. 5 shows a shooting sequence in the imaging device according to the present embodiment. In FIG. 5, the same time and period as those in the photographing sequence shown in FIG. 3 of the first embodiment are designated by the same reference numerals as those in FIG. 3, and the description thereof will be omitted. Further, since the components of the image pickup apparatus are the same as those of the first embodiment, the description thereof will be omitted.

In FIG. 5, what is different from FIG. 3 is a shutter OFF t5, a shutter full press continuation period T5, an imaging preview mode OFF period T6, and an imaging preview mode ON period T7. In the first embodiment, an example of one-shot still image shooting with an auxiliary light such as a strobe is shown, whereas in the present embodiment, an example in which a photographer shoots a person in a still image shooting by continuous shooting is shown. ..

As a typical shooting operation during continuous shooting, the shooting operation is started at the shooting sequence start t0 shown in FIG. 5, the person to be the subject is framed during the first shooting framing period T0, and the AF operation is performed by pressing the shutter halfway t1. To start. Then, the shutter full press t2 is started after the second shooting framing period T11, and the still image shooting by continuous shooting is continued during the shutter full press continuation period T5. Still image shooting by continuous shooting is stopped by the shutter OFFt5, and the image shot as the final frame of continuous shooting is displayed on the rear liquid crystal panel 106.

In the system control unit 113, the period from the shooting sequence start t0 to the shutter OFF t5 is set to the imaging preview mode OFF period T6, and the period from the shutter OFF t5 to the next shooting sequence start t4 is set to the imaging preview mode ON period T7. Since the image pickup preview mode OFF period T6 is during continuous shooting, the distance map generation mode is set to the AF mode. Further, the image pickup preview mode OFF period T6 corresponds to the first period of the first embodiment, and the image capture preview mode ON period T7 corresponds to the second period of the first embodiment. In the imaging preview mode ON period T7, as in the first embodiment, the photographer prepares for the next shooting while framing the image pickup device with respect to the subject, or temporarily stops framing the subject and shoots by looking at the back panel. Branch to one of the actions to check the result. The distance map generation mode switching determination period T2 is the same as that of the first embodiment in that the distance map generation mode switching determination period T2 is a period for determining the above two operation branches.

In the second embodiment, since it is an application example of continuous shooting, it is not premised on auxiliary light emission such as a strobe. The captured image is processed based on the shape information and distance information of the subject obtained based on the distance information and the position of the virtual light source. The process of generating a corrected image by irradiating the subject with light by the virtual light source does not necessarily presuppose the auxiliary light emission, and is established as a distance map application image process.

FIG. 6 is a flowchart of the operation branch determination in the present embodiment. The difference from FIG. 4 is the determination of whether or not the auxiliary light emission function such as the strobe is enabled (step S402 in FIG. 4) and the determination of whether or not the next emission is possible (step of FIG. 4) from the flowchart. There is no S403). Since the other processes of the flowchart are the same as the description of the first embodiment shown in FIG. 4, detailed description will be omitted.

In the second period, by performing the second and third determinations as the operation branch determination, the distance map generation mode is set to either the AF mode or the distance map application image processing mode. When the above operation is completed, the operation branch determination process is terminated in step S409. As a result of the operation branch determination, the operation after either the AF mode or the distance map application image processing mode is set as the distance map generation mode is not the essence of the present invention, but is the same as that of the first embodiment. The explanation is omitted.

As described above, even when the still image shooting by continuous shooting is performed without the auxiliary light such as the strobe, the distance map generation mode is set to the AF mode for the first period. Then, in the second period that follows, the distance map generation mode is set to either the AF mode or the distance map application image processing mode using at least one of the face detection data, the charge remaining amount signal for light emission, and the camera attitude information. do. This makes it possible to exclusively use the common distance map generation circuit and control the image pickup device so as not to give a sense of discomfort to the photographer.

(Third Embodiment)
In the present embodiment, as in the first embodiment, a case where one-shot still image shooting with auxiliary light such as a strobe is performed will be described. As a condition different from the first embodiment, it is assumed that the shooting mode is set when the image pickup device is fixed by a tripod or the like from the user instruction unit 109 such as the shooting mode. The photographing sequence of the present embodiment is the same as that of the first embodiment shown in FIG. 3, and the components of the image pickup apparatus are the same as those of the first embodiment.

FIG. 7 is a flowchart showing an operation branch determination in the image pickup apparatus according to the present embodiment. There are three differences from the flow chart of the operation branch determination in the first embodiment shown in FIG. The first point is that there is no determination (third determination of motion branch determination) as to whether or not the entire image pickup apparatus has transitioned downward based on the camera attitude information S112 in step S406 from the flowchart. be. The second point is that in the determination of whether the face detection function of step S404 is enabled, if it is not enabled, the process proceeds to step S407, and the distance map generation mode is set to AF mode. The third point is that in step S405, when the face is not detected in the determination of whether or not the face is detected based on the face detection data S104, the distance map application image processing mode is set as the distance map generation mode. It is a point to set. Others are the same as those in the first embodiment, and thus the description thereof will be omitted.

In the present embodiment, since the image pickup device is fixed by a tripod or the like, the lens surface of the image pickup device is obliquely lowered in the operation of the photographer looking at the rear panel to check the shooting result as in the first embodiment. That is, the back panel does not face the photographer's face. Therefore, the third determination of the motion branch determination is not performed, the first determination of the motion branch determination and the second determination of the motion branch determination are performed, and the face is detected in the second determination of the motion branch determination. It is assumed that the next shooting preparation is not made in the state where it is not. In addition, when the face detection function is not enabled, the flowchart emphasizes the possibility of preparing for the next shooting.

As described above, even when the image pickup apparatus is fixed by a tripod or the like, the distance map generation mode is set to the AF mode for the first period. Then, in the second period that follows, the distance map generation mode is set to either the AF mode or the distance map application image processing mode using at least one of the face detection data, the charge remaining amount signal for light emission, and the camera attitude information. do. This makes it possible to exclusively use the common distance map generation circuit and control the image pickup device so as not to give a sense of discomfort to the photographer.

Moreover, although the preferred embodiment of the present invention has been described, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

104 Distance map generation circuit (calculation circuit)
105 Distance map application image processing circuit (image processing circuit)
107 AF control circuit (focus detection circuit)
113 System control unit (control means)

Claims (10)

A calculation circuit that calculates distance information of a subject corresponding to the plurality of pixel signals based on a plurality of pixel signals having different viewpoints, and a calculation circuit.
A control means for controlling the calculation circuit and
A focus detection circuit that detects the focus of an image signal based on the plurality of pixel signals based on the distance information, and a focus detection circuit.
An image processing circuit that performs image processing on an image signal based on the plurality of pixel signals based on the distance information, and an image processing circuit.
Have,
The control means controls the number of pixels of the pixel signal used by the calculation circuit and the resolution of the generated distance information.
The calculation circuit is characterized in that it outputs distance information calculated to any one of the focus detection circuit and the image processing circuit based on the setting of the number of pixels and the resolution by the control means. Device. The feature is that the focus detection is performed by the focus detection circuit in the first period of the period from the start of shooting to the next shooting, and the image processing is performed by the image processing circuit in the second period. The image processing apparatus according to claim 1. 2. The first period is a period from the start to the completion of the imaging, and the second period is a period from the first period to the next imaging. The image processing apparatus according to. The image processing apparatus according to any one of claims 1 to 3 , wherein the plurality of pixel signals having different viewpoints are output from an image pickup device including a plurality of photoelectric conversion means in one microlens. .. The imaging according to claim 2, wherein in the second period, when the auxiliary light emitting unit is effective and when the subject is detected, the focus detection is performed by the focus detection circuit. Device. The fifth aspect of the present invention is characterized in that the image processing is performed by the image processing circuit when the next light emission cannot be performed by the auxiliary light emitting unit even when the auxiliary light emitting unit is effective. Imaging device. Even when the subject is detected in the second period, if the posture of the image pickup apparatus changes so as to face the ground, the image processing is performed by the image processing circuit. The image pickup apparatus according to claim 5. The claim is characterized in that when the image pickup apparatus is in a fixed state, the focus detection is performed by the focus detection circuit even when the subject is not detected in the second period. 5. The imaging device according to 5. Even when the subject is detected in the second period, if the subject is a person and the face of the person is not detected, the image processing by the image processing circuit is performed. The image pickup apparatus according to claim 8, wherein the image pickup apparatus is provided. A calculation process for calculating distance information of a subject corresponding to the plurality of pixel signals based on a plurality of pixel signals having different viewpoints, and a calculation process.
A focus detection step of performing focus detection of an image signal based on the plurality of pixel signals based on the distance information, and
An image processing step of performing image processing on an image signal based on the plurality of pixel signals based on the distance information, and an image processing step.
A control step that controls the number of pixels of the pixel signal used in the calculation step and the resolution of the generated distance information, and
Control of an image processing apparatus characterized in that the distance information calculated in any one of the focus detection step and the image processing step is output in the control step based on the setting of the number of pixels and the resolution. Method.

Images