WO2023042453A1

WO2023042453A1 - Imaging device, image processing method, and program

Info

Publication number: WO2023042453A1
Application number: PCT/JP2022/013430
Authority: WO
Inventors: 大水落; 龍之介横矢; 祐基明壁; 貴洸小杉; 洋司山本
Original assignee: ソニーグループ株式会社
Priority date: 2021-09-14
Filing date: 2022-03-23
Publication date: 2023-03-23

Abstract

Provided are a device and a method for calculating an image blend ratio and a distance to an object in pixel region units on the basis of a defocus amount in pixel region units of a captured image and carrying out various kinds of image processing using calculated data. The present invention comprises a defocus amount calculation unit for calculating a defocus amount in pixel region units of a captured image which is output from an imaging element of an imaging device, and carries out an image blend process by using the defocus amount in pixel region units, which has been calculated by the defocus amount calculation unit, to calculate an image blend ratio in pixel region units, a three-dimensional image generation process by calculating a distance to an object in pixel region units on the basis of the defocus amount in pixel region units and using the distance to the object in pixel region units, and an image stabilization amount calculation process in pixel region units. Alternatively, the present invention carries out various processes such as a green-background image generation process by determining a mask region with respect to the captured image on the basis of the defocus amount in pixel region units.

Description

IMAGING DEVICE, IMAGE PROCESSING METHOD, AND PROGRAM

The present disclosure relates to imaging devices, image processing methods, and programs. More specifically, the present invention relates to an imaging apparatus, an image processing method, and a program that apply detection information of image plane phase difference detection pixels not only to focus control but also to various other processes.

In imaging devices (cameras), there is an image plane phase difference method using an image plane phase difference pixel as a method for detecting a focus position (in-focus position).
This image plane phase difference method divides the light passing through the imaging lens into a pair of images and analyzes the phase difference between the generated pair of images to detect the focus position (in-focus position). It is a method to

In focus control using the image plane phase difference method, the light flux that has passed through the imaging lens is split into two, and the two split light fluxes are received by a set of image plane phase difference detection pixels that function as focus detection sensors. The focus lens is adjusted by detecting the degree of focus based on the shift amount of the signal output according to the amount of light received by each of the set of image plane phase difference detection pixels.

The detection information of the image plane phase difference detection pixels is mainly used for focus control, but it can be applied to other processing as well as focus control.
For example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2012-142952) discloses a configuration that performs blurring processing on a captured image using detection information of an image plane phase difference detection pixel.
Specifically, the detection information of the image plane phase difference detection pixels is used to analyze the distribution of the defocus amount in the captured image, and the analysis result is used to apply blur processing to the captured image.

Further, Patent Document 2 (Japanese Patent Application Laid-Open No. 2019-035967) discloses a configuration for changing and outputting the gradation characteristics of the defocus amount and the distance information obtained from the detection information of the image plane phase difference detection pixels according to the output destination. ing.
For example, different information is generated based on the detection information of the image plane phase difference detection pixels according to the information required by the output destination, such as an output destination that requires distance resolution near in-focus and an output destination that requires distance measurement range information. A configuration for outputting by

JP 2012-142952 A JP 2019-035967 A

An object of the present disclosure is to provide an imaging device, an image processing method, and a program that realize a new configuration for using detection information of image plane phase difference detection pixels.

For example, in an embodiment configuration of the present disclosure, a configuration is realized in which the detection information of the image plane phase difference detection pixels is used to calculate the image blend ratio when generating a composite image of a plurality of images.
Furthermore, in an embodiment configuration of the present disclosure, a configuration is realized in which distance information to be used for three-dimensional image generation is generated from detection information of image plane phase difference detection pixels.
Furthermore, in an embodiment configuration of the present disclosure, a configuration is realized in which information for determining a correction mode of camera shake correction is generated based on detection information of image plane phase difference detection pixels and camera shake correction is performed.

A first aspect of the present disclosure includes:
a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an image sensor of an imaging device;
an image blend ratio calculation unit that calculates an image blend ratio between the captured image and another image;
The image blend ratio calculation unit
In the imaging device, an image blend ratio for each pixel area is calculated based on the defocus amount for each pixel area calculated by the defocus amount calculation unit.

Furthermore, a second aspect of the present disclosure is
a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an image sensor of an imaging device;
The imaging apparatus includes a distance information calculation unit that calculates the object distance for each pixel area based on the defocus amount for each pixel area calculated by the defocus amount calculation unit.

Furthermore, a third aspect of the present disclosure is
a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an image sensor of an imaging device;
The imaging apparatus includes a mask area determination unit that determines a mask area for the captured image based on the defocus amount for each pixel area calculated by the defocus amount calculation unit.

Furthermore, a fourth aspect of the present disclosure is
An image processing method executed in an imaging device,
A defocus amount calculation step in which the defocus amount calculation unit calculates a defocus amount for each pixel area of a captured image output from an imaging device of an imaging device;
A defocus amount calculation unit executes an image blend ratio calculation step of calculating an image blend ratio between the captured image and another image,
The image blend ratio calculation step includes:
The image processing method includes a step of calculating an image blend ratio for each pixel area based on the defocus amount for each pixel area calculated in the defocus amount calculation step.

Furthermore, a fifth aspect of the present disclosure is
A program for executing image processing in an imaging device,
a defocus amount calculation step of causing a defocus amount calculation unit to calculate a defocus amount for each pixel area of a captured image output from an imaging element of an imaging device;
causing a defocus amount calculation unit to perform an image blend ratio calculation step of calculating an image blend ratio between the captured image and another image;
In the image blend ratio calculation step,
In the program, the image blend ratio for each pixel area is calculated based on the defocus amount for each pixel area calculated in the defocus amount calculation step.

It should be noted that the program of the present disclosure is, for example, a program that can be provided in a computer-readable format to an information processing device or computer system capable of executing various program codes via a storage medium or communication medium. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or computer system.

Still other objects, features, and advantages of the present disclosure will become apparent from more detailed descriptions based on the embodiments of the present disclosure and the accompanying drawings, which will be described later. In this specification, a system is a logical collective configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same housing.

According to the configuration of the embodiment of the present disclosure, the image blend ratio and the subject distance are calculated for each pixel region based on the defocus amount for each pixel region of the captured image, and various image processing is performed using the calculated data. An apparatus and method are realized.
Specifically, for example, a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an imaging device of an imaging device, and a defocus amount for each pixel area calculated by the defocus amount calculation unit. Calculating the image blend ratio for each pixel area using the defocus amount for each pixel area, and calculating the subject distance for each pixel area based on the defocus amount for each pixel area 3D image generation processing using , and shake correction amount calculation processing for each pixel region. Alternatively, a mask area for the captured image is determined based on the defocus amount for each pixel area, and various processes such as green screen image generation are performed.
With this configuration, it is possible to realize an apparatus and method for calculating the image blend ratio and the subject distance for each pixel area based on the defocus amount for each pixel area of the captured image, and performing various image processing using the calculated data. .
Note that the effects described in this specification are merely examples and are not limited, and additional effects may be provided.

It is a figure explaining the example of composition of the imaging device of this indication. It is a figure explaining the structural example of the image pick-up element which has a phase difference detection pixel. FIG. 10 is a diagram illustrating an outline of focus detection processing using a phase difference detection method; FIG. 10 is a diagram illustrating an outline of focus detection processing using a phase difference detection method; FIG. 10 is a diagram illustrating an outline of focus detection processing using a phase difference detection method; 4 is a diagram illustrating a configuration example of a digital signal processing unit of the image pickup apparatus of Example 1; FIG. FIG. 4 is a diagram illustrating a specific example of a pixel configuration of an image sensor and a pixel region as a defocus amount calculation unit; 4A and 4B are diagrams for explaining a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the first embodiment; FIG. 4A and 4B are diagrams for explaining a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the first embodiment; FIG. It is a figure explaining the structural example of the apparatus which performs the process of this indication. It is a figure explaining the structural example of the apparatus which performs the process of this indication. FIG. 10 is a diagram illustrating a configuration example of a digital signal processing unit of the imaging apparatus of Example 2; FIG. 11 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the second embodiment; FIG. 11 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging apparatus according to Example 3; FIG. 11 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus of Example 3; FIG. 12 is a diagram illustrating a configuration example of an imaging device and a keyer device that execute processing of the third embodiment; FIG. 11 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging apparatus according to a fourth embodiment; FIG. 12 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the fourth embodiment; FIG. 12 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the fourth embodiment; FIG. 5 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging device that executes processing of Examples 1 to 4; FIG. 2 is a diagram illustrating a hardware configuration example of an image processing device that executes processing using data input from an imaging device;

Hereinafter, details of an imaging device, an image processing method, and a program according to the present disclosure will be described with reference to the drawings. The description will be made according to the following items.
1. Configuration Example of Imaging Apparatus of Present Disclosure 2. Configuration of phase difference detection pixels, outline of focus control using detection signals from phase difference detection pixels3. Regarding processing using detection information of phase difference detection pixels 3-1. (Embodiment 1) Embodiment in which image blend ratio is calculated using detection information of phase difference detection pixels 3-2. (Embodiment 2) Embodiment in which distance information applied to three-dimensional image generation is calculated using detection information of phase difference detection pixels 3-3. (Embodiment 3) Embodiment in which a mask area for background image synthesis is determined using detection information of a phase difference detection pixel, and an image is generated by synthesizing the mask image with an area other than the selected subject area 3-4. (Embodiment 4) Embodiment in which camera shake correction amount for each pixel region is calculated using detection information of phase difference detection pixels. Other Examples 5. Hardware configuration example of an external device such as a PC connected to the imaging device6. SUMMARY OF THE STRUCTURE OF THE DISCLOSURE

[1. Regarding the configuration example of the imaging device of the present disclosure]
First, a configuration example of an imaging device according to the present disclosure will be described.

FIG. 1 is a block diagram showing a configuration example of an imaging device 100 of the present disclosure.
A configuration example of an imaging device 100 of the present disclosure will be described with reference to FIG.
Incident light passing through the focus lens 101 and the zoom lens 102 is input to an imaging device 103 such as a CMOS or CCD, and photoelectrically converted in the imaging device 103 .

The image pickup device 103 has a plurality of pixels each having a photodiode, for example, arranged two-dimensionally in a matrix. It has normal pixels in which a B (blue) color filter is arranged, and phase difference detection pixels for pupil-dividing subject light and performing focus detection.

The normal pixels of the image sensor 103 generate analog electrical signals (image signals) of R (red), G (green), and B (blue) color components of the subject image, and generate R, G, and B color image signals. Output.
A phase difference detection pixel of the image sensor 103 outputs a phase difference detection signal (detection information). A phase difference detection signal (detection information) is a signal mainly used for autofocus control.
The configuration of the phase difference detection pixels, phase difference detection signals generated by the phase difference detection pixels, and focus control using the phase difference detection signals will be described later in detail.

Photoelectrically converted data output from the image sensor 103 in this manner includes RGB image signals and phase difference detection signals from phase difference detection pixels.
Each of these signals is input to the analog signal processing unit 104 , subjected to processing such as noise removal in the analog signal processing unit 104 , and converted into a digital signal in the A/D conversion unit 105 .

A digital signal digitally converted by the A/D conversion unit 105 is input to a digital signal processing unit (DSP) 108 and subjected to various signal processing.
Various image signal processing such as demosaic processing, white balance adjustment, gamma correction, etc. is performed on the RGB image signal, and the processed image is recorded in a recording device 115 such as a flash memory.
Further, it is displayed on the monitor 117 and viewfinder (EVF) 116 . An image through the lens is displayed as a through image on the monitor 117 and the viewfinder (EVF) 116 regardless of whether or not shooting is performed.

Phase difference detection pixel information (detection signal) output from the phase difference detection pixels of the image sensor 103 is also input to the digital signal processing unit (DSP) 108 via the AD conversion unit 106 .
A digital signal processing unit (DSP) 108 analyzes the phase difference between the pair of images generated by the phase difference detection pixel information (detection signal), and focuses on the subject (focusing object) to be focused. , that is, the amount of deviation between the in-focus distance and the object distance (defocus amount (DF)) is calculated.

An input unit (operation unit) 118 is an operation unit including an input unit for inputting various operation information such as shutter and zoom buttons on the camera body, a mode dial for setting the shooting mode, and the like.

The control unit 110 has a CPU, and controls various processes executed by the imaging device according to programs stored in advance in a memory (ROM) 120 or the like. A memory (EEPROM) 119 is a non-volatile memory and stores image data, various auxiliary information, programs, and the like.

The memory (ROM) 120 stores programs, calculation parameters, etc. used by the control unit (CPU) 110 . A memory (RAM) 121 stores programs used in the control unit (CPU) 110, the AF control unit 112a, and the like, parameters that change as appropriate during the execution of the programs, and the like.

The gyro 131 is a sensor that measures the inclination, angle, inclination velocity (angular velocity), etc. of the imaging device 100 . The detection information of the gyro 131 is used, for example, for calculating the amount of camera shake during image capturing.

The AF control unit 112a drives the focus lens drive motor 113a set corresponding to the focus lens 101, and executes autofocus control (AF control) processing. For example, the focus lens 101 is moved to the focus position of the focus lens 101 with respect to the subject included in the area selected by the user to obtain the focused state.

The zoom control unit 112b drives a zoom lens driving motor 113b set corresponding to the zoom lens 102. FIG. A vertical driver 107 drives an image sensor (CCD) 103 . The timing generator 106 generates control signals for processing timings of the image sensor 103 and the analog signal processing unit 104, and controls the processing timings of these processing units.
Note that the focus lens 101 is driven in the optical axis direction under the control of the AF control section 112a.

[2. Configuration of Phase Difference Detection Pixels, Overview of Focus Control Using Detection Signals from Phase Difference Detection Pixels]
Next, the configuration of the phase difference detection pixels and the outline of focus control using detection signals from the phase difference detection pixels will be described.

As described above, the imaging device 103 of the imaging apparatus 100 shown in FIG. , G, and B image signals, and phase difference detection signals (detection information) from the phase difference detection pixels, which are signals used for autofocus control, are also output.
A specific pixel configuration example of the image sensor 103 will be described with reference to FIG.

FIG. 2 is a diagram showing a pixel configuration example of the image sensor 103. As shown in FIG.
FIG. 2 shows a pixel configuration example of the image sensor 103 corresponding to (A) a partial area of the captured image.
The vertical direction is the Y-axis, and the horizontal direction is the X-axis. In FIG. 2, one pixel is indicated by one square.
The RGB pixels shown in FIG. 2 are pixels for normal image capturing. RGB pixels have, for example, a Bayer array configuration.

Detection information acquisition pixels applied to the autofocus process, ie, phase difference detection pixels 151 for acquiring phase difference information, are discretely set in some (rows) of RGB pixels having a Bayer array.
A phase difference detection pixel is configured by a pair of a right opening phase difference detection pixel Pa and a left opening phase difference detection pixel Pb.

The following two data outputs are individually performed from the imaging device 103 .
(1) Output of pixel information (image signal) from pixels (RGB pixels) for captured images;
(2) phase difference detection pixel information ((AF) detection signal) output by the phase difference detection pixel 151;

"(1) Output of pixel information (image signal) by pixels (RGB pixels) for captured image" is output in accordance with image capturing timing by the user (photographer). A display image (live view image) to be displayed on the display is output. A display image (live view image) is output at a frame rate corresponding to the image display rate of the monitor 117 or the like.

"(2) Phase difference detection pixel information (detection signal) output by phase difference detection pixel 151" is the same interval as the image output interval or a shorter interval, for example, (1/60) sec interval (=16.7 msec interval). is done in

The phase difference detection pixel information (detection signal) output from the phase difference detection pixel 151 is input to the digital signal processor (DSP) 108 via the AD converter 106 .
A digital signal processing unit (DSP) 108 analyzes the phase difference between the pair of images generated by the phase difference detection pixel information (detection signal), and focuses on the subject (focusing object) to be focused. , that is, the amount of deviation between the in-focus distance and the object distance (defocus amount (DF)) is calculated.

An outline of focus detection processing of the phase difference detection method will be described with reference to FIGS. 3 to 5. FIG.
As described above, in the phase difference detection method, the defocus of the focus lens is detected based on the shift amount of the signal output according to the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor. The amount is calculated, and the focus lens is set to the focus position (focus position) based on this defocus amount.

The set of phase difference detection pixels described with reference to FIG. 2 will be referred to as pixel Pa and pixel Pb, respectively, and the details of light incident on these pixels will be described with reference to FIG.

As shown in FIG. 3, the phase difference detection unit stores a luminous flux Ta from a right portion (also referred to as a “right partial pupil region” or simply a “right pupil region”) Qa of the exit pupil EY of the photographing optical system, and a light beam Ta from the left side. Phase difference detection pixels Pa and Pb configured by a pair of photodetectors (PD) for receiving a light flux Tb from a portion (also referred to as “left partial pupil region” or simply “left pupil region”) Qb are arranged in the horizontal direction. It is Here, in the drawing, the +X direction side is expressed as the right side, and the −X direction side is expressed as the left side.

Of the pair of phase difference detection pixels Pa and Pb, one phase difference detection pixel (hereinafter referred to as “first phase difference detection pixel”) Pa collects incident light to the first phase difference detection pixel Pa. A first light shielding plate AS1 having a microlens ML, a first slit (rectangular) opening OP1, and a second slit (rectangular) opening OP2 arranged below the first light shielding plate AS1. It is composed of a photoelectric conversion part PD that receives light through the second light shielding plate AS2.

The first opening OP1 in the first phase difference detection pixel Pa is set in a specific direction (here, the right direction (+X direction) with respect to the central axis CL passing through the center of the light receiving element PD and parallel to the optical axis LT as a reference (starting point). ). In addition, the second opening OP2 in the first phase difference detection pixel Pa is provided at a position biased in a direction opposite to the specific direction (also referred to as "anti-specific direction") with respect to the central axis CL. .

Further, of the pair of phase difference detection pixels Pa and Pb, the other phase difference detection pixel (hereinafter referred to as “second phase difference detection pixel”) Pb is a first pixel having a slit-shaped first opening OP1. A light shielding plate AS1 and a second light shielding plate AS2 arranged below the first light shielding plate AS1 and having a slit-shaped second opening OP2 are provided. The first opening OP1 in the second phase difference detection pixel Pb is provided at a position biased in the direction opposite to the specific direction with reference to the central axis CL. Also, the second opening OP2 in the second phase difference detection pixel Pb is provided at a position biased in the specific direction with reference to the central axis CL.

That is, in the pair of phase difference detection pixels Pa and Pb, the first openings OP1 are arranged to be biased in different directions. Also, the second openings OP2 are arranged to be shifted in different directions with respect to the corresponding first openings OP1 in the phase difference detection pixels Pa and Pb.

The pair of phase difference detection pixels Pa and Pb configured as described above acquire subject light that has passed through different regions (parts) in the exit pupil EY.
Specifically, the luminous flux Ta that has passed through the right pupil region Qa of the exit pupil EY passes through the microlens ML corresponding to the phase difference detection pixel Pa and the first opening OP1 of the first light shielding plate AS1. After being limited (limited) by the light shielding plate AS2, the light is received by the light receiving element PD of the first phase difference detection pixel Pa.

Further, the light flux Tb that has passed through the left pupil region Qb of the exit pupil EY passes through the microlens ML corresponding to the phase difference detection pixel Pb and the first opening OP1 of the second light shielding plate AS2, and further passes through the second light shielding plate AS2. , the light is received by the light receiving element PD of the second phase difference detection pixel Pb.

Fig. 4 shows an example of the output of the light-receiving element obtained at each pixel of Pa and Pb. As shown in FIG. 4, the output line from the pixel Pa and the output line from the pixel Pb are signals having a predetermined amount of shift Sf.

FIG. 5(a) shows the shift amount Sfa generated between the pixels Pa and Pb when the focus lens is set at a position corresponding to the subject distance and the focus is achieved, that is, in the in-focus state.
5B1 and 5B2 show shifts occurring between pixels Pa and Pb when the focus lens is not set to a position corresponding to the subject distance and is out of focus, i.e., in an out-of-focus state. Quantity Sfa is shown.
(b1) is an example in which the shift amount is larger than that at the in-focus state, and (b2) is an example in which the shift amount is smaller than that at the in-focus state.

In the cases shown in FIGS. 5B1 and 5B2, it is possible to focus by moving the focus lens so as to achieve the shift amount during focusing.
This process is the focusing process according to the "phase difference detection method".
Focusing processing according to this "phase difference detection method" enables setting of the focus lens to the in-focus position, and the focus lens can be set to a position according to the object distance.

The shift amount described with reference to FIG. 5 can be measured for each set of pixels Pa and Pb, which are phase difference detection pixels configured in the image sensor shown in FIG. It is possible to individually calculate the in-focus position (focus point) and the defocus amount for the subject image captured in the pixel combination area).

5(b1) and (b2), there is a deviation from the shift amount at the time of focusing. can be calculated.

[3. Regarding processing using detection information of phase difference detection pixels]
Processing using the detection information of the phase difference detection pixels will be described.

A plurality of examples will be described below as processing using detection information of phase difference detection pixels executed by the imaging apparatus of the present disclosure.
(Example 1) Example of calculating image blend ratio using detection information of phase difference detection pixels (Example 2) Using detection information of phase difference detection pixels to calculate distance information applied to 3D image generation Embodiment (Embodiment 3) An embodiment in which a mask area for background image synthesis is determined using detection information of a phase difference detection pixel, and an image is generated by synthesizing the mask image with an area other than the selected subject area (Embodiment 4) ) Embodiment in which image stabilization amount for each pixel area is calculated using detection information of phase difference detection pixels

[3-1. (Example 1) Example of calculating an image blend ratio using detection information of a phase difference detection pixel]
First, as Example 1, an example in which an image blend ratio is calculated using detection information of a phase difference detection pixel will be described.

FIG. 6 is a block diagram for explaining the configuration of the first embodiment.
FIG. 6 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 6, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, an image information acquisition unit 202, a defocus amount calculation unit 203, an AF control signal generation unit 204, an image blend ratio calculation unit 205, an image signal It has a processing unit 206 and an image blend processing execution unit 221 .

The digital signal processing unit 108 receives the RGB image signals from the preceding A/D conversion unit 105 and the phase difference detection signals (detection information) output from the phase difference detection pixels.

A phase difference information acquisition unit 201 shown in FIG. 6 selects only phase difference detection signals (detection information), which are outputs of phase difference detection pixels, from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 202 shown in FIG. 6 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

The image signal acquired by the image information acquisition unit 202 is input to the image signal processing unit 206 .
The image signal processing unit 206 performs various image signal processing such as demosaicing, white balance adjustment, and gamma correction on the image signal, and inputs the processed image to the image blend processing execution unit 221 .

The phase difference information acquisition unit 201 selects a phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105, and obtains the selected phase difference detection signal (detection information). is output to the defocus amount calculation unit 203 .

The defocus amount calculation unit 202 calculates the defocus amount, that is, the amount of deviation between the in-focus distance and the subject distance (defocus amount (DF)) for each minute image area unit, for example, each pixel area composed of a plurality of pixels. calculate.
As described above, in the phase difference detection method, the defocus of the focus lens is detected based on the shift amount of the signal output according to the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor. Amount is calculated.

The AF control signal generation unit 204 generates an autofocus control signal (AF control signal) for setting the focus lens to an in-focus position (focus position) for a subject designated by the user based on the defocus amount. , and outputs the generated AF control signal to the AF control unit 112a.

The AF control unit 112a drives the focus lens according to an autofocus control signal (AF control signal) input from the AF control signal generation unit 204, and moves the focus lens to the in-focus position (focus position) for the subject designated by the user, for example. set.

Note that the subject to be set at the in-focus position (focus position) is not the entire image area of the captured image, but a subject specified by the user such as a person. Images of other objects, such as the background, are out of focus and appear blurred.

For example, the shift amount described above with reference to FIG. 5 is measured in units of a set of pixels Pa and Pb, which are phase difference detection pixels configured in the imaging device shown in FIG.
That is, the defocus amount calculation unit 203 can calculate the defocus amount of the subject image for each minute pixel area of the captured image.

FIG. 7 shows an example of a pixel region that serves as a defocus amount calculation unit.
FIG. 7 shows the pixel configuration of the imaging element 103 similar to that of FIG. 2 described above.
As described above with reference to FIG. 2, the phase difference detection pixels 151 for acquiring phase difference information are discretely set in some (rows) of RGB pixels having a Bayer array.
A phase difference detection pixel is configured by a pair of a right opening phase difference detection pixel Pa and a left opening phase difference detection pixel Pb.

A pixel region that serves as a unit for calculating the defocus amount can be set, for example, as a pixel region 152 shown in FIG.
The example shown in FIG. 7 is an example in which the pixel region 152, which is the unit for calculating the defocus amount, is a fine pixel region of n×m pixels, such as 6×5 pixels.
A plurality of sets of phase difference detection pixels are included in the n×m fine pixel area. The shift amount described above with reference to FIG. 5 is measured from each of the plurality of sets of phase difference detection pixels.

The defocus amount calculation unit 203 calculates, for example, an average value of a plurality of sets of shift amounts in a pixel region 152 as shown in FIG. 7 as a shift amount of the pixel region 152 of n×m pixels. Further, the defocus amount, that is, the defocus amount corresponding to the amount of deviation between the in-focus distance and the subject distance, is calculated from the shift amount calculated and the shift amount of the in-focus pixel area.
In this manner, the defocus amount calculation unit 203 calculates the defocus amount for each pixel region 152 as shown in FIG. 7, for example.

The image blending ratio calculation unit 205 inputs the fine defocus amount for each pixel area of the captured image calculated by the defocus amount calculation unit 203, and combines the two images based on this defocus amount for each pixel area. Calculate the image blend ratio to apply to the process.

One of the images to be combined is the captured image output by the image signal processing unit 206 . Another image is, for example, an image stored in advance in the storage unit, such as a background image.

The image blend ratio calculation unit 205 calculates the image blend ratio for each pixel area based on the fine defocus amount for each pixel area. That is, the image blending ratio to be applied to the process of synthesizing two images is calculated for each fine pixel area such as n×m pixels, and output to the image blending process execution unit 221 .

The image blending processing execution unit 221 executes blending processing, that is, synthesis processing, of two images according to the image blending ratio for each pixel area input from the image blending ratio calculation unit 205 .

A specific example of image blend processing will be described with reference to FIG.
8, the image blend processing execution unit 221 receives the (A) captured image output from the image signal processing unit 206, and applies a background image stored in advance in the storage unit to the (A) captured image. It is a figure which shows the example of a process in the case of blending and producing|generating a (C) blended image.

An image defining the blend ratio of the (A) photographed image to be blended and the background image is the "(B) blend ratio output image" shown in FIG.
The image blend ratio calculation unit 205 generates a “(B) blend ratio output image” in which pixel values having brightness corresponding to the image blend ratio for each pixel area are set.

The photographed image (A) output from the image signal processing unit 206 is, for example, an image photographed with a “stuffed bear” placed outdoors. This image was taken with the setting set to .
Therefore, the "stuffed bear" region is a focused image, but the background portion is not a focused image but a blurred image.

The user generates a blended image (composite image) by replacing this blurred background image area with the background image stored in advance in the storage unit. This blend image is the "(C) blend image" generated by the image blend processing unit 221 shown in FIG.

The image blend processing unit 221 shown in FIG. 8 applies a fine blending ratio for each pixel area when blending the background image stored in advance in the storage unit with (A) the captured image. Blend processing.

An image in which the blend ratio for each pixel area is defined is the "(B) blend ratio output image" shown in FIG.
This “(B) blend ratio output image” is an image generated by the image blend ratio calculation unit 205 . The image blend ratio calculation unit 205 calculates the image blend ratio for each pixel area based on the defocus amount for each pixel area input from the defocus amount calculation unit 203 .
The “(B) blend ratio output image” shown in FIG. 8 is an image in which the image blend ratio for each pixel region is output as the pixel value of each pixel.

The “(B) blend ratio output image” is a monochrome image with a brightness value (pixel value)=0 to 255, for example.
The black portion (pixel value ≈ 0) is (A) the blend ratio of the captured image = 0%, the blend ratio of the background image = 100%,
The white portion (pixel value ≈ 255) is (A) the blend ratio of the captured image = 100%, the blend ratio of the background image = 0%,
is.
In the gray area, for example, the pixel value=127 area, (A) the blend ratio of the photographed image=50% and the blend ratio of the background image=50%.

In the "(B) blend ratio output image" shown in FIG. 8, the subject at the in-focus position of the (A) photographed image, that is, the "stuffed bear" portion is almost white (pixel value=255), and this portion is , (A) The blending ratio of the captured image is set to 100%, and the blending ratio of the background image is set to 0%.

On the other hand, (A) the subject that is not in focus in the captured image, that is, the background portion other than the "stuffed bear" is almost black (pixel value = 0), and this portion is (A) blend of the captured image. The ratio is set to 0%, and the background image blend ratio is set to 100%.

Furthermore, the boundary between the “stuffed bear” portion and the background portion is set to gray, that is, the “stuffed bear” portion is almost white (pixel value = 255), and the background portion is gradually black (pixel value = 255). 0), the pixel value is set to change gently.

At the boundary between the "stuffed bear" part and the background part, the blending ratio between (A) the photographed image and the background image obtained from the storage unit changes gently.

The image blend ratio calculation unit 205 shown in FIG. 8 generates a "(B) blend ratio output image" having such pixel value output values and outputs it to the image blend processing execution unit 221.

The image blend processing execution unit 221 combines (A) the captured image and the stored A blending ratio with the background image obtained from the storage unit is determined, and (A) the photographed image and the background image obtained from the storage unit are blended pixel by pixel according to the determined blending ratio.
By this blending process, "(C) blended image" shown in FIG. 8 is generated.

The “(C) blended image” generated by the image blend processing execution unit 221 is a blended image obtained by blending (synthesizing) the “stuffed bear” set by the user as the focus position and the background image acquired from the storage unit. Become.
In the boundary area between the "stuffed bear" in the captured image and the background image obtained from the storage unit, the pixel values of each image are blended, and an image that softens the sense of incongruity as if different images were cut and pasted can be created. can.

A specific example of processing for calculating the image blend ratio for each pixel region based on the defocus amount for each pixel region, which is executed by the image blend ratio calculation unit 205, will be described with reference to FIG.

The graph shown in FIG. 9 is a graph in which the horizontal axis indicates the defocus amount for each pixel area, and the vertical axis indicates the shot image blend ratio (0 to 100%) for each pixel area.
Two thresholds Th1 and Th2 are shown on the axis of the defocus amount in units of pixel regions shown on the horizontal axis.
These thresholds are predetermined thresholds in the image blend ratio calculation unit 205 .

The graph shown in FIG. 9 has the following settings.
Range of defocus amount from 0 (in focus) to Th1=blend ratio of captured image=100%,
Range of defocus amount from Th1 to Th2 = blending ratio of captured image = transition from 100% to 0%,
Range of defocus amount from Th2=blend ratio of captured image=0%,
This is the setting.

This can be represented by the following formula.
(a) Defocus amount ≤ Th1 → photographed image blend ratio = 100%
(b) Th1<defocus amount<Th2→captured image blend ratio=((defocus amount−Th1)/(Th2−Th1))×100(%)
(c) Th2 ≤ defocus amount → photographed image blend ratio = 0%

As shown in FIG. 9, the blend ratio of the captured image is set as follows.
The image area of the "stuffed bear" in the captured image set by the user as the focus position has a defocus amount in the range of 0 (focus) to the threshold value Th1, and the blend ratio of the captured image is 100% (background image blending ratio=0%).
In addition, the background area other than the "stuffed bear" in the captured image is an area where the defocus amount is equal to or greater than the threshold value Th2, and the blend ratio of the captured image is 0% (the blend ratio of the background image is 100%). .

In addition, the boundary area between the "stuffed bear" and the background area in the captured image becomes an area of Th1 to Th2 when the defocus amount is between the two threshold values, and the blend ratio of the captured image is set to 100% to 0%. Change. That is, in the boundary area between the "stuffed bear" in the captured image and the background image obtained from the storage unit, the pixel values of each image are blended to produce an image that softens the sense of incongruity as if different images were cut and pasted.

As described above, the first embodiment is an embodiment in which the image blend ratio is calculated using the detection information of the phase difference detection pixels, and more natural blend image generation processing can be performed.

Note that the digital signal processing unit 108 described with reference to FIG. , the image blending ratio calculation unit 205, the image signal processing unit 206, and the image blending processing execution unit 221, but this configuration is an example.

A part of the configuration described with reference to FIG. Alternatively, the data processing may be performed by an external device different from the imaging device 100 .
Specifically, for example, the image blend processing execution unit 221 can be configured outside the digital signal processing unit 108 .

Various different configuration examples of the image blend processing execution unit 221 will be described with reference to FIGS. 10 and 11 .
FIG. 10A has the same configuration as described with reference to FIG. 6, and is an example in which the image blend processing execution unit 221 is the internal configuration of the digital signal processing unit 108. FIG.

On the other hand, FIG. 10B shows an example in which the image blend processing execution unit 221 is configured externally of the digital signal processing unit 108 . For example, the application execution unit 150 is set in the imaging device 100 . The image blend ratio calculation unit 205 of the digital signal processing unit 108 outputs the calculated image blend ratio information for each pixel area to the application execution unit 150 .

The application execution unit 150 uses the image blend ratio information for each pixel area input from the image blend ratio calculation unit 205 of the digital signal processing unit 108 to execute blend processing of multiple images.

Furthermore, as shown in FIG. 11C, an external device other than the imaging device 100, for example, an external device 180 such as a PC may be used.
The image blend ratio calculation unit 205 of the digital signal processing unit 108 of the imaging apparatus 100 outputs the calculated image blend ratio information for each pixel region to the external device 180 such as a PC.

The external device 180 such as a PC uses the image blending ratio information for each pixel area input from the image blending ratio calculating unit 205 of the digital signal processing unit 108 to execute blending processing of a plurality of images.
In this way, it is possible to perform processing using various configurations.

[3-2. (Example 2) Example of calculating distance information applied to three-dimensional image generation using detection information of phase difference detection pixels]
Next, as Example 2, an example in which distance information to be applied to three-dimensional image generation is calculated using detection information of phase difference detection pixels will be described.

FIG. 12 is a block diagram for explaining the configuration of the second embodiment.
FIG. 12 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 12, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, an image information acquisition unit 202, a defocus amount calculation unit 203, an AF control signal generation unit 204, an image signal processing unit 206, a distance information calculation unit. It has a unit 207 and a three-dimensional image generation unit 222 .

A phase difference information acquisition unit 201 shown in FIG. 12 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 202 shown in FIG. 12 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

The image signal acquired by the image information acquisition unit 202 is input to the image signal processing unit 206 .
The image signal processing unit 206 performs various image signal processing such as demosaic processing, white balance adjustment, and gamma correction on the image signal, and inputs the processed image to the three-dimensional image generation unit 222 .

The defocus amount calculation unit 202 calculates a defocus amount, that is, a defocus amount (DF) between the in-focus distance and the object distance, for each predetermined pixel area.
As described above, in the phase difference detection method, the defocus of the focus lens is detected based on the shift amount of the signal output according to the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor. Amount is calculated.

For example, as described above with reference to FIG. 7, the defocus amount calculation unit 203 calculates the defocus amount, that is, the amount of deviation between the in-focus distance and the subject distance, for each minute pixel area such as n×m pixels. A defocus amount equivalent to is calculated.

The distance information calculation unit 207 receives the fine defocus amount for each pixel area of the captured image calculated by the defocus amount calculation unit 203, and based on this defocus amount for each pixel area, determines the distance information for each pixel area of the captured image. Calculate the distance information of
The distance information calculation unit 207 generates, for example, a depth map indicating the distance value for each image area by pixel values (eg, 0 to 255).

A specific example of distance information calculation processing in the distance information calculation unit 207 will be described later.
The distance information calculation unit 207 outputs the calculated distance information for each pixel area to the three-dimensional image generation unit 222 .

The three-dimensional image generation unit 222 receives the captured image from the image signal processing unit 206 and further receives distance information for each pixel area from the distance information calculation unit 207 .
The 3D image generation unit 222 generates a 3D image corresponding to the captured image according to the distance information for each pixel area input from the distance information calculation unit 207 .

A specific example of the three-dimensional image generation processing according to the second embodiment will be described with reference to FIG.
13, the three-dimensional image generation unit 222 generates (A) the captured image output from the image signal processing unit 206 and (B) distance information (depth map) for each pixel region generated by the distance information calculation unit 207. FIG. 10 is a diagram showing an example of processing when inputting and using (B) distance information (depth map) to generate (C) a three-dimensional image corresponding to (A) a photographed image;

The photographed image (A) output from the image signal processing unit 206 is, for example, an image photographed with a “stuffed bear” placed outdoors. This image was taken with the setting set to .

The (C) subject distance information used by the three-dimensional image generation unit 222 to generate a three-dimensional image is (B) distance information (depth map) for each pixel region generated by the distance information calculation unit 207 .

The (B) distance information (depth map) for each pixel area generated by the distance information calculation unit 207 is based on the defocus amount for each minute pixel area such as n×m pixels calculated by the defocus amount calculation unit 203. 4 is a depth map showing the value of the object distance for each image area calculated in the above manner as a pixel value (0 to 255, for example).
A high luminance (high pixel value) area that looks white is an area where the object distance is short. A low luminance (low pixel value) area that looks black is an area with a long subject distance.

Note that the process of calculating the subject distance from the defocus amount can be calculated by applying parameters such as the focal length of the lens (focus lens) of the imaging device.
Specifically, the object distance for each pixel area is calculated according to the following (Equation 1).

... (Formula 1)

The distance information calculation unit 207 calculates the object distance for each pixel area according to the above (Equation 1).
The distance information calculation unit 207 outputs the calculated subject distance information for each pixel area to the three-dimensional image generation unit 222 .

The three-dimensional image generation unit 222 generates three-dimensional image data corresponding to (A) the photographed image input from the image signal processing unit 206 using the subject distance information for each pixel area input from the distance information calculation unit 207. do. FIG. 13C is a three-dimensional image shown in FIG.

The “(C) three-dimensional image” generated by the three-dimensional image generation unit 222 is a three-dimensional image that reflects subject distance information in units of minute areas generated by the distance information calculation unit 207, for example, minute pixel areas of n×m pixels. It becomes an image, and becomes a highly accurate three-dimensional image reflecting detailed distance information.

As described above, the second embodiment is an embodiment in which the subject distance information for each minute pixel area is calculated using the detection information of the phase difference detection pixels, and the three-dimensional image is generated using the calculated subject distance information. High-precision three-dimensional image generation processing can be performed.

Note that the digital signal processing unit 108 described with reference to FIG. , the image signal processing unit 206, the distance information calculation unit 207, and the three-dimensional image generation unit 222, but this configuration is an example.

A part of the configuration described with reference to FIG. Alternatively, the data processing may be performed by an external device different from the imaging device 100 .
Specifically, for example, the three-dimensional image generation unit 222 can be configured outside the digital signal processing unit 108 .

For example, similar to the configuration described above with reference to FIGS. can be set in the application execution unit 150 configured as follows.
Furthermore, as in the configuration described with reference to FIG. 11C, a configuration in which the three-dimensional image generation unit 222 is provided in an external device other than the imaging device 100, for example, an external device 180 such as a PC may be employed.
In this case, the distance information calculation unit 207 of the digital signal processing unit 108 of the imaging apparatus 100 outputs the calculated distance information for each pixel area to the external device 180 such as a PC.

The external device 180 such as a PC uses the distance information for each pixel area input from the distance information calculation unit 207 of the digital signal processing unit 108 to execute a three-dimensional image generation process.
In this way, it is possible to perform processing using various configurations.

[3-3. (Example 3) Example of determining a mask region for background image synthesis using detection information of a phase difference detection pixel, and generating an image by synthesizing the mask image with a region other than the selected subject region]
Next, as Example 3, an example will be described in which a mask region for background image synthesis is determined using detection information of a phase difference detection pixel, and an image is generated by synthesizing the mask image with a region other than the selected subject region. .

For example, mask processing is performed to set the background region other than a specific subject selected from the photographed image to a uniform color such as green background or blue background, and the green background region or blue background region of the image subjected to such mask processing is synthesized with another image, for example, a new background image to generate a synthesized image in which the selected specific subject is displayed on the new background image.

FIG. 14 is a block diagram for explaining the configuration of the third embodiment.
FIG. 14 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 14, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, an image information acquisition unit 202, a defocus amount calculation unit 203, an AF control signal generation unit 204, an image signal processing unit 206, a mask area determination unit It has a section 208 , an output subject color analyzing section 209 , a mask output color determining section 210 , a mask synthesizing section 211 and an image synthesizing section 223 .

A phase difference information acquisition unit 201 shown in FIG. 14 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 202 shown in FIG. 14 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

The output subject color analysis unit 209 analyzes the color of the subject to be displayed on the finally generated synthetic image.
The subject to be displayed on the finally generated synthesized image is specified by the user through the input unit 118, for example. That is, output subject selection information 232 shown in the drawing is input to the output subject color analysis section 209 via the input section 118, and the output subject color analysis section 209 analyzes the color of the selected output subject according to this input information.
A color analysis of this output subject is performed to determine the color of the mask area.

For example, in the case of using a green screen, if a green pixel area is included as the color of the output object, the background image to be combined is also output in that area. Therefore, the green screen cannot be used when the output subject includes green.
Similarly, when using a blue background, if a blue pixel area is included in the color of the output subject, the background image to be synthesized is also output in that area. Therefore, the blue background cannot be used when the output object contains blue.

Thus, it is necessary to set the color of the mask area to a color that is not included in the output object.
The output subject color analysis unit 209 analyzes the color of the output subject in order to set the color of the mask area to a color different from the color of the output subject displayed on the composite image.

The mask output color determination unit 210 determines a color that is not included in the output subject analyzed by the output subject color analysis unit 209 as the color of the mask area.

The mask area determination unit 208 receives the output subject selection information 232 from the input unit 118, and further inputs the defocus amount for each fine pixel area of the captured image calculated by the defocus amount calculation unit 203.

The mask area determination unit 208 determines the mask area based on the output subject selection information 232 input from the input unit 118 and the defocus amount for each pixel area input from the defocus amount calculation unit 203 . That is, the mask area to be set to green screen or blue screen is determined.

The mask area determining unit 208 sets an area not included in the output subject specified by the output subject selection information 232 input from the input unit 118 as a mask area. In this mask area determination process, the defocus amount for each pixel area input from the defocus amount calculation unit 203 is used as auxiliary information.

For example, a pixel region having a defocus amount different from the defocus amount of the output subject specified by the output subject selection information 232 input from the input unit 118 by a predetermined threshold value or more is determined as the mask area.
The mask area information determined by the mask area determining unit 208 is output to the mask synthesizing unit 211 .

The mask synthesizing unit 211 receives the captured image from the image signal processing unit 206 , the color information of the mask area from the mask output color determination unit 210 , and the mask area information from the mask area determination unit 208 .

The mask synthesizing unit 211 first determines a mask setting area for the captured image input from the image signal processing unit 206 . A mask setting area is determined according to the mask area information input from the mask area determination unit 208 .

The mask synthesizing unit 211 next sets a color mask in the determined mask setting area according to the mask area color information input from the mask output color determining unit 210 .
For example, a mask such as green screen or blue screen is set.

The mask setting image generated by the mask synthesizing unit 211 is an image in which areas other than the output subject specified by the output subject selection information 232 input from the input unit 118 are replaced with a mask image such as a green screen or blue screen.
The mask setting image generated by the mask synthesizing section 211 is output to the image synthesizing section 223 .

The image synthesizing unit 223 generates a synthesized image by pasting another image, for example, a background image, input from, for example, the storage unit or the outside onto the mask area of the mask setting image generated by the mask synthesizing unit 211 .

A specific example of composite image generation processing in the third embodiment will be described with reference to FIG. 15 .
15, the image synthesizing unit 223 inputs (C) the mask setting image from the mask synthesizing unit 211, and (C) stores another image input from the storage unit or the outside in the mask area of the mask setting image. For example, it is a figure which shows the example of a process in the case of producing|generating the (D) composite image which pasted the background image.

In FIG. 15 , the user (A) displays a captured image on the display section, selects an output subject that is not to be masked, and inputs output subject selection information 232 via the input section 118 .
The example shown in FIG. 15 is an example in which the user selects (A) the area of “stuffed bear” in the captured image as an output subject that is not to be masked.
The output subject selection information 232 input via the input unit 118 is input to the output subject color analysis unit 209 and mask area determination unit 208 .

The output subject color analysis unit 209 analyzes the color of the output subject in order to set the color of the mask area to a color different from the color of the output subject displayed on the composite image.

As described above, the mask area determination unit 208 sets, as a mask area, an area that is not included in the output subject specified by the output subject selection information 232 input from the input unit 118. During this mask area determination process, the defocus The defocus amount for each pixel area input from the amount calculation unit 203 is used as auxiliary information.

For example, a pixel area having a defocus amount having a difference equal to or greater than a predetermined threshold value with respect to the defocus amount of the output subject specified by the output subject selection information 232 input from the input unit 118 is determined as the mask area. do. For example, a “(B) mask area designating image” as shown in FIG. 15 is generated and output to the mask synthesis unit 211 .

The (B) mask area designation image shown in FIG. 15 is an example in which a pixel area other than the "stuffed bear" area in the (A) photographed image selected as the output subject not to be masked by the user is determined as the mask area.
(A) The defocus amount of the "stuffed bear" area in the photographed image is almost 0, that is, the in-focus area, and (A) the background area other than the "stuffed bear" in the photographed image is out of focus. This is an area where the defocus amount is large.

That is, the background area is a predetermined threshold value for the defocus amount (approximately 0) of the output subject ("stuffed bear" area) specified by the output subject selection information 232 input from the input unit 118. It is a pixel region having a defocus amount with the above difference. The mask area determining unit 208 determines such an area as a mask area, and generates an image in which the mask area is identifiable, for example, "(B) mask area designation image" in which the mask area is set to black, gray, or a specific color. ” is generated and output to the mask synthesizing unit 211 .

The mask synthesizing unit 211 determines the mask setting area according to the mask area information input from the mask area determining unit 208, for example, the (B) mask area designating image as shown in FIG.

The mask synthesizing unit 211 next sets a color mask in the determined mask setting area according to the mask area color information input from the mask output color determining unit 210 .
For example, a mask such as a green screen or a blue screen is set to generate a mask setting image (C) shown in FIG.

The (C) mask setting image generated by the mask synthesizing unit 211 is an image in which the area other than the output subject specified by the output subject selection information 232 input from the input unit 118 is replaced with a mask image such as a green screen or a blue screen. becomes.
The (C) mask setting image generated by the mask synthesizing unit 211 is output to the image synthesizing unit 223 .

The image synthesizing unit 223 pastes another image input from, for example, a storage unit or the outside, such as a background image, to the mask area of the (C) mask setting image generated by the mask synthesizing unit 211, as shown in FIG. ) to generate a composite image.

As described above, in the third embodiment, the subject distance information for each minute pixel area is calculated using the detection information of the phase difference detection pixels, the calculated subject distance information is used to determine the mask area, and the determined mask area is determined. In this embodiment, a composite image is generated by pasting a background image or the like to the mask area. It is possible to perform the mask area determination processing in units of fine pixel regions, and to perform highly accurate composite image generation processing.

Note that the digital signal processing unit 108 described with reference to FIG. , an image signal processing unit 206, a mask area determination unit 208, an output object color analysis unit 209, a mask output color determination unit 210, a mask synthesis unit 211, and an image synthesis unit 223. The configuration is an example.

A part of the configuration described with reference to FIG. Alternatively, the data processing may be performed by an external device different from the imaging device 100 .

Specifically, for example, as shown in FIG. 16, the image synthesizing process of the image synthesizing unit is configured to be executed using a keyer device 240, which is a dedicated processing device for synthesizing images for a green screen or a blue screen. good too.

In this case, the mask synthesizing unit 211 of the digital signal processing unit 108 of the imaging device 100 outputs the generated mask synthesized image to the external keyer device 240 .

A keying processing unit (image synthesizing unit) 241 in the keyer device 240 inserts another image, for example, input from a storage unit, into the mask area of the mask synthesized image input from the mask synthesizing unit 211 of the digital signal processing unit 108 . Generates a composite image pasted with a background image.

In addition, for example, similar to the configuration described above with reference to FIGS. 10 and 11, the image combining unit 223 of the digital signal processing unit 108 in FIG. In addition, it is possible to configure the setting in the application execution unit 150 configured in the imaging device 100 .

Further, as in the configuration described with reference to FIG. 11C, an external device other than the imaging device 100, for example, an external device 180 such as a PC may be provided with the image synthesizing unit 223. FIG.
In this case, the mask synthesizing unit 211 of the digital signal processing unit 108 of the imaging device 100 outputs the generated mask synthesized image to the external device 180 such as a PC.

The external device 180 such as a PC creates a composite image obtained by pasting another image input from, for example, a storage unit or the outside, such as a background image, to the mask area of the mask composite image input from the mask synthesis unit 211 of the digital signal processing unit 108 . Generate.
In this way, it is possible to perform processing using various configurations.

[3-4. (Embodiment 4) Embodiment of Calculating Camera Shake Correction Amount for Each Pixel Area Using Detection Information of Phase Difference Detection Pixel]
Next, as Example 4, an example in which the detection information of the phase difference detection pixels is used to calculate the camera shake correction amount for each pixel region will be described.

FIG. 17 is a block diagram for explaining the configuration of the fourth embodiment.
FIG. 17 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 17, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, an image information acquisition unit 202, a defocus amount calculation unit 203, an AF control signal generation unit 204, an image signal processing unit 206, a distance information calculation unit. A camera shake amount calculator 262 , a camera shake correction amount calculator 263 , and a camera shake correction execution unit 264 .

A phase difference information acquisition unit 201 shown in FIG. 17 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 202 shown in FIG. 17 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

The distance information calculation unit 251 inputs the fine defocus amount of each pixel area of the captured image calculated by the defocus amount calculation unit 203, and based on this defocus amount of each pixel area, calculates the pixel area of the captured image. Calculate the distance information of
The distance information calculation unit 251 generates, for example, a depth map indicating the distance value for each image area by pixel values (eg, 0 to 255).

The distance information calculation processing in the distance information calculation unit 251 is the same as the distance information calculation processing executed by the distance information calculation unit 207 of the second embodiment described above with reference to FIGS. Using the defocus amount for each pixel area according to (Equation 1), the subject distance for each pixel area is calculated.
The distance information calculation unit 251 outputs the calculated distance information for each pixel area to the camera shake correction amount calculation unit 263 .

The shake amount calculation unit 262 receives detection information such as the tilt, angle, and tilt speed (angular velocity) of the imaging device 100 from the gyro 131, calculates the shake amount at the time of image capturing, and uses the calculated shake amount as the shake correction amount. Output to the calculation unit 263 .

The camera shake correction amount calculation unit 263 receives the amount of camera shake during image shooting from the camera shake amount calculation unit 262 and also receives the distance information for each pixel area from the distance information calculation unit 251 .

The camera shake correction amount calculation unit 263 uses these two pieces of input information to calculate the camera shake correction amount according to the subject distance of the captured image.
A specific example of calculation processing of the camera shake correction amount according to the subject distance by the camera shake correction amount calculation unit 263 will be described with reference to FIG. 18 .
FIG. 18 shows (A) a captured image captured by the imaging device 100 and (B) distance information (depth map) generated by the distance information calculation unit 251 based on the (A) captured image. .

The (B) distance information (depth map) for each pixel area generated by the distance information calculation unit 251 is based on the defocus amount for each minute pixel area such as n×m pixels calculated by the defocus amount calculation unit 203. 4 is a depth map showing the value of the object distance for each image area calculated in the above manner as a pixel value (0 to 255, for example).
A high luminance (high pixel value) area that looks white is an area where the object distance is short. A low luminance (low pixel value) area that looks black is an area with a long subject distance.

The camera shake correction amount calculation unit 263 calculates the subject distance of the captured image by using the camera shake amount during image shooting input from the camera shake amount calculation unit 262 and (B) the distance information (depth map) generated by the distance information calculation unit 251 . determines the amount of correction according to .

As shown in FIG. 18, the shake correction amount calculator 263 determines the correction amount as follows according to the subject distance of the captured image.
A small correction amount is set for a pixel area with a long object distance.
A medium correction amount is set for a pixel area with a medium object distance.
A large correction amount is set for a pixel area with a short subject distance.
This is because the closer the object is to the object, the greater the shake of the image caused by camera shake.

The camera shake correction amount calculation unit 263 thus uses the camera shake amount during image shooting input from the camera shake amount calculation unit 262 and (B) the distance information (depth map) generated by the distance information calculation unit 251 to calculate the pixel A correction amount (shake correction amount) for each pixel area is calculated according to the distance information for each area.
The camera shake correction amount for each pixel region calculated by the camera shake correction amount calculator 263 is input to the camera shake correction execution unit 264 .

The camera shake correction execution unit 264 receives the camera shake correction amount for each pixel region calculated by the camera shake correction amount calculation unit 263, and executes camera shake correction processing on the captured image.
The camera shake correction execution unit 264 performs correction processing using a correction amount corresponding to the object distance in pixel area units according to the camera shake correction amount in pixel area units calculated by the camera shake correction amount calculation unit 263 .

A specific example of camera shake correction processing according to the fourth embodiment will be described with reference to FIG.
FIG. 19 shows an image of a subject for each pixel area, according to (A) the captured image output from the image signal processing unit 206 and the camera shake correction amount for each pixel area calculated by the camera shake correction amount calculation unit 263 . It is a figure which shows the example of a process in the case of performing the correction process which applied the correction amount according to distance.

The subject distance information used by the camera shake correction execution unit 264 to perform camera shake correction processing is (B) distance information (depth map) for each pixel region generated by the distance information calculation unit 251 .

Note that the process of calculating the subject distance from the defocus amount is the same process as the distance information calculation process executed by the distance information calculation unit 207 of the second embodiment described above with reference to FIGS. The object distance for each pixel area is calculated using the defocus amount for each pixel area according to Equation 1 described above.
The distance information calculation unit 251 outputs the calculated distance information for each pixel area to the camera shake correction amount calculation unit 263 .

The camera shake correction amount calculation unit 263 uses these two pieces of input information to calculate the camera shake correction amount according to the subject distance of the captured image.
As described above with reference to FIG. 18, the shake correction amount calculator 263 determines the correction amount as follows according to the subject distance of the captured image.
A small correction amount is set for a pixel area with a long object distance.
A medium correction amount is set for a pixel area with a medium object distance.
A large correction amount is set for a pixel area with a short subject distance.
This is because the closer the object is to the object, the greater the shake of the image caused by camera shake.

As described above, in the fourth embodiment, the detection information of the phase difference detection pixels is used to calculate the subject distance information for each minute pixel area, and the calculated subject distance information is used to determine the correction amount of camera shake correction. That is, the correction amount is reduced for a pixel area with a longer subject distance, and the correction amount is increased for a pixel area with a shorter subject distance.

In this way, by performing camera shake correction by adjusting the correction amount according to the subject distance, it is possible to perform optimum correction processing and generate a high-quality image with optimum camera shake correction.

Note that the digital signal processing unit 108 described with reference to FIG. , image signal processing unit 206, distance information calculation unit 251, camera shake amount calculation unit 262, camera shake correction amount calculation unit 263, and camera shake correction execution unit 264, but this configuration is an example. .

A part of the configuration described with reference to FIG. Alternatively, the data processing may be performed by an external device different from the imaging device 100 .
Specifically, for example, the shake correction execution unit 264 can be configured outside the digital signal processing unit 108 .

For example, similar to the configuration described above with reference to FIGS. A configuration is possible in which an image correction unit other than the signal processing unit 108 executes the processing.

Furthermore, as in the configuration described with reference to FIG. 11C, an external device other than the imaging device 100, for example, an external device 180 such as a PC may be provided with an image correction unit.
In this case, the camera shake correction amount calculation unit 263 of the digital signal processing unit 108 of the imaging apparatus 100 outputs the calculated camera shake correction amount for each pixel area to the external device 180 such as a PC.

The external device 180 such as a PC performs camera shake correction processing corresponding to the object distance in pixel area units using the camera shake correction amount in pixel area units calculated by the camera shake correction amount calculation unit 263 of the digital signal processing unit 108 .
In this way, it is possible to perform processing using various configurations.

[4. Other Examples]
Next, another embodiment will be described.

So far, the following four embodiments have been described with reference to FIGS.
(Example 1) Example of calculating image blend ratio using detection information of phase difference detection pixels (Example 2) Using detection information of phase difference detection pixels to calculate distance information applied to 3D image generation Embodiment (Embodiment 3) An embodiment in which a mask area for background image synthesis is determined using detection information of a phase difference detection pixel, and an image is generated by synthesizing the mask image with an area other than the selected subject area (Embodiment 4) ) Embodiment in which image stabilization amount for each pixel area is calculated using detection information of phase difference detection pixels

Each of these four embodiments can be configured independently, but can also be configured as a device or system having a combination configuration of any of a plurality of embodiments.
In addition, the purpose is determined from the object, detection information, preset settings, etc., and one or more examples are selected to be used, or one or more examples are proposed to the user. You may make it

For example, FIG. 20 is a diagram showing the configuration of the digital signal processing unit 108 of the imaging device 100 capable of executing all the processes of the above four embodiments.

FIG. 20 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging device 100 described with reference to FIG.

As shown in FIG. 20, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, an image information acquisition unit 202, a defocus amount calculation unit 203, an AF control signal generation unit 204, an image blend ratio calculation unit 205, an image signal Processing unit 206, distance information calculation unit 207, mask area determination unit 208, output subject color analysis unit 209, mask output color determination unit 210, mask synthesis unit 211, image blend processing execution unit 221, 3D image generation unit 222, image It has a synthesizing unit 223 , a camera shake amount calculator 262 , a camera shake correction amount calculator 263 , and a camera shake correction execution unit 264 .

The digital signal processing unit 108 shown in FIG. 20 becomes a digital signal processing unit capable of executing all the processes of the four embodiments described above.
For example, an imaging device having such a digital signal processing unit 108 can also be configured.

Note that some of the constituent elements of the digital signal processing unit 108 shown in FIG. 20 can be used as constituent elements of the imaging apparatus outside the digital signal processing unit 108.
Moreover, it is also possible to provide a part of the constituent elements of the digital signal processing unit 108 shown in FIG.

[5. Hardware configuration example of external device such as PC connected to imaging device]
Next, a hardware configuration example of an external device such as a PC connected to the imaging device will be described.

As described above, part of the processing of the digital signal processing section described in each embodiment can be executed in an external image processing device such as a PC.
For example, as described above with reference to FIG. 11, the external device 180 such as a PC inputs data calculated by the imaging device 100 and executes image processing based on the input data.

A hardware configuration example of an image processing apparatus such as a PC that inputs calculated data of the imaging apparatus 100 and executes image processing based on the input data will be described with reference to FIG.

A CPU (Central Processing Unit) 301 functions as a control section and a data processing section that execute various processes according to programs stored in a ROM (Read Only Memory) 302 or a storage section 308 . For example, the processing described in the above embodiment is executed. A RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301 . These CPU 301 , ROM 302 and RAM 303 are interconnected by a bus 304 .

The CPU 301 is connected to an input/output interface 305 via a bus 304. The input/output interface 305 is connected to an input unit 306 including a camera, various switches, microphones, sensors, etc., and an output unit 307 including a display, speakers, etc. there is
The CPU 301 executes various types of processing in response to commands input from the input unit 306 and outputs processing results to the output unit 307, for example.

A storage unit 308 connected to the input/output interface 305 is composed of, for example, a flash memory, and stores programs executed by the CPU 301 and various data. A communication unit 309 includes a communication unit for Wi-Fi communication, Bluetooth (registered trademark) (BT) communication, and other data communication via a network such as the Internet or a local area network, and communicates with an external device.

A drive 310 connected to the input/output interface 305 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card to record or read data.

[6. Summary of the configuration of the present disclosure]
Embodiments of the present disclosure have been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of this disclosure. That is, the present invention has been disclosed in the form of examples and should not be construed as limiting. In order to determine the gist of the present disclosure, the scope of claims should be considered.

In addition, the technique disclosed in this specification can take the following configurations.
(1) a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an image sensor of an imaging device;
an image blend ratio calculation unit that calculates an image blend ratio between the captured image and another image;
The image blend ratio calculation unit
An imaging device for calculating an image blend ratio for each pixel area based on the defocus amount for each pixel area calculated by the defocus amount calculation unit.

(2) The image blend ratio calculation unit
The imaging apparatus according to (1), which generates a blend ratio output image in which a pixel value having a luminance corresponding to the image blend ratio for each pixel area is set.

(3) The image blend ratio calculation unit
setting a blend ratio of the captured image to a large ratio for a pixel region having a small defocus amount of the captured image;
The image capturing apparatus according to (1) or (2), wherein a blend ratio of the captured image is set to a small ratio for a pixel region having a large defocus amount of the captured image.

(4) The image blend ratio calculation unit
setting the blending ratio of the captured image to 100% for pixel regions in which the defocus amount of the captured image is equal to or less than a prescribed threshold value Th1;
setting a blending ratio of the captured image to 0% for a pixel region in which the defocus amount of the captured image is equal to or greater than a prescribed threshold value Th2;
For pixel regions where the defocus amount of the captured image is greater than the specified threshold value Th1 and less than the specified threshold value Th2, the blend ratio of the captured image is set to change according to the defocus amount of the captured image. (1) to (3).

(5) The imaging device further includes:
Any one of (1) to (4) having an image blend processing execution unit that applies the image blend ratio for each pixel area calculated by the image blend ratio calculation unit and executes image blend processing of the photographed image and another image. 1. The imaging device according to 1.

(6) a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an image sensor of an imaging device;
An imaging apparatus comprising a distance information calculation unit that calculates a subject distance for each pixel area based on the defocus amount for each pixel area calculated by the defocus amount calculation unit.

(7) The distance information calculation unit
The imaging apparatus according to (6), which generates a distance image in which a pixel value having a brightness corresponding to the subject distance in pixel area units is set.

(8) The distance information calculation unit
(6) or ( 7) The imaging device described in 7).

(9) The imaging device further includes
The method according to any one of (6) to (8), further comprising a three-dimensional image generation unit that generates a three-dimensional image based on the captured image by applying the subject distance information for each pixel area calculated by the distance information calculation unit. Imaging device.

(10) The imaging device further includes:
any one of (6) to (9), further comprising a camera shake correction amount calculation unit that applies the subject distance information for each pixel area calculated by the distance information calculation unit to calculate a camera shake correction amount for each pixel area for the captured image; The imaging device according to .

(11) The camera shake correction amount calculation unit
The image pickup apparatus according to (10), wherein a larger shake correction amount is set for a pixel area with a shorter subject distance, and a smaller shake correction amount is set for a pixel area with a longer subject distance.

(12) The imaging device further includes
a camera shake correction execution unit that executes camera shake correction processing on the captured image;
The camera shake correction execution unit
The image pickup apparatus according to (10) or (11), wherein image stabilization processing is performed in units of pixel areas according to the amount of image stabilization in units of pixel areas calculated by the image stabilization amount calculation unit.

(13) a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an image sensor of an imaging device;
An imaging apparatus comprising a mask area determination unit that determines a mask area for the captured image based on the defocus amount for each pixel area calculated by the defocus amount calculation unit.

(14) The mask area determination unit
The imaging apparatus according to (13), wherein a pixel region having a defocus amount having a difference equal to or greater than a predetermined threshold value with respect to the defocus amount of the output subject specified by the user is determined as the mask area.

(15) The mask area determination unit
The imaging apparatus according to (13) or (14), which generates a mask area indication image in which the mask area is distinguishable from the area of the output object specified by the user.

(16) The imaging device further includes:
The imaging apparatus according to any one of (13) to (15), further comprising a mask synthesizing unit that synthesizes a mask image of a predetermined color with the mask area determined by the mask area determining unit to generate a mask setting image.

(17) The imaging device further includes
The imaging apparatus according to (16), further comprising an image synthesizing unit that superimposes another image on the mask area of the mask setting image generated by the mask synthesizing unit to generate a synthesized image with the output subject specified by the user.

(18) The mask synthesizing unit
The imaging apparatus according to (16) or (17), which outputs the generated mask setting image to an external device that executes composite image generation processing.

(19) An image processing method executed in an imaging device,
A defocus amount calculation step in which the defocus amount calculation unit calculates a defocus amount for each pixel area of a captured image output from an imaging device of an imaging device;
A defocus amount calculation unit executes an image blend ratio calculation step of calculating an image blend ratio between the captured image and another image,
The image blend ratio calculation step includes:
An image processing method, comprising: calculating an image blend ratio for each pixel area based on the defocus amount for each pixel area calculated in the defocus amount calculation step.

(20) A program for executing image processing in an imaging device,
a defocus amount calculation step of causing a defocus amount calculation unit to calculate a defocus amount for each pixel area of a captured image output from an imaging element of an imaging device;
causing a defocus amount calculation unit to perform an image blend ratio calculation step of calculating an image blend ratio between the captured image and another image;
In the image blend ratio calculation step,
A program for calculating an image blend ratio for each pixel area based on the defocus amount for each pixel area calculated in the defocus amount calculation step.

It should be noted that the series of processes described in the specification can be executed by hardware, software, or a composite configuration of both. When executing processing by software, a program recording the processing sequence is installed in the memory of a computer built into dedicated hardware and executed, or the program is loaded into a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be pre-recorded on a recording medium. In addition to being installed in a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet and installed in a recording medium such as an internal hard disk.

In addition, the various types of processing described in the specification may not only be executed in chronological order according to the description, but may also be executed in parallel or individually according to the processing capacity of the device that executes the processing or as necessary. Further, in this specification, a system is a logical collective configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same housing.

As described above, according to the configuration of the embodiment of the present disclosure, the image blend ratio and the subject distance for each pixel region are calculated based on the defocus amount for each pixel region of the captured image, and the calculated data is used to An apparatus and method for executing various image processing are realized.
Specifically, for example, a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an imaging device of an imaging device, and a defocus amount for each pixel area calculated by the defocus amount calculation unit. Calculating the image blend ratio for each pixel area using the defocus amount for each pixel area, and calculating the subject distance for each pixel area based on the defocus amount for each pixel area , and image stabilization amount calculation processing for each pixel region. Alternatively, various processes are executed such as determining a mask area for the captured image based on the defocus amount for each pixel area and performing processing for generating a green screen image.
With this configuration, it is possible to realize an apparatus and method for calculating the image blend ratio and the subject distance for each pixel area based on the defocus amount for each pixel area of the captured image, and performing various image processing using the calculated data. .

REFERENCE SIGNS LIST 100 imaging device 101 focus lens 102 zoom lens 103 imaging element 104 analog signal processing section 105 A/D conversion section 106 timing generator (TG)
107 vertical driver 108 digital signal processor (DSP)
110 control unit 112a AF control unit 112b zoom control unit 113 motor 115 recording device 116 viewfinder 117 monitor 118 input unit (operation unit)
122 pixel region 131 gyro 151 phase difference detection pixel 152 pixel region 201 phase difference information acquisition unit 202 image information acquisition unit 203 defocus amount calculation unit 204 AF control signal generation unit 205 image blend ratio calculation unit 206 image signal processing unit 207 distance information Calculation unit 208 Mask area determination unit 209 Output subject color analysis unit 210 Mask output color determination unit 211 Mask synthesis unit 221 Image blend processing execution unit 222 Three-dimensional image generation unit 223 Image synthesis unit 251 Distance information calculation unit 262 Shake amount calculation unit 263 Camera shake correction amount calculator 264 Camera shake correction execution unit 301 CPU
302 ROMs
303 RAM
304 bus 305 input/output interface 306 input unit 307 output unit 308 storage unit 309 communication unit 310 drive 311 removable media

Claims

a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an image sensor of an imaging device;
an image blend ratio calculation unit that calculates an image blend ratio between the captured image and another image;
The image blend ratio calculation unit
An imaging device for calculating an image blend ratio for each pixel area based on the defocus amount for each pixel area calculated by the defocus amount calculation unit.
The image blend ratio calculation unit
2. The imaging apparatus according to claim 1, wherein a blend ratio output image is generated in which a pixel value having a luminance corresponding to the image blend ratio for each pixel area is set.
The image blend ratio calculation unit
setting a blend ratio of the captured image to a large ratio for a pixel region having a small defocus amount of the captured image;
2. The imaging apparatus according to claim 1, wherein the blending ratio of the photographed image is set to a small ratio for a pixel region having a large defocus amount of the photographed image.
The image blend ratio calculation unit
setting the blending ratio of the captured image to 100% for pixel regions in which the defocus amount of the captured image is equal to or less than a prescribed threshold value Th1;
setting a blending ratio of the captured image to 0% for a pixel region in which the defocus amount of the captured image is equal to or greater than a prescribed threshold value Th2;
For pixel regions where the defocus amount of the captured image is greater than the specified threshold value Th1 and less than the specified threshold value Th2, the blend ratio of the captured image is set to change according to the defocus amount of the captured image. The imaging device according to claim 1 .
The imaging device further comprises
2. The imaging apparatus according to claim 1, further comprising an image blend processing execution unit that applies the image blend ratio for each pixel region calculated by the image blend ratio calculation unit and executes image blend processing between the captured image and another image. .
a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an image sensor of an imaging device;
An imaging apparatus comprising a distance information calculation unit that calculates a subject distance for each pixel area based on the defocus amount for each pixel area calculated by the defocus amount calculation unit.
The distance information calculation unit
7. The imaging apparatus according to claim 6, which generates a distance image in which pixel values having luminance corresponding to the object distance in pixel area units are set.
The distance information calculation unit
7. The object distance according to claim 6, wherein the subject distance is calculated for each pixel area by a calculation formula using the defocus amount for each pixel area calculated by the defocus amount calculation unit and the focal length information of the lens of the imaging device. imaging device.
The imaging device further comprises
7. The imaging apparatus according to claim 6, further comprising a three-dimensional image generation unit that generates a three-dimensional image based on the captured image by applying the subject distance information for each pixel area calculated by the distance information calculation unit.
The imaging device further comprises
7. The imaging apparatus according to claim 6, further comprising a camera shake correction amount calculation unit that calculates a camera shake correction amount for each pixel area of the captured image by applying the object distance information for each pixel area calculated by the distance information calculation unit.
The camera shake correction amount calculation unit
11. The image pickup apparatus according to claim 10, wherein a larger image stabilization amount is set for a pixel area with a shorter subject distance, and a smaller image stabilization amount is set for a pixel area with a longer subject distance.
The imaging device further comprises
a camera shake correction execution unit that executes camera shake correction processing on the captured image;
The camera shake correction execution unit
11. The imaging apparatus according to claim 10, wherein image stabilization processing is performed in units of pixel areas according to the amount of image stabilization in units of pixel areas calculated by the image stabilization amount calculation unit.
a defocus amount calculation unit that calculates a defocus amount for each pixel area of a captured image output from an image sensor of an imaging device;
An imaging apparatus comprising a mask area determination unit that determines a mask area for the captured image based on the defocus amount for each pixel area calculated by the defocus amount calculation unit.
The mask area determination unit
14. The image pickup apparatus according to claim 13, wherein a pixel region having a defocus amount having a difference equal to or greater than a predetermined threshold value with respect to the defocus amount of the output subject specified by the user is determined as the mask area.
The mask area determination unit
14. The imaging apparatus according to claim 13, which generates a mask area indication image in which the mask area is distinguishable from the output subject area specified by the user.
The imaging device further comprises
14. The imaging apparatus according to claim 13, further comprising a mask synthesizing unit that synthesizes a mask image of a predetermined color with the mask area determined by the mask area determining unit to generate a mask setting image.
The imaging device further comprises
17. The imaging apparatus according to claim 16, further comprising an image synthesizing section that superimposes another image on the mask area of the mask setting image generated by the mask synthesizing section to generate a synthetic image with an output subject specified by the user.
The mask synthesizing unit
17. The imaging apparatus according to claim 16, wherein the generated mask setting image is output to an external device that executes composite image generation processing.
An image processing method executed in an imaging device,
A defocus amount calculation step in which the defocus amount calculation unit calculates a defocus amount for each pixel area of a captured image output from an imaging device of an imaging device;
A defocus amount calculation unit executes an image blend ratio calculation step of calculating an image blend ratio between the captured image and another image,
The image blend ratio calculation step includes:
An image processing method, comprising: calculating an image blend ratio for each pixel area based on the defocus amount for each pixel area calculated in the defocus amount calculation step.
A program for executing image processing in an imaging device,
a defocus amount calculation step of causing a defocus amount calculation unit to calculate a defocus amount for each pixel area of a captured image output from an imaging element of an imaging device;
causing a defocus amount calculation unit to perform an image blend ratio calculation step of calculating an image blend ratio between the captured image and another image;
In the image blend ratio calculation step,
A program for calculating an image blend ratio for each pixel area based on the defocus amount for each pixel area calculated in the defocus amount calculation step.