WO2022244311A1

WO2022244311A1 - Imaging device, image processing method, and program

Info

Publication number: WO2022244311A1
Application number: PCT/JP2022/003019
Authority: WO
Inventors: 龍之介横矢; 大水落; 祐基明壁; 貴洸小杉; 洋司山本
Original assignee: ソニーグループ株式会社
Priority date: 2021-05-20
Filing date: 2022-01-27
Publication date: 2022-11-24
Also published as: JPWO2022244311A1

Abstract

Provided are a device and a method for calculating the reliability of a defocus amount and a distance value for each image area of a captured image, and executing a display process for graphic data indicating the reliability, a distance ratio display process of an object, an exposure time control process, and the like. The defocus amount and distance value for each image area of the captured image are calculated, the reliability of the calculated defocus amount and the distance value for each image area is calculated, and control is executed in accordance with the calculated reliability of the defocus amount and the distance value for each image area. For example, a process for superimposing and displaying graphic data indicating the reliability of the defocus amount for each image area on the captured image, a process for displaying the distance ratio of the object, a process for controlling the exposure time, and the like are executed.

Description

IMAGING DEVICE, IMAGE PROCESSING METHOD, AND PROGRAM

The present disclosure relates to imaging devices, image processing methods, and programs. More specifically, the reliability of the physical quantity for each image region, such as the defocus amount and the distance value for each image region obtained from the detection information of the image plane phase difference detection pixels, is calculated, and various values are calculated according to the calculated reliability. The present invention relates to an imaging device that performs processing, an image processing method, and a program.

In imaging devices (cameras), there is an image plane phase difference method using an image plane phase difference pixel as a method for detecting a focus position (in-focus position).
This image plane phase difference method divides the light passing through the imaging lens into a pair of images and analyzes the phase difference between the generated pair of images to detect the focus position (in-focus position). It is a method to

In focus control using the image plane phase difference method, the light flux that has passed through the imaging lens is split into two, and the two split light fluxes are received by a set of image plane phase difference detection pixels that function as focus detection sensors. The focus lens is adjusted by detecting the degree of focus based on the shift amount of the signal output according to the amount of light received by each of the set of image plane phase difference detection pixels.

The detection information of the image plane phase difference detection pixels is mainly used for focus control, but it can be applied to other processing as well as focus control.
For example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2012-142952) discloses a configuration that performs blurring processing on a captured image using detection information of an image plane phase difference detection pixel.
Specifically, the detection information of the image plane phase difference detection pixels is used to analyze the distribution of the defocus amount in the captured image, and the analysis result is used to apply blur processing to the captured image.

Further, Patent Document 2 (Japanese Patent Application Laid-Open No. 2019-035967) discloses a configuration for changing and outputting the gradation characteristics of the defocus amount and the distance information obtained from the detection information of the image plane phase difference detection pixels according to the output destination. ing.
For example, different information is generated based on the detection information of the image plane phase difference detection pixels according to the information required by the output destination, such as an output destination that requires distance resolution near in-focus and an output destination that requires distance measurement range information. A configuration for outputting by

Furthermore, Patent Document 3 (Japanese Patent Application Laid-Open No. 2011-053378) discloses a configuration for performing exposure control according to the defocus amount of each image area of a captured image and the photometric value.
Specifically, for example, a configuration is disclosed in which exposure control is performed in consideration of the photometric value of an out-of-focus area in addition to the photometric value of a focus detection area to be focused.

JP 2012-142952 A JP 2019-035967 A JP 2011-053378 A JP 2011-004088 A

The present disclosure calculates the reliability of physical quantities for each image region, such as the defocus amount and distance value for each image region obtained from the detection information of the image plane phase difference detection pixels, and performs various processing according to the calculated reliability. It is an object of the present invention to provide an imaging device, an image processing method, and a program that perform

For example, in the configuration of one embodiment of the present disclosure, a configuration is realized in which an image is generated and output in which the reliability of the defocus amount for each image area can be identified.
Furthermore, in an embodiment configuration of the present disclosure, a configuration for outputting the distance ratio between the in-focus object and the background object is realized.
For example, in the configuration of the embodiment of the present disclosure, an image is generated and output in which the stability of the mask set in a partial area of the captured image can be identified according to the reliability of the defocus amount for each image area. Realize a configuration that
Furthermore, in an embodiment configuration of the present disclosure, a configuration is realized that performs exposure control for each image area according to the reliability of the defocus amount for each image area.

A first aspect of the present disclosure includes:
an image region physical quantity calculator that calculates physical quantities that change according to the subject distance for each image region that is a segmented region of a captured image;
an image area unit physical quantity reliability calculation unit that calculates the reliability of the image area unit physical quantity calculated by the image area physical quantity calculation unit;
The imaging apparatus includes an image area unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the image area unit physical quantity calculated by the image area unit physical quantity reliability calculation unit.

Furthermore, a second aspect of the present disclosure is
An image processing method executed in an imaging device,
an image region physical quantity calculation step in which the image region physical quantity calculation unit calculates a physical quantity that changes according to the subject distance for each image region that is a segmented region of the captured image;
an image region unit physical quantity reliability calculation step in which the image region unit physical quantity reliability calculation unit calculates the reliability of the image region unit physical quantity calculated in the image region physical quantity calculation step;
an image area unit physical quantity reliability correspondence process execution step in which the image area unit physical quantity reliability correspondence process execution unit executes control processing according to the reliability of the image area unit physical quantity reliability calculated in the image area unit physical quantity reliability calculation step; An image processing method for executing

Furthermore, a third aspect of the present disclosure is
A program for executing image processing in an imaging device,
an image region physical quantity calculation step of causing an image region physical quantity calculation unit to calculate a physical quantity that changes according to a subject distance for each image region that is a segmented region of a captured image;
an image region unit physical quantity reliability calculation step for causing an image region unit physical quantity reliability calculation unit to calculate the reliability of the image region unit physical quantity calculated in the image region physical quantity calculation step;
An image area unit physical quantity reliability correspondence process execution step for causing an image area unit physical quantity reliability correspondence process execution unit to execute a control process according to the reliability of the image area unit physical quantity reliability calculated in the image area unit physical quantity reliability calculation step. is in the program that runs the

It should be noted that the program of the present disclosure is, for example, a program that can be provided in a computer-readable format to an information processing device or computer system capable of executing various program codes via a storage medium or communication medium. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or computer system.

Still other objects, features, and advantages of the present disclosure will become apparent from more detailed descriptions based on the embodiments of the present disclosure and the accompanying drawings, which will be described later. In this specification, a system is a logical collective configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same housing.

According to the configuration of the embodiment of the present disclosure, the reliability of the defocus amount and the distance value for each image area of the captured image is calculated, the display processing of the graphic data indicating the reliability, the distance ratio display processing of the subject, and the , an apparatus and a method for executing exposure time control processing and the like are realized.
Specifically, for example, the defocus amount and the distance value are calculated for each image area of the captured image, and the reliability of the calculated defocus amount and distance value for each image area is calculated. Control is executed according to the reliability of the defocus amount and the distance value. For example, it executes a process of superimposing and displaying graphic data indicating the reliability of the defocus amount for each image area on the captured image, a process of displaying the distance ratio of the object, a process of controlling the exposure time, and the like.
With this configuration, the reliability of the defocus amount and distance value for each image area of the captured image is calculated, and display processing of graphic data indicating the reliability, processing of displaying the distance ratio of the subject, processing of exposure time control, etc. are executed. An apparatus and method are realized.
Note that the effects described in this specification are merely examples and are not limited, and additional effects may be provided.

It is a figure explaining the example of composition of the imaging device of this indication. It is a figure explaining the structural example of the image pick-up element which has a phase difference detection pixel. FIG. 10 is a diagram illustrating an outline of focus detection processing using a phase difference detection method; FIG. 10 is a diagram illustrating an outline of focus detection processing using a phase difference detection method; FIG. 10 is a diagram illustrating an outline of focus detection processing using a phase difference detection method; FIG. 2 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging device; FIG. 4 is a diagram illustrating a specific example of a pixel configuration of an image pickup device and an image area as a defocus amount calculation unit; FIG. 4 is a diagram illustrating a specific example of a defocus map generated by a digital signal processing unit of an imaging device; 4 is a diagram illustrating a configuration example of a digital signal processing unit of the image pickup apparatus of Example 1; FIG. 4A and 4B are diagrams for explaining a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the first embodiment; FIG. 4A and 4B are diagrams for explaining a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the first embodiment; FIG. 4A and 4B are diagrams for explaining a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the first embodiment; FIG. FIG. 10 is a diagram illustrating a configuration example of a digital signal processing unit of the imaging apparatus of Example 2; FIG. 11 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the second embodiment; FIG. 11 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging apparatus according to Example 3; FIG. 11 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus of Example 3; FIG. 11 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus of Example 3; FIG. 11 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus of Example 3; FIG. 11 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging apparatus according to a fourth embodiment; FIG. 12 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the fourth embodiment; FIG. 12 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the fourth embodiment; FIG. 11 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging apparatus according to Example 5; FIG. 12 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus of Example 5; FIG. 12 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus of Example 5; FIG. 12 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging apparatus according to Example 6; FIG. 14 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the sixth embodiment; FIG. 12 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging apparatus according to Example 6; FIG. 14 is a diagram illustrating a specific example of processing executed by a digital signal processing unit of the imaging apparatus according to the sixth embodiment; FIG. 12 is a diagram illustrating a configuration example of a digital signal processing unit of an imaging apparatus according to Example 6;

Hereinafter, details of an imaging device, an image processing method, and a program according to the present disclosure will be described with reference to the drawings. The description will be made according to the following items.
1. Configuration Example of Imaging Apparatus of Present Disclosure 2. Configuration of phase difference detection pixels, outline of focus control using detection signals from phase difference detection pixels3. 3. Basic configuration example for calculating the reliability of physical quantities in units of image areas, such as defocus amount and distance value in units of image areas, and executing various processes according to the calculated reliability. Concrete Embodiments of Calculating Reliabilities of Physical Quantities in Units of Image Areas such as Defocus Amounts and Distance Values in Units of Image Areas and Executing Various Processes According to the Calculated Reliabilities 4-1. (Embodiment 1) Embodiment of generating and outputting an image in which the reliability of the defocus amount for each image area can be identified 4-2. (Embodiment 2) Embodiment of generating and outputting an image in which the distance ratio between a focused subject and other background subjects is superimposed on a photographed image 4-3. (Embodiment 3) Embodiment of generating and outputting an image in which mask stability information is superimposed on a photographed image 4-4. (Embodiment 4) Embodiment of generating and outputting an image in which a color map outputting a color corresponding to a defocus amount for each image area is superimposed on a photographed image 4-5. (Embodiment 5) In addition to outputting a color corresponding to the defocus amount for each image area, an image is generated and output by superimposing a color map that makes it possible to identify the reliability of the defocus amount for each image area on the captured image. Example 4-6. (Embodiment 6) Embodiment in which the exposure time is controlled according to the reliability of the defocus amount for each image area, and the image is captured5. Other Examples 6. SUMMARY OF THE STRUCTURE OF THE DISCLOSURE

[1. Regarding the configuration example of the imaging device of the present disclosure]
First, a configuration example of an imaging device according to the present disclosure will be described.

FIG. 1 is a block diagram showing a configuration example of an imaging device 100 of the present disclosure.
A configuration example of an imaging device 100 of the present disclosure will be described with reference to FIG.
Incident light passing through the focus lens 101 and the zoom lens 102 is input to an imaging device 103 such as a CMOS or CCD, and photoelectrically converted in the imaging device 103 .

The image pickup device 103 has a plurality of pixels each having a photodiode, for example, arranged two-dimensionally in a matrix. It has normal pixels in which a B (blue) color filter is arranged, and phase difference detection pixels for pupil-dividing subject light and performing focus detection.

The normal pixels of the image sensor 103 generate analog electrical signals (image signals) of R (red), G (green), and B (blue) color components of the subject image, and generate R, G, and B color image signals. Output.
A phase difference detection pixel of the image sensor 103 outputs a phase difference detection signal (detection information). A phase difference detection signal (detection information) is a signal mainly used for autofocus control.
The configuration of the phase difference detection pixels, phase difference detection signals generated by the phase difference detection pixels, and focus control using the phase difference detection signals will be described later in detail.

Photoelectrically converted data output from the image sensor 103 in this manner includes RGB image signals and phase difference detection signals from phase difference detection pixels.
Each of these signals is input to the analog signal processing unit 104 , subjected to processing such as noise removal in the analog signal processing unit 104 , and converted into a digital signal in the A/D conversion unit 105 .

A digital signal digitally converted by the A/D conversion unit 105 is input to a digital signal processing unit (DSP) 108 and subjected to various signal processing.
Various image signal processing such as demosaic processing, white balance adjustment, gamma correction, etc. is performed on the RGB image signal, and the processed image is recorded in a recording device 115 such as a flash memory.
Further, it is displayed on the monitor 117 and viewfinder (EVF) 116 . An image through the lens is displayed as a through image on the monitor 117 and the viewfinder (EVF) 116 regardless of whether or not shooting is performed.

Phase difference detection pixel information (detection signal) output from the phase difference detection pixels of the image sensor 103 is also input to the digital signal processing unit (DSP) 108 via the AD conversion unit 106 .
A digital signal processing unit (DSP) 108 analyzes the phase difference between the pair of images generated by the phase difference detection pixel information (detection signal), and focuses on the subject (focusing object) to be focused. , that is, the amount of deviation between the in-focus distance and the object distance (defocus amount (DF)) is calculated.

An input unit (operation unit) 118 is an operation unit including an input unit for inputting various operation information such as shutter and zoom buttons on the camera body, a mode dial for setting the shooting mode, and the like.

The control unit 110 has a CPU, and controls various processes executed by the imaging device according to programs stored in advance in a memory (ROM) 120 or the like. A memory (EEPROM) 119 is a non-volatile memory and stores image data, various auxiliary information, programs, and the like.

The memory (ROM) 120 stores programs, calculation parameters, etc. used by the control unit (CPU) 110 . A memory (RAM) 121 stores programs used in the control unit (CPU) 110, the AF control unit 112a, and the like, parameters that change as appropriate during the execution of the programs, and the like.

The gyro 131 is a sensor that measures the inclination, angle, inclination velocity (angular velocity), etc. of the imaging device 100 . The detection information of the gyro 131 is used, for example, for calculating the amount of camera shake during image capturing.

The AF control unit 112a drives the focus lens drive motor 113a set corresponding to the focus lens 101, and executes autofocus control (AF control) processing. For example, the focus lens 101 is moved to the focus position of the focus lens 101 with respect to the subject included in the area selected by the user to obtain the focused state.

The zoom control unit 112b drives a zoom lens driving motor 113b set corresponding to the zoom lens 102. FIG. A vertical driver 107 drives an image sensor (CCD) 103 . The timing generator 106 generates control signals for processing timings of the image sensor 103 and the analog signal processing unit 104, and controls the processing timings of these processing units.
Note that the focus lens 101 is driven in the optical axis direction under the control of the AF control section 112a.

[2. Configuration of Phase Difference Detection Pixels, Overview of Focus Control Using Detection Signals from Phase Difference Detection Pixels]
Next, the configuration of the phase difference detection pixels and the outline of focus control using detection signals from the phase difference detection pixels will be described.

As described above, the imaging device 103 of the imaging apparatus 100 shown in FIG. , G, and B image signals, and phase difference detection signals (detection information) from the phase difference detection pixels, which are signals used for autofocus control, are also output.
A specific pixel configuration example of the image sensor 103 will be described with reference to FIG.

FIG. 2 is a diagram showing a pixel configuration example of the image sensor 103. As shown in FIG.
FIG. 2 shows a pixel configuration example of the image sensor 103 corresponding to (A) a partial area of the captured image.
The vertical direction is the Y-axis, and the horizontal direction is the X-axis. In FIG. 2, one pixel is indicated by one square.
The RGB pixels shown in FIG. 2 are pixels for normal image capturing. RGB pixels have, for example, a Bayer array configuration.

Detection information acquisition pixels applied to the autofocus process, ie, phase difference detection pixels 151 for acquiring phase difference information, are discretely set in some (rows) of RGB pixels having a Bayer array.
A phase difference detection pixel is configured by a pair of a right opening phase difference detection pixel Pa and a left opening phase difference detection pixel Pb.

The following two data outputs are individually performed from the imaging device 103 .
(1) Output of pixel information (image signal) from pixels (RGB pixels) for captured images;
(2) phase difference detection pixel information ((AF) detection signal) output by the phase difference detection pixel 151;

"(1) Output of pixel information (image signal) by pixels (RGB pixels) for captured image" is output in accordance with image capturing timing by the user (photographer). A display image (live view image) to be displayed on the display is output. A display image (live view image) is output at a frame rate corresponding to the image display rate of the monitor 117 or the like.

"(2) Phase difference detection pixel information (detection signal) output by phase difference detection pixel 151" is the same interval as the image output interval or a shorter interval, for example, (1/60) sec interval (=16.7 msec interval). is done in

The phase difference detection pixel information (detection signal) output from the phase difference detection pixel 151 is input to the digital signal processor (DSP) 108 via the AD converter 106 .
A digital signal processing unit (DSP) 108 analyzes the phase difference between the pair of images generated by the phase difference detection pixel information (detection signal), and focuses on the subject (focusing object) to be focused. , that is, the amount of deviation between the in-focus distance and the object distance (defocus amount (DF)) is calculated.

An outline of focus detection processing of the phase difference detection method will be described with reference to FIGS. 3 to 5. FIG.
As described above, in the phase difference detection method, the defocus of the focus lens is detected based on the shift amount of the signal output according to the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor. The amount is calculated, and the focus lens is set to the focus position (focus position) based on this defocus amount.

The set of phase difference detection pixels described with reference to FIG. 2 will be referred to as pixel Pa and pixel Pb, respectively, and the details of light incident on these pixels will be described with reference to FIG.

As shown in FIG. 3, the phase difference detection unit stores a luminous flux Ta from a right portion (also referred to as a “right partial pupil region” or simply a “right pupil region”) Qa of the exit pupil EY of the photographing optical system, and a light beam Ta from the left side. Phase difference detection pixels Pa and Pb configured by a pair of photodetectors (PD) for receiving a light flux Tb from a portion (also referred to as “left partial pupil region” or simply “left pupil region”) Qb are arranged in the horizontal direction. It is Here, in the drawing, the +X direction side is expressed as the right side, and the −X direction side is expressed as the left side.

Of the pair of phase difference detection pixels Pa and Pb, one phase difference detection pixel (hereinafter referred to as “first phase difference detection pixel”) Pa collects incident light to the first phase difference detection pixel Pa. A first light shielding plate AS1 having a microlens ML, a first slit (rectangular) opening OP1, and a second slit (rectangular) opening OP2 arranged below the first light shielding plate AS1. It is composed of a photoelectric conversion part PD that receives light through the second light shielding plate AS2.

The first opening OP1 in the first phase difference detection pixel Pa is set in a specific direction (here, the right direction (+X direction) with respect to the central axis CL passing through the center of the light receiving element PD and parallel to the optical axis LT as a reference (starting point). ). In addition, the second opening OP2 in the first phase difference detection pixel Pa is provided at a position biased in a direction opposite to the specific direction (also referred to as "anti-specific direction") with respect to the central axis CL. .

Further, of the pair of phase difference detection pixels Pa and Pb, the other phase difference detection pixel (hereinafter referred to as “second phase difference detection pixel”) Pb is a first pixel having a slit-shaped first opening OP1. A light shielding plate AS1 and a second light shielding plate AS2 arranged below the first light shielding plate AS1 and having a slit-shaped second opening OP2 are provided. The first opening OP1 in the second phase difference detection pixel Pb is provided at a position biased in the direction opposite to the specific direction with reference to the central axis CL. Also, the second opening OP2 in the second phase difference detection pixel Pb is provided at a position biased in the specific direction with reference to the central axis CL.

That is, in the pair of phase difference detection pixels Pa and Pb, the first openings OP1 are arranged to be biased in different directions. Also, the second openings OP2 are arranged to be shifted in different directions with respect to the corresponding first openings OP1 in the phase difference detection pixels Pa and Pb.

The pair of phase difference detection pixels Pa and Pb configured as described above acquire subject light that has passed through different regions (parts) in the exit pupil EY.
Specifically, the luminous flux Ta that has passed through the right pupil region Qa of the exit pupil EY passes through the microlens ML corresponding to the phase difference detection pixel Pa and the first opening OP1 of the first light shielding plate AS1. After being limited (limited) by the light shielding plate AS2, the light is received by the light receiving element PD of the first phase difference detection pixel Pa.

Further, the light flux Tb that has passed through the left pupil region Qb of the exit pupil EY passes through the microlens ML corresponding to the phase difference detection pixel Pb and the first opening OP1 of the second light shielding plate AS2, and further passes through the second light shielding plate AS2. , the light is received by the light receiving element PD of the second phase difference detection pixel Pb.

Fig. 4 shows an example of the output of the light-receiving element obtained at each pixel of Pa and Pb. As shown in FIG. 4, the output line from the pixel Pa and the output line from the pixel Pb are signals having a predetermined amount of shift Sf.

FIG. 5(a) shows the shift amount Sfa generated between the pixels Pa and Pb when the focus lens is set at a position corresponding to the subject distance and the focus is achieved, that is, in the in-focus state.
5B1 and 5B2 show shifts occurring between pixels Pa and Pb when the focus lens is not set to a position corresponding to the subject distance and is out of focus, i.e., in an out-of-focus state. Quantity Sfa is shown.
(b1) is an example in which the shift amount is larger than that at the in-focus state, and (b2) is an example in which the shift amount is smaller than that at the in-focus state.

In the cases shown in FIGS. 5B1 and 5B2, it is possible to focus by moving the focus lens so as to achieve the shift amount during focusing.
This process is the focusing process according to the "phase difference detection method".
Focusing processing according to this "phase difference detection method" enables setting of the focus lens to the in-focus position, and the focus lens can be set to a position according to the object distance.

The shift amount described with reference to FIG. 5 can be measured for each set of pixels Pa and Pb, which are phase difference detection pixels configured in the image sensor shown in FIG. It is possible to individually calculate the in-focus position (focus point) and the defocus amount for the subject image captured in the pixel combination area).

5(b1) and (b2), there is a deviation from the shift amount at the time of focusing. can be calculated.

[3. Example of Basic Configuration for Calculating Reliability of Physical Quantities for Each Image Region, such as Defocus Amount and Distance Value for Each Image Region, and Executing Various Processes According to the Calculated Reliability]
Next, a basic configuration example for calculating the reliability of physical quantities in units of image areas such as the defocus amount and the distance value in units of image areas and executing various processes according to the calculated reliability will be described.

FIG. 6 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.
The digital signal processing unit 108 shown in FIG. 6 calculates the reliability of the physical quantity for each image region, such as the defocus amount and the distance value for each image region, using the detection information of the phase difference detection pixel, and the calculated reliability is Executes various processes according to the

As shown in FIG. 6, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit. 205 , an image area unit physical quantity reliability correspondence processing execution unit 206 , an image information acquisition unit 211 , an image signal processing unit 212 , and an image output unit 213 .

The digital signal processing unit 108 receives the RGB image signals from the preceding A/D conversion unit 105 and the phase difference detection signals (detection information) output from the phase difference detection pixels.

A phase difference information acquisition unit 201 shown in FIG. 6 selects only phase difference detection signals (detection information), which are outputs of phase difference detection pixels, from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 211 shown in FIG. 6 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

The image signal acquired by the image information acquiring section 211 is input to the image signal processing section 212 .
The image signal processing unit 212 performs various image signal processing such as demosaic processing, white balance adjustment, and gamma correction on the image signal, and outputs the processed image (eg, RGB image) to the image output unit 213 .

In some cases, the image region unit physical quantity reliability corresponding processing execution unit 206 may further perform image control processing on the image (for example, an RGB image) generated by the image signal processing unit 212 . For example, superimposition processing of defocus amount reliability identification data is performed on an RGB image. Specific examples of these processes will be described later.

An image (for example, an RGB image) generated by the image signal processing unit 212, or an image generated by the image signal processing unit 212 changed or processed by the image region unit physical quantity reliability correspondence processing execution unit 206 is output to the image output unit. 213.

The image output unit 213 outputs the image input from the image signal processing unit 212 . For example, image output to the monitor 117, viewfinder (EVF) 116, and recording device 115 is executed.

The phase difference information acquisition unit 201 selects a phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105, and obtains the selected phase difference detection signal (detection information). is output to the defocus amount calculation unit 202 .

The defocus amount calculation unit 202 calculates the defocus amount, that is, the amount of deviation between the in-focus distance and the object distance (defocus amount (DF)) for each minute image area unit, for example, an image area unit composed of a plurality of pixels. calculate.
A defocus amount calculation unit 202 calculates a defocus amount, which is a physical quantity that changes according to the subject distance, for each image area.

As described above, in the phase difference detection method, the defocus of the focus lens is detected based on the shift amount of the signal output according to the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor. Amount is calculated.

The AF control signal generation unit 203 generates an autofocus control signal (AF control signal) for setting the focus lens to an in-focus position (focus position) for a subject designated by the user based on the defocus amount. , and outputs the generated AF control signal to the AF control unit 112a.

The AF control unit 112a drives the focus lens according to an autofocus control signal (AF control signal) input from the AF control signal generation unit 203, and moves the focus lens to the in-focus position (focus position) for the subject specified by the user, for example. set.

Note that the subject to be set at the in-focus position (focus position) is not the entire image area of the captured image, but a subject specified by the user such as a person. Images of other objects, such as the background, are out of focus and appear blurred.

For example, the shift amount described above with reference to FIG. 5 is measured in units of a set of pixels Pa and Pb, which are phase difference detection pixels configured in the imaging device shown in FIG.
That is, the defocus amount calculation unit 202 can calculate the defocus amount of the subject image for each fine image area unit of the captured image.

FIG. 7 shows an example of an image area that is used as a defocus amount calculation unit.
FIG. 7 shows the pixel configuration of the imaging element 103 similar to that of FIG. 2 described above.
As described above with reference to FIG. 2, the phase difference detection pixels 151 for acquiring phase difference information are discretely set in some (rows) of RGB pixels having a Bayer array.
A phase difference detection pixel is configured by a pair of a right opening phase difference detection pixel Pa and a left opening phase difference detection pixel Pb.

An image area that is a unit for calculating the defocus amount can be set, for example, as an image area 152 shown in FIG.
In the example shown in FIG. 7, the image area 152, which is the unit for calculating the defocus amount, is a fine image area of n×m pixels, such as 6×5 pixels.
A plurality of sets of phase difference detection pixels are included in this fine image area of n×m pixels. The shift amount described above with reference to FIG. 5 is measured from each of the plurality of sets of phase difference detection pixels.

The defocus amount calculation unit 202 calculates, for example, an average value of a plurality of sets of shift amounts in an image area 152 as shown in FIG. 7 as a shift amount of the image area 152 of n×m pixels. Further, the defocus amount, that is, the defocus amount corresponding to the amount of deviation between the in-focus distance and the subject distance, is calculated from the shift amount calculated and the shift amount of the focused image area.
In this manner, the defocus amount calculation unit 202 calculates the defocus amount for each image area 152 as shown in FIG. 7, for example.

The defocus map generation unit 204 receives the fine defocus amount of the captured image calculated by the defocus amount calculation unit 202, and calculates the defocus amount of each image area based on the defocus amount of each image area. To generate a defocus map in which the amount of focus can be identified.

Based on the defocus amount calculated by the defocus amount calculation unit 202 for each image area 152 as shown in FIG. 7, the defocus amount for each image area 152 as shown in FIG. is calculated to generate a defocus map.

An example of the defocus map generated by the defocus map generation unit 204 will be described with reference to FIG.

FIG. 8A is an example of an image captured by the imaging device 100. FIG. Note that the photographed image (A) shown in FIG. 8 is not limited to the photographed image recorded by the user's shutter operation on the imaging apparatus 100, and is input through the lens of the imaging apparatus 100 regardless of whether the shutter operation is performed and is monitored. 117 or the like, which includes a so-called through image.

The defocus map of FIG. 8B is a defocus map corresponding to the captured image of (A), and is a defocus map generated by the defocus map generation unit 204 .
As described above, the defocus map generation unit 204 calculates the image area 152 as shown in FIG. 7 based on the defocus amount for each image area 152 as shown in FIG. A unit defocus amount is calculated to generate a defocus map.

The rectangular area shown in the defocus map in FIG. 8(B) is the image area of the defocus amount calculation unit, and corresponds to the image area 152 shown in FIG. It should be noted that the rectangular area shown in the defocus map of FIG. 8B is shown as a large size for easy understanding. The image area as the actual defocus amount calculation unit can be set as a finer image area. That is, as described above with reference to FIG. 7, it is possible to set a pixel region of several pixels to several tens of pixels.

The defocus map shown in FIG. 8B is a map in which luminance values (pixel values) are set according to defocus amounts in units of pixel regions (rectangular regions).
The higher the brightness (white) area, the smaller the defocus amount, that is, the pixel area with the higher degree of focus.
On the other hand, the lower luminance (black) region is a pixel region with a larger defocus amount, that is, a pixel region with a lower degree of focus.

For example, when the luminance value (pixel value) is set to 0 to 255, the closer the luminance value (pixel value) is to 255 (maximum luminance (white)), the smaller the defocus amount, that is, the pixel area with a high degree of focus. The closer the luminance value (pixel value) is to 0 (minimum luminance (black)), the larger the defocus amount, that is, the pixel region with the lower degree of focus.

In the example shown in FIG. 8, (A) the person or house shown in the captured image is set as the focus target subject, and (B) in the defocus map, the image area corresponding to the person or house has high brightness. It can be seen that the pixel area is set to the (white) area and has a small defocus amount and a high degree of focus. On the other hand, it can be seen that the background area other than the person and the house is set to a low luminance (black) area or a gray area, and is a pixel area with a large defocus amount and a low focus degree.

In this way, the defocus map generation unit 204 calculates the defocus amount for each image area based on the defocus amount for each image area 152 as shown in FIG. 7 calculated by the defocus amount calculation unit 202. , to generate a defocus map as shown in FIG. 8(B).

The image area unit physical quantity reliability calculation unit 205 calculates the reliability of the image area unit physical quantity that changes according to the subject distance, such as the image area unit physical quantity, for example, the image area unit defocus amount and distance value.

For example, the image area unit physical quantity reliability calculation unit 205 inputs the defocus amount for each image area calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204, and inputs these inputs. Based on the data, the reliability of the defocus amount calculated for each image area is calculated.
Further, the image area unit physical quantity reliability calculation unit 205 calculates the reliability of the distance value calculated from the defocus amount for each image area.
Specific examples of these processes will be described later.

The reliability of the defocus amount for each image region or the reliability of the distance value for each image region calculated by the image region unit physical quantity reliability calculation unit 205 is output to the image region unit physical quantity reliability correspondence processing execution unit 206. .

The image area unit physical quantity reliability correspondence processing execution unit 206 performs the following operations according to the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit 205 or the reliability of the distance value for each image area. Various controls such as image control and shooting control are performed on the captured image (for example, an RGB image) generated by the image signal processing unit 212 .

As image control processing for the captured image (eg, RGB image) generated by the image signal processing unit 212, for example, the following processing is executed.
(a) Generation of an output image that makes it possible to identify the reliability of the defocus amount for each image area;
(b) generating an output diagram image by superimposing the distance ratio data between the focused subject and the background subject on the captured image;
(c) Generating an image that makes it possible to identify the stability of the mask set in a partial area of the captured image;
(d) Generating an output image by superimposing a color map that makes it possible to identify the defocus amount for each image area on the captured image;
Also, as the shooting control process, exposure control for each image area is executed according to the reliability of the defocus amount for each image area.

[4. Concrete Embodiment of Calculating Reliability of Physical Quantities for Each Image Region, such as Defocus Amount and Distance Value for Each Image Region, and Executing Various Processing According to the Calculated Reliability]
Next, a specific embodiment will be described in which the reliability of physical quantities in units of image areas, such as the defocus amount and the distance value in units of image areas, is calculated, and various processes are executed according to the calculated reliability.

As specific examples of executing various processes according to the reliability of physical quantities in units of image areas, such as defocus amounts and distance values in units of image areas, which are executed by the imaging apparatus of the present disclosure, the following embodiments are provided. will be explained.
(Embodiment 1) Embodiment in which an image is generated and output in which the reliability of the defocus amount for each image area can be identified (Embodiment 2) The distance ratio between the in-focus subject and other background subjects is displayed on the captured image (Embodiment 3) Example of generating and outputting an image in which mask stability information is superimposed on a photographed image (Embodiment 4) Defocus amount for each image area Example of generating and outputting an image in which a color map outputting corresponding colors is superimposed on a captured image (Embodiment 5) Outputting colors corresponding to the defocus amount of each image area, and defocusing of each image area Example for generating and outputting an image in which a color map that enables identification of the amount reliability is superimposed on a captured image (Embodiment 6) Exposure time is controlled according to the reliability of the defocus amount for each image area. Example of image capturing

[4-1. (Embodiment 1) Embodiment of generating and outputting an image in which the reliability of the defocus amount for each image area can be identified]
First, as Example 1, an example of generating and outputting an image in which the reliability of the defocus amount for each image area can be identified will be described.

FIG. 9 is a block diagram for explaining the configuration of the first embodiment.
FIG. 9 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 9, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit. 205 , an image area unit physical quantity reliability correspondence processing execution unit 206 , an image information acquisition unit 211 , an image signal processing unit 212 , and an image output unit 213 .

A phase difference information acquisition unit 201 shown in FIG. 9 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 211 shown in FIG. 9 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

In addition, in the first embodiment, the image region unit physical quantity reliability corresponding processing execution unit 206 further performs image control processing on the image (eg, RGB image) generated by the image signal processing unit 212 .

In the first embodiment, the image (for example, the RGB image) generated by the image signal processing unit 212 is superimposed with the reliability identification data of the defocus amount.

For example, as described above with reference to FIG. 7, the defocus amount calculation unit 202 calculates the defocus amount, that is, the amount of deviation between the in-focus distance and the subject distance, in units of minute image areas such as n×m pixels. A defocus amount equivalent to is calculated.

For example, a defocus map as described above with reference to FIG. 8B is generated.
For example, in the case of a defocus map with a brightness value (pixel value) set to 0 to 255, the closer the brightness value (pixel value) to 255 (maximum brightness (white)), the smaller the defocus amount, that is, the degree of focus is a pixel region with a high luminance value (pixel value), and the closer the luminance value (pixel value) is to 0 (minimum luminance (black)), the larger the defocus amount, that is, the pixel region has a lower degree of focus.
Thus, the defocus map generation unit 204 generates a defocus map as shown in FIG. 8B, for example.

The image area unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image area, which is the physical quantity for each image area.
The image area unit physical quantity reliability calculation unit 205 receives the defocus amount for each image area calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204, and converts these input data into Based on this, the reliability of the defocus amount calculated for each image area is calculated.

Note that various methods can be applied as the method of calculating the reliability of the defocus amount for each image area.
For example, a method of calculating the defocus amount reliability for each image area using the contrast of the image (RGB image) generated by the image signal processing unit 212, the image frequency, the edge detection result, or the like can be used.
A method of calculating the defocus amount reliability for each image area using a cross-correlation function of two waveforms of parallax data used for defocus amount calculation is also possible.

In the following, as one specific example of the reliability calculation process of the defocus amount in units of image areas executed by the image area unit physical quantity reliability calculation unit 205, cross correlation between two waveforms of parallax data used for defocus amount calculation will be described. A technique using functions will be described.

This defocus amount reliability calculation method uses two waveforms of the parallax data shown in FIG. , a technique that uses the cross-correlation function of these two waveforms.

As described above with reference to FIG. 3, the pair of phase difference detection pixels Pa and Pb acquire subject light that has passed through different regions (parts) in the exit pupil EY.
As shown in FIG. 4, the outputs of the light-receiving elements acquired by the pixels Pa and Pb are signals having a predetermined amount of shift Sf for the output line from the pixel Pa and the output line from the pixel Pb.

FIG. 5(a) shows the shift amount Sfa generated between the pixels Pa and Pb when the focus lens is set at a position corresponding to the subject distance and the focus is achieved, that is, in the in-focus state.
5B1 and 5B2 show shifts occurring between pixels Pa and Pb when the focus lens is not set to a position corresponding to the subject distance and is out of focus, i.e., in an out-of-focus state. Quantity Sfa is shown.

The image area unit physical quantity reliability calculation unit 205 indicates the reliability of the defocus amount for each image area by using two waveforms of the parallax data used for calculating the defocus amount, that is, the output from the pixel Pa shown in FIG. It is calculated using the waveform, the waveform indicating the output from the pixel Pb, and the cross-correlation function of these two waveforms. A waveform indicating the output from the pixel Pa and a waveform indicating the output from the pixel Pb, and these two waveform data are acquired by the phase information acquisition unit 201 and passed through the defocus amount calculation unit 202. It is input to the reliability calculation unit 205 .

The image area unit physical quantity reliability calculation unit 205 calculates the defocus amount reliability from the two waveform data as follows.
Let f(t) be the waveform data representing the output from the pixel Pa shown in FIG. 4, and g(t) be the waveform data representing the output from the pixel Pb. A cross-correlation function h(τ) of these two waveform data f(t) and g(t) can be calculated by the following (equation 1). Note that t indicates the position within each pixel Pa, Pb.

(Formula 1)

In the above (Equation 1), h(τ _max ) can be calculated as the reliability of the defocus amount, where τ _max is the (τ) at which the cross-correlation function h(τ) reaches its maximum value.

Note that the defocus amount reliability calculated in this example is the reliability of the defocus amount for each pixel region.
As described above with reference to FIG. 7, one image region 152 includes a plurality of sets of phase difference detection pixels. The defocus amount calculation unit 202 calculates, for example, an average value of a plurality of sets of shift amounts in an image area 152 as shown in FIG. 7 as a shift amount of the image area 152 of n×m pixels. Further, the defocus amount, that is, the defocus amount corresponding to the amount of deviation between the in-focus distance and the subject distance, is calculated from the shift amount calculated and the shift amount of the focused image area.

The image area unit physical quantity reliability calculation unit 205 calculates the defocus amount for each image area calculated by the defocus amount calculation unit 202, that is, the image area which is one rectangular area in the defocus map shown in FIG. Using the unit defocus amount, the reliability of the defocus amount for each image area is calculated.
Note that the waveform data f(t) and g(t) that serve as the basis for calculating the reliability are, for example, the waveform data f( t) and the average waveform of the waveform data g(t) are calculated and used.

As described above, the image region unit physical quantity reliability calculation unit 205 uses, for example, the cross-correlation function of the two waveforms of the parallax data used to calculate the defocus amount, and the image region calculated by the defocus amount calculation unit 202. Calculate the reliability of the unit defocus amount.

The reliability data of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit 205 is output to the image area unit physical quantity reliability correspondence processing execution unit 206 .

The image area unit physical quantity reliability correspondence processing execution unit 206 calculates the captured image generated by the image signal processing unit 212 according to the reliability data of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit 205. (e.g., RGB images).
Specifically, an image generation process is performed in which the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit 205 can be identified.

That is, the image area unit physical quantity reliability correspondence processing execution unit 206 performs image control processing on an image (for example, an RGB image) generated by the image signal processing unit 212. The defocus amount calculated for each image area is calculated for each image area. and a process of generating an output image in which high-reliability regions can be identified.
A specific example of this processing will be described with reference to FIG. 10 and subsequent drawings.

FIG. 10 shows the following data.
(A) Captured image (B) Defocus map

(A) A captured image is a captured image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 .
(B) A defocus map is a defocus map generated by the defocus map generation unit 204 . That is, based on the defocus amount for each image area calculated by the defocus amount calculation unit 202 based on the pixel value signal of the phase difference detection pixel acquired by the phase difference information acquisition unit 201, the defocus map generation unit 204 generates This is a defocus map that outputs the defocus amount for each image area as a luminance value (for example, 0 to 255).

As described above, the image area unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image area in the defocus map generated by the defocus map generation unit 204, and uses the calculated reliability data. It is output to the image area unit physical quantity reliability correspondence processing execution unit 206 .

Based on the reliability data of the defocus amount for each image region input from the image region unit physical quantity reliability calculation unit 205, the image region unit physical quantity reliability corresponding processing execution unit 206 calculates, for example, as shown in FIG. , a defocus amount low-reliability area and a defocus amount high-reliability area are extracted.

Specifically, the defocus amount reliability for each image area input from the image area unit physical quantity reliability calculation unit 205 is compared with a predetermined reliability threshold to determine the defocus amount low reliability area. , to extract the defocus amount high reliability region.
For example, using a predefined low-reliability threshold Th1 and a high-reliability threshold Th2,
(Decision formula 1) Defocus amount reliability ≤ Th1
An image region that satisfies the above determination formula 1 is extracted as a defocus amount low reliability region.

moreover,
(Judgment Expression 2) Th2≦Defocus Amount Reliability An image region that satisfies the above judgment expression 2 is extracted as a defocus amount high reliability region.

The image area unit physical quantity reliability correspondence processing execution unit 206 extracts a defocus amount low reliability area and a defocus amount high reliability area according to these determination formulas, and based on the extraction result, is calculated for each image area. Then, a process of generating an output image in which a low-reliability region and a high-reliability region of the defocus amount can be identified is executed.
A specific example will be described with reference to FIG. 11 and subsequent figures.

FIG. 11 shows the following data.
(A) Captured image (B) Defocus map (C) Output image

(A) A captured image is a captured image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 .
(B) A defocus map is a defocus map generated by the defocus map generation unit 204 .
(C) An output image is an example of an output image generated by the image area unit physical quantity reliability correspondence processing execution unit 206 based on (A) a photographed image and (B) a defocus map.

An example of an output image (C) shown in FIG. 11 is an example of an output image in which a defocus amount low-reliability region can be identified.

The image area unit physical quantity reliability correspondence processing execution unit 206 performs (B) an area where the reliability of the defocus amount for each image area set in the defocus map is equal to or lower than the low reliability threshold Th1 described above, that is,
(Decision formula 1) Defocus amount reliability ≤ Th1
An image area that satisfies the above determination formula 1 is extracted as a defocus amount low-reliability area, and graphic data that makes it possible to identify this low-reliability area is superimposed on (A) the captured image to generate (C) an output image. do.

The defocus amount low-reliability area shown in the output image of FIG. 11(C) is composed of, for example, graphic data of a plurality of translucent red rectangular blocks. Each rectangular block corresponds to one image area that is a defocus amount calculation unit. Note that red is an example, and other colors may be set.

This (C) output image is output to the monitor 117 of the imaging apparatus 100, for example.
By looking at the image output to the monitor 117, the user can easily check the area where the defocus amount may not be calculated correctly.

The (C) output image in FIG. 11 is an example of an output image in which a defocus amount low-reliability region can be identified. Next, with reference to FIG. 12, an example of an output image in which the defocus amount high reliability area can be identified will be described.
Similar to FIG. 11, FIG. 12 shows the following data.
(A) Captured image (B) Defocus map (C) Output image

An example of the output image (C) shown in FIG. 12 is an example of an output image in which the defocus amount high reliability region can be identified.

The image area unit physical quantity reliability correspondence processing execution unit 206 performs (B) an area where the reliability of the defocus amount for each image area set in the defocus map is equal to or higher than the above-described high reliability threshold Th2, that is,
(Decision formula 2) Th2 ≤ defocus amount reliability An image area that satisfies the above determination formula 2 is extracted as a defocus amount high reliability area, and the graphic data that makes this high reliability area identifiable (A) Photographed image Superimpose on top to generate (C) the output image.

The defocus amount high-reliability area shown in the output image of FIG. 12(C) is composed of, for example, graphic data of a plurality of translucent blue rectangular blocks. Each rectangular block corresponds to one image area that is a defocus amount calculation unit. Note that blue is an example, and other colors may be set.

This (C) output image is output to the monitor 117 of the imaging apparatus 100, for example.
By looking at the image output to the monitor 117, the user can easily confirm the area where the defocus amount is calculated correctly.

As described above, the first embodiment is an embodiment for generating and outputting an image in which the graphic data that enables identification of the reliability of the defocus amount for each image area is superimposed on the captured image. With the graphic data superimposed on , it is possible to distinguish and confirm an area where the defocus amount is calculated correctly and an area where the defocus amount is not calculated correctly.

Note that the digital signal processing unit 108 described with reference to FIG. 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, an image output unit 213, and a setting having all these configurations. However, this configuration is an example.

A part of the configuration described with reference to FIG. Alternatively, the data processing may be performed by an external device different from the imaging device 100 .
Specifically, for example, the image signal processing unit 212 can be configured outside the digital signal processing unit 108 . Also, the image signal processing may be performed by an external device other than the imaging device 100, such as a PC.

[4-2. (Example 2) Example of generating and outputting an image in which a distance ratio between a focused subject and other background subjects is superimposed on a photographed image]
Next, as Example 2, an example of generating and outputting an image in which the distance ratio between the in-focus object and other background objects is superimposed on the captured image will be described.

FIG. 13 is a block diagram for explaining the configuration of the second embodiment.
FIG. 13 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 13, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit. 205 , an image area unit physical quantity reliability correspondence processing execution unit 206 , an image information acquisition unit 211 , an image signal processing unit 212 , an image output unit 213 , and a distance information calculation unit 221 .

A phase difference information acquisition unit 201 shown in FIG. 13 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 211 shown in FIG. 13 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

Note that, in the second embodiment, as in the first embodiment described above, image control by the image region unit physical quantity reliability corresponding processing execution unit 206 is performed on an image (for example, an RGB image) generated by the image signal processing unit 212. processing takes place.

In the second embodiment, an image in which the distance ratio between the in-focus subject and other background subjects is superimposed on the image (for example, the RGB image) generated by the image signal processing unit 212 is generated and output.

The defocus amount calculation unit 202 calculates the defocus amount, that is, the amount of deviation between the in-focus distance and the object distance (defocus amount (DF)) for each minute image area unit, for example, an image area unit composed of a plurality of pixels. calculate.
As described above, in the phase difference detection method, the defocus of the focus lens is detected based on the shift amount of the signal output according to the amount of light received by each of a set of phase difference detection pixels that function as a focus detection sensor. Amount is calculated.

The distance information calculation unit 221 receives the defocus amount for each image area of the captured image calculated by the defocus amount calculation unit 202, and calculates the distance for each image area of the captured image based on the defocus amount for each image area. Calculate information.
The distance information calculation unit 221 calculates a distance value, which is a physical quantity that changes according to the object distance, for each image area.
The distance information calculation unit 221 generates, for example, a depth map indicating the distance value for each image area by pixel values (eg, 0 to 255).

The (B) distance information (depth map) for each image area generated by the distance information calculation unit 221 is based on the defocus amount for each minute image area such as n×m pixels calculated by the defocus amount calculation unit 202. 4 is a depth map showing the value of the object distance for each image area calculated in the above manner as a pixel value (0 to 255, for example).
A high luminance (high pixel value) area is an area with a short subject distance, and a low luminance (low pixel value) area is an area with a long subject distance.

Note that the process of calculating the subject distance from the defocus amount can be calculated by applying parameters such as the focal length of the lens (focus lens) of the imaging device.
Specifically, the subject distance for each image area is calculated according to (Equation 2) below.

... (Formula 2)

The distance information calculation unit 221 calculates the subject distance for each image area according to the above (Formula 2).

The object distance information for each image area calculated by the distance information calculation unit 221 is output to the image area unit physical quantity reliability calculation unit 205 .

The image area unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image area, which is the physical quantity for each image area, and the reliability of the distance information for each image area.

Note that the image area unit physical quantity reliability calculation unit 205 calculates the reliability of the distance information for each image area as the reliability according to the reliability of the defocus amount for each image area.
That is, it is determined that the reliability of the distance value calculated based on the defocus amount is low for an image area with a low reliability of the defocus amount, and the image area with a high reliability of the defocus amount is determined based on the defocus amount. It is judged that the reliability of the distance value calculated by

As in the first embodiment, the image region unit physical quantity reliability calculation unit 205 calculates the defocus amount for each image region calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204. Based on these input data, the reliability of the defocus amount calculated for each image area is calculated.

As described in the first embodiment, various methods can be applied as the method of calculating the reliability of the defocus amount for each image area.
For example, a method of calculating the defocus amount reliability for each image area using the contrast of the image (RGB image) generated by the image signal processing unit 212, the image frequency, the edge detection result, or the like can be used.
Further, the defocus amount reliability for each image area is calculated using the method using (Equation 1) described in the first embodiment, that is, the cross-correlation function of the two waveforms of the parallax data used for defocus amount calculation. method is also possible.

Note that the defocus amount reliability calculated in the second embodiment is also the reliability of the defocus amount for each pixel area, as described in the first embodiment.
The image area unit physical quantity reliability calculation unit 205 calculates the defocus amount for each image area calculated by the defocus amount calculation unit 202, that is, the image area unit which is one rectangular area in the defocus map shown in FIG. is used to calculate the reliability of the defocus amount for each image area.

Furthermore, the image area unit physical quantity reliability calculation unit 205 calculates the reliability of the distance information for each image area according to the reliability of the defocus amount for each image area.
As described above, the reliability of the distance information for each image area is calculated as the reliability corresponding to the reliability of the defocus amount for each image area.
That is, it is determined that the reliability of the distance value calculated based on the defocus amount is low for an image area with a low reliability of the defocus amount, and the image area with a high reliability of the defocus amount is determined based on the defocus amount. It is judged that the reliability of the distance value calculated by

The reliability data of the distance information for each image area calculated by the image area unit physical quantity reliability calculation unit 205 is output to the image area unit physical quantity reliability correspondence processing execution unit 206 .

The image area unit physical quantity reliability correspondence processing execution unit 206 calculates the photographed image generated by the image signal processing unit 212 ( image control for RGB images).
Specifically, according to the reliability of the distance information for each image area calculated by the image area unit physical quantity reliability calculation unit 205, distance ratio data between areas having highly reliable distance information is generated to generate the captured image. Displayed superimposed on top.
For example, an image area having a distance value with a degree of reliability greater than or equal to a predetermined threshold is selected, and distance ratio data between subjects in the selected image area is generated and displayed superimposed on the captured image.

A specific example of this process will be described with reference to FIG.

FIG. 14 shows the following data.
(A) Photographed image (B) Output image (p) Separation degree (distance ratio) calculation example

(A) A captured image is a captured image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 .
The (B) output image is an example of an output image generated by the image-region unit physical quantity reliability correspondence processing execution unit 206 based on the (A) captured image.

(B) The degree of separation (=distance ratio) between the focused subject and the background subject is superimposed on the upper left of the output image.
This separability (=distance ratio) data is data generated by the image area unit physical quantity reliability calculation unit 205 . The image area unit physical quantity reliability calculation unit 205 selects the in-focus subject area and a part of the background area as areas having highly reliable distance information according to the reliability of the distance information for each image area, A distance ratio between these regions is calculated and displayed as separation degree (distance ratio) data superimposed on the captured image.

A calculation example of the separation (distance ratio) data is as shown in FIG. 14(p) separation (distance ratio) calculation example. For example, let a be the distance from the camera to the in-focus object, and b be the distance from the camera to the background area.
At this time, the degree of separation (distance ratio) between the focused object and the background area is b/a.
In the upper left of the output image in FIG.
is displayed.

This (B) output image is output to the monitor 117 of the imaging apparatus 100, for example.
By looking at the image output to the monitor 117, the user can easily check the degree of separation (distance ratio) between the regions whose distances are calculated correctly.

As described above, in the second embodiment, the reliability of distance information for each image area is calculated, and data that enables confirmation of the degree of separation (distance ratio) between areas having highly reliable distance information is displayed on the captured image. This is an embodiment in which superimposed images are generated and output, and the user sees the degree of separation (distance ratio) data superimposed on the captured image, and the degree of separation (distance ratio) between regions whose distances are correctly calculated. can be easily confirmed.

Note that the digital signal processing unit 108 described with reference to FIG. 204, image region unit physical quantity reliability calculation unit 205, image region unit physical quantity reliability correspondence processing execution unit 206, image information acquisition unit 211, image signal processing unit 212, image output unit 213, distance information calculation unit 221, all of these This configuration is an example.

[4-3. (Example 3) Example of generating and outputting an image in which mask stability information is superimposed on a photographed image]
Next, as Example 3, an example of generating and outputting an image in which mask stability information is superimposed on a photographed image will be described.

For example, mask processing may be performed to set a background area other than a specific subject selected from a captured image to a uniform color such as green background or blue background.
By synthesizing another image, such as a new background image, with the green background area or blue background area of the image subjected to such mask processing, a composite image is generated in which the selected specific subject is displayed on the new background image. It becomes possible to
The present embodiment is an embodiment for generating mask stability information indicating whether or not highly accurate mask setting is possible when performing such mask processing, and outputting the information superimposed on the captured image.

FIG. 15 is a block diagram for explaining the configuration of the third embodiment.
FIG. 15 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 15, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image area unit physical quantity reliability calculation unit. 205 , an image area unit physical quantity reliability correspondence processing execution unit 206 , an image information acquisition unit 211 , an image signal processing unit 212 , and an image output unit 213 .

A phase difference information acquisition unit 201 shown in FIG. 15 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 211 shown in FIG. 15 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

In the third embodiment, as in the first and second embodiments described above, the image (for example, RGB image) generated by the image signal processing unit 212 is further processed by the image region unit physical quantity reliability corresponding processing executing unit. 206 performs image control processing.

In the third embodiment, a process of superimposing and outputting mask stability information according to the reliability of the defocus amount is performed on an image (for example, an RGB image) generated by the image signal processing unit 212 .

The image area unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image area, which is the physical quantity for each image area.
As in the first embodiment, the image region unit physical quantity reliability calculation unit 205 calculates the defocus amount for each image region calculated by the defocus amount calculation unit 202 and the defocus map generated by the defocus map generation unit 204. Based on these input data, the reliability of the defocus amount calculated for each image area is calculated.

Note that the defocus amount reliability calculated in the third embodiment is also the reliability of the defocus amount for each pixel area, as described in the first embodiment.
The image area unit physical quantity reliability calculation unit 205 calculates the defocus amount for each image area calculated by the defocus amount calculation unit 202, that is, the image area unit which is one rectangular area in the defocus map shown in FIG. is used to calculate the reliability of the defocus amount for each image area.

The image area unit physical quantity reliability correspondence processing execution unit 206 calculates the captured image generated by the image signal processing unit 212 according to the reliability data of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit 205. (e.g., RGB images).
Specifically, mask stability information corresponding to the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit 205 is generated and superimposed on the captured image and output.

The mask stability information is information indicating whether or not it is possible to set the mask accurately when setting the mask on an area other than the selected subject of the captured image, for example, the background area.
As described above, for example, mask processing is performed to set a background region other than a specific subject selected from a photographed image to a uniform color such as green background or blue background, and the green background region or blue background of the image subjected to mask processing is performed. By synthesizing another image, such as a new background image, in the background area, it is possible to generate a synthesized image in which the selected specific subject is displayed on the new background image.

The present embodiment is an embodiment in which mask stability information indicating whether or not highly accurate mask setting is possible is generated and superimposed on the captured image and output when such mask processing is performed.
The image region unit physical quantity reliability correspondence processing execution unit 206 generates mask stability information according to the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit 205, and displays it on the captured image. superimposed on the output.
A specific example of this processing will be described with reference to FIG. 16 and subsequent drawings.

First, with reference to FIG. 16, an example of mask processing for a captured image and an example of composite image generation processing will be described.
For example, the photographed image in FIG. 16A is a photographed image in which a person and a house are set as focused subjects. The defocus amount of this in-focus object is almost zero. A mask process is performed to set the background area other than the in-focus object area as a mask area to a uniform color such as green background or blue background. (B) a mask setting image is generated by this mask setting process.

Next, another image such as a new background image is combined with the green background area or blue background area of the masked image. By this processing, it is possible to generate a composite image as shown in FIG. 16(C).
In order to generate such a composite image, it is essential to set an accurate mask area.

For example, the following processing is possible as a specific mask area determination process when an area where no mask is set is set as a focus area such as a person or a house, and a background area other than the focus area is set as a mask area. be.
A region in which the defocus amount per image region is equal to or less than a predetermined threshold value Th, that is,
(Decision formula a) defocus amount ≤ Th
An area that satisfies this determination formula a is determined as an in-focus area and determined as an area not to be masked.
On the other hand, an area that does not satisfy the determination formula a is determined as an out-of-focus area and determined as a mask area to be masked.
A mask area can be determined by such processing.

However, if the defocus amount is an inaccurate value, the mask area cannot be determined accurately even if the mask area determination process is performed using the above (determination formula a).
In the present embodiment, when performing mask processing, mask stability information indicating whether or not highly accurate mask setting is possible is generated, superimposed on the photographed image, and output to provide the user with stable mask area setting. This is an embodiment in which it is possible to notify whether or not it is possible.

The image region unit physical quantity reliability correspondence processing execution unit 206 generates mask stability information according to the reliability of the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit 205, and displays it on the captured image. superimposed on the output.
A specific processing example will be described with reference to FIG. 17 and the subsequent drawings.

FIG. 17 shows the following data.
(A) Captured image (B) Defocus map (C) Output image

The (C) output image is an example of the output image generated by the image region unit physical quantity reliability correspondence processing execution unit 206 based on the (A) captured image.
(C) Mask stability information is displayed in the upper right of the output image.

Based on the reliability data of the defocus amount for each image region input from the image region unit physical quantity reliability calculation unit 205, the image region unit physical quantity reliability corresponding processing execution unit 206 calculates, for example, as shown in FIG. , a defocus amount low-reliability region is extracted.

Specifically, the defocus amount reliability for each image area input from the image area unit physical quantity reliability calculation unit 205 is compared with a predetermined reliability threshold to determine a defocus amount low reliability area. Extract.
For example, using a predefined low reliability threshold Th1,
(Decision formula 1) Defocus amount reliability ≤ Th1
An image region that satisfies the above determination formula 1 is extracted as a defocus amount low reliability region.

The defocus map shown in FIG. 17B includes a defocus amount low reliability region.
Thus, when a defocus amount low-reliability area is detected, the image area unit physical quantity reliability correspondence processing execution unit 206 determines that stable mask area determination processing is difficult. Furthermore, according to this determination, an output image is generated and output by superimposing mask stability information indicating "mask instability" on the photographed image as shown in FIG. 17(C).

In the example of the output image (C) shown in FIG. 17, a defocus amount low-reliability region is detected, and mask stability information (=“mask instability”) indicating that it is difficult to determine a stable mask region is displayed. It is an example of an output image superimposed on a captured image.

This (C) output image is output to the monitor 117 of the imaging apparatus 100, for example.
The user can see the image output to the monitor 117 and recognize that stable mask setting is difficult.

Next, an example of processing when stable mask setting is possible will be described with reference to FIG.
Similar to FIG. 17, FIG. 18 also shows the following data.
(A) Captured image (B) Defocus map (C) Output image

The image area unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image area in the defocus map generated by the defocus map generation unit 204, and uses the calculated reliability data as the image area unit physical quantity reliability. Output to the degree correspondence processing execution unit 206 .

The example shown in FIG. 18 is an example in which no defocus amount low-reliability region is detected from the defocus map shown in FIG. 18B.
That is, as explained earlier,
(Decision formula 1) Defocus amount reliability ≤ Th1
This is an example of a case where an image area that satisfies the determination formula 1 is not detected.

The defocus map shown in FIG. 18B does not include the defocus amount low reliability region.
In this way, when no defocus amount low-reliability region is detected, the image region unit physical quantity reliability handling processing execution unit 206 determines that stable mask region determination processing is possible. Further, according to this determination, an output image is generated and output by superimposing mask stability information indicating "mask stability" on the captured image as shown in FIG. 18(C).

An example of the output image (C) shown in FIG. 18 contains mask stability information (=“mask stable”) indicating that a defocus amount low-reliability region is not detected and a stable mask region can be determined. It is an example of an output image superimposed on a captured image.

This (C) output image is output to the monitor 117 of the imaging apparatus 100, for example.
The user can see the image output to the monitor 117 and recognize that stable mask setting is possible.

The examples of the mask stability information shown in FIGS. 17 and 18 are only examples, and the mask stability information can be displayed using various characters, symbols, icons, icon displays of different colors, and the like. It is possible.

Further, the digital signal processing unit 108 described with reference to FIG. 204, an image region unit physical quantity reliability calculation unit 205, an image region unit physical quantity reliability corresponding processing execution unit 206, an image information acquisition unit 211, an image signal processing unit 212, an image output unit 213, and a setting having all these configurations. However, this configuration is an example.

[4-4. (Embodiment 4) Embodiment of generating and outputting an image in which a color map outputting a color corresponding to a defocus amount for each image area is superimposed on a captured image]
Next, as Example 4, an example will be described in which an image is generated and output by superimposing a color map outputting a color corresponding to the defocus amount for each image area on a photographed image.

For example, a photographed image includes subjects having various defocus amounts, such as a focused subject whose defocus amount is almost 0, and a background subject whose defocus amount is large.
This embodiment is an embodiment for generating and outputting an output image in which a color corresponding to the defocus amount is set for a photographed image.

FIG. 19 is a block diagram for explaining the configuration of the fourth embodiment.
FIG. 19 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 19, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image region unit physical quantity reliability calculation unit. 205 , an image area unit physical quantity reliability correspondence processing execution unit 206 , an image information acquisition unit 211 , an image signal processing unit 212 , and an image output unit 213 .

A phase difference information acquisition unit 201 shown in FIG. 19 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 211 shown in FIG. 19 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

In the fourth embodiment, as in the first to third embodiments described above, an image (for example, an RGB image) generated by the image signal processing unit 212 is further processed by the image region unit physical quantity reliability correspondence processing execution unit. 206 performs image control processing.

In the fourth embodiment, an image is generated and output by superimposing a color map outputting colors according to the defocus amount for each image area on an image (for example, an RGB image) generated by the image signal processing unit 212 . processing takes place.

Note that the defocus amount reliability calculated in the fourth embodiment is also the reliability of the defocus amount for each pixel area, as described in the first embodiment.
The image area unit physical quantity reliability calculation unit 205 calculates the defocus amount for each image area calculated by the defocus amount calculation unit 202, that is, the image area unit which is one rectangular area in the defocus map shown in FIG. is used to calculate the reliability of the defocus amount for each image area.

The image area unit physical quantity reliability correspondence processing execution unit 206 calculates the captured image generated by the image signal processing unit 212 according to the reliability data of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit 205. (e.g., RGB images).
Specifically, an image is generated and output by superimposing a color map outputting a color corresponding to the defocus amount for each image region calculated by the image region unit physical quantity reliability calculation unit 205 on the captured image.
A specific example of this process will be described with reference to FIG. 20 and subsequent figures.

FIG. 20 shows the following data.
(A) Captured image (B) Defocus map (C) Defocus color map

(C) The defocus color map is a map obtained by coloring the defocus map that outputs the defocus amount as a luminance value (for example, 0 to 255), and a map in which different colors are set according to the defocus amount. is.
For example, it is a color map in which yellow is set for areas where the defocus amount is 0 to small, green is set for areas where the defocus amount is medium, and blue is set for areas where the defocus amount is large.

Various settings are possible for the color setting.
For example, in addition to the 3-level or 5-level color setting as described above, it is also possible to set the color to smoothly change from yellow to blue according to the change in the defocus amount.

In addition, a setting may be made in which the color of the in-focus area where the defocus amount is almost 0 is not set and the color of the original photographed image is output as it is.
Further, for example, by performing face recognition, semantic segmentation, or the like, identification of the subject object may be performed, and a specific color that is different only for a person's face area or a specific object may be output.

In this way, the image area unit physical quantity reliability correspondence processing execution unit 206 first generates a color map that outputs colors according to the image area unit defocus amount calculated by the image area unit physical quantity reliability calculation unit 205. .
Furthermore, the image area unit physical quantity reliability correspondence processing execution unit 206 generates and outputs an output image in which the generated color map is superimposed on the captured image.
A specific example of this processing will be described with reference to FIG.

FIG. 21 shows the following data.
(A) Photographed image (C) Defocused color map (D) Output image

(A) A captured image is a captured image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 .
(C) The defocus color map is a map obtained by coloring the defocus map that outputs the defocus amount as a luminance value (for example, 0 to 255), and a map in which different colors are set according to the defocus amount. is.
For example, it is a color map in which yellow is set for areas where the defocus amount is 0 to small, green is set for areas where the defocus amount is medium, and blue is set for areas where the defocus amount is large.

(C) The output image is an output image generated by superimposing (B) the defocus color map on (A) the captured image.

In this manner, the image region unit physical quantity reliability correspondence processing execution unit 206 generates and outputs an output image in which the generated color map is superimposed on the captured image.
This (C) output image is output to the monitor 117 of the imaging apparatus 100, for example.
The user can easily and reliably determine the defocus amount of each area of the captured image by viewing the image output to the monitor 117 .

[4-5. (Embodiment 5) In addition to outputting a color corresponding to the defocus amount for each image area, an image is generated and output by superimposing a color map that makes it possible to identify the reliability of the defocus amount for each image area on the captured image. Example to be performed]
Next, as a fifth embodiment, an image is generated in which a color map is superimposed on the captured image so as to output a color corresponding to the defocus amount for each image area, and to make it possible to identify the reliability of the defocus amount for each image area. An embodiment of outputting by

The fifth embodiment is a modification of the fourth embodiment described above, and is an example in which the color map described in the fourth embodiment is changed to a color map capable of identifying the defocus amount reliability described in the first embodiment. be.

FIG. 22 is a block diagram for explaining the configuration of the fifth embodiment.
FIG. 22 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 22, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image area unit physical quantity reliability calculation unit. 205 , an image area unit physical quantity reliability correspondence processing execution unit 206 , an image information acquisition unit 211 , an image signal processing unit 212 , and an image output unit 213 .

A phase difference information acquisition unit 201 shown in FIG. 22 selects only the phase difference detection signal (detection information) that is the output of the phase difference detection pixel from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 211 shown in FIG. 22 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

In the fifth embodiment, as in the first to fourth embodiments described above, an image (for example, an RGB image) generated by the image signal processing unit 212 is further processed by the image region unit physical quantity reliability correspondence processing execution unit. 206 performs image control processing.

In the fifth embodiment, for an image (for example, an RGB image) generated by the image signal processing unit 212, a color corresponding to the defocus amount for each image area is output. A process of generating and outputting an image superimposed with a color map that makes it possible to distinguish degrees is performed.

Note that the defocus amount reliability calculated in the fifth embodiment is also the reliability of the defocus amount for each pixel area, as described in the first embodiment.
The image area unit physical quantity reliability calculation unit 205 calculates the defocus amount for each image area calculated by the defocus amount calculation unit 202, that is, the image area unit which is one rectangular area in the defocus map shown in FIG. is used to calculate the reliability of the defocus amount for each image area.

The image area unit physical quantity reliability correspondence processing execution unit 206 calculates the captured image generated by the image signal processing unit 212 according to the reliability data of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit 205. (e.g., RGB images).
Specifically, a color map that outputs a color corresponding to the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit 205, and that can identify the reliability of the defocus amount for each image area. is superimposed on the captured image and output.
A specific example of this process will be described with reference to FIG. 23 and subsequent figures.

FIG. 23 shows the following data.
(A) Captured image (B) Defocus map (C) Defocus color map

The (B) defocus map shown in FIG. 23 includes a defocus amount low reliability region.
As described above, the image area unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image area in the defocus map generated by the defocus map generation unit 204, and uses the calculated reliability data. It is output to the image area unit physical quantity reliability correspondence processing execution unit 206 .

The defocus map shown in FIG. 23B includes a defocus amount low reliability region.
In this way, when a defocus amount low reliability area is detected, the image area unit physical quantity reliability correspondence processing execution unit 206 generates a defocus color map that enables identification of the defocus amount low reliability area.

The defocus color map in FIG. 23(C) is a map obtained by coloring the defocus map outputting the defocus amount as a luminance value (for example, 0 to 255), and different colors are set according to the defocus amount. Further, it is a color map in which a specific color is set in the defocus amount low-reliability area.

For example, the area where the defocus amount is 0 to small is set in yellow, the area where the defocus amount is medium is set in green, the area where the defocus amount is large is set in blue, and the defocus amount low reliability area is set in red. is a color map.
Various settings are possible for the color setting.

In this manner, the image-region unit physical quantity reliability correspondence processing execution unit 206 first sets a color according to the defocus amount for each image region calculated by the image-region unit physical quantity reliability calculation unit 205, and then sets the defocus amount. Generate a color map with low-confidence regions set to red.
Furthermore, the image area unit physical quantity reliability correspondence processing execution unit 206 generates and outputs an output image in which the generated color map is superimposed on the captured image.
A specific example of this processing will be described with reference to FIG.

FIG. 24 shows the following data.
(A) Photographed image (C) Defocused color map (D) Output image

(A) A captured image is a captured image (RGB image) generated by the image signal processing unit 212 based on the image signal input from the image information acquisition unit 211 to the image signal processing unit 212 .
(C) The defocus color map sets a different color according to the defocus amount with respect to the defocus map that outputs the defocus amount as a luminance value (for example, 0 to 255), and the defocus amount low reliability region is a specific color (for example, set to red) that can be identified.

For example, a color map that sets yellow for areas where the defocus amount is 0 to small, green for areas with medium defocus amounts, blue for areas with large defocus amounts, and red for low-reliability defocus amount areas. is.

(D) The output image is an output image generated by superimposing (C) the defocus color map on (A) the captured image.

In this manner, the image region unit physical quantity reliability correspondence processing execution unit 206 generates and outputs an output image in which the generated color map is superimposed on the captured image.
This (D) output image is output to the monitor 117 of the imaging apparatus 100, for example.
The user can easily and reliably determine the defocus amount of each area of the captured image by looking at the image output to the monitor 117, and can also reliably determine the low-reliability area of the defocus amount. It becomes possible to recognize

[4-6. (Embodiment 6) Embodiment in which image shooting is performed by controlling exposure time according to reliability of defocus amount for each image area]
Next, as Example 6, an example in which image shooting is performed by controlling the exposure time according to the reliability of the defocus amount for each image area will be described.

An image area with a low reliability of the defocus amount is often an area where the subject distance is not accurately measured, and is likely an image area with a poor S/N ratio of the captured image. For such an image area, it is possible to improve the S/N by setting a long exposure time.
In the embodiment described below, for example, in an image area where the reliability of the defocus amount is low, the exposure time is set long and the image is captured, thereby making it possible to capture a high-quality image. It is an example.

FIG. 25 is a block diagram for explaining the configuration of the sixth embodiment.
FIG. 25 shows an internal configuration example of the digital signal processing unit 108, which is a component of the imaging apparatus 100 described with reference to FIG.

As shown in FIG. 25, the digital signal processing unit 108 includes a phase difference information acquisition unit 201, a defocus amount calculation unit 202, an AF control signal generation unit 203, a defocus map generation unit 204, an image area unit physical quantity reliability calculation unit. 205 , an image area unit physical quantity reliability correspondence processing execution unit 206 , an image information acquisition unit 211 , an image signal processing unit 212 , and an image output unit 213 .

A phase difference information acquisition unit 201 shown in FIG. 25 selects only phase difference detection signals (detection information) that are outputs of phase difference detection pixels from the input signal from the A/D conversion unit 105 .
On the other hand, the image information acquisition unit 211 shown in FIG. 25 selects only image signals (for example, RGB image signals) from the input signals from the A/D conversion unit 105 .

An image (for example, an RGB image) generated by the image signal processing unit 212 is output to the image output unit 213 .
The image output unit 213 outputs the image input from the image signal processing unit 212 . For example, image output to the monitor 117, viewfinder (EVF) 116, and recording device 115 is executed.

Note that, in the sixth embodiment, unlike the first to fifth embodiments described above, the image region unit physical quantity reliability corresponding processing execution unit 206 performs image processing on an image (for example, an RGB image) generated by the image signal processing unit 212. not performed.
The image area unit physical quantity reliability correspondence processing execution unit 206 outputs a shooting control command to the control unit 110 .
Specifically, it outputs an exposure time control command for executing image capturing by exposure control with a long exposure time set for an image region with a low-reliability defocus amount.

It should be noted that the exposure time during image shooting can be controlled to be changed on a pixel-by-pixel basis. For example, by applying the configuration described in Patent Document 4 (Japanese Unexamined Patent Application Publication No. 2011-004088), it is possible to perform exposure time control for each pixel.

Note that the defocus amount reliability calculated in the sixth embodiment is also the reliability of the defocus amount for each pixel area, as described in the first embodiment.
The image area unit physical quantity reliability calculation unit 205 calculates the defocus amount for each image area calculated by the defocus amount calculation unit 202, that is, the image area unit which is one rectangular area in the defocus map shown in FIG. is used to calculate the reliability of the defocus amount for each image area.

The image area unit physical quantity reliability corresponding processing execution unit 206 outputs a shooting control command to the control unit 110 according to the reliability data of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit 205. .
Specifically, an exposure time control command for setting a long exposure time and executing image capturing is output for an image region with a low reliability defocus amount.
A specific example of this processing will be described with reference to FIG.

FIG. 26 shows the following data.
(A) Photographed image (B) Defocus map (C) Shooting control example (exposure time control example for each image area)

The (B) defocus map shown in FIG. 26 includes a defocus amount low-reliability region.
As described above, the image area unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image area in the defocus map generated by the defocus map generation unit 204, and uses the calculated reliability data. It is output to the image area unit physical quantity reliability correspondence processing execution unit 206 .

The defocus map shown in FIG. 26B includes a defocus amount low reliability area.
In this way, when a defocus amount low-reliability region is detected, the image region unit physical quantity reliability correspondence processing execution unit 206 outputs an exposure time control command to the control unit 110 that performs exposure control during image shooting. do.
Specifically, it outputs an exposure time control command for setting the exposure time of an image area with a low reliability defocus amount to be longer than the exposure time of other image areas and performing image shooting.

That is, as shown in FIG. 26C, an example of shooting control (an example of exposure time control for each image area), an image area with a low-reliability defocus amount is specified, and the exposure time of this specified area is set to another image. An exposure time control command is output for setting the exposure time longer than the exposure time of the area to perform image shooting.

By performing such exposure time control, image shooting is performed in which the exposure time of the image area with the low reliability of the defocus amount is longer than the exposure time of the other image areas.
As a result, it is possible to capture a high-quality image with an improved S/N in an image area where the defocus amount has a low reliability.

In this way, the image area unit physical quantity reliability correspondence processing execution unit 206 executes image shooting control for image shooting by controlling the exposure time according to the reliability of the defocus amount in image area units.

As described above, an image area with a low degree of reliability of the defocus amount is often an area where the object distance is not accurately measured, and is likely an image area with a poor S/N of the captured image. For such an image area, it is possible to improve the S/N by setting a long exposure time.
As described above, the sixth embodiment is an embodiment in which high-quality images can be captured by performing image capturing control by setting a long exposure time for an image area with a low degree of reliability of the defocus amount and performing image capturing. be.

It should be noted that the sixth embodiment described with reference to FIGS. 25 and 26 is an embodiment in which photographing control is performed to execute image photographing by setting a long exposure time for an image region with a low degree of reliability of the defocus amount. As a modified example of the sixth embodiment, it is also possible to adopt a configuration that performs the following photographing control.
(Modification 1) A configuration in which photographing control is performed to set a short exposure time for an image region with a high degree of reliability of the defocus amount.
(Modification 2) A configuration in which a long exposure time is set for an image region with a low reliability of the defocus amount, and a short exposure time is set for an image region with a high reliability of the defocus amount.

FIG. 27 shows a configuration example of the digital signal processing unit 108 of the image pickup apparatus that performs the above (Modification 1) shooting control.
The configuration of digital signal processing section 108 shown in FIG. 27 has the same components as the configuration of digital signal processing section 108 shown in FIG. 25 described above.
However, the process executed by the image area unit physical quantity reliability corresponding process execution unit 206 is different.

The unit-image-area physical quantity reliability correspondence processing execution unit 206 of the digital signal processing unit 108 shown in FIG. , and outputs an exposure time control command for setting a short exposure time and executing image shooting.
A specific example of this processing will be described with reference to FIG.

FIG. 28 shows the following data.
(A) Photographed image (B) Defocus map (C) Shooting control example (exposure time control example for each image area)

The (B) defocus map shown in FIG. 28 includes a defocus amount high reliability region.
As described above, the image area unit physical quantity reliability calculation unit 205 calculates the reliability of the defocus amount for each image area in the defocus map generated by the defocus map generation unit 204, and uses the calculated reliability data. It is output to the image area unit physical quantity reliability correspondence processing execution unit 206 .

Based on the reliability data of the defocus amount for each image region input from the image region unit physical quantity reliability calculation unit 205, the image region unit physical quantity reliability corresponding processing execution unit 206 calculates, for example, as shown in FIG. to extract the defocus amount high reliability region.

Specifically, the defocus amount reliability for each image area input from the image area unit physical quantity reliability calculation unit 205 is compared with a predetermined reliability threshold to determine a defocus amount high reliability area. Extract.
For example, using a predefined low reliability threshold Th2,
(Judgment Expression 2) Th2≦Defocus Amount Reliability An image region that satisfies the above judgment expression 2 is extracted as a defocus amount high reliability region.

The defocus map shown in FIG. 28B includes a defocus amount high reliability area.
In this way, when a defocus amount high-reliability region is detected, the image region unit physical quantity reliability correspondence processing execution unit 206 outputs an exposure time control command to the control unit 110 that performs exposure control during image shooting. do.
Specifically, it outputs an exposure time control command for setting the exposure time of an image area with a highly reliable defocus amount to be shorter than the exposure time of other image areas and performing image shooting.

That is, as shown in FIG. 28C, an example of shooting control (an example of exposure time control for each image area), an image area with a highly reliable defocus amount is specified, and the exposure time of this specified area is set to another image. An exposure time control command is output for setting the exposure time to be shorter than the exposure time of the area to perform image shooting.

By performing such exposure time control, image shooting is performed in which the exposure time of an image area with a highly reliable defocus amount is shorter than the exposure time of other image areas.
As a result, it is possible to shoot a high-quality image with reduced subject blur in an image area where the defocus amount is highly reliable.

FIG. 29 shows the above (modified example 2), that is, the exposure time is set long for an image area with a low reliability of the defocus amount, and the exposure time is set short for an image area with a high reliability of the defocus amount. 1 shows a configuration example of a digital signal processing unit 108 of an imaging apparatus that performs control.
The configuration of digital signal processing section 108 shown in FIG. 29 has the same components as the configuration of digital signal processing section 108 shown in FIG. 25 described above.
However, the process executed by the image area unit physical quantity reliability corresponding process execution unit 206 is different.

The unit physical quantity reliability correspondence processing execution unit 206 of the digital signal processing unit 108 shown in FIG. An exposure time control command is output for setting a long exposure time and setting a short exposure time for a high-reliability image area to execute image shooting.

As a result of these processes, both the effects of the sixth embodiment described with reference to FIGS. 25 and 26 and the (modification 1) described with reference to FIGS. 27 and 28 are obtained.
That is, it is possible to improve the S/N ratio by setting a long exposure time for an image region where the reliability of the defocus amount is low.
In addition, it is possible to reduce subject blurring in an image area where the defocus amount is highly reliable.

As described above, the sixth embodiment is an embodiment in which it is possible to capture a high-quality image by performing image capturing by performing exposure time control for each image area according to the reliability of the defocus amount.

[5. Other Examples]
Next, another embodiment will be described.

So far, the following six embodiments have been described with reference to FIGS.
(Embodiment 1) Embodiment in which an image is generated and output in which the reliability of the defocus amount for each image area can be identified (Embodiment 2) The distance ratio between the in-focus subject and other background subjects is displayed on the captured image (Embodiment 3) Example of generating and outputting an image in which mask stability information is superimposed on a photographed image (Embodiment 4) Defocus amount for each image area Example of generating and outputting an image in which a color map outputting corresponding colors is superimposed on a captured image (Embodiment 5) Outputting colors corresponding to the defocus amount of each image area, and defocusing of each image area Example for generating and outputting an image in which a color map that enables identification of the amount reliability is superimposed on a captured image (Embodiment 6) Exposure time is controlled according to the reliability of the defocus amount for each image area. Example of image capturing

Each of these six embodiments can be configured independently, but can also be configured as a device or system having a combination configuration of any of a plurality of embodiments.

Further, for example, in the above-described second embodiment and the like, the following processing has been described as the calculation processing of the reliability of the distance value for each image region, which is executed by the image region unit physical quantity reliability calculation unit 205 .
That is, the image area unit physical quantity reliability calculation unit 205 determines that the reliability of the distance value calculated based on the defocus amount is low for an image area with a low reliability of the defocus amount, and the reliability of the defocus amount is high. For the image area, it is determined that the reliability of the distance value calculated based on the defocus amount is also high.
An example of such processing has been described.

Regarding the reliability determination processing of the distance value for each image region, for example, when the imaging device is a stereo camera with a two-lens structure, the following processing may be performed.
That is, the error amount of the matching process of the parallax map generated from the captured image of the stereo camera is calculated for each image area, and the distance value reliability for each image area is calculated based on the calculated error amount for each image area.
In this process, it is determined that an image area with a large error amount per image area has a low distance value reliability, and an image area with a small error amount per image area has a high distance value reliability.

Further, the processing of the first to sixth embodiments described above may be configured to be continuously executed during operation of the imaging apparatus 100, but may be executed in response to a specific operation on the input unit (operation unit) 118 by the user, for example. It is good also as a structure which carries out.
Further, the processing may be stopped when the focus position has changed significantly. Further, the processing may be stopped when it is determined that the imaging device 100 has moved greatly based on the detection value of the gyro 131 .
Conversely, when the focus position changes significantly, a new process may be started. Further, when it is determined that the imaging device 100 has moved greatly based on the detection value of the gyro 131, new processing may be started.

[6. Summary of the configuration of the present disclosure]
Embodiments of the present disclosure have been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of this disclosure. That is, the present invention has been disclosed in the form of examples and should not be construed as limiting. In order to determine the gist of the present disclosure, the scope of claims should be considered.

In addition, the technique disclosed in this specification can take the following configurations.
(1) an image region physical quantity calculator that calculates physical quantities that change according to the subject distance for each image region that is a segmented region of a captured image;
an image area unit physical quantity reliability calculation unit that calculates the reliability of the image area unit physical quantity calculated by the image area physical quantity calculation unit;
An imaging apparatus having an image area unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image area units calculated by the image area unit physical quantity reliability calculation unit.

(2) The image region physical quantity calculation unit
A defocus amount calculation unit that calculates a defocus amount for each image area,
The image area unit physical quantity reliability calculation unit,
The imaging apparatus according to (1), wherein reliability of the defocus amount for each image area calculated by the defocus amount calculation unit is calculated.

(3) The image area unit physical quantity reliability corresponding processing execution unit,
Based on the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit,
The imaging apparatus according to (2), wherein graphic data indicating the reliability of the defocus amount for each image area is superimposed and displayed on the captured image.

(4) The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1,
The image pickup apparatus according to (3), wherein graphic data that makes it possible to identify an image area whose reliability of the defocus amount for each image area is equal to or lower than the low reliability threshold value Th1 is superimposed on the captured image and displayed.

(5) The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined high reliability threshold value Th2,
The method according to (3) or (4), wherein graphic data that makes it possible to identify an image area whose reliability of the defocus amount for each image area is equal to or higher than the high reliability threshold value Th2 is displayed superimposed on the captured image. imaging device.

(6) The image area unit physical quantity reliability corresponding processing execution unit,
The imaging apparatus according to any one of (2) to (5), wherein mask stability information indicating whether highly accurate mask setting is possible when masking an area other than a specific subject is displayed.

(7) The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1,
When an image area equal to or lower than the low-reliability threshold value Th1 is detected in the captured image,
The imaging apparatus according to (6), wherein mask stability information indicating that highly accurate mask setting is difficult is displayed on the captured image.

(8) The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1,
When an image area equal to or lower than the low-reliability threshold value Th1 is not detected in the captured image,
The imaging apparatus according to (6) or (7), wherein mask stability information indicating that highly accurate mask setting is possible is displayed on the captured image.

(9) The image area unit physical quantity reliability corresponding processing execution unit,
The imaging apparatus according to any one of (2) to (8), wherein a color map in which different colors are set according to the defocus amount of each image area is superimposed and displayed on the captured image.

(10) The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1,
The image pickup apparatus according to (9), wherein a color map that makes it possible to identify image regions whose reliability of the defocus amount for each image region is equal to or lower than the low reliability threshold value Th1 is superimposed on the captured image and displayed. .

(11) The image area unit physical quantity reliability corresponding processing execution unit,
The imaging apparatus according to any one of (2) to (10), wherein exposure time control is performed for each image area according to the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit. .

(12) The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1,
The image pickup apparatus according to (11), wherein the exposure time control is performed such that the exposure time of an image area whose reliability of the defocus amount for each image area is equal to or lower than the low reliability threshold value Th1 is longer than that of other image areas.

(13) The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined high reliability threshold value Th2,
According to (11) or (12), the exposure time control is performed such that the exposure time of the image area whose reliability of the defocus amount for each image area is equal to or higher than the high reliability threshold value Th2 is shorter than that of other image areas. Imaging device.

(14) The image region physical quantity calculation unit
A distance information calculation unit that calculates a distance value for each image area,
The image area unit physical quantity reliability calculation unit,
The imaging apparatus according to any one of (1) to (13), wherein the reliability of the distance information for each image area calculated by the distance information calculation unit is calculated.

(15) The image area unit physical quantity reliability corresponding processing execution unit,
The imaging device according to (14), which displays distance ratios of subjects at a plurality of different distances.

(16) The image area unit physical quantity reliability corresponding processing execution unit,
selecting an image region in which the reliability of the distance information in the image region unit calculated by the image region unit physical quantity reliability calculation unit is equal to or higher than a predetermined threshold value, and calculating a distance ratio between subjects in the selected image region; , the imaging device according to (14) or (15) for displaying on the captured image of the imaging device.

(17) An image processing method executed in an imaging device,
an image region physical quantity calculation step in which the image region physical quantity calculation unit calculates a physical quantity that changes according to the subject distance for each image region that is a segmented region of the captured image;
an image region unit physical quantity reliability calculation step in which the image region unit physical quantity reliability calculation unit calculates the reliability of the image region unit physical quantity calculated in the image region physical quantity calculation step;
an image area unit physical quantity reliability correspondence process execution step in which the image area unit physical quantity reliability correspondence process execution unit executes control processing according to the reliability of the image area unit physical quantity reliability calculated in the image area unit physical quantity reliability calculation step; An image processing method that performs

(18) A program for executing image processing in an imaging device,
an image region physical quantity calculation step of causing an image region physical quantity calculation unit to calculate a physical quantity that changes according to a subject distance for each image region that is a segmented region of a captured image;
an image region unit physical quantity reliability calculation step for causing an image region unit physical quantity reliability calculation unit to calculate the reliability of the image region unit physical quantity calculated in the image region physical quantity calculation step;
An image area unit physical quantity reliability correspondence process execution step for causing an image area unit physical quantity reliability correspondence process execution unit to execute a control process according to the reliability of the image area unit physical quantity reliability calculated in the image area unit physical quantity reliability calculation step. program to run.

It should be noted that the series of processes described in the specification can be executed by hardware, software, or a composite configuration of both. When executing processing by software, a program recording the processing sequence is installed in the memory of a computer built into dedicated hardware and executed, or the program is loaded into a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be pre-recorded on a recording medium. In addition to being installed in a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet and installed in a recording medium such as an internal hard disk.

In addition, the various types of processing described in the specification may not only be executed in chronological order according to the description, but may also be executed in parallel or individually according to the processing capacity of the device that executes the processing or as necessary. Further, in this specification, a system is a logical collective configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same housing.

As described above, according to the configuration of the embodiment of the present disclosure, the reliability of the defocus amount and the distance value for each image area of the captured image is calculated, and the display processing of the graphic data indicating the reliability, An apparatus and method for executing subject distance ratio display processing, exposure time control processing, and the like are realized.
Specifically, for example, the defocus amount and the distance value are calculated for each image area of the captured image, and the reliability of the calculated defocus amount and distance value for each image area is calculated. Control is executed according to the reliability of the defocus amount and the distance value. For example, it executes a process of superimposing and displaying graphic data indicating the reliability of the defocus amount for each image area on the captured image, a process of displaying the distance ratio of the object, a process of controlling the exposure time, and the like.
With this configuration, the reliability of the defocus amount and distance value for each image area of the captured image is calculated, and the display processing of the graphic data indicating the reliability, the distance ratio display processing of the subject, the exposure time control processing, etc. are executed. An apparatus and method are realized.

REFERENCE SIGNS LIST 100 imaging device 101 focus lens 102 zoom lens 103 imaging element 104 analog signal processing section 105 A/D conversion section 106 timing generator (TG)
107 vertical driver 108 digital signal processor (DSP)
110 control unit 112a AF control unit 112b zoom control unit 113 motor 115 recording device 116 viewfinder 117 monitor 118 input unit (operation unit)
122 image area 131 gyro 151 phase difference detection pixel 152 image area 201 phase difference information acquisition unit 202 defocus amount calculation unit 203 AF control signal generation unit 204 defocus map generation unit 205 image area unit physical quantity reliability calculation unit 206 image area unit Physical quantity reliability corresponding processing execution unit 211 Image information acquisition unit 212 Image signal processing unit 213 Image output unit 221 Distance information calculation unit

Claims

an image region physical quantity calculator that calculates physical quantities that change according to the subject distance for each image region that is a segmented region of a captured image;
an image area unit physical quantity reliability calculation unit that calculates the reliability of the image area unit physical quantity calculated by the image area physical quantity calculation unit;
An imaging apparatus having an image area unit physical quantity reliability corresponding processing execution unit that executes control processing according to the reliability of the physical quantity in image area units calculated by the image area unit physical quantity reliability calculation unit.
The image region physical quantity calculation unit
A defocus amount calculation unit that calculates a defocus amount for each image area,
The image area unit physical quantity reliability calculation unit,
2. The imaging apparatus according to claim 1, wherein reliability of the defocus amount for each image area calculated by the defocus amount calculation unit is calculated.
The image area unit physical quantity reliability corresponding processing execution unit,
Based on the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit,
3. The imaging apparatus according to claim 2, wherein graphic data indicating the reliability of the defocus amount for each image area is superimposed and displayed on the captured image.
The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1,
4. The image pickup apparatus according to claim 3, wherein graphic data that makes it possible to identify an image area whose reliability of the defocus amount for each image area is equal to or lower than the low reliability threshold value Th1 is superimposed on the captured image and displayed.
The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined high reliability threshold value Th2,
4. The image pickup apparatus according to claim 3, wherein graphic data that makes it possible to identify an image area whose reliability of the defocus amount for each image area is equal to or higher than the high reliability threshold value Th2 is superimposed on the captured image and displayed.
The image area unit physical quantity reliability corresponding processing execution unit,
3. The imaging apparatus according to claim 2, wherein mask stability information indicating whether highly accurate mask setting is possible when masking an area other than a specific subject is displayed.
The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1,
When an image area equal to or lower than the low-reliability threshold value Th1 is detected in the captured image,
7. The imaging apparatus according to claim 6, wherein mask stability information indicating that highly accurate mask setting is difficult is displayed on the captured image.
The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1,
When an image area equal to or lower than the low-reliability threshold value Th1 is not detected in the captured image,
7. The imaging apparatus according to claim 6, wherein mask stability information indicating that highly accurate mask setting is possible is displayed on the captured image.
The image area unit physical quantity reliability corresponding processing execution unit,
3. The imaging apparatus according to claim 2, wherein a color map in which different colors are set according to the defocus amount of each image area is superimposed and displayed on the captured image.
The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold value Th1,
10. The image pickup apparatus according to claim 9, wherein a color map is superimposed on the captured image and displayed so that image regions whose reliability of the defocus amount for each image region is equal to or lower than the low reliability threshold value Th1 can be identified. .
The image area unit physical quantity reliability corresponding processing execution unit,
3. The imaging apparatus according to claim 2, wherein exposure time control is performed for each image area according to the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit.
The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined low reliability threshold Th1,
12. The imaging apparatus according to claim 11, wherein the exposure time control is performed such that the exposure time of an image area whose reliability of the defocus amount for each image area is equal to or less than the low reliability threshold value Th1 is longer than that of other image areas.
The image area unit physical quantity reliability corresponding processing execution unit,
comparing the reliability of the defocus amount for each image area calculated by the image area unit physical quantity reliability calculation unit with a predetermined high reliability threshold value Th2,
12. The imaging apparatus according to claim 11, wherein the exposure time control is performed such that the exposure time of image areas whose reliability of the defocus amount for each image area is equal to or higher than the high reliability threshold value Th2 is shorter than that of other image areas.
The image region physical quantity calculation unit
A distance information calculation unit that calculates a distance value for each image area,
The image area unit physical quantity reliability calculation unit,
2. The imaging apparatus according to claim 1, wherein reliability of the distance information for each image area calculated by the distance information calculation unit is calculated.
The image area unit physical quantity reliability corresponding processing execution unit,
15. The imaging device according to claim 14, which displays distance ratios of subjects at a plurality of different distances.
The image area unit physical quantity reliability corresponding processing execution unit,
selecting an image region in which the reliability of the distance information in the image region unit calculated by the image region unit physical quantity reliability calculation unit is equal to or higher than a predetermined threshold value, and calculating a distance ratio between subjects in the selected image region; 15. The imaging device according to claim 14, wherein the image is displayed on the captured image of the imaging device.
An image processing method executed in an imaging device,
an image region physical quantity calculation step in which the image region physical quantity calculation unit calculates a physical quantity that changes according to the subject distance for each image region that is a segmented region of the captured image;
an image region unit physical quantity reliability calculation step in which the image region unit physical quantity reliability calculation unit calculates the reliability of the image region unit physical quantity calculated in the image region physical quantity calculation step;
an image area unit physical quantity reliability correspondence process execution step in which the image area unit physical quantity reliability correspondence process execution unit executes control processing according to the reliability of the image area unit physical quantity reliability calculated in the image area unit physical quantity reliability calculation step; An image processing method that performs
A program for executing image processing in an imaging device,
an image region physical quantity calculation step of causing an image region physical quantity calculation unit to calculate a physical quantity that changes according to a subject distance for each image region that is a segmented region of a captured image;
an image region unit physical quantity reliability calculation step for causing an image region unit physical quantity reliability calculation unit to calculate the reliability of the image region unit physical quantity calculated in the image region physical quantity calculation step;
An image area unit physical quantity reliability correspondence process execution step for causing an image area unit physical quantity reliability correspondence process execution unit to execute a control process according to the reliability of the image area unit physical quantity reliability calculated in the image area unit physical quantity reliability calculation step. program to run.