JP2023033355A - Image processing device and control method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device for assisting capturing of images with various depths of field, and to provide a control method therefor.

SOLUTION: An image processing device is configured to acquire distance information indicative of a distribution of distances to scenes captured in an image, and then determine a plurality of aperture values to be used in bracket shooting with different apertures on the basis of the distance information.

SELECTED DRAWING: Figure 3

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to an image processing apparatus and its control method.

A method is known in which the background and foreground are blurred by photographing the main subject with a shallow (narrow) depth of field to obtain an image in which the main subject is in harmony. We also proposed a method to generate an image with a depth of field corresponding to a specified virtual aperture value by combining blurred images of the foreground and background and a focused image of the main subject. (Patent Document 1).

Japanese Patent Application Laid-Open No. 2003-209727

By using the method of Patent Document 1, it is possible to obtain an image with a depth of field that is not restricted by the imaging environment or imaging apparatus. However, since the depth of field depends not only on the aperture value, but also on the focusing distance and the focal length (angle of view) of the lens, it is easy to determine the virtual aperture value to set to obtain the desired image. not. Also, since a plurality of images are synthesized, the quality of the synthesized image deteriorates when the subject moves.

The present invention has been made in view of such problems of the prior art. SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide an image processing apparatus and a control method thereof that support shooting of images having various depths of field.

The above-mentioned object is an acquisition means for acquiring a photographed image and distance information representing the distribution of the distance of the scene photographed in the image, and a plurality of diaphragms used in bracket photography in which the aperture is changed based on the distance information. and control means for determining the value.

According to the present invention, it is possible to provide an image processing apparatus and a control method thereof that support shooting of images having various depths of field.

1 is a block diagram showing a functional configuration example of a digital camera as an example of an image processing apparatus according to an embodiment; FIG. Flowchart relating to operations during bracket shooting in the embodiment Schematic diagram for explaining the operation of the scene analysis unit Schematic diagram for explaining the operation of the scene analysis unit Schematic diagram for explaining the operation of the bokeh amount calculator Schematic diagram for explaining the operation of the bokeh amount calculator Diagram for explaining how the aperture value is determined Schematic diagram for explaining the reliability of distance information

Exemplary embodiments of the invention will now be described in detail with reference to the accompanying drawings. It should be noted that the invention is not limited to the described embodiments. Also, not all of the components described in the embodiments are essential to the present invention. Individual functional blocks in the embodiments can be realized by hardware such as programmable logic and circuit components, software executable by a programmable processor, or a combination of these hardware and software. Also, one functional block may be realized by multiple pieces of hardware. Also, one piece of hardware may implement a plurality of functional blocks. Also, one or more functional blocks may be implemented by one or more programmable processors (CPU, MPU, etc.) executing a computer program loaded into memory.

The invention will now be described in detail on the basis of its exemplary embodiments with reference to the accompanying drawings. It should be noted that the described embodiments are merely examples and do not limit the scope of the present invention. For example, an embodiment in which the present invention is applied to a digital camera will be described below. However, a digital camera is only one example of an image processing device to which the present invention can be applied. The present invention can be implemented in any electronic device. Such electronic devices include, but are not limited to, imaging devices such as digital cameras and digital video cameras, as well as personal computers, tablet terminals, mobile phones, game machines, drive recorders, robots, and drones. Note that the present invention does not require a photographing function, and can be implemented by a device that obtains an image from an external imaging device, determines photographing conditions, and controls the photographing operation of the external photographing device.

(Configuration of digital camera)
FIG. 1 is a block diagram showing a functional configuration example of a digital camera 100 according to an embodiment. The digital camera 100 can shoot and record moving and still images. Each functional block within the digital camera 100 is communicably connected to each other via a bus 160 . The operation of the digital camera 100 is realized by one or more programmable processors of the main control unit 151 reading, for example, a program stored in the ROM 155 into the RAM 154 and executing the program to control each functional block. Moreover, unless otherwise specified, each functional block can be implemented by a hardware circuit such as an ASIC, or by one or more programmable processors reading a program into a memory and executing it.

The taking lens 101 (lens unit) includes a fixed 1st group lens 102, a zoom lens 111, an aperture 103, a fixed 3rd group lens 121, a focus lens 131, a zoom motor (ZM) 112, an aperture motor (AM) 104, and a focus motor ( FM) 132. A fixed 1st group lens 102, a zoom lens 111, an aperture 103, a fixed 3rd group lens 121, and a focus lens 131 constitute a photographing optical system. Although the lenses 102, 111, 121, and 131 are illustrated as one lens for convenience, they may each be composed of a plurality of lenses. Also, the photographing lens 101 may be configured as an interchangeable lens that can be removed from the imaging device 100 .

The aperture control unit 105 controls the operation of the aperture motor 104 according to the command from the main control unit 151 to change the aperture diameter of the aperture 103 .
The zoom control unit 113 controls the operation of the zoom motor 112 according to commands from the main control unit 151 to change the focal length (angle of view) of the photographing lens 101 .

The focus control unit 133 calculates the defocus amount and the defocus direction of the photographing lens 101 based on the phase difference between the pair of focus detection signals obtained from the image sensor 141, for example. The focus control unit 133 then converts the defocus amount and defocus direction into the drive amount and drive direction of the focus motor 132 . The focus control unit 133 controls the operation of the focus motor 132 based on the drive amount and the drive direction, and drives the focus lens 131 to control the focus state of the photographing lens 101 .

In this manner, the focus control unit 133 performs automatic focus detection (AF) using a phase difference detection method. may be executed. Alternatively, phase difference detection AF may be performed using a focus detection signal obtained from an AF sensor provided separately from the imaging device 141 . In the AF operation of the focus control section 133, the focus detection area can be set to the area of the main subject detected by the image processing section 152, which will be described later.

A subject image formed on the imaging plane of the imaging element 141 by the imaging lens 101 is converted into an electric signal (image signal) by a photoelectric conversion element of each of a plurality of pixels arranged on the imaging element 141 . In this embodiment, m pixels in the horizontal direction and n pixels in the vertical direction (where n and m are plural) are arranged in a matrix in the image sensor 141, and each pixel has two photoelectric conversion elements (photoelectric conversion regions). ) is provided. Signal reading from the imaging device 141 is controlled by the sensor control section 143 according to instructions from the main control section 151 .

The two photoelectric conversion regions of each pixel are called region A and region B. An image composed of a group of image signals read out from region A of each pixel is called A image, and a group of image signals read out from region B of each pixel. is called a B-image. An image obtained by adding the A image and the B image pixel by pixel is called an A+B image. The A and B images form a parallax image pair. A+B images are used for display and recording. Further, the A image and the B image are used to generate a focus detection signal used in phase difference detection AF and to generate a distance map.

An image signal read out from the imaging element 141 is supplied to the signal processing section 142 . The signal processing unit 142 applies signal processing such as noise reduction processing, A/D conversion processing, and automatic gain control processing to the image signal, and outputs the image signal to the sensor control unit 143 as image data. The sensor control unit 143 stores the image data received from the signal processing unit 142 in a RAM (random access memory) 154 .

When recording the image data stored in the RAM 154, the main control unit 151 adds, for example, a predetermined header to the image data to generate a data file according to the recording format. At this time, the main control unit 151 encodes the image data in the compression/decompression unit 153 as necessary, and stores the encoded data in the data file. The main controller 151 records the generated data file in a recording medium 157 such as a memory card.

When displaying the image data stored in the RAM 154, the main control unit 151 scales the image data in the image processing unit 152 so as to fit the display size of the display unit 150, and then uses the RAM 154 as a video memory. Write to the area (VRAM area). The display unit 150 reads image data for display from the VRAM area of the RAM 154, and displays it on a display device such as an LCD or an organic EL display. The display unit 150 also displays the detection result of the main subject (moving object) detected by the image processing unit 152 (the frame indicating the main subject area, etc.).

The digital camera 100 causes the display unit 150 to function as an electronic viewfinder (EVF) by immediately displaying the captured moving image on the display unit 150 during moving image capturing (shooting standby state or moving image recording). A moving image and its frame images displayed when the display unit 150 functions as an EVF are called a live view image or a through image. Further, when a still image is captured, the digital camera 100 displays the last captured still image on the display unit 150 for a certain period of time so that the user can confirm the result of the capturing. These display operations are also realized under the control of the main control unit 151 .

A compression/decompression unit 153 encodes and decodes image data. For example, when recording still images and moving images, image data and audio data are encoded by a predetermined encoding method. Also, when reproducing a still image data file or a moving image data file recorded on the recording medium 157 , the compression/decompression unit 153 decodes the encoded data and stores it in the RAM 154 .

The RAM 154 is used as system memory, video memory, buffer memory, etc. for executing programs.
The ROM 155 stores programs that can be executed by the processor of the main control unit 151, various setting values, unique information of the digital camera 100, GUI data, and the like. The ROM 155 may be electrically rewritable.

The operation unit 156 is a general term for a group of input devices such as switches, buttons, keys, and touch panels for the user to input instructions to the digital camera 100 . An input through the operation unit 156 is detected by the main control unit 151 through the bus 160, and the main control unit 151 controls each unit in order to realize operations according to the input.

The main control unit 151 has one or more programmable processors such as a CPU and an MPU, and controls each unit by reading a program stored in the ROM 155 into the RAM 154 and executing it, thereby realizing the functions of the digital camera 100 . The main control unit 151 also executes AE processing for automatically determining exposure conditions (shutter speed or accumulation time, aperture value, sensitivity) based on subject brightness information. Information on subject brightness can be acquired from the image processing unit 152, for example. The main control unit 151 can also determine exposure conditions based on luminance information of a specific subject area such as a person's face.

The main control unit 151 fixes the aperture 103 during moving image shooting, and controls the exposure based on the electronic shutter speed (accumulation time) and the magnitude of the gain. The main control unit 151 notifies the sensor control unit 143 of the determined sense of accumulation and the magnitude of the gain. The sensor control unit 143 controls the operation of the imaging device 141 so that shooting is performed according to the notified exposure conditions.

The distance map generator 161 (distance detector) generates a distance map using image data stored in the RAM 154, for example. The distance map is information indicating the distribution of distances (object distances) from the digital camera 100 to objects existing within the shooting range. The distance map may be in the form of, for example, a two-dimensional image in which the brightness value represents the object distance, and is sometimes called a depth map distance image, a depth image, or the like. A distance map can be generated by a known method. For example, the distance map generation unit 161 obtains the defocus amount (the amount of deviation from the in-focus position of the focus lens and the direction of deviation) at each pixel position from the amount of image deviation of the parallax images (images A and B described above). be able to. Since the defocus amount represents the amount of defocus based on the current subject distance, the defocus amount can be regarded as distance information. Of course, the in-focus position of the focus lens may be obtained based on the defocus amount, and the object distance corresponding to the in-focus position may be obtained. Note that parallax images may be acquired by using the imaging device 100 as a multi-view camera such as a stereo camera, or parallax images may be acquired from a storage medium or an external device.

Also, the distance map can be generated without using parallax images. By finding the focus lens position that maximizes the contrast evaluation value for each pixel, the subject distance can be obtained for each pixel. In addition, from the image data obtained by shooting the same scene multiple times with different focal distances and the point spread function (PSF) of the optical system, distance information for each pixel is calculated based on the correlation between the amount of bokeh and the distance. can also be asked for. The distance map generator 161 may generate a distance map for the entire image, or may generate a distance map only for a partial area of the image that is necessary for detecting a moving object. Distance map generator 161 stores the generated distance map in RAM 154 . The distance map is referred to by the image processing unit 152 .

Furthermore, the distance map generation unit 161 can calculate the reliability for each area of the distance map and store it together with the distance map. There is no particular limitation on the method of calculating the reliability. For example, when generating a distance map using parallax images, in order to obtain an image shift amount between parallax images, a correlation amount (similarity) such as SAD is calculated while changing the relative shift amount, and the correlation is The amount of shift that maximizes (minimizes the amount of correlation) is detected as the amount of image deviation. It is considered that the larger the difference between the calculated average value and the maximum value of the correlation amount, the higher the reliability of the detected image shift amount (defocus amount). Therefore, the difference between the average value and the maximum value of correlation amounts calculated at each pixel position can be used as the pixel position reliability. Note that the distance map generation unit 161 may generate the distance map by obtaining the subject distance and its reliability for each small area (pixel block) instead of obtaining it for each pixel.

The scene analysis unit 162 analyzes the shooting scene from the distance map generated by the distance map generation unit 161, the image used to generate the distance map, and the imaging conditions thereof. The scene analysis unit 162 acquires, for example, the position (image height) and contrast information of the subject in focus, and the recognition information of the subject by scene analysis. For example, the scene analysis unit 162 detects the range of the position (image height) where the in-focus subject area exists from the distance map. Also, the scene analysis unit 162 can apply a band-pass filter to the image of the focused subject area to detect the edge strength, and acquire the integrated value as the contrast information. Also, the scene analysis unit 162 determines whether or not the focused subject region is a human body based on the result of subject detection by the image processing unit 152, and uses the determination result as recognition information. The scene analysis unit 162 saves the acquired information in the RAM 154 .

The blur amount calculation unit 163 calculates the blur amount of the background (out-of-focus subject) based on the scene analysis result stored in the RAM 154 . Details of the operation of the blur amount calculation unit 163 will be described later.

The image processing unit 152 applies predetermined image processing to the image data accumulated in the RAM 154 . Image processing applied by the image processing unit 152 includes, but is not limited to, so-called development processing such as white balance adjustment processing, color interpolation (demosaicing) processing, and gamma correction processing, as well as signal format conversion processing and scaling processing. . Furthermore, the image processing unit 152 detects a region having specific characteristics as a subject region. A region with specific features may be, for example, a human body region or a human face region.

Information about the detected subject area may be used for other image processing (for example, white balance adjustment processing, subject brightness information generation processing, etc.). When the focus control unit 133 performs contrast detection AF, the image processing unit 152 can generate a contrast evaluation value and supply it to the focus control unit 133 . The image processing unit 152 stores the processed image data, information about the subject area, and the like in the RAM 154 .

FIG. 2 is a flowchart relating to the shooting operation in the focus bracket mode of the digital camera 100 of this embodiment. The focus bracket mode may be set by operating a menu screen through the operation unit 156, for example.

The processing shown in FIG. 2 may be executed in response to a shooting preparation instruction being input through the operation unit 156, for example, in the shooting standby state. The shooting preparation instruction may be, for example, a half-press of the shutter button. It is assumed that moving image shooting is performed in the shooting standby state, and the display unit 150 functions as an EVF.

In S201, the main control unit 151 supplies the moving image data shot for EVF display to the distance map generation unit 161, and the distance map generation unit 161 generates distance information. Note that the main control unit 151 drives the focus lens 131 through the focus control unit 133 or supplies the data of the A image and the B image to the distance map generation unit 161 according to the method of generating the distance map. Distance map generation section 161 outputs the generated distance information to scene analysis section 162 . Also, the main control unit 151 supplies the above-described A+B image to the image processing unit 152 . The image processing unit 152 applies subject detection processing to the A+B image, and generates the position, size, reliability, etc. of each detected subject area as subject detection results. Also, the image processing unit 152 generates an image for EVF display from the A+B image.

In S<b>202 , the scene analysis unit 162 acquires distance information from the distance map generation unit 161 . The scene analysis unit 162 also acquires from the main control unit 151 the image capturing conditions (focus distance, aperture value, shutter speed, image capturing sensitivity, etc.) used to generate the distance map. Furthermore, the scene analysis unit 162 acquires the detection result of the subject area from the image processing unit 152 .

In S203, the scene analysis unit 162 executes scene analysis processing. The scene analysis unit 162 first divides the distance range into multiple parts using the distance information. At this time, the scene analysis unit 162 can generate, for example, a subject distance histogram from the distance information, and divide the distance range based on the generated histogram. The number of histogram classes can be determined in advance. For example, when there is a peak (maximum point) in the frequency of the histogram, the scene analysis unit 162 can divide the distance range so as to include each peak, with each peak as the representative distance of the subject. The peak of the histogram can be detected using the same method as the peak detection of the signal waveform.

For example, there are three peaks 322-324 in the frequency of histogram 501 in FIG. Therefore, the scene analysis unit 162 divides the distance range (range 1 to range 3) at the position where the count value first becomes minimum on the short distance side from each peak.

On the other hand, when there is one peak as shown in FIG. 4, the scene analysis unit 162 can divide the distance range (background distance range) other than the distance range (range 1) including the peak into a plurality of parts. For example, the scene analysis unit 162 may divide the distance range of the background by predetermined distances. Alternatively, the division unit may be changed according to conditions. For example, the scene analysis unit 162 can divide more finely when the distance of the focused subject is close (a predetermined distance or less) than when it is not. As a result, bracket photography with a finely controlled depth of field can be achieved in photography with a shallow depth of field, such as macro photography.

Further, for example, when there is a distance section in which the count value continues with little fluctuation in the distance range of the background, the dividing method may be changed depending on whether the section corresponds to the same subject or a plurality of subjects. When it is determined that the images correspond to the same subject, the scene analysis unit 162 can divide the images more finely than usual. On the other hand, if it is determined to correspond to different subjects, the scene analysis unit 162 can divide one section so that two subjects are not included. Whether or not a certain distance range in the distance histogram corresponds to the same subject can be determined based on the distribution of pixels belonging to that distance range in the distance map. For example, if the distribution is concentrated in a certain range, it can be determined that they correspond to the same subject.

Next, the scene analysis unit 162 analyzes the position, size, and contrast of the subject area within the image. For example, the scene analysis unit 162 can detect a subject existing within a predetermined distance range including the peak position for each peak of the distance information histogram, and detect the area of the subject and the position in the image. Here, the position of the subject area is detected as the image height of the representative position (for example, the barycentric position) of the subject area. The image height is the distance with the origin at the intersection of the optical axis of the imaging optical system and the imaging surface. In addition, the scene analysis unit 162 determines whether each of the detected subject regions is a person region based on the subject detection result by the image processing unit 152 . Note that the scene analysis section 162 may use the subject area and its position detected by the image processing section 152 instead of detecting the subject area and its position in the image using the distance information.

ｆｒ（ｉ，ｊ）：個々の画素のエッジ強度値
Ｆｒ：コントラスト情報 The scene analysis unit 162 also acquires contrast information for each subject area. The contrast information may be edge strength information, for example. The scene analysis unit uses the integrated value of the edge intensity calculated by applying the spatial band-pass filtering process to each pixel of a rectangular area of a predetermined size in the subject area as the contrast information of the subject area. This contrast information can be represented by the following formula (1), and the larger the value of the contrast information, the higher the contrast of the subject area.

fr(i, j): edge intensity value of individual pixels Fr: contrast information

Then, the scene analysis unit 162 uses the divided distance range information, the position in the image of each subject area, the contrast information, and the information as to whether or not the area is determined to be a human area as scene information, and the blur amount calculation unit 163 output to

In S204, the blur amount calculation unit 163 calculates the blur amount for each out-of-focus subject or divided range based on the scene information. The blur amount calculation unit 163 can calculate the blur amount based on one or more pieces of scene information.
First, an example of calculation of the blur amount based on the position of the in-focus subject area will be described. For example, the blur amount calculation unit 163 can regard the subject area with the largest contrast information among the scene information as the in-focus subject area.

FIG. 6 shows a captured image 601 in which the position (image height) of the focused object region 602 is small, a captured image 611 in which the position (image height) of the focused object region 612 is large, and an image height position 621 in the image. The image height is the distance from the point of intersection between the image (imaging device) and the optical axis of the imaging optical system. Here, it is assumed that the imaging device and the optical axis intersect at the center of the imaging device. Also, the image coordinates are expressed in an orthogonal coordinate system with the coordinates of the pixel 623 at the upper left corner of the image as the origin (0, 0).

In this case, the image height is maximum at the four vertices of the image, and if the image size is horizontal M pixels and vertical N pixels, the maximum image height H _max is H _max =√(M ² +N ² )/2. Also, the image height H _s at the position 624 (x, y) of the focused object is H _s =√(|(x−1)−M/2| ² +|(y−1)−N/2| ² ) is. Here, the magnitude of the image height is evaluated by the ratio H _s /H _max ×100[%] with the maximum value being 100% (100%).

When the image height of the focused subject area is small (for example, 0≦image height≦30%), the focused subject exists near the center of the image. In this case, it is highly likely that the photographer is mainly focusing on the in-focus subject. Therefore, the bokeh amount calculation unit 163 calculates the bokeh amount such that the bracket photographing is performed under the condition that the out-of-focus subject has a large blur amount. As a result, a captured image in which the focused subject stands out is obtained.

On the other hand, when the image height of the in-focus subject area is large (for example, 30%<image height), the photographer may be focusing not only on the in-focus subject but also on its background (out-of-focus subject). It is considered to be of high quality. Therefore, the amount of blur is calculated so that bracket photography can be performed under the condition that blur is reduced even for an out-of-focus subject. This makes it possible to obtain a captured image that also makes use of background information.

Reference numeral 501 in FIG. 5 indicates an example of the relationship between the position (image height) of the focused subject area and the blur amount δ of the background (out-of-focus subject) calculated by the blur amount calculation unit 163 . Here, thresholds th1 and th2 are set for the image height [%] of the in-focus subject area,
・When 0 ≤ image height [%] < th1, blur amount δ is maximized (upper limit)
・When th2 < image height [%] ≤ 100, the blur amount δ is the minimum (lower limit value)
· th1 ≤ image height [%] ≤ th2 shows an example in which the blur amount δ is reduced in inverse proportion to the image height. Note that th1 and th2 can be obtained in advance, for example, experimentally. Note that the relationship between the image height and the blur amount δ is not limited to the relationship shown in 501 as long as the background blur amount is basically large in a range where the image height is small.

Next, an example of calculating the blur amount based on the contrast value of the out-of-focus subject area will be described. When the contrast value of the out-of-focus subject area is large, the blur amount calculation unit 163 gives priority to keeping the contrast of the out-of-focus subject rather than making the in-focus subject stand out, and reduces the blur of the out-of-focus subject. On the other hand, when the contrast value of the out-of-focus subject area is small, the blur amount calculation unit 163 gives priority to making the focused subject stand out rather than leaving the contrast of the out-of-focus subject, and increases the blur of the out-of-focus subject. do.

Reference numeral 502 in FIG. 5 indicates an example of the relationship between the contrast value Fr of the out-of-focus subject area and the background blur amount δ calculated by the blur amount calculation unit 163 . Here, thresholds th3 and th4 are set for the contrast value Fr of the out-of-focus object,
・Bokeh amount δ is maximized (upper limit value) when 0≦out-of-focus subject area contrast value Fr<th3
・When th4<contrast value of out-of-focus subject area Fr≦100, blur amount δ is minimized (lower limit)
When th3≦contrast value Fr of out-of-focus object region≦th4, an example is shown in which the blur amount δ is reduced in inverse proportion to the contrast value Fr of the out-of-focus object region. Note that th3 and th4 can be obtained in advance, for example, experimentally. Here, the amount of blur is calculated so that the amount of blur when the contrast value of the out-of-focus subject area is the first value is less than or equal to the amount of blur when the contrast value is the second value smaller than the first value. An example was given, but this is only an example.

Next, an example of calculation of the blur amount of the background (out-of-focus subject) when the in-focus subject area is determined to be the area of a person will be described. If the in-focus subject area is determined to be the area of a person, and if the in-focus subject area occupies a large proportion of the entire image (e.g., 50% or more), the in-focus subject is focused on and photographed. Probability is high. In this case, similarly to the case where the image height of the in-focus object area is small, the blur amount of the background (out-of-focus object) is calculated to be large so that the in-focus object stands out in an image.

For example, the blur amount calculation unit 163
・Bokeh amount δ is maximum (upper limit) at 50 ≤ the ratio [%] of the in-focus subject to the image
If the ratio [%] of the focused object in the image is <50, the amount of blur δ can be calculated so as to decrease in proportion to the ratio [%] of the focused object in the image.

If the in-focus subject area is determined to be a human area and the peak count value of the in-focus subject area in the distance histogram is greater than the peak count value of the out-of-focus subject, the in-focus subject area It may be determined that there is a high possibility of photographing with the main subject. For example, this corresponds to the case where a distance histogram as indicated by 321 in FIG. 3 is obtained.

In this case, the blur amount calculation unit 163
・Bokeh amount δ is maximized (upper limit) when th5 < the ratio [%] of the in-focus subject in the image
・Bokeh amount δ is the minimum (lower limit value) when the ratio [%] of the focused subject in the image < th6
When th6≤ratio [%] of the focused subject in the image≤th5, the blur amount δ can be calculated so as to increase in proportion to the percentage [%] of the focused subject in the image. Note that th5 and th6 can be obtained in advance, for example, experimentally.

On the other hand, if the in-focus subject area is not determined to be a person's area, the background blur amount is calculated for each distance range divided by scene analysis based on the distance of the in-focus subject and the distance distribution of the subject. can be calculated.
For example, when the distance to the in-focus subject is very short (for example, 30 cm or less), there is a high possibility that macro photography is being performed. The depth of field is very shallow when shooting close-up, such as macro photography. Therefore, the blur amount δ of the background is basically calculated to be small.

For example, the blur amount calculation unit 163
・The shortest shooting distance is the distance of the in-focus subject, and the amount of bokeh δ is the lower limit.
When the distance of the in-focus subject is th7, the blur amount δ becomes a predetermined value (for example, the upper limit value/3),
In the range of shortest photographable distance<distance of focused subject<th7, the amount of blur δ can be calculated so that the amount of blur δ increases in proportion to the distance of the focused subject. Note that th7 can be predetermined as, for example, the shortest shooting distance of the lens +1 m. In this way, when it is determined that close-up photography is performed, the blur amount is calculated within a range in which the maximum value of the blur amount δ is smaller than the upper limit value. As a result, in close-up photography with a shallow depth of field, bracket photography with finely controlled depth of field can be realized.

For a plurality of distance ranges corresponding to the same subject, the blur amount calculation unit 163 calculates the blur amount δ for each distance range so that the absolute amount of blur changes within a small range.
On the other hand, distance ranges corresponding to different subjects are calculated by adding a difference to the amount of blur δ so that the difference in the amount of blur for each subject becomes clear.

δ_ｎ：情報毎に決定したボケ量
Δ：撮影で使用するボケ量 From one or more blur amounts δ calculated for each out-of-focus object or divided range as described above, the blur amount calculator 163 calculates the blur amount Δ for finally determining the bracket shooting conditions. The blur amount calculator 163 can calculate the blur amount Δ using, for example, the following equation (2). Here, α _n represents the weight set for the calculated blur amount for each piece of information, and n represents the number of acquired information. The weight α _n is determined in advance according to the combination of the blur amounts δ used to calculate the blur amount Δ, and is stored in the ROM 155, for example. The weight α _n can be determined according to the priority of each of the plurality of blur amounts δ.

_δn : Amount of bokeh determined for each information Δ: Amount of bokeh used for shooting

For example, when the image height of the in-focus subject is large and the contrast of the out-of-focus subject is high, the area of the out-of-focus subject that is not hidden by the in-focus subject is large, and the contrast of the out-of-focus subject is preserved when shooting. is considered suitable. As described above, the blur amount δ of the out-of-focus subject is calculated to be small both when the image height of the in-focus subject is large and when the contrast of the out-of-focus subject is high. Therefore, in the case of such a combination, the blur amount Δ of the out-of-focus subject is also small.

On the other hand, when the image height of the in-focus subject is small and the contrast of the out-of-focus subject is low, most of the out-of-focus subject is hidden behind the in-focus subject, and there is little need to preserve the contrast of the out-of-focus subject. Conceivable. As described above, the blur amount δ of the out-of-focus subject is calculated to be large both when the image height of the in-focus subject is small and when the contrast of the out-of-focus subject is low. Therefore, in such a combination, the blur amount Δ of the out-of-focus subject also increases.

In S205, the main control unit 151 uses the blur amount Δ calculated by the blur amount calculation unit 163, the camera information acquired from the camera information input unit 102, and the subject distance information acquired from the distance information input unit 103 for bracket shooting. Calculate multiple aperture values. FIG. 7 is a diagram showing the relationship between the subject distance, focal length, and blur amount.

In FIG. 7, the following formula (3) holds.
dF=Δ×S2/(S2-S1) (3)
here,
dF: size of lens aperture diameter Δ: blur amount S1: distance from the optical center to the imaging plane of the in-focus subject S2: distance from the optical center to the imaging plane of the out-of-focus subject.

Equations (4) and (5) hold from the lens formula.
S1=d1×f/(d1−f) (4)
S2=d2×f/(d2−f) (5)
here,
d1: Distance from the optical center to the focused subject d2: Distance from the optical center to the out-of-focus subject f: Focal length of the imaging optical system.

Equation (6) holds from equations (3) to (5).
dF=Δ×S2×(d1−f)/{f×(d1−d2)} (6)

Also, the aperture diameter dF of the lens can be calculated from the relationship between the focal length f and the aperture value F using equation (7).
dF=f/F (7)

Substituting equation (7) into equation (6), the aperture value used in bracket shooting is determined using equation (8).
F=f ² ×(d1−d2)/{Δ×d2×(d1−f)} (8)

Specifically, the main control unit 151 substitutes the focal length f of the photographing optical system at the time of photographing and the distance d1 of the focused object into the equation (8). Then, by substituting the calculated blur amount Δ and the distance d2 for each out-of-focus subject or divided range, a plurality of aperture values F used in bracket photography are determined. Here, the distances d1 and d2 may be representative distances of individual distance ranges divided by the distance histogram. For example, the distance with the maximum count value within the distance range can be used as the representative distance.

Note that when the distance map generator 161 generates the object distances d1 and d2 used to calculate the aperture value based on the parallax image, the accuracy of the distance information is affected by the contrast of the object and the like. For example, the accuracy of distance information decreases for a subject with low contrast. Therefore, by selecting information with a high degree of reliability as the subject distance information used to calculate the aperture value used in bracket shooting, it is possible to improve the accuracy of the aperture value determined by equation (8).

Taking the photographed scene 801 in FIG. 8 as an example, highly reliable distance information can be calculated for subject regions 802 to 804 with high contrast. On the other hand, the reliability of the distance information decreases for the remaining low-contrast regions. In addition, since the image of the edge portion of the subject includes subjects at different distances, the reliability of the distance information is lowered. The reliability map 811 binarizes the reliability of the distance information generated by the distance map generation unit 161 for the shooting scene 801, and schematically shows areas with high reliability in white and areas with low reliability in black. is. The edge portion of the object and the area other than the object region have low contrast or are edge portions, so the reliability is low.

In such a case, the main control unit 151 acquires the distance d1 used in Equation (8) from the distance information obtained inside the in-focus subject area 812 . Also, the main control unit 151 acquires the distance d2 used in Equation (8) from the distance information obtained inside the out-of-focus subject areas 813 and 814 . For example, for the out-of-focus object 1, the representative distance can be determined based on the distance corresponding to the pixels included in the area 813 among the pixels corresponding to the range 2 in the distance map.

For example, when the distance range is divided for each subject as in the shooting scene A (FIG. 3), the number of bracket shots may be smaller than the number of out-of-focus subjects for which the blur amount Δ is calculated. In this case, bracketing conditions can be determined with priority given to an out-of-focus subject that occupies a large area in the image. This is because a subject that occupies a large proportion of the image has a large visual impact.

For example, when the distance histogram 821 is obtained, since the integrated value of the count values in the range 2 corresponding to the out-of-focus subject 1 is larger than the integrated value of the count values in the range 3 corresponding to the out-of-focus subject 2, The out-of-focus subject 1 is prioritized over the out-of-focus subject 2. - 特許庁It should be noted that priority may be given to an out-of-focus subject with a large corresponding maximum count value instead of the integral value of the count value. In the example of FIG. 8, since the maximum count value 823 is greater than the maximum count value 824, the out-of-focus subject 1 is prioritized over the out-of-focus subject 2. In FIG. For an out-of-focus subject for which highly reliable distance information cannot be obtained, the aperture value may be calculated with the distance set to infinity.

Also, for example, when the background distance range is divided into a predetermined number as in the shooting scene B (FIG. 4), the number of bracket shots may be smaller than the number of divisions of the distance range for which the blur amount Δ is calculated. In this case, some of the plurality of calculated aperture values can be selected according to the number of bracket shots. For example, by adjusting the interval of aperture values to be selected, it is possible to select such that the number of bracket shots is satisfied.

The main control unit 151 determines other shooting conditions (shutter speed and shooting sensitivity) for shooting with the aperture value F calculated based on Equation (8). The photographing conditions are determined so that the exposure amount does not change even if the aperture value F changes, and the photographing sensitivity does not change unless the shutter speed becomes slower than a predetermined time. For example, if the shutter speed Tv for satisfying the exposure without changing the sensitivity is the reciprocal of the focal length f (1/f) or faster, the sensitivity is not changed and only the shutter speed is changed. If the shutter speed Tv for satisfying the exposure amount without changing the photographic sensitivity is slower than 1/f, the photographic sensitivity is increased so that the shutter speed becomes 1/f or faster. The main control unit 151 thus determines a plurality of imaging conditions used in bracket imaging.

In S206, the main control unit 151 checks whether or not a photographing instruction (for example, a full press of the shutter button) has been detected. If a photographing instruction has been detected, the process proceeds to S207. return.

In S207, the main control unit 151 starts bracket imaging using the imaging conditions determined in S205. During bracket shooting, the main control unit 151 continuously adjusts the position of the focus lens so as to maintain the focus on the in-focus subject during scene analysis.

When the image data for recording is stored in the RAM 154 after one shot (one frame) of shooting is completed, the main control unit 151 determines in S208 whether the aperture values that have not yet been shot remain among the plurality of aperture values determined in S205. determine whether or not there is If there remains an unphotographed aperture value, the process returns to S206, and if not, the process proceeds to S209.

In S209, the main control unit 151 sequentially records the image data for recording stored in the RAM 154 on the recording medium 157, and ends the process. Note that, for example, a list of thumbnails of images obtained by bracket photography may be displayed on the display unit 150 to allow the user to select an image to be recorded. Alternatively, instead of separately recording the bracketed images, or separately, a composite image using the bracketed images may be recorded.

As described above, according to this embodiment, a plurality of aperture values to be used for bracket shooting in which aperture values are changed are automatically determined based on the distribution of distances of subjects present in the shooting scene. Therefore, the user can easily obtain an image having a desired depth of field without having to consider conditions such as the aperture value, the focal length of the imaging optical system, and the subject distance.

(Other embodiments)
Implementation of the above-described bracket shooting operation is not limited to the case where the depth of field (aperture) bracket shooting mode is explicitly set. Even if the bracket shooting mode is not set, it may be decided to perform aperture bracket shooting according to the scene determination result. For example, when it is determined that the in-focus subject is a person, aperture bracket photography may be performed.

Note that in the above-described embodiment, a configuration has been described in which the aperture value used for bracket shooting is determined based on the blur amount of the out-of-focus subject and the background calculated based on the distance information of the subject. However, the aperture value used for bracket shooting may be determined by another method based on the distance information of the object.

For example, it is possible to determine the aperture value used for bracketing so that the closest out-of-focus subject behind the in-focus subject does not fall within the depth of field. For the scene shown in FIG. 3, the aperture value F can be determined so that the distance corresponding to the peak 323 in the distance range corresponding to the out-of-focus subject 1 is not included in the depth of field of the in-focus subject. . As a result, the out-of-focus subject 1 (and the out-of-focus subject 2) can be blurred, and an image in which the in-focus subject stands out can be captured.

For example, the main control unit 151 acquires from the scene analysis unit 162 the representative distance of the in-focus subject and the representative distance of the out-of-focus subject behind the in-focus subject and closest to the subject. In addition, the main control unit 151 acquires at least the focal length of the imaging optical system among the imaging conditions for the scene-analyzed image. Then, the main control unit 151 calculates the aperture value at which the difference in the representative distance is equal to or less than the depth of field based on a known formula (the pixel pitch of the image sensor 141 can be used as the permissible circle of confusion diameter). . The main control unit 151 selects an aperture value to be used for bracket shooting from aperture values on the open side of the calculated aperture value.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

DESCRIPTION OF SYMBOLS 100... Digital camera, 101... Lens unit, 141... Imaging element, 142... Imaging signal processing part, 151... Main control part, 152... Image processing part, 154... RAM, 155... ROM, 161... Distance map generation part, 162 ... scene analysis unit, 163 ... blur amount calculation unit

The above-mentioned object is an acquisition means for acquiring a photographed image and distance information representing the distribution of the distance of the scene photographed in the image, and a plurality of diaphragms used in bracket photography in which the aperture is changed based on the distance information. a control means for determining a value, wherein the control means includes an analysis means for detecting an out-of-focus subject existing farther than the in-focus subject based on the distance information; calculating means for calculating an amount of blur; and setting means for determining the plurality of aperture values based on the amount of blur within a range in which the amount of blur of the focused subject is within a predetermined value, wherein the setting means is achieved by an image processing apparatus that determines the plurality of aperture values so that the out-of-focus subject does not enter the depth of field .

Claims (15)

Acquisition means for acquiring a photographed image and distance information representing the distribution of distances of the scene photographed in the image;
a control means for determining a plurality of aperture values used in bracket photography with different apertures based on the distance information;
An image processing device comprising: The control means is
analysis means for detecting an out-of-focus subject existing farther than a focused subject based on the distance information;
calculating means for calculating a blur amount for each out-of-focus subject;
determining means for determining the plurality of aperture values based on the amount of blur;
2. The image processing apparatus according to claim 1, comprising: The calculation means is
a position in the image of the focused object;
the proportion of the focused object in the image;
high contrast of said out-of-focus object;
3. The image processing apparatus according to claim 2, wherein the magnitude of said blur amount is varied according to at least one of the above. The calculation means increases the blur amount of the out-of-focus subject when the focused subject is at a second position closer to the center of the image than when the focused subject is at the first position. 4. The image processing apparatus according to claim 3, wherein the calculation is performed. The calculation means calculates the blur amount of the out-of-focus object when the proportion of the focused object in the image is a second value larger than the first value than when the ratio is the first value. 4. The image processing apparatus according to claim 3, wherein the calculation is large. When the contrast value of the out-of-focus object is a second value indicating that the contrast value of the out-of-focus object is higher than the first value, the out-of-focus object 6. The image processing apparatus according to any one of claims 3 to 5, wherein the blur amount of is calculated to be small. 3. The determining means determines the aperture value based on the blur amount obtained by weighting addition of the plurality of blur amounts when a plurality of blur amounts are calculated for the out-of-focus object. 7. The image processing device according to any one of 2 to 6. The control means is
an analysis means for dividing a distance range farther than a focused object into a plurality of parts;
calculation means for calculating a blur amount for each of the divided distance ranges;
determining means for determining the plurality of aperture values based on the amount of blur;
2. The image processing apparatus according to claim 1, comprising: 9. The image processing apparatus according to claim 8, wherein when the distance of the in-focus object is equal to or less than a predetermined distance, the analysis means divides the distance range more finely than otherwise. . 9. The image processing apparatus according to claim 8, wherein the analysis means divides the distance range corresponding to the same subject more finely than the distance range not corresponding to the same subject. 9. The image processing apparatus according to claim 8, wherein said analysis means divides said distance range so that two subjects are not included in one range. The control means is
detecting an out-of-focus subject that is farther than the focused subject and closest to the focused subject based on the distance information;
determining the plurality of aperture values so that the detected out-of-focus object does not fall within the depth of field;
2. The image processing apparatus according to claim 1, wherein: a photographing means for photographing the image;
An image processing device according to any one of claims 1 to 12,
An imaging apparatus, wherein the photographing means executes bracket photographing using a plurality of aperture values determined by the control means. an acquisition step in which the acquisition means acquires the photographed image and distance information representing the distribution of distances of the scene photographed in the image;
a control step in which the control means determines a plurality of aperture values to be used in bracket photography with different apertures based on the distance information;
A control method for an image processing device, comprising: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 12.

Images