JP7297566B2 - Image processing device, imaging device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an imaging device, an image processing method, and a program.

2. Description of the Related Art Conventionally, there has been known a process of applying image processing (for example, dodging processing, HDR, etc.) for correcting clarity, contrast, brightness, etc. of an image to a partial region within an image. In order to extract a part of the area, there is a technique of referring to distance information and detecting a specific subject based on the distance distribution. Patent Document 1 discloses a technique of predicting the position of an object in the current frame based on distance distribution data before the current frame and tracking the object in subsequent frames.

WO2017/130639

However, in the configuration disclosed in Patent Document 1, in which the distance information is always referred to, for example, the distribution of the distances shown in the image is evenly distributed, or the scene where a plurality of subjects are present at the same distance. Therefore, it is difficult to detect the subject with high accuracy. If the accuracy of object detection or area extraction is low, there is a risk that desired effects cannot be obtained in image processing performed on a part of the extracted area.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus capable of separating regions in an image with high accuracy.

In order to solve the above-mentioned problems, the image processing apparatus of the present invention includes acquisition means for acquiring distance information corresponding to an image, and first division means for dividing the image into a plurality of areas according to the distance information. a detection means for detecting a subject; a second division means for dividing an image into a plurality of areas according to the detection result of the subject detection; and a division for dividing the image into a plurality of areas based on the distance information. determining means for determining means, wherein the determining means extracts a subject region including the subject based on the distance information and the detection result of the subject detection, and determines the distance between the subject region and a region farther than the subject region; A dividing means is determined based on the statistic of the distance information .

According to the present invention, it is possible to provide an image processing apparatus that can accurately separate regions in an image.

It is a figure which shows the structure of an imaging device. 4 is a diagram showing the configuration of an image processing unit; FIG. It is a figure which shows the structure of an imaging part. 4 is a flowchart for explaining image processing for a partial area of an image; 4A and 4B are diagrams illustrating an example of an input image and subject detection results; FIG. It is a figure which shows an example of a defocus map. FIG. 5 is a diagram for explaining a defocus histogram and a threshold calculation result; FIG. 4 is a diagram illustrating a kakiwari map based on a defocus map; FIG. 10 is a diagram illustrating a Kakizori map based on a subject detection result;

In this embodiment, as an example of image processing for a partial area of an image, an image captured using an imaging device such as a digital camera is processed using information related to the shooting scene such as subject area detection information, distance information, and shooting conditions. A case where gradation processing is performed by dodging processing will be described. Although an imaging device will be described below as an example of an image processing device, the image processing device of this embodiment is not limited to an imaging device, and may be, for example, a personal computer (PC).

FIG. 1 is a diagram showing the configuration of an imaging device 100. As shown in FIG. The imaging device 100 is, for example, a digital single-lens reflex camera, digital video camera, compact digital camera, or other imaging device capable of processing digital images. The imaging device 100 includes an optical system 101 , an imaging unit 102 , an A/D conversion unit 103 , an image processing unit 104 , a control unit 105 , a display unit 106 and a recording unit 107 . Note that in this embodiment, an example of an imaging device in which the main body of the imaging device 100 and the lens device (optical system 101) are integrated will be described. There may be.

The optical system 101 is an imaging optical system including a lens group including a zoom lens and a focus lens, an aperture, and a shutter. The optical system 101 adjusts the magnification, focus position, or amount of light of a subject image reaching the imaging unit 102 .
The image pickup unit 102 has an image pickup device such as a CCD or CMOS sensor that photoelectrically converts the light flux of the subject that has passed through the optical system 101 and converts the photoelectric signal into an electric signal. A detailed configuration of the imaging unit 102 will be described later with reference to FIGS. 3(A) and 3(B).

The A/D converter 103 converts the video signal input from the imaging unit 102 into a digital image. An image processing unit 104 performs noise reduction, development processing, signal processing such as gradation processing, processing of images for display, and the like. Note that the image processing unit 104 can perform similar image processing not only on the image output from the A/D conversion unit 103 but also on the image read from the recording unit 107 .

The control unit 105 is, for example, a CPU (Central Processing Unit), and controls the imaging apparatus 100 as a whole. For example, in order to obtain an input image with proper brightness, the amount of exposure at the time of shooting is calculated, and in order to obtain an image with the calculated amount of exposure, the optical system 101 and the imaging unit 102 are controlled, and the diaphragm and shutter are controlled. Controls speed, sensor analog gain, etc.

The display unit 106 functions as an electronic viewfinder (EVF) by sequentially displaying images output from the image processing unit 104 on a display member such as an LCD (liquid crystal display). Although an example of an electronic viewfinder will be described in this embodiment, the imaging device 100 may be provided with an optical viewfinder. Further, the display unit 106 may have a display member having a touch panel function that displays information such as operation details and various settings on the UI and receives operations from the photographer.

The recording unit 107 has a function of recording an image, and may include, for example, an information recording medium using a package containing a rotary recording medium such as a memory card equipped with a semiconductor memory or a magneto-optical disk. The recording unit 107 may be provided in the imaging device 100 or may be a detachable memory.

FIG. 2 is a diagram for explaining the configuration of the image processing unit 104. As shown in FIG. The image processing unit 104 includes a signal processing unit 201 , a subject detection unit 202 , a shooting information acquisition unit 203 , a defocus calculation unit 204 and a hist calculation unit 205 . One or more of the functional blocks shown in FIG. 2 may be implemented by hardware such as an ASIC or programmable logic array (PLA), or implemented by a programmable processor such as a CPU or MPU executing software. may It may also be implemented by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as main entities, the same hardware can be implemented as the main entities.

The signal processing unit 201 performs various signal processing such as noise reduction processing and development processing, as well as gradation compression processing such as gamma conversion, and processing for correcting image brightness, contrast, hue, saturation, sharpness, and the like. Perform image processing. Further, when image processing is performed for each region, the signal processing unit 201 includes determination means for determining a separation method for separating a partial region and separation means for separating image regions by the determined separation method. also has the function of

A subject detection unit 202 detects whether or not a specific subject exists in the image. Then, when a specific subject exists in the image, the subject detection unit 202 outputs the position (coordinates in the image) and size of the specific subject. Various objects such as a person's face or entire body, various animals, and vehicles can be specified as the specific subject.
A shooting information acquisition unit 203 acquires various types of information such as the shooting mode at the time of shooting, the focal length, the aperture value, and the exposure time.

The defocus calculation unit 204 calculates the amount of image shift based on the phase difference between a plurality of subject images generated by light beams arriving from different pupil regions of the optical system 101, and calculates the defocus distribution based on the amount of image shift. Generate a focus map. As for a specific method of generating the defocus map, for example, a known method such as that described in Japanese Unexamined Patent Application Publication No. 2016-9062 is used to calculate the defocus amount for each pixel and treat it as a defocus map. Further, the defocus calculation unit 204 may acquire distance map data between the imaging device 100 and the object converted from the defocus amount. In this embodiment, an example of using a defocus map as an example of distance information will be described. It may be a map based on absolute distance values.

A histogram calculation unit 205 performs processing for generating a histogram for an image. An image for generating a histogram may be an image having luminance and color signals or a multi-valued signal indicating distance.

FIG. 3A is a diagram showing an arrangement configuration of pixels 302 of the imaging unit 102. As shown in FIG. FIG. 3B is a diagram showing the structure of the pixel 302. As shown in FIG. The imaging element of the imaging unit 102 includes a plurality of pixels 302 arranged two-dimensionally. The pixel 302 includes one microlens 301 and a pair of photoelectric conversion units 303A and 303B. The photoelectric conversion unit 303A and the photoelectric conversion unit 303B are capable of receiving light beams pupil-divided for one microlens and outputting a pair of focus detection signals having a phase difference. In this embodiment, an example in which each pixel unit includes two photoelectric conversion units will be described, but the present invention is not limited to this, and a configuration including a plurality of photoelectric conversion units may be used.

Next, as an example of performing image processing on a partial area of an image, an example of performing gradation processing for correcting contrast only on a background area will be described. FIG. 4A is a flowchart for explaining image processing for a partial area of an image. FIG. 5A is a diagram showing an example of an input image 501 to be subjected to image processing. An input image 501 is a scene in which a person 502 presents a flower to the front. In the input image 501, the person 502 near the imaging device 100 has a sharp contrast, whereas the background is dim and the contrast is low.

In step S<b>401 , the subject detection unit 202 performs subject detection processing on the input image 501 . As the object detection method, an existing technique such as a method of determination based on an edge pattern as described in Japanese Patent Application Laid-Open No. 10-162118 may be used. In this embodiment, the subject is a person, and face detection processing is performed as the subject detection processing. FIG. 5B is a diagram showing an example of subject detection results. A frame 510 indicates the position of the face of the person 502, which is the result of face detection.

In step S<b>402 , the defocus calculation unit 204 generates a defocus map of the input image 501 . FIG. 6 is a diagram showing an example of a defocus map. A defocus map 601 is a defocus map corresponding to the input image 501 . In the defocus map 601, the defocus amount is indicated by shading. A region 602 indicated by intermediate gradation (gray) is a region corresponding to the person 502 in the input image 501 and indicates focus. A region shown in a lighter color (white) than the region 602 is a region corresponding to the flower on the near side in the input image 501, and indicates out-of-focus foreground blur. A region shown in a darker color (black) than the region 602 is a region corresponding to the background on the far side in the input image 501, and indicates out-of-focus background blur. In step S402 of the present embodiment, an example of generating a defocus map as distance information has been described. good too.

In step S403, the histogram calculation unit 205 generates a defocus histogram based on the defocus map 601 generated in step S402. FIG. 7A is a diagram showing an example of a histogram. The histogram 701 indicates the defocus value in the horizontal direction and the number of pixels in the vertical direction. A defocus value of 128 indicates in-focus, a value of 127 or less indicates back blur (back side), and a value of 129 or more indicates front blur (front side).

In step S404, the signal processing unit 201 calculates a subject distance threshold based on the defocus histogram generated in step S403. The subject distance threshold is a value indicating the defocus range of the subject area including the subject. In this embodiment, the defocus range corresponding to the person 502 as the subject is calculated. A specific method for calculating the defocus range will be described with reference to the flowchart of FIG. 4(B).

FIG. 4B is a flowchart showing subject distance threshold calculation processing.
In step S450, the signal processing unit 201 calculates the average defocus value Def_face at the position of the face detection result (frame 510) in step S401 with reference to the defocus map generated in step S402. FIG. 7B is a diagram for explaining the subject distance threshold. A point 710 indicates a value on the histogram corresponding to the calculated average defocus value Def_face. Point 710 is close to the defocus value of 128. It can be seen that the defocus amount of the face area of the person 502 is close to 128 because the input image 501 is shot with the person 502 in focus.

In step S451, the signal processing unit 201 calculates the lower limit defocus value th1 indicating the person 502 based on the average defocus value Def_face calculated in step S450. The defocus lower limit value th1 is a defocus amount indicating the boundary of the distance between the subject and the background. Specifically, the histogram values are sequentially referred from the average defocus value Def_face to the left (the defocus value decreases) on the histogram to detect the lower limit value th1. When the amount of change in the defocus value changes from the negative direction (decrease) to the positive direction (increase) by a predetermined value or more, or when the value of 0 continues for a predetermined value or more, either condition is satisfied. is the lower limit value th1. In this way, by detecting the troughs of the histogram, that is, the portions where the distance distribution changes significantly, it is possible to detect the defocus amount indicating the boundary of the distance between the subject and the background.

In step S452, the signal processing unit 201 calculates a defocus upper limit value th2 indicating the person 502 based on the average defocus value Def_face calculated in step S450. The defocus upper limit value th2 is a defocus amount indicating the boundary of the distance between the subject and the foreground.
Specifically, the histogram values are sequentially referred to from the average defocus value Def_face to the right (the defocus value increases) on the histogram to detect the upper limit value Th2. Then, when the amount of change in the defocus value changes from the negative direction (decrease) to the positive direction (increase) by a predetermined amount or more, or when the value of 0 continues for a predetermined amount or more, the defocus value is defocused. Let the focus value be the upper limit value th2. In this way, by detecting the troughs of the histogram, that is, the portions where the distance distribution changes significantly, it is possible to detect the defocus amount indicating the boundary of the distance between the person and the foreground.

In addition, although the method of calculating the threshold value based on the change in the increase and decrease of the histogram and the continuity in steps S451 and S452 has been described, the threshold value is not limited to this, and other methods may be used to calculate the threshold value. . For example, a method of searching for the optimum threshold based on the distribution of the histograms of the two groups after division may be used. Further, the count number may be integrated in the horizontal direction from the average defocus value Def_face, and the values when the integrated value reaches a predetermined threshold value based on the face size may be set as the lower limit value and the upper limit value, respectively.

Returning to the description of FIG. In this embodiment, since the contrast correction process is applied only to the background area as the process for each area, the distance information (defocus amount) is used to separate the background and the person. At this time, if the area-by-area processing is applied without correctly separating the background, there will be adverse effects such as unfavorable correction being applied to the person who is the main subject, or desired correction not being applied to the background.

For example, in a scene where the background is far away from the person, the defocus distribution is dispersed, but in a scene with depth where the distance changes slowly, the defocus value becomes uniform, and where It becomes difficult to separate the background. In the present embodiment, distance dispersion, that is, the degree of scattering of the distribution, is used as an index of how easily the person and the background can be separated by distance. If the variance is large, it can be determined that the person and the background are easily separated. In this embodiment, the processing for extracting a partial region is changed depending on whether the person and the background can be separated by the defocus amount, which is the distance information, and when they cannot be separated, that is, according to the separation accuracy. Specifically, when the variance is large and it is easy to separate the person and the background, a part of the area is extracted based on the defocus map. Detect some regions based on the results. Details of these processes will be described in steps S405 to S409.

In step S405, the signal processing unit 201 calculates the distribution of the defocus value 0 to the upper limit value th2 in the histogram 701, that is, the defocus variance including the person and the background. As a specific calculation method, first, the average value Dave of the defocus value 0 to the upper limit value th2 in the distribution is calculated by the following formulas (1) and (2). Here, Hist[i] represents the value of the hist for the defocus value i, and N represents the total count number of the defocus value 0 to the upper limit value th2. The reason why the defocus range for calculating the variance is limited to the defocus value 0 to the upper limit value th2 of the subject area and the area farther than the subject area is that it is difficult to determine the ease of separation between the person and the background. This is to avoid being affected by the distribution of the distance of the foreground that is closer to the target than the target.

Then, using the calculated average value Dave, the defocus dispersion Ddisp is calculated by the following equation (3).

In step S406, the signal processing unit 201 determines whether the separation accuracy is high by determining whether the defocus variance Ddisp calculated in step S405 is equal to or greater than a predetermined threshold value Dth. If the defocus variance Ddisp is equal to or greater than the predetermined threshold value Dth and it can be determined that the separation accuracy is high, the process proceeds to step S407. On the other hand, if the defocus variance Ddisp is less than the predetermined threshold value Dth and it can be determined that the separation accuracy is low, the process proceeds to step S408.

In step S<b>407 , the signal processing unit 201 generates a kakiwari map including the person 502 and the foreground based on the defocus map 601 . Specifically, in the defocus map 601, a defocus map is generated by binarizing the areas where the defocus value is equal to or greater than the lower limit value th1 and the areas where the defocus value is not. FIG. 8 is a diagram illustrating a kakiwari map based on a defocus map. In the kakiwari map 801, the person 502 and the flower area in front of it are white, and the other background is black. Note that shaping processing may be performed with reference to the pixel values of the input image 501 as described in Japanese Unexamined Patent Application Publication No. 2017-11652, for example, so that the drawing map 801 better matches the outline of the subject.

In step S408, the signal processing unit 201 generates a writing map including the person 502 based on the subject detection result in step S401. First, based on the position and size of the face of the person 502 detected by the subject detection unit 202, a silhouette image of a person is generated. FIGS. 9A and 9B are diagrams for explaining the kakiwari map based on the subject detection result. In FIGS. 9A and 9B, the area of the person 502 is white, and the other areas are black.

In an image 901 in FIG. 9A, a circle 902 aligned with the position of the frame 510 that is the result of face detection and an ellipse 903 below it are generated as silhouettes. Further, the image 901 is subjected to shaping processing with reference to the pixel values of the input image 501 as described in Japanese Unexamined Patent Application Publication No. 2017-11652, for example. , a kakiwari map 910 can be generated that fits the outline of the subject.

In step S409, the signal processing unit 201 performs image processing on each of the partial regions extracted by the kakiwari map. In this embodiment, the signal processing unit 201 generates an image in which the contrast of only a partial region is corrected from the input image 501 using the split map generated in step S407 or step S408. Here, the partial area is a black area that does not include a person in the kakiwari map.

As a specific correction method, first, an image in which the contrast of the entire image 501 is corrected is generated. As a method of correcting the contrast, an existing technique such as a method of correcting only local contrast without changing the brightness of the low frequency region as described in Japanese Patent Laid-Open No. 2019-28537 may be used. . Then, the contrast-corrected image is synthesized with an image developed without contrast correction based on the writing division map. As a result, it is possible to generate an image in which the contrast is corrected only for the background area shown in black in the writing map.

In this embodiment, the input image 501 is corrected using either one of the two writing map 801 and the writing map 910, but both of the two writing maps may be used. When using two writing division maps, the ratio of using two writing division maps may be changed according to the dispersion Ddisp calculated in step S405. For example, the larger the variance, the greater the weight of the map 801 based on the distance information, and the weighted addition of the two maps may be performed. In step S409, weighted addition is performed on the contrast-corrected image and the contrast-uncorrected image based on the multi-valued writing division map, thereby making it possible to suppress abrupt switching of the effect.

As described above, according to the present embodiment, when controlling the effect of the image processing for each region based on the distance information, it is determined whether the region can be divided based on the defocus distribution. It is possible to suppress the occurrence of adverse effects due to

In this embodiment, as means for acquiring distance information, a configuration for generating distance information based on phase differences between a plurality of subject images generated by light beams arriving from different pupil regions of the imaging optical system has been described. You may substitute or use together the structure and means of . For example, by configuring a compound eye camera having a plurality of lenses and imaging elements, the amount of image shift may be detected with higher accuracy. Further, by adopting a configuration in which the distance can be measured by a TOF (Time Of Flight) camera or ultrasonic waves, it is possible to improve the distance measurement performance for subjects with little pattern variation.

Also, in this embodiment, the case where the main subject is a person and the contrast of the background is corrected has been described, but the present invention is not limited to this, and any subject may be used. For example, the main subject may be an animal, a building, a vehicle, or the like, and the present embodiment can be applied by using a technique for detecting the subject. Image processing for a partial region is not limited to contrast correction processing, and may be processing for correcting brightness, hue, saturation, sharpness, and the like, for example.

Also, in the present embodiment, the case where the background separation accuracy is determined using the variance of the distance has been described. For example, the larger the aperture value, the deeper the depth and the smaller the defocus amount of the background, so separation by the defocus value tends to be difficult. Therefore, the larger the aperture value, the smaller the threshold Dth, and the smaller the aperture value, the larger the threshold Dth. Also, the higher the ISO sensitivity, the greater the noise in the signal and the greater the variance. Therefore, the threshold Dth may be set larger for higher sensitivity and smaller for lower sensitivity. In addition, the determination may be made based on other statistical quantities such as the standard deviation and the difference between the maximum value and the minimum value of values having a count number equal to or greater than a predetermined value on the histogram, in addition to the defocus variance.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

104 Image processing unit 105 Control unit 201 Signal processing unit 202 Subject detection unit 203 Shooting information acquisition unit 204 Defocus calculation unit 205 Hist calculation unit

Claims (12)

an acquisition means for acquiring distance information corresponding to an image;
a first dividing means for dividing an image into a plurality of regions according to the distance information;
a detecting means for detecting a subject;
a second dividing means for dividing an image into a plurality of regions according to the detection result of the subject detection;
determining means for determining dividing means for dividing an image into a plurality of regions based on the distance information ;
The determination means extracts a subject area including the subject based on the distance information and the detection result of the subject detection, and divides the subject area based on the statistics of the distance information of the subject area and an area farther than the subject area. An image processing apparatus characterized by determining The determining means determines accuracy of separation of a subject area including a subject and a background area based on the distance information, and divides the image into a plurality of areas according to the accuracy of separation based on the distance information. 2. The image processing apparatus according to claim 1, wherein the dividing means for dividing is determined. 3. The image processing apparatus according to claim 2, wherein said determining means changes a ratio of using a dividing means for dividing an image into a plurality of regions according to the separation accuracy based on said distance information. The determining means divides the image into a plurality of regions by the first dividing means when the separation accuracy based on the distance information is high, and divides the image into a plurality of regions when the separation accuracy based on the distance information is low. 3. An image processing apparatus according to claim 2, wherein means determines to divide the image into a plurality of regions. The statistic of the distance information is any one of the variance of the distance information, the standard deviation of the distance information, and the difference between the maximum value and the minimum value of values having a count number equal to or greater than a predetermined value in the distance information. Item 5. The image processing apparatus according to any one of Items 1 to 4 . 6. The image processing apparatus according to claim 1, wherein said determining means determines said dividing means based on an aperture value or an ISO sensitivity. The acquisition means is characterized in that the distance information is generated from a defocus amount or an image shift amount obtained based on a plurality of images generated by respectively receiving light beams that have passed through different pupil regions of the imaging optical system. The image processing apparatus according to any one of claims 1 to 6 . 8. The image processing apparatus according to claim 1, further comprising image processing means for performing image processing for each of said divided areas. 9. The image processing apparatus according to claim 8 , wherein the image processing is processing for correcting any one of brightness, contrast, hue, saturation, and sharpness. 10. An imaging device comprising: an image processing apparatus according to claim 1; and imaging means for receiving light beams passing through different pupil regions of an imaging optical system and outputting a plurality of images. Device. obtaining distance information corresponding to the image;
dividing an image into a plurality of regions by a first separation method according to the distance information;
a step of detecting an object;
a step of dividing an image into a plurality of regions by a second separation method according to the detection result of the subject detection;
determining a division method for dividing the image into a plurality of regions based on the distance information ;
In the step of determining the division method, a subject area including the subject is extracted based on the distance information and the detection result of the subject detection, and a statistic of the distance information of the subject area and an area farther than the subject area is calculated. An image processing method, characterized in that a division method is determined based on . A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 9 .

Images