JP2021164063A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To enable suppression of generation of an artifact due to insufficient extraction accuracy of a subject region.SOLUTION: An image processing apparatus includes: map obtaining means (301) configured to obtain an evaluation value distribution corresponding to an image as an evaluation value map; map generating means (301) configured to generate a first subject map based on a subject region extracted from the image using the evaluation value map; degree obtaining means (303) configured to obtain a sparse degree indicating a degree of a sparse region included in the first subject map; and correction means (303, 305) configured to execute correction processing on the image using at least any one of the first subject map and a second subject map (307) generated without using the evaluation value map. The correction means (303, 305) executes the correction processing using the second subject map preferentially rather than the first subject map, as the sparse degree becomes higher.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing technique for captured images.

Conventionally, there are known cameras that extract a subject area from a captured image, correct the brightness only in the subject area, or apply a background blur effect to the area other than the subject area. Patent Document 1 discloses a technique of extracting a subject region based on a defocus amount distribution and electronically adjusting the background blur effect by blurring a region other than the subject region.

Japanese Unexamined Patent Publication No. 2008-15754

However, in the prior art disclosed in the above-mentioned patent document, since the histogram of the defocus amount distribution is analyzed to determine the defocus amount range of the subject area, the subject such as a person and the background such as a wall are close to each other. In that case, the defocus amount range cannot be determined accurately. As a result, part of the background is sparsely included in the subject area, or conversely, part of the subject area is sparsely lacking, and image correction and image effects are applied sparsely, causing artifacts such as spots in the output image. Resulting in.

Therefore, an object of the present invention is to make it possible to suppress the occurrence of artifacts due to insufficient extraction accuracy of the subject area.

The image processing apparatus of the present invention generates a map acquisition means for acquiring an evaluation value distribution corresponding to an image as an evaluation value map, and a first subject map based on a subject area extracted from the image using the evaluation value map. The map generation means for acquiring the degree of sparseness, which represents the degree of the sparse area included in the first subject map, the first subject map, and the evaluation value map are not used. The image has a correction means for correcting the image using at least one of the second subject map, and the correction means has a higher degree of sparseness than the first subject map. The correction process is performed by preferentially using the second subject map.

According to the present invention, it is possible to suppress the occurrence of artifacts due to insufficient extraction accuracy of the subject area.

It is a figure which shows the structural example of the digital camera which concerns on embodiment. It is a figure for demonstrating the structure of the image pickup part. It is a figure which shows the structural example of the image processing part. It is a figure for demonstrating operation of a subject area extraction part. It is a figure for demonstrating the operation of the subject map synthesis part. It is a figure for demonstrating the structure of the sparseness determination part. It is a figure which shows the structural example of the expansion filter part. It is a figure which shows the structural example of the shrinkage filter part. It is an operation flowchart of the MAX / MEDIAN / MIN filter unit. It is an operation flowchart of the control unit at the time of still image shooting. It is a figure for demonstrating another structure of a sparseness determination part.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The configuration shown in the following embodiments is only an example, and the present invention is not limited to the illustrated configuration. The same configuration or processing will be described with the same reference numerals.

FIG. 1 is a block diagram showing a schematic configuration example of an image pickup apparatus (hereinafter referred to as a digital camera 100) as an application example of the image processing apparatus according to the embodiment of the present invention.
The control unit 101 is, for example, a CPU. The ROM 102 is a rewritable non-volatile memory, and stores, in addition to an operation program for controlling the operation of each block included in the digital camera 100, parameters and the like necessary for the operation of each block. The control unit 101 reads an operation program from the ROM 102, expands it into the RAM 103, and executes it to control the operation of each block included in the digital camera 100 of the present embodiment. The RAM 103 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block included in the digital camera 100.

The optical system 104 forms an optical image of a subject or the like on the imaging surface of the imaging unit 105.
The image pickup unit 105 is, for example, an image pickup element such as a CCD or a CMOS sensor. The optical image image formed on the image pickup device by the optical system 104 is photoelectrically converted, and the obtained image pickup signal (analog signal) is converted to an A / D converter. Output to 106.
The A / D conversion unit 106 applies the A / D conversion process to the input imaging signal, outputs the obtained imaging data (digital imaging signal) to the RAM 103, and stores the obtained imaging data (digital imaging signal).

The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, reduction / enlargement, and filtering to the imaged data stored in the RAM 103, and outputs the obtained image data to the RAM 103. And memorize it. The correction process described later according to this embodiment is performed by the image processing unit 107.

The recording medium 108 is a detachable memory card or the like, such as image data that has been image-processed by the image processing unit 107 and stored in the RAM 103, image data that has been A / D-converted by the A / D conversion unit 106, and the like. Is recorded as a recorded image.
The display unit 109 is a display device such as a liquid crystal display (LCD), and displays various information such as a through display of a subject image captured by the image pickup unit 105. When the subject image captured by the imaging unit 105 is displayed through, the display unit 109 functions as an EVF (electronic viewfinder).

The focus map processing unit 110 analyzes the image pickup signal by the image pickup unit 105 to generate information related to the focus distribution of the subject or the like as a focus map, and outputs the focus map data to the RAM 103 for storage. The details of the focus map generation process in the focus map processing unit 110 will be described later. In the case of the present embodiment, the focus map, which is the information related to the focus distribution of the subject or the like acquired by the focus map processing unit 110, is used as an evaluation value map showing the evaluation value distribution for the captured image.

FIG. 2 is a diagram for explaining a configuration example of the imaging surface of the imaging unit 105.
The pixel 202 is composed of a microlens 201 and a pair of photoelectric conversion units 203 and 204. On the imaging surface of the imaging unit 105 of FIG. 1, pixels 202 composed of the microlens 201 and a pair of photoelectric conversion units 203 and 204 are arranged two-dimensionally and regularly. In the imaging unit 105 having the configuration shown in FIG. 2, A image and B image are output as a pair of images from the outputs of the pair of photoelectric conversion units 203 and 204 of the pixels 202 arranged two-dimensionally regularly. .. That is, according to the imaging unit 105, a pair of light fluxes passing through different regions of the pupil of the optical system 104 of FIG. 1 are imaged as a pair of optical images, and these are formed as a pair of images, A image and B image. Can be output.

The focus map processing unit 110 of FIG. 1 outputs the phase difference distribution between the A image and the B image, that is, the parallax information distribution acquired from two image groups having different viewpoints as an evaluation value map (focus map). .. As the phase difference distribution between the A image and the B image, for example, the defocus amount distribution using the method disclosed in Patent Document 1 may be obtained.

FIG. 3 is a block diagram showing a configuration example of the image processing unit 107. As shown in FIG. 3, the image processing unit 107 includes a subject area extraction unit 301, a sparseness determination unit 302, a subject map composition unit 303, and a correction processing unit 305.

The subject area extraction unit 301 performs a subject map acquisition process of extracting a subject area using a focus map (evaluation value map) input from the focus map processing unit 110 to generate a subject map. Then, the subject area extraction unit 301 outputs the information of the subject map to the subject map synthesizing unit 303 and the sparse determination unit 302. The subject region can be extracted using, for example, the defocus amount distribution disclosed in Patent Document 1.

4 (a) to 4 (d) are diagrams for explaining the operation of the subject map generation process in the subject area extraction unit 301.
FIG. 4A is a diagram for explaining a subject image formed on the imaging surface 401 of the imaging unit 105 of FIG. In the example of FIG. 4A, it is assumed that the person 412, which is the main subject, is in focus, and the person 411 stands in front of the person 412. In the case of the example of FIG. 4A, it is assumed that the person 412 of the main subject stands in front of the wall of the room. Further, in the case of FIG. 4A, there is also furniture 413 in the room, and the furniture 413 is installed so that the back surface is attached to the wall. Therefore, when the person 412 of the main subject is in focus, it is assumed that the furniture 413 is also in focus. On the other hand, since the person 411 stands in front of the person 412, the person 411 is out of focus.

FIG. 4B is a diagram showing the focus map 402. The focus map 402 shown in FIG. 4B is represented in gray scale, and the larger the defocus amount is, the whiter it is, and the smaller the defocus amount is, the grayer it is. The area 422 in FIG. 4B is the area corresponding to the person 412 in FIG. 4A, the area 423 is the area corresponding to the furniture 413 in FIG. 4A, and the area 421 is the area corresponding to FIG. 4 (a). This is the area corresponding to the person 411 in a). Since the main subject person 412 and furniture 413 in FIG. 4A and the wall behind them are close to each other and the amount of defocus is small, the areas 422 and 423 and the wall are shown in gray, while the person 411 on the front side. Since the amount of defocus is large, the area 421 is represented by white.

FIG. 4D is a diagram showing a defocus amount frequency distribution. The frequency distribution 404 in FIG. 4 (d) shows the defocus amount frequency distribution of the person 411, the frequency distribution 405 shows the defocus amount frequency distribution of the person 412 and the furniture 413, and the frequency distribution 406 shows the defocus amount frequency distribution of the wall. Shown. As described above, since the person 412 and the furniture 413 are close to the wall, the boundary between the defocus frequency distribution 405 of the person 412 and the furniture 413 and the defocus frequency distribution 406 of the wall (the valley part of the frequency distribution) is unclear. It has become. On the other hand, the defocus frequency distribution 404 of the person 412 and the furniture 413 or the person 411 away from the wall is clearly distinguishable from the defocus frequency distributions 405 and 406 of the person 412 and the furniture 413 or the wall. ..

FIG. 4C is a diagram showing a subject map 403 generated based on the defocus amount frequency distribution of FIG. 4D. The subject map is an 8-bit binary map in which white is 255 and black is 0. The white portion is used as a subject label representing the subject area, and the black portion is used as a non-subject label representing the outside of the subject area (non-subject area). As described above, the subject map is classified into at least two label areas, that is, a subject label represented by a white portion and a non-subject label represented by a black portion. When the subject map is generated, the area included in the L2 range from one valley d1 to the other valley d2 sandwiching the peak of the defocus amount frequency distribution 405 including the defocus amount 0 is represented by a white (255) subject label. It is the subject area. The area outside the L2 range (L1 range or L3 range) is a non-subject area represented by a black (0) non-subject label. The area 432 in FIG. 4C is the area corresponding to the person 412 in FIG. 4A, and the area 433 is the area corresponding to the furniture 413 in FIG. 4A.

However, in the case of the defocus amount frequency distribution example of FIG. 4D, the boundary between the defocus amount frequency distributions 405 and 406 is unclear. Therefore, in the subject map 403 of FIG. 4C, not only the area 432 corresponding to the person 412 and the area 433 corresponding to the furniture 413, but also a white (255) area in which a part of the wall of the room represents the subject label. Generated as 434. Since this region 434 is a region that is not the original main subject, it is not a region that is organized like the person 412 of the main subject, but is often a small region such as sparsely scattered spots. Hereinafter, each small area 434 such as spots scattered sparsely in the subject map will be referred to as a sparse area.

The explanation is returned to FIG.
The sparseness determination unit 302 detects a sparse area in the subject map generated by the subject area extraction unit 301, and performs a sparseness degree acquisition process for determining the degree of sparseness indicating how much the sparse area is included in the subject map. .. Then, the sparseness determination unit 302 outputs the acquired information indicating the degree of sparseness to the subject map composition unit 303.

In the case of the present embodiment, the sparseness determination unit 302 obtains the area of each sparse area included in the subject map, and determines the degree of sparseness of the subject map based on the area of the sparse area. The degree of sparseness is set to a value indicating a ratio of 0 to 100% as an example, and the sparseness determination unit 302 increases the degree of sparseness of the subject map as the area of the sparse area becomes relatively larger than the total area of the subject map. To a high value. In the case of the subject map 403 illustrated in FIG. 4C, a small area 414 of a part of the wall is detected as a sparse area, and the larger the area of the sparse area with respect to the subject map, the higher the degree of sparseness. It is output. A detailed description of the configuration of the sparseness determination unit 302, the detection of the sparse area, the determination process of the degree of sparseness, and the like will be described later.

Then, the subject map synthesizing unit 303 synthesizes the subject map from the subject area extraction unit 301 and the alternative silhouette 307 prepared in advance based on the degree of sparseness, and the combined subject map is sent to the correction processing unit 305. Output. Although the details will be described later, in the case of the present embodiment, the subject map compositing unit 303 generates a post-composite subject map synthesized so that the higher the degree of sparseness, the more preferentially the alternative silhouette 307 is used over the subject map 403. Will be done. In this case, the correction processing unit 305 performs correction processing based on the synthesized subject map so that the alternative silhouette 307 is used preferentially over the subject map 403 as the degree of sparseness increases.

5 (a) to 5 (d) are diagrams for explaining the operation of the subject map compositing unit 303.
FIG. 5B is a diagram showing an example of the alternative silhouette 307. The alternative silhouette 307 shown in FIG. 5B is a fixed shape map including the humanoid 510. The subject map described above is the first subject map generated using the focus map (evaluation value map), while the alternative silhouette is the second subject map generated in advance without using the evaluation value map. .. The information of the alternative silhouette 307 is held by, for example, the ROM 102 of FIG. 1, and when it is used by the image processing unit 107, it is expanded in the RAM 103 of FIG. 1 and sent to the subject map compositing unit 303. When the alternative silhouette 307 is synthesized based on the degree of sparseness, the image processing unit 107 detects a humanoid region by, for example, a known person image recognition process from the captured image. Then, the subject map synthesizing unit 303 synthesizes the humanoid 510 of the alternative silhouette 307 so as to match the detected position.

Graph 501 of FIG. 5A is a diagram showing the relationship between the alternative silhouette usage rate and the degree of sparseness. The vertical axis of FIG. 5A shows the alternative silhouette usage rate [%], and the horizontal axis shows the degree of sparseness [%]. As shown in Graph 501, when the degree of sparseness is less than the first threshold value TH1, the alternative silhouette usage rate is set to 0%, and when the degree of sparseness is greater than or equal to the second threshold value TH2, the alternative silhouette usage rate is set to 0%. It is made 100%. Further, when the degree of sparseness is equal to or higher than the first threshold value TH1 and less than the second threshold value TH2, the higher the degree of sparseness, the higher the usage rate of the alternative silhouette.

FIG. 5D is a diagram showing the subject map 403 shown in FIG. 4C.
5 (c) shows the subject map 403 of FIG. 5 (d) and the alternative silhouette 307 of FIG. 5 (b) in the subject map synthesizer 303 based on the degree of sparseness of the graph 501 of FIG. 5 (a). It is a figure which showed the subject map 503 after composition after composition. The details of the compositing process in the subject map compositing unit 303 will be described later.

In the example of FIG. 5 (c), since the degree of sparseness of the subject map is, for example, the second threshold value TH2 or more of FIG. 5 (a), the combined subject map 503 when the alternative silhouette usage rate is 100% is used. Shown. In the example of FIG. 5 (c), the area 433 corresponding to the furniture 413 of FIG. 4 (a) in the subject map 403 of FIG. 5 (d) does not become the subject level of white (255) and is black (0). Although it is a non-subject level, all the sparse areas 434 are black (0) non-subject levels. When the combined subject map 503 is used in the correction processing described later in the correction processing unit 305 in the subsequent stage, it is possible to prevent the sparse region from being corrected and unnecessary artifacts from being generated.

In the description of FIG. 5, the alternative silhouette 307 is prepared in advance and combined with the subject map 403 without being processed or the like, but the present embodiment is not limited to this. The alternative silhouette 307 may be processed and combined with the subject map 403. For example, the facial organ position, joint position, posture information, etc. of the person 412 of the main subject in FIG. 4A are detected, and the position and shape of the humanoid 510 of the alternative silhouette 307 are determined based on the detection results. It may be deformed, enlarged / reduced, etc. so as to match the position and shape of the person 412. In addition, the alternative silhouette may be generated by analyzing the captured image. For example, a person map of the person 412 of the main subject may be detected by using a technique such as semantic region division based on machine learning, and the person map may be used as an alternative silhouette.

The explanation is returned to FIG.
The addition unit 304 inputs a pair of parallax images of the A image 308 and the B image 309, and adds the pair of parallax images of the A image 308 and the B image 309. The image after addition by the addition unit 304 (addition image) is sent to the correction processing unit 305.

The correction processing unit 305 corrects the brightness of the added image based on the subject map after composition. The correction processing in the correction processing unit 305 is expressed by the calculation of the following equation (1). In the equation (1), X is the pixel value of the added image, G is the pixel value of the subject map after composition, and Y is the pixel value of the image after the correction process is performed. The image after the correction processing by the correction processing unit 305 is output as the corrected image 310 in the image processing unit 107.

Y = X · (1 + G / 255) Equation (1)

Since the equation (1) is an calculation example of the correction process for correcting the brightness of the added image based on the subject map after composition, the corrected image 310 seems to illuminate the subject of interest, that is, the person 412 which is the main subject. The image will have a good lighting correction effect. In the present embodiment, the correction process for imparting the lighting correction effect is mentioned, but the correction process is not limited to this example. For example, a correction process that adjusts the sharpness of the added image based on the combined subject map may be used, and in this case, it is possible to acquire an image with improved sharpness of the subject of interest. In addition, instead of the correction processing for the subject of interest, the correction processing may be performed such that the background blur or the background contrast is adjusted for the background area outside the area of the subject of interest based on the combined subject map. In this case, it is possible to acquire an image in which the background outside the region of the subject of interest is blurred or an image in which the background contrast is adjusted. Further, these lighting corrections, sharpness adjustments, background blurring, background contrast adjustments, etc. may be performed separately, or may be performed in combination of two or more.

6 (a) is a diagram showing a configuration example of the sparseness determination unit 302 of FIG. 3, and FIGS. 6 (b) to 6 (g) are diagrams for explaining the operation in the configuration of FIG. 6 (a). Is.
As shown in FIG. 6A, the sparseness determination unit 302 includes an expansion filter unit 601, a contraction filter unit 602, a difference detection unit 603, a difference detection unit 604, a map integration unit 605, a median filter unit 606, and a sparseness degree calculation. It has a part 607.

The expansion filter unit 601 is a filter unit that expands the white (255) subject label portion of the input subject map 608. FIG. 6B shows that when the input subject map 608 is the subject map 403 shown in FIG. 4C described above, the subject map 403 is expanded and filtered by the expansion filter unit 601 after expansion. It is a figure which showed the subject map 611. That is, when the expansion filter processing is performed on the subject map 403 of FIG. 4 (c), the white portions (subject label portions) that are close to each other in the subject map 403 are connected, as shown in FIG. 6 (b). After expansion, the subject map 611 is generated. In the case of the present embodiment, the expanded subject map 611 corresponds to the third subject map.

The shrink filter unit 602 is a filter unit that shrinks the white (255) subject label of the input subject map 608. FIG. 6 (c) shows a post-shrinkage subject map after the input subject map 608 is the subject map 403 shown in FIG. 4 (c) and the subject map 403 is shrink-filtered by the shrink filter unit 602. It is a figure which showed 612. That is, when the subject map 403 of FIG. 4 (c) is subjected to the shrinkage filter processing, the black portions close to each other in the subject map 403 are connected to each other, and the contracted subject map 612 as shown in FIG. 6 (c). Is generated. In the case of the present embodiment, the contracted subject map 612 corresponds to the fourth subject map.

The difference detection unit 603 performs the difference detection process so that the white (255) subject label is used for the difference in the subject map before and after the expansion filter processing, and the black (0) non-subject label is used for the other parts. In FIG. 6 (d), the difference detection unit 603 is different from the subject map 403 of FIG. 4 (c) before the expansion filter processing and the post-expansion subject map 611 of FIG. 6 (b) after the expansion filter processing. It is a figure which showed the difference detection map 613 after performing the detection process. That is, when the difference detection process between the subject map 403 of FIG. 4 (c) and the expanded subject map 611 of FIG. 6 (b) is performed, the difference portion becomes white (255) and the other portion becomes black (0). A difference detection map 613 as shown in FIG. 6D, which is a non-subject label, is generated. In the case of the present embodiment, the difference detection map 613 corresponds to the first sparse area map.

The difference detection unit 604 performs the difference detection process so that the white (255) subject label is used as the difference in the subject map before and after the shrinkage filter processing, and the other black (0) non-subject label is used. In FIG. 6 (e), the difference detection unit 604 is different from the subject map 403 of FIG. 4 (c) before the shrinkage filter processing and the post-shrinkage subject map 612 of FIG. 6 (c) after the contraction filter processing. It is a figure which showed the difference detection map 614 after performing the detection process. That is, according to the difference detection process between the subject map 403 of FIG. 4 (c) and the contracted subject map 612 of FIG. 6 (c), the difference portion is the subject label of white (255), and the rest is black (0). The difference detection map 614 as shown in FIG. 6E, which is the non-subject label of the above, is generated. In the case of the present embodiment, the difference detection map 614 corresponds to the second sparse area map.

The difference detection process in the difference detection units 603 and 604 is represented by the following equation (2). In the equation (2), X0 and X1 are subject maps input to the difference detection unit. That is, X0 and X1 are subject maps before and after the expansion filter processing in the case of the difference detection unit 603, and subject maps before and after the expansion filter processing in the case of the difference detection unit 604. Further, in the equation (2), ABS is an absolute value function, and S is an output of the difference detection unit.

S = ABS (X0-X1) Equation (2)

In this way, by using the subject map before and after the expansion filter processing by the expansion filter unit 601 and configuring the difference detection unit 603 to perform the difference detection processing, it is possible to detect the black portion in the sparse region. That is, the black part (non-subject label) of the sparse area of the subject map changes (changes to the white subject label) by the expansion filter processing of the expansion filter unit 601, and further, the sparse area by the difference detection process by the difference detection unit 603. It will be possible to detect the black part in. Further, by using the subject map before and after the contraction filter processing by the contraction filter unit 602 and configuring the difference detection unit 604 to perform the difference detection process, the white portion in the sparse region can be detected. That is, the white part of the sparse area of the subject map changes (changes to the black part) by the shrinkage filter processing of the shrinkage filter unit 602, and the difference detection process of the difference detection unit 604 further changes the white part in the sparse area. The part can be detected.

In the present embodiment, the difference detection process in the difference detection units 603 and 604 is an operation represented by the equation (2), but the present invention is not limited to this example. For example, after the white (255) part is TRUE (true) and the black (0) part is FALSE (false), the difference is detected by performing the logical operation of XOR (exclusive OR). Is also good.

The difference detection map by the difference detection unit 603 and the difference detection map by the difference detection unit 604 are input to the map integration unit 605.
If any of the input difference detection maps is white, the map integration unit 605 sets the subject label as white (255), and sets the other labels as non-subject labels of black (0) and outputs the map integration process. FIG. 6 (f) is a diagram showing an integrated map 615 after the difference detection map 613 shown in FIG. 6 (d) and the difference detection map 614 shown in FIG. 6 (e) are subjected to map integration processing. be.

The map integration process in the map integration unit 605 is represented by the following equation (3). In the equation (3), X2 and X3 are difference detection maps input to the map integration unit. Further, in the equation (3), CLIP is a clip function that clips the value to 0 if it is 0 or less and 255 if it is 255 or more, and I is the output of the map integration unit.

I = CLIP (X2 + X3) Equation (3)

By performing the map integration process as represented by the equation (3) in the map integration unit 605, the entire sparse region can be detected from the subject map 403 of FIG. 4 (c). That is, the integrated map 615 shown in FIG. 6 (f) is a sparse area map 615 detected from the subject map 403.

In the present embodiment, an example in which the calculation of the equation (3) is performed in the map integration unit 605 is given, but the present invention is not limited to this. For example, the white (255) part may be TRUE, the black (0) part may be FALSE, and then OR (logical sum) logical operation may be performed to integrate the map.

Further, in the present embodiment, both the black portion detection result of the sparse region by the difference detection unit 603 and the white portion detection result of the sparse region by the difference detection unit 604 are evaluated as the sparse region, but the configuration is limited to this. It is not something that is done. For example, only the detection result of the difference detection unit 603 may be used, or conversely, only the detection result of the difference detection unit 604 may be used.

The MEDIAN filter unit 606 of FIG. 6A is a filter processing unit that applies a median filter to the sparse area map 615 generated by the map integration unit 605. FIG. 6 (g) is a diagram showing a sparse region map 616 after the median filter processing is performed on the sparse region map 615 shown in FIG. 6 (f). The median filter process for the sparse area map 615 is an isolated area removal process for removing the isolated area of the sparse area map 615. That is, when the median filter processing is performed on the sparse area map 615, fine lines and fine points in the sparse area map 615 can be removed, and an erroneous determination area of the sparse area map generated in the contour portion of the subject or the like can be removed. Can be removed.

The sparseness degree calculation unit 607 of FIG. 6A measures the number of pixels of the white subject label portion of the sparse area map 616 after the median filter processing, and the measured pixel number occupies the total number of pixels in the map. The sparseness degree calculation process is performed so that the ratio is calculated as the sparseness degree. Then, the sparseness degree calculation unit 607 outputs the calculated information of the sparseness degree degree 609 to the subject map synthesis unit 303 of FIG. 3 described above. In the present embodiment, the degree of sparseness is calculated as a ratio of the number of pixels, but for example, the number of pixels of the white subject label portion of the sparse area map may be used as the value of the degree of sparseness as it is.

FIG. 7 is a diagram showing a configuration example of the expansion filter unit 601 of FIG. 6A. The expansion filter unit 601 includes a MAX filter unit 701, a MEDIAn filter unit 702, and a MIN filter unit 703.
FIG. 8 is a diagram showing a configuration example of the contraction filter unit 602 of FIG. 6A. The contraction filter unit 602 includes a MIN filter unit 801 and a MEDIAn filter unit 802, and a MAX filter unit 803.
The operations of the MAX filter units 701 and 803, the MEDIA filter units 702 and 802, and the MIN filter units 703 and 801 shown in FIGS. 7 and 8 will be described with reference to the flowchart of FIG. In the flowchart of FIG. 9, the MAX filter, the MEDIAn filter unit, and the MIN filter unit are not distinguished and are simply referred to as a filter unit.

In step S901, the filter unit integrates the values of the peripheral pixels of the pixel of interest for each of the pixels of interest in the input subject map 608 of FIG. 6A, and compares the integrated value ΣPix with the threshold value th. Then, if the integrated value ΣPix is equal to or more than the threshold value th, the process proceeds to step S902 and outputs a white value (255), while if the integrated value ΣPix is less than the threshold value th, the process proceeds to step S903. The process proceeds and the black value (0) is output.

Here, when the range of peripheral pixels (referred to as a reference range) with respect to the pixel of interest is a range of 7 × 7 pixels in the vertical and horizontal directions centered on the pixel of interest, the threshold value th is set to 255 × 1 = 255, so that the filter unit is concerned. Operates as a MAX filter unit. Further, by setting the threshold value th to 255 × 7 × 7 = 12495, the filter unit operates as a MIX filter unit. Further, by setting the threshold value th to 255 × (7 × 7/2) = 255 × (24.5) = 255 × 25 = 6375, the filter unit operates as a median filter unit.

Due to the operation of the flowchart of FIG. 9, in the case of the expansion filter unit 601 of FIG. 7, in the MAX filter unit 701, the white (255) subject label portion of the input subject map 608 is uniformly expanded, and the black (0) of the sparse region is uniformly expanded. ) Non-subject label part will be filled with white. After that, the portion where the contour portion of the subject is expanded by the MIN filter unit 703 is contracted and restored. As a result, only the black (0) non-subject label portion in the sparse region can be changed. Further, the median filter unit 702 fills the unfilled portion of the black (0) non-subject label portion in the sparse region with white (255). As a result, the unfilled portion of the black (0) non-subject label portion does not spread again in the processing of the MIN filter unit 703 in the subsequent stage.

Further, in the case of the contraction filter unit 602 of FIG. 8, in the MIN filter unit 801 the white (255) portion of the input subject map 608 is uniformly contracted, and the white subject label portion of the sparse region is filled with black (0). After that, the part where the subject contour portion is contracted by the MAX filter unit 803 is expanded and restored. As a result, only the white portion of the sparse region can be changed. Further, the median filter unit 802 fills the unfilled portion of the white subject label portion in the sparse region with black, so that the unfilled portion of the white portion does not spread again in the processing of the MAX filter unit 803 in the subsequent stage.

FIG. 10 is a flowchart for explaining the operation of the control unit 101 when still image shooting is performed in the digital camera 100 of FIG.
As a process of step S1001, the control unit 101 performs EVF imaging control for displaying the EVF image on the display unit 109 until the shutter button (not shown) is in a so-called half-pressed state (SW1 is on).

Next, in step S1002, the control unit 101 determines whether or not the shutter button is in the half-pressed state (SW1 on). When the control unit 101 determines that it is not in the half-pressed state (SW1 on), the process returns to step S1001, while the shutter button is operated by the user to enter the half-pressed state (SW1 on). If it is determined, the process proceeds to step S1003.

Proceeding to step S1003, the control unit 101 performs an autofocus (AF) process for driving and controlling the focus lens of the optical system 104 to focus on the subject.

Next, in step S1004, the control unit 101 determines whether or not the shutter button is in the so-called fully pressed state (SW2 on). When the control unit 101 determines that it is not in the fully pressed state (SW2 on), it returns the process to step S1001, while when it determines that it is in the fully pressed state (SW2 on), it performs the process in step S1005. Proceed.

When the process proceeds to step S1005, the control unit 101 controls each unit to capture a still image.
After that, when the process proceeds to step S1006, the control unit 101 controls the image processing unit 107 to perform image processing including the correction processing according to the present embodiment. The correction process in the image processing unit 107 is a correction process as described with reference to FIG. 3 and the like described above, and is a combination of any or two or more of, for example, lighting correction, sharpness adjustment, background blurring, and background contrast adjustment. This is a correction process.

In the example of the flowchart of FIG. 10, an example in which correction processing is performed only on the captured still image is given, but the present invention is not limited to this example. For example, correction processing may be performed during EVF imaging. If the correction process is performed during the EVF imaging, the user can release the recorded still image while checking the finish condition of the recorded still image in advance by viewing the EVF image, which is highly convenient.

Further, in the description of the above-described embodiment, the sparse region is detected by detecting and integrating the black (non-subject label) portion and the white (subject label) portion of the sparse region of the subject map. It is not limited. For example, the sparseness determination unit 302 shown in FIG. 3 may have a configuration as shown in FIG. 11 (a). In the configuration example of FIG. 11, the expansion filter unit 601, the contraction filter unit 602, the median filter unit 606, the sparseness degree calculation unit 607, the subject map 608, and the sparseness degree degree 609 are the same as those in FIG. Omit.

In the sparseness determination unit 302 having the configuration of FIG. 11, the expansion filter unit 601 outputs the post-expansion subject map 611 shown in FIG. 6 (b), and the contraction filter unit 602 displays the above-mentioned FIG. 6 (c). After the contraction shown, the subject map 612 is output. The expanded subject map 611 and the contracted subject map 612 are input to the difference detection unit 1105.

The difference detection unit 1105 performs the same calculation as the above-mentioned equation (2) to detect the difference between the expanded subject map 611 and the contracted subject map 612. As a result of the difference detection process by the difference detection unit 1105, a sparse region map 615 similar to that shown in FIG. 6 (f) described above is generated. In the case of the configuration example of FIG. 11, the map of the difference detection processing result by the difference detection unit 1105 corresponds to the third sparse area map.

The median filter unit 606 eliminates erroneous detection of the subject contour portion of the sparse area map 615 and outputs the sparse area map 616 similar to FIG. 6 (g) described above.
In the case of the configuration example of FIG. 11, the configuration and calculation for the difference detection process and the map integration process of the subject map can be reduced.

As described above, according to the present embodiment, for example, in an image processing device that adaptively applies image correction and image effects to each subject area, it is possible to suppress the occurrence of artifacts due to insufficient subject area extraction accuracy. ..

In the above-described embodiment, the spatial filter such as the MAX filter unit and the MIN filter unit detects the change in the sparse region and calculates the degree of sparseness, but the present invention is not limited to this. For example, the degree of sparseness may be calculated by scanning the subject map vertically or horizontally and counting the number of times the white subject label and the black non-subject label are toggled. Also, for areas where white (subject label) and black (non-subject label) are frequently toggled, the width at which white persists when changing from white to black is defined as the number of white areas in the sparse area, and from black to white. The width in which black lasts when it changes may be counted as the number of black portions in the sparse region. With this configuration, the calculation cost can be reduced as compared with the use of a spatial filter.

Further, for example, the degree of sparseness may be calculated by analyzing the frequency band corresponding to the sparse region of the subject map by using the discrete Fourier transform (FFT) or the discrete cosine transform (DCT). That is, the sparseness determination unit 302 performs frequency domain conversion processing for converting the subject map extracted using the evaluation value map into a frequency domain to generate a frequency domain map, and calculates the degree of sparseness based on the frequency domain map. In the case of this example, the sparseness determination unit 302 has a sparse frequency range determined in advance, and the subject map is divided into small blocks and frequency conversion processing is performed for each small block of the frequency domain map. It is determined whether or not the response is equal to or higher than the threshold value. Then, the sparseness determination unit 302 calculates the degree of sparseness according to the number of small blocks showing a response equal to or greater than a predetermined value. More specifically, the sparseness determination unit 302 performs FFT processing on the subject map, for example, for each block of 8 × 8 pixels, and determines the number of blocks of blocks having a frequency response higher than the threshold value corresponding to the sparse area. Count and let the percentage of those blocks be the degree of sparseness. With this configuration, the frequency band in the sparse region can be determined more finely than using a spatial filter.

Further, in the present embodiment, the phase difference distribution (for example, the focus distribution based on the defocus amount distribution) is used as the evaluation value distribution, but the present invention is not limited to this. The evaluation value distribution may be, for example, a distribution of a shift amount representing the amount of deviation (that is, parallax) between the A image and the B image. The shift amount may be expressed in units of length such as micrometer by multiplying the detection pitch (arrangement pitch of pixels of the same type). Further, for example, the evaluation value distribution may be a distribution of values obtained by normalizing the defocus amount with the depth of focus (2Fδ or 1Fδ. F is the aperture value and δ is the permissible circle of confusion diameter). The aperture value F may be a fixed value on the entire surface with the aperture value near the center of the image height as a representative value, or an aperture value distribution that takes into account that the aperture value of the peripheral image height becomes dark due to the eclipse of the optical system 104 is applied. You may do so.

Further, in the present embodiment, an example of acquiring the focus information distribution of an image, for example, the defocus amount distribution by the phase difference distance measurement method as an evaluation value map has been given, but the present invention is not limited to this. For example, the evaluation value map may be generated based on the contrast information distribution acquired from the image group obtained by sequentially changing the subject distance, that is, the focus position by the contrast ranging method. Further, for example, the evaluation value map may be generated based on the distance information distribution obtained by converting the defocus amount on the image plane side into the distance value on the object plane side. Further, the distance measurement method for acquiring the distance information distribution is not limited to the phase difference distance measurement method, the contrast distance measurement method, or the passive method based on image features. For example, as the distance measuring method, a TOF (Time Of Flight) method or an active method for comparing the presence or absence of strobe reflected light may be used. Furthermore, a method that does not depend on the subject distance may be used. For example, a subject map is generated based on an optical flow that maps a motion vector distribution, a color label map that is labeled based on color information, and a semantic region division based on machine learning. May be done. When using semantic region division, it is necessary to use another method for the alternative silhouette, but it is sufficient to select a robust method depending on the scene change, such as using a humanoid fixed shape map. That is, the evaluation value map may be generated based on at least one of the focus information distribution, the distance information distribution, the motion vector information distribution, the color labeling information distribution, and the semantic region division by machine learning of the image.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
The above-described embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

100: Digital camera, 101: Control unit, 107: Image processing unit, 110: Focus map processing unit, 301: Subject area extraction unit, 302: Sparseness determination unit, 303: Subject map composition unit, 304: Addition unit, 305: Correction processing unit

Claims (15)

A map acquisition method that acquires the evaluation value distribution corresponding to the image as an evaluation value map,
A map generation means for generating a first subject map based on a subject area extracted from the image using the evaluation value map, and a map generation means.
A degree acquisition means for acquiring the degree of sparseness, which represents the degree of the sparse area included in the first subject map, and
It has a correction means for performing correction processing on the image using at least one of the first subject map and the second subject map generated without using the evaluation value map.
The image processing apparatus is characterized in that the correction processing is performed by preferentially using the second subject map over the first subject map as the degree of sparseness increases. The map generation means generates the first subject map classified into at least two label areas, that is, a subject label representing a subject area and a non-subject label representing a non-subject.
The degree acquisition means
A third subject map in which the region where the subject labels are sparsely distributed is changed by a predetermined process on the first subject map is generated.
A fourth subject map in which the region where the non-subject labels are sparsely distributed is changed by a predetermined process on the first subject map is generated.
The image processing apparatus according to claim 1, wherein the degree of sparseness is calculated based on at least one of the third subject map and the fourth subject map. The degree acquisition means
A first sparse area map is generated based on the first subject map and the third subject map.
A second sparse area map is generated based on the first subject map and the fourth subject map.
The image processing apparatus according to claim 2, wherein the degree of sparseness is calculated based on at least one of the first sparse area map and the second sparse area map. The degree acquisition means generates a third sparse area map based on the third subject map and the fourth subject map, and calculates the degree of sparseness based on the third sparse area map. The image processing apparatus according to claim 2. The predetermined process for generating the third subject map includes a filter process for contracting the label area of the subject label and then expanding at least the label area of the subject label. The image processing apparatus according to any one of claims 2 to 4. The predetermined process for generating the fourth subject map is characterized by including a filter process of shrinking the label area of the non-subject label and then expanding at least the label area of the non-subject label. The image processing apparatus according to any one of claims 2 to 4. The image processing apparatus according to any one of claims 3 to 6, wherein the degree acquisition means also performs an isolated area removing process for removing an isolated area from the sparse area map. The map generation means generates the first subject map classified into at least two label areas, that is, a subject label representing a subject area and a non-subject label representing a non-subject.
The image processing apparatus according to claim 1, wherein the degree acquisition means calculates the degree of sparseness based on the number of times the subject label and the non-subject label toggle. The first aspect of claim 1, wherein the degree acquisition means converts the first subject map into a frequency domain to generate a frequency domain map, and calculates the sparseness based on the frequency domain map. Image processing device. The degree acquisition means converts the first subject map into a frequency domain for each small block to generate a frequency domain map, and determines a predetermined sparse frequency range for each small block of the frequency domain map. The image according to claim 9, wherein it is determined whether or not the response is equal to or higher than the threshold value, and the degree of sparseness is calculated according to the number of small blocks showing the response equal to or higher than a predetermined threshold value. Processing equipment. The evaluation value map is characterized by including one of a focus information distribution, a distance information distribution, a motion vector information distribution, a color labeling information distribution, and a semantic region division by machine learning of the image. The image processing apparatus according to any one of claims 1 to 10. The eleventh aspect of claim 11 is characterized in that the focus information distribution is a parallax information distribution acquired from image groups having different viewpoints, or a contrast information distribution acquired from image groups obtained by sequentially changing the focus positions. The image processing apparatus described. The image processing according to claim 12, wherein the parallax information distribution includes one of a map based on a shift amount representing parallax, a map based on a defocus amount, and a map based on a distance value. Device. An image processing method executed by an image processing device.
A map acquisition process that acquires the evaluation value distribution corresponding to the image as an evaluation value map, and
A map generation step of generating a first subject map based on a subject area extracted from the image using the evaluation value map, and a map generation step.
A degree acquisition step for acquiring the degree of sparseness, which represents the degree of the sparse area included in the first subject map, and
It has a correction step of performing correction processing on the image using at least one of the first subject map and the second subject map generated without using the evaluation value map.
The image processing method is characterized in that, in the correction step, the correction processing is performed by preferentially using the second subject map over the first subject map as the degree of sparseness increases. A program for causing a computer to function as each means included in the image processing apparatus according to any one of claims 1 to 13.

Images