JP7534866B2 - Image processing device and method, program, and storage medium - Google Patents

Description

The present invention relates to an image processing device that performs image processing on an area-by-area basis on an image captured by a digital camera or the like.

Conventionally, correction methods have been proposed that extract a subject area and correct the brightness of the subject area, as well as image processing devices that correct the gradation of areas other than the subject area.

For example, Patent Document 1 discloses a technique for determining a subject area and correcting the lighting of the subject area. By using a relighting process that uses a virtual light source, it is possible to brighten dark areas such as shadows caused by ambient light, thereby obtaining a desirable image.

In such cases, a method for detecting the subject area is known in which distance information is acquired and an area that is within a predetermined distance range relative to the distance value of the main subject is detected as the subject area.

There are several methods for acquiring distance information, but a typical method is the stereo method, in which images are acquired from multiple viewpoints, and distance is calculated based on the principle of triangulation using the parallax calculated from the pixel correspondence in each image. By acquiring distance information and detecting the subject area, it becomes possible to distinguish between subject areas in the foreground and those in the background.

For example, when a virtual light source is directed at a subject in the foreground and when it is directed at a subject in the background, it is possible to control the amount of illumination (intensity) of the virtual light source depending on the difference in distance from the virtual light source to the subject.

Patent Document 2 also discloses a method for detecting multiple subject regions and performing gradation correction for each subject region, such as the region of the main subject and regions other than the main subject.

The subject area estimation methods used in these methods include a method of estimating the human body area from the results of face detection, a method of estimating the subject area using machine learning, and a method of detecting the subject area by obtaining distance information.

However, if the subject area estimation does not go well, unnatural corrections can be made, such as when correcting the brightness of a person area, the background area also being brightened, or when correcting the background area to improve contrast, the contrast can be improved not only in the background area but also in the main subject area.

To address these issues, Patent Document 1 discloses a method for switching between a means for acquiring subject area information from subject distance information and a means for acquiring subject area information from an image, based on the relationship between the shooting conditions and the subject conditions, and the results of image analysis of the subject.

JP 2018-182700 A JP 2014-153959 A

However, subject area estimation methods that use image information such as face detection information and subject continuity have difficulty correctly estimating the subject area when the boundaries between the skin or clothing color and the background color are ambiguous. Subject area detection methods that use machine learning have the same problem. Methods that use distance information have the problem of being unable to correctly estimate the subject area for objects that are close to the subject.

The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device that can perform more natural correction when performing image correction for each area of a subject.

The image processing device of the present invention is characterized in that it comprises a detection means for detecting a subject area from an image, a first image processing means for performing a relighting process on the subject area to correct brightness using a virtual light source , a second image processing means for performing a contrast correction process on an area other than the subject area, a first determination means for determining a first reference area in the subject area , which is close in size to the subject area and smaller than the subject area and which is to be subjected to the relighting process , and a second determination means for determining a second reference area , which is close in size to the subject area and larger than the subject area, in order to determine the area to be subjected to the contrast correction process.

The present invention makes it possible to perform more natural corrections when performing image corrections for each area of a subject.

1 is a block diagram showing the configuration of a digital camera according to an embodiment of the present invention; FIG. 2 is a block diagram showing a process flow according to an embodiment. FIG. 4 is a block diagram showing the flow of background region correction processing in an embodiment. FIG. 11 is a block diagram showing the flow of a person area correction process according to an embodiment. 1 is a conceptual diagram illustrating a shooting scene to which the present invention is applied.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

The following describes a digital camera, which is one embodiment of an image processing device of the present invention. In this embodiment, a digital camera (imaging device) is described as an example of an image processing device, but the image processing device of the present invention is not limited to an imaging device and may be, for example, a personal computer (PC), etc.

In general, when photographing a person in a backlit scene, if the photograph is taken so that the person subject is properly exposed, the background area will be overexposed. Conversely, if the photograph is taken so that the background area is properly exposed, the person area will be underexposed. In such a case, the photograph will be taken under shooting conditions that are intermediate between the shooting conditions in which the person area is properly exposed and the shooting conditions in which the background area is properly exposed. However, an image taken in this way will have a dark person and insufficient contrast in the background. Figure 5(a) is a conceptual diagram that assumes a shooting scene to which the present invention is applied. It shows a situation in which the contrast of the background is reduced and the person subject is photographed dark.

To make such images look good, it is desirable to brighten the subject area and also to perform processing to improve the contrast of the background area. In particular, in landscape scenes where the background area is a distant mountain, ocean, or lake, the effects of water vapor and haze in the air often reduce contrast, so there is a strong demand to improve the contrast of the background area.

In this embodiment, the background region is corrected by a correction process that improves contrast, and the subject region is corrected by a relighting correction process that corrects lighting, such as the brightness of the subject and the direction of the light source illuminating the subject.

<Digital Camera Configuration>
FIG. 1 is a block diagram showing the configuration of a digital camera 100 according to the present embodiment.

In the digital camera 100 shown in FIG. 1, light incident via a lens group 101 (image capture optical system) including a zoom lens and a focus lens, and a shutter 102 with an aperture function is photoelectrically converted in an image capture unit 103. The image capture unit 103 is configured with image capture elements such as CCD and CMOS elements, and the electrical signal obtained by photoelectric conversion is output as an image signal to an A/D converter 104. The A/D converter 104 converts the analog image signal output from the image capture unit 103 into a digital image signal (image data) and outputs it to an image processing unit 105.

The image processing unit 105 performs various image processing such as color conversion processing such as white balance, gamma processing, contour enhancement processing, and color correction processing on image data from the A/D converter 104 or image data read from the image memory 106 via the memory control unit 107. The image data output from the image processing unit 105 is written to the image memory 106 via the memory control unit 107. The image memory 106 stores the image data output from the image processing unit 105 and image data to be displayed on the display unit 109. The face/facial organ detection unit 113 detects facial areas and facial organ areas where a person's face and organs are present from the captured image.

The image processing unit 105 performs a predetermined evaluation value calculation process using the face detection results and face organ detection results of the face and face organ detection unit 113 and the captured image data, and the system control unit 50 performs exposure control and focus adjustment control based on the obtained evaluation value. This allows TTL (through-the-lens) type AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, etc. to be performed.

The D/A converter 108 also converts the digital image data for display stored in the image memory 106 into an analog signal and supplies it to the display unit 109. The display unit 109 displays on a display device such as an LCD in accordance with the analog signal from the D/A converter 108.

The codec unit 110 compresses and encodes the image data stored in the image memory 106 based on standards such as JPEG and MPEG. The system control unit 50 stores the encoded image data in a recording medium 112 such as a memory card or a hard disk via an interface (I/F) 111. The codec unit 110 also decodes and expands image data read from the recording medium 112 via the I/F 111, and stores the image in the image memory 106. The image data stored in the image memory 106 can then be displayed on the display unit 109 via the memory control unit 107 and D/A converter 108, allowing the image to be reproduced and displayed.

The relighting processing unit 114 performs relighting processing (re-illumination processing) to correct the brightness of the captured image by applying a virtual light source to it. The image synthesis processing unit 115 synthesizes two types of images according to a synthesis map. The synthesis map represents the synthesis ratio of the two types of images.

The system control unit 50 controls the entire system of the digital camera 100. The non-volatile memory 121 is composed of memory such as an EEPROM, and stores programs and parameters necessary for processing by the system control unit 50. The system control unit 50 implements each process of this embodiment, which will be described later, by expanding the programs recorded in the non-volatile memory 121 and the constants and variables for the operation of the system control unit 50 into the system memory 122 and executing them. The operation unit 120 accepts operations such as menu settings and image selection by the user.

<Flow of image correction process>
FIG. 2 is a block diagram showing the process flow in this embodiment.
2, an input image is input to an image processing unit 105, and a reference image is generated. The reference image generated here is a reference image for face detection and estimation of a human body region, and is an image with image quality settings suitable for face detection, region detection, etc., which are different from the image with white balance, color, etc. set by the user. For example, it is preferable to set the settings to make it easy to detect a face or human body region even in an image with many dark areas or high color saturation, such as auto white balance, color and saturation: standard setting, dark area correction: strong, brightness setting: bright, etc.

The face and facial organ detection unit 113 generates facial area information 203 using the reference image generated by the image processing unit 105. The facial area information 203 includes information such as contour information of the facial area, position information of facial organs such as the eyes, nose, and mouth in the face and their angles relative to the image, and face detection reliability indicating the detection accuracy for such position information. Generally, methods using template matching or machine learning are used for detecting faces and facial organs.

The background contrast correction area determination unit 300 determines an area in which contrast is to be improved based on face information and a reference image. The contrast enhancement unit 310 enhances the contrast of the contrast enhancement area (contrast correction processing area) determined by the background contrast correction area determination unit 300.

The relighting correction area determination unit 400 determines the area for which lighting correction is to be performed based on the face information and the reference image. The relighting processing unit 410 performs relighting processing on the lighting correction area determined by the relighting correction area determination unit 400.

The contrast enhancement processing performed by the contrast enhancement processing unit 310 may be a commonly used method, and may be gamma processing or sharpness enhancement processing that improves the ratio of dark and light areas as a tone conversion characteristic, or processing that enhances only local contrast. In this embodiment, local contrast enhancement processing is applied.

Figure 3 is a block diagram showing the processing flow of the background contrast correction area determination unit 300 and the contrast enhancement unit 310 in this embodiment.

In FIG. 3, the subject region setting unit for background correction 301 determines the subject region for background correction from the face region information 203. The person region specification unit for background correction 302 specifies the background region from the reference image 202 and the subject region determined by the subject region setting unit for background correction 301, and generates the background contrast correction region map 210. The synthesis processing unit 115 determines the synthesis processing ratio for each region based on the region and level information of the background contrast correction region map 210, and performs synthesis processing of the input image 201 and the output image of the contrast enhancement processing unit 311.

The background contrast correction area map 210 generated by the background correction person area identification unit 302 is area information indicating areas determined to be background areas, and is information on parts not determined to be person areas. In other words, it is area information in which the area determined to be a person area and the determination level at which it was determined to be a person are inverted, and is information equivalent to area information indicating the correction level of contrast correction in the synthesis processing unit 115.

Figure 4 is a block diagram showing the flow of relighting processing in the relighting correction area determination unit 400 and the relighting processing unit 410 in Figure 2.

The subject area setting unit 401 sets the subject area based on the face area information 203. The subject area identification unit 402 generates the relighting correction area map 220 using the subject area set by the subject area setting unit 401 and the reference image 202.

The virtual light source calculation unit 411 generates light source information of a virtual light source using the input image 201 and the face region information 203. The virtual light source component calculation unit 412 generates a relighting gain map 221 using the virtual light source information calculated by the virtual light source calculation unit 411 and the relighting correction region map 220. The relighting gain map 221 is a gain map that expresses a correction gain that corrects the brightness of the subject region for each image region. The gain multiplication unit 413 multiplies the contrast-multiplied image 211 by a gain for each subject region according to the relighting gain map.

The method of generating light source information of a virtual light source in virtual light source calculation unit 411 and the method of calculating a virtual light source component in virtual light source component 412 are described in detail in Patent Document 1, and therefore will not be described in this embodiment. However, Patent Document 1 performs relighting processing during processing in a signal processing circuit, but the relighting processing is performed after degamma processing in the processing flow, and performing relighting processing before performing signal processing in this embodiment can provide approximately the same effect.

<How to apply human body reference image>
Next, a subject region setting method in the background correction subject region setting unit 301 in FIG. 3 and the subject region setting unit 401 in FIG. 4 will be described.

In this embodiment, a person reference image (person reference region) including the face and human body regions is set in advance. The face region of the person reference image has facial organ information such as the eyes, nose, and mouth. The person reference image is enlarged, reduced, and rotated so that the positions of the organ information in the person reference image match those of the facial organ information such as the eyes, nose, and mouth detected from the captured image, thereby adapting it to the region of the actual image. In addition, the person reference image is set to have a gradation that decreases the judgment level of the person region as it approaches the periphery of the person region.

The person reference image for background region correction uses a model (subject model image) in which the human body region of the body is set wider than a general human body model, and the person reference image used by the subject region setting unit 401 uses a model in which the human body region of the body is set narrower than a general human body model. Similarly, for the region corresponding to the head in the person region, the person reference image for background region correction uses a model in which the head region is set wider than a general human body model, and the person reference image used by the subject region setting unit 401 uses a model in which the head region is set narrower than a general human body model.

Figure 5 is a conceptual diagram showing an example of application of a subject model in this embodiment. Figure 5(b) shows a reference image that is applied (targeted) in the case of relighting processing that corrects the brightness of a person area, and for the image of a backlit person in Figure 5(a), the person reference image is set so that it is inside the person.

Figure 5 (c) shows a reference image that is applied when correcting the background region, and for the backlit image of a person in Figure 5 (a), the person reference image is set so that it is outside the person. Also, since there is more organ information such as the eyes and mouth in the face region than in the body region, the accuracy of detecting the face region is high. Therefore, the size of the head region in the reference image may be set to be about the same as the actual image, or the enlargement/reduction ratio with respect to the actual image may be set smaller than that of the human body region. Figures 5 (b) and (c) also show how the ratio of the head region to the actual image of the reference image is set smaller than that of the body region.

The subject region setting method in the background correction person region identification unit 302 in FIG. 3 and the subject region identification unit 402 in FIG. 4 performs subject region identification processing to match the contours of the subject by performing shaping processing with reference to the pixel values of the reference image 202, as described, for example, in JP 2017-11652 A.

In this way, when the subject region identification process that matches the contours is applied to the subject reference image by the subject region identification unit, the background correction person region identification unit 302 for correcting the contrast of the background region sets a person identification region that extends outside the person. Also, because the person identification level decreases from the inside to the outside of the person, the background contrast correction region map that inverts the identified region setting level acts to strengthen the contrast outside the person. In other words, as you get closer to the person region, it is no longer determined to be a background region, and therefore the contrast enhancement process is not applied the closer you get to the person region.

On the other hand, the subject area identification unit 402 in relighting is set so that the inside of the subject area is the correction area, and the amount of subject correction decreases from the inside to the outside of the subject area. At the same time, processing is performed so that the subject area determination does not extend outside the subject. In other words, brightness correction is performed mainly inside the subject, and the brightness correction level decreases toward the outside of the subject. Therefore, it acts to prevent the brightness correction of the person area from extending outside the person area.

The subject region identification process also performs a shaping process on the subject region based on the contour information of the image. However, even in cases where the contour of the subject region cannot be detected, correction problems can be effectively avoided because the person reference image is selected so that it does not overlap with each correction region.

In this embodiment, the person reference image used in the background correction subject region setting unit 301 in FIG. 3 and the subject region setting unit 401 in FIG. 4 may be prepared for each left-right direction of the face. Then, a person reference image with an optimal angle may be selected according to the information of the face direction detected from the image. Also, a three-dimensional model including the face region and the human body region may be prepared, and a human body reference image may be generated according to the face angle. Also, in addition to the reference image for each face angle, a reference image of the face and a reference image of the human body region may be prepared for each angle. Then, a face angle model may be selected according to the face angle detected from the image, and multiple human body regions with different angles from the neck down may be applied to the lower face region of the actual image, and the angle at which the variance of the human body region is the smallest or the similarity within the region is the highest may be used as the reference image for the human body region.

In addition, in this embodiment, it has been described that there are multiple reference images, one for correcting the subject area and one for correcting outside the subject area, and the reference image to be used is selected from these. However, it is also possible to have a common reference image for both correcting the subject area and correcting outside the subject area, and change the magnification of the enlargement/reduction for each of the head area and human body area to which it is applied.

In this embodiment, the subject region is estimated based on the size and orientation of the face region, but this method is not limited to this, and there are other methods such as region estimation methods using machine learning. Even with these methods, if the color and pattern of the background region and the human body region are similar, the human body region is often not extracted correctly, and the present invention can be applied even in such cases.

Other Embodiments
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

50: System control unit, 101: Optical system, 103: Imaging unit, 105: Image processing unit, 113: Face/facial organ detection unit, 114: Relighting processing unit, 115: Composition processing unit

Claims (9)

A detection means for detecting a subject area from an image;
a first image processing means for performing a relighting process on the subject region to correct brightness using a virtual light source ;
a second image processing means for performing a contrast correction process on an area other than the subject area;
a first determination means for determining a first reference area, which is a target of the relighting process in the subject area and has a size close to the subject area and smaller than the subject area;
a second determination means for determining a second reference area having a size close to and larger than the subject area in order to determine an area to be subjected to the contrast correction process ;
An image processing device comprising: 2. The image processing device according to claim 1, wherein the detection means detects a face area and a body area as the subject area, and the first determination means sets a degree to which the first reference area is made smaller than the subject area to be different for the face area and the body area. 3. The image processing apparatus according to claim 2, wherein the first determining means determines a degree to which the first reference region is made smaller than the subject region such that the body region is larger than the face region. 2. The image processing device according to claim 1, wherein the detection means detects a face area and a body area as the subject area, and the second determination means sets a degree to which the second reference area is made larger than the subject area to be different for the face area and the body area. 5. The image processing apparatus according to claim 4, wherein the second determining means determines the extent to which the second reference region is to be made larger than the subject region such that the body region is larger than the face region. 6. The image processing device according to claim 1, wherein the first determination means and the second determination means determine the first reference area and the second reference area by transforming a subject model image based on a face area detection result and the image. A detection step of detecting a subject region from an image;
a first image processing step of performing a relighting process for correcting brightness of the subject region using a virtual light source ;
a second image processing step of performing a contrast correction process on an area other than the subject area;
a first determination step of determining a first reference area, which is a target area for the relighting process in the subject area and has a size close to the subject area and smaller than the subject area;
a second determination step of determining a second reference area having a size close to and larger than the subject area in order to determine an area to be subjected to the contrast correction process ;
13. An image processing method comprising the steps of: A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 5 . 6. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing apparatus according to claim 1.

Images