JP7318251B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

2. Description of the Related Art Conventionally, skin beautification processing has been used to make the skin of a person included in an image look beautiful.
For example, Patent Literature 1 discloses a technique that enables easy setting of the effect of processing (whitening processing) for making the skin of a person look brighter.

JP 2016-012890 A

However, when image processing is used to make a person's skin look beautiful, it has been desired to output a more natural image.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus that outputs a more natural image after performing skin beautification processing.

In order to achieve the above object, an image processing device according to one embodiment of the present invention includes:
an image acquiring means for acquiring a face image including a face area;
a skin-beautifying means for performing skin-beautifying processing on the face image acquired by the image acquisition means;
a first identifying means for identifying a first area, which is an area including a skin beautification reduction area, in the image before skin beautification is performed on the face image;
a second identifying means for identifying a second area, which is an area including a skin beautification reduction area, in the image after performing the skin beautifying process on the face image;
Blending for creating a blended image by blending the image corresponding to the first area specified by the first specifying means and the image corresponding to the second area specified by the second specifying means. means and
a reduction processing means for performing a reduction process for reducing the effect of the skin beautifying process on the blended image created by the blending means;
has
The reduction processing means is characterized by blending the blend image and the blend image that has been subjected to the reduction processing .

According to the present invention, it is possible to provide an image processing apparatus that outputs a more natural image after performing skin beautification processing.

1 is a block diagram showing the hardware configuration of an imaging device according to an embodiment of an image processing device of the present invention; FIG. FIG. 4 is a schematic diagram for explaining generation of a beautiful skin image according to the present embodiment; FIG. 4 is a schematic diagram showing an example of a specific method of the first α-blending; FIG. 4 is a schematic diagram showing an example of creating an organ map used to create a mask image; FIG. 10 is a schematic diagram showing an example of creating a mask image; FIG. 10 is a diagram showing a histogram and a lip color level weighting table relating to setting of lip color level weighting; It is a figure which shows LUT (Look Up Table) of V correction. FIG. 10 is a schematic diagram showing an example of creating a partially corrected image; FIG. 2 is a functional block diagram showing a functional configuration for executing skin beautification processing among the functional configurations of the imaging apparatus of FIG. 1; FIG. 10 is a flowchart for explaining the flow of beautiful skin image generation processing executed by the imaging apparatus of FIG. 1 having the functional configuration of FIG. 9; FIG. 10 is a flowchart for explaining the flow of mask image creation processing in beautiful skin image generation processing. 10 is a flowchart for explaining the flow of partial correction image generation processing in the beautiful skin image generation processing.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing the hardware configuration of an imaging device 1 according to an embodiment of the image processing device of the present invention.
The imaging device 1 is configured as, for example, a digital camera.

As shown in FIG. 1, the imaging device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input/output interface 15, and an imaging device. A unit 16 , an input unit 17 , an output unit 18 , a storage unit 19 , a communication unit 20 and a drive 21 are provided.

The CPU 11 executes various processes according to programs recorded in the ROM 12 or programs loaded from the storage unit 19 to the RAM 13 .

The RAM 13 also stores data necessary for the CPU 11 to execute various processes.

The CPU 11 , ROM 12 and RAM 13 are interconnected via a bus 14 . An input/output interface 15 is also connected to this bus 14 . The input/output interface 15 is connected with an imaging unit 16 , an input unit 17 , an output unit 18 , a storage unit 19 , a communication unit 20 and a drive 21 .

The imaging unit 16 includes an optical lens unit and an image sensor (not shown).

The optical lens section is composed of a lens for condensing light, such as a focus lens and a zoom lens, in order to photograph an object.
A focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. A zoom lens is a lens that can freely change the focal length within a certain range.
The optical lens unit is also provided with peripheral circuits for adjusting setting parameters such as focus, exposure, white balance, etc., as required.

The image sensor is composed of a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens section. Therefore, the photoelectric conversion element photoelectrically converts (captures) a subject image, accumulates an image signal for a certain period of time, and sequentially supplies the accumulated image signal to the AFE as an analog signal.
The AFE executes various signal processing such as A/D (Analog/Digital) conversion processing on this analog image signal. A digital signal is generated by various signal processing and output as an output signal of the imaging unit 16 .
Such an output signal of the imaging unit 16 is appropriately supplied to the CPU 11, an image processing unit (not shown), or the like as captured image data.

The input unit 17 is composed of various buttons and the like, and inputs various kinds of information according to user's instruction operation.
The output unit 18 includes a display, a speaker, and the like, and outputs images and sounds.
The storage unit 19 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores data of various images.
The communication unit 20 controls communication with another device (not shown) via a network including the Internet.

A removable medium 31 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is mounted in the drive 21 as appropriate. A program read from the removable medium 31 by the drive 21 is installed in the storage unit 19 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 19 in the same manner as the storage unit 19 .

In the imaging apparatus 1 configured as described above, skin beautification processing is applied to the skin in an image obtained by photographing a face. is appropriately corrected (hereinafter referred to as “beautiful skin image”). That is, the imaging device 1 generates a beautiful skin image by correcting the lip color corrected by the skin beautifying process applied to the facial skin to an appropriate color to restore (reduce) the effect of the skin beautifying process.

That is, the skin beautifying process is performed only on the skin color area, except for the target part, which is masked. It will be affected, such as the color of the lips will become lighter. On the other hand, simply pasting the lip portion of the original image before skin beautification to the lip portion after skin beautification increases discomfort with the surroundings.

Therefore, in the imaging apparatus 1 of the present embodiment, the lip portion in the image before the skin beautification process (hereinafter referred to as the "original image") and the image after the skin beautification process (hereinafter referred to as the "whole skin beautification image"). ) is blended with the lip portion to generate an image (hereinafter referred to as “first blended image”) in which the skin beautifying process for the lip portion is reduced. Then, an image (hereinafter referred to as a "partially corrected image") is generated by performing predetermined correction processing (color correction, smoothing processing, etc.) on the first blended image, and the partially corrected image is blended with the first blended image. An image (hereinafter referred to as a "second blend image") is generated. Then, by pasting the second blend image to the original position (lip portion) of the overall beautiful skin image, a beautiful skin image in which the lip color is corrected to an appropriate color is generated.

FIG. 2 is a schematic diagram for explaining generation of a beautiful skin image in this embodiment.
When the beautiful skin image of this embodiment is generated, as shown in FIG. 2, regions including the lips are cut out from the original image and the overall beautiful skin image, respectively, and mask images are used for processing the lip regions. is synthesized by the first α-blending. In the present embodiment, in the first α-blending, different coefficients are set for the luminance component and the color component (saturation) of the image, respectively, and α-blending is performed, as will be described later. As a result, the first blended image is generated in which the skin beautifying process for the lip portion is reduced. Then, an image (hereinafter referred to as a "color-corrected image") is created by applying color correction processing so that the color of the lips in the first blended image is made more appropriate. Furthermore, a partially corrected image is created by applying smoothing processing to the color corrected image to suppress thick wrinkles, and applying edge enhancement processing to emphasize thin edges. In the partially corrected image, color correction, smoothing, or edge enhancement has been applied to areas other than the lips, so that the effect of these correction processes appears only on the lips, a mask image that masks areas other than the lips is used. , the partially corrected image and the first blended image are synthesized by the second α-blending. As a result, a second blended image in which the color of the lip portion and the skin color around the lip are color-corrected in a well-balanced manner is created. Note that the pixel value of the mask image is the α value that is referred to when synthesizing images by α blending.
After that, the created second blend image is pasted to the original position (the lip portion) of the whole skin-beautifying image to generate a skin-beautifying image.

[First α blend]
FIG. 3 is a schematic diagram showing an example of a specific method of the first α-blending.
In the present embodiment, regions including the lips cut out from the original image and the overall beautiful skin image are combined by the first α-blending using a mask image for processing the lip region.
At this time, α-blending is performed using a different coefficient for each channel of the image of the region including the lips cut out from the original image and the overall beautiful skin image.

That is, as shown in FIG. 3, among the channels of each image represented in the YUV color space, for the Y value (luminance component), the α value represented by the pixel of the mask image is added by a first coefficient (for example, 0.5). is multiplied by to adjust the α value, and the Y value of each image is α-blended by the adjusted α value (first adjustment value). In addition, among the channels of each image represented in the YUV color space, the U value (blue hue and saturation components) and V value (red hue and saturation components) are represented by the pixels of the mask image. The α value is adjusted by multiplying the α value by a second coefficient (eg, 0.75), and the U and V values of each image are α-blended by the adjusted α value (second adjustment value). Note that, among the channels of each image expressed in the YUV color space, different α values may be adjusted for the U value and the V value.
Then, the Y value calculated by the α value (first adjustment value) adjusted by the first coefficient, and the U value and V value calculated by the α value (second adjustment value) adjusted by the second coefficient. is created as each component of the YUV color space (first blended image).

[Creation of organ map]
FIG. 4 is a schematic diagram showing an example of creating an organ map used to create a mask image.
The organ map is a map representing the area of organs (lips in this case) forming the face of the subject.
In this embodiment, the face of the user, who is the subject, is detected from the original image, and each organ that constitutes the face is detected in the detected face. Techniques for realizing detection of the face and each organ are not particularly limited, and existing face detection techniques and existing organ detection techniques can be used.
Then, as shown in FIG. 4A, among the detected organs, the target organ (here, the lip) for which the skin beautifying process is reduced is specified, and the lip organ point P1 is detected. Multiple organ points P1 are detected for each of the upper lip and the lower lip. In the drawing, each of the plurality of organ points P1 is indicated by a white circle, but only one organ point P1 among them is denoted by a reference numeral for convenience of illustration.

Then, based on the positions of the plurality of organ points P1, a region corresponding to the lips is detected. Furthermore, all the coordinate values indicating the positions of the plurality of organ points P1 in the coordinate system used for processing the image are obtained, and the position corresponding to the average value of all the obtained coordinate values is obtained. is identified as the reference point P2.
Note that FIG. 4A also shows measurement regions R1 and R2 set for measuring the color of the lips. The measurement regions R1 and R2 are used in creating a HUE map, which will be described later.
In the organ map shown in FIG. 4(B), the central area including the reference point P2 is set to white (that is, the α value is "1"), and the other areas are set to be black (that is, the α value is "0"). there is
However, if the organ map in which the α value is set in this way is used as it is for image correction, the boundary between the area where image correction is performed and the area where no correction is performed becomes clear, and the image correction becomes unnatural. . Therefore, in order to blur this boundary, the shape of the region corresponding to the lips is reduced at equal ratios in each direction around the reference point P2 to set concentric regions. At this time, concentric regions with different reduction ratios are set stepwise, and the α value decreases stepwise from the central region of the organ map toward the outer region, so that the degree of image correction is suppressed. Then, modify the organ map.
As a result, as shown in FIG. 4(C), the shape of the region corresponding to the detected lips is maintained, and the α value decreases stepwise from the inside to the outside of the organ map. It is possible to create an organ map that is set so that the Note that in this embodiment, the shape of the region corresponding to the lips is reduced at equal ratios in each direction around the reference point P2 to set concentric regions, and from the central region of the organ map to the outer region. Although the α value is decreased stepwise as it approaches, the concentric area may be set by enlarging the organ map in stages obtained by organ detection around the reference point P2.

[Create mask image]
FIG. 5 is a schematic diagram showing an example of creating a mask image.
In this embodiment, a mask image corresponding to a region corresponding to the lips is created in order to perform correction processing for reproducing the color of the lip portion that has become light due to the skin beautifying process.
As shown in FIG. 5, the mask image indicates that the darker the color-corrected image composition ratio is, the whiter the color-corrected image composition ratio is. In this embodiment, the center portion of the lips is strongly color-corrected, and the portion near the outer edge of the lips is configured to be weakly color-corrected in order to reduce the influence of the skin.

Such a mask image is created based on the peak hue value of the HSV color space in the area including the identified lip position, after identifying the position of the lip by detecting facial organs. In facial organ detection according to the present embodiment, as shown in FIG. 4 described above, the positions of a plurality of organ points P1 are detected in order to specify the lip position or the like.

In this embodiment, an organ map representing the lip region is combined with the mask image created based on the peak hue value of the HSV color space, in order to eliminate unnecessary portions such as the nose. to create a realistic mask image.

Specifically, the mask image of the present embodiment is obtained by creating a HUE map and an organ map from an image obtained by cutting out a region including lips from an original image (hereinafter referred to as a “cutout image”), and converting the HUE map into an organ map. It is created by combining with In synthesizing the HUE map and the organ map, the mask image of the present embodiment is obtained by multiplying the α values of the HUE map and the organ map in order to create a map from which spatially unnecessary regions are removed from the HUE map. will be created.

[Create HUE map]
A HUE map is a map that identifies a portion corresponding to the lip color in the clipped image. For the HUE map, first, the clipped image in the YUV color space is converted into the HSV (hue: Hue, saturation: Saturation/Chroma, brightness: Value/Lightness/Brightness) color space. Thereafter, in order to remove noise, the image is blurred with an ε filter and then analyzed (HSV analysis processing) after the HSV conversion. Then, a HUE map is created using the values calculated from the processing result of the HSV analysis processing as the map values of the HUE map.

In the HSV analysis process, lip color level weighting is set to determine the map value at each pixel. The weighting of the lip color level is set based on histograms created for each HSV channel.

FIG. 6 is a diagram showing a histogram and a lip color level weighting table relating to setting of lip color level weighting.
In setting the lip color level, first, measurement regions R1 and R2 (see FIG. 4A) are set near the centers of the upper and lower lips as measurement regions based on a plurality of organ points P1 detected by the organ detection shown in FIG. Then, a histogram of each HSV in each measurement area is created.
Note that the measurement areas on the lips for generating the histogram may be the number of areas, various shapes, positions, and sizes as long as the lip color can be obtained.

Then, a lip color level weighting table is set as shown in FIG. 6(b) corresponding to the mode value of each HSV histogram as shown in FIG. 6(a). In the case of the H channel, weighting is evenly distributed to plus and minus with respect to the mode, and the weighting is set to decrease as the distance from the mode increases. In addition, in the case of the S(V) channel, the peak value has a predetermined width with the mode of the histogram as the reference at the upper left end (right end), and The weighting is set so that the two-stage gradient is gentler than the other.

After that, a HUE map is created using the result of multiplying each pixel by the lip color level calculated according to the weighting for each HSV channel as a map value.
That is, the map value of the HUE map is calculated by the following formula (1).
Map value (Map) of HUE map=Lh×Ls×Lv (1)
"Lh" indicates the H lip color level, "Ls" indicates the S lip color level, and "Lv" indicates the V lip color level.

[Creation of Color Corrected Image]
The color corrected image is created by correcting the V value of the first blended image in this embodiment.
In the present embodiment, for example, when the original lip color is dark, for example, the lip color is corrected so that the correction by the skin beautifying process is not too effective.

FIG. 7 is a diagram showing a V correction LUT (Look Up Table).
As shown in FIG. 7, the V-correction LUT is configured to apply V-correction to dark-colored portions of the lips.

[Create partial correction image]
The partially corrected image is created by performing smoothing processing and edge enhancement processing on the color corrected image created from the first blended image.
FIG. 8 is a schematic diagram showing an example of creating a partially corrected image.
In this embodiment, among the channels of the second blended image expressed in the YUV color space, a color-corrected image is created by correcting the V value (red hue and saturation components). A partially corrected image is created by performing smoothing processing and edge enhancement processing on the value (luminance component).
As a result, thick wrinkles on the lips are suppressed, and a partially corrected image with enhanced resolution can be created.

[Creation of second blend image]
The second blended image is created by alpha-blending the first blended image and the partially corrected image and synthesizing the images based on the mask image. A beautiful skin image is generated by pasting the created second blend image to the original position (lip portion) of the entire beautiful skin image. That is, a partially corrected image in which only the lip region is subjected to color correction and the other skin portion is subjected to skin beautifying processing is pasted on the entire skin beautifying image to which the same skin beautifying processing is performed to obtain a beautiful skin image. generated.

FIG. 9 is a functional block diagram showing a functional configuration for executing skin beautifying processing among the functional configurations of the imaging device 1 of FIG.
Skin-beautifying image generation processing is to apply skin-beautifying processing to the original image that includes the photographed human face, and to correct the lips, which are similar in color to the skin and have been affected by the skin-beautifying processing and have become lighter in color. A series of processing to generate a beautiful skin image by correcting to a natural color.

When executing the beautiful skin image generation process, as shown in FIG. The creation processing unit 55 functions.

An image storage unit 71 and a mask image storage unit 72 are set in one area of the storage unit 19 .

Image data output from the imaging unit 16 is stored in the image storage unit 71 .
The mask image storage unit 72 stores mask image data for masking areas other than the lips.

The image acquisition unit 51 acquires data of a captured image obtained by subjecting an image captured by the imaging unit 16 to development processing, or data of an image to be processed from the image storage unit 71 .

The face detection unit 52 detects a face from an image, and also detects each organ that constitutes the face in the detected face. The face detection unit 52 detects at least left and right pupils and lip organs from the contour shape. As for the lips, the face detection unit 52 detects at least two points above and below the center of the upper lip, two points above and below the center of the lower lip, and two points at the left and right ends of the lip. Detected as P1. Face detection and detection of each organ use known face detection technology and organ detection technology. Further, the face detection and the detection of each organ may be performed by any detection method as long as the detection method is based on information different from the color information of the HSV color space, such as luminance and contour extraction.

The image processing unit 53 performs various image processing on the original image.
Specifically, the image processing unit 53 performs skin beautification processing on the original image, for example, mainly by correcting three elements of saturation reduction, brightness increase, and hue rotation toward blue. As a result, an overall beautiful skin image is created. Further, the image processing unit 53 cuts out a region including the lips from the original image and the entire skin-beautifying image, as shown in FIG.
In addition, the image processing unit 53 performs the first α-blending on the two cutout images obtained by cutting out the region including the lips from the original image and the overall beautiful skin image, based on the mask image created by the mask image creation processing unit 54. The image is synthesized by applying. As a result, a first blended image is created.

In the present embodiment, when performing the first α-blending, the image processing unit 53 converts the Y value (luminance component) among the channels of each image represented in the YUV color space to α The value is multiplied by a first factor (eg, 0.5) to adjust the α value, and the Y value of each image is α-blended by the adjusted α value (first adjustment value). In addition, the image processing unit 53, among the channels of each image represented in the YUV color space, for the U value (blue hue and saturation components) and V value (red hue and saturation components), The α value represented by the pixels of the mask image is multiplied by a second coefficient (for example, 0.75) to adjust the α value, and the U value and V value of each image are adjusted by the adjusted α value (second adjustment value). α blend. Note that, among the channels of each image expressed in the YUV color space, different α values may be adjusted for the U value and the V value.

Then, the image processing unit 53 calculates the Y value calculated by the α value (first adjustment value) adjusted by the first coefficient and the α value (second adjustment value) adjusted by the second coefficient. An image (first blended image) is created in which the U value and V value are each component of the YUV color space.
Further, the image processing unit 53 converts the first blended image and the partial correction image created by the partial correction image creation processing unit 55 into a second blend image based on the mask image created by the mask image creation processing unit 54 . Images are synthesized by applying α-blending. As a result, a second blended image is created. Further, the image processing unit 53 pastes the second blend image to the original position (lip portion) of the overall beautiful skin image. As a result, a beautiful skin image is generated.

The mask image creation processing unit 54 creates mask images used for the first α-blending and the second α-blending so as to obtain an image in which only the lip region is subjected to color correction.

Specifically, as shown in FIG. 4, the mask image creation processing unit 54 identifies the target organ (here, lips) for which the skin beautifying process is reduced from among the various organs detected by the face detection unit 52. , and the lip organ point P1 is detected. At this time, the mask image creation processing unit 54 detects a plurality of organ points P1 for each of the upper lip and the lower lip. Then, the mask image creation processing unit 54 detects the region corresponding to the lips based on the positions of the plurality of organ points P1. Further, the mask image creation processing unit 54 acquires all coordinate values indicating the positions of the plurality of organ points P1 in the coordinate system used for processing the image, and extracts all acquired coordinate values. A position corresponding to the average value of is specified as a reference point P2. Then, the mask image creation processing unit 54 reduces the shape of the region corresponding to the lips at equal ratios in each direction around the reference point P2 to set concentric regions. At this time, the mask image creation processing unit 54 gradually sets concentric regions with different reduction ratios, and the α value decreases step by step from the central region of the organ map toward the outer region, Modify the organ map so that the degree of image correction is reduced.

Also, the mask image creation processing unit 54 converts the YUV color space of the clipped image obtained by clipping the region including the lips from the original image into the HSV color space. In addition, the mask image creation processing unit 54 performs a process of blurring the image with an ε filter for the purpose of noise removal.
Then, the mask image creation processing unit 54 executes HSV analysis processing. In the HSV analysis process, a histogram is created for each HSV channel of the measurement regions R1 and R2 (see FIG. 4A) set near the center of the upper and lower lips (see FIG. 6A). Then, the weighting of the lip color level calculated from the created histogram is created (see FIG. 6(b)). After that, based on the created lip color level weighting, the map value of the HUE map for each pixel is calculated using Equation (1) described above. A HUE map is created from the calculated map value of each pixel.

Also, the mask image creation processing unit 54 synthesizes the created organ map and the HUE map, as shown in FIG. At this time, the mask image creation processing unit 54 creates a mask image by multiplying both the HUE map and the α value of the organ map in order to remove spatially unnecessary regions such as the nose from the HUE map. The mask image creation processing unit 54 stores the created mask image in the mask image storage unit 72 .

The partial correction image creation processing unit 55 executes color correction processing on the first blend image.
Specifically, the partially corrected image creation processing unit 55 performs correction (color correction) on the V value of the first blend image based on the LUT (see FIG. 7). As a result, a color-corrected image is created by performing color correction on the first blended image.

Further, the partial correction image creation processing unit 55 performs smoothing processing and edge enhancement processing on the color correction image.
Specifically, the partial correction image creation processing unit 55 performs smoothing processing and edge enhancement processing on the Y value (luminance component) of the channels of the color correction image expressed in the YUV color space. As a result, a partially corrected image is created.

FIG. 10 is a flowchart for explaining the flow of beautiful skin image generation processing executed by the imaging device 1 of FIG. 1 having the functional configuration of FIG.
The beautiful skin image generation process is started when the user operates the input unit 17 to start the beautiful skin image generation process.
The operation for starting the beautiful skin image generation processing is a photographing instruction operation, and the image captured by the imaging unit 16 in response to the photographing instruction operation is subjected to the development process, and the beautiful skin image is subsequently produced. The generation process may be performed, or the data of the captured image stored in the image storage unit 71 may be selected, and the beautiful skin image generation process may be started for the selected captured image data.

In step S<b>11 , the image acquisition unit 51 uses data of a captured image obtained by subjecting an image captured by the imaging unit 16 to development processing, or data of an image to be processed stored in the image storage unit 71 as an original image. get.

In step S12, the face detection unit 52 performs face detection on the original image and determines whether or not a face has been detected.
If the face is not detected, NO is determined in step S12, and the beautiful skin image generation process ends.
If a face is detected, YES is determined in step S12, and the process proceeds to step S13.

In step S13, the image processing unit 53 performs skin beautification processing on the original image to create an overall skin beautification image. The skin beautification process reduces the saturation of the original image, increases the brightness, and rotates the hue toward blue.

In step S14, the face detection unit 52 detects facial organs in the original image. As a result, at least the positions of the left and right pupils and the lip organs are detected in the original image.

In step S15, the image processing unit 53 cuts out a region containing the detected lips in the original image, and uses the same region as the region cut out from the original image in the whole beautiful skin image as a region containing the lips (cutout image). break the ice.

In step S16, the mask image creation processing unit 54 executes mask image creation processing. As a result of executing the mask image creation process, a mask image for alpha blending is created. Details of the mask image creation process will be described later.

In step S17, the image processing unit 53 synthesizes the two cutout images cut out from the original image and the overall beautiful skin image by applying the first α-blending based on the mask image. At this time, the image processing unit 53 adds a first coefficient (for example, 0.5) to the α value represented by the pixel of the mask image for the Y value (luminance component) among the channels of each image represented in the YUV color space. The α value (first adjustment value) adjusted by multiplying is used to perform α blending, and the U value (blue hue and saturation component) and V value (red hue and saturation component) are , α-blending is performed using an α-value (second adjustment value) adjusted by multiplying the α-value represented by the pixel of the mask image by a second coefficient (for example, 0.75). As a result, a first blended image is created.

In step S18, the partial correction image creation processing unit 55 executes partial correction image creation processing. As a result of executing the partial correction image creation process, a partial correction image is created by performing color correction, smoothing processing, and edge enhancement processing on the first blend image. Details of the partial correction image creation processing will be described later.

In step S19, the image processing unit 53 synthesizes the first blended image and the partially corrected image by applying the second α-blending based on the mask image. As a result, a second blended image is created.

In step S20, the image processing unit 53 pastes the second blended image to the original position (lip portion) of the overall beautiful skin image. As a result, a beautiful skin image is generated in which the skin has been subjected to skin beautification processing and the lip portion has been corrected to an appropriate color.
After that, the beautiful skin image generation process ends.

FIG. 11 is a flowchart for explaining the flow of mask image creation processing in the beautiful skin image creation processing.

In step S31, the mask image creation processing unit 54 converts the YUV color space of the clipped image clipped from the original image into the HSV color space.

In step S32, the mask image creation processing unit 54 blurs the clipped image, which has undergone color space conversion to the HSV color space, using an ε filter.

In step S33, the mask image creation processing unit 54 executes HSV analysis processing. In the HSV analysis process, a histogram is created for each HSV channel of the measurement regions R1 and R2 set near the center of the upper and lower lips (see FIG. 4A). Then, the weighting of the lip color level calculated from the created histogram is set (see FIG. 4(b)). The weighting of the lip color level is set such that, in the case of the H channel, the weighting is evenly distributed between plus and minus with respect to the mode, and the weighting decreases as the distance from the mode increases. In addition, in the case of the S(V) channel, the peak value has a predetermined width with the mode of the histogram as the reference at the upper left end (right end), and The weighting is set so that the two-stage gradient is gentler than the other.

In step S34, the mask image creation processing unit 54 creates a HUE map. The map value in the HUE map is calculated using the above-described formula (1) based on the created lip color level weighting.

In step S35, the mask image creation processing unit 54 creates an organ map. At this time, the mask image creation processing unit 54 creates the organ map so that the α value decreases stepwise from the central area of the organ map toward the outer area, thereby suppressing the degree of image correction.

In step S36, the mask image creation processing unit 54 synthesizes the created organ map and the HUE map. At this time, the mask image creation processing unit 54 creates a mask image by multiplying both the HUE map and the α value of the organ map in order to remove spatially unnecessary regions such as the nose from the HUE map. The created mask image is stored in the mask image storage unit 72 .

FIG. 12 is a flowchart for explaining the flow of partial correction image generation processing in the beautiful skin image generation processing.

In step S51, the partially corrected image creation processing unit 55 reads the LUT (see FIG. 7) for color correcting the first blended image.

In step S52, the partially corrected image creation processing unit 55 performs color correction on the first blended image based on the LUT. As a result, a color-corrected image is created by performing color correction on the first blended image.
In step S53, the partially corrected image creation processing unit 55 performs smoothing processing and edge enhancement processing on the color corrected image.

Various imaging devices have a function that allows the user to select a skin color, and for example, are configured to allow the user to select "natural" to improve the complexion of the face and "beautiful skin" to whiten the color of the face. The natural color gives good color reproduction, so the color of the lips can also be improved. On the other hand, for beautiful skin, the saturation of the skin color is reduced, so the lip color is also dragged by it, resulting in a pale and dull complexion.
Therefore, in the imaging apparatus 1 of the present embodiment, the lip region is extracted from the original image before the skin beautification process is performed, and the color of the lip portion and the skin color around the lip are corrected in a well-balanced manner, and the beautification image is obtained. can be created.

The imaging device 1 configured as described above includes an image acquisition unit 51 and an image processing unit 53 .
The image acquisition unit 51 (image acquisition means) acquires a face image including a face area.
The image processing unit 53 (skin beautification processing means) applies skin beautification processing to the face image acquired by the image acquisition unit 51 .
The image processing unit 53 (first identifying means) identifies a first area, which is an area including a skin beautification reduction area, in the image before performing the skin beautifying process on the face image.
The image processing unit 53 (second identifying means) identifies a second area, which is an area including the skin beautifying reduction area, in the image after performing the skin beautifying process on the face image.
The image processing unit 53 (first blending means) blends the image corresponding to the specified first area and the image corresponding to the second area specified by the second specifying means to create a first blended image. do.
The image processing unit 53 (reduction processing means) performs processing for reducing the effect of the skin beautification processing on the created first blend image.
Accordingly, in the imaging device 1, it is possible to provide an image processing device that outputs a more natural image after performing skin beautification processing.

The imaging device 1 also includes a mask image creation processing unit 54 .
The mask image creation processing unit 54 (map information creation means) creates an α map indicating the degree of image processing intensity based on the image of the first region specified by the image processing unit 53 .
The image processing unit 53 creates a first blended image by blending the image corresponding to the first area and the image corresponding to the second area based on the α map.
As a result, in the imaging device 1, the processing for reducing the effect of the skin beautification processing can be applied to an appropriate region.

The image processing unit 53 blends the first blended image and the first blended image that has been subjected to the processing for reducing the effect of the skin beautification processing.
As a result, in the imaging device 1, based on the first blended image obtained by blending the images before and after performing the skin beautifying process, processing for reducing the effect of the skin beautifying process can be performed and blended with the first blended image. , can output a more natural image.

Processing for reducing the effect of skin beautification processing includes color correction processing or edge correction processing.
As a result, the imaging apparatus 1 can output an image in which the skin beautifying process is reduced by color correction or edge correction.

The imaging device 1 also includes an image acquisition unit 51 and an image processing unit 53 .
The image acquisition unit 51 (image acquisition means) acquires a face image including a face area.
The image processing unit 53 (skin beautification processing means) applies skin beautification processing to the face image acquired by the image acquisition unit 51 .
The image processing unit 53 (first identifying means) identifies a first area, which is an area including a skin beautification reduction area, in the image before performing the skin beautifying process on the face image.
The image processing unit 53 (reduction processing means) performs processing for reducing the effect of the skin beautifying process that will be performed later on the first region specified by the first specifying means.
The image processing unit 53 acquires a face image with the effect of the skin beautifying process reduced based on the first region with the effect of the skin beautifying process reduced and the image subjected to the skin beautifying process.
Accordingly, it is possible to provide an image processing apparatus that outputs a more natural image after performing skin beautification processing.

Further, the processing for reducing the effect of the skin beautification processing includes color correction processing or edge correction processing.
The image processing unit 53 performs the face beautifying effect reduced based on the image corresponding to the first region subjected to the color correction processing or the edge correction processing and the image subjected to the skin beautification processing by the skin beautification processing means. Get an image.
As a result, the imaging apparatus 1 can output an image in which the skin beautifying process is reduced by color correction or edge correction.

The image processing unit 53 sets a first adjustment value for brightness and a second adjustment value for saturation, blends the brightness parameter in the image using the first adjustment value, and adjusts the saturation in the image. The degree parameter is blended using the second adjustment value.
Accordingly, in the imaging apparatus 1, the brightness and the saturation can be appropriately adjusted and α-blended, so that a more appropriate image can be output.

The mask image creation processing unit 54 creates an α map corresponding to the region of the organ by detecting the organ from the face image.
As a result, the imaging device 1 can more appropriately specify the region of the target organ and create an image in which the skin beautifying process is reduced for the facial image that has been subjected to the beautifying process.

It should be noted that the present invention is not limited to the above-described embodiments, and includes modifications, improvements, and the like within the scope of achieving the object of the present invention.

For example, in the above-described embodiment, the lip region may be specified first, and the specified lip region may be specified as the skin mask region during the skin beautifying process applied to the entire face.
Further, in the above-described embodiment, when creating a partially corrected image, the color corrected image is subjected to smoothing processing and edge enhancement processing, but the present invention is not limited to this. For example, when the face area of the original image is dark, the partial correction image may be created by performing only smoothing processing without performing edge enhancement processing on the color-corrected image.
Further, in the above-described embodiment, the cut-out image is cut out from the original image and the whole skin-beautifying image and various processes are performed, but the present invention is not limited to this. That is, if the region including the lips can be processed, it is also possible to process the image without clipping it. may be processed.

In the above-described embodiment, in synthesizing the HUE map and the organ map when creating the mask image, α Although both values are multiplied, the present invention is not limited to this. For example, the HUE map and the organ map may be synthesized by taking the minimum value of the HUE map and the organ map.

In the above-described embodiment, when creating a color-corrected image, the correction strength is set according to the V value in the YUV color space measured from the skin regions below the left and right pupils of the original image detected by organ detection. By doing so, color correction (color correction for reducing the effect of skin beautification processing) may be performed on the first blend image. This makes it possible to balance the effect of the skin beautifying process and the correction of the lip color with the original skin color of the subject as a reference. Note that such color correction can be performed together with the correction using the LUT shown in FIG. 7 or independently.

Also, in the above-described embodiments, the case where the image processing apparatus of the present invention is implemented as an image processing apparatus having an imaging function has been described, but the present invention is not limited to this. That is, the image processing apparatus of the present invention is realized as a device having an imaging function, and is also a device (for example, a server, a personal computer, a smartphone, etc.).

Further, in the above-described embodiment, the imaging apparatus 1 to which the present invention is applied is described as an example of a digital camera, but it is not particularly limited to this.
For example, the present invention can be applied to electronic devices in general that have a skin beautifying function. Specifically, for example, the present invention is applicable to notebook personal computers, printers, television receivers, video cameras, portable navigation devices, mobile phones, smart phones, portable game machines, and the like.

The series of processes described above can be executed by hardware or by software.
In other words, the functional configuration of FIG. 9 is merely an example and is not particularly limited. That is, it is sufficient that the imaging apparatus 1 has a function capable of executing the above-described series of processes as a whole, and what kind of functional blocks are used to realize this function is not particularly limited to the example in FIG.
Also, one functional block may be composed of hardware alone, software alone, or a combination thereof.
The functional configuration in this embodiment is realized by a processor that executes arithmetic processing, and processors that can be used in this embodiment are composed of various single processing units such as single processors, multiprocessors, and multicore processors. In addition, it includes a combination of these various processing devices and a processing circuit such as ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array).

When a series of processes is to be executed by software, a program constituting the software is installed in a computer or the like from a network or a recording medium.
The computer may be a computer built into dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a general-purpose personal computer.

A recording medium containing such a program is not only constituted by the removable medium 31 of FIG. It consists of a recording medium, etc. provided to The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. Optical discs are composed of, for example, CD-ROMs (Compact Disk-Read Only Memory), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, and the like. The magneto-optical disk is composed of an MD (Mini-Disk) or the like. Also, the recording medium provided to the user in a state of being pre-installed in the apparatus main body is composed of, for example, the ROM 12 in FIG. 1 in which the program is recorded, the hard disk included in the storage section 19 in FIG.

In this specification, the steps of writing a program recorded on a recording medium are not only processes that are performed chronologically in that order, but also processes that are not necessarily chronologically processed, and that are performed in parallel or individually. It also includes the processing to be executed.

Although several embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the scope of the invention described in the claims and equivalents thereof.

The invention described in the scope of claims at the time of filing of the present application will be additionally described below.
[Appendix 1]
an image acquiring means for acquiring a face image including a face area;
a skin-beautifying means for performing skin-beautifying processing on the face image acquired by the image acquisition means;
a first identifying means for identifying a first area, which is an area including a skin beautification reduction area, in the image before skin beautification is performed on the face image;
a second identifying means for identifying a second area, which is an area including a skin beautification reduction area, in the image after performing the skin beautifying process on the face image;
a first blending image for creating a first blended image by blending an image corresponding to the first area specified by the first specifying means and an image corresponding to the second area specified by the second specifying means; a blending means;
reduction processing means for performing processing for reducing the effect of the skin beautifying process using the first blended image created by the first blending means;
An image processing device comprising:
[Appendix 2]
map information creating means for creating an α map indicating the degree of intensity of image processing based on the image of the first region specified by the first specifying means;
The first blending means creates a first blended image by blending an image corresponding to the first area and an image corresponding to the second area based on the α map. 1. The image processing apparatus according to 1.
[Appendix 3]
3. The image processing apparatus according to appendix 1 or 2, wherein the reduction processing unit blends the first blended image with a first blended image that has been subjected to processing for reducing the effect of the skin beautification processing.
[Appendix 4]
4. The image processing apparatus according to any one of appendices 1 to 3, wherein the processing for reducing the effects of the skin beautification processing includes color correction processing or edge correction processing.
[Appendix 5]
an image acquiring means for acquiring a face image including a face area;
a skin-beautifying means for performing skin-beautifying processing on the face image acquired by the image acquisition means;
a first identifying means for identifying a first area, which is an area including a skin beautification reduction area, in the image before skin beautification is performed on the face image;
a reduction processing means for performing a process of reducing the effect of skin beautifying treatment to be performed later on the first region identified by the first identification means;
The reduction processing means obtains a face image with a reduced skin beautifying effect based on the first region with the reduced skin beautifying effect and the image subjected to the skin beautifying process by the skin beautifying means. An image processing device characterized by:
[Appendix 6]
The processing for reducing the effect of the skin beautification processing includes color correction processing or edge correction processing,
The reduction processing means reduces the effect of the skin beautifying process based on the image corresponding to the first region subjected to the color correction process or the edge correction process and the image subjected to the skin beautifying process by the skin beautifying means. 6. The image processing device according to Supplementary Note 5, wherein the image processing apparatus obtains a face image obtained by applying a facial image.
[Appendix 7]
The first blending means sets a first adjustment value for brightness and a second adjustment value for saturation, blends a parameter of brightness in the image using the first adjustment value, and 5. The image processing apparatus according to any one of appendices 1 to 4, wherein a saturation parameter in is blended using the second adjustment value.
[Appendix 8]
3. The image processing apparatus according to Supplementary Note 2, wherein the map information creating means creates the α map corresponding to the region of the organ by detecting the organ from the face image.
[Appendix 9]
An image processing method executed by an image processing device,
an image acquisition step of acquiring a face image including a region of the face;
a skin beautifying step of performing skin beautifying processing on the face image acquired in the image acquiring step;
a first specifying step of specifying a first area, which is an area including a skin beautification reduction area, in the image before skin beautification is performed on the face image;
a second identifying step of identifying a second area, which is an area including a skin beautification reduction area, in the image after performing the skin beautifying process on the face image;
A first blending image corresponding to the first area specified in the first specifying step and an image corresponding to the second area specified in the second specifying step to create a first blended image; a blending step;
a reduction processing step of performing processing for reducing the effect of the skin beautification processing using the first blended image created in the first blending step;
An image processing method comprising:
[Appendix 10]
An image processing method executed by an image processing device,
an image acquisition step of acquiring a face image including a region of the face;
a skin beautifying step of performing skin beautifying processing on the face image acquired in the image acquiring step;
a first specifying step of specifying a first area, which is an area including a skin beautification reduction area, in the image before skin beautification is performed on the face image;
a reduction processing step that performs processing to reduce the effect of skin beautification processing that will be performed later on the first region identified in the first identification step;
In the reduction processing step, a face image with a reduced effect of the skin beautifying process is acquired based on the first region in which the effect of the skin beautifying process is reduced and the image subjected to the skin beautifying process in the skin beautifying step. An image processing method characterized by:
[Appendix 11]
to the computer,
an image acquisition function for acquiring a face image including a face area;
a skin beautification function that performs skin beautification processing on the face image acquired by the image acquisition function;
a first specifying function for specifying a first region, which is a region including a skin beautification reduction region, in the image before skin beautification processing is performed on the face image;
a second specifying function for specifying a second area, which is an area including a skin beautification reduction area, in the image after performing the skin beautification process on the face image;
a first blending image for creating a first blended image by blending an image corresponding to the first area specified by the first specifying function and an image corresponding to the second area specified by the second specifying function; blend function and
a reduction processing function that uses the first blended image created by the first blending function to reduce the effect of the skin beautifying process;
A program characterized by realizing
[Appendix 12]
to the computer,
an image acquisition function for acquiring a face image including a face area;
a skin beautification function that performs skin beautification processing on the face image acquired by the image acquisition function;
a first specifying function for specifying a first region, which is a region including a skin beautification reduction region, in the image before skin beautification processing is performed on the face image;
a reduction processing function that performs processing for reducing the effect of skin beautification processing that will be performed later on the first region specified by the first specifying function;
The reduction processing function acquires a face image with reduced skin beautifying effect based on the first area with reduced skin beautifying effect and the image subjected to skin beautifying processing by the skin beautifying function. Program characterized.

Reference Signs List 1 imaging device 11 CPU 12 ROM 13 RAM 14 bus 15 input/output interface 16 imaging unit 17 Input unit 18 Output unit 19 Storage unit 20 Communication unit 21 Drive 31 Removable medium 51 Image acquisition unit 52 Face Detection unit 53 Image processing unit 54 Mask image creation processing unit 55 Partial correction image creation processing unit 71 Image storage unit 72 Mask image storage unit

Claims (7)

an image acquiring means for acquiring a face image including a face area;
a skin-beautifying means for performing skin-beautifying processing on the face image acquired by the image acquisition means;
a first identifying means for identifying a first area, which is an area including a skin beautification reduction area, in the image before skin beautification is performed on the face image;
a second identifying means for identifying a second area, which is an area including a skin beautification reduction area, in the image after performing the skin beautifying process on the face image;
Blending for creating a blended image by blending the image corresponding to the first area specified by the first specifying means and the image corresponding to the second area specified by the second specifying means. means and
a reduction processing means for performing a reduction process for reducing the effect of the skin beautifying process on the blended image created by the blending means;
has
The image processing device, wherein the reduction processing means blends the blended image and the blended image subjected to the reduction processing . map information creating means for creating an α map indicating the degree of intensity of image processing based on the image of the first region specified by the first specifying means;
2. The method according to claim 1, wherein said blending means creates said blended image by blending an image corresponding to said first area and an image corresponding to said second area based on said α map. The described image processing device. 3. The image processing apparatus according to claim 1, wherein the processing for reducing the effect of the skin beautification processing includes color correction processing or edge correction processing. The blending means sets a first adjustment value for brightness and a second adjustment value for saturation, blends a brightness parameter in the image using the first adjustment value, and 4. The image processing apparatus according to any one of claims 1 to 3 , wherein a saturation parameter in is blended using the second adjustment value. 3. The image processing apparatus according to claim 2, wherein said map information creating means creates said .alpha.-map corresponding to said organ area by detecting said organ from said face image. An image processing method executed by an image processing device,
an image acquisition step of acquiring a face image including a region of the face;
a skin beautifying step of performing skin beautifying processing on the face image acquired in the image acquiring step;
a first specifying step of specifying a first area, which is an area including a skin beautification reduction area, in the image before skin beautification is performed on the face image;
a second identifying step of identifying a second area, which is an area including a skin beautification reduction area, in the image after performing the skin beautifying process on the face image;
Blending for creating a blended image by blending an image corresponding to the first region identified in the first identifying step and an image corresponding to the second region identified in the second identifying step a step;
a reduction processing step of performing a reduction processing for reducing the effect of the skin beautification processing using the blended image created in the blending step;
including
The image processing method , wherein the reduction processing step blends the blend image and the blend image subjected to the reduction processing . to the computer,
an image acquisition function for acquiring a face image including a face area;
a skin beautification function that performs skin beautification processing on the face image acquired by the image acquisition function;
a first specifying function for specifying a first region, which is a region including a skin beautification reduction region, in the image before skin beautification processing is performed on the face image;
a second specifying function for specifying a second area, which is an area including a skin beautification reduction area, in the image after performing the skin beautification process on the face image;
Blending for creating a blended image by blending an image corresponding to the first area specified by the first specifying function and an image corresponding to the second area specified by the second specifying function. function and
a reduction processing function that performs a reduction processing for reducing the effect of the skin beautification processing using the blended image created by the blending function;
to realize
A program , wherein the reduction processing function blends the blend image and the blend image subjected to the reduction processing .

Images