JP7015009B2 - Image processing equipment, image processing methods and programs - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method and a program.

Conventionally, a whitening process for whitening the face of a person included in an image has been used. For example, Patent Document 1 discloses a technique that makes it possible to easily set the effect of whitening treatment.

Japanese Unexamined Patent Publication No. 2016-012890

However, the technique disclosed in Patent Document 1 described above has a problem that the color of the lips becomes lighter by applying the whitening treatment to the face of a person.

The present invention has been made in view of such a situation, and an object of the present invention is to apply an appropriate whitening treatment to the entire face of a person.

In order to achieve the above object, the image processing apparatus according to one aspect of the present invention is
Lip part identification means for specifying the lip part in the face by detecting each organ from the face of the person included in the image by contour extraction, and
A color that is close to the color information of the skin color in a predetermined range in the HSV color space in the specified lip portion is specified as a lip color that should reduce the whitening treatment applied to the region having the skin color. Color identification means and
When the whitening treatment is applied to the region including the lip portion in the image, the reduction treatment means for reducing the whitening treatment of the region having the color specified by the color specifying means, and the reduction treatment means.
It is characterized by having.
Further, the image processing method according to one aspect of the present invention is:
Lip part identification processing that identifies the lip part of the face by detecting each organ from the face of the person included in the image by contour extraction, and
A color that is close to the color information of the skin color in a predetermined range in the HSV color space in the specified lip portion is specified as a lip color that should reduce the whitening treatment applied to the region having the skin color. Color identification processing and
When the whitening treatment is applied to the region including the lip portion in the image, the mitigation processing means for reducing the whitening treatment of the region having the color specified in the color specifying treatment and the reduction processing means.
It is characterized by including.
Further, the program of one aspect of the present invention is
Computer,
A lip portion specifying means for identifying a lip portion in the face by detecting each organ from the face of a person included in the image by contour extraction.
A color that is close to the color information of the skin color in a predetermined range in the HSV color space in the specified lip portion is specified as a lip color that should reduce the whitening treatment applied to the region having the skin color. Color identification means,
A mitigation processing means for reducing the whitening treatment of a region having a color specified by the color specifying means when the whitening treatment is applied to the region including the lip portion in the image.
It is characterized by functioning as.

According to the present invention, an appropriate whitening treatment can be applied to the entire face of a person.

It is a block diagram which shows the hardware structure of the image pickup apparatus 1 which concerns on one Embodiment of the image processing apparatus of this invention. It is a schematic diagram for demonstrating the generation of the whitening image in this embodiment. It is a schematic diagram which shows the creation example of a mask image. It is a figure which shows the histogram which concerns the setting of the weighting of the lip color level, and the weighting table of the lip color level. It is a schematic diagram for demonstrating the creation of a fixed map. It is a figure which shows the LUT of V correction. It is a functional block diagram which shows the functional configuration for executing the whitening image generation processing among the functional configurations of the image pickup apparatus 1 of FIG. 6 is a flowchart illustrating a flow of a whitening image generation process executed by the image pickup apparatus 1 of FIG. 1 having the functional configuration of FIG. 7. It is a flowchart explaining the flow of the mask image creation process in the whitening image generation process. It is a flowchart explaining the flow of the color correction image creation process in the whitening image generation process.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of an image pickup apparatus 1 according to an embodiment of the image processing apparatus of the present invention.
The image pickup apparatus 1 is configured as, for example, a digital camera.

As shown in FIG. 1, the image pickup apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, and an image pickup. It includes a unit 16, an input unit 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21.

The CPU 11 executes various processes according to the program recorded in the ROM 12 or the program loaded from the storage unit 19 into the RAM 13.

Data and the like necessary for the CPU 11 to execute various processes are also appropriately stored in the RAM 13.

The CPU 11, ROM 12 and RAM 13 are connected to each other via the bus 14. An input / output interface 15 is also connected to the bus 14. An image pickup unit 16, an input unit 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21 are connected to the input / output interface 15.

Although not shown, the image pickup unit 16 includes an optical lens unit and an image sensor.

The optical lens unit is composed of a lens that collects light, such as a focus lens and a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. A zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with peripheral circuits for adjusting setting parameters such as focus, exposure, and white balance, if necessary.

The image sensor is composed of a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element performs photoelectric conversion (imaging) of the subject image, accumulates an image signal for a certain period of time, and sequentially supplies the accumulated image signal as an analog signal to AFE.
The AFE executes various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and is output as an output signal of the image pickup unit 16.
The output signal of the image pickup unit 16 is appropriately supplied to the CPU 11 or an image processing unit (not shown) as data of the image pickup image.

The input unit 17 is composed of various buttons and the like, and inputs various information according to a user's instruction operation.
The output unit 18 is composed of a display, a speaker, or the like, and outputs images and sounds.
The storage unit 19 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various image data.
The communication unit 20 controls communication with other devices (not shown) via a network including the Internet.

A removable media 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 21. The program read from the removable media 31 by the drive 21 is installed in the storage unit 19 as needed. Further, the removable media 31 can also store various data such as image data stored in the storage unit 19 in the same manner as the storage unit 19.

In the image pickup apparatus 1 configured in this way, the image of the photographed face is subjected to whitening treatment on the skin, and is affected by the fact that the skin color and hue are close to each other and the color is lightened by the whitening treatment. It has a function of being able to generate an image in which the lips are appropriately corrected (hereinafter referred to as "whitening image"). That is, the image pickup apparatus 1 generates a whitening image by correcting the color of the lips corrected by the whitening treatment applied to the skin of the face to an appropriate color and restoring (reducing) the effect of the whitening treatment.

That is, the whitening treatment is masked except for the target portion so that it is applied only to the skin color region, but the color of the lips is close to the skin color and is included in the skin color region, so that the whitening treatment is performed. It will be affected by the lightening of the color of the lips. Further, the whitening process is mainly performed by correcting three elements of saturation reduction, brightness increase, and hue rotation in the blue direction.
Therefore, in the image pickup apparatus 1 of the present embodiment, for the lightened lip color, only the saturation reduction among the three elements is restored, and the other effects are maintained, so that the lip color is changed. Correct to the appropriate color.

In the present embodiment, the whitening image is first subjected to whitening treatment on the entire face, and then the lips affected by the whitening treatment are separately color-corrected to be reproduced in an appropriate color. Generated.

FIG. 2 is a schematic diagram for explaining the generation of a whitening image in the present embodiment.
In the whitening image of the present embodiment, as shown in FIG. 2, a region including lips is cut out from an image (hereinafter, referred to as “original image”) in which the entire image is whitened. Then, the area including the lips is subjected to color correction processing so as to reproduce the color of the lips that has been lightened by the whitening treatment to an appropriate color, and the image (hereinafter referred to as "color correction image") is obtained. create. Since the color correction image is color-corrected for the entire image other than the lips, the original image and the color-corrected image are used by using a mask image that masks the parts other than the lips so that the color correction effect appears only on the lips. And are image-combined by α-blend to create an image in which only the lips are color-corrected (hereinafter referred to as “lip color-corrected image”). The mask image becomes an α value at the time of image composition by α blending.
After that, the created lip color correction image is combined with the cutout position of the original image to generate a whitening image.

[Create mask image]
FIG. 3 is a schematic diagram showing an example of creating a mask image.
In the present embodiment, in order to perform a correction process for reproducing the color of the lip portion that has become lighter due to the whitening process, a mask image corresponding to the region corresponding to the lips is generated.
As shown in FIG. 3, in the mask image, the black portion indicates that the proportion of the color-corrected image composite is low, and the white portion indicates that the proportion of the color-corrected image composite is high. In the present embodiment, the central portion of the lips is strongly color-corrected, and the portion near the outer edge of the lips is strongly color-corrected in order to reduce the influence on the skin.

Such a mask image is first created by identifying the position of the lips by detecting facial organs and based on the hue value that is the peak of the HSV color space in the region containing the identified lip position. In the detection of facial organs of the present embodiment, as shown in FIG. 2, at least two points P1 and P2 above and below the center of the upper lip and two points P3 above and below the center of the lower lip are used to specify the position of the lips. , P4, 2 points P5, P6 on the left and right edges of the lips, a total of 6 points are detected.

In the present embodiment, in order to eliminate unnecessary parts such as the nose, a fixed map that imitates the shape of the lips is synthesized in advance with the mask image generated based on the hue value that is the peak of the HSV color space. , Create the final mask image.

Specifically, the mask image of the present embodiment is a fixed map created in advance by creating a HUE map from an image obtained by cutting out an area including lips from the original image (hereinafter referred to as “cutout image”). It is created by synthesizing with. In the composition of the HUE map and the fixed map, the minimum value of the HUE map and the fixed map is taken in order to remove the spatially unnecessary area from the HUE map.

[Create HUE map]
The HUE map is a map that identifies a portion corresponding to the color of the lips in the cutout image. The HUE map first converts a cut-out image of the YUV color space into an HSV (hue: Hue, saturation: Saturation / Chroma, lightness: Value / Lightness / Brightness) color space. Then, in order to remove noise, the image after HSV conversion is analyzed (HSV analysis processing) after blurring with an ε filter. Then, the HUE map is created by using the value calculated from the processing result of the HSV analysis process as the map value of the HUE map.

In the HSV analysis process, weighting of the lip color level is set in order to determine the map value in each pixel. The weighting of the lip color level is set based on the histogram created for each channel of HSV.

FIG. 4 is a diagram showing a histogram and a weighting table of the lip color level related to the setting of the weighting of the lip color level.
In setting the lip color level, first, measurement areas R1 and R2 are set near the center of the upper and lower lips as measurement areas from points P1 to P6 detected by organ detection shown in FIG. 2, and histograms of each HSV in each measurement area. To create. The measurement area of the lips related to the creation of the histogram may be the number of areas, various shapes, positions, and sizes as long as the color of the lips can be acquired.

Then, a weighting table for the lip color level is set as shown in FIG. 4 (b) corresponding to the mode of each HSV histogram as shown in FIG. 4 (a). In the case of the H channel, the weighting is evenly distributed between plus and minus based on the mode value, and the weighting is set to be lower as the distance from the mode value increases. Further, in the case of the S (V) channel, the mode has a predetermined range in the peak value with the mode value of the histogram as the reference of the upper left end (right end), and the peak value is in the second stage rather than the first stage. Weighting is set so that the slope becomes a gentler two-step slope.

After that, a HUE map is created using the result of multiplying each pixel by the lip color level calculated according to the weighting for each channel of HSV as a map value.
That is, the map value of the HUE map is calculated by the following equation (1).
Map value (Map) of HUE map = Lh x Ls x Lv ... (1)
In addition, "Lh" indicates the lip color level of H, "Ls" indicates the lip color level of S, and "Lv" indicates the lip color level of V.

[Create fixed map]
FIG. 5 is a schematic diagram for explaining the creation of a fixed map.

The fixed map is a map that imitates the shape of the lips on a general face, and is a map created in advance for color correction.
The fixed map develops a reduced size map from the data, as shown in FIG. 5 (a).
Then, as shown in FIG. 5B, the angle of inclination of the lips is calculated from the contour information of the lips in the image (positions of the two points P5 and P6 in the present embodiment) and fixed corresponding to the angle. Rotate the map.
Finally, the fixed map is used in combination with the HUE map to resize to the size corresponding to the image and remove the non-lip portion of the HUE map. At the time of composition, the minimum values of the HUE map and the fixed map are taken.

[Creating a color-corrected image]
In the present embodiment, the color-corrected image is cut out by correcting the cut-out image with the correction intensity determined according to the V value of the YUV color space measured from the skin area below the left and right pupils detected by the organ detection. Is created. The skin area below the left and right pupils is a position that avoids the bear under the eyes, and is determined as the position that most expresses the skin color on the face, and the skin color on the face can be extracted. As long as it is a position, it may be in various positions.

If the original color of the lips is dark due to lips or the like, the V value of the pixels of the lips is corrected so that the correction does not work too much.
FIG. 6 is a diagram showing a V-corrected LUT (Look Up Table).
In the V-correction LUT, as shown in FIG. 6, the V-correction is applied to the dark-colored portion of the lips.

[Creating a lip color correction image]
The lip color correction image is created by alpha-blending a color correction image and a cutout image based on a mask image and synthesizing the images. The created color correction image is pasted at the cutout position of the original image to generate a whitening image. That is, the color correction is applied only to the lip area, and the color correction image to which the whitening treatment is applied to the other skin parts is pasted on the original image to which the same whitening treatment is applied to generate a whitening image. Will be done.

FIG. 7 is a functional block diagram showing a functional configuration for executing a whitening process among the functional configurations of the image pickup apparatus 1 of FIG. 1.
The whitening image generation process is to apply whitening treatment to the original image including the photographed person's face, and to properly apply the lips whose color is close to the skin and whose color has become lighter due to the influence of the whitening treatment. It is a series of processes to generate a whitening image by correcting it to a different color.

When executing the whitening image generation process, as shown in FIG. 7, in the CPU 11, the image acquisition unit 51, the face detection unit 52, the image processing unit 53, the mask image creation processing unit 54, and the color correction image are executed. The creation processing unit 55 and the function.

Further, an image storage unit 71 and a fixed map storage unit 72 are set in one area of the storage unit 19.

The image storage unit 71 stores image data output from the image pickup unit 16.
The fixed map storage unit 72 stores fixed map data used for creating a mask image. Since the fixed map has an arc shape imitating the lips in order to reduce the amount of data, it is possible to store the data obtained by dividing the arc into quarters. In this case, the fixed map is expanded so that it becomes an arc during processing.

The image acquisition unit 51 acquires the data of the captured image obtained by developing the image captured by the image pickup unit 16 or the data of the image to be processed from the image storage unit 71.

The face detection unit 52 detects the face from the image and also detects each organ constituting the face in the detected face. The face detection unit 52 detects the left and right pupils and the organs of the lips from at least the contour shape. Regarding the lips, the face detection unit 52 has at least two points P1 and P2 above and below the center of the upper lip, two points P3 and P4 above and below the center of the lower lip, and two points P5 and P6 at the left and right ends of the lips. The positions of a total of 6 points are detected. For face detection and detection of each organ, existing face detection techniques and known techniques of organ techniques are used. Further, the face detection and the detection of each organ are not limited as long as they are detection methods based on information different from the color information in the HSV color space, such as luminance and contour extraction.

The image processing unit 53 executes various image processing on the original image.
Specifically, the image processing unit 53 mainly executes a whitening process for correcting three elements of saturation reduction, brightness increase, and hue rotation in the blue direction. Further, as shown in FIG. 2, the image processing unit 53 cuts out a region including the lips from the original image. Further, the image processing unit 53 α-blends the cut-out image and the created color-corrected image based on the created mask image and synthesizes the image. As a result, a lip color correction image is created. Further, the image processing unit 53 pastes the image at the original cutout position. As a result, a whitening image is generated.

The mask image creation processing unit 54 generates a mask image to be used for the α blend so that the image is color-corrected only in the lip region.

Specifically, the mask image creation processing unit 54 converts the YUV color space of the cut-out image into the HSV color space, as shown in FIG. Further, the mask image creation processing unit 54 performs a process of blurring an image with an ε filter for the purpose of removing noise.

Further, the mask image creation processing unit 54 executes the HSV analysis processing. In the HSV analysis process, histograms are created for each channel of HSV in the measurement areas R1 and R2 (see FIG. 2) set near the center of the upper and lower lips (see FIG. 4A). Then, the weighting of the lip color level calculated from the created histogram is created (see FIG. 4B). Then, based on the created weighting of the lip color level, the map value of the HUE map in each pixel is calculated using the above-mentioned equation (1). A HUE map is created from the calculated map values of each pixel.

Further, as shown in FIGS. 5A and 5B, the mask image creation processing unit 54 adjusts the angle of the fixed map stored in the fixed map storage unit 72 so as to correspond to the lips of the cut-out image. Adjust the size.

Further, as shown in FIG. 3, the mask image creation processing unit 54 synthesizes the fixed map whose angle and size have been adjusted and the created HUE map. At this time, the mask image creation processing unit 54 synthesizes the HUE map and the fixed map so as to take the minimum values in order to remove an unnecessary spatial region from the HUE map such as the nose. As a result of compositing, a mask image is created.

The color correction image creation processing unit 55 executes color correction processing on the cut-out image.
Specifically, the color correction image creation processing unit 55 measures the V value of the detected skin region below the left and right pupils. Then, the color correction image creation processing unit 55 performs color correction on the cut-out image according to the measured V value. As a result, a color-corrected image in which the cut-out image is color-corrected is created.

FIG. 8 is a flowchart illustrating a flow of a whitening image generation process executed by the image pickup apparatus 1 of FIG. 1 having the functional configuration of FIG. 7.
The whitening image generation process is started by the operation of the user to start the whitening image generation process to the input unit 17.
The operation for starting the whitening image generation process is a shooting instruction operation, and the image captured by the image pickup unit 16 in response to the shooting instruction operation is continuously whitened with respect to the data of the captured image that has been developed. The generation process may be performed, or the operation of selecting the captured image data stored in the image storage unit 71 and starting the whitening image generation process for the selected captured image data may be performed.

In step S11, the image acquisition unit 51 acquires the data of the captured image obtained by developing the image captured by the image pickup unit 16 or the original image to be processed from the image storage unit 71.

In step S12, the face detection unit 52 detects the face in the original image and determines whether or not the face is detected.
If no face is detected, NO is determined in step S12, and the whitening image generation process ends.
If a face is detected, YES is determined in step S12, and the process proceeds to step S13.

In step S13, the image processing unit 53 executes whitening processing on the original image. By the whitening process, the original image is desaturated, brightened, and hue-rotated in the blue direction.

In step S14, the face detection unit 52 detects the detected facial organs. As a result, at least the positions of the left and right pupils and the organs of the lips are detected in the original image.

In step S15, the image processing unit 53 cuts out a region including the detected lips.

In step S16, the mask image creation processing unit 54 executes the mask image creation process. As a result of executing the mask image creation process, a mask image for alpha blending is created. The details of the mask image creation process will be described later.

In step S17, the color correction image creation processing unit 55 executes the color correction image creation processing. As a result of executing the color correction image creation process, a color correction image is created in which the cutout image is color-corrected to make the thinned lips an appropriate color. The details of the color correction image creation process will be described later.

In step S18, the image processing unit 53 α-blends the cut-out image and the created color-corrected image based on the created mask image and synthesizes the image. As a result, a lip color correction image in which color correction is applied only to the lip portion is created.

In step S19, the image processing unit 53 pastes the lip color correction image at the original cutout position. As a result, a whitening treatment is applied to the skin, and a whitening image corrected to an appropriate color is generated for the lip portion.
After that, the whitening image generation process is completed.

FIG. 9 is a flowchart illustrating the flow of the mask image creation process in the whitening image generation process.

In step S31, the mask image creation processing unit 54 converts the YUV color space of the cutout image into the HSV color space.

In step S32, the mask image creation processing unit 54 performs a blurring process with an ε filter.

In step S33, the mask image creation processing unit 54 executes the HSV analysis processing. In the HSV analysis process, as shown in FIG. 2, histograms are created in each channel of HSV of the measurement areas R1 and R2 set near the center of the upper and lower lips (see FIG. 4A). Then, the weighting of the lip color level calculated from the created histogram is set (see FIG. 4B). In the case of the H channel, the weighting of the lip color level is set so that the weighting is evenly distributed to plus and minus based on the mode value, and the weighting becomes lower as the distance from the mode value increases. Further, in the case of the S (V) channel, the mode has a predetermined range in the peak value with the mode value of the histogram as the reference of the upper left end (right end), and the peak value is in the second stage rather than the first stage. Weighting is set so that the slope becomes a gentler two-step slope.

In step S34, the mask image creation processing unit 54 creates a HUE map. The map value in the HUE map is calculated using the above-mentioned equation (1) based on the weighting of the created lip color level.

In step S35, the mask image creation processing unit 54 adjusts the angle and size of the fixed map stored in the fixed map storage unit 72 so as to correspond to the lips of the cut-out image.

In step S36, the mask image creation processing unit 54 synthesizes the fixed map whose angle and size have been adjusted and the created HUE map. At this time, the mask image creation processing unit 54 synthesizes the HUE map and the fixed map so as to take the minimum values in order to remove a spatially unnecessary region from the HUE map such as the nose. As a result of compositing, a mask image is created.

FIG. 10 is a flowchart illustrating the flow of the color correction image creation process in the whitening image generation process.

In step S51, the color correction image creation processing unit 55 measures the V value of the detected skin region below the left and right pupils.

In step S52, the color correction image creation processing unit 55 performs color correction on the cut-out image according to the measured V value. As a result, a color-corrected image in which the cut-out image is color-corrected is created.

Various image pickup devices have a function that allows the user to select a skin color, and for example, "natural" that improves the complexion of the face and "whitening" that whitens the face color are configured to be selectable. With natural, the color is reproduced with good color, so the color of the lips can also be improved. On the other hand, in whitening, the saturation of the skin color is reduced, so that the lip color is also dragged by it, and the color becomes light and pale.
Therefore, in the image pickup apparatus 1 of the present embodiment, the lip region is extracted from the detected face region using the result of detecting the contour of the image of the person, and the lip color is brightened or the lip is painted. Can be corrected to. The extraction of the lip color region analyzes the image expressed in the HSV color space (HSV image), and creates a mask image for α blend from the HSV image. When making corrections, the image represented in the YUV color space can be analyzed and corrected for each YUV (particularly V).

The image pickup apparatus 1 configured as described above includes an image processing unit 53, a mask image creation processing unit 54, and a color correction image creation processing unit 55.
The image processing unit 53 performs whitening processing for whitening the skin color on the face area of a person included in the image.
The mask image creation processing unit 54 specifies a portion of the face area of the person to be whitened to reduce the whitening process.
The color correction image creation processing unit 55 performs a process of reducing the whitening process on the specified portion to be reduced.
Thereby, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and it can be used for the whitening treatment applied to the entire face area of the person, and the appropriate whitening treatment is applied to the entire face of the person. be able to.

The whitening process consists of a plurality of types of processes.
The mask image creation processing unit 54 reduces the effect of some of the processing of the plurality of types of processing in the face area of the person to which the whitening processing consisting of the plurality of types of processing is performed, and suppresses the effect of the other processing. Identify the parts that should be maintained.
As a result, in the image pickup apparatus 1, for example, when the whitening process is performed by the three elements of saturation reduction, brightness increase, and hue rotation in the blue direction, only the saturation reduction is restored to restore the lips. The color of the lips can be appropriately specified, and it can be used for whitening treatment applied to the entire face area of a person.

Further, the image processing unit 53 specifies a skin color portion in the image based on the color information in the HSV color space.
Further, the image processing unit 53 performs whitening processing on the specified skin color portion.
The mask image creation processing unit 54 identifies a portion included in the specified skin color portion and a portion other than the skin whose hue value is close to the skin color as a portion for which the whitening process should be reduced.
As a result, in the image pickup apparatus 1, the portion included in the specified skin color portion and the portion other than the skin whose hue value is close to the skin color is specified as the portion for which the whitening process should be reduced. , Appropriate whitening treatment can be applied to the entire face of a person.

Further, the image pickup apparatus 1 further includes a face detection unit 52 for specifying a face portion based on a standard different from the color information in the HSV color space.
The face detection unit 52 identifies a portion of the specified face other than the skin whose color information in the HSV color space is close to the skin color, as a portion for which the whitening process should be reduced.
Thereby, in the image pickup apparatus 1, in order to identify the part other than the skin whose color information in the HSV color space is close to the skin color as the part where the whitening process should be reduced, it is appropriate for the entire face of the person. Whitening treatment can be applied.

Further, the color correction image creation processing unit 55 specifies the color density of the specified portion for which the whitening process should be reduced based on the color information in the YUV color space.
The color correction image creation processing unit 55 performs a process of reducing the whitening process according to the specified density.
Thereby, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and it can be used for the whitening process applied to the entire face region of the person.

The face detection unit 52 further identifies a portion for which the whitening process should be reduced by using the information prepared in advance as a reference for the shape of the facial portion.
As a result, in the image pickup apparatus 1, the position of the lips can be appropriately specified, and an appropriate whitening process can be applied to the entire face of the person.

The mask image creation processing unit 54 specifies the lip portion of a person's face as a portion for which whitening processing should be reduced.
By setting the lip portion of the person's face as a portion where the whitening treatment should be reduced, an appropriate whitening treatment can be applied to the entire face of the person.

The face detection unit 52 identifies the lip portion of the face of a person included in the image based on a standard different from the color information in the HSV color space.
The mask image creation processing unit 54 specifies the color of the lips based on the color information in the HSV color space in the specified lip portion.
Thereby, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and it can be used for the whitening process applied to the entire face region of the person.

In the HSV color space, the mask image creation processing unit 54 specifies as a lip color a color in which the color information of the specified lip portion is close to the color information of the skin color within a predetermined range.
Thereby, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and it can be used for the whitening process applied to the entire face region of the person.

The mask image creation processing unit 54 specifies the color of the lips based on the distribution of the color information of the specified lip portion in the HSV color space.
Thereby, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and it can be used for the whitening process applied to the entire face region of the person.

The face detection unit 52 identifies the lip portion based on a standard different from the color information in the HSV color space.
Thereby, in the image pickup apparatus 1, the position of the lips can be appropriately specified, and it can be used for the whitening process applied to the entire face region of the person.

Further, the image processing unit 53 performs predetermined image processing based on the color information of the YUV color space on the lip portion of the face of the person specified by the face detection unit 52 according to the specified lip color.
As a result, the image pickup apparatus 1 can perform image processing according to the lip color.

The present invention is not limited to the above-described embodiment, and modifications, improvements, and the like to the extent that the object of the present invention can be achieved are included in the present invention.

In the above-described embodiment, the color of the lips affected by the whitening treatment is corrected to an appropriate color, but the present invention is not limited to this. It may be corrected to change the color of the lips for the specified lip area. Specifically, you can make different lips and lipstick colors, apply lip, lipstick, gloss, etc. to make-up lips that have not been applied with lips, lipstick, gloss, etc., and adjust the degree of complexion. You may. At this time, the saturation may be positively changed, and further, the lightness and the hue may be changed.

Further, in the above-described embodiment, the lip region may be specified first, and the lip region specified as the skin mask region may be designated during the whitening treatment applied to the entire face.

Further, in the above-described embodiment, the image pickup apparatus 1 to which the present invention is applied has been described by taking a digital camera as an example, but the present invention is not particularly limited thereto.
For example, the present invention can be generally applied to electronic devices having a whitening processing function. Specifically, for example, the present invention can be applied to a notebook personal computer, a printer, a television receiver, a video camera, a portable navigation device, a mobile phone, a smartphone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or software.
In other words, the functional configuration of FIG. 7 is merely an example and is not particularly limited. That is, it suffices if the image pickup apparatus 1 is provided with a function capable of executing the above-mentioned series of processes as a whole, and what kind of functional block is used to realize this function is not particularly limited to the example of FIG.
Further, one functional block may be configured by a single piece of hardware, a single piece of software, or a combination thereof.
The functional configuration in the present embodiment is realized by a processor that executes arithmetic processing, and the processor that can be used in the present embodiment is composed of various processing devices such as a single processor, a multiprocessor, and a multicore processor. In addition to the above, the present invention includes a combination of these various processing devices and a processing circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

When a series of processes are executed by software, a program constituting the software is installed in a computer or the like from a network or a recording medium.
The computer may be a computer embedded in dedicated hardware. Further, the computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

The recording medium including such a program is not only composed of the removable media 31 of FIG. 1 distributed separately from the main body of the device in order to provide the program to the user, but also is preliminarily incorporated in the main body of the device of the user. It is composed of a recording medium or the like provided in. The removable media 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versaille Disk), a Blu-ray (registered trademark) Disc (Blu-ray Disc), or the like. The magneto-optical disk is composed of MD (Mini-Disc) or the like. Further, the recording medium provided to the user in a state of being preliminarily incorporated in the main body of the apparatus is composed of, for example, the ROM 12 of FIG. 1 in which the program is recorded, the hard disk included in the storage unit 19 of FIG. 1, and the like.

In this specification, the steps for describing a program to be recorded on a recording medium are not only processed in chronological order but also in parallel or individually, even if they are not necessarily processed in chronological order. It also includes the processing to be executed.

Although some embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and further, various modifications such as omission and substitution can be made without departing from the gist of the present invention. These embodiments and variations thereof are included in the scope and gist of the invention described in the present specification and the like, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

The inventions described in the claims at the time of filing the application of the present application are described below.
[Appendix 1]
A whitening treatment method for whitening the skin color of a person's face area included in the image, and a whitening treatment means.
In the face area of the person to which the whitening treatment is applied, a reduction portion specifying means for specifying a portion to be reduced in the whitening treatment, and a reduction portion specifying means.
A reduction treatment means for performing a treatment for reducing the whitening treatment on the specified portion for which the whitening treatment should be reduced, and a reduction treatment means.
An image processing device characterized by comprising.
[Appendix 2]
The whitening process consists of a plurality of types of processes.
The mitigation portion specifying means reduces the effect of some of the plurality of types of processing in the face area of the person to which the whitening treatment consisting of the plurality of types of processing is performed, and maintains the effect of the other processing. Identify the part to be done,
The image processing apparatus according to Appendix 1, wherein the image processing apparatus is characterized by the above.
[Appendix 3]
Further provided with a skin color part specifying means for specifying the skin color part in the image based on the color information in the HSV color space.
The whitening treatment means applies whitening treatment to the specified skin color portion, and then performs whitening treatment.
The mitigation portion specifying means identifies a portion included in the specified skin color portion and a portion other than the skin whose hue value is close to the skin color as a portion for which the whitening treatment should be mitigated.
The image processing apparatus according to Appendix 1 or 2, characterized in that.
[Appendix 4]
Further provided with a part identification means for specifying a part of the face based on a standard different from the color information of the HSV color space.
The site specifying means identifies a portion of the identified face other than the skin whose color information in the HSV color space is close to the skin color, as a portion for which the whitening treatment should be reduced. ,
The image processing apparatus according to Appendix 3, wherein the image processing apparatus is characterized in that.
[Appendix 5]
Further provided with a density specifying means for specifying the color density of the portion to be reduced in the specified whitening treatment based on the color information in the YUV color space.
The mitigation treatment means performs a treatment for reducing the whitening treatment according to the specified concentration.
The image processing apparatus according to Appendix 3 or 4, characterized in that.
[Appendix 6]
The site specifying means further identifies a portion for which the whitening treatment should be reduced by using information that is prepared in advance as a reference for the shape of the facial region.
The image processing apparatus according to any one of Supplementary note 3 to 5, wherein the image processing apparatus is characterized by the above.
[Appendix 7]
The mitigation portion specifying means identifies the lip portion of a person's face as a portion for which the whitening treatment should be mitigated.
The image processing apparatus according to any one of Supplementary note 1 to 6, wherein the image processing apparatus is characterized by the above.
[Appendix 8]
Lip part identification means for specifying the lip part of a person's face included in the image based on a standard different from the color information in the HSV color space, and
A lip color specifying means for specifying the color of the lips based on the color information of the HSV color space in the specified lip portion,
An image processing device characterized by comprising.
[Appendix 9]
The lip color specifying means specifies a color in the HSV color space whose color information of the specified lip portion is close to the color information of the skin color within a predetermined range as the color of the lips.
The image processing apparatus according to Appendix 8, wherein the image processing apparatus is characterized in that.
[Appendix 10]
The lip color specifying means identifies the color of the lips in the HSV color space based on the distribution of the color information of the specified lip portion.
The image processing apparatus according to Appendix 8 or 9, wherein the image processing apparatus is characterized by the above.
[Appendix 11]
The lip portion specifying means identifies the lip portion based on a standard different from the color information in the HSV color space.
The image processing apparatus according to any one of Supplementary Provisions 8 to 10, wherein the image processing apparatus is characterized by the above.
[Appendix 12]
Further provided with an image processing means for performing predetermined image processing based on the color information of the YUV color space on the lip portion of the face of the person specified by the lip portion specifying means according to the specified lip color.
The image processing apparatus according to any one of Supplementary note 8 to 11, wherein the image processing apparatus is characterized by the above.
[Appendix 13]
Whitening treatment that whitens the skin color of the face area of the person included in the image,
Within the face area of the person to whom the whitening treatment is applied, the mitigation portion specifying process for specifying the portion to be mitigated by the whitening treatment, and the mitigation portion specifying process.
A reduction process for performing a process for reducing the whitening process on the specified portion to be reduced for the whitening process, and a reduction process for reducing the whitening process.
An image processing method comprising.
[Appendix 14]
Lip part identification processing that identifies the lip part of the face of a person included in the image based on a standard different from the color information in the HSV color space, and
A lip color specifying process for specifying the color of the lips based on the color information of the HSV color space in the specified lip portion,
An image processing method comprising.
[Appendix 15]
On the computer
A whitening treatment method that applies whitening treatment to whiten the skin color of the face area of a person included in the image.
A mitigation part specifying means for specifying a part to be reduced in the face area of a person to be whitened.
A reduction treatment means for performing a treatment for reducing the whitening treatment on the specified portion for which the whitening treatment should be reduced, and a reduction treatment means.
A program characterized by realizing.
[Appendix 16]
On the computer
Lip part identification means for specifying the lip part of a person's face included in an image based on a standard different from the color information in the HSV color space.
A lip color specifying means for specifying the color of the lips based on the color information of the HSV color space in the specified lip portion.
A program characterized by realizing.

1 ... image pickup device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... bus, 15 ... input / output interface, 16 ... image pickup unit, 17 ... Input unit, 18 ... Output unit, 19 ... Storage unit, 20 ... Communication unit, 21 ... Drive, 31 ... Removable media, 51 ... Image acquisition unit, 52 ... Face Detection unit, 53 ... Image processing unit, 54 ... Mask image creation processing unit, 55 ... Color correction image creation processing unit, 71 ... Image storage unit, 72 ... Fixed map storage unit

Claims (5)

Lip part identification means for specifying the lip part in the face by detecting each organ from the face of the person included in the image by contour extraction, and
A color that is close to the color information of the skin color in a predetermined range in the HSV color space in the specified lip portion is specified as a lip color that should reduce the whitening treatment applied to the region having the skin color. Color identification means and
When the whitening treatment is applied to the region including the lip portion in the image, the reduction treatment means for reducing the whitening treatment of the region having the color specified by the color specifying means, and the reduction treatment means.
An image processing device characterized by comprising. The color specifying means identifies the color of the lips in the HSV color space based on the distribution of the color information of the specified lip portion.
The image processing apparatus according to claim 1. Further provided with an image processing means for performing predetermined image processing based on the color information of the YUV color space on the lip portion of the face of the person specified by the lip portion specifying means according to the specified lip color.
The image processing apparatus according to claim 1 or 2. Lip part identification processing that identifies the lip part of the face by detecting each organ from the face of the person included in the image by contour extraction, and
A color that is close to the color information of the skin color in a predetermined range in the HSV color space in the specified lip portion is specified as a lip color that should reduce the whitening treatment applied to the region having the skin color. Color identification processing and
When the whitening treatment is applied to the region including the lip portion in the image, the mitigation processing means for reducing the whitening treatment of the region having the color specified in the color specifying treatment and the reduction processing means.
An image processing method comprising. Computer,
A lip portion specifying means for identifying a lip portion in the face by detecting each organ from the face of a person included in the image by contour extraction.
A color that is close to the color information of the skin color in a predetermined range in the HSV color space in the specified lip portion is specified as a lip color that should reduce the whitening treatment applied to the region having the skin color. Color identification means,
A mitigation processing means for reducing the whitening treatment of a region having a color specified by the color specifying means when the whitening treatment is applied to the region including the lip portion in the image.
A program characterized by functioning as.

Images