JP2020074100A - Image processing device, image processing method, and program - Google Patents

Abstract

To apply appropriate whitening treatment to the entire face of a person.SOLUTION: An imaging device 1 includes an image processing unit 53, a mask image creation processing unit 54, and a color correction image creation processing unit 55. The image processing unit 53 performs a whitening process for whitening the color of a skin with respect to a face area of a person included in an image. The mask image creation processing unit 54 identifies a part to be reduced in the whitening process in the face area of the person to be subject to the whitening process. The color correction image creation processing unit 55 performs a process of reducing the whitening process on the identified part to be reduced in the whitening process.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an image processing device, an image processing method, and a program.

  Conventionally, whitening processing for whitening the face of a person included in an image has been used. For example, Japanese Patent Application Laid-Open No. 2004-242242 discloses a technique that enables the effect of whitening treatment to be easily set.

JP, 2016-012890, A

  However, the technique disclosed in the above-mentioned Patent Document 1 has a problem that the color of the lips becomes lighter by whitening the face of the person.

  The present invention has been made in view of such a situation, and an object thereof is to perform an appropriate whitening treatment on the entire face of a person.

In order to achieve the above object, an image processing device according to one embodiment of the present invention is
Whitening processing means for performing whitening processing for whitening the skin color for the face area of the person included in the image;
Within the face area of the person to which the whitening process is applied, a reduced portion specifying unit that specifies a portion where the whitening process should be reduced,
Reduction processing means for performing a process of reducing the whitening process on the identified part where the whitening process should be reduced,
It is characterized by including.

  According to the present invention, an appropriate whitening process can be applied to the entire face of a person.

It is a block diagram which shows the hardware constitutions of the imaging device 1 which concerns on one Embodiment of the image processing apparatus of this invention. It is a schematic diagram for explaining generation of a whitening image in the present embodiment. It is a schematic diagram which shows the example of preparation of a mask image. It is a figure which shows the histogram and the weighting table of a lip color level which concern on the setting of weighting of a lip color level. It is a schematic diagram for demonstrating preparation of a fixed map. It is a figure which shows the LUT of V correction. It is a functional block diagram which shows the functional structure for performing whitening image generation processing among the functional structures of the imaging device 1 of FIG. 8 is a flowchart illustrating a flow of whitening image generation processing executed by the image pickup apparatus 1 of FIG. 1 having the functional configuration of FIG. 7. It is a flow chart explaining the flow of mask image creation processing of whitening image generation processing. It is a flowchart explaining the flow of the color correction image creation process of the whitening image creation process.

  Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing a hardware configuration of an image pickup apparatus 1 according to an embodiment of an image processing apparatus of the present invention.
The imaging device 1 is configured as a digital camera, for example.

  As shown in FIG. 1, the image pickup apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, and an image pickup. It includes a unit 16, an input unit 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21.

  The CPU 11 executes various processes according to the program recorded in the ROM 12 or the program loaded from the storage unit 19 into the RAM 13.

  The RAM 13 also stores data necessary for the CPU 11 to execute various processes.

  The CPU 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output interface 15 is also connected to the bus 14. An imaging unit 16, an input unit 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21 are connected to the input / output interface 15.

  Although not shown, the imaging unit 16 includes an optical lens unit and an image sensor.

The optical lens unit is configured by a lens that condenses light, such as a focus lens and a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens section is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance, if necessary.

The image sensor includes a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal to the AFE as an analog signal.
The AFE executes various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and is output as an output signal of the image pickup unit 16.
Such an output signal of the image pickup unit 16 is appropriately supplied to the CPU 11, an image processing unit (not shown), or the like as data of a picked-up image.

The input unit 17 is configured by various buttons and the like, and inputs various information according to a user's instruction operation.
The output unit 18 includes a display, a speaker, and the like, and outputs an image and sound.
The storage unit 19 includes a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various image data.
The communication unit 20 controls communication with other devices (not shown) via a network including the Internet.

  A removable medium 31 made up of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 21. The program read from the removable medium 31 by the drive 21 is installed in the storage unit 19 as needed. Further, the removable medium 31 can also store various data such as image data stored in the storage unit 19 in the same manner as the storage unit 19.

  In the image pickup apparatus 1 configured as described above, the image of the face is subjected to the whitening process on the skin, and the skin color and the hue are close to each other. It has a function of generating an image (hereinafter, referred to as “whitening image”) in which the lips are appropriately corrected. That is, the imaging device 1 generates a whitening image by restoring (reducing) the effect of the whitening process by correcting the lip color corrected by the whitening process applied to the skin of the face to an appropriate color.

That is, the whitening process is masked except for the target area so that it is applied only to the skin color region, but since the hue of the lips is close to the skin color and is included in the skin color region, the whitening process is performed. It will be affected by the color of the lips becoming lighter. The whitening process is mainly performed by correcting three elements of saturation reduction, brightness increase, and hue rotation in the blue direction.
Therefore, in the image pickup apparatus 1 according to the present embodiment, of the three elements, only the saturation reduction is restored with respect to the thinned lip color, and the other effects are maintained, and the lip color is changed. Correct to an appropriate color.

  In the present embodiment, the whitening image is first subjected to whitening processing for the entire face, and then separately subjected to color correction for reproducing an appropriate color for the lips affected by the whitening processing. Is generated.

FIG. 2 is a schematic diagram for explaining generation of a whitening image in this embodiment.
As for the whitening image of the present embodiment, as shown in FIG. 2, a region including lips is cut out from an image obtained by whitening the entire image (hereinafter, referred to as “original image”). Then, an image (hereinafter referred to as a "color-corrected image") is obtained by performing color correction processing on the region including the lips so that the color of the lips that has been lightened by the whitening process is reproduced to an appropriate color. create. Since the color-corrected image is color-corrected for the entire image other than the lips, a mask image that masks the parts other than the lips is used so that the effect of the color correction is exerted only on the lips. The images are combined by α blending to create an image in which only the lips are color-corrected (hereinafter referred to as “lip color-corrected image”). The mask image becomes an α value when the images are combined by α blending.
Then, the created lip color correction image is combined with the cutout position of the original image to generate a whitening image.

[Create mask image]
FIG. 3 is a schematic diagram showing an example of creating a mask image.
In the present embodiment, a mask image corresponding to the region corresponding to the lips is generated in order to perform the correction process for reproducing the color of the lip portion that has become thin due to the whitening process.
As shown in FIG. 3, in the mask image, the black portion has a lower proportion of color-corrected image composition, and the white portion has a higher proportion of color-corrected image composition. In the present embodiment, the central portion of the lip is strongly color-corrected, and the portion near the outer edge of the lip is strongly color-corrected in order to reduce the influence on the skin.

  Such a mask image is created by first identifying the position of the lips by detecting the facial organs, and based on the hue value that is the peak of the HSV color space in the area including the identified position of the lips. In the face organ detection of the present embodiment, as shown in FIG. 2, at least two points P1 and P2 above and below the center of the upper lip and two points P3 above and below the center of the lower lip, for the purpose of specifying the lip position and the like. , P4, and two points P5 and P6 at the left and right ends of the lips, a total of 6 positions are detected.

  In the present embodiment, a fixed map simulating the shape of the lips is synthesized in advance in the mask image generated based on the peak hue value of the HSV color space in order to eliminate an unnecessary portion such as the nose. , Create the final mask image.

  Specifically, the mask image of the present embodiment creates a HUE map from an image obtained by cutting out an area including lips from the original image (hereinafter referred to as a “cutout image”), and a fixed map created in advance. It is created by combining with. In the combination of the HUE map and the fixed map, the minimum value of the HUE map and the fixed map is taken in order to remove a spatially unnecessary area from the HUE map.

[Creation of HUE map]
The HUE map is a map that specifies a portion corresponding to the lip color in the cutout image. The HUE map first converts the clipped image in the YUV color space into the HSV (hue: Hue, saturation: Saturation / Chroma, lightness: Value / Lightness / Brightness) color space. After that, in order to remove noise, the image is subjected to blurring with the ε filter and then the image after HSV conversion is analyzed (HSV analysis processing). Then, the HUE map is created using the value calculated from the processing result of the HSV analysis processing as the map value of the HUE map.

  In the HSV analysis processing, lip color level weighting is set in order to determine the map value at each pixel. The weighting of the lip color level is set based on the histogram created for each HSV channel.

FIG. 4 is a diagram showing a histogram and a lip color level weighting table relating to setting of lip color level weighting.
In setting the lip color level, first, measurement areas R1 and R2 are set near the center of the upper and lower lips as measurement areas from the points P1 to P6 detected by the organ detection shown in FIG. 2, and the histogram of each HSV in each measurement area is set. To create. Note that the measurement area on the lips relating to the creation of the histogram may be the number of areas, various shapes, positions, and sizes as long as the lip color can be acquired.

  Then, a lip color level weighting table is set as shown in FIG. 4B corresponding to the mode of each histogram of HSV as shown in FIG. 4A. In the case of the H channel, weighting is equally distributed to plus and minus with reference to the mode value, and the weighting is set to be lower as the mode value is further away. In the case of the S (V) channel, the peak value has a predetermined width with the mode of the histogram being the reference of the upper left end (right end), and the second stage is higher than the first stage. The weighting is set so that the gradient becomes two-step gradient with a gentler gradient.

Then, a HUE map is created with the result of multiplying the lip color level calculated according to the weighting for each HSV channel for each pixel as a map value.
That is, the map value of the HUE map is calculated by the following equation (1).
Map value of HUE map (Map) = Lh × Ls × Lv (1)
In addition, "Lh" shows the lip color level of H, "Ls" shows the lip color level of S, and "Lv" shows the lip color level of V.

[Create fixed map]
FIG. 5 is a schematic diagram for explaining the creation of a fixed map.

The fixed map is a map that imitates the shape of a lip on a general face, and is a map created in advance for color correction.
As the fixed map, as shown in FIG. 5A, a reduced size map is developed from the data.
Then, as shown in FIG. 5B, the angle of the lip inclination is calculated from the contour information of the lips in the image (the positions of the two points P5 and P6 in the present embodiment), and fixed in correspondence with the angle. Rotate the map.
Finally, the fixed map is used in combination with the HUE map to resize it to a size corresponding to the image and remove parts other than the lips in the HUE map. When combining, the minimum value of the HUE map and the fixed map is taken.

[Create Color Corrected Image]
In the present embodiment, the color-corrected image is subjected to correction processing on the cut-out image with the correction intensity determined according to the V value of the YUV color space measured from the skin regions under the left and right pupils detected by the organ detection. Created. The skin areas below the left and right pupils are positions that avoid the bear under the eyes, and are determined as the positions that most represent the skin color on the face, and the skin color on the face can be extracted. Various positions may be used as long as they are positions.

If the original lip color is dark due to a lip or the like, the V value of the pixel of the lip is corrected so that the correction is not overly effective.
FIG. 6 is a diagram showing a V correction LUT (Look Up Table).
As shown in FIG. 6, the V-correction LUT is configured so that the V-correction is applied to a portion where the lip color is dark.

[Create Lip Color Correction Image]
The lip color correction image is created by performing α blending of the color correction image and the cutout image based on the mask image to perform image blending. The created color-corrected image is pasted on the cutout position of the original image to generate a whitening image. That is, the color correction is applied only to the lips area, and the color-corrected image with the whitening treatment is applied to the other skin parts, and the whitening image is generated by pasting it on the original image with the same whitening treatment. To be done.

FIG. 7 is a functional block diagram showing a functional configuration for executing whitening processing, among the functional configurations of the image pickup apparatus 1 of FIG.
Whitening image generation processing applies whitening processing to the original image including the photographed person's face, and also makes it possible to properly treat lips whose color has become pale due to the effect of whitening processing on skin. It refers to a series of processes for generating a whitening image by correcting various colors.

  When executing the whitening image generation process, as shown in FIG. 7, in the CPU 11, the image acquisition unit 51, the face detection unit 52, the image processing unit 53, the mask image creation processing unit 54, and the color correction image. The creation processing unit 55 and functions.

  Further, an image storage unit 71 and a fixed map storage unit 72 are set in one area of the storage unit 19.

The image storage unit 71 stores image data output from the imaging unit 16.
The fixed map storage unit 72 stores fixed map data used to create a mask image. Since the fixed map has an arc shape imitating the lips in order to reduce the amount of data, the arc may be divided into quarters and data may be stored. In this case, the fixed map is developed so that it becomes a circular arc during processing.

  The image acquisition unit 51 acquires the data of the captured image in which the image captured by the image capturing unit 16 is subjected to the development processing, or the data of the image to be processed from the image storage unit 71.

  The face detection unit 52 detects a face from the image and also detects each organ constituting the face in the detected face. The face detection unit 52 detects at least the left and right pupils and the organ of the lip from the contour shape. Regarding the lips, the face detection unit 52 has at least two upper and lower points P1 and P2 at the center of the upper lip, two upper and lower points P3 and P4 at the center of the lower lip, and two points P5 and P6 at the left and right ends of the lip. The positions of 6 points in total are detected. For face detection and detection of each organ, known techniques of existing face detection technology and organ technology are used. Further, the face detection and the detection of each organ may be any detection method as long as it is a detection method based on information different from the color information of the HSV color space, such as brightness and contour extraction.

The image processing unit 53 executes various kinds of image processing on the original image.
Specifically, for example, the image processing unit 53 mainly performs whitening processing for correcting three elements, that is, saturation reduction, brightness increase, and hue rotation in the blue direction. Further, the image processing unit 53 cuts out a region including lips from the original image as shown in FIG. Further, the image processing unit 53 performs α blending of the cut-out image and the created color-corrected image based on the created mask image to perform image composition. As a result, a lip color corrected image is created. Further, the image processing unit 53 pastes the image at the original cutout position. As a result, a whitening image is generated.

  The mask image creation processing unit 54 generates a mask image used for α blending so that the image is color-corrected only in the lip region.

  Specifically, the mask image creation processing unit 54 converts the YUV color space of the cutout image into the HSV color space, as shown in FIG. Further, the mask image creation processing unit 54 performs a process of blurring an image with an ε filter for the purpose of removing noise.

  Further, the mask image creation processing unit 54 executes HSV analysis processing. In the HSV analysis processing, a histogram is created for each HSV channel of the measurement regions R1 and R2 (see FIG. 2) set near the center of the upper and lower lips (see FIG. 4A). Then, the weighting of the lip color level calculated from the created histogram is created (see FIG. 4B). Then, based on the created weighting of the lip color level, the map value of the HUE map in each pixel is calculated using the above-mentioned formula (1). A HUE map is created from the calculated map value of each pixel.

  Further, as shown in FIGS. 5A and 5B, the mask image creation processing unit 54 adjusts the angle of the fixed map stored in the fixed map storage unit 72 so as to correspond to the lips of the clipped image. Adjust the size.

  Further, as shown in FIG. 3, the mask image creation processing unit 54 combines the fixed map with the angle adjustment and the size adjustment with the created HUE map. At this time, the mask image creation processing unit 54 combines the HUE map and the fixed map so as to take the minimum value in order to remove a spatially unnecessary area from the HUE map such as the nose. As a result of the synthesis, a mask image is created.

The color correction image creation processing unit 55 executes color correction processing on the cutout image.
Specifically, the color correction image creation processing unit 55 measures the V value of the skin area below the detected left and right pupils. Then, the color correction image creation processing unit 55 performs color correction on the clipped image according to the measured V value. As a result, a color-corrected image that is color-corrected with respect to the cut-out image is created.

FIG. 8 is a flowchart illustrating the flow of whitening image generation processing executed by the image pickup apparatus 1 of FIG. 1 having the functional configuration of FIG. 7.
The whitening image generation process is started by the user's operation of starting the whitening image generation process on the input unit 17.
The operation for starting the whitening image generation processing is a shooting instruction operation, and the image captured by the image capturing unit 16 in response to the shooting instruction operation continues to be a whitening image for the data of the captured image subjected to the development processing. The generation process may be performed, or the operation of selecting the captured image data stored in the image storage unit 71 and starting the whitening image generation process for the selected captured image data may be performed.

  In step S <b> 11, the image acquisition unit 51 acquires the data of the captured image in which the image captured by the image capturing unit 16 is subjected to the development processing, or the original image to be processed from the image storage unit 71.

In step S12, the face detection unit 52 performs face detection in the original image and determines whether a face has been detected.
If no face is detected, NO is determined in step S12, and the whitening image generation process ends.
If a face is detected, YES is determined in step S12, and the process proceeds to step S13.

  In step S13, the image processing unit 53 executes whitening processing on the original image. By the whitening process, the original image has reduced saturation, increased brightness, and has its hue rotated in the blue direction.

  In step S14, the face detection unit 52 detects the detected facial organs. As a result, at least the positions of the left and right pupils and the organ of the lip are detected in the original image.

  In step S15, the image processing unit 53 cuts out a region including the detected lip.

  In step S16, the mask image creation processing unit 54 executes mask image creation processing. As a result of the execution of the mask image creation processing, a mask image for α blending is created. Details of the mask image creation processing will be described later.

  In step S17, the color correction image creation processing unit 55 executes color correction image creation processing. As a result of the execution of the color-corrected image creation processing, a color-corrected image is created in which the cut-out image is color-corrected to make the thinned lips an appropriate color. Details of the color correction image creation processing will be described later.

  In step S18, the image processing unit 53 performs α blending on the cut-out image and the created color-corrected image based on the created mask image to perform image composition. As a result, a lip color-corrected image in which only the lip portion is color-corrected is created.

In step S19, the image processing unit 53 attaches the lip color corrected image to the original cutout position. As a result, a whitening image is generated in which the skin is whitened and the lips are corrected to have an appropriate color.
Then, the whitening image generation process ends.

  FIG. 9 is a flowchart illustrating the flow of the mask image creation process of the whitening image creation process.

  In step S31, the mask image creation processing unit 54 converts the YUV color space of the cutout image into the HSV color space.

  In step S32, the mask image creation processing unit 54 performs a blurring process using the ε filter.

  In step S33, the mask image creation processing unit 54 executes the HSV analysis processing. In the HSV analysis process, a histogram is created for each HSV channel of the measurement regions R1 and R2 set near the center of the upper and lower lips as shown in FIG. 2 (see FIG. 4A). Then, the weighting of the lip color level calculated from the created histogram is set (see FIG. 4B). In the case of the H channel, the weighting of the lip color level is set such that the weighting is equally distributed to the plus and the minus with respect to the mode value, and the weighting becomes lower as the mode value is further away. In the case of the S (V) channel, the peak value has a predetermined width with the mode of the histogram being the reference of the upper left end (right end), and the second stage is higher than the first stage. The weighting is set so that the gradient becomes two-step gradient with a gentler gradient.

  In step S34, the mask image creation processing unit 54 creates a HUE map. The map value in the HUE map is calculated using the above-mentioned formula (1) based on the created weighting of the lip color level.

  In step S35, the mask image creation processing unit 54 performs angle adjustment and size adjustment so that the fixed map stored in the fixed map storage unit 72 corresponds to the lips of the clipped image.

  In step S36, the mask image creation processing unit 54 combines the fixed map that has undergone the angle adjustment and the size adjustment with the created HUE map. At this time, the mask image creation processing unit 54 combines the HUE map and the fixed map so as to take the minimum value in order to remove a spatially unnecessary region from the HUE map such as the nose. As a result of the synthesis, a mask image is created.

  FIG. 10 is a flowchart illustrating the flow of the color-corrected image creating process in the whitening image creating process.

  In step S51, the color correction image creation processing unit 55 measures the V value of the skin area below the detected left and right pupils.

  In step S52, the color correction image creation processing unit 55 performs color correction on the cutout image according to the measured V value. As a result, a color-corrected image that is color-corrected with respect to the cut-out image is created.

Various image pickup apparatuses have a function of allowing the user to select a flesh color, and for example, “natural” for improving the complexion of the face and “whitening” for whitening the face are selectable. Natural colors produce good color reproduction, and lip colors can also develop well. On the other hand, with whitening, the saturation of the skin color is lightened, and the lip color is also dragged by it, resulting in a lighter and less complexion.
Therefore, in the image pickup apparatus 1 of the present embodiment, the lip area is extracted from the detected face area by using the result of detecting the contour of the image of a person, and the lip color is made vivid or lip is applied. Can be corrected to. To extract the lip color region, an image (HSV image) expressed in the HSV color space is analyzed, and a mask image for α blending is created from the HSV image. At the time of correction, the image expressed in the YUV color space can be analyzed and corrected in each of YUV (particularly V).

The imaging device 1 configured as described above includes the image processing unit 53, the mask image creation processing unit 54, and the color correction image creation processing unit 55.
The image processing unit 53 performs a whitening process for whitening the skin color on the face area of the person included in the image.
The mask image creation processing unit 54 identifies a portion in the face area of the person to which the whitening process is applied, where the whitening process should be reduced.
The color-corrected image creation processing unit 55 performs a process of reducing the whitening process on the specified portion where the whitening process should be reduced.
As a result, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and the image can be used for the whitening process performed on the entire face area of the person, and the appropriate whitening process is performed on the entire face of the person. be able to.

The whitening treatment consists of a plurality of types of treatment.
The mask image creation processing unit 54 reduces the effects of some of the multiple types of processing and reduces the effects of other types of processing within the face area of a person to which the whitening process including multiple types of processing is performed. Identify the parts to keep.
As a result, in the image pickup apparatus 1, for example, when the whitening process is performed with the three elements of the color saturation reduction, the lightness increase, and the hue rotation in the blue direction, only the color saturation reduction is restored, and Can be appropriately specified, and can be used for whitening treatment applied to the entire face area of a person.

Further, the image processing unit 53 identifies the skin color portion in the image based on the color information of the HSV color space.
Further, the image processing unit 53 performs whitening processing on the specified skin color portion.
The mask image creation processing unit 54 identifies a portion included in the identified skin color portion and a portion other than the skin having a hue value similar to the skin color as a portion where the whitening process should be reduced.
Thereby, in the image pickup apparatus 1, a part included in the specified skin color part and a part other than the skin having a hue value similar to the skin color is specified as a part in which the whitening process should be reduced. , Appropriate whitening treatment can be applied to the entire face of a person.

Further, the image pickup apparatus 1 further includes a face detection unit 52 that specifies a face part based on a reference different from the color information of the HSV color space.
The face detection unit 52 identifies a portion of the identified face, other than the skin whose color information in the HSV color space approximates the skin color, as the portion where the whitening process should be reduced.
As a result, in the image pickup apparatus 1, in order to specify a portion other than the skin whose color information in the HSV color space approximates the skin color as a portion where the whitening process should be reduced, it is appropriate for the entire face of the person. A whitening treatment can be applied.

In addition, the color correction image creation processing unit 55 specifies the density of the color of the specified portion where the whitening processing should be reduced based on the color information of the YUV color space.
The color correction image creation processing unit 55 performs processing for reducing the whitening processing according to the specified density.
With this, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and the image can be used for the whitening process performed on the entire face area of the person.

The face detection unit 52 further identifies the portion where the whitening process should be reduced by using the information that is prepared in advance and serves as a reference for the shape of the face part.
As a result, in the image pickup apparatus 1, the position of the lips can be appropriately specified, and appropriate whitening processing can be performed on the entire face of the person.

The mask image creation processing unit 54 identifies the lip portion of the person's face as the portion where the whitening process should be reduced.
By setting the lip portion of the person's face as a portion where the whitening process should be reduced, it is possible to perform appropriate whitening processing on the entire face of the person.

The face detection unit 52 specifies the lip portion of the person's face included in the image based on a standard different from the color information of the HSV color space.
The mask image creation processing unit 54 identifies the lip color based on the color information of the HSV color space in the identified lip portion.
With this, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and the image can be used for the whitening process performed on the entire face area of the person.

The mask image creation processing unit 54 specifies, in the HSV color space, a color in which the color information of the specified lip portion approximates the color information of the skin color within a predetermined range as the lip color.
With this, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and the image can be used for the whitening process performed on the entire face area of the person.

The mask image creation processing unit 54 specifies the lip color based on the distribution of the color information of the specified lip portion in the HSV color space.
With this, in the image pickup apparatus 1, the color of the lips can be appropriately specified, and the image can be used for the whitening process performed on the entire face area of the person.

The face detection unit 52 specifies the lip portion based on a standard different from the color information of the HSV color space.
As a result, in the imaging device 1, the position of the lips can be appropriately specified, and the lips can be used for the whitening process performed on the entire face area of the person.

Further, the image processing unit 53 performs predetermined image processing on the lip portion of the face of the person identified by the face detection unit 52 based on the identified lip color using the color information in the YUV color space.
As a result, the image pickup apparatus 1 can perform image processing according to the lip color.

  It should be noted that the present invention is not limited to the above-described embodiments, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

  In the above-described embodiment, the lip color affected by the whitening process is corrected to an appropriate color, but the present invention is not limited to this. The specified lip region may be corrected so as to change the lip color. Specifically, you can apply different lips and lipstick colors, or apply makeup to lip without lip, lipstick, gloss, etc. by applying lip, lipstick, gloss, etc., or adjust the degree of blood color. You may. At this time, the saturation may be positively changed, and further, the brightness and the hue may be changed.

  Further, in the above-described embodiment, the lip region may be specified first, and the lip region specified as the skin mask region may be specified during the whitening process performed on the entire face.

Further, in the above-described embodiment, the image pickup apparatus 1 to which the present invention is applied has been described by taking a digital camera as an example, but the present invention is not particularly limited to this.
For example, the present invention can be applied to general electronic devices having a whitening treatment function. Specifically, for example, the present invention is applicable to a notebook personal computer, a printer, a television receiver, a video camera, a portable navigation device, a mobile phone, a smartphone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or software.
In other words, the functional configuration of FIG. 7 is merely an example and is not particularly limited. That is, it is sufficient if the image pickup apparatus 1 has a function capable of executing the series of processes described above as a whole, and what kind of functional block is used to realize this function is not particularly limited to the example of FIG. 7.
Further, one functional block may be configured by hardware alone, software alone, or a combination thereof.
The functional configuration in the present embodiment is realized by a processor that executes arithmetic processing, and the processor that can be used in the present embodiment is configured by a single processing device such as a single processor, a multiprocessor, and a multicore processor. In addition to the above, these various processing devices are combined with a processing circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

When the series of processes is executed by software, the program forming the software is installed in a computer or the like from a network or a recording medium.
The computer may be a computer embedded in dedicated hardware. Further, the computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium containing such a program is not only configured by the removable medium 31 of FIG. 1 which is distributed separately from the apparatus main body to provide the program to the user, but is also installed in the apparatus main body in advance. It is composed of a recording medium provided to the. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disc includes, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), and a Blu-ray (registered trademark) Disc (Blu-ray disc). The magneto-optical disk is composed of MD (Mini-Disk) or the like. Further, the recording medium provided to the user in the state of being incorporated in the apparatus main body in advance is constituted by, for example, the ROM 12 in FIG. 1 in which the program is recorded, the hard disk included in the storage unit 19 in FIG. 1, or the like.

  In the present specification, the steps of writing the program recorded on the recording medium are not limited to the processes performed in time series according to the order, but may be performed in parallel or individually even if they are not necessarily performed in time sequence. It also includes the processing to be executed.

  Although some embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are also included in the invention described in the claims and the scope of equivalents thereof.

The inventions described in the claims at the time of filing the present application will be additionally described below.
[Appendix 1]
Whitening processing means for performing whitening processing for whitening the skin color for the face area of the person included in the image;
Within the face area of the person to which the whitening process is applied, a reduced portion specifying unit that specifies a portion where the whitening process should be reduced,
Reduction processing means for performing a process of reducing the whitening process on the identified part where the whitening process should be reduced,
An image processing apparatus comprising:
[Appendix 2]
The whitening treatment consists of multiple types of treatment,
The reduced portion specifying means reduces the effect of a part of the plurality of types of processing within the face area of the person to which the whitening processing including a plurality of types of processing is applied, and maintains the effects of other processing. Identify the part to be done,
The image processing device according to appendix 1, characterized in that.
[Appendix 3]
A flesh color part specifying unit for specifying a flesh color part in the image based on the color information of the HSV color space;
The whitening processing means performs whitening processing on the identified skin color portion,
The reduced portion identifying means identifies a portion included in the identified skin color portion, where the hue value is similar to the skin color, other than the skin, is a portion where the whitening process should be reduced,
The image processing device according to appendix 1 or 2, characterized in that.
[Appendix 4]
Further comprising part specifying means for specifying a face part based on a reference different from the color information of the HSV color space,
The part specifying unit specifies a part of the specified face that is other than the skin in which the color information of the HSV color space approximates the skin color, as the part in which the whitening process should be reduced. ,
The image processing device according to appendix 3, wherein.
[Appendix 5]
Further comprising density specifying means for specifying the density of the color of the specified portion to be subjected to the whitening treatment based on the color information of the YUV color space,
The reduction processing unit performs processing for reducing the whitening processing according to the specified density,
The image processing device according to supplementary note 3 or 4, characterized in that.
[Appendix 6]
The part specifying unit further specifies a part in which the whitening process should be reduced by using information that serves as a reference for the shape of a face part prepared in advance.
6. The image processing apparatus according to any one of appendices 3 to 5, characterized in that.
[Appendix 7]
The reduced portion specifying unit specifies a lip portion of a human face as a portion where the whitening process should be reduced,
7. The image processing device according to any one of appendices 1 to 6, characterized in that.
[Appendix 8]
A lip part specifying means for specifying a lip part of a person's face included in the image based on a standard different from the color information of the HSV color space;
Lip color specifying means for specifying a lip color based on the color information of the HSV color space in the specified lip portion,
An image processing apparatus comprising:
[Appendix 9]
The lip color identifying means identifies, in the HSV color space, a color in which the color information of the identified lip portion approximates to the color information of the skin color within a predetermined range, as the lip color.
The image processing device according to appendix 8, characterized in that.
[Appendix 10]
The lip color specifying means specifies a lip color based on the distribution of the color information of the specified lip portion in the HSV color space,
The image processing device according to appendix 8 or 9, characterized in that.
[Appendix 11]
The lip portion identifying means identifies the lip portion based on a criterion different from the color information of the HSV color space,
11. The image processing apparatus according to any one of appendices 8 to 10, characterized in that.
[Appendix 12]
Image processing means for performing a predetermined image processing on the lip portion of the person's face identified by the lip portion identifying means in accordance with the identified lip color, based on the color information in the YUV color space,
12. The image processing apparatus according to any one of appendices 8 to 11, characterized in that.
[Appendix 13]
Whitening processing that whitens the skin color for the face area of the person contained in the image,
Within the face area of the person to which the whitening process is applied, a reduced part specifying process for specifying a part where the whitening process should be reduced,
A reduction process for performing a process of reducing the whitening process on a portion where the identified whitening process should be reduced,
An image processing method comprising:
[Appendix 14]
A lip part specifying process for specifying a lip part of a person's face included in the image based on a criterion different from the color information of the HSV color space;
A lip color specifying process for specifying a lip color based on color information of the HSV color space in the specified lip portion;
An image processing method comprising:
[Appendix 15]
On the computer,
Whitening processing means for performing whitening processing for whitening the skin color for the face area of the person included in the image,
In the face area of the person to which the whitening process is applied, a reduced part specifying unit that specifies a part in which the whitening process should be reduced,
Reduction processing means for performing a process of reducing the whitening process on the identified part to reduce the whitening process,
A program characterized by realizing.
[Appendix 16]
On the computer,
A lip part specifying means for specifying the lip part of the person's face included in the image based on a criterion different from the color information of the HSV color space;
Lip color specifying means for specifying a lip color based on color information of the HSV color space in the specified lip portion,
A program characterized by realizing.

  1 ... Imaging device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Bus, 15 ... Input / output interface, 16 ... Imaging unit, 17 ... Input unit, 18 ... Output unit, 19 ... Storage unit, 20 ... Communication unit, 21 ... Drive, 31 ... Removable media, 51 ... Image acquisition unit, 52 ... Face Detection unit, 53 ... Image processing unit, 54 ... Mask image creation processing unit, 55 ... Color correction image creation processing unit, 71 ... Image storage unit, 72 ... Fixed map storage unit

In order to achieve the above object, an image processing device according to one embodiment of the present invention is
A lip part specifying means for specifying a lip part of a person's face included in the image based on a standard different from the color information of the HSV color space;
Lip color specifying means for specifying a lip color based on the color information of the HSV color space in the specified lip portion,
It is characterized by including.

Claims (16)

Whitening processing means for performing whitening processing for whitening the skin color for the face area of the person included in the image;
Within the face area of the person to which the whitening process is applied, a reduced portion specifying unit that specifies a portion where the whitening process should be reduced,
Reduction processing means for performing a process of reducing the whitening process on the identified part where the whitening process should be reduced,
An image processing apparatus comprising: The whitening treatment consists of multiple types of treatment,
The reduced portion specifying means reduces the effect of a part of the plurality of types of processing within the face area of the person to which the whitening processing including a plurality of types of processing is applied, and maintains the effects of other processing. Identify the part to be done,
The image processing apparatus according to claim 1, wherein: A flesh color part specifying unit for specifying a flesh color part in the image based on the color information of the HSV color space;
The whitening processing means performs whitening processing on the identified skin color portion,
The reduced portion identifying means identifies a portion included in the identified skin color portion, where the hue value is similar to the skin color, other than the skin, is a portion where the whitening process should be reduced,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Further comprising part specifying means for specifying a face part based on a reference different from the color information of the HSV color space,
The part specifying unit specifies a part of the specified face that is other than the skin in which the color information of the HSV color space approximates the skin color, as the part in which the whitening process should be reduced. ,
The image processing device according to claim 3, wherein Further comprising density specifying means for specifying the density of the color of the specified portion to be subjected to the whitening treatment based on the color information of the YUV color space,
The reduction processing unit performs processing for reducing the whitening processing according to the specified density,
The image processing device according to claim 3 or 4, wherein The part specifying unit further specifies a part in which the whitening process should be reduced by using information that serves as a reference for the shape of a face part prepared in advance.
The image processing apparatus according to any one of claims 3 to 5, characterized in that. The reduced portion specifying unit specifies a lip portion of a human face as a portion where the whitening process should be reduced,
The image processing apparatus according to any one of claims 1 to 6, wherein A lip part specifying means for specifying a lip part of a person's face included in the image based on a standard different from the color information of the HSV color space;
Lip color specifying means for specifying a lip color based on the color information of the HSV color space in the specified lip portion,
An image processing apparatus comprising: The lip color identifying means identifies, in the HSV color space, a color in which the color information of the identified lip portion approximates to the color information of the skin color within a predetermined range, as the lip color.
The image processing device according to claim 8, wherein The lip color specifying means specifies a lip color based on the distribution of the color information of the specified lip portion in the HSV color space,
The image processing apparatus according to claim 8 or 9, characterized in that. The lip portion identifying means identifies the lip portion based on a criterion different from the color information of the HSV color space,
The image processing apparatus according to claim 8, wherein the image processing apparatus is an image processing apparatus. Image processing means for performing a predetermined image processing on the lip portion of the person's face identified by the lip portion identifying means in accordance with the identified lip color, based on the color information in the YUV color space,
The image processing apparatus according to any one of claims 8 to 11, characterized in that. Whitening processing that whitens the skin color for the face area of the person contained in the image,
Within the face area of the person to which the whitening process is applied, a reduced part specifying process for specifying a part where the whitening process should be reduced,
A reduction process for performing a process of reducing the whitening process on a portion where the identified whitening process should be reduced,
An image processing method comprising: A lip part specifying process for specifying a lip part of a person's face included in the image based on a criterion different from the color information of the HSV color space;
A lip color specifying process for specifying a lip color based on color information of the HSV color space in the specified lip portion;
An image processing method comprising: On the computer,
Whitening processing means for performing whitening processing for whitening the skin color for the face area of the person included in the image,
In the face area of the person to which the whitening process is applied, a reduced part specifying unit that specifies a part in which the whitening process should be reduced,
Reduction processing means for performing a process of reducing the whitening process on the identified part where the whitening process should be reduced,
A program characterized by realizing. On the computer,
A lip part specifying means for specifying the lip part of the face of the person included in the image based on a standard different from the color information of the HSV color space;
Lip color specifying means for specifying the color of the lip based on the color information of the HSV color space in the specified lip portion,
A program characterized by realizing.