JP4522229B2 - Image processing method and apparatus - Google Patents

Description

  The present invention relates to an image processing method and apparatus, and more particularly, to a color image device such as a digital camera, a display, a scanner, and a printer, and a color image correction process in image processing software of the color image device.

  Conventionally, various image processing such as adjustment of saturation, color tone, contrast, and gradation has been performed on digital images. Conventionally, when performing such image processing, an operator who has specialized knowledge about the image uses specialized or dedicated software and makes full use of empirical knowledge so that the image processing can be performed on a computer monitor screen. In general, trial and error are performed while confirming the result to obtain a suitable image.

  In recent years, the popularity of digital cameras has been increasing, driven by the popularity of the Internet. This is because the result (data) obtained by photographing with a digital camera is data in a file format that can be easily read by a computer. For example, it is possible to easily store an image photographed by a digital camera on a WWW (World Wide Web) server and open it to a third party. With the spread of such digital cameras, it is also true that digital image data has increased around users who have never had much connection with images.

  However, users of digital cameras are not users who have sufficient knowledge about cameras that can freely use conventional analog cameras, but rather users who are familiar with computers but do not have enough knowledge about cameras. It is done. Therefore, images taken with a digital camera are not necessarily only images taken under suitable conditions. That said, even if an image was taken under unfavorable conditions, it may not always be discarded if the content is important to the photographer. For this reason, there is a demand for an image correction method that can obtain an appropriate image from this image even if the image is taken under unfavorable conditions. In that case, as described above, the image correction method is preferably automatic or semi-automatic in view of the fact that it is not always the user who is familiar with the camera or the image or that the image data increases. It can be said.

  For example, the color matching technique corrects an image so as not to be affected by the characteristics of input / output devices, and requires color correction faithful to the actual product. On the other hand, when dealing with a photographic image taken with a digital camera, it is also important to perform image correction so that a person feels better than being true to the real thing. In the case of portraits, the main area of interest is the face of the person who is the subject, and it is preferable that the face area has the appropriate brightness (skin color). In such a case, it can be said that it is preferable that the correction is made so that the brightness (skin color) is adjusted. Furthermore, when men and women are shown in one image, it is generally preferred that a woman be brighter than a man.

  Regarding the brightness adjustment described above, for example, as disclosed in Patent Document 1, the brightness distribution state of an image is examined, and several percent levels are set as a highlight point and a shadow point, respectively, from the bright side and the dark side. To do. From the process of level expansion so that the highlight point becomes almost the maximum value of the brightness and the shadow point becomes the almost minimum value of the brightness, and the average luminance of the entire image, etc. Correction was performed.

  As another known example, Patent Document 2 corrects the lightness so that the automatically detected face area has a lightness set in advance under shooting conditions with limited poses and light sources such as a proof photograph. To avoid variations among operators, preset values were prepared for each gender and eye color, and the operator determined the gender and eye color, set the reference brightness data, and made corrections. .

Further, as a method for the operator to select and correct the correction contents when the correction method is not uniquely determined, as shown in Patent Document 3, the entire correction result image is displayed as a list or a comparison display before and after correction. The method and correction details were adjusted.
JP 2001-189862 A JP 2000-261650 A Japanese Patent Laid-Open No. 10-221773

  However, in the method according to Patent Document 1, a region other than the face is subject to correction, and preferable image correction cannot be performed. In Patent Document 2, the detected face is discriminated by the operator. In addition, because the brightness of the face area is forcibly converted to a predetermined brightness value, it is limited to images taken under limited shooting conditions such as lighting photos, as if taken with a general digital camera It could not be applied to photographs taken under various light sources. Further, in Patent Document 3, when there are a plurality of regions to be corrected and there are a plurality of correction methods, the number of combinations becomes enormous, and the operability is remarkably deteriorated by the list display method.

  In view of the above-described conventional problems, the present invention provides an image processing method capable of performing preferable image correction on a plurality of target areas with a simple operation even when there are a plurality of target areas for which different correction methods are required in an input image. The device is provided.

In order to achieve the above object, an image processing apparatus according to the present invention detects image input means for inputting image information, and whether or not a preset target is included in the image information input from the image input means, an area setting means for setting a detected region, for each region set by the region setting means, to determine the type of the region, and the region discriminating means for outputting a likelihood of discrimination with the type of the region, the region discrimination based on the type discriminated by the section, have a image correcting means for performing image correction for each area set, the image correcting means, the area in any region where the region setting means has set When the probability of discrimination output by the discrimination means is less than a predetermined threshold, an image correction method planning means for planning image correction method candidates and a correction method for the area to be corrected are presented. And a correcting method instruction unit seek instruction over data, and correcting the image in accordance with an instruction content according to said correction method instruction unit.

Here, the preset target is a human face. In addition, the type of the region is any one or a combination of gender, age, and biological classification of an individual or a human being. The image correction is correction of any one or a combination of lightness, hue, saturation, sharpness, and blurring for pixels included in the specific color gamut in the specific space region . Also, the correction method instruction means displays the image to be corrected, to place the operation button for instructing the correction method in the vicinity of the correction target region. The correction method instructing unit displays an image to be corrected and displays a circumscribed polygon in which the correction target region can be confirmed. The correction method instruction means sets the degree of correction and reflects it on the correction target image being displayed in accordance with the set contents.

Further, according to the present invention, the image the image processing method corresponding to the image processing apparatus, the image program order to execute the respective steps in a computer processing method, computer-readable recording medium storing the program Is provided.

  As described above, according to the present invention, in view of the above-described conventional problems, even when there are a plurality of target areas for which different correction methods are required in an input image, the present invention can be performed with a simple operation. There is an effect that preferable image correction can be performed on the region.

  Hereinafter, the configuration and operation of an image processing apparatus according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

<Conceptual Configuration Example of Image Processing Apparatus of Present Embodiment>
FIG. 1 is a diagram illustrating a conceptual configuration of an image processing apparatus according to the present embodiment.

  In FIG. 1, reference numeral 101 denotes an image input unit for inputting image information. The image input unit 101 of this example includes input from an image input device such as a digital camera or a scanner, an image storage medium such as a floppy (registered trademark) disk, CD, or memory card, and reception via a network.

  Reference numeral 102 denotes an area setting unit that detects whether or not a preset target is included in the image information input from the image input unit 101 and sets the detected area. In this example, a human face is set as a preset target. Whether or not the face is a person is detected based on, for example, the position of a contour or a component (such as an eye or a mouth). Reference numeral 103 denotes an area determination unit that determines the type of each area set by the area setting unit 102. Here, the types determined in this example are sex, age, race, and the like.

  Reference numeral 104 denotes an image correction method planning unit that plans image correction method candidates when the likelihood output by the region determination unit 103 is less than a predetermined threshold in any of the regions set by the region setting unit 102. A correction method selection unit 105 presents a correction method for a region to be corrected and asks for an instruction from the operator.

  An image correction unit 106 performs image correction on the region by the correction method selected by the selection unit 105 based on the type determined by the region determination unit 103. The image correction method planning unit 104 and the selection unit 105 are optional components, and the image correction unit 106 is a correction method determined in advance or determined by the user based on the type determined by the region determination unit 103. A configuration in which image correction is performed in the above may be used.

  An image output unit 107 outputs the corrected image. For example, the image output unit 107 includes output to an output device such as a display or a printer, output to an image storage medium such as a floppy (registered trademark) disk, CD, or memory card, and further transmission via a network.

<Specific Configuration Example of Image Processing Apparatus of Present Embodiment>
FIG. 2 is a block diagram illustrating a specific configuration example of the image processing apparatus according to the present embodiment.

  The image processing apparatus stores an arithmetic processing CPU 201 that executes instructions according to programs stored in the ROM 202 and the RAM 203, a ROM 202 that stores programs and data necessary for the control according to the present embodiment and other controls, and temporary data. Input images from devices such as RAM 203, drive I / F 204 that realizes an interface with an external storage device such as IDE or SCSI, HDD 205 that stores moving image or feature amount or program for moving image search, digital camera or scanner, etc. An image input unit 206, a buffer memory 207 for storing temporary data, an input unit 208 for inputting from an operator such as a keyboard and a mouse, a display unit 209 such as a cathode ray tube or a liquid crystal display, and a network such as the Internet or an intranet. Network I / F210 such as a modem or LAN for performing connection, image is constituted by a printer 212 for printing out, and inputs and outputs data to each other via a bus 213 that connects these.

  In this image processing apparatus, WINDOWS (registered trademark) XP of Microsoft Corporation is installed as an operating system, and the image processing apparatus is mounted as an application that operates on the XP.

  The RAM 203 of the present embodiment has a data storage area that is temporarily used during the arithmetic processing of the CPU 201 and a program load area for loading the application program of the present embodiment executed by the CPU 201 from the HDD 205 or the like.

  The data storage area includes an input image data area 203a for storing input image data, and an image data area 203b, a type flag area 203c, a calculation likelihood area 203d, and a correction method selection for each detection area i detected as a face area. It includes a flag area 203e, a correction data area 203f, a likelihood threshold area 203g, and an output image data area 203h.

  The program load area includes an image processing program 203i (FIG. 3), an image input module 203k, a face area detection module (FIG. 4), a face area determination module 203m (FIG. 6), and an image correction method planning module 203n (FIG. 7). ), The correction method selection module 203p, the image correction module 203q (FIG. 11), and the image output module 203r are loaded and executed by the CPU 201.

  A table for storing correction target values corresponding to the sex, age, and biological classification of human beings used in this embodiment is held in the ROM 202 if it is fixed, and in the RAM 203 if it is variable. . In FIG. 2, a correction target value table 202 a is illustrated in the ROM 202.

<Example of Operation of Image Processing Apparatus of Present Embodiment>
Next, an outline of image correction processing in the image processing apparatus having such a configuration will be described with reference to the flowchart of FIG.

  After inputting the image data, a face area is detected from the image to be corrected in step S301. Processing branches at step S302 depending on whether an area has been detected. If an area is detected, the process proceeds to step S303, and if not, the process ends.

  In step S303, the type of the face area is determined for each detected face area. Here, the type of face and its likelihood are output for each face area. In this embodiment, three types of faces are defined: gender, age, and human biological classification (so-called race). Age was divided into younger age groups with large growth changes, and was set to 6 levels, 4 levels up to the early 20s and 2 levels thereafter. As a biological classification method of human beings, Saitou 1995, Human Evolution, vol. 10, pp. With reference to 17-33, 6 types were selected: African, West Eurasian, East Eurasian, Safur, North American, and South American. In this case, it can be classified into 2 × 6 × 6 = 72 types.

  In step S304, an image correction method is devised for each detected face area in accordance with a correction method defined for each type of face. In step S305, it is determined whether there is a correction method. Since there is an option of not correcting, if there is only one correction method, the process proceeds to step S306, and if not, the process proceeds to step S307. In step S306, the operator is made to select a correction method. It should be noted that it is possible to omit the operator's selection work in step S306 and execute the correction process completely automatically by setting the correction to be performed using only the discrimination information with the highest likelihood in advance. .

  In step S307, the image input by the selected correction method is corrected. This method is performed on the image to which the selected correction method is output. The correction method will be described later as a detailed description of the image correction method planning process in step S304.

  Of course, it is not necessary to carry out image correction to compensate for color reproduction characteristics that differ for each input / output device before or after this processing, such as saturation enhancement of the entire image, adjustment of luminance distribution, and compensation for different color reproduction characteristics for each input / output device. This is not against the gist of the invention.

  Hereinafter, each process described above will be described in detail.

(Face region detection processing: S301)
Hereinafter, an example of the face area detection process in step S301 of FIG. 3 will be described in detail.

  Details of the face area detection processing will be described with reference to the flowchart of FIG. FIG. 5 is a diagram for explaining face area detection processing.

  First, in step S401, an area having a low brightness is marked based on the input image. In step S402, two regions marked in step S401 are grouped, and it is determined whether the two groups are eyes from the uniformity of the size of the regions, the difference in luminance, the angle with respect to the horizontal, etc. (FIG. 5 (a)).

  In step S403, it is verified whether or not the face region. That is, for the group determined to be an eye, a rectangular area centering on the two is set (FIG. 5B). Then, the edge near the boundary of the region and the color information in the region are determined. It is determined whether the edge is a face outline. Further, the color information determines whether or not the average value of the RGB values in the rectangular area falls within a preset skin color area ((c) in FIG. 5). In this way, the face area is determined by detecting the position and size of the face. The face area is given by the rectangular area (rectangle parallel to the horizontal / vertical direction).

  In step S404, area information of the determined face area is created. In this embodiment, the area information is a set of position data of the vertices of the circumscribed rectangle of the detected face area, but may be position data of the center of both eyes. Moreover, you may approximate by a polygon, an ellipse, etc.

(Face region discrimination processing: S303)
Hereinafter, an example of the face area determination process in step S303 of FIG. 3 will be described in detail.

  Details of the face area determination processing will be described with reference to the flowchart of FIG.

  First, in step S601, the face area is affine transformed into an area of 48 pixels horizontally and 64 pixels vertically so that the center of the left and right eyes is located in the horizontal direction at the 16th pixel from the left and right edges and the 24th pixel from the top. To do. Also, normalization is performed with respect to the luminance distribution and the face orientation.

  Next, feature amounts are extracted in step S602. In a simple method, the luminance value of each pixel may be used. In this case, the feature amount is 48 × 64 = 3072 dimensions. This is projected onto a partial space specified by a basis vector obtained by performing principal component analysis in advance, and the number of dimensions is reduced by ignoring the axis with little variance, thereby obtaining a feature amount.

  In step S603, the Euclidean distance from the representative feature amount in each type is obtained on the projected subspace. The partial space is specified by performing a discriminant analysis of the feature amount to obtain a base vector in advance. In the present embodiment, since an alternative set of discriminant analysis is performed, if there are two types of discrimination, such as men and women, this basis vector may be one. In the case of classification into three or more types, a discrimination analysis is performed to determine whether or not each type is included, a base vector is obtained in advance for each type, and a distance is obtained for each type. If implementation that emphasizes speed over accuracy is necessary, three or more or one kind of basis vector may be used.

  In step S604, the distance is normalized to the likelihood. This conversion function is determined from a distance from a representative feature amount obtained using a large number of samples in advance and a probability distribution as to whether or not it is actually included in a certain type.

  Depending on how the samples are divided when obtaining basis vectors for discriminant analysis, classifications regarding gender, age, and biological characteristics are possible. In addition, in this embodiment, the type of discrimination is performed for three items, but is not limited to this. The discriminant analysis may be performed by defining the desired classification and classifying the learning sample into the defined classes.

(Image correction method planning process: S304)
An example of the image correction method planning process in step S304 in FIG. 3 will be described in detail below. 7 to 11 show the operation of the image correction method planning process. Here, with respect to each face area detected from the above-described image, each skin correction process is performed on the identified area type having a high likelihood.

  Details of the image correction method planning process will be described with reference to the flowchart of FIG.

  When the image correction method planning process is started, one unprocessed face area information detected in step S301 in FIG. 3 is acquired in step S701. In step S702, one candidate of the unprocessed face type determined in step S303 in FIG. In step S703, it is determined whether the likelihood is greater than or equal to a predetermined threshold value. If the likelihood is less than the threshold value, the process proceeds to step S711, and the region is invalid and is not subject to correction. If it is equal to or greater than the threshold, the process proceeds to step S704.

  In step S704, the input image represented by each value in the RGB color space is converted into values of lightness L, saturation C, and hue H. This color space called LCH is calculated from the CIEL * a * b * color space by the following equation (1).

This LCH is an example of a color space that expresses a color in terms of brightness, saturation, and hue. In addition, other color spaces such as YCbCr, L * u * v *, HSV, and HSL may be used.
(Area representative color extraction processing: S705)
In step S705, a representative color in each region is obtained for pixels in the human skin region represented by saturation C and hue H.

  The process of step S705 will be described in detail with reference to the flowchart of FIG.

  First, in step S801, target face image area data is acquired. Steps S802 to S805 constitute a processing loop for each pixel in this region.

In step S802, an unprocessed pixel included in the image area currently focused on is determined. Next, in step S803, it is determined whether the brightness L of the pixel is equal to or greater than a certain value L _TH . If eyes, eyebrows, eyelids, or shadows are included in the face area extracted in step S301, the representative color of the color to be corrected extracted from the human skin area may not be obtained appropriately. Since the shadow in the human skin area or the area such as eyes, eyebrows, and eyelids is considered to have a lightness lower than a predetermined color such as a skin color to be corrected, the shadow in the human skin area is obtained by using the condition Alternatively, pixels in the area such as eyes, eyebrows, and eyelids can be excluded. For the pixel that satisfies the condition in step S803, the color data of the saturation C and hue H of the pixel are totaled in step S804. This aggregation method varies depending on what is used as the representative color to be corrected. For example, when the representative color to be corrected is the average value of the pixels in the human skin region, Mean simple addition of degree C and hue H. At the same time, the number of pixels is counted. The counting process in steps S802 to S805 is repeated until it is determined in step S805 that scanning has been completed for all pixels.

If it is determined that scanning has been completed for all the pixels of the input image, an area representative color of the color to be corrected is calculated for the image area currently focused on in step S806. When the representative color to be corrected is expressed by the average value of the saturation and hue of the pixels in the human skin region, in step S806, a value ( _CR (k, l), H _R (k, l)) is set as the region representative color. Here, k is the index number of the face area, and l is the index number of the type candidate in the discrimination result of each face area. Using the region representative color thus obtained, a correction coefficient for color correction is set in the following step S707.

  Returning to FIG. 7, in step S <b> 706, the correction target color set in advance for each type of correction target is acquired with reference to the correction target value table 202 a.

  This corrected target value table 202a has a skin color target color (in this example, hue and color) for 2 × 6 × 6 = 72 types of 6 stages of age, 6 types of biological classification of mankind, and 2 types of gender. Saturation) is defined, for example, if an Eurasian 19-24 year old woman has a target hue of “0.7”, a saturation of “27”, an East Eurasian 19-24 year old For a male, the target hue is defined as “0.75”, the saturation as “44”, and the like. This target color is determined by performing a subjective evaluation experiment of a preferable image correction.

(Correction coefficient setting process: S707)
In step S707, a correction coefficient for controlling the correction amount for each area is obtained using the area representative color extracted in the previous step S705 and the correction target color acquired in step S706. In the correction coefficient setting process, the correction coefficient is a predetermined coefficient for controlling the color correction amount according to the difference between the region representative color and the target color. Since a correction amount in color correction described later is determined for each hue and saturation, a correction coefficient is also set for each hue and saturation. In this embodiment, the color complementation coefficient and the saturation correction coefficient are set so that the target color decreases as the difference between the representative colors increases. Further, the saturation correction coefficient is set so that the correction amount is smaller when the saturation is corrected so as to decrease the saturation than when the saturation is corrected so as to increase the saturation.

  The correction coefficient setting process in step S707 will be described in more detail below.

The hue correction coefficient P _H (k, l) of the k, l-th (0 ≦ k, l <N) region is a region representative for the hue H _T of the correction target color and the l-th discrimination candidate of the k-th region. The difference H _dist (k, l) between the hues H _R k, l) of the colors is set so as to decrease as it increases. As an example of a method for setting the hue correction coefficient, a method using an expression such as Expression (2) is shown. However, K ₁ and t ₁ in Equation (2) are constants.

That is, the closer the representative hue of the color to be corrected is to the correction target hue, the closer the hue of the representative color is closer to the hue of the correction target color, and the representative hue of the color to be corrected is far away from the correction target hue. If so, the correction coefficient is set so as not to make much correction. The reason for setting the correction coefficient in this way is that when the representative hue of the color to be corrected is far away from the correction target hue, the correction is performed so that the representative color is closer to the correction target color. Is unbalanced.
Similarly, the saturation correction coefficient l-th determination candidates in k-th region P _C (k, l) is the region of l-th determination candidates in the k-th region and the chroma value C _T of the correction target color The difference C _dist (k, l) between the saturation values C _R (k, l) of the representative colors is set so as to decrease as it increases. In other words, correction is performed so that the saturation of the representative color is closer to the saturation of the correction target color as the representative saturation of the color to be corrected is closer to the correction target saturation, and the representative saturation of the color to be corrected is corrected. When it is far from the target saturation, the correction coefficient is set so as not to make much correction. However, the method of setting the saturation correction coefficient is different depending on whether the correction is performed so as to decrease the saturation or the correction is performed so as to increase the saturation.

When correcting to reduce saturation, it is desirable to set a correction coefficient so as to prevent the phenomenon that the impression of the image becomes plain by correcting, and conversely when correcting to increase saturation This is because, even if the representative saturation of the color to be corrected is far from the correction target saturation, the effect of improving the impression of the image can be expected by performing correction so as to approach the target saturation. As an example of a method for setting such a saturation correction coefficient, a method using a setting equation such as equation (3) in the case of correction for reducing saturation and a method using equation (4) in the case of correction for increasing saturation is shown. It is. It is assumed that a ₁ and b _{1 in} equation (3) and K ₂ and t ₂ in equation (4) are constants.

FIG. 9 shows a graph representing the expressions (3) and (4). In FIG. 9, the horizontal axis represents the difference C _dist (k, l) between the saturation value C _T of the correction target color and the saturation value C _R (k, l) of the representative color, and the vertical axis represents the saturation correction coefficient P. _C (k, l) is shown.
Even when the difference between the correction target saturation and the saturation of the representative color is equal, the saturation correction coefficient in the correction for decreasing the saturation is different from the saturation correction coefficient in the correction for increasing the saturation, and the correction for decreasing the saturation. It can be seen from FIG. 9 that the correction coefficient at is smaller. This indicates that when the correction is made to decrease the saturation, the correction amount is made smaller than when the correction is made to increase the saturation.

  As described above, the hue and saturation correction coefficients calculated in step S707 are used as coefficients for controlling the color correction amount used in the correction process (step S710) described later.

(Color area setting processing: S708)
Returning to FIG. 7, in step S <b> 708, color area setting processing is performed to obtain a color area for each area to be subjected to correction processing including area representative colors. Here, attention is paid to the hue of the pixels in the human skin region.

  In the present embodiment, a color area including a representative color to be corrected is set, and color correction is performed for the area. This is because, for example, if correction is performed to make the skin color a preferable skin color for the entire area of the input image, there is a problem that the colors of other areas in the image that were originally preferable colors become unnatural. It is. The color area to be corrected set in the correction coefficient processing in step S707 is indicated by a specific hue, and color area weights are set according to the correction range indicated by the specific hue and the representative color of the color to be corrected, which will be described later. Used for color correction processing.

  The processing in step S708 will be described in detail using the flowchart in FIG.

  First, in step S1001, face area data belonging to the k-th image area extracted in step S707 of FIG. 7 is acquired.

  Steps S1002 to S1006 constitute a processing loop for each pixel in this region.

In step S1002, an unprocessed pixel included in the currently focused image area is determined. Next, in step S1003, lightness L of the pixel to determine whether there are L _TH than to indicate any brightness. If eyes, eyebrows, eyelids, or shadows are included in the human skin area extracted in step S707 of FIG. 7, the representative color of the correction target extracted from the human skin area cannot be obtained appropriately. There is a case. Since the shadow in the human skin area or the area such as eyes, eyebrows, and eyelids is considered to have a lightness lower than a predetermined color such as a skin color to be corrected, the shadow in the human skin area can be obtained by using the condition. Alternatively, pixels in the area such as the eyes, eyebrows, and eyelids can be excluded.

Further, in step S1004, the chroma C of the pixel is determined whether a predetermined value or more C _T H indicating the saturation of the achromatic. At this time, the fact that the saturation of the pixel is equal to or less than the above-mentioned value C _TH means that the pixel is close to an achromatic color. Here, the reason for excluding the pixels in the achromatic color region is that in the achromatic color region, the hue changes greatly due to a slight difference in RGB values, and the hue region corresponding to the human skin region cannot be obtained with high accuracy. In the color area setting process in step S708, the color area is set with the hue of a specific color, paying attention to the hue of the pixel in the human skin area.

For the pixel that satisfies the condition of step S1004, the value of the hue H of the pixel is tabulated in step S1005. This aggregation method varies depending on what is used as a specific hue indicating a color area. For example, the hue distribution of pixels that exist in the human skin area and satisfy the conditions of steps S1003 and S1004 is analyzed, and the maximum A method of indicating a color area to be corrected by the hue H _max (k, l) and the minimum hue H _min (k, l) is exemplified. The counting process is repeated until it is determined in step S1006 that scanning has been completed for all pixels.

If it is determined that scanning has been completed for all pixels of the input image, a color region to be corrected is set in step S1007. When the maximum hue H _max (k, l) and the minimum hue H _min (k, l) indicate the color area to be corrected, in step S1007, the maximum hue H _max (k , L) and the minimum hue H _min (k, l), the color area to be corrected is set from the difference between the color to be corrected and the representative hue (formula (5)). Here, H _R (k, l) in the equation (5) is the region representative hue of the color to be corrected of the l th discrimination candidate in the k th region, and ΔH (k, l) is the correction target. The hue range of the color area in the kth image area is shown.

Further, the present invention is not limited to the color area setting process shown here, and for example, a method for manually setting an arbitrary specific hue may be used. In this case, it is desirable to set the color area to be corrected as a hue range that includes the human skin area extracted in step S301 in FIG.
For the color area set by the method as described above, a color area weight to be used in color correction processing described later is determined for each hue range and saturation range. The hue range ΔH indicating the color area to be corrected so that the weight W _H (k, l) in the hue H of the pixel to be corrected is 1 for the representative color to be corrected and 0 outside the correction target area. Based on (k, l) and the hue H _R (k, l) of the representative color of the color to be corrected, it is determined as follows (Formula 6).

However, the method of setting the weight of the hue area to be corrected is not limited to the method represented by the equation (6), and for example, the hue range ΔH (k, l) indicating the color area to be corrected is used. Further, a method of setting the color to be changed nonlinearly using the representative color H _R (k, l) of the color to be corrected and an arbitrary constant is exemplified.
As described above, the reason why the color region to be corrected is weighted in step S1008 is to prevent the color continuity from breaking inside and outside the hue region to be corrected when performing color interpolation described later. It is done.

Further, the weight W _C (k, l) in the saturation C of the pixel to be corrected is represented by the correction target target saturation CT and a predetermined value C _TH indicating the saturation of the achromatic color used in step S1004. (Expression 7).

However, the method of setting the weight of the saturation region to be corrected is not limited to the method represented by the equation (7), and for example, the saturation C _R of the representative color of the color to be corrected and the person A saturation range ΔC (k, l) that includes the skin region is obtained by the same method as the hue region setting method, and the weight of the saturation region to be corrected is set.
Returning to FIG. 7, next, in step S <b> 709, image area weights used for color correction processing described later are set from the position information of the human skin area in the input image.

  In the present embodiment, in a color correction process described later, color correction is performed only for a predetermined color such as a skin color using a weight for the specific color area set in step S708, so that an originally preferable color is obtained. This avoids the phenomenon of unnatural colors in other areas of the image. However, when only the color area is set as the correction target, for example, in the corrected input image, if a pixel other than the human skin included in the color area set in step S708 exists outside the human skin area, the pixel is color-interpolated. It becomes the target of. As a method for solving such a problem, the image area weight set in step S709 is used.

Wherein the image region weight, the skin and the color correction target pixels in the region, the human skin is not included in the region pixel color correction excluded to become as 0 above, the value of 1 or less W _P (k, l, x, y). At this time, it is desirable to set the image area weight so as to prevent color continuity from breaking near the boundary of the human skin area by a color correction process described later.
The image area weight setting method is not limited, and examples include a method of obtaining a centroid in the human skin area and assigning a weight according to the distance of each pixel from the centroid. When this setting method is used, in the color correction processing described later, the correction amount of pixels near the boundary of the human skin region is smaller than that of the pixel near the center of the human skin region, and the color near the boundary of the human skin region Can be prevented from failing. In addition, there is a method in which the human skin area is subjected to low-pass processing using a Gaussian filter or the like, and the vicinity of the boundary of the human skin area is blurred, thereby relieving the failure of color continuity inside and outside the human skin area.

(Correction process: S710)
Next, in step S710, image correction is performed according to each area correction information obtained previously. This process will be described in detail with reference to the flowchart of FIG.

  First, in step S1101, face area data belonging to the kth image area extracted in step S707 is acquired.

  Steps S1102 to S1106 constitute a processing loop for each pixel in this region.

In step S1103, hue correction processing is performed. First, the hue correction amount Dh (j) for the representative color to be corrected is calculated. The hue correction amount in the representative color of the color to be corrected is the difference between the target hue H _T to be corrected and the representative hue H _R (j) of the color to be corrected, and the hue correction set in the correction coefficient setting process in step S707. It is expressed by the product of the coefficient P _H (j) (formula (9)). Next, the hue correction amount Dh is set in the hue region weight W _H (j) and saturation region weight W _C (j) set in the color region setting process in step S9 and in the image region setting process in step S709. The hue of the pixel in the input image is corrected by the amount multiplied by the image area weight W _P (j) (formula (8)). In Equation (8), H represents the hue of the pixel to be corrected before the color correction process, and H ′ represents the hue of the pixel to be corrected after the color correction process.

Similarly, in step S1104, the saturation correction factor Kc (j) for the representative color is calculated from the saturation correction factor Pc (j) obtained in the correction factor setting process in step S707, and the saturation correction factor Kc is set in step S1104. The hue area weight W _H (j), the saturation area weight W _C (j) set in the color area setting process of S708, the image area weight W _P (j) set in the image area setting process of step S709, and the above-mentioned The saturation of the pixels in the input image is corrected by the amount corresponding to the saturation C of the correction target pixel before the color correction processing (Formula (11)).
The saturation correction rate Kc is expressed by the ratio of the difference between the corrected target saturation C _T and the correction target representative saturation C _R (j) and the corrected representative saturation C _R (j) as shown in Expression (12). It is represented by the product of the corrected saturation coefficient Pc (j). In Equation (11), C ′ represents the saturation of the correction target pixel after the color correction process.

Next, it is determined whether or not the above process has been completed for all the pixels based on the determination in step S1105. If completed, the correction process is terminated. If not, the process returns to step S1102.
When the likelihood of determining the type of correction target is low, a new target color is synthesized based on the likelihood based on the likelihood from a plurality of defined correction target colors, and the determination is made using this combined target color. You may make it a candidate.

As an example of a target color synthesis method, the female target saturation is C _F male target saturation C _M , the femininity likelihood is T, the gender determination likelihood is T _L , If the likelihood that there is no determination reliability is T _I (T _I <T _L ), the target saturation C _T is expressed by the following equation (14).

FIG. 12 is a graph showing an example of the relationship between the femininity likelihood T of discrimination between male and female on the horizontal axis and the target saturation (C _T ), which is a part of the target color, on the vertical axis. When the likelihood of femininity is T, the masculinity is calculated as (1-T).
In the section of 1−T _L ≦ T ≦ 1−T _I in FIG. 12, two correction proposals using target colors synthesized for men are presented as options. Further, in the section of T _I ≦ T ≦ T _L , two correction plans using the target color combined with those for women are processed after step S304 in FIG. 3 and presented as options in step S306. In the case where there are two correction proposals as described above, it is desirable that the combined target color is a second correction proposal candidate. Further, in the section of 1−T _I <T <T _I , a correction plan is created using only the synthesized target color and presented as an option in step S306.

  As a result, even when it is difficult to discriminate the region, it is possible to compensate for appropriate correction processing as a face region. The correction processing performed here is performed for each region type candidate, but is for confirming the correction effect on the display unit 209, and does not require a high resolution like an image to be printed. For this reason, an image with a reduced resolution may be used as a target. Further, in the present embodiment, for the sake of explanation, it is configured that the totalization processing is performed for all combinations for each area determination in each loop in the loop of steps S701 to S713, but drawing on the display unit 209 is required according to a user instruction Only the correction method that has become may be processed after the user's instruction.

(Correction method selection process: S306)
Hereinafter, an example of the correction method selection in step S306 in FIG. 3 will be described in detail.

  FIG. 12A shows an example of a selection window displayed on the display unit 209 in this process. 1201 is an example of an image to be corrected. Reference numerals 1202 and 1203 denote area cursors indicating the face areas detected in step S301. As a result of discriminating each face area, skin color correction processing is performed according to the type of correction method having the highest likelihood. Here, when one of the area cursors is clicked by the input unit 208, the area cursor becomes a focused state, the circumscribed polygon of the area to be corrected is displayed, and the area where the correction effect is exerted can be confirmed.

  Further, when the same area is double-clicked, a dialog window 1204 as shown in FIG. 12B is displayed in order to present the current correction method in this area and change the correction method. Each part of this window will be described.

  Reference numeral 1205 denotes the area determination content, and 1206 denotes a button for changing the area determination content. When this area is clicked by the input device 208, a list of other area types whose likelihood is higher than a predetermined value is displayed. The desired area type can be selected. 1207 displays and sets the degree of effect. The left end is in a state where no correction is applied. The squares in the middle black are the current settings. It can be set in two levels of strength, centering on the correction amount in the region determination result with the highest likelihood. By clicking on a desired location, the degree of effect can be set. Reference numeral 1208 denotes an OK button, which updates the correction method according to the set contents, closes the dialog, and applies correction to the 1201 image. A cancel button 1209 discards the set contents and closes the window. By clicking an OK button 1210 arranged in the background area of the image 1201, the processing in step S306 is terminated.

<Another Specific Configuration Example of the Image Processing Apparatus of the Present Embodiment>
FIG. 14 is a block diagram illustrating another specific configuration example of the image processing apparatus according to the present embodiment. The same number is given to the part which is common in the said embodiment.

  The block configuration is different from the above embodiment in that a CD / DVD drive 1101 is connected to the drive interface 214 and an optical disc 215 such as a DVD or CD in which the program of this embodiment is recorded is added.

  When the optical disc 215 in which the program of the present embodiment is recorded is inserted into the CD / DVD drive, the CPU 201 reads the program from the recording medium and develops it in the RAM 203, thereby realizing the same processing as in the above embodiment.

  In the present embodiment, the skin color correction process has been described in detail. However, as another correction method, a Gaussian filter may be applied to perform blurring, an edge enhancement filter, or the like. By applying the Gaussian filter to the skin color area around the neck, cheeks and eyes, the apparent age can be made younger. In this case, the degree of effect is defined for each type of face, and image area weight setting similar to that in step S709 is appropriately performed, and an image on which a Gaussian filter is applied according to the weight for each pixel in the area. A weighted average may be obtained for each LHC channel of the original image. Such spatial correction is particularly effective for low resolution images such as moving images and highly portable cameras.

<Other embodiments>
When discriminating the face area, the feature extraction of the face is not limited to the method shown in the first embodiment. For example, a face identification feature descriptor defined in ISO / IEC 15938-3 may be used. Further, as a discriminator, in addition to discriminant analysis, a boost method, a support vector machine combined with a kernel trick, a neural network, or the like may be used. In any case, by preparing a training image for each area type and performing training, it is possible to determine the desired area type. In the present embodiment, the case where the skin color of a person is preferably corrected has been described. However, it is needless to say that the present invention can also be applied to a case where other memory colors such as sky blue and green of plants are preferably corrected.

  Further, as a storage medium for supplying the program code of this embodiment, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD, a DVD, a magnetic tape, a nonvolatile memory card, a ROM, or the like is used. it can.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  Furthermore, in an image input device such as a camera or scanner, an image output device such as a printer, or a device in which these are combined or connected, a CPU or the like provided in both or any of the devices performs part or all of the actual processing. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

It is a block diagram which shows the structural example of the image processing apparatus of this embodiment. It is a figure which shows the hardware structural example of this embodiment. It is a flowchart which shows the outline | summary of the operation | movement procedure of this embodiment. It is a flowchart which shows the example of a procedure of the face area | region detection process (S301) of FIG. It is a figure explaining the process example of the face area | region detection process of FIG. It is a flowchart which shows the example of a procedure of the face area determination process (S303) of FIG. It is a flowchart which shows the example of a procedure of the image correction method planning process (S304) of FIG. It is a flowchart which shows the example of a procedure of the area | region representative color extraction process (S705) of FIG. It is a figure explaining the process example of the area | region representative color extraction process of FIG. FIG. 8 is a flowchart illustrating an exemplary procedure of color area setting processing (S708) in FIG. 7. FIG. It is a flowchart which shows the example of a procedure of the correction process (S710) of FIG. It is a figure explaining the addition and the synthesis | combination method of the target color in this embodiment. It is a figure which shows an example of the screen display in the correction method selection of FIG. It is a figure which shows the other hardware structural example of the image processing apparatus of this embodiment.

Claims (10)

Image input means for inputting image information;
An area setting means for detecting whether or not a preset object is included in the image information input from the image input means, and setting the detected area;
For each area set by the area setting means, an area determination means for determining the type of the area, and outputting the probability of determination together with the type of the area;
Based on the type is determined by the area discriminating means, have a image correcting means for performing image correction for each area set,
The image correcting means includes
Image correction method planning means for planning image correction method candidates when the probability of determination output by the region determination means in any region set by the region setting means is less than a predetermined threshold;
A correction method for the region to be corrected, and a correction method instruction means for seeking an operator's instruction;
An image processing apparatus, wherein an image is corrected in accordance with an instruction content by the correction method instruction means .   The image processing apparatus according to claim 1, wherein the preset target is a human face.   The image processing apparatus according to claim 2, wherein the type of the region is any one or a combination of gender and age of a person's face in the region, or a biological classification of an individual or a human being.   2. The image correction according to claim 1, wherein the image correction is correction of any one or a combination of lightness, hue, saturation, sharpness, and blurring for pixels included in the specific color gamut in the specific space region. Image processing device. The correcting method instruction means displays the image to be corrected, according to any one of claims 1 to 4, characterized in that placing the operation button for instructing the correction method in the vicinity of the correction target region Image processing apparatus. The correcting method instruction means displays the image to be corrected, the image processing apparatus according to any one of claims 1 to 5 area to be corrected and displaying the identifiable circumscribed polygon. The correcting method instruction unit sets the depth of the correction, the image processing apparatus according to any one of claims 1 to 6, characterized in that is reflected to the correction object image being displayed according to the settings. An image processing method in an image processing apparatus,
An image input step in which the image input means inputs image information;
An area setting step for detecting whether or not the preset object is included in the image information input in the image input process, and setting the detected area;
An area determination unit that determines an area type for each area set in the area setting step, and outputs an accuracy of determination together with the type of the area; and
Image correction means, based on the type discriminated by the region discrimination step, have a image correction step of performing image correction for each area set,
The image correction step includes
An image correction method for planning an image correction method candidate when the image correction method planning means does not satisfy a predetermined threshold in any of the regions set in the region setting step. A method planning process;
A correction method instructing means that presents a correction method for a region to be corrected and requests an operator's instruction;
An image processing method, wherein an image is corrected in accordance with an instruction content in the correction method instruction step . Program for executing the respective steps of the image processing method according to claim 8, wherein the computer. Computer readable storage medium storing a program according to claim 9.

Images