JP2010102426A - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically perform level calculation of backlight and follow light of a photographic image, and to perform image correction according to the calculation result. <P>SOLUTION: This image processing apparatus is provided with: an object detection means 1 which detects an object in an image from an input image, and searches region information showing in which region of the image the object is present; a featured value extraction means 2 which calculates a featured value based on each luminance or color among the region of the object and a partial or all background region other than the region of the object and the whole region of the image on the basis of the image and the region information from the object detection means 1; a backlight/follow light level calculation means 3 which calculates the levels of the backlight and follow light of the image from the featured value of the region of the object and the featured value of the background region or the whole region from the featured value extraction means 2; and an image correction means 4 which corrects the image on the basis of the region information from the object detection means 1 and the featured value of the whole image from the featured value extraction means 2 and the levels of the backlight and follow light from the backlight/follow light level calculation means 3. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method that determine the normal light and backlight conditions of an image and perform appropriate image correction.

  In recent years, with the widespread use of digital cameras, there is an increasing demand for taking pictures with digital cameras and printing the taken images. However, because digital cameras are so easy, even users who do not have enough knowledge about cameras can easily handle them. There are many cases where the scene is not only an image photographed under suitable conditions but also a backlight or a high-contrast scene. However, even if an image was taken under unfavorable conditions, if the contents are important to the photographer, it cannot be discarded, and appropriate image correction is performed, and a suitable state is stored or printed. It is hoped that.

  Therefore, in a conventional image processing apparatus, image state determination method, and storage medium, an image that is a backlight image is input in advance, the degree of backlight of the image is determined based on a histogram calculated from the image, and the backlight Image correction is performed according to the degree. (For example, refer to Patent Document 1).

JP 2002-92607 A (pages 3 to 4, FIGS. 1 to 3)

  In conventional image processing apparatuses, image state determination methods, and storage media, an image that is to be a backlight image is subjected to image correction by determining the degree of backlight, so whether or not it is a backlight image must be known in advance. In addition, since image correction is often performed on a large number of captured images, there is a problem that automatic processing and high-speed processing are difficult.

  The present invention has been made to solve the above-described problems, and provides an image processing apparatus and an image processing method for automatically calculating a back light level and a forward light level of a captured image, and performing image correction based on the calculation. To get.

  An image processing apparatus according to the present invention includes: an object detection unit that detects an object in an image from an input image and obtains region information indicating in which region the object is in the image; and an image and an object detection unit Feature amount extraction means for calculating a feature amount based on the brightness or color of each of the object region, a part or all of the background region other than the object region, and the entire region of the image based on the region information The backlight follow-up light level calculating means for calculating the backlight level and the follow light level of the image from the feature quantity of the object area from the feature quantity extracting means and the feature quantity of the background area or the entire area, and the object detection means An image for correcting the image based on the region information, the overall feature amount of the image from the feature amount extraction means, and the backlight and follow light levels from the backlight follow light level calculation means. In which and a correcting means.

  According to the present invention, an object in an input image is detected, a feature amount is calculated based on the object area information, and the image correction is performed by calculating the backlight and follow light levels of the image. An image processing apparatus capable of high-speed processing is obtained.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing an image processing apparatus according to Embodiment 1 of the present invention. Here, a case where an object in the image area is a human face will be described. The object detection means 1 detects an object (in this case, a human face) in the image by pattern matching or the like from the input image, and object area information (in this case) indicating in which area the object is in the image Obtain face area information). The feature amount extraction unit 2 calculates a statistic amount of each pixel value as a feature amount by dividing into a face region and other regions based on the face region information from the object detection unit 1, and also calculates the feature amount of the entire image. calculate. The backlight follow light level calculation unit 3 calculates the backlight and follow light levels based on the feature amount of the image calculated by the feature amount extraction unit. The image correction unit 4 includes the object area information output from the object detection unit 1, the feature amount of the entire image output from the feature amount extraction unit 2, and the backlight follow light level output from the backlight follow light level calculation unit 3. Based on the calculation result, image correction suitable for each image is performed.

  The operation of the image processing apparatus configured as described above will be described with reference to the flowchart of FIG. First, in the object detection step S101, the object detection means 1 detects a human face area in the image from the input image. Specifically, detection is performed using various known methods such as feature extraction, skin color detection, pattern and template matching as described in JP-T-2004-527863.

An example of face area information detected by the object detection means 1 is shown in FIG. In FIG.
The object detection means 1 detects a face area 12 and a background area 13 (indicated with dots in the figure) indicating areas other than the face area 12 from the entire photographic image 11. In the present embodiment, it is assumed that the face area 12 is displayed as a rectangle as shown in the figure. A face coordinate (x, y) 14 indicates a pixel position at the upper left of the face area 12 detected by the object detection unit 1 on the screen. Further, the width 15 is the horizontal length of the face area, and the height 16 is the vertical length of the face area, each indicated by the number of pixels. As described above, the face area information output from the object detection unit 1 in the present embodiment is the face coordinates (x, y) 14, the width 15, and the height 16, and when there are a plurality of faces, These pieces of information are detected for each face.

  Next, in the feature quantity extraction step S102, based on the face area information detected by the object detection means 1, the feature quantity extraction means 2 divides it into a face area and a background area other than the face, and the statistics of the pixel values of the respective areas. The amount is calculated as a feature amount. In addition, the entire inputted photographic image 11 is also calculated as a feature amount together with a statistic of pixel values of this region as one region. Specifically, as a statistic of pixel values, a luminance histogram and average luminance for each region are calculated as feature amounts. Note that the luminance here is 0 to 255 gradations.

  Next, in the backlight follow-up light level calculation step S103, in the feature quantity extraction means 2, the backlight follow-up light level calculation means 3 calculates the backlight and follow light levels based on the feature values of the average brightness calculated for each region. . Specifically, the backlight intensity level is calculated for each area by calculating the average brightness for each area and calculating the difference in average brightness between the areas as the backlight intensity value g. Hereinafter, in the present embodiment, a method for calculating the backlight follow-up light level when there is one person's face in the input photo image will be described.

  When the number of faces is one, the backlight follow light value g is determined by the difference between the average brightness face_ave of one face area and the average brightness back_ave of a background area other than the face area, for example, as shown in Equation (1). The average luminance of each region is calculated by dividing the sum of the luminance values of the pixels included in each region by the number of pixels included in each region.

  Details of the backlight follow light level calculation method will be described. First, when the backlight follow light value g is positive, that is, when the average brightness of the face area is higher than the average brightness of the background area other than the face, it is determined as follow light. On the other hand, when the backlight follow-up light value g is 0 or less, that is, when the average brightness of the face area is the same or lower than the average brightness of the background area other than the face, it is determined that the backlight is back. FIG. 4 shows an example of setting conditions for setting a backlight follow-up light level indicating the degree of backlight using the backlight follow-up light value g.

  As described above, the backlight follow-up light level calculation unit 3 sets the backlight follow-up light level based on the backlight follow-up light value g, and outputs the result to the image correction unit 4 as a backlight follow-up light level calculation result.

  In the image correction step S104, the image correction means 4 is the object area information (in this case, face area information) from the object detection means 1, the feature quantity of the entire image from the feature quantity extraction means 2, and the backlight follow light level calculation means. Based on the result of backlight backlight level calculation from 3, image correction suitable for each image is performed.

  Here, several image correction methods performed in the image correction unit 4 will be described. First, as a first image correction method, for example, “Fundamentals of Digital image Processing” by Prince-Hall International, Inc., written by ANIL K. JAIN. There is gradation correction using a histogram equalization method as described in Publishing, 1989, pp. 241-244.

  This is because, for example, the luminance value of each pixel is calculated over the entire screen of the input image, and the pixels having the same gradation value or the pixels included in the same section divided into sections smaller than the quantization number of gradations Is created, a single mapping table is created for tone conversion so that the shape of the created histogram (appearance frequency distribution) is optimized, and using this mapping table, Tone correction processing is performed over the entire screen. By using this histogram equalization method, the tone characteristics of the entire screen can be improved by optimizing the tone distribution when the tone in the screen is biased to a certain level. However, since only one type of mapping table is created for the entire screen, if the gradation width of a certain gradation level range is widened, the gradation width of another gradation level range is narrowed instead. In some cases, the level range may be crushed. Hereinafter, this correction is referred to as a contrast correction method.

  Specifically, this method creates a tone correction γ curve (mapping curve) by a histogram equalization method, and performs image quality adjustment based on the γ curve. FIG. 5 shows a procedure for creating a γ curve by the histogram equalization method. First, the horizontal axis represents the gradation (in this case, the gradation from 0 to 255), the vertical axis represents the frequency, and a histogram of the luminance signal of the entire screen is created (FIG. 5 (a)). Normalization is performed so that the total sum of the histograms becomes 255 (FIG. 5B). Subsequently, a cumulative histogram is created from this figure (FIG. 5C), and a representative value 21 of the cumulative histogram is calculated. Finally, these representative values are connected smoothly to create a contrast correction γ curve 22 (FIG. 5D). FIG. 5E shows a luminance histogram when the entire image is corrected using the contrast correction γ curve 22. It can be seen that the histogram is equalized.

  Next, as a second image correction method, there is local gradation correction, which is an improvement of the above-described histogram equalization method in which gradation correction is performed using one type of mapping table for the entire screen (see, for example, Japanese Patent No. 4011073). ). This is because by creating a mapping table for each local area of the screen according to the average brightness of the local area, it is possible to widen the necessary gradation width in each area, and to change the gradation of other areas Therefore, it is possible to suppress the adverse effect of the histogram equalization method in which the gradation width of other regions is narrowed by increasing the gradation width of the local region. In addition, according to the local gradation correction, an image in which both the area that is crushed in black and the area that is brightly overexposed exist as in the case where the target image is captured in a backlight state. Even in this case, more effective gradation correction can be performed for both regions. Hereinafter, this correction is referred to as a backlight correction method.

  Specifically, this method changes the γ curve used for gradation correction according to the backlight correction parameter. A method for determining the γ curve will be described with reference to FIG. In the figure, the horizontal axis indicates input of 255 gradations, and the vertical axis indicates output of 255 gradations. There are two types of γ curves, one for low luminance and one for high luminance, and the low luminance backlight correction γ curves 31 and 32 have a large inclination in the low luminance region and improve black crushing. The high-brightness backlight correction γ curves 36 and 37 have a large inclination in the high-brightness region and improve whiteout. The first low-brightness backlight correction γ curve 31 and the first high-brightness backlight correction γ curve 37 having the largest change in inclination are fixed regardless of the image, and the second low-brightness backlight correction γ of each image is fixed. A curve 32 and a second high-brightness backlight correction γ curve 36 are generated according to the backlight correction parameter of the image. The γ curve is a curve that approaches a fixed γ curve as the backlight correction parameter increases, and approaches a straight line 34 having a slope of 1 as the backlight correction parameter decreases.

Subsequently, gradation correction is performed for each local region. The correction amount of gradation is changed based on the average luminance of the local area. The gradation correction method will be described with reference to FIGS. When the average luminance of the local area is less than 128, correction is performed using the low luminance backlight correction γ curve, and when it is 128 or more, correction is performed using the high luminance backlight correction γ curve. The output value Y _{out with} respect to the luminance Y _in of each pixel in the local area is the output value when using a low-luminance / high-luminance γ curve, Y _out1 , and the output value when using a straight line 34 with a slope 1. _{Is determined} as Y _out2 , and the average luminance of the local region is Y _ave, as in Expression (2-1) and Expression (2-3). Consequently, Y _out for Y _in the a ride on the curve shown by a dotted line (a third of the low-intensity for backlight correction γ curve 33 or the third high-intensity for backlight correction γ curve 35) in Fig. 7 and 8 Become. In the figure, the relationship between α and β is as shown in equations (2-2) and (2-4). As described above, correction according to the average luminance of the local region can be performed.

  Next, as a third image correction method, there is a skin beautification correction method that can apply the above-mentioned local tone correction and further detect the skin color level of a person to make the skin of the person in the image look fine and beautiful. (For example, see Japanese Patent No. 4054360).

  This beautiful skin correction method detects a skin color portion, lowers the gradation to reduce noise, and makes it inconspicuous. The beautiful skin correction process is performed by locally changing the γ curve in the same manner as the backlight correction. The skin beautification correction γ curve is a broken line composed of three straight lines as shown in FIGS. By the straight line portion having a small inclination, the tone difference in the skin color area is reduced, and the skin color is corrected to a smooth skin color from which noise has been removed.

  Note that the skin beautification correction γ curve of each image is determined in the same manner as the backlight correction γ curve based on the skin beautification correction parameter. A method for determining the γ curve will be described with reference to FIG. The first skin correction γ curve 41 having a straight line with the smallest inclination is fixed regardless of the image, and the second skin correction γ curve 42 of each image is generated according to the skin correction parameter of the image. The γ curve is a curve that approaches a fixed γ curve as the skin beautification correction parameter is larger, and approaches a straight line 34 having a slope of 1 as it is smaller.

Further, the correction is limited to a local region that is a skin color. At this time, the correction amount is changed according to the “skin degree” indicating the degree of skin color in the local region. As shown in FIG. 10, the output value Y _out after correcting the luminance Y _in of each pixel in the local region is obtained by applying the output value Y _out1 when the skin softening correction γ curve is applied and the straight line 34 with the slope 1. In this case, the output value is Y _out2 , and the closer the skin beautification is, the closer to Y _out1, and the lower the skin beautification is, the closer to Y _out2 .

  In order for the image correction means having the above-described three image correction methods to operate appropriately according to the contents of the input image, it is necessary to set the degree of correction from the outside. Here, the backlight follow light level calculation result for the human image obtained by the backlight follow light level calculation means 3, the object area information (face area information) output from the object detection means 1, and the feature amount extraction means 2 are output. A method for setting the degree of correction for the above three image correction methods will be described based on the feature amount of the entire image.

  First, the setting value of the correction degree to the backlight correction method among the image correction methods in the image correction means 4 will be described. FIG. 11 shows an example of setting values of the backlight correction method for the backlight follow light level in FIG. Since Level 1 is weak forward light, backlight correction is not performed, and hereinafter, as the degree of backlight increases from Level 2 to Level 4 in order, the correction level of the backlight correction method is set to increase gradually from weak to strong. Since Level 0 is strong forward light, weak correction is performed. In other words, a region such as a face with a tendency to whiteout is slightly darkened, and a region such as a background with a blackout tendency is slightly brightened.

  Next, the setting value of the degree of correction to the contrast correction method using the histogram equalization method among the image correction methods in the image correction unit 4 will be described. Since contrast correction is effective when the distribution of the luminance histogram is biased during backlighting, this is also set according to the backlight follow-up light level in FIG. 4 as in the case of setting the correction degree in the backlight correction method. . FIG. 12 shows an example of setting values of the contrast correction method for the backlight normal light level in FIG. Levels 0 and 1 are forward light, but the image quality is improved by correcting the gradation width of the uneven gradation level to the upper and lower gradations by slightly correcting the uneven portion of the histogram by the histogram equalization method. There is expected. In addition, with respect to Level 2 to Level 4, a larger histogram bias is expected as the degree of backlight increases, so the degree of histogram equalization is increased, and the correction degree is gradually increased from weak to slightly stronger in the direction of eliminating the bias. Set to increase.

  Next, the setting value of the correction degree to the skin beautification correction method among the image correction methods in the image correction means 4 will be described. FIG. 13 shows an example of a skin beautification parameter setting value for the backlight normal light level in FIG. Note that the skin beautification correction has an effect of making a person's skin spots, wrinkles, breakouts, etc. inconspicuous, and is effective when the face area is large to some extent. On the other hand, when the face area is small, the correction works in such a direction that the necessary face parts such as the eyes and mouth become inconspicuous. Based on this, the correction degree to the skin beautification correction method is further set according to the width 15 and the height 16 indicating the size of the face area in the face area information output from the object detection means 1.

  FIG. 14 shows an example of setting values of the skin beautification correction method for the size of the face area. The size of the face area is represented by the width 15 and the height 16 of the face area. Here, the face area has the same width 15 and height 16, that is, the square is assumed to be treated as a square area. The length of one side is set as the size of the face area (represented as size in the figure), and the correction degree to the skin beautification correction method is set according to the face area size. Further, min_size shown in the figure is, for example, 1/3 of the short side of the image, and is used as a reference for setting the skin beautification correction method. As shown in FIG. 14, when the face area size of Level 0 is smaller than min_size, since necessary portions of the face such as eyes and mouth may be inconspicuous, skin beautification correction is not performed. When the face has an intermediate size (Level 1), the correction is made in stages from weak to slightly strong according to the backlight follow light level of the parameter shown in FIG. When the face is large (Level 2), the blurred skin looks more beautiful skin, so the parameters in FIG. Set to increase.

  Although an example of setting values is shown in FIG. 12 for contrast correction, conditions to be considered in setting the correction degree of contrast correction will be described in addition to the backlight normal light level. As described above, contrast correction is correction based on the histogram equalization method. Therefore, when the face area size is small during backlighting, and the gradation to which the majority of the small face area occupies a small percentage of the entire screen, the gradation equalization method increases the gradation distribution to other levels. A value is assigned from the key, and there is a possibility that the correction will work in the direction of squashing the face area. Therefore, it is set by adjusting so as not to perform the contrast correction or to weaken the degree of correction.

  Further, when the distribution of the luminance histogram of the entire screen is very biased, it is set so that contrast correction is performed with the parameters shown in FIG. For example, when the luminance histogram of the entire screen is created by dividing it into 8 sections, the distribution of the histogram is very biased when the most frequently occurring gradation section includes 1/2 or more of the entire pixels. Therefore, the correction degree of contrast correction is set to medium to high.

  Although the luminance histogram and the average luminance are mainly extracted here as features, the histogram of the color signal may be extracted as the features, and the analysis may be performed based on this.

  Further, the method of obtaining the backlight follow-up light value g is not limited to the method shown in the equation (1), and the average luminance difference between the face area and the entire screen area, or the average between the face area and a part of the background area. Any method can be used as long as it can capture a luminance change in the screen, such as a luminance difference.

  FIG. 4 shows an example of the setting condition of the backlight follow light level based on the backlight follow light value g, FIG. 11 shows an example of the setting value of the backlight correction method, FIG. 12 shows an example of the setting value of the contrast correction method, and FIGS. An example of the setting value of the skin beautification correction method is shown, and the condition for setting the degree of contrast correction according to the distribution of the luminance histogram is shown in FIG. 15, but the setting condition and the setting method are not limited to this. The set value may be appropriately determined according to the characteristics of the image correction method.

  In addition, although the object area and the skin softening correction use a rectangular area here, the present invention is not limited to this. The object is not limited to a person's face, and any object can be used as long as it can be detected from an image such as a car or a building. At this time, if an object of particular interest is targeted in the image, a correction amount for image correction is set according to the degree of backlight of the object, so that a more effective corrected image can be obtained.

  In the image processing apparatus and the image processing method configured as described above, an object is detected from an input image by pattern, template matching, or the like, and the feature amount of the object region and the other background region or the entire image region is detected. And the image correction is performed by calculating the backlight follow light level with a simple calculation, so that an image processing apparatus capable of automatic processing and high-speed processing can be obtained.

  In addition, in the case of a method that automatically analyzes a histogram of a signal related to brightness such as luminance and determines backlight and performs image correction, complicated analysis calculation is required, but the present invention detects an object by pattern matching or the like. As a result, simple and high-speed computation is possible. In addition, by using object region information, the feature amount of the entire image, and the backlight follow-up light level calculation result that can calculate not only the backlight but also the follow-up light level, various images captured under various environments can be obtained. Image correction suitable for each image can be performed.

Embodiment 2. FIG.
In Embodiment 1, the backlight follow-up light level calculation method when there is one person's face in the input image is shown, but here, the backlight follow-up light level calculation method when there are a plurality of person faces is shown.

  FIG. 16 is a block diagram showing an image processing apparatus according to Embodiment 2 of the present invention apparatus. The backlight follow light level calculation means 3a in this figure is different from that in FIG. The backlight follow light level calculating unit 3 according to the first embodiment calculates the backlight follow light level based on the feature of the image calculated by the feature amount extracting unit 2 when the input image has one person's face. The backlight follow-up light level calculation unit 3a in FIG. 16 calculates the backlight follow-up light level based on the feature of the image calculated by the feature amount extraction unit 2 when there are a plurality of human faces in the input image.

  The operation of the image processing apparatus configured as described above will be described. The basic processing flow is the same as that in FIG. 2 of the first embodiment. First, when a plurality of objects are detected in the image, the object detection means 1 obtains a plurality of pieces of face area information shown in FIG. The face area information is composed of face coordinates (x, y) 14 indicating the upper left pixel position on the screen of the face area detected as a rectangle, and a width 15 and a height 16 indicating the size of the face area. Are detected for each face.

  Based on the face area information detected by the object detection means 1, the feature quantity extraction means 2 divides a plurality of face areas and a background area other than the face area, and uses the statistic of the pixel value of each area as a feature quantity. calculate. In addition, the entire inputted photographic image 11 is also calculated as a feature amount together with a statistic of pixel values of this region as one region. Specifically, as a statistic of pixel values, a luminance histogram and average luminance for each region are calculated as feature amounts.

  Next, the backlight follow-up light level calculation unit 3 a calculates the backlight follow-up light level based on the plurality of face area information and the feature value of the average luminance obtained for each region by the feature value extraction unit 2. Specifically, the backlight intensity level is calculated for each area by calculating the average brightness for each area and calculating the difference in average brightness between the areas as the backlight intensity value g. Hereinafter, an operation for calculating the backlight follow-up light level in the case where there are a plurality of human faces in the input photographic image by the backlight follow-up light level calculation unit 3a of the present embodiment will be described. In the present embodiment, a backlight follow light level calculation method for the case where there are three faces is shown.

  In the case of three faces, the backlight follow-up light value g is set as a reference face among the three face areas by the following four setting methods, and the average brightness face_ave of the reference face area and the face It is determined by the difference in the average luminance back_ave of the background area other than the area. As four setting methods, (A) a face having the largest face area, (B) a face having the darkest average brightness, (C) a face having a median average brightness of three faces, and (D) an average of three faces The case of luminance will be described.

  First, when the face having the largest face area (A) is used as a reference face, the backlight follow-up light value g is the average brightness face_ave (max_size) of the face having the largest face area and the background area other than the face area. Depending on the difference in average luminance back_ave, it can be determined as shown in equation (3).

  When the second face (B) having the darkest average brightness is used as the reference face, the backlight follow-up light value g is the average brightness face_ave (max_dark) of the face with the darkest face among the three face areas. It can be defined as the equation (4) by the difference of the average luminance back_ave of the background region other than the region.

  Further, when the third face (C) whose average brightness is the median face of the three faces is used as the reference face, the backlight follow-up light value g is the average brightness face_ave of the face whose average brightness is the median of the three faces. By the difference between (middle_dark) and the average luminance back_ave of the background area other than the face area, it can be determined as in equation (5).

  In addition, when the fourth (D) average brightness of three faces is used as a reference, the backlight follow light value g is a difference between the average brightness face_ave (face_all_ave) of the three faces and the average brightness back_ave of the background area other than the face area. Thus, it can be determined as shown in equation (6).

  Note that the average luminance of each region is calculated by dividing the sum of the luminance values of the pixels included in each region by the number of pixels included in each region, as shown in the expression (1) of the first embodiment. It shall be.

  As described above, the method of setting the reference face by the four setting methods and obtaining the backlight follow light value g has been shown. These four setting methods may be selected as appropriate, but may be selected in accordance with, for example, the flowchart shown in FIG. FIG. 17B is an image diagram when selecting a reference face. In these figures, for example, when there are three faces, the difference in size of the face area is the average value of the difference in size between the maximum size face and the remaining two faces, and the difference in size of the face area is equal to or greater than a predetermined value. In this case, the reference face is (A) the face having the largest face area. If it is smaller than the predetermined value, the determination is made based on the difference in brightness of the face area. In this case, when the average value of the average brightness difference between the darkest face and the remaining two faces is equal to or greater than a predetermined value, the reference face is (B) the face with the darkest average brightness, which is smaller than the predetermined value. (D) The average luminance of the three faces is used. If the average brightness of the three faces does not change much, the reference face may be (C) a face whose average brightness is the median of the three faces. When the reference face is determined by such a method, the backlight follow-up light value g is obtained correspondingly using the equations (3) to (6). Note that this is an example, and any calculation method may be used as long as variations in face size and differences in face brightness can be obtained.

  As described above, the backlight follow light level calculation unit 3 a sets the backlight follow light level based on the backlight follow light value g and outputs it to the image correction unit 4. In the image processing unit 4, as in the case of the first embodiment, the backlight follow light level calculation result, the feature amount of the entire image output from the feature amount extraction unit 2, and the object area information from the object detection unit 1 are used. Based on the image correction methods, the backlight correction method, the contrast correction method, and the skin beautification correction method, image correction is performed by setting a necessary correction degree.

  According to the backlight normal light level shown in FIG. 4, the backlight correction method sets the degree of backlight correction as shown in FIG. 11, and the contrast correction method sets the degree of contrast correction as shown in FIG. .

  As for the skin beautification correction method, as shown in FIGS. 13 and 14, the degree of skin beautification is set according to the face area size. Here, it is the size of the face area that becomes the reference for setting when setting the degree of correction of skin beautification correction. (A) When the face with the largest face area is used as the reference for calculating the backlight follow-up light level, When the face size is set as the reference face area size and (B) the face with the darkest average brightness is used as the reference for calculating the backlight follow-up light level, the face size with the darkest average brightness is set as the reference face area size, and (C ) When a face with a median value of three faces with average brightness is used as a reference for calculating the backlight follow-up light level, a face size with a median value of three faces with average brightness is set as a reference face area size. Further, when the face serving as a reference for calculating the backlight normal light level is (D) the average luminance of the three faces, the size of the reference face area is obtained by, for example, obtaining the sum of the areas of the three faces and taking this square root. Then, it is calculated as one side of the square face area.

  As for the contrast correction, as described in the first embodiment, when the face area size is small at the time of backlighting and the gradation to which most of the small face area belongs corresponds to a gradation with a small proportion of the entire screen, the contrast is corrected. Set to not perform correction or to weaken the degree of correction. At this time, not only the case where the size of the reference face area is small and the gradation to which most of the small reference face area belongs corresponds to a gradation with a small proportion of the entire screen, this embodiment also has a face of 3 Therefore, for two face areas other than the reference face area, it is checked whether the gradation to which most of these face areas belong corresponds to a gradation with a small proportion of the entire screen. If the other two face areas correspond to gradations with a small proportion of the entire screen, adjustments are made such that contrast correction is not performed or the degree of correction is weakened. Similar to the first embodiment, when the distribution of the luminance histogram of the entire screen is extremely biased, it is preferable to set so that contrast correction is performed using the parameters shown in FIG.

  Note that the four methods for setting the face as a reference for calculating the backlight follow-up light level in the above-described (A) to (D) are not limited to this, and the backlight in each method is also used. The method for obtaining the follow-up light value g is not limited to that shown in the equations (3) to (6), and is based on the difference in average luminance from the background with the two faces of the maximum face and the face having an intermediate size as a reference. You may obtain | require by the difference of the average brightness | luminance with a background on the basis of two faces of the darkest face and the face of the middle darkness. Further, any method can be used as long as it can capture a luminance change in the screen, such as a difference in average luminance between the face area and the entire screen, or a difference in average luminance between the face area and a part of the background area.

  In the image processing apparatus and the image processing method configured as described above, a plurality of objects are detected from the input image by pattern matching or the like, and a plurality of object areas and other background areas or entire image areas are detected. Since the image correction is performed by calculating the feature amount and calculating the backlight follow light level with a simple calculation, an image processing apparatus capable of automatic processing and high-speed processing is obtained. In addition, by using a plurality of object area information, the characteristics of the entire image, and the backlight direct light level calculation result that can calculate not only the backlight but also the backlight level, it was photographed under various environments, Image correction suitable for each of various images can be performed.

  By the way, in the above description, the case where the present invention is applied to photographing and printing of a digital camera has been described. However, the present invention is not limited thereto. For example, when an image signal is input by shooting with a video camera, a surveillance camera, or the like, the same processing as in the above embodiment can be performed based on an image from this image signal. This makes it possible to correct to an appropriate image even when the face or the like of the person is blacked out or blown out due to backlight or direct light during photographing and the face or the like is difficult to check. The image processing method of the present invention is not limited to a camera or the like, and can be applied to, for example, image processing software for an electronic computer. In this case, the image processing software automatically executes the same processing as in the above embodiment using the CPU, memory, LSI, input / output device, etc. of the electronic computer. Thus, it goes without saying that even when the captured image is not appropriate due to backlight or direct light, it can be corrected to an appropriate image.

It is a block diagram which shows Embodiment 1 of this invention. It is a flowchart which shows operation | movement of Embodiment 1, 2 of this invention. It is explanatory drawing which shows the area | region information of the object which the object detection means 1 in Embodiment 1, 2 of this invention outputs. It is a figure which shows the relationship between the backlight follow light level which the backlight follow light level calculation means 3 in Embodiment 1, 2 of this invention outputs, and the backlight follow light value g. It is explanatory drawing of the contrast correction method in Embodiment 1 of this invention. It is explanatory drawing of the backlight correction method in Embodiment 1 of this invention. It is explanatory drawing of the backlight correction method in the case of the low luminance in Embodiment 1 of this invention. It is explanatory drawing of the backlight correction method in the case of high brightness in Embodiment 1 of this invention. It is explanatory drawing of the skin beautification correction method in Embodiment 1 of this invention. It is explanatory drawing of the skin beautification correction method in Embodiment 1 of this invention. It is a figure which shows the relationship between the backlight follow light level which the image correction means 4 in Embodiment 1, 2 of this invention sets, and a backlight correction parameter. It is a figure which shows the relationship between the backlight follow light level which the image correction means 4 in Embodiment 1, 2 in this invention sets, and a contrast correction parameter. It is a figure which shows the relationship between the backlight follow light level which the image correction means 4 in Embodiment 1, 2 of this invention sets, and the beautiful skin correction parameter. It is a figure which shows the relationship between the face area size which the image correction means 4 in Embodiment 1, 2 of this invention sets, and the beautiful skin correction parameter. It is a figure which shows the relationship between the backlight follow light level and contrast correction parameter in case the distribution of the histogram which the image correction means 4 in Embodiment 1, 2 of this invention sets is biased. It is a block diagram which shows Embodiment 2 of this invention. It is a flowchart which determines the face used as the reference | standard when there are two or more faces in Embodiment 2 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Object detection means 2 Feature-value extraction means 3, 3a Backlight follow light level calculation means 4 Image correction means 11 Whole photo image 12 Face area 13 Background area 14 Face coordinate 15 Width 16 Height 21 Representative value of cumulative histogram 22 Contrast correction γ Curve 31 First low-brightness backlight correction γ curve 32 Second low-brightness backlight correction γ curve 33 Third low-brightness backlight correction γ curve 34 Straight line with slope 35 35 Third high-brightness backlight correction γ curve 36 2nd backlight correction gamma curve for high brightness 37 1st backlight correction gamma curve for high brightness 41 First skin correction gamma curve 42 Second skin correction gamma curve

Claims (10)

Object detection means for detecting an object in the image from an input image and obtaining region information indicating in which region the image is in the image;
Based on the image and the area information from the object detection means, based on the brightness or color of each of the object area, a part or all of the background area other than the object area, and the entire area of the image. Feature amount extraction means for calculating the feature amount,
A backlight follow-up light level calculating means for calculating a back light and follow light level of the image from the feature quantity of the object region from the feature quantity extracting means and the feature quantity of the background region or the entire region;
An image for correcting the image based on the region information from the object detection unit, the entire feature amount of the image from the feature amount extraction unit, and the backlight and follow light levels from the backlight follow light level calculation unit. An image processing apparatus comprising correction means.   The image processing apparatus according to claim 1, wherein the feature amount extraction unit calculates an average luminance of each region as the feature amount. The object detection means obtains area information of each of the plurality of objects when a plurality of objects are detected in the image,
The feature amount extraction unit further calculates a feature amount of each region of the plurality of objects,
The backlight follow-up light level calculating means determines a region of the reference object from the region information of the plurality of objects and their feature amounts, and the feature amount of the reference object region and the background region or the whole region The image processing apparatus according to claim 1, wherein a level of backlight and direct light of the image is calculated from a feature amount of the area.   The image correction means is a contrast correction that corrects a histogram distribution of the luminance of the entire image, a backlight correction that corrects the gradation of the image for each local region in the image using a γ curve, and a person's The image processing apparatus according to claim 1, wherein at least one image correction is performed among beautiful skin corrections for reducing skin noise.   The image processing apparatus according to claim 1, wherein the object detection unit detects a human face as an object and obtains face area information. An object detection step of detecting an object in the image from an input image and obtaining region information indicating in which region of the image the object is located;
Based on the image and the region information from the object detection step, based on the luminance or color of the object region, a part or all of the background region other than the object region, and the entire region of the image. A feature amount extraction step for calculating the feature amount,
A backlight follow-up light level calculating step for calculating a backlight level and a follow-up light level of the image from the feature value of the object region from the feature value extraction step and the feature value of the background region or the entire region;
An image for correcting the image based on the region information from the object detection step, the entire feature amount of the image from the feature amount extraction step, and the backlight and follow light levels from the backlight follow light level calculation step. An image processing method comprising a correction step.   The image processing method according to claim 6, wherein the feature amount extraction step calculates an average luminance of each region as the feature amount. The object detection step obtains area information of each of the plurality of objects when a plurality of objects are detected in the image,
The feature amount extraction step further calculates a feature amount of each region of the plurality of objects,
The backlight follow light level calculating step determines a region of the reference object from the region information of the plurality of objects and the feature amounts thereof, and determines the feature amount of the reference object region and the background region or the whole region. The image processing method according to claim 6, wherein a level of backlight and direct light of the image is calculated from a feature amount of the region.   The image correction step includes contrast correction for correcting the distribution of the luminance histogram of the entire image, backlight correction for correcting the gradation of the image for each local region in the image using a γ curve, and The image processing method according to claim 6, wherein at least one image correction is performed among beautiful skin corrections for reducing skin noise.   The image processing method according to claim 6, wherein the object detection step detects a human face as an object and obtains face area information.

Images