JPH0862741A - Gradation correcting device - Google Patents

Abstract

PURPOSE: To judge reliability with which an input image is backlight scene and a backlight degree in a backlight scene by a picture processing and simultaneously to judge reliability with which an input image is figure photographing scene and a figure degree in order to achieve highly precise discrimination process for forming an optimum gradation correcting curve by judging an input image to make a video- print having a backlight condition or non-beautiful skin color of a person higher image quality. CONSTITUTION: A backlight judging part 103 judges backlight from brightness and shape of a dark part by blocking and binarizing the image and calculates backlight degree from estimation of face brightness of the person. A person judging part 104 judges the presence of the person from chromaticity and brightness of skin color and calculates pereon degree from skin color brightness. Reliability of a scene characteristic is calculated from the backlight judging part 103, the figure judging part 104, a category classifying part 105 and a scene information deciding part 106 and the gradation correcting of the optimum picture is performed by inputting these pieces of information into a gradation correcting part 107. Consequently, an image judging result with higher precision hereto can be obtained and optimum gradation correcting to the image is attained.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gradation correction device for improving the image quality of an image in a video printer or the like.

[0002]

2. Description of the Related Art Images printed by a video printer
Many videos were shot by amateurs with home-use movie cameras. Images taken outdoors during the daytime are often backlit scenes because there is no illumination such as a dedicated video light, and if they are printed as they are, the faces of the people will appear dark. Therefore, it is necessary to determine that the input image is a backlit scene, and if it is determined to be backlit, a technique for increasing the brightness and saturation of the image to beautifully reproduce a dark face is required. It is also important to change the flesh color gradation to a specially adjusted one in a person shooting scene in which the face of a person is up to reproduce a more beautiful flesh color. In this case, a person determination technique is required.

In the prior art of backlight scene determination,
The one that automatically controls the iris that is mounted on a video camera and optimally controls the exposure so that the subject is not darkened by center-weighted photometry (Japanese Patent Laid-Open No. 5-122600) and the gradation curve is optimally selected for a deteriorated image. There are two types of gradation correction (Japanese Patent Laid-Open No. 3-106269). However, in both cases, in the backlight scene determination, the brightness ratio between the central portion of the screen and the entire screen is calculated for the determination. The feature seems to be that it is assumed that the subject always exists in the center of the screen or that the average brightness of the subject corresponds to the degree of backlight. There is one in which the screen is divided into blocks with some degree of freedom in the position of the subject and the main subject is found by correlation with the luminance distribution in the central block of the screen (JP-A-4-340875), and there is a special division of the screen. There is a method (Japanese Unexamined Patent Publication No. 5-122600) that eliminates the influence of bright or dark clothes worn by the subject when obtaining the average luminance of the subject by photometry.

On the other hand, a conventional technique for determining a person is to detect a hue in the vicinity of a skin color (Japanese Patent Laid-Open No. 23592/1992), and to use image area division by a two-dimensional histogram of hue and saturation (Japanese Patent Laid-Open No. No. 4-346334), all of them determine a region that seems to be a skin color within the chromaticity plane and process and judge only by the chromaticity.

[0005]

Conventionally, in backlight determination, it is based on the premise that the subject is present in the center of the screen. Therefore, when the subject is offset to the left or right on the screen,
Alternatively, when the two subjects are separated, there is a problem that the backlight determination is not performed correctly. There is also a problem that an image with a large black area such as an image with only a dark background or an image of a woman with long black hair is erroneously determined as backlight. In addition, the standard for calculating the degree of back light was ambiguous, such that the degree of back light changed greatly between black and white clothes under the same lighting conditions. On the other hand, in person determination, since it is difficult to model the shape of a person, there is a technique that maps the color distribution on the image to a chromaticity diagram and detects the location with a high probability of skin color by hue and saturation. Many. However, outside, the hue and saturation of the skin color are deviated due to the influence of illumination light. In addition, the chromaticity of skin color varies depending on race and makeup. Furthermore, since there are as many flesh-colored colors in the natural world, there is a problem that they are excessively detected in a flesh-colored area such as a lower part of an image where a person's face cannot be originally supposed.

The present invention solves the above-mentioned problems of the prior art, and makes it possible to perform backlight determination with higher accuracy than before, and to determine a person without being affected by illumination, and as a result, a high-quality video print can be obtained. An object of the present invention is to provide an adjustment device.

[0007]

In order to achieve this object, a gradation correction apparatus according to the present invention comprises a block information acquisition unit for dividing an input image into blocks to obtain luminance / chromaticity information, and an output of the block information acquisition unit. According to the feature presence determination means for determining the presence or absence of the feature for each feature of the input image such as a backlit scene and a person scene, and the brightness degree of a specific portion of the input image is determined by image processing that matches the feature. And a category classification unit that classifies the input image into individual categories by the combination of the outputs of the respective feature presence / absence determining units, and to what extent the input image identifies each feature for each individual category. Scene information deciding unit for deciding whether or not the input image has the reliability, and the input image according to the output of the degree information deciding unit and the output of the scene information deciding unit. Those having a gradation correction unit for performing optimal gradation correction.

[0008]

With this configuration, the input image is first roughly divided into blocks vertically and horizontally, and the backlight information and the person are determined by processing the block information.

In the backlight judgment, since the backlight scene is judged based on the shape and brightness of the dark part without using the assumption that the backlight object exists in the center of the screen, the judgment can be performed with higher accuracy than before, and the degree of backlight can also be determined as a human face. Since the part is estimated and evaluated, a reliable value can be obtained. In the human judgment, the skin color judgment is used as the basis, but the skin color distribution is calculated by the principal component analysis method including the skin color gradation based on the candidate data from the chromaticity, and the judgment is made based on the angle. , Can be detected relatively without being affected by illumination fluctuation.

Since the respective degrees and reliability are obtained from the above-described backlight judgment and person judgment, an optimum gradation correction curve is generated from them, and gradation correction can be performed,
Even if the quality of the original image is not very high, a high quality video print can be obtained.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The overall structure of an embodiment of the present invention will be described below with reference to FIG. FIG. 1 is a block connection diagram of a gradation correction apparatus according to an embodiment of the present invention. In FIG. 1, 101 is an input image, which is an image input from a video printer, and 102 is a block information acquisition unit.
The input image 101 is vertically and horizontally divided into blocks, a representative value in each block is determined, and the representative value is output as block information. A backlight determination unit 103 receives the block information and calculates a backlight presence determination 108 and a backlight intensity 110 by a backlight presence determination unit 103a and a backlight intensity information determination unit 103b, respectively. A person determination unit 104 receives the block information and calculates a person presence / absence determination 109 and a person degree 111 by a person presence / absence determination unit 104a and a person degree information determination unit 104b. 105
Is a category classifying unit that combines two pieces of information, that is, backlight presence / absence determination 108 and person presence / absence determination 110, into an input image as a first "a backlit portrait shooting scene" and a second "a backlight but a portrait shooting scene." It is classified into one of four categories: “no”, thirdly “a person shooting scene, not backlight”, and fourth “not a person shooting scene, not backlight”. Reference numeral 106 denotes a scene information determination unit, which has three characteristics of an image scene, “backlight scene”, “portrait shooting scene”, and “others”, which are required when synthesizing a correction curve in gradation correction from the selected category information. Reliability information 1
Decide 12 Reference numeral 107 denotes a gradation correction unit, which is reliability information 1
A gradation correction curve is generated based on 12, the degree of backlight 110, and the degree of person 111.

Each block of one embodiment of the present invention will be described below. First, the block information acquisition unit 102 will be described. In the input image 101 input to the block information acquisition unit 102, each pixel obtained by A / D converting an analog signal from a video printer or the like has a luminance Y and color differences RY and B.
It is a full-color image represented by Y. Where RY =
RY and BY = BY. If this image is used as it is for backlight determination and person determination, the amount of data processing increases, so the block information acquisition unit 102 divides the screen into blocks vertically and horizontally, and uses only the representative value of each block as block information for determination. The block information includes (1) average value, average value of colors in each block, (2) WP value, color at pixel (location) where luminance Y is maximum in each block, (3) BP value, luminance in each block It consists of three types of color information of the color at the pixel (location) where Y is the smallest, and all of them are luminance Y and color difference R.
It is expressed by Y and BY. It can be said that this information includes both the color distribution information within the block and the gradation information.
For example, even if the color is the same as the skin color, if one block is all "solid color" like a painted wall, all three types of information match, but an uneven object such as a human face has shadows in the block. Generally, three skin colors are different and WP
Is almost highlighted, and BP is almost black. As described above, the block information has some resolution information, but has some gradation information.

Although not described in the present embodiment, this block information is white balance correction processing for removing the influence of the illumination of the subject and standard brightness correction A
It is assumed that GC (auto gain control) processing and the like have already been performed.

Next, the details of the backlight determining unit 103 will be described with reference to FIG.
Will be explained. FIG. 2 is a block connection diagram of the backlight determination unit 103, which is a main part of the tone correction apparatus in one embodiment of the present invention. First, the determination policy of the backlight image will be described. In the present embodiment, it is considered that "the difference between the backlit image and the normal image is the distribution pattern of the shadows in the image". Since the illumination light is projected from the front of the subject in the normal light state, the shadow blocks when the image is represented by the block average image are sparsely distributed in the entire image without deviation. In the backlight image, the illumination light is projected onto the subject from above or from the back surface, so that the shadow block exists as a block at the bottom of the image or from the top to the bottom of the image. It is decided to determine the backlight by using the brightness of the shadow block and the lump pattern (the brightness and the shape). Since the center of the image may be bright when two persons are backlit, this method does not use the assumption that the shadow is in the center of the image. According to this policy, the average brightness of the block information is mainly used, and after determining the blocks in the shadow (dark area) by the brightness threshold processing, the dark area average brightness value,
The backlight judgment is performed in consideration of all three conditions, that is, the brightness ratio of the bright part / dark part and the shape of the dark part.

In FIG. 2, input image block information 201
Is an image made of the block average color among the three types of information of the block information, and the average luminance and the average chromaticity are calculated for each block. The average value of the average brightness of all blocks is calculated by the image average brightness calculation means 202, and the image is binarized by the light / dark block brightness processing means 203 to be classified into two types of blocks, a dark part and a bright part. FIG. 3 shows an example of the above processing. FIG. 3A is a color image to be input, and in this case, it is a bust shot of a backlit person. In FIG. 3B, the screen is vertically (x direction) 7 blocks horizontally (y
Direction) shows an average luminance image of a block average color image obtained by dividing the block into a total of 70 blocks, and when this is binarized by the average of the entire luminance of each block, FIG. Like The shape of the person in the backlight part corresponds to the dark part. Here, the block average luminance image and the binarized image are two-dimensional arrays AV_YY [x] [y] and dark_tbl, respectively.
When expressed as [x] [y], the mean value mean_y of the average luminance of all blocks is

[0016]

[Equation 1]

In the binarized image creation, the bright part if AV_YY [x] [y]> mean_y then dark_tbl [x] [y] = 0 the dark part if AV_YY [x] [y] <mean_y then dark_tbl [x ] [y] = 1. Next, in the same brightness / darkness block brightness processing means 203, (1) dark part average brightness dark_mean,
(2) Bright part average brightness light_mean, (3) Bright / dark brightness ratio
LDratio is calculated.

[0018]

[Equation 2]

[0019]

(Equation 3)

[0020]

[Equation 4]

Here, the total number of dark block is dark_large
And The dark block saturation processing means 204 is used when a backlit person is photographed and a background with relatively high saturation and brightness (green of mountains, green of trees, etc.) is connected to a backlit person subject in front. It is introduced to solve the problem of forming dark areas in a binarized image. The content of the processing is to re-binarize only the dark part by the average saturation for the binarized image, and leave only the monochrome area part having the low saturation and the dark part as the binary image. However, re-binarization by saturation processing is not effective for all images, and in the originally low-saturation monochrome backlight area, almost the same monochrome area is re-binarized using a slight saturation variation. Because it does, it has the opposite effect. Further, in a backlit person image of clothes with high saturation and low brightness, all the clothes are not in the backlit area, so only the face is separated and the typical backlit pattern is destroyed. Therefore, the average saturation in the dark area is first obtained, and if the average saturation is very low or very high, re-binarization is not performed. The specific processing procedure is as follows. (1) Obtain the squared saturation average value (dark_mean_s) in the dark area.

[0022]

(Equation 5)

(2) When the value of dark_mean_s is larger than the threshold value S1 (S_CHECK = 1), the saturation is sufficiently high, so that binarization is not performed. (3) If the value of dark_mean_s is smaller than the threshold value S2 (S_CHECK = 2), the saturation is low enough so that binarization is not performed. (4) When the value of dark_mean_s is in the middle of S1 S2 (S_
Re-binarize to CHECK = 0). Re-binarization is performed as if (AV_RY [x] [y]) ² × (AV_BY [x] [y]) ² <dark_mean_s then dark_tbl [x] [y] = 0. The dark block shape processing means 205 calculates four peripheral features of the dark part. As described above, the backlit subject has a shape feature that occurs as a block of shadows from the bottom to the top on the screen. Therefore, the image is binarized, and the number of consecutive black blocks until the black block disappears is added from each of the four surrounding edges to the four peripheral features (Peripheral Characteris
tics). Peripheral feature values are ranked from 1st to 4th
By performing the process up to the position, the shape and position of the lump of the dark part in the image is expressed. The processing method will be described with reference to FIG. FIG. 4 is a conceptual diagram of an embodiment of the dark block shape processing. In FIG. 4, the upper left is the origin, the horizontal is the x-axis, and the vertical is the y-axis, and the blocks are divided into 10 blocks in the x direction and 7 blocks in the y direction. The direction from the right side of the image is the (0) direction, the left side is the (1) direction, the upper side is the (2) direction, and the lower side is the (3) direction. The peripheral feature is the amount obtained by accumulating black blocks continuous from these four directions in the opposite direction and adding them on the side. These quantities are accumulated in periph [0] to periph [3].

[0024]

(Equation 6)

[0025]

(Equation 7)

[0026]

[Equation 8]

[0027]

[Equation 9]

Here, how to obtain S0, S1, S2 and S3 will be described with reference to FIG. FIG. 5 is a flowchart of the dark block shape processing. Since S0 to S3 are obtained by the same process, only S0 is y = 0
The case will be described. In FIG. 5, first, S0 = 0, y =
Set 0 and x = 9 (steps (a) and (b) in FIG. 5), and if x is not 0, determine whether dark_tbl [x] [y] = 1 (step
(C), (D)). If dark_tbl [x] [y] = 0, the process ends. dar
If k_tbl [x] [y] = 1, increase S0 by 1 (step (e)) and set x to 1
Decrease (step (f)) and return to the judgment whether x is 0 or more. If x = 0, the process ends (step (g)).
This process is performed from y = 0 to y = 6 and the sum is calculated. According to the embodiment shown in FIG. 4, the following peripheral area information of the dark block is obtained.

Periph [0] = 17 periph [1] = 14 periph [2] = 0 periph [3] = 37 In the dark block peripheral information ordering means 206, the dark part peripheral information periph [0] to periph [3] are larger or smaller. Order by relationship or
Set the order 0,1,2,3 from der [0] to order [3]. In the embodiment of FIG. 4, in order of increasing peripheral information,
37, 17, 14, 0, so if you set that direction, order [0] = 3 (downward) order [1] = 0 (rightward) order [2] = 1 (leftward) order [3 ] = 2 (upward). This order relationship is simplified from the direction with the greater magnitude relationship and expressed as (3 0 1 2) as pattern shape information in four directions. The pattern information 212 means position / shape information that "there is a dark portion from the lower portion to the right portion of the image and there is no upper portion".

The backlight final determination means 207 obtains two pieces of information 211 of the dark area average luminance dark_mean and the light / dark luminance ratio LDratio from the light / dark block luminance processing means 203, and the dark portion pattern shape information 212 from the dark portion block peripheral information ordering means 206. obtain. And (1) The average brightness value of the dark part is darker than the light / dark threshold dark_mean_t (dark_mean <= dark_mean_t) (2) The brightness ratio of light / dark is larger than the brightness ratio threshold LD_t (LDratio> = LD_t ) (3) The shape pattern of the dark part is (3 xxx) or (2 3
xx). The presence / absence of backlight is determined as "backlight" when the three conditions are simultaneously satisfied and "not backlight" when any one of the conditions is not satisfied. Here, x indicates that it is arbitrary. That is, the shape of the dark part is the pattern with the largest 3 (down) direction, and 2 (up) and then 3 (down).
If it is large, it is against the light. The former corresponds to the most common backlit bust shot of a person, and the latter corresponds to a backlit face up of a person. This is the backlight presence / absence determination 1
It will be 08.

Next, the dark part block center of gravity processing means 208 obtains the center of gravity of the dark part for the subsequent estimation of the human face position. It is important to note that when processing the center of gravity, in a backlit scene, the dark area in front of the subject other than the subject is often reflected in the left and right edges of the image. This is the point that they are misaligned. Therefore, in the dark part, one block at the left and right ends of the image, x = 0 and x
The block of = 9 is excluded, and the center of gravity is calculated using blocks 1 to 8 in the x direction. If the center of gravity is (gcenterX, gcenterY) in block coordinates (x, y),

[0032]

[Equation 10]

[0033]

[Equation 11]

It becomes Next, the inside / outside of the dark part of the center of gravity, which is used for estimating the position of the face of the person at the time of calculating the backlight intensity, is determined. When the center of gravity is inside the dark part, the distribution of the dark part is concentrated, and it can be estimated that the backlit subject exists in a mountain shape at the center of the image. On the other hand, when the center of gravity is outside the dark area, it can be estimated that the backlit subjects are two persons and the dark area has a bimodal mountain shape. This information is displayed in some cases, such as when the block at the position of the center of gravity is expressed as in FIG. 8, G ... The center of gravity exists inside the dark part, g ... The center of gravity exists outside the dark part.

The dark block human face luminance processing means 209 calculates the degree of backlighting. At first, I thought that the average brightness in the dark area would be an index of the degree, but in reality, the average brightness is high because the face of the person is dark but the clothes are white. There wasn't. Therefore, we decided to use the brightness of the person's face as an index, following the human subjective evaluation. In many cases, skin color search is used for face search, but the face of a person in a backlit state is shaded and there is no color information, so it is impossible to search using colors equally. However, since the backlight image is characterized in that a person is present in the backlight portion, the face position may be estimated only within the dark portion with reference to the position of the center of gravity of the dark portion. At this time, considering the result of the dark area block peripheral information ordering process, the shape of the dark area is divided into the following three cases, and in each case, the average brightness f
Calculate acelight. Hereinafter, a facelight calculation method will be described with reference to FIG. 6, which is a conceptual diagram of a dark block human face estimation example. In FIG. 6, the black part indicates the dark part block, and the shaded part indicates the part estimated to be the human face in the dark part block, and the number of blocks judged to be the human face part is fa
ce_num. First, the case where the dark center of gravity is outside the dark part will be described with reference to FIG. Since the dark part has a bimodal shape, the center-of-gravity block g exists outside near the valley of the dark part, and it is considered that two or more persons are backlit. At this time, "the entire dark area above the center of gravity"
The average brightness is calculated by considering that the faces are the faces of two persons.

[0036]

[Equation 12]

Secondly, a case where the center of gravity of the dark portion is in the dark portion and the shape of the dark portion is (2 3 XX) (where X is arbitrary) will be described with reference to FIG. 6B. This is a case where the dark part exists from the upper part to the lower part of the image, and corresponds to a case where a backlit person is photographed up, or a case where there is a distant view part such as a forest in the background which hits the backlit person. At this time, if the upper part of the dark part is considered to be a face, the brightness may be slightly brighter in the distant view part. Therefore, "the dark part inside the rectangular area below the center of gravity and within three blocks to the left and right of the center of gravity" The average brightness is calculated by considering it as a face.

[0038]

[Equation 13]

Thirdly, with reference to FIG. 6C, the case where the center of gravity of the dark portion is inside the dark portion and the shape of the dark portion is (3 XXX) will be described. This is the case where the dark part exists at the bottom of the image, which is a standard case when a bust shot of a backlit person is taken. In this case, there is a high possibility that a person's "clothes" will exist in the lower part of the dark part, and the average brightness will be greatly affected by whether the clothes are black or white. Therefore, in order to avoid this, the average brightness is calculated by considering the "dark portion above the center of gravity position and inside the rectangular area of three blocks to the left and right of the center of gravity" as the face of the person.

[0040]

[Equation 14]

In some cases, there are no dark blocks in the face candidate area determined from the dark center of gravity (face_num == 0). For example, (1) the center of gravity is outside the dark area, and the number of face candidate blocks =
0 (2) Center of gravity is in the dark area and shape pattern = 2xxx 3xxx
Other than the above, in this case the average face brightness facel
ight has a maximum value of 255. In the backlight intensity calculation means 210, the backlight intensity is determined as an evaluation value (0 -255) as to how dark the face luminance is compared with the brightness threshold dark_mean_t. Also, the face brightness most_dark when the maximum backlight intensity is 255
Normalize by deciding. That is,

[0042]

(Equation 15)

When the facelight has a negative backlighting such as 255, the backlighting is set to 0. This is the backlight intensity 110.

Next, the details of the person determination unit 104 will be described with reference to FIG.
Will be explained. FIG. 7 is a block connection diagram of the person determination unit 104 which is a main part of the tone correction apparatus according to the embodiment of the present invention. First, a determination method of a person image will be described.

For the detection of a person, the detection of a skin color such as a face is used as in the prior art, but the characteristic of gradation information as well as the chromaticity of the facial color is extremely important. On the face of a person, the skin color has a bright shade, and the nose, cheeks, and forehead are almost white, while the openings such as hair, eyes, nostrils, and mouth are almost black. This is in the block information,
In each block, a portion close to white can be detected as a WP value, and a portion close to black can be detected as a BP value, which means that the overall skin color can be detected as an average value. Because of the existence of low-saturation white and black, and high-saturation flesh color, when these are plotted on the chromaticity diagram, in the flesh color image, the gradation highlight part, the average value, and the dark part form the same hue line. Tend. However, a wall having a cream color system of the same hue does not have a clear hue line on the chromaticity diagram because it has a uniform color and has little shading. In addition, since red, which has the same hue as the flesh color, has high saturation in the dark portion, it can be clearly distinguished. From the above consideration, in the present invention, all the information of the WP value, the BP value, and the average value in each block of the image is plotted on the chromaticity diagram, and the hue line is statistically extracted using the principal component analysis method, which seems to be the skin color. Whether or not it will be detected.
When performing this calculation, (1) use only the sample data existing in the second quadrant of the simple chromaticity diagram to detect the hue line by principal component analysis, and (2) simply detect the angle of the detected hue line. By detecting in two steps, that is, in the range of π / 4 to π in the chromaticity diagram, the angle formed by the entire data distribution even if the hue line moves in parallel without passing through the origin due to the influence of illumination light. Is calculated almost correctly.

In FIG. 7, the skin color candidate block selection means 3
In 01, the input image block information 201 is processed to select the block information existing within the hue angle which seems to be skin color. If this candidate block selection is not performed, statistical processing will be performed including colors completely unrelated to the skin color, and a meaningful result cannot be obtained. Therefore, "Skin color is R
It exists in the range from the yellow hue to the red hue in the GB space ", and a simple color difference (RG) is used as a chromaticity diagram.
(BG) Use chromaticity diagram (color moment diagram). In the simple color difference chromaticity diagram, the hues from yellow to red in the hues defined in the RGB color space correspond to the second quadrant, so the candidate block can be judged by simple positive / negative judgment, and there is no need to calculate the angle. is there. Therefore, in the following explanation, RY means RG and BY means BG. FIG. 8 illustrates how the hue line 901 is represented on the simple color difference chromaticity diagram. Skin colors photographed outdoors as in this example often exist in the second quadrant from yellow to red in the simple chromaticity diagram.

Therefore, the candidate block is selected by using the condition that the BP value is not set and that both the WP value and the average value are present in the second quadrant in the simplified chromaticity / color difference diagram of FIG. The block information that is originally expressed as (Y, RY = RY, BY = BY) is converted into simple color difference (Y, RG, BG) again (1) Average value AV_YY [x] [y], AV_RY [x] [y], AV_BY [x] [y] (2) WP value WP_YY [x] [y], WP_RY [x] [y], WP_BY [x] [y] (3) BP value BP_YY [x ] [y], BP_RY [x] [y], BP_BY [x] [y], only the blocks existing in the second quadrant of the simplified color difference chromaticity diagram shown in FIG. Using table Zin_tbl [x] [y], if (WP_RY [x] [y]> = 0 WP_BY [x] [y] <= 0 AV_RY [x] [y]> = 0 AV_BY [x] [y ] <= 0) then Zin_tbl [x] [y] = 1 else Zin_tbl [x] [y] = 0. Another condition is that the saturation of the WP value is not extremely large Add In this way, the number of skin color candidate blocks detected with Zin_tbl [x] [y] == 1 is set as hada_num.
The covariance matrix calculation unit 302 uses three types of block information, that is, the WP value, the BP value, and the average value in the skin color candidate block, and obtains the color distribution statistics using the gradation information in these blocks. Since the statistical method uses the principal component analysis method for obtaining the first eigenvector from the covariance matrix, the overall average value, variance value, and covariance value of each chromaticity of WP, BP, and average are calculated for the skin color candidate block. Average of RY and BY
ry_av, by_av is calculated by the following formula. However, the sum shall be taken for all x and y where Zin_tbl [x] [y] = 1.

[0048]

[Equation 16]

[0049]

[Equation 17]

Here, hada_num is the number of flesh color candidate blocks, and 3 × hada_num is the total number of chromaticity samples. R
The variance values Srr and Sbb of Y and BY and the covariance value Srb are obtained by the following equations. The same sum is taken for all x and y where Zin_tbl [x] [y] = 1.

[0051]

(Equation 18)

[0052]

[Formula 19]

[0053]

[Equation 20]

As a result, the covariance matrix calculation means 302 calculates the matrix

[0055]

[Equation 21]

Is calculated. Next, the flesh color hue line detection means 303 determines the flesh color hue line to be the first principal component axis of this statistical distribution,
That is, it is calculated as an eigenvector having the maximum eigenvalue. Specifically, TanValue = tan 2θ, where θ is the angle between the hue line, which is the main component axis, and the BY positive axis.

[0057]

[Equation 22]

It is calculated by the following relational expression. In the skin color hue line detection means 303, TanVal is not calculated directly for the purpose of simplifying the calculation.
The value of ue will be used instead.

Next, the skin color hue angle existing area detecting means 304
Then, it is determined whether or not the hue angle θ of the flesh color line exists within a range reserved beforehand. The problem here is
This is a multi-valued function of Tan ^{- 1} , but this problem is 2
This can be solved by considering the quadrant in which θ exists. In FIG. 9 which is a conceptual diagram showing the existence region of the skin color hue angle 2θ and in FIG. 10 which is a principle diagram of the existence region determination of the skin color hue angle, the quadrant where 2θ exists is Srb> = 0 and (Sbb-Srr)> = 0 => 1st quadrant Srb> = 0 and (Sbb-Srr) <0 => 2nd quadrant Srb <0 and (Sbb-Srr) <0 => 3rd quadrant Srb <0 and (Sbb-Srr)> = 0 => Determined as the 4th quadrant. In the present invention, it is assumed that the range of the detected angle θ of the skin color hue line is from π / 4 to π shown by the shaded range in the simplified color difference chromaticity diagram as shown in FIG. Therefore 2θ
Is twice that, and as shown in Fig. 9, from π / 2 of tan 2θ to 2π
Range. Therefore, 2θ exists from the second quadrant through the third quadrant to the fourth quadrant as shown by the angle in the shaded area in FIG. 9, but cannot exist in the first quadrant. Utilizing this property, the specification of the angle range of θ can be replaced with the thresholding of TanValue = tan 2 θ. 2θ in FIG.
And tan2θ are shown. In FIG. 10, from the above discussion, the linearity and singularity of tan2θ can be maintained in the second, third, and part of the fourth quadrant.
Determine H2. Skin color hue angle existing area detecting means 30
The output of 4 is as follows: if (TanValue> TH1 or TanValue <TH2) else It is a skin color and not a skin color. However, TH1 must be greater than TH2,
Without this condition, monovalency cannot be established.

The skin color candidate block center of gravity processing means 305 calculates the center of gravity of the position of the skin color candidate block. This process is introduced for threshold value processing of the center of gravity position for preventing erroneous determination of a region similar to the skin color such as "desk" or "ground" at the bottom of the image. Center of gravity position in block coordinates (x, y) (hada_gcenterX, hada_gcenterY)
Then,

[0061]

[Equation 23]

[0062]

[Equation 24]

It becomes The person final determination processing unit 306 finally determines whether or not the input image is a person image based on the following determination criteria. That is, (1) the output from the flesh color hue angle existing area detection unit 304 falls into one of the following three ranges.

(A) 2θ is in the second quadrant and TanValue> TH1 (b) 2θ is in the third quadrant (c) 2θ is in the fourth quadrant, and TanValue <th2 (2) Skin color candidate block centroid The output from the processing means 305 is in the following range.

If both conditions of hada_gcenterY <hada_gcenter_max (that is, the Y coordinate on the screen of the center of gravity of the flesh color is higher than the fourth block in the image) are both human shooting scenes, the others are human shooting scenes. There is no. This is the person presence / absence determination 109 in the person determination unit 104.

The flesh color brightness processing means 307 calculates the average flesh color brightness h of the flesh color candidate blocks in order to obtain the degree of person.
Ask for ada_Y. The average value of (highlight luminance value) + (average luminance value) of the skin color candidates is calculated only for the blocks in which the array Zin_tbl [] [] is 1.

[0067]

(Equation 25)

In the person degree calculating means 308, "darkest skin color luminance" hada_most_dark, "lightest skin color luminance"
Determine hada_most_light by referring to "skin brightness" obtained from image processing experiments, and set hada_Y to hada_most_dark
Normalize during hada_most_light.

[0069]

(Equation 26)

If the person presence / absence determination is NO, the degree of person = 0
I am trying. This is the degree of person 111.

Details of the category classifying unit 105 and the scene information determining unit 106 will be described below. The category classification unit 105 combines the two pieces of information of the backlight presence / absence determination 108 and the person presence / absence determination 110 into the input image as “backlight YES person YE
"S""Backlight YES person NO""Backlight NO person YES""Backlight NO person N"
It is classified into one of the four categories of "O".
In the scene information determination unit 106, the reliability of each of the three characteristics of the image scenes “backlight scene”, “portrait shooting scene”, and “other” necessary when synthesizing the correction curve in gradation correction is selected from the selected category information. Determine the degree. The sum of reliability is 1. The reliability value is determined in advance for each category. For example, the image shows "Backlight YES Person Y
If it is determined that the image belongs to the category of “ES”, the three types of reliability are automatically determined, and the reliability is calculated for each image like the degree of backlight and the degree of person. Not of. Therefore, it is necessary to perform a pre-processing to obtain the correspondence between the category and the reliability in advance.

The preprocessing will be described below with reference to FIG. 11, which is a block connection diagram showing the preprocessing for reliability calculation. In FIG. 11, a sample image 401 is a large number of images used for preprocessing. It is assumed that these images are labeled in advance as either "backlit images", "portrait images", or "other images" (landscapes, still lifes, etc. are collectively referred to as "others") according to human subjective evaluation.

The backlight judgment unit 1 for each sample image
03, the backlight presence / absence determining unit 103a performs processing, and the backlight presence / absence determining unit 108 outputs, and at the same time, the person determining unit 104 performs processing by the person presence / absence determining unit 104a.
09 is output. The category classification unit 105 has 2 ² = 4 categories based on the combination of the presence / absence of the two characteristics of “backlight scene” and “person shooting scene”, that is, “backlight YES person YES”, “backlight YES person NO”, and “backlight NO person”. YES "
Automatically classified into "backlit NO person NO". After processing all sample images, as shown in FIG. 4, N1 image samples in total are classified into the category of “backlight YES person YES”, and the breakdown is the image labeled “backlight image” in the subjective evaluation.
It is assumed that one example is the same "person image" as the Z1 example, and the same "other image" is the A1 example. At this time, "Backlight YE
The reliability in the "S person YES" category is defined as follows.

[0074]

[Equation 27]

Similarly, the reliability in the “backlight YES person NO” category is

[0076]

[Equation 28]

It becomes The same applies to other categories. The scene information determination unit 106 determines the reliability of the input image based on the reliability of each feature obtained by such preprocessing. The gradation correction unit 107 generates a gradation correction curve based on the reliability information 112 of the backlight reliability, person reliability, and other reliability of the input image, the backlight 110, and the person 111. The gradation correction unit 107 facilitates three types of templates, a backlight correction gradation curve, a person correction gradation curve, and other correction gradation curves. Generation of the gradation curve is performed in two steps. First, the correction degree of the backlight correction gradation curve and the correction degree of the person correction gradation curve are adjusted based on the degree information of the backlight and the person. The correction gradation curve is determined independently of the degree. Next, the three types of generated curves are weighted and combined with the three types of reliability information of backlight reliability, person reliability, and other reliability to generate a final curve. By the above operation, the optimum gradation correction curve according to the image scene can be generated.

[0078]

According to the present invention, it is possible to determine a backlight scene and a person scene from an input image and optimally generate a gradation correction curve in accordance with the degree and reliability of the scene so that gradation correction can be performed. Even if the quality of the original image is not very high, a high quality video print can be obtained.

In the backlight judgment, since it is possible to judge the backlight scene in the shape and brightness of the dark part without using the assumption that the backlight object exists in the center of the screen, it is possible to make the judgment with higher accuracy than before, and the degree of backlight can also be determined by the human face. Since the part is estimated and evaluated, a reliable value can be obtained.

In the person determination, the skin color determination is used as a basis, but a method of obtaining the skin color distribution by the principal component analysis method including the skin color gradation based on the candidate data from the chromaticity and making the determination based on the angle is used. Therefore, it can be detected relatively without being affected by the fluctuation of illumination.

[Brief description of drawings]

FIG. 1 is a block connection diagram of a gradation correction apparatus according to an embodiment of the present invention.

FIG. 2 is a block connection diagram showing a backlight judgment unit which is a main part of the tone correction apparatus of the embodiment.

FIG. 3 is a conceptual diagram showing a brightness / darkness block luminance process in the tone correction apparatus of the embodiment.

FIG. 4 is a conceptual diagram showing an example of dark block shape processing in the tone correction apparatus of the embodiment.

FIG. 5 is a processing flowchart of dark block shape processing in the tone correction apparatus of the embodiment.

FIG. 6 is a conceptual diagram showing an example of estimating a dark block human face portion in the tone correction apparatus of the embodiment.

FIG. 7 is a block connection diagram showing a person determination unit which is a main part of the tone correction apparatus of the embodiment.

FIG. 8 is a simplified color difference chromaticity diagram showing a range of an angle θ of a skin color hue line in the tone correction apparatus of the same embodiment.

FIG. 9 is a flesh color hue angle 2 in the tone correction apparatus of the embodiment.
Conceptual diagram showing the existence region of θ

FIG. 10 is a principle diagram of determination of a region where a flesh color hue angle exists in the tone correction apparatus of the same embodiment.

FIG. 11 is a block connection diagram showing a preliminary process for reliability calculation in the tone correction apparatus of the embodiment.

[Explanation of symbols]

 101 Input Image 102 Block Information Acquiring Section 103 Backlight Determining Section 103a Backlight Existence Determining Means 103b Backlight Depth Information Determining Section 104 Person Determining Section 104a Person Existence Determining Section 104b Person Degree Information Determining Section 105 Category Classification Section 106 Scene Information Determining Section 107 Gradation Correction unit 108 Backlight presence / absence determination 109 Person presence / absence determination 110 Backlight degree information 111 Person degree information 112 Reliability information

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI technical display location H04N 9/79

Claims (13)

[Claims] 1. A block information acquisition unit that divides an input image into blocks to obtain luminance and chromaticity information, and a feature presence / absence determination unit that determines the presence or absence of the feature for each feature of the input image by the output of the block information acquisition unit. And a degree information determining unit that determines the degree of brightness of a specific portion of the input image by image processing that matches the features, and the input image is classified into individual categories by a combination of outputs of the feature presence / absence determining units. A category classification unit, a scene information determination unit that determines with what degree of reliability the input image has each feature for each of the individual categories, an output of the degree information determination unit, and the scene A gradation correction device comprising: a gradation correction unit that performs gradation correction on the input image according to the output of the information determination unit. 2. The feature presence / absence determining means, an image average brightness calculating means for obtaining an average brightness of the entire input image from the output of the block information acquiring part, and a bright / dark block for binarizing the input image into a bright part block and a dark part block. 2. A brightness processing means, a dark part block shape processing means for processing the dark part block and outputting dark part peripheral information, and a dark part block peripheral information ordered processing means for ordering the dark part peripheral information.
The gradation correction device described. 3. The feature presence / absence determining means, an image average brightness calculating means for obtaining an average brightness of the entire input image from the output of the block information acquiring part, and a bright / dark block for binarizing the input image into a bright block and a dark block. Luminance processing means, dark area block saturation processing means for binarizing the dark area block by saturation, dark area block shape processing means for processing the output of the dark area block saturation processing means and outputting dark area peripheral information, The gradation correction apparatus according to claim 1, further comprising a dark area block peripheral information ordering processing unit that orders the dark area peripheral information. 4. The degree information determining means, an image average luminance calculating means for obtaining an average luminance of the entire input image from the output of the block information acquiring portion, and binarizing the input image into a bright portion block and a dark portion block for processing. Bright / dark block brightness processing means, dark part block center of gravity processing means for processing the dark part block and outputting the center of gravity of the dark part block, and output of the dark part block center of gravity processing means to determine the face area of a person in the dark part block. The tone correction apparatus according to claim 1, further comprising a dark block human face luminance processing unit that estimates and calculates the luminance. 5. The feature presence / absence determining means processes the output of the block information acquiring section to select a skin color candidate block, and a covariance matrix that performs statistical processing from the output of the skin color candidate block selecting means. Computation means, flesh color hue line detection means for detecting a flesh color hue line from the output of the covariance matrix calculation means, and flesh color hue angle existence area detection means for detecting the existence area of the hue angle of the flesh color hue line. Item 1. The gradation correction device according to item 1. 6. The feature presence / absence determining means processes the output of the block information acquiring section to select a skin color candidate block, and a covariance matrix which performs statistical processing from the output of the skin color candidate block selecting means. Calculating means, a skin color hue line detecting means for detecting a skin color hue line from the output of the covariance matrix calculating means, a skin color hue angle existing area detecting means for detecting an existing area of a hue angle of the skin color hue line, and the skin color The tone correction apparatus according to claim 1, further comprising: a skin color candidate block centroid processing unit that processes the output of the candidate block selection unit to output the centroid of the skin color candidate block. 7. The degree information determination means processes the output of the block information acquisition part to select a skin color candidate block, and the skin color candidate block processing means processes the skin color candidate block to output the brightness of the skin color. The gradation correction device according to claim 1, further comprising a brightness processing unit. 8. The category classifying unit, based on a combination of the presence / absence of a backlight scene feature and the presence / absence of a person scene feature in the input image, firstly a backlighting person shooting scene, and secondly a backlighting person shooting scene. The gradation correction device according to claim 1, wherein the gradation correction device is classified into four categories, namely, no, third is a person shooting scene, not backlight, and fourth is not a backlight, not person shooting scene. 9. The scene information determining unit determines, by the category classifying unit, a large number of image samples pre-labeled with each one of the M features Cm (m = 1 ... M). The number of samples of the i-th classified Ni image samples labeled with the feature Cm is Lm, and each feature Cm of the i-th sorted image is automatically classified into 2 ^M pieces from the combination. 2. The gradation correction device according to claim 1, wherein the reliability of Lm / Ni is Lm / Ni. 10. The features of the input image are firstly a backlit scene,
The gradation correction device according to claim 1, wherein there are three scenes of a person photographing scene, a scene excluding the first scene and a scene of the second scene. 11. The block information acquisition unit divides an input image vertically and horizontally into coarse blocks, and acquires average values of luminance and chromaticity of all pixels in each block for each block. Item 1. The gradation correction device according to item 1. 12. A block information acquisition unit divides an input image vertically and horizontally into coarse blocks, defines a pixel set for each block, and the pixel set having the maximum luminance of color for all the pixel sets in the block. The gradation correction device according to claim 1, wherein an average luminance and an average chromaticity within the image are acquired. 13. A block information acquisition unit divides an input image vertically and horizontally into coarse blocks, defines a pixel set for each block, and sets the pixel set in which the luminance of color is minimum for all the pixel sets in the block. The gradation correction device according to claim 1, wherein an average luminance and an average chromaticity within the image are acquired.