JP2019040382A - Image processing device - Google Patents

Abstract

To generate a desirable image with wide dynamic range and high contrast by reducing coloration generated at boundaries between areas in a scene photographed at different color temperature when correcting by setting gradation characteristic for each area.SOLUTION: An image processing device determines a subject area of an input image, and detects a first image recognition area and a second image recognition area other than the first image recognition area from the result of subject area determination, and determines first gradation correction amount on the basis of image evaluation values detected in the first image recognition area and the second image recognition area, then detects color temperature of the first image recognition area and the second image recognition area, further, calculates color temperature difference, and determines second gradation correction amount on the basis of gradation characteristic determined from the color temperature difference and the first gradation correction amount.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an image processing technique for correcting gradation of an output image.

  Conventionally, there is a process of performing gradation compression on an input signal with an expanded dynamic range, such as high dynamic range synthesis (HDR) or dodging, and outputting the result.

  In Patent Document 1, a subject area is determined for an input image, a representative luminance value and a histogram for each subject area are calculated using the determination result of the subject area, and gradation correction for each subject area is performed from the representative luminance value. A technique for calculating the amount and determining the gradation characteristic has been proposed.

JP 2014-153959 A

  However, in Patent Document 1, there is a problem that coloring occurs in the edge portion of the subject when there is a color temperature difference between the background region and the subject region, particularly the person region.

  In particular, when strobe light is emitted, the color temperature of the portion where the subject is exposed to strobe light and the strobe of the background portion are often different in the non-irradiated region, and this is likely to occur remarkably.

In order to solve the above problems, an image processing apparatus according to the present invention provides:
An image processing apparatus that performs gradation correction of an image,
Subject region determination means for determining a subject region for an input image;
From the result of the subject area determination means, a first image recognition area,
Means for detecting a second image recognition area other than the first image recognition area;
Based on the image evaluation value detected by the image quality evaluation value calculation means in the first image recognition area and the image evaluation value detected by the image quality evaluation value calculation means in the second image recognition area. First gradation correction amount determining means for determining a tone correction amount;
Color temperature difference detection means for detecting a color temperature of the first image recognition area and the second image recognition area and calculating a color temperature difference;
Second gradation characteristic determining means for determining a second gradation correction amount based on the color temperature difference and the gradation characteristic determined by the first gradation correction amount determining means;
Have
The image processing apparatus further comprises means for correcting the previous image gradation based on the gradation correction amount determined by the second gradation correction amount determining means.

  According to the image processing apparatus of the present invention, in setting a gradation characteristic for each area and performing correction, in a scene shot at a different color temperature, the coloring generated at the boundary portion for each area is reduced. Thus, a preferable image having a wide dynamic range and high contrast can be generated.

The block diagram which shows the apparatus structure of embodiment which concerns on this invention. Block diagram showing a part of the configuration of the image processing unit Block diagram showing the configuration of the exposure calculation unit Flow chart showing processing for calculating gradation correction amount Block diagram showing the configuration of the gain processing unit Flow chart showing exposure calculation processing Flow chart showing gradation processing by image processing unit Flow chart showing processing by gradation characteristic calculation unit The figure explaining the target exposure amount and the determination method of the exposure amount in addition to the luminance step between regions The figure explaining the calculation method of the brightness | luminance level difference between area | regions Conceptual diagram showing the relationship between the color temperature on the blackbody radiation axis and the WB evaluation value Diagram explaining input / output characteristics to be realized in each area The figure explaining the calculation method of HUMAN_POINT, BACK_POINT, SKY_POINT Diagram explaining input / output characteristics to be realized in each area and dynamic range priority and contrast priority gradation characteristics The figure explaining the range of the input signal which adapts the gradation correction amount for every area The figure explaining the effect acquired by using the brightness | luminance level difference between area | regions as the weighted addition coefficient of two types of gradation characteristics The figure explaining the conversion from a gradation characteristic to a gain table The figure explaining the gradation compression processing by γ characteristics The flowchart which showed the flow of the 2nd gradation correction amount calculation means Example of linear interpolation coefficient based on face size, adjacent block, and color temperature difference Timing chart showing timing of image acquisition for calculating strobe light emission area in the second embodiment Example of calculating the strobe emission area in the second embodiment

  Hereinafter, embodiments for carrying out the present invention will be described in detail. The embodiment described below is an example for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

[Example 1]
Hereinafter, an embodiment in which the image processing apparatus of the present invention is applied to an imaging apparatus represented by, for example, a digital still camera or a digital video camera will be described. With reference to FIG. 1, an outline of a configuration and functions of an imaging apparatus according to an embodiment of the present invention will be described.

  In the present embodiment, optimum gradation processing is performed using information on the image area. In the present embodiment, a gradation process is described assuming a scene of a backlit person scene having the highest effect of the present invention and assuming a scene where the person is darkest and the sky is brightest. However, it goes without saying that the technical idea of the present invention can be applied to other than the above scenes.

  In FIG. 1, the optical system 101 includes a lens group including a zoom lens and a focus lens, a diaphragm adjusting device, and a shutter device. The optical system 101 adjusts the magnification, focus position, or incident light quantity of the subject image formed on the imaging unit 102. The imaging unit 102 includes an image sensor including a photoelectric conversion element such as a CCD or a CMOS that photoelectrically converts a light beam of a subject that has passed through the optical system 101. The A / D conversion unit 103 converts the video signal input from the imaging unit 102 into a digital image.

  The image processing unit 104 performs gradation processing described later in addition to normal signal processing. The image processing unit 104 can perform similar image processing not only on the digital image signal output from the A / D conversion unit 103 but also on the image data read from the recording unit 110.

  The exposure amount calculation unit 105 calculates an exposure amount at the time of shooting in order to obtain an optimal input image for performing the gradation processing of the present embodiment. The exposure amount calculation unit 105 inputs the processing result obtained by the image processing unit 104 and outputs the calculated exposure amount to the exposure amount control unit 106. Details of the exposure amount calculation unit 105 will be described later. The exposure amount control unit 106 controls the optical system 101 and the imaging unit 102 according to the exposure amount calculated by the exposure amount calculation unit 105, and controls the aperture, shutter speed, and analog gain of the imaging unit 102.

  The system control unit 107 controls the overall operation of the apparatus according to the present embodiment. Based on the luminance value obtained from the image signal processed by the image processing unit 104 and the instruction signal transmitted from the operation unit 108, drive control of the optical system 101 and the imaging unit 102 is also performed. The display unit 109 is configured by a liquid crystal display or an organic EL (Electro Luminescence) display, and displays the image signal generated by the imaging unit 102 and the image data read from the recording unit 110.

  The recording unit 110 has a function of recording image data, and may include, for example, an information recording medium using a package containing a rotating recording body such as a memory card on which a semiconductor memory is mounted or a magneto-optical disk. The information recording medium may be detachable. A bus 111 is a transmission path for transmitting image signals and control signals among the image processing unit 104, the system control unit 107, the display unit 109, and the recording unit 110.

  Next, the configuration and function of the exposure amount calculation unit 105 will be described with reference to FIG. The exposure amount calculation unit 105 calculates an exposure amount necessary for obtaining an image signal photographed with an optimum exposure amount for performing gradation processing. In the present embodiment, image area information is also used for calculating the exposure amount. In the following, as the input information to the exposure amount calculation unit 105, the output result of the area-specific luminance value calculation unit 203 of the image processing unit 104 shown in FIG. 2 is used. The configuration and function of the image processing unit 104 will be described later.

  The exposure amount calculation unit 105 receives the output result of the region-specific luminance value calculation unit 203 and performs processing by the main subject output luminance value calculation unit 301 and the inter-region luminance level calculation unit 302, and then the exposure amount determination unit 303. Then, the exposure amount is determined and output to the exposure amount control unit 106.

  FIG. 6 shows processing from acquisition of an image signal for calculating the exposure amount to calculation of the exposure amount by the exposure amount calculation unit 105. Processing from S601 to S603 is performed by the image processing unit 104, S604. The process from step S606 to step S606 is performed by the exposure amount calculation unit 105. 6 is realized by the system control unit 107 developing a program stored in the nonvolatile memory in the work area of the volatile memory and controlling the above-described units.

  In step S601, the image processing unit 104 acquires an image signal captured by the imaging unit 102 in order to calculate an exposure amount. In step S602, the image processing unit 104 recognizes a region from the image signal acquired in step S601. In the present embodiment, a total of four areas are recognized as areas of a person's face area, a body area other than the person's face, a sky area including clouds and the sun, and other areas as background areas. FIG. 9A illustrates the result of area recognition performed on the input image. As the region recognition method, a known method such as object recognition using learning data by a neural network may be used (Japanese Patent Laid-Open No. 2006-39666). In addition, the region recognition method may be a method of recognizing every block divided by a certain size, not every pixel.

  In S603, the image processing unit 104 calculates a representative luminance value for each region using the region recognition result in S602. The representative luminance value is an average value of luminance signals in the corresponding area or a weighted average value by weighting according to coordinates. As areas for calculating the representative luminance values, three areas are calculated: a human face area, an empty area, a human face area, and a background area other than the empty area.

  Hereinafter, the representative luminance values of the respective regions are referred to as HUMAN_Y, BACK_Y, and SKY_Y, respectively. In this embodiment, the representative luminance values of the human face area and sky area are calculated from the average value of the luminance signals in the corresponding area, and the representative luminance value of the background area is weighted by weighting according to the coordinates. Calculate from the average value. In step S604, the exposure amount calculation unit 105 calculates a luminance step between regions using the representative luminance value of each region calculated in step S603. Hereinafter, the calculated output luminance value of the face area of the person who is the main subject is referred to as HUMAN_OUT_Y. Details will be described later.

  In S605, the exposure amount calculation unit 105 calculates the output luminance value of the main subject using the input image and the representative luminance value of each area calculated in S603. Hereinafter, it is assumed that the luminance step between the human face region and the background region is ΔHB, the luminance step between the background region and the sky region is ΔBS, and the luminance step between the human face region and the sky region is ΔHS.

  In step S606, the exposure amount calculation unit 105 receives the luminance steps ΔHB, ΔBS, and ΔHS between the areas calculated in step S604 and the output luminance value HUMAN_OUT_Y of the main subject calculated in step S605, and is optimal for performing gradation processing. An exposure amount for taking an image is determined. Details will be described later.

<Luminance level calculation method>
Next, the calculation of the luminance step between the areas in S604 will be described with reference to FIG. FIG. 9B illustrates the target exposure amount of each region based on the exposure amount of the input image. The target exposure amount indicates how many steps over and under the exposure amount should be set from the exposure amount of the input image in order to set the representative luminance value of the calculation target region to the target luminance value. .

  Overexposure means to make it larger than the exposure amount of the input image, and underexposure means to make it smaller than the exposure amount of the input image. The target luminance value is a value determined in advance by each area.

  The representative luminance values of each area are HUMAN_Y, BACK_Y, and SKY_Y, and the target luminance values of each area are HUMAN_ref_Y, BACK_ref_Y, and SKY_ref_Y. The target exposure amounts ΔBV_HUMAN_ref, ΔBV_BACK_ref, and ΔBV_SKY_ref for each region are calculated by Equation 1.

ΔBV_HUMAN_ref = LOG2 (HUMAN_ref_Y / HUMAN_Y)
ΔBV_BACK_ref = LOG2 (BACK_ref_Y / BACK_Y)
ΔBV_SKY_ref = LOG2 (SKY_ref_Y / SKY_Y)
... (1)
A luminance level difference between the areas is calculated from the target exposure amount of each area calculated by Expression 1. ΔHB, ΔBS, and ΔHS are calculated from Equation 2, respectively.

ΔHB = ABS (ΔBV_HUMAN_ref−ΔBV_BACK_ref)
ΔBS = ABS (ΔBV_BACK_ref−ΔBV_SKY_ref)
ΔHS = ABS (ΔBV_HUMAN_ref−ΔBV_SKY_ref)
... (2)
The luminance steps ΔHB, ΔBS, and ΔHS between the areas calculated by Expression 2 are used in the process of determining the exposure amount in S606.

<Target luminance value calculation method>
Next, a method for calculating the output luminance value of the main subject in S605 will be described.

  The output luminance value of the main subject is different from the above-described target luminance value, and indicates the luminance value of the main subject region that is finally desired to be output in a later captured image. In the present embodiment, in the case of a scene including a person, the output luminance value of the face area of the person is calculated in order to consider the person as the main subject. In this embodiment, as a method for calculating the output luminance value, a known method for determining the final luminance value of the human face region by using the relationship between the luminance value of only the human face region and the luminance value of the entire image is used. (For example, Japanese Patent No. 4778976).

  Note that the present invention is not limited to this method, and a method of determining the luminance value of the person's face region using a luminance step between the regions with reference to the subject located at the intermediate luminance may be used. Further, the target luminance value of the person's face area may be used as the output luminance value as it is.

<Exposure amount determination method>
Next, the method for determining the exposure amount of the main subject in S606 will be described with reference to FIG. FIG. 9C illustrates the output exposure amount of each region with respect to the exposure amount of the input image signal. The output exposure amount indicates how many steps over and under the exposure amount should be set with respect to the exposure amount of the input image in order to realize the brightness of each area to be finally output. It is. Assuming that the output exposure amounts of the human face region, background region, and sky region are ΔBV_HUMAN, ΔBV_BACK, and ΔBV_SKY, respectively, the output exposure amount of each region is calculated from Equation 3.

ΔBV_HUMAN = LOG2 (HUMAN_OUT_Y / HUMAN_Y)
ΔBV_BACK = ΔBV_HUMAN−ΔHB / 2
ΔBV_SKY = ΔBV_HUMAN−ΔHS / 2
... (3)
As shown in Equation 3, the main subject is maintained so as to maintain the level difference between the areas calculated in S604 by ½, based on the exposure amount of the main object determined by the output luminance value calculated in S605. The exposure amount of the area other than is determined.

  The output exposure amount having the smallest exposure amount among the output exposure amounts of the respective regions is determined as the exposure amount for photographing the optimum image for performing the gradation processing. That is, among each region, ΔBV_SKY of the empty region that is the smallest output exposure amount is determined as the exposure amount at the time of photographing. In this embodiment, the exposure amount at the time of shooting is determined by the method described above. In addition to this method, the exposure amount can be reduced, for example, the exposure amount can be reduced only to a certain value with respect to the output exposure amount of the main subject. An upper limit may be provided to determine the exposure amount at the time of shooting.

  As described above, the exposure amount calculation unit 105 acquires an input image and determines an optimum exposure amount.

<Tone processing>
Next, gradation processing by the image processing unit 104 will be described with reference to FIGS.

  The image processing unit 104 performs gradation processing by inputting an image signal photographed with the exposure amount determined by the exposure amount calculation unit 105.

  Note that the gradation processing according to the present embodiment applies gradation characteristics that reduce a luminance step between regions to the entire image. In addition, the main subject area is given the highest priority, and an area other than the main subject is assigned with a well-balanced gradation in accordance with the scene, thereby generating an output image with contrast and brightness as seen by humans.

  FIG. 2 shows a part of the configuration of the image processing unit 104, and FIG. 7 shows gradation processing. 7 is realized by the system control unit 107 expanding the program stored in the nonvolatile memory in the work area of the volatile memory and controlling the image processing unit. In step S <b> 701, the image processing unit 104 acquires an image signal captured by the imaging unit 102. In step S702, the image processing unit 104 recognizes a region for the image signal acquired in step S701. This region recognition processing is the same as the processing in S602 described above.

  In S703, the image processing unit 104 calculates a representative luminance value for each area recognized in S702, using the image signal acquired in S701 as input information. The calculation method of the representative luminance value of each area is the same as that in S603 described above. In step S704, the image processing unit 104 calculates a histogram for each area recognized in step S702, using the image signal acquired in step S701 as input information. In the present embodiment, a histogram of the luminance signal is calculated. The calculation target areas of the histogram are a human face area, a sky area, and a background area, as in the calculation of the representative luminance value.

  In step S <b> 705, the image processing unit 104 calculates a first gradation correction amount for each region using the representative luminance value calculated for each region as input information. In S706, the color temperature is detected for the main subject area and the background area. As a method of detecting the color temperature of the main subject area, the color of the face area is analyzed based on the face detection result. Details will be described later. In step S707, it is determined whether or not the main subject area and the non-main subject area are adjacent to each other. The determination method is to divide the image area into multiple blocks, label the area discrimination results for each block area, and count the number of blocks where the main subject area and non-main subject area labels are adjacent to adjacent blocks. Just do it.

  In S708, the image processing unit 104 determines the second gradation correction amount using the gradation correction amount for each region calculated in S707, the color temperature difference calculated in S706, and the region adjacency determination result calculated in S708. . A method for calculating the gradation correction amount will be described later. In step S709, tone characteristics are calculated based on the histogram for each region calculated in step S704 and the second tone correction amount calculated in step S708. Details will be described later.

  Note that the gradation characteristic calculated in S708 is an input / output characteristic that represents an output signal corresponding to an input signal, with the horizontal axis representing the input signal and the vertical axis representing the output signal, as shown in FIG. For example, when the gradation characteristics are not changed, the graph is indicated by a dotted line, and the values of the input signal and the output signal are equal. In the present embodiment, the input signal and output signal in FIG. 17A indicate luminance signals.

  In S710, the image processing unit 104 uses the tone characteristics calculated in S708, and the gain indicating the gain according to the input signal with the horizontal axis as the input signal and the vertical axis as the gain as shown in FIG. 17B. Convert to a table (gain table conversion process). In the present embodiment, the input signal in FIG. 17B represents a luminance signal. When the input signal is x and the output signal corresponding to the input signal is represented by y according to the gradation characteristics, the gain Gain is calculated by Equation 4.

Gain = y / x (4)
In step S <b> 711, the image processing unit 104 refers to the gain table obtained in step S <b> 706 and performs a process of applying gain to the captured image.

  Here, gain processing by the gain processing unit 206 will be described with reference to FIG. When the pixel located at the upper left is (0, 0) and the pixel position of the input image is (x, y), the gain output from the input signal Yin (x, y) is Gain (x, y). When the gain table obtained in S706 is expressed as a GainTbl function, Equation 10 is obtained.

Gain (x, y) = GainTbl (Yin (x, y)) (5)
Gain (x, y) is a gain corresponding to the pixel located at (x, y) of the input image.

  Yin, which is an input signal for calculating the gain, is generated by the gain calculation image generation unit 501 using the captured input image. As an image generation method for gain calculation, in order to improve contrast, an input image is converted into luminance image data and an averaging process is performed. By calculating the gain using the luminance image data as an input, an effect of further improving the contrast can be obtained. The gain converter 502 uses the Yin signal generated by the gain calculation image generator 501 to perform a process of converting into a gain signal as shown in Expression 17.

  The gain multiplier 503 performs a process of multiplying the input image signal by the gain using the gain calculated by the gain converter 502. In the present embodiment, a process of multiplying an R, G, B image signal that has already undergone demosaicing processing from the captured input image by a gain is performed. The R, G, and B signals of the input image positioned at (x, y) before gain multiplication are Rin (x, y), Gin (x, y), and Bin (x, y), respectively. In that case, the R, G, B signals Rout (x, y), Gout (x, y), and Bout (x, y) after gain multiplication are calculated by Equation 6 using the GainTbl function of Equation 10. Can do.

Rout (x, y) = Rin (x, y) × GainTbl (Yin (x, y))
Gout (x, y) = Gin (x, y) × GainTbl (Yin (x, y))
Bout (x, y) = Rin (x, y) × GainTbl (Yin (x, y))
... (6)
As in Expression 6, if the pixel position is (x, y), the gain applied to the R, G, and B signals is the same.

  In the present embodiment, the gain processing is performed on the RGB signal, but the above-described gain processing may be performed on the YUV signal.

  In step S712, the image processing unit 104 performs gradation compression processing with the γ characteristic to match the R, G, and B m-bit signals that have been subjected to the gain processing in step S711 to the characteristics of the display unit 109, and n (m ≧ n) A bit signal is output. In this embodiment, as the γ characteristic, two types of characteristics indicated by a solid line and a dotted line are used as shown in FIG. The γ characteristic indicated by the dotted line is a characteristic that improves the contrast of the low-brightness part as a measure for reducing the contrast of the face area of the person in the backlight due to the influence of flare or the like. The γ characteristic indicated by the dotted line is used for a human scene at the time of backlighting, and the γ characteristic indicated by a solid line is used for other than the human scene at the time of backlighting.

  In the tone processing of FIG. 7, the image processing unit 104 shown in FIG. 2 includes the area detection units 201 and 213 in S702, the region-specific histogram calculation unit 202 in S704, and the region-specific luminance value calculation unit 203 in S703. Perform each process.

  Also, the first gradation correction amount calculation unit 208 performs S705, the color temperature detection unit 209 and the color temperature difference detection unit 210 perform S706, and the area adjacency determination unit 212 performs S707. Then, the second gradation correction amount calculation unit 14 performs the process of S708. The gradation characteristic calculation unit 204 performs S709, the gain table calculation unit 205 performs S710, the gain processing unit 206 performs S711, and the gradation compression processing unit 207 performs S712.

<Subject area color temperature detection method>
Next, the detection of the color temperature for the main subject area and the background area performed in S706 will be described.

  In this embodiment, the color temperature detection method for the main subject area is performed by analyzing the hue of the face area based on the face detection result, and using the result of normal auto white balance control for the background area. .

  The auto white balance control method can automatically detect white parts from the captured image, determine the white balance coefficient from the average value of each color component on the entire screen, and apply the calculated white balance coefficient to the entire screen. It is common. Conventional auto white balance control will be described. The signal output from the image sensor is digitized by A / D conversion and divided into a plurality of blocks.

  Each block is composed of RGB color pixels, and color evaluation values (Cx [i], Cy [i]) are obtained for each block, for example, by the following calculation formula.

Cx [i] = (R [i] −B [i]) / Y [i] × 1024
Cy [i] = (R [i] + B [i] -2G [i]) / Y [i] × 1024
... (7)
(Where i is a block number, R [i], G [i], and B [i] are average values of RGB pixels included in the block i, Y [i] = (R [i] + 2G [i] + B [I] / 4))
Determine that the block is white. Then, integral values SumR, SumG, and SumB of the color pixels included in the block are calculated, and white balance coefficients (WBCo_R, WBCo_G, WBCo_B) are calculated as in the following equations.

WBCo_R = SumY × 1024 / sumR
WBCo_G = SumY × 1024 / sumG
... (8)
WBCo_B = SumY × 1024 / sumB
However, SumY = (sumR + 2 × sumG + SumB) / 4
Regarding the relationship between the white balance coefficient and the color temperature, the relationship between the black body radiation axis and the WB coefficient is calculated in advance, and the color temperature is calculated from the WB coefficient.

  A graph a) and a table b) shown in FIG. 11 are black body radiation axes (hereinafter referred to as white axes), and show the relationship between the color temperature and the CxCy value. The correspondence between the color temperature and the CxCy value can be clarified by acquiring image data with a camera with a known color temperature light source on the blackbody radiation axis in advance and calculating CxCy. Of course, not only in the color temperature direction but also in the green direction and the magenta direction of the white fluorescent lamp described above, it is possible to clarify the correspondence relationship with the CxCy value.

  Further, by acquiring the CxCy value corresponding to the above-described flash emission color temperature, if the flash emission color temperature is known, the WB correction value for the flash light source can be calculated. The method for calculating the WB coefficient based on the face area is disclosed in detail in Japanese Patent No.

  In the sky region, the color temperature is determined to be high because the bluish color is strong. The color temperature value of the sky region may be determined in advance, or the color temperature may be determined in accordance with subject luminance (hereinafter referred to as Bv value) and block average value. In the latter case, the CxCy values for each block are averaged, and the color temperature may be determined by performing clipping in a predetermined range according to the Bv value as shown in FIG. 11c).

<First Gradation Correction Amount Calculation Method>
FIG. 4 shows the first gradation correction amount calculation process. 4 is realized by the system control unit 107 developing a program stored in the nonvolatile memory in the work area of the volatile memory and controlling each unit.

  In step S <b> 401, the image processing unit 104 calculates a luminance step between regions using the representative luminance value calculated for each region. As for the method of calculating the luminance step, as shown in FIG. 10A, the exposure amount may be replaced with the gain amount in the process of S604. In S402, the image processing unit 104 calculates the output luminance value of the main subject. The output luminance value of the main subject is the luminance value of the main subject area to be finally output, as in S605. In the present embodiment, the person's face area is set as the main subject area as described above, and the calculation method is the same as that in step S605.

  In S403, the image processing unit 104 calculates the first gradation correction amount of each region using the luminance step between the regions calculated in S401 and the output luminance value of the main subject calculated in S402 as input information. Note that the first tone correction amount indicates a uniform gain amount in the present embodiment, and the uniform gain amount desired to be applied to each target region is calculated. As for the calculation method of the gradation correction amount for each region, as shown in FIG. 10B, the exposure amount may be replaced with the gradation correction amount in S606.

  The calculation method of the first gradation correction amount of each area is calculated by Expression 4 when the gain amount of the human face area, the gain amount of the background area, and the gain amount of the sky area are GAIN_HUMAN, GAIN_BACK, and GAIN_SKY, respectively. The HUMAN_Y is the representative luminance value of the person's face area, HUMAN_OUT_Y is the output luminance value of the person's face area as the main subject, ΔHB is the luminance step between the person's face area and the background area, and ΔHS is the person's face area. And the luminance step between the sky regions.

GAIN_HUMAN = LOG2 (HUMAN_OUT_Y / HUMAN_Y)
GAIN_BACK = GAIN_HUMAN−ΔHB / 2
GAIN_SKY = GAIN_HUMAN−ΔHS / 2
... (9)

  In this embodiment, since the gain amount is not less than 0 stage, after calculating GAIN_HUMAN, GAIN_BACK, and GAIN_SKY with Expression 7, if GAIN_HUMAN, GAIN_BACK, and GAIN_SKY have negative values, clip to 0 To do. The first gradation correction amount for each region is represented by input / output characteristics as shown in FIG. Assuming that the input signal is X and the output signal is Y, the input / output characteristics to be realized in each region are obtained by Equation 10.

Input / output characteristics of human face area: Y = 2 ^(GAIN_HUMAN) × X
Input / output characteristics of background area: Y = 2 ^(GAIN_BACK) × X (10)
Input / output characteristics of empty area: Y = 2 ^(GAIN_SKY) × X
The input / output characteristics of each area are gradation characteristics applied to each area. The slope of the input / output characteristics of each region is the contrast desired to be realized in each region, and the output signal represents the brightness desired to be realized in each region.

<Second Gradation Correction Amount Calculation Method>
Next, the second gradation correction amount calculation method in S708 will be described. In step S708, the image processing unit 104 inputs the first gradation correction amount calculated for each region calculated in step S705, the color temperature difference calculated in step S706, and the adjacency determination result calculated in step S707. A gradation correction amount is calculated. Note that the gain amount of the human face region, the gain amount of the background region, and the gain amount of the sky region are GAIN_HUMAN_2, GAIN_BACK_2, and GAIN_SKY_2, respectively.

  FIG. 19 is a flowchart showing the flow of the second gradation correction amount calculating means in the present embodiment. In step S901, it is determined whether the person area and the background area are adjacent to each other. If they are adjacent to each other, S902 determines whether the color temperature difference between the person area and the background area is greater than a predetermined value. In step S903, it is determined whether the face area is large.

In S904, if the color temperature difference is large in S901-S903 and the face size is greater than or equal to a predetermined value set in advance,
GAIN_HUMAN_2 = GAIN_HUMAN
GAIN_BACK_2 = GAIN_HUMAN (11)
Set to.

In S905, when the color temperature difference is large and the face size is equal to or smaller than a predetermined value set in advance,
GAIN_HUMAN_2 = 2 = GAIN_BACK
GAIN_BACK_2 = GAIN_BACK (12)
And set.

If the color temperature difference is small or there is no adjacent area in S901 and S902, GAIN_HUMAN_2 = GAIN_HUMAN
GAIN_BACK_2 = GAIN_BACK (13)
And set.

  Further, the steps of S901 to S905 can be substituted by the following equation. The adjacency degree interpolation coefficient α [0-1] determined to be larger as the degree of adjacency increases in a predetermined range, and the color temperature difference linear interpolation coefficient β determined to be larger as the color temperature difference increases to a predetermined range. [0-1] The face size linear interpolation coefficient, which is determined to increase as the face size increases within a predetermined range, is set to γ [0-1], and the gain for person and background is determined as follows. May be.

GAIN_HUMAN_2 =
α × β × (γ × GAIN_HUMAN + (1-γ) × GAIN_BACK)) +
(1-α × β) × (GAIN_HUMAN)
GAIN_BACK_2 =
α × β × (γ × GAIN_HUMAN + (1-γ) × GAIN_BACK)) +
(1-α × β) × (GAIN_BACK) (14)

  FIG. 21 is an example of a linear interpolation coefficient based on a face size / adjacent block and a color temperature difference. In S906, it is determined whether the background area is adjacent to the sky area. In S907, it is determined whether the difference in color temperature between the background area and the sky area is large. In S908, if the color temperature difference is large and adjacent, the gain amount in the sky area is set as the gain amount in the background area. Make the same setting.

GAIN_SKY_2 = GAIN_BACK_2 (15)
Of course, as in Equation 14, the degree of adjacency interpolation coefficient α [0-1], which is determined to be larger as the degree of adjacency increases in a predetermined range, is determined to be a larger value as the color temperature difference increases to a predetermined range. The gain for the sky region may be determined as follows as the color temperature difference linear interpolation coefficient β [0-1].

GAIN_SKY_2 =
α × β × (GAIN_SKY_2) + (1−α × β) × (GAIN_BACK)
... (16)
Further, when the input signal is X and the output signal is Y, the input / output characteristics to be realized in each region can be obtained by Expression 17.

Input / output characteristics of human face area: Y = 2 ^{(GAIN_HUMAN_2)} × X
Input / output characteristics of background area: Y = 2 ^{(GAIN_BACK_2)} × X (17)
Input / output characteristics of empty area: Y = 2 ^(GAIN_SKY_2) × X

<Method of calculating gradation characteristics>
Next, the gradation characteristic calculation method in S709 will be described. With reference to FIG. 8, a description will be given of processing for creating gradation characteristics to be applied uniformly on the screen with reference to input / output characteristics for each region, regarding gradation characteristic calculation processing by the gradation characteristic calculation unit 204. In step S804, the gradation characteristic calculation unit 204 determines a range of the input signal to which the second gradation correction amount for each region calculated in step S708 is applied, using the luminance histogram for each region.

  The three input / output characteristics shown in FIG. 15 are input / output characteristics that are desired to be realized in each region shown in Expression 17. HUMAN_POINT, BACK_POINT, and SKY_POINT on the horizontal axis of the input / output characteristics serve as an index for determining the range of the input signal to which the input / output characteristics of each area are applied. The input signal range from 0 to HUMAN_POINT is shown as (1), the input signal range from 0 to BACK_POINT is shown as (2), and the input signal range from SKY_POINT to the maximum value that can be expressed by the input signal is shown as (3). Yes. (1) is the input signal range to which the human face area input / output characteristics are to be applied, (2) is the input signal range to which the background area input / output characteristics are to be applied, (3) is the sky area input / output characteristics Each of the input signal ranges desired is shown.

  A method for calculating HUMAN_POINT, BACK_POINT, and SKY_POINT will be described with reference to FIG. The calculation of HUMAN_POINT, BACK_POINT, and SKY_POINT uses a luminance histogram calculated for each region as shown in FIG. The calculation of HUMAN_POINT uses the luminance histogram of the human face area, the calculation of BACK_POINT uses the luminance histogram of the background area, and the calculation of SKY_POINT uses the luminance histogram of the sky area.

  FIG. 13B illustrates a method for calculating HUMAN_POINT, BACK_POINT, and SKY_POINT using the luminance histogram of each region. First, calculation of HUMAN_POINT will be described.

  In the present embodiment, the frequency is added to the luminance histogram of the human face area from the smallest signal value to the gradually larger signal value. Then, the value of the input signal obtained by adding the frequency at the end when the added value is equal to or greater than a predetermined threshold is defined as HUMAN_POINT. If the threshold is HUMAN_TH, it can be calculated by Equation 18.

HUMAN_TH = HUMAN_AREASUM × P_HUMAN (18)
(However, 0.0 <P_HUMAN ≦ 1.0)
HUMAN_AREASUM is the total number of pixels recognized as a human face area, and P_HUMAN is an adjustment parameter. As shown in Expression 18, HUMAN_POINT is determined based on whether the addition value of the frequency reaches a predetermined threshold (%) with respect to the total number of pixels in the human face area.

  Next, calculation of BACK_POINT will be described. In the present embodiment, as with HUMAN_POINT, the frequency is added to the luminance histogram of the background area from the smallest signal value to the gradually larger signal value. Then, when the added value becomes equal to or greater than a predetermined threshold, the value of the input signal obtained by adding the frequency at the end is defined as BACK_POINT. The threshold is set by the same method as HUMAN_TH.

  Finally, calculation of SKY_POINT will be described. In the present embodiment, contrary to HUMAN_POINT and BACK_POINT, the frequency is subtracted from the total number of pixels recognized as the sky area from the minimum signal value to the gradually larger signal value with respect to the brightness histogram of the sky area. . Then, when the subtraction value becomes equal to or less than a predetermined threshold value, the value of the input signal obtained by subtracting the frequent number at the end is defined as SKY_POINT. The threshold is set by the same method as HUMAN_TH.

  As described above, the calculation methods are different for the sky region that is a high-luminance subject, the background region that is a medium-luminance subject and the low-luminance subject, and the human face region. In this embodiment, the scene in which the sky area is the brightest and the human face area is the darkest has been described. However, when the background area is the brightest subject, the same calculation method as SKY_POINT is used. In some cases, the same calculation method as BACK_POINT is used.

  In S805 and S806, the gradation characteristic calculation unit 204 creates gradation characteristics having different characteristics. S805 and S806 are the second gradation correction amounts HUMAN_GAIN_2, BACK_GAIN_2, and SKY_GAIN_2 for each area calculated in S708, and HUMAN_POINT, BACK_POINT, and SKY_POINT that are indices of ranges to which the gradation correction amounts for each area calculated in S804 are applied. Is used to create gradation characteristics.

  In the present embodiment, a plurality of (two types) gradation characteristics having different characteristics are created, and the final gradation characteristics are calculated by weighted addition of the two created gradation characteristics.

  The three input / output characteristics shown in FIG. 14A are input / output characteristics that are desired to be realized in each region shown in Expression 17. HUMAN_POINT, BACK_POINT, and SKY_POINT on the horizontal axis of the input / output characteristics are indicators for determining the input signal range to which the input / output characteristics of each area are applied. In contrast to FIG. 14A, in this embodiment, there are two types of gradation characteristics called dynamic range priority shown in FIG. 14B and gradation characteristics called contrast priority shown in FIG. create.

  The two types of gradation characteristics created in S805 and S806 will be described in detail with reference to FIG. First, the dynamic range priority gradation characteristic shown in FIG. 14B will be described. The characteristic of the gradation characteristics with priority on the dynamic range is that the input / output characteristics of the human face area and the input / output characteristics of the sky area shown in FIG. The characteristic of FIG. 14A is that the brightness and contrast of the human face area and the sky area are realized.

  The section on the low brightness side from 0 to HUMAN_POINT of the input signal has the input / output characteristics of the human face area. Conversely, the section on the high brightness side from SKY_POINT to the maximum value that can be represented by the input signal is an empty area. Input / output characteristics. The section located at the intermediate luminance from HUMAN_POINT to SKY_POINT connects the input / output characteristics of the human face area on the low luminance side and the input / output characteristics of the sky area on the high luminance side. If the input signal is X and the output signal is Y, the dynamic range priority gradation is expressed by Equation 19.

Y = 2 ^{(GAIN_HUMAN_2)} × X (0 ≦ X <HUMAN_POINT)
Y = (2 ^(GAIN_SKY_2) * SKY_POINT-2 ^{(GAIN_HUMAN_2)} * HUMAN_POINT) * X / (SKY_POINT-HUMAN_POINT) +2 ^(GAIN_HUMAN) * HUMAN_POINT <HUMINT_POINT <X
Y = 2 ^(GAIN_SKY_2) × X (SKY_POINT ≦ X)
... (19)
In the present embodiment, the input / output characteristics are such that the high luminance side is saturated, but the input / output characteristics are set so as not to saturate the high luminance side as much as possible, as shown by the dotted line and the curve in FIG. Also good. In the present embodiment, the input / output characteristics are created with a broken line, but the input / output characteristics may be created with a curve approximating the broken line. The dynamic range priority gradation characteristics are created in S805.

  Next, the contrast priority gradation characteristics shown in FIG. 14C will be described. A feature of the contrast-priority gradation characteristic is that only the slopes of the input / output characteristics of the background area and the sky area are incorporated in addition to the input / output characteristics of the human face area shown in FIG. The characteristic of FIG. 14A is that the face area of a person realizes brightness and contrast, and the background area and sky area realize contrast.

  Among the input signals, the section from 0 to HUMAN_POINT is equivalent to the dynamic range priority gradation characteristic in that the input / output characteristic of the human face area is used. In the section from HUMAN_POINT to BACK_POINT, only the slope of the input / output characteristics of the background area is realized. In the section from BACK_POINT to the maximum value that can be expressed by the input signal, only the slope of the input / output characteristics of the empty area is realized. It has become. When the input signal is X and the output signal is Y, the contrast priority gradation is expressed by Equation 20.

Y = 2 ^{(GAIN_HUMAN_2)} × X (0 ≦ X <HUMAN_POINT)
Y = 2 ^{(GAIN_BACK_2)} × X + 2 ^{(GAIN_HUMAN_2)} × HUMAN_POINT
(HUMAN_POINT ≦ X <BACK_POINT)
Y = 2 ^(GAIN_SKY_2) × X + 2 ^{(GAIN_BACK_2)} × (BACK_POINT−HUMAN_POINT) +2 ^{(GAIN_HUMAN_2)} × HUMAN_POINT
(BACK_POINT ≦ X)
... (20)

  In the present embodiment, the input / output characteristics are such that the high luminance side is saturated, but the input / output characteristics are set so as not to saturate the high luminance side as much as possible, as shown by the dotted line and the curve in FIG. Also good. In the present embodiment, the input / output characteristics are created with a broken line, but the input / output characteristics may be created with a curve approximating the broken line. The contrast priority gradation characteristics are created in S806.

  Although the two types of gradation characteristics of dynamic range priority and contrast priority have been described above, the common point of the two types of gradation characteristics is that both of the input / output characteristics of the human face area are realized. This is because the person's face area in the person scene in this embodiment is the main subject area, and therefore it is necessary to determine the tone characteristics with the highest priority.

  The method of creating two gradation characteristics is the same when there are only two regions. For example, in the case of a scene with only a sky, which is a person and a high-luminance subject, BACK_POINT and BACK_GAIN are replaced with SKY_POINT and SKY_GAIN, respectively, for the two gradation characteristics shown in FIG.

  As described above, the gradation characteristics corresponding to the two types of priority, the dynamic range priority and the contrast priority, are created by the processing of S805 and S806. The created two types of gradation characteristics are weighted and added in S807 to determine the gradation characteristics of the captured image.

  Here, with reference to FIG. 16, the gradation characteristic determination processing in S807 will be described in detail. As a method of performing weighted addition of gradation characteristics according to two types of priority, dynamic range priority and contrast priority, in the present embodiment, the luminance step between regions calculated in S801 is used (weighted addition coefficient calculation processing). Based on the background area, which is a subject located at intermediate luminance, the luminance step ΔHB between the human face area and the background area, and the luminance difference ΔBS between the background area and the sky area are used as weighted addition coefficients for two types of gradation characteristics.

  Assume that the input signal of the gradation characteristic is X and the output signal is Y, and the dynamic range priority gradation characteristic and the contrast priority gradation characteristic are expressed by a distance function and a contrast function, respectively, as shown in Equation 21.

Y = Drange (X)
Y = Contrast (X) (21)

When the gradation characteristic after weighted addition, that is, the gradation characteristic of the photographed image is expressed by the MIX_OUT function, Expression 15 is obtained.

Y = MIX_OUT (X)
= (ΔHB × Drange (X) + ΔBS × Contrast (X)) / (ΔHB + ΔBS) (22)
As shown in Equation 22, the luminance step ΔHB between the human face region and the background region and the luminance step ΔBS between the background region and the sky region are used as weighted addition coefficients for dynamic range priority and contrast priority, respectively.

  The effect obtained by using the luminance step ΔHB between the human face region and the background region and the luminance step ΔBS between the background region and the sky region for weighted addition will be described in detail with reference to FIG.

  The three input / output characteristics shown in FIG. 16A are input / output characteristics that are desired to be realized in each region shown in Expression 17. The three input / output characteristics shown in FIG. 16A are determined using luminance steps between regions. Since the luminance level difference between areas varies depending on the photographed scene, the input / output characteristics to be realized in each area also vary depending on the scene. The left side of FIG. 16 (b) shows an input to be realized in each area in a scene where the luminance step ΔBS between the background region and the sky region is extremely small compared to the luminance step ΔHB between the human face region and the background region. Output characteristics. Since the slope of the input / output characteristics to be realized in each area is determined by Equation 7, the input / output characteristics of the background area tend to approach the input / output characteristics of the sky area.

  In the above case, the input / output characteristics of the background area tend to be more easily realized when the dynamic range priority gradation characteristics shown on the right side of FIG. The reason for this is that when the luminance step ΔBS is extremely small, there is a tendency for many input signals in the background area to be distributed around and after SKY_POINT, and many input signals in the background area are in the dynamic range priority gradation characteristics after SKY_POINT. This is the case.

  In contrast to FIG. 16B, the left side of FIG. 16C shows that the luminance step ΔBS between the background region and the sky region is extremely different from the luminance step ΔHB between the human face region and the background region. This is an input / output characteristic that is desired to be realized in each area in a large scene. Since the slope of the input / output characteristics desired to be realized in each area is determined by Equation 17, the input / output characteristics of the background area tend to approach the input / output characteristics of the human face area.

  In the above case, the input / output characteristics desired to be realized in the background region tend to be more easily realized when the contrast-priority gradation characteristics shown on the right side of FIG. When the luminance step ΔBS is extremely large with respect to the luminance step ΔHB, there is a tendency that many input signals in the background region are distributed from around HUMAN_POINT to BACK_POINT, and the input / output characteristics of the human face region and the background region are close to each other. It becomes. As a result, the input / output characteristics desired to be realized in the background area tend to approach the contrast priority gradation characteristics of BACK_POINT or lower.

  As shown in Expression 17, the luminance difference ΔHB between the human face area and the background area and the luminance difference ΔBS between the background area and the sky area are used for weighted addition, and the three regions are photographed using the input / output characteristics desired to be realized. It is possible to efficiently create gradation characteristics optimal for an image.

  In the present embodiment, the gradation characteristic of the captured image is created using the luminance step between the areas from Expression 22, but as shown in Expression 23, different information is used with the weighted addition coefficients α and β. It may be realized. A value corresponding to the area ratio between regions, a luminance step between regions, or a fixed value independent of the scene may be given.

Y = MIX_OUT (X)
= (Α × Drange (X) + β × Contrast (X)) / (α + β)
(23)

For example, in a scene where the area where the background area exists is smaller than that of the sky area, the image may look better when the sky area is closer to the brightness and contrast desired to be realized. On the other hand, in a scene where the area where the sky region exists is smaller than that of the background region, there are cases where the image looks better when the background region is brought closer to the contrast desired to be realized. For this reason, gradation characteristics may be created by applying Equation 23, where α is the total number of pixels occupied by the sky region and β is the total number of pixels occupied by the background region.

  Also, in the case of a scene where there are only two subjects, it is better to calculate a weighted addition coefficient according to the value of the luminance step between the areas and weighted and add the two gradation characteristics. This is because, for example, when the scene is only a person and the sky, in a scene where the luminance step between the areas is originally small, it is better to prioritize the high contrast than the wide dynamic range. In addition, in a scene where the luminance difference between regions is originally large, it is better to give priority to the dynamic range than the contrast so that the image looks better. Therefore, the weighted addition coefficients α and β in Expression 16 are calculated by Expression 17. Δ is a luminance step between the two regions. For example, in the case of a scene with only a person and the sky, the value of the luminance step ΔHS between the face area of the person and the sky area becomes the value of Δ. The thresholds TH1 and TH2 are adjustment parameters.

α = 0.0, β = 1.0 (TH1 ≧ Δ)
α = 1.0 / (TH2−TH1) × (Δ−TH1), β = 1.0−α
(TH1 ≦ Δ <TH2)
α = 1.0, β = 0.0 (Δ ≦ TH2) (24)

  As described above, the weighted addition coefficients α and β of Expression 24 are not limited to the method of the present embodiment, and may be calculated using various information.

  In the above-described embodiment, the case where the present invention is applied to an imaging apparatus such as a digital video camera has been described as an example. However, the present invention is not limited thereto, and any apparatus that performs gradation processing using area information may be used. It can be applied to other devices.

  The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of this embodiment is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program. It is processing to do.

[Example 2]
Next, a second embodiment will be described. In the second embodiment, a case where strobe light is emitted during main exposure photography will be described. In the present embodiment, the second image recognition area is a strobe irradiation area.

  FIG. 21 is a timing chart for capturing a live image and a main light emission image at the time of main image capturing with strobe irradiation. Integration is performed for each region from the live image and the main flash image, and the integral value of the live image is calculated to match the main exposure by calculating the difference in exposure amount and sensitivity. The strobe irradiation amount can be acquired by performing a value difference calculation.

  FIG. 22 is a block diagram showing a flow for generating the strobe irradiation amount and the external light ratio from the live image and the main exposure image.

  From the block ratio in the outside light strobe light ratio generation means 405 for each area, the area where the strobe light exceeds a predetermined ratio is set as a strobe irradiation area, and the outside light block and strobe area are set as outside light block areas at a ratio equal to or less than a predetermined value. Can be labeled. The second image recognition area is set such that the strobe irradiation area is the main subject area and the outside light area is the non-main subject area.

  As in the first embodiment, the strobe area color temperature calculating means detects the color temperature from the strobe area and detects the color temperature, the light quantity ratio between the external light area and the strobe area, and the strobe color temperature. The difference between the color temperatures of the corresponding blocks may be calculated from the ambient light color temperature during live shooting. The color temperature difference calculating means may calculate a color temperature difference between adjacent blocks and set a maximum value, an average value, a median value, etc. of the color temperature difference as the color temperature difference value.

DESCRIPTION OF SYMBOLS 101 Optical system, 102 Imaging part, 103 A / D conversion part, 104 Image processing part

Claims (6)

An image processing apparatus that performs gradation correction of an image,
An input means for inputting an image;
Subject region determination means for determining a subject region for an input image;
First image recognition detection means for detecting a first image recognition area from the result of the subject area determination means;
Second image recognition area means for detecting a second image recognition area other than the first image recognition area;
First gradation correction amount determining means for determining a first gradation correction amount based on the image evaluation value detected by the image quality evaluation value calculating means in the first and second image recognition regions;
First color temperature detecting means for detecting a color temperature in the first image recognition area;
Second color temperature detection means for detecting a color temperature for the second image recognition region;
A color temperature difference detecting means for calculating a difference between the first color temperature and the second color temperature;
Second gradation characteristic determining means for determining a second gradation correction amount based on the color temperature difference and the gradation characteristic determined by the first gradation correction amount determining means;
Have
An image processing apparatus comprising: means for performing a previous image gradation correction based on the gradation correction amount determined by the second gradation correction amount determination means. The second gradation characteristic correction amount determination means compares the gradation characteristics determined by the first gradation correction amount determination means when the color temperature difference is large, and the gradation characteristic difference falls within a predetermined range. The image processing apparatus according to claim 1, wherein the gradation characteristic is determined to be Adjacent determination means for determining whether the second image recognition area is adjacent to the first image recognition area, and determining the second gradation correction amount when the adjacent determination means determines that the second image recognition area is adjacent. The image processing apparatus according to claim 1, wherein the gradation correction amount is determined by a unit. The first image area recognition means is a person area determination means for performing face recognition, and the person area determination means has face detection means for detecting the size of the face of the knowledge area, and the second tone characteristic correction The image processing apparatus according to claim 3, wherein the amount determining unit determines the correction amount using the correction amount calculated from the face area when the face size of the face detection is large. The first image area recognition means is a person area determination means for performing face recognition, and the person area determination means has face detection means for detecting the size of the face of the knowledge area, and the second tone characteristic correction The image processing apparatus according to claim 3, wherein the amount determining unit determines the correction amount using the correction amount calculated from the background region when the face size of the face detection is small. 6. The image processing apparatus according to claim 1, further comprising a strobe light emitting device, wherein the first image recognition area is an area determination unit that determines a strobe irradiation area.