JP6366432B2 - Image processing apparatus, control method therefor, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, a control method thereof, and a program, and more particularly to a gradation processing technique.

  2. Description of the Related Art Conventionally, gradation compression processing is known at the stage of outputting a signal whose input dynamic range is expanded, such as HDR (High Dynamic Range) composition and dodging. In Patent Document 1, an image is divided into a plurality of subject areas, a histogram for each subject area is calculated based on the representative pixel value and edge component calculated from each subject area, and the subject area is calculated based on the calculated histogram. A technique for generating a corresponding gradation conversion curve is disclosed.

JP 2010-130150 A

  However, in the technique disclosed in Patent Document 1, since a gradation conversion curve is generated without considering the brightness balance between subject areas, an image obtained such that the contrast decreases as the dynamic range is expanded. May become unnatural.

  Furthermore, in the technique disclosed in Patent Document 1, since the tone conversion curve is generated without considering the difference between the shooting scenes such as the sunset scene and the daytime scene, the atmosphere of the shooting scene is lost. There is. For example, by generating a gradation conversion curve that brightens the brightness of the background area of an image taken of a dim sunset scene to an appropriate value equivalent to that of a daytime scene, the dimness of the background area characteristic of the sunset scene The contrast between the sky area and the background area may be lost.

  The present invention has been made in view of the above-described problems of the prior art, and an image processing apparatus capable of performing gradation processing in consideration of a balance between subject areas characteristic of a shooting scene and a control method thereof As well as providing a program.

  In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is, a discriminating unit that discriminates a plurality of subject areas from the image, a degree calculating unit that calculates a degree indicating the possibility that the image is a predetermined shooting scene, based on a feature amount obtained from the image, and a plurality of subject regions Calculation means for calculating a gradation correction amount for each of a plurality of subject areas based on the luminance difference, a correction means for correcting the gradation correction amount based on the degree, and a gradation correction amount corrected by the correction means And a gradation processing means for applying to a plurality of subject areas.

  According to the present invention, it is possible to perform gradation processing in consideration of the balance between subject areas characteristic of a shooting scene.

1 is a block diagram illustrating a functional configuration example of a digital camera 100 as an example of an image processing apparatus according to an embodiment of the present invention. FIG. 3 is a block diagram illustrating an example of a functional configuration of the image processing unit 104 according to the present embodiment. FIG. 3 is a block diagram illustrating an example of a functional configuration of an exposure amount calculation unit 105 according to the present embodiment. FIG. 2 is a block diagram illustrating an example of a functional configuration of a gradation characteristic calculation unit 204 according to the present embodiment. FIG. 2 is a block diagram illustrating an example of a functional configuration of a gain processing unit 206 according to the present embodiment. The flowchart which shows a series of operation | movement which concerns on the exposure amount calculation process which concerns on this embodiment. A flowchart showing a series of operations related to gradation processing according to the present embodiment. Flowchart showing a series of operations related to the gradation characteristic calculation processing according to the present embodiment. The flowchart which shows a series of operation | movement which concerns on the gain amount correction process which concerns on this embodiment. The figure which shows an example of the relationship between the parameter for calculating the evening scene degree, and the evening scene degree The figure which illustrates the gain amount (A) concerning the whole image according to the evening scene degree, and the difference (B) between the gain amount concerning the background area and the gain quantity concerning the sky area according to the evening scene degree FIG. 6 is a diagram for explaining an example of determining a subject area (A), a luminance step (B) between the subject areas, and an output exposure amount (C). The figure explaining the target gain amount (A) for each subject area and the gain amount (B) applied to each subject area The figure which illustrates the gradation characteristic (A) with respect to a luminance signal, and a gain table (B) The figure which shows the example (A) of the input / output characteristic computed by the gradation characteristic preparation part 405 which concerns on this embodiment, and the range (B) of the input signal for every object area with respect to an input / output characteristic The figure explaining the method of calculating the range of the input signal for every object field to input / output characteristics The figure which shows an example of the gradation characteristic obtained by this embodiment The figure which shows the outline of the gradation process for every to-be-photographed object area | region which concerns on this embodiment. The figure which shows an example of the relationship between the parameter for calculating cloudiness, and cloudiness

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, an example in which the present invention is applied to an arbitrary digital camera capable of gradation processing will be described as an example of an image processing apparatus. However, the present invention is not limited to a digital camera and can be applied to any device capable of performing gradation processing. These devices may include, for example, a mobile phone, a game machine, a tablet terminal, a personal computer, a clock-type or glasses-type information terminal.

  In this embodiment, a subject area on the image is determined, and gradation processing (also referred to as area-specific tone mapping) is performed for each subject area. As shown in FIG. 17, in the tone mapping for each region, first, an optimum gain amount is calculated for each subject region based on the representative luminance value, and a plurality of images to which the calculated gain amount for each subject region is applied are generated. Then, each subject region is extracted from each image to obtain an output image. In the present embodiment, the tone mapping for each area will be described by taking as an example a sunset scene that is one of the most effective shooting scenes, with the darkest background and the brightest sky.

(1 Digital camera configuration)
FIG. 1 is a block diagram illustrating a functional configuration example of a digital camera 100 as an example of an image processing apparatus according to the present embodiment. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. May be. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as the operation subject, the same hardware can be realized as the subject.

  The optical system 101 includes a lens group including a zoom lens and a focus lens, an aperture adjustment device, and a shutter device. The optical system 101 adjusts the magnification of the subject image that reaches the imaging unit 102, the focus position, or the amount of light incident on the imaging unit 102. The imaging unit 102 is a photoelectric conversion element such as a CCD or a CMOS sensor that photoelectrically converts a light beam of a subject that has passed through the optical system 101 into an electric signal.

  The A / D conversion unit 103 performs analog / digital conversion on the input analog signal by an A / D conversion circuit, and outputs a digital signal in units of pixels.

  The image processing unit 104 performs development processing such as pixel interpolation processing and predetermined compression processing for recording an image on the recording unit 110. Further, gradation processing according to this embodiment, which will be described later, is performed. The image processing unit 104 generates a plurality of images to which gradation processing is applied, and generates an output image in which each subject area is extracted from each image. The image processing unit 104 is not limited to the image output from the A / D conversion unit 103, and can perform similar image processing and expansion processing on the image read from the recording unit 110.

  The exposure amount calculation unit 105 calculates an exposure amount at the time of shooting in order to obtain an optimal input image for performing gradation processing according to the present embodiment. Using the processing result of the image processing unit 104 as an input, an exposure amount calculation process described later is performed and output to the exposure amount control unit 106.

  The exposure amount control unit 106 controls the aperture, shutter speed, and sensor analog gain of the optical system 101 and the imaging unit 102 based on the exposure amount input by the exposure amount calculation unit 105.

  The control unit 107 is, for example, a CPU or MPU, and controls the entire digital camera 100 by developing and executing a program stored in a ROM (not shown) in a work area of a RAM (not shown).

  The recording unit 110 has a function of recording a photographed image, and includes, for example, a recording medium using a package containing a rotary recording body such as a memory card on which a semiconductor memory is mounted or a magneto-optical disk.

(2-1. Series of operations related to exposure amount calculation processing)
In order to perform appropriate gradation processing, it is necessary to obtain an image shot with an optimal exposure amount, and thus the exposure amount calculation unit 105 calculates an appropriate exposure amount for shooting. The exposure amount calculation unit 105 includes, for example, functional blocks shown in FIG. The exposure amount calculation unit 105 receives a representative luminance value for each subject area output from the luminance value calculation unit 203 in the image processing unit 104. The processing of each part constituting the exposure amount calculation unit 105 will be described later together with the exposure amount calculation process.

  A series of operations from when an image is acquired until the exposure amount is calculated by the exposure amount calculation unit 105 will be described with reference to FIG. This process is started when, for example, a shutter button (not shown) of the digital camera 100 is pressed halfway and a photographed image is input. Of the steps described below, the image processing unit 104 executes the processing of S601 to S603, the exposure amount calculation unit 105 executes the processing of S604 to S606, and the control unit 107 controls each unit. The entire exposure calculation process is controlled.

  In step S <b> 601, the image processing unit 104 acquires an image captured for calculating the exposure amount from the A / D conversion unit 103 based on an instruction from the control unit 107.

  In step S602, the image processing unit 104 determines the subject area from the acquired image. In the present embodiment, a total of four areas are discriminated by using a person's face area, a person's body area, a sky area including clouds and the sun, and other areas as background areas. FIG. 11A shows an example of a result of subject area discrimination processing performed on an input image. Discrimination of the subject area can be performed using a known method such as object recognition using learning data by a neural network as described in Japanese Patent Application Laid-Open No. 2006-39666. Omitted. The subject area discrimination accuracy may be either one pixel unit or a block unit of a certain size, but in the present embodiment, it will be described as one pixel unit.

  In step S603, the image processing unit 104 calculates a representative luminance value for each subject region using the subject region determination result in step S602. The representative luminance value indicates either an average value of luminance signals in the corresponding subject area or a weighted average value by weighting according to coordinates. As in the present embodiment, in a photographic scene where no person is detected, representative luminance values are calculated for two areas, the sky area and the background area. For example, the image processing unit 104 calculates the representative luminance value (SKY_Y) of the sky region from the average value of the luminance signals in the corresponding subject region, and sets the representative luminance value (BACK_Y) of the background region with a large weight at the center of the image. It calculates from the weighted average value by weighting. The image processing unit 104 outputs the calculated representative luminance value of each subject area to the exposure amount calculation unit 105.

  In step S604, the luminance level calculation unit 302 of the exposure amount calculation unit 105 calculates the luminance level difference between the subject areas from the input representative luminance value of each subject area. As shown in FIG. 11B, the calculation of the luminance step is obtained by obtaining an absolute value of a difference between the target exposure amounts of the subject areas obtained on the basis of the exposure amount of the input image. The target exposure amount is the number of steps over or under from the exposure amount of the input image in order to set the representative luminance value of the target subject area to the target luminance value (ie, exposure correction amount). Is shown. Increasing the exposure amount of the input image is called over, and reducing the exposure amount of the input image is called under. It is assumed that the target luminance value is a predetermined value for each subject area.

  Assuming that the target luminance values of the subject areas are BACK_ref_Y and SKY_ref_Y with respect to the representative luminance values BACK_Y and SKY_Y, the target exposure amounts ΔBV_BACK_ref and ΔBV_SKY_ref of the subject areas are calculated by (Equation 1).

  The luminance step calculation unit 302 calculates a luminance step between the subject areas from the target exposure amount calculated for each subject area calculated by (Equation 1). The luminance step ΔBS between the background area and the sky area is calculated by (Equation 2). Here, ABS represents a function for calculating an absolute value.

  In S605, the output luminance value calculation unit 301 of the exposure amount calculation unit 105 calculates the output luminance value of the main subject based on the target luminance value of each subject area calculated in S603. The output luminance value of the main subject is different from the above-described target luminance value, and refers to the luminance value of the main subject region that is finally desired to be output in a later captured image. In this embodiment, the background is the main subject, and the target luminance value of the background area is directly used as the output luminance value (BACK_OUT_Y) of the main subject.

  In S606, the exposure amount determination unit 303 of the exposure amount calculation unit 105 inputs the luminance step ΔBS between the subject areas calculated in S604 and the output luminance value BACK_OUT_Y of the main subject calculated in S605, and is suitable for gradation processing. The amount of exposure for shooting is determined. FIG. 11C shows the output exposure amount of each subject area with respect to the exposure amount of the input image. In order to realize the brightness of each subject area to be finally output, the output exposure amount should be an exposure amount that is over or under the exposure amount of the input image (ie, exposure correction amount). Indicates. When the output exposure amounts of the background area and the sky area are ΔBV_BACK and ΔBV_SKY, respectively, the output exposure amount of each subject area is calculated by (Equation 3). The exposure amount determining unit 303 is configured to maintain a main luminance difference ΔBS between the regions calculated in S604 based on the output exposure amount ΔBV_BACK of the main subject determined based on the output luminance value calculated in S605. An output exposure amount of a subject area other than the subject (that is, an empty area) is determined.

  The exposure amount determination unit 303 determines the smallest output exposure amount among the output exposure amounts of each subject area as an exposure correction amount for performing photographing suitable for gradation processing. That is, the exposure amount BV_Capture that reflects ΔBV_SKY of the sky region, which is the smallest output exposure amount among the subject regions, in the exposure amount of the input image is determined as the exposure amount at the time of shooting. When the exposure amount calculation unit 105 outputs the exposure amount at the time of shooting, the control unit 107 ends a series of processes related to this process.

  If the exposure amount suitable for the gradation processing as described above is not calculated, the gradation processing cannot be performed with an appropriate exposure amount. For example, if the exposure amount becomes larger than necessary, an image with a wide dynamic range cannot be realized due to the occurrence of a saturated region or the like. Therefore, the image quality is deteriorated due to the amplified noise. For this reason, it is possible to input an image optimal for gradation processing by calculating the exposure amount described above.

(3. Configuration of the image processing unit 104)
FIG. 2 is a block diagram illustrating a functional configuration example of the image processing unit 104. Each of the following functional blocks performs gradation processing on the input image as a whole. The subject area determination unit 201 determines a subject area for the input image. The luminance value calculation unit 203 calculates a representative luminance value for each discriminated subject area, and the histogram calculation unit 202 calculates a histogram for each subject region. The gradation characteristic calculation unit 204 calculates gradation characteristics for the input image using the calculation results obtained by the luminance value calculation unit 203 and the histogram calculation unit 202. The gradation characteristics are calculated by, for example, calculating a gain amount to be applied to each subject area after calculating the degree of a shooting scene such as a sunset scene. The gain table calculation unit 205 calculates a gain table for tone conversion corresponding to the tone characteristics, and the gain processing unit 206 performs a process of applying gain to the input image. Thereafter, the tone compression unit 207 performs tone compression processing in order to match the signal after gain processing to the display system.

(4-1. A series of operations related to gradation processing)
Next, a series of operations related to gradation processing will be described with reference to FIG. The gradation processing according to the present invention is characterized in that gradation characteristics capable of reducing a luminance step between subject areas are applied to the entire image. Also, by assigning gradations in a well-balanced manner according to the scene other than the main subject area with the main subject area as a reference, an image close to the contrast and brightness as seen by humans can be output.

  After the optical system 101 is adjusted based on the exposure amount BV_Capture determined by the exposure amount calculation process described above, for example, when shooting is performed according to a shooting instruction by fully pressing the shutter button, the image processing unit 104 has been shot. Tone processing is performed on the image.

  In step S701, the image processing unit 104 acquires an image captured for gradation processing as an input image.

  In step S <b> 702, the subject region determination unit 201 performs subject region determination processing on the input image acquired in step S <b> 701. The subject area determination process may be performed in the same manner as in S602, and thus the description thereof is omitted.

  In step S703, the luminance value calculation unit 203 calculates a representative luminance value for each subject area determined in step S702. The processing for calculating the representative luminance value of each subject area may be performed in the same manner as in S603, and the description thereof will be omitted.

  In step S <b> 704, the histogram calculation unit 202 calculates a luminance signal histogram for each subject area.

  In step S <b> 705, the gradation characteristic calculation unit 204 receives the representative luminance values calculated in step S <b> 703 and the histogram for each subject area calculated in step S <b> 704, and calculates gradation characteristics for the luminance signal of the input image. For example, as shown in FIG. 13A, the gradation characteristic is an input / output characteristic indicating an output signal corresponding to the input signal, with the horizontal axis representing the input signal and the vertical axis representing the output signal. For example, when the gradation characteristic is not corrected, as shown by the dotted line, the input signal and the output signal have the same input / output characteristic. Details of the gradation characteristic calculation processing in this step will be described in detail with reference to FIG. 8A and the like.

  In step S706, the gain table calculation unit 205 calculates a gain table based on the gradation characteristics for the luminance signal calculated in step S705. As shown in FIG. 13B, the gain table is a table indicating the gain according to the input signal, with the horizontal axis representing the input signal and the vertical axis representing the gain. When the input signal is x and the output signal converted by the gradation characteristic is y, the gain Gain is expressed by (Equation 4).

  In step S707, the gain processing unit 206 applies a gain to the input image using the gain table calculated in step S706. FIG. 5 shows a functional configuration example in the gain processing unit 206 corresponding to the gain processing in this step.

  The gain Gain (x, y) applied to the input signal Yin (x, y) when the pixel position is indicated by (x, y) with the pixel at the top left of the image as (0,0) is the gain When the table is expressed by the GainTbl function, (Formula 5) is obtained.

  Gain (x, y) is a gain corresponding to the pixel located at (x, y) of the input image. Yin (x, y), which is an input signal for calculating the gain, is a gain calculation image signal generated based on the input image by the gain calculation image generation unit 501. The gain calculation image generation unit 501 converts the input image into a luminance image and performs blurring processing such as average value processing in order to reduce the sensitivity of the gain to the fine texture of the input image and maintain better contrast. To generate an image for gain calculation.

  The gain converter 502 converts Yin (x, y), which is a pixel value located at (x, y) of the gain calculation image, into a gain signal to be applied to the pixel located at (x, y) of the input image. That is, the process shown in (Formula 5) is performed. The gain multiplication unit 503 performs processing for multiplying the input image by a gain based on the gain calculated by the gain conversion unit 502. In the present embodiment, a process of multiplying an R, G, B image that has already undergone demosaicing processing from the captured input image by a gain is performed. For example, R, G, and B signals of a pixel located at (x, y) before gain multiplication are Rin (x, y), Gin (x, y), and Bin (x, y), respectively. At this time, R, G, and B signals Rout (x, y), Gout (x, y), and Bout (x, y) after gain multiplication are calculated according to (Equation 6) using the GainTbl function.

  Here, at the pixel position (x, y), the gains applied to the R, G, and B signals are the same. In the present embodiment, the gain processing is performed on the RGB signals, but the above gain processing may be performed on the YUV signals.

  In step S708, the gradation compression unit 207 performs gradation compression processing having γ characteristics to match the R, G, and B m-bit signals that have been subjected to gain processing in step S707 to the display system, and n bits (n is m The following signal is output. When the image processing unit 104 outputs the signal subjected to the gradation compression process, the image processing unit 104 ends a series of operations related to the gradation process.

(4-2. A series of operations related to calculation processing of gradation characteristics)
A series of operations for calculating gradation characteristics, which is the main feature of the present invention, will be described with reference to 800 shown in FIG. 8A. FIG. 4 shows a functional configuration example of the gradation characteristic calculation unit 204 that executes a series of operations of the gradation characteristic calculation process. The gradation characteristic calculation process is started when the above-described process of S705 is called.

  In step S810, the luminance level calculation unit 401 calculates the luminance level difference between the subject areas using the input representative luminance value for each subject area. The method for calculating the luminance step is the same as the processing shown in FIGS. 11B, 11C, and S604. As shown in FIGS. 12A and 12B, the zero step is used as a reference. The exposure amount (exposure correction amount) may be replaced with the gain amount.

  In step S820, the output luminance value determination unit 402 determines an output luminance value for the main subject. The output luminance value of the main subject is the luminance value of the main subject area to be finally output, like the output luminance value in S605. In the present embodiment, the background area is set as the main subject area as described above. The calculation method of the output luminance value is the same as the processing content shown in S605, and thus the description is omitted.

  In step S830, the gradation correction amount calculation unit 403 calculates the gradation correction amount of each subject area using the luminance step between the subject areas calculated in step S810 and the output luminance value of the main subject calculated in step S802 as inputs. The gradation correction amount is a gain amount that is uniformly applied to a target subject area in the present embodiment. A method of calculating the gradation correction amount for each subject area will be described with reference to the flowchart 801 shown in FIG. 8A and the flowchart shown in FIG. 8B.

(4-3. Series of operations related to determination processing of gradation correction amount)
In S831, the gradation correction amount calculation unit 403 sets the gradation correction amount [stage] of each subject region as the gain amount of the background region (GAIN_BACK) and the gain amount of the sky region (GAIN_SKY), respectively. It calculates by Formula 7). These gain amounts are the same as the output exposure amounts obtained in (Equation 3). BACK_Y is a representative luminance value of the background area, BACK_OUT_Y is an output luminance value of the background area as the main subject, and ΔBS is a luminance step between the background area and the sky area.

  In this embodiment, tone correction is not applied to gain amounts less than 0. Therefore, after calculating GAIN_BACK and GAIN_SKY in (Equation 7), there may be negative values among GAIN_BACK and GAIN_SKY. For example, these values are clipped to 0.

(4-3. Series of operations related to gain amount correction processing)
In S832, the scene degree calculation unit 406 calculates a feature amount based on each region, calculates the degree of the scene indicated by the image, and performs a process of further correcting the gain amount calculated in S831. More detailed steps (S833 to S838) relating to the gain amount correction processing will be described with reference to the flowchart shown in FIG. 8B.

  In step S833, the scene degree calculation unit 406 determines whether or not a person is detected in the image. If no person is detected, the process proceeds to step S834. If the person is detected, the gain amount correction process ends.

  In S834, the scene degree calculation unit 406 calculates three coefficients α, β, and γ (that is, feature amounts) based on the pixel value and the luminance step for each subject area, and determines the possibility that the scene is an evening scene based on these coefficients. The evening scene score Score_Sunset shown is calculated.

  A method of calculating the three coefficients will be described with reference to FIG. The scene degree calculation unit 406 accounts for the evening scene degree α of the total number of pixels determined to be the sky region by the subject region determination unit 201 and the number of pixels whose hues are red to yellow (hue angle: 0 to 60) Calculate based on (sunset sky ratio). In FIG. 9A, the scene degree calculation unit 406 sets the evening scene degree α to a large value (100 at the maximum) so as to determine that the sunset sky ratio is higher as the sunset sky ratio is higher.

  Also, the scene degree calculation unit 406 sets the evening scene degree β to the brightness of the sky area determined by the subject area determination unit 201, that is, whether the Bv value (Brightness Value) has a value that seems to be the sky of the evening scene. Calculate based on whether or not. For example, the Bv value of the sky region is calculated by the following equation using the representative brightness value SKY_Y of the sky region calculated in S703, the target brightness value SKY_REF_Y of the sky region (for example, 133 at 10 bits), and the exposure amount BV_Capture at the time of shooting. Calculated.

  For example, in FIG. 9B, when the scene degree calculation unit 406 determines that there is a high possibility of the evening sky when the Bv value of the sky region is 4 to 6, the evening scene degree β is set to a large value. (Maximum 100).

  In calculating the evening scene degree γ, the scene degree calculating unit 406 is calculated based on the luminance step ΔBS between the sky area and the background area calculated in S831. In FIG. 9C, the scene degree calculation unit 406 determines that the greater the value of ΔBS, that is, the greater the difference in brightness between the sky and the background, the higher the possibility that the scene shows an evening scene atmosphere. The evening scene degree γ is set to a large value (100 at the maximum).

  The scene degree calculation unit 406 selects a large value from the calculated α and β, and sets a value obtained by multiplying the selected value by γ as the final evening scene score Score_Sunset. In sunset scenes, α takes a large value because the hue of the sky often shows red to yellow before sunset. On the other hand, after sunset, the hue of the sky does not show red to yellow, but the Bv value is smaller than that in sunny daytime, so β is set to take a large value. Therefore, in the present embodiment, the evening scene degrees α and β are selectively used. Further, as will be described later, since the difference between the brightness of the sky and the background (luminance step) can be used as a signal representing the atmosphere of the evening scene, the evening scene degree γ is used for determining the evening scene degree. For example, when the input image is an evening scene before sunset, the value of the evening scene degree α increases because the sky hue includes an area indicating red to yellow. Then, since the brightness difference between the sky including the sun and the dim background increases, the value of the evening scene degree γ increases, and as a result, the evening scene degree Score_Sunset equal to or higher than a predetermined threshold value is obtained.

  In S835, the scene degree calculation unit 406 determines whether the evening scene score Score_Sunset calculated in S834 is smaller than the threshold. If the threshold is smaller than the threshold, the process proceeds to S836. If the threshold is greater than or equal to the threshold, the process proceeds to S838.

  In S836, the scene degree calculation unit 406 calculates a cloudy score Score_Cloudy indicating the possibility of a cloudy scene. Specifically, the coefficients α2, β2, and γ2 are calculated as described in the sunset scene. The cloudiness degree α2 is calculated based on the ratio (the ratio of clouds) of the total number of pixels determined as the cloud area to the total number of pixels determined as the sky area by the subject area determination unit 201. To do. As shown in FIG. 18 (A), α2 is set to a large value (100 at the maximum) so as to determine that the larger the cloud ratio, the higher the possibility of being cloudy. The cloudy degree β2 is calculated based on Expression 8 in the same manner as the determination at sunset. As shown in FIG. 18B, the cloudiness β2 is set to a large value (100 at the maximum) so that it is determined that the sky is likely to be cloudy when the sky Bv value is 4-6. Further, the cloudy degree γ2 is calculated based on the luminance step ΔBS in the same manner as in the determination at sunset. As shown in FIG. 18C, that is, γ2 is set to a large value (100 at the maximum) so that it is determined that there is a high possibility that the scene has a cloudy atmosphere as the brightness difference ΔBS between the sky and the background increases. In the subsequent processing, similarly to the determination at the time of sunset, a larger value is selected from α2 and β2, and a value obtained by multiplying it by γ2 is set as the final cloudiness Score_Cloudy.

  In step S827, the scene degree calculation unit 406 determines whether the calculated cloudiness score Score_Cloudy is smaller than the threshold value. If the calculated cloudy degree Score_Cloudy is smaller than the threshold value, the gain amount correction process is terminated.

  In S838, the gradation characteristic creating unit 405 corrects the gain amounts GAIN_BACK and GAIN_SKY calculated in S831 based on the calculated evening scene degree (or cloudy degree). In the following description, a case where the evening scene degree is high will be described. However, even when the cloudy degree is high, the gain amounts GAIN_BACK and GAIN_SKY may be corrected as in the case of the evening scene. The gain amount correction processing according to the present embodiment will be further described with reference to FIG. FIG. 10A shows a coefficient (OUT_GAIN) for correcting the amount of gain in the background area according to the evening scene when the gain is 1024 gradations, and FIG. 10B shows the coefficient according to the evening scene. The coefficient COMP_H for correcting the gain amount in the sky region is shown. Then, using these coefficients, the correction gain amounts GAIN_BACK2 and GAIN_SKY2 are calculated by the following equations.

  Since COMP_H increases as the evening scene degree increases, the difference between the gain amount applied to the background region and the gain amount applied to the sky region increases. Further, since the OUT_GAIN decreases as the evening scene degree increases, the gain amount applied to the background area decreases, and as a result, the gain related to the sky area also decreases. That is, by setting OUT_GAIN and COMP_H to values that change as shown in FIG. 10 according to the evening scene degree, the brightness difference between the sky and the background area remains as the atmosphere during the evening scene, and the brightness of both areas is adjusted to the daytime. It leads to an underscene than the scene. In this way, it is possible to leave the atmosphere of the scene by controlling the gain amount of each region according to the degree of suitability of the scene. When the gain degree calculation process is completed, the scene degree calculation unit 406 ends the gain amount correction process and returns the process to S832.

  In S839, the gradation characteristic creating unit 405 calculates the input / output characteristics shown in FIG. 14A from the gradation correction amount for each subject area. When the input signal is X and the output signal is Y, the input / output characteristics realized in each subject area are expressed by (Equation 10).

  The input / output characteristics of each subject area obtained by this equation are the gradation characteristics applied to each subject area in the area-specific tone mapping. The slope of the input / output characteristics of each subject area is the contrast to be realized in each subject area, and the output signal indicates the brightness to be realized in each subject area. After calculating the input / output characteristics, the gradation characteristic creating unit 405 advances the process to S840.

  Next, a process for determining the input signal range based on the luminance histogram for each subject area will be described.

  In S840, the application range calculation unit 404 determines an input signal range to which the gradation correction amount for each subject area calculated in S830 is applied based on the luminance histogram for each subject area. The range of the input signal will be described with reference to FIG.

  The two input / output characteristics shown in FIG. 14B are input / output characteristics to be realized in each subject area shown in (Equation 10). BACK_POINT and SKY_POINT described on the horizontal axis of the input / output characteristics are indices that determine the range of the input signal to which the input / output characteristics are applied in each corresponding subject area. The X axis in FIG. 14B shows an example in which the input signal range from 0 to BACK_POINT is the input range 141 and the input signal range from SKY_POINT to the maximum value is the input range 142, for example. The input range 141 is an input signal range to which the input / output characteristics for the background area are applied, and the input range 142 is an input signal range to which the input / output characteristics for the sky area are applied.

  The calculation of BACK_POINT and SKY_POINT is performed using a luminance histogram calculated for each subject area, as shown in FIG. FIG. 15A shows a state in which a luminance histogram 1501 in the background area determined from the input image and a luminance histogram 1502 in the sky area determined from the input image are generated. Is formed. A specific BACK_POINT and SKY_POINT calculation process using the histogram of each subject area will be described with reference to FIG. First, the application range calculation unit 404 adds the frequency of each signal value from the smallest signal value to the larger signal value in the luminance histogram 1501 of the background region. Regarding the luminance histogram 1501, the relationship between the input signal value and the cumulative addition value of the frequency when the frequency of each signal value is added is shown in 1503. The application range calculation unit 404 sets the value of the input signal to which the frequency is added at the end as the BACK_POINT when the cumulative addition value of the frequency reaches a certain threshold (BACK_TH) or more. When the threshold is BACK_TH, it can be expressed as (Equation 11). Here, BACK_AREASUM is the total number of pixels determined as the background area, and P_BACK is an adjustment parameter.

  In (Expression 11), for example, the BACK_POINT is determined when the addition value of the frequency reaches a predetermined ratio with respect to the total number of pixels in the background area.

  On the other hand, with respect to the calculation of SKY_POINT, the application range calculation unit 404 determines each signal value from the total number of pixels determined to be a sky region from the minimum signal value toward a larger signal value with respect to the brightness histogram 1502 of the sky region. The value obtained by subtracting the frequency of is calculated. Then, when the subtracted value falls below a certain threshold value (SKY_TH), the value of the input signal from which the frequency is finally subtracted is defined as SKY_POINT. For determination of SKY_TH, a value that reaches a predetermined ratio of the total number of pixels in the sky region may be used as in BACK_TH. In this embodiment, the scene where the sky area is the brightest and the background area is the darkest is described. On the contrary, in the case of the scene where the background area is the brightest subject, the same calculation process as SKY_POINT is performed. Can be done. Further, when the sky area is a scene of an intermediate luminance subject, the calculation process similar to BACK_POINT may be performed.

  In step S850, the gradation characteristic determination unit 407 determines gradation characteristics to be applied uniformly to the image using the above-described gradation correction amount and application range. That is, the tone characteristics are determined using the tone correction amounts GAIN_BACK2 and GAIN_SKY2 calculated in S830 and the index (BACK_POINT, SKY_POINT) of the range to which the tone correction amount calculated in S840 is applied. FIG. 16 shows the gradation characteristics generated by this embodiment. A period (interval 1601) in which the input signal X is 0 to BACK_POINT is an input / output characteristic applied to the background area, and a period from SKY_POINT to the maximum value (interval 1602) is an input / output characteristic applied to the empty area. In the period from BACK_POINT to SKY_POINT, the output values at BACK_POINT and SKY_POINT are linearly changed. In this way, for each subject area in the image, by determining the input / output characteristics according to the shooting scene, tone correction that leaves the scene atmosphere (for example, the brightness difference between the sky and the background area remains, and It is possible to perform gradation correction such that the brightness of both areas is lower than the daytime scene.

  When the image processing unit 104 completes the process in S850, the image processing unit 104 ends the series of operations related to the gradation characteristic calculation process and returns the process to S705.

  In the present embodiment, a scene that does not include a person is described as an example. However, the present invention can be applied even when a person exists. When a person is detected, a target luminance value and an output luminance value are calculated from the face area of the person who is the main subject. Then, the gradation correction amount for the background and sky regions may be controlled by the method described in this embodiment. Further, in the present embodiment, two examples of sunset scene and cloudy sky are illustrated as scenes in which the atmosphere is desired to be left in the gradation correction processing, but the present invention is not limited to this. For example, in a scene where there is a person who is spotlighted, the area of this person is bright, and the background area is dark, the tone correction amount is controlled to reduce the gain applied to the background area. It becomes possible to leave an atmosphere.

  As described above, in this embodiment, the degree of a predetermined shooting scene is calculated, and the gradation characteristics are calculated according to the degree. At this time, the difference in gain amount between the subject areas and the gain amount to be applied to each subject area are set according to the degree of the shooting scene, and the gradation characteristic applied to each subject area depends on the degree of the shooting scene. Relevant. In this way, it is possible to perform gradation processing in consideration of the balance between the subject areas characteristic of the shooting scene. In other words, it is possible to generate an image having optimum contrast and brightness for each subject area while leaving the atmosphere of the shooting scene.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 102 ... Imaging part, 104 ... Image processing part, 107 ... Control part, 201 ... Subject area discrimination | determination part, 202 ... Histogram calculation part, 203 ... Luminance value calculation part, 204 ... Tone characteristic calculation part, 205 ... Gain table calculation part , 406 ... Scene degree calculation unit, 407 ... Tone characteristic determination unit

Claims (12)

Discriminating means for discriminating a plurality of subject areas from the image;
A degree calculating means for calculating a degree indicating the possibility that the image is a predetermined shooting scene based on a feature amount obtained from the image;
Calculating means for calculating a gradation correction amount for each of the plurality of subject areas based on a luminance difference between the plurality of subject areas;
Correction means for correcting the gradation correction amount based on the degree;
An image processing apparatus comprising: a gradation processing unit that applies the gradation correction amount corrected by the correction unit to the plurality of subject areas. A range calculation means for calculating a histogram of pixel values constituting the subject area, and calculating a range of the pixel values to which the gradation correction amount is applied according to the histogram;
The image processing apparatus according to claim 1, wherein the gradation processing unit applies the gradation correction amount for each range of the pixel values to the plurality of subject areas.   The calculation means calculates a gradation correction amount for a main subject region among the plurality of subject regions, and for a subject region other than the main subject region among the plurality of subjects, a gradation correction amount for the main subject region The image processing apparatus according to claim 1, wherein a gradation correction amount is calculated based on a difference in luminance from the main subject area.   The image processing apparatus according to claim 3, wherein the calculation unit sets the subject area having the lowest luminance value representing the subject area as the main subject area among the subject areas.   The image processing apparatus according to claim 1, wherein the determination unit determines at least a background area and an empty area as the plurality of subject areas.   The image processing apparatus according to claim 1, wherein the degree calculation unit calculates the feature amount based on a ratio of a predetermined pixel to a pixel of the predetermined subject area. .   The degree calculation means calculates the feature amount based on whether a luminance value representing the predetermined subject area is within a predetermined luminance value range for the predetermined shooting scene. The image processing apparatus according to any one of claims 1 to 6.   The degree calculation means calculates the feature amount based on whether a luminance difference between the plurality of subject areas of at least two of the subject areas is larger than a predetermined value for the predetermined shooting scene. The image processing apparatus according to claim 1, wherein the image processing apparatus calculates the image processing apparatus.   The correction means decreases the magnitude of the gradation correction amount for each of the subject areas and increases the difference between the gradation correction amounts for the two predetermined subject areas as the degree increases. The image processing apparatus according to any one of claims 1 to 8.   The image processing apparatus according to claim 5, wherein the predetermined shooting scene is a scene including an evening scene or a cloudy sky. A determination step of determining a plurality of subject areas from the image;
A degree calculation step in which a degree calculation unit calculates a degree indicating the possibility that the image is a predetermined shooting scene based on a feature amount obtained from the image;
A calculating step for calculating a gradation correction amount for each of the plurality of subject areas based on a luminance difference between the plurality of subject areas;
A correction step in which the correction means corrects the gradation correction amount based on the degree;
A method for controlling an image processing apparatus, comprising: a gradation processing step in which gradation processing means applies the gradation correction amount corrected by the correction means to the plurality of subject areas.   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 thru | or 10.

Images