JP2016213717A - Image processing apparatus, image processing method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus that, when performing a relighting process by using a virtual light source, can obtain a desired relighting result by preventing excessive correction or insufficient correction.SOLUTION: The image processing apparatus comprises: a setting unit that, with respect to a photographed image, sets a virtual light source which is a virtual light source corresponding to a portion whose brightness is to be corrected in the image; a correction unit that corrects the brightness of the portion whose brightness is to be corrected by using the virtual light source set by the setting unit; and a calculation unit that calculates the maximum value of a brightness correction amount by the correction unit on the basis of a distribution of luminance values of the image and a distribution of reflections in a case where the light of the virtual light source is reflected by a subject.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to a technique for correcting image brightness.

  Conventionally, a technique (relighting) that corrects brightness by irradiating light from a virtual light source onto a subject in an image is known (Patent Document 1). As a result, dark areas such as shadows caused by ambient light can be brightened, and the image can be corrected so that, for example, a subject existing in a blackened portion can be identified.

  For example, Patent Literature 1 discloses a lighting processing technique for performing pseudo lighting processing on a captured image. Specifically, a luminance area lower than the average luminance of the entire face area is extracted as a shadow area. And the process which raises the brightness of the extracted shadow area | region is performed. Thereby, the shadow of the subject can be suppressed.

JP 2010-135996 A

  When correcting the shadow by the relighting process, it is necessary to appropriately determine the relighting correction amount. Increasing the relighting correction amount saturates the pixel value, and depending on the picture, the brightness balance of the image after relighting is lost. Conversely, if the relighting correction amount is reduced so as not to saturate, there is a problem that the relighting effect is reduced.

  In the method disclosed in Patent Document 1, the gain value of the relighting correction amount is determined based on the luminance value of the original image before the relighting process. However, in this method, the positional relationship between the subject and the virtual light source and the reflection distribution of the virtual light source are not considered, and the gain value of the relighting correction amount is controlled to be small if the luminance value of the original image is high. For example, when the virtual light source is set obliquely with respect to the subject, there are a region where the virtual light source is strongly applied and a region where the virtual light source is weakly applied even within the same subject. Therefore, even if the luminance value of the original image in the area where the virtual light source is weak is high, if the gain value of the relighting correction amount is set small, the effect of relighting becomes small.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to perform overcorrection or correction when brightness is corrected by irradiating light from a virtual light source to a subject in an image. An object of the present invention is to provide an image processing apparatus capable of suppressing the insufficient correction.

  An image processing apparatus according to the present invention includes: a setting unit that sets a virtual light source that is a virtual light source corresponding to a portion of the image whose brightness is to be corrected; Correction means for correcting the brightness of the portion for correcting the brightness using the virtual light source set by the following: a distribution of luminance values of the image and a distribution of reflection when the virtual light source is reflected by a subject And a calculating means for calculating a maximum value of the brightness correction amount by the correcting means.

  According to the present invention, when correcting brightness by irradiating light from a virtual light source to a subject in an image, image processing that can suppress overcorrection or insufficient correction. An apparatus can be provided.

1 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention. The block diagram which shows the structure of an image process part. The block diagram which shows the structure of a relighting process part. The figure explaining reflection by irradiation of a virtual light source. The flowchart which shows the calculation process of a maximum gain value. The figure explaining the image before and after a relighting process. The flowchart which shows an auto rewriting process. The figure explaining the intensity setting screen of rewriting.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the embodiment described below, an example in which the image processing apparatus according to the present invention is applied to a digital camera will be described. Note that a digital camera means an electronic device having a photographing function using a photoelectric conversion element, and includes any electronic device having or usable with a camera, such as a mobile phone, a game machine, and a personal computer. In addition, the photographing function is not essential in the present invention, and the image processing apparatus according to the present invention can be applied to any electronic device capable of image processing.

  FIG. 1 is a block diagram illustrating a configuration example of a digital camera 100 according to an embodiment of the present invention.

  In FIG. 1, a lens group 101 is a zoom lens including a focus lens. A shutter 102 having a diaphragm function is provided between the lens group 101 and the imaging unit 103. The imaging unit 103 includes an imaging element represented by a CCD / CMOS image sensor that converts an optical image formed on the imaging surface by the lens group 101 into an electrical signal in pixel units. The A / D converter 104 converts the analog signal output from the imaging unit 103 into a digital signal (image data).

  The image processing unit 105 performs various image processing such as color interpolation (demosaic), white balance adjustment, γ correction, contour enhancement, noise reduction, and color correction on the image data output from the A / D converter 104. The image memory 106 temporarily stores image data. A memory control unit 107 controls reading and writing of the image memory 106. The D / A converter 108 converts the image data into an analog signal. The display unit 109 includes a display device such as an LCD or an organic EL display, and displays various GUIs, live view images, images read out from the recording medium 112 and reproduced. The codec unit 110 encodes the image data stored in the image memory 106 by a predetermined method for recording on the recording medium, or decodes the encoded image data included in the image file for display, for example. Or

  The interface (I / F) 111 mechanically and electrically connects a detachable recording medium 112 such as a semiconductor memory card or a card type hard disk to the digital camera 100. The system control unit 50 may be a programmable processor such as a CPU or MPU. The system control unit 50 realizes the functions of the digital camera 100 by executing a program recorded in, for example, the nonvolatile memory 121 or the built-in nonvolatile memory and controlling necessary blocks and circuits. The relighting processing unit 114 performs relighting processing on the captured image.

  The operation unit 120 collectively describes buttons and switches for the user to input various instructions to the digital camera 100.

  The nonvolatile memory 121 may be an electrically erasable / recordable, for example, an EEPROM. The nonvolatile memory 121 stores various setting values and GUI data, and programs to be executed by the system control unit 50 when the system control unit 50 is an MPU or CPU.

  The system memory 122 is used to develop constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like. The distance measuring sensor 123 measures the distance to the subject and outputs distance information corresponding to the pixel unit of the photographic pixel as a two-dimensional distance map image.

  Next, the operation at the time of shooting in the digital camera 100 will be described.

  For example, the imaging unit 103 photoelectrically converts a subject image formed on the imaging surface by the lens group 101 when the shutter 102 is open, and outputs the subject image to the A / D converter 104 as an analog image signal. The A / D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal (image data) and outputs the digital image signal to the image processing unit 105.

  The image processing unit 105 performs various processes such as color interpolation (demosaic), γ correction, edge enhancement, noise reduction, and color correction on the image data from the A / D converter 104 or the image data from the memory control unit 107. Perform image processing.

  In addition, the image processing unit 105 performs predetermined calculation processing relating to luminance, contrast, and the like using image data obtained by photographing, and the system control unit 50 performs distance measurement control and exposure control based on the obtained calculation result. Do. As described above, the digital camera 100 according to the present embodiment performs TTL (through the lens) AF (autofocus) processing and AE (automatic exposure) processing. The image processing unit 105 further performs auto white balance (AWB) adjustment using image data obtained by shooting.

  The image data output from the image processing unit 105 is written into the image memory 106 via the memory control unit 107. The image memory 106 stores image data output from the imaging unit 103 and image data to be displayed on the display unit 109.

  The D / A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109. The display unit 109 performs display in accordance with an analog signal from the D / A converter 108 on a display device such as an LCD.

  The codec unit 110 encodes the image data recorded in the image memory 106 based on a standard such as JPEG or MPEG. The system control unit 50 assigns a predetermined header or the like to the encoded image data, forms an image file, and records the image file on the recording medium 112 via the interface 111.

  Note that with current digital cameras, it is common to perform moving image shooting in the shooting standby state, and keep the captured moving image displayed on the display unit 109 so that the display unit 109 functions as an electronic viewfinder (EVF). is there. In this case, the shutter 102 is opened and a so-called electronic shutter of the imaging unit 103 is used to capture an image at, for example, 30 frames / second.

  When the shutter button included in the operation unit 120 is half-pressed, the above-described AF and AE control is performed. When the shutter button is fully pressed, the still image shooting for recording is performed by the main shooting and is recorded on the recording medium 112. The Also, when moving image shooting is instructed by a moving image shooting button or the like, moving image recording on the recording medium 112 is started.

  In addition to the above basic operation, the system control unit 50 executes each process of the present embodiment, which will be described later, by executing the program recorded in the nonvolatile memory 121 described above. The program here is a program for executing various flowcharts described later in the present embodiment. At this time, constants, variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like are expanded in the system memory 122.

  Next, details of the image processing unit 105 will be described with reference to FIG. FIG. 2 is a block diagram illustrating a configuration of the image processing unit 105.

  2, the image processing unit 105 includes a synchronization processing unit 200, a WB amplification unit 201, a luminance / color signal generation unit 202, a contour enhancement processing unit 203, and a luminance gamma processing unit 204. Further, the image processing unit 105 includes a color conversion processing unit 205, a color γ processing unit 206, a color difference signal generation unit 207, and a shadow information acquisition unit 208.

  Next, processing in the image processing unit 105 will be described. The image signal input from the A / D conversion unit 104 in FIG. 1 is input to the image processing unit 105. The image signal input to the image processing unit 105 is input to the synchronization processing unit 200. The synchronization processing unit 200 performs synchronization processing on the input RGB image data of the Bayer array to generate color signals R, G, and B. Based on the white balance gain value calculated by the system control unit 50, the WB amplification unit 201 applies a gain to the RGB color signals and adjusts the white balance. The RGB signal output from the WB amplification unit 201 is input to the luminance / color signal generation unit 202. The luminance / color signal generation unit 202 generates a luminance signal Y from the RGB signals, and outputs the generated luminance signal Y to the contour enhancement processing unit 203 and the RGB color signals to the color conversion processing unit 205.

  The contour enhancement processing unit 203 performs contour enhancement processing on the luminance signal and outputs the luminance signal to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y and outputs the luminance signal Y to the image memory 106.

  The color conversion processing unit 205 converts the RGB signal into a desired color balance by matrix calculation or the like. The color gamma processing unit 206 performs gamma correction on the RGB color signals. The color difference signal generation unit 207 generates color difference signals BY and RY from RGB signals. The Y signal, the BY signal, and the RY signal, which are image signals output to the image memory 106, are compression encoded by the codec unit 110 and recorded on the recording medium 112.

  Further, the RGB signal output from the color conversion processing unit 205 is also input to the shadow information acquisition unit 208. The shadow information acquisition unit 208 acquires information for analyzing the shadow state (brightness / darkness state) generated in the subject by the environmental light source. For example, average luminance information of the subject, luminance histogram information of the face area, and the like are acquired as shadow information.

  Next, the configuration and operation of the relighting processing unit 114 that performs brightness correction processing (relighting processing) by irradiating light from a virtual light source to the subject in the image will be described with reference to FIG. To do. When the relighting process is selected by the user's operation, the data output from the image processing unit 105 is input to the relighting process unit 114, and the relighting process using the virtual light source is performed.

  FIG. 3 is a block diagram showing a configuration of the relighting processing unit 114. In FIG. 3, a signal conversion unit 301 converts an input luminance / color difference signal (Y, BY, RY) into an RGB signal. The degamma processing unit 302 performs degamma processing. In addition, the distance calculation unit 303 acquires distance information from the imaging device output from the distance measuring sensor 123 to the subject (distance information acquisition). The normal calculation unit 304 calculates the normal of the subject (obtains normal information). The virtual light source reflection component calculation unit 305 calculates a component reflected by the virtual light source and hitting the subject. The virtual light source addition processing unit 306 adds the relighting effect by the virtual light source to the image. The gamma processing unit 307 applies gamma characteristics to the RGB signals, and the luminance / color difference signal conversion unit 308 converts the RGB signals into luminance / color difference signals (Y, BY, RY).

  The operation of the relighting processing unit 114 configured as described above will be described. The relighting processing unit 114 receives luminance / color difference signals (Y, BY, RY) recorded in the image memory 106. The RGB signal conversion unit 301 converts the input luminance / color difference signals (Y, BY, RY) into RGB signals and outputs them to the degamma processing unit 302. The de-gamma processing unit 302 performs a calculation of a characteristic opposite to the gamma characteristic multiplied by the gamma processing unit of the image processing unit 105, and converts the RGB signal into linear data. The degamma processing unit 302 outputs the RGB signals (Rt, Gt, Bt) after linear conversion to the virtual light source reflection component calculation unit 305 and the virtual light source addition processing unit 306.

  On the other hand, the distance calculation unit 303 calculates a distance map from the distance information of the subject acquired from the distance measuring sensor 123. The subject distance information is two-dimensional distance information obtained for each area obtained by dividing a captured image. This region may be a single pixel unit or a plurality of pixel units. The normal calculation unit 304 calculates a normal map from the distance information acquired from the distance calculation unit 303.

  A known technique may be used as a method for generating a normal map from a distance map, but a specific processing example will be described with reference to FIG. FIG. 4 is a diagram showing the relationship between the shooting coordinates of the camera and the subject. For example, as shown in FIG. 4, for a subject 401, the gradient information of the surface 401a is calculated from the difference ΔD of the distance (depth) D with respect to the difference ΔH in the horizontal direction of the captured image, and the method of the surface 401a is calculated from the gradient information. Line N can be calculated. By performing the above processing on each photographed pixel, it is possible to calculate normal line information N corresponding to each pixel of the photographed image. The normal calculation unit 304 outputs normal information N corresponding to each pixel of the captured image to the virtual light source reflection component calculation unit 305 as a normal map.

  Based on the distance K between the light source and the subject, the normal line information N, and the virtual light source parameter L, the virtual light source reflection component calculation unit 305 calculates a component that reflects the reflected light from the installed virtual light source. Specifically, the reflection component at the coordinate position corresponding to the captured image is calculated so as to be inversely proportional to the square of the distance K between the light source and the subject and proportional to the inner product of the normal vector N of the subject and the light source direction vector L. .

  This will be further described with reference to FIG. FIG. 4 shows the position of the subject 401 and the position of the set virtual light source 402. The reflection component at the horizontal pixel position H1 (the vertical pixel position is omitted for easy understanding) of the photographed image taken by the camera 100 is proportional to the inner product of the normal line N1 at the coordinate H1 and the direction vector L1 of the virtual light source. The value is inversely proportional to the square of the distance K1 between the virtual light source and the subject.

  When this relationship is expressed by a mathematical expression, reflection components (Ra, Ga, Ba) from the subject by the virtual light source are as follows.

Ra = α × (−L · N) / K ² × Rt
Ga = α × (−L · N) / K ² × Gt (1)
Ba = α × (−L · N) / K ² × Bt
Here, α is the strength of the virtual light source and the gain value of the relighting correction amount. In the present embodiment, the maximum gain value αmax of the relighting correction amount is calculated, and control is performed so that the relighting process is performed with a gain value equal to or less than αmax. L is the three-dimensional direction vector of the virtual light source, N is the three-dimensional normal vector of the subject, and K is the distance between the virtual light source and the subject. Rt, Gt, and Bt are photographing RGB data representing the luminance value of the photographed image output from the degamma processing unit 302.

  The reflection component (Ra, Ga, Ba) by the virtual light source calculated as described above is output to the virtual light source addition processing unit 306. The virtual light source addition processing unit 306 performs the following processing for adding virtual light source components (Ra, Ga, Ba).

Rout = Rt + Ra
Gout = Gt + Ga
Bout = Bt + Ba
The image signals (Rout, Gout, Bout) output from the virtual light source addition processing unit 306 are input to the gamma processing unit 307. The gamma processing unit 307 performs gamma correction on the RGB input signals. The color difference signal generation unit 308 generates a luminance signal Y and color difference signals BY and RY from the RGB signals. The above is the operation of the relighting processing unit 114.

  The system control unit 50 stores the luminance / color difference signal output from the relighting processing unit 114 in the image memory 106 under the control of the memory control unit 107 and then compression-codes the codec unit 110. Further, recording is performed on the recording medium 112 via the I / F 111.

  Next, the flow of processing in which the system control unit 50 determines the maximum gain value αmax of the relighting processing unit 114 that operates as described above will be described. Specifically, the maximum gain value αmax is determined based on the calculation results of the distance calculation unit 303, normal calculation unit 304, and virtual light source reflection component calculation unit 305. These operations will be described below.

  FIG. 5 is a flowchart showing the operation of the system control unit 50. The operation of the system control unit 50 for determining the maximum gain value αmax of the relighting parameter will be described below with reference to the flowchart of FIG.

  In step S <b> 501, it is determined whether relighting processing is selected by an operation from the user via the operation unit 120. That is, it is determined whether or not processing by the relighting processing unit 114 is performed. When rewriting is performed, the process proceeds to step 502.

In step S502, the subject is photographed. In step S503, the distance calculation unit 303 generates a weighted map image (mapK) based on the distance between the imaging device and the subject. Specifically, the distance calculation unit 303 calculates the distance K in units of pixels based on the two-dimensional subject distance information obtained in units of pixels of the captured image acquired from the distance measuring sensor 123. A value obtained by normalizing 1 / K ² with an arbitrary bit width in units of pixels is set as a weighted map image (mapK) based on distance.

  In step S504, the normal line calculation unit 304 generates a normal map image (mapN) based on the distance information acquired from the distance calculation unit 303. Specifically, as described in FIG. 4, the subject normal vector N is calculated in units of pixels, and the direction cosine for each coordinate axis direction is obtained. The obtained direction cosine is expressed by an arbitrary bit width to obtain a normal weighting map (mapN).

  In step S505, the virtual light source reflection component calculation unit 305 generates a weighted map image (mapL) using the virtual light source. Specifically, the light source direction vector -L of the virtual light source is calculated in units of pixels, and the direction cosine for each coordinate axis direction is obtained. The obtained direction cosine is expressed by an arbitrary bit width to obtain a weighting map (mapL) by a virtual light source.

  In step S506, an area E equal to or greater than an arbitrary threshold is detected in the weighted map image (mapK) based on distance, the weighted map image (mapN) based on the normal of the subject, and the weighted map image (mapL) based on the virtual light source. Specifically, a region E where mapK ≧ TH_Ksatu, mapN ≧ TH_Nsatu, and mapL ≧ TH_Lsatu is detected. The reflection components (Ra, Ga, Ba) of the virtual light source are as shown in the above-described equation (1).

  When this equation (1) is replaced with each weighted map image, the following equation is obtained.

Ra = α × mapL / mapN / mapK × Rt
Ga = α × mapL / mapN / mapK × Gt
Ba = α × mapL / mapN / mapK × Bt
That is, the region E that satisfies mapK ≧ TH_Ksatu, mapN ≧ TH_Nsatu, and mapL ≧ TH_Lsatu is a region that is easily saturated when the gain value α that is the strength of the virtual light source is increased.

  In step S507, it is determined whether or not the area corresponding to the area E is small. If the area corresponding to the area E is small, the process proceeds to step S509. If the area corresponding to the area E is large, the process proceeds to step S508. In step S508, the values of TH_Ksatu, TH_Nsatu, and TH_Lsatu are increased, and the process proceeds to step S506 to detect the region E again. Then, step S508 is repeated until the area E becomes small to some extent. The reason why Step S508 is repeated until the area E becomes smaller is to reduce the amount of calculation when obtaining the maximum gain value αmax. Whether or not the area E is large is determined based on whether or not the area E is larger than a predetermined threshold TH_areaE.

In step S509, the regain maximum gain value αmax in which the region E is not saturated is calculated. Specifically, when the virtual light source addition processing unit 306 adds the output (Rt, Gt, Bt) of the degamma processing unit 302 and the virtual light source component (Ra, Ga, Ba) obtained in step S506, (Rout , Gout, Bout) is calculated as the maximum regaining gain αmax. That means
(Rout, Gout, Bout) = (Rt, Gt, Bt) + (Ra, Ga, Ba)
Therefore, by expanding the above equation, αrmax, αgmax, and αbmax are calculated by the following equations. Routmax, Goutmax, and Boutmax are 2 ⁸ −1 = 255 if ⁸ bits, and 2 ¹⁰ −1 = 1023 if 10 bits. The αmax value is the smallest value among αrmax, αgmax, and αbmax.

αrmax = ((Routmax / Rt) −1) · (K ² / (− L · N))
αgmax = ((Goutmax / Gt) −1) · (K ² / (− L · N))
αbmax = ((Boutmax / Bt) −1) · (K ² / (− L · N))
FIG. 6 shows an example in which the maximum gain value αmax of the relighting processing unit 114 is determined by the above processing and the relighting processing unit performs the relighting processing. FIG. 6A shows an example of a photographed image before the relighting process, and FIG. 6B shows an example of a photographed image after the relighting process. The dark subject in FIG. 6A is brightly corrected as shown in FIG. 6B by performing relighting processing with a virtual light source.

  Next, processing when the system control unit 50 performs auto-relighting using the maximum gain value αmax of the relighting processing calculated as described above will be described with reference to the flowchart of FIG.

  In step S <b> 701, it is determined whether relighting processing is selected by a user operation via the operation unit 120. That is, it is determined whether or not processing by the relighting processing unit 114 is performed. When rewriting is performed, the process proceeds to step 702.

  In step S702, the subject is photographed. In step S703, a relighting gain value αmax at which the relighted image is not saturated is calculated. Specifically, as described above with reference to FIG.

  In step S704, based on the maximum value αmax of the relighting gain calculated in step S703, the value of α when the relighting intensity is large, medium, or small is calculated with a value less than or equal to the maximum value αmax (maximum value or less). . For example, as shown in the display example of the display unit 109 shown in FIG. 8, the relighting intensity is expressed as 0% to 100%, the gain value αmax is 100%, and the value of α when the intensity is large / medium / small is set. calculate.

  In step S705, it is determined whether an arbitrary relighting intensity is set by the user via the operation unit 120. If it has been set, the process proceeds to step S706. If it has not been set, step S705 is repeated.

In step S706, brightness automatic correction processing by relighting is performed based on the gain value α set by the user. In particular,
(Ra, Ga, Ba) = α × (−L · N) / K ² × (Rt, Gt, Bt)
Substituting the value of α set by the user for α in the equation (1). Through the above processing, the relighting processing unit 114 can perform appropriate auto-relighting that does not cause undercorrection or overcorrection.

  In the present embodiment, the user sets an arbitrary relighting intensity via the operation unit 120, but the present invention is not limited to this. For example, the α value may be determined based on the luminance information of the subject acquired by the shadow information acquisition unit 208 below the relighting maximum gain value αmax.

  Further, in the present embodiment, the case of performing bright correction by relighting has been described, but conversely, relighting that darkens may be performed. In that case, the gain value α of the virtual light source is made negative. As a result, the signal based on the brightness of the virtual light source is subtracted. In this case, it is necessary to calculate a gain value αmin that does not cause black crushing.

  The calculation method of the distance D between the position of the virtual light source and the processing target pixel is not limited to the method of the present embodiment, and any calculation method may be used. For example, the camera position and the subject position may be acquired as a three-dimensional position and calculated by a three-dimensional distance.

  In addition, when a virtual light source is added, the calculation is performed using an equation that is inversely proportional to the square of the distance, but the amount of addition of the virtual light source is not limited to that calculated by this method. For example, an expression that is inversely proportional to the distance D or an expression in which the irradiation range changes in a Gaussian distribution may be used.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

50: System control unit, 101: Optical system, 102: Shutter, 103: Imaging unit, 104: A / D converter, 105: Image processing unit, 106: Image memory, 107: Memory control unit, 109: Display unit, 110: Codec section, 111: Recording I / F, 112: Recording medium, 113: Face detection section, 114: Relighting processing section, 120: Operation section, 121: Non-volatile memory, 122: System memory, 123: Distance sensor

Claims (13)

Setting means for setting a virtual light source, which is a virtual light source, corresponding to a portion whose brightness is corrected in the image,
Correction means for correcting the brightness of the portion for correcting the brightness using the virtual light source set by the setting means;
A calculation unit that calculates a maximum value of a correction amount of brightness by the correction unit based on a distribution of luminance values of the image and a distribution of reflection when the virtual light source is reflected by a subject;
An image processing apparatus comprising:   The said correction | amendment means correct | amends brightness by adding or subtracting the signal based on the brightness of the said virtual light source with respect to the signal of the part which correct | amends the said brightness. Image processing device.   The image processing apparatus according to claim 2, wherein the brightness correction amount is a gain value of a signal based on brightness of the virtual light source.   The calculation unit includes an analysis unit that analyzes a shadow state of a subject, and a reflection component calculation unit that calculates a reflection distribution of the virtual light source set by the setting unit, and the shadow state output by the analysis unit 4. The image processing according to claim 1, wherein a maximum value of a correction amount of brightness is obtained based on the reflection distribution of the virtual light source calculated by the reflection component calculation unit. apparatus.   The image processing apparatus according to claim 4, wherein the shaded state analyzed by the analyzing unit is a distribution of light and darkness generated in a subject by an environmental light source.   The said reflection component calculation means is provided with the distance information acquisition means which acquires the distance information of an imaging device and a to-be-photographed object, and the normal line information acquisition means to acquire the normal line information of a to-be-photographed object. An image processing apparatus according to 1.   The calculating means determines whether or not the area is likely to be saturated from the shaded state output by the analyzing means, and calculates the maximum value of the brightness correction amount in the area determined to be saturated. The image processing apparatus according to any one of claims 4 to 6.   8. The correction unit according to claim 1, wherein the correction unit automatically corrects the brightness of a portion for correcting the brightness with a correction amount equal to or less than a maximum correction amount calculated by the calculation unit. The image processing apparatus according to item 1.   The image processing apparatus according to claim 1, further comprising a setting unit that allows a user to set a correction amount that is equal to or less than a maximum value of the correction amount calculated by the calculation unit.   The image processing apparatus according to claim 9, wherein the correction unit automatically corrects the brightness of the portion where the brightness is corrected using a correction amount set by the user by the setting unit. A setting step of setting a virtual light source, which is a virtual light source, corresponding to a portion whose brightness is corrected in the image,
A correction step of correcting the brightness of the portion for correcting the brightness using the virtual light source set in the setting step;
A calculation step of calculating a maximum value of a correction amount of brightness by the correction unit based on a distribution of luminance values of the image and a distribution of reflection when the virtual light source is reflected by a subject;
An image processing method comprising:   The program for making a computer perform each process of the image processing method of Claim 11.   A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 11.