JP2021087125A - Image processing device, control method thereof, and program - Google Patents

Abstract

To eliminate a complicated work such as manually switching of a color filter by a user, and enable background color shift processing with a simple operation.SOLUTION: An image processing device includes white balance correction means that corrects the white balance of a captured image, virtual lighting means that imparts the effect of irradiating a subject of the captured image with virtual light, and means that calculates first white balance gain on the basis of the color of a light source irradiating the subject, and the white balance correction means sets second white balance gain different from the first white balance gain, and in the virtual lighting means, the color of the virtual light is controlled such that a region to which the effect of irradiating the virtual light is given approaches the color when white balance correction means sets the first white balance gain.SELECTED DRAWING: Figure 5

Description

The present invention relates to an image processing device, and more particularly to an image processing device that corrects the color of an input image.

Conventionally, in photography, background color shift photography has been performed in which only the background area is photographed by changing the color balance. In the background color shift shooting method, a color filter of an arbitrary color is attached to the light emitting portion of the strobe device, and the white balance gain is set so that the color of the strobe light passing through the color filter becomes white. As a result, an appropriate white balance is applied to the subject area exposed to the strobe light, and an image in which the color balance is shifted to the opposite color of the color of the color filter can be taken in the background area where the strobe light does not reach.

Further, Patent Document 1 discloses a method of changing the color temperature of strobe light by attaching a filter holder to which a color filter is attached in front of a light emitting portion of a strobe device. In Patent Document 1, identification information for identifying the type of the color filter is added, and the identification information of the color filter is read by a reading unit provided on the strobe device side to identify the type of the color filter attached to the filter holder. doing. Then, the strobe device determines the color temperature of the irradiation light from the type of the identified color filter and transmits it to the mounted camera.

JP-A-2009-20298

By using the apparatus of Patent Document 1, it is possible to save the trouble of manually setting the white balance according to the type of the color filter attached to the strobe apparatus.

However, in the apparatus of Patent Document 1, the user needs to select and switch the color filter. When performing background color shift shooting, it is complicated because the user must manually select a color filter of the opposite color of the color to be set as the background color.

Therefore, an object of the present invention is to reduce the trouble of complicated work such as manually switching the color filter by the user, and to easily change the background color.

In order to solve the above problems, the image processing apparatus according to the present invention imparts a white balance correction means for performing white balance correction on a captured image and an effect of irradiating a subject of the captured image with virtual light. It has a virtual lighting means and a means for calculating a first white balance gain based on the color of the light source irradiating the subject, and the white balance correction means is different from the first white balance gain. The second white balance gain is set, and the virtual lighting means so that the region to which the effect of the virtual light is applied approaches the color when the white balance correction means sets the first white balance gain. It is characterized in that the color of the virtual light is controlled.

According to the present invention, it is possible to reduce the troublesome work of the user and easily change the background color.

It is a block diagram which shows the structure of the digital camera in this invention. It is a block diagram which shows the structure of the image processing part in this invention. It is a block diagram which shows the structure of the rewriting processing part in this invention. It is a figure which shows the relationship between a normal and a virtual light source in this invention. It is a figure which shows the parameter calculation flow of the virtual light source in this invention. It is a figure which shows the example of the block division in this invention. It is a figure which shows the irradiation angle and position of the virtual light source in this invention. It is a figure which shows the irradiation range of the virtual light source in this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this embodiment, an example applied to a digital camera as an image processing device will be described.

<First Embodiment>
Hereinafter, the digital camera according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 8.

FIG. 1 is a block diagram showing a configuration example of a digital camera according to an embodiment of the present invention.

In FIG. 1, 100 is an entire digital camera, 101 is a lens group including a zoom lens and a focus lens, 102 is a shutter having an aperture function, and 103 is an image pickup composed of a CCD or CMOS element that converts an optical image into an electric signal. Department. Reference numeral 104 is an A / D converter that converts an analog signal into a digital signal, and 105 is an image data output from the A / D converter 104 that is subjected to white balance processing, γ processing, contour enhancement, color correction processing, and the like. It is an image processing unit that performs various image processing. 106 is an image memory, 107 is a memory control unit that controls the image memory 106, 108 is a D / A converter that converts an input digital signal into an analog signal, 109 is a display such as an LCD, and 110 is a compression codec for image data. -The codec part to be decrypted.

Reference numeral 111 is an interface I / F with the recording medium 200, and 112 is a recording medium such as a memory card or a hard disk. Reference numeral 113 denotes a face detection processing unit that detects a region in which a face is reflected in the captured image, and 114 is a rewriting processing unit that performs rewriting processing on the captured image. Reference numeral 50 denotes a system control unit that controls the entire system of the digital camera 100.

Further, 121 is a non-volatile memory such as EEPROM for storing programs and parameters, and 122 is a system memory for developing constants and variables for operation of the system control unit 50, programs read from the non-volatile memory 124, and the like. Is. Reference numeral 123 denotes a light source device such as a strobe, and 124 is a distance measuring sensor that measures the distance to the subject and outputs distance information corresponding to each pixel of the photographing pixel as a two-dimensional distance map image.

Next, the basic operation at the time of shooting a subject with the digital camera 100 configured as described above will be described. The imaging unit 103 photoelectrically converts the light incident through the lens 101 and the shutter 102, and outputs the light as an input image signal to the A / D converter 104. The A / D converter 104 converts the analog image signal output from the image pickup unit 103 into a digital image signal and outputs the analog image signal to the image processing unit 105.

The image processing unit 105 performs color conversion processing such as white balance correction, γ processing, contour enhancement processing, and the like on the image data from the A / D converter 104 or the image data from the memory control unit 107. Further, the image processing unit 105 performs a predetermined evaluation value calculation process (not shown) using the face detection result of the face detection unit 113 and the captured image data, and the system control unit is based on the obtained evaluation value result. 50 performs exposure control and distance measurement control. As a result, TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, AWB (auto white balance) processing, and the like are performed.

The image data output from the image processing unit 105 is written to the image memory 106 via the memory control unit 107. The image memory 106 stores the image data output from the imaging unit 103 and the image data to be displayed on the display unit 109.

Further, the D / A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies it to the display unit 109. The display unit 109 displays on a display such as an LCD according to the analog signal from the D / A converter 108.

The codec unit 110 compresses and encodes the image data recorded in the image memory 106 based on standards such as JPEG and MPEG. The system control unit 50 associates the encoded image data and stores it in the recording medium via the recording interface 111.

The basic operation when shooting a subject has been described above.

In addition to the above basic operations, the system control unit 50 realizes each process of the present embodiment described later by executing the program recorded in the non-volatile memory 124 described above. The program referred to here is a program for executing various flowcharts described later in this embodiment. At this time, the constants and variables for the operation of the system control unit 50, the program read from the non-volatile memory 121, and the like are expanded in the system memory 122.

In this embodiment, the image processing described later is performed under the control of the system control unit 50, but the hardware configuration is not limited to this embodiment. For example, one hardware may function as various means according to an input signal or a program, or a plurality of hardware may cooperate to function as one means. Further, a part of the processing may be realized by using a circuit instead of the processing according to the so-called software program.

Further, the display unit and the recording medium do not necessarily have to be possessed by the device itself, and may be configured to connect an external display or an external recording medium. It suffices that the device has at least means for performing display control and recording control.

Next, the details of the image processing unit 105 will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the image processing unit 105.

In FIG. 2, 200 is a simultaneous processing unit, 201 is a WB amplification unit, 202 is a brightness / color signal generation unit, 203 is a contour enhancement processing unit, 204 is a brightness gamma processing unit, 205 is a color conversion processing unit, and 206 is a color. The γ processing unit and 207 are color difference signal generation units. Reference numeral 208 denotes an evaluation value generation unit that outputs feature information of the image.

Next, the processing in the image processing unit 105 will be described. The image signal input from the A / D conversion unit 104 of FIG. 1 is input to the image processing unit 105.

The image signal input to the image processing unit 105 is input to the simultaneous processing unit 200. The simultaneous processing unit 200 performs simultaneous processing on the input Bayer RGB image data to generate color signals R, G, and B. The WB amplification unit 201 applies a gain to the RGB color signal based on the white balance gain value set by the user according to the flow described later, and performs the white balance correction. The RGB signal output by the WB amplification unit 201 is input to the luminance / color signal generation unit 202. Luminance / color signal generation unit 202 A luminance signal Y is generated from the RGB signal, the generated luminance signal Y is output to the contour enhancement processing unit 203, and the color signal RGB is output to the color conversion processing unit 205.

The contour enhancement processing unit 203 performs contour enhancement processing on the luminance signal and outputs it to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y and outputs the luminance signal Y to the image memory 106.

The color conversion processing unit 205 converts the RGB signal into a desired color balance by performing a matrix calculation or the like. The color gamma processing unit 206 performs gamma correction on the RGB color signal. The color difference signal generation unit 207 generates color difference signals RY and BY signals from RGB signals.

The image signals Y, RY, and BY signals output to the image memory 106 are compressed and encoded by the codec unit 110 and recorded on the recording medium 200.

Further, the output RGB signal of the simultaneous processing unit 200 is also input to the evaluation value generation unit 208. The evaluation value generation unit 208 divides the input image into a plurality of blocks as shown in FIG. 6, and calculates the average value of the RGB values for each block as the evaluation value. The evaluation value generation unit 208 outputs the calculated evaluation value to the system memory 122.

Next, the configuration and operation of the rewriting processing unit 114 will be described with reference to FIG.

When the "background color shift mode" is turned on by the flow described later, the data output from the image processing unit 105 is input to the rewriting processing unit 114, and the effect of irradiating the virtual light by the virtual light source is given. Performs (virtual lighting) processing.

FIG. 3 is a block diagram showing the configuration of the rewriting processing unit 114.

In FIG. 3, 301 is an RGB signal conversion unit that converts input luminance / color difference signals (Y, BY, RY) into RGB signals, and 302 is a degamma processing unit that performs degamma processing.

Reference numeral 303 denotes a virtual light source processing unit that calculates a component reflected by the virtual light source on the subject and reflects the influence of the virtual light source on the input image.

Further, 304 is a gamma processing unit that applies a gamma characteristic to an RGB signal, and 305 is a luminance / color difference signal conversion unit that converts an RGB signal into a luminance / color difference signal (Y, BY, RY).

Further, 306 is a normal calculation unit for calculating a normal from distance information, and 307 is a virtual light source reflection component calculation unit for calculating a component reflected by a virtual light source on a subject.

The operation of the rewriting processing unit 114 having the above configuration will be described.

The rewriting processing unit 114 reads out the luminance / color difference signals (Y, BY, RY) recorded in the image memory 106 and uses them as inputs. The RGB signal conversion unit 301 converts the input luminance / color difference signals (Y, BY, RY) into RGB signals and outputs them to the degamma processing unit 302.

The degamma processing unit 302 performs an operation of a characteristic opposite to the gamma characteristic multiplied by the gamma processing unit of the image processing unit 105 and converts it into linear data. The degamma processing unit 302 outputs the RGB signals (Rin, Gin, Bin) after linear conversion to the virtual light source processing unit 303.

The normal calculation unit 306 calculates a normal map from the subject distance information acquired from the distance measuring sensor (124 in FIG. 1). The subject distance information is two-dimensional distance information obtained in pixel units of a captured image. As for the method of generating the normal map from the subject distance information, a known technique is used, but a specific processing example will be described with reference to FIG. FIG. 4 is a diagram showing the relationship between the camera shooting coordinates and the subject.

When calculating the normal N402 of the subject 401 shown in FIG. 4, the gradient information can be calculated from the difference ΔD of the distance (depth) D with respect to the horizontal difference ΔH of the captured image, and the normal N can be calculated from the gradient information. It is possible. The normal calculation unit 306 calculates the normal information N corresponding to each pixel of the photographed image by performing the above processing on each pixel of the photographed subject. The normal calculation unit 306 outputs the normal information calculated for each pixel of the captured image as a normal map to the virtual light source processing unit 303.

The virtual light source processing unit 303 applies the influence of the reflection component reflected by the installed virtual light source to the subject to the input signal (Rin, Gin, Bin).

The calculation of the reflection component of the virtual light source will be described with reference to FIG. In FIG. 4, consider the case where the virtual light source is installed at the position of 403. The reflection component at the horizontal pixel position H1 (the vertical pixel position is omitted for simplification of explanation) of the captured image taken by the camera 100 is proportional to the inner product of the normal N1 at the camera coordinates H1 and the direction vector L1 of the virtual light source. .. Then, the value is inversely proportional to the square of the distance K1 between the virtual light source and the subject position.

When this relationship is expressed by a mathematical formula, the reflection components (Rout, Gout, Bout) by the virtual light source are as follows.
(Equation 1)
Rout = β × Rin × Rcolor + (1-β) × Rin
Gout = Gin
Bout = β × B × Bcolor + (1-β) × Bin

Here, Rcolor and Bcolor are parameters that control the color of the virtual light source, and are determined based on the processing described later. Further, β indicates the degree of contribution of the virtual light source to the subject including the positional relationship between the virtual light source and the subject, and is calculated by the following formula.

Here, α is the magnitude of the gain of the virtual light source, L is the three-dimensional direction vector of the virtual light source, N is the three-dimensional normal vector of the subject, and K is the distance between the virtual light source and the subject. However, β ≦ 1 is set.

According to the above equation, the pixel strongly affected by the virtual light source has β close to 1, and is strongly subjected to color correction by Rcolor and Bcolor. On the other hand, the pixels that are not affected by the virtual light source have β = 0, and the input signals (Rin, Gin, Bin) are output as they are.

The above is an example when there is one virtual light source, but it is possible to set multiple virtual light sources. In that case, the β value is calculated for each light source and the influence of each virtual light source is weighted and added. To generate an output signal.

The image signals (Rout, Gout, Bout) output from the virtual light source addition processing unit 303 are input to the gamma processing unit 304. The gamma processing unit 304 performs gamma correction on the RGB input signal, and outputs the RGB signals (R'out, G'out, B'out) after the gamma to the color difference signal generation unit 305. The color difference signal generation unit 305 generates luminance Y, color difference signals RY, and BY signals from RGB signals.

The above is the operation of the rewriting processing unit 114.

The system control unit 50 stores the luminance / color difference signal output by the rewriting correction unit 114 in the image memory 106 under the control of the memory control unit 107, and then compresses and encodes it in the codec unit 110. Also, recording is performed on the recording medium 112 via the I / F 111.

Next, a processing flow in which the system control unit 50 determines various parameters such as the color parameters (Rcolor, Bcolor) of the virtual light source and the direction of the light source will be described with reference to FIG. When the processing of the image processing unit 105 is completed, the system control unit 50 calculates the rewriting parameters according to the flow of FIG.

In step S501 in FIG. 5, it is determined whether or not the "background color shift mode" is set to ON. Here, prior to the flow shown in FIG. 5, it is assumed that the user inputs whether to turn the "background color shift mode" ON (enabled) or OFF (disabled) by a menu operation (not shown). ..

The "background color shift mode" is a mode in which the adjustment result of the manual white balance gain is mainly reflected in the background area when the white balance gain is manually adjusted by the user. When the "background color shift mode" is ON, the area exposed to the virtual light source is controlled so as to have an appropriate color balance (white approaches white) regardless of the value of the manual white balance gain. As a result, the subject area exposed to the virtual light source always approaches the state in which the white balance is matched, and the color tone for which the user manually adjusts the white balance is reflected in the background area not exposed to the virtual light source.

If the "background color shift mode" is ON in step S501, the process proceeds to step S502. If the "background color shift mode" is OFF, the process ends.

In step S502, the input of the manual white balance gain value is accepted by the menu operation by the user. This is done, for example, by allowing the user to specify a preferred color temperature between 2000K and 9000K. The white balance gain value corresponding to the color temperature specified by the user is set in the WB amplification unit (201 in FIG. 2) as a manual white balance gain. The manual white balance gain at this time is described as follows.
Manual R gain = Rshift
Manual G gain = 1
Manual B gain = Bshift

For example, consider the case where the manual white balance is set to 8000K when the color temperature of the ambient light is 3000K. In this case, since the gain for 8000K is set for the ambient light of 3000K, the output from the WB amplification unit (201 in FIG. 2) is an image in which the entire image is shifted to a reddish color. It will be output.

In step S503, the color temperature of the ambient light is estimated from the evaluation value generated by the evaluation value generation unit (208 in FIG. 2), and the white balance gain for appropriately correcting the ambient light is calculated. Specifically, a block having a color ratio close to white (achromatic color) is extracted from the block evaluation value (FIG. 6) output from the evaluation value generation unit, and the average RGB value of the extracted plurality of blocks is calculated. .. The white balance gain that corrects the calculated block average RGB value close to white (achromatic color) to achromatic color (R = G = B) is calculated as the white balance gain for ambient light correction. The white balance gain for ambient light correction is expressed as follows.
Ambient light correction R gain = Rambient
Ambient light correction G gain = 1
Ambient light correction B gain = Bambient

In step S504, the manual white ballast gain set in step S502 is compared with the ambient light correction white balance gain calculated in step S503, and it is determined whether or not there is a difference of a predetermined threshold value or more. If there is a difference of the predetermined threshold value or more, the process proceeds to step S505, and if there is no difference, the process ends.

In step S505, the light source color of the virtual light source used for rewriting is determined. In this embodiment, the light source color of the virtual light source is determined based on the manual white balance gain acquired in step S502 and the ambient light correction white balance gain calculated from the ambient light. Specifically, as a result of irradiating the virtual light source, the color of the virtual light source (Rcolor, Bcolor) is based on the following formula so as to cancel the manual white balance gain component and approach the state in which the ambient light correction white balance gain is applied. To set.
Rcolor = Rambient / Rshift
Bcolor = Bambient / Bshift

In step S506, the irradiation direction and irradiation range of the virtual light source are determined based on the characteristics of the main subject. In this embodiment, the main subject is detected, and the irradiation direction of the virtual light source is determined based on the orientation of the main subject. In this embodiment, the position and direction of the virtual light source are controlled so as to irradiate the virtual light source from the front side of the main subject in a direction inclined by 45 degrees with respect to the optical axis direction of the camera. A specific example of this will be described with reference to FIG.

FIG. 7 is a diagram showing a positional relationship between the main subject and the information of the camera 100 and the virtual light source. In FIG. 7, 100 indicates the position of the camera, and 701 indicates the position of the detected main subject. As shown in FIG. 7, the direction tilted by 45 degrees with respect to the optical axis of the camera and the direction in which the angle with the main subject front direction vector becomes smaller is set as the light source direction. Further, the position of the virtual light source is determined at a position (position 702 in FIG. 7) separated from the position of the main subject by a predetermined distance.

By setting the light source direction so that the subject is illuminated diagonally as described above, the color on the front side of the subject is corrected for correction, and the side surface becomes a color close to the background, making it possible to create a three-dimensional effect. Become.

Next, the irradiation range of the virtual light source is determined based on the area of the main subject. FIG. 8 shows an example in which the main subject is a person. FIG. 8A shows a photographed image, and 801 is a person area of the main subject. FIG. 8B is an extract of the main subject area (personal area), and 802 shows the main subject area. Further, the light irradiation range 803 is determined so as to include the main subject area. An example of the irradiation range of the virtual light source determined to include the subject area 803 is shown in FIG. 804 of FIG. 8C. Here, the light source intensity characteristic of the virtual light source is such that the intensity of the virtual light source gradually weakens from the irradiation center to the periphery, instead of irradiating the entire irradiation range with a uniform intensity. Determine the characteristics of the light source. Therefore, when irradiating the main subject with virtual light, the intensity of the virtual light decreases toward the edge of the main subject area.

Returning to FIG. 5, in step S507, parameters such as the color, position, and irradiation range of the virtual light source calculated in the above step are set in the virtual light source processing unit (303 in FIG. 3).

The processing flow for determining the parameters of the virtual light source has been described above. By performing the lighting process based on the parameters of the virtual light source determined as described above, the color of the subject area illuminated by the light of the virtual light source becomes the color shot with the appropriate white balance (white is reproduced in white). Get closer.

As described above, in this embodiment, the color of the virtual light source is controlled based on the difference between the white balance value manually set by the user and the white balance value based on the ambient light. This makes it possible to bring the color of the subject area irradiated with the light of the virtual light source closer to the color captured with an appropriate white balance with respect to the ambient light.

In addition, the irradiation direction of the virtual light source is controlled so as to irradiate the virtual light source diagonally according to the direction of the camera. This makes it possible to generate an image having a stereoscopic effect as compared with the case where the color of the subject area is simply changed uniformly.

In the above embodiment, the background color shift control using the virtual light source is described when the strobe device is not used, but the background color shift control using the virtual light source is also performed when the strobe device is used. Is possible.

In that case, instead of calculating the white balance gain from the color of the ambient light in steps S503 and S504 of FIG. 5, the white balance gain corresponding to the strobe light (making the color of the strobe light white) is acquired. The white balance gain corresponding to the strobe light shall be recorded in advance in the non-volatile memory 121 of the camera 100. The white balance gain corresponding to the strobe light is described as follows.
Strobe light correction R gain = R strobe
Strobe light correction G gain = 1
Strobe light correction B gain = B strobe

The colors of the virtual light source (Rcolor, Bcolor) are also calculated as follows by replacing the white balance gain of the ambient light of S505 with the white balance gain of the strobe light.
Rcolor = Rstrobe / Rshift
Bcolor = Bstrom / Bshift

The method of determining the direction and position of the virtual light source may be controlled in the same manner as described above, but may be determined by referring to the positional relationship between the actual strobe device and the camera. For example, when the strobe device is placed at a position different from that of the camera body, the virtual light source is placed on the side where the strobe device is located.

Further, instead of determining the position and range of the virtual light source based on the position of the strobe device, it is also possible to directly calculate the β value (Equation 2) indicating how the virtual light source hits by using the strobe light emission image. It is possible. In this case, it is calculated so that β approaches 1 as the difference in brightness between non-strobe light emission and strobe light emission increases. Since β becomes large in the place where the strobe light is strongly hit, it is possible to generate an image with an appropriate color balance (white becomes white) in the region where the strobe light is hit.

By controlling as described above, the background color can be shifted even when the strobe light is shining.

Further, in this embodiment, the direction of the virtual light source is set to be a direction shifted at an angle of 45 degrees with respect to the optical axis direction of the camera, but the method of determining the direction of the virtual light source is not limited to this, and the light of the camera is used. Any method may be used as long as the direction is deviated from the axial direction by a predetermined angle or more.

Further, in the above embodiment, the irradiation range of the virtual light source is set so as to include the person area as the main subject, but any configuration may be adopted as long as the foreground and the background can be separated. For example, the main subject is not limited to a person, and other animals or object areas may be detected and the same processing may be performed as the main subject. In addition to the method of using object recognition such as person recognition, it is also possible to compare images that emit light with strobe light and images that do not emit light, and to adopt a configuration in which a region having a large difference is the main subject.

Further, the distance information may be used to determine the light irradiation range so as to include the subject area at the same distance as the main subject, or only the distance information is referred to and only the area where the distance is close to the camera is mainly used. It may be a subject area.

<Other Embodiments>
Further, in the above embodiment, an example in which the camera performs image processing has been described, but the image taken by the camera is configured to perform white balance correction and rewriting processing when another image processing device performs so-called development processing. May be.

The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed.

Claims (11)

A white balance correction means that corrects the white balance of the captured image,
A virtual lighting means that imparts the effect of irradiating the subject of the captured image with virtual light,
It has a means for calculating a first white balance gain based on the color of the light source irradiating the subject.
The white balance correction means sets a second white balance gain different from the first white balance gain, and sets the second white balance gain.
The virtual lighting means controls the color of the virtual light so that the region to which the effect of the virtual light is applied approaches the color when the white balance correction means sets the first white balance gain. An image processing device characterized by. It also has a mode to change the background color of the image,
In the virtual lighting means, when the mode for changing the background color of the image is effective, the white balance correction means sets the first white balance gain in the region to which the effect of being irradiated with the virtual light is given. Control the color of the virtual light so that it approaches the color when
When another mode different from the mode for changing the background color of the image is effective, the area to which the effect of irradiating the virtual light is applied is the area where the white balance correction means obtains the second white balance gain. The image processing apparatus according to claim 1, wherein the color of the virtual light is controlled so as to approach the color when the setting is made. The virtual lighting means imparts the effect of being irradiated with the virtual light when the difference between the first white balance gain calculated based on the color of the ambient light and the second white balance gain is larger than a predetermined threshold value. The image processing apparatus according to claim 1 or 2, wherein the color of the virtual light is controlled so that the area to be processed approaches the color when the white balance correction means sets the first white balance gain. .. The virtual lighting means calculates the light source intensity characteristic of the virtual light for each subject area, and the region where the light source intensity characteristic of the virtual light is higher is the case where the white balance correction means sets the first white balance gain. The image processing apparatus according to any one of claims 1 to 3, wherein the color of the virtual light is controlled so as to approach the color. The virtual lighting means
The image processing device according to any one of claims 1 to 4, wherein the irradiation direction of the virtual light is set in a direction different from the optical axis direction of the image pickup device that has captured the captured image. Further having a detection means for detecting the orientation of the main subject of the captured image,
The image processing apparatus according to claim 5, wherein the virtual lighting means controls the irradiation direction of virtual light according to the direction of the main subject. Further having an extraction means for extracting a region of the main subject of the captured image,
The image processing apparatus according to any one of claims 1 to 6, wherein the virtual lighting means determines an irradiation range of virtual light so as to include an area of the main subject. The image processing apparatus according to claim 7, wherein the virtual lighting means controls the virtual light so that the intensity of the virtual light is reduced at an end portion of a region of the main subject. Further having a means for acquiring information regarding the distance between the subject of the captured image and the imaging device that captured the captured image.
The image processing device according to claim 7 or 8, wherein the main subject area is closer to the image pickup device than the distance of the subject is closer to the image pickup device. It is a control method of an image processing device.
A white balance correction process that corrects the white balance of the captured image,
A virtual lighting process that imparts the effect of irradiating the subject of the captured image with virtual light, and
It has a calculation step of calculating a first white balance gain based on the color of the light source irradiating the subject.
In the white balance correction step, a second white balance gain different from the first white balance gain is set.
In the virtual lighting step, the color of the virtual light is controlled so that the region to which the effect of the virtual light is applied approaches the color when the first white balance gain is set in the white balance correction step. A control method for an image processing device. A program that can be executed by a computer, which causes the computer to function as the image processing device according to any one of claims 1 to 9.

Images