JP2019004290A - Image processing apparatus, imaging apparatus, image processing method and program - Google Patents

Abstract

To provide a technology that calculates a specular reflection component considering a color of a processing target pixel for the processing target pixel in re-lighting an image on the basis of a virtual light source.SOLUTION: An image processing apparatus that sets a virtual light source having a predetermined color for an image, and that performs image processing such that a light irradiation effect due to the virtual light source is imparted to the image is provided, and the image processing apparatus comprises: first acquisition means that acquires a first specular reflection component of light due to the virtual light source having the predetermined color for a processing target pixel in the image; second acquisition means that acquires a second specular reflection component of light due to the virtual light source corresponding to a case in which the virtual light source has a color of the processing target pixel instead of the predetermined color, for the processing target pixel; and addition means that adds the first specular reflection component and the second specular reflection component using a predetermined ratio for the processing target pixel.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an image processing device, an imaging device, an image processing method, and a program.

  2. Description of the Related Art Conventionally, a technique for performing relighting based on a virtual light source on a subject in an image after shooting is known. According to this technique, the user can add an arbitrary shadow to the subject of the captured image. Further, in lighting using a virtual light source, a technique for simulating a component (specular reflection component) in which light from a virtual light source is reflected on the surface of the subject at a reflection angle that is the same as an incident angle and adding the component to the subject is known. According to this technology, it is possible to reproduce the texture of the subject such as gloss. However, since the specular reflection component is generated based on the information of the light source, it may not match the color of the subject even if it is added to the subject as it is. Therefore, Patent Document 1 proposes a method for reproducing the color of a desired subject by changing the color information of the specular reflection component based on the distance from the specular reflection center.

JP 2008-077741A

  Depending on the type of subject, a sharp color change may occur partially in the image. For example, when the subject is a person, there are pores and moles on the person's skin, and a sharp color change may occur in this part. However, the technique of Patent Document 1 does not consider such a partial color change of the subject, and the specular reflection component added to the subject cannot be adapted to the partial color change of the subject.

  The present invention has been made in view of such a situation, and when performing relighting of an image based on a virtual light source, a specular reflection component considering the color of the pixel to be processed is applied to the pixel to be processed. The purpose is to provide a technique for adding.

  In order to solve the above-mentioned problems, the present invention sets an virtual light source having a predetermined color for an image, and performs image processing so that the effect of light irradiation by the virtual light source is given to the image. A first acquisition unit configured to acquire a first specular reflection component of light from the virtual light source having the predetermined color for the pixel to be processed in the image; and the virtual for the pixel to be processed. In response to the case where the light source has the color of the pixel to be processed instead of the predetermined color, second acquisition means for acquiring a second specular reflection component of the light from the virtual light source; There is provided an image processing apparatus comprising: an adding unit that adds the first specular reflection component and the second specular reflection component to a pixel at a predetermined ratio.

  Other features of the present invention will become more apparent from the accompanying drawings and the following description of the preferred embodiments.

  According to the present invention, when image relighting is performed based on a virtual light source, it is possible to add a specular reflection component in consideration of the color of the processing target pixel to the processing target pixel.

1 is a block diagram showing a configuration of a digital camera 100. FIG. FIG. 2 is a block diagram showing a detailed configuration of an image processing unit 104. The block diagram which shows the detailed structure of the relighting process part 203 based on 1st Embodiment. The flowchart of a relighting process. The figure which shows the relationship between the imaging | photography coordinate of the digital camera 100, and a to-be-photographed object. The flowchart of the calculation process of the brightness | luminance average value Y_ave of a to-be-photographed object. The figure explaining the calculation method of N_value. The figure which shows the relationship between evaluation value Eval and the synthetic | combination ratio a. The figure which shows the relationship between distance C_dist and the synthetic | combination ratio a. The block diagram which shows the detailed structure of the relighting process part 203 based on 2nd Embodiment. The figure which shows the relationship between evaluation value Eval_c and the synthetic | combination ratio a.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. The technical scope of the present invention is determined by the claims, and is not limited by the following individual embodiments. In addition, not all combinations of features described in the embodiments are essential to the present invention. Moreover, it is possible to appropriately combine the features described in different embodiments.

[First Embodiment]
FIG. 1 is a block diagram illustrating a configuration of a digital camera 100 that is an example of an image processing apparatus. In the following description, it is assumed that the target image for the relighting process is an image captured by the digital camera 100. However, the target image of the relighting process is not limited to the captured image, and an arbitrary image (for example, computer graphics) having a predetermined viewpoint in the three-dimensional space can be used as the target image.

  The digital camera 100 includes an optical system 101, an image sensor 102, an A / D conversion unit 103, an image processing unit 104, a distance measurement unit 105, a face detection unit 106, a recording unit 107, a control unit 108, a memory 109, an operation unit 110, A display unit 111 and a strobe 112 are provided.

  The optical system 101 includes a focus lens, a diaphragm, and a shutter. The optical system 101 adjusts the exposure amount by driving a focus lens to focus a subject at the time of shooting, and controlling an aperture and a shutter. The imaging element 102 is a photoelectric conversion element such as a CCD or CMOS that converts the amount of light of the subject imaged in the optical system 101 into an electrical signal by photoelectric conversion. The A / D converter 103 digitizes the input electrical signal. The digitized image signal is subjected to synchronization processing, white balance correction processing, relighting processing, gamma processing, and the like in the image processing unit 104 and is output to the recording unit 107. The distance measuring unit 105 acquires distance information to the subject at the time of shooting and generates a distance map. The distance map is a two-dimensional array indicating distance information to a subject in pixel units of a captured image.

  The face detection unit 106 detects a face region of a person in a captured image in order to determine a subject region that is a target region for relighting processing performed by the image processing unit 104. At this time, the face detection unit 106 detects the coordinates of the facial organs such as the right eye, the left eye, and the mouth, and calculates the distance between the facial organs such as the distance between the right eye and the left eye. The face detection unit 106 sets an elliptical area centered on the barycentric coordinates of the coordinates of each facial organ based on the distance between the facial organs so as not to include the background area, and detects the elliptical area as a facial area. To do.

  The recording unit 107 converts the image information output from the image processing unit 104 into an image format such as JPEG and records it. Alternatively, the recording unit 107 records, in a data format such as RAW, information obtained by photoelectric conversion in the image sensor 102 and information at the time of shooting without performing conversion to JPEG or the like. .

  The control unit 108 controls the operation of the entire digital camera 100. For example, the control unit 108 calculates a target exposure amount by the optical system 101 from the brightness of the subject immediately before shooting. Further, the control unit 108 calculates a predetermined evaluation value based on the captured image, and determines parameters for image processing performed by the image processing unit 104.

  The memory 109 stores information used by the image processing unit 104 and outputs the information to the image processing unit 104 as necessary. The operation unit 110 is a part where the user gives an operation instruction to the digital camera 100. The display unit 111 is, for example, a liquid crystal display installed on the back of the digital camera 100, and displays a screen for assisting operations during shooting, an image stored in the recording unit 107, and the like. . The strobe 112 emits light in response to a user operation and irradiates the subject with light.

  Details of the processing of the image processing unit 104 will be described with reference to FIG. The image processing unit 104 includes a synchronization processing unit 201, a white balance correction unit 202, a relighting processing unit 203, and a gamma processing unit 204.

  A processing flow of the image processing unit 104 will be described. The Bayer array image signal (RGB signal) digitized by the A / D conversion unit 103 is input to the image processing unit 104. In the image processing unit 104, the synchronization processing unit 201 performs synchronization processing on the RGB signals in the Bayer array, and generates color signals R, G, and B. The generated color signals R, G, and B are input to the white balance correction unit 202. The white balance correction unit 202 adjusts the white balance by applying gains to the color signals R, G, and B based on the white balance gain value calculated by the control unit 108. The color signals R, G, and B whose white balance has been adjusted are input to the relighting processing unit 203. The relighting processing unit 203 acquires color signals R, G, and B from the white balance correction unit 202 and acquires a distance map D_MAP from the distance measurement unit 105. Based on the distance map D_MAP, the relighting processing unit 203 generates an image obtained by irradiating the color signals R, G, and B with a virtual light source, and outputs the processed color signals R_out, G_out, and B_out to the gamma processing unit 204 To do. The gamma processing unit 204 performs gamma processing on the color signals R_out, G_out, and B_out after the relighting processing, and outputs the color signals Rg, Gg, and Bg after the gamma processing to the recording unit 107.

  Details of the relighting process will be described with reference to FIGS. 3 and 4. FIG. 3 is a block diagram illustrating a detailed configuration of the relighting processing unit 203. FIG. 4 is a flowchart of the relighting process.

  In the description of the present embodiment, a human face is used as an example of a subject to be irradiated with a virtual light source by relighting processing. Therefore, the digital camera 100 includes a face detection unit 106 that detects a face area in the image. However, the irradiation target of the virtual light source is not limited to a human face, and any object can be set as the irradiation target of the virtual light source. Therefore, the digital camera 100 may include an object detection unit corresponding to the object to be irradiated.

  The relighting processing unit 203 includes a normal calculation unit 301, a shine correction amount calculation unit 302, a shine correction unit 303, a diffuse reflection component calculation unit 304, a specular reflection component calculation unit 305, and a virtual light source addition unit 306.

  First, in S 401, the relighting processing unit 203 acquires a distance map from the distance measuring unit 105, acquires color signals R, G, and B from the white balance correction unit 202, and acquires face area information from the face detection unit 106. To do.

  In step S <b> 402, the normal calculation unit 301 generates a normal map based on the distance map acquired from the distance measurement unit 105. As a method for generating a normal map based on a distance map, any known technique can be used. An example will be described with reference to FIG.

  FIG. 5 is a diagram illustrating the relationship between the shooting coordinates of the digital camera 100 and the subject. For example, when calculating the normal 502 of the subject 501, the normal calculation unit 301 calculates gradient information from the difference ΔD of the distance (depth) D with respect to the difference ΔH in the horizontal direction of the captured image, and calculates the normal 502 from the gradient information. It is possible to calculate. The normal line calculation unit 301 performs a similar process on each captured pixel, thereby calculating a normal line corresponding to each pixel of the captured image and generating a normal map. The normal calculation unit 301 outputs the normal map to the diffuse reflection component calculation unit 304 and the specular reflection component calculation unit 305.

  In step S403, the shine correction amount calculation unit 302 calculates correction amounts Rh, Gh, and Bh of the saturation region (shine) of the subject that is generated due to strong reflection of ambient light during shooting. The shine correction amount calculation unit 302 acquires color signals R, G, and B from the white balance correction unit 202 and acquires face area information from the face detection unit 106. The correction amounts Rh, Gh, and Bh are complementary colors of the subject color that is the correction target color. The correction target color is, for example, an average value of the subject color. As described above, in the description of the present embodiment, the subject is a human face. Therefore, the shine correction amount calculation unit 302 calculates the simple average values of the color signals R, G, and B in the face area detected by the face detection unit 106 as the correction target colors Rt, Gt, and Bt, respectively.

Here, the complementary color is a color having characteristics opposite to those of the target color. If the color signals R, G, and B each take a value in the range of 0 to 255, the complementary color signal, that is, the correction amount Rh, Gh and Bh can be obtained by the following equation 1.

Rh = 255-Rt
Gh = 255−Gt (1)
Bh = 255-Bt

In step S <b> 404, the shine correction unit 303 acquires color signals R, G, and B from the white balance correction unit 202, and acquires correction amounts Rh, Gh, and Bh from the shine correction amount calculation unit 302. The shine correction unit 303 determines whether or not the color signals R, G, and B are saturated for each pixel in the face area. The color signal saturation determination is performed based on whether the value of the color signal is equal to or greater than a threshold value. For example, when the value of the color signal R is 251 and the threshold is 250, it is determined that the color signal R is saturated. Based on the color signals R, G, B and the correction amounts Rh, Gh, Bh, the shine correction unit 303 calculates the shine correction color signals R ′, G ′, B ′. Specifically, the shine correction unit 303 subtracts the shine correction amounts Rh, Gh, and Bh from the color signals R, G, and B in accordance with the following equation 2, so that the color signals R ′, G after the shine correction are performed. ', B' is calculated. For the color signal that is not saturated, the shine correction unit 303 does not perform the calculation according to Equation 2, and outputs the color signal acquired from the white balance correction unit 202 as the color signal after the shine correction.

R ′ = R−ky × Rh
G ′ = G−ky × Gh (2)
B ′ = B−ky × Bh

Here, ky in Equation 2 is the gain of the shine correction amount, and the shine correction amount can be adjusted by adjusting ky.

  In this way, it is possible to obtain post-correction color signals R ′, G ′, and B ′ that suppress the influence of ambient light. The shine correction unit 303 outputs the shine correction color signals R ′, G ′, and B ′ to the diffuse reflection component calculation unit 304, the specular reflection component calculation unit 305, and the virtual light source addition unit 306. As described above, the shine correction process is performed before the calculation of the specular reflection component by the specular reflection component calculation unit 305 described later (before the acquisition).

Ｒｄ＝Ｐｄ×Ｒｗ×Ｒ’
Ｇｄ＝Ｐｄ×Ｇｗ×Ｇ’ ・・・（４）
Ｂｄ＝Ｐｄ×Ｂｗ×Ｂ’
式３において、αは仮想光源の強さ、Ｌは仮想光源の３次元方向ベクトル、Ｎは被写体の３次元法線ベクトル、Ｋは仮想光源と被写体の距離、ｋ_ｄは被写体の拡散反射率である。また、式４において、Ｒｗ、Ｇｗ、Ｂｗは仮想光源の色を示すパラメータである。 In S405, the diffuse reflection component calculation unit 304 calculates the diffuse reflection components Rd, Gd, and Bd. Details of the calculation process will be described with reference to FIG. 5 in which a virtual light source 503 having a predetermined color is arranged. Pay attention to the horizontal pixel position H1 of the image (the vertical pixel position is omitted for the sake of simplicity). The diffuse reflection component of the subject at the horizontal pixel position H1 is proportional to the inner product of the normal N1 at the horizontal pixel position H1 and the direction vector L1 of the virtual light source 503, and is a value inversely proportional to the square of the distance K1 between the virtual light source 503 and the subject position. Become. The diffuse reflection component intensity Pd by the virtual light source can be expressed by the following formula 3, and the diffuse reflection components Rd, Gd, and Bd for each color can be expressed by the following formula 4.

Rd = Pd × Rw × R ′
Gd = Pd × Gw × G ′ (4)
Bd = Pd × Bw × B ′
In Equation 3, α is the intensity of the virtual light source, L is the three-dimensional direction vector of the virtual light source, N is the three-dimensional normal vector of the subject, K is the distance between the virtual light source and the subject, and k _d is the diffuse reflectance of the subject. is there. In Expression 4, Rw, Gw, and Bw are parameters indicating the color of the virtual light source.

  A plurality of virtual light sources can be arranged. In this case, various parameters such as intensity α and colors Rw, Gw, and Bw can be controlled for each virtual light source.

式５において、Ｓは仮想光源の鏡面反射ベクトル、Ｖはデジタルカメラ１００から被写体位置への方向を示す視線方向ベクトル、ｋ_ｓは被写体の鏡面反射率である。また、βは反射した光の広がり具合を示す輝き係数であり、この値が大きくなると鏡面反射特性が急峻になる。 In S406, the specular reflection component calculation unit 305 calculates specular reflection components Rs, Gs, and Bs. Details of the calculation process will be described with reference to FIG. 5 in which a virtual light source 503 having a predetermined color is arranged. Pay attention to the horizontal pixel position H1 of the image (the vertical pixel position is omitted for the sake of simplicity). The intensity of the specular reflection component of the subject at the horizontal pixel position H1 is proportional to the inner product of the specular reflection direction S1 with respect to the subject and the direction V1 (the direction of the line of sight) of the digital camera 100 at the time of shooting from the subject position. The specular reflection component intensity Ps by the virtual light source can be expressed by the following formula 5.

In Equation 5, S is the specular reflection vector of the virtual light source, V is the line-of-sight vector indicating the direction from the digital camera 100 to the subject position, and k _s is the specular reflectance of the subject. Β is a brightness coefficient indicating the extent of the reflected light, and the specular reflection characteristic becomes steep as this value increases.

Next, the specular reflection component calculation unit 305 uses the specular reflection component intensity Ps to calculate a specular reflection component that is reflected on the irradiation target when the virtual light source is irradiated. The specular reflection color components Rs1, Gs1, and Bs1 reflecting the color of the virtual light source can be expressed by the following Expression 6 using the color information Rw, Gw, and Bw of the virtual light source.

Rs1 = Ps × Rw
Gs1 = Ps × Gw (6)
Bs1 = Ps × Bw

The specular reflection component calculation unit 305 also calculates specular reflection color components Rs2, Gs2, and Bs2 reflecting the color of the subject. The “specular reflection color component reflecting the color of the subject” means the specular reflection color component of the virtual light source corresponding to the case where the virtual light source has the subject color instead of the original virtual light source color. The specular reflection color components Rs2, Gs2, and Bs2 reflecting the color of the subject can be expressed by the following Expression 7 using the color signals R ′, G ′, and B ′ after the shine correction.

Rs2 = Ps × R ′
Gs2 = Ps × G ′ (7)
Bs2 = Ps × B ′

Further, the specular reflection component calculation unit 305 performs specular reflection color components Rs1, Gs1, Bs1, specular reflection color components Rs2, Gs2, Bs2, and a combination ratio a that takes a value of 0 to 1 according to the following equation 8: The specular reflection components Rs, Gs, and Bs are calculated.

Rs = a * Rs1 + (1-a) * Rs2
Gs = a × Gs1 + (1−a) × Gs2 (8)
Bs = a * Bs1 + (1-a) * Bs2

  The color of the specular reflection components Rs, Gs, and Bs approaches the virtual light source as the value of the combination ratio a approaches 1, and the color of the specular reflection components Rs, Gs, and Bs as the value of the combination ratio a approaches 0. Approaches the color of the subject. A specific example of the method for determining the composition ratio a will be described later, but the present embodiment is not limited to a specific determination method.

In step S407, the virtual light source adding unit 306 converts the diffuse reflection components Rd, Gd, and Bd calculated in step S405 and the specular reflection components Rs, Gs, and Bs calculated in step S406 into color signals R ′, G after the shine correction. Processing to add to “, B” is performed. The relighting color signals R_out, G_out, and B_out obtained by the addition here can be expressed by the following Expression 9.

R_out = R ′ + Rd + Rs
G_out = G ′ + Gd + Gs (9)
B_out = B ′ + Bd + Bs

  In step S <b> 408, the virtual light source adding unit 306 outputs the relighting color signals R_out, G_out, and B_out to the gamma processing unit 204.

Next, a specific example of the method for determining the composition ratio a in S406 will be described. The specular reflection component calculation unit 305 calculates the synthesis ratio a based on the evaluation value Eval related to the captured image. The evaluation value Eval is a value calculated based on the brightness of the subject and the amount of noise in the image, and the evaluation value Eval increases as the luminance value of the subject increases and the amount of noise in the image increases. . The evaluation value Eval is calculated according to the following equation 10 based on the average luminance value Y_ave of the subject and the value N_value determined by the light receiving sensitivity of the image sensor 102 at the time of shooting.

Eval = Y_ave × N_value (10)

  With reference to FIG. 6, the calculation method of Y_ave in Expression 10 will be described. FIG. 6 is a flowchart of the calculation process of the luminance average value Y_ave of the subject executed by the specular reflection component calculation unit 305.

In step S601, the specular reflection component calculation unit 305 calculates the luminance signal Y based on the shine correction corrected color signals R ′, G ′, and B ′. The luminance signal Y can be expressed by the following equation 11, for example.

Y = 0.299R ′ + 0.587G ′ + 0.114B ′ (11)

  In step S <b> 602, the specular reflection component calculation unit 305 acquires a subject region that is irradiated with a virtual light source. Here, the face area detected by the face detection unit 106 is acquired as the subject area.

  In step S603, the specular reflection component calculation unit 305 calculates the average luminance value Y_ave of the subject based on a simple average of the luminance values in the subject area.

  With reference to FIG. 7, the calculation method of N_value of Expression 10 will be described. The specular reflection component calculation unit 305 calculates N_value based on the ISO sensitivity that is the light reception sensitivity of the image sensor 102 in accordance with the characteristics shown in FIG. N_value increases as the ISO sensitivity increases. Here, Th in FIG. 7 is an ISO sensitivity in which a noise component of a captured image is not so conspicuous. Th is the upper limit of the ISO sensitivity that can maintain the evaluation value of the noise component expressed by, for example, PSNR (Peak Signal-to-Noise Ratio) or the like at a predetermined value or higher.

  After calculating the evaluation value Eval, the specular reflection component calculation unit 305 calculates the composite ratio a based on the value of the evaluation value Eval according to the characteristics shown in FIG. The composite ratio a has a characteristic that approaches 1 as the evaluation value Eval increases. Accordingly, in the addition (addition) of the specular reflection component, the higher the luminance of the subject and the higher the ISO sensitivity at the time of image capturing, the greater the proportion of the specular reflection component reflecting the color of the virtual light source. The specular reflection component calculation unit 305 calculates the specular reflection components Rs, Gs, and Bs using the composite ratio a thus calculated.

  Next, another example of the method for calculating the composition ratio a will be described. The specular reflection component calculation unit 305 calculates a distance C_dist on a color space having three axes of R, G, and B for the virtual light source colors Rw, Gw, and Bw and the subject colors Rt, Gt, and Bt, and the distance C_dist Based on the above, the composition ratio a is calculated. Here, the subject colors Rt, Gt, and Bt are simple average values of the post-shine correction color signals R ′, G ′, and B ′ of the face area detected by the face detection unit 106.

The distance C_dist is calculated based on the Euclidean distance using the virtual light source colors Rw, Gw, and Bw and the subject colors Rt, Gt, and Bt as shown in Expression 12 below.

  The specular reflection component calculation unit 305 calculates the composite ratio a based on the value of the distance C_dist according to the characteristics shown in FIG. The composition ratio a has a characteristic of approaching 1 as the distance C_dist becomes smaller. Therefore, in the addition (addition) of the specular reflection component, the closer the subject color is to the color of the virtual light source, the greater the proportion of the specular reflection component reflecting the color of the virtual light source. The specular reflection component calculation unit 305 calculates the specular reflection components Rs, Gs, and Bs using the composite ratio a thus calculated.

  As described above, according to the first embodiment, the digital camera 100 sets a virtual light source having a predetermined color for an image, and the effect of light irradiation by the virtual light source is given to the image. Then, image processing (relighting processing) is performed. In the relighting process, the digital camera 100 calculates a specular reflection component of a virtual light source having a predetermined color and a specular reflection component corresponding to the case where the virtual light source has a subject color instead of the predetermined color. Then, the digital camera 100 adds these two types of specular reflection components at a predetermined ratio to the pixel to be processed. As a result, it is possible to add a specular reflection component adapted to the color of the subject.

[Second Embodiment]
In the second embodiment, a relighting process when a subject area cannot be detected (or is not detected) will be described. In this embodiment, the basic configuration of the digital camera 100 is the same as that of the first embodiment, but the relighting processing unit 203 in the image processing unit 104 is shown in FIG. 10 instead of the configuration shown in FIG. Has the configuration shown. 10, blocks having the same or similar functions as those in FIG. 3 are denoted by the same reference numerals as those in FIG. Hereinafter, differences from the first embodiment will be mainly described.

  If the subject area cannot be detected, the correction target color cannot be calculated in the shine correction amount calculation unit 302 in FIG. Therefore, in FIG. 10, the shine correction amount calculation unit 302 and the shine correction unit 303 are removed.

  In the first embodiment, the post-correction color signals R ′, G ′, and B ′ are input to the diffuse reflection component calculation unit 304, the specular reflection component calculation unit 305, and the virtual light source addition unit 306. In the present embodiment, color signals R, G, and B input from the white balance correction unit 202 instead of the shine correction color signals R ′, G ′, and B ′ are converted into the diffuse reflection component calculation unit 304, the specular reflection component, and the like. The data is input to the calculation unit 305 and the virtual light source addition unit 306.

  In the present embodiment, the specular reflection component calculation unit 305 calculates the composition ratio a from the parameters at the time of shooting. The control unit 108 calculates an appropriate exposure amount in the optical system 101 and an appropriate light emission amount in the strobe 112 immediately before taking an image (exposure detection process and light emission detection process). The user can adjust the exposure amount and the light emission amount actually applied to the optical system 101 and the strobe 112 at the time of photographing using the operation unit 110 in order to photograph the subject with a desired brightness.

Therefore, the appropriate exposure amount and appropriate light emission amount calculated by the control unit 108 are compared with the exposure amount and light emission amount set by the user, and an evaluation value Eval_c indicating the degree to which the user attempts to photograph the subject brightly is expressed as follows: This is expressed as in Equation 13.

Eval_c = EV_rate × F_rate × N_value (13)

A method for calculating EV_rate in Expression 13 will be described. EV_rate is expressed as in Expression 14 below using the exposure amount EV_actual set by the user at the time of shooting with the optical system 101 and the appropriate exposure amount EV_correct in the shooting scene calculated by the control unit 108 immediately before shooting.

EV_rate = EV_actual / EV_correct (14)

A method for calculating F_rate in Expression 13 will be described. F_rate is expressed by the following Expression 15 using the actual light emission amount F_actual of the strobe 112 set by the user at the time of shooting and the appropriate light emission amount F_correct in the shooting scene calculated by the control unit 108 immediately before shooting. Note that the value of F_rate is 1 when the strobe 112 does not emit light.

F_rate = F_actual / F_correct (15)

  The value of N_value in Expression 13 is calculated by the same method as the calculation method described with reference to Expression 10 in the first embodiment.

  After calculating the evaluation value Eval_c, the specular reflection component calculation unit 305 calculates the composite ratio a based on the value of the evaluation value Eval_c according to the characteristics shown in FIG. The composition ratio a has a characteristic that approaches 1 as the evaluation value Eval_c increases. Therefore, in the addition (addition) of the specular reflection component, the larger the ratio of the exposure amount at the time of imaging to the appropriate exposure amount, and the greater the ratio of the emission amount at the time of imaging to the appropriate light emission amount, the more the color of the virtual light source is reflected. The ratio of the specular reflection component is increased. In addition, the higher the ISO sensitivity at the time of capturing an image, the greater the proportion of the specular reflection component that reflects the color of the virtual light source. Further, when the exposure conditions and the strobe light emission amount at the time of photographing are appropriate and the ISO sensitivity is lower than Th in FIG. 8, the evaluation value Eval_c is 1.0 and the composition ratio a is 0.5. The specular reflection component calculation unit 305 calculates the specular reflection components Rs, Gs, and Bs using the composite ratio a thus calculated.

  As described above, according to the second embodiment, the digital camera 100 uses the specular reflection component of the virtual light source having a predetermined color and the pixel to be processed instead of the predetermined color in the relighting process. And a specular reflection component corresponding to the case of having a color of. Then, the digital camera 100 adds these two types of specular reflection components at a predetermined ratio to the pixel to be processed. Thereby, it is possible to add a specular reflection component that conforms to the color of the pixel to be processed.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  DESCRIPTION OF SYMBOLS 100 ... Digital camera, 104 ... Image processing part, 203 ... Relighting processing part, 301 ... Normal calculation part, 304 ... Diffuse reflection component calculation part, 305 ... Specular reflection component calculation part, 306 ... Virtual light source addition part

Claims (13)

An image processing apparatus that sets a virtual light source having a predetermined color for an image and performs image processing so that the effect of light irradiation by the virtual light source is imparted to the image,
First acquisition means for acquiring a first specular reflection component of light from the virtual light source having the predetermined color for a pixel to be processed in the image;
A second specular reflection component of light from the virtual light source is acquired for the pixel to be processed, corresponding to the case where the virtual light source has the color of the pixel to be processed instead of the predetermined color. Acquisition means of
Adding means for adding the first specular reflection component and the second specular reflection component at a predetermined ratio to the pixel to be processed;
An image processing apparatus comprising: The image is an image captured by an imaging unit,
The pixel to be processed is a pixel included in a predetermined subject,
The image processing apparatus includes:
Detecting means for detecting the predetermined subject from the image;
Determining means for determining the predetermined ratio based on at least one of the luminance of the predetermined subject, the ISO sensitivity at the time of capturing the image, and the color of the subject;
The image processing apparatus according to claim 1, further comprising: The image processing apparatus according to claim 2, wherein the determination unit determines the predetermined ratio such that the higher the luminance of the subject, the larger the ratio of the first specular reflection component. The said determination means determines the said predetermined ratio so that the ratio of a said 1st specular reflection component may become large, so that the ISO sensitivity at the time of the said imaging of an image is high. Image processing apparatus. 5. The determination unit according to claim 2, wherein the determination unit determines the predetermined ratio such that the ratio of the first specular reflection component increases as the color of the subject is closer to the predetermined color. The image processing apparatus according to any one of the above. In a case where the pixel to be processed includes a color component having a value equal to or greater than a threshold value, the acquisition of the second specular reflection component by the second acquisition unit is performed based on the color of the subject. The image processing apparatus according to claim 2, further comprising a correction unit that corrects a color component having a value equal to or greater than the threshold value of the pixel. The image is an image captured by an imaging unit,
The image processing apparatus determines the predetermined ratio based on at least one of ISO sensitivity at the time of capturing the image, an exposure amount at the time of capturing the image, and a light emission amount of a strobe at the time of capturing the image. The image processing apparatus according to claim 1, further comprising a determination unit that performs the determination. 8. The image according to claim 7, wherein the determination unit determines the predetermined ratio such that the higher the ISO sensitivity at the time of capturing the image, the higher the ratio of the first specular reflection component. Processing equipment. Further comprising an exposure detection means for detecting an appropriate exposure amount at the time of capturing the image;
The said determination means determines the said predetermined ratio so that the ratio of the said 1st specular reflection component may become large, so that the ratio of the said exposure amount with respect to the said appropriate exposure amount is large. An image processing apparatus according to 1. Further comprising a light emission detecting means for detecting an appropriate light emission amount of the strobe when the image is captured;
The determination unit determines the predetermined ratio such that the ratio of the first specular reflection component increases as the ratio of the light emission amount to the appropriate light emission amount increases. The image processing apparatus according to any one of the above. The image processing apparatus according to any one of claims 1 to 10,
Imaging means for capturing the image;
An imaging apparatus comprising: An image processing method executed by an image processing apparatus for performing image processing so that a virtual light source having a predetermined color is set for an image and an effect of light irradiation by the virtual light source is imparted to the image,
A first acquisition step of acquiring a first specular reflection component of light from a virtual light source having the predetermined color for a pixel to be processed in the image;
A second specular reflection component of light from the virtual light source is acquired for the pixel to be processed, corresponding to the case where the virtual light source has the color of the pixel to be processed instead of the predetermined color. Acquisition process,
An adding step of adding the first specular reflection component and the second specular reflection component at a predetermined ratio to the pixel to be processed;
An image processing method comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 10.

Images