JP7311995B2 - IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD, PROGRAM, STORAGE MEDIUM - Google Patents

Description

The present invention relates to an image processing apparatus that reflects the influence of a virtual light source on an input image.

2. Description of the Related Art Conventionally, there has been known a technique of performing relighting by irradiating light from a virtual light source to a subject in an image after photographing. This allows the user to add arbitrary shadows and virtual light source colors to the subject in the image.

For example, in Patent Document 1, shadow information of a subject is analyzed, and the direction of a virtual light source for irradiation is determined so as to satisfy a predetermined target condition. This makes it possible to automatically obtain an image of a subject having shadows that match predetermined conditions without the user having to directly specify the direction of the virtual light source.

JP 2016-72692 A

However, in the relighting method described in Patent Document 1, only a virtual light source with a predetermined light source color can be applied. For example, if a subject in a photographed image is illuminated by multiple light sources of different colors, the subject will be in a state where multiple colors are mixed, and the subject will be illuminated by a light source of a specific color. The bright light may give a bad impression of the subject. As in such an example, the technique described in Japanese Patent Application Laid-Open No. 2002-200013 cannot handle the case where the color tone differs for each area of the subject.

SUMMARY OF THE INVENTION The present invention has been made in view of the problems described above, and its object is to provide an image processing apparatus capable of correcting a subject photographed under a plurality of environmental light sources having different colors so as to have a favorable impression. is.

An image processing apparatus according to the present invention comprises selection means for selecting a subject from an input image, color information acquisition means for acquiring color information on the selected subject, and shape information acquisition for acquiring three-dimensional shape information on the selected subject. target color acquisition means for acquiring target color information for the selected subject; correction means for performing correction by adding or subtracting light from a virtual light source to the selected subject; and setting means for setting at least the color and orientation of the virtual light source using the three-dimensional shape information so that the corrected color information of the subject approaches the target color information when correction is performed. , the setting means further sets the intensity of the virtual light source based on the target color information , and the three-dimensional shape information is normal information that is information of a normal direction to the surface of the subject; The setting means acquires the normal line information and the color information for each pixel in the subject, and based on the information obtained by weighting the normal line information of each pixel according to the color information of each pixel, determines the It is characterized by setting the direction of the virtual light source .

According to the present invention, it is possible to correct a subject photographed under, for example, a plurality of environmental light sources with different colors so that the subject has a favorable impression.

1 is a block diagram showing the configuration of a digital camera that is an embodiment of the image processing apparatus of the present invention; FIG. FIG. 2 is a block diagram showing the configuration of the image processing section of the digital camera of one embodiment; FIG. 3 is a block diagram showing the configuration of a relighting processing unit according to one embodiment; FIG. 4 is a diagram showing calculation of reflection components by relighting in one embodiment; 4 is a flow chart showing processing of a virtual light source information calculation unit in one embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

FIG. 1 is a block diagram showing the configuration of a digital camera 100 which is an embodiment of the image processing apparatus of the invention.

The digital camera 100 in this embodiment includes an optical system 101, an image sensor 102, an A/D conversion unit 103, an image processing unit 104, a distance measurement unit 105, a face detection unit 106, a recording unit 107, a control unit 108, a memory 109, An operation unit 110 and a display unit 111 are provided. The optical system 101 includes a focus lens, diaphragm, and shutter. The optical system 101 drives a focus lens to focus an object and adjusts the amount of exposure by controlling an aperture and a shutter during photographing. The imaging device 102 is composed of a CCD, a CMOS sensor, or the like that converts an object image formed by the optical system 101 into an analog electric signal by photoelectric conversion.

The A/D converter 103 converts the input analog electric signal into a digital signal. The image signal converted into a digital signal is subjected to synchronization processing, white balance correction processing, focus degree calculation processing, relighting processing, gamma processing, etc. in the image processing unit 104 and output to the recording unit 107 . A distance measuring unit 105 acquires distance information to a subject at the time of shooting and generates a distance map. A distance map is two-dimensional array information indicating distance information to a subject in units of pixels of a photographed image.

A face detection unit 106 detects a person's face area in a captured image in order to determine a subject area for relighting processing performed by the image processing unit 104 . At this time, the coordinates of the facial features such as the right eye, the left eye, and the mouth are detected, and the distance between the facial features, such as the distance between the right eye and the left eye, is calculated. Based on the distance between each facial organ, an elliptical area centered on the barycentric coordinates of each organ is set so as not to include the background area, and is detected as a facial area. A recording unit 107 converts the image signal output from the image processing unit 104 into a format such as JPEG and records the converted signal.

The control unit 108 controls the operation of the entire imaging apparatus 100 of this embodiment. For example, the target exposure amount is calculated from the brightness of the subject immediately before shooting. In addition, a predetermined evaluation value is calculated based on the captured image, and parameters for image processing performed by the image processing unit 104 are determined. A memory 109 stores information used by the image processing unit 104 as well as settings and illumination conditions at the time of shooting, and outputs the information to the image processing unit 104 as necessary. The operation unit 110 has a plurality of members such as switches with which the user gives operation instructions to the imaging device. A display unit 111 is, for example, a liquid crystal display or the like installed on the rear surface of the camera, and displays a screen for assisting operations during shooting, images stored in the recording unit 107, and the like.

FIG. 2 is a block diagram showing the configuration of the image processing unit 104 in this embodiment. The image processing unit 104 includes a synchronization processing unit 201 , a white balance correction processing unit 202 , a relighting processing unit 203 and a gamma processing unit 204 .

A processing flow of the image processing unit 104 will be described. A video signal in the Bayer array converted into a digital signal by the A/D conversion unit 103 is input to the image processing unit 104 . The image signal input to the image processing unit 104 is input to the synchronization processing unit 201, and the input Bayer array RGB signals are subjected to synchronization processing to generate color signals R, G, and B. FIG. The generated R, G, and B color signals are input to the white balance correction processing unit 202, gains are applied to the RGB color signals based on the white balance gain values calculated by the control unit 107, and the white balance is adjusted. be.

The color signals RGB whose white balance has been adjusted are output to the relighting processing unit 203 . The relighting processing unit 203 receives color signals RGB from the white balance correction processing unit 202 , distance map D_MAP from the distance measurement unit 105 , and information indicating the coordinates and size of the face in the image from the face detection unit 106 . Then, an image is generated by illuminating the color signals R, G, and B with a virtual light source, and the processed color signals R_out, G_out, and B_out are output to the gamma processing unit 204 . The gamma processing unit 204 receives the signals R_out, G_out, and B_out after the relighting process, performs gamma processing on the input signals, and outputs color signals Rg, Gg, and Bg after the gamma processing to the recording unit 107 .

Next, FIG. 3 is a block diagram showing the configuration of the relighting processing unit 203. As shown in FIG.

The relighting processing unit 203 includes a normal acquisition unit 301 , a virtual light source information calculation unit 302 , a virtual light source diffuse reflection component calculation unit 303 , and a virtual light source addition processing unit 304 . The relighting processing unit 203 receives distance information from the distance measurement unit 105 , color signals R, G, and B from the white balance correction processing unit 202 , and face area information from the face detection unit 106 .

The normal acquisition unit 301 acquires the normal direction of the surface of the subject at the face coordinate position input from the face detection unit 106 . In the present embodiment, the normal obtaining unit 301 refers to the distance information measured by the distance measuring unit 105 and calculates the normal information of the subject area (acquisition of three-dimensional shape information). The normal information N obtained in this manner is output to the virtual light source information calculation unit 302 and the virtual light source diffuse reflection component calculation unit 303 .

The virtual light source information calculation unit 302 receives the color signals R, G, and B (color information acquisition) from the white balance processing unit 202 and the normal line N information from the normal line acquisition unit 301 . The virtual light source information calculation unit 302 calculates a three-dimensional direction vector L indicating the orientation of the virtual light source, α indicating the irradiation intensity of the virtual light source, light source color components Rw, Gw, and Bw by a method described later, and calculates the virtual light source diffuse reflection component Output to the calculation unit 303 .

The virtual light source diffuse reflection component calculator 303 calculates diffuse reflection components Rd, Gd, and Bd. A method of obtaining the diffuse reflection component will be described with reference to FIG.

FIG. 4 is a diagram showing the positional relationship among the digital camera 100, the subject 401, and the virtual light source 402 at the time of photographing, and the reflection characteristics of the virtual light source.

The diffuse reflection component at the horizontal pixel position H1 (the vertical pixel position is omitted for the sake of clarity) of the captured image captured by the camera 100 is the inner product of the normal N1 at the camera coordinates H1 and the direction vector L1 of the virtual light source. It becomes a value proportional to the square of the distance K1 between the virtual light source and the object position. The diffuse reflection component intensity Pd by the virtual light source can be expressed by (Equation 1), and the diffuse reflection components Rd, Gd, and Bd for each color can be expressed by (Equation 2).

Rd = Pd x Rw x R
Gd = Pd x Gw x G
Bd=Pd×Bw×B (Formula 2)
Here, N in (Equation 1) is the three-dimensional normal vector of the subject, and K is the distance between the virtual light source and the subject. kd is the diffuse reflectance of the object. α and L are the intensity of the virtual light source input from the virtual light source information calculation unit 302 and the three-dimensional directional vector of the virtual light source, respectively. Also, R, G, and B in (Equation 2) are the color signals R, G, and B input from the white balance correction processing unit 202 . Rw, Gw, and Bw are parameters indicating the color of the virtual light source. A method of calculating these parameters will be described later.

A virtual light source addition processing unit 304 performs processing for adding the diffuse reflection components Rd, Gd, and Bd calculated by the virtual light source diffuse reflection component calculation unit 303 to the input color signals R, G, and B. FIG. Color signals R_out, G_out, and B_out after irradiation with a virtual light source can be expressed as in (Equation 3).

R_out = R + Rd
G_out=G+Gd
B_out=B+Bd (Formula 3)
The post-irradiation color signals R_out, G_out, and B_out calculated in this manner are output to the gamma processing unit 205 .

Next, FIG. 5 is a flowchart showing processing of the virtual light source information calculation unit 302. As shown in FIG.

In step S<b>501 , the virtual light source information calculation unit 302 receives face detection information from the face detection unit 106 , color signals R, G, and B of the input image from the white balance correction processing unit 202 , and the face area normal from the normal acquisition unit 301 . Get information.

In S502, correction target colors Rt, Gt, and Bt, which are target skin colors in the face area acquired from the face detection unit 106, are calculated (target color acquisition). The correction target colors Rt, Gt, and Bt are, for example, the average values of the color signals R, G, and B in the face area. Statistically processed values such as values and modes can be used.

In S503, the correction target colors Rs, Gs, and Bs in the face area obtained from the face detection unit 106 are obtained. Here, the correction target colors Rs, Gs, and Bs indicate the balance of the skin color under the environmental light source at the time of photographing, and satisfy the relationship Rs+Gs+Bs=1. When the light source colors to be corrected are known, the correction target colors Rs, Gs, and Bs are obtained by referring to the known light source color values stored in advance in the memory 109 . The light source color to be corrected can be known by referring to the lighting conditions at the time of shooting stored in the memory 109 or by receiving a color input from the user through the operation unit 110 .

If the light source to be corrected is unknown, among the color signals R, G, and B of each pixel in the face region acquired from the face detection unit 106, the color signals R, G, and B that are most different from the correction target colors Rt, Gt, and Bt. Based on G and B, it is determined as in (Equation 4). Re, Ge, and Be in (Equation 4) are the values of the color signals R, G, and B that are most different from the correction target colors Rt, Gt, and Bt among the color signals R, G, and B in the face area. A method for comparing the color signals is a known method. should be set to Re, Ge, and Be.

Rs=Re/(Re+Ge+Be)
Gs=Ge/(Re+Ge+Be)
Bs=Be/(Re+Ge+Be) (Formula 4)
In S504, the correction target color weight Cw of each pixel is acquired. The correction target color weight Cw is a value that indicates how much the color signals R, G, and B of each pixel have the components of the correction target colors Rs, Gs, and Bs, and is expressed as in (Equation 5). (Equation 5) indicates that the inner product value of the color signals R, G, B of the target pixel and the correction target colors Rs, Gs, Bs is used as the correction target color weight Cw.

Cw=R×Rs+G×Gs+B×Bs (Formula 5)
In S505, the irradiation direction vector L of the virtual light source is calculated. The irradiation vector L of the virtual light source is expressed as in (Equation 6). (Equation 6) is a directional vector obtained by averaging the normal vectors N1 to Nm of pixels 1 to m in the face region obtained from the face detection unit 106 using the correction target color weights Cw1 to Cwm of each pixel. is the irradiation vector L of the virtual light source.

In S506, the intensity α of the virtual light source is calculated. The intensity α of the virtual light source is represented by (Equation 7). (Formula 7) is a formula for the purpose of calculating the correction intensity α that brings the subject closer to the correction target colors Rt, Gt, and Bt by irradiating the virtual light source. The intensity α of the virtual light source can have both positive and negative values. Adds a light subtraction effect.

α = -1 × Ct / (Cmax - Ct) ... (Equation 7)
In (Equation 7), Ct is a value indicating how much the correction target colors Rt, Gt, and Bt have the components of the correction target colors Rs, Gs, and Bs. and the inner product value of the correction target colors Rs, Gs, and Bs. Cmax is the value of Cw of a pixel that has many components of the correction target colors Rs, Gs, and Bs in the face area obtained from the face detection unit 106 and is most different from the correction target colors Rt, Gt, and Bt. In this embodiment, the value of Cmax is the value of Cw that maximizes the difference absolute value between the inner product values Cw and Ct of the color signals R, G, B of each pixel and the correction target colors Rs, Gs, Bs. Accordingly, when the value of Cmax is smaller than Ct, α takes a positive value, and the irradiation effect by the virtual light source can be added so as to approach the correction target value. Also, when the value of Cmax is greater than Ct, α takes a negative value, and the illumination effect of the virtual light source can be subtracted so as to approach the correction target.

In S507, it is determined whether or not α<0. If α<0, proceed to S508; otherwise, proceed to S509.

In S508, virtual light source colors Rw, Gw, and Bw are calculated when α<0. The virtual light source colors Rw, Gw, and Bw indicate the color balance of the virtual light sources, and have the characteristic of Rw+Gw+Bw=1.

When α<0, the virtual light source colors Rw, Gw, and Bw are corrected to the correction target colors Rs, Gs, and Bs as shown in (Equation 8) because the correction target colors are approximated by subtracting the correction color components. can be equivalent to

Rw = Rs
Gw = Gs
Bw=Bs (Formula 8)
In S509, virtual light source colors Rw, Gw, and Bw are calculated when α≧0. When α≧0, since the correction target color is approximated by adding the irradiation effect in the virtual light source, the virtual light source colors Rw, Gw, and Bw are symmetrical with the correction target colors Rs, Gs, and Bs as shown in (Equation 9). It has a color balance of MAX(Rs, Gs, Bs) in (Expression 9) is a function that selects the maximum value among the correction target colors Rs, Gs, and Bs.

Rw'=max(Rs, Gs, Bs)-Rs
Gw'=max(Rs, Gs, Bs)-Gs
Bw'=max(Rs, Gs, Bs)-Bs
Rw=Rw'/(Rw'+Gw'+Bw')
Gw=Gw'/(Rw'+Gw'+Bw')
Bw=Bw'/(Rw'+Gw'+Bw') ... (Formula 9)
In S 510 , the direction vector L of the virtual light source, the intensity α of the virtual light source, and the colors Rw, Gw, and Bw of the virtual light source calculated as described above are output to the virtual light source diffuse reflection component calculator 303 .

In this way, it is possible to set a virtual light source that corrects the bad impression caused by the color of the light source that illuminates the subject at the time of photographing so as to obtain a favorable impression.

In this embodiment, an example of directly acquiring normal line information from the memory 109 has been described. do not have. For example, it may be three-dimensional wireframe information indicating the three-dimensional shape of the subject. In this case, the normal line information can be calculated from the angle information of each point of the wire frame.

(Other embodiments)
In addition, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads the program. It can also be realized by executing processing. It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

100: Digital camera, 101: Optical system, 102: Imaging element, 104: Image processing unit, 105: Distance measurement unit, 106: Face detection unit, 108: Control unit

Claims (14)

a selection means for selecting a subject from an input image;
a color information obtaining means for obtaining color information of the selected subject;
shape information acquiring means for acquiring three-dimensional shape information of the selected subject;
target color acquisition means for acquiring target color information about the selected subject;
correction means for correcting the selected subject by adding or subtracting light from a virtual light source;
setting means for setting at least the color and direction of the virtual light source using the three-dimensional shape information so that the corrected color information of the subject approaches the target color information when the correction is performed by the correction means; and
The setting means further sets the intensity of the virtual light source based on the target color information ,
the three-dimensional shape information is normal information that is information about a normal direction to the surface of the subject;
The setting means acquires the normal line information and the color information for each pixel in the subject, and based on the information obtained by weighting the normal line information of each pixel according to the color information of each pixel, determines the An image processing device that sets the direction of a virtual light source . 2. The image processing apparatus according to claim 1, wherein the subject is a human face, and the target color information is target skin color information. 3. The image processing apparatus according to claim 2, wherein said color information acquiring means acquires color information of a skin color of said subject. 4. The image processing apparatus according to claim 2, wherein said target color information is an average value of color signals of said person's face. 4. The image processing apparatus according to claim 2, wherein said target color information is a median value of color signals in said person's face. 4. The image processing apparatus according to claim 2, wherein the target color information is a mode of color signals of the person's face. 2. The setting device according to claim 1 , wherein the setting means sets a direction obtained by averaging normal directions weighted according to the color information of each pixel as the direction of the virtual light source. Image processing device. 8. The image processing apparatus according to claim 1 , wherein said setting means selects a color most different from said target color information in said object as a correction target color to be corrected. 9. The image processing apparatus according to claim 8 , wherein the weight is determined based on how much the color component of each pixel contains the correction target color. 10. The image processing apparatus according to claim 8 , wherein the setting unit sets the correction target color based on lighting conditions when the input image was captured. 10. The image processing apparatus according to claim 8 , wherein said setting means sets a color specified by a user as said color to be corrected. a selection step of selecting a subject from the input image;
a color information acquisition step of acquiring color information of the selected subject;
shape information acquiring means for acquiring three-dimensional shape information of the selected subject;
a target color acquisition step of acquiring target color information about the selected subject;
a correction step of correcting the selected subject by adding or subtracting light from a virtual light source;
A setting step of setting at least the color and orientation of the virtual light source using the three-dimensional shape information so that the corrected color information of the subject approaches the target color information when correction is performed in the correcting step. and
In the setting step, further setting the intensity of the virtual light source based on the target color information ,
the three-dimensional shape information is normal information that is information about a normal direction to the surface of the subject;
In the setting step, the normal information and the color information are acquired for each pixel of the subject, and based on the information obtained by weighting the normal information of each pixel according to the color information of each pixel, the An image processing method characterized by setting the direction of a virtual light source . A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 11 . A computer-readable storage medium storing a program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 11 .

Images