JP7356255B2 - Image processing device and its processing method, imaging device, program, storage medium - Google Patents

Description

The present invention relates to an image processing device that reflects the influence of a virtual light source on an input image.

2. Description of the Related Art Conventionally, a technique is known in which a subject in a photographed image is irradiated with light from a virtual light source to perform relighting processing. The relighting process brightens dark areas such as shadows caused by ambient light, making it possible to obtain a preferable image.

For example, Patent Document 1 discloses relighting processing that can appropriately correct the state of shadows of a subject. Specifically, the state of the shadow in a predetermined area of the photographed image is detected, and the characteristics of the virtual light source are determined based on the state of the detected shadow. Then, the photographed image is corrected so that it will be in a shadow state when irradiated with light by a virtual light source having the determined characteristics.

Japanese Patent Application Publication No. 2016-72694

In the method described in Patent Document 1 mentioned above, in a scene in which multiple people are photographed, the shadow state of each detected subject is detected, and the characteristics of a virtual light source suitable for each subject are determined. It turns out.

However, in a scene where multiple people are photographed, some of the people may not be detected as subjects due to reasons such as the tilt angle of their faces being large or their faces being too dark. In such a case, relighting processing is performed on the detected subject, but no processing is performed on the undetected subject. As a result, in a backlit scene with a dark face, the face of a certain person becomes brighter through relighting processing, but the face of another person remains dark, resulting in an unnatural image.

The present invention has been made in view of the above-mentioned problems, and its purpose is to suppress unnatural relighting processing results when performing relighting processing on a scene in which multiple subjects are photographed.

The image processing device according to the present invention sets an acquisition unit that acquires a captured image, a plurality of detection units that detects a subject included in the captured image, and a virtual light source that is a virtual light source that illuminates the subject. The setting means includes a setting means, and a correction means for correcting the brightness of the photographed image using a virtual light source set by the setting means, and the setting means is configured to adjust the brightness of the photographed image based on the detection results for one subject by the plurality of detection means. Based on this, the reliability of object detection is calculated separately for each of the detection results of the plurality of detection means for the one object, and the separately calculated reliability is averaged to determine the object whose brightness should be corrected. The present invention is characterized in that a relighting reliability indicating a reliability is calculated, and a parameter of the virtual light source is set based on the calculated relighting reliability.

According to the present invention, when performing relighting processing on a scene in which multiple subjects are photographed, it is possible to suppress unnatural relighting processing results.

1 is a block diagram showing the configuration of a digital camera that is an embodiment of an image processing device of the present invention. FIG. 2 is a block diagram showing the configuration of an image processing section in one embodiment. 5 is a flowchart showing processing of an image processing unit in one embodiment. FIG. 2 is a block diagram showing the configuration of a rewriting processing unit in an embodiment. FIG. 3 is a schematic diagram illustrating reflection of virtual light from a virtual light source in one embodiment. FIG. 3 is an explanatory diagram of parameter settings of a virtual light source in one embodiment. FIG. 3 is a diagram illustrating an example of images before and after rewriting processing in an embodiment. 5 is a flowchart showing rewriting processing in one embodiment. 5 is a flowchart showing rewriting processing in one embodiment. FIG. 3 is a diagram illustrating rewriting processing for multiple people in one embodiment. FIG. 2 is an explanatory diagram of rewriting reliability calculation in one embodiment. FIG. 2 is an explanatory diagram of rewriting reliability calculation in one embodiment. FIG. 2 is an explanatory diagram of rewriting reliability calculation in one embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

FIG. 1 is a block diagram showing the configuration of a digital camera 100, which is an embodiment of the image processing apparatus of the present invention.

In the digital camera 100 shown in FIG. 1, light that enters through a lens group 101 (imaging optical system) including a zoom lens and a focus lens, and a shutter 102 having an aperture function is photoelectrically converted in an imaging unit 103. The imaging unit 103 is composed of a CCD, a CMOS element, etc., and an electrical signal obtained by photoelectric conversion is output to an A/D converter 104 as an image signal. The A/D converter 104 converts the analog image signal output from the imaging section 103 into a digital image signal (image data) and outputs it to the image processing section 105.

The image processing unit 105 performs color conversion processing such as white balance processing, and γ processing on the image data from the A/D converter 104 or the image data read from the image memory 106 via the memory control unit 107. , performs various image processing such as contour enhancement processing and color correction processing. Image data output from the image processing section 105 is written into the image memory 106 via the memory control section 107. The image memory 106 stores image data output from the image processing section 105 and image data to be displayed on the display section 109.

The face/facial organ detection unit 113 detects a face area and a facial organ area where a person's face and organs are present from the captured image. The head detection unit 115 detects a head region in which a person's head exists from a captured image using a method based on pattern recognition or machine learning.

The image processing unit 105 performs a predetermined evaluation value calculation process using the face detection result and facial organ detection result of the face/facial organ detection unit 113, the head detection result of the head detection unit 115, and the captured image data. . Then, the system control unit 50 performs exposure control and focus detection control based on the obtained evaluation value. As a result, TTL (through-the-lens) type AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, etc. are performed.

Further, the D/A converter 108 converts the digital image data for display stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109. The display unit 109 displays on a display such as an LCD according to the analog signal from the D/A converter 108.

The codec unit 110 compresses and encodes the image data stored in the image memory 106 based on standards such as JPEG and MPEG. The system control unit 50 stores the encoded image data in a recording medium 112 such as a memory card or a hard disk via an interface (I/F) 111. Further, image data read from the recording medium 112 via the I/F 111 is decoded and expanded by the codec unit 110, and stored in the image memory 106. Then, by displaying the image data stored in the image memory 106 on the display unit 109 via the memory control unit 107 and the D/A converter 108, the image can be reproduced and displayed.

The relighting processing unit 114 performs a relighting process (relighting process) that corrects brightness by applying a virtual light source to a captured image. Note that details of the rewriting processing performed by the rewriting processing unit 114 will be described later.

The system control unit 50 controls the entire system of the digital camera 100. The nonvolatile memory 121 is constituted by a memory such as an EEPROM, and stores programs, parameters, etc. necessary for processing by the system control unit 50. The system control unit 50 expands programs recorded in the non-volatile memory 121 and constants and variables for operation of the system control unit 50 into the system memory 122 and executes them, thereby executing each process of the present embodiment described later. Realize.

The operation unit 120 accepts operations such as menu settings and image selection by the user. The distance sensor 123 measures the distance to the subject and outputs distance information corresponding to each pixel of the photographed pixels (distance information detection).

Next, details of the image processing unit 105 will be explained using FIGS. 2 and 3. FIG. 2 is a block diagram showing the configuration of the image processing unit 105, and FIG. 3 is a flowchart showing the processing in the image processing unit 105. In this embodiment, it is assumed that the imaging unit 103 is covered with a Bayer array color filter. Therefore, each pixel of the imaging unit 103 outputs an R, G, or B pixel signal.

First, in S301, Bayer RGB image data input from the A/D converter 104 in FIG. 1 is input to the synchronization processing unit 200. The synchronization processing unit 200 performs synchronization processing on the input R, G, and B image signals, and generates color signals RGB for each pixel.

Next, in S302, the WB amplification unit 201 adjusts the white balance by applying a gain to the generated color signal RGB of each pixel based on the white balance gain value calculated by the system control unit 50 using known processing. The color signal RGB whose white balance has been adjusted by the WB amplification unit 201 is input to the luminance/color signal generation unit 202 in S303. The brightness/color signal generation unit 202 generates a brightness signal Y from the color signals RGB, and outputs the generated brightness signal Y to the edge enhancement processing unit 203 and the color signal RGB to the color conversion processing unit 205.

In S<b>304 , the edge enhancement processing unit 203 performs edge enhancement processing on the luminance signal Y and outputs it to the luminance gamma processing unit 204 . On the other hand, the color conversion processing unit 205 performs, for example, matrix calculation on the color signals RGB to convert the signal into a desired color balance, and outputs the result to the color gamma processing unit 206 and the subject information detection unit 208 .

In S305, the subject information detection unit 208 detects the face/facial organ detection information output from the face/facial organ detection unit 113, the head detection information output from the head detection unit 115, and the color conversion processing unit 205. Information regarding the subject in the photographed image is detected from the output color signals RGB (subject information detection). Here, the information regarding the subjects includes the number of subjects in the photographed image, the size of the subjects, the positional relationship between the subjects, the way the subjects are hit by light, the shadow information of the subjects, and the like. For example, the number of subjects, their sizes, and positional relationships are detected from the coordinate position information of each face/facial organ output by the face/facial organ detection unit 113 or the coordinate position information of the head output by the head detection unit 115. do. In addition, the way the light hits and the shadow information are detected from the average brightness information and brightness histogram information of the entire photographed image and each subject.

In S306, the luminance gamma processing unit 204 performs gamma correction on the luminance signal Y, and outputs the gamma-corrected luminance signal Y to the image memory 106 via the memory control unit 107. On the other hand, the color gamma processing unit 206 performs gamma correction on the color signals RGB and outputs them to the color difference signal generation unit 207.

In S307, the color difference signal generation unit 207 generates color difference signals RY and BY signals from the RGB signals, and outputs them to the image memory 106 via the memory control unit 107.

Next, rewriting processing in this embodiment will be explained.

FIG. 4 is a block diagram showing the configuration of the rewriting processing section 114. The relighting processing unit 114 reads out the luminance signal Y and color difference signals BY and RY processed by the image processing unit 105 and recorded in the image memory 106 as input, and performs relighting processing using a virtual light source.

First, the RGB signal converter 401 converts the input luminance signal Y and color difference signals BY, RY into RGB signals and outputs them to the degamma processor 402. The degamma processing unit 402 performs calculation (degamma processing) of a gamma characteristic inverse to the gamma characteristics of the gamma correction in the luminance gamma processing unit 204 and color gamma processing unit 206 of the image processing unit 105, and converts it into linear data. Then, the degamma processing unit 402 outputs the RGB signals (Rt, Gt, Bt) converted into linear data to the virtual light source reflection component calculation unit 406 and the virtual light source addition processing unit 407.

On the other hand, the distance calculation unit 403 creates a distance map from the object distance information acquired from the distance measurement sensor 123. The object distance information is two-dimensional distance information obtained for each pixel of a photographed image. The normal calculation unit 404 creates a normal map from the distance map created by the distance calculation unit 403 as shape information representing the shape of the subject. As for the method of generating a normal map from a distance map, a known technique will be used, and a specific processing example will be explained using FIG. 5.

FIG. 5 is a diagram showing the relationship between camera shooting coordinates and a subject. For example, for a certain subject 501 as shown in FIG. Gradient information in a part of the subject is calculated from the difference ΔDV in the distance D with respect to the difference ΔV in the direction (direction). Then, it is possible to calculate the normal N from the obtained gradient information for the part of the subject. By performing the above processing on each photographed pixel, it is possible to calculate the normal N corresponding to each pixel of the photographed image. The normal calculation unit 404 outputs information on the normal N corresponding to each pixel of the photographed image to the virtual light source reflection component calculation unit 406 as a normal map.

Note that although the distance calculation unit 403 and the normal line calculation unit 404 have been described as being configured in the relighting processing unit 114, the present invention is not limited to this. It may be configured within 105 or may be configured independently.

The virtual light source setting unit 405 sets parameters of the virtual light source based on subject information input from the subject information detecting unit 208 of the image processing unit 105. For example, if you want to brighten the entire face of a subject whose entire face is dark, you can adjust parameters such as the position, irradiation range, and intensity of the virtual light source so that the entire face is included in the irradiation range of the virtual light source. control.

Here, the parameters to be set for the virtual light source will be explained using FIG. 6, taking as an example the case where there is only one subject. FIG. 6(a) is a perspective view showing the positional relationship between the subject and the virtual light source, and FIG. 6(b) is a plan view showing the positional relationship between the subject and the virtual light source.

Regarding the position of the virtual light source, if you set the distance from the virtual light source to the subject short, the light from the virtual light source will hit the subject more strongly, and conversely, if you set the distance from the virtual light source to the subject longer, the light from the virtual light source will hit the subject more strongly. It starts to hit weakly. Regarding the irradiation range of the virtual light source, if the irradiation range of the virtual light source is set wide, the entire subject can be illuminated, and conversely, if the irradiation range is set narrow, only a part of the subject can be illuminated. Regarding the intensity of the virtual light source, if the intensity of the virtual light source is set high, the subject will be illuminated strongly, and conversely, if the intensity is set low, the subject will be illuminated weakly.

The virtual light source reflection component calculation unit 406 calculates the amount of light emitted virtually from the set virtual light source based on the distance K between the light source and the subject, the normal information N, and the parameters of the virtual light source set by the virtual light source setting unit 405. , calculate the component reflected by the object. Hereinafter, light radiated virtually from a virtual light source will be referred to as "virtual light." Specifically, the coordinate position of the captured image is inversely proportional to the square of the distance K between the virtual light source and the part of the subject corresponding to each pixel, and proportional to the inner product of the vector of the normal N and the vector of the light source direction L. The reflected component of the virtual light in the part of the subject corresponding to is calculated.

Here, a general method for calculating the reflected component of virtual light will be explained using FIG. 5. Note that in FIG. 5, only the horizontal direction of the photographed image is shown to make the explanation easier to understand, but as described above, the direction perpendicular to the plane of the paper is the vertical direction of the photographed image. In the following description, a method for calculating the reflection component of virtual light at a point P1 on the subject 501, which corresponds to a horizontal pixel position H1 in a captured image and a vertical pixel position V1 (not shown), will be described.

In FIG. 5, a virtual light source 502 is a virtual light source set for the subject 501. The reflection component of the virtual light at the position (H1, V1) of the photographed image photographed by the camera 100 is proportional to the inner product of the normal vector N1 at the point P1 on the subject 501 and the light source direction vector L1 of the virtual light source 502, The value is inversely proportional to the square of the distance K1 between the virtual light source 502 and the point P1. Note that the normal vector N1 and the light source direction vector L1 are three-dimensional vectors consisting of a horizontal direction, a vertical direction, and a depth direction (direction indicated by distance D in FIG. 5). If this relationship is expressed mathematically, the reflected components (Ra, Ga, Ba) of the virtual light at point P1 on the subject 501 will be as shown in the following equation (1).

Ra=α×(-L1・N1)/K1 ² ×Rt
Ga=α×(−L1・N1)/K1 ² ×Gt…(1)
Ba=α×(-L1・N1)/K1 ² ×Bt
Here, α is the intensity of light from the virtual light source, the gain value of the relighting correction amount, Rt, Gt, and Bt are RGB signals output from the degamma processing unit 402.

The reflected components (Ra, Ga, Ba) of the virtual light calculated as described above are output to the virtual light source addition processing section 407. The virtual light source addition processing unit 407 performs processing shown in the following equation (2) to add the reflected components (Ra, Ga, Ba) of the virtual light to the RGB signal output from the degamma processing unit 402.

Rout=Rt+Ra
Gout=Gt+Ga...(2)
Bout=Bt+Ba
The RBG signals (Rout, Gout, Bout) that have been relighted by the virtual light source addition processing unit 407 are input to the gamma processing unit 408, where gamma correction is performed. Then, the luminance/color difference signal generation unit 409 generates and outputs a luminance signal Y and color difference signals RY, BY signals from the gamma-processed RGB signals (R'out, G'out, B'out). do.

FIG. 7 shows an example in which the rewriting processing unit 114 performs the above-mentioned rewriting processing. FIG. 7(a) is an example of a captured image before relighting processing, and FIG. 7(b) is an example of a captured image after relighting processing. The dark object shown in FIG. 7(a) is corrected to be brighter as shown in FIG. 7(b) by applying virtual light and performing relighting processing.

The system control unit 50 stores the luminance signal Y and the color difference signals RY and BY outputted by the relighting processing unit 114 in the image memory 106 under the control of the memory control unit 107, and then uses the codec unit 110 to compress and code them. make a change. Further, the information is recorded on the recording medium 112 via the I/F 111.

Next, rewriting processing by the rewriting processing unit 114 in this embodiment will be explained along the flowcharts of FIGS. 8 and 9. This process is performed when the relighting process is selected by the user's operation via the operation unit 120, and the image processed by the image processing unit 105 and stored in the image memory 106 (luminance signal Y and color difference signal R -Y, BY).

In S801, the virtual light source setting unit 405 sets, as the first subject information, information about the subject included in the image to be relighted, which the subject information detection unit 208 acquired from the face/facial organ detection unit 113 in S305 of FIG. Obtain subject information such as number of people, size, and positional relationship.

In S802, the virtual light source setting unit 405 sets, as second subject information, the number, size, Obtain subject information such as positional relationship.

In S803, the distance calculation unit 403 further generates a weighted map (mapK) based on the distance between the digital camera 100 and the subject. Specifically, first, the distance calculation unit 403 calculates the distance K in pixel units based on two-dimensional object distance information obtained in pixel units of the captured image acquired from the distance measurement sensor 123 (distance map). Then, a value obtained by normalizing 1/(K ² ) with an arbitrary bit width for each pixel is used as a distance weighting map (mapK).

In S804, the normal line calculation unit 404 generates a normal line map (mapN) based on the distance map acquired from the distance calculation unit 403. Specifically, as described with reference to FIG. 5, the object normal vector N is calculated for each pixel, and the direction cosine with respect to each coordinate axis direction is determined. Then, the obtained direction cosine is expressed in an arbitrary bit width for each pixel to form a weighting map (mapN) based on the normal line.

In S805, the virtual light source setting unit 405 calculates the relighting reliability of each subject. A method for calculating rewriting reliability will be explained using FIGS. 10 and 11.

FIG. 10 shows an example of a captured image when there are two subjects. FIG. 10(a) is a photographed image before relighting processing, and FIG. 10(b) is an image diagram showing the position and irradiation range of a virtual light source with respect to the photographed image. Furthermore, FIG. 10(c) is an image diagram of the virtual light reflection component calculation result calculated by the virtual light source reflection component calculation unit 406, and FIG. 10(d) is an example of the image after the relighting process. The two subjects, which were dark as shown in FIG. 10(a), are corrected to be brighter as shown in FIG. 10(d) by applying virtual light and performing relighting processing. FIG. 10(e) shows an example in which relighting processing is performed using only that information when, for example, a subject on the right side is detected and a subject on the left side is not detected. ing. In FIG. 10E, the relighting process is performed only on the detected right subject, resulting in an unnatural image.

FIG. 11 is a diagram illustrating a method for calculating rewriting reliability from the first subject information acquired in S801 and the second subject information acquired in S802. Here, an example will be described in which subject detection is performed on captured images of two people as shown in FIG. 10(a). Note that in the example given here, the number of subjects detected by the face/facial organ detection unit 113 as the first subject information and the number of subjects detected by the head detection unit 115 as the second subject information match. This is an example of a case where

FIG. 11A shows a detected image detected by the subject information detection unit 208 as the first subject information. FIG. 11B shows information acquired by the subject information detection unit 208 as the first subject information. 1101 is the image No. , 1102 is the type of object detection, and 1103 is the object number. , 1104 indicates the face center coordinates of the detected face, 1105 indicates the face size of the detected face, and 1106 indicates the reliability of subject detection.

FIG. 11C shows a detected image detected by the subject information detection unit 208 as second subject information. FIG. 11D shows information acquired by the subject information detection unit 208 as the second subject information. 1107 is image No. , 1108 is the type of object detection, and 1109 is the object number. , 1110 indicates the face center coordinates of the detected face, 1111 indicates the face size of the detected face, and 1112 indicates the reliability of subject detection.

FIG. 11(e) shows the results of calculating the rewriting reliability from the first and second object detection information shown in FIGS. 11(a) to 11(d). 1113 is image No. , 1114 is the subject No. , 1115 indicates the type of subject detection, 1116 indicates the face center coordinates of the detected face, and 1117 indicates the face size of the detected face. Further, 1118 indicates a flag indicating whether the detected subject is a main subject group, 1119 indicates the reliability of subject detection, and 1120 indicates the rewriting reliability.

Here, the flag indicating whether or not the subject shown at 1118 is a main subject group is determined using a known main subject detection technique. For example, if the positional coordinates of the subject are close to the center of the captured image and the face area of the subject in the captured image is large, it is determined that the subject is in the main subject group, and the flag is set to 1. As a method for calculating the rewriting reliability shown in 1120, for example, the average value of the reliability of each object detection shown in 1119 is set as the rewriting reliability.

Returning to the description of FIG. 8, in S806, the virtual light source setting unit 405 determines whether the rewriting reliability of all the people included in the captured image is equal to or greater than a predetermined threshold Th. If everyone's rewriting reliability is equal to or higher than the threshold, the process advances to S807, and if it is less than the threshold, the process advances to S907 in FIG.

In S807, the virtual light source setting unit 405 generates a virtual light weighting map. Specifically, the light source direction vector -L of the virtual light source is calculated for each pixel, and its direction cosine with respect to each coordinate axis direction is determined. Then, the obtained direction cosine is expressed in an arbitrary bit width for each pixel to form a weighting map (mapL) based on the virtual light source. Note that the parameter settings of the virtual light source for calculating the light source vector -L of the virtual light source are determined based on the subject information input from the subject information detection unit 208. For example, if the brightness distribution within the acquired face area is biased, the position, irradiation range, and intensity of the virtual light source are determined so that the virtual light shines on the area where the brightness value is low.

For example, if the coordinates in the photographed image of an area with low luminance value are (x1, y1), the reflected components of virtual light by the subject (Ra(x1, y1), Ga(x1, y1), Ba(x1, y1) )) is expressed by the following equation (3).

Ra(x1, y1)=α×(-L(x1, y1)・N(x1, y1))/K(x1, y1) ² ×Rt
Ga(x1, y1)=α×(−L(x1, y1)・N(x1, y1))/K(x1, y1) ² ×Gt…(3)
Ba(x1, y1)=α×(-L(x1, y1)・N(x1, y1))/K(x1, y1) ² ×Bt
Note that in equation (3), α is the intensity of light from the virtual light source. Furthermore, L(x1, y1) is the light source direction vector of the virtual light source at the position on the subject corresponding to the coordinates (x1, y1), and N(x1, y1) is the light source direction vector on the subject corresponding to the coordinates (x1, y1). is the normal vector at the position. Further, K(x1, y1) indicates the distance between the virtual light source and the position on the subject corresponding to the coordinates (x1, y1). In order for the virtual light to hit the subject at coordinates (x1, y1), which is an area with low brightness, (Ra(x1, y1), Ga(x1, y1), Ba(x1, y1)) must be correct. The intensity α of the virtual light source and the distance K(x1, y1) to the subject are controlled so that the value . Furthermore, the irradiation range of the virtual light source is controlled by acquiring coordinate information that has a luminance value lower than a predetermined threshold value based on the acquired luminance distribution information within the face area, and controlling the irradiation range to include the coordinates. . In addition, if the strength α of the virtual light source is set too high, there is a possibility that problems such as overexposure or gradation inversion may occur, so the range of the strength α of the virtual light source should be set to a high luminance value that is outside the irradiation range. The brightness value is set within the range of the average brightness value of the area ±β, and is controlled within the range of a certain threshold value β. Through the above-described processing, the virtual light source setting unit 405 calculates the position range, irradiation range, and intensity range of the virtual light source, and determines the average value within the range as the position, irradiation range, and intensity of the virtual light source. do.

In S808, the virtual light source reflection component calculation unit 406 calculates virtual light reflection components (Ra, Ga, Ba) for the detected subject area. Note that the reflected components (Ra, Ga, Ba) can be calculated using equation (1) as described above. This equation (1) is replaced by a weighting map based on distance (mapK) in S803, a weighting map based on the normal to the subject (mapN) in S804, and a weighting map based on the virtual light source (mapL) in S807. That is, the reflected component of the virtual light can be calculated using the following equation (4).

Ra=α×mapL・mapN・mapK×Rt
Ga=α×mapL・mapN・mapK×Gt…(4)
Ba=α×mapL・mapN・mapK×Bt
The calculation result of the virtual light reflection component is the result of multiplying the gain α (intensity), the weighting map by the virtual light source (mapL), the weighting map by the normal (mapN), and the weighting map by distance (mapK). ing. The virtual light source reflection component calculation unit 406 combines the virtual light reflection components of each subject group calculated in S808, and calculates virtual light reflection components (Ra, Ga, Ba) from the virtual light source for the entire image. For example, in the case of a plurality of subjects as in the photographed image shown in FIG. 10, the virtual light reflection components of each subject are combined. The compositing method may be an existing method, but for example, a relatively bright compositing process may be used. A synthesized example is shown in FIG. 10(c).

In S809, rewriting processing is performed. Specifically, as shown in equation (2) above, the virtual light reflection components (Ra, Ga, Ba) calculated in S808 are converted to the output (Rt) of the degamma processing unit 402 in the virtual light source addition processing unit 407 , Gt, Bt). When the rewriting processing is finished, the processing by the rewriting processing unit 114 is ended.

Through the above processing, the relighting processing unit 114 performs relighting processing on the subject in the captured image.

On the other hand, if it is determined in S806 that the rewriting reliability of all the people included in the photographed image is not greater than or equal to the predetermined threshold, the process advances to S907 in FIG.

In S907, it is determined whether the relighting reliability of each subject in the main subject group is greater than or equal to a predetermined threshold. If it is smaller than the predetermined threshold, the process advances to S911, and if it is greater than or equal to the predetermined threshold, the process advances to S908. This determination method will be explained using FIG. 12.

FIG. 12 is a diagram illustrating a method of calculating rewriting reliability from the first subject information acquired in S801 and the second subject information acquired in S802. Here, a description will be given of an example in which a subject is detected in a photographed image such as that shown in FIG. 10(a) in which another subject appears small in the distance from two subjects.

FIG. 12A shows a detected image detected by the subject information detection unit 208 as the first subject information. FIG. 12(b) shows information acquired by the subject information detection unit 208 as the first subject information. 1201 is the image No. , 1202 is the type of subject detection, and 1203 is the subject number. , 1204 indicates the face center coordinates of the detected face, 1205 indicates the face size of the detected face, and 1206 indicates the reliability of subject detection.

FIG. 12C shows a detected image detected by the subject information detection unit 208 as second subject information. FIG. 12(d) shows information acquired by the subject information detection unit 208 as the second subject information. 1207 is image No. , 1208 is the type of object detection, and 1209 is the object number. , 1210 indicates the face center coordinates of the detected face, 1211 indicates the face size of the detected face, and 1212 indicates the reliability of subject detection.

FIG. 12(e) shows the results of calculating the rewriting reliability from the first and second object detection information shown in FIGS. 12(a) to 12(d). 1213 is image No. , 1214 is the subject No. , 1215 indicates the type of subject detection, 1216 indicates the face center coordinates of the detected face, and 1217 indicates the face size of the detected face. Further, 1218 indicates a flag indicating whether the detected subject is a main subject group, 1219 indicates the reliability of subject detection, and 1220 indicates the rewriting reliability.

Here, the flag indicating whether or not the subject shown at 1218 is a main subject group is determined by a known main subject detection technique. For example, if the positional coordinates of the subject are close to the center of the captured image and the face area of the subject in the captured image is large, it is determined that the subject is in the main subject group, and the flag is set to 1. As a method for calculating the rewriting reliability shown in 1220, the average value of the reliability of each object detection shown in 1219 is used as the rewriting reliability. In step S907 described above, it is determined whether the rewriting reliability of the subjects determined to be the main subject group is equal to or greater than a predetermined threshold. For example, when the reliability threshold is 80, in FIG. 12, the subject determined to be the main subject group of the captured image is No. 1 and no. It is 2. And No. 1 and no. The relighting reliability of subject No. 2 is 100 and 90, which exceeds the threshold of 80. Therefore, in S907, it is determined that the reliability is high, and the process advances to S908.

In S908, the virtual light source setting unit 405 sets parameters such as the position, irradiation range, and intensity of each virtual light source. Parameters such as the position, irradiation range, and intensity of this virtual light source are set in the same manner as in S807 described above. However, in S807, irradiation is performed using normal virtual light source parameters, whereas in S908, irradiation is performed according to the rewriting reliability. to control virtual light source parameters. As a control method, the higher the relighting reliability of each subject in the main subject group, the smaller the degree of the main subject in the non-main subject group (in other words, the higher the possibility that it is not the main subject), Control so that the setting values are the same as normal virtual light source parameters. In the opposite case, the effect is controlled to be weaker than the normal virtual light source parameters. Based on the parameters of the virtual light source set in this way, a weighting map based on the virtual light source is generated. Specifically, the light source direction vector -L of the virtual light source is calculated for each pixel, and its direction cosine with respect to each coordinate axis direction is determined. The obtained direction cosine is expressed with an arbitrary bit width to form a weighting map (mapL) based on the virtual light source.

In S909, the virtual light source reflection component calculation unit 406 calculates the virtual light reflection components (Ra, Ga, Ba) for the detected subject area. The calculation method is the same as the method described above in S808.

In S910, rewriting processing is performed. Specifically, the virtual light source addition processing unit 407 adds the reflection components (Ra, Ga, Ba) of the virtual light calculated in S908 to the output (Rt, Gt, Bt) of the degamma processing unit 402. When the rewriting processing is finished, the processing by the rewriting processing unit 114 is ended.

On the other hand, if the rewriting reliability of each subject in the main subject group is less than the predetermined threshold in S907, the process advances to S911. This determination method will be explained using FIG. 13.

FIG. 13 is a diagram illustrating a method for calculating rewriting reliability from the first subject information acquired in S801 and the second subject information acquired in S802. Here, we will explain an example in which two people are detected in the first object detection and only one is detected in the second object detection in the photographed image of two people as shown in FIG. 10(a). .

FIG. 13A shows a detected image detected by the subject information detection unit 208 as the first subject information. FIG. 13(b) shows information acquired by the subject information detection unit 208 as the first subject information. 1301 is image No. , 1302 is the type of subject detection, and 1303 is the subject number. , 1304 indicates the face center coordinates of the detected face, 1305 indicates the face size of the detected face, and 1306 indicates the reliability of subject detection.

FIG. 13(c) shows a detected image detected by the subject information detection unit 208 as second subject information. FIG. 13(d) shows information acquired by the subject information detection unit 208 as the second subject information. 1307 is image No. , 1308 is the type of subject detection, and 1309 is the subject number. , 1310 indicates the face center coordinates of the detected face, 1311 indicates the face size of the detected face, and 1312 indicates the reliability of subject detection.

FIG. 13(e) shows the results of calculating the rewriting reliability from the first and second object detection information shown in FIGS. 13(a) to 13(d). 1313 is image No. , 1314 is the subject No. , 1315 indicates the type of subject detection, 1316 indicates the face center coordinates of the detected face, and 1317 indicates the face size of the detected face. Further, 1318 indicates a flag indicating whether the detected subject is a main subject group, 1319 indicates the reliability of subject detection, and 1320 indicates the rewriting reliability. In addition, in FIG. 13(c), in the second subject detection information, No. 1 of the two subjects is detected. Subject 2 has not been detected. Therefore, this undetected No. The subject reliability of subject No. 2 is considered to be 0. Therefore, in the calculation of rewriting reliability in FIG. 13(e), No. For subject No. 2, the rewriting reliability is 50 by taking the average of the object detection reliability of 100 in the first object detection information and the object reliability of 0 in the second object detection information.

Here, the flag indicating whether or not the subject shown at 1318 is a main subject group is determined by a known main subject detection technique. For example, if the positional coordinates of the subject are close to the center of the captured image and the face area of the subject in the captured image is large, it is determined that the subject is in the main subject group, and the flag is set to 1. As a method for calculating the rewriting reliability shown in 1320, the average value of the reliability of each object detection shown in 1319 is used as the rewriting reliability. In step S907 described above, it is determined whether the rewriting reliability of the subjects determined to be the main subject group is equal to or higher than an arbitrary threshold value. For example, when the reliability threshold is 80, in FIG. 13, the subject determined to be the main subject group of the captured image is No. 1 and no. It is 2. And No. 1 and no. The relighting reliability of subject No. 2 is 100 and 50, and one of them is below the threshold of 80. Therefore, in S907, it is determined that the reliability is low, and the process advances to S911.

In S911, brightness correction processing is performed on the entire screen without performing relighting processing. For example, instead of correcting the brightness by setting a virtual light source for each subject, the brightness is corrected by gain processing such that the lower the brightness value, the brighter it becomes.

Through the above processing, the relighting processing unit 114 performs relighting processing on the subject in the captured image. This makes it possible to avoid generating an unnatural relighting-processed image in which some subjects become brighter and some subjects become darker, despite the subjects being in the main subject group.

Although the present embodiment has been described using the digital camera 100 as an example of an image processing device, the present invention can also be applied to an image processing device such as a personal computer. In that case, the image processing apparatus may be an image processing apparatus that acquires an image photographed by an imaging means such as a camera, and performs relighting processing by the user arbitrarily setting virtual light source parameters for the acquired image. If there is additional information such as face detection results, head detection results, distance information, normal information, etc. together with the image, the relighting process may be performed using such information. Even in that case, by dividing the subjects into groups, the user does not need to set virtual light source parameters for each subject individually, so relighting can be performed without complicating the virtual light source parameter setting method. It becomes possible.

Further, in this embodiment, the case where there is one virtual light source has been described, but the present invention is not limited to this. For example, a configuration may be adopted in which relighting processing is performed using a plurality of virtual light sources, such as one virtual light source from diagonally above and to the right of the subject, and another virtual light source from directly to the side of the subject.

Furthermore, in this embodiment, a case has been described in which brightening is corrected using added light, but relighting to darken may also be performed. In that case, the gain value α of the virtual light source is made negative (subtraction light). Alternatively, a configuration may be adopted in which specular reflection light is added to the subject. In this way, one of a plurality of types can be selected as virtual light.

In addition, the virtual light source parameters that are set include values related to the number of virtual light sources, such as the number of additional lights that brighten the subject, the number of subtractive lights that darken the subject, and the number of specular lights that add specular reflection to the subject. It may be included.

Further, the method of calculating the distance D between the position of the virtual light source and the pixel to be processed is not limited to the direction of this embodiment, and any calculation method may be used. For example, the camera position and subject position may be acquired as three-dimensional positions, and the distance may be calculated in three dimensions.

Further, when adding a virtual light source, calculation was performed using a formula that is inversely proportional to the square of the distance, but the amount of added virtual light sources is not limited to calculation using this method. For example, a formula that is inversely proportional to the distance D or a formula that changes the irradiation range in a Gaussian distribution may be used.

Furthermore, in this embodiment, the description has been made using an example of a face/facial organ detection means and a head detection means as a plurality of object detection means, but the present invention is not limited to this, and other object area detection means may also be used. I do not care.

Furthermore, in the above example, the case where a person is detected as the subject is explained, but the subject is not limited to a person, and a predetermined subject (for example, a car, an animal, a plant, etc.) may be detected. .

(Other embodiments)
The present invention also provides a system or device with a program that implements one or more functions of the above-described embodiments via a network or a storage medium, and one or more processors in a computer of the system or device reads the program. This can also be achieved by executing a process. It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

50: system control unit, 101: optical system, 102: shutter, 103: imaging unit, 104: A/D converter, 105: image processing unit, 106: image memory, 107: memory control unit, 109: display unit, 113: Face/facial organ detection section, 114: Rewriting processing section, 115: Head detection section

Claims (13)

an acquisition means for acquiring a photographed image;
a plurality of detection means for detecting subjects included in the captured image;
a setting means for setting a virtual light source that is a virtual light source that illuminates the subject;
a correction means for correcting the brightness of the photographed image using the virtual light source set by the setting means,
The setting means separately calculates reliability of object detection for each of the detection results of the plurality of detection means for the one object based on the detection results of the one object by the plurality of detection means, and A relighting reliability indicating the reliability of the subject whose brightness should be corrected is calculated by averaging the reliability calculated in , and parameters of the virtual light source are set based on the calculated relighting reliability. An image processing device characterized by: When the relighting reliabilities of the plurality of subjects included in the captured image are all equal to or higher than a predetermined threshold, the setting means adjusts the virtual light source so that all of the plurality of subjects are illuminated with the virtual light source. The image processing apparatus according to claim 1, wherein parameters are set. If some of the rewriting reliability of a plurality of subjects included in the photographed image is smaller than a predetermined threshold, the setting means selects one or more main subjects from the plurality of subjects, and selects one or more main subjects from the plurality of subjects, and Alternatively, when the relighting reliabilities of a plurality of main subjects are all greater than or equal to the predetermined threshold, the setting means sets the virtual light source to illuminate all of the one or more main subjects with the virtual light source. The image processing apparatus according to claim 1, wherein the image processing apparatus sets parameters of . If some of the rewriting reliability of a plurality of subjects included in the photographed image is smaller than a predetermined threshold, the setting means selects one or more main subjects from the plurality of subjects, and selects one or more main subjects from the plurality of subjects, and The image according to claim 1, wherein the correction means does not perform correction using the virtual light source if any of the relighting reliability of a plurality of main subjects is smaller than the predetermined threshold value. Processing equipment. 5. The image processing apparatus according to claim 4, wherein the correction means corrects the brightness of the entire screen when the correction using the virtual light source is not performed. The setting means calculates the rewriting reliability higher when the number of objects detected by the plurality of detection means matches than when the number of objects detected is different. 5. The image processing device according to any one of 5. The image according to any one of claims 1 to 6 , wherein the parameters of the virtual light source set by the setting means include a position of the virtual light source, an irradiation range of the virtual light, and an intensity of the virtual light. Processing equipment. The parameters of the virtual light source set by the setting means indicate any of a plurality of types of virtual light including additive light that brightens the subject, subtracted light that darkens the subject, and specular light that adds specular reflection to the subject. The image processing apparatus according to any one of claims 1 to 7 , characterized in that the image processing apparatus includes a parameter. The virtual light source parameters set by the setting means include a plurality of types of virtual light including the number of additional lights that brighten the subject, the number of subtractive lights that darken the subject, and the number of specularly reflected lights that add specular reflection to the subject. 9. The image processing apparatus according to claim 1, further comprising a parameter indicating the number of images. an imaging means for photographing a subject and outputting the photographed image;
An image processing device according to any one of claims 1 to 9 ,
An imaging device comprising: A method for controlling an image processing apparatus, comprising: an acquisition means for acquiring a photographed image; and a plurality of detection means for detecting a subject included in the photographed image, the method comprising:
a setting step of setting a virtual light source that is a virtual light source that illuminates the subject;
a correction step of correcting the brightness of the photographed image using the virtual light source set in the setting step,
In the setting step, the reliability of object detection is separately calculated for each of the detection results of the plurality of detection means for the one object based on the detection results of the one object by the plurality of detection means, and A relighting reliability indicating the reliability of the subject whose brightness should be corrected is calculated by averaging the reliability calculated in , and parameters of the virtual light source are set based on the calculated relighting reliability. A method for controlling an image processing device, characterized in that: A program for causing a computer to function as each means of the image processing apparatus according to claim 1 . A computer-readable storage medium storing a program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 9 .

Images