JP7169841B2 - IMAGE PROCESSING DEVICE, IMAGING DEVICE, IMAGE PROCESSING METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to an image processing device, an imaging device, an image processing method, and a program.

2. Description of the Related Art Conventionally, in photography, areas of light and shadow that appear on a subject are adjusted by adjusting light using auxiliary lighting or a reflector. This makes it possible to take photographs with various impressions of the subject. In addition, as a technique for performing these light adjustments after photographing, there is a technique (relighting) for adding highlight and shadow components by reflecting light to the subject area. As a result, it is possible to correct dark areas and generate an image with an emphasized stereoscopic effect.

Patent Document 1 acquires distance information of a subject, obtains shape information of the subject based on the distance information, processes a photographed image based on the shape information, the distance information, and the position of a virtual light source, A technique for generating an image irradiated with light from a virtual light source is disclosed.

JP 2016-86246 A

In Patent Document 1, a normal line is calculated from distance information of a subject, and relighting control is performed by obtaining reflection characteristics of a virtual light source based on the normal line, the subject distance, and the position, direction, and intensity of the virtual light source. This method requires a large amount of computation for calculating the object distance, the normal line, and the reflection characteristics based on the virtual light source, and the processing takes a long time. In particular, when the number of virtual light sources is increased and multi-light irradiation is performed, it is necessary to repeat the calculation for the number of virtual light sources, which increases the time required for processing.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a technique for reducing the processing load of relighting using a plurality of virtual light sources.

In order to solve the above problems, the present invention provides a first reflection characteristic indicating a reflection characteristic of a first virtual light source in a predetermined subject included in the input image before adding the effect of the virtual light source to the input image. a synthesizing unit for acquiring synthetic reflection characteristic data by performing synthesis processing for synthesizing the data and second reflection characteristic data indicating the reflection characteristics of the second virtual light source in the predetermined subject; and generating means for generating a reflection component of a subject included in the input image based on the reflection characteristic indicated by the obtained synthetic reflection characteristic data; and the reflection component generated by the generating means for the input image. and adding means for adding the image processing apparatus.

In addition, other features of the present invention will be further clarified by the description in the attached drawings and the following detailed description.

According to the present invention, it is possible to reduce the processing load of relighting using a plurality of virtual light sources.

2 is a block diagram showing the configuration of the digital camera 100. FIG. FIG. 2 is a block diagram showing the configuration of an image processing unit 105; FIG. 3 is a block diagram showing the configuration of a relighting processing unit 202; 4 is a flowchart of synthetic parameter setting processing executed by the system control unit 50 for the reflection characteristic map synthesizing unit 301 according to the first embodiment; FIG. 4 is a diagram showing a relighting mode according to the luminance of a subject; The figure which shows a reflection characteristic map. FIG. 5 is a diagram for explaining the relationship between relighting modes and reflection characteristic maps to be used; The figure which shows the reflection characteristic map after composition. The figure explaining a fitting process. 9 is a flowchart of synthetic parameter setting processing executed by the system control unit 50 for the reflection characteristic map synthesizing unit 301 according to the second embodiment. FIG. 10 is a diagram showing a user interface for user operation to install a virtual light source; FIG. 12 is a block diagram showing the configuration of a relighting processing unit 202 according to the third embodiment; FIG. 4A and 4B are diagrams for explaining a process of generating a diffuse reflection map D and a specular reflection map S; FIG.

Embodiments of the present invention will be described below with reference to the accompanying drawings. It should be noted that the technical scope of the present invention is determined by the claims and is not limited by the following individual embodiments. Also, not all combinations of features described in the embodiments are essential to the present invention. It is also possible to combine features described in separate embodiments as appropriate.

In each of the following embodiments, a configuration in which an image processing device is applied to a digital camera (imaging device) will be described as an example.

[First Embodiment]
FIG. 1 is a block diagram showing the configuration of the digital camera 100. As shown in FIG. In FIG. 1, 101 is a lens group including a zoom lens and a focus lens. A shutter 102 has an aperture function. An imaging unit 103 is composed of a CCD, a CMOS element, or the like that converts an optical image into an electric signal. 104 is an A/D converter that converts an analog signal into a digital signal. An image processing unit 105 performs various image processing such as white balance processing, relighting processing, γ processing, edge enhancement processing, and color correction processing on image data output from the A/D converter 104 . 106 is an image memory. A memory control unit 107 controls the image memory 106 . A D/A converter 108 converts an input digital signal into an analog signal. A display unit 109 includes an LCD and the like. A codec unit 110 compresses, encodes, and decodes image data.

111 is an interface (I/F) with the recording medium 112 . 112 is a recording medium such as a memory card or a hard disk. A face detection unit 113 detects an area in which a face appears in the captured image. A system control unit 50 controls the entire system of the digital camera 100 . 121 is a non-volatile memory such as an EEPROM that stores programs, parameters, and the like. A system memory 122 develops constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like. 123 is a strobe (light source device). A distance measuring sensor 124 measures the distance to a subject.

Next, the basic operation of the digital camera 100 configured as described above when photographing a subject will be described. The imaging unit 103 photoelectrically converts light incident through the lens 101 and the shutter 102 and outputs the converted image signal to the A/D converter 104 as an input image signal. The A/D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal and outputs the digital image signal to the image processing unit 105 . Note that the digital image signal may be recorded on the recording medium 112 as RAW image data.

The image processing unit 105 performs color conversion processing such as white balance, relighting processing, γ processing, edge enhancement processing, etc. on the image data from the A/D converter 104 or the image data from the memory control unit 107. . Further, the image processing unit 105 performs predetermined evaluation value calculation processing using the face detection result of the face detection unit 113 and captured image data. The system control unit 50 performs exposure control and ranging control based on the obtained evaluation value. As a result, TTL (through-the-lens) AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and the like can be performed.

Image data output from the image processing unit 105 is written into the image memory 106 via the memory control unit 107 . The image memory 106 stores image data output from the imaging unit 103 and image data to be displayed on the display unit 109 . The D/A converter 108 converts image display data stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109 . The display unit 109 displays on a display device such as an LCD according to the analog signal from the D/A converter 108 .

A codec unit 110 compresses and encodes the image data recorded in the image memory 106 based on standards such as JPEG and MPEG. The system control unit 50 stores the encoded image data in the recording medium 112 via the I/F 111 .

The basic operation of photographing a subject has been described above. In addition to the basic operations described above, the system control unit 50 executes the programs recorded in the non-volatile memory 121 described above to realize each process of the present embodiment described later. The program here is a program for executing various flowcharts described later in this embodiment. At this time, constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like are developed in the system memory 122. FIG.

Next, details of the image processing unit 105 will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the image processing unit 105. As shown in FIG. In FIG. 2, 200 is a synchronization processing unit, 201 is a WB amplification unit, 202 is a relighting processing unit, and 203 is a luminance/color signal generation unit. 204 is an edge enhancement processor, 205 is a brightness gamma processor, 206 is a color conversion processor, 207 is a color gamma processor, 208 is a color difference signal generator, and 209 is an evaluation value generator.

The operation of the image processing unit 105 will be described. An image signal output from the A/D converter 104 in FIG. 1 is input to the image processing unit 105 . An image signal input to the image processing unit 105 is input to the synchronization processing unit 200 . The synchronization processing unit 200 performs synchronization processing on the input image data in Bayer RGB format to generate R, G, and B color signals.

Based on the white balance gain value calculated by the system control unit 50, the WB amplification unit 201 applies a gain to the RGB color signals to adjust the white balance. The WB amplification unit 201 inputs the RGB signals whose white balance has been adjusted to the relighting processing unit 202 . Further, the WB amplification unit 201 converts the RGB signals whose white balance has been adjusted into the luminance/color signal generation unit 203 as R′G′B′ signals used to generate a luminance signal Y′ used to generate an evaluation value to be described later. to enter.

The relighting processing unit 202 outputs the RGB signals to the luminance/color signal generation unit 203 after performing relighting processing, which will be described later, on the input RGB signals. The luminance/color signal generation unit 203 generates a luminance signal Y from the RGB signals, and outputs the generated luminance signal Y to the contour enhancement processing unit 204 and the color signals RGB to the color conversion processing unit 206, respectively. Also, the luminance/chrominance signal generation unit 203 generates a luminance signal Y′ from the R′G′B′ signal input from the WB amplification unit 201 and inputs the luminance signal Y′ to the evaluation value generation unit 209 .

The edge enhancement processing unit 204 performs edge enhancement processing on the luminance signal Y and outputs the result to the luminance gamma processing unit 205 . A luminance gamma processing unit 205 performs gamma correction on the luminance signal Y and outputs the luminance signal Y to the image memory 106 . A color conversion processing unit 206 performs desired color balance conversion by performing matrix calculations on RGB signals. A color gamma processing unit 207 performs gamma correction on the RGB color signals. A color difference signal generator 208 generates color difference signals RY and BY from the RGB signals and outputs them to the image memory 106 . The image signals (Y, RY, BY) output to the image memory 106 are compression-encoded by the codec section 110 and recorded on the recording medium 112 .

The evaluation value generator 209 generates an evaluation value for estimating the actual state of the environment light source. The state of the environment light source in this embodiment is the direction and intensity (brightness) of the environment light source with respect to the object. Therefore, the evaluation value generator 209 generates and outputs an evaluation value for estimating the direction and intensity of the ambient light source. Specifically, the evaluation value generator 209 divides the image into a plurality of blocks as shown in FIG. 5, and calculates the average luminance value for each block. The evaluation value generator 209 records the calculated evaluation value in the system memory 122 .

Next, the configuration and operation of the relighting processing unit 202 will be described with reference to FIG. FIG. 3 is a block diagram showing the configuration of the relighting processing unit 202. As shown in FIG. In FIG. 3, reference numeral 301 denotes a reflection characteristic map synthesizing unit that reads and synthesizes a plurality of reflection characteristic maps indicating the characteristics of the reflection of the virtual light source on the subject (reflection characteristics of the virtual light source on the subject). Reference numeral 302 denotes a fitting processing unit that adapts the reflection characteristic map of the virtual light source to the subject. A virtual light source reflection component calculation unit 303 calculates a color component (virtual light source reflection component) reflected by the subject based on the reflection characteristic map. A virtual light source addition processing unit 304 adds a virtual light source reflection component to the input image. The operation of each unit of the relighting processing unit 202 will be described in detail below.

The reflection characteristic map synthesizing unit 301 reads and synthesizes a plurality of reflection characteristic maps (details will be described later) of virtual light sources prepared in advance. In the process of synthesizing a plurality of reflection characteristic maps, the system control unit 50 determines which reflection characteristic map to read, and determines synthesis parameters.

Synthesis parameter setting processing executed by the system control unit 50 for the reflection characteristic map synthesizing unit 301 will be described with reference to FIG. When the subject is photographed, the processing of the flowchart of FIG. 4 is started.

In S401, the system control unit 50 acquires the feature amount of the photographed subject. Specifically, the system control unit 50 acquires the average luminance value of each block of the image generated by the evaluation value generation unit 209 (FIG. 2). An example of block division is shown in FIG. The evaluation value generation unit 209 calculates the average brightness value of the subject for each divided block as shown in FIG. 5, and the system control unit 50 acquires the calculated average brightness value.

In S402, the system control unit 50 acquires strobe irradiation information at the time of photographing. The strobe irradiation information is information indicating whether or not the strobe 123 has emitted light at the time of photographing.

In S403, the system control unit 50 determines a relighting mode according to the feature amount of the subject acquired in S401. In this embodiment, there are four relighting modes: (1) oblique light correction mode, (2) top light correction mode, (3) backlight correction mode, and (4) flat correction mode. First, the system control unit 50 acquires information on the face area detected by the face detection unit 113 in the captured image. For example, the system control unit 50 acquires information on the face area 501 (see FIGS. 5A to 5D) from the face detection unit 113. FIG. Next, the system control unit 50 acquires the subject feature amount of each block included in the face area 501 among the subject feature amounts of each block acquired in S401. The system control unit 50 analyzes the distribution of subject feature amounts (brightness) in the face area 501 and determines to which of FIGS. 5A to 5D the distribution corresponds. As shown in FIG. 5A, when there is a difference in brightness between the left and right sides of the face area 501, the system control unit 50 selects the oblique light correction mode. As shown in FIG. 5B, when there is a difference in brightness between the upper and lower sides of the face area 501 and the upper side is bright, the system control unit 50 selects the top light correction mode. As shown in FIG. 5C, when the difference in brightness in the face area 501 is small and the face area 501 is darker than other areas, the system control unit 50 selects the backlight correction mode. As shown in FIG. 5D, when the difference in brightness is small in the face area 501 and the face area 501 is sufficiently bright, the system control unit 50 selects the flat correction mode. Although FIG. 5A shows a state in which the left side of the face area 501 is bright and the right side is dark, the oblique light correction mode is also selected when the left side of the face area 501 is dark and the right side is bright. If the distribution of subject feature amounts (luminance) in the face area 501 does not correspond to any of the above four relighting modes, the system control unit 50 determines not to perform image correction by relighting.

In S404, the system control unit 50 selects a reflection characteristic map (reflection characteristic data) corresponding to the relighting mode determined in S403, and determines the composite characteristic of the selected reflection characteristic map. A reflection characteristic map is an image showing the reflection characteristic of an average face shape model as a result of relighting from a predetermined light source position. An example of a reflection characteristic map is shown in FIG. The upper part of FIG. 6 shows a reflection characteristic map in which the position of the light source is changed from 90 degrees to the left (L90) to 90 degrees to the right (R90). The lower part of FIG. 6 shows reflection characteristic maps when the position of the light source is set to 60 degrees downward (Down60) and 60 degrees upward (Up60). It is assumed that such a reflection characteristic map is generated in advance based on an average face shape model and recorded in the nonvolatile memory 121 (FIG. 1).

The relationship between the relighting mode and the reflection characteristic map to be used will be described with reference to FIG. In the example of FIG. 7, when the relighting mode is the "oblique light correction mode", the left and right sides of the subject are compared, and the dark side is irradiated with the virtual light source at an angle of 60 degrees. It also dims light from an angle of 60 degrees to the bright side. In the example of FIG. 5A, the subject is dark on the right side and bright on the left side. Therefore, the reflection characteristic map (R60) when the virtual light source is irradiated from the right 60 degrees and the reflection characteristic map (L60) when the virtual light source is irradiated from the left 60 degrees are combined according to the following formula (1). .
M = α × (R60) - β × (L60) (1)
Here, M is a reflection characteristic map after synthesis, and α and β are coefficients (synthesis characteristics) at the time of synthesis and take values from 0 to 1. FIG. The system control unit 50 sets α larger as the dark part of the face is darker with respect to the predetermined reference luminance of the face, and sets β larger as the bright part of the face is brighter than the reference luminance. Since the β term is negative, it indicates extinction. FIG. 8A shows the reflection characteristic map after synthesis. The synthesized reflection characteristic map (synthetic reflection characteristic data) has a data format of an image (reflection characteristic image) expressing the reflection characteristic at each position of the subject. In FIG. 8A, parts brighter than the reference gradation (for example, 128 in the case of 8 bits) indicate addition light, and darker parts indicate subtraction light (the same applies to FIGS. 8B to 8D). .

Contrary to FIG. 5A, when the subject is dark on the left side and bright on the right side, the reflection characteristic map used in equation (1) is left-right reversed. That is, the reflection characteristic map (L60) when the virtual light source is irradiated from the left 60 degrees and the reflection characteristic map (R60) when the virtual light source is irradiated from the right 60 degrees are combined according to the following formula (2). .
M = α × (L60) - β × (R60) (2)

In equation (2), the system control unit 50 sets α larger as the darker part of the face is darker with respect to the predetermined reference luminance of the face, and sets β larger as the brighter part of the face is brighter than the reference luminance. .

Next, the case where the relighting mode is the "top light correction mode" will be described. In this case, as shown in FIG. 5B, the area around the subject's head is bright and the area below the mouth is dark. Therefore, by reducing the light from the upper side of the subject and irradiating the additional light from the lower side, the light striking the face is balanced. That is, the reflection characteristic map (Up60) when the virtual light source is irradiated from above 60 degrees and the reflection characteristic map (Down60) when the virtual light source is irradiated from below 60 degrees are synthesized according to the following formula (3). do.
M = α × (Down60) - β × (Up60) (3)

FIG. 8B shows a reflection characteristic map after synthesis in this case. In equation (3), the system control unit 50 sets α larger as the darker part of the face is darker with respect to the predetermined reference luminance of the face, and increases β as the brighter part of the face is brighter than the reference luminance. set.

When the relighting mode is the "backlight correction mode", the face is dark as a whole, so virtual light sources of added light are emitted from the front and from below the front. That is, the reflection characteristic map (F0) when the virtual light source is irradiated from the front and the reflection characteristic map (Down60) when the virtual light source is irradiated from below 60 degrees are combined according to the following formula (4).
M=α×(F0)+β×(Down60) (4)

FIG. 8C shows a reflection characteristic map after synthesis in this case. In equation (4), the system control unit 50 sets α and β to be larger as the face region 501 is darker with respect to the predetermined reference luminance of the face.

When the relighting mode is the "flat correction mode" and there is no strobe light emission, the virtual light source is emitted from either the left or the right at an angle of 60 degrees. In this case, reflection characteristic maps are not synthesized, and only gain adjustment is performed as shown in equation (5).
M=α×(L60 or R60) (5)

In equation (5), the system control unit 50 sets α larger as the face region 501 is darker with respect to the predetermined reference luminance of the face.

When the relighting mode is the “flat correction mode” and strobe light is emitted, the light from the front is reduced to cancel the strobe light, and the virtual light source is illuminated from the front upper side. Specifically, the reflection characteristic map (F0) when the virtual light source is irradiated from the front and the reflection characteristic map (Up60) when the virtual light source is irradiated from above 60 degrees are combined according to the following formula (6). do.
M = α × (Up60) - β × (F0) (6)

FIG. 8D shows a reflection characteristic map after synthesis in this case. In equation (6), the system control unit 50 sets α larger as the face region 501 is darker with respect to the predetermined reference luminance of the face, and sets β larger as the bright part of the face is brighter than the reference luminance. set.

The selection of the reflection characteristic map and the determination of the composite characteristic (the values and signs of the coefficients α and β) according to the relighting mode have been described above.

In S<b>405 , the system control unit 50 sets the synthesis parameters (reflection characteristic map and synthesis characteristics) determined in S<b>404 in the reflection characteristic map synthesis unit 301 .

The synthesizing parameter setting processing executed by the system control unit 50 for the reflection characteristic map synthesizing unit 301 has been described above. After that, the reflection characteristic map synthesizing unit 301 performs synthesis processing of the reflection characteristic maps based on the set synthesis parameters. The reflection characteristic map synthesizing section 301 outputs the synthesized reflection characteristic map to the fitting processing section 302 . The fitting processing unit 302 refers to the captured images (Rin, Gin, Bin) and performs fitting processing on the synthesized reflection characteristic map.

The fitting process will be described with reference to FIG. FIG. 9A shows a reflection characteristic map after synthesis. Feature points 901 to 903 corresponding to the positions of the eyes and mouth are preset in the reflection characteristic map. FIG. 9B shows captured images (Rin, Gin, Bin). First, the fitting processing unit 302 geometrically deforms (geometrically transforms) the reflection characteristic map based on the positions 904 to 906 of the feature points (eyes, mouth, etc.) of the photographed image. Specifically, the fitting processing unit 302 acquires the coordinate positions of the feature points (eyes, mouth, etc.) of the subject in the captured image from the face detection unit 113 (FIG. 1). Then, the fitting processing unit 302 enlarges, reduces, and rotates the reflection characteristic map so that the positional relationships between the feature points 901 to 903 and the positions 904 to 906 match. This makes it possible to compensate for the difference in shape between the average face shape model (predetermined subject) of the reflection characteristic map and the subject included in the captured image.

Next, the fitting processing unit 302 applies a joint bilateral filter to the reflection characteristic map using the captured image as a reference image. This makes it possible to apply the smoothing process to the reflection property map while preserving the edges of the reference image. The result of applying the joint bilateral filter is shown in FIG. 9(C). The fitting processing unit 302 outputs the reflection characteristic map Mf obtained by applying the joint bilateral filter to the virtual light source reflection component calculation unit 303 .

As in the case where the relighting mode is the "flat correction mode" and there is no strobe light emission, there is a case where the synthesis processing for synthesizing a plurality of reflection characteristic maps is not performed. In this case, as shown in the above equation (5), the reflection characteristic map synthesizing section 301 outputs the reflection characteristic map obtained by multiplying the selected reflection characteristic map by a coefficient to the fitting processing section 302. . Then, the fitting processing unit 302 performs the above-described fitting process on the reflection characteristic map output from the reflection characteristic map combining unit 301 in the same manner as when the combining process is performed.

The virtual light source reflection component calculation unit 303 multiplies the captured image (Rin, Gin, Bin) by the reflection characteristic map Mf after the fitting process according to Equation (7). Thereby, the reflected color components (Ra, Ga, Ba) of the subject by the virtual light source are calculated.
Ra = Mf x Rin
Ga=Mf×Gin (7)
Ba = Mf x Bin

The virtual light source reflection component calculation unit 303 outputs the calculated reflection components (Ra, Ga, Ba) of the subject due to the virtual light source to the virtual light source addition processing unit 304 .

The virtual light source addition processing unit 304 adds reflection components (Ra, Ga, Ba) of the virtual light source to the subject area according to Equation (8).
Rout = Rin + Ra
Gout=Gin+Ga (8)
Bout = Bin + Ba

As described above, according to the first embodiment, the digital camera 100 synthesizes a plurality of reflection characteristic maps, and calculates the reflection component of the subject included in the image based on the reflection characteristics indicated by the synthesized reflection characteristic map. and add the generated reflection component to the image. This makes it possible to reduce the processing load of relighting using a plurality of virtual light sources.

In the present embodiment, the case where the digital camera 100 has five levels of reflection characteristic maps in the horizontal direction and two levels in the vertical direction has been described as an example, but the way of holding the reflection characteristic maps is not limited to this. . For example, the digital camera 100 may hold more patterns of reflection characteristic maps, such as 10 levels for each of the top, bottom, left, and right. For a pair of symmetrical reflection characteristic maps, for example, 60 degrees to the left and 60 degrees to the right, only one of them is prepared, and the digital camera 100 flips the reflection characteristic map horizontally according to the state of the subject. You can use it by

Further, in the present embodiment, examples of addition and subtraction have been described as reflection characteristic map synthesizing operations, but operations for synthesizing reflection characteristic maps are not limited to addition and subtraction. For example, the compositing operation may include multiplication or division, or may include color conversion processing such as compositing by changing the ratio of RGB color components.

Also, in the present embodiment, a case where there are four relighting modes, namely, the "oblique light correction mode", the "top light correction mode", the "backlight correction mode", and the "flat correction mode", has been described. However, relighting modes are not limited to these four modes. Any mode may be prepared as long as the composition parameter of the reflection characteristic map is determined according to the relighting mode.

Also, in this embodiment, the configuration for determining the relighting mode according to the luminance distribution of the subject has been described, but the relighting mode may be determined by other methods. For example, it is possible to adopt a configuration in which a relighting mode is determined (selected) according to a user instruction.

Further, in the present embodiment, the case of synthesizing two reflection characteristic maps has been described as an example, but any number of reflection characteristic maps may be synthesized as long as they are configured to synthesize a plurality of reflection characteristic maps. For example, it is possible to synthesize three reflection characteristic maps in order to reproduce three-lamp illumination.

[Second embodiment]
In the first embodiment, a configuration has been described in which the relighting mode is selected according to the condition of the subject during shooting, and the composition parameter of the reflection characteristic map is controlled according to the relighting mode. In the second embodiment, a configuration will be described in which relighting is performed using a virtual light source according to a user instruction when developing a RAW image once recorded on the recording medium 112 . In the second embodiment, the basic configuration of the digital camera 100 is the same as in the first embodiment (see FIGS. 1 to 3). Differences from the first embodiment will be mainly described below.

The digital camera 100 does not immediately develop the captured image captured by the imaging unit 103 by the image processing unit 105 , but temporarily records the RAW image on the recording medium 112 . After that, when an image is selected by the user's operation, the digital camera 100 reads the RAW image from the recording medium 112 and the image processing unit 105 performs image processing.

This embodiment differs from the first embodiment in the method of determining synthesis parameters for reflection characteristic maps set in the reflection characteristic map synthesis unit 301 of FIG. A method for determining synthesis parameters will be described with reference to FIG.

FIG. 10 is a flowchart of synthesis parameter setting processing executed by the system control unit 50 for the reflection characteristic map synthesis unit 301 according to the second embodiment. When the user selects an image to be developed (an image to be relighted), the processing of this flowchart starts.

In S1001, the system control unit 50 determines the irradiation angle of the virtual light source for relighting. The irradiation angle is determined according to user's operation. FIG. 11 shows an example of a user interface for user operation to install a virtual light source.

In FIG. 11, 1101 denotes a camera casing, 1102 denotes an operation section, and 1103 denotes a display section. An operation unit 1102 is included in the operation unit 120 in FIG. A display unit 1103 is included in the display unit 109 in FIG. When the user selects an image to be relighted, the system control unit 50 displays the selected image on the display unit 1103 . 1104 to 1106 indicate the positions of the set virtual light sources. The digital camera 100 performs relighting processing based on the position and direction of this virtual light source. The position of the virtual light source is determined by a user's touch operation. As shown at 1104-1106, the user can place multiple virtual light sources at arbitrary positions. When the position of the light source is determined, the system control unit 50 determines the irradiation angle so that the virtual light source is projected from the set position of the virtual light source toward the center of the main subject. A plurality of virtual light sources can be installed, and FIG. 11 shows a case where three virtual light sources 1104, 1105, and 1106 are installed. After determining the irradiation angle of the virtual light source, the system control unit 50 executes the processing of S1002 to S1007 for each of the plurality of installed virtual light sources.

In S<b>1002 , the system control unit 50 determines whether or not the reflection characteristic map corresponding to the irradiation angle is recorded in the nonvolatile memory 121 . In this embodiment, as in the first embodiment, seven types of reflection characteristic maps shown in FIG. 6 are recorded in the nonvolatile memory 121. FIG. In the example of FIG. 11, the virtual light source 1104 is in front of the subject, and "F0" in FIG. 6 corresponds to the irradiation angle. The virtual light source 1106 is positioned 90 degrees to the left of the object, and "L90" in FIG. 6 corresponds to the irradiation angle. In the example of virtual light sources 1104 and 1106, there is a corresponding reflection property map, so processing proceeds to S1003. On the other hand, the virtual light source 1105 illuminates from 75 degrees to the left of the subject. Since there is no reflection characteristic map corresponding to this irradiation angle, the process proceeds to S1004.

In S1003, the system control unit 50 selects a reflection characteristic map corresponding to the irradiation angle of the virtual light source. In the example of FIG. 11 , “F0” in FIG. 6 is selected for the virtual light source 1104 and “L90” in FIG. 6 is selected for the virtual light source 1106 .

In S1004, the system control unit 50 selects two reflection characteristic maps (one larger than the determined irradiation angle and one smaller than the determined irradiation angle) that are closest to the irradiation angle of the virtual light source. In the example of FIG. 11, the virtual light source 1105 has an irradiation angle of 75 degrees to the left, so two reflection characteristic maps of 90 degrees to the left (L90) and 60 degrees to the left (L60), which are close to 75 degrees to the left, are selected. .

In S1005, the system control unit 50 determines a synthesis ratio for synthesizing the two selected reflection characteristic maps. In the case of the virtual light source 1105, 75 degrees on the left side is an intermediate angle between 90 degrees on the left side and 60 degrees on the left side. Synthesize at a ratio of 1. That is, by synthesizing the reflection characteristic maps according to Equation (9), it is possible to approximately generate a reflection characteristic map irradiated from the left 75 degrees.
M = 0.5 x (L90) + 0.5 x (L60) (9)

In S<b>1006 , the system control unit 50 sets synthesis parameters (reflection characteristic map and synthesis ratio) in the reflection characteristic map synthesis unit 301 .

In S1007, the system control unit 50 determines whether processing of all virtual light sources has been completed. Since three virtual light sources are set in the example of FIG. 11, the system control unit 50 determines whether or not the processing of S1002 to S1006 has been completed for the three virtual light sources. If all virtual light sources have been processed, the process proceeds to S1008; otherwise, the process returns to S1002.

In S1008, the system control unit 50 sets the reflection property map synthesizing unit 301 to add and synthesize the reflection property maps of all the selected or generated virtual light sources. In the example of FIG. 11, the system control unit 50 sets the reflection property map synthesizing unit 301 so as to synthesize three reflection property maps corresponding to the virtual light sources 1104 to 1106 and generate one synthetic reflection property map. .

The synthesizing parameter setting processing executed by the system control unit 50 for the reflection characteristic map synthesizing unit 301 has been described above. After that, the reflection characteristic map synthesizing unit 301 performs synthesis processing of the reflection characteristic maps based on the set synthesis parameters. Subsequent processing is the same as in the first embodiment.

As described above, according to the second embodiment, the digital camera 100 determines the irradiation angle of the virtual light source according to the user's operation when performing the relighting process. Then, the digital camera 100 generates a reflection characteristic map corresponding to the determined irradiation angle by synthesizing reflection characteristic maps prepared in advance. This makes it possible to generate a reflection characteristic map for a desired irradiation angle with a relatively small processing load.

Note that in the present embodiment, the final reflection characteristic map is generated by synthesizing reflection characteristic maps stored in advance. However, it is also possible to employ a configuration in which a reflection characteristic map calculated from the shape of the subject acquired at the time of photographing is also used in addition to the previously stored reflection characteristic map. In this case, the digital camera 100 acquires the shape of the subject and generates a reflection characteristic map based on the shape. Further, the digital camera 100 may be configured to selectively use a previously stored reflection characteristic map and a reflection characteristic map based on the shape of the subject. Specifically, the digital camera 100 calculates a normal line from the distance information acquired from the ranging sensor 124, and generates a reflection characteristic map based on the normal line and the position, direction, reflectance, etc. of the virtual light source. The reflection characteristic map generated using the distance measuring sensor 124 has high accuracy, but it takes time to generate. Therefore, when the speed is important, such as a preview, the reflection characteristic maps prepared in advance are synthesized and used, and when the accuracy is important, such as the final image, the reflection characteristic map generated using the distance measuring sensor 124 is used. It is also possible to take the method of using.

In addition, there are cases where the distance information cannot be obtained because the subject is too far away, or where the accuracy of the normal line deteriorates due to reasons such as the face being too small, too dark, or too bright. In such a case, even if the accuracy is given priority, it is possible to switch to a previously prepared reflection characteristic map and use it.

[Third Embodiment]
The third embodiment will be described below with reference to FIGS. 12 and 13. FIG. In this embodiment, an example will be described in which a plurality of reflection characteristic maps are generated in the camera according to the subject and synthesized. Furthermore, in this embodiment, two maps having different characteristics, a diffuse reflection characteristic map and a specular reflection characteristic map, are generated and synthesized.

Since the overall configuration of the digital camera 100 in this embodiment is the same as that described in the first embodiment (FIGS. 1 and 2), description thereof will be omitted. This embodiment differs from the first and second embodiments in the configuration of the relighting processing unit 202 . Differences from the first and second embodiments will mainly be described below.

FIG. 12 shows the configuration of the relighting processing unit 202 in this embodiment. In FIG. 12, reference numeral 1201 denotes a normal line calculation unit that calculates a normal line, and 1202 denotes a reflection characteristic map generation unit that generates a reflection characteristic map of a virtual light source based on normal line information and direction information of the virtual light source. Reference numeral 1203 denotes a reflection characteristic map synthesizing unit that synthesizes a plurality of reflection characteristic maps, and 1204 denotes a virtual light source reflection component calculation unit that calculates the reflection component of the virtual light source based on the synthesized reflection characteristic map.

The operation of the rewriting processing unit 202 having the above configuration will be described. A normal line calculation unit 1201 calculates a normal line map from subject distance information acquired from the distance measuring sensor 124 (FIG. 1). The object distance information is two-dimensional distance information obtained for each pixel of the captured image. Any known technique may be used for the method of generating the normal map from the subject distance information.

The reflection characteristic map generating unit 1202 sets a virtual light source for the subject, and calculates a diffuse reflection characteristic map D and a specular reflection map S of the subject from the relationship between the direction of the normal and the virtual light source.

The diffuse reflection map D, as shown in FIG. 13, has N as the normal vector of the subject at the pixel of interest P in the subject, L as the directional vector of the virtual light source from the pixel of interest P, and the distance between the virtual light source and the subject (pixel of interest) as K is calculated by the following formula. However, vectors L and N are assumed to be normalized.
D _i =(L·N)/K ²

A plurality of virtual light sources that impart diffuse reflection can be installed, and Di indicates a diffuse reflection characteristic map corresponding to the i-th virtual light source. According to the calculation of the above formula, the closer the direction of the normal vector N and the direction of the virtual light source direction vector L is, and the closer the distance K between the subject and the virtual light source is, the larger the diffuse reflection component is.

As shown in FIG. 13, the specular reflection map S is calculated by the following formula, where R is the direction vector of the specular reflection, and V is the direction vector (direction of line of sight) from the subject position (pixel of interest P) to the camera. be. That is, the specular reflection component increases as the direction of the camera approaches the direction of the specular reflection.
S _j = (R·V) ^m

Any known proposed calculation method may be used to calculate the direction vector R of the specular reflection. Here, when the angle of incidence is a vector from the position of the virtual light source to the position of the object (pixel of interest P), the direction vector R of the specular reflection is the vector when the light is reflected on the reflecting surface at the same angle as the angle of incidence. . Also, m is a brilliance coefficient that indicates the degree of spread of specularly reflected light, and the larger this value, the sharper the specular reflection.

A plurality of virtual light sources that impart specular reflection can be installed, and Si indicates a specular reflection characteristic map corresponding to the j-th virtual light source.

Reflection characteristic map generation unit 1202 installs a plurality of virtual light sources that impart diffuse reflection and specular reflection based on subject evaluation values acquired by evaluation value generation unit 209 (FIG. 2). For example, as shown in FIG. 5A, in the case of oblique light with different brightnesses on the left and right sides of the subject, a virtual light source that irradiates the subject with light from the left and a virtual light source that irradiates the subject with light from the right are installed. When the subject has different brightnesses above and below as shown in FIG. 5B, a virtual light source that illuminates the subject from above and a virtual light source that illuminates the subject from below are installed.

As described above, the reflection characteristic map generation unit 1202 calculates the diffuse reflection map D and the specular reflection map S for the virtual light sources at a plurality of installed positions, and outputs them to the reflection characteristic map synthesizing unit 1203 .

Reflection characteristic map synthesizing unit 1203 performs addition/subtraction on a plurality of input diffuse reflection maps D and specular reflection maps S at a ratio corresponding to the shading state of the subject, based on the following formula, to obtain a final reflection characteristic map M: Generate.
M=Σ _i (α _i ×D _i )+Σ _j (β _j ×S _j )

Here, α and β are weights for addition and subtraction and can take negative values. By setting α or β to a negative value, it is possible to make a particular virtual light source the subtractive light.

For example, in the case of an oblique light scene where the subject is bright on the left side and dark on the right side, such as the subject shown in FIG. do. Also, the α value of the virtual light source that illuminates from the right side is increased to illuminate the dark area with strong diffused light. By controlling the composition ratio of a plurality of diffuse reflection maps and specular reflection maps in this way, a reflection characteristic map M corresponding to the state of the subject is generated. The generated reflection characteristic map M is output to the virtual light source reflection component calculator 1204 .

The virtual light source reflection component calculator 1204 multiplies the captured image (Rin, Gin, Bin) by the reflection characteristic map M. FIG. Thereby, the reflected color components (Ra, Ga, Ba) of the subject by the virtual light source are calculated. The formula is as follows.
Ra = M x Rin
Ra = M x Gin
Ra = M x Bin

The reflection components (Ra, Ga, Ba) by the virtual light source calculated as described above are output to the virtual light source addition processing unit 304 . Since the processing of the virtual light source addition processing unit 304 is the same as that of the first embodiment, the description is omitted here.

As described above, according to the third embodiment, the digital camera 100 generates reflection characteristic maps corresponding to a plurality of set light sources when performing relighting processing. Then, the digital camera 100 obtains a desired reflection characteristic map by weighting and synthesizing the plurality of generated reflection characteristic maps. As a result, high-speed processing becomes possible without calculating reflected color components corresponding to each of a plurality of virtual light sources.

In the above description, the diffuse reflection map D and the specular reflection map S are combined to calculate the final reflection characteristic map M. , to calculate the final reflected color component.

[Other embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

50 System control unit 100 Digital camera 105 Image processing unit 202 Relighting processing unit 301 Reflection characteristic map synthesis unit 302 Fitting processing unit 303 Virtual light source reflection component calculation unit 304 Virtual light source Additional processing part

Claims (13)

Before adding the effect of the virtual light source to the input image, first reflection characteristic data indicating the reflection characteristic of a first virtual light source in a predetermined subject included in the input image, and a second virtual light source in the predetermined subject. Synthesizing means for acquiring synthetic reflection characteristic data by performing synthesis processing for synthesizing second reflection characteristic data indicating reflection characteristics of a light source;
generating means for generating a reflection component of a subject included in the input image based on the reflection characteristic indicated by the synthetic reflection characteristic data obtained by the synthesis processing;
adding means for adding the reflection component generated by the generating means to the input image;
An image processing device comprising: 2. The image processing apparatus according to claim 1, wherein said synthesizing means selects said first reflection characteristic data and said second reflection characteristic data from a plurality of reflection characteristic data. Further comprising determining means for determining the correction mode,
3. The image processing apparatus according to claim 2, wherein said synthesizing means selects said first reflection characteristic data and said second reflection characteristic data based on said correction mode. 4. The image processing apparatus according to claim 3, wherein said determining means determines said correction mode based on the luminance of said subject included in said input image. 3. The image processing apparatus according to claim 2, wherein said synthesizing means selects said first reflection characteristic data and said second reflection characteristic data according to a user instruction. 6. The method according to any one of claims 1 to 5, wherein said synthesizing means multiplies said first reflection characteristic data by a first coefficient and multiplies said second reflection characteristic data by a second coefficient in said synthesizing process. The image processing device according to any one of items 1 and 2. 7. The image processing apparatus according to claim 6, wherein said synthesizing means determines said first coefficient and said second coefficient based on luminance of said subject included in said input image. further comprising correction means for correcting the synthetic reflection characteristic data so as to compensate for a difference in shape between the predetermined subject and the subject included in the input image;
The image processing apparatus according to any one of claims 1 to 7, wherein the generating means generates the reflection component based on the corrected combined reflection characteristic data. The synthetic reflection characteristic data includes a reflection characteristic image representing the reflection characteristic at each position of the predetermined subject,
9. The image according to claim 8, wherein said correction means geometrically transforms said reflection characteristic image so as to compensate for a difference in shape between said predetermined subject and said subject included in said input image. processing equipment. 10. The image processing apparatus according to claim 9, wherein said correction means further performs smoothing processing on said reflection characteristic image based on said input image. an image processing apparatus according to any one of claims 1 to 10;
imaging means for generating the input image;
An imaging device comprising: An image processing method executed by an image processing device,
Before adding the effect of the virtual light source to the input image, first reflection characteristic data indicating the reflection characteristic of a first virtual light source in a predetermined subject included in the input image, and a second virtual light source in the predetermined subject. a synthesizing step of acquiring synthetic reflection characteristic data by performing a synthesis process of synthesizing second reflection characteristic data indicating reflection characteristics of a light source;
a generation step of generating reflection components of a subject included in the input image based on the reflection characteristics indicated by the synthetic reflection characteristic data obtained by the synthesis process;
an adding step of adding the reflection component generated by the generating step to the input image;
An image processing method comprising: A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 10.

Images