JP2017147568A - Image processing apparatus, imaging apparatus, control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an image to which an appropriate lighting effect unique to backlight is added after imaging.SOLUTION: An occurrence rate of transmitted light by a virtual light source is determined on the basis of a shape of a subject region when the virtual light source that did not exist at a time of capturing the image is defined as a position that is in a backlight relationship with respect to the subject included in the image. An occurrence rate of reflected light by the virtual light source is determined on the basis of a position of the virtual light source and the position of the subject region. Then, on the basis of the generation rate of the transmitted light and the generation rate of the reflected light, the influence by the virtual light source is calculated, and an image added with the influence by the virtual light source is outputted.SELECTED DRAWING: Figure 8

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program, and more particularly to an image processing technique for adding an illumination effect by a virtual light source additionally defined after imaging to an image.

  There is a technique for adding a lighting effect by a virtual light source that does not exist in the shooting environment to an image obtained after shooting to reduce shadows generated on the subject. In the technique described in Patent Document 1, a face area in an image is detected, and brightness adjustment is performed using a virtual light source defined for the face area.

JP 2010-135996 A

  By the way, in the image processing for reducing the shadow by increasing the brightness as in Patent Document 1, the virtual light source is defined so as to have the relationship of the following light or the oblique light in the image capturing environment. On the other hand, the shooting environment is not always forward light or oblique light, and depending on the photographer, there are cases where shooting is performed with a desire for expression specific to backlight. By performing image processing as in Patent Document 1, a desired expression can be obtained. There was a possibility that it could not be obtained. That is, in the backlight scene, the illumination effect related to the transmitted light in which light from behind from around the subject passes through or the reflected light generated near the ridge line by hitting from behind the subject is not expressed.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide an image processing apparatus, an imaging apparatus, a control method, and a program that generate an image that preferably realizes expression specific to backlight after imaging. To do.

  In order to achieve the above-described object, an image processing apparatus according to the present invention includes an acquisition unit that acquires an image obtained by imaging, and a virtual object that does not exist at the time of imaging with respect to a subject included in the image acquired by the acquisition unit. A calculation unit that calculates the influence of the light source; and an output unit that outputs an image in which the influence of the virtual light source is added to the subject based on a calculation result of the calculation unit. The calculation unit determines a position where the virtual light source is defined. Determining means, specifying means for specifying the subject area included in the acquired image, and specifying position when the position for defining the virtual light source is determined by the determining means at a position that is in a backlight relationship Based on the shape of the subject area, the first determining means for determining the incidence rate of the transmitted light by the virtual light source, and based on the position of the virtual light source and the position of the subject area, Based on the second determining means for determining the generation rate of the reflected light by the light source, the generation rate of the transmitted light determined by the first determination unit, and the generation rate of the reflected light determined by the second determination unit And processing means for calculating the influence of the virtual light source.

  With such a configuration, according to the present invention, it is possible to generate an image that suitably realizes expression specific to backlight after imaging.

1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention. The block diagram which showed the detailed structure of the image process part 105 which concerns on embodiment of this invention. The block diagram which showed the detailed structure of the relighting process part 114 which concerns on embodiment of this invention. The figure for demonstrating the process which concerns on the determination of the contour distance weight based on embodiment of this invention. Another figure for demonstrating the process which concerns on the determination of the contour distance weight based on embodiment of this invention The figure for demonstrating the process which concerns on the determination of the light source angle weight which concerns on embodiment of this invention. The figure which illustrated the light source angle weight based on embodiment of this invention The figure which illustrated the total weight which concerns on embodiment of this invention The flowchart which illustrated the relighting control process performed with the digital camera 100 of embodiment of this invention. The figure for demonstrating the scene identification which concerns on embodiment of this invention The figure for demonstrating the process which concerns on the determination of an outline distance weight in consideration of the direction which an outline forms The figure for demonstrating the process which correct | amends the light source angle weight based on Embodiment 2 of this invention.

[Embodiment 1]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiment, an example in which the present invention is applied to a digital camera that can execute a relighting process that adds a lighting effect by a virtual light source as an example of an image processing apparatus will be described. However, the imaging function is not an essential configuration in the implementation of the present invention, and can be applied to any device capable of executing various processes for adding an illumination effect by an additional virtual light source to an image signal. Further, in this specification, “relighting processing” is to additionally define a virtual light source that did not actually exist in the shooting environment with respect to the image signal obtained by imaging, and to determine the lighting effect by the virtual light source. Description will be made assuming that the processing to be added is said.

<< Configuration of Digital Camera 100 >>
FIG. 1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention.

  The lens group 101 is an imaging optical system including a zoom lens and a focus lens, and guides a reflected light beam of a subject in a shooting environment and a light beam from a light source to the imaging unit 103. The light flux of the guided light beam is controlled by a shutter 102 having a diaphragm function, and converted to an analog image signal by an imaging unit 103. The imaging unit 103 is an imaging device such as a CCD or CMOS sensor, and photoelectrically converts an optical image formed on the imaging surface by the imaging unit 103 and outputs an analog image signal. The analog image signal output from the imaging unit 103 is converted into a digital image signal (image data) by the A / D converter 104, and the image processing unit 105 performs white balance processing, gamma correction processing, contour enhancement processing, and color correction processing. Various image processing such as these are applied. The face detection unit 113 executes a face detection process for detecting a human face area included in the input image data, and outputs a detection result. The relighting processing unit 114 applies relighting processing for adding a lighting effect by a virtual light source to the input image data. In various processing relating to images, the image memory 106 is provided so as to be usable as a work area and a data storage area, and the overall operation related to access such as writing / reading to the image memory 106 is controlled by the memory control unit 107. The

  Details of each of the image processing unit 105 and the relighting processing unit 114 will be described later with reference to another drawing. In the digital camera 100 of the present embodiment, various image processing is described as being configured by being separated into the image processing unit 105, the face detection unit 113, and the relighting processing unit 114. However, it will be readily understood that in the practice of the present invention, these image processes may be comprised of any number of processing circuits greater than one.

  Image data obtained by imaging and applied with various image processing and superimposition processing is converted by the D / A converter 108 to become an analog image signal for display. The analog image signal for display is displayed on the display unit 109 such as an LCD, for example, so that a function relating to the electronic viewfinder is provided to the photographer. When recording (storing) image data acquired in connection with a shooting operation, the codec unit 110 converts the data into recording data in a predetermined compression / encoding format, and the data is transmitted via the I / F 111. The data is recorded on a recording medium 112 that is a recording device such as a built-in memory or a memory card. When displaying the image data recorded on the recording medium 112, the codec unit 110 decodes the data.

  The system control unit 50 is an arithmetic device such as a CPU, for example, and controls the operation of each block included in the digital camera 100. More specifically, the system control unit 50 controls the operation of each block by reading the operation program for each block stored in the non-volatile memory 121 and developing it in the system memory 122 for execution. The nonvolatile memory 121 is a nonvolatile storage device such as an EEPROM. The nonvolatile memory 121 stores not only the operation program for each block but also various parameters necessary for the operation of each block. The system memory 122 may be a volatile memory configured to be rewritable. The system memory 122 not only serves as a development area for the operation program of each block but also temporarily stores intermediate data output in the operation of each block. Also used as a region.

  In addition, based on a predetermined evaluation value performed by the image processing unit 105 using a face detection result by the face detection unit 113 and image data obtained by imaging, the system control unit 50 performs exposure control and distance measurement control ( The lens group 101, the shutter 102, and the imaging unit 103 are controlled). As a result, TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and the like are realized.

  In addition, the digital camera 100 includes a strobe 123 that functions as an actual light source at the time of shooting, a distance measuring sensor 124 that measures the distance between the digital camera 100 and the subject, and an operation unit 120 that is a user interface that detects various operation inputs. Have. When the operation unit 120 detects that an operation input has been made, the operation unit 120 outputs a corresponding control signal to the system control unit 50.

  In the present embodiment, description will be made assuming that processing is realized by a circuit and a processor corresponding to each block included in the digital camera 100 as hardware. However, the implementation of the present invention is not limited to this, and the processing of each block may be realized by a program that performs the same processing as each block.

<Configuration of Image Processing Unit 105>
Next, the configuration of the image processing unit 105 will be described in detail with reference to the block diagram of FIG. In the digital camera 100 of the present embodiment, when an image is taken, the image processing unit 105 has a Bayer-type digital image in which each pixel indicates the signal level of any one of the R component, the G component, and the B component. Assume that a signal is input.

  The synchronization processing unit 200 performs synchronization processing on the input Bayer-format image signal, interpolates color components that do not indicate the signal level for each pixel of the image signal, and performs R component, G component, and B component. Image signals (RGB signals) are generated. Based on the white balance (WB) gain value determined by the system control unit 50, the WB amplification unit 201 amplifies the signal level for each color component of the RGB signal and adjusts the WB. Further, the luminance / color signal generation unit 202 generates a luminance signal (Y signal) from the RGB signal subjected to the WB adjustment and outputs the luminance signal (Y signal) to the contour enhancement processing unit 203. The luminance / color signal generation unit 202 outputs the input RGB signal to the color conversion processing unit 205.

  The contour enhancement processing unit 203 performs contour enhancement processing on the input Y signal and outputs the result to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction processing on the Y signal to which the contour enhancement processing is applied, and outputs the corrected Y signal to the image memory 106 via the memory control unit 107.

  On the other hand, the color conversion processing unit 205 applies a predetermined matrix operation to the input RGB signal and changes the signal level of each color component to a predetermined color balance. The color conversion processing unit 205 outputs the changed RGB signal to the color gamma processing unit 206 and the evaluation value generation unit 208. The color gamma processing unit 206 performs gamma correction processing on each color component of the RGB signal whose color balance has been changed, and outputs the result to the color difference signal generation unit 207. The color difference signal generation unit 207 generates color difference signals RY and BY based on the input RGB signals and outputs them to the image memory 106 via the memory control unit 107.

  In the digital camera 100 of the present embodiment, image data for recording is configured in the format of luminance / color difference signals Y, RY, and BY. That is, when an operation related to photographing is performed, the Y, RY, and BY signals stored in the image memory 106 are compression-coded by the codec unit 110 and recorded on the recording medium 112.

  Further, the evaluation value generation unit 208 generates an evaluation value used for estimating the state of the light source (environment light source) that actually exists in the shooting environment. In the later-described relighting control process performed by the digital camera 100 of the present embodiment, the evaluation value generation unit 208 estimates whether the scene is a backlight scene as the state of the environmental light source, so that the evaluation value generation unit 208 uses the luminance, hue, and saturation as evaluation values. Generate a distribution of values. More specifically, the evaluation value generation unit 208 divides the image area related to the input RGB signal into a plurality of blocks, and based on the luminance value, hue value, and saturation value of the pixels included in each block, A representative value for each such value is determined. The representative value may be various values such as a maximum value, an average value, a value of a pixel existing at a predetermined position in the block, and a total value. The evaluation value generation unit 208 outputs an evaluation value composed of representative value groups related to a plurality of blocks included in the image. The evaluation value information may be stored in the system memory 122, for example.

<Configuration of Relighting Processing Unit 114>
Next, the configuration of the relighting processing unit 114 will be described in detail with reference to the block diagram of FIG. In the digital camera 100 of the present embodiment, the relighting processing unit 114 performs the luminance / color difference signal generated by the image processing unit 105 when a predetermined mode is set to apply the relighting processing. It demonstrates as what performs a process. Alternatively, the relighting processing unit 114 may perform processing on the luminance / color difference signal read from the recording medium 112 and developed in the image memory 106.

The RGB signal conversion unit 301 applies a predetermined color conversion process to the input luminance / color difference signal, converts it to an RGB signal, and outputs the RGB signal to the degamma processing unit 302. The degamma processing unit 302 executes degamma processing for canceling the effect of gamma correction based on gamma characteristics related to gamma correction applied to the input RGB signal. At the time of imaging, the degamma processing unit 302 uses the inverse characteristic of the gamma characteristic related to the gamma correction applied by the color gamma processing unit 206 as the gamma characteristic, and converts the input RGB signal into a linear signal before gamma correction. Convert. The degamma processing unit 302 outputs the RGB signals (R _t , G _t , B _t ) after applying the degamma processing to the gain processing unit 303 and a virtual light source transmission / reflection component calculation unit 313 described later.

The gain processing unit 303 performs gain processing for amplifying the entire signal level of the image on the input RGB signal. The gain process may be performed, for example, to raise the signal level of the dark part regardless of the virtual light source to be defined. The gain processing unit 303 outputs the RGB signals (R _g , G _g , B _g ) after gain processing to the virtual light source addition processing unit 304.

The virtual light source addition processing unit 304 performs relighting processing that adds the influence of the virtual light source to the RGB signal to which the gain processing is applied in the gain processing unit 303. The influence of the virtual light source is performed by adding the additional RGB signals (R _a , G _a , B _a ) output by the virtual light source transmission / reflection component calculation unit 313.

  The virtual light source transmission / reflection component calculation unit 313 calculates, for each color component, the influence generated on the subject by arranging the virtual light source. The virtual light source transmission / reflection component calculation unit 313 according to the present embodiment calculates the influence that occurs when the virtual light source is arranged at a position that is particularly in a backlight relationship. Hereinafter, processing of each block performed in connection with the generation of the additional RGB signal in the virtual light source transmission / reflection component calculation unit 313 will be described mainly with respect to the processing when the virtual light source is arranged at the position where the backlight is related.

  The subject area extraction unit 310 identifies a subject area in an image related to RGB signals. In the present embodiment, the subject area extraction unit 310 will be described as specifying the person who is the main subject as the subject area (main subject area), but the subject area is not limited to a person, and may be various subjects. . Therefore, the subject region extraction unit 310 identifies the region related to the person who is the main subject in the image as the subject region based on the detection result (face position information) of the face detection unit 113 and the output (distance information) of the distance measuring sensor 124. To do.

  For example, a case will be described in which processing is performed on an image captured with a composition in which the main subject is arranged in the center as shown in FIG. At this time, since the position and size of the face area of the person in the image are specified by the face position information, the subject area extraction unit 310 first calculates the average distance (depth) for the subject included in the face area. The average distance related to the face area is calculated as the average value of the corresponding pixel values in the distance information. Then, the subject area extraction unit 310 identifies pixels included in a predetermined distance range as a subject area based on the average distance related to the face area. The predetermined distance range may be defined on the basis of a threshold value before and after the depth direction so as to extract a region corresponding to the human body based on the face region. The range is from the subtracted distance to the distance obtained by adding the threshold in the back direction to the average distance. The subject area extraction unit 310 grasps pixel positions included in a predetermined distance range based on the distance information, and identifies a subject area in the image. Note that in specifying the subject area based on the face area, pixels that do not belong to the area that includes the face area may be excluded from the subject area even if they are included in a predetermined distance range. In this way, the subject area extraction unit 310 can generate information (subject area information) indicating the identified subject area as a binary image signal as shown in FIG. 4B, for example. The generated subject area information is output to the contour distance weight calculation unit 311 and the light source angle weight calculation unit 312.

The contour distance weight calculation unit 311 determines a contour distance weight W _d that has the maximum weight at the end (contour) of the specified subject region and that decreases in weight according to the distance from the end in the region. To do. The contour distance weight is the probability that the light beam emitted from the virtual light source will exit in the direction of the digital camera 100 (front side) when there is a virtual light source in a backlight relationship with the subject (a virtual light source is arranged behind the main subject) ( The weight is determined according to the height of the transmission light generation rate. Specifically, the contour distance weight calculation unit 311 first applies a band-pass filter to the image signal that is the input subject area information, so that a high-frequency portion as shown in FIG. An extracted image signal (contour signal) indicating a contour portion is generated. Then, the contour distance weight calculation unit 311 applies a contour distance weight calculation filter as shown in FIG. 5A to the contour signal, and further performs an AND operation with the subject area information, thereby performing the contour distance. The image signal shown in FIG. 4D showing the weight is obtained. As shown in FIG. 4D, the contour distance weight indicates a weight (closer to white) that is closer to the contour in the subject area. In other words, the contour distance weight calculation unit 311 generates the contour distance weight so that the incidence rate of transmitted light decreases as the distance from the subject region boundary increases. The contour distance weight calculation unit 311 outputs information on the obtained contour distance weight W _d to the virtual light source transmission / reflection component calculation unit 313.

On the other hand, the light source angle weight calculation unit 312 determines _a light source angle weight W _a that indicates the generation degree (occurrence rate) of reflected light from the virtual light source (point light source) defined in the relighting process. The light source angle weight is a weight determined to indicate the degree of brightness change caused by reflection, for example, for the specified subject region, and is determined according to the incident angle of the light beam from the virtual light source (incident on the subject). The smaller the corner, the higher).

  Information (virtual light source information) on the position and attributes (forward / backlight, intensity, light source color) of the defined virtual light source is determined by the system control unit 50 and supplied to the relighting processing unit 114. The attribute relating to the forward / backlight of the virtual light source may be configured such that, for example, whether or not the relighting process is performed with respect to the virtual light source having the relationship between forward light and backlight by an operation input by the user may be selected. Also, it is not an aspect in which the user determines whether the virtual light source is in the following light or backlight, depending on the relationship between the position of the main subject in the distance information and the position of the virtual light source determined by the user's operation input The system control unit 50 may determine the attribute. In this case, the position of the virtual light source determined by the user may be a depth distance, for example. The system control unit 50 generates virtual light source information based on the information obtained by setting the virtual light source, and inputs the virtual light source information to the relighting processing unit 114. The position of the virtual light source may be configured such that the system control unit 50 determines a suitable position based on the state of the environment light source in the shooting environment, as will be described later.

  Here, an example of generation of light source angle weights will be described for both normal light and backlight. 6A and 6B show the positional relationship between the main subject 601 and the virtual light source 602 defined. When the virtual light source 602 is arranged in front of the main subject 601 in the depth direction, FIG. 6A shows the virtual light source 602 arranged behind the main subject 601 when forward light (oblique light) is shown in FIG. ) Indicates backlight. For the sake of simplicity, the light source angle weight calculation unit 312 of the present embodiment will be described assuming that the main subject 601 is a subject area-shaped flat plate that does not have a length (thickness) in the depth direction.

  In FIG. 6A, the virtual light source 602 is arranged in the left direction as viewed from the digital camera 100 (x is smaller than the digital camera 100). Therefore, the angle θ formed by the normal vector of the main subject 601 (the side toward the digital camera 100) and the incident vector of the light beam incident on the main subject 601 from the virtual light source 602 is the main subject (as viewed from the digital camera 100). 601 is different on the left and right. The light source angle weight may be calculated as the cosine cos θ of the angle θ formed because the (reflection) contribution by the virtual light source is based on the inner product of these vectors, and the weight distribution as shown in FIG. Become. The weight becomes larger as the formed angle θ is closer to a right angle.

  On the other hand, in FIG. 6B, the virtual light source 602 is arranged in the right direction (x is larger than the digital camera 100) when viewed from the digital camera 100. Accordingly, the angle θ formed between the normal vector of the main subject 601 (the side far from the digital camera 100) and the incident vector from the virtual light source 602 differs between the left and right of the main subject 601 (as viewed from the digital camera 100). Therefore, the light source angle weight has a weight distribution as shown in FIG. Similarly, the weight becomes larger as the angle θ formed is closer to the vertical.

Source angle weight calculator 312, the information of the thus obtained light angle weights W _a, and outputs to a virtual light source transmission-reflection component calculation unit 313.

The virtual light source transmission / reflection component calculation unit 313 adds RGB signals (R _a , G _a , B) indicating the influence of the virtual light source based on the input information on the contour distance weight W _{d and} the information on the light source angle weight W _{a. a} ) is generated. When the virtual light source is defined behind the main subject, the virtual light source transmission / reflection component calculation unit 313 is affected by the fact that the light beam from the light source is transmitted through the main subject or reflected by the main subject. Two types of weights are used to illustrate. First, the virtual light source transmission / reflection component calculation unit 313 multiplies the contour distance weight W _d and the light source angle weight W _a for each pixel to obtain a total weight W _t (= W _d ×) related to the transmission component and reflection component of the virtual light source. W _a ) is calculated. In the case where the virtual light source position where the relationship between the forward light is defined, for transmission component does not occur, the contour distance weight W _d is next 1 over the entire pixel, and W _t = W _a. For example, when a virtual light source is defined at a position having a backlight relationship as shown in FIG. 6B, the contour distance weight shown in FIG. 4D and the light source angle weight shown in FIG. 7B. Based on the above, the total weight as shown in FIG.

The virtual light source transmission / reflection component calculation unit 313 also adds the RGB signals (R _a , G _a , B _t ) from the RGB signals (R _t , G _t , B _t ) after applying the degamma processing based on the obtained total weight. _a ) Calculate each signal level related to. Each signal level related to the additional RGB signal can be calculated by the following formula using the intensity α (luminance, etc.) of the virtual light source and the control values of the light source color (red component R _w and blue component B _w ).
R _a = α × W _t × R _w × R _t
G _a = α × W _t × G _t
B _a = α × W _t × B _w × B _t

Accordingly, the additional RGB signal, which is the calculation result obtained in this way, and the RGB signal (R _g , G _g , B _g ) after gain processing are added, and the virtual light source addition processing unit 304 realizes relighting. The output RGB signal thus generated is generated. That is, each color component (R _out , G _out , B _out ) of the output RGB signal is R _out = R _a + R _g
G _out = G _a + G _g
B _out = B _a + B _g
Is obtained. For example, when the virtual light source is arranged at a position that is in a backlight relationship, the output RGB signal is obtained by adding the additional RGB signal shown in FIG. 8A to the image signal shown in FIG. As shown in (b), light transmission and reflection expression peculiar to backlighting in the vicinity of the boundary of the subject area is realized.

  The gamma processing unit 305 performs gamma correction processing on the output RGB signal generated by the virtual light source addition processing unit 304. The output RGB signal to which the gamma correction is applied is input to the luminance / color difference signal conversion unit 306, converted into a luminance / color difference signal, and output.

《Relighting control processing》
With respect to the relighting control process executed prior to the relighting process in the digital camera 100 of the present embodiment having such a configuration, a specific process will be described with reference to the flowchart of FIG. In the present embodiment, the relighting control process will be described as being performed before the process in order to determine the operation parameter of each block of the relighting processing unit 114 when executing the relighting process. However, the parameter setting timing is not limited to this, and the relighting control process may be executed in parallel with the relighting process. The processing corresponding to the flowchart can be realized by the system control unit 50 reading out a corresponding processing program stored in, for example, the non-volatile memory 121, developing the program in the system memory 122, and executing the program. This relighting control process will be described as being started, for example, when it is detected that a shooting instruction has been made in a state where a shooting mode for performing the relighting process is set.

  In step S <b> 901, the system control unit 50 estimates the state of the environment light source based on the evaluation value generated by the evaluation value generation unit 208, and the shooting environment in which the target image signal is captured includes a backlight scene and a backlight scene (backlight scene). Or any other scene). Specifically, the system control unit 50 calculates the illuminance based on the exposure information used for photographing and the luminance signal related to the image signal obtained by photographing. The system control unit 50 determines that the shooting environment is a backlight scene when the illuminance is higher than a predetermined threshold and the evaluation values obtained for the plurality of blocks related to the image signal have a predetermined distribution, When it does not correspond to this, it determines with it being a follow light scene.

  For example, as shown in FIG. 10A, when there are more than a predetermined number of blocks whose luminance values (EV values) exceed the predetermined threshold and are classified as high luminance in the upper part of the image, the system control unit 50 It is determined that the scene is a backlight scene. On the other hand, as shown in FIG. 10B, for example, when the number of blocks classified as high luminance is less than a predetermined number at the top of the image, it is determined that the scene is not a backlight scene. Further, the system control unit 50 identifies the direction in which the environment light source is radiating the subject from the brightness distribution. In the example of FIG. 10A, it is specified that the person who is the main subject is irradiated with the environmental light source from the back right direction when viewed from the digital camera 100. On the other hand, in the example of FIG. 10B, it is specified that the person who is the main subject is irradiated with the environmental light source from the left front side when viewed from the digital camera 100.

  In step S <b> 902, the system control unit 50 determines whether or not the relighting process to be performed is to add an illumination effect to a virtual light source having a backlight relationship. As described above, the determination in this step is performed based on the contents and settings of the operation input related to the virtual light source to which the lighting effect is added in the relighting process. If the system control unit 50 determines that the relighting process is to add an illumination effect for the virtual light source having a backlight relationship, the system control unit 50 moves the process to S903. If the system control unit 50 determines that the relighting process is to add an illumination effect for the virtual light source having the following light relationship, the system control unit 50 proceeds to S906.

  In step S903, the system control unit 50 determines whether the shooting environment in which the target image signal is shot is a backlight scene based on the scene determination result in step S901. If the system control unit 50 determines that the shooting environment is a backlight scene, it moves the process to S904, and if it determines that the shooting environment is not a backlight scene, it moves the process to S905.

In step S904, the system control unit 50 determines a position where the virtual light source is defined based on the irradiation direction of the environmental light source estimated in step S901. In the present embodiment, the position defining the virtual light source is determined so that the direction from the virtual light source toward the main subject is the same as the irradiation direction of the environment light source or has an equivalent direction component. Further, the system control unit 50 determines parameters related to the intensity (α) and the light source color (R _w , B _w ) related to the virtual light source used in the virtual light source transmission / reflection component calculation unit 313. The light source intensity may be determined based on, for example, an operation input made by the user, or may be determined according to the environmental light source, such as increasing the intensity as the environmental light source is brighter. Further, since the light source color is applied in a backlight scene, it is determined so that the relationship of R _w > B _w is realized so that the color component is closer to red. The system control unit 50 configures virtual light source information based on the virtual light source position and various attribute information determined as described above, and outputs the virtual light source information to the relighting processing unit 114 to complete the relighting control process.

  On the other hand, if it is determined in step S903 that the scene is not a backlight scene, the system control unit 50 notifies the error notifying that the relighting process related to the set position of the virtual light source is not executed in step S905, thereby completing the relighting control process. This is because an unnatural expression is obtained by adding an illumination effect by a virtual light source having a backlight relationship to an image that is not a backlight scene. The error notification may be performed via the display unit 109. Further, the system control unit 50 controls the operation so that the relighting processing unit 114 does not perform the relighting processing on the image signal obtained by photographing.

If it is determined in step S902 that the relighting process is to add an illumination effect for the virtual light source that has the relationship of direct light, the system control unit 50 determines a position in which the virtual light source is defined in step S906. The position that defines the virtual light source determined in this step is determined based on the estimated irradiation direction of the environmental light source, as in S903. More specifically, the system control unit 50 determines the position of the virtual light source so that the direction from the virtual light source toward the main subject is the same as the irradiation direction of the environment light source or has an equivalent direction component. Further, the system control unit 50 determines parameters related to the intensity and the light source color related to the virtual light source used in the virtual light source transmission / reflection component calculation unit 313 as in S903. Then, the system control unit 50 configures virtual light source information based on the virtual light source position and various attribute information determined as described above, supplies the virtual light source information to the relighting processing unit 114, and completes the relighting control process. Note that the system control unit 50 does not need to express the transmission component of the light source when adding the illumination effect by the virtual light source having the relationship of the follow light, so that the contour distance weight W _d becomes 1 for all the pixels. The operation of the contour distance weight calculation unit 311 is also controlled.

  By doing in this way, in the digital camera 100 of this embodiment, a suitable relighting result can be obtained even when the lighting effect by the virtual light source having a backlight relationship is added in executing the relighting process.

  In this embodiment, the example of using the 7 × 7 filter as shown in FIG. 5A in order to calculate the contour distance weight has been described, but the embodiment of the present invention is not limited to this. Any method may be used as long as weighting is performed according to the distance from the contour. For example, a filter with a large number of taps of 9 × 9 or more may be used to express the characteristics of light transmission and wraparound to the inside of the subject. Alternatively, for example, a method may be used in which a contour pixel closest to the processing target pixel is specified and a weight is determined according to the distance between the contour pixel and the processing target.

  In addition, in determining the light source angle weight, the direction (normal direction or tangent direction) formed by the partial contour may be detected and further weighted based on the direction. The direction in which the contour is formed may be detected using any conventional method. For example, as shown in FIG. 11, for a processing target pixel 1101 whose weight is to be calculated, a normal vector 1103 of a partial contour including the pixel and a virtual light source vector 1104 from the virtual light source 1102 toward the pixel form θ. In this case, the weight of the pixel is changed according to θ. Specifically, the light source angle weight calculation unit 312 changes the signal level by multiplying the pixel extracted as the contour in the contour signal by the inner product (cos θ) of the normal vector and the virtual light source vector related to the pixel, The light source angle weight is calculated based on the changed contour signal. By detecting the direction in which the contour is formed in units of 45 degrees, the light source angle weight is determined by such a method so that the illumination effect by the transmitted light appears only in the contour that forms the direction facing the virtual light source. Can do.

  In the present embodiment, when relighting related to a virtual light source having a backlight relationship is described, only the illumination effect of transmitted light and reflected light in the contour is expressed. The virtual light source may be defined to change the lighting effect. In this way, it is also possible to express the sneak light from the environment.

[Modification]
In the embodiment described above, the contour distance weight W _d related to the transmitted light around the contour is determined so as to emphasize the person who is the main subject. However, the implementation of the present invention is not limited to this. For example, the contour distance weight W _d may be determined in consideration of the type, material, and reflection characteristics of the subject to which the illumination effect is added. Further, hair even a person, the skin, and recognized separated according to the site or materials such as clothes, may have different processing according to the determination of the contour distance weight W _d for each site. At this time, for example, in the case of hair, the contour distance weight W _d may be determined using the filter shown in FIG. 5A so that transmitted light is emphasized. On the other hand, for other human bodies, for example, a filter as shown in FIG. 5B may be used so that the reflected light is more emphasized than the transmitted light, and the weight closer to the contour may be increased.

[Embodiment 2]
In the first embodiment, for the sake of simplicity, the light source angle weight W _a is determined on the assumption that the subject area is a flat plate. However, the normal of each pixel determined based on the estimation of the three-dimensional shape of the subject is considered. Also good. For example, consider a case where the normal vector based on the shape estimated for the subject region 1201 shows a distribution as shown in FIG. At this time, the light source angle weight calculation unit 312 considers the inner product of the virtual light source vector 1203 from the virtual light source 1202 to each pixel of the subject region and the normal vector of the pixel, and 1-cos θ is set as the light source angle weight related to the pixel. Multiply W _a . Such a comprehensive weight W _t by, a contour of the object can be more virtual light source vector 1203 against rewriting target object is vertical, constituting so that the reflected light becomes stronger.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  100: Digital camera, 103: Imaging unit, 105: Image processing unit, 113: Face detection unit, 114: Relighting processing unit, 50: System control unit, 120: Operation unit, 124: Distance sensor, 208: Evaluation value generation Unit 304: virtual light source addition processing unit 310: subject region extraction unit 311: contour distance weight calculation unit 312: light source angle weight calculation unit 313: virtual light source transmission / reflection component calculation unit

Claims (10)

Acquisition means for acquiring an image obtained by imaging;
A computing means for computing the influence of a virtual light source that did not exist at the time of imaging for a subject included in the image obtained by the obtaining means;
Output means for outputting an image in which the influence of the virtual light source is added to the subject based on a calculation result by the calculation means;
The computing means is
Determining means for determining a position defining the virtual light source;
Specifying means for specifying a subject area included in the acquired image;
When the position that defines the virtual light source is determined at a position that is in a backlight relationship by the determining means,
First determining means for determining the incidence rate of transmitted light by the virtual light source based on the shape of the subject area specified by the specifying means;
Second determining means for determining a rate of occurrence of reflected light by the virtual light source based on a position of the virtual light source and a position of the subject region;
A processing unit that calculates the influence of the virtual light source based on the generation rate of the transmitted light determined by the first determination unit and the generation rate of the reflected light determined by the second determination unit;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the generation rate of the transmitted light is determined so as to decrease as the distance from the boundary of the subject region increases.   The image processing apparatus according to claim 1, wherein the generation rate of the transmitted light is determined in consideration of at least one of a type, a material, and a reflection characteristic of a subject corresponding to the subject region.   The incidence rate of the reflected light is determined so as to increase as the incident angle of light from the position defining the virtual light source to the subject region is closer to a right angle. The image processing apparatus according to item.   The image processing apparatus according to claim 1, wherein the incidence rate of the reflected light is determined according to an inclination at a boundary of the subject area. The calculation means further includes estimation means for estimating a three-dimensional shape of a subject corresponding to the subject area,
The incidence rate of the reflected light is determined based on a vector from the position defining the virtual light source toward the pixel and a normal vector of a three-dimensional shape estimated for the pixel for the pixel in the subject area. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A judgment means for judging whether or not the environment in which the acquired image is captured is a backlight scene;
The computing means performs computations related to the first determining means, the second determining means, and the processing means when the determining means determines that the captured environment is a backlight scene. The image processing apparatus according to claim 1, wherein: Imaging means for generating an image by imaging;
An image processing apparatus comprising: the image processing apparatus according to claim 1, which acquires an image generated by the imaging unit and outputs an image to which an influence of a virtual light source is added. . An acquisition step of acquiring an image obtained by imaging;
A calculation step of calculating the influence of the virtual light source that did not exist at the time of imaging for the subject included in the image acquired in the acquisition step;
An output step of outputting an image in which the influence of the virtual light source is added to the subject based on the calculation result in the calculation step;
The calculation step includes
A determining step for determining a position defining the virtual light source;
A specifying step of specifying a subject area included in the acquired image;
When the position that defines the virtual light source is determined at a position that is in a backlight relationship in the determination step,
A first determination step of determining a generation rate of transmitted light by the virtual light source based on the shape of the subject region specified in the specification step;
A second determining step of determining an incidence rate of reflected light by the virtual light source based on a position of the virtual light source and a position of the subject region;
A processing step of calculating the influence of the virtual light source based on the generation rate of the transmitted light determined in the first determination step and the generation rate of the reflected light determined in the second determination step;
A control method for an image processing apparatus.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 7.