JP2018185576A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem that effect of relighting processing is changed, when setting a parameter of the relighting processing for each frame image, in a moving picture photographing a moving subject, and the moving picture after the relighting processing is difficult for a user to see.SOLUTION: An image processing device performs image processing which gives relighting effect by a virtual light source to a plurality of frame images constituting a moving picture. The image processing device comprises: processing means which performs the image processing for giving the relighting effect to the frame image; and setting means which sets a parameter used for the image processing by the processing means. The setting means switches fixedly setting of the parameter, or variably setting of the parameter for each frame image, to the frame image of a predetermined period continuous in the plurality of frame images.SELECTED DRAWING: Figure 1

Description

  The present invention relates to image processing for providing a relighting effect by a virtual light source to a plurality of frame images constituting a moving image.

  Conventionally, a virtual light source (hereinafter referred to as a virtual light source) that has not existed at the time of photographing is set in a photographed image, and an image that gives an effect as if light is emitted from the virtual light source to a subject in the image Processing is known. Such image processing is called relighting processing, and it is possible to brighten dark areas such as shadows caused by ambient light at the time of shooting.

  Patent Document 1 below discloses a method of performing relighting processing in units of image areas in accordance with information on face areas in the image. Specifically, the relighting process is performed on the face area of the image based on the information about the face area such as the presence / absence and position of the face in the image and the scene discrimination result of the image.

JP 2007-148537 A

  When correcting the shadow by performing the relighting process on the image, it is necessary to appropriately set parameters used for the relighting process such as the position, intensity, and angle of the virtual light source according to the state of the subject in the image.

  The method of Patent Document 1 targets still images and does not consider moving images. Therefore, when the method of Patent Document 1 is applied to a moving image obtained by shooting a moving subject, the parameters of the relighting process are set for each frame image, so that the effect of the relighting process changes one after another. There is a problem that it is difficult for viewers to see the video.

  Therefore, the present invention provides image processing for appropriately giving a relighting effect by a virtual light source to a moving image.

  One embodiment of the present invention is an image processing apparatus that performs image processing for imparting a relighting effect by a virtual light source to a plurality of frame images constituting a moving image, and performs image processing for imparting a relighting effect to the frame image. Processing means, and setting means for setting parameters used for image processing by the processing means, wherein the setting means fixes the parameters for a series of frame images in the plurality of frame images, An image processing apparatus is characterized in that the parameter can be changed for each frame image.

  According to the present invention, it is possible to perform appropriate relighting processing on a moving image.

It is a block diagram which shows the structure of the digital camera in this invention. It is a block diagram which shows the structure of the image process part in this invention. It is a block diagram which shows the structure of the relighting process part in this invention. It is a figure explaining the reflection with respect to irradiation by the virtual light source in this invention It is a figure explaining the image before and after the relighting process in this invention. It is a flowchart which determines the virtual light source parameter in this invention. It is a table | surface which shows a response | compatibility with the virtual light source parameter and tracking in this invention. It is an image figure of the to-be-photographed object and virtual light source in this invention. It is a flowchart which determines the virtual light source parameter in this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Although the present invention is described using an embodiment in which the present invention is applied to a digital camera, the present invention is not limited to a digital camera and can be applied to other image processing apparatuses.

(First embodiment)
A first embodiment of the present invention will be described with reference to FIGS.

  FIG. 1 is a block diagram illustrating a configuration example of a digital camera according to the first embodiment.

  In FIG. 1, reference numeral 100 denotes an entire digital camera, 101 denotes a lens group including a zoom lens and a focus lens, 102 denotes a shutter having a diaphragm function, and 103 denotes an image pickup composed of a CCD or a CMOS element that converts an optical image into an electric signal. , 104 is an A / D converter that converts an analog signal into a digital signal, and 105 is white balance processing, γ processing, contour enhancement, and color correction processing for image data output from the A / D converter 104. It is an image processing unit that performs various image processing such as.

  Reference numeral 106 denotes an image memory, 107 denotes a memory control unit that controls the image memory 106, 108 denotes a D / A converter that converts an input digital signal into an analog signal, 109 denotes a display such as an LCD, and 110 denotes compression of image data. It is a codec unit for encoding / decoding.

  111 is an interface I / F with the recording medium 112, 112 is a recording medium such as a memory card or a hard disk, 113 is a subject information acquisition unit that detects subject information such as the position of the subject and the orientation of the face from the captured image, and 114 A relighting processing unit 50 that performs relighting processing on a captured image, and a system control unit 50 that controls the entire system of the digital camera 100.

  Reference numeral 121 denotes a nonvolatile memory such as an EEPROM for storing programs and parameters. Reference numeral 122 denotes a system memory for developing constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 124, and the like. is there. A distance measuring sensor 123 measures the distance to the subject and outputs distance information corresponding to the pixel unit of the photographic pixel as a two-dimensional distance map image.

  Next, a basic operation at the time of photographing an object in the digital camera 100 configured as described above will be described. The imaging unit 103 photoelectrically converts light that has entered through the lens 101 and the shutter 102 and outputs it to the A / D converter 104 as an input image signal. The A / D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal and outputs the digital image signal to the image processing unit 105.

  The image processing unit 105 performs color conversion processing such as white balance, γ processing, and contour enhancement processing on the image data from the A / D converter 104 or the image data from the memory control unit 107. Further, the image processing unit 105 performs a predetermined evaluation value calculation process (not shown) using the result of the subject information acquired by the subject information acquisition unit 113 and the captured image data, and based on the obtained evaluation value result. Then, the system control unit 50 performs exposure control and distance measurement control. Thus, TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and the like are performed.

  The image data output from the image processing unit 105 is written into the image memory 106 via the memory control unit 107. The image memory 106 stores image data output from the imaging unit 103 and image data to be displayed on the display unit 109.

  The D / A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109. The display unit 109 performs display according to an analog signal from the D / A converter 108 on a display such as an LCD.

  The codec unit 110 compresses and encodes the image data recorded in the image memory 106 based on a standard such as MPEG. The system control unit 50 associates the encoded image data and stores them in the recording medium 112 via the recording interface 111.

  In addition to the above basic operations, the system control unit 50 executes the programs recorded in the nonvolatile memory 124 described above, thereby realizing each process described later. The program here is a program for executing various flowcharts described later. At this time, constants, variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like are expanded in the system memory 122.

  Next, details of the image processing unit 105 will be described with reference to FIG. FIG. 2 is a block diagram illustrating a configuration of the image processing unit 105.

  In FIG. 2, reference numeral 200 denotes a synchronization processing unit, 201 denotes a WB amplification unit, 202 denotes a luminance / color signal generation unit, 203 denotes a contour enhancement processing unit, 204 denotes a luminance gamma processing unit, 205 denotes a color conversion processing unit, and 206 denotes a color. A γ processing unit, 207 is a color difference signal generation unit, and 208 is a shadow information acquisition unit.

  Next, processing in the image processing unit 105 will be described. The image signal input from the A / D conversion unit 104 in FIG. 1 is input to the image processing unit 105.

  The image signal input to the image processing unit 105 is input to the synchronization processing unit 200. The synchronization processing unit 200 performs synchronization processing on the input Bayer RGB image data to generate color signals R, G, and B. Based on the white balance gain value calculated by the system control unit 50, the WB amplification unit 201 applies a gain to the RGB color signals and adjusts the white balance. The RGB signal output from the WB amplification unit 201 is input to the luminance / color signal generation unit 202. The luminance / color signal generation unit 202 generates a luminance signal Y from the RGB signals, and outputs the generated luminance signal Y to the contour enhancement processing unit 203 and the color signal RGB to the color conversion processing unit 205.

  The contour enhancement processing unit 203 performs contour enhancement processing on the luminance signal and outputs the luminance signal to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y and outputs the luminance signal Y to the image memory 106.

  The color conversion processing unit 205 performs conversion to a desired color balance by performing a matrix operation on the RGB signals. The color gamma processing unit 206 performs gamma correction on the RGB color signals. The color difference signal generation unit 207 generates color difference signals RY and BY signals from the RGB signals.

  The image signal (Y, RY, BY) signal output to the image memory 106 is compression encoded by the codec unit 110 and recorded on the recording medium 112.

  The output RGB signal of the color conversion processing unit 205 is also input to the shadow information acquisition unit 208. The shadow information acquisition unit 208 acquires information for analyzing the state of the shadow generated on the subject by the environmental light source. For example, average luminance information of the subject, luminance histogram information of the face area, and the like are acquired as shadow information.

  Next, the configuration and operation of the relighting processing unit 114 will be described with reference to FIG.

  For example, when the relighting process is selected by a user operation, the data output from the image processing unit 105 is input to the relighting process unit 114, and the relighting process using a virtual light source is performed.

  FIG. 3 is a block diagram showing a configuration of the relighting processing unit 114.

  In FIG. 3, reference numeral 301 denotes an RGB signal conversion unit that converts input luminance / color difference signals (Y, BY, RY) into RGB signals, and 302 denotes a degamma processing unit that performs degamma processing. Reference numeral 303 denotes a distance calculation unit that acquires distance information between the imaging device output from the distance measuring sensor 123 and the subject, 304 denotes a normal calculation unit that calculates the normal of the subject, and 305 denotes a virtual light source that reflects the subject. A virtual light source reflection component calculation unit 306 that calculates the calculated component is a virtual light source addition processing unit that adds a relighting effect by the virtual light source.

  Reference numeral 307 denotes a gamma processing unit that applies gamma characteristics to the RGB signal, and reference numeral 308 denotes a luminance / color difference signal conversion unit that converts the RGB signal into luminance / color difference signals (Y, BY, RY).

  The operation of the relighting processing unit 114 configured as described above will be described.

  The relighting processing unit 114 reads out luminance / color difference signals (Y, BY, RY) from the image data recorded in the image memory 106 and inputs them. The RGB signal conversion unit 301 converts the input luminance / color difference signals (Y, BY, RY) into RGB signals and outputs them to the degamma processing unit 302.

  The de-gamma processing unit 302 calculates a characteristic opposite to the gamma characteristic multiplied by the gamma processing unit of the image processing unit 105 and converts it to linear data. The degamma processing unit 302 outputs the RGB signals (Rt, Gt, Bt) after linear conversion to the virtual light source reflection component calculation unit 305 and the virtual light source addition processing unit 306.

  On the other hand, the distance calculation unit 303 calculates a distance map from the subject distance information acquired from the distance measuring sensor 123. The subject distance information is two-dimensional distance information obtained for each pixel of the captured image. The normal calculation unit 304 calculates a normal map from the distance information acquired from the distance calculation unit 303. As a method for generating a normal map from a distance map, a known technique is used. A specific processing example will be described with reference to FIG.

  FIG. 4 is a diagram illustrating the relationship between camera shooting coordinates and a subject. For example, as shown in FIG. 4, for a subject 401, gradient information is calculated from a difference ΔD of a distance (depth) D with respect to a difference ΔH in the horizontal direction of a captured image, and a normal N is calculated from the gradient information. Is possible. By performing the above processing on each pixel, it is possible to calculate normal information N corresponding to each pixel of the captured image. The normal calculation unit 304 outputs normal information corresponding to each pixel of the captured image to the virtual light source reflection component calculation unit 305 as a normal map.

  The virtual light source reflection component calculation unit 305 calculates a component that the virtual light source reflects to the subject based on the distance K between the light source and the subject, the normal information N, and the virtual light source parameter L.

  Specifically, the reflection component at the coordinate position corresponding to the captured image is calculated so as to be inversely proportional to the square of the distance K between the light source and the subject and proportional to the inner product of the subject normal vector N and the light source direction vector L.

  This will be described with reference to FIG. In FIG. 4, 401 indicates the subject, and 402 indicates the position of the set virtual light source. The reflection component at the horizontal pixel position H1 (the vertical pixel position is omitted for the sake of simplicity) of the captured image captured by the digital camera 100 is proportional to the inner product of the normal N1 at the camera coordinate H1 and the direction vector L1 of the virtual light source. The value is inversely proportional to the distance K1 between the virtual light source and the subject position.

When this relationship is expressed by a mathematical expression, the subject reflection components (Ra, Ga, Ba) by the virtual light source are as follows (Formula 1).
Ra = α × (−L · N) / K ² × Rt
Ga = α × (−L · N) / K ² × Gt (Formula 1)
Ba = α × (−L · N) / K ² × Bt

  Here, α is the strength of the virtual light source and the gain value of the relighting correction amount. L is the three-dimensional direction vector of the virtual light source, N is the three-dimensional normal vector of the subject, and K is the distance between the virtual light source and the subject. Rt, Gt, and Bt are photographing RGB data output from the degamma processing unit 302.

The reflection components (Ra, Ga, Ba) by the virtual light source calculated as described above are output to the virtual light source addition processing unit 306. The virtual light source addition processing unit 306 performs processing according to the following (Equation 2) for adding virtual light source components (Ra, Ga, Ba).
Rout = Rt + Ra
Gout = Gt + Ga (Formula 2)
Bout = Bt + Ba

  The image signals (Rout, Gout, Bout) output from the virtual light source addition processing unit 306 are input to the gamma processing unit 307. The gamma processing unit 307 performs gamma correction on the RGB input signals. The color difference signal generation unit 308 generates luminance Y, color difference signals RY, and BY signals from the RGB signals.

  The above is the operation of the relighting processing unit 114. An example of the relighting process performed by the relighting processing unit 114 is shown in FIG. FIG. 5A shows a photographed image before the relighting process, and FIG. 5B shows a photographed image after the relighting process. The dark subject in FIG. 5A is brightly corrected as shown in FIG. 5B by applying relighting processing to the virtual light source.

  The system control unit 50 accumulates the luminance / color difference signal output from the relighting correction unit 114 in the image memory 106 under the control of the memory control unit 107, and then performs compression encoding in the codec unit 110. In addition, recording is performed on the recording medium 112 via the I / F 111.

  Next, a processing flow in which the system control unit 50 determines the parameters of the virtual light source in the relighting processing unit 114 based on the subject information acquired by the subject information acquisition unit 113 will be described below.

  Here, the subject information acquisition unit 113 includes a main subject detection unit that detects a main subject, a face organ detection unit that detects a facial organ of the main subject detected by the main subject detection unit, and a main subject detection unit that detects the main subject. It consists of a tracking unit that tracks the subject. Known techniques are used for the main subject detection method, face organ detection method, and tracking method. For example, as a main subject detection method, there is a method of estimating the position of a subject from feature points in the screen and determining a region having the largest feature amount as a main subject. For example, as a face organ detection method, there is a method of detecting a plurality of parts such as a face outline and eyes from an edge and detecting a face direction based on the positional relationship. For example, as a tracking method, a method of obtaining a movement vector by comparing different images in the time direction, or a method of making a subject or a part thereof a characteristic color pattern and tracking an image taken at different times by characteristic color pattern matching and so on.

  FIG. 6 is a flowchart showing the operation of the system control unit 50. Hereinafter, the operation of the relighting processing unit 114 for determining the parameters of the virtual light source based on the subject information acquired by the subject information acquisition unit 113 will be described with reference to the flowchart of FIG.

  In step S601, it is determined whether relighting processing is selected by an operation from the user via the operation unit 120. That is, it is determined whether or not processing by the relighting processing unit 114 is performed. If rewriting is performed, the process proceeds to step 602.

  In step S602, the subject is photographed.

  In step S603, the subject information acquisition unit 113 determines whether the subject can be tracked in each frame. If tracking has succeeded, the process proceeds to step S604, and if tracking has failed, the process proceeds to step S605.

  Specifically, the subject information shown in the correspondence table between the frame number (frame No.) and the tracking result shown in FIG. In FIG. 7A, reference numeral 701 denotes a frame number. Reference numeral 702 denotes a flag indicating whether or not tracking is possible.

  The flag is set to 1 when tracking is successful, and the flag is set to 0 when tracking fails. Reference numeral 703 denotes the tracking result, that is, the coordinates of the face position of the main subject. In accordance with the flag indicating whether or not the tracking indicated by 702 in FIG. 7A is successful, the process proceeds to step S604 if the flag is 1, and proceeds to step S605 if the flag is 0.

  In step S604, if tracking is successful in step S603, the relighting processing unit 114 determines the parameters of the virtual light source based on the parameter setting table of the virtual light source when tracking is OK. Specifically, it is determined based on the virtual light source setting table at the time of tracking OK shown in FIG. In FIG. 7B, 704 indicates a mode, and 705 indicates a virtual light source setting. When the mode is 0, control is performed so that the position of the virtual light source is not changed for each frame.

For example, when the frame number is 3, the tracking flag is 1, and the parameters of the virtual light source are controlled so as to be the same position as the virtual light source at the time of frame number 2 one frame before.
When the mode is 1, control is performed to change the position of the virtual light source for each frame in accordance with the tracking result. For example, when the frame number is 2, the virtual light source parameters such as the position, angle, and intensity of the virtual light source are determined so that the virtual light source hits the face position (x ₂ , y ₂ ) of the main subject, and the frame number is 3 In this case, parameters such as the position, angle, and intensity of the virtual light source are determined so that the virtual light source hits the face position (x ₃ , y ₃ ) of the main subject. That is, the subject reflection components (Ra (x ₂ , y ₂ ), Ga (x ₂ , y ₂ ), Ba (x ₂ , y ₂ )) by the virtual light source when the frame number is 2 are expressed by the following formulas (formula 3 ). It should be noted that the setting of mode 0 and mode 1 may be determined by an instruction operation from the user, or may be determined automatically according to the state of the subject by analyzing the image.
Ra (x ₂ , y ₂ ) =
α × (−L (x ₂ , y ₂ ) · N (x ₂ , y ₂ )) / K (x ₂ , y ₂ ) ² × Rt
Ga (x ₂ , y ₂ ) =
α × (−L (x ₂ , y ₂ ) · N (x ₂ , y ₂ )) / K (x ₂ , y ₂ ) ² × Gt
Ba (x ₂ , y ₂ ) =
α × (−L (x ₂ , y ₂ ) · N (x ₂ , y ₂ )) / K (x ₂ , y ₂ ) ² × Bt
(Formula 3)

L (x ₂ , y ₂ ) is the three-dimensional direction vector of the virtual light source, N (x ₂ , y ₂ ) is the three-dimensional normal vector of the subject at the position (x ₂ , y ₂ ), and K (x ₂ , y ₂ ) indicates the distance between the virtual light source and the subject at the position (x ₂ , y ₂ ). In order to hit a virtual light source, it is only necessary to satisfy the condition that (Ra (x ₂ , y ₂ ), Ga (x ₂ , y ₂ ), Ba (x ₂ , y ₂ )) is a positive value.

Similarly, the subject reflection components (Ra (x ₃ , y ₃ ), Ga (x ₃ , y ₃ ), Ba (x ₃ , y ₃ )) by the virtual light source when the frame number is 3 are expressed by the following equations (formulas) 4).
Ra (x ₃ , y ₃ ) =
α × (−L (x ₃ , y ₃ ) · N (x ₃ , y ₃ )) / K (x ₃ , y ₃ ) ² × Rt
Ga (x ₃ , y ₃ ) =
α × (−L (x ₃ , y ₃ ) · N (x ₃ , y ₃ )) / K (x ₃ , y ₃ ) ² × Gt
Ba (x ₃ , y ₃ ) =
α × (−L (x ₃ , y ₃ ) · N (x ₃ , y ₃ )) / K (x ₃ , y ₃ ) ² × Bt
(Formula 4)

L (x ₃ , y ₃ ) is the three-dimensional direction vector of the virtual light source, N (x ₃ , y ₃ ) is the three-dimensional normal vector of the subject at the position (x ₃ , y ₃ ), and K (x ₃ , y ₃ ) indicates the distance between the virtual light source and the subject at the position (x ₃ , y ₃ ). In order to hit the virtual light source, it is only necessary to satisfy the condition that (Ra (x ₃ , y ₃ ), Ga (x ₃ , y ₃ ), Ba (x ₃ , y ₃ )) is a positive value.

  In step S605, when tracking cannot be performed in step S603, the relighting processing unit 114 determines the parameters of the virtual light source based on the virtual light source parameter setting table at the time of tracking NG. Specifically, it is determined based on the virtual light source setting table at the time of tracking NG shown in FIG.

  In FIG. 7C, 706 indicates a mode, and 707 indicates a virtual light source setting. When the mode is 0, the virtual light source is turned off. For example, when the frame number is 101, the tracking flag is 0, and tracking is not performed, so the virtual light source is turned off. Specifically, the gain α indicating the strength of the virtual light source is set to zero.

  When the mode is 1, control is performed so as to weaken the intensity of the virtual light source. For example, when the frame number is 101, the intensity of the virtual light source is controlled to be weaker than when the frame number is 100. Specifically, the value of the gain α indicating the strength of the virtual light source is set to a value smaller than that when the frame number is 100.

  Based on the above processing, the control unit 50 determines the parameters of the virtual light source of the relighting processing unit 114 based on the subject information acquired by the subject information acquisition unit 113.

  Further, in this embodiment, the case where there are two modes of 0 and 1 in the virtual light source setting table at the time of tracking OK and tracking NG has been described, but the virtual information is based on the subject information acquired by the subject information acquisition unit 113. The present invention is not limited to this as long as it controls parameters such as the position, angle, and intensity of the light source.

  For example, the face organ detection unit of the subject information acquisition unit 113 acquires subject information about the face orientation of the main subject, and controls to change the angle of the virtual light source based on the face orientation information. Also good.

  In this embodiment, the case of correcting brightly by relighting has been described, but conversely, relighting that darkens may be performed. In that case, the gain value α of the virtual light source is made negative.

  Further, the calculation method of the distance D between the position of the virtual light source and the processing target pixel is not limited to the method of this embodiment, and any calculation method may be used. For example, the camera position and the subject position may be acquired as a three-dimensional position and calculated by a three-dimensional distance.

  In addition, when the relighting effect by the virtual light source is added, the calculation is performed using an equation that is inversely proportional to the square of the distance. However, the amount of the relighting effect added is not limited to that calculated by this method. For example, an expression that is inversely proportional to the distance D or an expression in which the irradiation range changes in a Gaussian distribution may be used.

  According to the above embodiment, it is possible to switch between setting a parameter used for relighting processing fixedly for a frame image of a predetermined period in a plurality of frame images or setting a variable for each frame image. Therefore, it is possible to perform appropriate relighting processing according to the state of the subject in the moving image.

(Second Embodiment)
Next, as a second embodiment, a method for determining the parameters of the virtual light source of the relighting processing unit 114 based on the subject information by the subject information acquisition unit 113 composed only of the main subject detection unit will be described.

  In the second embodiment, an example in which the present invention is applied to a digital camera as in Example 1 will be described as an example of an imaging apparatus, and the basic configuration is the same as that in Example 1. In the following description, Will describe only the differences from the first embodiment.

  In the first embodiment, the subject information acquisition unit 113 includes a main subject detection unit, an organ detection unit, and a tracking unit. However, in the second embodiment, the subject information acquisition unit 113 includes only the main subject detection unit. This is different from the first embodiment.

  In the first embodiment, the relighting process can be processed in real time almost simultaneously with the shooting based on the tracking result. However, in the second embodiment, the relighting is performed as a post process after shooting instead of in real time. The point which can perform a process is different from 1st Embodiment.

  FIG. 8 is a flowchart showing the operation of the system control unit 50. Hereinafter, the operation of the relighting processing unit 114 for determining the parameters of the virtual light source based on the subject information acquired by the subject information acquisition unit 113 will be described with reference to the flowchart of FIG.

  In step S <b> 801, it is determined whether relighting processing is selected by an operation from the user via the operation unit 120. That is, it is determined whether or not processing by the relighting processing unit 114 is performed. When rewriting is performed, the process proceeds to step 802.

  In step S802, the subject is photographed.

In step S803, the photographic subject information obtaining unit 113 obtains the position information of the main subject at the time _{t 1.}

At step S804, the in the photographic subject information obtaining unit 113 obtains the position information of the main subject at time _{t n.} Here, time t _{1 and} time t _n may be images of frame number 1 and frame number n, respectively.

In step S805, the virtual light source parameter of the relighting processing unit 114 is determined from the position information of the main subject acquired in steps S802 and S803. For example, FIG. 9 (a), the case where moving the subject (b), the object position at time _{t 1 (x t1, y t1} ), object position at time _{t n (x tn, y tn} ) And position information of the subject.

Based on the information that the subject has moved from (x _t1 , y _t1 ) to (x _tn , y _tn ), the relighting processing unit 114 makes the virtual light source so that the light from the virtual light source strikes the subject at any position. Determine the parameters of position, intensity and angle.

For example, the position of the virtual light source in the case of moving from the position of the virtual light source when the subject did not move from the _{(x t1, y} t1), the subject from _{(x t1, y} t1) to _{(x tn, y} tn) is set far from the object, the angle _{(x t1, y t1)} be _{(x tn, y tn)} at the position of the angular range which corresponds at the position of, and _{(x t1, y t1)} with respect to strength at the position (x _The intensity corresponding to the position of _t1 , _yt1 ) is set.

Specifically, subject reflection components (Ra (x _t1 , y _t1 ), Ga (x _t1 , y _t1 ), Ba (x _t1 , y _t1 ) by the virtual light source when the subject position is (x _t1 , y _t1 ). )) Is expressed by the following formula (Formula 5).
Ra (x _t1 , y _t1 ) =
α × (−L (x _t1 , y _t1 ) · N (x _t1 , y _t1 )) / K (x _t1 , y _t1 ) ² × Rt
Ga (x _t1 , y _t1 ) =
α × (−L (x _t1 , y _t1 ) · N (x _t1, y _t1 )) / K (x _t1 , y _t1 ) ² × Gt
Ba (x _t1 , y _t1 ) =
α × (−L (x _t1 , y _t1 ) · N (x _t1 , y _t1 )) / K (x _t1 , y _t1 ) ² × Bt
(Formula 5)

L (x _t1 , y _t1 ) is the three-dimensional direction vector of the virtual light source, N (x _t1 , y _t1 ) is the three-dimensional normal vector of the subject at the position (x _t1 , y _t1 ), and K (x _t1 , _yt1 ) indicates the distance between the virtual light source and the subject at the position of ( _xt1 , _yt1 ). In order to hit the virtual light source, it is only necessary to satisfy a condition in which (Ra ( _xt1 , _yt1 ), Ga ( _xt1 , _yt1 ), Ba ( _xt1 , _yt1 )) is a positive value.

Similarly, the subject position _{(x tn, y} tn) subject reflected component due to virtual light source when the _{(Ra (x tn, y tn} ), Ga (x tn, y tn), Ba (x tn, y tn)) Is represented by the following formula (formula 6).
Ra (x _tn , y _tn ) =
α × (−L (x _tn , y _tn ) · N (x _tn , y _tn )) / K (x _tn , y _tn ) ² × Rt
Ga (x _tn , y _tn ) =
α × (−L (x _tn , y _tn ) · N (x _tn, y _tn )) / K (x _tn , y _tn ) ² × Gt
Ba (x _tn , y _tn ) =
α × (−L (x _tn , y _tn ) · N (x _tn , y _tn )) / K (x _tn , y _tn ) ² × Bt
(Formula 6)

L (x _tn , y _tn ) is the three-dimensional direction vector of the virtual light source, N (x _tn , y _tn ) is the three-dimensional normal vector of the subject at the position (x _tn , y _tn ), and K (x _tn , y _tn ) indicates the distance between the virtual light source and the subject at the position of (x _tn , y _tn ). In order to hit the virtual light source, it is only necessary to satisfy the condition that (Ra ( _xtn , _ytn ), Ga ( _xtn , _ytn ), Ba ( _xtn , _ytn )) is a positive value.

Based on the above processing, the control unit 50 determines the parameters of the virtual light source of the relighting processing unit 114 based on the position information of the subjects at different times t ₁ and t _n acquired by the subject information acquisition unit 113.

  In the present embodiment, the virtual light source is such that the subject is moving and the light from the virtual light source shines from the position of the subject at the movement source and the movement destination regardless of the position of the movement source or the movement destination. The method of determining the parameters of was described. However, it is not limited to this. For example, the position of the virtual light source may be moved in parallel based on a movement line determined by the movement of the subject.

  In the present embodiment, the method for determining the parameters of the virtual light source based on the position information of the subject has been described. However, the present invention is not limited to the position information. Not only the position information but also the subject information acquisition unit 113 discriminates scene changes such as changes in the facial expression of the subject, whether the subject is speaking, whether the number of subjects has changed, etc. Based on this, the parameters of the virtual light source may be determined.

  In this embodiment, the example in which the digital camera as the imaging apparatus determines the parameters of the virtual light source based on the subject information and performs the relighting process is described, but the present invention is not limited to this. For example, in a configuration where a personal computer or the like captures image data captured by a digital camera, analyzes the subject information of the captured image data, determines virtual light source parameters, and performs relighting processing in the personal computer. good.

  In addition, in a personal computer or the like, image data before relighting processing taken with a digital camera having a relighting processing function, subject information acquired by the digital camera, and virtual light source parameter information are captured. As long as the scene is not relighted as intended, the virtual light source parameters may be edited again and the relighting process may be performed again.

50 System Control Unit 100 Digital Camera 105 Image Processing Unit 114 Relighting Processing Unit 120 Operation Unit 123 Distance Sensor

Claims (6)

An image processing apparatus for performing image processing for providing a relighting effect by a virtual light source to a plurality of frame images constituting a moving image,
Processing means for performing image processing for providing a relighting effect to the frame image;
Setting means for setting parameters used for image processing by the processing means,
The setting means switches between setting the parameter fixedly or setting the parameter variably for each frame image for a frame image of a predetermined period that is continuous in the plurality of frame images. An image processing apparatus. Tracking means for tracking the subject in the plurality of frame images;
2. The image processing according to claim 1, wherein the setting unit switches between setting the parameter fixedly and setting the parameter variably for each frame image during a period in which tracking is successful. apparatus. The parameter used for the image processing is the position of the virtual light source,
The image processing apparatus according to claim 2, wherein when the parameter is changed for each frame image, the setting unit changes a position of the virtual light source according to a subject to be tracked. Detecting means for detecting a position of a subject in the plurality of frame images;
The imaging apparatus according to claim 1, wherein the setting unit sets the parameter based on a change in a position of a subject detected from the plurality of frame images when the parameter is fixedly set. .   The imaging apparatus according to claim 4, wherein the parameter is at least one of a position, an angle, and an intensity of the virtual light source. An image processing method for performing image processing to give a relighting effect by a virtual light source to a plurality of frame images constituting a moving image,
A processing step of performing image processing for imparting a relighting effect to the frame image;
A setting step for setting parameters used for image processing in the processing step;
With
In the setting step, whether to set the parameter fixedly or to set the parameter variably for each frame image is switched for a frame image of a predetermined period that is continuous in the plurality of frame images. An image processing method.

Images