JP6921606B2 - Image processing equipment, image processing methods and programs - Google Patents

Description

The present invention relates to image processing for imparting a rewriting effect by a virtual light source to a plurality of frame images constituting a moving image.

An image in which a virtual light source (hereinafter referred to as a virtual light source) that did not exist at the time of shooting is set in the captured image to give an effect as if the subject in the image was irradiated with light from the virtual light source. The processing is known. Such image processing is called rewriting processing, and it is possible to brighten dark areas such as shadows generated by ambient light at the time of shooting.

The following Patent Document 1 discloses a method of performing rewriting processing for each area of an image according to information about a face area in the image. Specifically, the rewriting process is performed on the face area of the image based on the information on the face area such as the presence / absence and position of the face in the image and the scene discrimination result of the image.

JP-A-2007-148537

When rewriting an image to correct shadows, it is necessary to appropriately set parameters used for the rewriting process, such as the position, intensity, and angle of the virtual light source, according to the state of the subject in the image.

The method of Patent Document 1 targets still images and does not consider moving images. Therefore, when the method of Patent Document 1 is applied to a moving image of a moving subject, the parameters of the rewriting process are set for each frame image, so that the effect of the rewriting process changes one after another, and after the rewriting process. There is a problem that it is difficult for viewers to see the video.

Therefore, the present invention provides image processing that appropriately imparts a rewriting effect by a virtual light source to a moving image.

One embodiment of the present invention is an image processing device that performs image processing for imparting a rewriting effect by a virtual light source to a plurality of frame images constituting a moving image, and performs image processing for imparting a rewriting effect to the frame image. A processing means to be performed, a setting means for setting parameters used for image processing by the processing means, and a tracking means for tracking a subject with the plurality of frame images are provided, and the setting means is among the plurality of frame images. An image processing apparatus characterized in that the parameters are fixedly set or the parameters are variably set for each frame image for continuous frame images during the period in which the tracking is successful. be.

According to the present invention, it is possible to perform an appropriate rewriting process on a moving image.

It is a block diagram which shows the structure of the digital camera in this invention. It is a block diagram which shows the structure of the image processing part in this invention. It is a block diagram which shows the structure of the rewriting processing part in this invention. It is a figure explaining the reflection with respect to the irradiation by a virtual light source in this invention. It is a figure explaining the image before and after the rewriting process in this invention. It is a flowchart which determines the virtual light source parameter in this invention. It is a table which shows the correspondence between the virtual light source parameter and tracking in this invention. It is an image diagram of a subject and a virtual light source in this invention. It is a flowchart which determines the virtual light source parameter in this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Although the present invention will be described using an embodiment in which the present invention is applied to a digital camera, the present invention is not limited to the digital camera and can be applied to other image processing devices.

(First Embodiment)
The first embodiment of the present invention will be described with reference to FIGS. 1 to 7.

FIG. 1 is a block diagram showing a configuration example of a digital camera according to the first embodiment.

In FIG. 1, 100 is an entire digital camera, 101 is a lens group including a zoom lens and a focus lens, 102 is a shutter having an aperture function, and 103 is an image image composed of a CCD or CMOS element that converts an optical image into an electric signal. Part 104 is an A / D converter that converts an analog signal into a digital signal, and 105 is an image data output from the A / D converter 104 that undergoes white balance processing, γ processing, contour enhancement, and color correction processing. It is an image processing unit that performs various image processing such as.

Further, 106 is an image memory, 107 is a memory control unit that controls the image memory 106, 108 is a D / A converter that converts an input digital signal into an analog signal, 109 is a display such as an LCD, and 110 is a compression of image data. This is a codec unit that encodes / decodes.

111 is an interface I / F with the recording medium 112, 112 is a recording medium such as a memory card or a hard disk, 113 is a subject information acquisition unit that detects subject information such as the position of the subject and the orientation of the face from the captured image, and 114 is a subject information acquisition unit. The rewriting processing unit 50 that performs rewriting processing on the captured image is a system control unit that controls the entire system of the digital camera 100.

Further, 121 is a non-volatile memory such as EEPROM for storing programs and parameters, and 122 is a system memory for expanding constants and variables for operation of the system control unit 50, programs read from the non-volatile memory 124, and the like. be. Reference numeral 123 denotes a distance measuring sensor that measures the distance to the subject and outputs the distance information corresponding to each pixel of the photographing pixel as a two-dimensional distance map image.

Next, the basic operation at the time of shooting a subject with the digital camera 100 configured as described above will be described. The imaging unit 103 photoelectrically converts the light incident through the lens 101 and the shutter 102, and outputs the light as an input image signal to the A / D converter 104. The A / D converter 104 converts the analog image signal output from the image pickup unit 103 into a digital image signal and outputs the analog image signal to the image processing unit 105.

The image processing unit 105 performs color conversion processing such as white balance, γ processing, contour enhancement processing, and the like on the image data from the A / D converter 104 or the image data from the memory control unit 107. Further, the image processing unit 105 performs a predetermined evaluation value calculation process (not shown) using the result of the subject information acquired by the subject information acquisition unit 113 and the captured image data, and is based on the obtained evaluation value result. The system control unit 50 performs exposure control and distance measurement control. As a result, TTL (through-the-lens) AF (autofocus) processing, AE (autoexposure) processing, AWB (auto white balance) processing, and the like are performed.

The image data output from the image processing unit 105 is written to the image memory 106 via the memory control unit 107. The image memory 106 stores the image data output from the imaging unit 103 and the image data to be displayed on the display unit 109.

Further, the D / A converter 108 converts the image display data stored in the image memory 106 into an analog signal and supplies it to the display unit 109. The display unit 109 displays on a display such as an LCD according to the analog signal from the D / A converter 108.

The codec unit 110 compresses and encodes the image data recorded in the image memory 106 based on a standard such as MPEG. The system control unit 50 associates the encoded image data and stores it in the recording medium 112 via the recording interface 111.

In addition to the above basic operations, the system control unit 50 realizes each of the processes described below by executing the program recorded in the non-volatile memory 124 described above. The program referred to here is a program for executing various flowcharts described later. At this time, the constants and variables for the operation of the system control unit 50, the program read from the non-volatile memory 121, and the like are expanded in the system memory 122.

Next, the details of the image processing unit 105 will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the image processing unit 105.

In FIG. 2, 200 is a simultaneous processing unit, 201 is a WB amplification unit, 202 is a brightness / color signal generation unit, 203 is a contour enhancement processing unit, 204 is a brightness gamma processing unit, 205 is a color conversion processing unit, and 206 is a color. The γ processing unit, 207 is a color difference signal generation unit, and 208 is a shadow information acquisition unit.

Next, the processing in the image processing unit 105 will be described. The image signal input from the A / D conversion unit 104 of FIG. 1 is input to the image processing unit 105.

The image signal input to the image processing unit 105 is input to the simultaneous processing unit 200. The simultaneous processing unit 200 performs simultaneous processing on the input Bayer RGB image data to generate color signals R, G, and B. The WB amplification unit 201 applies a gain to the RGB color signal based on the white balance gain value calculated by the system control unit 50, and adjusts the white balance. The RGB signal output by the WB amplification unit 201 is input to the luminance / color signal generation unit 202. The luminance / color signal generation unit 202 generates a luminance signal Y from the RGB signal, outputs the generated luminance signal Y to the contour enhancement processing unit 203, and outputs the color signal RGB to the color conversion processing unit 205.

The contour enhancement processing unit 203 performs contour enhancement processing on the luminance signal and outputs it to the luminance gamma processing unit 204. The luminance gamma processing unit 204 performs gamma correction on the luminance signal Y and outputs the luminance signal Y to the image memory 106.

The color conversion processing unit 205 converts the RGB signal into a desired color balance by performing a matrix calculation or the like. The color gamma processing unit 206 performs gamma correction on the RGB color signal. The color difference signal generation unit 207 generates color difference signals RY and BY signals from RGB signals.

The image signal (Y, RY, BY) signal output to the image memory 106 is compressed and encoded by the codec unit 110 and recorded on the recording medium 112.

Further, the output RGB signal of the color conversion processing unit 205 is also input to the shadow information acquisition unit 208. The shadow information acquisition unit 208 acquires information for analyzing the state of the shadow generated on the subject by the environmental light source. For example, the average brightness information of the subject, the brightness histogram information of the face area, and the like are acquired as shadow information.

Next, the configuration and operation of the rewriting processing unit 114 will be described with reference to FIG.

For example, when the rewriting process is selected by the user operation, the data output from the image processing unit 105 is input to the rewriting processing unit 114, and the rewriting process is performed by the virtual light source.

FIG. 3 is a block diagram showing the configuration of the rewriting processing unit 114.

In FIG. 3, 301 is an RGB signal conversion unit that converts input luminance / color difference signals (Y, BY, RY) into RGB signals, and 302 is a degamma processing unit that performs degamma processing. Further, 303 is a distance calculation unit that acquires distance information between the image pickup device and the subject output from the distance measurement sensor 123, 304 is a normal calculation unit that calculates the normal of the subject, and 305 is a virtual light source that reflects off the subject. The virtual light source reflection component calculation unit, 306, is a virtual light source addition processing unit that adds a rewriting effect by the virtual light source.

Further, 307 is a gamma processing unit that applies gamma characteristics to an RGB signal, and 308 is a luminance / color difference signal conversion unit that converts an RGB signal into a luminance / color difference signal (Y, BY, RY).

The operation of the rewriting processing unit 114 having the above configuration will be described.

The rewriting processing unit 114 reads the luminance / color difference signals (Y, BY, RY) from the image data recorded in the image memory 106 and inputs them. The RGB signal conversion unit 301 converts the input luminance / color difference signals (Y, BY, RY) into RGB signals and outputs them to the degamma processing unit 302.

The degamma processing unit 302 calculates the gamma characteristic multiplied by the gamma processing unit of the image processing unit 105 and converts it into linear data. The degamma processing unit 302 outputs the RGB signals (Rt, Gt, Bt) after linear conversion to the virtual light source reflection component calculation unit 305 and the virtual light source addition processing unit 306.

On the other hand, the distance calculation unit 303 calculates the distance map from the subject distance information acquired from the distance measurement sensor 123. The subject distance information is two-dimensional distance information obtained in pixel units of a captured image. The normal calculation unit 304 calculates a normal map from the distance information acquired from the distance calculation unit 303. As for the method of generating the normal map from the distance map, a known technique is used, but a specific processing example will be described with reference to FIG.

FIG. 4 is a diagram showing the relationship between the camera shooting coordinates and the subject. For example, for a subject 401 as shown in FIG. 4, the gradient information can be calculated from the difference ΔD of the distance (depth) D with respect to the horizontal difference ΔH of the captured image, and the normal N can be calculated from the gradient information. It is possible. By performing the above processing on each pixel, the normal information N corresponding to each pixel of the captured image can be calculated. The normal calculation unit 304 outputs the normal information corresponding to each pixel of the captured image as a normal map to the virtual light source reflection component calculation unit 305.

The virtual light source reflection component calculation unit 305 calculates the component that the virtual light source reflects on the subject based on the distance K between the light source and the subject, the normal information N, and the virtual light source parameter L.

Specifically, the reflection component of the coordinate position corresponding to the captured image is calculated so as to be inversely proportional to the square of the distance K between the light source and the subject and proportional to the inner product of the subject normal vector N and the light source direction vector L.

This will be described with reference to FIG. In FIG. 4, 401 indicates the subject and 402 indicates the position of the set virtual light source. The reflection component at the horizontal pixel position H1 (the vertical pixel position is omitted for simplification of explanation) of the captured image taken by the digital camera 100 is proportional to the inner product of the normal N1 at the camera coordinates H1 and the direction vector L1 of the virtual light source. However, the value is inversely proportional to the distance K1 between the virtual light source and the subject position.

Expressing this relationship with a mathematical formula, the subject reflection components (Ra, Ga, Ba) by the virtual light source are as follows (Equation 1).
Ra = α × (−L ・ N) / K ² × Rt
Ga = α × (−L · N) / K ² × Gt (Equation 1)
Ba = α × (−L ・ N) / K ² × Bt

Here, α is the intensity of the virtual light source and is the gain value of the rewriting correction amount. L is the three-dimensional direction vector of the virtual light source, N is the three-dimensional normal vector of the subject, and K is the distance between the virtual light source and the subject. Rt, Gt, and Bt are captured RGB data output from the degamma processing unit 302.

The reflection components (Ra, Ga, Ba) by the virtual light source calculated as described above are output to the virtual light source addition processing unit 306. The virtual light source addition processing unit 306 performs processing according to the following (Equation 2) for adding virtual light source components (Ra, Ga, Ba).
Rout = Rt + Ra
Gout = Gt + Ga (Equation 2)
Bout = Bt + Ba

The image signals (Rout, Gout, Bout) output from the virtual light source addition processing unit 306 are input to the gamma processing unit 307. The gamma processing unit 307 performs gamma correction on the RGB input signal. The color difference signal generation unit 308 generates brightness Y, color difference signals RY, and BY signals from RGB signals.

The above is the operation of the rewriting processing unit 114. FIG. 5 shows an example of rewriting processing by the rewriting processing unit 114. FIG. 5A is a photographed image before the rewriting process, and FIG. 5B is a photographed image after the rewriting process. The dark subject in FIG. 5 (a) is brightly corrected as shown in FIG. 5 (b) by applying a virtual light source and performing a rewriting process.

The system control unit 50 stores the luminance / color difference signal output by the rewriting correction unit 114 in the image memory 106 under the control of the memory control unit 107, and then compresses and encodes it in the codec unit 110. Also, recording is performed on the recording medium 112 via the I / F 111.

Next, a processing flow in which the system control unit 50 determines the parameters of the virtual light source in the rewriting processing unit 114 based on the subject information acquired by the subject information acquisition unit 113 will be described below.

The subject information acquisition unit 113 here includes a main subject detection unit that detects the main subject, a face organ detection unit that detects the face organ of the main subject detected by the main subject detection unit, and a main subject detection unit that detects the main subject. It consists of a tracking unit that tracks the subject. Known techniques shall be used for the main subject detection method, facial organ detection method, and tracking method. For example, as a main subject detection method, there is a method of estimating the position of the subject from the feature points on the screen and determining the region having the largest feature amount as the main subject. For example, as a method for detecting facial organs, there is a method of detecting a plurality of parts such as facial contours and eyes from edges and detecting the orientation of the face based on the positional relationship. For example, as a tracking method, a method of comparing different images in the time direction to obtain a movement vector, or a method of patterning a subject or a part thereof into a feature color pattern and tracking images taken at different times by feature color pattern matching. and so on.

FIG. 6 is a flowchart showing the operation of the system control unit 50. Hereinafter, an operation in which the system control unit 50 determines the parameters of the virtual light source of the rewriting processing unit 114 based on the subject information acquired by the subject information acquisition unit 113 will be described according to the flowchart of FIG.

In step S601, it is determined whether the rewriting process is selected by the operation from the user via the operation unit 120. That is, it is determined whether or not the processing by the rewriting processing unit 114 is performed. If rewriting is to be performed, the process proceeds to step 602.

In step S602, the subject is photographed.

In step S603, the subject information acquisition unit 113 determines whether or not the subject can be tracked in each frame. If the tracking is successful, the process proceeds to step S604, and if the tracking is unsuccessful, the process proceeds to step S605.

Specifically, the subject information shown in the correspondence table between the frame number (frame No.) shown in FIG. 7A and the tracking result is acquired. In FIG. 7A, 701 indicates a frame number. 702 indicates a flag as to whether or not tracking is possible.

If the tracking is successful, the flag is set to 1, and if the tracking is unsuccessful, the flag is set to 0. Reference numeral 703 indicates the tracking result, that is, the coordinates of the face position of the main subject. If the flag is 1, the process proceeds to step S604, and if the flag is 0, the process proceeds to step S605, depending on the flag indicating whether or not the tracking shown in FIG. 7 (a) is successful.

In step S604, when the tracking is successful in step S603, the rewriting processing unit 114 determines the parameters of the virtual light source based on the parameter setting table of the virtual light source when the tracking is OK. Specifically, the determination is made based on the virtual light source setting table at the time of tracking OK shown in FIG. 7 (b). In FIG. 7B, 704 indicates the mode and 705 indicates the virtual light source setting. When the mode is 0, the position of the virtual light source is controlled so as not to be changed for each frame.

For example, when the frame number is 3, the tracking flag is 1, and the parameter of the virtual light source is controlled so that the position is the same as the position of the virtual light source when the frame number is 2 one frame before.
When the mode is 1, the position of the virtual light source is controlled to be changed for each frame according to the tracking result. For example, when the frame number is 2, the position of the virtual light source as a virtual light source hits the face position of the main object (x _{2, y 2),} the angle determines the parameters of the virtual light source such as strength, frame number 3 when determines parameters such position, angle, intensity of the virtual light source as a virtual light source hits the face position of the main object (x _{3, y 3)} of the. That is, the subject reflected component frame number by the virtual light source when the _{2 (Ra (x 2, y} 2), Ga (x 2, y 2), Ba (x 2, y 2)) , the following equation (Equation 3 ). The settings of mode 0 and mode 1 may be determined by an instruction operation from the user, or may be automatically determined according to the state of the subject by analyzing the image.
_{Ra (x 2, y 2)} =
α × (−L (x ₂ , y ₂ ) ・ N (x ₂ , y ₂ )) / K (x ₂ , y ₂ ) ² × Rt
Ga (x ₂ , y ₂ ) =
α × (−L (x ₂ , y ₂ ) ・ N (x ₂ , y ₂ )) / K (x ₂ , y ₂ ) ² × Gt
Ba (x ₂ , y ₂ ) =
α × (−L (x ₂ , y ₂ ) ・ N (x ₂ , y ₂ )) / K (x ₂ , y ₂ ) ² × Bt
(Equation 3)

L (x ₂ , y ₂ ) is the three-dimensional direction vector of the virtual light source, N (x ₂ , y ₂ ) is the three-dimensional normal vector of the subject at the position _{(x 2} , y _{2), and K (x). 2} , y ₂ ) indicates the distance between the virtual light source and the _{subject at the position (x 2} , y _2). A virtual light source hits _{are, (Ra (x 2, y} 2), Ga (x 2, y 2), Ba (x 2, y 2)) may satisfy the condition that a positive value.

Similarly, when the frame number is 3, the subject reflection component (Ra (x ₃ , y ₃ ), Ga (x ₃ , y ₃ ), Ba (x ₃ , y ₃ )) by the virtual light source is given by the following equation (Equation). It is represented by 4).
Ra (x ₃ , y ₃ ) =
α × (−L (x ₃ , y ₃ ) ・ N (x ₃ , y ₃ )) / K (x ₃ , y ₃ ) ² × Rt
Ga (x ₃ , y ₃ ) =
α × (−L (x ₃ , y ₃ ) ・ N (x ₃ , y ₃ )) / K (x ₃ , y ₃ ) ² × Gt
Ba (x ₃ , y ₃ ) =
α × (−L (x ₃ , y ₃ ) ・ N (x ₃ , y ₃ )) / K (x ₃ , y ₃ ) ² × Bt
(Equation 4)

L (x ₃ , y ₃ ) is the 3D direction vector of the virtual light source, N (x ₃ , y ₃ ) is the _{3D normal vector of the subject at the position (x 3} , y ₃ ), and K (x). ₃ , y ₃ ) indicates the distance between the virtual light source and the _{subject at the position (x 3} , y _3). The virtual light source is defined as long as it satisfies the condition that _{(Ra (x 3} , y ₃ ), Ga (x ₃ , y ₃ ), Ba (x ₃ , y _{3)) has a positive value.}

In step S605, when tracking cannot be performed in step S603, the rewriting processing unit 114 determines the parameters of the virtual light source based on the parameter setting table of the virtual light source at the time of tracking NG. Specifically, the determination is made based on the virtual light source setting table at the time of tracking NG shown in FIG. 7 (c).

In FIG. 7 (c), 706 indicates a mode and 707 indicates a virtual light source setting. When the mode is 0, the virtual light source is turned off. For example, when the frame number is 101, the tracking flag becomes 0 and tracking is not completed, so the virtual light source is turned off. Specifically, the gain α indicating the strength of the virtual light source is set to 0.

When the mode is 1, the intensity of the virtual light source is controlled to be weakened. For example, when the frame number is 101, the intensity of the virtual light source is controlled to be weaker than when the frame number is 100. Specifically, the value of the gain α indicating the intensity of the virtual light source is set to a value smaller than that when the frame number is 100.

Through the above processing, the control unit 50 determines the parameters of the virtual light source of the rewriting processing unit 114 based on the subject information acquired by the subject information acquisition unit 113.

Further, in this embodiment, the case where there are two modes 0 and 1 in the virtual light source setting table when tracking is OK and when tracking is NG has been described, but it is virtual based on the subject information acquired by the subject information acquisition unit 113. It is not limited to this as long as it controls parameters such as the position, angle, and intensity of the light source.

For example, the face organ detection unit of the subject information acquisition unit 113 acquires subject information about the orientation of the face of the main subject, and controls to change the angle of the virtual light source based on the orientation information of the face. Is also good.

Further, in this embodiment, the case where the correction is made brighter by rewriting has been described, but conversely, the rewriting which makes the correction darker may be performed. In that case, the gain value α of the virtual light source is set to minus.

Further, the method of calculating the position of the virtual light source and the distance D of the pixel to be processed is not limited to the method of the present embodiment, and any calculation method may be used. For example, the position of the camera and the position of the subject may be acquired as three-dimensional positions and calculated by the distance in three dimensions.

Further, when the rewriting effect by the virtual light source is added, the calculation is performed by an equation inversely proportional to the square of the distance, but the addition amount of the rewriting effect is not limited to the one calculated by this method. For example, it may be an equation that is inversely proportional to the distance D or an equation in which the irradiation range changes in a Gaussian distribution.

According to the above embodiment, it is possible to switch between fixing the parameters used for the rewriting process and setting them variably for each frame image for the frame images of a plurality of frame images for a predetermined period of time. Therefore, it is possible to perform an appropriate rewriting process according to the state of the subject in the moving image.

(Second embodiment)
Next, as a second embodiment, a method of determining the parameters of the virtual light source of the rewriting processing unit 114 based on the subject information by the subject information acquisition unit 113 composed of only the main subject detection unit will be described.

In the second embodiment, an example in which the present invention is applied to a digital camera as in the first embodiment will be described as an example of the imaging device, and the basic configuration is the same as that of the first embodiment. Will describe only the differences from Example 1.

In the first embodiment, the subject information acquisition unit 113 is composed of a main subject detection unit, an organ detection unit, and a tracking unit, but in the second embodiment, the subject information acquisition unit 113 has only the main subject detection unit. It is different from the first embodiment in that it is composed of.

Further, in the first embodiment, it was possible to process the rewriting process in real time at almost the same time as the shooting based on the tracking result, but in the second embodiment, the rewriting is not in real time but as post-processing after shooting. It differs from the first embodiment in that the process can be executed.

FIG. 8 is a flowchart showing the operation of the system control unit 50. Hereinafter, an operation in which the system control unit 50 determines the parameters of the virtual light source of the rewriting processing unit 114 based on the subject information acquired by the subject information acquisition unit 113 will be described according to the flowchart of FIG.

In step S801, it is determined whether or not the rewriting process is selected by the operation from the user via the operation unit 120. That is, it is determined whether or not the processing by the rewriting processing unit 114 is performed. To perform rewriting, the process proceeds to step 802.

In step S802, the subject is photographed.

In step S803, the photographic subject information obtaining unit 113 obtains the position information of the main subject at the time _{t 1.}

At step S804, the in the photographic subject information obtaining unit 113 obtains the position information of the main subject at time _{t n.} Here, the time t _{1 and the} time t _n may be images having frame numbers 1 and n, respectively.

In step S805, the virtual light source parameter of the rewriting processing unit 114 is determined from the position information of the main subject acquired in steps S802 and S803. For example, FIG. 9 (a), the case where moving the subject (b), the object position at time _{t 1 (x t1, y t1} ), object position at time _{t n (x tn, y tn} ) And acquire the position information of the subject.

In the rewriting processing unit 114, _{based on the information that the subject has moved from (x t1} , y _t1 ) to (x _tn , y _tun ), the virtual light source so that the light from the virtual light source hits the subject at either position. Determine the position, strength, and angle parameters of.

For example, the position of the virtual light source in the case of moving from the position of the virtual light source when the subject did not move from the _{(x t1, y} t1), the subject from _{(x t1, y} t1) to _{(x tn, y} tn) is set far from the object, the angle _{(x t1, y t1)} be _{(x tn, y tn)} at the position of the angular range which corresponds at the position of, and _{(x t1, y t1)} with respect to strength at the position (x Set the strength to correspond to the position of _t1 and y _t1).

Specifically, when the subject position is (x _t1 , y _t1 ), the subject reflection component (Ra (x _t1 , y _t1 )), Ga (x _t1 , y _t1 ), Ba (x _t1 , y _{t1) by the virtual light source is used.} )) Is expressed by the following equation (Equation 5).
Ra (x _t1 , y _t1 ) =
α × (−L (x _t1 , y _t1 ) ・ N (x _t1 , y _t1 )) / K (x _t1 , y _t1 ) ² × Rt
Ga (x _t1 , y _t1 ) =
α × (−L (x _t1 , y _t1 ) ・ N (x _t1, y _t1 )) / K (x _t1 , y _t1 ) ² × Gt
Ba (x _t1 , y _t1 ) =
α × (−L (x _t1 , y _t1 ) ・ N (x _t1 , y _t1 )) / K (x _t1 , y _t1 ) ² × Bt
(Equation 5)

L (x _t1 , y _t1 ) is the three-dimensional direction vector of the virtual light source, and N (x _t1 , y _t1 ) is the three-dimensional normal vector of the subject at the position _{(x t1} , y _{t1), K (x). t1} , y _t1 ) indicates the distance between the virtual light source and the _{subject at the position (x t1} , y _t1). The virtual light source does not have to satisfy the condition that _{(Ra (x t1} , y _t1 ), Ga (x _t1 , y _t1 ), Ba (x _t1 , y _{t1)) has a positive value.}

Similarly, the subject reflection component by the virtual light source when the subject position is (x _nt , y _{nt ) (Ra (x tun} , y _tun ), Ga (x _tun , y _tun ), Ba (x _tun , y _tun )). Is expressed by the following equation (Equation 6).
Ra (x _tun , y _tun ) =
α × (−L (x _tun , y _tun ) ・ N (x _tun , y _tun )) / K (x _tun , y _tun ) ² × Rt
Ga (x _tun , y _tun ) =
α × (−L (x _tun , y _tun ) ・ N (x _tun, y _tun )) / K (x _tun , y _tun ) ² × Gt
Ba (x _tun , y _tun ) =
α × (−L (x _tun , y _tun ) ・ N (x _tun , y _tun )) / K (x _tun , y _tun ) ² × Bt
(Equation 6)

L (x _nt , y _tun ) is the three-dimensional direction vector of the virtual light source, N (x _tun , y _tun ) is the three-dimensional normal vector of the subject at the position _{(x tun} , y _{tun), and K (x). tn, y tn)} denotes the distance to the subject who is in the position of the virtual light source _{(x tn, y} tn). The virtual light source does not have to satisfy the condition that _{(Ra (x tun} , y _tun ), Ga (x _tun , y _tun ), Ba (x _tun , y _{tun)) has a positive value.}

Through the above processing, the control unit 50 determines the parameters of the virtual light source of the rewriting processing unit 114 based on the position information of the subject at _{different times t 1} and t _{n acquired by the subject information acquisition unit 113.}

Further, in the present embodiment, the subject is moving, and the virtual light source so that the light from the virtual light source shines from the position of the subject at the moving source and the moving destination so that the subject is at either the moving source or the moving destination position. The method of determining the parameters of is explained. However, it is not limited to this. For example, the positions of the virtual light sources may be moved in parallel based on the movement line determined by the movement of the subject.

Further, in the present embodiment, the method of determining the parameter of the virtual light source based on the position information of the subject has been described, but the method is not limited to the position information. Not limited to the position information, the subject information acquisition unit 113 determines, for example, a change in the facial expression of the subject, whether or not the subject is speaking, a change in the number of subjects, and the like, and obtains information on the change of the scene. The parameters of the virtual light source may be determined based on this.

Further, in the present embodiment, an example in which a digital camera as an imaging device determines parameters of a virtual light source based on subject information and performs rewriting processing has been described, but the present invention is not limited to this. For example, in a personal computer or the like, even if the personal computer has a configuration in which the parameters of the virtual light source are determined by capturing the image data taken by the digital camera and analyzing the subject information of the captured image data, the personal computer performs the rewriting process. good.

In addition, in a personal computer or the like, the image data before the rewriting process taken by a digital camera having a rewriting process function, the subject information acquired by the digital camera, and the parameter information of the virtual light source are taken in, and the user in the rewriting process by the digital camera. Only in the scene where the rewriting is not performed as intended, the parameters of the virtual light source may be edited again and the rewriting process may be performed again.

50 System control unit 100 Digital camera 105 Image processing unit 114 Rewriting processing unit 120 Operation unit 123 Distance measurement sensor

Claims (7)

An image processing device that performs image processing that imparts a rewriting effect by a virtual light source to a plurality of frame images that make up a moving image.
A processing means for performing image processing for imparting a rewriting effect to the frame image, and
Setting means for setting parameters used for image processing by the processing means, and
A tracking means for tracking a subject with the plurality of frame images, and
With
The setting means either fixes the parameter or sets the parameter variably for each frame image with respect to the frame images that are continuous during the period in which the tracking is successful among the plurality of frame images. An image processing device characterized by switching between. The parameter used for the image processing is the position of the virtual light source.
The setting means, if the variable to set the parameters for each frame image in the period in which the tracking is successful, according to claim 1, characterized in that changing the position of the virtual light source according to the object to be tracked Image processing equipment. The image processing apparatus according to claim 1, wherein the parameter is at least one of the position, angle, and intensity of the virtual light source. The setting means is any one of claims 1 to 3, wherein when the parameter is fixedly set during the period when the tracking is successful, the virtual light source is turned off during the period when the tracking is not successful. The image processing apparatus according to item 1. When the setting means variably sets the parameter for each frame image during the period when the tracking is successful, the intensity of the virtual light source is weaker during the period when the tracking is not successful than during the period when the tracking is successful. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is used. This is an image processing method that performs image processing that imparts a rewriting effect by a virtual light source to a plurality of frame images that make up a moving image.
A processing step of performing image processing for imparting a rewriting effect to the frame image, and
A setting process for setting parameters used for image processing in the processing process , and a setting process for setting parameters.
The tracking process of tracking the subject with the plurality of frame images,
With
In the setting step , the parameters are fixedly set for the frame images of the plurality of frame images that are continuous in the period in which the tracking is successful, or the parameters are changed for each frame image. An image processing method characterized by switching between setting and setting. A program that causes a computer to function as each means of the image processing apparatus according to any one of claims 1 to 5 .

Images