JP5956756B2 - Video processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to a video processing technique, and more particularly, to a video processing technique for applying a lighting effect to a foreground image in accordance with a background image and a synthesis position during image synthesis.

  It is common in video editing to cut out a subject from a foreground image and combine it with a background image. For example, a typical example is that a subject is cut out with a chroma key from a video shot with a blue background and then combined with a different video. At that time, in order to make the synthesized video natural, it is necessary that the lighting of the background video and the lighting of the foreground video match. In order to match the lighting, a lighting effect is applied to a real image using CG.

  As lighting using CG, lighting is often performed using a ray tracing or radiosity that makes a lighting environment a 3D model and calculates a light path.

  In addition, image-based lighting is often performed in which a background image of a background image is captured and a lighting effect is applied using the captured environment image as a light source image (Non-Patent Document 1, Non-Patent Document). 2). In addition, a light source in the distance is performed by image-based lighting, and a lighting effect is performed using a 3D model for a light source in the vicinity. (Non Patent Literature 3)

Peter-Pike Sloan and Ben Luna and John Snyder ”Local, Deformable Precomputed Radiance Transfer” SIGGRAPH ACM ACM SIGGRAPH 2005 Papers Zhong Ren and Rui Wang and John Snyder and Kun Zhou and Xinguo Liu and Bo Sun and Peter-Pike Sloan and Hujun Bao and Qunsheng Peng and Baining Guo ”Real-time Soft Shadows in Dynamic Scenes using Spherical Harmonic Exponentiation”, SIGGRAPH ACM SIGGRAPH 2006 Papers Prof. Shiho Furuya, Takayuki Ito “Image-based Drying of Scenes Including Neighborhood Light Sources” Information Processing Society of Japan Annual Conference

  In the conventional method using image-based lighting, when the synthesis position is changed, it is necessary to prepare an all-around image corresponding to the synthesis position in advance. This is because when the composition position changes, the positional relationship between the neighboring light source and the composition position changes greatly, and the lighting changes greatly. If an all-around image corresponding to the synthesis position is not used, an unnatural lighting effect is applied. If the positions to be combined are determined in advance, a finite number of all-around images may be prepared. Since there are an infinite number of combining positions, it is impossible to prepare all the surrounding images according to the combining positions in advance.

  In the conventional method using ray tracing or radiosity, the lighting environment in the field where the background image is taken must be manually modeled as a 3D model, which is troublesome. Ray tracing and radiosity perform lighting by calculating ray paths, but in order to achieve more natural lighting, the ray paths branch many times and the amount of calculations increases. .

  The present invention seeks to provide a technique that solves the above-described problems and that can easily provide a lighting effect according to a synthesis position.

In order to solve this problem, for example, a video processing apparatus of the present invention has the following configuration. That is,
Applying lighting effects to the foreground picture image data, an image processing apparatus for synthesizing the background image data,
Storage means for storing the background image data, light source information representing a position of at least one light source with respect to the imaging device when the background image data is captured, and moving image data of the subject as the foreground image data;
A composite position acquisition means for acquiring an initial composite position of the subject in the background image data ;
The synthesized moving image that generates the synthesized moving image data by sequentially generating a synthesized frame of the subject and the background image data, in which the obtained initial synthesized position is the initial arrangement position of the subject in the moving image data. Generating means,
The synthesized moving image generating means
Spatial coordinate acquisition means for acquiring the spatial coordinates of the composite position of the subject in the background image data each time the composite frame is generated ;
Calculation means for calculating a distance between the spatial coordinates of the subject acquired by the spatial coordinate acquisition means and each light source indicated by the light source information acquired from the storage means ;
Among the light sources indicated by the light source information acquired from the storage unit, the position of the light source whose distance calculated by the calculation unit exceeds a preset threshold is not changed, and the distance calculated by the calculation unit is equal to or less than the threshold. For the light source to be obtained, the position of the light source viewed from the subject, the changing means for changing the position of the corresponding light source in the acquired light source information according to the obtained position ;
Lighting means for applying a lighting effect to the subject using the position of each light source indicated by the light source information after the change processing by the changing means;
The subject lighting effect said writing means has been performed, characterized in that it comprises a synthesizing means for generating the composite frame by synthesizing the arrangement position of the object in the background image data.

  According to the present invention, it is possible to easily apply a lighting effect corresponding to the synthesis position to the foreground image, and to make a natural video even if the video is synthesized.

The hardware block diagram of the video editing apparatus in 1st Embodiment. The figure of the external appearance of the photo frame in a 1st embodiment. The figure which shows the screen shot before and behind changing the synthetic | combination position of a to-be-photographed object in the video editing screen in 1st Embodiment. The functional block diagram of the video editing apparatus in 1st Embodiment. The figure showing the light source parameter table in a 1st embodiment. The figure showing synthetic position coordinate data in a 1st embodiment. The figure showing the data of projection information in a 1st embodiment. 5 is a flowchart of processing performed by the video editing apparatus according to the first embodiment. The flowchart of the video editing process in 1st Embodiment. The flowchart of the light source parameter change process in 1st Embodiment. The flowchart of the lighting process in 1st Embodiment. The flowchart of the synthetic | combination process in 1st Embodiment. The figure showing the light source parameter table in 2nd Embodiment. The flowchart of the video editing apparatus in 2nd Embodiment. The flowchart of the video editing process in 2nd Embodiment. The flowchart of the light source parameter change process in 2nd Embodiment. The flowchart of the video editing process in 3rd Embodiment. The flowchart of the preparation process in 4th Embodiment. The figure showing the light source parameter table in 4th Embodiment. The flowchart of the video editing process in 4th Embodiment.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
Hereinafter, the present embodiment will be described with reference to the drawings.

  FIG. 1 shows a hardware configuration diagram for realizing the present embodiment. The CPU 101 is related to all the processes of each configuration, reads and interprets the instructions stored in the ROM 102 and the RAM 103 in order, and processes the processes according to the results. The ROM 102 and RAM 103 provide the CPU 101 with programs, data, work areas, and the like necessary for the processing. The storage medium 104 is a storage medium (or device) that stores image data and the like, and is, for example, a hard disk, a CF card, an SD card, a USB memory, a memory card, or the like. The input device 105 is an input device such as a touch screen, a keyboard, and a mouse, and receives instructions from the user. In general, a liquid crystal display is widely used as the output device 106, and displays images and characters. Further, a touch panel function may be provided, and in that case, the input device 105 can be handled.

  In the present embodiment, an example in which the present invention is applied to a photo frame as an embodiment of the video editing apparatus will be described.

  FIG. 2 is a front view of a photo frame which is a video processing apparatus in the present embodiment. The touch screen display 201 is a display for displaying an image and is an input device that receives a user input. The touch screen display 201 corresponds to the output device 106 and the input device 105 in FIG. 1, and can simultaneously detect the coordinates of a plurality of inputs. Reference numeral 202 denotes a power button.

  FIG. 3 shows screenshots before and after the subject composition position is changed on the video editing screen in the present embodiment. Reference numeral 301 denotes a background image, and 302 denotes a synthesized foreground image. A large number of background images and foreground images are distinguished and stored in the storage medium 104, and the illustrated background image 301 and foreground image 302 are those specified by the user. An α value (transparency) is added to the foreground image, and a desired subject is cut out. In the present embodiment, the foreground image 302 is a moving image, and when it is reproduced, the subject moves and its composition position changes. Along with the change of the composition position, the foreground image 303 is subjected to a lighting effect corresponding to the composition position. In this embodiment, since the foreground image is a moving image, the synthesis position is changed. However, this is an example, and the present invention is not limited to this. For example, the synthesis position of the foreground image may be changed by the user dragging the foreground image. Alternatively, the scale of the foreground image may be changed, the background image may be moved, or the scale may be changed according to a user operation.

  FIG. 4 shows a functional block diagram of the apparatus of the present embodiment. In the drawing, 401 is a foreground image. Reference numeral 402 denotes a background image. Reference numeral 403 denotes background image depth information representing the depth of each pixel of the background image 402. Depth information is recorded in the RGB component of each pixel in an image format with the same resolution as the background image 402. Also, the background image depth information is generated by photographing the background image as a stereo image and using the stereo matching method. Note that the generation of depth information using the stereo matching method is an example, and the present invention is not limited to this. For example, the straight line and vanishing point of the background image may be detected to estimate the depth information. Also, the user may specify. Reference numeral 404 denotes a light source parameter representing position information or illuminance information of the light source viewed from the position where the background image is captured. The position information of the light source is acquired by generating an all-around image of the position where the background image is taken and its depth information using Japanese Patent Application Laid-Open No. 2010-181826. As the illuminance information of the light source, the entire surrounding image at the position where the background image is captured is captured as an HDR image, and the luminance value of the image is acquired as the illuminance information. These are only examples, and the present invention is not limited to these. For example, the position information of the light source may be specified by the user, or knowledge information up to the light source may be acquired using a distance measuring sensor. Further, the illuminance information of the light source may be specified by the user or measured with an illuminometer.

  FIG. 5 shows an example of a light source parameter table. Reference numeral 501 denotes an ID for identifying each light source. An area having a certain luminance value or more in the entire surrounding image is set as a light source, and its center of gravity is set as the position of the light source. Reference numeral 502 denotes distance information from the background image shooting position to the light source. The unit is meters. When the distance information cannot be acquired, “−1” is held as being present at infinity. Reference numerals 503 and 504 denote the direction of the light source as viewed from the background image photographing position, which are an azimuth angle and an elevation angle. The unit is degrees. In this embodiment, an area having a certain luminance value in the entire surrounding image is regarded as a light source area. However, this is an example, and the present invention is not limited to this. For example, each pixel may be treated as a light source. Moreover, in this embodiment, although the light source position is hold | maintained by the polar coordinate system, this is an example and is not restricted to this. For example, you may hold | maintain with a rectangular coordinate system.

  Returning to the description of FIG. 405 in the figure is the result of calculating the light propagation in advance. It is a light propagation prior calculation result. The calculation is performed using Non-Patent Document 1 or Non-Patent Document 2 described above. A composite image 406 is an editing result of the video editing apparatus according to the present embodiment. The foreground image 401, the background image 402, the background image depth information 403, the light source parameter 404, the light propagation pre-calculation result 405, and the composite image 406 are recorded on the recording medium 104. Reference numeral 410 denotes a video editing apparatus according to the present embodiment. Reference numeral 411 denotes a composite position acquisition unit that acquires coordinates on the background image as a composite position. The coordinates on the background image designated by the user using the input device 105 are acquired as the synthesis position. In addition, the coordinates on the background image of the pixel on the foreground image that is the reference point for synthesis designated in advance by the user are acquired as the synthesis position. Although the user designates the reference point for synthesis, this is merely an example and the present invention is not limited to this. For example, face detection may be performed and the position of the face may be used as a reference for the combined position. Further, the lowest part of the opaque part of the foreground image may be used as a reference for the composition position.

  FIG. 6 shows an example of the combined position information. Reference numeral 601 denotes an X coordinate on the background image at the synthesis position, and reference numeral 602 denotes a Y coordinate.

  Next, the present embodiment will be described with reference to FIG. Reference numeral 412 denotes a projection unit that generates projection information obtained by projecting space coordinates generated from the depth information of the background image onto two-dimensional coordinates on the background image. Projection is performed using a perspective projection method.

  FIG. 7 shows an example of projection information. Reference numeral 701 denotes two-dimensional coordinates on the background image. Reference numeral 702 denotes an X coordinate. Reference numeral 703 denotes a Y coordinate. Reference numeral 711 denotes spatial coordinates generated from the depth information of the background image. Reference numeral 712 denotes an X coordinate. Reference numeral 713 denotes a Y coordinate. Reference numeral 714 denotes a Z coordinate. This is an example, and the present invention is not limited to this. For example, the X coordinate, Y coordinate, and Z coordinate corresponding to each pixel on the background image may be stored as image information as R, G, and B information.

  Next, the present embodiment will be described with reference to FIG. Reference numeral 413 denotes a spatial coordinate acquisition unit that acquires spatial coordinates corresponding to the coordinates on the background image acquired by the composite position acquisition unit 411 based on the projection information generated by the projection unit 412.

  Reference numeral 414 denotes a light source distance or illuminance calculator that calculates the illuminance of the light source viewed from the distance between the spatial coordinates of the combined position and the light source or the spatial coordinates of the combined position. Details will be described later. Reference numeral 415 denotes light source parameter changing means for changing the light source parameter. Details will be described later. A lighting unit 416 applies a lighting effect to the foreground image based on the light source parameter changed by the light source parameter changing unit 415. Details will be described later. Here, the lighting effect may be applied to the foreground image after performing the process of removing the shadow portion. The removal of the shadow portion is performed using a technique disclosed in Japanese Patent Application Laid-Open No. 2010-135996. Reference numeral 417 denotes a combining unit that combines the foreground image with the lighting effect applied by the writing unit 416 at the combining position acquired by 411 with the background image. The synthesis result of the synthesis unit 416 is output to the recording medium 104 and the output device 106.

  FIG. 8 shows a flowchart of a control processing procedure executed by the CPU 101 of the video editing apparatus in the present embodiment. When the video editing apparatus is activated, the foreground image selected by the user is acquired from the recording medium 104 (S801). In the present embodiment, a moving image is acquired. Next, the background image selected by the user is acquired from the recording medium 104 (S802). An initial composite position is acquired from user input from the input device 105 (S803). The depth information corresponding to the background image acquired in S802 is acquired from the recording medium 104 (S804). A light source parameter table corresponding to the background image acquired in S802 is acquired from the recording medium 104 (S805). Video editing processing is started (S806). Details of this video editing process will be described later. The edited image is recorded on the recording medium 104 (S807). It is determined whether an end operation has been performed (S808). If it is determined that the end operation has not been performed (S808), a foreground image is acquired (S801). If it is determined that the end operation has been performed (S808), the video editing apparatus is ended.

  FIG. 9 shows a flowchart of the video editing process of S806 in the present embodiment. When the video editing process is started, the coordinates of the space generated from the background image depth information acquired in S804 are projected onto the two-dimensional coordinates on the background image to generate projection information (S901). The first frame of the foreground image acquired in S801 is acquired (S902). A light source parameter change process is performed (S903). Details of the light source parameter changing process will be described later. Writing processing is performed (S904). Details of the lighting process will be described later. A synthesis process is performed (S905). Details of the synthesis process will be described later. The composite image is displayed on the display device 106 (S906). It is determined whether a video editing end operation has been performed (S907). If it is determined that the end operation has been performed (S907), the video editing process is ended. If it is determined that the ending operation has not been performed (S907), it is determined whether or not there is a next frame in the moving image used as the foreground image (S908). If it is determined that there is a next frame (S908), the next frame image of the moving image is acquired as the foreground image (S909). A predetermined pixel in the foreground is detected, and the composition position is changed only by a change in position with the previous frame (S910). If it is determined that there is no next frame (S908), the video editing process is terminated.

  FIG. 10 shows a flowchart of the light source parameter change processing 903 in the present embodiment. When the light source parameter changing process is started, the spatial coordinates of the composite position are acquired from the projection information generated in S901 (S1001). Light source parameters are acquired from the light source parameter table acquired in S805 (S1002). From the spatial coordinates of the combined position acquired in S1001 and the light source parameters acquired in S1002, the distance of the light source is calculated based on Equations 1 and 2 (S1003).

The distance information 502 of the light source parameter is R, the azimuth angle 503 is θ, and the elevation angle 504 is φ. The x coordinate of the space coordinates of the composite position is X _c , the y coordinate is Y _c , and the z coordinate is Z _c . A distance calculating and R _l.
X ₁ = R · cos (θ) cos (φ) −X _c
Y _l = R · sin (θ) cos (φ) −Y _c
Z _l = R · sin (θ) −Z _c (1)

^{_{R l = {X l 2 +}} Y l 2 + Z l 2} 1/2 ... (2)

In S1004, it is determined whether or not the value calculated in S1003 is equal to or less than a threshold value. Only when it is determined that the threshold value is equal to or less than the threshold value, the light source parameter changing process is performed based on the spatial coordinates of the combined position acquired in S1001 and the light source parameter acquired in S1002 (S1005). The parameter is changed based on the equations (3) and (4).

The distance information 502 of the light source parameter is R, the azimuth angle 503 is θ, and the elevation angle 504 is φ. The x coordinate of the space coordinates of the composite position is X _c , the y coordinate is Y _c , and the z coordinate is Z _c . The x coordinate of the light source after changing the light source parameters is X _l , the y coordinate is Y _l , and the z coordinate is Z _l . The distance information 502 after the light source parameter change is R _l , the azimuth angle 503 is θ _l , and the elevation angle 504 is φ _l .
X ₁ = R · cos (θ) cos (φ) −X _c
Y _l = R · sin (θ) cos (φ) −Y _c
Z _l = R · sin (θ) −Z _c (3)

^{_{R l = {X l 2 +}} Y l 2 + Z l 2} 1/2
θ _l = arccos {X _l / {(X _l ² + Y _l ² ) ^1/2 }}
φ _l = arcsin {Z _l / {(X _l ² + Y _l ² ) ^1/2 }} (4)

  It is determined whether all the light source parameters have been processed (S1006). When it is determined that all the light source parameters are not processed (S1006). The next light source parameter is acquired (S1001). When all the light source parameters have been processed (S1006), the light source parameter changing process is terminated. The light source parameter changing process is an example, and the present invention is not limited to this. For example, the light source parameter is held as an all-around image and its depth information as image information. Divide the image for each light source. Rotation, parallel movement, enlargement, and reduction processing are performed on the divided images of the light sources so that the directions of the light sources viewed from the combination position coincide. You may process in this way.

  FIG. 11 shows a flowchart of the lighting process S904 in the present embodiment. Based on the shape of the foreground image, light propagation information such as self-shielding information and shielding information calculated in advance is acquired (S1101). The shape of the foreground image is estimated based on depth information obtained using stereo matching by taking a stereo image of the foreground. A table of light source parameters subjected to the change process in S903 is acquired (S1102). Based on the acquired light source parameter and light propagation information, the radiance of each pixel is calculated (S1103). Based on the calculated radiance, the color of each pixel of the foreground image is changed (S1104). Note that calculation of light propagation and calculation of radiance are performed using Non-Patent Document 1, Non-Patent Document 2, and the like.

FIG. 12 shows a flowchart of the synthesis process S905 in the present embodiment. Pixels of the foreground image are acquired (S1201). A background pixel at a position where the acquired pixel is synthesized is acquired (S1202). Assuming that the pixel value of the foreground image is F, the α value of the foreground image is α, the pixel value of the background image is B, and the pixel value C after the combination is combined, the combining process is performed based on Expression (5) (S1203).
C = Fα + B (1−α) (5)

  By configuring and processing in this way, a lighting effect corresponding to the synthesis position can be easily applied to the foreground image, and even if the video is synthesized, the synthesized image 406 is generated as a natural video. Thereafter, the composite image 406 is reproduced according to the user's designation.

[Second Embodiment]
The following second embodiment will be described. In addition, the description according to the first embodiment is omitted. In the second embodiment, a still image is used as the foreground image, and the user designates a desired synthesis position by performing a drag operation or the like. At that time, a lighting effect is applied to the foreground image according to the change in the composition position. Further, it is determined whether or not to change the light source parameter based on the intensity of the light source.

FIG. 13 shows an example of a light source parameter in the second embodiment. The description according to the first embodiment is omitted. Reference numeral 1301 denotes illuminance information of the light source. In the present embodiment, Expression (6) is used to convert the RGB value of each pixel of the all-around image into an XYZ value, and the Y value is acquired as illuminance.
X = 0.412453R + 0.35758G + 0.180423B
Y = 0.212671R + 0.71516G + 0.072169B
Z = 0.019334R + 0.119193G + 0.950227B (6)
This is an example, and the present invention is not limited to this. For example, it may be converted into HSV and the value of V may be illuminance.

  FIG. 14 shows a flowchart of the video editing apparatus according to the second embodiment. The description according to the first embodiment is omitted. A still image selected by the user as a foreground image is acquired from the recording medium 104 (S1401).

  FIG. 15 shows a flowchart of the video editing process S806 in the second embodiment. The description according to the first embodiment is omitted. The composite position is acquired from the user input from the input device 105 (S1501).

FIG. 16 shows a flowchart of the light source parameter change processing S903 in the second embodiment. The description according to the first embodiment is omitted. Using formulas (1) and (2), the distance to the light source is calculated from the spatial coordinates of the composite position acquired in step S906. From the calculated distance and the light source parameter, the illuminance viewed from the spatial coordinates of the combined position is calculated using Equation (7) (S1601). The illuminance of the light source parameter is E, and the calculated illuminance is _El .
_{E l = E · (R ·} R) / (R l · R l) ... (7)
This is an example, and the present invention is not limited to this. The illuminance E of the light source parameter may be used as a criterion. It is determined whether the calculated illuminance is greater than or equal to a predetermined threshold (S1602). If the calculated illuminance is greater than or equal to the threshold (S1602), the light source parameter is changed (S1005). In S1005, the position information of the light source parameter is changed, but this is an example, and the present invention is not limited to this. You may change the illumination intensity information of a light source parameter based on Formula (5). If the calculated illuminance is less than the threshold (S1602), it is determined whether all light source parameters have been processed (S1006).

  By configuring and processing in this way, a lighting effect corresponding to the combining position can be easily applied to the foreground image, and a natural image can be obtained even if the images are combined.

[Third Embodiment]
Next, a third embodiment will be described. In addition, the description according to the first embodiment and the second embodiment is omitted. In the third embodiment, whether to perform the calculation process in S907, the determination process in S908, and the light source parameter change process in S909 is determined according to the amount of change in the combined position.

  FIG. 17 shows a flowchart of the video editing process in the third embodiment. In addition, the description according to the first embodiment and the second embodiment is omitted. As shown in the figure, the third embodiment includes a step (S1701) of determining whether or not the amount of change in the composite position is greater than or equal to a threshold (whether or not the amount of change in the composite position is greater than or equal to a predetermined range). If it is determined that it is less than the threshold value, that is, the amount of change in the composite position is within a predetermined range (No in S1701), lighting processing is performed (S904). And when it determines with more than a threshold value (S1701 is Yes), a light source parameter change process is performed (S903).

  By configuring and processing in this way, a lighting effect corresponding to the combining position can be easily applied to the foreground image, and a natural image can be obtained even if the images are combined.

[Fourth Embodiment]
Next, a fourth embodiment will be described. In addition, the description according to the first to third embodiments is omitted. In the fourth embodiment, as a preparatory process before video editing, a large number of light source parameters corresponding to the synthesis position are created in advance. During the video editing process, the light source parameter corresponding to the synthesis position is acquired and the lighting effect is applied.

  FIG. 18 shows a flowchart of the preparation process in the fourth embodiment. In addition, the description according to the first to third embodiments is omitted. The positions sampled at regular intervals on the background image are acquired as candidate positions for generating light source parameters in advance (S1801). This is an example, and the present invention is not limited to this. For example, a position designated in advance by the user may be set as the creation candidate position. Moreover, you may create only the point where distance is below fixed. The changed light source parameter is recorded on the recording medium (S1802).

  FIG. 19 shows an example of light source parameters recorded on a recording medium in the fourth embodiment. FIG. 5A is a table in which a table of light source parameters corresponding to the synthesis position is associated. Reference numeral 1901 denotes an x coordinate on the background image at the synthesis position. The unit is pixel. Reference numeral 1902 denotes a y-coordinate on the background image at the synthesis position. The unit is pixel. Reference numeral 1903 denotes a table ID for uniquely determining the light source parameter table. FIG. 4B is a light source parameter table.

  The preparatory process in this embodiment will be described with reference to FIG. It is determined whether light source parameters have been generated at all candidate positions (S1803). When it is determined that the light source parameters are not generated at all candidate positions (No in S1803), the next candidate position is acquired (S1801). If it is determined that light source parameters have been created at all candidate positions (Yes in S1803), the preparation process ends.

  FIG. 20 shows a flowchart of the video editing process in the fourth embodiment. In addition, the description according to the first to third embodiments is omitted. Among the light source parameters created in the preparation process, the light source parameter created at the point closest to the combining position is acquired from the recording medium (S2001). In the fourth embodiment, the light source parameter closest to the combination position is acquired from the recording medium. However, this is an example, and the present invention is not limited to this. For example, a plurality of light source parameters created in the vicinity of the synthesis position may be acquired, and the light source parameters may be generated by linear interpolation according to the distance to each light source parameter.

  By configuring and processing in this way, a lighting effect corresponding to the combining position can be easily applied to the foreground image, and a natural image can be obtained even if the images are combined.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (6)

Applying lighting effects to the foreground picture image data, an image processing apparatus for synthesizing the background image data,
Storage means for storing the background image data, light source information representing a position of at least one light source with respect to the imaging device when the background image data is captured, and moving image data of the subject as the foreground image data;
A composite position acquisition means for acquiring an initial composite position of the subject in the background image data ;
The synthesized moving image that generates the synthesized moving image data by sequentially generating a synthesized frame of the subject and the background image data, in which the obtained initial synthesized position is the initial arrangement position of the subject in the moving image data. Generating means,
The synthesized moving image generating means
Spatial coordinate acquisition means for acquiring the spatial coordinates of the composite position of the subject in the background image data each time the composite frame is generated ;
Calculation means for calculating a distance between the spatial coordinates of the subject acquired by the spatial coordinate acquisition means and each light source indicated by the light source information acquired from the storage means ;
Among the light sources indicated by the light source information acquired from the storage unit, the position of the light source whose distance calculated by the calculation unit exceeds a preset threshold is not changed, and the distance calculated by the calculation unit is equal to or less than the threshold. For the light source to be obtained, the position of the light source viewed from the subject, the changing means for changing the position of the corresponding light source in the acquired light source information according to the obtained position ;
Lighting means for applying a lighting effect to the subject using the position of each light source indicated by the light source information after the change processing by the changing means;
A video processing apparatus , comprising: a compositing unit that generates the composite frame by compositing the subject on which the lighting effect has been applied by the lighting unit with the subject arrangement position in the background image data . A video processing device that applies a lighting effect to foreground image data and combines it with background image data,
Storage means for storing the background image data, light source information representing a position of at least one light source with respect to the imaging device when the background image data is captured, and moving image data of the subject as the foreground image data;
A composite position acquisition means for acquiring an initial composite position of the subject in the background image data;
The synthesized moving image that generates the synthesized moving image data by sequentially generating a synthesized frame of the subject and the background image data, in which the obtained initial synthesized position is the initial arrangement position of the subject in the moving image data. Generating means,
The synthesized moving image generating means
Spatial coordinate acquisition means for acquiring the spatial coordinates of the composite position of the subject in the background image data each time the composite frame is generated;
The difference between the spatial coordinates of the subject acquired by the spatial coordinate acquisition means and the spatial coordinates of the subject in the immediately preceding frame is calculated, and it is determined whether or not the difference is equal to or greater than a preset first threshold value. Determination means to perform,
When the determination unit determines that the difference is equal to or greater than the first threshold, the distance between the spatial coordinates of the subject in the composite frame and each light source indicated by the light source information acquired from the storage unit is determined. A calculating means for calculating;
Among the light sources indicated by the light source information acquired from the storage means, the position of the light source whose distance calculated by the calculation means exceeds a preset second threshold is not changed, and the distance calculated by the calculation means is For a light source that is equal to or less than a second threshold value, a position of the light source viewed from the subject is obtained, and a changing unit that changes the position of the corresponding light source in the acquired light source information according to the obtained position;
When the determination unit determines that the difference is not equal to or greater than the first threshold, the lighting effect is applied to the subject using the position of each light source indicated by the light source information used in the immediately preceding frame,
A lighting unit that applies a lighting effect to the subject using the position of each light source indicated by the light source information obtained by the changing unit when the determination unit determines that the difference is equal to or greater than the first threshold. When,
Synthesizing means for generating the synthesized frame by synthesizing the subject on which the lighting effect has been applied by the writing means with the subject arrangement position in the background image data.
A video processing apparatus characterized by that. Storage means for storing background image data, light source information representing the position of at least one light source with respect to the imaging device when the background image data is captured, and moving image data of the subject as foreground image data, applying lighting effects to images data, a control method of an image processing apparatus for synthesizing the background image data,
A composite position acquisition unit that acquires an initial composite position of the subject in the background image data; and
The synthesized moving image generating means sequentially generates a synthesized frame of the subject and the background image data by using the acquired initial synthesized position as the initial arrangement position of the subject in the moving image data. A synthetic moving image generation process for generating data,
The synthetic moving image generating step includes
A spatial coordinate acquisition step of acquiring a spatial coordinate of the composite position of the subject in the background image data each time the spatial coordinate acquisition unit generates the composite frame ;
A calculating step of calculating a distance between the spatial coordinates of the subject acquired in the spatial coordinate acquiring step and each light source indicated by the light source information acquired from the storage unit ;
Distance change means, among the light source indicated by the light source information obtained from the storage unit, the position of the light source exceeds a threshold distance calculated in the calculation step is set in advance without changing is calculated in the calculating step For a light source that is less than or equal to the threshold value, the position of the light source viewed from the subject is determined, and the position of the corresponding light source in the acquired light source information is changed according to the determined position ; and
A writing process in which a lighting unit applies a lighting effect to the subject using the position of each light source indicated by the light source information after the change process in the change process ;
Synthesis hand stage, the subject lighting effect by the lighting step has been performed, characterized in that it comprises a synthesis step of generating the composite frame by synthesizing the arrangement position of the object in the background image data Control method of video processing apparatus. Storage means for storing background image data, light source information representing the position of at least one light source with respect to the imaging device when the background image data is captured, and moving image data of the subject as foreground image data, A control method of a video processing apparatus that performs a lighting effect on image data and combines the image data with the background image data,
A composite position acquisition unit that acquires an initial composite position of the subject in the background image data; and
The synthesized moving image generating means sequentially generates a synthesized frame of the subject and the background image data by using the acquired initial synthesized position as the initial arrangement position of the subject in the moving image data. A synthetic moving image generation process for generating data,
The synthetic moving image generating step includes
A spatial coordinate acquisition step of acquiring a spatial coordinate of the composite position of the subject in the background image data each time the spatial coordinate acquisition unit generates the composite frame;
The determination unit calculates a difference between the spatial coordinates of the subject acquired in the spatial coordinate acquisition step and the spatial coordinates of the subject in the immediately preceding frame, and whether the difference is equal to or greater than a preset first threshold value. A determination step of determining whether or not,
When the calculation unit determines that the difference is greater than or equal to the first threshold value in the determination step, each light source indicated by the spatial coordinates of the subject in the composite frame and the light source information acquired from the storage unit A calculation step of calculating a distance between and
Of the light sources indicated by the light source information acquired from the storage means, the changing means does not change the position of the light source whose distance calculated in the calculation step exceeds a preset second threshold value, and is calculated in the calculation step. For a light source whose distance is equal to or less than the second threshold, a position of the light source viewed from the subject is obtained, and a change step of changing the position of the corresponding light source in the obtained light source information according to the obtained position;
Writing means
When the determination step determines that the difference is not equal to or greater than the first threshold, the lighting effect is applied to the subject using the position of each light source indicated by the light source information used in the immediately preceding frame,
A lighting step of applying a lighting effect to the subject using the position of each light source indicated by the light source information obtained in the changing step when the difference is determined to be greater than or equal to the first threshold value in the determining step. When,
A compositing unit that generates the composite frame by compositing the subject on which the lighting effect has been applied in the lighting step with the subject arrangement position in the background image data.
A control method for a video processing apparatus. The program for making a computer function as each process of the control method of the video processing apparatus of Claim 3 or 4 by making a computer read and run. A computer-readable storage medium storing the program according to claim 5 .

Images