JP2013149219A - Video processor and its control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily obtain a natural synthetic video by giving lighting effects corresponding to a synthetic position to an image for foreground.SOLUTION: A projection part 412 projects the coordinates of a space generated from the depth information of an image for background to the coordinates of an image for background to generate projection information. A spatial coordinate acquisition part 413 acquires the spatial coordinates of a synthetic position from the projection information and the synthetic position. A calculation part 413 calculates a distance between a light source and the spatial coordinates of the synthetic position or the illuminance of the light source when viewed from the spatial coordinates of the synthetic position from a light source parameter holding the information of the position and illuminance of the light source when viewed from the position where the image for background has been photographed and the spatial coordinates of the synthetic position. A light source parameter changing part 415 changes a light source parameter in accordance with the spatial coordinates of the synthetic position and the calculated value. A lighting part 416 gives lighting effects to the image for foreground by using the light source parameter, and a synthesis part 417 synthesizes the image for foreground given the lighting effects with the synthetic position set on the background image.

Description

  The present invention relates to a video processing technique, and more particularly, to a video processing technique for applying a lighting effect to a foreground image in accordance with a background image and a synthesis position during image synthesis.

  It is common in video editing to cut out a subject from a foreground image and combine it with a background image. For example, a typical example is that a subject is cut out with a chroma key from a video shot with a blue background and then combined with a different video. At that time, in order to make the synthesized video natural, it is necessary that the lighting of the background video and the lighting of the foreground video match. In order to match the lighting, a lighting effect is applied to a real image using CG.

  As lighting using CG, lighting is often performed using a ray tracing or radiosity that makes a lighting environment a 3D model and calculates a light path.

  In addition, image-based lighting is often performed in which a background image of a background image is captured and a lighting effect is applied using the captured environment image as a light source image (Non-Patent Document 1, Non-Patent Document). 2). In addition, a light source in the distance is performed by image-based lighting, and a lighting effect is performed using a 3D model for a light source in the vicinity. (Non Patent Literature 3)

Peter-Pike Sloan and Ben Luna and John Snyder ”Local, Deformable Precomputed Radiance Transfer” SIGGRAPH ACM ACM SIGGRAPH 2005 Papers Zhong Ren and Rui Wang and John Snyder and Kun Zhou and Xinguo Liu and Bo Sun and Peter-Pike Sloan and Hujun Bao and Qunsheng Peng and Baining Guo ”Real-time Soft Shadows in Dynamic Scenes using Spherical Harmonic Exponentiation”, SIGGRAPH ACM SIGGRAPH 2006 Papers Prof. Shiho Furuya, Takayuki Ito “Image-based Drying of Scenes Including Neighborhood Light Sources” Information Processing Society of Japan Annual Conference

  In the conventional method using image-based lighting, when the synthesis position is changed, it is necessary to prepare an all-around image corresponding to the synthesis position in advance. This is because when the composition position changes, the positional relationship between the neighboring light source and the composition position changes greatly, and the lighting changes greatly. If an all-around image corresponding to the synthesis position is not used, an unnatural lighting effect is applied. If the positions to be combined are determined in advance, a finite number of all-around images may be prepared. Since there are an infinite number of combining positions, it is impossible to prepare all the surrounding images according to the combining positions in advance.

  In the conventional method using ray tracing or radiosity, the lighting environment in the field where the background image is taken must be manually modeled as a 3D model, which is troublesome. Ray tracing and radiosity perform lighting by calculating ray paths, but in order to achieve more natural lighting, the ray paths branch many times and the amount of calculations increases. .

  The present invention seeks to provide a technique that solves the above-described problems and that can easily provide a lighting effect according to a synthesis position.

In order to solve this problem, for example, a video processing apparatus of the present invention has the following configuration. That is,
A video processing device that applies a lighting effect to a foreground image and combines it with a background image,
Storage means for storing a foreground image, a background image as a background of the foreground image, and depth information indicating the depth of the background image;
Combined position acquisition means for acquiring a combined position of the foreground image on the background image;
Projecting means for projecting the coordinates of the space generated from the depth information of the background image onto the coordinates of the background image;
Spatial coordinate acquisition means for acquiring spatial coordinates of the composite position from projection information by the projection means and the composite position;
The distance between the light source and the spatial coordinates of the combined position, or the combined position of the light source parameter that holds information on the light source position and illuminance as viewed from the position where the background image was captured and the spatial coordinates of the combined position, or Calculating means for calculating the illuminance of the light source as seen from the spatial coordinates;
Light source parameter changing means for changing a light source parameter according to the spatial coordinates of the combined position and the calculated value;
Lighting means for applying a lighting effect to the foreground image using the light source parameter;
Synthesizing means for synthesizing the foreground image on which the lighting effect has been applied by the lighting means at the synthesis position of the background image.

  According to the present invention, it is possible to easily apply a lighting effect corresponding to the synthesis position to the foreground image, and to make a natural video even if the video is synthesized.

The hardware block diagram of the video editing apparatus in 1st Embodiment. The figure of the external appearance of the photo frame in a 1st embodiment. The figure which shows the screen shot before and behind changing the synthetic | combination position of a to-be-photographed object in the video editing screen in 1st Embodiment. The functional block diagram of the video editing apparatus in 1st Embodiment. The figure showing the light source parameter table in a 1st embodiment. The figure showing synthetic position coordinate data in a 1st embodiment. The figure showing the data of projection information in a 1st embodiment. 5 is a flowchart of processing performed by the video editing apparatus according to the first embodiment. The flowchart of the video editing process in 1st Embodiment. The flowchart of the light source parameter change process in 1st Embodiment. The flowchart of the lighting process in 1st Embodiment. The flowchart of the synthetic | combination process in 1st Embodiment. The figure showing the light source parameter table in 2nd Embodiment. The flowchart of the video editing apparatus in 2nd Embodiment. The flowchart of the video editing process in 2nd Embodiment. The flowchart of the light source parameter change process in 2nd Embodiment. The flowchart of the video editing process in 3rd Embodiment. The flowchart of the preparation process in 4th Embodiment. The figure showing the light source parameter table in 4th Embodiment. The flowchart of the video editing process in 4th Embodiment.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
Hereinafter, the present embodiment will be described with reference to the drawings.

  FIG. 1 shows a hardware configuration diagram for realizing the present embodiment. The CPU 101 is related to all the processes of each configuration, reads and interprets the instructions stored in the ROM 102 and the RAM 103 in order, and processes the processes according to the results. The ROM 102 and RAM 103 provide the CPU 101 with programs, data, work areas, and the like necessary for the processing. The storage medium 104 is a storage medium (or device) that stores image data and the like, and is, for example, a hard disk, a CF card, an SD card, a USB memory, a memory card, or the like. The input device 105 is an input device such as a touch screen, a keyboard, and a mouse, and receives instructions from the user. In general, a liquid crystal display is widely used as the output device 106, and displays images and characters. Further, a touch panel function may be provided, and in that case, the input device 105 can be handled.

  In the present embodiment, an example in which the present invention is applied to a photo frame as an embodiment of the video editing apparatus will be described.

  FIG. 2 is a front view of a photo frame which is a video processing apparatus in the present embodiment. The touch screen display 201 is a display for displaying an image and is an input device that receives a user input. The touch screen display 201 corresponds to the output device 106 and the input device 105 in FIG. 1, and can simultaneously detect the coordinates of a plurality of inputs. Reference numeral 202 denotes a power button.

  FIG. 3 shows screenshots before and after the subject composition position is changed on the video editing screen in the present embodiment. Reference numeral 301 denotes a background image, and 302 denotes a synthesized foreground image. A large number of background images and foreground images are distinguished and stored in the storage medium 104, and the illustrated background image 301 and foreground image 302 are those specified by the user. An α value (transparency) is added to the foreground image, and a desired subject is cut out. In the present embodiment, the foreground image 302 is a moving image, and when it is reproduced, the subject moves and its composition position changes. Along with the change of the composition position, the foreground image 303 is subjected to a lighting effect corresponding to the composition position. In this embodiment, since the foreground image is a moving image, the synthesis position is changed. However, this is an example, and the present invention is not limited to this. For example, the synthesis position of the foreground image may be changed by the user dragging the foreground image. Alternatively, the scale of the foreground image may be changed, the background image may be moved, or the scale may be changed according to a user operation.

  FIG. 4 shows a functional block diagram of the apparatus of the present embodiment. In the drawing, 401 is a foreground image. Reference numeral 402 denotes a background image. Reference numeral 403 denotes background image depth information representing the depth of each pixel of the background image 402. Depth information is recorded in the RGB component of each pixel in an image format with the same resolution as the background image 402. Also, the background image depth information is generated by photographing the background image as a stereo image and using the stereo matching method. Note that the generation of depth information using the stereo matching method is an example, and the present invention is not limited to this. For example, the straight line and vanishing point of the background image may be detected to estimate the depth information. Also, the user may specify. Reference numeral 404 denotes a light source parameter representing position information or illuminance information of the light source viewed from the position where the background image is captured. The position information of the light source is acquired by generating an all-around image of the position where the background image is taken and its depth information using Japanese Patent Application Laid-Open No. 2010-181826. As the illuminance information of the light source, the entire surrounding image at the position where the background image is captured is captured as an HDR image, and the luminance value of the image is acquired as the illuminance information. These are only examples, and the present invention is not limited to these. For example, the position information of the light source may be specified by the user, or knowledge information up to the light source may be acquired using a distance measuring sensor. Further, the illuminance information of the light source may be specified by the user or measured with an illuminometer.

  FIG. 5 shows an example of a light source parameter table. Reference numeral 501 denotes an ID for identifying each light source. An area having a certain luminance value or more in the entire surrounding image is set as a light source, and its center of gravity is set as the position of the light source. Reference numeral 502 denotes distance information from the background image shooting position to the light source. The unit is meters. When the distance information cannot be acquired, “−1” is held as being present at infinity. Reference numerals 503 and 504 denote the direction of the light source as viewed from the background image photographing position, which are an azimuth angle and an elevation angle. The unit is degrees. In this embodiment, an area having a certain luminance value in the entire surrounding image is regarded as a light source area. However, this is an example, and the present invention is not limited to this. For example, each pixel may be treated as a light source. Moreover, in this embodiment, although the light source position is hold | maintained by the polar coordinate system, this is an example and is not restricted to this. For example, you may hold | maintain with a rectangular coordinate system.

  Returning to the description of FIG. 405 in the figure is the result of calculating the light propagation in advance. It is a light propagation prior calculation result. The calculation is performed using Non-Patent Document 1 or Non-Patent Document 2 described above. A composite image 406 is an editing result of the video editing apparatus according to the present embodiment. The foreground image 401, the background image 402, the background image depth information 403, the light source parameter 404, the light propagation pre-calculation result 405, and the composite image 406 are recorded on the recording medium 104. Reference numeral 410 denotes a video editing apparatus according to the present embodiment. Reference numeral 411 denotes a composite position acquisition unit that acquires coordinates on the background image as a composite position. The coordinates on the background image designated by the user using the input device 105 are acquired as the synthesis position. In addition, the coordinates on the background image of the pixel on the foreground image that is the reference point for synthesis designated in advance by the user are acquired as the synthesis position. Although the user designates the reference point for synthesis, this is merely an example and the present invention is not limited to this. For example, face detection may be performed and the position of the face may be used as a reference for the combined position. Further, the lowest part of the opaque part of the foreground image may be used as a reference for the composition position.

  FIG. 6 shows an example of the combined position information. Reference numeral 601 denotes an X coordinate on the background image at the synthesis position, and reference numeral 602 denotes a Y coordinate.

  Next, the present embodiment will be described with reference to FIG. Reference numeral 412 denotes a projection unit that generates projection information obtained by projecting space coordinates generated from the depth information of the background image onto two-dimensional coordinates on the background image. Projection is performed using a perspective projection method.

  FIG. 7 shows an example of projection information. Reference numeral 701 denotes two-dimensional coordinates on the background image. Reference numeral 702 denotes an X coordinate. Reference numeral 703 denotes a Y coordinate. Reference numeral 711 denotes spatial coordinates generated from the depth information of the background image. Reference numeral 712 denotes an X coordinate. Reference numeral 713 denotes a Y coordinate. Reference numeral 714 denotes a Z coordinate. This is an example, and the present invention is not limited to this. For example, the X coordinate, Y coordinate, and Z coordinate corresponding to each pixel on the background image may be stored as image information as R, G, and B information.

  Next, the present embodiment will be described with reference to FIG. Reference numeral 413 denotes a spatial coordinate acquisition unit that acquires spatial coordinates corresponding to the coordinates on the background image acquired by the composite position acquisition unit 411 based on the projection information generated by the projection unit 412.

  Reference numeral 414 denotes a light source distance or illuminance calculator that calculates the illuminance of the light source viewed from the distance between the spatial coordinates of the combined position and the light source or the spatial coordinates of the combined position. Details will be described later. Reference numeral 415 denotes light source parameter changing means for changing the light source parameter. Details will be described later. A lighting unit 416 applies a lighting effect to the foreground image based on the light source parameter changed by the light source parameter changing unit 415. Details will be described later. Here, the lighting effect may be applied to the foreground image after performing the process of removing the shadow portion. The removal of the shadow portion is performed using a technique disclosed in Japanese Patent Application Laid-Open No. 2010-135996. Reference numeral 417 denotes a combining unit that combines the foreground image with the lighting effect applied by the writing unit 416 at the combining position acquired by 411 with the background image. The synthesis result of the synthesis unit 416 is output to the recording medium 104 and the output device 106.

  FIG. 8 shows a flowchart of a control processing procedure executed by the CPU 101 of the video editing apparatus in the present embodiment. When the video editing apparatus is activated, the foreground image selected by the user is acquired from the recording medium 104 (S801). In the present embodiment, a moving image is acquired. Next, the background image selected by the user is acquired from the recording medium 104 (S802). An initial composite position is acquired from user input from the input device 105 (S803). The depth information corresponding to the background image acquired in S802 is acquired from the recording medium 104 (S804). A light source parameter table corresponding to the background image acquired in S802 is acquired from the recording medium 104 (S805). Video editing processing is started (S806). Details of this video editing process will be described later. The edited image is recorded on the recording medium 104 (S807). It is determined whether an end operation has been performed (S808). If it is determined that the end operation has not been performed (S808), a foreground image is acquired (S801). If it is determined that the end operation has been performed (S808), the video editing apparatus is ended.

  FIG. 9 shows a flowchart of the video editing process of S806 in the present embodiment. When the video editing process is started, the coordinates of the space generated from the background image depth information acquired in S804 are projected onto the two-dimensional coordinates on the background image to generate projection information (S901). The first frame of the foreground image acquired in S801 is acquired (S902). A light source parameter change process is performed (S903). Details of the light source parameter changing process will be described later. Writing processing is performed (S904). Details of the lighting process will be described later. A synthesis process is performed (S905). Details of the synthesis process will be described later. The composite image is displayed on the display device 106 (S906). It is determined whether a video editing end operation has been performed (S907). If it is determined that the end operation has been performed (S907), the video editing process is ended. If it is determined that the ending operation has not been performed (S907), it is determined whether or not there is a next frame in the moving image used as the foreground image (S908). If it is determined that there is a next frame (S908), the next frame image of the moving image is acquired as the foreground image (S909). A predetermined pixel in the foreground is detected, and the composition position is changed only by a change in position with the previous frame (S910). If it is determined that there is no next frame (S908), the video editing process is terminated.

  FIG. 10 shows a flowchart of the light source parameter change processing 903 in the present embodiment. When the light source parameter changing process is started, the spatial coordinates of the composite position are acquired from the projection information generated in S901 (S1001). Light source parameters are acquired from the light source parameter table acquired in S805 (S1002). From the spatial coordinates of the combined position acquired in S1001 and the light source parameters acquired in S1002, the distance of the light source is calculated based on Equations 1 and 2 (S1003).

The distance information 502 of the light source parameter is R, the azimuth angle 503 is θ, and the elevation angle 504 is φ. The x coordinate of the space coordinates of the composite position is X _c , the y coordinate is Y _c , and the z coordinate is Z _c . A distance calculating and R _l.
X ₁ = R · cos (θ) cos (φ) −X _c
Y _l = R · sin (θ) cos (φ) −Y _c
Z _l = R · sin (θ) −Z _c (1)

^{_{R l = {X l 2 +}} Y l 2 + Z l 2} 1/2 ... (2)

  In S1004, it is determined whether or not the value calculated in S1003 is equal to or less than a threshold value. Only when it is determined that it is not less than or equal to the threshold value, light source parameter change processing is performed based on the spatial coordinates of the composite position acquired in S1001 and the light source parameter acquired in S1002 (S1005). The parameter is changed based on the equations (3) and (4).

The distance information 502 of the light source parameter is R, the azimuth angle 503 is θ, and the elevation angle 504 is φ. The x coordinate of the space coordinates of the composite position is X _c , the y coordinate is Y _c , and the z coordinate is Z _c . The x coordinate of the light source after changing the light source parameters is X _l , the y coordinate is Y _l , and the z coordinate is Z _l . The distance information 502 after the light source parameter change is R _l , the azimuth angle 503 is θ _l , and the elevation angle 504 is φ _l .
X ₁ = R · cos (θ) cos (φ) −X _c
Y _l = R · sin (θ) cos (φ) −Y _c
Z _l = R · sin (θ) −Z _c (3)

^{_{R l = {X l 2 +}} Y l 2 + Z l 2} 1/2
θ _l = arccos {X _l / {(X _l ² + Y _l ² ) ^1/2 }}
φ _l = arcsin {Z _l / {(X _l ² + Y _l ² ) ^1/2 }} (4)

  It is determined whether all the light source parameters have been processed (S1006). When it is determined that all the light source parameters are not processed (S1006). The next light source parameter is acquired (S1001). When all the light source parameters have been processed (S1006), the light source parameter changing process is terminated. The light source parameter changing process is an example, and the present invention is not limited to this. For example, the light source parameter is held as an all-around image and its depth information as image information. Divide the image for each light source. Rotation, parallel movement, enlargement, and reduction processing are performed on the divided images of the light sources so that the directions of the light sources viewed from the combination position coincide. You may process in this way.

  FIG. 11 shows a flowchart of the lighting process S904 in the present embodiment. Based on the shape of the foreground image, light propagation information such as self-shielding information and shielding information calculated in advance is acquired (S1101). The shape of the foreground image is estimated based on depth information obtained using stereo matching by taking a stereo image of the foreground. A table of light source parameters subjected to the change process in S903 is acquired (S1102). Based on the acquired light source parameter and light propagation information, the radiance of each pixel is calculated (S1103). Based on the calculated radiance, the color of each pixel of the foreground image is changed (S1104). Note that calculation of light propagation and calculation of radiance are performed using Non-Patent Document 1, Non-Patent Document 2, and the like.

FIG. 12 shows a flowchart of the synthesis process S905 in the present embodiment. Pixels of the foreground image are acquired (S1201). A background pixel at a position where the acquired pixel is synthesized is acquired (S1202). Assuming that the pixel value of the foreground image is F, the α value of the foreground image is α, the pixel value of the background image is B, and the pixel value C after the combination is combined, the combining process is performed based on Expression (5) (S1203).
C = Fα + B (1−α) (5)

  By configuring and processing in this way, a lighting effect corresponding to the synthesis position can be easily applied to the foreground image, and even if the video is synthesized, the synthesized image 406 is generated as a natural video. Thereafter, the composite image 406 is reproduced according to the user's designation.

[Second Embodiment]
The following second embodiment will be described. In addition, the description according to the first embodiment is omitted. In the second embodiment, a still image is used as the foreground image, and the user designates a desired synthesis position by performing a drag operation or the like. At that time, a lighting effect is applied to the foreground image according to the change in the composition position. Further, it is determined whether or not to change the light source parameter based on the intensity of the light source.

FIG. 13 shows an example of a light source parameter in the second embodiment. The description according to the first embodiment is omitted. Reference numeral 1301 denotes illuminance information of the light source. In the present embodiment, Expression (6) is used to convert the RGB value of each pixel of the all-around image into an XYZ value, and the Y value is acquired as illuminance.
X = 0.412453R + 0.35758G + 0.180423B
Y = 0.212671R + 0.71516G + 0.072169B
Z = 0.019334R + 0.119193G + 0.950227B (6)
This is an example, and the present invention is not limited to this. For example, it may be converted into HSV and the value of V may be illuminance.

  FIG. 14 shows a flowchart of the video editing apparatus according to the second embodiment. The description according to the first embodiment is omitted. A still image selected by the user as a foreground image is acquired from the recording medium 104 (S1401).

  FIG. 15 shows a flowchart of the video editing process S806 in the second embodiment. The description according to the first embodiment is omitted. The composite position is acquired from the user input from the input device 105 (S1501).

FIG. 16 shows a flowchart of the light source parameter change processing S903 in the second embodiment. The description according to the first embodiment is omitted. Using formulas (1) and (2), the distance to the light source is calculated from the spatial coordinates of the composite position acquired in step S906. From the calculated distance and the light source parameter, the illuminance viewed from the spatial coordinates of the combined position is calculated using Equation (7) (S1601). The illuminance of the light source parameter is E, and the calculated illuminance is _El .
_{E l = E · (R ·} R) / (R l · R l) ... (7)
This is an example, and the present invention is not limited to this. The illuminance E of the light source parameter may be used as a criterion. It is determined whether the calculated illuminance is greater than or equal to a predetermined threshold (S1602). If the calculated illuminance is greater than or equal to the threshold (S1602), the light source parameter is changed (S1005). In S1005, the position information of the light source parameter is changed, but this is an example, and the present invention is not limited to this. You may change the illumination intensity information of a light source parameter based on Formula (5). If the calculated illuminance is less than the threshold (S1602), it is determined whether all light source parameters have been processed (S1006).

  By configuring and processing in this way, a lighting effect corresponding to the combining position can be easily applied to the foreground image, and a natural image can be obtained even if the images are combined.

[Third Embodiment]
Next, a third embodiment will be described. In addition, the description according to the first embodiment and the second embodiment is omitted. In the third embodiment, whether to perform the calculation process in S907, the determination process in S908, and the light source parameter change process in S909 is determined according to the amount of change in the combined position.

  FIG. 17 shows a flowchart of the video editing process in the third embodiment. In addition, the description according to the first embodiment and the second embodiment is omitted. As shown in the figure, the third embodiment includes a step (S1701) of determining whether or not the amount of change in the composite position is greater than or equal to a threshold (whether or not the amount of change in the composite position is greater than or equal to a predetermined range). If it is determined that it is less than the threshold value, that is, the amount of change in the composite position is within a predetermined range (No in S1701), lighting processing is performed (S904). And when it determines with more than a threshold value (S1701 is Yes), a light source parameter change process is performed (S903).

  By configuring and processing in this way, a lighting effect corresponding to the combining position can be easily applied to the foreground image, and a natural image can be obtained even if the images are combined.

[Fourth Embodiment]
Next, a fourth embodiment will be described. In addition, the description according to the first to third embodiments is omitted. In the fourth embodiment, as a preparatory process before video editing, a large number of light source parameters corresponding to the synthesis position are created in advance. During the video editing process, the light source parameter corresponding to the synthesis position is acquired and the lighting effect is applied.

  FIG. 18 shows a flowchart of the preparation process in the fourth embodiment. In addition, the description according to the first to third embodiments is omitted. The positions sampled at regular intervals on the background image are acquired as candidate positions for generating light source parameters in advance (S1801). This is an example, and the present invention is not limited to this. For example, a position designated in advance by the user may be set as the creation candidate position. Moreover, you may create only the point where distance is below fixed. The changed light source parameter is recorded on the recording medium (S1802).

  FIG. 19 shows an example of light source parameters recorded on a recording medium in the fourth embodiment. FIG. 5A is a table in which a table of light source parameters corresponding to the synthesis position is associated. Reference numeral 1901 denotes an x coordinate on the background image at the synthesis position. The unit is pixel. Reference numeral 1902 denotes a y-coordinate on the background image at the synthesis position. The unit is pixel. Reference numeral 1903 denotes a table ID for uniquely determining the light source parameter table. FIG. 4B is a light source parameter table.

  The preparatory process in this embodiment will be described with reference to FIG. It is determined whether light source parameters have been generated at all candidate positions (S1803). When it is determined that the light source parameters are not generated at all candidate positions (No in S1803), the next candidate position is acquired (S1801). If it is determined that light source parameters have been created at all candidate positions (Yes in S1803), the preparation process ends.

  FIG. 20 shows a flowchart of the video editing process in the fourth embodiment. In addition, the description according to the first to third embodiments is omitted. Among the light source parameters created in the preparation process, the light source parameter created at the point closest to the combining position is acquired from the recording medium (S2001). In the fourth embodiment, the light source parameter closest to the combination position is acquired from the recording medium. However, this is an example, and the present invention is not limited to this. For example, a plurality of light source parameters created in the vicinity of the synthesis position may be acquired, and the light source parameters may be generated by linear interpolation according to the distance to each light source parameter.

  By configuring and processing in this way, a lighting effect corresponding to the combining position can be easily applied to the foreground image, and a natural image can be obtained even if the images are combined.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (9)

A video processing device that applies a lighting effect to a foreground image and combines it with a background image,
Storage means for storing a foreground image, a background image as a background of the foreground image, and depth information indicating the depth of the background image;
Combined position acquisition means for acquiring a combined position of the foreground image on the background image;
Projecting means for projecting the coordinates of the space generated from the depth information of the background image onto the coordinates of the background image;
Spatial coordinate acquisition means for acquiring spatial coordinates of the composite position from projection information by the projection means and the composite position;
The distance between the light source and the spatial coordinates of the combined position, or the combined position of the light source parameter that holds information on the light source position and illuminance as viewed from the position where the background image was captured and the spatial coordinates of the combined position, or Calculating means for calculating the illuminance of the light source as seen from the spatial coordinates;
Light source parameter changing means for changing a light source parameter according to the spatial coordinates of the combined position and the calculated value;
Lighting means for applying a lighting effect to the foreground image using the light source parameter;
Synthesizing means for synthesizing the foreground image on which the lighting effect has been applied by the lighting means at the synthesis position of the background image;
A video processing apparatus comprising:   The video processing apparatus according to claim 1, wherein the light source parameter changing unit performs a light source parameter changing process when a change amount of a spatial coordinate of the combined position is equal to or greater than a predetermined range.   The light source parameter changing means changes the position information of the light source parameter within the range in which the value calculated by the calculating means is determined in advance to a value obtained by subtracting the value of the spatial coordinate of the combined position. Item 3. The video processing apparatus according to Item 1 or 2. The storage means holds all-around image and depth information of all-around image as image information,
The light source parameter changing means calculates the parameters of the light source viewed from the spatial coordinates of the combined position of the light sources, divides the entire surrounding image for each light source, and rotates the image of each light source to match the calculated parameters. 3. The video processing apparatus according to claim 1, wherein the parallel movement, enlargement, and reduction processes are performed.   The light source parameter changing means generates light source parameters corresponding to a plurality of spatial coordinates in advance, and from among the light source parameters generated in advance, a light source corresponding to spatial coordinates in the vicinity of the spatial coordinates of the combined position 5. The video processing apparatus according to claim 3, wherein a parameter is acquired.   The light source parameter changing unit obtains a plurality of light source parameters corresponding to spatial coordinates in the vicinity of the spatial coordinates of the synthesis position from the light source parameters generated in advance, and generates the light source parameters by linearly interpolating each parameter. The video processing apparatus according to claim 5, wherein: A foreground image, a background image serving as a background of the foreground image, and storage means for storing depth information indicating the depth of the background image are provided, and a lighting effect is applied to the foreground image and combined with the background image. A method of controlling a video processing apparatus for performing
A combined position acquisition unit for acquiring a combined position of the foreground image on the background image; and
A projecting step for projecting the coordinates of the space generated from the depth information of the background image onto the coordinates of the background image;
Spatial coordinate acquisition means acquires the spatial coordinates of the combined position from the projection information and the combined position by the projection step;
The calculation means determines the distance between the light source and the spatial coordinates of the combined position from the light source parameter that holds the position of the light source and the illuminance information viewed from the position where the background image is captured and the spatial coordinates of the combined position, or A calculation step of calculating the illuminance of the light source viewed from the spatial coordinates of the combined position;
A light source parameter changing unit that changes a light source parameter according to the spatial coordinates of the combined position and the calculated value;
A lighting step of applying a lighting effect to the foreground image using the light source parameter; and
A synthesizing unit that synthesizes the foreground image on which the lighting effect has been applied in the lighting step with the synthesis position of the background image;
A control method for a video processing apparatus, comprising:   A program for causing a computer to function as the video processing apparatus according to claim 1 by being read and executed by a computer.   A computer-readable storage medium storing the program according to claim 8.

Images