JP2010278813A - Video synthesizing method, video synthesis system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video synthesizing method and video synthesis system for obtaining synthesizing video obtained by extracting only material for synthesis from video photographed in an optional background when generating the synthesizing video to be used for synthesis. <P>SOLUTION: The video synthesizing method and the video synthesis system perform angle calculation processing for calculating a pan angle and a tilt angle during photographing each frame of background side video 10 by comparing the position of an index point showing the same background position in a plurality of frames of the background side video 10, background selection processing for selecting a comparing frame 10a from a frame group of the background side video 10 by frame of material side video 12 on the basis of a pan angle and a tilt angle during photographing each frame of the material side video 12, and background comparison processing for generating synthesizing video 14 obtained by extracting synthesizing material 20 by comparing the selected comparing frame 10a with each frame of the material side video 12. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for generating a composite video by cutting out only a composite material from a video shot without using a special background such as a blue background.

  Conventionally, composition techniques are indispensable in the production of video works such as movies. Images used for composition are often shot using a blue-back composition technique.

  The blue back composition is a technique that uses a background such as a blue cloth (hereinafter referred to as a blue back) when shooting an image used for composition. An image special process that treats the blue back portion as transparent is applied to the captured image to generate an image for synthesis, and another background image is synthesized there. For example, when composing a person and a background, a person as a composition material is photographed before the blue background, and a separately prepared background image is synthesized with the blue background portion of the captured image (for example, patent Reference 1 or 2).

Japanese Patent Laid-Open No. 10-042307 JP 2007-142726 A

  In this blue background composition, a composition image photographed using a blue background is used. For this reason, it is necessary to prepare a blue background as a temporary background for blue background synthesis. However, there is a problem that the blue back is prepared only in a part of the shooting studios, and that the use and preparation of the shooting studio and the blue back are costly and troublesome.

  In addition, since the number of shooting studios is limited, there is a problem that it is difficult to secure a shooting studio when re-shooting is required suddenly due to re-shooting or the like. Furthermore, since the shootable range is limited to the size of the shooting studio or the blue background, there is a problem that it is difficult to shoot a moving image that uses a lot of camera pan and tilt.

  Also, if you try to synthesize an image shot under artificial light called a shooting studio and a background image shot under sunlight, the image after the composition process will become unnatural due to subtle differences in light. There was also.

  In addition, as a method of generating a composite video without using a blue background, it is also possible to manually cut out a composite material from video shot by a normal method using image processing software that operates on a digital computer. It is done. However, manual operation takes time and labor costs, and details cannot be cut out precisely, resulting in an unnatural composite image in which fine nuances are lost.

  In addition, software that automatically detects the outline of the composition material from the video containing the composition material and cuts out the composition material is also conceivable, but it detects and cuts out the dynamically changing outline. In the process of going, there is a problem that information processing becomes complicated and enormous.

  The present invention has been made in view of the above-described problems. When generating a composite image used for composition, the present invention combines the image captured with an arbitrary background without using the image captured with the blue background. It is an object of the present invention to provide a video composition method and a video composition system capable of obtaining a composite image in which only a material for use is extracted.

  The present invention provides a background-side image in which a multi-directional background that can be photographed by a background camera arranged so as to be capable of panning and tilting is photographed by the background camera, and is arranged so as to be able to pan and tilt at the same position as the background camera. A material-side image captured so that the composition material is included under the background by the prepared material camera, and the position of the index point indicating the same background position in a plurality of frames of the background-side image is prepared Based on the comparison, an angle calculation process for calculating a pan angle and a tilt angle for each frame of the background-side video for each frame, and a pan angle and a tilt angle for the frame of the material-side video at the time of shooting. A background selection process for selecting a comparison frame from the frame group of the background side video for each frame of the material side video, and the selected comparison frame. By comparing each frame of over arm and the material-side image, and background comparison process of generating a composition image of the composition material is extracted, and performing a video composition method.

  According to the present invention, in the video composition method, the angle calculation processing sets a virtual sphere centered on the background camera position, and sets the index point on one reference frame of two frames to be compared. The reference projection index point projected on the surface of the virtual sphere toward the radial direction of the virtual sphere, and the index point on the other comparison frame are directed on the surface of the virtual sphere toward the radial direction of the virtual sphere. A projected projection index point is obtained, and a pan angle and a tilt angle at the time of shooting each frame of the background side image are calculated by comparing the positions of the reference projection index point and the contrast projection index point. And

  The present invention is also characterized in that, in the video composition method, the angle calculation processing sets a radius of the virtual sphere based on a field angle of the background camera and a frame size of the background side video.

  According to the present invention, in the video composition method, in the angle calculation processing, the center of the virtual sphere is set as a principal point of the lens of the background camera, and the radius of the virtual sphere is set as the lens of the background camera. The focal length is set to be the same.

  The present invention is also characterized in that, in the video composition method, the index point is a point corresponding to the optical center of the reference frame.

  According to the present invention, in the video composition method, the angle calculation process generates a reference projection frame by projecting the reference frame on a surface of the virtual sphere in a radial direction of the virtual sphere, By projecting the contrast frame onto the surface of the virtual sphere in the radial direction of the virtual sphere, generating a contrast projection frame, and performing pattern matching on the reference projection frame and the contrast projection frame, the index A plurality of projection specific points indicating the same background position other than the points are obtained, and the projection specific point on the reference projection frame is set as a reference projection specific point, and the projection specific point on the contrast projection frame is contrasted and specified. And calculating the position of the contrast projection index point based on the positions of the reference projection specific point and the contrast projection specific point. To.

  According to the present invention, in the video composition method, the angle calculation process calculates a position of the contrast projection index point based on a relative positional relationship between the reference projection specific point and the reference projection index point. Features.

  According to the present invention, in the video composition method, the angle calculation processing sets a first projection specific point and a second projection specific point as the projection specific point, and the reference projection of the first projection specific point. A first distance between a specific point and the reference projection index point, and a second distance between the reference projection specific point and the reference projection index point of the second projection specific point are obtained, and the first distance The point on the surface of the virtual sphere located at the first distance from the contrast projection specific point of the projection specific point and the second distance from the contrast projection specific point of the second projection specific point is the contrast. A projection index point is used.

  According to the present invention, in the video composition method, the angle calculation processing calculates a roll angle at the time of shooting each frame of the background side video for each frame of the background side video, and the background comparison processing includes the background comparison The correction is performed based on the roll angle at the time of photographing each frame of the side image.

  The present invention is also the video composition method, wherein the angle calculation process includes a straight line connecting the reference projection specific point of the predetermined projection specific point and the reference projection index point, and the contrast projection of the predetermined projection specific point. Based on an angle formed by a specific point and a straight line connecting the contrast projection index points, a roll angle at the time of shooting each frame of the background side image is obtained, and the background comparison processing is performed by shooting each frame of the background side image. The correction is performed based on the current roll angle.

  The present invention is also the video composition method, wherein the background selection processing includes comparing the positions of index points for selection indicating the same background position in the frame of the background side video and the frame of the material side video. A pan angle and a tilt angle at the time of photographing each frame of the material side image are calculated for each frame.

  According to the present invention, in the video composition method, when the background selection process cannot select a pan angle and a tilt angle comparison frame that are the same as the pan angle and the tilt angle at the time of shooting the frame of the material side video, A plurality of frames for comparison of pan angle and tilt angle close to the pan angle and tilt angle at the time of photographing the frame of the material side image are selected.

  According to the present invention, in the video composition method, the background selection processing calculates a roll angle at the time of shooting each frame of the material-side video for each frame of the material-side video, and the background comparison processing includes the material comparison The correction is performed based on the roll angle at the time of photographing each frame of the side image.

  According to the present invention, in the video composition method, in the background comparison process, if the pan angle or the tilt angle at the time of shooting of the material side video frame to be compared and the comparison frame are not the same, the frame of the material side video A comparison correction frame is generated by correcting the comparison frame so as to coincide with the pan angle and tilt angle at the time of shooting, and the comparison correction frame and the frame of the material side image are compared.

  According to the present invention, in the video composition method, the background comparison processing generates the comparative correction frame in which the comparison frame is corrected so as to coincide with a roll angle at the time of shooting the frame of the material side video. It is characterized by.

  According to the present invention, in the video composition method, the background comparison process projects each pixel of the comparison frame on the same plane as the frame of the material-side video in the radial direction of the virtual sphere, The modification frame for comparison is generated.

  The present invention also provides background-side image storage means for storing a background-side image in which a multi-directional background that can be photographed by a background camera arranged so as to be capable of panning and tilting is photographed by the background camera, and the background camera. Material-side video storage means for storing a material-side image shot so that the material for composition is included under the background by a material camera arranged so as to be capable of panning and tilting at the same position; and a plurality of background-side images By comparing the positions of index points indicating the same background position in the frame, angle calculation means for calculating the pan angle and the tilt angle at the time of shooting of each frame of the background side image for each frame, and the material side image Based on the pan angle and tilt angle at the time of shooting each frame, the frame for comparison is extracted from the frame group of the background side image for each frame of the material side image. A background selection means for selecting an image, and a background comparison means for generating a composition image from which the composition material is extracted by comparing the selected comparison frame and each frame of the material side image. This is a video composition system.

  According to the present invention, in the video composition system, the angle calculation unit calculates a roll angle at the time of shooting each frame of the background side video for each frame of the background side video, and the background comparison unit includes the background comparison unit. The correction is performed based on the roll angle at the time of photographing each frame of the side image.

  The present invention also provides the video composition system, wherein the background selection means compares the position of the index point for selection indicating the same background position in the frame of the background side video and the frame of the material side video. A pan angle and a tilt angle at the time of photographing each frame of the material side image are calculated for each frame.

  According to the present invention, in the video composition system, the background selection unit calculates a roll angle at the time of shooting each frame of the material side video for each frame of the material side video, and the background comparison unit includes the material comparison unit. The correction is performed based on the roll angle at the time of photographing each frame of the side image.

  According to the present invention, in the video composition system, the background comparison unit is configured such that when the pan angle or the tilt angle at the time of photographing of the material side video to be compared and the comparison frame are not the same, the frame of the material side video A comparison correction frame is generated by correcting the comparison frame so as to coincide with the pan angle and tilt angle at the time of shooting, and the comparison correction frame and the frame of the material side image are compared.

  According to the present invention, in the video composition system, the background comparison unit generates the comparative correction frame in which the comparison frame is corrected so as to coincide with a roll angle at the time of shooting the frame of the material side video. It is characterized by.

  According to the video composition method or the video composition system of the present invention, it is possible to generate a composite image in any background environment. As a result, it is possible to reduce the burden of the usage fee for the shooting studio for the blue background and the time and effort for advance preparation. Furthermore, even when the composition material is photographed while panning or tilting the camera, and the position of the composition material with respect to the background changes greatly, the composition image can be generated efficiently.

It is the figure which showed the concept of the image | video synthetic | combination method of embodiment of this invention. 1 is a schematic configuration diagram of a video composition system according to an embodiment of the present invention. It is the figure which showed the internal structure with which a computer is provided. It is the figure which showed pan angle (theta) and tilt angle (phi) in embodiment of this invention. (A) And (b) It is the figure which showed roll angle (psi) in embodiment of this invention. It is the figure which showed typically imaging | photography with a video camera. It is the schematic diagram which replaced the rotation of the video camera with the movement of the background along a celestial sphere. It is the figure which showed how to obtain | require the projection point on a virtual sphere. (A)-(c) It is the figure which showed the example of the frame of the background side image | video. (A) And (b) It is the figure which showed the case where each point on the flame | frame shown in FIG. 9 was projected on the virtual sphere. (A)-(c) It is the schematic which showed the example of the three-dimensional image for selection. It is the flowchart which showed the procedure of the synthetic | combination video generation process. It is the flowchart which showed the procedure of the angle calculation process. It is the figure which showed the specific method of calculating each angle (theta), (phi), and (psi). It is the figure which showed the specific method of calculating each angle (theta), (phi), and (psi). It is the flowchart which showed the procedure of the background selection process. It is the figure which showed the specific method of (a) and (b) background selection processing. It is the flowchart which showed the procedure of the background comparison process. It is the figure which showed the specific method of (a) and (b) background comparison processing. It is the figure which showed the specific method of (a) and (b) background comparison processing.

  Embodiments of the present invention will be described below with reference to the drawings. It should be noted in advance that the present invention is not limited to these drawings.

<Basic idea>
First, the basic idea throughout the present embodiment will be described. FIG. 1 is a diagram showing the concept of the video composition method of the present embodiment.

  As shown in the figure, in this embodiment, first, a background side image 10 generated by photographing an arbitrary background is prepared. And the material side image | video 12 image | photographed so that the synthetic | combination material (for example, person) 20 may be included on the same background, and, for example, the plum 22 and the horsetail 24 inserted later as the background of the synthetic material 20 were photographed. A background image 14 for insertion is prepared.

  Next, a comparison frame 10a having the same background as each frame of the material-side video 12 (for example, the background where the tulip 26 is photographed) is selected for each frame of the material-side video 12 from the frame group of the background-side video 10. And extract (background selection processing). As a result, two images (frames) 12 and 10a that differ only in whether the composition material 20 is shown are prepared. Then, each frame of the material-side image 12 and the comparison frame 10a are compared, and a different portion is extracted to obtain a composition image 14 in which only the composition material 20 is cut out (background comparison processing). By synthesizing the insertion background image 16 with the composition image 14, for example, the final image 18 of the composition material (person) 20 with the background of the plum 22 and the horsetail 24 is completed.

  The background side image 10 and the material side image 12 are shot by panning or tilting a camera (video camera) arranged at the same position. In this way, by fixing the camera position, when processing the image of the background portion, it is possible to treat it as a background that exists at a certain distance from the camera, that is, a background that exists on the celestial sphere of an arbitrary radius. . The camera that captures the background-side video 10 and the camera that captures the material-side video 12 may be either the same camera or different cameras as long as they are arranged at the same position.

<Video composition system>
Next, a video composition system according to an embodiment of the present invention will be described. FIG. 2 is a schematic configuration diagram of the video composition system 100.

  In FIG. 2, reference numeral 101 denotes a background side image taken by the video camera, and reference numeral 102 denotes a material side image taken by the video camera. Reference numeral 104 denotes a hard disk (background-side video storage unit) in which the background-side video 101 is stored, and reference numeral 105 denotes a hard disk (material-side video storage unit) in which the material-side video 102 is stored. Reference numeral 106 denotes a frame constituting the background side video 101, and 107 denotes a frame constituting the background side video 103.

  Reference numeral 108 denotes a computer having a CPU (central processing unit), a memory, a communication bus, a plurality of input / output interfaces, and the like. Reference numeral 109 denotes a hard disk (program storage) in which a program executed by the computer 108 is stored. Means). Reference numeral 110 denotes a compositing frame (a frame in which only the compositing material is cut out) generated after the background comparison processing is performed by the computer 108, and 111 denotes a hard disk (compositing video) that stores the compositing frame 110. Storage means). Reference numeral 112 denotes a composition video composed of the composition frame 110.

  FIG. 3 is a diagram showing an internal configuration of the computer 108. This internal configuration also includes various functions obtained by temporarily storing the program in the memory 121 of the computer 108 and executing the program by a CPU (Central Processing Unit) 120.

  The computer 108 includes a CPU 120 that controls the entire computer and a memory 121 including a RAM and the like, and also includes a background-side image storage unit 104, a material-side image storage unit 105, a program storage unit 109, and a composition image storage unit. 111 are provided with input / output interfaces (I / O) 122, 123, 124, and 125 respectively connected to 111. Further, as a functional configuration obtained by executing the program by the CPU 120, an angle calculation unit 130, a background side video angle database generation unit 132, a selection three-dimensional image generation unit 134, a background selection unit 136, and a background comparison unit 138 are provided. I have.

<Angle calculation means>
Next, the function of the angle calculation unit 130 will be described. The angle calculation means 130 executes an angle calculation process to be described later, and calculates a pan angle θ and a tilt angle φ when each frame 106 constituting the background side image 101 is captured. FIG. 4 is a diagram showing the pan angle θ and the tilt angle φ in the present embodiment. As shown in the figure, in this embodiment, a background side image is shot by a video camera 200 arranged at an arbitrary position such as in a shooting studio or outdoors.

  The video camera 200 is placed on a pan head 300, and the video camera 200 can be directed in an arbitrary shooting direction by panning and tilting. In this embodiment, the video camera 200 is rotated about the vertical axis and the operation of changing the shooting direction in the horizontal plane is “pan”, and the video camera 200 is rotated about the horizontal axis to perform shooting. The operation of changing the direction in the vertical plane is called “tilt”. By obtaining the pan angle θ and the tilt angle φ at the time of photographing each frame 106, the photographing direction at the time of photographing each frame 106 can be specified.

  When shooting with the video camera 200 fixed at a predetermined position and panning and tilting, the distance between the video camera 200 and the background does not change. It can be considered to be above (on the surface of the celestial sphere 400). Therefore, by calculating the pan angle θ and the tilt angle φ at the time of shooting each frame 106 and specifying the shooting direction, the position of the shooting area 402 of each frame 106 of the background side image 101 on the celestial sphere 400 is determined. Can be identified.

  In addition, the angle calculation unit 130 calculates a roll angle ψ at the time of shooting each frame 106 constituting the background side image 101. FIGS. 5A and 5B are views showing the roll angle ψ in the present embodiment. As shown in FIG. 5A, when the video camera 200 is panned while being tilted at a predetermined angle, the video camera 200 rolls around the optical center with respect to the background. . Therefore, for example, when the images displayed on the video camera 200 when shooting the shooting areas 402a, 402b, and 402c in FIG. 6A are 502a, 502b, and 502c, respectively, these images 502a to 502c are shown in FIG. As shown in b), the image is projected so as to move in an arc shape with respect to the background and to rotate about the optical center O. The angle calculation means 130 calculates the roll rotation angle around the optical center 0 as the roll angle ψ.

  Next, a method for calculating the above-described pan angle θ, tilt angle φ, and roll angle ψ by the angle calculation unit 130 of the present embodiment will be described. First, the principle of the calculation method of each angle θ, φ, ψ will be described with reference to FIGS.

  FIG. 6 is a diagram schematically showing shooting by the video camera 200. In the figure, a plurality of lenses included in the video camera 200 is shown as a virtual lens 202 regarded as one lens, and the center thereof is a main point 204. An imaging element 206 made of, for example, a CCD is disposed at the focal position of the virtual lens 202, and receives light that has passed through the virtual lens 202.

  This figure shows a case where the video displayed on the image sensor 206 changes from the shooting area 402d to the shooting area 402e by rotating or panning the video camera 200 around the principal point 204. The object 404 that is a part of the background is stopped on the celestial sphere 400 centered on the principal point 204, and the direction of light incident from the object 404 to the principal point 204 of the virtual lens 202 is determined by the position of the video camera 200. It is constant because it is fixed. Therefore, the position of the mapping 504 of the object 404 received by the image sensor 206 changes according to the movement of the image sensor 206 due to the rotation of the video camera 200. Specifically, in this example, the position of the mapping 504 changes toward the end portion side (left end portion side in the figure) of the image sensor 200.

  Since the positional relationship between the background and the video camera 200 is relative, rotation of the video camera 200 can be replaced by movement of the background along the celestial sphere 400. FIG. 7 is a schematic diagram showing the rotation of the video camera 200 replaced with the movement of the background along the celestial sphere 400.

  As shown in the figure, the object 404 moves on the celestial sphere 400 in the imaging region 402 as the video camera 200 rotates. In accordance with the movement of the object 404, the position of the mapping 504 of the object 404 received by the image sensor 206 changes. However, the movement of the map 504 on the image sensor 206 is obtained by converting the movement of the object 404 on the spherical surface into the movement on the plane. For this reason, the mapping 504 moves in a different manner depending on the position on the image sensor 206 with respect to the rotation of the same video camera 200. Further, when the video camera 200 rotates in a roll, the movement of the map 504 is further complicated by adding a rotational movement proportional to the distance from the optical center.

  Therefore, in the present embodiment, a virtual sphere 600 that is in contact with the optical center O of the imaging element 206 with the principal point 204 of the virtual lens 202 as the center is set, and the mapping 504 on the virtual sphere 600 (on the surface of the virtual sphere 600). The angles θ, φ, and ψ are calculated based on the movement of the point (projection point) 604 corresponding to. The movement of the projection point 604 on the virtual sphere 600 is similar to the relative movement of the object 404 on the celestial sphere 400 by panning or tilting the video camera 200 (the path is the ratio of the radius of the celestial sphere 400 and the virtual sphere 600). This is the reduced movement) plus the rotational movement due to roll rotation.

そして、本実施形態では、撮像素子２０６上（フレーム上）の各点（画素）を仮想球体６００の半径方向、すなわち仮想球体６００の中心である主点２０４に向けて仮想球体６００の表面上に投射した点を、撮像素子２０６上（フレーム上）の各点に対応する投射点とする。 FIG. 8 is a diagram showing how to obtain the projection points on the virtual sphere 600. As described above, the virtual sphere 600 is a sphere centered on the principal point 204 of the virtual lens 202 and in contact with the optical center O of the image sensor 206. Accordingly, the radius of the virtual sphere 600 can be regarded as being equal to the focal length of the virtual lens 202. Further, when the diagonal angle of view of the video camera 200 is α and the length of the diagonal line of the image sensor 206 is L, the radius d of the virtual sphere 600 can be obtained by the following equation, for example.

In this embodiment, each point (pixel) on the image sensor 206 (on the frame) is directed on the surface of the virtual sphere 600 in the radial direction of the virtual sphere 600, that is, toward the principal point 204 that is the center of the virtual sphere 600. Let the projected point be the projection point corresponding to each point on the image sensor 206 (on the frame).

  For example, in the figure, the projection point of the point 506 a on the image sensor 206 is a point 606 a on the virtual sphere 600, and the projection point of the point 506 b on the image sensor 206 is a point 606 b on the virtual sphere 600. In this way, by setting the point projected on the surface of the virtual sphere 600 with the point on the image sensor 206 directed in the radial direction of the virtual sphere 600 as a projection point, for example, the mapping of the point 406a on the celestial sphere 400 is a point. In the case of 506a, since the point 606a is positioned in a straight line connecting the point 406a, the main point 204, and the point 506a, the point 606a is a point that accurately indicates the position of the point 406a on the virtual sphere 600. ing. Similarly, the point 606b is a point that accurately indicates the position of the point 406b on the virtual sphere 600.

  Projection points of other points on the image sensor 206, for example, points 506c, 506d, and 506e, are points 606c, 606d, and 606e, respectively. Further, the point 506f corresponding to the optical center O of the image sensor 206 and the projection point 606f are located at the same point.

図９（ａ）〜（ｃ）は、背景側映像１０１のフレーム１０６の例を示した図である。この例では、同図（ａ）に示されるように、フレーム１０６ａにおいて、光学中心に対応する点Ｏ、ならびに点Ｐおよび点Ｑから構成される三角形ＯＰＱが映し出されている。そして、フレーム１０６ａの撮影方向からビデオカメラ２００をパンおよびチルトさせて撮影したフレームがフレーム１０６ｂである。同図（ｂ）に示されるように、フレーム１０６ｂでは、ビデオカメラ２００のパンおよびチルトに伴って背景が相対的に移動することにより、点Ｏが点Ｏ'に移動し、点Ｐが点Ｐ'に移動し、点Ｑが点Ｑ'に移動している。すなわち、フレーム１０６ａにおける三角形ＯＰＱは、フレーム１０６ｂでは三角形Ｏ'Ｐ'Ｑ'となって映し出されている。また、フレーム１０６ｂは、ビデオカメラ２００がパンおよびチルトされた結果、背景である三角形Ｏ'Ｐ'Ｑ'に対してロール回転した状態となっている。 In this embodiment, the spatial coordinates (x, y, z) are set with the principal point 204 as the origin, the horizontal direction as the x direction, the vertical direction as the y direction, and the axial direction of the optical center of the video camera 200 as the z direction. is doing. Therefore, assuming that the radius of the virtual sphere 600 is d, the coordinates (x ′, y ′, z ′) of the projection point on the virtual sphere 600 corresponding to the point of the coordinates (x, y, z) on the image sensor 206 are For example, it is calculated | required by the following numerical formula.

FIGS. 9A to 9C are diagrams illustrating an example of the frame 106 of the background side image 101. FIG. In this example, as shown in FIG. 9A, a point O corresponding to the optical center and a triangle OPQ composed of points P and Q are displayed in the frame 106a. A frame 106b is obtained by panning and tilting the video camera 200 from the shooting direction of the frame 106a. As shown in FIG. 6B, in the frame 106b, the background moves relatively with the panning and tilting of the video camera 200, so that the point O moves to the point O ′, and the point P changes to the point P. Has moved to 'and point Q has moved to point Q'. That is, the triangle OPQ in the frame 106a is displayed as a triangle O′P′Q ′ in the frame 106b. Further, the frame 106b is in a state of being rolled with respect to the triangle O′P′Q ′ as a background as a result of the video camera 200 being panned and tilted.

  FIG. 4C shows the movement from the triangle OPQ to the triangle O′P′Q ′ on one frame. As described above, due to the conversion from the celestial sphere 400 to the movement on the plane and the influence of the roll rotation of the video camera 200, the movement from the point O to the point O ′ on the frame, the movement from the point P to the point P ′. , And the movement from the point Q to the point Q ′ are different from each other. That is, on the frame, the triangle OPQ and the triangle O′P′Q ′ are different because the movement distance and the movement direction change according to the position where each point is projected, as shown in FIG. It becomes a shape.

  Since the triangle OPQ and the triangle O′P′Q ′ are the same background, it is generally recognized that the triangle OPQ and the triangle O′P′Q ′ are projected in the same shape in both frames 106a and 106b. However, actually, when the positions in the frame 106a and the frame 106b are different, the triangle OPQ and the triangle O'P'Q 'are projected in different shapes. That is, when the video camera 200 is arranged at a predetermined position and the background side image 101 or the material side image 102 is shot by panning and tilting as in the present embodiment, the same image is taken on frames with different shooting directions. Even the background is projected in different shapes. For this reason, two frames are extracted from each frame 106 constituting the background side video 101 or each frame 107 constituting the material side video 102 and pattern matching is performed, thereby indicating the same background position in the two frames. It is difficult to seek.

  FIGS. 10A and 10B are diagrams showing a case where each point on the frame 106a and the frame 106b shown in FIG. 9 is projected onto the virtual sphere 600 shown in FIG. In these figures, point o is the projection point of point O, point p is the projection point of point P, and point q is the projection point of point Q. Further, the point o ′ is the projection point of the point O ′, the point p ′ is the projection point of the point P ′, and the point q ′ is the projection point of the point Q ′.

  A triangle opq and a triangle o′p′q ′ projected onto the virtual sphere 600 are similar to the same triangle in the background on the celestial sphere 400 (a shape reduced by the ratio of the radius of the celestial sphere 400 and the virtual sphere 600). ) Therefore, the triangle opq and the triangle o′p′q ′ have the same shape (congruent) regardless of the position on the virtual sphere 600, as shown in FIG. That is, in the case of shooting as in the present embodiment, in a partially spherical projection frame obtained by projecting a point (pixel) on the frame onto the virtual sphere 600, even if the shooting direction is different, If it is the same background, it is projected in the same shape. Therefore, if pattern matching is performed for two projection frames, a point indicating the same background position in the two frames can be identified relatively easily.

  In FIG. 9A, point o corresponds to the optical center of the frame 106a. Therefore, the movement from the point o to the point o ′ on the virtual sphere 600 indicates the movement of the optical center of the video camera 200, that is, the movement in the photographing direction according to the pan angle θ and the tilt angle φ. Therefore, the pan angle θ and the tilt angle φ between the frame 106a and the frame 106b can be obtained based on the movement from the point o on the virtual sphere 600 to the point o ′.

また、点ｐから点ｐ'への移動および点ｑから点ｑ'への移動は、フレーム１０６ａからフレーム１０６ｂまでのパン角度θおよびチルト角度φに加え、ロール角度ψに応じたものとなっている。ロール回転は点ｏまたは点ｏ'を中心に生じるため、同図（ｂ）に示されるように、三角形ｏ'ｐ'ｑ'を仮想球体６００上で移動させて点ｏと点ｏ'を一致させた場合、三角形ｏｐｑおよび三角形ｏ'ｐ'ｑ'はロール角度ψだけ傾いた状態で互いに重なる。点ｐの座標が（ｘ_１，ｙ_１，ｚ_１）、同図（ｂ）に示される移動後の点ｐ'の座標が（ｘ_２，ｙ_２，ｚ_２）である場合、例えば次の式でロール角度ψを求めることができる。

＜コンピュータの備えるその他の機能構成＞
次に、コンピュータ１０８の備えるその他の機能構成について説明する。 The coordinates of the point o are the same as the point O on the frame 106a, and when the radius of the virtual sphere 600 is d, the coordinates are (0, 0, d). Therefore, when the coordinates of the point o ′ are (x _s , y _s , z _s ), for example, the pan angle θ and the tilt angle φ can be obtained by the following equations.

Further, the movement from the point p to the point p ′ and the movement from the point q to the point q ′ are in accordance with the roll angle ψ in addition to the pan angle θ and the tilt angle φ from the frame 106a to the frame 106b. Yes. Since the roll rotation occurs around the point o or the point o ′, the triangle o′p′q ′ is moved on the virtual sphere 600 to match the point o and the point o ′ as shown in FIG. In this case, the triangle opq and the triangle o′p′q ′ are overlapped with each other while being inclined by the roll angle ψ. When the coordinates of the point p are (x ₁ , y ₁ , z ₁ ) and the coordinates of the point p ′ after the movement shown in FIG. 5B are (x ₂ , y ₂ , z ₂ ), for example, The roll angle ψ can be obtained by the equation.

<Other functional configurations of the computer>
Next, other functional configurations included in the computer 108 will be described.

  The background side video angle database generation unit 132 associates each angle θ, φ, ψ calculated by the angle calculation unit 130 with each frame 106 of the background side video by executing a background side video angle database generation process described later. A background side video angle database is generated. The background selection unit 136 (to be described later) refers to the background side video angle database and selects a comparison frame from the plurality of frames 106 constituting the background side video 101.

  The selection three-dimensional image generation unit 134 executes selection three-dimensional image generation processing described later, thereby selecting each frame 106 of the background side video 101 mapped on the virtual sphere 600 based on the background side video angle database. A three-dimensional image is generated.

  FIGS. 11A to 11C are schematic diagrams illustrating examples of a selection three-dimensional image. In addition, the same figure (a) and (c) has shown the one part cross section of the three-dimensional image for three choices. As shown in FIG. 6A, the selection three-dimensional image generation means 134 uses the pan angle θ and the tilt angle φ of each frame 106 as the spherical coordinates (θ, φ) on the virtual sphere 600, and the background side video. A three-dimensional image 700 for selection constituted by arranging 101 frames 106 on a virtual sphere is generated. Specifically, as shown in FIG. 4B, each point O corresponding to the optical center of each frame 106 is positioned at the coordinate point corresponding to the pan angle θ and the tilt angle φ on the virtual sphere 600. Each frame 106 is arranged so that the frame 106 contacts the virtual sphere 600. Further, each frame 106 is rotated about the point O by the roll angle ψ, and correction is performed so that the background of each frame 106 matches the position corresponding to the background on the celestial sphere 400. Accordingly, the selection three-dimensional image 700 is generated as image data imitating the background on the celestial sphere 400 by arranging the two-dimensional frames 106 in a three-dimensional manner.

  In this way, by mapping each frame 106 of the background side image 101 on the virtual sphere 600 based on the pan angle θ and the tilt angle φ, it corresponds to each point (pixel) on each frame 106 of the background side image 101. It is possible to specify the spherical coordinates to be played.

  As shown in FIG. 6C, a spherical panoramic image is synthesized by projecting the mapped pixels of each frame 106 onto the virtual sphere 600, and this is used as a selection three-dimensional image 700. Good. Projection of the pixels of each frame 106 onto the virtual sphere 600 is performed in the radial direction of the virtual sphere 600. As described above, the selection three-dimensional image 700 configured as a spherical panoramic image is image data having a similar shape to the celestial sphere 400 (or part of the celestial sphere 400) including the background.

  The background selection unit 136 performs background selection processing described later, and selects a comparison frame including the same background portion from a plurality of frames 106 constituting the background side video 101 for each frame 107 of the material side video 102. To extract. Details of processing executed by the background selection unit 136 will be described later.

  The background comparison unit 138 executes background comparison processing described later, extracts a background portion from each frame 107 of the material-side video 102, and generates a composition frame 110 in which only the composition material is cut out. Details of the processing executed by the background comparison unit 138 will be described later.

<Synthesis video generation processing>
Next, the procedure of the synthesis video generation process for generating the synthesis video 112 from the background side video 101 and the material side video 102 will be described. FIG. 12 is a flowchart showing the procedure of the composition video generation process.

  First, in step S <b> 101, the angle calculation unit 130 executes an angle calculation process for the frame 106 of the background side image 101, and calculates each angle θ, φ, ψ for each frame 106 of the background side image 101. The detailed procedure of the angle calculation process will be described later.

  In step S102, it is determined whether or not the angles θ, φ, and ψ have been calculated for all the frames 106 of the background side video 101. If the angles θ, φ, and ψ have not been calculated for all the frames 106, the process returns to step S101, and angle calculation processing is executed for the remaining frames 106 in a predetermined order. When the angles θ, φ, and ψ are calculated for all the frames 106, the process proceeds to step S103.

  In step S103, the background side video angle database generation unit 132 executes background side video angle database generation processing to generate a background side video angle database. The generated background side video angle database is stored in the background side video storage means 104. In step S104, the selection three-dimensional image generation means executes a selection three-dimensional image generation process to generate a selection three-dimensional image 700. The generated selection three-dimensional image 700 is stored in the background side image storage unit 104. The background side video angle database and the selection three-dimensional image 700 may be stored in a dedicated storage unit other than the background side video storage unit 104.

  In step S <b> 105, the background selection unit 136 performs background selection processing for the frame 107 of the material side video 102 and selects a comparison frame for each frame 107 of the material side video 102. The detailed procedure of the background selection process will be described later.

  In step S <b> 106, the background comparison unit performs background comparison processing on the frame 107 of the material-side video 102 to generate a composition frame 110. Here, the background comparison process is executed using the comparison frame selected in step S105. The generated synthesis frame 110 is stored in the synthesis video storage unit 111. The detailed procedure of the background comparison process will be described later.

  In step S107, it is determined whether background comparison processing has been executed for all frames 107 of the material-side video 102. When the background comparison process is not executed for all the frames 107, the process returns to step S105, and the background selection process and the background comparison process are executed for the remaining frames 106 in a predetermined order. If the background comparison process has been executed for all frames 106, the process ends.

  Through the above procedure, a composite video 112 composed of a plurality of composite frames 110 stored in the composite video storage unit 111 is generated. Then, the final image is completed by synthesizing the background image for insertion with each of the synthesis frames 11 of the synthesis video 112.

<Angle calculation processing>
Next, the details of the procedure of the angle calculation process in step S101 of the composition video generation process will be described. FIG. 13 is a flowchart showing the procedure of angle calculation processing, and FIGS. 14 and 15 are diagrams showing a specific method for calculating the angles θ, φ, and ψ.

  First, in step S201, a reference frame 106S (see FIG. 14A) used as a reference for each angle θ, φ, ψ from a plurality of frames 106 constituting the background-side video 101 and a comparison to be made with this reference frame 106S. The frame 106C (see FIG. 14B) is extracted. In the present embodiment, the angles θ, φ, and ψ at the time of shooting the contrast frame 106C are obtained by obtaining the angles θ, φ, and ψ of the contrast frame 106C with respect to the reference frame 106S.

  In this embodiment, first, the angles θ, φ, and ψ at the time of shooting of the first first frame 106 are defined as reference angles (for example, all are set to 0), and the first frame 106 is set as the reference frame 106S. The next second frame 106 is set as a comparison frame 106C, and the angles θ, φ, and ψ of the second frame 106 are calculated in the following steps. In the next step S201, the second frame 106 is set as the reference frame 106S, the next third frame 106 is set as the comparison frame 106C, and the angles θ, φ, and ψ of the third frame 106 are calculated. By repeating this up to the last frame 106, the angles θ, φ, and ψ are calculated for all the frames 106 of the background side image 101.

  In step S202, the point O corresponding to the optical center of the reference frame 106S is set as an index point S for obtaining the pan angle θ and the tilt angle φ (see FIG. 14A). Further, the projection point obtained by projecting the index point S on the reference frame 106S onto the virtual sphere 600 is set as the reference projection index point s. Since the reference projection index point s is the same point as the point o that is the projection point of the point O corresponding to the optical center of the reference frame 106S, the coordinates of the index point S on the reference frame 106S and the reference projection index point s The coordinates are the same.

  In step S203, the spherical reference projection frame 607S (see FIG. 14A) obtained by projecting all the pixels of the reference frame 106S onto the virtual sphere 600 in the radial direction and all the pixels of the comparison frame 106C are obtained. A spherical contrast projection frame 607C (see FIG. 14B) obtained by projecting onto the virtual sphere 600 in the radial direction is generated. Then, for the reference projection frame 607S and the contrast projection frame 607C, pattern matching is performed for each comparison area having a predetermined size (for example, an area having 64 × 64 pixels), and the same in the reference projection frame 607S and the contrast projection frame 607C. At least two specific comparison areas indicating patterns (for example, color information patterns) are extracted. As this pattern matching method, various known methods can be adopted, but it is necessary to adopt a pattern matching method considering roll rotation.

  FIGS. 14A and 14B show an example in which specific comparison areas Aa and Aa ′ showing the same pattern and specific comparison areas Ab and Ab ′ are extracted. The specific comparison area Aa of the reference projection frame 607S showing the same pattern and the specific comparison area Aa ′ of the contrast projection frame 607C indicate the same background portion on the celestial sphere 400. Similarly, the specific comparison area Ab and the specific comparison area Ab ′ indicate the same background portion on the celestial sphere 400.

  In step S204, the coordinates of the center of each specific comparison area Aa, Aa ′, Ab, Ab ′ are obtained, and the area center of the specific comparison area Aa is the first reference projection specific point a, the area of the specific comparison area Aa ′. The center is set as the first contrast projection specific point a ′, the region center of the specific comparison region Ab is set as the second reference projection specific point b, and the region center of the specific comparison region Ab ′ is set as the second contrast projection specific point b ′. (See FIG. 15 (a)). The first reference projection specific point a and the first contrast projection specific point a ′ set in this way are points on the virtual sphere 600 indicating the same background position. Similarly, the second reference projection specific point b and the second contrast projection specific point b ′ are points on the virtual sphere 600 that indicate the same background position.

  In step S205, the comparison frame 106C is based on the first reference projection specific point a, the first contrast projection specific point a ′, the second reference projection specific point b, and the second contrast projection specific point b ′. The coordinates of the contrast projection index point s ′ obtained by projecting the upper index point (a point indicating the same background position as the point S of the reference frame 106S) onto the virtual sphere 600 is calculated.

  In calculating the coordinates of the contrast projection index point s ′, first, as shown in FIG. 15A, from the reference projection index point s, the first reference projection specific point a, and the second reference projection specific point b. And a triangle s′a′b ′ composed of a contrast projection index point s ′, a first contrast projection specific point a ′, and a second contrast projection specific point b ′. Then, based on the fact that the triangle sab and the triangle s′a′b ′ are congruent, the coordinates of the contrast projection index point s ′ are calculated.

  Specifically, as shown in FIG. 15A, the distance between the reference projection index point s and the first reference projection specific point a is sa, the reference projection index point s and the second reference projection specific point. When the distance between b is sb, the distance between the contrast projection index point s ′ and the first contrast projection specific point a ′ is sa, and the contrast projection index point s ′ and the second contrast projection specific point b. The distance between 'is sb. Therefore, for example, as shown in FIG. 15B, a sphere 800 having a radius sa with the first contrast projection specific point a ′ as the center, and a radius sb with the second contrast projection specific point b ′ as the center. The coordinates of the contrast projection specific point s ′ can be obtained by geometrically obtaining the intersection of the three spheres of the sphere 802 and the virtual sphere 600.

  In step S206, based on the coordinates of the contrast projection specific point s ′, the first reference projection specific point a, and the first contrast projection specific point a ′, each angle θ, φ, ψ of the contrast frame 106C with respect to the reference frame 106S. Is calculated. Here, for example, Equation 3 and Equation 4 described above are used.

  In step S207, the angles θ, φ, and ψ obtained in step S206 are corrected based on the angles θ, φ, and ψ of the reference frame 106S, and the angles θ, φ, and ψ at the time of contrast frame shooting are obtained.

  In the angle calculation process, a plurality of comparison frames 106C may be compared with one reference frame 106S. That is, in the present embodiment, when the nth frame 106 is the reference frame 106S, only the n + 1th frame 106 is the comparison frame 106C. The n + 3th three frames 106 may be compared as a comparison frame 106C.

  Also, a plurality of first reference projection specific points a and first contrast projection specific points a ′, and second reference projection specific points b and second contrast projection specific points b ′ are set, and a comparison is made for each set. The coordinates of the projection index point s ′ may be calculated. In this case, by comparing the calculation results of the respective sets, it is possible to obtain a more accurate coordinate of the contrast projection index point s ′ that eliminates the effects of pattern matching errors, lens distortion (distortion aberration), and the like.

  Further, depending on conditions such as the presence of a feature in the background, pattern matching may be directly performed on the reference frame 106S and the contrast frame 106C. By doing so, there is a possibility that the processing load of the angle calculation means 130 can be reduced. In this case, the reference projection specific point and the contrast projection specific point can be obtained by projecting the region center of the extracted specific comparison region onto the virtual sphere 600.

  If a point indicating the same background position as the reference projection index point s can be extracted by pattern matching, the coordinates of the extracted point may be obtained as the coordinates of the contrast projection index point s ′. Good.

<Background selection processing>
Next, the details of the procedure of the background selection process in step S105 of the composition video generation process will be described. FIG. 16 is a flowchart showing the procedure of the background selection process, and FIGS. 17A and 17B are diagrams showing a specific method of the background selection process.

  First, in step S301, a spherical material-side projection frame 608 obtained by projecting all the pixels of the frame 107 of the material-side image 102 onto the virtual sphere 600 in the radial direction is generated (see FIG. 17A). ).

  In step S302, the material-side projection frame 608 and the selection three-dimensional image 700 generated in step S301 are subjected to pattern matching for each comparison area having a predetermined size, and the same in the material-side projection frame and the selection three-dimensional image 700. The specific comparison area Ac showing the pattern is extracted (see FIG. 17A). Here, first, a background-side projection frame 609 is generated by projecting the frame 106 of the background image 101 mapped in the selection three-dimensional image 700 onto the virtual sphere 600 (see FIG. 17A). Then, pattern matching is performed on the background-side projection frame 609 and the material-side projection frame 608 by the same method as in step S203 of the angle calculation process. When the selection three-dimensional image 700 is configured as a panoramic image, the selection three-dimensional image 700 can be used as it is for pattern matching. If the specific comparison area Ac is extracted, the area center of the specific comparison area Ac is set as the selection index point c, and the selection index is obtained from the spherical coordinates (θ, φ) of the point o corresponding to the optical center of the background-side projection frame 609. The spherical coordinates (θ, φ) on the virtual sphere 600 of the point c are obtained.

  In step S303, the spherical coordinates (θ, φ) of the point o ′ corresponding to the optical center of the material-side projection frame 608 are determined from the positional relationship between the selection index point c and the point o ′ corresponding to the optical center of the material-side projection frame 608. ) Is calculated geometrically. Each component of the spherical coordinates (θ, φ) of the point o ′ corresponding to the calculated optical center of the material-side projection frame 608 becomes a pan angle θ and a tilt angle φ when the material-side frame 107 is captured.

  In step S304, the roll angle ψ at the time of photographing the material side frame 107 is calculated from the inclination of the material side projection frame 608 relative to the background side projection frame 609 and the roll angle ψ of the background side projection frame 609.

  In step S305, the background side video angle database is referred to based on the pan angle θ and tilt angle φ at the time of photographing the material side frame 107 obtained in step S303, and the pan angle θ and tilt angle φ at the time of photographing the material side frame 107 are obtained. A plurality of frames 106 of the background side image 101 having the close pan angle θ and tilt angle φ are selected as the comparison frames 113 (see FIG. 17B). Specifically, the number of frames 106 of the background-side video 101 that is sufficient to cover the entire shooting area of the frame 107 of the material-side video 102 is selected as the comparison frames 113.

  Further, when there is a frame 106 of the background side image 101 having the same pan angle θ and tilt angle φ as the pan angle θ and tilt angle φ at the time of shooting the frame 107 of the material side image 102, this frame 106 is used for comparison. Select as frame 113. If there is an uncovered area because the sizes of the frame 106 and the frame 107 are different, the frame 106 for covering the uncovered area is further selected as the comparison frame 113.

  Depending on the conditions, pattern matching may be directly performed on the frame 106 of the background-side video 101 and the frame 107 of the material-side video 102. By doing so, there is a possibility that the processing load of the angle calculation means 130 can be reduced.

  Further, by performing the same process as the angle calculation process described above using the frame 106 of the background-side video 101 as a reference frame and the frame 107 of the material-side video 102 as a comparison frame, each angle θ of the frame 107 of the material-side video 102 is performed. , Φ, ψ may be obtained.

<Background comparison processing>
Next, details of the procedure of the background comparison process in step S106 of the composition video generation process will be described. FIG. 18 is a flowchart showing the procedure of the background comparison process. FIGS. 19A and 19B and FIGS. 20A and 20B show a specific method of the background comparison process. It is.

  First, in step S401, the frame 107 of the material side image 102 and the selected comparison frame 113 are mapped onto the virtual sphere 600 based on the spherical coordinates (θ, φ) corresponding to the pan angle θ and the tilt angle φ. (See FIG. 19A). In this mapping, the comparison frame 113 is rotated about a point corresponding to the optical center by its own roll angle ψ so that the background of the comparison frame 113 matches the position corresponding to the background on the celestial sphere 400. Make corrections. Similarly, the frame 107 of the material-side image 102 is also rotated around the point corresponding to the optical center by its own roll angle ψ so that the background coincides with the position corresponding to the background on the celestial sphere 400. To correct.

  In Step S402, the comparison correction frame 113M is generated by projecting the pixels of the comparison frame 113 mapped on the virtual sphere 600 onto the same plane as the frame 107 mapped on the virtual sphere 600 (FIG. 19 ( b)). The projection of the pixels of the comparison frame 113 is performed in the radial direction of the virtual sphere 600. For this reason, each pixel of the comparison frame 113 is projected on the same plane as the frame 107 without shifting from a position corresponding to the background position on the celestial sphere 400.

  With the above processing, the comparison frame 113M having the same angles θ, φ, and ψ as the frame 107 of the material-side video 102 can be generated. That is, the frame 107 of the material-side video 102 and the comparison frame 113M have the same background portion excluding the composition material 20 included in the frame 107 of the material-side video 102, as shown in FIG. It has become.

  In step S403, the frame 107 for the material-side video 102 and the comparison correction frame 113M are compared to generate the synthesis frame 110. Specifically, first, as shown in FIG. 20 (a), the frame 107 and the comparative correction frame 113M are divided into comparison areas CA and CA ′, which are areas of one pixel or more, and are placed at the same position. The difference between the color values in a certain comparison area CA and CA ′ is calculated. Here, being in the same position indicates that the coordinates are the same in the plane coordinate system on the frame, not in the spatial coordinate system described above.

  After the color value difference is calculated for all the comparison areas CA and CA ′, it is next determined whether or not the difference value in each comparison area CA and CA ′ exceeds a predetermined threshold value. If the color information of a specific color (for example, blue) that is set in advance is assumed to be the same background part, and the difference value exceeds a predetermined threshold value. The color information of the frame 107 is set on the assumption that the portion shows the composition material 20 portion. By executing the above determination for each comparison region in order and setting color information according to the determination result, it is possible to generate a composition frame 110 in which only the composition material 20 is cut out.

  As shown in FIG. 20A, the comparative correction frame 113M does not generate one frame having the same size as the frame 107 of the material-side video 102, but as shown in FIG. 20B. In addition, a plurality of comparison correction frames 113M may be generated for each of the plurality of selected comparison frames 113. In this case, for example, as shown in FIG. 20B, if four comparison frames 113Ma to 113Md are generated, the region 107 of the frame 107 covered by the comparison correction frame 113Ma is compared with the frame 107 and the comparison correction. The difference between the color values of the comparison areas CA and CA ′ is calculated for the frame 113Ma, and similarly, the difference between the color values of the comparison areas CA and CA ′ for the area Tb of the frame 107 and the comparison frame 113Mb for the comparison correction frame 113Mb. For the region Tc of the frame 107, the difference between the color values of the comparison regions CA and CA ′ is calculated for the frame 107 and the comparison correction frame 113Mc. For the region Td of the frame 107, the frame 107 and the comparison correction frame are calculated. The color of the comparison areas CA and CA 'for 113Md By calculating the difference between the values, the synthesis frame 110 can be generated.

  As described above, the video composition method and video composition system 100 according to the present embodiment uses a background camera to capture a multidirectional background that can be photographed by a background camera (video camera 200) arranged so as to be capable of panning and tilting. The material side photographed so that the composition material 20 is included under the same background by the photographed background image 101 and the material camera (video camera 200) arranged so as to be able to pan and tilt at the same position as the background camera. The image 102 is prepared, and the position of the index point S indicating the same background position in the plurality of frames 106 of the background-side image 101 is compared, so that the pan angle θ at the time of shooting of each frame 106 of the background-side image 101 is obtained. And an angle calculation process for calculating the tilt angle φ for each frame 106, and at the time of shooting each frame 107 of the material side image 102 Based on the pan angle θ and the tilt angle φ, a background selection process for selecting the comparison frame 113 from the frame 106 group of the background side image 101 for each frame 107 of the material side image 102, and the selected comparison frame 113 and the material side By comparing each frame 107 of the video 102, a background comparison process for generating a composite video 112 from which the composite material 20 is extracted (only the composite material 20 is cut out) is performed.

  Therefore, each frame of the background-side video 101 is not attached to the video camera 200 without attaching an encoder or the like for measuring the pan angle θ and the tilt angle φ, or using expensive and complicated equipment such as a motion control camera. The pan angle θ and the tilt angle φ at the time of shooting 106 can be obtained. Therefore, it is possible to generate the composition video 112 more efficiently than in the past while using the arbitrarily photographed background side video 101.

  In the present embodiment, an example in which the video camera 200 pans and tilts around the principal point 204 of the video camera 200 virtual lens 202 is shown, but the present invention is not limited to this. Even when the rotation center of panning and tilting of the video camera 200 is deviated from the principal point 204, for example, each frame 106 of the background side video 101 and each frame 107 of the material side video 102 are appropriately subjected to image processing. Can correct the deviation.

  In the angle calculation process, a virtual sphere 600 centered on the position of the video camera 200 (specifically, the main point 204 of the virtual lens 202) is set, and the index point S on the reference frame 106S is set as the radius of the virtual sphere 600. The reference projection index point s projected onto the surface of the virtual sphere 600 toward the direction and the index point S ′ on the contrast frame 106C toward the direction toward the radial direction of the virtual sphere 600 and projected onto the surface of the virtual sphere 600 By obtaining the index point s ′ and comparing the positions of the reference projection index point s and the contrast projection index point s ′, the pan angle θ and the tilt angle φ at the time of shooting each frame 106 of the background side image 101 are calculated.

  Therefore, the pan angle θ and the tilt angle φ at the time of shooting each frame 106 of the background side image 101 can be easily and accurately obtained without being affected by the image distortion on the frame 106.

  In the angle calculation process, the radius of the virtual sphere 600 is set based on the angle of view of the video camera 200 and the size of the frame 106 of the background side image 101. More specifically, the center of the virtual sphere 600 is set to the principal point 204 of the virtual lens 202 of the video camera 200, and the radius of the virtual sphere 600 is set to be the same as the focal length of the virtual lens 202 of the video camera 200. In this way, when a point on the frame 106 is projected onto the virtual sphere 600, a similar shape of the background on the celestial sphere 400 can be obtained.

  In addition, since the index point S is a point O corresponding to the optical center of the reference frame 106S, it is possible to eliminate the influence of roll rotation and to easily calculate the pan angle θ and the tilt angle φ. .

  Note that when the video camera 200 is panned with the tilt angle of the video camera 200 fixed in the horizontal direction and the background video 101 is shot, or when the video camera 200 is tilted and the background video 101 is shot, roll rotation is performed. Therefore, a point other than the point O corresponding to the optical center can be set as the index point S.

  In addition, the angle calculation process projects the reference frame 106S on the surface of the virtual sphere 600 in the radial direction of the virtual sphere 600, thereby generating the reference projection frame 607S and the comparison frame 106C in the radial direction of the virtual sphere 600. Is projected onto the surface of the virtual sphere 600 to generate a contrast projection frame 607C, and pattern matching is performed on the reference projection frame 607S and the contrast projection frame 607C, so that the same background position other than the index point S can be obtained. A plurality of projection specific points to be shown (region centers of specific comparison regions) are obtained, the projection specific points on the reference projection frame 607S are set as the reference projection specific points a and b, and the projection specific points on the contrast projection frame 607C are contrasted. Set to specific points a ′ and b ′ and based on the positions of reference projection specific points a and b and contrast projection specific points a ′ and b ′ Then, the position of the contrast projection index point s ′ is calculated. More specifically, the position of the contrast projection index point s ′ is calculated based on the relative positional relationship between the reference projection specific points a and b and the reference projection index point s.

  For this reason, even if it is difficult to find a point indicating the same background position as the index point S on the contrast projection frame 607C by pattern matching, the position of the contrast projection index point s ′ is calculated. be able to.

  In addition, the angle calculation process sets the first projection specific point and the second projection specific point as the projection specific points, and sets the first projection specific point a between the reference projection specific point a and the reference projection index point s. The first distance sa and the second distance sb between the reference projection specific point b of the second projection specific point and the reference projection index point s are obtained, and the second projection s is determined from the contrast projection specific point a ′ of the first projection specific point. A point on the surface of the virtual sphere 600 located at the second distance sa and the second projection specific point b 'from the contrast projection specific point b' is the contrast projection index point s'.

  For this reason, the position of the contrast projection index point s ′ can be accurately calculated by a geometric calculation method. However, the present invention is not limited to this, and the position of the contrast projection index point s ′ may be calculated by an approximate solution method.

  In addition, the angle calculation processing calculates the roll angle ψ at the time of shooting each frame 106 of the background side video 101 for each frame 106 of the background side video 101, and the background comparison processing shots each frame 106 of the background side video 101. Correction based on the current roll angle ψ is performed.

  For this reason, even if roll rotation occurs with panning or tilting of the video camera 200 at the time of shooting the background side image 101, the comparison frame 113 (or the comparison correction frame 113M) is used as the material side image 102. It is possible to match the background portion of the frame 107 and perform background comparison processing with high accuracy.

  Further, the angle calculation process connects a straight line connecting the reference projection specific point a of the predetermined projection specific point and the reference projection index point s, and the contrast projection specific point a ′ of the predetermined projection specific point and the contrast projection index point s ′. Based on the angle formed by the straight line, the roll angle ψ at the time of shooting each frame 106 of the background side image 101 is obtained.

  For this reason, since the roll angle ψ can be calculated using the points for calculating the pan angle θ and the tilt angle φ, the processing load of the angle calculation process can be reduced.

  Further, the background selection process compares the positions of the selection index points c indicating the same background position in the frame 106 of the background side video 101 and the frame 107 of the material side video 102, thereby making each frame 107 of the material side video 102. The pan angle θ and the tilt angle φ at the time of shooting are calculated for each frame 107.

  Therefore, the pan angle θ and the tilt angle φ at the time of shooting each frame 107 of the material-side image 102 are obtained without attaching an encoder or the like for measuring the pan angle θ and the tilt angle φ to the video camera 200. Can do. The comparison frame 113 can be efficiently selected based on the pan angle θ and the tilt angle φ.

  Further, in the background selection process, when the comparison frame 113 having the same pan angle θ and tilt angle φ as the pan angle θ and tilt angle φ at the time of shooting the frame 107 of the material side image 102 cannot be selected, A plurality of comparison frames 113 having a pan angle θ and a tilt angle φ close to the pan angle θ and the tilt angle φ at the time of shooting the frame 107 are selected.

  Therefore, regardless of the frame configuration of the background side video 101 and the material side video 102, the comparison frame 113 can be selected to perform the background comparison process.

  The background selection process calculates the roll angle ψ at the time of shooting each frame 107 of the material-side video 102 for each frame 107 of the material-side video 102, and the background comparison process captures each frame 107 of the material-side video 102. Correction based on the current roll angle ψ is performed.

  Therefore, the background comparison process can be performed with high accuracy even in the case where roll rotation occurs with panning or tilting of the video camera 200 at the time of shooting the material-side image 102.

  Further, in the background comparison process, when the pan angle θ or the tilt angle φ at the time of shooting of the frame 107 of the material side video 102 and the comparison frame 113 are not the same, the pan angle at the time of shooting of the frame 107 of the material side video 102 A comparison correction frame 113M in which the comparison frame 113 is corrected so as to coincide with θ and the tilt angle φ is generated, and the comparison correction frame 113M and the frame 107 of the material side image 102 are compared.

  Therefore, regardless of the frame configuration of the background side video 101 and the material side video 102, a comparative correction frame 113M that matches the background portion of the frame 107 of the material side video 102 is generated to compare the background with high accuracy. Processing can be performed.

  In the background comparison process, each pixel of the comparison frame 113 is projected on the same plane as the frame 107 of the material-side image 102 in the radial direction of the virtual sphere 600, thereby generating a comparative correction frame 113M. .

  Therefore, the comparison correction frame 113M can be exactly matched with the background portion of the frame 107 of the material-side video 102.

  Although the embodiments of the present invention have been described above, the video composition method and the video composition system of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention. Of course, can be added.

  The video composition method and video composition system of the present invention can be used in the field of various video generation.

DESCRIPTION OF SYMBOLS 10,101 Background side image 10a, 113 Comparison frame 12, 102 Material side image 14, 112 Composition image 20 Composition material 100 Image composition system 104 Background side image storage means 105 Material side image storage means 106 Frame of background side image 106C Contrast frame 106S Reference frame 107 Material side image frame 113M Comparison correction frame 130 Angle calculation means 136 Background selection means 138 Background comparison means 200 Video camera 202 Virtual lens 204 Virtual lens principal point 600 Virtual sphere 607C Contrast projection frame 607S Reference Projection frame O Point corresponding to optical center of frame a First reference projection specific point a 'First contrast projection specific point b Second reference projection specific point b' Second contrast projection specific point c Selection index point S, S 'Index point s Reference projection Mark s' Contrast projection index point sa Distance between the reference projection index point s and the first reference projection specific point a sb Distance between the reference projection index point s and the second reference projection specific point b θ Pan angle φ Tilt angle ψ Roll angle

Claims (22)

A background side image obtained by photographing a multidirectional background that can be photographed by a background camera arranged so as to be capable of panning and tilting, and the background camera;
A material-side image shot so that the composition material is included under the background by a material camera arranged so as to be able to pan and tilt at the same position as the background camera; and
An angle calculation process for calculating a pan angle and a tilt angle for each frame of the background side video at the time of shooting by comparing the positions of index points indicating the same background position in a plurality of frames of the background side video. When,
A background selection process for selecting a comparison frame from a frame group of the background-side video for each frame of the material-side video based on a pan angle and a tilt angle at the time of shooting each frame of the material-side video;
A background comparison process for generating a composition image from which the composition material is extracted is performed by comparing the selected comparison frame and each frame of the material-side image.
Video composition method. The angle calculation process includes:
Set a virtual sphere centered on the background camera position,
Of the two frames to be compared, the reference projection index point projected on the surface of the virtual sphere with the index point on one reference frame directed in the radial direction of the virtual sphere, and the index on the other contrast frame Find a contrast projection index point projected on the surface of the virtual sphere with the point in the radial direction of the virtual sphere,
By comparing the positions of the reference projection index point and the contrast projection index point, a pan angle and a tilt angle at the time of shooting each frame of the background side image are calculated.
The video composition method according to claim 1. The angle calculation process includes:
The radius of the virtual sphere is set based on the angle of view of the background camera and the frame size of the background side image,
The video composition method according to claim 2. The angle calculation process includes:
Set the center of the virtual sphere as the principal point of the lens of the background camera,
The radius of the virtual sphere is set equal to the focal length of the lens of the background camera,
The video composition method according to claim 3. The index point is a point corresponding to the optical center of the reference frame,
The video composition method according to claim 2. The angle calculation process includes:
By projecting the reference frame on the surface of the virtual sphere in the radial direction of the virtual sphere, a reference projection frame is generated,
By projecting the contrast frame on the surface of the virtual sphere in the radial direction of the virtual sphere, a contrast projection frame is generated,
By performing pattern matching for the reference projection frame and the contrast projection frame, a plurality of projection specific points indicating the same background position other than the index point are obtained,
While setting the projection specific point on the reference projection frame as a reference projection specific point, setting the projection specific point on the contrast projection frame as a contrast projection specific point,
Based on the position of the reference projection specific point and the contrast projection specific point, the position of the contrast projection index point is calculated,
The video composition method according to claim 5. The angle calculation process calculates a position of the contrast projection index point based on a relative positional relationship between the reference projection specific point and the reference projection index point.
The video composition method according to claim 6. The angle calculation process includes:
Set the first projection specific point and the second projection specific point as the projection specific point,
A first distance between the reference projection specific point of the first projection specific point and the reference projection index point, and between the reference projection specific point of the second projection specific point and the reference projection index point Find the second distance,
On the surface of the virtual sphere located at the first distance from the contrast projection specific point of the first projection specific point and the second distance from the contrast projection specific point of the second projection specific point. A point is the contrast projection index point,
The video composition method according to claim 7. The angle calculation process calculates a roll angle at the time of shooting each frame of the background side video for each frame of the background side video,
The background comparison process is characterized by performing correction based on a roll angle at the time of shooting each frame of the background side video,
The video composition method according to claim 1. The angle calculation process includes a straight line connecting the reference projection specific point of the predetermined projection specific point and the reference projection index point, and a straight line connecting the contrast projection specific point of the predetermined projection specific point and the contrast projection index point. Based on the angle formed by and, obtain the roll angle at the time of shooting each frame of the background side video,
The background comparison process is characterized by performing correction based on a roll angle at the time of shooting each frame of the background side video,
The video composition method according to claim 6. The background selection process compares the position of the index point for selection indicating the same background position in the frame of the background side video and the frame of the material side video, thereby panning at the time of shooting each frame of the material side video. The angle and tilt angle are calculated for each frame,
The video composition method according to claim 1. When the background selection process cannot select a pan angle and tilt angle comparison frame that is the same as the pan angle and tilt angle at the time of capturing the material-side image frame, the pan angle at the time of capturing the material-side image frame And selecting a plurality of pan and tilt angle comparison frames close to the tilt angle.
The video composition method according to claim 11. The background selection process calculates a roll angle for each frame of the material-side video for each frame of the material-side video,
The background comparison process is characterized by performing correction based on a roll angle at the time of shooting each frame of the material side video.
The video composition method according to claim 11 or 12. In the background comparison process, when the pan angle or tilt angle at the time of shooting the material side video frame and the comparison frame are not the same, the background comparison processing matches the pan angle and tilt angle at the time of shooting the material side video frame. Generating a comparative correction frame in which the comparison frame is corrected, and comparing the comparative correction frame and the frame of the material side video,
The video composition method according to claim 1. In the background comparison process, the comparison correction frame is generated by correcting the comparison frame so as to coincide with a roll angle at the time of shooting the frame of the material side image.
The video composition method according to claim 14. The background comparison process generates the comparison correction frame by projecting each pixel of the comparison frame on the same plane as the frame of the material side image in the radial direction of the virtual sphere. And
The video composition method according to claim 14 or 15. Background-side image storage means for storing a background-side image obtained by photographing the multi-directional background that can be photographed by the background camera arranged so as to be capable of panning and tilting;
Material-side video storage means for storing a material-side video shot so that a material for composition is included under the background by a material camera arranged so as to be capable of panning and tilting at the same position as the background camera;
An angle calculation means for calculating the pan angle and the tilt angle for each frame of the background-side video for each frame by comparing the positions of index points indicating the same background position in the plurality of frames of the background-side video. When,
Background selection means for selecting a comparison frame from a frame group of the background-side video for each frame of the material-side video based on a pan angle and a tilt angle at the time of shooting each frame of the material-side video;
A background comparison unit that generates a composition image in which the composition material is extracted by comparing the selected comparison frame and each frame of the material-side image,
Video composition system. The angle calculation means calculates a roll angle at the time of shooting each frame of the background side video for each frame of the background side video,
The background comparison means performs correction based on a roll angle at the time of shooting each frame of the background side video,
The video composition system according to claim 17. The background selection means compares the position of the index point for selection indicating the same background position in the frame of the background side video and the frame of the material side video, thereby panning at the time of shooting each frame of the material side video. The angle and tilt angle are calculated for each frame,
The video composition system according to claim 17 or 18. The background selection means calculates a roll angle at the time of shooting each frame of the material side video for each frame of the material side video,
The background comparison means performs correction based on a roll angle at the time of shooting each frame of the material side video,
The video composition system according to claim 19. The background comparing means matches the pan angle and tilt angle at the time of shooting the frame of the material side image when the frame of the material side image to be compared and the pan angle or tilt angle at the time of shooting of the comparison frame are not the same. Generating a comparative correction frame in which the comparison frame is corrected, and comparing the comparative correction frame and the frame of the material side video,
The video composition system according to any one of claims 17 to 20. The background comparison means generates the comparison correction frame in which the comparison frame is corrected so as to coincide with a roll angle at the time of shooting the frame of the material-side video.
The video composition system according to claim 21.

Images