JP2007195091A - Synthetic image generating system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable to make setting so that a virtual camera cannot be located nor directed in the direction to which one does not want to direct a position to which one does not want the one to move a visual point, and line of sight at a content maker side. <P>SOLUTION: At a transmitting side, limiting/permission information of visual point position of the virtual camera or the direction of the line of sight are beforehand defined by a content maker. At a recipient side, the above-mentioned limiting/permission information are acquired with video data. A virtual camera position/direction decision unit 204 of the recipient side forms control information from intelligence on the visual point position/direction of the line of sight of the virtual camera input from an input device 206 and the acquired above-mentioned limiting/permission information, and sends to an arbitrary visual point image formation 203. The arbitrary visual point image formation 203 synthesizes the arbitrary visual point image, using received video data and camera parameter and control information sent from the virtual camera position/direction decision unit 204. The compound video image is output to a display unit 205. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a composite video generation system, and more specifically, from video information obtained by a large number of cameras arranged in different viewpoint positions and line-of-sight directions, in other words, videos in viewpoint positions and line-of-sight directions where no cameras actually exist, The present invention relates to a composite video generation system that synthesizes video at a virtual camera position and direction.

  Arbitrary viewpoint video generation technology is being researched by various research institutions and universities. This is a technique for synthesizing a video when a virtual camera is placed at a point where a camera does not exist by shooting a scene using a large number of cameras and processing the video data. For example, there are technologies such as Non-Patent Document 1 and Non-Patent Document 2.

  An example of the arbitrary viewpoint video generation system is shown in FIG. FIG. 9 schematically shows the stage 1 being photographed using a plurality of cameras 2. Arbitrary viewpoint video generation technology generates video from these video images of the camera 2 as if it were shot from the position of the virtual camera 3 where no camera actually exists.

By using this technique, for example, the user can operate the position and direction of the virtual camera 3 to track the specific subject and view the video. For example, taking sports broadcasts as an example, you can always track and watch your favorite players. In addition, if used for theatrical performances and concerts, you can see the performers and performers from their own direction. It can also be used for content such as dramas.
Kitahara, Sato, Ota, “3D image display with motion parallax reproducible using multi-eye stereo method-Generation and evaluation of display image-”, Television Society Journal, Vol. 50, no. 9, (1996) Primm, Fujii, Tanimoto, "Real-time system for free viewpoint television", IEICE Technical Report, IE2002-120 (2002)

  In the case of such an arbitrary viewpoint video system, the content provider side may have a position where the viewpoint is not desired to move or a direction where the viewpoint is not desired to be directed. For example, there may be a case where it is not desired to move to a viewpoint position that can be seen up to the back of the stage set in a live broadcast. Or, in content such as dramas, you don't want the line of sight to look in the direction of the equipment and staff on the shooting side.

  In addition, there may be a viewpoint position and a line-of-sight direction in which an arbitrary viewpoint video cannot be generated or only an error-prone video can be generated due to the camera arrangement. For example, if the viewpoint is too close to the subject, the resolution of the camera video becomes insufficient, and only a blurred video can be generated. Or, if the viewpoint goes under the ground, the video cannot be generated. Alternatively, there may be a line-of-sight direction that is not taken from any camera and cannot be combined. In such a case, it is desirable to restrict the viewpoint from being close to a certain distance from the beginning, or restrict the viewpoint position so that it cannot move below the ground, or restrict the line-of-sight direction.

  The present invention has been made in view of the circumstances as described above, and allows a content provider in an arbitrary viewpoint video system to set an area in which the viewpoint cannot be moved in advance and a direction in which the line of sight cannot be directed, Or it makes it a subject to enable it to set the area | region which may move a viewpoint conversely, and the direction which may turn a line of sight.

  The first means of the present invention for solving the above-mentioned problem is to generate a synthesized video that synthesizes videos from a plurality of different viewpoint positions and line-of-sight directions and synthesizes videos at arbitrary virtual viewpoint positions and line-of-sight directions. The system is characterized in that the restriction information for the virtual viewpoint position when compositing the video is added to the video data and transmitted and recorded.

  The second technical means is characterized in that, in the first technical means, permission information for a virtual viewpoint position when the video is synthesized is added to the video data and transmitted and recorded.

  The third technical means is characterized in that, in the first or second technical means, restriction information for a virtual line-of-sight direction when compositing video is added to video data and transmitted and recorded. is there.

  According to a fourth technical means, in any one of the first to third technical means, permission information for a virtual line-of-sight direction when compositing a video is added to video data and transmitted and recorded. It is what.

  The fifth technical means is any one of the first to fourth technical means, wherein the restriction information for the virtual viewpoint position, the permission information for the virtual viewpoint position, the restriction information for the virtual gaze direction, and the virtual One or a plurality of pieces of permission information for a specific line-of-sight direction is designated by a mathematical expression representing a plane.

  The sixth technical means is any one of the first to fourth technical means, wherein the restriction information for the virtual viewpoint position, the permission information for the virtual viewpoint position, the restriction information for the virtual line-of-sight direction, and the virtual One or a plurality of pieces of permission information for a specific line-of-sight direction is designated by a mathematical expression representing a spherical surface.

  The seventh technical means is the fifth or sixth technical means, wherein the restriction information for the virtual viewpoint position, the permission information for the virtual viewpoint position, the restriction information for the virtual viewing direction, and the virtual viewing direction One or more of the permission information for is an AND or logical sum of one or more pieces of restriction information specified by a mathematical expression representing a plane or a sphere, or one or more permission information specified by a mathematical expression representing a plane or a sphere It is characterized by being specified by the logical product or logical sum of.

  In an eighth technical means, in any one of the first to seventh technical means, restriction information for a virtual viewpoint position, permission information for a virtual viewpoint position, restriction information for a virtual gaze direction, and virtual One or a plurality of frame rates of the permission information for a specific line-of-sight direction is the same as the frame rate of the video data.

  The ninth technical means is any one of the first to seventh technical means, wherein the restriction information for the virtual viewpoint position, the permission information for the virtual viewpoint position, the restriction information for the virtual line-of-sight direction, and the virtual One or a plurality of frame rates of permission information for a specific line-of-sight direction is different from the frame rate of video data.

  According to a tenth technical means, in any one of the first to ninth technical means, information on a predetermined standard viewpoint position / line-of-sight direction with respect to a virtual viewpoint position and line-of-sight direction when the video is synthesized Is added to video data and transmitted and recorded.

  The eleventh technical means is characterized in that, in the tenth technical means, the frame rate of the standard viewpoint position / gaze direction information is the same as the frame rate of the video data.

  The twelfth technical means is characterized in that, in the tenth technical means, the frame rate of the standard viewpoint position / gaze direction information is different from the frame rate of the video data.

  A thirteenth technical means is the first technical means according to any one of the first to twelfth technical means, which has an input device on the data reception side or reproduction side, and receives an input of a virtual viewpoint position and line-of-sight direction by the input device. It is characterized by that.

  The fourteenth technical means is characterized in that in the thirteenth technical means, the video is synthesized in accordance with the virtual viewpoint position and line-of-sight direction input by the input device.

  In a fourteenth technical means, a fifteenth technical means determines and processes the virtual viewpoint position and line-of-sight direction input by the input device according to the restriction information and / or permission information added to the video data. Control is performed so as not to generate an image at the viewpoint position and the line-of-sight direction that matches the information or does not match the permission information.

  The sixteenth technical means is any one of the thirteenth to fifteenth technical means, wherein the input device has a force feedback mechanism.

  The seventeenth technical means is the sixteenth technical means, in which the virtual viewpoint position / gaze direction input by the input device matches the restriction information added to the video data, or added to the video data. When the information does not match the permission information, a control signal is sent to the force feedback mechanism to control the user to give a physical stimulus.

  An eighteenth technical means includes a changeover switch according to any one of the thirteenth to seventeenth technical means, and synthesizes an image according to a virtual viewpoint position / line-of-sight direction input by an input device, or a standard It is characterized in that it is possible to switch with a changeover switch whether to synthesize a video according to information on a viewpoint position and a line-of-sight direction.

  In a nineteenth technical means according to any one of the thirteenth to eighteenth technical means, when the input of the virtual viewpoint position / gaze direction by the input device is not performed for a certain period of time, the standard viewpoint position / It is characterized in that it is automatically switched to synthesize a video according to the information on the line of sight.

  In a twentieth technical means, in any one of the thirteenth to nineteenth technical means, when the video is synthesized in accordance with the information of the standard viewpoint position / gaze direction, the virtual viewpoint position / When the line-of-sight direction is input, the video is automatically switched to synthesize the video according to the input virtual viewpoint position and line-of-sight direction.

  In a twenty-first technical means, in any one of the thirteenth to twentieth technical means, the virtual viewpoint position / line-of-sight direction input by the input device is used as the restriction information and / or permission information added to the video data. If the virtual viewpoint position / gaze direction matches the restriction information or does not match the permission information, the virtual viewpoint position / gaze direction input by the input device is not used. An image is generated using information on a standard viewpoint position and line-of-sight direction.

According to the present invention, in the arbitrary viewpoint video system, it is possible to set a region in which the viewpoint cannot be moved and a direction in which the line of sight cannot be directed in advance on the content provider side. It is possible to set a good region and a direction in which the line of sight may be directed.
In addition, by adding information on the predetermined standard viewpoint position / gaze direction to the video data, if the viewpoint position / gaze direction input by the user is not appropriate, instead of the standard viewpoint position / gaze direction, Arbitrary viewpoint video can be synthesized using information on the line of sight. Further, even when there is no input of the viewpoint position / gaze direction by the user, it is possible to synthesize an arbitrary viewpoint video using standard viewpoint position / gaze direction information.
Also, by having a viewpoint position / line-of-sight input device and adding a force feedback mechanism to this, when the viewpoint position / line-of-sight input by the user is not appropriate, the user is physically stimulated You can let it know.

<First embodiment>
A first embodiment of the present invention will be described below. The first embodiment shows an application example when this patent is applied to television broadcasting.
FIG. 1 is a diagram for explaining a first embodiment of a composite video generation system of the present invention, and shows a block diagram of a broadcasting station side system of the composite video system. The system on the broadcasting station side includes a camera 101, a camera parameter recording unit 102, a virtual camera position / direction restriction / permission information recording unit 103, a data multiplexing processing unit 104, and a transmission unit 105.

One or more cameras 101 are arranged around the subject to obtain video data obtained by photographing the subject from one or more viewpoint positions and directions.
The camera parameter recording unit 102 is a part that holds camera parameters. The camera parameters include so-called internal parameters such as the focal length of each camera and lens distortion information, and external parameters representing the relative positional relationship between the cameras. These camera parameters are required at the stage of synthesizing an arbitrary viewpoint video.

  The virtual camera position / direction restriction / permission information recording unit 103 is a part for recording the virtual camera viewpoint position and line-of-sight direction restriction / permission information. These pieces of information are preliminarily determined and recorded by the content producer. The restriction / permission information regarding the viewpoint position and the line-of-sight direction of the virtual camera may be data having the same frame rate as the frame rate of the video data, or may be data having a different frame rate. For example, when the restriction position / permission information of the viewpoint position and the line-of-sight direction of the virtual camera is the same over a plurality of frames, the data may not be sent during that time.

The data multiplexing processing unit 104 is recorded in the video data captured by the camera 101, the camera parameters recorded in the camera parameter recording unit 102, and the virtual camera position / direction restriction / permission information recording unit 103. The multiplexed stream is generated by multiplexing the restriction / permission information of the viewpoint position and line-of-sight direction of the virtual camera.
The transmission unit 105 places the multiplexed stream on the broadcast radio wave.
This process is merely an example. In addition to this, a method of compressing and multiplexing the video from the camera 101, and a method of transmitting each data in the form of a plurality of streams without multiplexing each data are also conceivable.

  FIG. 2 is a diagram for explaining an example of a description method of restriction / permission information of the viewpoint position of the virtual camera. Consider a rectangular parallelepiped 111 having sides of x1, y1, and z1 as shown in FIG.

First, as a condition for limiting the viewpoint position of the virtual camera, the inside of the rectangular parallelepiped 111 is set as a prohibited area. That is, it is assumed that no virtual camera can be placed inside the rectangular parallelepiped 111.
The six planes of the rectangular parallelepiped 111 are described by the following six mathematical expressions.

x = 0 (1)
x = x1 (2)
y = 0 (3)
y = y1 (4)
z = 0 (5)
z = z1 (6)
Get in.

From here, the inner region of the rectangular parallelepiped 111 can be described by the logical product of the following six inequalities.
x> 0 (7)
x <x1 (8)
y> 0 (9)
y <y1 (10)
z> 0 (11)
z <z1 (12)

If the coordinates of a point apply to all six inequalities, the point is in the inner region of the cuboid 111. If even one of them does not apply, the point is in the outer region of the rectangular parallelepiped 111.
For example, when (x1, y1, z1) = (1, 1, 1), the coordinates are (x, y, z) = (0.5, 0.5, 0.5). 7) thru | or all the inequalities of Formula (12), and it turns out that it is located in the internal region of the rectangular parallelepiped 111. For this reason, the viewpoint position of the virtual camera cannot move to this point. Further, for example, the point of coordinates (x, y, z) = (1.5, 0.5, 0.5) does not apply to the equation (8), and therefore may be located in the external region of the rectangular parallelepiped 111. Recognize. For this reason, the viewpoint position of the virtual camera is movable at this point.

Further, the outer region of the rectangular parallelepiped 111 can be described by the logical sum of the following six inequalities.
x <0 (13)
x> x1 (14)
y <0 (15)
y> y1 (16)
z <0 (17)
z> z1 (18)

  If the coordinates of a point apply to one of these six inequalities, the point is in the outer region of the cuboid 111. If not all of them, the point is in the inner region of the rectangular parallelepiped 111.

For example, it can be seen that the coordinates (x, y, z) = (1.5, 0.5, 0.5) are located in the external region of the rectangular parallelepiped 111 because at least the expression (14) applies. For this reason, the viewpoint position of the virtual camera is movable at this point.
Further, for example, the point that the coordinates are (x, y, z) = (0.5, 0.5, 0.5) does not apply to all the inequalities of the equations (13) to (18), and thus the rectangular parallelepiped 111 It can be seen that it is located in the inner region. For this reason, the viewpoint position of the virtual camera cannot move to this point.

  Thus, an area that can be expressed by a combination of one or more planes can be defined by an inequality that describes the plane and its logical product or logical sum. In addition, it is possible to determine which region a point belongs to from their definition.

In the same way, it is possible to use the outside of the rectangular parallelepiped 111 as a restricted area, that is, the viewpoint position of the virtual camera can move only inside the rectangular parallelepiped 111. Furthermore, in the same way, it is possible to set the inside of the rectangular parallelepiped 111 as a permission region, that is, the viewpoint position of the virtual camera can move only within the rectangular parallelepiped 111. Furthermore, in the same way, it is also possible to set the outside of the rectangular parallelepiped 111 as a permitted area, that is, the viewpoint position of the virtual camera can move only outside the rectangular parallelepiped 111.
As in the above example, it is possible to describe and determine virtual camera position restriction / permission information.

In the above, for simplicity, an example where the plane is simple such as x <0 is taken as an example. A plane can generally be described as ax + by + cz = 1. The same thing as above can be realized by using a plane expression based on such a general description.
In addition, a method of defining a region by a curved surface that can be described by a mathematical expression such as a spherical surface, or a method of defining a region by one or more polygons is also conceivable. In the present invention, all of these methods can be used as the description method of the restriction / permission information of the viewpoint position of the virtual camera.
Furthermore, the above-described area description method is an example, and the present invention can be applied to cases where other area description methods are used.

  Next, an example of a description method of restriction / permission information on the viewing direction of the virtual camera will be described. First, a schematic diagram of the camera is shown in FIG. FIG. 3 shows a state where the lens 122 is attached to the camera body 121. Further, a direction toward the subject through the center of the imaging surface and the center of the lens is set as a camera viewing direction 123.

FIG. 4 is a diagram for explaining an example of a description method of restriction / permission information on the viewing direction of the virtual camera. As shown in FIG. 4, consider a sphere 112 whose center is (x2, y2, z2) and whose radius is R.
First, as a condition for limiting the viewing direction of the virtual camera, consider a case where a viewing direction in which a straight line formed by the viewing direction 123 of the camera passes through any region inside the sphere 112 is prohibited.

In this case, the distance between the straight line in the line-of-sight direction 123 of the camera and the center of the sphere 112 is calculated, and the distance is compared with the radius R. If the distance is smaller than the radius R, the viewing direction 123 of the camera passes through the area inside the sphere 112. If the distance is larger than the radius R, the camera viewing direction 123 does not pass through the area inside the sphere 112.
Therefore, a viewing direction in which the distance between the straight line formed by the viewing direction 123 of the camera and the center of the sphere 112 is smaller than the radius R is prohibited. Note that this restriction condition is the same as a condition that permits a line-of-sight direction in which a straight line formed by the line-of-sight direction 123 of the camera does not pass through any region inside the sphere 112.

On the contrary, when the direction in which the straight line formed by the camera line-of-sight direction 123 passes through any region inside the sphere 112, the distance between the straight line formed by the camera line-of-sight direction 123 and the center of the sphere 112 is A line-of-sight direction that is smaller than the radius R is permitted. This permission condition is the same as the condition that restricts the line-of-sight direction in which the straight line formed by the line-of-sight direction 123 of the camera does not pass through any region inside the sphere 112.
As in the above example, it is feasible to describe and determine the restriction / permission information of the viewing direction of the virtual camera.
It should be noted that the above-described description method of the restriction / permission information of the viewing direction of the virtual camera is an example, and the present invention can also be applied when other description methods are used.

  FIG. 5 is another diagram for explaining the first embodiment of the composite video generation system of the present invention, and shows a block diagram of the receiving side system of the composite video system. The reception-side system includes a reception unit 201, a multiplexed data separation processing unit 202, an arbitrary viewpoint video generation unit 203, a virtual camera position / direction determination unit 204, a display unit 205, and an input device 206.

The receiving unit 201 receives the multiplexed stream transmitted by the transmitting unit 105 in FIG. 1 and sends it to the multiplexed data separation processing unit 202.
The multiplexed data separation processing unit 202 separates the multiplexed stream, and obtains video data, camera parameters, and virtual camera viewpoint position and gaze direction restriction / permission information. The video data and camera parameters are sent to the arbitrary viewpoint video generation unit 203, and the virtual camera viewpoint / line-of-sight direction restriction / permission information is sent to the virtual camera position / direction determination unit 204.

The input device 206 is a device for inputting information on the viewpoint position and line-of-sight direction of the virtual camera. For example, a joystick can be used. Further, the input device 206 may include a force feedback mechanism.
The virtual camera position / direction determination unit 204 receives information on the viewpoint position / line-of-sight direction of the virtual camera input from the input device 206 and the viewpoint position and line-of-sight direction of the virtual camera sent from the multiplexed data separation processing unit 202. Control information is generated from the restriction / permission information, and the control information is sent to the arbitrary viewpoint video generation unit 203. If the input device 206 has a force feedback mechanism, the control is performed. An example of the procedure will be described later.

The arbitrary viewpoint video generation unit 203 synthesizes an arbitrary viewpoint video using the video data and camera parameters sent from the multiplexed data separation processing unit 202 and the control information sent from the virtual camera position / direction determination unit 204. The synthesized video is output to the display unit 205.
The display unit 205 is a device that displays the video generated by the arbitrary viewpoint video generation unit 203. A case of a general two-dimensional display or a case of a three-dimensional display capable of stereoscopic viewing can be considered.

An example of a processing procedure in the virtual camera position / direction determination unit 204 is shown in FIG.
First, the viewpoint position / gaze direction data of the virtual camera is acquired from the input device 206 (S1). Then, it is determined whether or not the acquired viewpoint position / line-of-sight direction satisfies a restriction condition determined by restriction information on the viewpoint position and line-of-sight direction of the virtual camera (S2).

If it does not match the determination of S2 (in the case of No), next, whether the viewpoint position / gaze direction acquired from the input device 206 matches the permission conditions determined by the permission information of the viewpoint position and the gaze direction of the virtual camera. A determination process is performed (S3).
If the determination in S3 is met (in the case of Yes), the viewpoint position / gaze direction data is sent to the arbitrary viewpoint video generation unit 203 (S4).

If the determination in S2 is met (in the case of Yes), that is, the viewpoint position / gaze direction acquired from the input device 206 is not appropriate, it is fed back to the input device 206 (S5). As a feedback method, a method of causing the input device 206 to vibrate and notifying the user, or a method of generating a reaction force so that the viewpoint cannot be moved or the direction of the line of sight cannot be changed is considered.
Thereafter, data is generated by shifting the viewpoint position / gaze direction from the viewpoint position / gaze direction acquired from the input device 206 so as to satisfy the restriction / permission information of the viewpoint position and gaze direction of the virtual camera (S6). Then, the generated new viewpoint position / gaze direction data is sent to the arbitrary viewpoint video generation unit 203 (S4).

If the determination at S3 does not match (No), the viewpoint position / line-of-sight direction acquired from the input device 206 is not appropriate, and it is fed back to the input device 206 (S5).
Thereafter, data is generated by shifting the viewpoint position / gaze direction from the viewpoint position / gaze direction acquired from the input device 206 so as to satisfy the restriction / permission information of the viewpoint position and gaze direction of the virtual camera (S6). The generated viewpoint position / gaze direction data is sent to the arbitrary viewpoint video generation unit 203 (S4).

  Among the above procedures, data is generated by shifting the viewpoint position / gaze direction from the viewpoint position / gaze direction acquired from the input device 206 so as to satisfy the restriction / permission information of the viewpoint position and gaze direction of the virtual camera in S6. A specific example of the method will be described.

  First, a method of shifting the viewpoint position so as to satisfy the virtual camera position restriction / permission information will be described. In the example described with reference to FIG. 2, a combination of planes is used as the virtual camera position restriction / permission information. Therefore, the distance between the viewpoint position acquired from the input device 206 and each plane is obtained. A point closest to the viewpoint position acquired from the input device 206 is obtained on the plane having the closest distance. Then, the closest point is set as a new viewpoint position corrected.

  Next, a method for shifting the line-of-sight direction so as to satisfy the virtual camera direction restriction / permission information will be described. In the example of the description using FIGS. 3 and 4 described above, a spherical surface is used as the restriction / permission information in the virtual camera direction. Therefore, a point closest to the center point of the sphere on the straight line formed by the line-of-sight direction acquired from the input device 206 is obtained. The intersection point between the straight line formed by this point and the center point of the sphere and the spherical surface is obtained. The line-of-sight direction is corrected so that a straight line formed by the line-of-sight direction of the virtual camera passes through this intersection.

By adopting the above procedure, data obtained by shifting the viewpoint position / gaze direction from the viewpoint position / gaze direction acquired from the input device 206 so as to satisfy the restriction / permission information of the viewpoint position and gaze direction of the virtual camera is obtained. It is possible to create.
In addition, said method is an example and this invention is applicable also when other methods are used.

<Second embodiment>
A second embodiment of the present invention will be described below. The second embodiment shows another application example when the present invention is applied to television broadcasting.
FIG. 7 is a diagram for explaining a second embodiment of the composite video generation system of the present invention, and shows a block diagram of a broadcasting station side system of the composite video system. The system on the broadcasting station side in this embodiment is almost the same as the configuration in FIG. 1, but a standard viewpoint position / gaze direction information recording unit 106 is added.

  The standard viewpoint position / line-of-sight direction information recording unit 106 is a part for recording information on a standard viewpoint position / line-of-sight direction when an arbitrary viewpoint video is synthesized. This information is determined and recorded in advance by the content producer. Note that the standard viewpoint position / line-of-sight information when combining arbitrary viewpoint videos may be data having the same frame rate as the frame rate of the video data, or data having a different frame rate. Also good. For example, when the standard viewpoint position and line-of-sight direction are the same over a plurality of frames, it is not necessary to send data during that time.

  In the data multiplexing processing unit 104, video data captured by the camera 101, camera parameters recorded in the camera parameter recording unit 102, and virtual camera position / direction restriction / permission information recording unit 103 are recorded. Restriction / permission information of viewpoint position and line-of-sight direction of virtual camera and standard viewpoint position / line-of-sight information when combining arbitrary viewpoint video recorded in standard viewpoint position / line-of-sight direction information recording unit 106 are multiplexed. To generate a multiplexed stream.

FIG. 8 is another diagram for explaining the second embodiment of the composite video generation system of the present invention, and shows a block diagram of the reception side system of the composite video system. The system on the receiving side is almost the same as the configuration in FIG. 5, but a switch 207 is added.
The multiplexed data separation processing unit 202 separates the multiplexed stream, and standard viewpoint position when synthesizing video data and camera parameters, virtual camera viewpoint position and gaze direction restriction / permission information, and arbitrary viewpoint video・ Get information on the line of sight. The video data and camera parameters are sent to the arbitrary viewpoint video generation unit 203, and the viewpoint position and line-of-sight direction restriction / permission information of the virtual camera and the standard viewpoint position / line-of-sight information when combining arbitrary viewpoint video are virtual. It is sent to the camera position / direction determination unit 204.

  When the virtual camera position / direction determination unit 204 generates control information, the switch 207 uses information on the viewpoint position / line-of-sight direction of the virtual camera input from the input device 206, or from the multiplexed data separation processing unit 202. This is used to select whether to use standard viewpoint position / line-of-sight direction information when the sent arbitrary viewpoint video is synthesized.

  Although it is convenient for the user of the system to freely select the viewpoint position and line-of-sight direction of the virtual camera, it can be said that it is troublesome to always change the viewpoint position and line-of-sight direction of the virtual camera. In some cases, it may be desirable to automatically change the viewpoint position and line-of-sight direction of the virtual camera. Therefore, if you want to automatically change the viewpoint position and line-of-sight direction of the virtual camera, use the standard viewpoint position and line-of-sight information when combining arbitrary viewpoint videos sent from the broadcast station. This problem can be solved by generating The switch 207 can freely select manual determination and automatic determination of the viewpoint position and line-of-sight direction of the virtual camera.

  In addition, when there is no input from the input device 206 for a certain period of time, there is a mechanism for switching to use standard viewpoint position / line-of-sight information when automatically synthesizing an arbitrary viewpoint video sent from the broadcasting station side. It is also possible to add. Furthermore, when there is an input from the input device 206 when synthesizing the video using the information of the standard viewpoint position / gaze direction when synthesizing the arbitrary viewpoint video sent from the broadcasting station side, It is also possible to add a mechanism for switching to synthesize a video using the viewpoint position / gaze direction automatically acquired from the input device 206.

  Furthermore, when the video is synthesized using the viewpoint position / gaze direction acquired from the input device 206, the information on the viewpoint position / gaze direction acquired from the input device 206 is limited to the viewpoint position and the gaze direction of the virtual camera. In the case where the permission information is not satisfied, it is also possible to adopt a mechanism for generating a video by using standard viewpoint position / gaze direction information instead of the viewpoint position / gaze direction acquired from the input device 206.

  In the first and second embodiments described above, the case where the present invention is applied to television broadcasting has been described. However, the present invention can be similarly applied to recording on package media.

It is a block diagram of the system by the side of the broadcasting station in the 1st Example of this invention. It is a figure explaining an example of the description method of the restrictions / permission information of the position of the virtual camera of this invention. It is a schematic diagram of the camera for demonstrating this invention. It is a figure explaining the other example of the description method of the limitation / permission information of the gaze direction of a virtual camera. It is a block diagram of the system of the receiving side in 1st Example of this invention. It is a flowchart for demonstrating the example of the process sequence of the virtual camera position and direction determination part in 1st Example of this invention. It is a block diagram of the system by the side of the broadcasting station in the 2nd Example of this invention. It is a block diagram of the system of the receiving side in 2nd Example of this invention. It is a figure for demonstrating the example of an arbitrary viewpoint video generation system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Stage, 2 ... Camera, 3 ... Virtual camera, 101 ... Camera, 102 ... Camera parameter recording part, 103 ... Virtual camera position / direction restriction / permission information recording part, 104 ... Data multiplexing processing part, 105 ... Transmission 106: Standard viewpoint position / gaze direction information recording unit 107 ... Video delay unit 111 ... Cuboid, 112 ... Sphere, 121 ... Camera body, 122 ... Lens, 123 ... Camera gaze direction, 201 ... Receiver unit, 202 ... multiplexed data separation processing unit, 203 ... arbitrary viewpoint video generation unit, 204 ... virtual camera position / direction determination unit, 205 ... display unit, 206 ... input device, 207 ... switch.

Claims (21)

  Restriction on virtual viewpoint position when synthesizing video in a synthesized video generation system that synthesizes videos from arbitrary virtual viewpoint positions and gaze directions by synthesizing videos from multiple different viewpoint positions and gaze directions A composite video generation system characterized in that information is added to video data and transmitted and recorded.   2. The composite video generation system according to claim 1, wherein permission information for a virtual viewpoint position when the video is synthesized is added to the video data and transmitted and recorded.   3. The composite video generation system according to claim 1, wherein restriction information on a virtual line-of-sight direction when the video is synthesized is added to the video data and transmitted and recorded.   4. The composite video generation system according to claim 1, wherein permission information for a virtual line-of-sight direction when the video is synthesized is added to the video data and transmitted and recorded.   One or more of the restriction information for the virtual viewpoint position, the permission information for the virtual viewpoint position, the restriction information for the virtual viewing direction, and the permission information for the virtual viewing direction represent a plane. 5. The composite video generation system according to claim 1, wherein the composite video generation system is specified by a mathematical expression.   One or more of the restriction information for the virtual viewpoint position, the permission information for the virtual viewpoint position, the restriction information for the virtual line-of-sight direction, and the permission information for the virtual line-of-sight direction represent a spherical surface. 5. The composite video generation system according to claim 1, wherein the composite video generation system is specified by a mathematical expression. One or more of the restriction information for the virtual viewpoint position, the permission information for the virtual viewpoint position, the restriction information for the virtual line-of-sight direction, and the permission information for the virtual line-of-sight direction are:
A logical product or logical sum of one or more pieces of restriction information specified by a mathematical expression representing the plane or the spherical surface, or a logical product or logical sum of one or more pieces of permission information designated by a mathematical formula representing the plane or the spherical surface. The composite video generation system according to claim 5 or 6, characterized in that:   One or more frame rates of the restriction information for the virtual viewpoint position, the permission information for the virtual viewpoint position, the restriction information for the virtual line-of-sight direction, and the permission information for the virtual line-of-sight direction are: 8. The composite video generation system according to claim 1, wherein the composite video generation system has the same frame rate as the video data.   One or more frame rates of the restriction information for the virtual viewpoint position, the permission information for the virtual viewpoint position, the restriction information for the virtual line-of-sight direction, and the permission information for the virtual line-of-sight direction are: The composite video generation system according to claim 1, wherein the composite video generation system is different from a frame rate of the video data.   2. The virtual viewpoint position and line-of-sight direction when synthesizing video are transmitted and recorded by adding predetermined standard viewpoint position and line-of-sight information to the video data. 10. The composite video generation system according to any one of 9 to 9.   11. The composite video generation system according to claim 10, wherein a frame rate of the standard viewpoint position / line-of-sight direction information is the same as a frame rate of the video data.   The composite video generation system according to claim 10, wherein a frame rate of the standard viewpoint position / line-of-sight direction information is different from a frame rate of the video data.   13. The composite video generation system according to claim 1, further comprising an input device on a data reception side or a reproduction side, and receiving an input of a virtual viewpoint position and line-of-sight direction by the input device. .   14. The composite video generation system according to claim 13, wherein video is synthesized according to the virtual viewpoint position and line-of-sight direction input by the input device.   The virtual viewpoint position / line-of-sight direction input by the input device is determined and processed according to the restriction information and / or permission information added to the video data, and matches the restriction information, or 15. The composite video generation system according to claim 14, wherein control is performed so as not to generate an image at a viewpoint position and a line-of-sight direction that does not match the permission information.   16. The composite video generation system according to claim 13, wherein the input device has a force feedback mechanism.   When the virtual viewpoint position / gaze direction input by the input device matches the restriction information added to the video data, or does not match the permission information added to the video data 17. The composite video generation system according to claim 16, wherein a control signal is sent to the force feedback mechanism so as to give a physical stimulus to the user.   A change-over switch, wherein the change-over switch determines whether to synthesize an image according to the virtual viewpoint position / line-of-sight direction input by the input device or to synthesize an image according to the information of the standard viewpoint position / line-of-sight direction. 18. The composite video generation system according to claim 13, wherein the composite video generation system can be switched.   When the input of the virtual viewpoint position / gaze direction by the input device has not been performed for a certain period of time, it is automatically switched to synthesize a video according to the information of the standard viewpoint position / gaze direction. The composite video generation system according to any one of claims 13 to 18.   When the video is synthesized according to the information on the standard viewpoint position / gaze direction, when the virtual viewpoint position / gaze direction is input by the input device, the input virtual 20. The composite video generation system according to claim 13, wherein the composite video generation system automatically switches to synthesize a video according to a viewpoint position and a line-of-sight direction.   The virtual viewpoint position / gaze direction input by the input device is determined and processed according to the restriction information and / or the permission information added to the video data, and the virtual viewpoint position / gaze direction is determined. When it matches the restriction information or when it does not match the permission information, the information on the standard viewpoint position / line-of-sight direction is used without using the virtual viewpoint position / line-of-sight direction input by the input device. 21. The composite video generation system according to claim 13, wherein an image is generated by using.

Images