JP2019145017A - System and method for reproducing replay video of free viewpoint video - Google Patents

Abstract

To reproduce a replay video of a free viewpoint video at a low load, and reproduce a replay video of a free viewpoint video even on a viewing terminal in which processing capacity is low.SOLUTION: A replay video reproduction system of a free viewpoint video comprises: a free viewpoint video generation part 201 for generating a free viewpoint video, based on a plurality of camera videos and viewpoint information of a virtual viewpoint; a format recording part 202 for recording a replay format in generation process of the free viewpoint video; a rendering information acquisition part 303 for acquiring information required for replay, based on the replay format; and a replay video reproduction part 304 for reproducing a replay video, based on the replay format and the acquired rendering information. The free viewpoint video generation part 201 and format recording part 202 are implemented on a server on a cloud, and the rendering information acquisition part 303 and the replay video reproduction part 304 are implemented on a viewing terminal 3.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system and method for reproducing a replay video of a free viewpoint video, and more particularly, to replay a free viewpoint video even on a viewing terminal having inferior processing capability by reducing the load related to the construction of the replay video of a free viewpoint video. The present invention relates to a system and a method for enabling video playback.

  Free viewpoint video technologies such as Patent Literature 1 and Non-Patent Literature 1 are proposed as technologies that enable video viewing from a virtual viewpoint where cameras are not actually placed based on videos taken from a plurality of cameras. It has been. By arranging multiple cameras at sports stadiums and generating free viewpoint images based on the images from these cameras, users can enjoy viewing videos from any virtual viewpoint they want to see. Is possible.

  When recording a video from a virtual viewpoint selected by the user using such a free viewpoint video technology, the displayed screen is captured, for example, H.264 described in Non-Patent Document 2. It is conceivable to save the video file in an existing video format such as (MPEG-4 AVC). The moving image file recorded in this manner is stored in a storage or the like, and can be played back when the user wants to view it again, or can be shared with another user by sending the stored file to another user.

  Moreover, as an example of recording a video from a specific viewpoint selected by the user without directly recording a moving image and reproducing the video later, there is a game device and a game replay method as shown in Patent Document 2. is there. According to the present invention, a history relating to a game operation of a player is stored as replay data, and a play image of the player is reproduced based on the replay data later.

Japanese Patent Application No. 2017-167472 JP 2010-264173 A

T. Koyama, I. Kitahara and Y. ohta, "Live Mixed-Reality 3D Video in Soccer Stadium" In Proc of IEEE / ACM Conference on ISMAR, pp. 178-187 (2003) ITU-T Recommendation H.264 "Advanced video coding for generic audiovisual services," May 2003

  Using free-viewpoint video technology, users can freely select a viewpoint based on touch operations on the screens of dedicated controllers, smartphones, and tablets, and enjoy viewing videos from any viewpoint. Unlike television images such as terrestrial broadcasts that are normally broadcast, each user can freely select and move a viewpoint in a free viewpoint video. The video you are viewing varies depending on the viewpoint position and how you move the viewpoint. Therefore, it is possible to generate original video content by recording video from the viewpoint that the user is viewing.

  When thinking about the future of free-viewpoint video technology, it is natural that videos from such a specific point of view will be exchanged via the Internet, and new ways of enjoying will be born, such as evaluation and comments via SNS. Conceivable.

  In this way, you want to play back a video from a specific viewpoint (hereinafter referred to as a replay video) that was recorded on a specific terminal, and then play it back on another terminal, or share it with many users. In response to demand, it is possible to save in the existing video encoding format as described in Non-Patent Document 2 and exchange the created video, but there is a problem that the capacity of the video will increase Was.

  In particular, free viewpoint video allows users to generate videos from different viewpoints, and it is possible to create a huge variety of video content from a single free viewpoint video. It is thought to appear.

  Also, if the movement of the viewpoint is known, an image from a specific viewpoint can be obtained by passing the viewpoint information to the free viewpoint video generation device. For example, like the game replay function shown in Patent Document 2, it is possible to recalculate and display the video from a given viewpoint based on the viewpoint information recorded in advance. .

  However, unlike games, it is necessary to first reconstruct a three-dimensional space from images from a plurality of cameras in a free viewpoint image, and the calculation cost for reconstructing this 3D space is very high. In particular, when considering a case where a large number of users simultaneously request the server to reconstruct a replay video, it is difficult to reconstruct the replay video without delay.

  An object of the present invention is to solve the above technical problem, to reduce the load related to the construction of a replay video of a free viewpoint video, and to make it possible to play back a replay video of a free viewpoint video even with a viewing terminal having inferior processing capability It is to provide a method.

  In order to achieve the above object, the present invention is characterized in that a system for reproducing a replay video of a free viewpoint video has the following configuration.

  (1) means for generating a free viewpoint video based on viewpoint information of a plurality of camera videos and virtual viewpoints, means for recording a replay format in a free viewpoint video generation process, and replaying based on the replay format Means for obtaining necessary information, and means for reproducing a replay video based on the replay format and the obtained information.

  (2) The information necessary for the replay includes a background 3D model and spatial information for rendering an object on the background 3D model.

  (3) The spatial information includes position information for arranging a mask image of each object and a model of each object.

  (4) A means for generating a free viewpoint video and a means for recording a replay format are implemented in a server on the cloud, and a means for acquiring information necessary for replay and a means for playing back the replay video are provided in the terminal for viewing the replay video. Implemented, the replay format is transferred from the server to the viewing terminal and stored on the viewing terminal.

  According to the present invention, the following effects are achieved.

  (1) In a free viewpoint video generation process, information such as identifiers of information that can be used for replay video reconstruction and parameters necessary for replay video reconstruction are recorded in the replay format. Therefore, when replaying a replay video, information that can be used for replay video reconstruction is acquired based on the identification information recorded in the replay format, and the information recorded in the replay format is used as a parameter. Thus, the replay video can be played with a light load.

  (2) As the information necessary for replay video reconstruction, the background 3D model and spatial information for rendering objects on the background 3D model are acquired, reducing the processing load when replay video is reconstructed. The

  (3) Since the spatial information includes the position information for placing the mask image of each object and the model of each object, when reconstructing the replay video, calculations for obtaining such information with high processing load are unnecessary. Become.

  (4) A means for generating a free viewpoint video and a means for recording a replay format are implemented in a server on the cloud, and a means for acquiring information necessary for replay and a means for reproducing the replay video are provided as a viewing terminal for the replay video. If implemented, it is generally possible to place a high processing load on a server with high processing capability. Therefore, it is possible to replay a free viewpoint video even on a viewing terminal having generally low processing capability.

It is the block diagram which showed the structure of the principal part of the free viewpoint image | video delivery system which concerns on one Embodiment of this invention. FIG. 6 is a diagram for explaining a method of switching an object on which a polygon is to be placed according to a virtual viewpoint P. It is the figure which showed the 1st example of the format for replays. It is a figure for demonstrating the definition of viewpoint information. It is the sequence flow which showed the procedure from construction of the format for replay to reproduction | regeneration of a replay image | video. It is the figure which showed the 2nd example of the format for replays. It is the figure which showed the 3rd example of the format for replays. It is the figure which showed the production | generation method of the free viewpoint image | video in 2nd Embodiment of this invention. It is the figure which showed the 3rd example of the format for replays.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the main part of a free viewpoint video distribution system according to an embodiment of the present invention, and here, the configuration unnecessary for the description of the present invention is omitted.

  The free viewpoint video distribution system of the present invention is a content server 1 that stores a plurality of camera cams that are arranged in a stadium or the like and that captures an object such as an athlete from different viewpoints, videos captured by each camera cam, and camera parameters. Free viewpoint video generation device 2 that generates a free viewpoint video based on a plurality of camera videos, camera parameters, and viewpoint information, and free viewpoint video information (viewpoint information) of virtual viewpoint P according to the operation of terminal user U The viewing terminal 3 that is provided to the device 2 and acquires and reproduces the free viewpoint video generated by the free viewpoint video generation device 2 is a main configuration.

  The content server 1, the free viewpoint video generation device 2, and the viewing terminal 3 may be configured by mounting an application (program) for realizing each function described later on a general-purpose computer, or a part of the application may be hardware. It may be configured as a dedicated machine or a single-function machine in the form of a wear or ROM.

  In the content server 1, the captured camera video and the content including the camera parameters are managed with a unique ID. In the illustrated example, ID1 is assigned to the free viewpoint content of soccer, ID2 is assigned to the free viewpoint content of valley, and ID3 is assigned to the free viewpoint content of judo.

  The free viewpoint video generation device 2 includes a free viewpoint video generation unit 201 and a format recording unit 202. By implementing these functions on a server placed on the cloud, the free viewpoint video generation unit 2 can be configured as a free viewpoint video generation server. it can.

  The free viewpoint video generation unit 201 generates a free viewpoint video by applying a plurality of camera videos with different viewpoints, camera parameters, and viewpoint information selected by the terminal user U in the viewing terminal 3 to a known free viewpoint technology. To do.

  In the first embodiment of the present invention, similar to Non-Patent Document 1, an object in a three-dimensional space is approximated by one rectangular polygon, and acquired from a plurality of camera images according to a virtual viewpoint P selected by the user. An example of application to free viewpoint technology that employs a billboard method for appropriately mapping texture information to rectangular polygons will be described.

  The free viewpoint video generation unit 201 extracts objects from a plurality of camera videos and estimates their positions. In the case of a free viewpoint video of a soccer game, a person such as a player appearing on the stadium becomes an object. It is assumed that a 3D model is manually created in advance as a preset for backgrounds such as stadium pitches and spectator seats other than objects.

  As shown in FIG. 2, the free viewpoint video generation unit 201 is further configured to face the visual line direction of the virtual viewpoint P at a position where each object is estimated to exist according to the selected virtual viewpoint P. Two rectangular polygons are installed, and the texture of the object extracted from the camera image of the real camera cam having the closest angle in the line-of-sight direction to the virtual viewpoint P is displayed on the rectangular polygons.

  In the example of FIG. 2, since the line-of-sight angle of the virtual viewpoint P is closest to the line-of-sight angle of the real camera cam3, the texture of the target object is cut out from the camera image of the real camera cam3 and pasted on the rectangular polygon. When displaying this texture, by using a binary mask image representing the shape of the target object, only the portion of the rectangular polygon where the target object exists is displayed, and the other portions are transparent. . When the virtual viewpoint P moves, the rectangular polygon is also rotated in accordance with this movement, and viewing from the virtual viewpoint P can be realized without a sense of incongruity by always facing the virtual viewpoint P.

  The format recording unit 202 provides the free viewpoint video generation unit 201 with various parameters for enabling the viewing terminal 3 to replay the free viewpoint video generated by the free viewpoint video generation unit 201 later with a small calculation load. Acquired in the process of generating the viewpoint video and recorded in the replay format.

  The recording of the replay format is started in response to the recording start request RQ2 by the viewing terminal 3 that is reproducing the free viewpoint video, and is ended in response to the recording end request RQ3. The completed replay format is transferred to the viewing terminal 3 that transmitted the requests RQ2 and RQ3 and managed on the storage 301.

  In the format recording unit 202, the free viewpoint information recording unit 202a records viewpoint information including the position, orientation, and orientation of the virtual viewpoint P selected by the terminal user U operating the viewing terminal 3 in the replay format. .

  The spatial information recording unit 202b records spatial information that can reduce the processing load when the viewing terminal 3 renders an image seen from the selected virtual viewpoint P. In the present embodiment, it is considered that an object visible from the viewpoint is specified based on the viewpoint information, and only the visible object is rendered and reconstructed into a three-dimensional space.

  Also, a parameter that places a high processing load on the rendering process, specifically the position of the billboard that displays each object, and a mask that allows a part of the billboard to be transmitted in accordance with the object shape when pasting the texture on each billboard. For the image, only the identifier for the viewing terminal 3 to acquire these parameters from the free viewpoint video generation device 2 is recorded.

  In the case of FIG. 2, four objects (ID1 to ID4) are standing on the background model, but only two objects (ID3 and ID4) are visible from the selected virtual viewpoint P. In addition, since the angle of the line of sight of the virtual viewpoint P is close to the angle of the real camera cam3, it is assumed that the texture acquired from the camera image of the real camera cam3 is pasted on the billboard installed at the positions of the object ID3 and ID4 To do.

  In such a case, in the present embodiment, only two objects are reconstructed into a three-dimensional space, and only the billboard position, texture, and mask image are acquired. Since the object, its billboard position, the texture attached to the billboard, the actual camera, and the mask image for masking the texture can be linked to each other, all of these can be recorded in the replay format. It becomes redundant.

  Therefore, in this embodiment, as shown in FIG. 3 described in detail later, only “camera number”, “texture number”, and “billboard position” are recorded as the spatial information, and the identifier of the mask image is “texture number”. Is summarized. Note that the position of each billboard is not necessarily recorded as long as the free viewpoint video generation device 2 records the billboard in association with the texture number.

  The viewing terminal 3 has a function of selecting a viewpoint by connecting a dedicated controller to a TV capable of reproducing video, a function of selecting a viewpoint by a touch operation or a swipe operation on a touch screen provided on a display of a smartphone or a tablet, or This can be realized with the function of viewing a free viewpoint video on a VR terminal equipped with an acceleration sensor and selecting the viewpoint according to the user's movement.

  In the viewing terminal 3, the storage 301 stores the replay format and various spatial information transmitted from the free viewpoint video generation device 2 via the network. The user operation detection unit 302 detects a virtual viewpoint operation RQ1, a replay format recording start request RQ2, a recording end request RQ3, and a replay request RQ4 by the terminal user U, and a free viewpoint video generation device via the network 2 to send.

  In response to the replay operation RQ4, the rendering information acquisition unit 303 refers to the replay format stored in the storage 301 and identifies which free viewpoint video the format is for. Then, the background 3D model of the specified free viewpoint video is acquired from the free viewpoint video generation device 2, and further, based on the background 3D model, the texture, mask image, billboard position, etc. necessary for reconstruction of each frame are acquired. Spatial information is acquired from the free viewpoint video generation device 2. Each acquired information is temporarily stored in the storage 301.

  The replay video reproduction unit 304 performs rendering based on the replay format and the spatial information acquired from the free viewpoint video generation device 2 and reproduces the replay video of the free viewpoint video.

  In the present embodiment, rendering of the free viewpoint video is performed by the viewing terminal 3, but the viewing terminal 3 needs to render a video from the virtual viewpoint P obtained from the viewpoint information. That is, at the standing position of each billboard in the acquired background 3D model, the billboard is set up in a form facing the virtual viewpoint P, and the acquired texture is masked and pasted with the mask image to create the 3D space. Reconfigure.

  On the other hand, since the costly process of estimating the standing position of the billboard from a plurality of camera images and generating a mask image is not performed, the above rendering is performed on the viewing terminal side in view of the specifications of recent smartphones and the like. It is well possible to do.

  According to this embodiment, as compared with the case where rendering is performed in the free viewpoint video generation device 2, not only the background 3D model needs to be transmitted once, but also only the spatial information of the visible object in other frames. It may be transmitted. For example, when considering soccer video, the area where players are present is often very small relative to the whole picture, and only the texture and mask images are transmitted rather than sending the rendered video every time. However, there are many cases where the amount of data is light.

  In addition, if each texture is once transmitted and then stored in the viewing terminal, the texture remains in the viewing terminal 3 that has been replayed once. There is no need to download, and replay video can be reconstructed without a network.

  FIG. 3 is a diagram showing a first example of the replay format, which is composed of header information and time-series information.

  In the header information, the “free viewpoint video ID” is used to uniquely identify each replay format. Since this free viewpoint video ID only needs to be recorded once, it is information written in the format header. The “total number of frames” is the number of frames of the replay video reconstructed based on the replay format, and corresponds to the playback time.

  The time-series information is generated for each frame, and information specifying the position (time) of each frame in the replay video is recorded in the “playback time identifier”. In the illustrated example, only the viewpoint time-series information (21, 22) for two frames is shown. However, if the “total number of frames” is 200, the time-series information for 200 frames is connected. Become.

  For example, when a recording start request RQ2 is detected at a timing of 10 seconds from the start and a recording end request RQ3 is detected at a timing of 20 seconds for a free viewpoint video of 1 minute at 30 frames per second, the time series information is: The “reproduction time identifier” is configured by connecting information from 300 to 600 in time series. The “reproduction time identifier” is not limited to the frame number, and may be absolute time information or relative time information. As described above, in this embodiment, information for replaying each frame is managed in time series. Therefore, if the audio recorded in the content server is managed by the time information, the video and audio can be displayed on a time basis. Synchronized playback can be easily performed.

  In the present embodiment, as information for specifying the virtual viewpoint P, as shown in FIG. 4, “viewpoint position E (ex, ey, ez)” representing the three-dimensional position coordinates of the viewpoint, and the viewpoint direction (line of sight) "Gaze direction D (dx, dy, dz)" representing the position and "Attitude direction U (ux, uy, uz)" representing the posture information of the viewpoint are adopted, and the viewpoint information is a total of nine parameters of three three-dimensional vectors. Specified by

  The “posture direction” is information indicating which of the screens used for display is on when viewing a certain direction from a certain viewpoint position. Even if the viewpoint position and line-of-sight direction are the same, the image is upside down between the image viewed in an upright state and the image viewed in an upside-down state. Replay video can be reconstructed.

  The “camera number” is an identifier of the real camera cam whose direction is closest to the virtual viewpoint P. “Texture number” is the number of the texture of the object visible at the current virtual viewpoint P. “Billboard position” represents the relationship between the coordinate position of the billboard that models the object viewed at the current virtual viewpoint and the identifier of the texture that is attached to the billboard. In this embodiment, such time-series information is constructed in a predetermined cycle, for example, in units of frames, managed by the “reproduction time identifier”, and sequentially connected.

  FIG. 5 is a sequence flow showing a procedure from construction of the replay format to replay video playback based on the replay format.

  At time t1, a video viewing request RQ1 is transmitted from the viewing terminal 3 to the free viewpoint video generation device 2. In response to the viewing request RQ1, the free viewpoint video generation device 2 starts distributing video content at time t2. At time t3, the video is played on the viewing terminal 3 that acquired the video content.

  At time t4, the terminal user U performs a viewpoint operation for viewing a free viewpoint video on the viewing terminal 3, and when this is detected by the user operation detection unit 302, at time t5, the terminal user U has selected. The viewpoint information for specifying the virtual viewpoint P is transferred from the viewing terminal 3 to the free viewpoint video generation apparatus 2. In the free viewpoint video generation device 2, at time t6, the free viewpoint video generation unit 201 performs rendering based on the viewpoint information and each camera video to generate a free viewpoint video. At time t7, the free viewpoint video is distributed to the viewing terminal 3 and reproduced at time t8.

  At time t9, the terminal user U operates the viewing terminal 3 to request the start of replay video recording, and when this is detected by the user operation detection unit 302, the recording start request RQ2 is received from the viewing terminal 3 at time t10. It is transmitted to the free viewpoint video generation device 2. In the free viewpoint video generation device 2, at time t11, the format recording unit 202 starts recording the replay format for the free viewpoint video being reproduced in response to the recording start request RQ2. The recording of the replay format is repeated in units of frames until a recording end request RQ3 from the viewing terminal 3 is detected.

  In general, when the free viewpoint video generation unit 201 reproduces a free viewpoint video, information on the virtual viewpoint P selected by the user can be acquired from the free viewpoint video generation unit 201. Also in this embodiment, when the replay video recording start RQ2 is detected, the format recording unit 202 acquires viewpoint information from the free viewpoint video generation unit 201 in units of frames and records the parameters in the replay format.

  In this embodiment, “viewpoint position E (ex, ey, ez)”, “line-of-sight direction D (dx, dy, dz)”, and “posture direction U (ux, uy, uz)” are recorded in units of frames. . Further, the position information of the position where the billboard of the object seen from the current viewpoint is raised is recorded. Further, the texture number and camera number of the corresponding object to be attached to each billboard are recorded.

  Thereafter, at time t12, the terminal user U operates the viewing terminal 3 to request the end of recording of the replay video, and when this is detected by the user operation detection unit 302, at time t13, the recording end request RQ3 is transmitted to the viewing terminal. 3 to the free viewpoint video generation device 2. In the free viewpoint video generation device 2, the format recording unit 202 ends the recording of the replay format at time t14. At time t15, the generated replay format is transferred to the viewing terminal 3, and is stored in the storage 301 of the viewing terminal 3 at time t16.

  After that, at time t17, a user who desires to replay the free viewpoint video designates a replay format on the storage and requests replay. When this is detected by the user operation detection unit 302, the rendering information The acquisition unit 303 interprets the replay format and determines which free viewpoint video the replay video is based on the free viewpoint video ID described in the format.

  At time t18, the viewing terminal 3 requests (RQ4) the information required for video replay to the free viewpoint video generation device 2 based on the replay format. In the present embodiment, a background 3D model associated with the free viewpoint video ID in the replay format is requested, and at time t19, the free viewpoint video generation device 2 delivers the background 3D model in response to the request.

  At time t20, the replay video playback unit 304 performs rendering based on the replay format and the acquired background 3D model, and playback of the replay video of the free viewpoint video is started. During playback of the replay video, the rendering information acquisition unit 303 requests and acquires spatial information such as the position of the billboard, the mask image, and the texture from the free viewpoint video generation device 2 based on the replay format in units of frames.

  Then, the replay video playback unit 304 reconstructs the 3D space with high efficiency based on the free viewpoint space information described in the format, and is recorded in the format for the reconstructed space. The replay video is reconstructed by rendering the image viewed from the viewpoint position based on the acquired spatial information.

FIG. 6 is a diagram showing another example of the replay format. In the above embodiment, the viewpoint information includes the viewpoint position E (e _x , e _y , e _z ), the line-of-sight direction D (d _x , d _y , d _z ), and the posture direction U (u _x , u _y , u _z ). Each of the three-dimensional vectors is described as being expressed by nine parameters. However, when the parameter does not change, it may have a function to reduce the data size by eliminating the redundancy, and this is realized by not describing the subsequent parameter when the parameter does not change between frames. it can.

  For example, when the viewpoint moves in parallel, the line-of-sight direction D and the posture direction U do not change, and only the viewpoint position E may change. In the present embodiment, when such a viewpoint movement is detected, as shown in FIG. 6, the recording of the line-of-sight direction D and the posture direction U is omitted for the time-series information of the next frame, thereby reducing the data size. And the processing load can be reduced. The parameter to be recorded may be rounded by a fixed number of digits and recorded as an approximate value in order to reduce the data size.

FIG. 7 is a diagram showing still another example of the replay format. The present embodiment is characterized in that the gazing point position F (f _x , f _y , f _z ) is stored instead of the line-of-sight direction D (d _x , d _y , d _z ). The gaze point position F indicates the position of a specific point on the line-of-sight direction D. The line-of-sight direction D is obtained from the viewpoint position E and the gazing point position F by the following equation (1).

In this way, redundancy may be eliminated by adopting the gaze point position F (f _x , f _y , f _z ) instead of the viewpoint direction D (d _x , d _y , d _z ). For example, if a movement that rotates around the gazing point F is assigned to a swipe operation or the like of the viewing terminal 3, it can be considered that many movements that rotate around the gazing point F appear. In such a case, according to the present embodiment, the gazing point position F (f _x , f _y , f _z ) does not change, so that redundancy can be eliminated.

  As still another example of the viewpoint information, a rotational movement amount (rotation angle) and a parallel movement amount may be employed as the viewpoint information parameters.

To obtain a certain viewpoint, the viewpoint is rotated by the rotation amount θ _x around the x axis, the rotation amount θ _y around the y axis, and the rotation amount θ _z around the z axis around the origin of the world coordinate system. Further, by performing parallel movement T (t _x , t _y , t _z ) to the viewpoint position, the position of the viewpoint, the line-of-sight direction and the posture can be specified. Therefore, the viewpoint can be reconstructed from the six parameters of the rotation amounts θx, θy, θz and the parallel movement amount T (tx, ty, tz).

  In this example, the viewpoint can be reconstructed from a small number of parameters, but the default position, direction, and orientation of the viewpoint need to be clearly determined before rotation or translation. This means that if you do not perform any rotation or translation, the viewpoint position is at the origin of the world coordinate system, facing the positive direction of the z axis, and the posture is facing the positive direction of the y axis. This means that an initial value such as “Yes” needs to be determined, and the replay video reproduction unit 304 of the viewing terminal 3 also needs to recognize the initial viewpoint information. This information may be written and exchanged in the format itself, but the initial position when the free viewpoint video generation unit 201 reproduces the free viewpoint video may be determined as it is as the initial position.

  As still another example of the viewpoint information, a form in which a view transformation matrix is recorded may be employed. The view transformation matrix indicates a transformation matrix that performs transformation from the world coordinate system to the viewpoint coordinate system (camera coordinate system). By using this transformation matrix, the position and direction of the viewpoint and the posture information are specified. It is possible. Here, if the view coordinate matrix is expressed in a homogeneous coordinate system, a 4 × 4 transformation matrix M is expressed by the following equation (2).

Such matrices are frequently used in popular 3D display libraries such as OpenGL and DirectX, and include the viewpoint position E (e _x , e _y , e _z ) and the line-of-sight direction D (d _x , d _y , D _z ), the orientation direction U (u _x , u _y , u _z ), and the like are often calculated. Therefore, if the view transformation matrix is stored in advance, the processing cost can be reduced because the transformation matrix can be most easily acquired when considering use in a library or the like.

  Furthermore, in the example of each format described above, the viewpoint information is recorded as it is in principle. However, as shown in FIG. 7, only the difference value from the previous frame may be recorded.

  Such a format has a feature that many small values are written because the difference value between frames tends to be small. When the value is small, it is likely that the same value sequence such as “0” is likely to occur. By performing entropy coding represented by Huffman coding on such a code string in which the same value sequence is likely to occur, there is a possibility that the data size can be further reduced.

  However, when it is desired to reproduce from the middle, the calculation cost tends to increase because the value of the middle frame must be calculated by adding the differences from the first frame. Therefore, it is also possible to set a format in which one frame is described as a parameter that is not a normal difference and a difference value from the previous frame is recorded in other frames.

  In this case, it is necessary to set the format so that it can be understood which frame holds all the information and which frame holds the difference information. In the example of FIG. 7, each frame is distinguished by attaching an identifier “D” to the difference frame and an identifier “I” other than the difference frame.

  The format recording unit 202 records viewpoint information by describing various parameters over each frame by the above-described methods. The free viewpoint information recording unit 202a must have a function of converting and shaping information about the viewpoint received from the free viewpoint video generation unit 201 so that the information is described in the format.

  For example, the free viewpoint video generation unit 201 uses the transformation matrix (view transformation matrix) of the viewpoint from the world coordinate system to the camera coordinate system to obtain the user's viewpoint. If the video is generated, this transformation matrix is acquired, and the calculation process until obtaining information such as the three-dimensional vector of the position coordinates of the viewpoint from the transformation matrix, or the numerical value recorded with a fixed number of digits is truncated. , And processing to convert to a format suitable for the format, such as rounding processing.

  FIG. 8 is a diagram for explaining the free viewpoint technology adopted by the second embodiment of the present invention, and FIG. 8 is a diagram showing an example of a replay format in the second embodiment.

  In the first embodiment, the free viewpoint video generation unit 201 has been described as generating a free viewpoint video using the billboard method. On the other hand, this embodiment expresses a method for accurately restoring the shape of a 3D model of an object (here, “full model method using a backprojection plane”) as disclosed in Patent Document 1. ) To generate a free viewpoint video.

  When the free viewpoint video generation unit 201 adopts the full model method, a large number of backprojection planes P1, P2,... Are arranged in a form facing the virtual viewpoint P in order to restore the 3D shape of the object. Next, a 3D model is created for each backprojection plane by projecting the mask image of the target object obtained by the background subtraction method to each backprojection plane P1, P2,... Further, the back projection plane is colored by mapping the texture image of the target object. Therefore, it is possible to restore the 3D model by cutting out the backprojection surface appropriately.

  In such a method, each backprojection plane P1, P2,... Is always arranged in a shape orthogonal to the virtual line of sight P. Therefore, the position of each backprojection plane P1, P2... Changes depending on the position of the virtual viewpoint P. To do. The format recording unit 202 starts recording the format in response to the replay video recording start request RQ2 from the viewing terminal 3. At this time, as in the first embodiment, the free viewpoint video ID and the total number of frames are recorded in the header, and the viewpoint information is also recorded for each frame in the same manner as in the first embodiment.

  The spatial information recording unit 202b records, as spatial information, only the index of the surface on which the model is generated among the many backprojection surfaces P1, P2,. That is, in this embodiment, the index of the surface on which no model is generated is not recorded. For example, in the example shown in FIG. 8, a cylindrical object exists in the space, but the model is generated only for P2, P3, and P4. Therefore, as shown in FIG. 9, only “2 3 4” is recorded as the index.

  For example, in a free viewpoint video in which players are discretely present in a part of a wide field, such as soccer, if a backprojection plane is placed over the entire field, a lot of useless backprojection planes in which a model is not generated are generated. It is useless to perform a correct calculation again during replay video playback.

  On the other hand, in this embodiment, the backprojection plane where the model is generated and the backprojection plane where the model is not generated can be identified in advance, so that efficient memory can be secured, and the backprojection plane where the model is not generated is masked. Since the calculation for back projecting the image is not necessary, the calculation load is reduced.

  In particular, the full model method of Patent Document 1 employed by this embodiment is also described in Patent Document 1 that performs parallel computation using a GPU, and reducing the number of backprojection planes leads to memory saving. . As a result, it is possible to reduce the time required for memory access and the like, so that it is possible to save calculation resources and increase the calculation speed.

  The spatial information recording unit 202b records the projection plane index that needs to be calculated when reproducing the 3D space, thereby speeding up the calculation and saving the calculation resources. As for the index attached to the backprojection plane, when there are 1000 backprojection planes directly facing the viewpoint, a method of assigning the index in order from 1 to 1000 in order from the side closest to the viewpoint is conceivable. In the example of the format shown in FIG. 9, “2 3 4” is recorded as an index representing the three back projection planes P2, P3, and P4 on which the model is generated. The replay format recorded in this way is transferred to the viewing terminal 3 and stored in the same manner as in the first embodiment, and is later referred to during replay.

  In the viewing terminal 3, the replay video playback unit 304 reconstructs the replay video based on the stored replay format. At this time, the viewpoint is determined based on the viewpoint information of the replay video format, and the backprojection plane is arranged based on this viewpoint, but the backprojection plane where no model is generated is identified by referring to the index, and the backprojection plane is identified. Arrangement and calculation are not performed for the projection plane.

  As a result, the 3D space can be reproduced by calculating only the backprojection plane on which the model is promised to be generated. After that, the video from the viewpoint is rendered on this 3D space, and the rendered image is transmitted to the viewing terminal 3 to realize replay video reproduction.

  In this embodiment, rendering is performed in the free-viewpoint video generation device 2, but for example, all videos for constituting a free-viewpoint video are distributed in advance to the viewing terminal 3 and rendered on the terminal side. Also good. Even in this case, if the index of the backprojection plane in which no model is generated is recorded in advance, it is possible to increase the calculation speed and save the calculation resources.

  In such a configuration, the free viewpoint video generation device 2 is provided with a replay video playback function, and a complete reconstruction of the 3D space is possible. For this reason, the replay video playback function receives information related to the specifications of the viewing terminal 3, and renders the rendered image so that the visual field and the aspect ratio of the image change depending on the resolution and screen size of the playback device, although the viewpoint is the same. You may provide the function to output.

  In addition, when a plurality of viewing terminals 3 request the free viewpoint video generation device 2 to replay the replay video at the same time, if there are requests for rendering frames at the same viewpoint position and the same time, rendering is performed. A mechanism for storing and reusing the result may be provided.

  DESCRIPTION OF SYMBOLS 1 ... Content server, 2 ... Free viewpoint video production | generation apparatus, 3 ... Viewing terminal, 201 ... Free viewpoint video production | generation part, 202 ... Format recording part, 202a ... Free viewpoint information recording part, 202b ... Spatial information recording part, 301 ... Storage , 302 ... User operation detection unit, 303 ... Rendering information acquisition unit, 304 ... Replay video reproduction unit

Claims (15)

In a system that plays replay video of free viewpoint video,
Means for generating a free viewpoint video based on viewpoint information of a plurality of camera videos and virtual viewpoints;
Means for recording a replay format in the free viewpoint video generation process;
Means for obtaining information necessary for replay based on the replay format;
A system for playing back a replay video of a free viewpoint video, comprising: means for playing back a replay video based on the replay format and acquired information.   The system for reproducing a replay video of a free viewpoint video according to claim 1, wherein the information necessary for the replay is a background 3D model and spatial information for rendering an object on the background 3D model. The replay format includes header information and time series information,
The time series information is configured by connecting a plurality of time series information recorded at a predetermined period,
3. The system for reproducing a replay video of a free viewpoint video according to claim 1, wherein a reproduction time ID unique to a reproduction position of the replay video based on the time series information is recorded in each time series information. The generation process includes a generation process of a background 3D model of a free viewpoint video;
In the replay format, the ID of the free viewpoint video is recorded,
The system for reproducing a replay video of a free viewpoint video according to any one of claims 1 to 3, wherein the acquiring unit acquires the background 3D model based on an ID of the free viewpoint video. The generation process includes a viewpoint information generation process;
The system for reproducing a replay video of a free viewpoint video according to any one of claims 1 to 4, wherein the viewpoint information is recorded in the replay format.   6. A system for reproducing a replay video of a free viewpoint video according to claim 1, wherein the means for generating the free viewpoint video employs a billboard free viewpoint technology. The generation process includes a process of identifying objects visible in a virtual perspective;
A general ID associated with the visible object is recorded in the replay format,
The system for reproducing a replay video of a free viewpoint video according to claim 6, wherein the acquiring unit acquires rendering information of each object based on the overall ID. The generating process includes generating a mask image of each object;
The mask image is linked to the general ID,
8. The system for reproducing a replay video of a free viewpoint video according to claim 7, wherein the acquiring unit acquires a mask image of each object based on the overall ID. The generating process includes extracting a texture of each object from a camera image;
The texture is linked to the general ID,
8. The system for reproducing a replay video of a free viewpoint video according to claim 7, wherein the acquiring unit acquires a texture of each object based on the overall ID.   7. The system for reproducing a replay video of a free viewpoint video according to claim 6, wherein the means for generating the free viewpoint video employs a full-model free viewpoint technology using a backprojection plane. The generation process includes a process of projecting a mask image of the object by arranging a plurality of back projection planes facing the virtual viewpoint at the position of the object, and performing 3D modeling for each back projection plane to restore the 3D model ,
In the replay format, record the index of the backprojection plane where the 3D model exists,
The system for reproducing a replay video of a free viewpoint video according to claim 10, wherein the acquiring unit acquires information necessary for replay based on an index of the backprojection plane. The means for generating the free viewpoint video and the means for recording the replay format are mounted on a server on the cloud,
Means for acquiring information necessary for the replay and means for reproducing the replay video are implemented in a replay video viewing terminal,
12. The system for reproducing a replay video of a free viewpoint video according to claim 1, wherein the replay format is transferred from the server to the viewing terminal and stored on the viewing terminal. A means for generating the free viewpoint video, a means for recording a replay format, and a means for acquiring information necessary for the replay are mounted on a server on the cloud,
Means for playing the replay video is implemented in the viewing terminal;
12. The system for reproducing a replay video of a free viewpoint video according to claim 1, wherein the replay format is transferred from the server to the viewing terminal and stored on the viewing terminal.   13. The means for recording the replay format starts recording the replay format in response to a recording start request from a viewing terminal, and ends recording in response to a recording end request. The system which reproduces the replay image of the free viewpoint image according to 13. In a method for a computer to play a replay video of a free viewpoint video,
Generate free viewpoint video based on the viewpoint information of multiple camera videos and virtual viewpoints,
Record the replay format in the free viewpoint video generation process,
Obtain information necessary for replay based on the replay format,
A method of reproducing a replay image of a free viewpoint image, wherein the replay image is reproduced based on the replay format and the acquired information.

Images