JP7165254B2 - System and method for reproducing replay video of free-viewpoint video - Google Patents

Description

The present invention relates to a system and method for reproducing a replay video of a free viewpoint video, and in particular, by reducing the load related to constructing a replay video of a free viewpoint video, replaying a free viewpoint video even on a viewing terminal with inferior processing capability. The present invention relates to systems and methods for enabling video playback.

Free-viewpoint video technology, such as Patent Document 1 and Non-Patent Document 1, has been proposed as a technology that enables video viewing from virtual viewpoints where no cameras are actually placed, based on video captured by multiple cameras. It has been. By arranging multiple cameras in a sports stadium or the like and generating free-viewpoint images based on images from these multiple cameras, users can enjoy viewing images from arbitrary virtual viewpoints that they want to see. It is possible.

When considering recording a video from a virtual viewpoint selected by the user using such a free-viewpoint video technology, the displayed screen is captured and, for example, H.264 described in Non-Patent Document 2 is used. It is conceivable to save the video as a video file in an existing video format such as (MPEG-4 AVC). The video files recorded in this way are saved in a storage or the like, and can be played back when desired to be viewed again, or can be shared with other users by sending the saved files to other users.

In addition, as an example of recording a video from a specific viewpoint selected by a user without directly recording a moving image and playing it back later, there is a game device and a game replay method as disclosed in Patent Document 2. be. The present invention saves a history of game operations performed by a player as replay data, and later reproduces a play image of the player based on the replay data.

Japanese Patent Application No. 2017-167472 JP 2010-264173 A

T. Koyama, I. Kitahara and Y. ohta, "Live Mixed-Reality 3D Video in Soccer Stadium"In Proc of IEEE/ACM Conference on ISMAR, pp. 178-187(2003) ITU-T Recommendation H.264 "Advanced video coding for generic audiovisual services," May 2003

By using free-viewpoint video technology, users can freely select a viewpoint based on touch operations on the dedicated controller, smartphone, or tablet screen, and enjoy watching videos from any viewpoint. Unlike terrestrial TV images that are normally broadcast, free-viewpoint images allow each user to freely select and move their viewpoint. The video being viewed differs depending on the position of the viewpoint and how the viewpoint is moved. Therefore, it is possible to create original video content by recording the video from such a user's viewpoint.

When we think about the future where free-viewpoint video technology has spread, it is natural that video from such a specific viewpoint will be exchanged via the Internet, and a new way of enjoying it will be born, such as receiving evaluations and comments via SNS. Conceivable.

In this way, the video recorded on a specific terminal from a specific viewpoint (hereinafter referred to as replay video) can be played again later, played on other terminals, or shared with many users. In response to demand, it is possible to save in an existing video encoding format as described in Non-Patent Document 2 and exchange the created moving image, but there is a problem that the size of the moving image increases. Was.

In particular, with free-viewpoint video, the user can generate video from each viewpoint, and it is possible to create a huge variety of video content from one free-viewpoint video, so this problem is conspicuous. presumed to appear.

Also, if the movement of the viewpoint is known, an image from a specific viewpoint can be obtained by passing the viewpoint information to the free viewpoint video generation device. For example, it is possible to recalculate and display an image from a given viewpoint based on pre-recorded viewpoint information, such as the game replay function disclosed in Patent Document 2. .

However, unlike games, with free-viewpoint video, it is first necessary to reconstruct a 3D space from images captured by multiple cameras, and the computational cost for reconstructing this 3D space is very high. In particular, if a large number of users simultaneously request the server to reconstruct the replay video, it is difficult to reconstruct the replay video without delay.

An object of the present invention is to solve the above technical problems, reduce the load associated with constructing a replay video of a free viewpoint video, and enable a viewing terminal with inferior processing capability to reproduce the replay video of the free viewpoint video. It is to provide a method.

In order to achieve the above objects, the present invention provides a system configured by connecting a viewing terminal and a free-viewpoint video generation device via a network, and reproducing a replay video of the free-viewpoint video, having the following configuration. is characterized by

(1) A viewing terminal comprises means for requesting playback of free-viewpoint video and means for requesting recording of replay video for the free-viewpoint video being played,
means for the free-viewpoint video generation device to generate a free-viewpoint video based on viewpoint information of a plurality of camera videos and virtual viewpoints in response to the playback request; comprising means for recording a replay format in which a virtual viewpoint is described for each playback time; and means for transferring the recorded replay format to a viewing terminal;
The viewing terminal further comprises means for acquiring information necessary for replay based on the replay format, and means for reproducing replay video based on the replay format and the acquired information. The information includes a background 3D model and spatial information for rendering an object on the background 3D model.

(2) The spatial information includes position information for arranging the mask image of each object and the model of each object.

(3) The means for generating free-viewpoint video and the means for recording the format for replay are implemented on a server on the cloud, and the means for obtaining information necessary for replay and the means for reproducing replay video are on the replay video viewing terminal. The replay format is transferred from the server to the viewing terminal and stored on the viewing terminal.

According to the present invention, the following effects are achieved.

(1) In the free-viewpoint video generation process, information such as identifiers of information that can be used for reconstruction of the replay video and parameters necessary for reconstruction of the replay video are recorded in the replay format. Therefore, when reconstructing a replay video, information that can be used for reconstructing the replay video is obtained based on the identification information recorded in the replay format, and the information recorded in the replay format is used as a parameter. With this, replay videos can be played back with a light load.

(2) The background 3D model and spatial information for rendering objects on the background 3D model are acquired as the information necessary for reconstructing the replay video, so the processing load when reconstructing the replay video is reduced. be.

(3) Spatial information includes the mask image of each object and positional information for arranging the model of each object, so when reconstructing the replay video, computation to obtain this information, which has a high processing load, is not required. Become.

(4) The means for generating free-viewpoint video and the means for recording replay formats are implemented on a server on the cloud, and the means for acquiring information necessary for replays and the means for reproducing replay videos are installed on replay video viewing terminals. Once implemented, it can offload computationally intensive computations to typically more powerful servers. Therefore, it is possible to replay the free-viewpoint video even in a viewing terminal that generally has a low processing capability.

1 is a block diagram showing the configuration of main parts of a free-viewpoint video distribution system according to one embodiment of the present invention; FIG. FIG. 10 is a diagram for explaining a method of switching an object for which a polygon is to be installed according to a virtual viewpoint P; FIG. 4 is a diagram showing a first example of a replay format; FIG. FIG. 4 is a diagram for explaining the definition of viewpoint information; FIG. 11 is a sequence flow showing a procedure from construction of a replay format to reproduction of replay video; FIG. FIG. 10 is a diagram showing a second example of a replay format; FIG. 11 is a diagram showing a third example of a replay format; FIG. 10 is a diagram showing a method of generating a free viewpoint video according to the second embodiment of the present invention; FIG. 11 is a diagram showing a third example of a replay format;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the main parts of a free-viewpoint video distribution system according to an embodiment of the present invention, and the illustration of configurations unnecessary for the explanation of the present invention is omitted here.

The free-viewpoint video distribution system of the present invention includes a plurality of camera cams arranged in a stadium or the like to capture objects such as athletes from different viewpoints, and a content server 1 that stores the video captured by each camera cam and camera parameters. , a free-viewpoint video generation device 2 that generates a free-viewpoint video based on a plurality of camera videos, camera parameters, and viewpoint information; The viewing terminal 3 is provided to the device 2 and acquires and reproduces the free-viewpoint video generated by the free-viewpoint video generation device 2 .

The content server 1, the free-viewpoint video generation device 2, and the viewing terminal 3 may be configured by installing applications (programs) for realizing each function described later in a general-purpose computer, or a part of the application may be implemented by hardware. It may be configured as a dedicated machine or a single-function machine in the form of hardware or ROM.

In the content server 1, content including captured camera images and their camera parameters is managed with unique IDs. In the illustrated example, ID1 is assigned to soccer free-viewpoint content, ID2 is assigned to volleyball free-viewpoint content, and ID3 is assigned to judo free-viewpoint content.

The free-viewpoint video generation device 2 includes a free-viewpoint video generation unit 201 and a format recording unit 202, and by implementing these functions in a server placed on the cloud, it can be configured as a server for free-viewpoint video generation. can.

The free-viewpoint video generation unit 201 generates a free-viewpoint video by applying a known free-viewpoint technique to a plurality of camera videos with different viewpoints, each camera parameter, and viewpoint information selected by the terminal user U in the viewing terminal 3. do.

In the first embodiment of the present invention, as in Non-Patent Document 1, an object in a three-dimensional space is approximated by a single rectangular polygon, and the virtual viewpoint P selected by the user is obtained from multiple camera images. An example of application to a free-viewpoint technology that employs a billboard method that appropriately maps texture information onto rectangular polygons will be described.

The free-viewpoint video generation unit 201 extracts an object from multiple camera videos and estimates its position. In the case of a free-viewpoint video of a soccer match, the object is a person such as a player appearing on the stadium. In addition, it is assumed that a 3D model is manually created in advance as a preset and exists for the stadium pitch, spectator seats, and other backgrounds other than objects.

Further, as shown in FIG. 2, the free-viewpoint video generation unit 201, according to the selected virtual viewpoint P, places a 1 Rectangular polygons are set, and the texture of the object is displayed on the rectangular polygons, extracted from the camera image of the real camera cam whose line-of-sight direction angle is the closest to the virtual viewpoint P.

In the example of FIG. 2, the line-of-sight angle of the virtual viewpoint P is closest to the line-of-sight angle of the real camera cam3, so the texture of the target object is extracted from the camera image of the real camera cam3 and pasted on the rectangular polygon. When displaying this texture, by using a binary mask image that expresses the shape of the target object, only the part of the rectangular polygon where the target object exists is displayed, and the other parts are transparent. . Then, when the virtual viewpoint P moves, the rectangular polygon is also rotated according to this movement, and by always facing the virtual viewpoint P, viewing from the virtual viewpoint P can be realized without discomfort.

The format recording unit 202 allows the free viewpoint video generation unit 201 to freely store various parameters for enabling the viewing terminal 3 to replay the free viewpoint video generated by the free viewpoint video generation unit 201 later with a small calculation load. Acquired in the process of generating viewpoint video and recorded in a replay format.

Recording of the replay format is started in response to a recording start request RQ2 from the viewing terminal 3 that is reproducing the free viewpoint video, and ends in response to a recording end request RQ3. The completed replay format is transferred to the viewing terminal 3 that has transmitted the requests RQ2 and RQ3, and managed on the storage 301. FIG.

In the format recording unit 202, the free viewpoint information recording unit 202a records, in the replay format, viewpoint information including the position, orientation, and attitude of the virtual viewpoint P selected by the terminal user U operating the viewing terminal 3. .

The spatial information recording unit 202b records spatial information that can reduce the processing load when the viewing terminal 3 renders an image viewed from the selected virtual viewpoint P. FIG. In the present embodiment, it is considered that an object seen from the viewpoint is specified based on the viewpoint information, and only the visible object is rendered and reconstructed in a three-dimensional space.

In addition, parameters with a high processing load in the rendering process, specifically the position of the billboard that displays each object and the mask for making a part of the texture transparent according to the object shape when pasting the texture to each billboard For images, only identifiers for the viewing terminal 3 to acquire these parameters from the free-viewpoint video generating device 2 are recorded.

In the case of FIG. 2, four objects (ID1 to ID4) stand on the background model, but only two objects (ID3, ID4) are visible from the selected virtual viewpoint P. Also, since the angle of the viewing direction of the virtual viewpoint P is close to the angle of the real camera cam3, it is assumed that the billboards installed at the positions of the objects ID3 and ID4 will have textures obtained from the camera image of the real camera cam3. do.

In such a case, in this embodiment, only two objects are reconstructed in 3D space, and only their billboard positions, textures and mask images are obtained. The object, its billboard position, the texture to be pasted on the billboard, the actual camera, and the mask image for masking the texture can be associated with each other, so it is not possible to record all of these in the replay format. redundant.

Therefore, in this embodiment, as shown in FIG. 3, which will be described later in detail, only the "camera number", "texture number" and "billboard position" are recorded as the spatial information, and the identifier of the mask image is the "texture number". ” sums it up. It should be noted that the position of each billboard does not necessarily need to be recorded if the free-viewpoint video generation device 2 records it in association with the texture number.

The viewing terminal 3 has a function of selecting a viewpoint by connecting a dedicated controller to a television capable of reproducing video, a function of selecting a viewpoint by a touch operation or a swipe operation on a touch screen provided on a display of a smartphone or a tablet, or a function of selecting a viewpoint. It can be realized by viewing free-viewpoint video on a VR terminal equipped with an acceleration sensor and selecting a viewpoint according to the user's movement.

In the viewing terminal 3, the storage 301 stores the replay format and various spatial information transmitted from the free-viewpoint video generating device 2 via the network. The user operation detection unit 302 detects each operation of the virtual viewpoint operation RQ1 by the terminal user U, the recording start request RQ2 and recording end request RQ3 of the replay format, and the replay request RQ4, and outputs the free viewpoint video generation device via the network. 2.

In response to the replay request RQ4, the rendering information acquisition unit 303 refers to the replay format stored in the storage 301, and specifies which free viewpoint video the format corresponds to. Then, the background 3D model of the specified free-viewpoint video is acquired from the free-viewpoint video generation device 2, and based on the background 3D model, textures, mask images, billboard positions, etc. necessary for reconstruction of each frame are created. Spatial information is acquired from the free viewpoint video generation device 2 . Each acquired information is temporarily stored in the storage 301 .

The replay video reproduction unit 304 executes rendering based on the replay format and the spatial information acquired from the free viewpoint video generation device 2, and reproduces the replay video of the free viewpoint video.

In this embodiment, the viewing terminal 3 renders the free-viewpoint video, but the viewing terminal 3 needs processing to render the video from the virtual viewpoint P obtained from the viewpoint information. In other words, the 3D space is created by placing billboards facing the virtual viewpoint P at the standing position of each billboard in the acquired background 3D model, and masking the acquired texture with a mask image and pasting it there. Reconfigure.

On the other hand, since it does not perform high-cost processing such as estimating the standing position of the billboard from multiple camera images and generating a mask image, considering the specs of smartphones in recent years, the above rendering can be performed on the viewing terminal side. It is quite possible to do so.

According to this embodiment, compared to the case where the rendering is performed by the free-viewpoint video generation device 2, not only can the background 3D model be transmitted only once, but only the spatial information of the visible object can be transmitted in other frames as well. should be transmitted. For example, when considering a soccer video, it is often the case that the area where the player exists is very small compared to the entire picture, so it is better to transmit only the texture and mask image than to send the rendered video every time. However, in many cases, the amount of data is small.

Also, if each texture is transmitted once and then kept stored in the viewing terminal, the texture will continue to remain in the viewing terminal 3 where the replay reconstruction has been performed once. There is no need to download, and it is possible to reconstruct the replay video without a network.

FIG. 3 is a diagram showing a first example of the replay format, which consists of header information and chronological information.

In the header information, the "free viewpoint video ID" is used to uniquely identify each replay format. Since this free-viewpoint video ID only needs to be recorded once, it becomes information written in the header of the format. The "total number of frames" is the number of frames of the replay video reconstructed based on the replay format, and corresponds to the playback time.

Time-series information is generated for each frame, and information specifying the position (time) of each frame in the replay video is recorded in the "playback time identifier". In the illustrated example, only two frames of viewpoint time-series information (21, 22) are shown. Become.

For example, with respect to free-viewpoint video of 1 minute at 30 frames per second, when a recording start request RQ2 is detected at a timing of 10 seconds from the start and a recording end request RQ3 is detected at a timing of 20 seconds, the time-series information is as follows. A "playback time identifier" is configured by linking information from 300 to 600 in chronological order. Note that the "playback time identifier" is not limited to the frame number, and may be absolute time information or relative time information. As described above, in this embodiment, information for replaying each frame is managed in chronological order. Therefore, if the audio recorded in the content server is managed by time information, video and audio can be reproduced on a time basis. Synchronized playback is easy.

Further, in this embodiment, as information for specifying the virtual viewpoint P, as shown in FIG. and "posture direction U (ux, uy, uz)" representing the viewpoint information. identified by

Note that the “orientation direction” is information indicating which of the screens used for display is on top when viewing in a certain direction from a certain viewpoint position. Even if the viewpoint position and line-of-sight direction are the same, the image viewed upright and the image viewed upside down will be upside down. Reconstruction of the replay video becomes possible.

The "camera number" is the identifier of the real camera cam closest to the virtual viewpoint P in direction. “Texture number” is the number of the texture of the object visible at the current virtual viewpoint P. "Billboard position" represents the relationship between the coordinate position of the billboard that models the object seen at the current virtual viewpoint and the identifier of the texture that is attached to the billboard. In this embodiment, such time-series information is constructed in a predetermined cycle, for example, in units of frames, managed by the "reproduction time identifier", and sequentially linked.

FIG. 5 is a sequence flow showing the procedure from construction of the replay format to reproduction of replay video based on the replay format.

At time t1, a video viewing request RQ1 is transmitted from the viewing terminal 3 to the free viewpoint video generating device 2. FIG. In response to the viewing request RQ1, the free viewpoint video generating device 2 starts distributing the video content at time t2. At time t3, the video is reproduced on the viewing terminal 3 that has acquired the video content.

At time t4, the terminal user U performs a viewpoint operation for viewing a free viewpoint video on the viewing terminal 3. When this is detected by the user operation detection unit 302, at time t5, the terminal user U selects Viewpoint information specifying the virtual viewpoint P is transferred from the viewing terminal 3 to the free-viewpoint video generation device 2 . In the free viewpoint video generation device 2, at time t6, the free viewpoint video generation unit 201 performs rendering based on the viewpoint information and each camera video to generate a free viewpoint video. At time t7, the free viewpoint video is delivered to the viewing terminal 3, and reproduced at time t8.

At time t9, the terminal user U operates the viewing terminal 3 to request the recording of the replay video, and when this is detected by the user operation detection unit 302, the recording start request RQ2 is sent from the viewing terminal 3 at time t10. It is transmitted to the free viewpoint video generation device 2 . In the free viewpoint video generating device 2, at time t11, the format recording unit 202 responds to the recording start request RQ2 and starts recording the replay format for the free viewpoint video being reproduced. Recording of the replay format is repeated frame by frame until the recording end request RQ3 from the viewing terminal 3 is detected.

In general, when the free-viewpoint video generation unit 201 is reproducing the free-viewpoint video, information on the virtual viewpoint P selected by the user can be obtained from the free-viewpoint video generation unit 201 . Also in this embodiment, when the replay video recording start RQ2 is detected, the format recording unit 202 acquires the viewpoint information from the free viewpoint video generating unit 201 in units of frames, and records the parameters in the replay format.

In this embodiment, "viewpoint position E (ex, ey, ez)", "line-of-sight direction D (dx, dy, dz)", and "attitude direction U (ux, uy, uz)" are recorded in frame units. . In addition, the positional information of the billboard position of the object visible from the current viewpoint is recorded. In addition, the texture number and camera number of the corresponding object to be pasted on each billboard are recorded.

After that, at time t12, the terminal user U operates the viewing terminal 3 to request the end of recording of the replay video. 3 to the free viewpoint video generation device 2 . In the free viewpoint video generation device 2, the format recording unit 202 finishes recording the replay format at time t14. At time t15, the generated replay format is transferred to the viewing terminal 3, and stored in the storage 301 of the viewing terminal 3 at time t16.

Thereafter, at time t17, the user who desires replay of the free viewpoint video requests replay by specifying a replay format on the storage, and when this is detected by the user operation detection unit 302, the rendering information The acquisition unit 303 interprets the replay format, and finds out which free-viewpoint video replay video the format is based on the free-viewpoint video ID described in the format.

At time t18, the viewing terminal 3 requests (RQ4) information necessary for video replay from the free viewpoint video generation device 2 based on the replay format. In this embodiment, the background 3D model linked to the free-viewpoint video ID in the replay format is requested, and at time t19, the free-viewpoint video generation device 2 responds to the request and delivers the background 3D model.

At time t20, the replay video playback unit 304 performs rendering based on the replay format and the acquired background 3D model, and playback of the free viewpoint video replay video is started. During playback of the replay video, the rendering information acquisition unit 303 requests and acquires spatial information such as billboard positions, mask images, and textures from the free viewpoint video generation device 2 on a frame-by-frame basis based on the replay format.

Then, the replay video playback unit 304 highly efficiently reconstructs the 3D space based on the free viewpoint space information described in the format, and the reconstructed space is recorded in the format. The replay video is reconstructed by rendering the image viewed from the viewpoint position based on the acquired spatial information.

FIG. 6 is a diagram showing another example of the replay format. In the above embodiment, the viewpoint information is a three-dimensional vector of viewpoint position E (ex, ey, ez), line-of-sight direction D (dx, dy, dz), posture direction U (ux, uy, uz), and nine parameters. described as being represented. However, when the parameter does not change, it may have a function to eliminate the redundancy and reduce the data size. This is realized by not describing the later parameter when the parameter does not change between frames. can.

For example, when the viewpoint moves in parallel, the line-of-sight direction D and the posture direction U do not change, and only the viewpoint position E may change. In the present embodiment, when such a movement of the viewpoint is detected, as shown in FIG. 6, recording of the line-of-sight direction D and the posture direction U is omitted for the time-series information of the next frame, thereby reducing the data size. can be reduced and the processing load can be reduced. In order to reduce the data size, the parameters to be recorded may be rounded to a certain number of digits and recorded as approximate values.

FIG. 7 is a diagram showing still another example of the replay format. This embodiment is characterized in that the gaze point position F (fx, fy, fz) is stored instead of the line-of-sight direction D (dx, dy, dz). The point-of-regard position F indicates the position of a specific point on the line-of-sight direction D. FIG. The line-of-sight direction D is obtained from the viewpoint position E and the gazing point position F by the following equation (1).

In this way, it may be possible to eliminate redundancy by adopting the gaze point position F(fx, fy, fz) instead of the viewpoint direction D(dx, dy, dz). For example, if a motion rotating around the point of gaze F is assigned to a swipe operation of the viewing terminal 3 or the like, it is conceivable that many motions rotating around the point of gaze F appear. In such a case, according to this embodiment, since the point-of-regard position F(fx, fy, fz) does not change, redundancy can be eliminated.

As still another example of the viewpoint information, the amount of rotational movement (rotation angle) and the amount of parallel movement may be employed as parameters of the viewpoint information.

To obtain a certain viewpoint, the viewpoint is rotated around the origin of the world coordinate system by the amount of rotation θx about the x-axis, the amount of rotation θy about the y-axis, and the amount of rotation θz about the z-axis. The position, line-of-sight direction, and orientation of the viewpoint can be specified by performing parallel movement T(tx, ty, tz) to the position. Therefore, the viewpoint can be reconstructed from the six parameters of rotation amounts .theta.x, .theta.y, .theta.z and translation amount T(tx, ty, tz).

In this example, the viewpoint can be reconstructed from a small number of parameters, but the default position, direction, and orientation of the viewpoint must be clearly determined before rotation and translation are applied. In other words, when no rotation, translation, etc. This means that an initial value such as "is present" must be determined, and the replay video reproduction unit 304 of the viewing terminal 3 also needs to recognize the initial viewpoint information. Although this information may be written in the format itself and exchanged, the initial position when playing back the free-viewpoint video in the free-viewpoint video generation unit 201 may be determined as the initial position.

As still another example of the viewpoint information, a form of recording a view transformation matrix may be employed. The view transformation matrix indicates a transformation matrix that performs transformation from the world coordinate system to the viewpoint coordinate system (camera coordinate system). Using this transformation matrix, the position and direction of the viewpoint and orientation information can be specified. It is possible. Here, assuming that the view coordinate matrix is represented by a homogeneous coordinate system, the 4×4 transformation matrix M is represented by the following equation (2).

Such matrices are frequently used in popular 3D display libraries such as OpenGL and DirectX. The view transformation matrix M is often calculated from the direction U(ux, uy, uz) and so on. Therefore, if the view transformation matrix is stored in advance, the processing cost can be reduced because the transformation matrix can be obtained most easily when considering its use in a library or the like.

Furthermore, in the examples of each format described above, in principle, the viewpoint information is recorded as it is, but as shown in FIG. 7, only the difference value from the previous frame may be recorded.

In such a format, since the difference value between frames tends to be small, many small values are written. When the value is small, it is conceivable that a sequence of the same value such as "0" is more likely to occur. By performing entropy coding, typically Huffman coding, on code strings in which sequences of the same values tend to occur, there is a possibility that the data size can be further reduced.

However, when it is desired to start reproduction from the middle, the difference from the first frame must be added to calculate the value of the middle frame, which tends to increase the calculation cost. Therefore, it is possible to adopt a format in which parameters other than normal differences are described in one of several frames, and difference values from the previous frame are recorded in other frames.

In this case, it is necessary to use a format that makes it possible to identify which frame holds all information and which frame holds differential information. In the example of FIG. 7, each frame is distinguished by assigning an identifier "D" to a differential frame and an identifier "I" to a non-differential frame.

The format recording unit 202 records viewpoint information by describing various parameters over each frame in each of the above methods. The free-viewpoint information recording unit 202a must have a function of converting and shaping the information on the viewpoint received from the free-viewpoint video generation unit 201 into a format described in the format.

The conversion and shaping here means that, for example, the free-viewpoint video generation unit 201 uses a viewpoint conversion matrix (view conversion matrix) from the world coordinate system to the camera coordinate system to obtain the user's viewpoint, and converts the image from a specific viewpoint. Assuming that you are generating an image, you need to obtain this transformation matrix and obtain information such as a 3D vector of the position coordinates of the viewpoint from the transformation matrix, or truncate the numerical value to be recorded to a fixed number of digits. , rounding, etc.

FIG. 8 is a diagram for explaining the free viewpoint technique adopted by the second embodiment of the present invention, and FIG. 8 is a diagram showing an example of a replay format in the second embodiment.

In the first embodiment, the free-viewpoint video generation unit 201 employs the billboard method to generate the free-viewpoint video. On the other hand, as shown in Patent Document 1, the present embodiment is a method for accurately restoring the shape of a 3D model of an object (here, expressed as a "full model method using a back projection plane"). ) to generate free-viewpoint video.

When the free viewpoint video generation unit 201 adopts the full model method, a large number of back projection planes P1, P2, . Next, a mask image of the target object obtained by background subtraction or the like is projected onto each back projection plane P1, P2, etc., and the visual volume is calculated to create a 3D model for each back projection plane. Furthermore, the back projection plane is colored by mapping the texture image of the target object. Therefore, it is possible to restore the 3D model by properly cutting out the back projection plane.

In such a method, the backprojection planes P1, P2, etc. are always arranged orthogonally to the virtual line of sight P, so the positions of the backprojection planes P1, P2, etc. change depending on the position of the virtual viewpoint P. do. The format recording unit 202 starts recording the format in response to a replay video recording start request RQ2 from the viewing terminal 3 . At this time, as in the first embodiment, the free viewpoint video ID and the total number of frames are recorded in the header, and the viewpoint information is also recorded for each frame in the same manner as in the first embodiment.

The spatial information recording unit 202b records, as spatial information, only the index of the plane on which the model is generated among the many backprojection planes P1, P2, . . . . That is, in this embodiment, the indices of faces for which models are not generated are not recorded. For example, in the example shown in FIG. 8, a cylindrical object exists in space, but only P2, P3, and P4 are modeled for it. Therefore, as shown in FIG. 9, only "2 3 4" is recorded as the index.

For example, in free-viewpoint video, such as soccer, in which players exist discretely in one part of a wide field, if backprojection planes are arranged over the entire field, many useless backprojection planes are generated for which models are not generated. It is useless to re-calculate such aspects when reproducing a replay video.

On the other hand, in the present embodiment, it is possible to distinguish in advance between backprojection planes for which models are generated and those for which models are not generated. The calculation load is reduced because the calculation for backprojecting the image is also unnecessary.

In particular, Patent Document 1 mentions that the full model method of Patent Document 1, which is adopted in this embodiment, performs parallel calculation using a GPU, and reducing the number of back projection planes leads to memory saving. . As a result, it is possible to reduce the time required for memory access, etc., so that it is possible to save computational resources and increase the speed of computation.

The space information recording unit 202b records the index of the projection plane that needs to be calculated when reproducing the 3D space, thereby speeding up calculation and saving calculation resources. As for the indices assigned to the backprojection planes, if there are 1000 backprojection planes facing the viewpoint, a method of assigning indices from 1 to 1000 in order from the one closest to the viewpoint can be considered. In the format example shown in FIG. 9, "2 3 4" is recorded as an index representing the three backprojection planes P2, P3, and P4 on which models are generated. The replay format recorded in this manner is transferred to and stored in the viewing terminal 3 in the same manner as in the first embodiment, and will be referred to later during replay.

In the viewing terminal 3, the replay video is reconstructed based on the replay format in which the replay video playback unit 304 is stored. At this time, the viewpoint is determined based on the viewpoint information of the replay video format, and the back projection plane is arranged based on this viewpoint. No placement and calculations are performed for the projection plane.

As a result, it is possible to reproduce the 3D space by performing calculations only on the back projection plane for which the model is guaranteed to be generated. After that, the video from the viewpoint is rendered in this 3D space, and the rendered image is transmitted to the viewing terminal 3 to realize the reproduction of the replay video.

In this embodiment, rendering is performed in the free-viewpoint video generation device 2. However, for example, all videos for constructing the free-viewpoint video may be delivered to the viewing terminal 3 in advance and rendered on the terminal side. Also good. Even in this case, if the index of the back projection plane on which no model is generated is recorded in advance, it is possible to speed up the calculation and save the calculation resources.

Moreover, in such a configuration, the free-viewpoint video generation device 2 is provided with a replay video reproduction function, which enables complete reconstruction of the 3D space. For this reason, the replay video playback function receives information about the specs of the viewing terminal 3, and renders the rendered image so that the visual field and the aspect ratio of the image change, although the viewpoint is the same, according to the resolution and screen size of the playback device. A function to output may be provided.

In addition, when a plurality of viewing terminals 3 simultaneously request the free viewpoint video generation device 2 to reproduce a replay video, if there is a rendering request for a frame at the same viewpoint position and at the same time, rendering is performed. A mechanism for saving the result and reusing it may be provided.

REFERENCE SIGNS LIST 1 content server 2 free viewpoint video generation device 3 viewing terminal 201 free viewpoint video generation unit 202 format recording unit 202a free viewpoint information recording unit 202b spatial information recording unit 301 storage , 302... user operation detection unit, 303... rendering information acquisition unit, 304... replay video reproduction unit

Claims (14)

In a system configured by connecting a viewing terminal and a free-viewpoint video generation device via a network and reproducing a replay video of the free-viewpoint video,
the viewing device
means for requesting playback of free-viewpoint video;
means for requesting recording of a replay video for the free-viewpoint video being played;
The free-viewpoint video generation device
means for generating a free viewpoint video based on viewpoint information of a plurality of camera videos and virtual viewpoints in response to the playback request;
means for recording, in response to the recording request, a replay format in which a virtual viewpoint is described for each playback time of the replay video in the free viewpoint video generation process;
means for transferring the recorded replay format to the viewing terminal;
The viewing terminal further
means for acquiring information necessary for replay based on the replay format;
means for executing rendering based on the replay format and the acquired information and reproducing the replay video;
A system for reproducing a replay video of a free-viewpoint video, wherein the information necessary for the replay is a background 3D model and spatial information for rendering an object on the background 3D model. the replay format includes header information and chronological information;
The time-series information is configured by concatenating a plurality of time-series information recorded at a predetermined cycle,
2. The system for reproducing a replay video of a free-viewpoint video according to claim 1, wherein each piece of time-series information records a reproduction time ID unique to a reproduction position of the replay video based on the time-series information. the generating process includes a background 3D model generating process for the free-viewpoint video;
The ID of the free viewpoint video is recorded in the replay format,
3. The system according to claim 1, wherein said acquiring means acquires said background 3D model based on an ID of said free viewpoint video. the generation process includes a viewpoint information generation process;
4. The system for reproducing a replay video of a free viewpoint video according to claim 1, wherein the viewpoint information is recorded in the replay format. 5. A system for reproducing a replay video of a free viewpoint video according to claim 1, wherein the means for generating the free viewpoint video employs billboard type free viewpoint technology. the generating process includes identifying objects visible in a virtual viewpoint;
A general ID associated with the visible object is recorded in the replay format,
6. The system according to claim 5, wherein said acquiring means acquires rendering information of each object based on said general ID. the generating process includes generating a mask image for each object;
The mask image is associated with the general ID,
7. The system according to claim 6, wherein said acquiring means acquires a mask image of each object based on said general ID. the generating process includes extracting the texture of each object from camera footage;
The texture is associated with the general ID,
7. The system according to claim 6, wherein said acquiring means acquires the texture of each object based on said general ID. 5. The replay video of the free viewpoint video according to any one of claims 1 to 4, wherein the means for generating the free viewpoint video employs a full model type free viewpoint technology using a back projection plane. system to do. The generating process includes a process of arranging a plurality of back projection planes directly facing the virtual viewpoint at the position of the object, projecting a mask image of the object, and performing 3D modeling for each of the back projection planes to restore the 3D model. ,
Record the index of the back projection plane where the 3D model exists in the replay format;
10. The system according to claim 9, wherein said acquiring means acquires information necessary for replay based on the index of said back projection plane. the means for generating the free viewpoint video and the means for recording the replay format are mounted on a server on the cloud;
a means for acquiring information necessary for the replay and a means for reproducing the replay video are mounted on a replay video viewing terminal;
11. The system for reproducing replay video of free viewpoint video according to any one of claims 1 to 10, wherein the replay format is transferred from a server to a viewing terminal and stored on the viewing terminal. the means for generating the free viewpoint video, the means for recording the replay format, and the means for acquiring information necessary for the replay are implemented on a server on the cloud;
means for playing back the replay video is mounted on the viewing terminal;
11. The system for reproducing replay video of free viewpoint video according to any one of claims 1 to 10, wherein the replay format is transferred from a server to a viewing terminal and stored on the viewing terminal. 12. The means for recording the replay format starts recording the replay format in response to a recording start request from a viewing terminal, and finishes recording in response to a recording end request. 13. A system for reproducing a replay video of the free viewpoint video according to 12. In a method for reproducing a replay video of a free-viewpoint video, wherein a viewing terminal and a free-viewpoint video generation device are connected via a network,
the viewing device
request playback of free-viewpoint video,
Request recording of replay video for the free viewpoint video being played,
The free-viewpoint video generation device
Generating a free viewpoint video based on viewpoint information of a plurality of camera videos and a virtual viewpoint in response to the playback request;
recording a replay format in which a virtual viewpoint is described for each playback time of the replay video in the free viewpoint video generation process in response to the recording request;
Transfer the recorded replay format to the viewing terminal,
The viewing terminal further
Acquiring information necessary for replay based on the replay format;
Rendering is performed based on the replay format and the acquired information to reproduce the replay video,
A method for reproducing a replay video of a free-viewpoint video, wherein the information necessary for the replay is a background 3D model and spatial information for rendering an object on the background 3D model.

Images