JP2022036123A - System and method for reproducing replay video of free-viewpoint video - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable replay videos of free-viewpoint videos to be reproduced in a low-load manner, by which even audiovisual terminals with poor processing capacity can reproduce the replay videos of the free-viewpoint videos.

SOLUTION: A replay video reproduction system for free-viewpoint videos according to the present invention includes: a free-viewpoint video generation section 201 for generating free-viewpoint videos based on a plurality of camera videos and viewpoint information of virtual viewpoints; a format recording section 202 for recording formats for replays in a process generating the free-viewpoint videos; a rendering information acquisition section 303 for acquiring information required for the replays based on the formats for replays; and a replay video reproduction section 304 for reproducing replay videos based on the formats for replays and the acquired rendering information. The free-viewpoint video generation section 201 and the format recording section 202 are implemented on a cloud server, whereas the rendering information acquisition section 303 and the replay video reproduction section 304 are implemented on an audiovisual terminal 3.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a system and a method for reproducing a replay image of a free viewpoint image, and in particular, by reducing the load related to the construction of a replay image of the free viewpoint image, the replay of the free viewpoint image is performed even on a viewing terminal having inferior processing capacity. Related to systems and methods that allow video to be played.

Free-viewpoint video technologies such as Patent Document 1 and Non-Patent Document 1 have been proposed as technologies that enable video viewing from a virtual viewpoint in which cameras are not actually placed, based on video shot from multiple cameras. It has been. By arranging multiple cameras in sports stadiums and generating free-viewpoint images based on the images from these multiple cameras, users can enjoy viewing images from any virtual viewpoint they want to see. It is possible.

When considering recording an image from a virtual viewpoint selected by a user by using such a free viewpoint image technology, the displayed screen is captured, for example, H.264 described in Non-Patent Document 2. It is conceivable to save as a video file in an existing video format such as (MPEG-4 AVC). The video file recorded in this way is saved in a storage or the like, and can be played back when it is desired to be viewed again, or the saved file can be sent to another user to share the video with another user.

Further, as an example of recording a video from a specific viewpoint selected by the user without directly recording the video and playing it back later, a game device and a game replay method as shown in Patent Document 2 are used. be. INDUSTRIAL APPLICABILITY The present invention saves a history of a player's game operation as replay data, and later reproduces a player's play image based on the replay data.

Japanese Patent Application No. 2017-167472 Japanese Unexamined Patent Publication No. 2010-264173

T. Koyama, I. Kitahara and Y. ohta, "Live Mixed-Reality 3D Video in Soccer Stadium" In Proc of IEEE / ACM Conference on ISMAR, pp. 178-187 (2003) ITU-T Recommendation H.264 "Advanced video coding for generic audiovisual services," May 2003

By using the free viewpoint video technology, the user can freely select the viewpoint based on the touch operation of the screen of the dedicated controller, smartphone, or tablet, and enjoy watching the video from any viewpoint. Unlike TV images such as terrestrial broadcasting that are normally broadcast, each user can freely select and move the viewpoint in the free-viewpoint video, so even if the same free-viewpoint video is used, each user can freely select and move the viewpoint. The image you are watching depends on the position of the viewpoint and how you move the viewpoint. Therefore, it is possible to create original video content by recording the video from the viewpoint that the user is viewing.

When considering the future in which free-viewpoint video technology has become widespread, it is natural that videos from such specific viewpoints will be exchanged via the Internet, and new ways of enjoying will be born, such as evaluations and comments via SNS. Conceivable.

In this way, you want to play back the video from a specific viewpoint (hereinafter referred to as replay video) recorded on a specific terminal, play it on another terminal, or share it with a large number of users. In response to demand, it is possible to save in the existing video coding format as described in Non-Patent Document 2 and exchange the created video, but there is a problem that the capacity of the video becomes large. Was.

In particular, the free-viewpoint video allows the user to generate video from each viewpoint, and it is possible to generate a huge variety of video contents from one free-viewpoint video, so such a problem is remarkable. It is thought that it will appear.

Further, if the movement of the viewpoint is known, an image from a specific viewpoint can be obtained by passing the information of the viewpoint to the free viewpoint video generator. For example, like the game replay function shown in Patent Document 2, it is possible to recalculate and display the image from a given viewpoint based on the information of the viewpoint recorded in advance. ..

However, unlike games, it is necessary to first reconstruct a three-dimensional space from images from a plurality of cameras in a free-viewpoint image, and the calculation cost for reconstructing this 3D space is very high. In particular, when considering a case where a large number of users request the server to reconstruct the replay video at the same time, it is difficult to reconstruct the replay video without delay.

An object of the present invention is a system that solves the above technical problems, reduces the load related to the construction of a replay image of a free viewpoint image, and enables a viewing terminal having inferior processing capacity to reproduce the replay image of the free viewpoint image. To provide a method.

In order to achieve the above object, the present invention is configured by connecting a viewing terminal and a free viewpoint video generator via a network, and has the following configurations in a system for reproducing a replay video of a free viewpoint video. There is a feature in.

(1) The viewing terminal is provided with a means for requesting the reproduction of the free-viewpoint video and a means for requesting the recording of the replay video for the free-viewpoint video being played.
In response to the reproduction request, the free-viewpoint video generation device generates a free-viewpoint video based on viewpoint information of a plurality of camera images and a virtual viewpoint, and in the process of generating the free-viewpoint video, the replay video is It is provided with a means for recording a replay format in which a virtual viewpoint is described for each playback time and a means for transferring the recorded replay format to a viewing terminal.
The viewing terminal further includes a means for acquiring information necessary for replay based on the replay format and a means for reproducing the replay video based on the replay format and the acquired information, which is necessary for the replay. Information now includes the background 3D model and the spatial information for rendering the object on the background 3D model.

(2) The spatial information includes the mask image of each object and the position information in which the model of each object is placed.

(3) A means for generating a free-viewpoint video and a means for recording a replay format are implemented in a server on the cloud, and a means for acquiring information necessary for replay and a means for playing the replay video are provided on the replay video viewing terminal. Implemented so that the replay format is transferred from the server to the viewing terminal and stored on the viewing terminal.

According to the present invention, the following effects are achieved.

(1) In the process of generating a free-viewpoint video, information such as an identifier of information that can be diverted to the reconstruction of the replay video and parameters necessary for the reconstruction of the replay video is recorded in the replay format. Therefore, when reconstructing the replay video, the information that can be diverted to the reconstruction of the replay video is acquired based on the identification information recorded in the replay format, and the information recorded in the replay format is used as a parameter. Then, the replay video can be played with a light load.

(2) Since the background 3D model and the spatial information for rendering the object on the background 3D model are acquired as the information necessary for reconstructing the replay video, the processing load when reconstructing the replay video is reduced. Ru.

(3) Since the spatial information includes the mask image of each object and the position information for arranging the model of each object, when reconstructing the replay video, there is no need for calculation to obtain such information having a high processing load. Become.

(4) Implement a means to generate a free-viewpoint video and a means to record a replay format on a server on the cloud, and provide a means to acquire information necessary for replay and a means to play the replay video on the replay video viewing terminal. If implemented, it is possible to impose a high processing load calculation on a server with high processing power in general. Therefore, it is possible to replay the free-viewpoint video even on a viewing terminal having generally low processing power.

It is a block diagram which showed the structure of the main part of the free viewpoint video distribution system which concerns on one Embodiment of this invention. It is a figure for demonstrating the method of switching the object which is the object of setting of a polygon according to the virtual viewpoint P. It is a figure which showed the 1st example of the format for replay. It is a figure for demonstrating the definition of viewpoint information. It is a sequence flow showing the procedure from the construction of the replay format to the reproduction of the replay video. It is a figure which showed the 2nd example of the format for replay. It is a figure which showed the 3rd example of the format for replay. It is a figure which showed the generation method of the free viewpoint image in 2nd Embodiment of this invention. It is a figure which showed the 3rd example of the format for replay.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a main part of a free-viewpoint video distribution system according to an embodiment of the present invention, and the configuration unnecessary for the description of the present invention is omitted here.

The free-viewpoint video distribution system of the present invention is a content server 1 that is arranged in a stadium or the like and stores a plurality of camera cams that shoot objects such as athletes from different viewpoints, videos taken by each camera cam, and camera parameters. , Free-viewpoint video generator 2 that generates free-viewpoint video based on multiple camera images, camera parameters, and viewpoint information, and free-viewpoint video generation of virtual viewpoint P information (viewpoint information) according to the operation of terminal user U. The main configuration is a viewing terminal 3 that is provided to the device 2 and acquires and reproduces the free-viewpoint video generated by the free-viewpoint video generation device 2.

The content server 1, the free-viewpoint video generation device 2, and the viewing terminal 3 may be configured by mounting an application (program) that realizes each function described later on a general-purpose computer, or a part of the application is hardware. It may be configured as a dedicated machine or a single-purpose machine that has been made into hardware or ROM.

In the content server 1, the captured camera image and the content including the camera parameters thereof are managed by a unique ID. In the illustrated example, ID1 is assigned to the free-viewpoint content of soccer, ID2 is assigned to the free-viewpoint content of volleyball, and ID3 is assigned to the free-viewpoint content of judo.

The free-viewpoint video generation device 2 includes a free-viewpoint video generation unit 201 and a format recording unit 202, and by implementing these functions on a server placed on the cloud, it can be configured as a free-viewpoint video generation server. can.

The free viewpoint image generation unit 201 generates a free viewpoint image by applying a plurality of camera images having different viewpoints, each camera parameter, and viewpoint information selected by the terminal user U in the viewing terminal 3 to a known free viewpoint technique. do.

In the first embodiment of the present invention, as in Non-Patent Document 1, an object in a three-dimensional space is approximated by one rectangular polygon, and is acquired from a plurality of camera images according to a virtual viewpoint P selected by the user. An example of application to a free-viewpoint technology that employs a billboard method that appropriately maps the texture information to a rectangular polygon will be described.

The free viewpoint image generation unit 201 extracts an object from a plurality of camera images and estimates its position. In the case of a free-viewpoint video of a soccer game, a person such as a player appearing on the stadium is an object. For backgrounds such as stadium pitches and spectator seats other than objects, it is assumed that a 3D model is manually created and exists as a preset.

Further, as shown in FIG. 2, the free viewpoint image generation unit 201 faces the line-of-sight direction of the virtual viewpoint P at a position where each object is presumed to exist according to the selected virtual viewpoint P1. A piece of rectangular polygon is installed, and the texture of the object extracted from the camera image of the real camera cam having the closest angle in the line-of-sight direction to the virtual viewpoint P is displayed on the rectangular polygon.

In the example of FIG. 2, since the line-of-sight angle of the virtual viewpoint P is the closest to the line-of-sight angle of the real camera cam3, the texture of the target object is cut out from the camera image of the real camera cam3 and pasted on the rectangular polygon. When displaying this texture, by using a binary mask image that expresses the shape of the target object, only the part of the rectangular polygon where the target object exists is displayed, and the other parts are transparent. .. Then, when the virtual viewpoint P moves, the rectangular polygon is also rotated in response to this movement, and by always facing the virtual viewpoint P, viewing from the virtual viewpoint P can be realized without discomfort.

In the format recording unit 202, the free viewpoint video generation unit 201 is free to set various parameters for allowing the viewing terminal 3 to later replay the free viewpoint video generated by the free viewpoint video generation unit 201 with a small calculation load. Obtained in the process of generating the viewpoint image and recorded in the replay format.

Recording in the replay format is started in response to the recording start request RQ2 by the viewing terminal 3 playing the free viewpoint video, and ends in response to the recording end request RQ3. The completed replay format is transferred to the viewing terminal 3 that has transmitted the requested RQ2 and RQ3, and is managed on the storage 301.

In the format recording unit 202, the free viewpoint information recording unit 202a records the viewpoint information including the position, orientation, and posture of the virtual viewpoint P selected by the terminal user U by operating the viewing terminal 3 in the replay format. ..

The spatial information recording unit 202b records spatial information that can reduce the processing load when the viewing terminal 3 renders an image that can be seen from the selected virtual viewpoint P. In the present embodiment, it is considered that an object visible from the viewpoint is specified based on the viewpoint information, and only the visible object is rendered and reconstructed in a three-dimensional space.

In addition, parameters with a high processing load in the rendering process, specifically, the position of the billboard that displays each object and the mask for making a part of it transparent according to the object shape when pasting the texture on each billboard. For the image, only the identifier for the viewing terminal 3 to acquire these parameters from the free viewpoint image generation device 2 is recorded.

In the case of FIG. 2, four objects (ID1 to ID4) stand on the background model, but only two objects (ID3 and ID4) can be seen from the selected virtual viewpoint P. Also, since the angle of the line-of-sight direction of the virtual viewpoint P is close to the angle of the real camera cam3, it is assumed that the texture acquired from the camera image of the real camera cam3 will be attached to the billboard installed at the positions of objects ID3 and ID4. do.

In such a case, in the present embodiment, only two objects are reconstructed in the three-dimensional space, and only these billboard positions, textures, and mask images are acquired. Since the object, its billboard position, the texture to be attached to the billboard, its actual camera, and the mask image for masking the texture can be linked to each other, it is not possible to record all of them in the replay format. It becomes redundant.

Therefore, in this embodiment, as shown in FIG. 3 to be described in detail later, only the "camera number", "texture number" and "bilboard position" are recorded as spatial information, and the identifier of the mask image is "texture number". Is summarized in. It should be noted that the position of each billboard does not necessarily have to be recorded as long as the free viewpoint video generation device 2 records the position in association with the texture number.

The viewing terminal 3 has a function of connecting a dedicated controller to a TV capable of playing video and selecting a viewpoint, a function of selecting a viewpoint by a touch operation or a swipe operation on a touch screen provided on a display of a smartphone or tablet, or a function of selecting a viewpoint. This can be achieved by viewing a free-viewpoint video on a VR terminal equipped with an acceleration sensor and selecting the viewpoint according to the movement of the user.

In the viewing terminal 3, the storage 301 stores a replay format and various spatial information transmitted from the free viewpoint video generation device 2 via a network. The user operation detection unit 302 detects each operation of the virtual viewpoint operation RQ1, the recording start request RQ2, the recording end request RQ3, and the replay request RQ4 by the terminal user U, and is a free viewpoint video generation device via the network. Send to 2.

In response to the replay request RQ4, the rendering information acquisition unit 303 refers to the replay format stored in the storage 301, and specifies which free-viewpoint video the format belongs to. Then, the background 3D model of the specified free-viewpoint video is acquired from the free-viewpoint video generator 2, and based on the background 3D model, the texture, mask image, billboard position, etc. required for reconstructing each frame are obtained. Spatial information is acquired from the free viewpoint image generator 2. Each acquired information is temporarily stored in the storage 301.

The replay video reproduction unit 304 executes rendering based on the replay format and the spatial information acquired from the free viewpoint video generation device 2, and reproduces the replay video of the free viewpoint video.

In this embodiment, the free viewpoint video is rendered on the viewing terminal 3, but the viewing terminal 3 needs a process of rendering the video from the virtual viewpoint P obtained from the viewpoint information. In other words, a billboard is erected at the standing position of each billboard of the acquired background 3D model so as to face the virtual viewpoint P, and the acquired texture is masked with a mask image and pasted to create a 3D space. Reconstruct.

On the other hand, since high-cost processing such as estimating the standing position of the billboard from multiple camera images and generating a mask image is not performed, the above rendering is performed on the viewing terminal side in view of the specifications of recent smartphones and the like. It is quite possible to do.

According to the present embodiment, as compared with the case where the rendering is performed by the free viewpoint video generator 2, not only the background 3D model can be transmitted only once, but also the spatial information of the visible object can be obtained only in other frames. It should be transmitted. For example, when considering a soccer image, the area where the player exists is often very small for the entire picture, and it is better to transmit only the texture or mask image rather than sending the rendered image each time. However, there are many cases where the amount of data is lightweight.

Also, if each texture is transmitted once and then kept saved in the viewing terminal, the texture will continue to remain in the viewing terminal 3 that has been replayed once, so the texture numbers that have already been downloaded will be re-recorded. There is no need to download and it will be possible to reconstruct the replay video without a network.

FIG. 3 is a diagram showing a first example of the replay format, and is composed of header information and time-series information.

In the header information, the "free viewpoint video ID" is used to uniquely identify each replay format. Since this free viewpoint video ID only needs to be recorded once, it is information to be written in the header of the format. The "total number of frames" is the number of frames of the replay video reconstructed based on the replay format, and corresponds to the playback time.

Time-series information is generated for each frame, and information that identifies the position (time) of each frame in the replay video is recorded in the "playback time identifier". In the illustrated example, only the viewpoint time series information (21, 22) for two frames is shown, but if the "total number of frames" is 200, the time series information for 200 frames will be concatenated. Become.

For example, for a 1-minute free-viewpoint video at 30 frames per second, when the recording start request RQ2 is detected at the timing of 10 seconds from the start and the recording end request RQ3 is detected at the timing of 20 seconds, the time-series information is The "playback time identifier" is configured by concatenating 300 information to 600 information in chronological order. The "playback time identifier" is not limited to the frame number, and may be absolute time information or relative time information. As described above, in the present embodiment, the information for replaying each frame is managed in chronological order. Therefore, if the audio recorded in the content server is managed by the time information, the video and audio can be recorded on a time basis. You will be able to easily play in sync.

Further, in the present embodiment, as the information for specifying the virtual viewpoint P, as shown in FIG. 4, the “viewpoint position E (ex, ey, ez)” representing the three-dimensional position coordinates of the viewpoint, the direction of the viewpoint (line of sight). The "line-of-sight direction D (dx, dy, dz)" that represents the viewpoint information and the "posture direction U (ux, uy, uz)" that represents the posture information of the viewpoint are adopted, and the viewpoint information is a total of nine parameters of three-dimensional vectors. Specified by.

The "posture direction" is information indicating which of the screens used for display is on top when looking in a certain direction from a certain viewpoint position. Even if the viewpoint position and line-of-sight direction are the same, the image is upside down between the image viewed in an upright position and the image viewed in an upright position. Replay videos can be reconstructed.

The "camera number" is an identifier of the real camera cam whose direction is closest to the virtual viewpoint P. The "texture number" is the number of the texture of the object visible in the current virtual viewpoint P. The "billboard position" represents the relationship between the coordinate position of the billboard that models the object visible in the current virtual viewpoint and the identifier of the texture attached to the billboard. In the present embodiment, such time-series information is constructed in a predetermined cycle, for example, in frame units, managed by the "reproduction time identifier", and sequentially connected.

FIG. 5 is a sequence flow showing a procedure from the construction of the replay format to the reproduction of the replay video based on the replay format.

At time t1, the video viewing request RQ1 is transmitted from the viewing terminal 3 to the free viewpoint video generator 2. The free-viewpoint video generation device 2 starts distribution of video content at time t2 in response to the viewing request RQ1. At time t3, the video is played back on the viewing terminal 3 that has acquired the video content.

At time t4, the terminal user U performs a viewpoint operation for viewing the free viewpoint video to the viewing terminal 3, and when this is detected by the user operation detection unit 302, the terminal user U selects at time t5. The viewpoint information that identifies the virtual viewpoint P is transferred from the viewing terminal 3 to the free viewpoint video generator 2. In the free viewpoint image generation device 2, at time t6, the free viewpoint image generation unit 201 performs rendering based on the viewpoint information and each camera image, and generates a free viewpoint image. At time t7, the free viewpoint video is delivered to the viewing terminal 3 and played back at time t8.

At time t9, the terminal user U operates the viewing terminal 3 to request the start of recording of the replay video, and when this is detected by the user operation detection unit 302, at time t10, the recording start request RQ2 is sent from the viewing terminal 3. It is transmitted to the free viewpoint video generator 2. In the free-viewpoint video generation device 2, at time t11, the format recording unit 202 starts recording the replay format for the free-viewpoint video being reproduced in response to the recording start request RQ2. Recording of the replay format is repeated in frame units until the recording end request RQ3 from the viewing terminal 3 is detected.

Generally, when the free viewpoint video generation unit 201 is playing back the free viewpoint video, the information of the virtual viewpoint P selected by the user can be acquired from the free viewpoint video generation unit 201. Also in this embodiment, when the recording start RQ2 of the replay video is detected, the format recording unit 202 acquires the viewpoint information from the free viewpoint video generation unit 201 in frame units, and records the parameters in the replay format.

In this embodiment, "viewpoint position E (ex, ey, ez)", "line-of-sight direction D (dx, dy, dz)" and "posture direction U (ux, uy, uz)" are recorded in frame units. .. In addition, the position information of the position where the billboard of the object seen from the current viewpoint is erected is recorded. In addition, the texture number and camera number of the corresponding object to be attached to each billboard are recorded.

After that, at time t12, the terminal user U operates the viewing terminal 3 to request the end of recording of the replay video, and when this is detected by the user operation detection unit 302, at time t13, the recording end request RQ3 is sent to the viewing terminal. It is transmitted from 3 to the free viewpoint image generation device 2. In the free viewpoint video generation device 2, at time t14, the format recording unit 202 ends recording of the replay format. At time t15, the generated replay format is transferred to the viewing terminal 3, and at time t16, it is stored in the storage 301 of the viewing terminal 3.

After that, at time t17, a user who desires to replay the free viewpoint video specifies a replay format on the storage and requests replay, and when this is detected by the user operation detection unit 302, the rendering information. The acquisition unit 303 interprets the replay format and determines which free-viewpoint video replay video the format is based on the free-viewpoint video ID described in the format.

At time t18, the viewing terminal 3 requests the free viewpoint video generator 2 (RQ4) for information necessary for video replay based on the replay format. In the present embodiment, a background 3D model associated with the free-viewpoint video ID in the replay format is requested, and at time t19, the free-viewpoint video generator 2 distributes the background 3D model in response to the request.

At time t20, the replay video reproduction unit 304 renders based on the replay format and the acquired background 3D model, and the reproduction of the replay video of the free viewpoint video is started. During playback of the replay video, the rendering information acquisition unit 303 requests and acquires spatial information such as a billboard position, a mask image, and a texture from the free viewpoint video generation device 2 based on the replay format on a frame-by-frame basis.

Then, the replay image reproduction unit 304 reconstructs the 3D space with high efficiency based on the free viewpoint space information described in the format, and the reconstructed space is recorded in the format. The replay image is reconstructed by rendering the image viewed from the viewpoint position based on the acquired spatial information.

FIG. 6 is a diagram showing another example of the replay format. In the above embodiment, the viewpoint information is a three-dimensional vector of the viewpoint position E (ex, ey, ez), the line-of-sight direction D (dx, dy, dz), and the posture direction U (ux, uy, uz), and 9 parameters. Explained as being expressed. However, when the parameter does not change, it may be provided with a function to eliminate the redundancy and reduce the data size, which is realized by not describing the later parameter when the parameter does not change between frames. can.

For example, when the viewpoint moves in parallel, the line-of-sight direction D and the posture direction U do not change, and only the viewpoint position E may change. In the present embodiment, when such a movement of the viewpoint is detected, as shown in FIG. 6, the data size of the time-series information of the next frame is omitted by omitting the recording of the line-of-sight direction D and the posture direction U. And the processing load can be reduced. Further, the parameters to be recorded may be rounded to a certain number of digits and recorded as approximate values in order to reduce the data size.

FIG. 7 is a diagram showing still another example of the replay format. The present embodiment is characterized in that the gazing point position F (fx, fy, fz) is stored instead of the line-of-sight direction D (dx, dy, dz). The gazing point position F indicates the position of a specific point on the line-of-sight direction D. The line-of-sight direction D is obtained from the viewpoint position E and the gazing point position F by the following equation (1).

In this way, redundancy may be eliminated by adopting the gazing point position F (fx, fy, fz) instead of the viewpoint direction D (dx, dy, dz). For example, if a movement that rotates around the gazing point F is assigned to a swipe operation of the viewing terminal 3, it is conceivable that many movements that rotate around the gazing point F will appear. In such a case, according to the present embodiment, the gazing point position F (fx, fy, fz) does not change, so that redundancy can be eliminated.

As still another example of the viewpoint information, a rotational movement amount (rotation angle) and a parallel movement amount may be adopted as parameters of the viewpoint information.

To obtain a certain viewpoint, rotate the viewpoint by the amount of rotation θx around the x-axis, the amount of rotation θy around the y-axis, and the amount of rotation θz around the z-axis around the origin of the world coordinate system. By translating to the position T (tx, ty, tz), the position of the viewpoint, the direction of the line of sight, and the posture can be specified. Therefore, the viewpoint can be reconstructed from the six parameters of the rotation amount θx, θy, θz and the translation amount T (tx, ty, tz).

In this example, the viewpoint can be reconstructed from a small number of parameters, but the default position, direction, and posture of the viewpoint must be clearly determined before rotation or translation. This means that if you do not rotate or translate anything, "the viewpoint position is at the origin of the world coordinate system, pointing in the positive direction of the z-axis, and the attitude is facing up in the positive direction of the y-axis. It means that it is necessary to determine an initial value such as "is", and the replay video reproduction unit 304 of the viewing terminal 3 also needs to recognize the initial viewpoint information. This information may be written in the format itself and exchanged, but the initial position when the free viewpoint image is reproduced in the free viewpoint image generation unit 201 may be set as the initial position as it is.

As yet another example of the viewpoint information, a form of recording a view transformation matrix may be adopted. The view transformation matrix points to a transformation matrix that transforms the world coordinate system to the viewpoint coordinate system (camera coordinate system), and by using this transformation matrix, the position, direction, and attitude information of the viewpoint are specified. It is possible. Here, assuming that the view coordinate matrix is represented by a homogeneous coordinate system, the 4 × 4 transformation matrix M is represented by the following equation (2).

Such a matrix is frequently used in popular 3D display libraries such as OpenGL and DirectX, and has a viewpoint position E (ex, ey, ez), a line-of-sight direction D (dx, dy, dz), and a posture. The view transformation matrix M is often calculated from the direction U (ux, uy, uz). Therefore, if the view transformation matrix is saved in advance, the transformation matrix can be obtained most easily when it is considered to be used in a library or the like, and the processing cost is reduced.

Further, in the above examples of each format, the viewpoint information is recorded as it is in principle, but as shown in FIG. 7, only the difference value from the previous frame may be recorded.

In such a format, the difference value between frames tends to be small, so that a large number of small values are written. When the value becomes small, it is considered that the same value sequence such as "0" is likely to occur. There is a possibility that the data size can be further reduced by performing entropy coding as typified by Huffman coding for such a code string in which the same sequence of values is likely to occur.

However, when it is desired to reproduce from the middle, the calculation cost tends to be large because the value of the frame in the middle must be calculated by adding the differences from the first frame. Therefore, it is possible to describe a parameter that is not a normal difference in one sheet in several frames, and to record the difference value from the previous frame in the other frames.

In this case, it is necessary to format the frame so that it can be understood which frame holds all the information and which frame holds the difference information. In the example of FIG. 7, each frame is distinguished by assigning an identifier "D" to the difference frame and an identifier "I" other than the difference frame.

The format recording unit 202 records information on the viewpoint by describing various parameters over each frame in each of the above methods. The free viewpoint information recording unit 202a must have a function of converting or shaping the information regarding the viewpoint received from the free viewpoint video generation unit 201 so as to be described in the format.

The transformation and shaping referred to here is, for example, in the free viewpoint image generation unit 201, using a viewpoint transformation matrix (view transformation matrix) from the world coordinate system to the camera coordinate system in order to obtain the user's viewpoint from a specific viewpoint. Assuming that an image is being generated, the calculation process from acquiring this transformation matrix to obtaining information such as the three-dimensional vector of the position coordinates of the viewpoint from the transformation matrix, or truncating the numerical value to be recorded with a fixed number of digits. , Refers to the process of converting to a format suitable for the format, such as the process of rounding.

FIG. 8 is a diagram for explaining a free viewpoint technique adopted by the second embodiment of the present invention, and FIG. 8 is a diagram showing an example of a replay format in the second embodiment.

In the first embodiment, it has been described that the free viewpoint image generation unit 201 adopts the billboard method to generate the free viewpoint image. On the other hand, as shown in Patent Document 1, this embodiment is expressed as a method of accurately restoring the shape of a 3D model of an object (here, a "full model method using a back projection plane"). ) Is adopted to generate a free-viewpoint image.

When the free viewpoint image generation unit 201 adopts the full model method, a large number of back projection planes P1, P2 ... Are arranged so as to face the virtual viewpoint P in order to restore the 3D shape of the object. Next, a mask image of the target object obtained by background subtraction is projected onto each back-projection surface P1, P2 ..., and the visual volume is calculated to create a 3D model for each back-projection surface. Furthermore, the back projection surface is colored by mapping the texture image of the target object. Therefore, it is possible to restore the 3D model by properly carving out the back projection plane.

In such a method, since the back projection planes P1, P2 ... Are always arranged orthogonal to the virtual line of sight P, the positions of the back projection planes P1, P2 ... change depending on the position of the virtual viewpoint P. do. The format recording unit 202 starts recording the format in response to the recording start request RQ2 of the replay video from the viewing terminal 3. At this time, as in the first embodiment, the free viewpoint video ID and the total number of frames are recorded in the header, and the viewpoint information is also recorded for each frame by the same method as in the first embodiment.

The spatial information recording unit 202b records only the index of the surface on which the model is generated as spatial information among a large number of back projection surfaces P1, P2 ... That is, in this embodiment, the index of the surface on which the model is not generated is not recorded. For example, in the example shown in FIG. 8, a cylindrical object exists in space, but the model is generated only in P2, P3, and P4. Therefore, as shown in FIG. 9, only "2 3 4" is recorded as the index.

For example, in a free-viewpoint image in which players are discretely present in a part of a wide field such as soccer, if a back projection plane is arranged over the entire field, many useless back projection planes that do not generate a model occur. It is useless to recalculate the various aspects when playing the replay video.

On the other hand, in the present embodiment, since the back projection plane in which the model is generated and the back projection plane in which the model is not generated can be distinguished in advance, efficient memory can be secured, and the back projection plane in which the model is not generated can be masked. Since the calculation of back-projecting the image is not required, the calculation load is reduced.

In particular, in the full model method of Patent Document 1 adopted in this embodiment, it is mentioned in Patent Document 1 that parallel calculation is performed using a GPU, and reducing the number of back projection planes leads to memory saving. .. As a result, the time required to access the memory can be reduced, so that it is possible to save computational resources and speed up the calculation.

The spatial information recording unit 202b records the index of the projection plane that needs to be calculated when reproducing the 3D space, thereby speeding up the calculation and saving computational resources. As for the index attached to the back-projection surface, if there are 1000 back-projection surfaces facing the viewpoint, a method of allocating the index in order from the one closest to the viewpoint, such as 1 to 1000, can be considered. In the example of the format shown in FIG. 9, "2 3 4" is recorded as an index representing the three back projection planes P2, P3, and P4 from which the model is generated. The replay format recorded in this way is transferred to the viewing terminal 3 and stored in the same manner as in the first embodiment, and is later referred to at the time of replay.

In the viewing terminal 3, the replay video reproduction unit 304 reconstructs the replay video based on the stored replay format. At this time, the viewpoint is determined based on the viewpoint information of the replay video format, and the back projection plane is arranged based on this viewpoint, but the back projection plane from which the model is not generated is identified by referring to the index, and the reverse projection plane is identified. No placement or calculation is performed on the projection plane.

As a result, it is possible to reproduce the 3D space by performing calculations only on the back projection plane where the model is promised to be generated. After that, the image from the viewpoint is rendered in this 3D space, and the rendered image is transmitted to the viewing terminal 3 to realize the reproduction of the replay image.

In this embodiment, rendering is performed by the free viewpoint video generation device 2, but for example, all the moving images for constituting the free viewpoint video are distributed in advance to the viewing terminal 3 and the rendering is performed on the terminal side. Is also good. Even in this case, if the index of the back projection plane where the model does not occur is recorded in advance, it is possible to speed up the calculation and save the calculation resources.

Further, in such a configuration, the free viewpoint video generation device 2 is provided with a replay video reproduction function, and the 3D space can be completely reconstructed. Therefore, the replay video playback function receives information about the specifications of the viewing terminal 3, and renders the rendered image so that the visible field of view and the aspect ratio of the image change according to the resolution and screen size of the playback device, although the viewpoint is the same. It may have a function to output.

In addition, when a plurality of viewing terminals 3 request the free viewpoint video generation device 2 to play the replay video at the same time, and when there is a request to render a frame at the same viewpoint position and at the same time, rendering is performed. It may be equipped with a mechanism for storing and reusing the results.

1 ... Content server, 2 ... Free viewpoint video generator, 3 ... Viewing terminal, 201 ... Free viewpoint video generation unit, 202 ... Format recording unit, 202a ... Free viewpoint information recording unit, 202b ... Spatial information recording unit, 301 ... Storage , 302 ... User operation detection unit, 303 ... Rendering information acquisition unit, 304 ... Replay video playback unit

Claims (14)

In a system that reproduces a replay image of a free-viewpoint video by connecting a viewing terminal and a free-viewpoint video generator via a network.
The viewing terminal is
A means to request the reproduction of free-viewpoint video,
Equipped with a means to request the recording of the replay video for the free viewpoint video being played.
The free viewpoint video generator
A means for generating a free viewpoint image based on a plurality of camera images and viewpoint information of a virtual viewpoint in response to the reproduction request.
In response to the recording request, a means for recording a replay format in which a virtual viewpoint is described for each playback time of the replay video in the process of generating the free viewpoint video.
It is equipped with a means to transfer the recorded replay format to the viewing terminal.
The viewing terminal further
A means for acquiring information necessary for replay based on the replay format, and
It is provided with a means for performing rendering based on the replay format and the acquired information and reproducing the replay video.
A system for reproducing a replay image of a free-viewpoint image, characterized in that the information required for the replay is a background 3D model and spatial information for rendering an object on the background 3D model. The replay format contains header information and time series information.
The time-series information is configured by concatenating a plurality of time-series information recorded in a predetermined cycle.
The system for reproducing a replay image of a free-viewpoint image according to claim 1, wherein a reproduction time ID unique to a reproduction position of the replay image based on the time-series information is recorded in each time-series information. The generation process includes the generation process of the background 3D model of the free viewpoint image.
The ID of the free viewpoint video is recorded in the replay format, and the ID is recorded.
The system for reproducing the replay image of the free viewpoint image according to claim 1 or 2, wherein the acquisition means acquires the background 3D model based on the ID of the free viewpoint image. The generation process includes a viewpoint information generation process.
The system for reproducing a replay image of a free viewpoint image according to any one of claims 1 to 3, wherein the viewpoint information is recorded in the replay format. The system for reproducing a replay image of a free-viewpoint image according to any one of claims 1 to 4, wherein the means for generating the free-viewpoint image adopts a billboard-type free-viewpoint technique. The generation process includes a process of identifying an object that can be seen from a virtual viewpoint.
The general ID associated with the visible object is recorded in the replay format, and
The system for reproducing a replay image of a free-viewpoint image according to claim 5, wherein the acquisition means acquires rendering information of each object based on the general ID. The generation process includes a process of generating a mask image of each object.
The mask image is associated with the general ID,
The system for reproducing a replay image of a free viewpoint image according to claim 6, wherein the acquisition means acquires a mask image of each object based on the general ID. The generation process includes the process of extracting the texture of each object from the camera image.
The texture is associated with the general ID,
The system for reproducing a replay image of a free-viewpoint image according to claim 6, wherein the acquisition means acquires the texture of each object based on the general ID. Reproduce the replayed image of the free-viewpoint image according to any one of claims 1 to 4, wherein the means for generating the free-viewpoint image adopts a full-model type free-viewpoint technique using a back projection surface. System to do. The generation process includes a process of arranging a plurality of back projection planes facing the virtual viewpoint at the position of the object, projecting a mask image of the object, performing 3D modeling for each back projection plane, and restoring the 3D model. ,
In the replay format, the index of the back projection plane where the 3D model exists is recorded.
The system for reproducing a replay image of a free viewpoint image according to claim 9, wherein the acquisition means acquires information necessary for replay based on the index of the back projection surface. The means for generating the free viewpoint video and the means for recording the replay format are implemented in the server on the cloud.
The means for acquiring the information necessary for the replay and the means for playing the replay video are implemented in the viewing terminal of the replay video.
The system for reproducing a replay image of a free viewpoint image according to any one of claims 1 to 10, wherein the replay format is transferred from a server to a viewing terminal and stored on the viewing terminal. A means for generating the free viewpoint video, a means for recording the format for replay, and a means for acquiring the information necessary for the replay are implemented in a server on the cloud.
The means for reproducing the replay video is implemented in the viewing terminal,
The system for reproducing a replay image of a free viewpoint image according to any one of claims 1 to 10, wherein the replay format is transferred from a server to a viewing terminal and stored on the viewing terminal. 11. A system for reproducing a replay image of the free viewpoint image according to 12. In a method of playing a replay video of a free viewpoint video by connecting a viewing terminal and a free viewpoint video generator via a network.
The viewing terminal is
Requesting the reproduction of free-viewpoint video,
Requested recording of replay video for free-viewpoint video being played,
The free viewpoint video generator
In response to the playback request, a free viewpoint image is generated based on the viewpoint information of a plurality of camera images and a virtual viewpoint.
In response to the recording request, in the process of generating the free viewpoint video, a replay format in which the virtual viewpoint is described is recorded for each playback time of the replay video.
Transfer the recorded replay format to the viewing terminal and
The viewing terminal further
Obtain the information required for replay based on the replay format, and
Rendering is executed based on the replay format and the acquired information, and the replay video is played back.
A method of reproducing a replay image of a free viewpoint image, characterized in that the information required for the replay is a background 3D model and spatial information for rendering an object on the background 3D model.

Images