WO2022124047A1

WO2022124047A1 - Information processing device, information processing system, and information processing method

Info

Publication number: WO2022124047A1
Application number: PCT/JP2021/042524
Authority: WO
Inventors: 健一郎細川
Original assignee: ソニーグループ株式会社
Priority date: 2020-12-07
Filing date: 2021-11-19
Publication date: 2022-06-16

Abstract

An information processing device (100) is provided with a generation unit (120) and a transmission unit (140). The generation unit (120) acquires a first rendering image viewed from each of a first number of viewpoint positions and visual line directions, and rendering information used in rendering the first rendering image. The generation unit (120) generates visual line information relating to a second number of viewpoint positions and visual line directions, the second number being greater than the first number . The generation unit (120) generates a second rendering image viewed from each of the second number of viewpoint positions and visual line directions, on the basis of the first rendering image and the rendering information. The transmission unit (140) transmits, to a terminal device (200), the visual line information and the second rendering image.

Description

Information processing equipment, information processing system and information processing method

This disclosure relates to an information processing device, an information processing system, and an information processing method.

Conventionally, there is a technique of presenting an image rendered by a rendering device such as a game image or a 3D image to a user via a network. For example, in game streaming, a game image of a viewpoint corresponding to a user's operation is rendered by a rendering device and presented to the user. Further, there is known a technique of displaying an image rendered by a rendering device using AR (Augment Reality) or VR (Virtual Reality) on a head-mounted display worn by a user, for example.

When the image rendered by the rendering device is presented to the user via the network, for example, the rendered image may be delayed and presented to the user depending on the traffic situation of the network. Therefore, there is known a technique of predicting the posture of the user after the lapse of delay from the posture of the user and rendering an image based on the predicted posture. In such a technique, the rendered image is corrected according to the difference between the predicted posture and the actual posture and presented to the user.

Japanese Unexamined Patent Publication No. 2007-79664

In the above technique, for example, when the user moves a lot, if the difference between the predicted posture and the actual posture is large, the correction amount becomes large, so that the image quality of the rendered image deteriorates and the quality of the provided service may deteriorate. There is.

Therefore, in this disclosure, we propose an information processing device, an information processing system, and an information processing method that can further suppress the deterioration of the quality of the service provided to the user.

It should be noted that the above-mentioned problem or purpose is only one of a plurality of problems or purposes that can be solved or achieved by the plurality of embodiments disclosed in the present specification.

According to the present disclosure, an information processing device is provided. The information processing device includes a generation unit and a transmission unit. The generation unit acquires the first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image. The generation unit generates line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number. The generation unit generates the second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number based on the first rendered image and the rendered information. The transmission unit transmits the line-of-sight information and the second rendered image to the terminal device.

It is a figure for demonstrating the outline of the proposed technique of this disclosure. It is a figure which shows an example of the structure of the information processing system which concerns on embodiment of this disclosure. It is a block diagram which shows the structural example of the multi-viewpoint conversion part which concerns on embodiment of this disclosure. It is a figure for demonstrating an example of viewpoint information determined by the viewpoint buffer control unit which concerns on embodiment of this disclosure. It is a figure for demonstrating an example of viewpoint information determined by the viewpoint buffer control unit which concerns on embodiment of this disclosure. It is a figure for demonstrating an example of viewpoint information determined by the viewpoint buffer control unit which concerns on embodiment of this disclosure. It is a figure for demonstrating an example of viewpoint information determined by the viewpoint buffer control unit which concerns on embodiment of this disclosure. It is a figure for demonstrating an example of viewpoint information which concerns on embodiment of this disclosure. It is a figure which shows an example of the conversion viewpoint information determined by the viewpoint buffer control unit which concerns on embodiment of this disclosure. It is a block diagram which shows the structural example of the viewpoint buffer control part which concerns on embodiment of this disclosure. It is a block diagram which shows the structural example of the decoding part which concerns on embodiment of this disclosure. It is a figure for demonstrating the decoding viewpoint information determined by the decoding control unit which concerns on embodiment of this disclosure. It is a flowchart which shows the flow of the multi-viewpoint conversion process executed by the server apparatus which concerns on embodiment of this disclosure. It is a flowchart which shows the flow of the display image generation processing executed by the client apparatus which concerns on embodiment of this disclosure.

The embodiments of the present disclosure will be described in detail with reference to the accompanying drawings below. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

Further, in the present specification and the drawings, similar components of the embodiment may be distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to distinguish each of the similar components, only the same reference numerals are given.

Each of one or more embodiments (including examples and modifications) described below can be implemented independently. On the other hand, at least a part of the plurality of embodiments described below may be carried out in combination with at least a part of other embodiments as appropriate. These plurality of embodiments may contain novel features that differ from each other. Therefore, these plurality of embodiments may contribute to solving different purposes or problems, and may have different effects.

The explanations will be given in the following order.
1. 1. Introduction 1.1. Outline of the proposed technology 2. Configuration example of information processing system 2.1. Information processing system 2.2. Server device 2.3. Client device 3. Information processing 3.1. Multi-viewpoint conversion process 3.2. Server device 4. Other embodiments 5. summary

<< 1. Introduction >>
<1.1. Outline of proposed technology ＞
First, the outline of the proposed technique according to the present disclosure will be described. FIG. 1 is a diagram for explaining an outline of the proposed technique of the present disclosure. The proposed technique according to the present disclosure is implemented in the information processing system shown in FIG. As shown in FIG. 1, the information processing system 10 includes a server device 100 and a client device 200. The server device 100 and the client device 200 are connected via a network N such as the Internet.

The server device 100 is, for example, a rendering server, and transmits a rendered image (hereinafter, also referred to as a rendered image) to the client device 200. For example, the server device 100 is a game server, and provides a game service to the client device 200 by transmitting a rendered image which is a game image to the client device 200. Alternatively, the server device 100 may be a device that provides AR or VR to the user via the client device 200. In this case, the server device 100 generates, for example, a rendered image according to the user's viewpoint and transmits it to the client device 200.

The client device 200 is an information processing device such as a smartphone, a PC, or a game device. Further, the client device 200 may be an information processing device worn on the user's head, such as a head-mounted display (HMD: Head Mounted Display) or smart glasses. The client device 200 displays the rendered image acquired from the server device 100 on the display unit of the own device or the display device connected to the own device.

When the server device 100 transmits a rendered image to the client device 200 via the network N, the display of the rendered image on the client device 200 may be delayed due to the communication delay generated in the network N. For example, in the case of game streaming in which the information processing system 10 provides a game service, such a delay may hinder a good game experience given to the user. In particular, in games involving viewpoint movement such as game streaming in which the user controls the virtual camera and game streaming using AR and VR technologies, the user's viewpoint moves while the communication delay occurs. , There is a possibility that a large difference may occur between the viewpoint of the displayed image and the viewpoint of the actual user. In VR, the user may feel VR sickness due to the deviation between the viewpoint of the displayed image and the viewpoint of the user.

As described above, for example, it takes a transmission time to transmit the image rendered by the server device 100 arranged on the cloud to the client device 200. In particular, when the network N is a general Internet line, the transmission time depends on the traffic of the Internet line. Therefore, depending on the amount of traffic on the Internet line, the quality of services provided to users such as games may be significantly deteriorated.

Therefore, in the technique of the present disclosure, the server device 100 converts the rendered image into a multi-viewpoint image, and transmits the converted multi-viewpoint image to the client device 200. At this time, the server device 100 generates a multi-viewpoint image with high accuracy by using the information used when generating the rendered image (hereinafter, also referred to as rendering incidental information).

The client device 200 generates a display image based on the received multi-viewpoint image according to the current user's viewpoint or the operation status of the game by the user, and presents the display image to the user.

Here, with reference to FIG. 1, an outline of information processing related to the technique of the present disclosure will be described.

First, the server device 100 acquires the rendered image M0 and the rendering incidental information (step S1). Here, the rendered image is an image generated by the rendering unit (not shown) of the server device 100 based on the operation status of the game and the viewpoint information of the user transmitted from the client device 200. Further, the rendering incidental information is information used by the rendering unit to generate a rendered image, and includes, for example, depth information. The rendering unit uses the rendering incidental information to generate a rendered image based on a predetermined number of viewpoint positions and line-of-sight directions.

In the example of FIG. 1, the server device 100 acquires a rendered image M0 viewed from one viewpoint position and line-of-sight direction from a plurality of images in a viewpoint position and a line-of-sight direction that can be generated by the rendering unit. In FIG. 1, a plurality of rendered images that can be generated by the rendering unit are arranged and shown in a matrix. Among the illustrated matrix-shaped rendered images, the rendered image with hatching is an image that is not actually generated by the rendering unit, and the rendering unit generates the rendered image M0 without hatching.

Next, the server device 100 generates the viewpoint buffer information (step S2). The viewpoint buffer information is information used to generate a multi-viewpoint image, and includes a plurality of viewpoint positions and line-of-sight directions. The server device 100 generates viewpoint buffer information based on a game status, a communication delay status (communication status), and the like.

The server device 100 generates a viewpoint buffer using the viewpoint buffer information and the rendering incidental information based on the rendered image (step S3). The viewpoint buffer includes a plurality of viewpoint positions included in the viewpoint buffer information and a plurality of images viewed from the line-of-sight direction (hereinafter, also referred to as multi-viewpoint images).

FIG. 1 shows a plurality of images that can be viewpoint-converted by the server device 100 arranged in a matrix. Among the illustrated matrix-shaped images, the hatched images are images that are not actually converted by the server device 100, and the server device 100 generates the unhatched images M1 to M5 by the viewpoint conversion.

The server device 100 can generate a highly accurate multi-viewpoint image by using the rendering incidental information.

The server device 100 transmits the viewpoint buffer information and the viewpoint buffer to the client device 200 via the network N (step S4).

The client device 200 receives the viewpoint buffer information and the viewpoint buffer from the server device 100 (step S5).

The client device 200 generates a display image from the multi-viewpoint image included in the viewpoint buffer based on the current viewpoint of the user (step S6), and presents the display image to the user.

As described above, in the information processing according to the embodiment of the present disclosure, the server device 100 transmits a plurality of multi-viewpoint images generated from the rendered image to the client device 200. At this time, the server device 100 can generate a high-quality multi-viewpoint image by using the rendering incidental information. Further, the client device 200 generates a display image according to the viewpoint of the current user from a plurality of multi-viewpoint images. As a result, the information processing system 10 can present the user with a high-quality display image according to the viewpoint of the current user even if a communication delay occurs in the network N, and the deterioration of the provided service due to the communication delay is deteriorated. It can be suppressed.

Here, as an application example of the technique of the present disclosure, a case where the information system 10 provides a game service will be described, but the service provided by the information system 10 is not limited to the game service. For example, the information processing system 10 may be an AR system or a VR system. As described above, the information processing system 10 may be any system that provides the user with an image corresponding to the movement of the viewpoint, and is not limited to the system that provides the game service.

<< 2. Information processing system configuration example >>
<2.1. Information processing system ＞
FIG. 2 is a diagram showing an example of the configuration of the information processing system 10 according to the embodiment of the present disclosure. The information processing system includes a server device 100, a client device 200, and a peripheral device 300.

In the information system according to the embodiment of the present disclosure, the server device 100 distributes a rendered game image (for example, a rendered image). The client device 200 receives the video of the game through the network N, which is a communication line such as the Internet, and reproduces it, for example, on the display unit (not shown) of the peripheral device 300. The peripheral device 300 includes, for example, a display, an HMD, a controller, and the like.

The client device 200 detects, for example, command information an operation or operation performed by the user in response to the reproduced video. When the user operates the controller, the client device 200 detects the operation on the controller as command information. Further, when the user wears the HMD, for example, the client device 200 detects the movement of the user's head as the movement of the HMD. The client device 200 transmits the detected command information to the server device 100 via the network N.

The server device 100 renders an image of the game at the next time according to the received command information and distributes it to the client device 200.

[Server device 100]
As shown in FIG. 2, the server device 100 is an information processing device that provides an application function to the client device 200. The server device 100 is, for example, a game server (Game Server) that provides a game service as an application function, and transmits a rendered image to be displayed as a game screen to the client device 200. Further, the server device 100 receives the command information from the client device 200. Such command information includes information that can identify the user's viewpoint, such as information on the movement of the user's head, and information on operations by the user with respect to the controller.

The server device 100 includes a rendering unit 110, a multi-viewpoint conversion unit 120, an encoding unit 130, a transmission / reception unit 140, and a command information acquisition unit 150.

(Rendering unit 110)
The rendering unit 110 renders the game image. The rendering unit 110 determines, for example, the camera position in the game (for example, the user's viewpoint position), the progress of the game, and the like based on the command information acquired from the client device 200, and produces a game image according to the determination result. Generate. As described above, the rendering unit 110 is an application that has a Game Logic function for controlling the progress of the game and provides a game service. The rendering unit 110 generates n1 viewpoint positions and line-of-sight directions (hereinafter, also simply referred to as viewpoints or viewpoint information) rendered images.

Generally, the rendering unit 110 generates a small number of rendered images such as one viewpoint (n1 = 1) due to restrictions such as the amount of calculation. Hereinafter, in order to simplify the explanation, the rendering unit 110 shall generate a rendered image viewed from one viewpoint. In other words, the rendering unit 110 generates a rendered image using the rendering incidental information based on the command information. The rendering incidental information includes, for example, depth information and normal information of objects included in the game. The details of the rendering incidental information will be described later. Further, the time when the rendering unit 110 generates the rendered image is referred to as a rendering time t1.

The rendering unit 110 outputs the rendered image, the rendering incidental information, and the information about the game to the multi-viewpoint conversion unit 120. Here, the information about the game includes, for example, information about the title of the game, information about the type of the game such as a race game, a simulation game, and an RPG game, and information such as a game play status.

(Multi-viewpoint conversion unit 120)
The multi-viewpoint conversion unit 120 generates a viewpoint buffer including a plurality of viewpoint conversion images from the rendered image generated by the rendering unit 110. In addition, a plurality of viewpoint conversion images are collectively referred to as a multi-viewpoint image.

The multi-viewpoint conversion unit 120 acquires information on the network traffic (delay) status from the transmission / reception unit 140 in addition to the rendered image acquired from the rendering unit 110, rendering incidental information, and information on the game. The multi-viewpoint conversion unit 120 generates viewpoint buffer information from the acquired information and converts the rendered image into a multi-viewpoint image viewed from n2 viewpoints. It is assumed that the number of viewpoints n2 of the multi-viewpoint image is larger than the number of viewpoints n1 of the rendered image (n1 <n2). Further, the viewpoint buffer information includes the coordinate information of the viewpoint of the multi-view image included in the viewpoint buffer. The details of the viewpoint buffer and the viewpoint buffer information generated by the multi-viewpoint conversion unit 120 will be described later.

The multi-viewpoint conversion unit 120 outputs the generated viewpoint buffer and viewpoint buffer information to the encoding unit 130.

Here, there is a corresponding delay depending on the traffic of the network N until the rendered image is reproduced by the user via the network N. Ideally, it is desired that the command information input to the client device 200 by the user from moment to moment is immediately reflected, and the game image is rendered and presented to the user from an appropriate viewpoint. However, due to the delay described above, GAP is generated between the rendering time t1 at which the rendered image is generated and the time at which the command information is input (hereinafter, also referred to as command generation time t2). Therefore, the viewpoint may be different between the image expected by the user and the image rendered by the server device 100.

Therefore, the multi-viewpoint conversion unit 120 generates in advance an image viewed from a viewpoint around the viewpoint of the rendered image (hereinafter, also referred to as a rendering viewpoint) as a multi-viewpoint image based on the rendered image rendered by the rendering unit 110. In other words, the multi-viewpoint conversion unit 120 builds a buffer for images of a plurality of viewpoints as a viewpoint buffer in advance.

The server device 100 transmits a game image having such a viewpoint buffer to the client device 200. As described above, it is assumed that the GAP between the rendering time t1 and the command time t2 causes a deviation between the viewpoint of the game image and the viewpoint of the image expected by the user.

Even in this case, if the deviation is within the range of the viewpoint buffer, the client device 200 can generate a game image in which the deviation is interpolated using the viewpoint buffer.

Further, even if the deviation is large and deviates from the range of the viewpoint buffer, the client device 200 is based on the viewpoint conversion image of the viewpoint included in the viewpoint buffer and close to the viewpoint of the image expected by the user. A game image can be generated by performing viewpoint conversion. In this way, even if the deviation is large, by generating the game image from the viewpoint conversion image included in the viewpoint buffer, the deviation of the viewpoint can be reduced as compared with the case where the game image is generated from the rendered image. Deterioration of the quality of the game screen can be further suppressed.

(Encoding unit 130)
The encoding unit 130 compresses the viewpoint buffer and the viewpoint buffer information generated by the multi-viewpoint conversion unit 120 into a small capacity, and outputs the compressed bit stream (hereinafter, also referred to as compressed data) to the transmission / reception unit 140. do.

(Transmission / reception unit 140)
The transmission / reception unit 140 is a communication interface (I / F) that communicates with an external device. The transmission / reception unit 140 is realized by, for example, a NIC (Network Interface Card) or the like. For example, the transmission / reception unit 140 converts the compressed data encoded by the encoding unit 130 into a transmission signal and transmits it to the client device 200. Further, the transmission / reception unit 140 receives command information from the client device 200 and notifies the viewpoint information acquisition unit 150 of the reception result.

(Command information acquisition unit 150)
The command information acquisition unit 150 acquires command information via the transmission / reception unit 140. The command information acquisition unit 150 outputs the acquired command information to the rendering unit 110.

[Client device 200]
The client device 200 is an information processing device that receives application functions from the server device 100. The client device 200 receives, for example, a game service as an application function.

The client device 200 receives an image (viewpoint buffer) related to the game transmitted from the server device 100, and displays the game image based on the received viewpoint buffer. The client device 200 transmits command information according to the operation or operation of the game by the user to the server device 100.

The client device 200 has a transmission / reception unit 210, a decoding unit 220, and an input information acquisition unit 230.

(Transmission / reception unit 210)
The transmission / reception unit 210 is a communication interface (I / F) that communicates with an external device. The transmission / reception unit 210 is realized by, for example, a NIC (Network Interface Card) or the like. For example, the transmission / reception unit 210 receives a transmission signal from the server device 100 as a reception signal, and converts the received reception signal into compressed data. Further, the transmission / reception unit 210 transmits command information to the server device 100.

(Decoding unit 220)
The decoding unit 220 decodes the compressed data received by the transmission / reception unit 210. The decoding unit 220 generates a game image to be presented to the user from the multi-viewpoint image included in the viewpoint buffer based on the user's viewpoint information and the viewpoint buffer information acquired by the input information acquisition unit 230. The user's viewpoint information acquired by the input information acquisition unit 230 is the latest viewpoint information in the current game, that is, information regarding the viewpoint of the game image expected by the user, and is simply "current". Also referred to as "user's viewpoint information".

The decoding unit 220 displays the generated game image on the display unit (not shown) of the peripheral device 300.

(Input information acquisition unit 230)
The input information acquisition unit 230 is a user interface (UI: User Interface) that accepts operations of the peripheral device 300 by the user. The input information acquisition unit 230 converts the operation or operation performed by the user in response to the game image displayed on the peripheral device 300 into command information that can be interpreted by the server device 100. Further, the input information acquisition unit 230 generates viewpoint information (viewpoint information of the current user) of the game image to be displayed on the peripheral device 300. The input information acquisition unit 230 outputs the generated viewpoint information to the decoding unit 220. Further, the input information acquisition unit 230 includes the viewpoint information in the command information and transmits the viewpoint information to the server device 100 via the transmission / reception unit 210.

[Peripheral device 300]
The peripheral device 300 is a device such as a display, smart glasses, HMD, controller, etc. that displays a game image output by the client device 200 and accepts operations by the user. When the peripheral device 300 is a display, the peripheral device 300 presents a game image to the user. When the peripheral device 300 is a controller, the peripheral device 300 accepts an operation performed by the user in response to a game image displayed on a display device (not shown). When the peripheral device 300 is an HMD, the peripheral device 300 displays a game image on a display unit (not shown) and detects the movement of the user's head as the movement of the HMD.

Here, it is assumed that the peripheral device 300 is a device different from the client device 200, but the present invention is not limited to this. For example, the peripheral device 300 may be realized as a function of the client device 200.

As described above, in the embodiment of the present disclosure, the server device 100 uses the rendering incidental information to generate the viewpoint buffer. Further, the client device 200 generates a game image (display image) according to the viewpoint information of the current user based on the viewpoint buffer. As a result, the information processing system 10 can display the game image as seen from the viewpoint of the current user regardless of the delay of the network N.

In this way, in the information processing system 10, the viewpoint buffer is generated by the server device 100. In other words, the server device 100 uses the rendering incidental information to perform viewpoint conversion of the rendered image. As described above, it is possible to obtain a higher image quality viewpoint conversion image by performing the viewpoint conversion of the rendered image using the rendering incidental information than by performing the viewpoint conversion only from the rendered image.

Here, it is conceivable that the client device 200 performs viewpoint conversion of the rendered image based on the viewpoint information of the current user.

However, in this case, it is difficult for the client device 200 to perform viewpoint conversion using rendering incidental information. This is because the rendering incidental information is inferior in coding efficiency as compared with the multi-viewpoint image, so that the communication amount becomes large. As will be described later, the rendering incidental information includes not only image information but also various information such as information about objects in the game. Therefore, the rendering incidental information has a lower coding efficiency than the viewpoint buffer composed of image information having different viewpoints.

Therefore, the information processing system 10 according to the embodiment of the present disclosure further reduces the image delay due to the viewpoint movement while suppressing the increase in network traffic, as compared with the case where the viewpoint conversion of the rendered image is performed by the client device 200. be able to. As described above, the information processing system 10 according to the embodiment of the present disclosure can provide a higher quality service.

<2.2. Server device>
Next, the details of each part of the server device 100 will be described.

[Multi-viewpoint conversion unit 120]
FIG. 3 is a block diagram showing a configuration example of the multi-viewpoint conversion unit 120 according to the embodiment of the present disclosure. As shown in FIG. 3, the multi-viewpoint conversion unit 120 includes a viewpoint buffer control unit 121 and a multi-viewpoint generation unit 122.

(Viewpoint buffer control unit 121)
The viewpoint buffer control unit 121 determines the viewpoint information of the multi-view image generated by the multi-view generation unit 122. The viewpoint buffer control unit 121 is based on at least one of the viewpoint information of the rendered image generated by the rendering unit 110, the information about the game such as the game title and the play status, and the traffic information (communication status) of the network N. To decide.

An example of viewpoint information determined by the viewpoint buffer control unit 121 will be described with reference to FIGS. 4 to 7. 4 to 7 are diagrams for explaining an example of viewpoint information determined by the viewpoint buffer control unit 121 according to the embodiment of the present disclosure.

Note that FIGS. 4 to 7 show a plurality of rendered images that can be generated by the rendering unit 110 and examples of the multi-viewpoint images that can be generated by the multi-viewpoint generation unit 122 side by side in a matrix.

The viewpoint buffer control unit 121 determines the viewpoint information of the multi-view image generated by the multi-view generation unit 122 from a predetermined range including the viewpoint information of the rendered image generated by the rendering unit 110 (hereinafter, also referred to as rendering viewpoint information). do.

In the example of FIG. 4, the viewpoint buffer control unit 121 determines the viewpoint information indicated by a cross from the circular range centered on the rendering viewpoint information as the viewpoint information of the viewpoint converted image (hereinafter, also referred to as converted viewpoint information). do. FIG. 4 shows an example in which the viewpoint buffer control unit 121 determines 12 conversion viewpoint information.

The viewpoint buffer control unit 121 determines the conversion viewpoint information according to the situation of the game or the network N.

For example, the viewpoint buffer control unit 121 determines the conversion viewpoint information according to the delay status (communication status) of the network N. More specifically, the viewpoint buffer control unit 121 determines the conversion viewpoint information from a larger predetermined range as the delay of the network N increases.

When the delay of the network N is small, the viewpoint buffer control unit 121 determines, for example, 12 conversion viewpoint information from the range R1 shown in FIG. Further, when the delay of the network N is large, as shown in FIG. 5, for example, 12 conversion viewpoint information is determined from the range R2 wider than the range R1 (see FIG. 4).

In this way, the viewpoint buffer control unit 121 determines the conversion viewpoint information according to the delay of the network N. As a result, even when the delay of the network N is large, the game image seen from the viewpoint of the current user can be generated on the client device 200 side, and the deterioration of the quality of the service provided to the user can be further suppressed.

In FIGS. 4 and 5, the viewpoint buffer control unit 121 evenly determines the conversion viewpoint information from the ranges R1 and R2. Therefore, the viewpoint buffer control unit 121 determines the conversion viewpoint information more densely as the delay of the network N is smaller, and determines the conversion viewpoint information sparsely as the delay is larger.

Here, the viewpoint buffer control unit 121 determines the conversion viewpoint information according to the traffic (delay) status of the network N, but the present invention is not limited to this. The viewpoint buffer control unit 121 may determine the conversion viewpoint information according to the information about the game such as the game title and the play situation (scene).

For example, the viewpoint buffer control unit 121 can determine the conversion viewpoint information according to the game title and the type of the game. For example, in the case of a game in which the viewpoint moves quickly, such as a racing game, the viewpoint buffer control unit 121 can determine the converted viewpoint information from a larger range. On the other hand, for example, in the case of a game in which the viewpoint movement is small, the viewpoint buffer control unit 121 can determine the converted viewpoint information from a smaller range.

Alternatively, the viewpoint buffer control unit 121 may determine the conversion viewpoint information according to the play situation. For example, the viewpoint buffer control unit 121 determines the conversion viewpoint information from a larger range when playing in a space with a wide movement range such as outdoors, and when playing in a space with a narrow movement range such as indoors. The conversion viewpoint information may be determined from a narrower range.

In this way, the viewpoint buffer control unit 121 can determine the conversion viewpoint information according to the information about the game such as the game title and the play status, in other words, the content including the rendered image and the scene of the rendered image. In particular, the viewpoint buffer control unit 121 can determine the converted viewpoint information according to the information related to the speed of the viewpoint movement among the information related to the game.

Further, in the above example, the viewpoint buffer control unit 121 evenly determines the conversion viewpoint information from the predetermined range, but the present invention is not limited to this. For example, as shown in FIG. 6, the viewpoint buffer control unit 121 is densely located in the vicinity of the rendering viewpoint information (center of the range R3) in the circular range R3 centered on the rendering viewpoint information, and around the range R3. The conversion viewpoint information may be determined so as to be sparse. For example, when the server device 100 provides a game service with less viewpoint movement and the delay of the network N is large, the viewpoint buffer control unit 121 can determine the conversion viewpoint information at the density shown in FIG.

Further, the viewpoint buffer control unit 121 may determine the conversion viewpoint information according to the rendering viewpoint information, particularly the history of the rendering viewpoint information. In this case, the viewpoint buffer control unit 121 predicts the viewpoint information of the future time (for example, the command generation time t2) from the change (history) of the past rendering viewpoint information. The viewpoint buffer control unit 121 determines the converted viewpoint information from the range including the predicted viewpoint information.

In the example of FIG. 7, the viewpoint buffer control unit 121 predicts the viewpoint information in a predetermined direction (P1 in FIG. 7), and determines the conversion viewpoint information from the long range R4 in the predicted direction (upper right direction in FIG. 7).

The number of converted viewpoint information (number of viewpoints) determined by the viewpoint buffer control unit 121 may be constant or variable. For example, when the server device 100 is constructed in a cloud system and the calculation resources of the server device 100 can be flexibly increased or decreased, the server device 100 may change the number of conversion viewpoint information according to the situation. This makes it possible to improve the efficiency of the calculation cost of the viewpoint conversion by the server device 100. Further, when it is desired to keep the amount of calculation constant due to restrictions such as a hardware solution, for example, when the server device 100 is constructed with specific hardware, the server device 100 determines a certain number of conversion viewpoint information. May be good.

Next, the conversion viewpoint information determined by the viewpoint buffer control unit 121 will be described. First, an example of viewpoint information according to the embodiment of the present disclosure will be described with reference to FIG. FIG. 8 is a diagram for explaining an example of viewpoint information according to the embodiment of the present disclosure.

As described above, the viewpoint information includes the viewpoint position and the line-of-sight direction. As shown in FIG. 8, the viewpoint position is represented by the viewpoint (camera) coordinates (x _wk , y _wk , z _wk ) representing the position of the camera (or the user's head) that saw the image to be rendered. The coordinate system of the viewpoint coordinates is, for example, a world coordinate system in which game objects are arranged. Further, the line-of-sight direction is represented by a line-of-sight unit vector (x _vk , y _vk , x _vk ) indicating the direction of the camera (or the user's head).

The viewpoint buffer control unit 121 determines a plurality of sets of the viewpoint position (viewpoint coordinates) and the line-of-sight direction (line-of-sight unit vector) as a plurality of converted viewpoint information. FIG. 9 is a diagram showing an example of conversion viewpoint information determined by the viewpoint buffer control unit 121 according to the embodiment of the present disclosure. FIG. 9 shows a case where the viewpoint buffer control unit 121 determines N conversion viewpoint information.

As shown in FIG. 9, the viewpoint buffer control unit 121 sets {(x _w1 , y _w1 , z _w1 ), (x _v1 , y _v1 , x _v1 )}, {(x w1, y w1, z w1)}, {(x w1, y w1, x v1)}, {(x w1, y w1, z w1)}, {(x w1, y v1, x v1)}, {(x w1, y w1, z w1), x _w2 , y _w2 , z _w2 ), (x _v2 , y _v2 , x _v2 )} ... {(x _w2 , y _w2 , z _w2 ), (x _v2 , y _v2 , x _v2 )} ..

The viewpoint buffer control unit 121 outputs the determined set of viewpoint coordinates and line-of-sight unit vector to the multi-viewpoint generation unit 122 (see FIG. 3).

Note that the conversion viewpoint information shown in FIGS. 8 and 9 is an example, and the present invention is not limited to this. The converted viewpoint information may be expressed in any format as long as the multi-viewpoint generation unit 122 can convert the viewpoint of the rendered image based on the converted viewpoint information.

Here, the viewpoint buffer control unit 121 may output, for example, the conversion viewpoint information stored in the storage unit (not shown) to the multi-viewpoint generation unit 122 as the determined conversion viewpoint information, or may be determined using a function. You may. Hereinafter, a case where the viewpoint buffer control unit 121 determines the conversion viewpoint information using a function will be described with reference to FIG. 10, but the method for determining the conversion viewpoint information is not limited to this.

FIG. 10 is a block diagram showing a configuration example of the viewpoint buffer control unit 121 according to the embodiment of the present disclosure. The viewpoint buffer control unit 121 includes a viewpoint function generation unit 1211, a line-of-sight function generation unit 1213, a sampling unit 1212, and a line-of-sight vector generation unit 1214.

The viewpoint buffer control unit 121 first determines a set of a viewpoint density function and a line-of-sight function in a three-dimensional space. The viewpoint buffer control unit 121 determines a predetermined viewpoint density function and line-of-sight function. Alternatively, the viewpoint buffer control unit 121 may determine the viewpoint density function and the line-of-sight function according to information about the game, information about network traffic, rendering viewpoint information, and the like.

More specifically, the viewpoint function generation unit 1211 determines, for example, an arbitrary probability density function P (x, y, z) as the viewpoint density function P. For example, the viewpoint function generation unit 1211 determines the viewpoint density function P according to information about the game, information about network traffic, rendering viewpoint information, etc. from a plurality of probability density functions stored in advance in the storage unit (not shown). do.

At this time, the viewpoint function generation unit 1211 may change the parameters of the viewpoint density function P according to the information about the game, the information about the network traffic, the rendering viewpoint information, and the like. The viewpoint function generation unit 1211 outputs the determined viewpoint density function P to the sampling unit 1212.

Further, the line-of-sight function generation unit 1213 determines a function that inputs an arbitrary viewpoint coordinate and outputs a line-of-sight unit vector as a line-of-sight function V. For example, the line-of-sight function generation unit 1213 determines the line-of-sight function V according to information about a game, information about network traffic, rendering viewpoint information, and the like from a plurality of line-of-sight functions stored in advance in a storage unit (not shown).

At this time, the line-of-sight function generation unit 1213 may change the parameters of the line-of-sight function V according to information about the game, information about network traffic, rendering viewpoint information, and the like. The line-of-sight function generation unit 1213 outputs the determined line-of-sight function V to the line-of-sight vector generation unit 1214.

The sampling unit 1212 determines, for example, N viewpoint coordinates to be used for the viewpoint conversion from the viewpoint density function P by using various sampling methods. Examples of the sampling method include an inverse function method, rejection sampling, and Markov chain Monte Carlo method. The sampling unit 1212 outputs the determined viewpoint coordinates to the line-of-sight vector generation unit 1214.

The line-of-sight vector generation unit 1214 determines the line-of-sight unit vector by inputting the viewpoint coordinates determined by the sampling unit 1212 into the line-of-sight function V. The line-of-sight vector generation unit 1214 outputs the determined N viewpoint coordinates and line-of-sight unit vectors to the multi-viewpoint generation unit 122 (see FIG. 3) as viewpoint buffer information.

Here, it is assumed that the line-of-sight vector generation unit 1214 outputs the viewpoint coordinates and the line-of-sight unit vector as the conversion viewpoint information, but the present invention is not limited to this. For example, the viewpoint buffer control unit 121 may output the viewpoint density function P and the line-of-sight function V to the multi-viewpoint generation unit 122 as viewpoint buffer information. Alternatively, the line-of-sight vector generation unit 1214 may determine the parameters of the predetermined viewpoint density function P and the predetermined line-of-sight function V, and output the determined parameters to the multi-viewpoint generation unit 122 as the viewpoint buffer information.

In this way, the viewpoint buffer control unit 121 may output the viewpoint density function P and the line-of-sight function V as the viewpoint buffer information, or information (for example, parameters, etc.) for uniquely determining the function. In this case, the sampling unit 1212 and the line-of-sight vector generation unit 1214 of the viewpoint buffer control unit 121 may be omitted. Further, in this case, it is assumed that the information on the sampling method by the sampling unit 1212 and the function to be used is shared by the multi-viewpoint generation unit 122, the decoding unit 220 of the client device 200 (see FIG. 2), and the like.

Here, it is assumed that the viewpoint buffer control unit 121 generates the viewpoint buffer information according to the communication status of the network N and the game information, but the present invention is not limited to this. For example, the multi-viewpoint conversion unit 120 may generate a viewpoint conversion image based on a predetermined constant viewpoint buffer information regardless of the above-mentioned information. In this case, the server device 100 may omit the viewpoint buffer control unit 121 by storing the viewpoint buffer information in a storage unit (not shown) in advance, for example.

Return to Fig. 3. Based on the rendered image, the multi-viewpoint generation unit 122 generates a viewpoint conversion image in which the line-of-sight unit vector direction is viewed from the viewpoint coordinates included in the conversion viewpoint information.

Note that FIG. 3 shows a plurality of rendered images that can be generated by the rendering unit 110 arranged in a matrix. Among the illustrated matrix-shaped rendered images, the hatched image is an image that is not actually rendered by the rendering unit 110, and the rendering unit 110 transfers the rendered image M0 without hatching to the multi-viewpoint generation unit 122. Output.

Further, in FIG. 3, a plurality of viewpoint conversion images that can be generated by the multi-viewpoint generation unit 122 are arranged and shown in a matrix. Of the matrix-shaped viewpoint conversion images shown in the figure, the viewpoint conversion image with hatching is an image that is not actually generated by the multi-viewpoint generation unit 122, and the multi-viewpoint generation unit 122 is a viewpoint conversion image without hatching. Generate M1 to M5.

The multi-viewpoint generation unit 122 generates a viewpoint conversion image using the rendering incidental information.

Here, an example of rendering incidental information will be described. The rendering incidental information includes, for example, metadata generated each time a game image is rendered (hereinafter referred to as run-time metadata) and metadata created during game production (hereinafter referred to as production-time metadata). Is included.

An example of run-time metadata is the normal information of objects in the game. Examples of run-time metadata include information related to light reflection, such as object specular, roughness, albedo, and transparency. One example of run-time metadata is information about the speed of objects in the game. An example of run-time metadata is information that represents the material of an object, such as the material ID of the object in the game. An example of run-time metadata is the Depth information of objects in the game.

The above-mentioned run-time metadata is metadata created depending on the player's viewpoint, but the run-time metadata also includes metadata created independently of the player's viewpoint. Examples of such run-time metadata include an in-game user interface (UI) showing the player's status, the progress of the game, and the Reflection map.

In addition, as an example of production metadata, there is data that is asynchronous to the frame of the game loop and does not depend on the player's viewpoint, such as mesh (3D model), texture, animation pattern of objects in the game, and layout data. Be done.

The multi-viewpoint generation unit 122 generates a viewpoint conversion image from the rendered image using the above-mentioned rendering incidental information. The multi-viewpoint generation unit 122 may generate a viewpoint-converted image from a rendered image by using, for example, a model-based method, or may generate a viewpoint-converted image by using a DNN-based method. In this way, the viewpoint conversion image can be generated by using various existing methods.

Note that the above-mentioned rendering incidental information may differ depending on the type of game or the title of the game. Therefore, the multi-viewpoint generation unit 122 may perform generation processing specialized for each type of game or each title of the game. For example, when the multi-viewpoint generation unit 122 executes a viewpoint conversion image generation process using a DNN-based method, the multi-viewpoint generation unit 122 can be generated by preparing a coefficient database according to the type of game or the like. It is possible to perform generation processing according to the type of game. Further, when the multi-viewpoint generation unit 122 executes the viewpoint conversion image generation process using the model-based method, the multi-viewpoint generation unit 122 can be used as a game by preparing a model according to the type of the game or the like. It is possible to perform generation processing according to the type of.

In this way, the multi-viewpoint generation unit 122 can perform viewpoint conversion with high accuracy by executing the generation process according to the type of the game, the title of the game, in other words, the rendering incidental information.

It is not necessary for the multi-viewpoint generation unit 122 to change the generation process for all game types or game titles. For example, the multi-viewpoint generation unit 122 may execute a specific generation process in a specific game.

When the viewpoint buffer information includes N conversion viewpoint information, the multi-view generation unit 122 generates N viewpoint conversion images. The multi-viewpoint generation unit 122 outputs the generated N viewpoint conversion images to the encoding unit 130 as a viewpoint buffer (multi-viewpoint image). At this time, the multi-viewpoint generation unit 122 outputs the viewpoint buffer information to the encoding unit 130 in addition to the viewpoint buffer.

Here, it is assumed that the viewpoint buffer control unit 121 generates the viewpoint buffer information, but the present invention is not limited to this. For example, the multi-viewpoint generation unit 122 may perform viewpoint conversion based on a certain number of conversion viewpoint information within a certain range regardless of information on a game, information on network traffic, and the like. In this case, for example, the multi-viewpoint generation unit 122 may generate the viewpoint conversion image based on the viewpoint buffer information stored in the storage unit (not shown), and the viewpoint buffer control unit 121 may be omitted.

[Encoder unit 130]
Return to FIG. Upon receiving the viewpoint buffer information and the viewpoint buffer from the multi-viewpoint conversion unit 120, the encoding unit 130 encodes the viewpoint buffer information and the viewpoint buffer for transmission to the client device 200. The encoding unit 130 encodes the viewpoint buffer information so that it can be independently decoded prior to the viewpoint buffer. The encoding method performed by the encoding unit 130 may be a model-based method or a DNN-based method.

Further, the encoding unit 130 may or may not perform encoding in consideration of the viewpoint buffer information and the viewpoint buffer in the time direction. That is, the coding performed by the encoding unit 130 may be intra-coding or inter-coding.

The multi-viewpoint conversion unit 120 generates a multi-viewpoint image with a slightly deviated viewpoint. From this, it is considered that each viewpoint-converted image included in the multi-viewpoint image does not change significantly, and the viewpoint-converted image has high redundancy. Therefore, the encoding unit 130 can compress the multi-viewpoint image (viewpoint buffer) at a high compression rate.

<2.3. Client device>
[Decoding unit 220]
Next, an example of the decoding unit 220 of the client device 200 according to the embodiment of the present disclosure will be described. The decoding unit 220 first decodes the viewpoint buffer information from the compressed data received by the transmission / reception unit 210, and decodes the viewpoint conversion image included in the viewpoint buffer from the decoded viewpoint buffer information and the current user's viewpoint information. The decoding unit 220 generates a display image (game image) based on the current user's viewpoint information and the decoded viewpoint conversion image.

FIG. 11 is a block diagram showing a configuration example of the decoding unit 220 according to the embodiment of the present disclosure. The decoding unit 220 shown in FIG. 11 includes a buffer information decoding unit 221, a decoding control unit 222, a viewpoint buffer decoding unit 223, and an image generation unit 224.

The buffer information decoding unit 221 receives compressed data as received data from the transmission / reception unit 210. The buffer information decoding unit 221 decodes the viewpoint buffer information from the compressed data and outputs it to the decoding control unit 222.

The decode control unit 222 acquires the viewpoint information of the current user from the input information acquisition unit 230. The current user's viewpoint information is viewpoint information determined by command information such as a user's operation or operation, and indicates a viewpoint position and a line-of-sight direction to be rendered as a display image.

When the coordinate system of the converted viewpoint information included in the viewpoint buffer information and the coordinate system of the viewpoint information acquired from the input information acquisition unit 230 are different, the decode control unit 222 coordinates one coordinate system to match the other coordinate system. Perform the conversion. Here, it is assumed that the decoding control unit 222 performs processing in a unified manner in the coordinate system of the conversion viewpoint information.

The decoding control unit 222 determines the viewpoint information (hereinafter referred to as decoding viewpoint information) of the viewpoint conversion image to be decoded by the viewpoint buffer decoding unit 223 based on the viewpoint information and the viewpoint buffer information of the current user.

FIG. 12 is a diagram for explaining the decoding viewpoint information determined by the decoding control unit 222 according to the embodiment of the present disclosure.

FIG. 12 shows a plurality of viewpoint conversion images that can be generated by the multi-viewpoint generation unit 122 side by side in a matrix. Further, in FIG. 12, the viewpoint information of the viewpoint converted image actually included in the viewpoint buffer, in other words, the converted viewpoint information included in the viewpoint buffer information is shown by a cross. Further, "x" in FIG. 12 indicates the viewpoint information VP0 of the current user.

The decoding control unit 222 determines M conversion viewpoint information from the conversion viewpoint information VP1 to VP12 as the decoding viewpoint information based on the current user viewpoint information VP0. For example, the decoding control unit 222 selects the upper M conversion viewpoint information as the decoding viewpoint information in order of increasing distance from the current user viewpoint information VP0.

In the example of FIG. 12, the decoding control unit 222 selects M = 4 conversion viewpoint information VP4, VP7, VP8, VP11 as the decoding viewpoint information in order of distance from the user viewpoint information VP0. In FIG. 12, the conversion viewpoint information VP4, VP7, VP8, and VP11 selected by the decoding control unit 222 are indicated by a circled cross.

The method and number of decoding viewpoint information selected by the decoding control unit 222 is not limited to the example shown in FIG. The conversion viewpoint information that can be generated by interpolating the display image corresponding to the current user's viewpoint information VP0 by the image generation unit 224 in the subsequent stage may be selected. The selection method and the number may be appropriately changed depending on the processing by the image generation unit 224.

Return to Fig. 11. The decoding control unit 222 outputs the selected decoding viewpoint information to the viewpoint buffer decoding unit 223. Further, the decoding control unit 222 outputs the current user's viewpoint information VP0 to the image generation unit 224.

The viewpoint buffer decoding unit 223 decodes the viewpoint conversion image corresponding to the decoding viewpoint information from the received data acquired from the transmission / reception unit 210. In the example of FIG. 12, the viewpoint buffer decoding unit 223 acquires the conversion viewpoint information VP4, VP7, VP8, and VP11 as the decoding viewpoint information. In this case, the viewpoint buffer decoding unit 223 decodes the viewpoint conversion image corresponding to the conversion viewpoint information VP4, VP7, VP8, and VP11.

The viewpoint buffer decoding unit 223 outputs the decoded M viewpoint conversion images and the decoded viewpoint information to the image generation unit 224.

The image generation unit 224 generates game images corresponding to the current user's viewpoint information acquired from the decoding control unit 222 based on the M viewpoint conversion images acquired from the viewpoint buffer decoding unit 223 and the decoding viewpoint information. .. The image generation unit 224 interpolates and generates a game image from, for example, M viewpoint conversion images. The image generation unit 224 can interpolate and generate a game image by using various methods. The image generation unit 224 can interpolate and generate a game image using, for example, a model-based method or a DNN-based method.

The image generation unit 224 presents the game image to the user by outputting the generated game image to the display unit (not shown) of the peripheral device 300.

In this way, the decoding unit 220 decodes the number of viewpoint conversion images used for generating the game image based on the viewpoint information VP0 of the current user. As a result, the decoding unit 220 can reduce the processing load as compared with the case of decoding all the viewpoint conversion images included in the viewpoint buffer.

[Input information acquisition unit 230]
Return to FIG. The input information acquisition unit 230 detects the operation of the peripheral device 300 (for example, the controller) by the user and the operation of the user by the peripheral device 300 (for example, HMD). The input information acquisition unit 230 converts the detection result into command information that can be interpreted by, for example, the rendering unit 110 of the server device 100, and generates the viewpoint information of the current user.

The input information acquisition unit 230 outputs the viewpoint information of the current user to the decoding unit 220. Further, the input information acquisition unit 230 includes the viewpoint information of the current user in the command information and transmits it to the server device 100 via the transmission / reception unit 210.

<< 3. Information processing >>
<3.1. Multi-viewpoint conversion process>
FIG. 13 is a flowchart showing the flow of the multi-viewpoint conversion process executed by the server device 100 according to the embodiment of the present disclosure. While providing the service to the client device 200, the server device 100 repeatedly executes the multi-viewpoint conversion process shown in FIG. 13, for example, at a predetermined cycle.

As shown in FIG. 13, the server device 100 acquires a rendered image viewed from the viewpoint position and the line-of-sight direction of the first number (for example, one) (step S101). Further, the server device 100 acquires the rendering incidental information used for generating the rendered image (step S102).

The server device 100 generates viewpoint buffer information including a second number (for example, 12) of viewpoint positions and line-of-sight directions, which is larger than the first number (step S103). The server device 100 generates a viewpoint buffer including the viewpoint position and the viewpoint conversion image seen from the line-of-sight direction included in the viewpoint buffer information (step S104). The server device 100 generates a viewpoint buffer based on the rendered image and the rendering incidental information.

The server device 100 encodes the viewpoint buffer information and the viewpoint buffer (step S105), and transmits the obtained transmission data to the client device 200 (step S106).

<3.2. Display image generation process>
FIG. 14 is a flowchart showing a flow of display image generation processing executed by the client device 200 according to the embodiment of the present disclosure. When the client device 200 receives a signal from the server device 100, the client device 200 executes the display image generation process shown in FIG.

As shown in FIG. 14, the client device 200 receives the received signal (step S201). The client device 200 first decodes the viewpoint buffer information from the received signal (step S202).

The client device 200 selects the converted viewpoint information corresponding to the viewpoint converted image to be decoded from the converted viewpoint information included in the viewpoint buffer information as the decoded viewpoint information (step S203).

The client device 200 decodes the viewpoint buffer and acquires a viewpoint conversion image corresponding to the decoded viewpoint information (step S204).

The client device 200 interpolates and generates a display image (game image) viewed from the current user's viewpoint using the decoded viewpoint conversion image (step S205).

<< 4. Other embodiments >>
The above embodiment shows an example, and various modifications and applications are possible.

The control device for controlling the server device 100 or the client device 200 of the present embodiment may be realized by a dedicated computer system or a general-purpose computer system.

For example, a program for executing the above operation is stored and distributed in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk. Then, for example, the control device is configured by installing the program in a computer and executing the above-mentioned processing. At this time, the control device may be an external device (for example, a personal computer) of the server device 100 or the client device 200. Further, the control device may be an internal device of the server device 100 or the client device 200.

Further, the above communication program may be stored in a disk device provided in a server device on a network such as the Internet so that it can be downloaded to a computer or the like. Further, the above-mentioned functions may be realized by cooperation between the OS (Operating System) and the application software. In this case, the part other than the OS may be stored in a medium and distributed, or the part other than the OS may be stored in the server device so that it can be downloaded to a computer or the like.

Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or part of it can be done automatically by a known method. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown in the figure.

Further, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Further, the above-described embodiments can be appropriately combined in an area where the processing contents do not contradict each other. Further, the order of each step shown in the sequence diagram or the flowchart of the present embodiment can be changed as appropriate.

Further, for example, the present embodiment includes a device or any configuration constituting the system, for example, a processor as a system LSI (Large Scale Integration), a module using a plurality of processors, a unit using a plurality of modules, and a unit. It can also be implemented as a set or the like (that is, a configuration of a part of the device) to which other functions are added.

In the present embodiment, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

Further, for example, the present embodiment can have a cloud computing configuration in which one function is shared by a plurality of devices via a network and jointly processed.

<< 5. Summary >>
As described above, the server device 100 (an example of an information processing device) according to the embodiment of the present disclosure includes a multi-viewpoint conversion unit 120 (an example of a generation unit) and a transmission / reception unit 140 (an example of a transmission unit). Be prepared. The multi-viewpoint conversion unit 120 includes a rendered image (an example of the first rendered image) and rendering as seen from the viewpoint position and the line-of-sight direction (an example of the first number of viewpoint positions and the line-of-sight direction) included in the rendered viewpoint information. Acquires incidental information (an example of rendering information used for rendering a rendered image). The multi-viewpoint conversion unit 120 generates viewpoint buffer information (an example of line-of-sight information regarding a second number of viewpoint positions and line-of-sight directions that is larger than the first number). The multi-viewpoint conversion unit 120 generates a viewpoint conversion image (an example of the second rendered image) seen from the viewpoint position and the line-of-sight direction included in the decoded viewpoint information based on the rendered image and the rendering incidental information. The multi-viewpoint conversion unit 120 transmits the viewpoint buffer including the viewpoint buffer information and the viewpoint conversion image to the client device 200 (an example of the terminal device).

Thereby, the server device 100 can present the user with a high-quality image with further reduced delay, and can further suppress the deterioration of the quality of the service provided to the user.

Further, the multi-viewpoint conversion unit 120 of the server device 100 according to the embodiment of the present disclosure determines the viewpoint position to be included in the viewpoint buffer information according to the delay situation generated in the communication with the client device 200.

As a result, the server device 100 can generate a viewpoint conversion image according to the delay situation, and can present the user with a high-quality image with further reduced delay.

Further, the multi-viewpoint conversion unit 120 of the server device 100 according to the embodiment of the present disclosure determines the viewpoint position to be included in the viewpoint buffer based on the history of the rendering viewpoint information (an example of the viewpoint position of the rendered image acquired in the past). ..

Thereby, the server device 100 can generate a viewpoint conversion image according to the user's operation or movement, and can present the user with a high-quality image with further reduced delay.

Further, the multi-viewpoint conversion unit 120 of the server device 100 according to the embodiment of the present disclosure determines the viewpoint position to be included in the viewpoint buffer information according to the type and title of the game (an example of the content including the rendered image).

As a result, the server device 100 can generate a viewpoint conversion image according to the user's operation or movement peculiar to the content, and can present the user with a high-quality image with further reduced delay.

Further, the multi-viewpoint conversion unit 120 of the server device 100 according to the embodiment of the present disclosure determines the viewpoint position to be included in the viewpoint buffer information according to the game play situation (an example of the scene of the rendered image).

As a result, the server device 100 can generate a viewpoint conversion image that can correspond to the user's operation and movement according to the scene of the content (for example, the play situation), and the user can obtain a high-quality image with further reduced delay. Can be presented to.

Further, the rendering information according to the embodiment of the present disclosure is run-time metadata (an example of information generated when rendering a rendered image).

As a result, the server device 100 can generate a viewpoint conversion image with higher image quality by generating the viewpoint conversion image using the runtime metadata.

Further, the rendering information according to the embodiment of the present disclosure is production-time metadata (an example of information preset for rendering a rendered image).

As a result, the server device 100 can generate a viewpoint-converted image with higher image quality by generating the viewpoint-converted image using the metadata at the time of production.

Further, the multi-viewpoint conversion unit 120 of the server device 100 according to the embodiment of the present disclosure calculates the viewpoint density function (an example of the viewpoint function representing the distribution of the viewpoint position), the line-of-sight function, and the calculated viewpoint density function. The viewpoint buffer information is generated using the line-of-sight function. The transmission / reception unit 140 of the server device 100 transmits information regarding the viewpoint density function and the line-of-sight function as viewpoint buffer information.

Thereby, the server device 100 can reduce the amount of information of the viewpoint buffer information.

Further, the client device 200 (an example of an information processing device) according to the embodiment of the present disclosure includes a transmission / reception unit 210 (an example of a reception unit) and a decoding unit 220 (an example of a generation unit). The receiving unit 210 is a viewpoint converted image generated based on the rendered image seen from the viewpoint position and the line-of-sight direction included in the rendered viewpoint information and the rendering incidental information, and the viewpoint position and the line of sight included in the converted viewpoint information. The viewpoint buffer including the viewpoint conversion image viewed from the direction and the viewpoint buffer information including the conversion viewpoint information are received. The decoding unit 220 generates a display image based on the viewpoint conversion image and the viewpoint buffer information.

Thereby, the client device 200 can present the user with a high-quality image with further reduced delay, and can further suppress the deterioration of the quality of the service provided to the user.

Further, the decoding unit 220 of the client device 200 according to the embodiment of the present disclosure selects and selects at least one viewpoint position and line-of-sight direction included in the viewpoint buffer information based on the user's current viewpoint position and line-of-sight direction. The display image is generated based on the viewpoint conversion image viewed from the viewpoint position and the line-of-sight direction.

As a result, the client device 200 can present a high-quality image to the user while reducing the number of viewpoint-converted images to be decoded.

Further, the information processing system 10 according to the embodiment of the present disclosure includes a server device 100 and a client device 200. The server device 100 includes a multi-viewpoint conversion unit 120 and a transmission / reception unit 140. The multi-viewpoint conversion unit 120 acquires the rendered image as seen from the viewpoint position and the line-of-sight direction included in the rendered viewpoint information, and the rendered image incidental information. The multi-viewpoint conversion unit 120 generates viewpoint buffer information. The multi-viewpoint conversion unit 120 generates a viewpoint conversion image seen from the viewpoint position and the line-of-sight direction included in the decoded viewpoint information based on the rendered image and the rendering incidental information. The transmission / reception unit 140 transmits the viewpoint buffer including the viewpoint buffer information and the viewpoint conversion image to the client device 200. The client device 200 includes a transmission / reception unit 210 and a decoding unit 220. The transmission / reception unit 210 receives the viewpoint conversion image and the viewpoint buffer information. The decoding unit 220 generates a display image based on the viewpoint conversion image and the viewpoint buffer information.

Thereby, the information processing system 10 can present the user with a high-quality image with further reduced delay, and can further suppress the deterioration of the quality of the service provided to the user.

Further, the effects described in the present specification are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may have other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

The present technology can also have the following configurations.
(1)
The first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image are acquired, and the rendering information is acquired.
The line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number, is generated.
A generation unit that generates a second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number based on the first rendered image and the rendered information.
A transmission unit that transmits the line-of-sight information and the second rendered image to the terminal device, and
Information processing device equipped with.
(2)
The information processing device according to (1), wherein the generation unit determines the viewpoint position to be included in the line-of-sight information according to a delay situation generated in communication with the terminal device.
(3)
The information processing apparatus according to (1) or (2), wherein the generation unit determines the viewpoint position to be included in the line-of-sight information based on the viewpoint position of the first rendered image acquired in the past.
(4)
The information processing apparatus according to any one of (1) to (3), wherein the generation unit determines the viewpoint position to be included in the line-of-sight information according to the content including the first rendered image.
(5)
The information processing apparatus according to any one of (1) to (4), wherein the generation unit determines the viewpoint position to be included in the line-of-sight information according to the scene of the first rendered image.
(6)
The information processing apparatus according to any one of (1) to (5), wherein the rendering information is information generated when the first rendered image is rendered.
(7)
The information processing apparatus according to any one of (1) to (5), wherein the rendering information is information preset for rendering the first rendered image.
(8)
The generation unit calculates a viewpoint function representing the distribution of the viewpoint position and a line-of-sight function representing the line-of-sight direction according to the viewpoint position, and uses the calculated viewpoint function and the line-of-sight function to obtain the second line-of-sight function. Generate the viewpoint position and the line-of-sight direction of the number,
The transmitting unit transmits information about the viewpoint function and the line-of-sight function as the line-of-sight information.
The information processing apparatus according to any one of (1) to (7).
(9)
The first rendered image viewed from the viewpoint position and the line-of-sight direction of the first number, and the second rendered image generated based on the rendering information used for rendering the first rendered image. Reception of receiving a second rendered image viewed from the viewpoint position and the line-of-sight direction of a second number larger than the first number, and line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number. Department and
A generation unit that generates a display image based on the second rendered image and the line-of-sight information,
Information processing device equipped with.
(10)
The generation unit selects at least one of the viewpoint positions and the line-of-sight directions included in the line-of-sight information based on the user's current viewpoint position and the line-of-sight direction, and views from the selected viewpoint position and the line-of-sight direction. The information processing apparatus according to (9), which generates the display image based on the second rendered image.
(11)
An information processing system including an information processing device and a terminal device.
The information processing device is
The first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image are acquired, and the rendering information is acquired.
The line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number, is generated.
A generation unit that generates a second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number based on the first rendered image and the rendered information.
A transmission unit that transmits the line-of-sight information and the second rendered image, and
Equipped with
The terminal device is
A receiving unit that receives the second rendered image and the line-of-sight information.
A generation unit that generates a display image based on the second rendered image and the line-of-sight information,
Information processing system equipped with.
(12)
The first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image are acquired, and the rendering information is acquired.
The line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number, is generated.
Based on the first rendered image and the rendered information, a second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number is generated.
The line-of-sight information and the second rendered image are transmitted to the terminal device.
Information processing method.

10 Information processing system 100 Server equipment 110 Rendering unit 120 Multi-viewpoint conversion unit 130

Encoding unit

140, 210 Transmission / reception unit 150 Command information acquisition unit 200 Client equipment 220 Decoding unit 230 Input information acquisition unit 300 Peripheral devices

Claims

The first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image are acquired, and the rendering information is acquired.
The line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number, is generated.
A generation unit that generates a second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number based on the first rendered image and the rendered information.
A transmission unit that transmits the line-of-sight information and the second rendered image to the terminal device, and
Information processing device equipped with.
The information processing device according to claim 1, wherein the generation unit determines the viewpoint position to be included in the line-of-sight information according to a delay situation generated in communication with the terminal device.
The information processing device according to claim 1, wherein the generation unit determines the viewpoint position to be included in the line-of-sight information based on the viewpoint position of the first rendered image acquired in the past.
The information processing device according to claim 1, wherein the generation unit determines the viewpoint position to be included in the line-of-sight information according to the content including the first rendered image.
The information processing device according to claim 1, wherein the generation unit determines the viewpoint position to be included in the line-of-sight information according to the scene of the first rendered image.
The information processing device according to claim 1, wherein the rendering information is information generated when the first rendered image is rendered.
The information processing device according to claim 1, wherein the rendering information is information preset for rendering the first rendered image.
The generation unit calculates a viewpoint function representing the distribution of the viewpoint position and a line-of-sight function representing the line-of-sight direction according to the viewpoint position, and uses the calculated viewpoint function and the line-of-sight function to obtain the second line-of-sight function. Generate the viewpoint position and the line-of-sight direction of the number,
The transmitting unit transmits information about the viewpoint function and the line-of-sight function as the line-of-sight information.
The information processing apparatus according to claim 1.
The first rendered image viewed from the viewpoint position and the line-of-sight direction of the first number, and the second rendered image generated based on the rendering information used for rendering the first rendered image. Reception of receiving a second rendered image viewed from the viewpoint position and the line-of-sight direction of a second number larger than the first number, and line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number. Department and
A generation unit that generates a display image based on the second rendered image and the line-of-sight information,
Information processing device equipped with.
The generation unit selects at least one of the viewpoint positions and the line-of-sight directions included in the line-of-sight information based on the user's current viewpoint position and the line-of-sight direction, and looks from the selected viewpoint position and the line-of-sight direction. The information processing apparatus according to claim 9, which generates the display image based on the rendered image.
An information processing system including an information processing device and a terminal device.
The information processing device is
The first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image are acquired, and the rendering information is acquired.
The line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number, is generated.
A generation unit that generates a second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number based on the first rendered image and the rendered information.
A transmission unit that transmits the line-of-sight information and the second rendered image, and
Equipped with
The terminal device is
A receiving unit that receives the second rendered image and the line-of-sight information.
A generation unit that generates a display image based on the second rendered image and the line-of-sight information.
Information processing system equipped with.
The first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image are acquired, and the rendering information is acquired.
The line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number, is generated.
Based on the first rendered image and the rendered information, a second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number is generated.
The line-of-sight information and the second rendered image are transmitted to the terminal device.
Information processing method.