WO2022124047A1 - Dispositif de traitement d'informations, système de traitement d'informations et procédé de traitement d'informations - Google Patents

Dispositif de traitement d'informations, système de traitement d'informations et procédé de traitement d'informations Download PDF

Info

Publication number
WO2022124047A1
WO2022124047A1 PCT/JP2021/042524 JP2021042524W WO2022124047A1 WO 2022124047 A1 WO2022124047 A1 WO 2022124047A1 JP 2021042524 W JP2021042524 W JP 2021042524W WO 2022124047 A1 WO2022124047 A1 WO 2022124047A1
Authority
WO
WIPO (PCT)
Prior art keywords
viewpoint
information
line
sight
image
Prior art date
Application number
PCT/JP2021/042524
Other languages
English (en)
Japanese (ja)
Inventor
健一郎 細川
Original Assignee
ソニーグループ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーグループ株式会社 filed Critical ソニーグループ株式会社
Publication of WO2022124047A1 publication Critical patent/WO2022124047A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]

Definitions

  • This disclosure relates to an information processing device, an information processing system, and an information processing method.
  • a technique of presenting an image rendered by a rendering device such as a game image or a 3D image to a user via a network.
  • a rendering device such as a game image or a 3D image
  • a game image of a viewpoint corresponding to a user's operation is rendered by a rendering device and presented to the user.
  • a technique of displaying an image rendered by a rendering device using AR (Augment Reality) or VR (Virtual Reality) on a head-mounted display worn by a user for example.
  • the rendered image When the image rendered by the rendering device is presented to the user via the network, for example, the rendered image may be delayed and presented to the user depending on the traffic situation of the network. Therefore, there is known a technique of predicting the posture of the user after the lapse of delay from the posture of the user and rendering an image based on the predicted posture. In such a technique, the rendered image is corrected according to the difference between the predicted posture and the actual posture and presented to the user.
  • an information processing device includes a generation unit and a transmission unit.
  • the generation unit acquires the first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image.
  • the generation unit generates line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number.
  • the generation unit generates the second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number based on the first rendered image and the rendered information.
  • the transmission unit transmits the line-of-sight information and the second rendered image to the terminal device.
  • Each of one or more embodiments (including examples and modifications) described below can be implemented independently. On the other hand, at least a part of the plurality of embodiments described below may be carried out in combination with at least a part of other embodiments as appropriate. These plurality of embodiments may contain novel features that differ from each other. Therefore, these plurality of embodiments may contribute to solving different purposes or problems, and may have different effects.
  • FIG. 1 is a diagram for explaining an outline of the proposed technique of the present disclosure.
  • the proposed technique according to the present disclosure is implemented in the information processing system shown in FIG.
  • the information processing system 10 includes a server device 100 and a client device 200.
  • the server device 100 and the client device 200 are connected via a network N such as the Internet.
  • the server device 100 is, for example, a rendering server, and transmits a rendered image (hereinafter, also referred to as a rendered image) to the client device 200.
  • the server device 100 is a game server, and provides a game service to the client device 200 by transmitting a rendered image which is a game image to the client device 200.
  • the server device 100 may be a device that provides AR or VR to the user via the client device 200. In this case, the server device 100 generates, for example, a rendered image according to the user's viewpoint and transmits it to the client device 200.
  • the client device 200 is an information processing device such as a smartphone, a PC, or a game device. Further, the client device 200 may be an information processing device worn on the user's head, such as a head-mounted display (HMD: Head Mounted Display) or smart glasses. The client device 200 displays the rendered image acquired from the server device 100 on the display unit of the own device or the display device connected to the own device.
  • HMD Head Mounted Display
  • the display of the rendered image on the client device 200 may be delayed due to the communication delay generated in the network N.
  • the communication delay generated in the network N.
  • the user's viewpoint moves while the communication delay occurs.
  • the user may feel VR sickness due to the deviation between the viewpoint of the displayed image and the viewpoint of the user.
  • the transmission time depends on the traffic of the Internet line. Therefore, depending on the amount of traffic on the Internet line, the quality of services provided to users such as games may be significantly deteriorated.
  • the server device 100 converts the rendered image into a multi-viewpoint image, and transmits the converted multi-viewpoint image to the client device 200. At this time, the server device 100 generates a multi-viewpoint image with high accuracy by using the information used when generating the rendered image (hereinafter, also referred to as rendering incidental information).
  • the client device 200 generates a display image based on the received multi-viewpoint image according to the current user's viewpoint or the operation status of the game by the user, and presents the display image to the user.
  • the server device 100 acquires the rendered image M0 and the rendering incidental information (step S1).
  • the rendered image is an image generated by the rendering unit (not shown) of the server device 100 based on the operation status of the game and the viewpoint information of the user transmitted from the client device 200.
  • the rendering incidental information is information used by the rendering unit to generate a rendered image, and includes, for example, depth information.
  • the rendering unit uses the rendering incidental information to generate a rendered image based on a predetermined number of viewpoint positions and line-of-sight directions.
  • the server device 100 acquires a rendered image M0 viewed from one viewpoint position and line-of-sight direction from a plurality of images in a viewpoint position and a line-of-sight direction that can be generated by the rendering unit.
  • a plurality of rendered images that can be generated by the rendering unit are arranged and shown in a matrix.
  • the rendered image with hatching is an image that is not actually generated by the rendering unit, and the rendering unit generates the rendered image M0 without hatching.
  • the server device 100 generates the viewpoint buffer information (step S2).
  • the viewpoint buffer information is information used to generate a multi-viewpoint image, and includes a plurality of viewpoint positions and line-of-sight directions.
  • the server device 100 generates viewpoint buffer information based on a game status, a communication delay status (communication status), and the like.
  • the server device 100 generates a viewpoint buffer using the viewpoint buffer information and the rendering incidental information based on the rendered image (step S3).
  • the viewpoint buffer includes a plurality of viewpoint positions included in the viewpoint buffer information and a plurality of images viewed from the line-of-sight direction (hereinafter, also referred to as multi-viewpoint images).
  • FIG. 1 shows a plurality of images that can be viewpoint-converted by the server device 100 arranged in a matrix.
  • the hatched images are images that are not actually converted by the server device 100, and the server device 100 generates the unhatched images M1 to M5 by the viewpoint conversion.
  • the server device 100 can generate a highly accurate multi-viewpoint image by using the rendering incidental information.
  • the server device 100 transmits the viewpoint buffer information and the viewpoint buffer to the client device 200 via the network N (step S4).
  • the client device 200 receives the viewpoint buffer information and the viewpoint buffer from the server device 100 (step S5).
  • the client device 200 generates a display image from the multi-viewpoint image included in the viewpoint buffer based on the current viewpoint of the user (step S6), and presents the display image to the user.
  • the server device 100 transmits a plurality of multi-viewpoint images generated from the rendered image to the client device 200.
  • the server device 100 can generate a high-quality multi-viewpoint image by using the rendering incidental information.
  • the client device 200 generates a display image according to the viewpoint of the current user from a plurality of multi-viewpoint images.
  • the information processing system 10 can present the user with a high-quality display image according to the viewpoint of the current user even if a communication delay occurs in the network N, and the deterioration of the provided service due to the communication delay is deteriorated. It can be suppressed.
  • the information processing system 10 may be an AR system or a VR system.
  • the information processing system 10 may be any system that provides the user with an image corresponding to the movement of the viewpoint, and is not limited to the system that provides the game service.
  • FIG. 2 is a diagram showing an example of the configuration of the information processing system 10 according to the embodiment of the present disclosure.
  • the information processing system includes a server device 100, a client device 200, and a peripheral device 300.
  • the server device 100 distributes a rendered game image (for example, a rendered image).
  • the client device 200 receives the video of the game through the network N, which is a communication line such as the Internet, and reproduces it, for example, on the display unit (not shown) of the peripheral device 300.
  • the peripheral device 300 includes, for example, a display, an HMD, a controller, and the like.
  • the client device 200 detects, for example, command information an operation or operation performed by the user in response to the reproduced video.
  • the client device 200 detects the operation on the controller as command information.
  • the client device 200 detects the movement of the user's head as the movement of the HMD.
  • the client device 200 transmits the detected command information to the server device 100 via the network N.
  • the server device 100 renders an image of the game at the next time according to the received command information and distributes it to the client device 200.
  • the server device 100 is an information processing device that provides an application function to the client device 200.
  • the server device 100 is, for example, a game server (Game Server) that provides a game service as an application function, and transmits a rendered image to be displayed as a game screen to the client device 200.
  • Game Server Game Server
  • the server device 100 receives the command information from the client device 200.
  • command information includes information that can identify the user's viewpoint, such as information on the movement of the user's head, and information on operations by the user with respect to the controller.
  • the server device 100 includes a rendering unit 110, a multi-viewpoint conversion unit 120, an encoding unit 130, a transmission / reception unit 140, and a command information acquisition unit 150.
  • the rendering unit 110 renders the game image.
  • the rendering unit 110 determines, for example, the camera position in the game (for example, the user's viewpoint position), the progress of the game, and the like based on the command information acquired from the client device 200, and produces a game image according to the determination result. Generate.
  • the rendering unit 110 is an application that has a Game Logic function for controlling the progress of the game and provides a game service.
  • the rendering unit 110 generates n1 viewpoint positions and line-of-sight directions (hereinafter, also simply referred to as viewpoints or viewpoint information) rendered images.
  • the rendering unit 110 shall generate a rendered image viewed from one viewpoint.
  • the rendering unit 110 generates a rendered image using the rendering incidental information based on the command information.
  • the rendering incidental information includes, for example, depth information and normal information of objects included in the game. The details of the rendering incidental information will be described later.
  • the time when the rendering unit 110 generates the rendered image is referred to as a rendering time t1.
  • the rendering unit 110 outputs the rendered image, the rendering incidental information, and the information about the game to the multi-viewpoint conversion unit 120.
  • the information about the game includes, for example, information about the title of the game, information about the type of the game such as a race game, a simulation game, and an RPG game, and information such as a game play status.
  • the multi-viewpoint conversion unit 120 generates a viewpoint buffer including a plurality of viewpoint conversion images from the rendered image generated by the rendering unit 110.
  • a plurality of viewpoint conversion images are collectively referred to as a multi-viewpoint image.
  • the multi-viewpoint conversion unit 120 acquires information on the network traffic (delay) status from the transmission / reception unit 140 in addition to the rendered image acquired from the rendering unit 110, rendering incidental information, and information on the game.
  • the multi-viewpoint conversion unit 120 generates viewpoint buffer information from the acquired information and converts the rendered image into a multi-viewpoint image viewed from n2 viewpoints. It is assumed that the number of viewpoints n2 of the multi-viewpoint image is larger than the number of viewpoints n1 of the rendered image (n1 ⁇ n2). Further, the viewpoint buffer information includes the coordinate information of the viewpoint of the multi-view image included in the viewpoint buffer. The details of the viewpoint buffer and the viewpoint buffer information generated by the multi-viewpoint conversion unit 120 will be described later.
  • the multi-viewpoint conversion unit 120 outputs the generated viewpoint buffer and viewpoint buffer information to the encoding unit 130.
  • the multi-viewpoint conversion unit 120 generates in advance an image viewed from a viewpoint around the viewpoint of the rendered image (hereinafter, also referred to as a rendering viewpoint) as a multi-viewpoint image based on the rendered image rendered by the rendering unit 110.
  • the multi-viewpoint conversion unit 120 builds a buffer for images of a plurality of viewpoints as a viewpoint buffer in advance.
  • the server device 100 transmits a game image having such a viewpoint buffer to the client device 200.
  • the GAP between the rendering time t1 and the command time t2 causes a deviation between the viewpoint of the game image and the viewpoint of the image expected by the user.
  • the client device 200 can generate a game image in which the deviation is interpolated using the viewpoint buffer.
  • the client device 200 is based on the viewpoint conversion image of the viewpoint included in the viewpoint buffer and close to the viewpoint of the image expected by the user.
  • a game image can be generated by performing viewpoint conversion. In this way, even if the deviation is large, by generating the game image from the viewpoint conversion image included in the viewpoint buffer, the deviation of the viewpoint can be reduced as compared with the case where the game image is generated from the rendered image. Deterioration of the quality of the game screen can be further suppressed.
  • the encoding unit 130 compresses the viewpoint buffer and the viewpoint buffer information generated by the multi-viewpoint conversion unit 120 into a small capacity, and outputs the compressed bit stream (hereinafter, also referred to as compressed data) to the transmission / reception unit 140. do.
  • the transmission / reception unit 140 is a communication interface (I / F) that communicates with an external device.
  • the transmission / reception unit 140 is realized by, for example, a NIC (Network Interface Card) or the like.
  • the transmission / reception unit 140 converts the compressed data encoded by the encoding unit 130 into a transmission signal and transmits it to the client device 200. Further, the transmission / reception unit 140 receives command information from the client device 200 and notifies the viewpoint information acquisition unit 150 of the reception result.
  • the command information acquisition unit 150 acquires command information via the transmission / reception unit 140.
  • the command information acquisition unit 150 outputs the acquired command information to the rendering unit 110.
  • the client device 200 is an information processing device that receives application functions from the server device 100.
  • the client device 200 receives, for example, a game service as an application function.
  • the client device 200 receives an image (viewpoint buffer) related to the game transmitted from the server device 100, and displays the game image based on the received viewpoint buffer.
  • the client device 200 transmits command information according to the operation or operation of the game by the user to the server device 100.
  • the client device 200 has a transmission / reception unit 210, a decoding unit 220, and an input information acquisition unit 230.
  • the transmission / reception unit 210 is a communication interface (I / F) that communicates with an external device.
  • the transmission / reception unit 210 is realized by, for example, a NIC (Network Interface Card) or the like.
  • the transmission / reception unit 210 receives a transmission signal from the server device 100 as a reception signal, and converts the received reception signal into compressed data. Further, the transmission / reception unit 210 transmits command information to the server device 100.
  • the decoding unit 220 decodes the compressed data received by the transmission / reception unit 210.
  • the decoding unit 220 generates a game image to be presented to the user from the multi-viewpoint image included in the viewpoint buffer based on the user's viewpoint information and the viewpoint buffer information acquired by the input information acquisition unit 230.
  • the user's viewpoint information acquired by the input information acquisition unit 230 is the latest viewpoint information in the current game, that is, information regarding the viewpoint of the game image expected by the user, and is simply "current". Also referred to as "user's viewpoint information”.
  • the decoding unit 220 displays the generated game image on the display unit (not shown) of the peripheral device 300.
  • the input information acquisition unit 230 is a user interface (UI: User Interface) that accepts operations of the peripheral device 300 by the user.
  • the input information acquisition unit 230 converts the operation or operation performed by the user in response to the game image displayed on the peripheral device 300 into command information that can be interpreted by the server device 100. Further, the input information acquisition unit 230 generates viewpoint information (viewpoint information of the current user) of the game image to be displayed on the peripheral device 300.
  • the input information acquisition unit 230 outputs the generated viewpoint information to the decoding unit 220. Further, the input information acquisition unit 230 includes the viewpoint information in the command information and transmits the viewpoint information to the server device 100 via the transmission / reception unit 210.
  • the peripheral device 300 is a device such as a display, smart glasses, HMD, controller, etc. that displays a game image output by the client device 200 and accepts operations by the user.
  • the peripheral device 300 is a display
  • the peripheral device 300 presents a game image to the user.
  • the peripheral device 300 is a controller
  • the peripheral device 300 accepts an operation performed by the user in response to a game image displayed on a display device (not shown).
  • the peripheral device 300 is an HMD
  • the peripheral device 300 displays a game image on a display unit (not shown) and detects the movement of the user's head as the movement of the HMD.
  • the peripheral device 300 is a device different from the client device 200, but the present invention is not limited to this.
  • the peripheral device 300 may be realized as a function of the client device 200.
  • the server device 100 uses the rendering incidental information to generate the viewpoint buffer. Further, the client device 200 generates a game image (display image) according to the viewpoint information of the current user based on the viewpoint buffer. As a result, the information processing system 10 can display the game image as seen from the viewpoint of the current user regardless of the delay of the network N.
  • the viewpoint buffer is generated by the server device 100.
  • the server device 100 uses the rendering incidental information to perform viewpoint conversion of the rendered image.
  • the client device 200 performs viewpoint conversion of the rendered image based on the viewpoint information of the current user.
  • the client device 200 it is difficult for the client device 200 to perform viewpoint conversion using rendering incidental information. This is because the rendering incidental information is inferior in coding efficiency as compared with the multi-viewpoint image, so that the communication amount becomes large.
  • the rendering incidental information includes not only image information but also various information such as information about objects in the game. Therefore, the rendering incidental information has a lower coding efficiency than the viewpoint buffer composed of image information having different viewpoints.
  • the information processing system 10 according to the embodiment of the present disclosure further reduces the image delay due to the viewpoint movement while suppressing the increase in network traffic, as compared with the case where the viewpoint conversion of the rendered image is performed by the client device 200. be able to. As described above, the information processing system 10 according to the embodiment of the present disclosure can provide a higher quality service.
  • FIG. 3 is a block diagram showing a configuration example of the multi-viewpoint conversion unit 120 according to the embodiment of the present disclosure. As shown in FIG. 3, the multi-viewpoint conversion unit 120 includes a viewpoint buffer control unit 121 and a multi-viewpoint generation unit 122.
  • the viewpoint buffer control unit 121 determines the viewpoint information of the multi-view image generated by the multi-view generation unit 122.
  • the viewpoint buffer control unit 121 is based on at least one of the viewpoint information of the rendered image generated by the rendering unit 110, the information about the game such as the game title and the play status, and the traffic information (communication status) of the network N. To decide.
  • FIGS. 4 to 7 are diagrams for explaining an example of viewpoint information determined by the viewpoint buffer control unit 121 according to the embodiment of the present disclosure.
  • FIGS. 4 to 7 show a plurality of rendered images that can be generated by the rendering unit 110 and examples of the multi-viewpoint images that can be generated by the multi-viewpoint generation unit 122 side by side in a matrix.
  • the viewpoint buffer control unit 121 determines the viewpoint information of the multi-view image generated by the multi-view generation unit 122 from a predetermined range including the viewpoint information of the rendered image generated by the rendering unit 110 (hereinafter, also referred to as rendering viewpoint information). do.
  • the viewpoint buffer control unit 121 determines the viewpoint information indicated by a cross from the circular range centered on the rendering viewpoint information as the viewpoint information of the viewpoint converted image (hereinafter, also referred to as converted viewpoint information). do.
  • FIG. 4 shows an example in which the viewpoint buffer control unit 121 determines 12 conversion viewpoint information.
  • the viewpoint buffer control unit 121 determines the conversion viewpoint information according to the situation of the game or the network N.
  • the viewpoint buffer control unit 121 determines the conversion viewpoint information according to the delay status (communication status) of the network N. More specifically, the viewpoint buffer control unit 121 determines the conversion viewpoint information from a larger predetermined range as the delay of the network N increases.
  • the viewpoint buffer control unit 121 determines, for example, 12 conversion viewpoint information from the range R1 shown in FIG. Further, when the delay of the network N is large, as shown in FIG. 5, for example, 12 conversion viewpoint information is determined from the range R2 wider than the range R1 (see FIG. 4).
  • the viewpoint buffer control unit 121 determines the conversion viewpoint information according to the delay of the network N. As a result, even when the delay of the network N is large, the game image seen from the viewpoint of the current user can be generated on the client device 200 side, and the deterioration of the quality of the service provided to the user can be further suppressed.
  • the viewpoint buffer control unit 121 evenly determines the conversion viewpoint information from the ranges R1 and R2. Therefore, the viewpoint buffer control unit 121 determines the conversion viewpoint information more densely as the delay of the network N is smaller, and determines the conversion viewpoint information sparsely as the delay is larger.
  • the viewpoint buffer control unit 121 determines the conversion viewpoint information according to the traffic (delay) status of the network N, but the present invention is not limited to this.
  • the viewpoint buffer control unit 121 may determine the conversion viewpoint information according to the information about the game such as the game title and the play situation (scene).
  • the viewpoint buffer control unit 121 can determine the conversion viewpoint information according to the game title and the type of the game. For example, in the case of a game in which the viewpoint moves quickly, such as a racing game, the viewpoint buffer control unit 121 can determine the converted viewpoint information from a larger range. On the other hand, for example, in the case of a game in which the viewpoint movement is small, the viewpoint buffer control unit 121 can determine the converted viewpoint information from a smaller range.
  • the viewpoint buffer control unit 121 may determine the conversion viewpoint information according to the play situation. For example, the viewpoint buffer control unit 121 determines the conversion viewpoint information from a larger range when playing in a space with a wide movement range such as outdoors, and when playing in a space with a narrow movement range such as indoors. The conversion viewpoint information may be determined from a narrower range.
  • the viewpoint buffer control unit 121 can determine the conversion viewpoint information according to the information about the game such as the game title and the play status, in other words, the content including the rendered image and the scene of the rendered image.
  • the viewpoint buffer control unit 121 can determine the converted viewpoint information according to the information related to the speed of the viewpoint movement among the information related to the game.
  • the viewpoint buffer control unit 121 evenly determines the conversion viewpoint information from the predetermined range, but the present invention is not limited to this.
  • the viewpoint buffer control unit 121 is densely located in the vicinity of the rendering viewpoint information (center of the range R3) in the circular range R3 centered on the rendering viewpoint information, and around the range R3.
  • the conversion viewpoint information may be determined so as to be sparse.
  • the viewpoint buffer control unit 121 can determine the conversion viewpoint information at the density shown in FIG.
  • the viewpoint buffer control unit 121 may determine the conversion viewpoint information according to the rendering viewpoint information, particularly the history of the rendering viewpoint information. In this case, the viewpoint buffer control unit 121 predicts the viewpoint information of the future time (for example, the command generation time t2) from the change (history) of the past rendering viewpoint information. The viewpoint buffer control unit 121 determines the converted viewpoint information from the range including the predicted viewpoint information.
  • the viewpoint buffer control unit 121 predicts the viewpoint information in a predetermined direction (P1 in FIG. 7), and determines the conversion viewpoint information from the long range R4 in the predicted direction (upper right direction in FIG. 7).
  • the number of converted viewpoint information (number of viewpoints) determined by the viewpoint buffer control unit 121 may be constant or variable.
  • the server device 100 may change the number of conversion viewpoint information according to the situation. This makes it possible to improve the efficiency of the calculation cost of the viewpoint conversion by the server device 100.
  • the server device 100 determines a certain number of conversion viewpoint information. May be good.
  • FIG. 8 is a diagram for explaining an example of viewpoint information according to the embodiment of the present disclosure.
  • the viewpoint information includes the viewpoint position and the line-of-sight direction.
  • the viewpoint position is represented by the viewpoint (camera) coordinates (x wk , y wk , z wk ) representing the position of the camera (or the user's head) that saw the image to be rendered.
  • the coordinate system of the viewpoint coordinates is, for example, a world coordinate system in which game objects are arranged.
  • the line-of-sight direction is represented by a line-of-sight unit vector (x vk , y vk , x vk ) indicating the direction of the camera (or the user's head).
  • the viewpoint buffer control unit 121 determines a plurality of sets of the viewpoint position (viewpoint coordinates) and the line-of-sight direction (line-of-sight unit vector) as a plurality of converted viewpoint information.
  • FIG. 9 is a diagram showing an example of conversion viewpoint information determined by the viewpoint buffer control unit 121 according to the embodiment of the present disclosure.
  • FIG. 9 shows a case where the viewpoint buffer control unit 121 determines N conversion viewpoint information.
  • the viewpoint buffer control unit 121 sets ⁇ (x w1 , y w1 , z w1 ), (x v1 , y v1 , x v1 ) ⁇ , ⁇ (x w1, y w1, z w1) ⁇ , ⁇ (x w1, y w1, x v1) ⁇ , ⁇ (x w1, y w1, z w1) ⁇ , ⁇ (x w1, y v1, x v1) ⁇ , ⁇ (x w1, y w1, z w1), x w2 , y w2 , z w2 ), (x v2 , y v2 , x v2 ) ⁇ ... ⁇ (x w2 , y w2 , z w2 ), (x v2 , y v2 , x v2 ) ⁇ ... ⁇ (x w2 , y w2
  • the viewpoint buffer control unit 121 outputs the determined set of viewpoint coordinates and line-of-sight unit vector to the multi-viewpoint generation unit 122 (see FIG. 3).
  • the converted viewpoint information shown in FIGS. 8 and 9 is an example, and the present invention is not limited to this.
  • the converted viewpoint information may be expressed in any format as long as the multi-viewpoint generation unit 122 can convert the viewpoint of the rendered image based on the converted viewpoint information.
  • the viewpoint buffer control unit 121 may output, for example, the conversion viewpoint information stored in the storage unit (not shown) to the multi-viewpoint generation unit 122 as the determined conversion viewpoint information, or may be determined using a function. You may.
  • the viewpoint buffer control unit 121 determines the conversion viewpoint information using a function will be described with reference to FIG. 10, but the method for determining the conversion viewpoint information is not limited to this.
  • FIG. 10 is a block diagram showing a configuration example of the viewpoint buffer control unit 121 according to the embodiment of the present disclosure.
  • the viewpoint buffer control unit 121 includes a viewpoint function generation unit 1211, a line-of-sight function generation unit 1213, a sampling unit 1212, and a line-of-sight vector generation unit 1214.
  • the viewpoint buffer control unit 121 first determines a set of a viewpoint density function and a line-of-sight function in a three-dimensional space.
  • the viewpoint buffer control unit 121 determines a predetermined viewpoint density function and line-of-sight function.
  • the viewpoint buffer control unit 121 may determine the viewpoint density function and the line-of-sight function according to information about the game, information about network traffic, rendering viewpoint information, and the like.
  • the viewpoint function generation unit 1211 determines, for example, an arbitrary probability density function P (x, y, z) as the viewpoint density function P. For example, the viewpoint function generation unit 1211 determines the viewpoint density function P according to information about the game, information about network traffic, rendering viewpoint information, etc. from a plurality of probability density functions stored in advance in the storage unit (not shown). do.
  • the viewpoint function generation unit 1211 may change the parameters of the viewpoint density function P according to the information about the game, the information about the network traffic, the rendering viewpoint information, and the like.
  • the viewpoint function generation unit 1211 outputs the determined viewpoint density function P to the sampling unit 1212.
  • the line-of-sight function generation unit 1213 determines a function that inputs an arbitrary viewpoint coordinate and outputs a line-of-sight unit vector as a line-of-sight function V. For example, the line-of-sight function generation unit 1213 determines the line-of-sight function V according to information about a game, information about network traffic, rendering viewpoint information, and the like from a plurality of line-of-sight functions stored in advance in a storage unit (not shown).
  • the line-of-sight function generation unit 1213 may change the parameters of the line-of-sight function V according to information about the game, information about network traffic, rendering viewpoint information, and the like.
  • the line-of-sight function generation unit 1213 outputs the determined line-of-sight function V to the line-of-sight vector generation unit 1214.
  • the sampling unit 1212 determines, for example, N viewpoint coordinates to be used for the viewpoint conversion from the viewpoint density function P by using various sampling methods. Examples of the sampling method include an inverse function method, rejection sampling, and Markov chain Monte Carlo method. The sampling unit 1212 outputs the determined viewpoint coordinates to the line-of-sight vector generation unit 1214.
  • the line-of-sight vector generation unit 1214 determines the line-of-sight unit vector by inputting the viewpoint coordinates determined by the sampling unit 1212 into the line-of-sight function V.
  • the line-of-sight vector generation unit 1214 outputs the determined N viewpoint coordinates and line-of-sight unit vectors to the multi-viewpoint generation unit 122 (see FIG. 3) as viewpoint buffer information.
  • the line-of-sight vector generation unit 1214 outputs the viewpoint coordinates and the line-of-sight unit vector as the conversion viewpoint information, but the present invention is not limited to this.
  • the viewpoint buffer control unit 121 may output the viewpoint density function P and the line-of-sight function V to the multi-viewpoint generation unit 122 as viewpoint buffer information.
  • the line-of-sight vector generation unit 1214 may determine the parameters of the predetermined viewpoint density function P and the predetermined line-of-sight function V, and output the determined parameters to the multi-viewpoint generation unit 122 as the viewpoint buffer information.
  • the viewpoint buffer control unit 121 may output the viewpoint density function P and the line-of-sight function V as the viewpoint buffer information, or information (for example, parameters, etc.) for uniquely determining the function.
  • the sampling unit 1212 and the line-of-sight vector generation unit 1214 of the viewpoint buffer control unit 121 may be omitted.
  • the information on the sampling method by the sampling unit 1212 and the function to be used is shared by the multi-viewpoint generation unit 122, the decoding unit 220 of the client device 200 (see FIG. 2), and the like.
  • the viewpoint buffer control unit 121 generates the viewpoint buffer information according to the communication status of the network N and the game information, but the present invention is not limited to this.
  • the multi-viewpoint conversion unit 120 may generate a viewpoint conversion image based on a predetermined constant viewpoint buffer information regardless of the above-mentioned information.
  • the server device 100 may omit the viewpoint buffer control unit 121 by storing the viewpoint buffer information in a storage unit (not shown) in advance, for example.
  • the multi-viewpoint generation unit 122 Based on the rendered image, the multi-viewpoint generation unit 122 generates a viewpoint conversion image in which the line-of-sight unit vector direction is viewed from the viewpoint coordinates included in the conversion viewpoint information.
  • FIG. 3 shows a plurality of rendered images that can be generated by the rendering unit 110 arranged in a matrix.
  • the hatched image is an image that is not actually rendered by the rendering unit 110, and the rendering unit 110 transfers the rendered image M0 without hatching to the multi-viewpoint generation unit 122. Output.
  • a plurality of viewpoint conversion images that can be generated by the multi-viewpoint generation unit 122 are arranged and shown in a matrix.
  • the viewpoint conversion image with hatching is an image that is not actually generated by the multi-viewpoint generation unit 122
  • the multi-viewpoint generation unit 122 is a viewpoint conversion image without hatching.
  • the multi-viewpoint generation unit 122 generates a viewpoint conversion image using the rendering incidental information.
  • the rendering incidental information includes, for example, metadata generated each time a game image is rendered (hereinafter referred to as run-time metadata) and metadata created during game production (hereinafter referred to as production-time metadata). Is included.
  • run-time metadata is the normal information of objects in the game.
  • run-time metadata include information related to light reflection, such as object specular, roughness, albedo, and transparency.
  • One example of run-time metadata is information about the speed of objects in the game.
  • An example of run-time metadata is information that represents the material of an object, such as the material ID of the object in the game.
  • An example of run-time metadata is the Depth information of objects in the game.
  • run-time metadata is metadata created depending on the player's viewpoint, but the run-time metadata also includes metadata created independently of the player's viewpoint.
  • run-time metadata include an in-game user interface (UI) showing the player's status, the progress of the game, and the Reflection map.
  • UI in-game user interface
  • production metadata there is data that is asynchronous to the frame of the game loop and does not depend on the player's viewpoint, such as mesh (3D model), texture, animation pattern of objects in the game, and layout data. Be done.
  • the multi-viewpoint generation unit 122 generates a viewpoint conversion image from the rendered image using the above-mentioned rendering incidental information.
  • the multi-viewpoint generation unit 122 may generate a viewpoint-converted image from a rendered image by using, for example, a model-based method, or may generate a viewpoint-converted image by using a DNN-based method. In this way, the viewpoint conversion image can be generated by using various existing methods.
  • the multi-viewpoint generation unit 122 may perform generation processing specialized for each type of game or each title of the game. For example, when the multi-viewpoint generation unit 122 executes a viewpoint conversion image generation process using a DNN-based method, the multi-viewpoint generation unit 122 can be generated by preparing a coefficient database according to the type of game or the like. It is possible to perform generation processing according to the type of game. Further, when the multi-viewpoint generation unit 122 executes the viewpoint conversion image generation process using the model-based method, the multi-viewpoint generation unit 122 can be used as a game by preparing a model according to the type of the game or the like. It is possible to perform generation processing according to the type of.
  • the multi-viewpoint generation unit 122 can perform viewpoint conversion with high accuracy by executing the generation process according to the type of the game, the title of the game, in other words, the rendering incidental information.
  • the multi-viewpoint generation unit 122 may change the generation process for all game types or game titles.
  • the multi-viewpoint generation unit 122 may execute a specific generation process in a specific game.
  • the multi-view generation unit 122 When the viewpoint buffer information includes N conversion viewpoint information, the multi-view generation unit 122 generates N viewpoint conversion images.
  • the multi-viewpoint generation unit 122 outputs the generated N viewpoint conversion images to the encoding unit 130 as a viewpoint buffer (multi-viewpoint image). At this time, the multi-viewpoint generation unit 122 outputs the viewpoint buffer information to the encoding unit 130 in addition to the viewpoint buffer.
  • the viewpoint buffer control unit 121 generates the viewpoint buffer information, but the present invention is not limited to this.
  • the multi-viewpoint generation unit 122 may perform viewpoint conversion based on a certain number of conversion viewpoint information within a certain range regardless of information on a game, information on network traffic, and the like.
  • the multi-viewpoint generation unit 122 may generate the viewpoint conversion image based on the viewpoint buffer information stored in the storage unit (not shown), and the viewpoint buffer control unit 121 may be omitted.
  • the encoding unit 130 Upon receiving the viewpoint buffer information and the viewpoint buffer from the multi-viewpoint conversion unit 120, the encoding unit 130 encodes the viewpoint buffer information and the viewpoint buffer for transmission to the client device 200. The encoding unit 130 encodes the viewpoint buffer information so that it can be independently decoded prior to the viewpoint buffer.
  • the encoding method performed by the encoding unit 130 may be a model-based method or a DNN-based method.
  • the encoding unit 130 may or may not perform encoding in consideration of the viewpoint buffer information and the viewpoint buffer in the time direction. That is, the coding performed by the encoding unit 130 may be intra-coding or inter-coding.
  • the multi-viewpoint conversion unit 120 generates a multi-viewpoint image with a slightly deviated viewpoint. From this, it is considered that each viewpoint-converted image included in the multi-viewpoint image does not change significantly, and the viewpoint-converted image has high redundancy. Therefore, the encoding unit 130 can compress the multi-viewpoint image (viewpoint buffer) at a high compression rate.
  • the decoding unit 220 first decodes the viewpoint buffer information from the compressed data received by the transmission / reception unit 210, and decodes the viewpoint conversion image included in the viewpoint buffer from the decoded viewpoint buffer information and the current user's viewpoint information.
  • the decoding unit 220 generates a display image (game image) based on the current user's viewpoint information and the decoded viewpoint conversion image.
  • FIG. 11 is a block diagram showing a configuration example of the decoding unit 220 according to the embodiment of the present disclosure.
  • the decoding unit 220 shown in FIG. 11 includes a buffer information decoding unit 221, a decoding control unit 222, a viewpoint buffer decoding unit 223, and an image generation unit 224.
  • the buffer information decoding unit 221 receives compressed data as received data from the transmission / reception unit 210.
  • the buffer information decoding unit 221 decodes the viewpoint buffer information from the compressed data and outputs it to the decoding control unit 222.
  • the decode control unit 222 acquires the viewpoint information of the current user from the input information acquisition unit 230.
  • the current user's viewpoint information is viewpoint information determined by command information such as a user's operation or operation, and indicates a viewpoint position and a line-of-sight direction to be rendered as a display image.
  • the decode control unit 222 coordinates one coordinate system to match the other coordinate system. Perform the conversion.
  • the decoding control unit 222 performs processing in a unified manner in the coordinate system of the conversion viewpoint information.
  • the decoding control unit 222 determines the viewpoint information (hereinafter referred to as decoding viewpoint information) of the viewpoint conversion image to be decoded by the viewpoint buffer decoding unit 223 based on the viewpoint information and the viewpoint buffer information of the current user.
  • FIG. 12 is a diagram for explaining the decoding viewpoint information determined by the decoding control unit 222 according to the embodiment of the present disclosure.
  • FIG. 12 shows a plurality of viewpoint conversion images that can be generated by the multi-viewpoint generation unit 122 side by side in a matrix. Further, in FIG. 12, the viewpoint information of the viewpoint converted image actually included in the viewpoint buffer, in other words, the converted viewpoint information included in the viewpoint buffer information is shown by a cross. Further, "x" in FIG. 12 indicates the viewpoint information VP0 of the current user.
  • the decoding control unit 222 determines M conversion viewpoint information from the conversion viewpoint information VP1 to VP12 as the decoding viewpoint information based on the current user viewpoint information VP0. For example, the decoding control unit 222 selects the upper M conversion viewpoint information as the decoding viewpoint information in order of increasing distance from the current user viewpoint information VP0.
  • the conversion viewpoint information VP4, VP7, VP8, and VP11 selected by the decoding control unit 222 are indicated by a circled cross.
  • the method and number of decoding viewpoint information selected by the decoding control unit 222 is not limited to the example shown in FIG.
  • the conversion viewpoint information that can be generated by interpolating the display image corresponding to the current user's viewpoint information VP0 by the image generation unit 224 in the subsequent stage may be selected.
  • the selection method and the number may be appropriately changed depending on the processing by the image generation unit 224.
  • the decoding control unit 222 outputs the selected decoding viewpoint information to the viewpoint buffer decoding unit 223. Further, the decoding control unit 222 outputs the current user's viewpoint information VP0 to the image generation unit 224.
  • the viewpoint buffer decoding unit 223 decodes the viewpoint conversion image corresponding to the decoding viewpoint information from the received data acquired from the transmission / reception unit 210.
  • the viewpoint buffer decoding unit 223 acquires the conversion viewpoint information VP4, VP7, VP8, and VP11 as the decoding viewpoint information.
  • the viewpoint buffer decoding unit 223 decodes the viewpoint conversion image corresponding to the conversion viewpoint information VP4, VP7, VP8, and VP11.
  • the viewpoint buffer decoding unit 223 outputs the decoded M viewpoint conversion images and the decoded viewpoint information to the image generation unit 224.
  • the image generation unit 224 generates game images corresponding to the current user's viewpoint information acquired from the decoding control unit 222 based on the M viewpoint conversion images acquired from the viewpoint buffer decoding unit 223 and the decoding viewpoint information. ..
  • the image generation unit 224 interpolates and generates a game image from, for example, M viewpoint conversion images.
  • the image generation unit 224 can interpolate and generate a game image by using various methods.
  • the image generation unit 224 can interpolate and generate a game image using, for example, a model-based method or a DNN-based method.
  • the image generation unit 224 presents the game image to the user by outputting the generated game image to the display unit (not shown) of the peripheral device 300.
  • the decoding unit 220 decodes the number of viewpoint conversion images used for generating the game image based on the viewpoint information VP0 of the current user. As a result, the decoding unit 220 can reduce the processing load as compared with the case of decoding all the viewpoint conversion images included in the viewpoint buffer.
  • the input information acquisition unit 230 detects the operation of the peripheral device 300 (for example, the controller) by the user and the operation of the user by the peripheral device 300 (for example, HMD).
  • the input information acquisition unit 230 converts the detection result into command information that can be interpreted by, for example, the rendering unit 110 of the server device 100, and generates the viewpoint information of the current user.
  • the input information acquisition unit 230 outputs the viewpoint information of the current user to the decoding unit 220. Further, the input information acquisition unit 230 includes the viewpoint information of the current user in the command information and transmits it to the server device 100 via the transmission / reception unit 210.
  • FIG. 13 is a flowchart showing the flow of the multi-viewpoint conversion process executed by the server device 100 according to the embodiment of the present disclosure. While providing the service to the client device 200, the server device 100 repeatedly executes the multi-viewpoint conversion process shown in FIG. 13, for example, at a predetermined cycle.
  • the server device 100 acquires a rendered image viewed from the viewpoint position and the line-of-sight direction of the first number (for example, one) (step S101). Further, the server device 100 acquires the rendering incidental information used for generating the rendered image (step S102).
  • the server device 100 generates viewpoint buffer information including a second number (for example, 12) of viewpoint positions and line-of-sight directions, which is larger than the first number (step S103).
  • the server device 100 generates a viewpoint buffer including the viewpoint position and the viewpoint conversion image seen from the line-of-sight direction included in the viewpoint buffer information (step S104).
  • the server device 100 generates a viewpoint buffer based on the rendered image and the rendering incidental information.
  • the server device 100 encodes the viewpoint buffer information and the viewpoint buffer (step S105), and transmits the obtained transmission data to the client device 200 (step S106).
  • FIG. 14 is a flowchart showing a flow of display image generation processing executed by the client device 200 according to the embodiment of the present disclosure.
  • the client device 200 receives a signal from the server device 100, the client device 200 executes the display image generation process shown in FIG.
  • the client device 200 receives the received signal (step S201).
  • the client device 200 first decodes the viewpoint buffer information from the received signal (step S202).
  • the client device 200 selects the converted viewpoint information corresponding to the viewpoint converted image to be decoded from the converted viewpoint information included in the viewpoint buffer information as the decoded viewpoint information (step S203).
  • the client device 200 decodes the viewpoint buffer and acquires a viewpoint conversion image corresponding to the decoded viewpoint information (step S204).
  • the client device 200 interpolates and generates a display image (game image) viewed from the current user's viewpoint using the decoded viewpoint conversion image (step S205).
  • the control device for controlling the server device 100 or the client device 200 of the present embodiment may be realized by a dedicated computer system or a general-purpose computer system.
  • a program for executing the above operation is stored and distributed in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk.
  • the control device is configured by installing the program in a computer and executing the above-mentioned processing.
  • the control device may be an external device (for example, a personal computer) of the server device 100 or the client device 200. Further, the control device may be an internal device of the server device 100 or the client device 200.
  • the above communication program may be stored in a disk device provided in a server device on a network such as the Internet so that it can be downloaded to a computer or the like.
  • the above-mentioned functions may be realized by cooperation between the OS (Operating System) and the application software.
  • the part other than the OS may be stored in a medium and distributed, or the part other than the OS may be stored in the server device so that it can be downloaded to a computer or the like.
  • each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. Can be integrated and configured.
  • the present embodiment includes a device or any configuration constituting the system, for example, a processor as a system LSI (Large Scale Integration), a module using a plurality of processors, a unit using a plurality of modules, and a unit. It can also be implemented as a set or the like (that is, a configuration of a part of the device) to which other functions are added.
  • a processor as a system LSI (Large Scale Integration)
  • a module using a plurality of processors a unit using a plurality of modules
  • a unit that is, a configuration of a part of the device
  • the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..
  • the present embodiment can have a cloud computing configuration in which one function is shared by a plurality of devices via a network and jointly processed.
  • the server device 100 (an example of an information processing device) according to the embodiment of the present disclosure includes a multi-viewpoint conversion unit 120 (an example of a generation unit) and a transmission / reception unit 140 (an example of a transmission unit). Be prepared.
  • the multi-viewpoint conversion unit 120 includes a rendered image (an example of the first rendered image) and rendering as seen from the viewpoint position and the line-of-sight direction (an example of the first number of viewpoint positions and the line-of-sight direction) included in the rendered viewpoint information.
  • Acquires incidental information (an example of rendering information used for rendering a rendered image).
  • the multi-viewpoint conversion unit 120 generates viewpoint buffer information (an example of line-of-sight information regarding a second number of viewpoint positions and line-of-sight directions that is larger than the first number).
  • the multi-viewpoint conversion unit 120 generates a viewpoint conversion image (an example of the second rendered image) seen from the viewpoint position and the line-of-sight direction included in the decoded viewpoint information based on the rendered image and the rendering incidental information.
  • the multi-viewpoint conversion unit 120 transmits the viewpoint buffer including the viewpoint buffer information and the viewpoint conversion image to the client device 200 (an example of the terminal device).
  • the server device 100 can present the user with a high-quality image with further reduced delay, and can further suppress the deterioration of the quality of the service provided to the user.
  • the multi-viewpoint conversion unit 120 of the server device 100 determines the viewpoint position to be included in the viewpoint buffer information according to the delay situation generated in the communication with the client device 200.
  • the server device 100 can generate a viewpoint conversion image according to the delay situation, and can present the user with a high-quality image with further reduced delay.
  • the multi-viewpoint conversion unit 120 of the server device 100 determines the viewpoint position to be included in the viewpoint buffer based on the history of the rendering viewpoint information (an example of the viewpoint position of the rendered image acquired in the past). ..
  • the server device 100 can generate a viewpoint conversion image according to the user's operation or movement, and can present the user with a high-quality image with further reduced delay.
  • the multi-viewpoint conversion unit 120 of the server device 100 determines the viewpoint position to be included in the viewpoint buffer information according to the type and title of the game (an example of the content including the rendered image).
  • the server device 100 can generate a viewpoint conversion image according to the user's operation or movement peculiar to the content, and can present the user with a high-quality image with further reduced delay.
  • the multi-viewpoint conversion unit 120 of the server device 100 determines the viewpoint position to be included in the viewpoint buffer information according to the game play situation (an example of the scene of the rendered image).
  • the server device 100 can generate a viewpoint conversion image that can correspond to the user's operation and movement according to the scene of the content (for example, the play situation), and the user can obtain a high-quality image with further reduced delay. Can be presented to.
  • the rendering information according to the embodiment of the present disclosure is run-time metadata (an example of information generated when rendering a rendered image).
  • the server device 100 can generate a viewpoint conversion image with higher image quality by generating the viewpoint conversion image using the runtime metadata.
  • the rendering information according to the embodiment of the present disclosure is production-time metadata (an example of information preset for rendering a rendered image).
  • the server device 100 can generate a viewpoint-converted image with higher image quality by generating the viewpoint-converted image using the metadata at the time of production.
  • the multi-viewpoint conversion unit 120 of the server device 100 calculates the viewpoint density function (an example of the viewpoint function representing the distribution of the viewpoint position), the line-of-sight function, and the calculated viewpoint density function.
  • the viewpoint buffer information is generated using the line-of-sight function.
  • the transmission / reception unit 140 of the server device 100 transmits information regarding the viewpoint density function and the line-of-sight function as viewpoint buffer information.
  • the server device 100 can reduce the amount of information of the viewpoint buffer information.
  • the client device 200 (an example of an information processing device) according to the embodiment of the present disclosure includes a transmission / reception unit 210 (an example of a reception unit) and a decoding unit 220 (an example of a generation unit).
  • the receiving unit 210 is a viewpoint converted image generated based on the rendered image seen from the viewpoint position and the line-of-sight direction included in the rendered viewpoint information and the rendering incidental information, and the viewpoint position and the line of sight included in the converted viewpoint information.
  • the viewpoint buffer including the viewpoint conversion image viewed from the direction and the viewpoint buffer information including the conversion viewpoint information are received.
  • the decoding unit 220 generates a display image based on the viewpoint conversion image and the viewpoint buffer information.
  • the client device 200 can present the user with a high-quality image with further reduced delay, and can further suppress the deterioration of the quality of the service provided to the user.
  • the decoding unit 220 of the client device 200 selects and selects at least one viewpoint position and line-of-sight direction included in the viewpoint buffer information based on the user's current viewpoint position and line-of-sight direction.
  • the display image is generated based on the viewpoint conversion image viewed from the viewpoint position and the line-of-sight direction.
  • the client device 200 can present a high-quality image to the user while reducing the number of viewpoint-converted images to be decoded.
  • the information processing system 10 includes a server device 100 and a client device 200.
  • the server device 100 includes a multi-viewpoint conversion unit 120 and a transmission / reception unit 140.
  • the multi-viewpoint conversion unit 120 acquires the rendered image as seen from the viewpoint position and the line-of-sight direction included in the rendered viewpoint information, and the rendered image incidental information.
  • the multi-viewpoint conversion unit 120 generates viewpoint buffer information.
  • the multi-viewpoint conversion unit 120 generates a viewpoint conversion image seen from the viewpoint position and the line-of-sight direction included in the decoded viewpoint information based on the rendered image and the rendering incidental information.
  • the transmission / reception unit 140 transmits the viewpoint buffer including the viewpoint buffer information and the viewpoint conversion image to the client device 200.
  • the client device 200 includes a transmission / reception unit 210 and a decoding unit 220.
  • the transmission / reception unit 210 receives the viewpoint conversion image and the viewpoint buffer information.
  • the decoding unit 220 generates a display image based on the viewpoint conversion image and the viewpoint buffer information.
  • the information processing system 10 can present the user with a high-quality image with further reduced delay, and can further suppress the deterioration of the quality of the service provided to the user.
  • the present technology can also have the following configurations.
  • (1) The first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image are acquired, and the rendering information is acquired.
  • the line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number, is generated.
  • a generation unit that generates a second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number based on the first rendered image and the rendered information.
  • a transmission unit that transmits the line-of-sight information and the second rendered image to the terminal device, and Information processing device equipped with.
  • the information processing apparatus according to any one of (1) to (5), wherein the rendering information is information generated when the first rendered image is rendered.
  • the rendering information is information preset for rendering the first rendered image.
  • the generation unit calculates a viewpoint function representing the distribution of the viewpoint position and a line-of-sight function representing the line-of-sight direction according to the viewpoint position, and uses the calculated viewpoint function and the line-of-sight function to obtain the second line-of-sight function. Generate the viewpoint position and the line-of-sight direction of the number, The transmitting unit transmits information about the viewpoint function and the line-of-sight function as the line-of-sight information.
  • the information processing apparatus according to any one of (1) to (7).
  • (9) The first rendered image viewed from the viewpoint position and the line-of-sight direction of the first number, and the second rendered image generated based on the rendering information used for rendering the first rendered image. Reception of receiving a second rendered image viewed from the viewpoint position and the line-of-sight direction of a second number larger than the first number, and line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number.
  • Department and A generation unit that generates a display image based on the second rendered image and the line-of-sight information, Information processing device equipped with.
  • the generation unit selects at least one of the viewpoint positions and the line-of-sight directions included in the line-of-sight information based on the user's current viewpoint position and the line-of-sight direction, and views from the selected viewpoint position and the line-of-sight direction.
  • the information processing apparatus according to (9), which generates the display image based on the second rendered image.
  • An information processing system including an information processing device and a terminal device.
  • the information processing device is The first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image are acquired, and the rendering information is acquired.
  • the line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number, is generated.
  • a generation unit that generates a second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number based on the first rendered image and the rendered information.
  • a transmission unit that transmits the line-of-sight information and the second rendered image, and Equipped with The terminal device is A receiving unit that receives the second rendered image and the line-of-sight information.
  • a generation unit that generates a display image based on the second rendered image and the line-of-sight information, Information processing system equipped with.
  • the first rendered image viewed from the first number of viewpoint positions and the line-of-sight direction, and the rendering information used for rendering the first rendered image are acquired, and the rendering information is acquired.
  • the line-of-sight information regarding the viewpoint position and the line-of-sight direction of the second number, which is larger than the first number, is generated.
  • a second rendered image viewed from the viewpoint position and the line-of-sight direction of the second number is generated.
  • the line-of-sight information and the second rendered image are transmitted to the terminal device. Information processing method.
  • Information processing system 100 Server equipment 110 Rendering unit 120 Multi-viewpoint conversion unit 130 Encoding unit 140, 210 Transmission / reception unit 150 Command information acquisition unit 200 Client equipment 220 Decoding unit 230 Input information acquisition unit 300 Peripheral devices

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Image Generation (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un dispositif de traitement d'informations (100) qui comprend une unité de génération (120) et une unité de transmission (140). L'unité de génération (120) acquiert une première image de rendu, vue à partir de chacune, d'un premier nombre, de positions de points de vue et de directions de lignes visuelles, et des informations de rendu utilisées dans le rendu de la première image de rendu. L'unité de génération (120) génère des informations de lignes visuelles concernant un second nombre de positions de points de vue et de directions de lignes visuelles, ce second nombre étant supérieur au premier nombre. L'unité de génération (120) génère une seconde image de rendu, vue à partir de chacune, d'un second nombre, de positions de points de vue et de directions de lignes visuelles, sur la base de la première image de rendu et des informations de rendu. L'unité de transmission (140) transmet, à un dispositif terminal (200), les informations de lignes visuelles et la seconde image de rendu.
PCT/JP2021/042524 2020-12-07 2021-11-19 Dispositif de traitement d'informations, système de traitement d'informations et procédé de traitement d'informations WO2022124047A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020202607 2020-12-07
JP2020-202607 2020-12-07

Publications (1)

Publication Number Publication Date
WO2022124047A1 true WO2022124047A1 (fr) 2022-06-16

Family

ID=81974406

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/042524 WO2022124047A1 (fr) 2020-12-07 2021-11-19 Dispositif de traitement d'informations, système de traitement d'informations et procédé de traitement d'informations

Country Status (1)

Country Link
WO (1) WO2022124047A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020010301A (ja) * 2018-07-12 2020-01-16 キヤノン株式会社 情報処理装置、情報処理装置の制御方法及びプログラム
JP2020095741A (ja) * 2020-03-03 2020-06-18 キヤノン株式会社 画像処理装置、画像処理システム、画像処理方法及びプログラム
JP2021114216A (ja) * 2020-01-21 2021-08-05 Kddi株式会社 情報システム、端末、サーバ及びプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020010301A (ja) * 2018-07-12 2020-01-16 キヤノン株式会社 情報処理装置、情報処理装置の制御方法及びプログラム
JP2021114216A (ja) * 2020-01-21 2021-08-05 Kddi株式会社 情報システム、端末、サーバ及びプログラム
JP2020095741A (ja) * 2020-03-03 2020-06-18 キヤノン株式会社 画像処理装置、画像処理システム、画像処理方法及びプログラム

Similar Documents

Publication Publication Date Title
US11909984B2 (en) Video encoding and decoding for cloud gaming
US11196973B2 (en) Providing apparatus, providing method and computer readable storage medium for performing processing relating to a virtual viewpoint image
TW202134890A (zh) 用於應用計算的分佈的方法和裝置
TW201921921A (zh) 基於紋理圖與網格之３ｄ影像資訊的處理
EP4037789A1 (fr) Application de jeu fournissant une indication de changement de scène destinée à être codée au niveau d'un serveur de jeu en nuage
US8860720B1 (en) System and method for delivering graphics over network
Joachimczak et al. Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors
JP6306089B2 (ja) ネットワークを通してメディアを伝送するシステム及び方法
KR102441514B1 (ko) 하이브리드 스트리밍
CN111464828A (zh) 虚拟特效显示方法、装置、终端及存储介质
CN110663067A (zh) 用于生成现实世界场景的定制视图的虚拟化投影以包括在虚拟现实媒体内容中的方法和系统
WO2022124047A1 (fr) Dispositif de traitement d'informations, système de traitement d'informations et procédé de traitement d'informations
US20220084282A1 (en) Image processing device, image generation method, and image processing method
JPWO2014171066A1 (ja) 三次元画像表示システム、三次元画像表示システムのサーバ及び三次元画像表示方法
US20220321861A1 (en) Method and apparatus for providing conversational services in mobile communication system
WO2023079987A1 (fr) Dispositif de distribution, procédé de distribution et programme
CN112930556A (zh) 集成用于可视化大数据集的第三方几何结构的系统和方法
KR102674943B1 (ko) 하이브리드 스트리밍
US12034787B2 (en) Hybrid streaming
US8897821B2 (en) Method for providing visual effect messages and associated communication system and transmitting end
US20230316656A1 (en) Information processing device, information processing method, and program
WO2022224964A1 (fr) Dispositif de traitement d'informations et procédé de traitement d'informations
KR20240083907A (ko) Vr 모션 예측 장치
EP3564905A1 (fr) Convertissement d'un objet volumetrique dans une scène 3d vers un modèle de représentation plus simple
KR20240052837A (ko) 몰입형 미디어를 위한 스트리밍 장면 우선순위화기

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21903151

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21903151

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP