WO2023029845A1 - 渲染方法、远端装置、计算设备集群、终端装置及设备 - Google Patents

渲染方法、远端装置、计算设备集群、终端装置及设备 Download PDF

Info

Publication number
WO2023029845A1
WO2023029845A1 PCT/CN2022/109107 CN2022109107W WO2023029845A1 WO 2023029845 A1 WO2023029845 A1 WO 2023029845A1 CN 2022109107 W CN2022109107 W CN 2022109107W WO 2023029845 A1 WO2023029845 A1 WO 2023029845A1
Authority
WO
WIPO (PCT)
Prior art keywords
rendered image
state information
information
terminal device
rendered
Prior art date
Application number
PCT/CN2022/109107
Other languages
English (en)
French (fr)
Inventor
张文晓
戴国辉
Original Assignee
华为云计算技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为云计算技术有限公司 filed Critical 华为云计算技术有限公司
Publication of WO2023029845A1 publication Critical patent/WO2023029845A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/04—Texture mapping
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects

Definitions

  • the present application relates to the technical field of image rendering, and in particular to a rendering method, remote device, computing device cluster, terminal device and equipment.
  • Virtual reality is a technology that simulates a three-dimensional virtual world through a computer and provides images to users so that users can experience the virtual world in three-dimensional space through images.
  • Augmented reality is a technology that integrates images obtained from virtual information into the real world through computers. When the user's viewing angle or position changes, the computer will perform complex calculations and image rendering, and provide the user with images under the new viewing angle or position, so that the user can experience virtual reality/augmented reality personally.
  • the existing mainstream solution is that the collection of observation angles or positions, image rendering at new angles of view or new positions, and displaying images at new angles of view or new positions to users are all completed by VR/AR devices.
  • a rendering method a remote device, a computing device cluster, a terminal device, and a device are proposed.
  • the bandwidth consumption of data transmission between the server and the VR/AR device can be reduced, so that Data transfer costs are reduced.
  • the embodiment of the present application provides a rendering method, the method is applied to a remote device, the rendering system where the remote device is located also includes a terminal device, and the remote device runs the server of the application,
  • the application includes a plurality of three-dimensional models, and the method includes: receiving first state information, the first state information including position information and posture information of the terminal device at a first moment; according to the first state information and At least one 3D model is rendered, and a first rendered image and depth information of the first rendered image are obtained; and the first rendered image and the depth information of the first rendered image are sent to the terminal device.
  • the remote device runs the server of the application, and the application includes multiple 3D models, so that the remote device can use at least one 3D model for rendering; the remote device uses at least one 3D model and received The received first state information can be rendered to obtain the first rendered image and the depth information of the first rendered image, and the remote device sends the first rendered image and the depth information of the first rendered image to the terminal device so that the terminal device can use the first The depth information of the rendered image and the first rendered image obtains multiple frames of images that can be displayed to the user.
  • the remote device only needs to transmit the first rendered image and the depth information of the first rendered image once, and can obtain multiple frames of images correspondingly on the terminal device, so that the remote device transmits the first rendered image and the depth information of the first rendered image.
  • the frequency is reduced.
  • the remote device corresponds to the cloud server and the terminal device corresponds to the VR/AR device
  • the bandwidth consumption of data transmission between the cloud and the VR/AR device can be reduced, reducing the cost of data transmission.
  • the first rendered image may include one frame of image.
  • executing the rendering method of the embodiment of the present application can enable the remote device to output the first rendered image in the cloud AR scene, so that when the terminal device receives the first rendered image Can make a response to realize the image display of the display area, and can enable the remote device to output the first rendered image in the cloud VR scene, so that the terminal device can respond when receiving the first rendered image to realize the corresponding corresponding to the user's left eye and An image of at least one of the two display areas for the right eye is displayed.
  • the first rendered image may include two frames of images.
  • executing the rendering method of the embodiment of the present application can enable the remote device to output the first rendered image in the cloud VR scene, so that when the terminal device receives the first rendered image It is possible to respond to realize image display of the two display areas respectively corresponding to the user's left and right eyes.
  • the sending the first rendered image and the depth information of the first rendered image to the terminal device includes: rate to send the first rendered image and the depth information of the first rendered image to the terminal device; after receiving the first state information, the method further includes: receiving second state information, the second The state information includes position information and posture information of the terminal device at a second moment after the first moment; performing rendering according to the second state information and the at least one three-dimensional model to obtain a second rendered image; at a low The second rendered image is sent to the terminal device at a second bit rate of the first bit rate.
  • the second state information includes the position information and posture information of the terminal device at a second moment after the first moment, and the state information is also used to obtain a second rendered image, so the second rendered image corresponds to the rendered image at the second moment image, by transmitting the second rendered image at a lower second bit rate, so that the second rendered image has a smaller amount of data, and the terminal device can obtain the rendered image corresponding to the first moment on the basis of The rendered image at the second moment can reduce the downlink transmission data cost of the remote device while ensuring that the user has a better experience in using the terminal device.
  • the second rendered image may include one frame of image.
  • executing the rendering method of the embodiment of the present application can enable the remote device to output the first rendered image and the second rendered image in the cloud AR scene, so that the terminal device can receive When the first rendered image and the second rendered image can respond to realize the image display of a display area corresponding to the user's left eye and right eye, and enable the remote device to output the first rendered image and the second rendered image in the cloud VR scene
  • the images are rendered, so that the terminal device can respond when receiving the first rendered image and the second rendered image to realize image display in at least one of the two display areas respectively corresponding to the left eye and the right eye of the user.
  • the second rendered image includes two frames of rendered images, and the virtual viewpoint corresponding to the two frames of rendered images The distance between them is equal to the viewing distance.
  • the second rendered image may include two frames of images.
  • executing the rendering method of the embodiment of the present application can enable the remote device to output the first rendered image and the second rendered image in the cloud VR scene, so that the terminal device can receive
  • the first rendered image and the second rendered image can respond, so as to realize the image display in the two display areas respectively corresponding to the left eye and the right eye of the user, and at the same time ensure the accuracy of the image corresponding to the second state information among the images displayed in the display area quality.
  • the first rendered image is used to provide the user with a left-eye or right-eye corresponding
  • the method further includes: performing rendering according to the first state information and the at least one 3D model to obtain another rendered image, the virtual viewpoint corresponding to the first rendered image and the other rendered image
  • the distance between virtual viewpoints corresponding to a rendered image is equal to the viewing distance; the other rendered image is sent to the terminal device at a second bit rate lower than the first bit rate, and the first rendered image is based on the The first code rate is sent to the terminal device.
  • the execution of the rendering method of the embodiment of the present application enables the remote device to output the first rendered image and the other rendered image in the cloud VR scene, so that the terminal device receives the first rendered image and the other rendered image Responses can be made to realize image display in the two display areas respectively corresponding to the user's left eye and right eye, while ensuring the quality of the image corresponding to the first state information among the images displayed in the display area.
  • the method further includes: determining the difference between the last rendered image sent to the terminal device and the first rendered image Interval a specified number of images; or, determine that the bandwidth margin for transmitting data to the terminal device at the moment when the first rendered image is obtained is greater than or equal to a bandwidth threshold.
  • the specified number can be preset according to the bandwidth upper limit and the bandwidth threshold of data transmitted from the remote device to the terminal device, so that the first rendered image is sent at intervals of the specified number of images, without causing the remote device to The bandwidth margin for data transmission to the terminal device is less than the bandwidth threshold.
  • the remote device by first determining the interval of a specified number of images between the last rendered image sent to the terminal device and the rendered image, and then converting the first rendered image and the second rendered image at a first code rate
  • the depth information of a rendered image is sent to the terminal device, so that the remote device will not output the first rendered image and the depth information of the first rendered image when the bandwidth margin is too small, which can keep the remote device from transmitting data information to the terminal device smoothness, and image display on the terminal device will not be delayed due to the first rendered image and the depth information of the first rendered image not being received in time.
  • the bandwidth margin for transmitting data to the terminal device at the moment when the rendered image is obtained is greater than or equal to a bandwidth threshold, and then sending the first rendered image and the depth information of the first rendered image at a first bit rate To the terminal device, so that the remote device will not output the first rendered image and the depth information of the first rendered image when the bandwidth margin is too small, the smoothness of data information transmission from the remote device to the terminal device can be maintained, and the terminal device The display of the image on will not be delayed due to the first rendered image and the depth information of the first rendered image not being received in time.
  • the sending of the first rendered image and the first rendered image Sending the depth information of the first rendered image to the terminal device includes: synthesizing the first rendered image and the depth information of the first rendered image into a rendered frame; and sending the rendered frame to the terminal device.
  • the rendering according to the first state information and at least one 3D model Before obtaining the first rendered image and the depth information of the first rendered image, the method further includes: determining the specified number of intervals between the state information corresponding to the last rendered image sent to the terminal device and the first state information or, determining that the bandwidth margin for data transmission to the terminal device at the time of receiving the first state information is greater than or equal to the bandwidth threshold.
  • the code rate is to send the first rendered image and the depth information of the first rendered image to the terminal device, so that the remote device will not output the first rendered image and the depth of the first rendered image when the bandwidth margin is too small
  • the information can maintain the smoothness of data information transmission from the remote device to the terminal device, and the image display on the terminal device will not be delayed because the first rendered image and the depth information of the first rendered image are not received in time.
  • the first rendered image is rendered at the first code rate
  • the image and the depth information of the first rendered image are sent to the terminal device, so that the remote device will not output the first rendered image and the depth information of the first rendered image when the bandwidth margin is too small, and the remote device can keep sending
  • the terminal device transmits data information smoothly, and image display on the terminal device will not be delayed because the first rendered image and the depth information of the first rendered image are not received in time.
  • an embodiment of the present application provides a rendering method, the method is applied to a terminal device, the rendering system where the terminal device is located also includes a remote device, the terminal device runs a client of an application, and the The application includes a plurality of three-dimensional models, and the method includes: sending first state information to the remote device, the first state information including position information and attitude information of the terminal device at a first moment; receiving the remote device The first rendered image sent by the end device and the depth information of the first rendered image, the first rendered image is obtained by the remote device through rendering according to the first state information and at least one 3D model; according to the The first rendered image and the depth information of the first rendered image are used to obtain a multi-frame rendered image.
  • the terminal device sends the first state information including the position information and posture information of the terminal device at the first moment to the remote device, so that the remote device State information, which can be rendered to obtain the first rendered image and the depth information of the first rendered image, and output to the terminal device, so that the terminal device can receive and use the first rendered image and the depth information of the first rendered image to obtain multiple frames Render the image.
  • the terminal device only needs to transmit the status information once, and the remote device only needs to transmit the first rendered image and the depth information of the first rendered image once, and then the terminal device can obtain multiple frames of rendered images correspondingly, so that the remote device transmits the first
  • the frequency of the depth information of the rendered image and the first rendered image is reduced.
  • the first rendered image may include one frame of image.
  • executing the rendering method of the embodiment of the present application can enable the terminal device to respond when receiving the first rendered image in the cloud AR scene to realize image display in the display area. , and enable the remote device to output the first rendered image in the cloud VR scene, so that the terminal device can respond when receiving the first rendered image to realize at least An image display of the display area.
  • the first rendered image may include two frames of images.
  • executing the rendering method of the embodiment of the present application can enable the terminal device to respond when receiving the first rendered image in the cloud VR scene so as to respectively correspond to the left eye of the user.
  • the images of the two display areas of the right eye and the right eye are displayed.
  • the obtaining multi-frame rendered images according to the first rendered image and the depth information of the first rendered image includes: according to the first rendered image A rendered image, depth information of the first rendered image, and second state information, obtaining a rendered image corresponding to the second state information, the second state information including the state of the terminal device after the first moment Position information and attitude information at the second moment.
  • the rendered image corresponding to the second state information can be obtained according to the second state information, and the second state information includes the position information and posture information of the terminal device at the second moment after the first moment, therefore, there are many
  • the rendered images corresponding to multiple different second state information can be obtained.
  • the rendered image corresponding to the different second state information can meet the requirement of presenting the rendered image conforming to the user's action to the user, so as to ensure the user experience.
  • the Depth information obtaining multiple frames of rendered images includes: obtaining another rendered image corresponding to the first state information according to the first rendered image and the depth information of the first rendered image, the first rendered image and the depth information of the first rendered image The distance between the virtual viewpoints corresponding to another rendered image is equal to the viewing distance.
  • the execution of the rendering method of the embodiment of the present application enables the terminal device to respond when receiving the first rendered image in the cloud VR scene, so as to use the first rendered image and another rendered image to correspond to the user's left and right images respectively.
  • obtaining a rendered image corresponding to the second state information includes: generating point cloud data according to the first rendered image and depth information of the first rendered image; generating point cloud data according to the second state information and the point The cloud data is rendered to obtain a rendered image corresponding to the second state information.
  • the point cloud data may include a plurality of three-dimensional points, which may represent information such as the position and shape of the object in the first rendered image in the three-dimensional scene.
  • the points that the user can see when observing the 3D scene under the second state information can be obtained, and according to the obtained points and the first rendered image, it can be obtained that under the second state information, when the user observes the 3D scene images that can be seen.
  • An image for presentation to the user can thus be obtained.
  • obtaining the rendered image corresponding to the second state information includes: determining a plurality of two-dimensional line segments according to the second state information, a preset near clipping plane and a far clipping plane; The plurality of two-dimensional line segments are projected to obtain a plurality of two-dimensional coordinates, wherein each two-dimensional line segment corresponds to a plurality of two-dimensional coordinates; obtaining depth information of the plurality of two-dimensional coordinates; The depth information of the coordinates is compared with the depth information of the first rendered image to determine the coordinates of points with the same depth value, wherein the depth information includes the depth values corresponding to each coordinate; according to the coordinates of the points with the same depth value Obtain a rendered image corresponding to the second state information based on the first rendered image.
  • the obtained Another rendered image corresponding to the first state information includes: generating point cloud data according to the first rendered image and depth information of the first rendered image; The point cloud data is rendered, and another rendered image corresponding to the first state information is obtained.
  • the obtained Another rendered image corresponding to the first state information includes: determining the position of the virtual viewpoint according to the first state information and the viewing distance; The line segments between the planes determine a plurality of two-dimensional line segments; the plurality of two-dimensional line segments are projected onto the viewing plane corresponding to the virtual viewpoint to obtain a plurality of two-dimensional coordinates, wherein each two-dimensional line segment corresponds to a plurality of two-dimensional line segments two-dimensional coordinates; obtain the depth information of the multiple two-dimensional coordinates; compare the depth information of the multiple two-dimensional coordinates with the depth information of the first rendered image, and determine the coordinates with the same depth value, wherein the The depth information includes a depth value corresponding to each coordinate; another rendered image corresponding to the first state information is obtained according to texture information of coordinates with the same depth value in the first rendered image.
  • the embodiment of the present application provides a rendering method, the method is applied to a terminal device, the rendering system where the terminal device is located also includes a remote device, the terminal device runs the client of the application, the The application includes a plurality of three-dimensional models, and the method includes: sending first state information to the remote device, the first state information including position information and attitude information of the terminal device at a first moment; receiving the remote device The first rendered image sent by the end device and the depth information of the first rendered image, the first rendered image is obtained by the remote device through rendering according to the first state information and at least one 3D model; sending the first rendered image Send state information to the remote device, the second state information includes position information and attitude information of the terminal device at a second moment after the first moment; receive a second rendered image, and the second rendered image The image is obtained by the remote device through rendering according to the second state information and the at least one three-dimensional model; according to the first rendered image, the depth information of the first rendered image, and the second state information and the second rendered image to obtain a
  • the terminal device sends the first state information including the position information and posture information of the terminal device at the first moment to the remote device, so that the remote device State information, which can be rendered to obtain the first rendered image and the depth information of the first rendered image, and output to the terminal device, and the terminal device sends the first state information including the position information and posture information of the terminal device at the second moment to the remote device , so that the remote device can render the second rendered image according to at least one 3D model and the received second state information and output it to the terminal device, so that the terminal device can receive and use the first rendered image, the first rendered image
  • the depth information of the second rendered image and the second state information on the terminal device are used to obtain a multi-frame rendered image. In this way, the quality of the obtained multi-frame rendered image can be guaranteed to be good, and the user experience can be improved when the multi-frame rendered image is displayed to the user.
  • obtaining multi-frame rendered images includes: generating point cloud data according to the first rendered image and depth information of the first rendered image; rendering the point cloud data according to the second state information, Obtain a first intermediate rendered image; use the second rendered image to fill the first intermediate rendered image, and obtain a second rendered image corresponding to the second state information.
  • the point cloud data may include a plurality of three-dimensional points, which may represent information such as the position and shape of the object in the first rendered image in the three-dimensional scene.
  • the points that the user can see when observing the 3D scene under the second state information can be obtained, and according to the obtained points and the first rendered image, it can be obtained that under the second state information, when the user observes the 3D scene
  • the image that can be seen, the first intermediate rendered image is also obtained according to the second state information, based on the second rendered image, the first intermediate rendered image may be processed to obtain a second rendered image corresponding to the second state information. An image for presentation to the user can thus be obtained.
  • obtaining multi-frame rendering images includes: determining the position of the virtual viewpoint according to the second state information and the viewing distance; intercepting the connection line between the virtual viewpoint and the far clipping plane between the near clipping plane and the far clipping plane The line segment between, determine a plurality of two-dimensional line segments; Project the plurality of two-dimensional line segments to the viewing plane corresponding to the virtual viewpoint to obtain a plurality of two-dimensional coordinates, wherein each two-dimensional line segment corresponds to a plurality of two-dimensional coordinates ; Obtain depth information of the plurality of two-dimensional coordinates; compare the depth information of the plurality of two-dimensional coordinates with the depth information of the first rendered image, and determine coordinates with the same depth value, wherein the depth information Including the depth value corresponding to each coordinate; according to the texture information in the first rendered image at the coordinates with the same depth
  • the third possible implementation manner of the rendering method includes: receiving another rendered image sent by the remote device, and the other rendered image is the remote device Obtained by rendering according to the first state information and the at least one 3D model, the distance between the virtual viewpoint corresponding to the first rendered image and the other rendered image is equal to the viewing distance; according to the first rendered image and the depth information of the first rendered image to generate point cloud data; according to the first state information and the viewing distance, render the point cloud data to obtain a third intermediate rendered image; use the remote device to send Fill the third intermediate rendered image with another rendered image to obtain another rendered image corresponding to the first state information.
  • the first rendered image The depth information, the second state information and the second rendered image, and obtaining the multi-frame rendered image includes: determining the position of the virtual viewpoint according to the first state information and the viewing distance; intercepting the virtual viewpoint and the far clipping plane The line segment between the near clipping plane and the far clipping plane determines a plurality of two-dimensional line segments; the plurality of two-dimensional line segments are projected to the visual plane corresponding to the virtual viewpoint to obtain a plurality of two-dimensional coordinates , wherein each two-dimensional line segment corresponds to a plurality of two-dimensional coordinates; obtaining depth information of the plurality of two-dimensional coordinates; comparing the depth information of the plurality of two-dimensional coordinates with the depth information of the first rendered image , determine the coordinates with the same depth value, wherein the depth information includes the depth value corresponding to each coordinate; according to the texture information in the first rendered
  • the embodiment of the present application provides a remote device, where the rendering system of the remote device further includes a terminal device, and the remote device runs a server of an application, and the application includes multiple 3D models
  • the device includes: a first receiving module, configured to receive first state information, the first state information including position information and posture information of the terminal device at a first moment; a first rendering module, configured to Perform rendering of the first state information and at least one 3D model to obtain a first rendered image and depth information of the first rendered image; a first sending module, configured to send the first rendered image and the depth information of the first rendered image Depth information to the terminal device.
  • the sending the first rendered image and the depth information of the first rendered image to the terminal device includes: The code rate sends the first rendered image and the depth information of the first rendered image to the terminal device; after the first receiving module, the device further includes: a second receiving module, configured to receive second state information , the second state information includes position information and posture information of the terminal device at a second moment after the first moment; a second rendering module, configured to use the second state information and the at least one three-dimensional The model is rendered to obtain a second rendered image; a second sending module is configured to send the second rendered image to the terminal device at a second bit rate lower than the first bit rate.
  • the second rendered image includes two frames of rendered images, and the two frames of rendered images correspond to virtual The distance between viewpoints is equal to the viewing distance.
  • the first rendered image is provided to the user's left or right eye
  • the device further includes: a third rendering module, configured to perform rendering according to the first state information and the at least one 3D model, and obtain another rendered image, and the first rendered image The distance between the corresponding virtual viewpoint and the virtual viewpoint corresponding to the other rendered image is equal to the viewing distance; the third sending module is configured to send the other rendered image to the For the terminal device, the first rendered image is sent to the terminal device at the first bit rate.
  • the The device before the first sending module, the The device further includes: a first determination module, configured to determine a specified number of images between the last rendered image sent to the terminal device and the rendered image; The bandwidth allowance for the device to transmit data is greater than or equal to the bandwidth threshold.
  • the sending of the first rendered image and the first rendered image Sending the depth information of the image to the terminal device includes: synthesizing the first rendered image and the depth information of the first rendered image into a rendered frame; and sending the rendered frame to the terminal device.
  • the embodiment of the present application provides a terminal device, where the rendering system of the terminal device further includes a remote device, and the terminal device runs a client of an application, and the application includes multiple 3D models, so
  • the device includes: a fourth sending module, configured to send first status information to the remote device, the first status information including position information and attitude information of the terminal device at the first moment; a fourth receiving module, for receiving a first rendered image sent by the remote device and depth information of the first rendered image, where the first rendered image is performed by the remote device according to the first state information and at least one 3D model Obtained by rendering; a first image obtaining module, configured to obtain multiple frames of rendered images according to the first rendered image and the depth information of the first rendered image.
  • the obtaining the multi-frame rendered image according to the first rendered image and the depth information of the first rendered image includes: according to the first rendered image A rendered image, depth information of the first rendered image, and second state information, obtaining a rendered image corresponding to the second state information, the second state information including the state of the terminal device after the first moment Position information and attitude information at the second moment.
  • the Depth information obtaining multiple frames of rendered images includes: obtaining another rendered image corresponding to the first state information according to the first rendered image and the depth information of the first rendered image, the first rendered image and the depth information of the first rendered image The distance between the virtual viewpoints corresponding to another rendered image is equal to the viewing distance.
  • obtaining a rendered image corresponding to the second state information includes: generating point cloud data according to the first rendered image and depth information of the first rendered image; generating point cloud data according to the second state information and the point The cloud data is rendered to obtain a rendered image corresponding to the second state information.
  • the embodiment of the present application provides a terminal device, where the rendering system of the terminal device further includes a remote device, and the terminal device runs a client of an application, and the application includes multiple 3D models, so
  • the device includes: a fifth sending module, configured to send first status information to the remote device, the first status information including position information and attitude information of the terminal device at the first moment; a fifth receiving module, for receiving a first rendered image sent by the remote device and depth information of the first rendered image, where the first rendered image is performed by the remote device according to the first state information and at least one 3D model Obtained by rendering; a sixth sending module, configured to send second state information to the remote device, where the second state information includes position information and attitude of the terminal device at a second moment after the first moment information; a sixth receiving module, configured to receive a second rendered image, the second rendered image is obtained by the remote device through rendering according to the second state information and the at least one three-dimensional model; the second image is obtained A module, configured to obtain multiple frames of rendered images according to the first
  • obtaining multi-frame rendered images includes: generating point cloud data according to the first rendered image and depth information of the first rendered image; rendering the point cloud data according to the second state information, Obtain a first intermediate rendered image; use the second rendered image to fill the first intermediate rendered image, and obtain a second rendered image corresponding to the second state information.
  • an embodiment of the present application provides a computing device cluster, which is characterized in that it includes at least one computing device, each computing device includes a processor and a memory, and the processor of the at least one computing device is used to execute the first A rendering method of one or more of the multiple possible implementations of the aspect or the first aspect.
  • an embodiment of the present application provides a terminal device, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to implement the above-mentioned second A rendering method of one or more of the multiple possible implementations of the aspect or the second aspect.
  • an embodiment of the present application provides a terminal device, including: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to implement the above third method when executing the instructions.
  • the embodiment of the present application provides a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned first aspect or multiple aspects of the first aspect are realized.
  • the embodiments of the present application provide a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium bearing computer-readable codes, when the computer-readable codes are stored in
  • the processor in the electronic device executes the rendering method of the first aspect or one or more of the multiple possible implementation manners of the first aspect.
  • FIG. 1 shows a schematic diagram of a cloud VR/AR system in the prior art.
  • Fig. 2a shows a schematic diagram of the working mode of the cloud VR system in the prior art.
  • Fig. 2b shows a schematic diagram of the working mode of the cloud AR system in the prior art.
  • Fig. 3 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud AR system according to an embodiment of the present application.
  • Fig. 4 shows an example of obtaining the first rendered image and the depth information of the first rendered image according to the three-dimensional model and the first state information according to the embodiment of the present application.
  • FIG. 5 shows an example of a first rendered image and a rendered frame synthesized with depth information of the first rendered image according to an embodiment of the present application.
  • Fig. 6 shows an example of determining point cloud data according to an embodiment of the present application.
  • Fig. 7 shows an example of determining multiple line segments in the world coordinate system according to the second state information.
  • Fig. 8 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud AR system according to an embodiment of the present application.
  • FIG. 9 shows an example of hole points appearing in the first intermediate rendered image according to an embodiment of the present application.
  • FIG. 10 shows an example of supplementing hole points in a second rendered image according to an embodiment of the present application.
  • FIG. 11 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud VR system according to an embodiment of the present application.
  • Fig. 12 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud VR system according to an embodiment of the present application.
  • Fig. 13 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud VR system according to an embodiment of the present application.
  • FIG. 14 shows an example of a transmission manner of a second rendered image according to an embodiment of the present application.
  • FIG. 15 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud VR system according to an embodiment of the present application.
  • Fig. 16 shows an exemplary schematic diagram of a rendering method according to an embodiment of the present application.
  • Fig. 17 shows an exemplary schematic diagram of a rendering method according to an embodiment of the present application.
  • Fig. 18 shows an exemplary schematic diagram of a rendering method according to an embodiment of the present application.
  • Fig. 19 shows an exemplary structural diagram of a remote device according to an embodiment of the present application.
  • FIG. 20 shows an exemplary structural diagram of a terminal device according to an embodiment of the present application.
  • Fig. 21 shows an exemplary structural diagram of a terminal device according to an embodiment of the present application.
  • Fig. 22 shows an exemplary structural diagram of a computing device cluster according to an embodiment of the present application.
  • Fig. 23 shows an exemplary structural diagram of a terminal device according to an embodiment of the present application.
  • FIG. 1 shows a schematic diagram of a cloud VR/AR system in the prior art.
  • the cloud VR/AR system of prior art is introduced below in conjunction with Fig. 1.
  • a cloud VR/AR system in the prior art includes a cloud server and a VR/AR device, and the cloud server and the VR/AR device perform data transmission through the Internet.
  • the cloud server may, for example, be provided with a processor, which may be a central processing unit (central processing unit, CPU), a graphics processing unit (graphics processing unit, GPU) and other types of processors that support image rendering.
  • a posture sensor such as a gyroscope
  • a display device such as a display device (not shown) corresponding to the left eye, may be provided on the VR/AR device.
  • VR/AR runtime VR Runtime/AR Runtime, the operating environment of VR/AR applications
  • VR/AR applications VR/AR applications
  • server programs are deployed on the cloud server
  • client programs are deployed on VR/AR devices.
  • other programs or applications the interaction between the cloud server and the VR/AR device is realized through the data exchange between the server program and the client program.
  • the cloud VR/AR system in FIG. 1 can be used as an exemplary application scenario of the rendering method of the embodiment of the present application.
  • this application scenario for the specific manner of executing the rendering method of the embodiment of the present application, reference may be made to the relevant descriptions in FIG. 3-FIG. 15 below.
  • Fig. 2a shows a schematic diagram of the working mode of the cloud VR system in the prior art.
  • the working mode of the cloud VR system in the prior art will be introduced below with reference to FIG. 2a.
  • the client program will actively establish a connection with the server program of the cloud server, and collect gestures at a certain frequency.
  • the client program transmits the posture information G1 to the server program.
  • the server program will transmit the attitude information G1 to the VR runtime, and the VR runtime will forward the attitude information G1 to the VR application.
  • the VR application will perform image rendering according to the posture information G1, and obtain two 2D images suitable for the posture information G1 according to the 3D data (such as the model in the 3D world pre-set on the VR application). eye and right eye.
  • the VR application transmits the rendered two 2D images to the server program.
  • the server program will synthesize a two-dimensional image according to the left and right eye images, so that the left half of the synthesized image includes the left eye image, and the right half includes the right eye image.
  • the server program can encode the synthesized image, for example, by using video encoding to obtain the corresponding encoding result P1, and transmit the encoding result P1 to the client program on the VR device.
  • the client program After the client program receives the encoding result P1, it decodes the image according to the encoding result P1, for example, by using video decoding to obtain a two-dimensional image in which the left half includes the left-eye image and the right half includes the right-eye image.
  • the two-dimensional image is rendered to obtain a left-eye image and a right-eye image, and the left-eye image and the right-eye image are respectively submitted to the display buffers of the display screens corresponding to the left eye and the right eye.
  • the two-dimensional image data in the display buffer is read by the controller (not shown), and the controller controls the display screen (not shown) to display the image, and draws the image data on the display screen.
  • the user can watch the image content through the display screens corresponding to the left eye and the right eye.
  • the images seen by the left and right eyes are different.
  • the objects in the two images will have position deviations, which is called parallax.
  • the user's brain will calculate the distance between the object and the body according to the parallax, thus creating a sense of depth, so that the user can experience the three-dimensional world under the posture information G1 personally.
  • Fig. 2b shows a schematic diagram of the working mode of the cloud AR system in the prior art.
  • the client program will actively establish a connection with the server program of the cloud server, and collect gestures at a certain frequency.
  • the client program transmits the attitude information G1 to the server program.
  • the server program will transmit the attitude information G1 to the AR runtime, and the AR runtime will forward the attitude information G1 to the AR application.
  • the AR application will perform image rendering according to the attitude information G1, and obtain a two-dimensional image suitable for the attitude information G1 according to the three-dimensional data (such as the model in the three-dimensional world preset on the AR application).
  • the AR application transmits the rendered 2D image to the server program.
  • the server program can encode the two-dimensional image, for example, by using video encoding to obtain the corresponding encoding result P1, and transmit the encoding result P1 to the client program on the AR device.
  • the client program After receiving the encoding result P1, the client program decodes the image according to the encoding result P1, for example, by using video decoding to obtain the two-dimensional image, renders the two-dimensional image, and submits it to the display buffer.
  • the two-dimensional image data in the display buffer is read by the controller (not shown), and the controller controls the image to be displayed on the display area. In this case, the user can watch the virtual image blended with the real world in the display area.
  • the disadvantage of the cloud VR/AR system in the prior art is that the data transmission between the cloud server and the VR/AR device is mainly completed through the Internet.
  • the resolution of the image content displayed on the VR/AR device is generally Above 3K (3000 ⁇ the number of horizontal pixels of the image ⁇ 3500)
  • the frame rate of the image is usually above 60 frames per second
  • the minimum network bandwidth requirement is 30Mbps. Therefore, using the existing cloud VR/AR system, if users want to experience the services brought by cloud VR/AR smoothly, they usually use high-bit-rate video streaming to transmit content, which requires a network bandwidth environment higher than 30 Mbps, for example, 50 Mbps. -100Mbps. This will lead to excessive bandwidth consumption of the system, which is not conducive to controlling the cost of data transmission between devices.
  • the present application proposes a rendering method, a remote device, a computing device cluster, a terminal device and equipment. According to the rendering method of the present application, the bandwidth consumption of data transmission between the cloud server and the VR/AR device can be reduced, so that Data transfer costs are reduced.
  • executing the rendering method of the present application can realize that the image is displayed to the left and right eyes of the user on the AR device.
  • executing the rendering method of the application For the specific manner of the rendering method of the embodiment, reference may be made to the relevant descriptions in FIG. 3 to FIG. 10 below.
  • the rendering method of the present application may be executed to display an image to the user's left eye (or right eye) on the VR device.
  • the rendering method of the present application may be executed to display an image to the user's left eye (or right eye) on the VR device.
  • the rendering method of the present application may be executed to display images to the left and right eyes of the user on the VR device.
  • the rendering method of the present application may be executed to display images to the left and right eyes of the user on the VR device.
  • FIG. 11-FIG. 15 for the specific manner of executing the rendering method of the embodiment of the present application, reference may be made to the relevant descriptions in FIG. 11-FIG. 15 below.
  • the cloud server may include a remote device
  • the AR device may include a terminal device
  • the remote device may run the server of the application
  • the terminal device may run the client of the application
  • the application may include multiple 3D model
  • the application can be an AR application.
  • the remote device and the terminal device can cooperate to implement rendering.
  • Fig. 3 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud AR system according to an embodiment of the present application.
  • the remote device can perform steps S1-S3:
  • the first state information g1 may be acquired by the remote device from the terminal device.
  • the first state information g1 includes position information and posture information of the terminal device at the first moment, and it can be considered that the first state information g1 corresponds to the first moment.
  • the remote device runs the server of the application, and can obtain the 3D model included in the application.
  • the remote device may, for example, obtain at least one 3D model, and perform rendering according to the first state information g1 and the obtained at least one 3D model, to obtain a first rendered image x1 and depth information x1' of the first rendered image.
  • the first state information g1 is state information corresponding to the first rendered image x1. That is, the first moment, the first state information g1, and the first rendered image x1 have a corresponding relationship.
  • the image displayed to the user's left eye and right eye at the same time may be one frame of image, therefore, it may be preset that the first rendered image x1 includes one frame of rendered image.
  • Fig. 4 shows an example of obtaining the first rendered image and the depth information of the first rendered image according to the three-dimensional model and the first state information according to the embodiment of the present application.
  • rendering when rendering is performed according to the first state information g1 and at least one 3D model, rendering may be performed in combination with rendering parameters.
  • the rendering parameters may be, for example, parameters such as the aspect ratio of the display area of the terminal device, and the rendering parameters may be actively transmitted from the terminal device to the remote device.
  • the first rendered image x1 may include the texture information of the 3D model that can be seen by the user's left and right eyes under the first state information g1, and the depth information x1' of the first rendered image may include the first state information g1, and the user's left eye The depth information of the 3D model in the scene that the eye and the right eye can see.
  • the aspect ratio of the first rendered image x1 may be, for example, the same as the aspect ratio indicated by the rendering parameter.
  • the depth information x1' of the first rendered image may include depths corresponding to multiple pixels in the first rendered image x1 under the aspect ratio.
  • the remote device may send the first rendered image x1 to the terminal device, so as to display the first rendered image x1' on the terminal device.
  • the first rendered image x1 may correspond to the first state information g1.
  • the remote device may simultaneously send the depth information x1' of the first rendered image to the terminal device.
  • the first rendered image x1 and the depth information x1' of the first rendered image may also be used for rendering to obtain multiple frames of rendered images, where the multiple frames of rendered images may respectively correspond to, in addition to the first state information g1,
  • Other status information eg g2-g6 collected by the device where the terminal device is located (eg AR device).
  • the remote device may set a flag for the first rendered image x1 and the depth information x1' of the first rendered image, and the flag may represent state information corresponding to the first rendered image x1 and the depth information x1' of the first rendered image.
  • step S3 may also include S4-S5:
  • the first rendered image x1 may include the texture information of the 3D model
  • the depth information x1' of the first rendered image may include the depth information of the 3D model in the scene.
  • the bit precision of the texture information is usually 8 bits
  • the bit precision of the depth information is usually 32 bits, so they need to be transmitted separately, and this transmission method will occupy a relatively high bandwidth.
  • the embodiment of the present application proposes to process the 32-bit digit precision data so that it has the same number of digits as the texture information, so that the texture information and the depth information can be transmitted together to reduce bandwidth consumption.
  • FIG. 5 shows an example of a first rendered image and a rendered frame synthesized with depth information of the first rendered image according to an embodiment of the present application.
  • a certain mapping method can be set, so that according to the 32-bit depth information and the preset mapping method, the depth information x1' of the first rendered image is converted into a texture indicating the depth information, so that the The texture indicating depth information has the same data structure as the first rendered image x1.
  • the texture indicating the depth information can be integrated with the first rendered image x1 into one image, and the integrated image is the rendered frame.
  • the left half of the rendered frame may include the first rendered image x1
  • the right half of the rendered frame may include the texture indicating the depth information converted from the depth information x1' of the first rendered image.
  • the texture indicating the depth information and the first rendered image x1 can be synthesized into a rendered frame in various ways, for example, as the left half or right half of the rendered frame, or respectively as the upper half of the rendered frame Or the second half, as long as the first rendered image x1 and the texture indicating the depth information can be embodied on the rendered frame, the embodiment of the present application does not limit the specific positions of the two on the rendered frame.
  • the remote device may set an identifier for the rendered frame, and the identifier may indicate state information corresponding to the rendered frame, that is, state information corresponding to the first rendered image x1 and the depth information x1' of the first rendered image.
  • the first rendered image x1 and the depth information x1' of the first rendered image can be transmitted to the terminal device at the same time, the first rendered image x1 and the depth information x1' of the first rendered image do not need to be transmitted separately, and the bandwidth can be reduced consumption.
  • the terminal device can perform steps S6-S8:
  • the device (such as the AR device) where the terminal device is located may include a sensor, and the sensor can collect status information at a preset frequency, for example, collect status information at a frequency of 60 times/s.
  • the state information collected at each moment may include position information and posture information of the terminal device at that moment, and it may be considered that different state information corresponds to different moments.
  • the first state information g1 corresponds to the first moment.
  • the first state information g1 can be sent to the remote device, so that the remote device can receive the first state information g1 including the position information and posture information of the terminal device at the first moment.
  • a preset condition may be set, so that the terminal device determines that the first status information g1 is sent to the remote device when the preset condition is met.
  • the terminal device does not send the first state information g1.
  • the preset condition may include: determining the interval between the first status information g1 and the status information sent to the remote device last time by a preset number of status information, or determining the bandwidth margin for data transmission from the remote device to the terminal device at the first moment The amount is greater than or equal to the bandwidth threshold.
  • the status information collected by the device where the terminal device is located can be selectively sent by the terminal device to the remote device, so that after sending the status information to the remote device, the rendered image and rendered depth corresponding to the status information can be received information.
  • the remote device may execute steps S2 and S3 in sequence to obtain the first rendered image x1 corresponding to the first state information g1 and the depth information x1 of the first rendered image ' and send it to the end device. In this way, it is not necessary to send all the state information collected by the equipment where the terminal device is located to the remote device, which can reduce the data transmission cost for the terminal device to transmit the state information to the remote device.
  • the first rendered image x1 is obtained by the remote device through rendering according to the first state information g1 and at least one 3D model of.
  • step S7 the first rendered image x1 and the depth information x1' of the first rendered image may be acquired by the terminal device from the remote device.
  • the remote device acquires the first rendered image x1 and the depth information x1' of the first rendered image.
  • the first rendered image x1 and the depth information x1' of the first rendered image may be sent in the form of rendered frames.
  • what the terminal device receives may also be a rendered frame.
  • the rendered frame includes the first rendered image x1 and the texture indicating the depth information, and the terminal device can use the mapping method used by the remote device in step S4 to map the texture indicating the depth information in the rendered frame back to the form of depth information, that is, obtain Depth information x1' of the first rendered image.
  • the terminal device can also obtain the first rendered image x1 and the depth information x1' of the first rendered image.
  • the first rendered image x1 and the depth information x1' of the first rendered image may include an identifier, and the identifier may indicate state information corresponding to the first rendered image x1 and the depth information x1' of the first rendered image.
  • the terminal device may determine that the first rendered image x1 and the depth information x1' of the first rendered image correspond to the first state information g1 according to the identifier.
  • the rendered frame may include an identifier. According to the identification, the terminal device can determine that the rendered frame corresponds to the first state information g1, so the first rendered image x1 and the depth information x1' of the first rendered image obtained according to the rendered frame also correspond to the first state information g1.
  • the first rendered image x1 is an image corresponding to the first state information g1, and can be directly displayed on the display area of the device where the terminal device is located. It can be seen from the above description that the status information collected by the device where the terminal device is located is sent to the remote device when the preset condition is met, and the status information that does not meet the preset condition (such as status information g2-g6) is not sent to the remote device. Therefore, the terminal device cannot acquire the rendering image corresponding to the status information (eg, g2-g6) that does not meet the preset condition from the remote device.
  • the status information collected by the device where the terminal device is located is sent to the remote device when the preset condition is met, and the status information that does not meet the preset condition (such as status information g2-g6) is not sent to the remote device. Therefore, the terminal device cannot acquire the rendering image corresponding to the status information (eg, g2-g6) that does not meet the preset condition from the remote device.
  • the rendered image corresponding to the second state information may be obtained according to the first rendered image x1, the depth information x1' of the first rendered image, and the second state information, and the second state information, and the second state information includes the terminal Position information and posture information of the device at a second moment after the first moment.
  • Each state information in the state information that does not meet the preset conditions (such as state information g2-g6) is sequentially used as the second state information to perform step S8, and multiple frames of rendered images x2-x6 corresponding to a plurality of state information g2-g6 can be obtained .
  • step S8 is to firstly reconstruct the point cloud model of the object in the first rendered image x1 in the three-dimensional world on the terminal device according to the first rendered image x1 and the depth information x1' of the first rendered image , that is, point cloud data (see step S9 below), and then render the point cloud data according to the second state information to obtain a corresponding rendered image corresponding to the second state information (see step S10 below).
  • step S8 may include S9-S10:
  • Fig. 6 shows an example of determining point cloud data according to an embodiment of the present application.
  • Determining the point cloud data according to the first rendered image x1 and the depth information x1' of the first rendered image can be realized based on existing technologies.
  • the observation point C can be determined, which is equivalent to the position of the user's eyes, and the observation direction can be determined, which is equivalent to the direction of the user's eyes.
  • the viewing angle range of the user eg L1, L2, L3, L4 in FIG. 6
  • the points on the near clipping plane may be pixels in the first rendered image x1.
  • a ray can be uniquely determined from the position of the observation point to a point on the near clipping plane (such as point M), according to the depth information corresponding to the point in the depth information x1' of the first rendered image, and the observation point in the world coordinate system
  • the coordinates of can determine a point in the world coordinate system (for example, N points).
  • Point cloud data may include 3D coordinates of multiple points in a 3D world.
  • the second state information may be any state information among the state information (such as state information g2-g6) that does not satisfy the preset condition.
  • the point cloud data can be projected to obtain a plurality of two-dimensional coordinates.
  • An example is coordinate transformations through multiple spaces such as world space, view space, clip space, screen space, etc.
  • the world space includes point cloud data, that is, three-dimensional coordinates of multiple points in the three-dimensional world.
  • the observation space (View Space) is also called the camera space (Camera Space) or the visual space (Eye Space), and the observation space can include the three-dimensional coordinates of the point of the object in front of the user (observation point) field of view (in the observation direction).
  • the clip space (clip sapce) is obtained by clipping the observation space.
  • the clip space can include the three-dimensional coordinates of the point of the object in front of the user (observation point) field of view (in the direction of observation), and all coordinates in the clip space can fall in Within a given range, points outside this range can be clipped.
  • Screen space (screen space) includes the two-dimensional coordinates of the pixels on the image that the viewer finally sees.
  • the observation point in the observation space can be determined, and the coordinate transformation matrix from the world space to the observation space can be further determined. Combined with the data in the world space, the coordinate transformation can be performed to obtain the data in the observation space.
  • the coordinate transformation can be performed to obtain the data in the screen space.
  • the points in the point cloud data can be mapped to the positions of multiple pixel points in the screen space, and multiple two-dimensional coordinates can be obtained by completing the projection.
  • a rendered image corresponding to the second state information may be obtained.
  • each pixel in the first rendered image x1 may correspond to a two-dimensional coordinate
  • the two-dimensional coordinate represents the position of the pixel in the first rendered image x1.
  • the plurality of two-dimensional coordinates obtained by projection among the pixels of the first rendered image x1, the pixels having the same coordinates as the plurality of two-dimensional coordinates obtained by projection can be found, and according to all the found pixels, it can be used as the second state
  • the information corresponds to the pixels in the rendered image. In this way, the rendered image corresponding to the second state information can be obtained.
  • the above steps S9 and S10 are executed to obtain the rendered images x2-x6 corresponding to the second state information respectively.
  • the multiple frames of rendered images may include rendered images x2-x6. It can be considered that the rendered images x2-x6 correspond to state information g2-g6, respectively.
  • step S8 is to first obtain multiple two-dimensional line segments according to the second state information (see step S11 below), and then project the multiple two-dimensional line segments according to the first state information g1 to obtain multiple two-dimensional line segments Coordinates (see step S12 below), obtain depth information of multiple two-dimensional coordinates (see step S13 below), and compare with the depth information x1' of the first rendered image to determine the coordinates of points with the same depth value (see below In step S14), the rendered image corresponding to the second state information is obtained according to the coordinates of the point with the same depth value and the first rendered image x1 (see step S15 below).
  • step S8 may include S11-S15:
  • the second state information may be any state information among the state information (such as state information g2-g6) that does not satisfy the preset condition.
  • Step S11 can be implemented based on existing technologies.
  • Fig. 7 shows an example of determining multiple line segments in the world coordinate system according to the second state information. As shown in FIG. 7 , according to the second state information, the observation point C can be determined, which is equivalent to the position of the user's eyes, and the observation direction can be determined, which is equivalent to the direction of the user's eyes. Based on this, the viewing angle range of the user (eg L1, L2, L3, L4 in FIG. 7) can be determined.
  • a ray can be uniquely determined from the position of the observation point to a point on the near clipping plane, and a two-dimensional line segment between the near clipping plane and the far clipping plane can be uniquely determined according to the spatial positional relationship between the far clipping plane and the near clipping plane (for example line segment MN).
  • a two-dimensional line segment between the near clipping plane and the far clipping plane can be uniquely determined, so that the world coordinate system can be obtained Multiple 2D line segments.
  • the realization of the projection can refer to the related description of obtaining multiple two-dimensional coordinates according to the projection of the point cloud data in step S10 above, which will not be repeated here.
  • Multiple 2D line segments can be projected on the viewing plane corresponding to the virtual viewpoint.
  • the multiple 2D line segments can occupy part of the pixel points.
  • the plurality of two-dimensional coordinates may be two-dimensional coordinates on the viewing plane of some pixel points occupied by the plurality of two-dimensional line segments.
  • the pixel points corresponding to the multiple two-dimensional coordinates are sampled to obtain the depth value of the pixel point, that is, the depth information of the multiple two-dimensional coordinates.
  • Each two-dimensional coordinate may correspond to a depth value.
  • the depth information x1' of the first rendered image may include depths corresponding to multiple pixels in the first rendered image under the aspect ratio indicated by the rendering parameters, and the depth information of multiple two-dimensional coordinates may include multiple The depth value at the pixel occupied by the two-dimensional line segment.
  • the two-dimensional coordinates of these determined points may be the coordinates of points with the same depth value.
  • the rendered image corresponding to the second state information can be obtained.
  • each pixel in the first rendered image x1 may correspond to a two-dimensional coordinate, and the two-dimensional coordinate represents the position of the pixel in the first rendered image x1.
  • the pixels with the same coordinates as the points with the same depth value can be found, and all the found pixels can be used as the corresponding to the second state information Pixels in the rendered image. In this way, the rendered image corresponding to the second state information can be obtained.
  • the above steps S9 and S10 are executed to obtain the rendered images x2-x6 corresponding to the second state information respectively.
  • the multiple frames of rendered images may include rendered images x2-x6. It can be considered that the rendered images x2-x6 correspond to state information g2-g6, respectively.
  • the device where the terminal device is located may display the first rendered image x1 corresponding to the state information g1 and the multi-frame rendering corresponding to the state information g2-g6 in the display area of the device where the terminal device is located according to the order in which the state information g1-g6 is collected.
  • the images x2-x6 enable the user to observe the virtual objects integrated into the real world through the display area. In this way, the image displayed in the display area is consistent with the change of the state information caused by the change of the user's action, and the user experience is improved.
  • the data processing cost of the AR device can be reduced by directly using the first rendered image for display.
  • the cloud server can include a remote device
  • the VR device can include a terminal device.
  • the remote device can run the server of the application
  • the terminal device can run the client of the application.
  • the application can include multiple 3D models, and the application can be a VR application.
  • the status information may include left-eye status information and right-eye status information, for example, the first status information may include first left-eye status information and first right-eye status information, and the second status information may include second left-eye status information and second Right eye status information.
  • the first rendered image and the depth information of the first rendered image may be obtained by rendering according to the first left-eye state information (or the first right-eye state information). Since the implementation of the rendering method of the present application is to display the image to the user's left eye (or right eye), the first rendered image may include a frame of rendered image.
  • the multi-frame rendered image obtained according to the first rendered image and the depth information of the first rendered image may be an image corresponding to the second left-eye state information (or the second right-eye state information).
  • the first rendered image and the multi-frame rendered images are sequentially displayed in the display area corresponding to the left eye (or right eye) of the VR device according to the collection sequence of the corresponding posture information.
  • the image corresponding to the posture information of the right eye (or left eye) of the VR device can be obtained based on the method in the prior art, and will not be repeated here.
  • the cloud server may include a remote device
  • the AR device may include a terminal device
  • the remote device may run the server of the application
  • the terminal device may run the client of the application
  • the application may include multiple 3D model
  • the application can be an AR application.
  • the remote device and the terminal device can cooperate to implement rendering.
  • Fig. 8 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud AR system according to an embodiment of the present application.
  • the remote device can execute steps S16-S21, wherein, steps S16, S17, and S18 can be executed in sequence, and steps S16, S19, S20, and S21 can be executed in sequence.
  • the order of execution is not limited.
  • S16 Receive first state information g1, where the first state information g1 includes position information and posture information of the terminal device at a first moment.
  • step S16 For the implementation of step S16, reference may be made to the example of step S1 above, which will not be repeated here.
  • S17 perform rendering according to the first state information g1 and at least one 3D model, and obtain a first rendered image x1 and depth information x1' of the first rendered image.
  • step S17 For the implementation of step S17, reference may be made to the example of step S2 above, which will not be repeated here.
  • step S18 can refer to the example of step S3 above, the first rendered image x1 and the depth information x1' of the first rendered image can be sent to the terminal device at the first bit rate.
  • the bit rate of an image is the encoding length required for unit pixel encoding.
  • the first code rate may be a relatively high code rate
  • sending the first rendered image x1 and the depth information x1' of the first rendered image at the first code rate is equivalent to sending the first rendered image x1 and the depth information x1' of the first rendered image at the first code rate
  • the depth information x1' of the first rendered image is encoded, and an encoding result corresponding to the first rendered image x1 and the depth information x1' of the first rendered image is obtained, and then the encoding result is sent.
  • the higher first code rate makes the data volume of the encoding result larger, and the information contained in the encoding result can be closer to the first rendered image x1 and the depth information x1' of the first rendered image, and the first encoding result can be downlinked to the terminal device, and the terminal device performs decoding after receiving the first rendered image x1 and the depth information x1' of the first rendered image.
  • the loss of information before and after encoding and decoding the first rendered image x1 and the depth information x1' of the first rendered image can be guaranteed to be smaller, so as to ensure the quality of image content displayed on the AR device.
  • step S18 may also include S22-S23:
  • S22 Synthesize the first rendered image x1 and the depth information x1' of the first rendered image into a rendered frame
  • S23 Send the rendered frame to the terminal device.
  • the rendered frame may be encoded at the first bit rate, and the encoding result corresponding to the rendered frame may be obtained and then sent. .
  • the method further includes:
  • Step S24 determine the interval between the last rendered image sent to the terminal device and the first rendered image x1 by a specified number of images; or, determine that the bandwidth margin for data transmission to the terminal device at the time of obtaining the first rendered image is greater than or equal to the bandwidth threshold.
  • a preset condition can be set so that the remote device sends the first rendered image x1 and the depth information x1' of the first rendered image to the terminal device when the first rendered image x1 satisfies the preset condition.
  • the preset condition may include: determining that there is a specified number of images between the last rendered image sent to the terminal device and the first rendered image x1; or determining that the bandwidth margin for transmitting data to the terminal device at the moment when the first rendered image is obtained is greater than or equal to the bandwidth threshold.
  • the state information collected by the device where the terminal device is located may be directly sent by the terminal device to the remote device, and the remote device judges whether the rendered image obtained according to the state information and the depth information of the rendered image are sent to the terminal device.
  • the remote device may perform step S17 to first obtain the first rendered image x1 and the depth information x1' of the first rendered image, and then determine whether the first rendered image x1 satisfies Preset conditions, and when the conditions are satisfied, step S18 is executed to send the first rendered image x1 and the depth information x1' of the first rendered image to the terminal device.
  • the terminal device does not need to select the state information sent to the remote device, and the data processing cost of the terminal device can be reduced.
  • the method further includes: determining that there is a specified number of state information between the state information corresponding to the last rendered image sent to the terminal device and the first state information g1; A bandwidth margin for transmitting data to the terminal device at all times of the status information is greater than or equal to the bandwidth threshold.
  • Preset conditions can be set so that the remote device determines that when the first state information g1 satisfies the preset condition, the first rendered image x1 and the depth information x1' of the first rendered image are obtained according to the first state information g1 and sent to the terminal device .
  • the preset conditions may include: determining that the state information corresponding to the last rendered image sent to the terminal device is separated from the first state information g1 by a specified number of state information; The data bandwidth allowance is greater than or equal to the bandwidth threshold.
  • the state information collected by the device where the terminal device is located may be directly sent by the terminal device to the remote device, and the remote device judges whether the state information is used to obtain the rendered image and the depth information of the rendered image sent to the terminal device.
  • the remote device can first determine whether the first state information g1 satisfies the preset condition, and if the condition is met, execute step S17 to obtain the first rendered image x1 and the first rendered image x1.
  • step S18 is performed to send the first rendered image x1 and the depth information x1' of the first rendered image to the terminal device.
  • the terminal device does not need to select the state information sent to the remote device, and the data processing cost of the terminal device can be reduced.
  • the remote device does not render the rendered image but does not use the rendered image, which can reduce the data processing cost of the remote device.
  • the second state information may be acquired by the remote device from the terminal device.
  • the device where the terminal device is located may include a sensor, and the sensor can collect status information at a preset frequency, for example, collect status information at a frequency of 60 times/s.
  • the state information collected at each moment may include position information and posture information of the terminal device at that moment, and it may be considered that different state information corresponds to different moments.
  • the second status information corresponds to the second moment.
  • the second state information may be, for example, any state information in state information g2-g6.
  • the second state information can be sent to the remote device, so that the remote device can receive the second state information including the position information and attitude information of the terminal device at the second moment.
  • step S20 For an exemplary implementation of step S20, reference may be made to the relevant description of obtaining the first rendered image x1 by rendering according to the first state information g1 and at least one 3D model in step S2 above, and details are not repeated here.
  • the above steps S19 and S20 are executed to obtain the rendered images x2-x6 corresponding to the second state information respectively.
  • the rendered images x2-x6 may serve as the second rendered images respectively. It can be considered that the rendered images x2-x6 correspond to state information g2-g6, respectively.
  • the remote device may send the second rendered image to the terminal device, so as to use the second rendered image on the terminal device to enhance the effect of the multi-frame rendered image rendered according to the first rendered image x1 and the depth information x1' of the first rendered image .
  • the remote device may set an identifier for the first rendered image x1, the depth information x1' of the first rendered image, and the second rendered image, and the identifier may represent the first rendered image x1, the depth information x1' of the first rendered image, the second The status information corresponding to each rendering image.
  • the identification of the first rendered image x1 and the depth information x1' of the first rendered image may be the same and correspond to the first state information g1, and the identification of the second rendered image is the same as that of the first rendered image x1, the first rendered image
  • the identifiers of the depth information x1' can be different, and correspond to the state information collected by the equipment to which the terminal device belongs except the first state information g1 (for example, any state information in state information g2-g6).
  • the second code rate may be a lower code rate, and the second code rate may be lower than the first code rate.
  • Sending the second rendered image at the second bit rate is equivalent to encoding the second rendered image at the second bit rate, obtaining an encoding result corresponding to the second rendered image and then sending the encoding result.
  • Encoding is performed according to the second code rate, so that the amount of data in the encoding result is small, and the amount of information included in the encoding result is reduced.
  • the encoding result can be transmitted downlink to the terminal device, and the terminal device decodes it after receiving it to obtain the second rendering image. By encoding at a low bit rate, network bandwidth consumption can be reduced.
  • the second rendered image may be down-sampled to obtain a down-sampled result of the second rendered image, and then the down-sampled result is sent at the second bit rate.
  • Downsampling can be implemented based on existing techniques. In this way, network bandwidth consumption can be further reduced.
  • the terminal device may execute steps S25-S29, wherein step S27 may also be executed prior to step S26.
  • step S25 For the implementation of step S25, reference may be made to the example of step S6, wherein the terminal device may not set a preset condition for sending the first status information.
  • step S26 For the implementation of step S26, reference may be made to the example of step S7, which will not be repeated here.
  • the first rendered image x1 and the depth information x1' of the first rendered image may include an identifier, and the identifier may indicate state information corresponding to the first rendered image x1 and the depth information x1' of the first rendered image.
  • the terminal device may determine that the first rendered image x1 and the depth information x1' of the first rendered image correspond to the first state information g1 according to the identifier.
  • the rendered frame may include an identifier. The terminal device can determine that the rendered frame corresponds to the first state information g1 according to the identifier, so the first rendered image x1 and the depth information x1' of the first rendered image obtained according to the rendered frame also correspond to the first state information g1.
  • step S27 refer to the example of sending the first state information g1 to the remote device in step S6.
  • the terminal device may not set a preset condition for sending the second state information.
  • the second rendered image may be acquired by the terminal device from the remote device.
  • the second rendered image may include an identifier, and the identifier may represent state information corresponding to the second rendered image.
  • the terminal device may determine according to the identifier that the second rendered image corresponds to the second state information.
  • the downsampling result may include an identification.
  • the terminal device may determine, according to the identifier, that the second rendered image obtained from the downsampling result corresponds to the second state information. For example, it may be determined according to the identifier that the second state information corresponding to the second rendered image is one of state information g2-g6.
  • the first rendered image x1 is an image corresponding to the first state information g1, and can be directly displayed on the display area of the device where the terminal device is located. It can be seen from the above description that the status information collected by the device where the terminal device is located is directly sent to the remote device, and the remote device sends the rendered image at a higher first bit rate when the rendered image obtained from the status information meets the preset conditions To the terminal device, the rendered image (second rendered image) that does not meet the preset conditions is sent to the terminal device at a lower second bit rate. Since the lower second bit rate has a greater impact on the quality of the rendered image, it may Not suitable for displaying directly on the display area of the device where the terminal device is located.
  • the first intermediate rendered image may be obtained according to the first rendered image x1, the depth information x1' of the first rendered image, and the second state information, and then The first intermediate rendered image is further processed according to the second rendered image to obtain second rendered images xx2-xx6 corresponding to the second state information.
  • Step 29 can have multiple implementations, optionally, step S29 can include S30-S32:
  • step S30 For the implementation manner of step S30, reference may be made to the example of step S9, which will not be repeated here.
  • step 31 reference may be made to the related description of rendering the point cloud data according to the second state information in step S10 to obtain a rendered image corresponding to the second state information, which will not be repeated here.
  • step S31 during the process of projecting the point cloud data according to the second state information to obtain multiple two-dimensional coordinates, it may occur that after coordinate transformation of multiple points in the three-dimensional point cloud data, the same two-dimensional coordinates may be obtained.
  • coordinates so that according to the two-dimensional coordinates and the texture information corresponding to multiple two-dimensional coordinates in the first rendered image x1, when the first intermediate rendered image is obtained, some pixels in the obtained first intermediate rendered image have no data , resulting in "hollow spots" in the first intermediate rendered image.
  • FIG. 9 shows an example of hole points (such as black points in FIG. 9 ) appearing in the first intermediate rendered image according to an embodiment of the present application.
  • the above steps S30-S32 are performed to obtain the rendered images xx2-xx6 corresponding to the second state information respectively.
  • the multiple frames of rendered images may include rendered images xx2-xx6. It can be considered that the rendered images xx2-xx6 correspond to state information g2-g6, respectively.
  • step S29 may include:
  • the virtual view point may be the observation point in the relevant description of step S11
  • the viewing distance may be the distance between the farthest point that the user's eyes (observation point) can see and the observation point in the relevant description of step S11.
  • each two-dimensional line segment corresponds to multiple two-dimensional coordinates.
  • the depth information of the multiple two-dimensional coordinates is compared with the depth information of the first rendered image, and coordinates with the same depth value are determined, wherein the depth information includes the depth value corresponding to each coordinate.
  • the depth information includes the depth value corresponding to each coordinate.
  • a second intermediate rendered image corresponding to the second state information is obtained according to texture information in the first rendered image at coordinates with the same depth value.
  • the second intermediate rendered image is filled with the second rendered image to obtain a rendered image corresponding to the second state information.
  • the second rendered image is used to fill in the first intermediate rendered image to obtain a related description of the second rendered image corresponding to the second state information.
  • the above step S29 is executed to obtain the rendered images xx2-xx6 corresponding to the second state information respectively.
  • the multiple frames of rendered images may include rendered images xx2-xx6. It can be considered that the rendered images xx2-xx6 correspond to state information g2-g6, respectively.
  • the device where the terminal device is located may display the first rendered image x1 corresponding to the state information g1 and the multi-frame rendering corresponding to the state information g2-g6 in the display area of the device where the terminal device is located according to the order in which the state information g1-g6 is collected.
  • the images xx2-xx6 enable the user to observe the virtual objects integrated into the real world through the display area. In this way, the image displayed in the display area is consistent with the change of the state information caused by the change of the user's action, and the user experience is improved. Directly using the first rendered image x1 for display can reduce the data processing cost of the AR device.
  • the cloud server can include a remote device
  • the VR device can include a terminal device.
  • the remote device can run the server of the application
  • the terminal device can run the client of the application.
  • the application can include multiple 3D models, and the application can be a VR application.
  • the status information may include left-eye status information and right-eye status information, for example, the first status information may include first left-eye status information and first right-eye status information, and the second status information may include second left-eye status information and second Right eye status information.
  • the first rendered image and the depth information of the first rendered image may be obtained by rendering according to the first left-eye state information (or the first right-eye state information). Since the implementation of the rendering method of the present application is to display the image to the user's left eye (or right eye), the first rendered image may include a frame of rendered image.
  • the multi-frame rendered image obtained according to the first rendered image and the depth information of the first rendered image may be an image corresponding to the second left-eye state information (or the second right-eye state information).
  • the first rendered image and the multi-frame rendered images are sequentially displayed in the display area corresponding to the left eye (or right eye) of the VR device according to the collection sequence of the corresponding posture information.
  • the image corresponding to the posture information of the right eye (or left eye) of the VR device can be obtained based on the method in the prior art, and will not be repeated here.
  • the cloud server may include a remote device
  • the VR device may include a terminal device
  • the remote device may run the server of the application
  • the terminal device may run the client of the application
  • the application may include multiple 3D model.
  • the application can be a VR application.
  • the remote device and the terminal device can cooperate to implement rendering.
  • FIG. 11 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud VR system according to an embodiment of the present application.
  • the remote device can perform steps S33-S35:
  • S33 Receive first state information g1, where the first state information g1 includes position information and posture information of the terminal device at a first moment.
  • step S33 For the implementation of step S33, reference may be made to the example of step S1 above, which will not be repeated here.
  • step S34 can refer to the example of step S2 above, wherein, according to the first state information g1, the state information g11 corresponding to the user’s left eye and the state information g12 corresponding to the user’s right eye can be obtained first, and the first rendered image x11 It may be obtained according to the state information g11 of the user's left eye (or the state information g12 of the right eye).
  • the first rendered image x11 is obtained according to the state information g11 of the user's left eye.
  • step S35 For the implementation of step S35, reference may be made to the example of step S18 above, which will not be repeated here.
  • step S35 may also include S36-S37:
  • S36 combining the first rendered image x11 and the depth information x11' of the first rendered image into a rendered frame
  • S37 sending the rendered frame to the terminal device.
  • the rendered frame may be encoded at the first bit rate, and the encoding result corresponding to the rendered frame may be obtained and then sent. .
  • the method further includes: determining that there is a specified number of state information between the state information corresponding to the last rendered image sent to the terminal device and the first state information g1; A bandwidth margin for transmitting data to the terminal device at all times of the status information is greater than or equal to the bandwidth threshold.
  • Preset conditions can be set so that the remote device determines that when the first state information g1 satisfies the preset condition, the first rendered image x11 and the depth information x11' of the first rendered image are obtained according to the first state information g1 and sent to the terminal device .
  • the preset conditions may include: determining that the state information corresponding to the last rendered image sent to the terminal device is separated from the first state information g1 by a specified number of state information; The data bandwidth allowance is greater than or equal to the bandwidth threshold.
  • the state information collected by the device where the terminal device is located may be directly sent by the terminal device to the remote device, and the remote device judges whether the state information is used to obtain the rendered image and the depth information of the rendered image sent to the terminal device.
  • the remote device may first determine whether the first state information g1 satisfies the preset condition, and if the condition is met, execute step S34 to obtain the first rendered image x11 and the first After rendering the depth information x11' of the image, step S35 is performed to send the first rendered image x11 and the depth information x1' of the first rendered image to the terminal device. In this way, the terminal device does not need to select the state information sent to the remote device, and the data processing cost of the terminal device can be reduced.
  • the terminal device can perform steps S38-S40:
  • step S38 For the implementation of step S38, reference may be made to the example of step S6 above.
  • the first rendered image x11 is obtained by rendering by the remote device according to the first state information g1 and at least one 3D model of.
  • step S39 For the implementation of step S39, reference may be made to the example of step S7 above.
  • the first rendered image x11 may be obtained according to the state information g11 of the user's left eye in the first state information g1.
  • the first rendered image x11 is an image corresponding to the state information g11 of the user's left eye in the first state information g1, and can be directly displayed on the display area corresponding to the left eye of the device where the terminal device is located.
  • the user's right eye in the first state information g1 can be obtained according to the first rendered image x11 and the depth information x11' of the first rendered image.
  • Another rendered image x12 corresponding to the state information g12 of so that the distance between the virtual viewpoints corresponding to the first rendered image x11 and the other rendered image x12 is equal to the viewing distance.
  • another rendered image is an image corresponding to the state information g12 of the user's right eye in the first state information, and may be displayed on the display area corresponding to the right eye of the terminal device.
  • point cloud data can be generated according to the first rendered image x11 and the depth information x11' of the first rendered image.
  • the point cloud data is rendered according to the state information g12 of the user's right eye in the first state information g1, and the rendered image x12 corresponding to the state information g12 is obtained.
  • the step S10 where the point cloud data is rendered according to the second state information, and a related description of the rendered image corresponding to the second state information is obtained.
  • multiple two-dimensional line segments may be determined according to the state information g12 of the user's right eye in the first state information, the preset near clipping plane and the far clipping plane.
  • step S11 in which descriptions of a plurality of two-dimensional line segments are determined according to the second state information, the preset near clipping plane and the far clipping plane.
  • the multiple two-dimensional line segments are projected according to the state information g11 of the user's left eye in the first state information to obtain multiple two-dimensional coordinates.
  • step S12 For an exemplary implementation manner thereof, reference may be made to the relevant description of step S12 above. Obtain depth information for multiple 2D coordinates.
  • step S13 reference may be made to the relevant description of step S13 above.
  • the depth information of multiple two-dimensional coordinates is compared with the depth information of the first rendered image x11 to determine the coordinates of points with the same depth value.
  • step S14 For an exemplary implementation, refer to the relevant description of step S14 above.
  • According to the coordinates of the points with the same depth value and the first rendered image x11 obtain the rendered image x12 corresponding to the state information g12 of the user's right eye in the first state information.
  • An exemplary implementation thereof may refer to the step S15 above, in which the description of the rendered image corresponding to the second state information is obtained according to the coordinates of the point with the same depth value and the first rendered image x1.
  • the terminal device cannot acquire the rendering image corresponding to the second state information from the remote device.
  • the state information corresponding to the user's left eye and the state information corresponding to the user's right eye may be obtained first according to the second state information, Then according to the first rendered image x11, the depth information x11' of the first rendered image, and the second state information (the state information corresponding to the user's left eye and the state information corresponding to the user's right eye), obtain the rendering corresponding to the second state information image, the second state information includes position information and posture information of the terminal device at a second moment after the first moment.
  • step S40 may include S41-S42:
  • step S41 For the implementation of step S41, reference may be made to the example of step S9 above.
  • the state information g11 corresponding to the user's left eye and the state information g12 corresponding to the user's right eye can be obtained according to the first state information g1, and the first rendered image x11 is determined according to the state information of the user's left eye in the first state information obtained by g11.
  • the point cloud data can be obtained.
  • step S42 For the implementation manner of step S42, reference may be made to the example of step S10 above.
  • the state information corresponding to the user's left eye and the state information corresponding to the user's right eye can be obtained according to the second state information.
  • the state information renders the point cloud data, and the rendered images x21-x61 corresponding to the user's left eye can be obtained.
  • the point cloud data is rendered according to the state information of the user's right eye in the second state information, and rendered images x22-x62 corresponding to the user's right eye can be obtained.
  • the rendered images corresponding to the second state information may include the aforementioned rendered images x21-x61 corresponding to the user's left eye and the rendered images x22-x62 corresponding to the user's right eye.
  • step S40 may include:
  • a plurality of two-dimensional line segments are determined according to the second state information, the preset near clipping plane and the far clipping plane.
  • the state information corresponding to the user's left eye and the state information corresponding to the user's right eye can be obtained according to the second state information processing.
  • the preset near The clipping plane and the far clipping plane can obtain the 2D line segment corresponding to the user's left eye.
  • a two-dimensional line segment corresponding to the user's right eye can be obtained.
  • the plurality of two-dimensional line segments may include the aforementioned two-dimensional line segment corresponding to the user's left eye and the above-mentioned two-dimensional line segment corresponding to the user's right eye.
  • a plurality of two-dimensional line segments are projected according to the first state information g1 to obtain a plurality of two-dimensional coordinates, wherein each two-dimensional line segment corresponds to a plurality of two-dimensional coordinates.
  • each two-dimensional line segment corresponds to a plurality of two-dimensional coordinates.
  • project the 2D line segment corresponding to the user's left eye according to the first state information g1 to obtain the 2D coordinates corresponding to the user's left eye, and project the 2D line segment corresponding to the user's right eye according to the first state information g1 , to obtain the two-dimensional coordinates corresponding to the user's right eye, and the plurality of two-dimensional coordinates may include the above-mentioned two-dimensional coordinates corresponding to the user's left eye and the two-dimensional coordinates corresponding to the user's right eye.
  • step S13 Obtain depth information for multiple 2D coordinates.
  • the depth information corresponding to the two-dimensional coordinates of the user's left eye can be correspondingly obtained
  • the depth information corresponding to the user's right eye can be correspondingly obtained
  • the depth information of two-dimensional coordinates, the plurality of depth information of two-dimensional coordinates may include the above-mentioned depth information of two-dimensional coordinates corresponding to the user's left eye and depth information of two-dimensional coordinates corresponding to the user's right eye.
  • the depth information of the multiple two-dimensional coordinates is compared with the depth information of the first rendered image, and coordinates of points with the same depth value are determined, wherein the depth information includes the depth value corresponding to each coordinate.
  • Its exemplary implementation can refer to the relevant description of step S14 above, wherein, comparing the depth information corresponding to the two-dimensional coordinates of the user's left eye with the depth information of the first rendered image, it can be correspondingly obtained corresponding to the depth information of the user's left eye. Coordinates of points with the same depth value.
  • the depth information corresponding to the two-dimensional coordinates of the user's right eye is compared with the depth information of the first rendered image, and the coordinates of points corresponding to the user's right eye with the same depth value can be correspondingly obtained.
  • the coordinates of the points with the same depth value may include the coordinates of the points with the same depth value corresponding to the user's left eye and the coordinates of the points with the same depth value corresponding to the user's right eye.
  • the rendered image corresponding to the second state information is obtained. Its implementation can refer to the example of step S15 above, wherein, according to the coordinates of the point corresponding to the user's left eye with the same depth value and the first rendered image, the rendered images x21-x61 corresponding to the user's left eye can be correspondingly obtained, according to Corresponding to the coordinates of the point with the same depth value of the user's right eye and the first rendered image, the rendered image x22-x62 corresponding to the user's right eye can be obtained correspondingly, and the rendered image corresponding to the second state information can include the above-mentioned corresponding to the user's left eye
  • the rendered images x21-x61 of and the rendered images x22-x62 corresponding to the user's right eye can include the above-mentioned corresponding to the user's left eye.
  • the device where the terminal device is located may display the first rendered image x11, another rendered image x12, and rendered images x21-x61 corresponding to the second state information in the display area of the device where the terminal device is located according to the order in which state information g1-g6 is collected , x22-x62, for example, x11, x21-x61 are displayed in the display area corresponding to the left eye, and x12-x62 are displayed in the display area corresponding to the right eye, so that the user can observe the virtual world in three-dimensional space through the display area.
  • images corresponding to the same state information can be displayed simultaneously.
  • the image displayed in the display area is consistent with the change of the state information caused by the change of the user's action, and the user experience is improved.
  • Directly using the first rendered image for display can reduce the data processing cost of the VR device.
  • the cloud server can include a remote device, and the VR device can include a terminal device.
  • the remote device can run the server of the application, and the terminal device can run the client of the application.
  • the application can include multiple 3D models.
  • the application could be a VR application.
  • the remote device and the terminal device can cooperate to implement rendering.
  • Fig. 12 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud VR system according to an embodiment of the present application.
  • the remote device can perform steps S43-S47:
  • step S43 For the implementation of step S43, reference may be made to the example of step S33 above, which will not be repeated here.
  • step S44 For the implementation of step S44, reference may be made to the example of step S34 above, which will not be repeated here.
  • the state information g11 corresponding to the user's left eye and the state information g12 corresponding to the user's right eye can be obtained first, and the first rendered image x11 can be obtained according to the state information g11 of the user's left eye (or the state of the right eye).
  • Information g12) obtained obtained.
  • the first rendered image x11 is obtained according to the state information g11 of the user's left eye.
  • step S45 For the implementation of step S45, reference may be made to the example of step S35 above, which will not be repeated here.
  • step S45 may also include S48-S49:
  • step S48 combining the first rendered image x11 and the depth information x11' of the first rendered image into a rendered frame
  • step S49 sending the rendered frame to the terminal device.
  • the rendered frame may be encoded at the first bit rate, and the encoded result corresponding to the rendered frame may be obtained and then sent.
  • the method also includes:
  • Step S50 determine that there is a specified number of images between the last rendered image sent to the terminal device and the first rendered image x11; or, determine that the bandwidth margin for data transmission to the terminal device at the moment when the first rendered image is obtained is greater than or equal to the bandwidth threshold.
  • a preset condition can be set so that the remote device determines to send the first rendered image x11 and the depth information x11' of the first rendered image to the terminal device when the first rendered image x11 satisfies the preset condition.
  • the preset condition may include: determining that there is a specified number of images between the last rendered image sent to the terminal device and the first rendered image x11; or determining that the bandwidth margin for transmitting data to the terminal device at the moment when the first rendered image is obtained is greater than or equal to the bandwidth threshold.
  • the state information collected by the device where the terminal device is located may be directly sent by the terminal device to the remote device, and the remote device judges whether the rendered image obtained according to the state information and the depth information of the rendered image are sent to the terminal device.
  • the remote device may perform step S44 to first obtain the first rendered image x11 and the depth information x11' of the first rendered image, and then determine whether the first rendered image x11 satisfies Preset conditions, and when the conditions are satisfied, step S45 is executed to send the first rendered image x11 and the depth information x11' of the first rendered image to the terminal device.
  • the terminal device does not need to select the state information sent to the remote device, and the data processing cost of the terminal device can be reduced.
  • step S46 may refer to the example of step S2 above, wherein another rendered image may be obtained according to the state information g12 of the user's right eye in the first state information. That is, the first rendered image x11 and the other rendered image x12 may respectively correspond to state information g11 of the user's left eye and state information g12 of the user's right eye in the first state information.
  • the terminal device can perform steps S51-S53:
  • step S51 For the implementation of step S51, reference may be made to the example of step S6 above, which will not be repeated here.
  • the first rendered image x11 is obtained by the remote device through rendering according to the first state information g1 and at least one 3D model of.
  • step S52 For the implementation of step S52, reference may be made to the example of step S7 above.
  • the first rendered image x11 may be obtained according to the state information g11 of the user's left eye in the first state information.
  • the first rendered image x11 is an image corresponding to the state information g11 of the user's left eye in the first state information g1, and can be directly displayed on the display area corresponding to the left eye of the device where the terminal device is located. It can be seen from the above description that the remote device also sends another rendered image corresponding to the state information g12 of the user's right eye in the first state information g1 to the terminal device at a second lower bit rate.
  • the bit rate has a great impact on the quality of the rendered image, and may not be suitable for displaying directly on the display area of the device where the terminal device is located.
  • step S53 may include:
  • Receive another rendered image x12 sent by the remote device The other rendered image is obtained by the remote device through rendering according to the first state information and the application.
  • the distance between the virtual view point corresponding to the first rendered image and the other rendered image is equal to the view point distance.
  • the first rendered image is the rendered image corresponding to the state information of the user's left eye (or right eye) in the first state information
  • the other rendered image may be the user's right eye (or left eye) in the first state information.
  • state information corresponding to the rendered image when the first rendered image is the rendered image corresponding to the state information of the user's left eye (or right eye) in the first state information.
  • the point cloud data is rendered to obtain a third intermediate rendered image.
  • the first rendered image x11 is the rendered image corresponding to the state information g11 of the user's left eye in the first state information
  • the point The cloud data is rendered to obtain a third intermediate rendered image.
  • the third intermediate rendered image is filled with another rendered image x12 sent by the remote device to obtain another rendered image xx12 corresponding to the first state information.
  • the third intermediate rendered image is filled with another rendered image x12 sent by the remote device to obtain another rendered image xx12 corresponding to the first state information.
  • step S53 also includes:
  • An exemplary implementation thereof may refer to the step S40 above, in which the relevant descriptions of multiple two-dimensional line segments are determined according to the second state information, the preset near clipping plane and the far clipping plane.
  • each two-dimensional line segment corresponds to multiple two-dimensional coordinates.
  • step S40 where multiple two-dimensional line segments are projected according to the first state information g1 to obtain related descriptions of multiple two-dimensional coordinates.
  • Obtain depth information for multiple 2D coordinates For an exemplary implementation manner thereof, reference may be made to the related description of obtaining depth information of multiple two-dimensional coordinates in step S40 above.
  • the depth information of the plurality of two-dimensional coordinates is compared with the depth information of the first rendered image, and coordinates having the same depth value are determined, wherein the depth information includes depth values corresponding to each coordinate.
  • An exemplary implementation thereof may refer to step S40 above, where the depth information of multiple two-dimensional coordinates is compared with the depth information of the first rendered image, and the relevant description of the coordinates of points with the same depth value is determined.
  • the fourth intermediate rendered image is filled with another rendered image sent by the remote device to obtain another rendered image xx12 corresponding to the first state information.
  • another rendered image xx12 corresponding to the first state information.
  • the terminal device cannot acquire the rendered image corresponding to the second state information from the remote device.
  • the state information corresponding to the user's left eye and the state information corresponding to the user's right eye may be obtained first according to the second state information, Then according to the first rendered image x11, the depth information x11' of the first rendered image, and the second state information (the state information corresponding to the user's left eye and the state information corresponding to the user's right eye), obtain the rendering corresponding to the second state information image, the second state information includes position information and posture information of the terminal device at a second moment after the first moment.
  • step S53 may include S54-S55:
  • step S41 For the implementation of step S41, reference may be made to the example of step S9 above.
  • the state information g11 corresponding to the user's left eye and the state information g12 corresponding to the user's right eye can be obtained according to the first state information g1
  • the first rendered image x11 is determined according to the state information of the user's left eye in the first state information obtained by g11.
  • the point cloud data can be obtained.
  • step S55 For the implementation manner of step S55, reference may be made to the example of step S10 above.
  • the state information corresponding to the user's left eye and the state information corresponding to the user's right eye can be obtained according to the second state information.
  • the state information renders the point cloud data, and the rendered images x21-x61 corresponding to the user's left eye can be obtained.
  • the point cloud data is rendered according to the state information of the user's right eye in the second state information, and rendered images x22-x62 corresponding to the user's right eye can be obtained.
  • the rendered images corresponding to the second state information may include the aforementioned rendered images x21-x61 corresponding to the user's left eye and the rendered images x22-x62 corresponding to the user's right eye.
  • step S53 may include:
  • a plurality of two-dimensional line segments are determined according to the second state information, the preset near clipping plane and the far clipping plane.
  • a plurality of two-dimensional line segments are projected according to the first state information g1 to obtain a plurality of two-dimensional coordinates, wherein each two-dimensional line segment corresponds to a plurality of two-dimensional coordinates.
  • the depth information of the multiple two-dimensional coordinates is compared with the depth information of the first rendered image, and coordinates of points with the same depth value are determined, wherein the depth information includes the depth value corresponding to each coordinate.
  • the rendered image corresponding to the second state information is obtained.
  • the rendered images corresponding to the second state information may include rendered images x21-x61 and rendered images x22-x62.
  • the device where the terminal device is located may display the first rendered image x11, the rendered image xx12, and the rendered images x21-x61, x22 corresponding to the second state information in the display area of the device where the terminal device is located according to the order in which state information g1-g6 is collected.
  • -x62 for example, display x11, x21-x61 in the display area corresponding to the left eye, and display xx12, x22-x62 in the display area corresponding to the right eye, so that the user can observe the virtual world in three-dimensional space through the display area.
  • images corresponding to the same state information can be displayed simultaneously.
  • the image displayed in the display area is consistent with the change of the state information caused by the change of the user's action, and the user experience is improved.
  • Directly using the first rendered image x11 for display can reduce the data processing cost of the VR device.
  • the cloud server can include a remote device, and the VR device can include a terminal device.
  • the remote device can run the server of the application, and the terminal device can run the client of the application.
  • the application can include multiple 3D models.
  • the application could be a VR application.
  • the rendering can be realized through the cooperation of the remote device and the terminal device.
  • Fig. 13 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud VR system according to an embodiment of the present application.
  • the remote device can execute steps S56-S61, wherein, steps S56, S57, and S58 can be executed in sequence, and steps S56, S59, S60, and S61 can be executed in sequence.
  • the order of execution is not limited.
  • step S56 For the implementation of step S56, reference may be made to the example of step S1 above, which will not be repeated here.
  • step S57 can refer to the example of step S2 above, wherein, according to the first state information g1, the state information g11 corresponding to the user’s left eye and the state information g12 corresponding to the user’s right eye can be obtained first, and the first rendered image x11 may be obtained according to the state information g12 of the user's left eye in the first state information.
  • step S58 can refer to the example of step S3 above, the first rendered image x11 and the depth information x11' of the first rendered image can be sent to the terminal device at the first bit rate.
  • step S58 may also include S62-S63:
  • S62 Synthesize the first rendered image and the depth information of the first rendered image into a rendered frame
  • S63 Send the rendered frame to the terminal device.
  • the rendered frame may be encoded at the first bit rate, and the encoded result corresponding to the rendered frame may be obtained and then sent.
  • the method also includes:
  • Step S64 determine that the last rendered image sent to the terminal device is separated from the first rendered image x11 by a specified number of images; or, determine that the bandwidth margin for transmitting data to the terminal device at the time of obtaining the first rendered image is greater than or equal to the bandwidth threshold.
  • a preset condition can be set so that the remote device determines to send the first rendered image x11 and the depth information x11' of the first rendered image to the terminal device when the first rendered image x11 satisfies the preset condition.
  • the preset condition may include: determining that there is a specified number of images between the last rendered image sent to the terminal device and the first rendered image x11; or determining that the bandwidth margin for transmitting data to the terminal device at the moment when the first rendered image is obtained is greater than or equal to the bandwidth threshold.
  • the state information collected by the device where the terminal device is located may be directly sent by the terminal device to the remote device, and the remote device judges whether the rendered image obtained according to the state information and the depth information of the rendered image are sent to the terminal device.
  • the remote device may perform step S57 to first obtain the first rendered image x11 and the depth information x11' of the first rendered image, and then determine whether the first rendered image x11 satisfies Preset conditions, and when the conditions are satisfied, step S58 is executed to send the first rendered image x11 and the depth information x11' of the first rendered image to the terminal device. In this way, the terminal device does not need to select the state information sent to the remote device, and the data processing cost of the terminal device can be reduced.
  • S59 Receive second state information, where the second state information includes position information and attitude information of the terminal device at a second moment after the first moment.
  • step S59 For the implementation of step S59, reference may be made to the example of step S19 above, which will not be repeated here.
  • S60 Perform rendering according to the second state information and at least one 3D model, to obtain a second rendered image.
  • step S60 can refer to the example of step S2 above, wherein the terminal device can obtain the state information corresponding to the user's left eye and the state information corresponding to the user's right eye according to the second state information, and the state information corresponding to the user's left eye
  • the state information and the state information corresponding to the user's right eye and at least one three-dimensional model are rendered, and the obtained second rendered image may include the rendered image corresponding to the state information of the user's left eye in the second state information, and the state information in the second state information
  • the rendered image corresponding to the state information of the user's right eye can refer to the example of step S2 above, wherein the terminal device can obtain the state information corresponding to the user's left eye and the state information corresponding to the user's right eye according to the second state information, and the state information corresponding to the user's left eye
  • the state information and the state information corresponding to the user's right eye and at least one three-dimensional model are rendered, and the obtained second rendered image may include the rendered
  • step S61 can refer to the example of step S21 above.
  • the second rendered image can include the rendered image corresponding to the state information of the user's left eye in the second state information, and the user's right eye in the second state information.
  • the rendered image corresponding to the state information of the eye may be down-sampled and sent.
  • FIG. 14 shows an example of a transmission manner of a second rendered image according to an embodiment of the present application.
  • the rendered image corresponding to the state information of the user's left eye in the second state information and the rendered image corresponding to the state information of the user's right eye in the second state information may be respectively down-sampled to obtain the first down-sampling result and the second downsampling result, in which case the two downsampling results can be combined into one texture.
  • the downsampling can be implemented based on the existing technology, which can reduce the amount of data sent by the remote device for the second rendered image.
  • the terminal device can perform steps S65-S69:
  • step S65 refer to the example of step S6, where the terminal device may not set a preset condition for sending the first status information.
  • step S66 may refer to the example of step S7, wherein the first rendered image may be obtained according to the state information of the user's left eye in the first state information.
  • the first rendered image and the depth information of the first rendered image may include an identifier, and the identifier may indicate state information corresponding to the first rendered image and the depth information of the first rendered image.
  • the terminal device may determine, according to the identifier, that the first rendered image and the depth information of the first rendered image correspond to the first state information.
  • the rendered frame may include an identification. According to the identification, the terminal device may determine that the first rendered image obtained from the rendered frame and the depth information of the first rendered image correspond to the first state information.
  • the identifier may also represent the state information of the user's left eye (or right eye) in the first state information corresponding to the first rendered image and the depth information of the first rendered image. According to the identification, the terminal device may determine that the first rendered image and the depth information of the first rendered image correspond to the state information of the user's left eye (or right eye) in the first state information.
  • step S67 refer to the example of step S27, where the terminal device may not set a preset condition for sending the second status information.
  • the second rendered image may include a rendered image corresponding to the state information of the user's left eye in the second state information, and a rendered image corresponding to the state information of the user's right eye in the second state information.
  • the second rendered image may include an identifier, and the identifier may represent state information corresponding to the second rendered image.
  • the terminal device may determine according to the identifier that the second rendered image corresponds to the second state information.
  • the downsampling result may include an identification.
  • the terminal device may determine, according to the identifier, that the second rendered image obtained from the downsampling result corresponds to the second state information.
  • the first rendered image x11 is an image corresponding to the state information g11 of the user's left eye in the first state information, and can be directly displayed on the display area corresponding to the left eye of the terminal device.
  • the user's right eye in the first state information g1 can be obtained according to the first rendered image x11 and the depth information x11' of the first rendered image.
  • Another rendered image x12 corresponding to the state information g12 of so that the distance between the virtual viewpoints corresponding to the first rendered image x11 and the other rendered image x12 is equal to the viewing distance.
  • another rendered image is an image corresponding to the state information g12 of the user's right eye in the first state information, and may be displayed on the display area corresponding to the right eye of the terminal device.
  • point cloud data can be generated according to the first rendered image x11 and the depth information x11' of the first rendered image.
  • the point cloud data is rendered according to the state information g12 of the user's right eye in the first state information g1, and the rendered image x12 corresponding to the state information g12 is obtained.
  • the step S10 where the point cloud data is rendered according to the second state information, and a related description of the rendered image corresponding to the second state information is obtained.
  • multiple two-dimensional line segments may be determined according to the state information g12 of the user's right eye in the first state information, the preset near clipping plane and the far clipping plane.
  • step S11 in which descriptions of a plurality of two-dimensional line segments are determined according to the second state information, the preset near clipping plane and the far clipping plane.
  • the multiple two-dimensional line segments are projected according to the state information g11 of the user's left eye in the first state information to obtain multiple two-dimensional coordinates.
  • step S12 For an exemplary implementation manner thereof, reference may be made to the relevant description of step S12 above. Obtain depth information for multiple 2D coordinates.
  • step S13 reference may be made to the relevant description of step S13 above.
  • the depth information of multiple two-dimensional coordinates is compared with the depth information of the first rendered image x11 to determine the coordinates of points with the same depth value.
  • the rendered image x12 corresponding to the state information g12 of the user's right eye in the first state information is obtained.
  • An exemplary implementation thereof may refer to the step S15 above, in which the description of the rendered image corresponding to the second state information is obtained according to the coordinates of the point with the same depth value and the first rendered image x1.
  • the terminal device sends the second rendered image to the terminal device at a lower second bit rate. Since the lower second bit rate has a greater impact on the quality of the rendered image, it may not be suitable for direct rendering on the terminal device. displayed in the display area of the device where it is located.
  • step S69 may include S70-S72:
  • step S71 Render the point cloud data according to the second state information to obtain a first intermediate rendered image.
  • the state information corresponding to the user's left eye and the state information corresponding to the user's right eye can be obtained according to the second state information.
  • the state information renders the point cloud data, and the rendered images x21-x61 corresponding to the user's left eye can be obtained.
  • the point cloud data is rendered according to the state information of the user's right eye in the second state information, and rendered images x22-x62 corresponding to the user's right eye can be obtained.
  • the first intermediate rendered images may include the aforementioned rendered images x21-x61 corresponding to the user's left eye and the rendered images x22-x62 corresponding to the user's right eye.
  • the second rendered image may include a rendered image corresponding to the status information of the user's left eye in the second status information, and a rendered image corresponding to the status information of the user's right eye in the second status information
  • the first intermediate rendered image may include a rendering image corresponding to the status information of the user's right eye in the second status information.
  • the rendered image corresponding to the state information of the user's left eye in the second state information may be used to fill the rendered image corresponding to the user's left eye in the first intermediate rendered image to obtain rendered images xx21-xx61.
  • the rendered image corresponding to the state information of the user's right eye in the second state information may be used to fill the rendered image corresponding to the user's right eye in the first intermediate rendered image to obtain rendered images xx22-xx62.
  • the second rendered images corresponding to the second state information may include rendered images xx21-xx61 and rendered images xx22-xx62. In this way, high-quality rendering can be guaranteed.
  • step S69 may include:
  • Step S40 Determine the position of the virtual viewpoint according to the second state information and the viewing distance; intercept the line segment between the virtual viewpoint and the far clipping plane between the near clipping plane and the far clipping plane, and determine a plurality of two-dimensional line segments;
  • the three-dimensional line segment is projected to the viewing plane corresponding to the virtual viewpoint to obtain a plurality of two-dimensional coordinates, wherein each two-dimensional line segment corresponds to a plurality of two-dimensional coordinates; the depth information of the plurality of two-dimensional coordinates is obtained; the plurality of two-dimensional coordinates
  • the depth information of the first rendered image is compared with the depth information of the first rendered image, and coordinates with the same depth value are determined, wherein the depth information includes the depth value corresponding to each coordinate.
  • a second intermediate rendered image corresponding to the second state information is obtained according to texture information in the first rendered image at coordinates with the same depth value.
  • the rendered images x21-x61 corresponding to the user's left eye can be correspondingly obtained, and according to the coordinates of the points corresponding to the same depth value of the user's right eye
  • the coordinates and the first rendered image can correspond to the rendered images x22-x62 corresponding to the user's right eye
  • the second intermediate rendered image can include the above-mentioned rendered images x21-x61 corresponding to the user's left eye and the rendered image corresponding to the user's right eye x22-x62.
  • the second intermediate rendered image is filled with the second rendered image to obtain a rendered image corresponding to the second state information.
  • the second rendered image may include a rendered image corresponding to the state information of the user's left eye in the second state information, and a rendered image corresponding to the state information of the user's right eye in the second state information, and the second intermediate rendered image may include a rendering image corresponding to the state information of the user's right eye in the second state information.
  • a rendered image for the user's left eye and a rendered image for the user's right eye may be filled with the second rendered image to obtain a rendered image corresponding to the second state information.
  • the rendered image corresponding to the state information of the user's left eye in the second state information may be used to fill the rendered image corresponding to the user's left eye in the second intermediate rendered image to obtain rendered images xx21-xx61.
  • the rendered image corresponding to the state information of the user's right eye in the second state information can be used to fill in the rendered image corresponding to the user's right eye in the second intermediate rendered image to obtain rendered images xx22-xx62.
  • the second rendered images corresponding to the second state information may include rendered images xx21-xx61 and rendered images xx22-xx62.
  • the device where the terminal device is located can display the first rendered image x11, another rendered image x12, and rendered images xx21-xx61 corresponding to the second state information in the display area of the device where the terminal device is located according to the order in which state information g1-g6 is collected , xx22-xx62, for example, x11, xx21-xx61 are displayed in the display area corresponding to the left eye, and x12, xx22-xx62 are displayed in the display area corresponding to the right eye, so that the user can observe the virtual world in three-dimensional space through the display area.
  • images corresponding to the same state information can be displayed simultaneously.
  • the image displayed in the display area is consistent with the change of the state information caused by the change of the user's action, and the user experience is improved.
  • Directly using the first rendered image for display can reduce the data processing cost of the VR device.
  • the cloud server can include a remote device, and the VR device can include a terminal device.
  • the remote device can run the server of the application, and the terminal device can run the client of the application.
  • the application can include multiple 3D models.
  • the application could be a VR application.
  • the remote device and the terminal device can cooperate to implement rendering.
  • FIG. 15 shows an example of working modes of a remote device and a terminal device in an application scenario of a cloud VR system according to an embodiment of the present application.
  • the remote device can perform steps S73-S78:
  • S73 Receive first state information g1, where the first state information g1 includes position information and attitude information of the terminal device at a first moment.
  • step S73 For the implementation of step S73, reference may be made to the example of step S33 above, which will not be repeated here.
  • step S74 For the implementation of step S74, reference may be made to the example of step S34 above, which will not be repeated here.
  • the state information g11 corresponding to the user's left eye and the state information g12 corresponding to the user's right eye can be obtained first, and the first rendered image x11 can be obtained according to the state information g11 of the user's left eye (or the state of the right eye).
  • Information g12) obtained obtained.
  • the first rendered image x11 is obtained according to the state information g11 of the user's left eye.
  • step S75 For the implementation of step S75, reference may be made to the example of step S35 above, which will not be repeated here.
  • step S75 may also include S79-S80:
  • S79 Synthesize the first rendered image and the depth information of the first rendered image into a rendered frame
  • S80 Send the rendered frame to the terminal device.
  • steps S79 and S80 reference may be made to the examples of steps S4 and S5 above.
  • the rendered frame when sending the first rendered image and the depth information of the first rendered image to the terminal device, the rendered frame may be encoded at the first bit rate, and the encoded result corresponding to the rendered frame may be obtained and then sent.
  • the method further includes:
  • Step S801 determine that the last rendered image sent to the terminal device is separated from the first rendered image x11 by a specified number of images; or, determine that the bandwidth margin for data transmission to the terminal device at the moment when the first rendered image is obtained is greater than or equal to the bandwidth threshold.
  • a preset condition can be set so that the remote device determines that when the first rendered image x11 meets the preset condition, it sends the first rendered image x11 and the depth information x11' of the first rendered image to the terminal device.
  • the preset condition may include: determining that there is a specified number of images between the last rendered image sent to the terminal device and the first rendered image x11; or determining that the bandwidth margin for transmitting data to the terminal device at the moment when the first rendered image is obtained is greater than or equal to the bandwidth threshold.
  • the state information collected by the device where the terminal device is located may be directly sent by the terminal device to the remote device, and the remote device judges whether the rendered image obtained according to the state information and the depth information of the rendered image are sent to the terminal device.
  • the remote device may perform step S74 to first obtain the first rendered image x11 and the depth information x11' of the first rendered image, and then determine whether the first rendered image x11 satisfies Preset conditions, and when the conditions are satisfied, step S75 is executed to send the first rendered image x11 and the depth information x11' of the first rendered image to the terminal device.
  • the terminal device does not need to select the state information sent to the remote device, and the data processing cost of the terminal device can be reduced.
  • S76 Receive second state information, where the second state information includes position information and attitude information of the terminal device at a second moment after the first moment.
  • step S76 For the implementation of step S76, reference may be made to the example of step S59 above, which will not be repeated here.
  • step S77 can refer to the example of step S2 above, wherein the terminal device can obtain the state information corresponding to the user's left eye and the state information corresponding to the user's right eye according to the second state information, and the state information corresponding to the user's left eye State information and state information corresponding to the user's right eye and at least one 3D model are rendered, and the obtained second rendered image may include the rendered images x21-x61 corresponding to the state information of the user's left eye in the second state information, and the second The rendered images x22-x62 corresponding to the state information of the user's right eye in the state information.
  • step S78 reference may be made to the example of step S61 above.
  • the remote device can also perform steps S81-S82:
  • step S81 may refer to the example of step S2 above, wherein another rendered image may be obtained according to the state information g12 of the user's right eye in the first state information. That is, the first rendered image x11 and the other rendered image x12 may respectively correspond to state information g11 of the user's left eye and state information g12 of the user's right eye in the first state information.
  • the terminal device can perform steps S83-S87:
  • step S83 refer to the example of step S6, where the terminal device may not set a preset condition for sending the first status information.
  • the first rendered image x11 is obtained by the remote device through rendering according to the first state information g1 and at least one 3D model of.
  • step S84 can refer to the example of step S66, wherein the first rendered image can be obtained according to the state information of the user's left eye in the first state information.
  • S85 Send second state information to the remote device, where the second state information includes position information and posture information of the terminal device at a second moment after the first moment.
  • step S85 reference may be made to the example of step S27, wherein the terminal device may not set a preset condition for sending the second state information.
  • the second rendered image may include a rendered image corresponding to the state information of the user's left eye in the second state information, and a rendered image corresponding to the state information of the user's right eye in the second state information.
  • the first rendered image x11 is an image corresponding to the state information g11 of the user's left eye in the first state information g1, and can be directly displayed on the display area corresponding to the left eye of the device where the terminal device is located. It can be seen from the above description that the remote device also sends another rendered image x12 corresponding to the state information g12 of the user's right eye in the first state information g1 to the terminal device at a second lower bit rate.
  • the second bit rate has a great influence on the quality of the rendered image, and may not be suitable for displaying directly on the display area of the device where the terminal device is located.
  • step S87 may include:
  • Receive another rendered image sent by the remote device The other rendered image is obtained by the remote device through rendering according to the first state information and the application.
  • the distance between the virtual viewpoint corresponding to the first rendered image and the other rendered image is equal to the viewing distance .
  • the point cloud data is rendered to obtain a third intermediate rendered image.
  • the third intermediate rendered image is filled with another rendered image sent by the remote device to obtain another rendered image corresponding to the first state information.
  • step S87 may include:
  • step S53 Obtain a fourth intermediate rendering image corresponding to the first state information according to texture information in the first rendering image at coordinates with the same depth value.
  • the fourth intermediate rendered image is filled with another rendered image sent by the remote device to obtain another rendered image xx12 corresponding to the first state information.
  • the terminal device sends the second rendered image to the terminal device at a lower second bit rate. Since the lower second bit rate has a greater impact on the quality of the rendered image, it may not be suitable for direct rendering on the terminal device. displayed in the display area of the device where it is located.
  • step S87 may include S88-S90:
  • step S88 Generate point cloud data according to the first rendered image x11 and the depth information x11' of the first rendered image. For its implementation, reference may be made to the example of step S70 above, which will not be repeated here.
  • the rendered images corresponding to the second state information may include rendered images xx21-xx61 and rendered images xx22-xx62.
  • step S69 may include:
  • Step S69 Determine the position of the virtual viewpoint according to the second state information and the viewing distance; intercept the line segment between the virtual viewpoint and the far clipping plane between the near clipping plane and the far clipping plane, and determine a plurality of two-dimensional line segments;
  • the three-dimensional line segment is projected to the viewing plane corresponding to the virtual viewpoint to obtain a plurality of two-dimensional coordinates, wherein each two-dimensional line segment corresponds to a plurality of two-dimensional coordinates; the depth information of the plurality of two-dimensional coordinates is obtained; the plurality of two-dimensional coordinates
  • the depth information of is compared with the depth information of the first rendered image to determine the coordinates with the same depth value, wherein the depth information includes the depth value corresponding to each coordinate; according to the texture information of the coordinates with the same depth value in the first rendered image, obtain A second intermediate rendering image corresponding to the second state information; filling the second intermediate rendering image with the second rendering image to obtain a rendering image corresponding to the second state information.
  • the device where the terminal device is located can display the first rendered image x11, another rendered image xx12 corresponding to the first state information and the second rendered image xx12 corresponding to the second state information in the display area of the device where the terminal device is located according to the order of collecting state information g1-g6.
  • the rendered images xx21-xx61, xx22-xx62, for example, x11, xx21-xx61 are displayed in the display area corresponding to the left eye, and xx12, xx22-xx62 are displayed in the display area corresponding to the right eye, so that the user can observe the three-dimensional space through the display area virtual world.
  • images corresponding to the same state information can be displayed simultaneously.
  • the image displayed in the display area is consistent with the change of the state information caused by the change of the user's action, and the user experience is improved.
  • Directly using the first rendered image for display can reduce the data processing cost of the VR device.
  • the encoding result of the first rendered image and the encoding result of the second rendered image that the remote device transmits downlink to the terminal device are transmitted serially, but Since the data volume of the encoding result of the second rendered image is small, the decoding speed of the encoding result of the second rendered image is faster than that of the encoding result of the first rendered image, which is originally later than that of the first rendered image on the remote device.
  • the obtained second rendered image may be decoded and obtained on the terminal device earlier than the first rendered image.
  • the second rendered image can be saved first, and the corresponding first rendered image can be obtained before rendering based on the first rendered image, and according to The second rendered image performs enhancement of the rendered image.
  • the first rendered image may be saved first, and at least one of the corresponding second rendered images may be acquired After one, render according to the first rendered image, and enhance the rendered image according to the second rendered image, or complete the rendering first to obtain the rendered image corresponding to the second state information, and wait for the corresponding second rendered image to be decoded
  • image effect enhancement those skilled in the art should understand that as long as a frame of second rendered image used for effect enhancement is collected at the same time as the image object (rendered image corresponding to the second state information) for effect enhancement, The present application does not limit the specific sequence of effect enhancement using the second rendered images. This mechanism also applies to the examples above that include a related description of another rendered image (eg, FIGS. 8 , 11 , 12 , 13 , 15 ).
  • the embodiment of the present application brings benefits of reducing network bandwidth, reducing delay and increasing frame rate.
  • high-bit-rate video streams are used to transmit content, which will occupy relatively high network bandwidth.
  • the high consumption of network bandwidth poses challenges to the popularization of cloud VR/AR.
  • the size of the first rendered image at the first bit rate is the same as the video frame
  • the second rendered image and/or another rendered image of the second bit rate is 1/5-1/6 of the video frame, which can reduce the bandwidth consumption of cloud VR/AR by about 60%, which will greatly reduce cloud VR/AR operating costs, promoting the widespread application of cloud VR/AR.
  • Fig. 16 shows an exemplary schematic diagram of a rendering method according to an embodiment of the present application.
  • an embodiment of the present application provides a rendering method, the method is applied to a remote device, the rendering system where the remote device is located also includes a terminal device, and the remote device runs the At the server end, the application includes a plurality of three-dimensional models, and the method includes: S3310, receiving first state information, the first state information including position information and posture information of the terminal device at the first moment; S3320, according to Render the first state information and at least one 3D model, and obtain a first rendered image and depth information of the first rendered image; S3330, send the first rendered image and the depth information of the first rendered image to the terminal device.
  • the remote device can refer to the example of the cloud server above
  • the terminal device can refer to the example of the VR/AR device above
  • the application can refer to the example of the VR/AR application above
  • the server of the application can refer to the above service
  • the multiple 3D models included in the application can refer to the example of the 3D models of objects or characters preset in the VR/AR application above.
  • step S3310 For an exemplary implementation manner of step S3310, reference may be made to relevant descriptions of steps S1, S16, S33, S43, S56, and S73 above.
  • step S3320 reference may be made to relevant descriptions of steps S2, S17, S34, S44, S57, and S74 above.
  • step S3330 reference may be made to relevant descriptions of steps S3, S18, S35, S45, S58, and S75 above.
  • the remote device runs the server of the application, and the application includes multiple 3D models, so that the remote device can use at least one 3D model for rendering; the remote device uses at least one 3D model and received The received first state information can be rendered to obtain the first rendered image and the depth information of the first rendered image, and the remote device sends the first rendered image and the depth information of the first rendered image to the terminal device so that the terminal device can use the first The depth information of the rendered image and the first rendered image obtains multiple frames of images that can be displayed to the user.
  • the remote device only needs to transmit the first rendered image and the depth information of the first rendered image once, and can obtain multiple frames of images correspondingly on the terminal device, so that the remote device transmits the first rendered image and the depth information of the first rendered image.
  • the frequency is reduced.
  • the remote device corresponds to the cloud server and the terminal device corresponds to the VR/AR device
  • the bandwidth consumption of data transmission between the cloud and the VR/AR device can be reduced, reducing the cost of data transmission.
  • the first rendered image may include one frame of image.
  • executing the rendering method of the embodiment of the present application can enable the remote device to output the first rendered image in the cloud AR scene, so that when the terminal device receives the first rendered image Can make a response to realize the image display of the display area, and can enable the remote device to output the first rendered image in the cloud VR scene, so that the terminal device can respond when receiving the first rendered image to realize the corresponding corresponding to the user's left eye and An image of at least one of the two display areas for the right eye is displayed.
  • the first rendered image may include two frames of images.
  • executing the rendering method of the embodiment of the present application can enable the remote device to output the first rendered image in the cloud VR scene, so that when the terminal device receives the first rendered image It is possible to respond to realize image display of the two display areas respectively corresponding to the user's left and right eyes.
  • the sending the first rendered image and the depth information of the first rendered image to the terminal device includes: transmitting the first rendered image and the The depth information of the first rendered image is sent to the terminal device; after receiving the first state information, the method further includes: receiving second state information, the second state information includes the terminal device in the first state information Position information and posture information at a second moment after a moment; rendering according to the second state information and the at least one 3D model to obtain a second rendered image; using a second code rate lower than the first code rate sending the second rendered image to the terminal device at a rate.
  • the first rendered image and the depth information of the first rendered image are sent to the terminal device at a first bit rate, for example, reference may be made to the above steps S18, S58, and S75.
  • receiving the second state information reference may be made to the examples of steps S19, S59, and S76 above.
  • performing rendering according to the second state information and the at least one 3D model to obtain a second rendered image reference may be made to the examples of steps S20, S60, and S77 above.
  • sending the second rendered image to the terminal device at a second code rate lower than the first code rate reference may be made to the examples of steps S21, S61, and S78 above.
  • the second state information includes the position information and posture information of the terminal device at a second moment after the first moment, and the state information is also used to obtain a second rendered image, so the second rendered image corresponds to the rendered image at the second moment image, by transmitting the second rendered image at a lower second bit rate, so that the second rendered image has a smaller amount of data, and the terminal device can obtain the rendered image corresponding to the first moment on the basis of The rendered image at the second moment can reduce the downlink transmission data cost of the remote device while ensuring that the user has a better experience in using the terminal device.
  • the second rendered image may include one frame of image.
  • executing the rendering method of the embodiment of the present application can enable the remote device to output the first rendered image and the second rendered image in the cloud AR scene, so that the terminal device can receive When the first rendered image and the second rendered image can respond to realize the image display of a display area corresponding to the user's left eye and right eye, and enable the remote device to output the first rendered image and the second rendered image in the cloud VR scene
  • the images are rendered, so that the terminal device can respond when receiving the first rendered image and the second rendered image to realize image display in at least one of the two display areas respectively corresponding to the left eye and the right eye of the user.
  • the second rendered image includes two frames of rendered images, and the distance between virtual viewpoints corresponding to the two frames of rendered images is equal to the viewing distance.
  • the second rendered image may include two frames of images.
  • executing the rendering method of the embodiment of the present application can enable the remote device to output the first rendered image and the second rendered image in the cloud VR scene, so that the terminal device can receive
  • the first rendered image and the second rendered image can respond, so as to realize the image display in the two display areas respectively corresponding to the left eye and the right eye of the user, and at the same time ensure the accuracy of the image corresponding to the second state information among the images displayed in the display area quality.
  • the method further includes: performing rendering according to the first state information and the at least one 3D model to obtain another rendered image, the virtual viewpoint corresponding to the first rendered image and the The distance between the virtual viewpoints corresponding to the other rendered image is equal to the viewing distance; the another rendered image is sent to the terminal device at a second bit rate lower than the first bit rate, and the first rendered image is The first code rate is sent to the terminal device.
  • Step S46 and S81 For sending the other rendered image to the terminal device at a second code rate lower than the first code rate, reference may be made to the examples of steps S47 and S82 above.
  • the execution of the rendering method of the embodiment of the present application enables the remote device to output the first rendered image and the other rendered image in the cloud VR scene, so that the terminal device receives the first rendered image and the other rendered image Responses can be made to realize image display in the two display areas respectively corresponding to the user's left eye and right eye, while ensuring the quality of the image corresponding to the first state information among the images displayed in the display area.
  • the method before sending the first rendered image and the depth information of the first rendered image to the terminal device at the first bit rate, the method further includes: determining to send to A specified number of images are separated between the last rendered image of the terminal device and the rendered image; or, determining that the bandwidth margin for transmitting data to the terminal device at the moment when the rendered image is obtained is greater than or equal to a bandwidth threshold.
  • the specified number can be preset according to the bandwidth upper limit and the bandwidth threshold of data transmitted from the remote device to the terminal device, so that the first rendered image is sent at intervals of the specified number of images, without causing the remote device to The bandwidth margin for data transmission to the terminal device is less than the bandwidth threshold.
  • the remote device by first determining the interval of a specified number of images between the last rendered image sent to the terminal device and the rendered image, and then converting the first rendered image and the second rendered image at a first code rate
  • the depth information of a rendered image is sent to the terminal device, so that the remote device will not output the first rendered image and the depth information of the first rendered image when the bandwidth margin is too small, which can keep the remote device from transmitting data information to the terminal device smoothness, and image display on the terminal device will not be delayed due to the first rendered image and the depth information of the first rendered image not being received in time.
  • the bandwidth margin for transmitting data to the terminal device at the moment when the rendered image is obtained is greater than or equal to a bandwidth threshold, and then sending the first rendered image and the depth information of the first rendered image at a first bit rate To the terminal device, so that the remote device will not output the first rendered image and the depth information of the first rendered image when the bandwidth margin is too small, the smoothness of data information transmission from the remote device to the terminal device can be maintained, and the terminal device The display of the image on will not be delayed due to the first rendered image and the depth information of the first rendered image not being received in time.
  • the sending the first rendered image and the depth information of the first rendered image to the terminal device includes: sending the first rendered image and the depth information of the first rendered image Synthesizing the rendered frame with the depth information; sending the rendered frame to the terminal device.
  • steps S4, S22, S36, S48, S62, and S79 For an implementation manner of synthesizing the first rendered image and the depth information of the first rendered image into a rendered frame, reference may be made to the examples of steps S4, S22, S36, S48, S62, and S79 above.
  • steps S5, S23, S37, S49, S63, and S80 For an implementation manner of sending the rendered frame to the terminal device, reference may be made to the examples of steps S5, S23, S37, S49, S63, and S80 above.
  • Fig. 17 shows an exemplary schematic diagram of a rendering method according to an embodiment of the present application.
  • the present application further provides a rendering method, the method is applied to a terminal device, the rendering system where the terminal device is located further includes a remote device, and the terminal device runs an application client,
  • the application includes multiple three-dimensional models, and the method includes: S3410, sending first state information to the remote device, where the first state information includes position information and posture information of the terminal device at a first moment; S3420. Receive a first rendered image sent by the remote device and depth information of the first rendered image, where the first rendered image is performed by the remote device according to the first state information and at least one 3D model Obtained by rendering; S3430. Obtain multiple frames of rendered images according to the first rendered image and the depth information of the first rendered image.
  • step S3410 reference may be made to the examples of steps S6, S38, and S51 above.
  • step S3420 reference may be made to the examples of steps S7, S39, and S52 above.
  • step S3430 reference may be made to the examples of steps S8, S40, and S53 above.
  • the terminal device sends the first state information including the position information and posture information of the terminal device at the first moment to the remote device, so that the remote device State information, which can be rendered to obtain the first rendered image and the depth information of the first rendered image, and output to the terminal device, so that the terminal device can receive and use the first rendered image and the depth information of the first rendered image to obtain multiple frames Render the image.
  • the terminal device only needs to transmit the status information once, and the remote device only needs to transmit the first rendered image and the depth information of the first rendered image once, and then the terminal device can obtain multiple frames of rendered images correspondingly, so that the remote device transmits the first
  • the frequency of the depth information of the rendered image and the first rendered image is reduced.
  • the first rendered image may include one frame of image.
  • executing the rendering method of the embodiment of the present application can enable the terminal device to respond when receiving the first rendered image in the cloud AR scene to realize image display in the display area. , and enable the remote device to output the first rendered image in the cloud VR scene, so that the terminal device can respond when receiving the first rendered image to realize at least An image display of the display area.
  • the first rendered image may include two frames of images.
  • executing the rendering method of the embodiment of the present application can enable the terminal device to respond when receiving the first rendered image in the cloud VR scene so as to respectively correspond to the left eye of the user.
  • the images of the two display areas of the right eye and the right eye are displayed.
  • the obtaining the multi-frame rendered image according to the first rendered image and the depth information of the first rendered image includes: according to the first rendered image, the Depth information and second state information, obtaining a rendered image corresponding to the second state information, where the second state information includes position information and attitude information of the terminal device at a second moment after the first moment.
  • the second state information includes position information and attitude information of the terminal device at a second moment after the first moment.
  • the rendered image corresponding to the second state information can be obtained according to the second state information, and the second state information includes the position information and posture information of the terminal device at the second moment after the first moment, therefore, there are many
  • the rendered images corresponding to multiple different second state information can be obtained.
  • the rendered image corresponding to the different second state information can meet the requirement of presenting the rendered image conforming to the user's action to the user, so as to ensure the user experience.
  • the obtaining multiple frames of rendered images according to the first rendered image and depth information of the first rendered image includes:
  • the depth information is to obtain another rendered image corresponding to the first state information, and the distance between the first rendered image and the virtual viewpoint corresponding to the other rendered image is equal to the viewing distance.
  • the execution of the rendering method of the embodiment of the present application enables the terminal device to respond when receiving the first rendered image in the cloud VR scene, so as to use the first rendered image and another rendered image to correspond to the user's left and right images respectively.
  • the obtaining the rendered image corresponding to the second state information according to the first rendered image, the depth information of the first rendered image, and the second state information includes: according to the The first rendered image and the depth information of the first rendered image generate point cloud data; perform rendering according to the second state information and the point cloud data to obtain a rendered image corresponding to the second state information.
  • the point cloud data may include a plurality of three-dimensional points, which may represent information such as the position and shape of the object in the first rendered image in the three-dimensional scene.
  • the points that the user can see when observing the 3D scene under the second state information can be obtained, and according to the obtained points and the first rendered image, it can be obtained that under the second state information, when the user observes the 3D scene images that can be seen.
  • An image for presentation to the user can thus be obtained.
  • Fig. 18 shows an exemplary schematic diagram of a rendering method according to an embodiment of the present application.
  • the embodiment of the present application also provides a rendering method, the method is applied to a terminal device, the rendering system where the terminal device is located also includes a remote device, the terminal device runs a client of an application, and the application includes multiple 3D model, the method includes: S3510, sending first state information to the remote device, the first state information including the position information and posture information of the terminal device at the first moment; S3520, receiving the remote The first rendered image sent by the device and the depth information of the first rendered image, where the first rendered image is obtained by the remote device through rendering according to the first state information and at least one 3D model; S3530, send Send second state information to the remote device, the second state information includes position information and attitude information of the terminal device at a second moment after the first moment; S3540, receiving a second rendered image, the The second rendered image is obtained by the remote device through rendering according to the second state information and the at least one 3D model; S3550, according to the first rendered image, the depth information of the first rendered image, the The second state information and the second rendered
  • step S3510 For the implementation of step S3510, reference may be made to the examples of steps S25, S65, and S83 above.
  • step S3520 For the implementation of step S3520, reference may be made to the examples of steps S26, S66, and S84 above.
  • step S3530 For the implementation of step S3530, reference may be made to the examples of steps S27, S67, and S85 above. Those skilled in the art should understand that step S3530 may also be executed before step S3520, as long as step S3530 is executed later than step S3510, and this application does not limit the execution order of step S3530 and step S3520.
  • step S3540 For the implementation of step S3540, reference may be made to the examples of steps S28, S68, and S86 above.
  • step S3550 For the implementation of step S3550, reference may be made to the examples of steps S29, S69, and S87 above.
  • the terminal device sends the first state information including the position information and posture information of the terminal device at the first moment to the remote device, so that the remote device State information, which can be rendered to obtain the first rendered image and the depth information of the first rendered image, and output to the terminal device, and the terminal device sends the first state information including the position information and posture information of the terminal device at the second moment to the remote device , so that the remote device can render the second rendered image according to at least one 3D model and the received second state information and output it to the terminal device, so that the terminal device can receive and use the first rendered image, the first rendered image
  • the depth information of the second rendered image and the second state information on the terminal device are used to obtain a multi-frame rendered image. In this way, the quality of the obtained multi-frame rendered image can be guaranteed to be good, and the user experience can be improved when the multi-frame rendered image is displayed to the user.
  • the obtaining the multi-frame rendered image according to the first rendered image, the depth information of the first rendered image, the second state information, and the second rendered image includes: Generate point cloud data according to the first rendered image and depth information of the first rendered image; render the point cloud data according to the second state information to obtain a first intermediate rendered image; use the first rendered image The second rendered image fills the first intermediate rendered image to obtain a second rendered image corresponding to the second state information.
  • the point cloud data may include a plurality of three-dimensional points, which may represent information such as the position and shape of the object in the first rendered image in the three-dimensional scene.
  • the points that the user can see when observing the 3D scene under the second state information can be obtained, and according to the obtained points and the first rendered image, it can be obtained that under the second state information, when the user observes the 3D scene
  • the image that can be seen, the first intermediate rendered image is also obtained according to the second state information, based on the second rendered image, the first intermediate rendered image may be processed to obtain a second rendered image corresponding to the second state information. An image for presentation to the user can thus be obtained.
  • Fig. 19 shows an exemplary structural diagram of a remote device according to an embodiment of the present application.
  • the present application provides a remote device, where the rendering system of the remote device further includes a terminal device, and the remote device runs a server of an application, and the application includes multiple 3D model, the device includes: a first receiving module 701, configured to receive first state information, the first state information including position information and posture information of the terminal device at a first moment; a first rendering module 702, used Perform rendering according to the first state information and at least one 3D model, and obtain a first rendered image and depth information of the first rendered image; a first sending module 703, configured to send the first rendered image and the The depth information of the first rendered image is sent to the terminal device.
  • a first receiving module 701 configured to receive first state information, the first state information including position information and posture information of the terminal device at a first moment
  • a first rendering module 702 used Perform rendering according to the first state information and at least one 3D model, and obtain a first rendered image and depth information of the first rendered image
  • a first sending module 703, configured to send the first rendered image and the The depth information of the first
  • the sending the first rendered image and the depth information of the first rendered image to the terminal device includes: transmitting the first rendered image and the The depth information of the first rendered image is sent to the terminal device; after the first receiving module, the device further includes: a second receiving module, configured to receive second state information, the second state information includes the terminal device Position information and posture information at a second moment after the first moment; a second rendering module, configured to perform rendering according to the second state information and the at least one 3D model, and obtain a second rendered image; second A sending module, configured to send the second rendered image to the terminal device at a second code rate lower than the first code rate.
  • the second rendered image includes two frames of rendered images, and the distance between virtual viewpoints corresponding to the two frames of rendered images is equal to the viewing distance.
  • the device further includes: a third rendering module, configured to perform rendering according to the first state information and the at least one 3D model, and obtain another rendered image, and the first rendering The distance between the virtual viewpoint corresponding to the image and the virtual viewpoint corresponding to the other rendered image is equal to the viewing distance; the third sending module is configured to send the another rendered image at a second bit rate lower than the first bit rate To the terminal device, the first rendered image is sent to the terminal device at the first bit rate.
  • a third rendering module configured to perform rendering according to the first state information and the at least one 3D model, and obtain another rendered image, and the first rendering The distance between the virtual viewpoint corresponding to the image and the virtual viewpoint corresponding to the other rendered image is equal to the viewing distance
  • the third sending module is configured to send the another rendered image at a second bit rate lower than the first bit rate
  • the first rendered image is sent to the terminal device at the first bit rate.
  • the device before the first sending module, the device further includes: a first determining module, configured to determine the interval between the last rendered image sent to the terminal device and the rendered image by a specified interval number of images; or, determine that the bandwidth margin for transmitting data to the terminal device at the moment when the rendered image is obtained is greater than or equal to a bandwidth threshold.
  • a first determining module configured to determine the interval between the last rendered image sent to the terminal device and the rendered image by a specified interval number of images; or, determine that the bandwidth margin for transmitting data to the terminal device at the moment when the rendered image is obtained is greater than or equal to a bandwidth threshold.
  • the sending the first rendered image and the depth information of the first rendered image to the terminal device includes: sending the first rendered image and the depth information of the first rendered image Synthesizing the rendered frame with the depth information; sending the rendered frame to the terminal device.
  • FIG. 20 shows an exemplary structural diagram of a terminal device according to an embodiment of the present application.
  • the present application provides a terminal device, where the rendering system where the terminal device is located further includes a remote device, where the terminal device runs a client of an application, and the application includes multiple 3D models
  • the device includes: a fourth sending module 704, configured to send first status information to the remote device, the first status information including position information and attitude information of the terminal device at the first moment; the fourth receiving Module 705, configured to receive a first rendered image and depth information of the first rendered image sent by the remote device, where the first rendered image is obtained by the remote device according to the first state information and at least one
  • the 3D model is obtained by rendering;
  • the first image obtaining module 706 is configured to obtain multiple frames of rendered images according to the first rendered image and the depth information of the first rendered image.
  • the obtaining the multi-frame rendered image according to the first rendered image and the depth information of the first rendered image includes: according to the first rendered image, the Depth information and second state information, obtaining a rendered image corresponding to the second state information, where the second state information includes position information and attitude information of the terminal device at a second moment after the first moment.
  • the obtaining multiple frames of rendered images according to the first rendered image and depth information of the first rendered image includes: The depth information is to obtain another rendered image corresponding to the first state information, and the distance between the first rendered image and the virtual viewpoint corresponding to the other rendered image is equal to the viewing distance.
  • the obtaining the rendered image corresponding to the second state information according to the first rendered image, the depth information of the first rendered image, and the second state information includes: according to the The first rendered image and the depth information of the first rendered image generate point cloud data; perform rendering according to the second state information and the point cloud data to obtain a rendered image corresponding to the second state information.
  • Fig. 21 shows an exemplary structural diagram of a terminal device according to an embodiment of the present application.
  • the present application provides a terminal device, where the rendering system where the terminal device is located further includes a remote device, where the terminal device runs a client of an application, and the application includes multiple 3D models
  • the device includes: a fifth sending module 707, configured to send first status information to the remote device, the first status information including position information and attitude information of the terminal device at the first moment; the fifth receiving Module 708, configured to receive a first rendered image and depth information of the first rendered image sent by the remote device, where the first rendered image is obtained by the remote device according to the first state information and at least one obtained by rendering the three-dimensional model; the sixth sending module 709 is configured to send second state information to the remote device, where the second state information includes the terminal device at a second moment after the first moment Position information and posture information; a sixth receiving module 710, configured to receive a second rendered image, the second rendered image is obtained by the remote device through rendering according to the second state information and the at least one 3D model
  • a second image obtaining module 711 configured to obtain multi-frame rendered
  • the obtaining the multi-frame rendered image according to the first rendered image, the depth information of the first rendered image, the second state information, and the second rendered image includes: Generate point cloud data according to the first rendered image and depth information of the first rendered image; render the point cloud data according to the second state information to obtain a first intermediate rendered image; use the first rendered image The second rendered image fills the first intermediate rendered image to obtain a second rendered image corresponding to the second state information.
  • An embodiment of the present application provides a cluster of computing devices, including at least one computing device, and each computing device includes a processor and a memory, wherein the processor of the at least one computing device is configured to execute the above method.
  • An embodiment of the present application provides a terminal device, including: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions.
  • An embodiment of the present application provides a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is realized.
  • An embodiment of the present application provides a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium bearing computer-readable codes, when the computer-readable codes are stored in a processor of an electronic device When running in the electronic device, the processor in the electronic device executes the above method.
  • Fig. 22 shows an exemplary structural diagram of a computing device cluster according to an embodiment of the present application.
  • the present application also provides a computing device cluster.
  • the computing device cluster includes at least one computing device 2000 , and each computing device 2000 includes a bus 2001 , a processor 2002 , a communication interface 2003 and a memory 2004 .
  • the processor 2002 , the memory 2004 and the communication interface 2003 communicate through the bus 2001 .
  • Communication paths can be established between different computing devices 2000 through a communication network. Any one or more of the first receiving module 701 , the first rendering module 702 and the first sending module 703 run on each computing device 2000 .
  • Any computing device 2000 may be a computing device in a cloud environment, or a computing device in an edge environment, or a computing device in a terminal environment.
  • Fig. 23 shows an exemplary structural diagram of a terminal device according to an embodiment of the present application.
  • terminal devices may include desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (ultra-mobile personal computers, UMPCs), netbooks, personal digital assistants (personal digital assistants, PDAs) ), augmented reality (augmented reality, AR) equipment, virtual reality (virtual reality, VR) equipment, artificial intelligence (artificial intelligence, AI) equipment, wearable equipment, vehicle equipment, smart home equipment, or smart city equipment, servers at least one of the devices.
  • augmented reality augmented reality, AR
  • VR virtual reality
  • AI artificial intelligence
  • wearable equipment wearable equipment
  • vehicle equipment smart home equipment
  • smart city equipment servers at least one of the devices.
  • servers at least one of the devices.
  • the embodiment of the present application does not specifically limit the specific type of the computing device cluster.
  • the terminal device may include a processor 110 and a memory 121 . It can be understood that, the structure shown in the embodiment of the present application does not constitute a specific limitation on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.
  • the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit
  • the processor 110 can generate an operation control signal according to the instruction operation code and the timing signal, and complete the control of obtaining and executing the instruction.
  • a memory may also be provided in the processor 110 for storing instructions and data.
  • the memory in processor 110 may be a cache memory.
  • the memory may store instructions or data used by the processor 110 or used frequently. If the processor 110 needs to use the instruction or data, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.
  • the memory 121 may be used to store computer-executable program code including instructions.
  • the memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, an application program required by at least one function, and the like.
  • the storage data area can store data and the like created during the use of the terminal device.
  • the memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.
  • the processor 110 executes various function methods of the terminal device or the above-mentioned rendering methods by executing instructions stored in the memory 121 and/or instructions stored in the memory provided in the processor.
  • a computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.
  • a computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Non-exhaustive list of computer-readable storage media include: portable computer disk, hard disk, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), erasable Electrically Programmable Read-Only-Memory (EPROM or flash memory), Static Random-Access Memory (Static Random-Access Memory, SRAM), Portable Compression Disk Read-Only Memory (Compact Disc Read-Only Memory, CD -ROM), Digital Video Disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves with instructions stored thereon, and any suitable combination of the foregoing .
  • RAM Random Access Memory
  • ROM read only memory
  • EPROM or flash memory erasable Electrically Programmable Read-Only-Memory
  • Static Random-Access Memory SRAM
  • Portable Compression Disk Read-Only Memory Compact Disc Read-Only Memory
  • CD -ROM Compact Disc Read-Only Memory
  • DVD Digital Video Disc
  • Computer readable program instructions or codes described herein may be downloaded from a computer readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, local area network, wide area network, and/or wireless network.
  • the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • Computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (Instruction Set Architecture, ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages.
  • Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer can be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer such as use an Internet service provider to connect via the Internet).
  • electronic circuits such as programmable logic circuits, field-programmable gate arrays (Field-Programmable Gate Array, FPGA) or programmable logic arrays (Programmable Logic Array, PLA), the electronic circuit can execute computer-readable program instructions, thereby realizing various aspects of the present application.
  • These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.
  • each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts can be implemented with hardware (such as circuits or ASIC (Application Specific Integrated Circuit, application-specific integrated circuit)), or can be implemented with a combination of hardware and software, such as firmware.
  • hardware such as circuits or ASIC (Application Specific Integrated Circuit, application-specific integrated circuit)
  • firmware such as firmware

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请涉及一种渲染方法、远端装置、计算设备集群、终端装置及设备，所述方法应用于远端装置，所述远端装置所在的渲染系统还包括终端装置，所述远端装置运行应用的服务端，所述应用包括多个三维模型，所述方法包括：接收第一状态信息，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；根据所述第一状态信息和至少一个三维模型进行渲染，获得第一渲染图像和所述第一渲染图像的深度信息；发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置。根据本申请实施例的渲染方法，能够降低服务器与VR/AR设备之间数据传输的带宽消耗，使得数据传输成本降低。

Description

渲染方法、远端装置、计算设备集群、终端装置及设备
本申请要求于2021年9月3日提交中国专利局、申请号为202111032167.2、申请名称为“渲染方法、远端装置、计算设备集群、终端装置及设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像渲染技术领域，尤其涉及一种渲染方法、远端装置、计算设备集群、终端装置及设备。
背景技术
虚拟现实(virtual reality，VR)是一种通过计算机模拟出三维空间的虚拟世界，并向用户提供图像使得用户通过图像感受三维空间的虚拟世界的技术。增强现实(augmented reality，AR)是一种通过计算机将根据虚拟信息得到的图像融合于真实世界的技术。当用户观察视角或位置发生变化时，计算机会进行复杂的运算及图像渲染，将新视角或新位置下的图像提供给用户，让用户可以身临其境地感受虚拟现实/增强现实。现有的主流方案是，观察视角或位置的采集、新视角或新位置下的图像渲染、展示新视角或新位置下的图像给用户，都由VR/AR设备完成。
随着VR/AR业务的普及与VR/AR内容的丰富化，对VR/AR设备提出了更高的性能要求与体验要求。由此，云VR/AR应运而生，其特点是将图像渲染过程中的绝大部分放在云端完成，VR设备主要用于用户姿态(观察视角/位置)的感知、上报，图像渲染过程剩余的较少部分，以及新姿态(新视角/新位置)下图像的显示。云VR/AR降低了VR/AR设备的性能要求与设备成本，实现了轻量化的VR/AR设备，摆脱了传统VR/AR设备的有线连接、增加了设备的自由度。但云端的加入使得通过云端与VR/AR设备之间行数据交换时，也面临着网络传输带来的带宽消耗的挑战。
发明内容
有鉴于此，提出了一种渲染方法、远端装置、计算设备集群、终端装置及设备，根据本申请实施例的渲染方法，能够降低服务器与VR/AR设备之间数据传输的带宽消耗，使得数据传输成本降低。
第一方面，本申请的实施例提供了一种渲染方法，所述方法应用于远端装置，所述远端装置所在的渲染系统还包括终端装置，所述远端装置运行应用的服务端，所述应用包括多个三维模型，所述方法包括：接收第一状态信息，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；根据所述第一状态信息和至少一个三维模型进行渲染，获得第一渲染图像和所述第一渲染图像的深度信息；发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置。
根据本申请实施例的渲染方法，远端装置上运行应用的服务端，且应用包括多个三维模型，使得远端装置能够使用至少一个三维模型进行渲染；远端装置根据至少一个三维模型以 及接收到的第一状态信息，可以渲染得到第一渲染图像和第一渲染图像的深度信息，远端装置将第一渲染图像和第一渲染图像的深度信息至终端装置，使得终端装置可以使用第一渲染图像和第一渲染图像的深度信息得到可以展示给用户的多帧图像。远端装置只需传输一次第一渲染图像和第一渲染图像的深度信息，即可在终端装置上对应得到多帧图像，使得远端装置传输第一渲染图像和第一渲染图像的深度信息的频率得以降低，远端装置对应云端服务器、终端装置对应VR/AR设备时，能够降低云端与VR/AR设备之间数据传输的带宽消耗，使得数据传输成本降低。
第一渲染图像可以包括一帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云AR场景下远端装置能够输出第一渲染图像，以便终端装置接收到第一渲染图像时能够作出响应以实现显示区域的图像显示，并能够使得在云VR场景下远端装置能够输出第一渲染图像，以便终端装置接收到第一渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域中的至少一个显示区域的图像显示。
第一渲染图像可以包括两帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云VR场景下远端装置能够输出第一渲染图像，以便终端装置接收到第一渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域的图像显示。
根据第一方面，在所述渲染方法的第一种可能的实现方式中，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置；所述接收第一状态信息后，所述方法还包括：接收第二状态信息，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；根据所述第二状态信息和所述至少一个三维模型进行渲染，获得第二渲染图像；以低于所述第一码率的第二码率将所述第二渲染图像发送至所述终端装置。
第二状态信息包括所述终端装置在第一时刻之后的第二时刻的位置信息和姿态信息，且状态信息还用于得到第二渲染图像，因此第二渲染图像是对应于第二时刻的渲染图像，通过以较低的第二码率传输第二渲染图像，使得第二渲染图像具有较小的数据量，终端装置在获得对应于第一时刻的渲染图像的基础上，还可以获得对应于第二时刻的渲染图像，在保证用户使用终端装置具有较好的体验的同时，可以降低远端装置的下行传输数据成本。
第二渲染图像可以包括一帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云AR场景下远端装置能够输出第一渲染图像以及第二渲染图像，以便终端装置接收到第一渲染图像以及第二渲染图像时能够作出响应以实现对应用户左眼和右眼的一个显示区域的图像显示，并能够使得在云VR场景下远端装置能够输出第一渲染图像以及第二渲染图像，以便终端装置接收到第一渲染图像以及第二渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域中的至少一个显示区域的图像显示。
根据第一方面的第一种可能的实现方式，在所述渲染方法的第二种可能的实现方式中，所述第二渲染图像包括两帧渲染图像，所述两帧渲染图像对应的虚拟视点之间距离等于视距。
第二渲染图像可以包括两帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云VR场景下远端装置能够输出第一渲染图像以及第二渲染图像，以便终端装置接收到第一渲染图像以及第二渲染图像时能够作出响应，以实现分别对应用户左眼和右眼的两个显示区域的图像显示，同时保证显示区域显示的图像中与第二状态信息对应的图像的质量。
根据第一方面，以及以上第一方面的任意一种可能的实现方式，在所述渲染方法的第三 种可能的实现方式中，第一渲染图像是用于提供给用户左眼或右眼对应的显示区域的图像时，所述方法还包括：根据所述第一状态信息和所述至少一个三维模型进行渲染，获得另一渲染图像，所述第一渲染图像对应的虚拟视点和所述另一渲染图像对应的虚拟视点之间距离等于视距；以低于第一码率的第二码率将所述另一渲染图像发送至所述终端装置，所述第一渲染图像是以所述第一码率发送至所述终端装置。
通过这种方式，使得执行本申请实施例的渲染方法能够使得云VR场景下，远端装置能够输出第一渲染图像以及另一渲染图像，以便终端装置接收到第一渲染图像以及另一渲染图像时能够作出响应，以实现分别对应用户左眼和右眼的两个显示区域的图像显示，同时保证显示区域显示的图像中与第一状态信息对应的图像的质量。
根据第一方面的第一种至第三种可能的实现方式中的任意一种可能的实现方式，在所述渲染方法的第四种可能的实现方式中，所述以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置之前，所述方法还包括：确定发送至所述终端装置的上一渲染图像与所述第一渲染图像之间间隔指定个数的图像；或者，确定获得所述第一渲染图像时刻向所述终端装置传输数据的带宽余量大于或等于带宽阈值。
可以在执行所述渲染方法前先根据远端装置向终端装置传输数据的带宽上限以及带宽阈值预设好指定个数，使得间隔指定个数的图像发送第一渲染图像，不会使得远端装置向终端装置传输数据的带宽余量小于带宽阈值。在此情况下，通过先确定发送至所述终端装置的上一渲染图像与所述渲染图像之间间隔指定个数的图像，再以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置，使得远端装置不会在带宽余量过小时输出第一渲染图像和第一渲染图像的深度信息，可以保持远端装置向终端装置传输数据信息的顺畅度，并且终端装置上的图像显示不会因第一渲染图像和和第一渲染图像的深度信息未及时接收到而出现延迟。
通过先确定获得所述渲染图像时刻向所述终端装置传输数据的带宽余量大于或等于带宽阈值，再以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置，使得远端装置不会在带宽余量过小时输出第一渲染图像和第一渲染图像的深度信息，可以保持远端装置向终端装置传输数据信息的顺畅度，并且终端装置上的图像显示不会因第一渲染图像和第一渲染图像的深度信息未及时接收到而出现延迟。
根据第一方面，以及以上第一方面的任意一种可能的实现方式，在所述渲染方法的第五种可能的实现方式中，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：将所述第一渲染图像和所述第一渲染图像的深度信息合成渲染帧；发送所述渲染帧至所述终端装置。
通过将第一渲染图像和第一渲染图像的深度信息合成后传输，避免了位数较高的深度信息单独传输，可以降低远端装置的数据传输成本。
根据第一方面，以及以上第一方面的任意一种可能的实现方式，在所述渲染方法的第六种可能的实现方式中，所述根据所述第一状态信息和至少一个三维模型进行渲染，获得第一渲染图像和所述第一渲染图像的深度信息之前，所述方法还包括：确定发送至终端装置的上一渲染图像所对应的状态信息与第一状态信息之间间隔指定个数的状态信息；或者，确定接收第一状态信息时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
通过先确定发送至所述终端装置的上一渲染图像所对应的状态信息与第一状态信息之间 间隔指定个数的状态信息，再根据第一状态信息渲染获得第一渲染图像，以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置，使得远端装置不会在带宽余量过小时输出第一渲染图像和第一渲染图像的深度信息，可以保持远端装置向终端装置传输数据信息的顺畅度，并且终端装置上的图像显示不会因第一渲染图像和和第一渲染图像的深度信息未及时接收到而出现延迟。
通过先确定接收第一状态信息时刻向所述终端装置传输数据的带宽余量大于或等于带宽阈值，再根据第一状态信息渲染获得第一渲染图像，以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置，使得远端装置不会在带宽余量过小时输出第一渲染图像和第一渲染图像的深度信息，可以保持远端装置向终端装置传输数据信息的顺畅度，并且终端装置上的图像显示不会因第一渲染图像和第一渲染图像的深度信息未及时接收到而出现延迟。
第二方面，本申请的实施例提供了一种渲染方法，所述方法应用于终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述方法包括：发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像。
根据本申请实施例的渲染方法，终端装置发送包括终端装置在第一时刻的位置信息和姿态信息的第一状态信息到远端装置，使得远端装置根据至少一个三维模型以及接收到的第一状态信息，可以渲染得到第一渲染图像和第一渲染图像的深度信息，并输出给终端装置，进而使得终端装置可以接收到并使用第一渲染图像和第一渲染图像的深度信息，得到多帧渲染图像。终端装置只需传输一次状态信息，且远端装置只需传输一次第一渲染图像和第一渲染图像的深度信息，即可在终端装置上对应得到多帧渲染图像，使得远端装置传输第一渲染图像和第一渲染图像的深度信息的频率得以降低，远端装置对应云端服务器、终端装置对应VR/AR设备时，能够降低云端与VR/AR设备之间数据传输的带宽消耗，使得数据传输成本降低。
第一渲染图像可以包括一帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云AR场景下，终端装置接收到第一渲染图像时能够作出响应以实现显示区域的图像显示，并能够使得在云VR场景下远端装置能够输出第一渲染图像，以便终端装置接收到第一渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域中的至少一个显示区域的图像显示。
第一渲染图像可以包括两帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云VR场景下，终端装置接收到第一渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域的图像显示。
根据第二方面，在所述渲染方法的第一种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。
根据第二状态信息可以获得多帧渲染图像中，第二状态信息对应的渲染图像，且第二状态信息包括终端装置在第一时刻之后的第二时刻的位置信息和姿态信息，因此，存在多个不同的第二状态信息时，可以获得多个不同的第二状态信息对应的渲染图像，若多个不同的第二状态信息与第一状态信息是符合用户动作的状态信息，使得根据多个不同的第二状态信息对应的渲染图像，可以满足呈现与用户动作相符合的渲染图像给用户，保证用户体验。
根据第二方面，或第二方面的第一种可能的实现方式，在所述渲染方法的第二种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，获得所述第一状态信息对应的另一渲染图像，所述第一渲染图像和所述另一渲染图像对应的虚拟视点之间距离等于视距。
通过这种方式，使得执行本申请实施例的渲染方法能够使得云VR场景下，终端装置接收到第一渲染图像时能够作出响应，以实现使用第一渲染图像和另一渲染图像分别对应用户左眼和右眼的两个显示区域的图像显示，同时保证显示区域显示的图像中与第一状态信息对应、且与第一渲染图像不同的另一渲染图像的质量。
根据第二方面的第一种可能的实现方式，在所述渲染方法的第三种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第二状态信息和所述点云数据进行渲染，获得所述第二状态信息对应的渲染图像。
点云数据可以包括多个三维的点，可以表示第一渲染图像中的物体在三维场景中的位置、形状等信息。结合第二状态信息，可以得到在第二状态信息下，用户观察三维场景时可以看到的点，根据得到的点以及第一渲染图像，可以得到在第二状态信息下，用户观察三维场景时可以看到的图像。从而能得到用于展示给用户的图像。
根据第二方面的第一种可能的实现方式，在所述渲染方法的第四种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像包括：根据所述第二状态信息、预设的近裁剪面和远裁剪面，确定多个二维线段；根据所述第一状态信息对所述多个二维线段进行投影，得到多个二维坐标，其中，每一二维线段对应多个二维坐标；获得所述多个二维坐标的深度信息；将所述多个二维坐标的深度信息与所述第一渲染图像的深度信息进行比较，确定深度值相同的点的坐标，其中，所述深度信息包括各坐标对应的深度值；根据所述深度值相同的点的坐标和所述第一渲染图像，获得所述第二状态信息对应的渲染图像。
根据第二方面的第二种可能的实现方式，在所述渲染方法的第五种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息，获得所述第一状态信息对应的另一渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第一状态信息和视距，对所述点云数据进行渲染，获得所述第一状态信息对应的另一渲染图像。
根据第二方面的第二种可能的实现方式，在所述渲染方法的第六种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息，获得所述第一状态信息对应的另一渲染图像包括：根据所述第一状态信息和视距确定虚拟视点的位置；截取所述虚拟视点与远裁剪面之间的连线在近裁剪面和远裁剪面之间的线段，确定多个二维线段；将所述多个 二维线段投影至所述虚拟视点对应的视平面，得到多个二维坐标，其中，每一二维线段对应多个二维坐标；获得所述多个二维坐标的深度信息；将所述多个二维坐标的深度信息与所述第一渲染图像的深度信息进行比较，确定深度值相同的坐标，其中，所述深度信息包括各坐标对应的深度值；根据所述深度值相同的坐标在所述第一渲染图像中的纹理信息，获得所述第一状态信息对应的另一渲染图像。
第三方面，本申请的实施例提供了一种渲染方法，所述方法应用于终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述方法包括：发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；发送第二状态信息至所述远端装置，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；接收第二渲染图像，所述第二渲染图像是所述远端装置根据所述第二状态信息和所述至少一个三维模型进行渲染获得的；根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像。
根据本申请实施例的渲染方法，终端装置发送包括终端装置在第一时刻的位置信息和姿态信息的第一状态信息到远端装置，使得远端装置根据至少一个三维模型以及接收到的第一状态信息，可以渲染得到第一渲染图像和第一渲染图像的深度信息，并输出给终端装置，终端装置发送包括终端装置在第二时刻的位置信息和姿态信息的第一状态信息到远端装置，使得远端装置根据至少一个三维模型以及接收到的第二状态信息，可以渲染得到第二渲染图像并输出给终端装置，进而使得终端装置可以接收到并使用第一渲染图像、第一渲染图像的深度信息、第二渲染图像以及使用终端装置上的第二状态信息，得到多帧渲染图像。通过这种方式，可以保证得到的多帧渲染图像的质量较好，将多帧渲染图像展示给用户时，可以提升用户体验。
根据第三方面，在所述渲染方法的第一种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第二状态信息对所述点云数据进行渲染，获得第一中间渲染图像；利用所述第二渲染图像对所述第一中间渲染图像进行填补，获得所述第二状态信息对应的第二渲染图像。
点云数据可以包括多个三维的点，可以表示第一渲染图像中的物体在三维场景中的位置、形状等信息。结合第二状态信息，可以得到在第二状态信息下，用户观察三维场景时可以看到的点，根据得到的点以及第一渲染图像，可以得到在第二状态信息下，用户观察三维场景时可以看到的图像，即第一中间渲染图像。第二渲染图像也是根据第二状态信息得到的，基于第二渲染图像可以对第一中间渲染图像进行处理，得到第二状态信息对应的第二渲染图像。从而能得到用于展示给用户的图像。
根据第三方面，在所述渲染方法的第二种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像包括：根据所述第二状态信息和视距确定虚拟视点的位置；截取所述虚拟视点与远裁剪面之间的连线在近裁剪面和远裁剪面之间的线段，确定多个二维线段；将所述多个二维线段 投影至所述虚拟视点对应的视平面，得到多个二维坐标，其中，每一二维线段对应多个二维坐标；获得所述多个二维坐标的深度信息；将所述多个二维坐标的深度信息与所述第一渲染图像的深度信息进行比较，确定深度值相同的坐标，其中，所述深度信息包括各坐标对应的深度值；根据所述深度值相同的坐标在所述第一渲染图像中的纹理信息，获得所述第二状态信息对应的第二中间渲染图像；利用所述第二渲染图像对所述第二中间渲染图像进行填补，获得所述第二状态信息对应的渲染图像。
根据第三方面，以及以上第三方面的任意一种可能的实现方式，在所述渲染方法的第三种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像包括：接收所述远端装置发送的另一渲染图像，所述另一渲染图像是所述远端装置根据所述第一状态信息和所述至少一个三维模型进行渲染获得的，所述第一渲染图像和所述另一渲染图像对应的虚拟视点之间距离等于视距；根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第一状态信息和视距，对所述点云数据进行渲染，获得第三中间渲染图像；利用所述远端装置发送的另一渲染图像对所述第三中间渲染图像进行填补，获得所述第一状态信息对应的另一渲染图像。
根据第三方面，以及以上第三方面的任意一种可能的实现方式，在所述渲染方法的第四种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像包括：根据所述第一状态信息和视距确定虚拟视点的位置；截取所述虚拟视点与远裁剪面之间的连线在近裁剪面和远裁剪面之间的线段，确定多个二维线段；将所述多个二维线段投影至所述虚拟视点对应的视平面，得到多个二维坐标，其中，每一二维线段对应多个二维坐标；获得所述多个二维坐标的深度信息；将所述多个二维坐标的深度信息与所述第一渲染图像的深度信息进行比较，确定深度值相同的坐标，其中，所述深度信息包括各坐标对应的深度值；根据所述深度值相同的坐标在所述第一渲染图像中的纹理信息，获得所述第一状态信息对应的第四中间渲染图像；利用所述远端装置发送的另一渲染图像对所述第四中间渲染图像进行填补，获得所述第一状态信息对应的另一渲染图像。
第四方面，本申请实施例提供了一种远端装置，所述远端装置所在的渲染系统还包括终端装置，所述远端装置运行应用的服务端，所述应用包括多个三维模型，所述装置包括：第一接收模块，用于接收第一状态信息，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；第一渲染模块，用于根据所述第一状态信息和至少一个三维模型进行渲染，获得第一渲染图像和所述第一渲染图像的深度信息；第一发送模块，用于发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置。
根据第四方面，在所述远端装置的第一种可能的实现方式中，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置；第一接收模块之后，所述装置还包括：第二接收模块，用于接收第二状态信息，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；第二渲染模块，用于根据所述第二状态信息和所述至少一个三维模型进行渲染，获得第二渲染图像；第二发送模块，用于以低于所述第一码率的第二码率将所述第二渲染图像发送至所述终端装置。
根据第四方面的第一种可能的实现方式，在所述远端装置的第二种可能的实现方式中，所述第二渲染图像包括两帧渲染图像，所述两帧渲染图像对应的虚拟视点之间距离等于视距。
根据第四方面，以及以上第四方面的任意一种可能的实现方式，在所述远端装置的第三种可能的实现方式中，第一渲染图像是用于提供给用户左眼或右眼对应的显示区域的图像时，所述装置还包括：第三渲染模块，用于根据所述第一状态信息和所述至少一个三维模型进行渲染，获得另一渲染图像，所述第一渲染图像对应的虚拟视点和所述另一渲染图像对应的虚拟视点之间距离等于视距；第三发送模块，用于以低于第一码率的第二码率将所述另一渲染图像发送至所述终端装置，所述第一渲染图像是以所述第一码率发送至所述终端装置。
根据第四方面的第一种至第三种可能的实现方式中的任意一种可能的实现方式，在所述远端装置的第四种可能的实现方式中，第一发送模块之前，所述装置还包括：第一确定模块，用于确定发送至所述终端装置的上一渲染图像与所述渲染图像之间间隔指定个数的图像；或者，确定获得所述渲染图像时刻向所述终端装置传输数据的带宽余量大于或等于带宽阈值。
根据第四方面，以及以上第四方面的任意一种可能的实现方式，在所述远端装置的第五种可能的实现方式中，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：将所述第一渲染图像和所述第一渲染图像的深度信息合成渲染帧；发送所述渲染帧至所述终端装置。
第五方面，本申请的实施例提供了一种终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述装置包括：第四发送模块，用于发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；第四接收模块，用于接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；第一图像获得模块，用于根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像。
根据第五方面，在所述终端装置的第一种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。
根据第五方面，或第五方面的第一种可能的实现方式，在所述终端装置的第二种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，获得所述第一状态信息对应的另一渲染图像，所述第一渲染图像和所述另一渲染图像对应的虚拟视点之间距离等于视距。
根据第五方面的第一种可能的实现方式，在所述终端装置的第三种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第二状态信息和所述点云数据进行渲染，获得所述第二状态信息对应的渲染图像。
第六方面，本申请的实施例提供了一种终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述装置包括：第五发送模块，用于发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装 置在第一时刻的位置信息和姿态信息；第五接收模块，用于接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；第六发送模块，用于发送第二状态信息至所述远端装置，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；第六接收模块，用于接收第二渲染图像，所述第二渲染图像是所述远端装置根据所述第二状态信息和所述至少一个三维模型进行渲染获得的；第二图像获得模块，用于根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像。
根据第六方面，在所述终端装置的第一种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第二状态信息对所述点云数据进行渲染，获得第一中间渲染图像；利用所述第二渲染图像对所述第一中间渲染图像进行填补，获得所述第二状态信息对应的第二渲染图像。
第七方面，本申请的实施例提供了一种计算设备集群，其特征在于，包括至少一个计算设备，每个计算设备包括处理器和存储器，所述至少一个计算设备的处理器用于执行第一方面或者第一方面的多种可能的实现方式中的一种或几种的渲染方法。
第八方面，本申请的实施例提供了一种终端设备，包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器被配置为执行所述指令时实现上述第二方面或者第二方面的多种可能的实现方式中的一种或几种的渲染方法。
第九方面，本申请的实施例提供了一种终端设备，包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器被配置为执行所述指令时实现上述第三方面或者第三方面的多种可能的实现方式中的一种或几种的渲染方法。
第十方面，本申请实施例提供一种非易失性计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现上述第一方面或者第一方面的多种可能的实现方式中的一种或几种的渲染方法。
第十一方面，本申请的实施例提供了一种计算机程序产品，包括计算机可读代码，或者承载有计算机可读代码的非易失性计算机可读存储介质，当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行上述第一方面或者第一方面的多种可能的实现方式中的一种或几种的渲染方法。
本申请的这些和其他方面在以下(多个)实施例的描述中会更加简明易懂。
附图说明
包含在说明书中并且构成说明书的一部分的附图与说明书一起示出了本申请的示例性实施例、特征和方面，并且用于解释本申请的原理。
图1示出现有技术的云VR/AR系统的示意图。
图2a示出现有技术的云VR系统工作方式的示意图。
图2b示出现有技术的云AR系统工作方式的示意图。
图3示出根据本申请实施例的远端装置和终端装置在云AR系统的应用场景下的工作方式的一个示例。
图4示出根据本申请实施例的三维模型、第一状态信息获得第一渲染图像和第一渲染图像的深度信息的一个示例。
图5示出根据本申请实施例的第一渲染图像和第一渲染图像的深度信息合成渲染帧的一个示例。
图6示出根据本申请实施例确定点云数据的一个示例。
图7示出根据第二状态信息确定世界坐标系下的多条线段的一个示例。
图8示出根据本申请实施例的远端装置和终端装置在云AR系统的应用场景下的工作方式的一个示例。
图9示出根据本申请实施例的第一中间渲染图像出现空洞点的一个示例。
图10示出根据本申请实施例的第二渲染图像补充空洞点的一个示例。
图11示出根据本申请实施例的远端装置和终端装置在云VR系统的应用场景下的工作方式的一个示例。
图12示出根据本申请实施例的远端装置和终端装置在云VR系统的应用场景下的工作方式的一个示例。
图13示出根据本申请实施例的远端装置和终端装置在云VR系统的应用场景下的工作方式的一个示例。
图14示出根据本申请实施例的第二渲染图像的传输方式的一个示例。
图15示出根据本申请实施例的远端装置和终端装置在云VR系统的应用场景下的工作方式的一个示例。
图16示出根据本申请实施例的渲染方法的示例性示意图。
图17示出根据本申请实施例的渲染方法的示例性示意图。
图18示出根据本申请实施例的渲染方法的示例性示意图。
图19示出根据本申请实施例的远端装置的示例性结构示意图。
图20示出根据本申请实施例的终端装置的示例性结构示意图。
图21示出根据本申请实施例的终端装置的示例性结构示意图。
图22示出根据本申请实施例的计算设备集群的示例性结构示意图。
图23示出根据本申请实施例的终端设备的示例性结构示意图。
具体实施方式
以下将参考附图详细说明本申请的各种示例性实施例、特征和方面。附图中相同的附图标记表示功能相同或相似的元件。尽管在附图中示出了实施例的各种方面，但是除非特别指出，不必按比例绘制附图。
在这里专用的词“示例性”意为“用作例子、实施例或说明性”。这里作为“示例性”所说明的任何实施例不必解释为优于或好于其它实施例。
另外，为了更好的说明本申请，在下文的具体实施方式中给出了众多的具体细节。本领域技术人员应当理解，没有某些具体细节，本申请同样可以实施。在一些实例中，对于本领域技术人员熟知的方法、手段、元件和电路未作详细描述，以便于凸显本申请的主旨。
图1示出现有技术的云VR/AR系统的示意图。下面结合图1介绍现有技术的云VR/AR系 统。
如图1所示，现有技术的云VR/AR系统包括云端服务器以及VR/AR设备，云端服务器和VR/AR设备通过因特网进行数据传输。云端服务器可例如设置有处理器，可以是中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit,GPU)等支持图像渲染的多种类型的处理器。VR/AR设备上可例如设置有姿态传感器(未示出)，例如陀螺仪等，用于采集用户的姿态信息，VR/AR设备上还设置有显示装置(未示出)，例如对应左眼和右眼的显示屏，用于展示图像给用户的左眼和右眼。云端服务器上部署有VR/AR运行时(VR Runtime/AR Runtime，即VR/AR应用的运行环境)、VR/AR应用和服务端程序等程序或应用，VR/AR设备上部署有客户端程序等程序或应用，云端服务器与VR/AR设备的交互通过服务端程序与客户端程序之间的数据交换实现。
图1的云VR/AR系统可以作为本申请实施例的渲染方法的一个示例性应用场景。该应用场景下，执行本申请实施例的渲染方法的具体方式，可以参照下文图3-图15的相关描述。
图2a示出现有技术的云VR系统工作方式的示意图。下面结合图2a介绍现有技术的云VR系统的工作方式。如图2a所示，VR设备上，在VR设备启动后，客户端程序会主动与云端服务器的服务端程序建立连接，并以一定的频率进行姿态采集。以某一次采集时VR设备采集到的姿态信息G1为例，客户端程序将姿态信息G1传输至服务端程序。服务端程序会将姿态信息G1传输给VR运行时，VR运行时将姿态信息G1再转发给VR应用。VR应用会根据姿态信息G1进行图像渲染，根据三维数据(例如VR应用上预先设置的三维世界中的模型)得到适合于姿态信息G1的两张二维图像，两张二维图像分别用于展示给用户的左眼和右眼。
VR应用将渲染好的两张二维图像传输给服务端程序。服务端程序会根据左右眼的图像合成一张二维图像，使得合成后的图像的左半部分包括左眼图像，右半部分包括右眼图像。服务端程序可以对合成后的图像进行编码，例如采用视频编码的编码方式，得到对应的编码结果P1，并将编码结果P1传输至VR设备上的客户端程序。
客户端程序接收到编码结果P1后，根据编码结果P1进行图像解码，例如采用视频解码的解码方式，得到该左半部分包括左眼图像、右半部分包括右眼图像的一张二维图像，并对二维图像进行渲染，得到左眼图像和右眼图像，将左眼图像和右眼图像分别提交至对应左眼和右眼的显示屏的显示缓冲区。显示缓冲区中的二维图像数据被控制器(未示出)读取，并由控制器控制显示屏(未示出)进行图像显示，将图像数据绘制在显示屏上。在此情况下，用户即可通过对应左眼和右眼的显示屏观看到图像内容。通过对左眼和右眼的图像进行设置，可以使得用户观看图像内容时，左右两只眼睛看到的图像是不一样的，视觉上两个图像中的物体会有位置的偏差，叫做视差，用户的大脑会根据视差计算物体离身体的距离，于是产生了景深感，从而能身临其境地感受在姿态信息G1下的三维世界。
图2b示出现有技术的云AR系统工作方式的示意图。下面结合图2b介绍现有技术的云AR系统的工作方式。如图2b所示，AR设备上，在AR设备启动后，客户端程序会主动与云端服务器的服务端程序建立连接，并以一定的频率进行姿态采集。以某一次采集时AR设备采集到的姿态信息G1为例，客户端程序将姿态信息G1传输至服务端程序。服务端程序会将姿态信息G1传输给AR运行时，AR运行时将姿态信息G1再转发给AR应用。AR应用会根据姿态信息G1进行图像渲染，根据三维数据(例如AR应用上预先设置的三维世界中的模型)得到适合于姿态信息G1的一张二维图像。
AR应用将渲染好的二维图像传输给服务端程序。服务端程序可以对二维图像进行编码，例如采用视频编码的编码方式，得到对应的编码结果P1，并将编码结果P1传输至AR设备上的客户端程序。
客户端程序接收到编码结果P1后，根据编码结果P1进行图像解码，例如采用视频解码的解码方式，得到该二维图像，并对二维图像进行渲染，提交至显示缓冲区。显示缓冲区中的二维图像数据被控制器(未示出)读取，并由控制器控制图像显示在显示区域。在此情况下，用户即可在显示区域观看到融合在真实世界的虚拟图像。
现有技术的云VR/AR系统的缺点在于，云端服务器和VR/AR设备之间的数据传输主要通过因特网完成，为了保证用户的使用体验，VR/AR设备上显示的图像内容的分辨率通常在3K(3000≤图像横向像素数量≤3500)以上，图像的帧率通常要在60帧/秒以上，而想要保证3K的图像分辨率和60帧/秒的帧率，网络带宽需求最低为30Mbps。因此，使用现有技术的云VR/AR系统，用户想要流畅的体验云VR/AR带来的服务，通常采用高码率视频流方式传输内容，需要高于30Mbps的网络带宽环境，例如50-100Mbps。这会导致系统的带宽消耗过大，不利于控制设备间的数据传输成本。
有鉴于此，本申请提出一种渲染方法、远端装置、计算设备集群、终端装置及设备，根据本申请的渲染方法，能够降低云端服务器与VR/AR设备之间数据传输的带宽消耗，使得数据传输成本降低。
在一种可能的实现方式中，云AR系统的应用场景下，执行本申请的渲染方法可以实现，在AR设备上将图像展示给用户的左眼和右眼，该应用场景下，执行本申请实施例的渲染方法的具体方式，可以参照下文图3-图10的相关描述。
在另一种可能的实现方式中，云VR系统的应用场景下，执行本申请的渲染方法，可以实现，在VR设备上将图像展示给用户的左眼(或右眼)。该应用场景下，执行本申请实施例的渲染方法的具体方式，可以参照下文图3-图10的相关描述。
在另一种可能的实现方式中，云VR系统的应用场景下，执行本申请的渲染方法，可以实现在VR设备上将图像分别展示给用户的左眼和右眼。该应用场景下，执行本申请实施例的渲染方法的具体方式，可以参照下文图11-图15的相关描述。
以下分别结合不同的应用场景，对本申请实施例的渲染方法实现不同效果的示例性方式进行描述。
下面介绍云AR系统的应用场景下，执行本申请的渲染方法实现在AR设备上将图像展示给用户的左眼和右眼的一个示例。在一种可能的实现方式中，云端服务器上可包括远端装置，AR设备上可包括终端装置，远端装置可运行应用的服务端，终端装置可运行应用的客户端，应用可包括多个三维模型，本应用场景下，应用可以是AR应用。本申请实施例可以通过远端装置和终端装置协作实现渲染。图3示出根据本申请实施例的远端装置和终端装置在云AR系统的应用场景下的工作方式的一个示例。
如图3所示，远端装置可执行步骤S1-S3：
S1，接收第一状态信息g1，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S1中，第一状态信息g1可以是远端装置从终端装置处获取的。终端装置获取第一 状态信息g1的示例性方式可以参照下文步骤S6的相关描述。第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息，可以认为第一状态信息g1对应于第一时刻。
S2，根据第一状态信息g1和至少一个三维模型进行渲染，获得第一渲染图像x1和第一渲染图像的深度信息x1’。
步骤S2中，远端装置运行应用的服务端，可以获取到应用包括的三维模型。远端装置可例如获取到至少一个三维模型，并根据第一状态信息g1和获取到的至少一个三维模型进行渲染，获得第一渲染图像x1和第一渲染图像的深度信息x1’。可以认为第一状态信息g1是对应于第一渲染图像x1的状态信息。即，第一时刻、第一状态信息g1、第一渲染图像x1具有对应关系。对于云AR系统，同一时刻展示图像给用户的左眼和右眼的图像可以是一帧图像，因此，可以预设第一渲染图像x1包括一帧渲染图像。
图4示出根据本申请实施例的三维模型、第一状态信息获得第一渲染图像和第一渲染图像的深度信息的一个示例。
其中，根据第一状态信息g1和至少一个三维模型进行渲染时，可以结合渲染参数进行渲染。渲染参数可例如是终端装置的显示区域的宽高比等参数，渲染参数可以由终端装置主动传输到远端装置。第一渲染图像x1可以包括第一状态信息g1下，用户左眼和右眼能够看到的三维模型的纹理信息，第一渲染图像的深度信息x1’可以包括第一状态信息g1下，用户左眼和右眼能够看到的三维模型在场景中的深度信息。其中，第一渲染图像x1的宽高比可例如与渲染参数指示的宽高比相同。第一渲染图像的深度信息x1’可以包括该宽高比下的第一渲染图像x1中多个像素点对应的深度。
S3，发送第一渲染图像x1和第一渲染图像的深度信息x1’至终端装置。
步骤S3中，远端装置可以发送第一渲染图像x1到终端装置，以在终端装置上显示第一渲染图像x1’。由上文描述可知，第一渲染图像x1可对应于第一状态信息g1。远端装置可以同时发送第一渲染图像的深度信息x1’到终端装置。在终端装置上，第一渲染图像x1和第一渲染图像的深度信息x1’还可以用于渲染得到多帧渲染图像，其中多帧渲染图像可分别对应于除第一状态信息g1之外的、终端装置所在的设备(例如AR设备)采集到的其他状态信息(例如g2-g6)。
其中，远端装置可以对第一渲染图像x1和第一渲染图像的深度信息x1’设置标识，该标识可以表示第一渲染图像x1和第一渲染图像的深度信息x1’对应的状态信息。
可选地，步骤S3还可以包括S4-S5：
S4，将第一渲染图像x1和第一渲染图像的深度信息x1’合成渲染帧。
由上文描述可知，第一渲染图像x1可以包括三维模型的纹理信息，第一渲染图像的深度信息x1’可以包括三维模型在场景中的的深度信息。纹理信息的位数精度通常是8位，深度信息的位数精度通常是32位，因此需要分别传输，这种传输方式会占用较高的带宽。本申请实施例提出对32位的位数精度的数据进行处理，使其具有与纹理信息相同的位数，从而能够将纹理信息与深度信息共同传输，降低带宽消耗。
图5示出根据本申请实施例的第一渲染图像和第一渲染图像的深度信息合成渲染帧的一个示例。
举例来说，可以基于现有技术，设置一定的映射方式，使得根据32位的深度信息和预设的映射方式，将第一渲染图像的深度信息x1’转换为指示深度信息的纹理，使得该指示深度信 息的纹理与第一渲染图像x1具有相同的数据结构。在此情况下，该指示深度信息的纹理能够与第一渲染图像x1整合到一张图像上，整合后的一张图像即渲染帧。例如，渲染帧的左半部分可以包括第一渲染图像x1，渲染帧的右半部分可以包括该由第一渲染图像的深度信息x1’转化的指示深度信息的纹理。
本领域技术人员应理解，指示深度信息的纹理与第一渲染图像x1合成渲染帧可以有多种方式，例如分别作为渲染帧的左半部分或右半部分，或者分别作为渲染帧的上半部分或下半部分，只要能够满足在渲染帧上体现第一渲染图像x1和指示深度信息的纹理即可，本申请实施例对渲染帧上二者的具体位置不作限制。
远端装置可以对渲染帧设置标识，该标识可以表示渲染帧对应的状态信息，也即，第一渲染图像x1和第一渲染图像的深度信息x1’对应的状态信息。
S5，发送渲染帧至终端装置。
通过发送渲染帧，使得第一渲染图像x1和第一渲染图像的深度信息x1’可以同时传输到终端装置，第一渲染图像x1和第一渲染图像的深度信息x1’不必分别传输，可以减少带宽的消耗。
如图3所示，终端装置可执行步骤S6-S8：
S6，发送第一状态信息g1至远端装置，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S6中，终端装置所在的设备(例如AR设备)可以包括传感器，传感器能够以预设的频率采集状态信息，例如以60次/s的频率进行状态信息采集。每一时刻采集到的状态信息可以包括终端装置在该时刻的位置信息和姿态信息，可以认为不同状态信息是对应于不同的时刻。例如第一状态信息g1对应于第一时刻。第一状态信息g1可以发送到远端装置，使得远端装置可以接收到该包括终端装置在第一时刻的位置信息和姿态信息的第一状态信息g1。
可选地，可以设置预设条件，使得终端装置确定第一状态信息g1在满足预设条件时发送到远端装置。在不满足预设条件时，终端装置不发送第一状态信息g1。预设条件可以包括：确定第一状态信息g1与前次发送到远端装置的状态信息之间间隔预设个数的状态信息，或者确定第一时刻远端装置向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，终端装置所在的设备采集的状态信息可以由终端装置有选择地发送给远端装置，使得发送状态信息给远端装置后可以对应接收到该状态信息对应的渲染图像和渲染的深度信息。在此情况下，步骤S1接收到第一状态信息g1后，远端装置可以依次执行步骤S2、S3，得到与第一状态信息g1对应的第一渲染图像x1和第一渲染图像的深度信息x1’并发送至终端装置。通过这种方式，使得终端装置所在的设备采集的状态信息不必全部发送给远端装置，可以降低终端装置向远端装置传输状态信息的数据传输成本。
S7，接收远端装置发送的第一渲染图像x1和第一渲染图像的深度信息x1’的深度信息，第一渲染图像x1是远端装置根据第一状态信息g1和至少一个三维模型进行渲染获得的。
步骤S7中，第一渲染图像x1和第一渲染图像的深度信息x1’可以是终端装置从远端装置处获取的。远端装置获取第一渲染图像x1和第一渲染图像的深度信息x1’的示例性方式可以参照上文步骤S2的相关描述。
可选地，结合上文步骤S4和S5的相关描述可知，第一渲染图像x1和第一渲染图像的深度信息x1’可能是以渲染帧的形式发送。在此情况下，步骤S7中，终端装置接收到的也可以 是渲染帧。渲染帧包括第一渲染图像x1以及指示深度信息的纹理，终端装置可以使用远端装置在步骤S4中使用的映射方式，将渲染帧中的指示深度信息的纹理映射回深度信息的形式，即得到第一渲染图像的深度信息x1’。通过这种方式，在远端装置传输渲染帧的情况下，终端装置也能得到第一渲染图像x1和第一渲染图像的深度信息x1’。
第一渲染图像x1和第一渲染图像的深度信息x1’可以包括标识，标识可以表示第一渲染图像x1和第一渲染图像的深度信息x1’对应的状态信息。在此情况下，终端装置根据标识可以确定第一渲染图像x1和第一渲染图像的深度信息x1’对应于第一状态信息g1。在第一渲染图像x1和第一渲染图像的深度信息x1’以渲染帧的形式发送时，渲染帧可以包括标识。终端装置根据标识可以确定由渲染帧对应于第一状态信息g1，因此根据渲染帧得到的第一渲染图像x1和第一渲染图像的深度信息x1’也对应于第一状态信息g1。
S8，根据第一渲染图像x1和第一渲染图像的深度信息x1’获得多帧渲染图像x2-x6。
第一渲染图像x1是对应于第一状态信息g1的图像，可以直接在终端装置所在的设备的显示区域进行显示。由上文描述可知，终端装置所在的设备采集的状态信息在满足预设条件时发送给远端装置，对于未满足预设条件的状态信息(例如状态信息g2-g6)，由于没有发送到远端装置，因此，终端装置无法从远端装置处获取到与该未满足预设条件的状态信息(例如g2-g6)对应的渲染图像。在此情况下，步骤S8中，可以根据第一渲染图像x1、第一渲染图像的深度信息x1’和第二状态信息，获得第二状态信息对应的渲染图像，第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。未满足预设条件的状态信息(例如状态信息g2-g6)中的每一状态信息依次作为第二状态信息执行步骤S8，可以获得多个状态信息g2-g6对应的多帧渲染图像x2-x6。
步骤S8的一种实现方式是，先根据第一渲染图像x1和第一渲染图像的深度信息x1’，在终端装置重建出第一渲染图像x1中的物体在三维世界中的点云形式的模型，即点云数据(参见下文的步骤S9)，再根据第二状态信息对点云数据进行渲染，得到第二状态信息对应的对应的渲染图像(参见下文的步骤S10)。
可选地，步骤S8可包括S9-S10：
S9，根据第一渲染图像x1和第一渲染图像的深度信息x1’，生成点云数据。
图6示出根据本申请实施例确定点云数据的一个示例。
根据第一渲染图像x1和第一渲染图像的深度信息x1’确定点云数据可以基于现有技术来实现。如图6所示，根据第一状态信息g1，可以确定观察点C，相当于用户的眼睛的位置，并确定观察方向，相当于用户的眼睛的方向。基于此，可以确定用户的视角范围(例如图6中的L1、L2、L3、L4)。在此基础上，确定垂直于观察方向的近裁剪面，使得近裁剪面上的点位于用户的视角范围内、并且是用户眼睛(观察点)能看到的最近的点的集合。其中，近裁剪面上的点可以是第一渲染图像x1中的像素点。从观察点位置到近裁剪面上的一个点(例如M点)可以唯一确定一条射线，根据第一渲染图像的深度信息x1’中与该点对应的深度信息，以及观察点在世界坐标系下的坐标，可以确定一个世界坐标系下的点(例如N点)。以此类推，对于近裁剪面上的每个点，结合第一渲染图像的深度信息x1’中与该点对应的深度信息，以及观察点在世界坐标系下的坐标，都可以确定与之对应的世界坐标系下的点，根据确定的全部的世界坐标系下的点，可以重建出第一渲染图像x1中的物体在三维世界中的点云形式的模型，即点云数据。点云数据可以包括三维世界的多个点的三维坐标。
S10，根据第二状态信息对点云数据进行渲染，获得第二状态信息对应的渲染图像。
其中，第二状态信息可以是未满足预设条件的状态信息(例如状态信息g2-g6)中的任一状态信息。根据第二状态信息可以对点云数据进行投影得到多个二维坐标。一个示例是通过世界空间、观察空间、裁剪空间、屏幕空间等多个空间进行坐标转换。其中，世界空间包括点云数据，即三维世界的多个点的三维坐标。观察空间(View Space)也被称为摄像机空间(Camera Space)或视觉空间(Eye Space)，观察空间可包括用户(观察点)视野前面(观察方向上)的对象的点的三维坐标。裁剪空间(clip sapce)是对观察空间进行裁剪得到的，裁剪空间可以包括用户(观察点)视野前面(观察方向上)的对象的点的三维坐标，且裁剪空间中所有的坐标都能落在一个给定的范围内，在这个范围之外的点可被裁剪掉。屏幕空间(screen space)包括观察者最终看到的图像上的像素点的二维坐标。根据第二状态信息，可以确定观察空间的观察点，进一步确定世界空间到观察空间的坐标变换矩阵，结合世界空间下的数据，可以进行坐标变换得到观察空间下的数据。同理，根据观察空间到裁剪空间、裁剪空间到屏幕空间的坐标变换矩阵，可以进行坐标变换得到屏幕空间下的数据。在此情况下，点云数据中的点可以映射到屏幕空间的多个像素点的位置上，完成投影得到多个二维坐标。
根据多个二维坐标以及第一渲染图像x1中与多个二维坐标对应的纹理信息，可以得到第二状态信息对应的渲染图像。举例来说，第一渲染图像x1中的每个像素点可以对应一个二维坐标，该二维坐标表示该像素点在第一渲染图像x1中的位置。根据投影得到的多个二维坐标，可以找到第一渲染图像x1的像素点中，与投影得到的多个二维坐标具有相同坐标的像素点，根据找到的全部像素点，可以作为第二状态信息对应的渲染图像中的像素点。通过这种方式，可以获得第二状态信息对应的渲染图像。
状态信息g2-g6分别作为第二状态信息时，执行上文的步骤S9和S10，可分别得到第二状态信息对应的渲染图像x2-x6。多帧渲染图像可以包括渲染图像x2-x6。可认为渲染图像x2-x6分别对应于状态信息g2-g6。
步骤S8的另一种实现方式是，先根据第二状态信息得到多个二维线段(参见下文的步骤S11)，再根据第一状态信息g1对多个二维线段进行投影得到多个二维坐标(参见下文的步骤S12)，获得多个二维坐标的深度信息(参见下文的步骤S13)，并与第一渲染图像的深度信息x1’进行比较确定深度值相同的点的坐标(参见下文的步骤S14)，根据深度值相同的点的坐标和第一渲染图像x1获得第二状态信息对应的渲染图像(参见下文的步骤S15)。
可选地，步骤S8可包括S11-S15：
S11，根据第二状态信息、预设的近裁剪面和远裁剪面，确定多个二维线段。
其中，第二状态信息可以是未满足预设条件的状态信息(例如状态信息g2-g6)中的任一状态信息。步骤S11可基于现有技术来实现。图7示出根据第二状态信息确定世界坐标系下的多条线段的一个示例。如图7所示，根据第二状态信息，可以确定观察点C，相当于用户的眼睛的位置，并确定观察方向，相当于用户的眼睛的方向。基于此，可以确定用户的视角范围(例如图7中的L1、L2、L3、L4)。在此基础上，确定垂直于观察方向的近裁剪面和远裁剪面，使得近裁剪面上的点位于用户的视角范围内、并且是用户眼睛(观察点)能看到的最近的点的集合，使得远裁剪面上的点位于用户的视角范围内、并且是用户眼睛(观察点)能看到的最远的点的集合。从观察点位置到近裁剪面上的一个点可以唯一确定一条射线，根据远裁剪面和近裁剪面的空间位置关系，可以唯一确定近裁剪面和远裁剪面之间的一条二维 线段(例如线段MN)。以此类推，根据观察点位置、远裁剪面位置和近裁剪面上的每个点，都可以唯一确定近裁剪面和远裁剪面之间的一条二维线段，从而可以得到世界坐标系下的多个二维线段。
S12，根据第一状态信息g1对多个二维线段进行投影，得到多个二维坐标，其中，每一二维线段对应多个二维坐标。
其中，投影的实现方式可以参照上文步骤S10中，根据点云数据投影得到多个二维坐标的相关描述，在此不再赘述。多个二维线段可以投影在虚拟视点对应的视平面上，在视平面按照预先设置的分辨率分为多个像素点时，多条二维线段可占据部分像素点。多个二维坐标可以是多条二维线段占据的部分像素点在视平面上的二维坐标。
S13，获得多个二维坐标的深度信息。
多个二维坐标对应的像素点进行采样，可以获得像素点的深度值，即多个二维坐标的深度信息。每一二维坐标可对应一个深度值。
S14，将多个二维坐标的深度信息与第一渲染图像的深度信息进行比较，确定深度值相同的点的坐标，其中，深度信息包括各坐标对应的深度值。
由上文描述可知，第一渲染图像的深度信息x1’可以包括渲染参数指示的宽高比下的第一渲染图像中多个像素点对应的深度，多个二维坐标的深度信息可包括多条二维线段占据的像素点处的深度值。对于第一渲染图像x1中的像素点和多条二维线段占据的像素点，确定二者对应的二维坐标和深度值均相同的点，在此情况下，这些确定的点可以是最终显示在显示区域的图像中包含图像信息的像素点。这些确定的点的二维坐标可以是深度值相同的点的坐标。
S15，根据深度值相同的点的坐标和第一渲染图像x1，获得第二状态信息对应的渲染图像。
根据深度值相同的点的坐标以及第一渲染图像x1中与深度值相同的点的坐标对应的纹理信息，可以得到第二状态信息对应的渲染图像。举例来说，第一渲染图像x1中的每个像素点可以对应一个二维坐标，该二维坐标表示该像素点在第一渲染图像x1中的位置。根据深度值相同的点的坐标，可以找到第一渲染图像x1的像素点中，与深度值相同的点的坐标具有相同坐标的像素点，找到的全部像素点，可以作为第二状态信息对应的渲染图像中的像素点。通过这种方式，可以获得第二状态信息对应的渲染图像。
状态信息g2-g6分别作为第二状态信息时，执行上文的步骤S9和S10，可分别得到第二状态信息对应的渲染图像x2-x6。多帧渲染图像可以包括渲染图像x2-x6。可认为渲染图像x2-x6分别对应于状态信息g2-g6。
终端装置所在的设备可以按照状态信息g1-g6的采集顺序，在终端装置所在的设备的显示区域，显示对应于状态信息g1的第一渲染图像x1以及对应于状态信息g2-g6的多帧渲染图像x2-x6，使得用户能够通过显示区域观察到融合于真实世界的虚拟物体。通过这种方式，使得显示区域显示的图像与用户的动作变化导致的状态信息变化相符合，提高用户体验。直接使用第一渲染图像进行显示，可以降低AR设备的数据处理成本。
云VR系统的应用场景下，执行本申请的渲染方法实现在VR设备上将图像展示给用户的左眼(或右眼)的示例性实现方式，可以参照上文云AR系统的应用场景下，执行本申请的渲染方法实现在AR设备上将图像展示给用户的左眼和右眼的示例。其中，云端服务器上可包括远端装置，VR设备上可包括终端装置，远端装置可运行应用的服务端，终端装置可运行应用 的客户端，应用可包括多个三维模型，应用可以是VR应用。状态信息可包括左眼状态信息和右眼状态信息，例如第一状态信息可包括第一左眼状态信息和第一右眼状态信息，第二状态信息可包括第二左眼状态信息和第二右眼状态信息。
远端装置上，第一渲染图像和第一渲染图像的深度信息，可以是根据第一左眼状态信息(或者第一右眼状态信息)渲染得到的。由于执行本申请的渲染方法实现的是将图像展示给用户的左眼(或右眼)，所以第一渲染图像可以包括一帧渲染图像。
终端装置上，根据第一渲染图像和第一渲染图像的深度信息得到的多帧渲染图像，可以是对应于第二左眼状态信息(或者第二右眼状态信息)的图像。第一渲染图像以及多帧渲染图像，按照各自对应的姿态信息的采集顺序依次显示在VR设备的左眼(或右眼)对应的显示区域。VR设备的右眼(或左眼)的姿态信息对应的图像，可以基于现有技术的方法获取，在此不再赘述。
下面介绍云AR系统的应用场景下，执行本申请的渲染方法实现在AR设备上将图像展示给用户的左眼和右眼的另一个示例。在一种可能的实现方式中，云端服务器上可包括远端装置，AR设备上可包括终端装置，远端装置可运行应用的服务端，终端装置可运行应用的客户端，应用可包括多个三维模型，本应用场景下，应用可以是AR应用。本申请实施例可以通过远端装置和终端装置协作实现渲染。图8示出根据本申请实施例的远端装置和终端装置在云AR系统的应用场景下的工作方式的一个示例。
如图8所示，远端装置可执行步骤S16-S21，其中，步骤S16、S17、S18可以按顺序执行，步骤S16、S19、S20、S21可以按顺序执行，本申请对于步骤S17与S19的执行顺序不作限制。
S16，接收第一状态信息g1，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S16的实现方式可以参照上文步骤S1的示例，在此不再赘述。
S17，根据第一状态信息g1和至少一个三维模型进行渲染，获得第一渲染图像x1和第一渲染图像的深度信息x1’。
步骤S17的实现方式可以参照上文步骤S2的示例，在此不再赘述。
S18，发送第一渲染图像x1和第一渲染图像的深度信息x1’至终端装置。
步骤S18的实现方式可以参照上文步骤S3的示例，第一渲染图像x1和第一渲染图像的深度信息x1’可以以第一码率发送至终端装置。图像的码率是单位像素编码所需要的编码长度。其中，第一码率可以是较高的码率，以第一码率发送第一渲染图像x1和第一渲染图像的深度信息x1’，相当于以第一码率对第一渲染图像x1和第一渲染图像的深度信息x1’进行编码，得到第一渲染图像x1和第一渲染图像的深度信息x1’对应的编码结果后发送该编码结果。较高的第一码率使得编码结果的数据量较大，编码结果包括的信息量可更接近于第一渲染图像x1和第一渲染图像的深度信息x1’，第一编码结果可下行传输至终端装置，并由终端装置在接收后进行解码获得第一渲染图像x1和第一渲染图像的深度信息x1’。通过以高码率进行编码的编码方式，可以保证第一渲染图像x1和第一渲染图像的深度信息x1’编解码前后的信息量损失更小，以保证在AR设备上显示的图像内容质量。
可选地，步骤S18还可以包括S22-S23：
S22，将第一渲染图像x1和第一渲染图像的深度信息x1’合成渲染帧；S23，发送渲染帧至终端装置。步骤S22、S23的实现方式可以参照上文步骤S4、S5的示例。在此情况下，发送第一渲染图像x1和第一渲染图像的深度信息x1’至终端装置时，可以以第一码率对渲染帧进行编码，得到渲染帧对应的编码结果后发送该编码结果。
可选地，步骤S18之前，所述方法还包括：
步骤S24,确定发送至终端装置的上一渲染图像与第一渲染图像x1之间间隔指定个数的图像；或者，确定获得第一渲染图像时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，可以设置预设条件，使得远端装置确定在第一渲染图像x1满足预设条件时，发送第一渲染图像x1和第一渲染图像的深度信息x1’到终端装置。预设条件可以包括：确定发送至终端装置的上一渲染图像与第一渲染图像x1之间间隔指定个数的图像；或者，确定获得第一渲染图像时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，终端装置所在的设备采集的状态信息可以由终端装置直接发送给远端装置，由远端装置判断根据状态信息得到的渲染图像和渲染图像的深度信息是否发送给终端装置。在此情况下，步骤S16接收到第一状态信息g1后，远端装置可以执行步骤S17先得到第一渲染图像x1和第一渲染图像的深度信息x1’，再确定第一渲染图像x1是否满足预设条件，在满足条件时，执行步骤S18，将第一渲染图像x1和第一渲染图像的深度信息x1’发送至终端装置。通过这种方式，使得终端装置不需对发送到远端装置的状态信息作选择，可以降低终端装置的数据处理成本。
可选地，步骤S17之前，所述方法还包括：确定发送至终端装置的上一渲染图像所对应的状态信息与第一状态信息g1之间间隔指定个数的状态信息；或者，确定接收第一状态信息时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
可以设置预设条件，使得远端装置确定在第一状态信息g1满足预设条件时，根据第一状态信息g1得到第一渲染图像x1和第一渲染图像的深度信息x1’并发送到终端装置。预设条件可以包括：确定发送至终端装置的上一渲染图像所对应的状态信息与第一状态信息g1之间间隔指定个数的状态信息；或者，确定接收第一状态信息时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，终端装置所在的设备采集的状态信息可以由终端装置直接发送给远端装置，由远端装置判断状态信息是否用于得到发送给终端装置的渲染图像和渲染图像的深度信息。在此情况下，步骤S16接收到第一状态信息后，远端装置可以先确定第一状态信息g1是否满足预设条件，在满足条件时，执行步骤S17得到第一渲染图像x1和第一渲染图像的深度信息x1’，再执行步骤S18，将第一渲染图像x1和第一渲染图像的深度信息x1’发送至终端装置。通过这种方式，使得终端装置不需对发送到远端装置的状态信息作选择，可以降低终端装置的数据处理成本。远端装置上不会出现渲染得到渲染图像但未使用该渲染图像的情况，可以降低远端装置的数据处理成本。
S19，接收第二状态信息，第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。
第二状态信息可以是远端装置从终端装置处获取的。终端装置所在的设备(例如AR设备)可以包括传感器，传感器能够以预设的频率采集状态信息，例如以60次/s的频率进行状态 信息采集。每一时刻采集到的状态信息可以包括终端装置在该时刻的位置信息和姿态信息，可以认为不同状态信息是对应于不同的时刻。例如第二状态信息对应于第二时刻。第二状态信息可例如是状态信息g2-g6中的任一状态信息。第二状态信息可以发送到远端装置，使得远端装置可以接收到该包括终端装置在第二时刻的位置信息和姿态信息的第二状态信息。
S20，根据第二状态信息和至少一个三维模型进行渲染，获得第二渲染图像。
步骤S20的示例性实现方式可以参照上文步骤S2中，根据第一状态信息g1和至少一个三维模型进行渲染获得第一渲染图像x1的相关描述，在此不再赘述。
状态信息g2-g6分别作为第二状态信息时，执行上文的步骤S19和S20，可分别得到第二状态信息对应的渲染图像x2-x6。渲染图像x2-x6可以分别作为第二渲染图像。可认为渲染图像x2-x6分别对应于状态信息g2-g6。
S21，以低于第一码率的第二码率将第二渲染图像发送至终端装置。
远端装置可以发送第二渲染图像到终端装置，以在终端装置上使用第二渲染图像对根据第一渲染图像x1和第一渲染图像的深度信息x1’渲染得到的多帧渲染图像进行效果增强。远端装置可以对第一渲染图像x1、第一渲染图像的深度信息x1’、第二渲染图像设置标识，该标识可以表示第一渲染图像x1、第一渲染图像的深度信息x1’、第二渲染图像各自对应的状态信息。在此情况下，第一渲染图像x1、第一渲染图像的深度信息x1’的标识可以相同且对应于第一状态信息g1，第二渲染图像的标识与第一渲染图像x1、第一渲染图像的深度信息x1’的标识可以不同，且对应于除第一状态信息g1之外的、终端装置所属的设备采集到的状态信息(例如状态信息g2-g6中的任一状态信息)。
第二码率可以是较低的码率，第二码率可以低于第一码率。以第二码率发送第二渲染图像，相当于以第二码率对第二渲染图像进行编码，得到第二渲染图像对应的编码结果后发送该编码结果。根据第二码率进行编码，使得编码结果中的数据量较小，该编码结果包括的信息量得以降低，该编码结果可下行传输至终端装置，由终端装置在接收后进行解码获得第二渲染图像。通过以低码率进行编码的编码方式，可以降低网络带宽消耗。
可选地，可以对第二渲染图像进行下采样，得到第二渲染图像的下采样结果，再以第二码率发送该下采样结果。下采样可以基于现有技术来实现。通过这种方式可以进一步降低网络带宽消耗。
如图8所示，终端装置可执行步骤S25-S29，其中，步骤S27也可以先于步骤S26执行。
S25，发送第一状态信息g1至远端装置，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S25的实现方式可以参照步骤S6的示例，其中终端装置可不设置发送第一状态信息的预设条件。
S26，接收远端装置发送的第一渲染图像x1和第一渲染图像的深度信息x1’的深度信息，第一渲染图像x1是远端装置根据第一状态信息g1和至少一个三维模型进行渲染获得的。
步骤S26的实现方式可以参照步骤S7的示例，在此不再赘述。
第一渲染图像x1和第一渲染图像的深度信息x1’可以包括标识，标识可以表示第一渲染图像x1和第一渲染图像的深度信息x1’对应的状态信息。在此情况下，终端装置根据标识可以确定第一渲染图像x1和第一渲染图像的深度信息x1’对应于第一状态信息g1。在第一渲染图像x1和第一渲染图像的深度信息x1’以渲染帧的形式发送时，渲染帧可以包括标识。终端 装置根据标识可以确定由渲染帧对应于第一状态信息g1，因此根据渲染帧得到的第一渲染图像x1和第一渲染图像的深度信息x1’也对应于第一状态信息g1。
S27，发送第二状态信息至远端装置，第二状态信息包括终端装置在第一时刻之后的第二时刻的位置信息和姿态信息。
步骤S27的实现方式可以参照步骤S6中，发送第一状态信息g1至远端装置的示例。其中终端装置可不设置发送第二状态信息的预设条件。
S28，接收第二渲染图像，第二渲染图像是远端装置根据第二状态信息和至少一个三维模型进行渲染获得的。
第二渲染图像可以是终端装置从远端装置处获取的。第二渲染图像可以包括标识，标识可以表示第二渲染图像对应的状态信息。在此情况下，终端装置根据标识可以确定第二渲染图像对应于第二状态信息。在第二渲染图像以下采样结果的形式发送时，下采样结果可以包括标识。终端装置根据标识可以确定由下采样结果得到的第二渲染图像对应于第二状态信息。例如可根据标识确定第二渲染图像对应的第二状态信息是状态信息g2-g6中的一个。
S29，根据第一渲染图像x1、第一渲染图像的深度信息x1’、第二状态信息和第二渲染图像，获得多帧渲染图像xx2-xx6。
第一渲染图像x1是对应于第一状态信息g1的图像，可以直接在终端装置所在的设备的显示区域进行显示。由上文描述可知，终端装置所在的设备采集的状态信息直接发送给远端装置，远端装置在由状态信息得到的渲染图像满足预设条件时将渲染图像以较高的第一码率发送至终端装置，对于未满足预设条件的渲染图像(第二渲染图像)以较低的第二码率发送至终端装置，由于较低的第二码率对渲染图像的质量影响较大，可能不适于直接在终端装置所在的设备的显示区域进行显示。
在此情况下，为了在终端装置上，显示质量较好的渲染图像，可以根据第一渲染图像x1、第一渲染图像的深度信息x1’和第二状态信息，获得第一中间渲染图像，再根据第二渲染图像对第一中间渲染图像进一步处理，得到第二状态信息对应的第二渲染图像xx2-xx6。
步骤29可以有多种实现方式，可选地，步骤S29可包括S30-S32：
S30，根据第一渲染图像x1和第一渲染图像的深度信息x1’，生成点云数据。
步骤S30的实现方式可以参照步骤S9的示例，在此不再赘述。
S31，根据第二状态信息对点云数据进行渲染，获得第一中间渲染图像。
步骤31的示例性实现方式，可以参照步骤S10中，根据第二状态信息对点云数据进行渲染获得第二状态信息对应的渲染图像的相关描述，在此不再赘述。
S32，利用第二渲染图像对第一中间渲染图像进行填补，获得第二状态信息对应的第二渲染图像。
举例来说，步骤S31根据第二状态信息对点云数据进行投影得到多个二维坐标的过程中，可能会出现三维点云数据中有多个点经坐标变换后，得到相同的一个二维坐标的情况，使得根据二维坐标以及第一渲染图像x1中与多个二维坐标对应的纹理信息，得到第一中间渲染图像时，得到的第一中间渲染图像中有部分像素点处没有数据，导致第一中间渲染图像出现“空洞点”。图9示出根据本申请实施例的第一中间渲染图像出现空洞点(如图9中的黑色点)的一个示例。对于这些少量“空洞点”，确定其在图像中的位置后，根据该第一中间渲染图像对应的姿态信息，找到对应相同姿态信息的第二渲染图像，并使用第二渲染图像的对应位置像素 点的数据，来补充在第一中间渲染图像的“空洞点”，例如可将第二渲染图像的对应位置像素点的数据直接补充，或处理后补充在第一中间渲染图像的“空洞点”处，得到第二状态信息对应的第二渲染图像，图10示出根据本申请实施例的第二渲染图像补充空洞点的一个示例。通过这种方式，可以保证高质量渲染。
渲染图像x2-x6分别作为第二渲染图像时，执行上文的步骤S30-S32，可分别得到第二状态信息对应的渲染图像xx2-xx6。多帧渲染图像可以包括渲染图像xx2-xx6。可认为渲染图像xx2-xx6分别对应于状态信息g2-g6。
可选地，步骤S29可包括：
根据第二状态信息和视距确定虚拟视点的位置；截取虚拟视点与远裁剪面之间的连线在近裁剪面和远裁剪面之间的线段，确定多个二维线段。其示例性实现方式可以参照上文步骤S11的相关描述。其中，虚拟视点可以是步骤S11的相关描述中的观察点，视距可以是步骤S11的相关描述中用户眼睛(观察点)能看到的最远的点与观察点的距离。
将多个二维线段投影至虚拟视点对应的视平面，得到多个二维坐标。其中，每一二维线段对应多个二维坐标。其示例性实现方式可以参照上文步骤S12的相关描述。
获得多个二维坐标的深度信息。其示例性实现方式可以参照上文步骤S13的相关描述。
将多个二维坐标的深度信息与第一渲染图像的深度信息进行比较，确定深度值相同的坐标，其中，深度信息包括各坐标对应的深度值。其示例性实现方式可以参照上文步骤S14的相关描述。
根据深度值相同的坐标在第一渲染图像中的纹理信息，获得第二状态信息对应的第二中间渲染图像。其示例性实现方式可以参照上文步骤S15中，根据深度值相同的点的坐标和第一渲染图像x1获得第二状态信息对应的渲染图像的相关描述。
利用第二渲染图像对第二中间渲染图像进行填补，获得第二状态信息对应的渲染图像。其示例性实现方式可以参照上文步骤S32中，利用第二渲染图像对第一中间渲染图像进行填补获得第二状态信息对应的第二渲染图像的相关描述。
渲染图像x2-x6分别作为第二渲染图像时，执行上文的步骤S29，可分别得到第二状态信息对应的渲染图像xx2-xx6。多帧渲染图像可以包括渲染图像xx2-xx6。可认为渲染图像xx2-xx6分别对应于状态信息g2-g6。
终端装置所在的设备可以按照状态信息g1-g6的采集顺序，在终端装置所在的设备的显示区域，显示对应于状态信息g1的第一渲染图像x1以及对应于状态信息g2-g6的多帧渲染图像xx2-xx6，使得用户能够通过显示区域观察到融合于真实世界的虚拟物体。通过这种方式，使得显示区域显示的图像与用户的动作变化导致的状态信息变化相符合，提高用户体验。直接使用第一渲染图像x1进行显示，可以降低AR设备的数据处理成本。
云VR系统的应用场景下，执行本申请的渲染方法实现在VR设备上将图像展示给用户的左眼(或右眼)的示例性实现方式，可以参照上文云AR系统的应用场景下，执行本申请的渲染方法实现在AR设备上将图像展示给用户的左眼和右眼的示例。其中，云端服务器上可包括远端装置，VR设备上可包括终端装置，远端装置可运行应用的服务端，终端装置可运行应用的客户端，应用可包括多个三维模型，应用可以是VR应用。状态信息可包括左眼状态信息和右眼状态信息，例如第一状态信息可包括第一左眼状态信息和第一右眼状态信息，第二状态信息可包括第二左眼状态信息和第二右眼状态信息。
远端装置上，第一渲染图像和第一渲染图像的深度信息，可以是根据第一左眼状态信息(或者第一右眼状态信息)渲染得到的。由于执行本申请的渲染方法实现的是将图像展示给用户的左眼(或右眼)，所以第一渲染图像可以包括一帧渲染图像。
终端装置上，根据第一渲染图像和第一渲染图像的深度信息得到的多帧渲染图像，可以是对应于第二左眼状态信息(或者第二右眼状态信息)的图像。第一渲染图像以及多帧渲染图像，按照各自对应的姿态信息的采集顺序依次显示在VR设备的左眼(或右眼)对应的显示区域。VR设备的右眼(或左眼)的姿态信息对应的图像，可以基于现有技术的方法获取，在此不再赘述。
下面介绍云VR系统的应用场景下，执行本申请的渲染方法实现在VR设备上将图像分别展示给用户的左眼和右眼的一个示例。在一种可能的实现方式中，云端服务器上可包括远端装置，VR设备上可包括终端装置，远端装置可运行应用的服务端，终端装置可运行应用的客户端，应用可包括多个三维模型，本应用场景下，应用可以是VR应用。本申请实施例可以通过远端装置和终端装置协作实现渲染。图11示出根据本申请实施例的远端装置和终端装置在云VR系统的应用场景下的工作方式的一个示例。
如图11所示，远端装置可执行步骤S33-S35：
S33，接收第一状态信息g1，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S33的实现方式可以参照上文步骤S1的示例，在此不再赘述。
S34，根据第一状态信息g1和至少一个三维模型进行渲染，获得第一渲染图像x11和第一渲染图像的深度信息x11’。
步骤S34的实现方式可以参照上文步骤S2的示例，其中，根据第一状态信息g1可以先得到对应于用户左眼的状态信息g11和对应于用户右眼的状态信息g12，第一渲染图像x11可以是根据用户左眼的状态信息g11(或右眼的状态信息g12)得到的。在此以第一渲染图像x11根据用户左眼的状态信息g11得到作为示例。
S35，发送第一渲染图像x11和第一渲染图像的深度信息x11’至终端装置。
步骤S35的实现方式可以参照上文步骤S18的示例，在此不再赘述。
可选地，步骤S35还可以包括S36-S37：
S36，将第一渲染图像x11和第一渲染图像的深度信息x11’合成渲染帧；S37，发送渲染帧至终端装置。步骤S36、S37的实现方式可以参照上文步骤S4、S5的示例。在此情况下，发送第一渲染图像x11和第一渲染图像的深度信息x11’至终端装置时，可以以第一码率对渲染帧进行编码，得到渲染帧对应的编码结果后发送该编码结果。
可选地，步骤S34之前，所述方法还包括：确定发送至终端装置的上一渲染图像所对应的状态信息与第一状态信息g1之间间隔指定个数的状态信息；或者，确定接收第一状态信息时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
可以设置预设条件，使得远端装置确定在第一状态信息g1满足预设条件时，根据第一状态信息g1得到第一渲染图像x11和第一渲染图像的深度信息x11’并发送到终端装置。预设条件可以包括：确定发送至终端装置的上一渲染图像所对应的状态信息与第一状态信息g1之间 间隔指定个数的状态信息；或者，确定接收第一状态信息时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，终端装置所在的设备采集的状态信息可以由终端装置直接发送给远端装置，由远端装置判断状态信息是否用于得到发送给终端装置的渲染图像和渲染图像的深度信息。在此情况下，步骤S33接收到第一状态信息g1后，远端装置可以先确定第一状态信息g1是否满足预设条件，在满足条件时，执行步骤S34得到第一渲染图像x11和第一渲染图像的深度信息x11’，再执行步骤S35，将第一渲染图像x11和第一渲染图像的深度信息x1’发送至终端装置。通过这种方式，使得终端装置不需对发送到远端装置的状态信息作选择，可以降低终端装置的数据处理成本。
如图11所示，终端装置可执行步骤S38-S40：
S38，发送第一状态信息g1至远端装置，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S38的实现方式可以参照上文步骤S6的示例。
S39，接收远端装置发送的第一渲染图像x11和第一渲染图像的深度信息x11’的深度信息，第一渲染图像x11是远端装置根据第一状态信息g1和至少一个三维模型进行渲染获得的。
步骤S39的实现方式可以参照上文步骤S7的示例。其中，第一渲染图像x11可以是根据第一状态信息g1中用户左眼的状态信息g11得到的。
S40，根据第一渲染图像x11和第一渲染图像的深度信息x11’获得多帧渲染图像x12、x21-x61、x22-x62。
第一渲染图像x11是对应于第一状态信息g1中用户左眼的状态信息g11的图像，可以直接在终端装置所在的设备的左眼对应的显示区域上进行显示。为了在终端装置所在的设备的左眼和右眼对应的显示区域均显示图像，可以根据第一渲染图像x11和第一渲染图像的深度信息x11’，获得与第一状态信息g1中用户右眼的状态信息g12对应的另一渲染图像x12，使得第一渲染图像x11和另一渲染图像x12对应的虚拟视点之间距离等于视距。此时另一渲染图像是对应于第一状态信息中用户右眼的状态信息g12的图像，可以在终端装置的右眼对应的显示区域上进行显示。
例如，可以根据第一渲染图像x11和第一渲染图像的深度信息x11’，生成点云数据。其示例性实现方式可以参照上文步骤S9的相关描述。根据第一状态信息g1中用户右眼的状态信息g12对点云数据进行渲染，获得状态信息g12对应的渲染图像x12。其示例性实现方式可以参照上文步骤S10中，根据第二状态信息对点云数据进行渲染，获得第二状态信息对应的渲染图像的相关描述。
又例如，可以根据第一状态信息中用户右眼的状态信息g12、预设的近裁剪面和远裁剪面，确定多个二维线段。其示例性实现方式可以参照上文步骤S11中，根据第二状态信息、预设的近裁剪面和远裁剪面，确定多个二维线段的相关描述。根据第一状态信息中用户左眼的状态信息g11对多个二维线段进行投影，得到多个二维坐标。其示例性实现方式可以参照上文步骤S12的相关描述。获得多个二维坐标的深度信息。其示例性实现方式可以参照上文步骤S13的相关描述。将多个二维坐标的深度信息与第一渲染图像x11的深度信息进行比较，确定深度值相同的点的坐标，其示例性实现方式可以参照上文步骤S14的相关描述。根据深度值相同的点的坐标和第一渲染图像x11，获得第一状态信息中用户右眼的状态信息g12对 应的渲染图像x12。其示例性实现方式可以参照上文步骤S15中，根据深度值相同的点的坐标和第一渲染图像x1，获得第二状态信息对应的渲染图像的相关描述。
由上文描述可知，终端装置无法从远端装置处获取到与第二状态信息对应的渲染图像。在此情况下，为了在终端装置上，显示与第二状态信息对应的渲染图像，可以根据第二状态信息，先处理得到对应于用户左眼的状态信息和对应于用户右眼的状态信息，再根据第一渲染图像x11、第一渲染图像的深度信息x11’和第二状态信息(对应于用户左眼的状态信息和对应于用户右眼的状态信息)，获得第二状态信息对应的渲染图像，第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。
可选地，步骤S40可包括S41-S42：
S41，根据第一渲染图像x11和第一渲染图像的深度信息x11’，生成点云数据。
步骤S41的实现方式可以参照上文步骤S9的示例。其中，可以根据第一状态信息g1得到对应于用户左眼的状态信息g11和对应于用户右眼的状态信息g12，并确定第一渲染图像x11是根据第一状态信息中用户左眼的状态信息g11得到的。在此情况下，参照上文步骤S9根据第一渲染图像x1和第一渲染图像的深度信息x1’，生成点云数据的方法，可以得到点云数据。
S42，根据第二状态信息对点云数据进行渲染，获得第二状态信息对应的渲染图像。
步骤S42的实现方式可以参照上文步骤S10的示例。其中，可以根据第二状态信息得到对应于用户左眼的状态信息和对应于用户右眼的状态信息，在此情况下，参照上文步骤S10的方法，根据第二状态信息中用户左眼的状态信息对点云数据进行渲染，可以获得对应于用户左眼的渲染图像x21-x61。根据第二状态信息中用户右眼的状态信息对点云数据进行渲染，可以获得对应于用户右眼的渲染图像x22-x62。第二状态信息对应的渲染图像可以包括上述对应于用户左眼的渲染图像x21-x61和对应于用户右眼的渲染图像x22-x62。
可选地，步骤S40可包括：
根据第二状态信息、预设的近裁剪面和远裁剪面，确定多个二维线段。其示例性实现方式可以参照上文步骤S11的相关描述。其中，可以根据第二状态信息处理得到对应于用户左眼的状态信息和对应于用户右眼的状态信息，在此情况下，根据第二状态信息中用户左眼的状态信息、预设的近裁剪面和远裁剪面，可获得对应于用户左眼的二维线段。根据第二状态信息中用户右眼的状态信息、预设的近裁剪面和远裁剪面，可获得对应于用户右眼的二维线段。多个二维线段可以包括上述对应于用户左眼的二维线段和对应于用户右眼的二维线段。
根据第一状态信息g1对多个二维线段进行投影，得到多个二维坐标，其中，每一二维线段对应多个二维坐标。其示例性的实现方式可以参照上文步骤S12的相关描述。其中，根据第一状态信息g1对对应于用户左眼的二维线段进行投影，得到对应于用户左眼的二维坐标，根据第一状态信息g1对对应于用户右眼的二维线段进行投影，得到对应于用户右眼的二维坐标，多个二维坐标可以包括上述对应于用户左眼的二维坐标和对应于用户右眼的二维坐标。
获得多个二维坐标的深度信息。其示例性实现方式可以参照上文步骤S13的相关描述。其中，对于对应于用户左眼的二维坐标，可对应得到对应于用户左眼的二维坐标的深度信息，对于对应于用户右眼的二维坐标，可对应得到对应于用户右眼的二维坐标的深度信息，多个二维坐标的深度信息可以包括上述对应于用户左眼的二维坐标的深度信息和对应于用户右眼的二维坐标的深度信息。
将多个二维坐标的深度信息与第一渲染图像的深度信息进行比较，确定深度值相同的点的坐标，其中，深度信息包括各坐标对应的深度值。其示例性实现方式可以参照上文步骤S14的相关描述，其中，将对应于用户左眼的二维坐标的深度信息与第一渲染图像的深度信息进行比较，可对应得到对应于用户左眼的深度值相同的点的坐标。对应于用户右眼的二维坐标的深度信息与第一渲染图像的深度信息进行比较，可对应得到对应于用户右眼的深度值相同的点的坐标。深度值相同的点的坐标可以包括上述对应于用户左眼的深度值相同的点的坐标和对应于用户右眼的深度值相同的点的坐标。
根据深度值相同的点的坐标和第一渲染图像，获得第二状态信息对应的渲染图像。其实现方式可以参照上文步骤S15的示例，其中，根据对应于用户左眼的深度值相同的点的坐标和第一渲染图像，可对应得到对应于用户左眼的渲染图像x21-x61，根据对应于用户右眼的深度值相同的点的坐标和第一渲染图像，可对应得到对应于用户右眼的渲染图像x22-x62，第二状态信息对应的渲染图像可以包括上述对应于用户左眼的渲染图像x21-x61和对应于用户右眼的渲染图像x22-x62。
终端装置所在的设备可以按照状态信息g1-g6的采集顺序，在终端装置所在的设备的显示区域，显示第一渲染图像x11、另一渲染图像x12和第二状态信息对应的渲染图像x21-x61、x22-x62，例如在左眼对应的显示区域显示x11、x21-x61、在右眼对应的显示区域显示x12-x62，使得用户能够通过显示区域观察到三维空间的虚拟世界。其中，对应于相同状态信息的图像可以同时显示。
通过这种方式，使得显示区域显示的图像与用户的动作变化导致的状态信息变化相符合，提高用户体验。直接使用第一渲染图像进行显示，可以降低VR设备的数据处理成本。
下面介绍云VR系统的应用场景下，执行本申请的渲染方法实现在VR设备上将图像分别展示给用户的左眼和右眼的另一个示例。云端服务器上可包括远端装置，VR设备上可包括终端装置，远端装置可运行应用的服务端，终端装置可运行应用的客户端，应用可包括多个三维模型，本应用场景下，应用可以是VR应用。本申请实施例可以通过远端装置和终端装置协作实现渲染。图12示出根据本申请实施例的远端装置和终端装置在云VR系统的应用场景下的工作方式的一个示例。
如图12所示，远端装置可执行步骤S43-S47：
S43，接收第一状态信息g1，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S43的实现方式可以参照上文步骤S33的示例，在此不再赘述。
S44，根据第一状态信息g1和至少一个三维模型进行渲染，获得第一渲染图像x11和第一渲染图像的深度信息x11’。
步骤S44的实现方式可以参照上文步骤S34的示例，在此不再赘述。根据第一状态信息g1可以先得到对应于用户左眼的状态信息g11和对应于用户右眼的状态信息g12，第一渲染图像x11可以是根据用户左眼的状态信息g11(或右眼的状态信息g12)得到的。在此以第一渲染图像x11根据用户左眼的状态信息g11得到作为示例。
S45，发送第一渲染图像x11和第一渲染图像的深度信息x11’至终端装置。
步骤S45的实现方式可以参照上文步骤S35的示例，在此不再赘述。
可选地，步骤S45还可以包括S48-S49：
S48，将第一渲染图像x11和第一渲染图像的深度信息x11’合成渲染帧；S49，发送渲染帧至终端装置。步骤S48、S49的实现方式可以参照上文步骤S4、S5的示例。在此情况下，发送第一渲染图像和第一渲染图像的深度信息至终端装置时，可以以第一码率对渲染帧进行编码，得到渲染帧对应的编码结果后发送该编码结果。
可选地，步骤S45之前，所述方法还包括：
步骤S50,确定发送至终端装置的上一渲染图像与第一渲染图像x11之间间隔指定个数的图像；或者，确定获得第一渲染图像时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，可以设置预设条件，使得远端装置确定在第一渲染图像x11满足预设条件时，发送第一渲染图像x11和第一渲染图像的深度信息x11’到终端装置。预设条件可以包括：确定发送至终端装置的上一渲染图像与第一渲染图像x11之间间隔指定个数的图像；或者，确定获得第一渲染图像时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，终端装置所在的设备采集的状态信息可以由终端装置直接发送给远端装置，由远端装置判断根据状态信息得到的渲染图像和渲染图像的深度信息是否发送给终端装置。在此情况下，步骤S43接收到第一状态信息g1后，远端装置可以执行步骤S44先得到第一渲染图像x11和第一渲染图像的深度信息x11’，再确定第一渲染图像x11是否满足预设条件，在满足条件时，执行步骤S45，将第一渲染图像x11和第一渲染图像的深度信息x11’发送至终端装置。通过这种方式，使得终端装置不需对发送到远端装置的状态信息作选择，可以降低终端装置的数据处理成本。
S46，根据第一状态信息g1和至少一个三维模型进行渲染，获得另一渲染图像x12，第一渲染图像x11对应的虚拟视点和另一渲染图像x12对应的虚拟视点之间距离等于视距。
步骤S46的实现方式可以参照上文步骤S2的示例，其中，另一渲染图像可以是根据第一状态信息中用户右眼的状态信息g12得到的。也即，第一渲染图像x11和另一渲染图像x12，可以分别对应于第一状态信息中用户左眼的状态信息g11和用户右眼的状态信息g12。
S47，以低于第一码率的第二码率将另一渲染图像x12发送至终端装置，第一渲染图像x11是以第一码率发送至终端装置。
以低于第二码率将另一渲染图像x12发送至终端装置的示例性实现方式，可以参照上文步骤S21中，以第二码率将第二渲染图像发送至终端装置的示例，在此不再赘述。
如图12所示，终端装置可执行步骤S51-S53：
S51，发送第一状态信息g1至远端装置，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S51的实现方式可以参照上文步骤S6的示例，在此不再赘述。
S52，接收远端装置发送的第一渲染图像x11和第一渲染图像的深度信息x11’的深度信息，第一渲染图像x11是远端装置根据第一状态信息g1和至少一个三维模型进行渲染获得的。
步骤S52的实现方式可以参照上文步骤S7的示例。其中，第一渲染图像x11可以是根据第一状态信息中用户左眼的状态信息g11得到的。
S53，根据第一渲染图像x11和一渲染图像的深度信息x11’获得多帧渲染图像。
第一渲染图像x11是对应于第一状态信息g1中用户左眼的状态信息g11的图像，可以直接在终端装置所在的设备的左眼对应的显示区域上进行显示。由上文描述可知，远端装置还以较低的第二码率将对应于第一状态信息g1中用户右眼的状态信息g12的的另一渲染图像发送至终端装置，由于较低的第二码率对渲染图像的质量影响较大，可能不适于直接在终端装置所在的设备的显示区域进行显示。
可选地，步骤S53可包括：
接收远端装置发送的另一渲染图像x12，另一渲染图像是远端装置根据第一状态信息和应用进行渲染获得的，第一渲染图像和另一渲染图像对应的虚拟视点之间距离等于视距。在此情况下，第一渲染图像是第一状态信息中用户左眼(或右眼)的状态信息对应的渲染图像时，另一渲染图像可以是第一状态信息中用户右眼(或左眼)的状态信息对应的渲染图像。
根据第一渲染图像和第一渲染图像的深度信息，生成点云数据。其实现方式可以参照上文步骤S9的示例。
根据第一状态信息和视距，对点云数据进行渲染，获得第三中间渲染图像。例如，第一渲染图像x11是第一状态信息中用户左眼的状态信息g11对应的渲染图像时，可以根据第一状态信息中的用户右眼的状态信息g12以及第一渲染图像x11，对点云数据进行渲染，得到第三中间渲染图像。其实现方式可以参照上文步骤S10的示例。
利用远端装置发送的另一渲染图像x12对第三中间渲染图像进行填补，获得第一状态信息对应的另一渲染图像xx12。其实现方式可以参照上文步骤S32中，利用第二渲染图像对第一中间渲染图像进行填补的示例。
可选地，步骤S53还包括：
根据第一状态信息和视距确定虚拟视点的位置；截取虚拟视点与远裁剪面之间的连线在近裁剪面和远裁剪面之间的线段，确定多个二维线段。其示例性实现方式可以参照上文步骤S40中，根据第二状态信息、预设的近裁剪面和远裁剪面，确定多个二维线段的相关描述。
将多个二维线段投影至所述虚拟视点对应的视平面，得到多个二维坐标，其中，每一二维线段对应多个二维坐标。其示例性实现方式可以参照上文步骤S40中，据第一状态信息g1对多个二维线段进行投影，得到多个二维坐标的相关描述。
获得多个二维坐标的深度信息。其示例性实现方式可以参照上文步骤S40中，获得多个二维坐标的深度信息的相关描述。
将多个二维坐标的深度信息与第一渲染图像的深度信息进行比较，确定深度值相同的坐标，其中，所述深度信息包括各坐标对应的深度值。其示例性实现方式可以参照上文步骤S40中，将多个二维坐标的深度信息与第一渲染图像的深度信息进行比较，确定深度值相同的点的坐标的相关描述。
根据深度值相同的坐标在所述第一渲染图像中的纹理信息，获得所述第一状态信息对应的第四中间渲染图像。其示例性实现方式可以参照上文步骤S40中，根据深度值相同的点的坐标和第一渲染图像获得第二状态信息对应的渲染图像的相关描述。
利用远端装置发送的另一渲染图像对第四中间渲染图像进行填补，获得所述第一状态信息对应的另一渲染图像xx12。其示例性实现方式可以参照上文步骤S32中，利用第二渲染图像对第一中间渲染图像进行填补的相关描述。
由上文描述可知，终端装置无法从远端装置处获取到与第二状态信息对应的渲染图像。 在此情况下，为了在终端装置上，显示与第二状态信息对应的渲染图像，可以根据第二状态信息，先处理得到对应于用户左眼的状态信息和对应于用户右眼的状态信息，再根据第一渲染图像x11、第一渲染图像的深度信息x11’和第二状态信息(对应于用户左眼的状态信息和对应于用户右眼的状态信息)，获得第二状态信息对应的渲染图像，第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。
可选地，步骤S53可包括S54-S55：
S54，根据第一渲染图像x11和第一渲染图像的深度信息x11’，生成点云数据。
步骤S41的实现方式可以参照上文步骤S9的示例。其中，可以根据第一状态信息g1得到对应于用户左眼的状态信息g11和对应于用户右眼的状态信息g12，并确定第一渲染图像x11是根据第一状态信息中用户左眼的状态信息g11得到的。在此情况下，参照上文步骤S9根据第一渲染图像和第一渲染图像的深度信息，生成点云数据的方法，可以得到点云数据。
S55，根据第二状态信息对点云数据进行渲染，获得第二状态信息对应的渲染图像。
步骤S55的实现方式可以参照上文步骤S10的示例。其中，可以根据第二状态信息得到对应于用户左眼的状态信息和对应于用户右眼的状态信息，在此情况下，参照上文步骤S10的方法，根据第二状态信息中用户左眼的状态信息对点云数据进行渲染，可以获得对应于用户左眼的渲染图像x21-x61。根据第二状态信息中用户右眼的状态信息对点云数据进行渲染，可以获得对应于用户右眼的渲染图像x22-x62。第二状态信息对应的渲染图像可以包括上述对应于用户左眼的渲染图像x21-x61和对应于用户右眼的渲染图像x22-x62。
可选地，步骤S53可包括：
根据第二状态信息、预设的近裁剪面和远裁剪面，确定多个二维线段。根据第一状态信息g1对多个二维线段进行投影，得到多个二维坐标，其中，每一二维线段对应多个二维坐标。获得多个二维坐标的深度信息。将多个二维坐标的深度信息与第一渲染图像的深度信息进行比较，确定深度值相同的点的坐标，其中，深度信息包括各坐标对应的深度值。根据深度值相同的点的坐标和第一渲染图像，获得第二状态信息对应的渲染图像。其实现方式可以参照上文步骤S40的示例。其中，第二状态信息对应的渲染图像可以包括渲染图像x21-x61和渲染图像x22-x62。
终端装置所在的设备可以按照状态信息g1-g6的采集顺序，在终端装置所在的设备的显示区域，显示第一渲染图像x11、渲染图像xx12和第二状态信息对应的渲染图像x21-x61、x22-x62，例如在左眼对应的显示区域显示x11、x21-x61、在右眼对应的显示区域显示xx12、x22-x62，使得用户能够通过显示区域观察到三维空间的虚拟世界。其中，对应于相同状态信息的图像可以同时显示。
通过这种方式，使得显示区域显示的图像与用户的动作变化导致的状态信息变化相符合，提高用户体验。直接使用第一渲染图像x11进行显示，可以降低VR设备的数据处理成本。
下面介绍云VR系统的应用场景下，执行本申请的渲染方法实现在VR设备上将图像分别展示给用户的左眼和右眼的另一个示例。云端服务器上可包括远端装置，VR设备上可包括终端装置，远端装置可运行应用的服务端，终端装置可运行应用的客户端，应用可包括多个三维模型，本应用场景下，应用可以是VR应用。本申请实施例可以通过远端装置和终端装置协 作实现渲染。图13示出根据本申请实施例的远端装置和终端装置在云VR系统的应用场景下的工作方式的一个示例。
如图13所示，远端装置可执行步骤S56-S61，其中，步骤S56、S57、S58可以按顺序执行，步骤S56、S59、S60、S61可以按顺序执行，本申请对于步骤S57与S59的执行顺序不作限制。
S56，接收第一状态信息g1，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S56的实现方式可以参照上文步骤S1的示例，在此不再赘述。
S57，根据第一状态信息g1和至少一个三维模型进行渲染，获得第一渲染图像x11和第一渲染图像的深度信息x11’。
步骤S57的实现方式可以参照上文步骤S2的示例，其中，根据第一状态信息g1可以先处理得到对应于用户左眼的状态信息g11和对应于用户右眼的状态信息g12，第一渲染图像x11可以是根据第一状态信息中用户左眼的状态信息g12得到的。
S58，发送第一渲染图像x11和第一渲染图像的深度信息x11’至终端装置。
步骤S58的实现方式可以参照上文步骤S3的示例，第一渲染图像x11和第一渲染图像的深度信息x11’可以以第一码率发送至终端装置。
可选地，步骤S58还可以包括S62-S63：
S62，将第一渲染图像和第一渲染图像的深度信息合成渲染帧；S63，发送渲染帧至终端装置。步骤S62、S63的实现方式可以参照上文步骤S4、S5的示例。在此情况下，发送第一渲染图像和第一渲染图像的深度信息至终端装置时，可以以第一码率对渲染帧进行编码，得到渲染帧对应的编码结果后发送该编码结果。
可选地，步骤S58之前，所述方法还包括：
步骤S64,确定发送至终端装置的上一渲染图像与第一渲染图像x11之间间隔指定个数的图像；或者，确定获得第一渲染图像时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，可以设置预设条件，使得远端装置确定在第一渲染图像x11满足预设条件时，发送第一渲染图像x11和第一渲染图像的深度信息x11’到终端装置。预设条件可以包括：确定发送至终端装置的上一渲染图像与第一渲染图像x11之间间隔指定个数的图像；或者，确定获得第一渲染图像时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，终端装置所在的设备采集的状态信息可以由终端装置直接发送给远端装置，由远端装置判断根据状态信息得到的渲染图像和渲染图像的深度信息是否发送给终端装置。在此情况下，步骤S56接收到第一状态信息g1后，远端装置可以执行步骤S57先得到第一渲染图像x11和第一渲染图像的深度信息x11’，再确定第一渲染图像x11是否满足预设条件，在满足条件时，执行步骤S58，将第一渲染图像x11和第一渲染图像的深度信息x11’发送至终端装置。通过这种方式，使得终端装置不需对发送到远端装置的状态信息作选择，可以降低终端装置的数据处理成本。
S59，接收第二状态信息，第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。
步骤S59的实现方式可以参照上文步骤S19的示例，在此不再赘述。
S60，根据第二状态信息和至少一个三维模型进行渲染，获得第二渲染图像。
步骤S60的实现方式可以参照上文步骤S2的示例，其中，终端装置可根据第二状态信息得到对应于用户左眼的状态信息和对应于用户右眼的状态信息，针对对应于用户左眼的状态信息和对应于用户右眼的状态信息和至少一个三维模型进行渲染，获得的第二渲染图像，可以包括第二状态信息中用户左眼的状态信息对应的渲染图像，以及第二状态信息中用户右眼的状态信息对应的渲染图像。
S61，以低于第一码率的第二码率将第二渲染图像发送至终端装置。
步骤S61的实现方式可以参照上文步骤S21的示例，由上文描述可知，第二渲染图像可以包括第二状态信息中用户左眼的状态信息对应的渲染图像，以及第二状态信息中用户右眼的状态信息对应的渲染图像。可选地，可以对第二渲染图像进行下采样后发送。图14示出根据本申请实施例的第二渲染图像的传输方式的一个示例。
如图14所示，可以对第二状态信息中用户左眼的状态信息对应的渲染图像和第二状态信息中用户右眼的状态信息对应的渲染图像分别进行下采样，得到第一下采样结果和第二下采样结果，在此情况下，两个下采样结果可以整合到一张纹理上。下采样可以基于现有技术实现，能够使得远端装置发送第二渲染图像的数据量降低。
本领域技术人员应理解，两个下采样结果整合得到一张纹理可以有多种方式，例如分别作为一张纹理的左半部分或右半部分，或者分别作为一张纹理的上半部分或下半部分，只要能够满足在一张纹理上体现第一下采样结果和第二下采样结果即可，本申请实施例对一张纹理上二者的具体位置不作限制。
如图13所示，终端装置可执行步骤S65-S69：
S65，发送第一状态信息g1至远端装置，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S65的实现方式可以参照步骤S6的示例，其中终端装置可不设置发送第一状态信息的预设条件。
S66，接收远端装置发送的第一渲染图像x11和第一渲染图像的深度信息x11’的深度信息，第一渲染图像x11是远端装置根据第一状态信息g1和至少一个三维模型进行渲染获得的。
步骤S66的实现方式可以参照步骤S7的示例，其中，第一渲染图像可以是根据第一状态信息中用户左眼的状态信息得到的。
第一渲染图像和第一渲染图像的深度信息可以包括标识，标识可以表示第一渲染图像和第一渲染图像的深度信息对应的状态信息。在此情况下，终端装置根据标识可以确定第一渲染图像和第一渲染图像的深度信息对应于第一状态信息。在第一渲染图像和第一渲染图像的深度信息以渲染帧的形式发送时，渲染帧可以包括标识。终端装置根据标识可以确定由渲染帧得到的第一渲染图像和第一渲染图像的深度信息对应于第一状态信息。进一步地，标识还可以表示第一渲染图像和第一渲染图像的深度信息对应的第一状态信息中的用户左眼(或右眼)的状态信息。终端装置根据标识可以确定第一渲染图像和第一渲染图像的深度信息对应于第一状态信息中的用户左眼(或右眼)的状态信息。
S67，发送第二状态信息至远端装置，第二状态信息包括终端装置在第一时刻之后的第二时刻的位置信息和姿态信息。
步骤S67的实现方式可以参照步骤S27的示例，其中终端装置可不设置发送第二状态信 息的预设条件。
S68，接收第二渲染图像，第二渲染图像是远端装置根据第二状态信息和至少一个三维模型进行渲染获得的。
步骤S68的实现方式可以参照步骤S28的示例。其中，第二渲染图像可以包括第二状态信息中用户左眼的状态信息对应的渲染图像，以及第二状态信息中用户右眼的状态信息对应的渲染图像。
第二渲染图像可以包括标识，标识可以表示第二渲染图像对应的状态信息。在此情况下，终端装置根据标识可以确定第二渲染图像对应于第二状态信息。在第二渲染图像以下采样结果的形式发送时，下采样结果可以包括标识。终端装置根据标识可以确定由下采样结果得到的第二渲染图像对应于第二状态信息。
S69，根据第一渲染图像、第一渲染图像的深度信息、第二状态信息和第二渲染图像，获得多帧渲染图像。
第一渲染图像x11是对应于第一状态信息中用户左眼的状态信息g11的图像，可以直接在终端装置的左眼对应的显示区域上进行显示。为了在终端装置所在的设备的左眼和右眼对应的显示区域均显示图像，可以根据第一渲染图像x11和第一渲染图像的深度信息x11’，获得与第一状态信息g1中用户右眼的状态信息g12对应的另一渲染图像x12，使得第一渲染图像x11和另一渲染图像x12对应的虚拟视点之间距离等于视距。此时另一渲染图像是对应于第一状态信息中用户右眼的状态信息g12的图像，可以在终端装置的右眼对应的显示区域上进行显示。
例如，可以根据第一渲染图像x11和第一渲染图像的深度信息x11’，生成点云数据。其示例性实现方式可以参照上文步骤S9的相关描述。根据第一状态信息g1中用户右眼的状态信息g12对点云数据进行渲染，获得状态信息g12对应的渲染图像x12。其示例性实现方式可以参照上文步骤S10中，根据第二状态信息对点云数据进行渲染，获得第二状态信息对应的渲染图像的相关描述。
又例如，可以根据第一状态信息中用户右眼的状态信息g12、预设的近裁剪面和远裁剪面，确定多个二维线段。其示例性实现方式可以参照上文步骤S11中，根据第二状态信息、预设的近裁剪面和远裁剪面，确定多个二维线段的相关描述。根据第一状态信息中用户左眼的状态信息g11对多个二维线段进行投影，得到多个二维坐标。其示例性实现方式可以参照上文步骤S12的相关描述。获得多个二维坐标的深度信息。其示例性实现方式可以参照上文步骤S13的相关描述。将多个二维坐标的深度信息与第一渲染图像x11的深度信息进行比较，确定深度值相同的点的坐标，其示例性实现方式可以参照上文步骤S14的相关描述。根据深度值相同的点的坐标和第一渲染图像x11，获得第一状态信息中用户右眼的状态信息g12对应的渲染图像x12。其示例性实现方式可以参照上文步骤S15中，根据深度值相同的点的坐标和第一渲染图像x1，获得第二状态信息对应的渲染图像的相关描述。
由上文描述可知，终端装置对于第二渲染图像以较低的第二码率发送至终端装置，由于较低的第二码率对渲染图像的质量影响较大，可能不适于直接在终端装置所在的设备的显示区域进行显示。
可选地，步骤S69可包括S70-S72：
S70，根据第一渲染图像x11和第一渲染图像的深度信息x11’，生成点云数据。其实现方 式可以参照上文步骤S41的示例，在此不再赘述。
S71，根据第二状态信息对所述点云数据进行渲染，获得第一中间渲染图像。其实现方式可以参照上文步骤S10的示例。其中，可以根据第二状态信息得到对应于用户左眼的状态信息和对应于用户右眼的状态信息，在此情况下，参照上文步骤S10的方法，根据第二状态信息中用户左眼的状态信息对点云数据进行渲染，可以获得对应于用户左眼的渲染图像x21-x61。根据第二状态信息中用户右眼的状态信息对点云数据进行渲染，可以获得对应于用户右眼的渲染图像x22-x62。第一中间渲染图像可以包括上述对应于用户左眼的渲染图像x21-x61和对应于用户右眼的渲染图像x22-x62。
S72，利用第二渲染图像对第一中间渲染图像进行填补，获得第二状态信息对应的第二渲染图像。其实现方式可以参照上文步骤S32中利用第二渲染图像对第一中间渲染图像进行填补的示例。其中，第二渲染图像可以包括第二状态信息中用户左眼的状态信息对应的渲染图像，以及第二状态信息中用户右眼的状态信息对应的渲染图像，第一中间渲染图像可以包括对应于用户左眼的渲染图像和对应于用户右眼的渲染图像。第二状态信息中用户左眼的状态信息对应的渲染图像可以用于填补第一中间渲染图像中的对应于用户左眼的渲染图像，得到渲染图像xx21-xx61。第二状态信息中用户右眼的状态信息对应的渲染图像可以用于填补第一中间渲染图像中的对应于用户右眼的渲染图像，得到渲染图像xx22-xx62。第二状态信息对应的第二渲染图像可以包括渲染图像xx21-xx61和渲染图像xx22-xx62。通过这种方式，可以保证高质量渲染。
可选地，步骤S69可包括：
根据第二状态信息和视距确定虚拟视点的位置；截取虚拟视点与远裁剪面之间的连线在近裁剪面和远裁剪面之间的线段，确定多个二维线段；将多个二维线段投影至所述虚拟视点对应的视平面，得到多个二维坐标，其中，每一二维线段对应多个二维坐标；获得多个二维坐标的深度信息；将多个二维坐标的深度信息与第一渲染图像的深度信息进行比较，确定深度值相同的坐标，其中，深度信息包括各坐标对应的深度值。其示例性实现方式可以参照步骤S40中的相关描述。
根据深度值相同的坐标在第一渲染图像中的纹理信息，获得第二状态信息对应的第二中间渲染图像。其示例性实现方式可以参照步骤S15中，根据深度值相同的点的坐标和第一渲染图像，获得第二状态信息对应的渲染图像的相关描述。其中，根据对应于用户左眼的深度值相同的点的坐标和第一渲染图像，可对应得到对应于用户左眼的渲染图像x21-x61，根据对应于用户右眼的深度值相同的点的坐标和第一渲染图像，可对应得到对应于用户右眼的渲染图像x22-x62，第二中间渲染图像可以包括上述对应于用户左眼的渲染图像x21-x61和对应于用户右眼的渲染图像x22-x62。
利用第二渲染图像对第二中间渲染图像进行填补，获得第二状态信息对应的渲染图像。其实现方式可以参照上文步骤S32中利用第二渲染图像对第一中间渲染图像进行填补的示例。其中，第二渲染图像可以包括第二状态信息中用户左眼的状态信息对应的渲染图像，以及第二状态信息中用户右眼的状态信息对应的渲染图像，第二中间渲染图像可以包括对应于用户左眼的渲染图像和对应于用户右眼的渲染图像。第二状态信息中用户左眼的状态信息对应的渲染图像可以用于填补第二中间渲染图像中的对应于用户左眼的渲染图像，得到渲染图像xx21-xx61。第二状态信息中用户右眼的状态信息对应的渲染图像可以用于填补第二中间渲染 图像中的对应于用户右眼的渲染图像，得到渲染图像xx22-xx62。第二状态信息对应的第二渲染图像可以包括渲染图像xx21-xx61和渲染图像xx22-xx62。
终端装置所在的设备可以按照状态信息g1-g6的采集顺序，在终端装置所在的设备的显示区域，显示第一渲染图像x11、另一渲染图像x12和第二状态信息对应的渲染图像xx21-xx61、xx22-xx62，例如在左眼对应的显示区域显示x11、xx21-xx61、在右眼对应的显示区域显示x12、xx22-xx62，使得用户能够通过显示区域观察到三维空间的虚拟世界。其中，对应于相同状态信息的图像可以同时显示。
通过这种方式，使得显示区域显示的图像与用户的动作变化导致的状态信息变化相符合，提高用户体验。直接使用第一渲染图像进行显示，可以降低VR设备的数据处理成本。
下面介绍云VR系统的应用场景下，执行本申请的渲染方法实现在VR设备上将两张图像分别展示给用户的左眼和右眼的另一个示例。云端服务器上可包括远端装置，VR设备上可包括终端装置，远端装置可运行应用的服务端，终端装置可运行应用的客户端，应用可包括多个三维模型，本应用场景下，应用可以是VR应用。本申请实施例可以通过远端装置和终端装置协作实现渲染。图15示出根据本申请实施例的远端装置和终端装置在云VR系统的应用场景下的工作方式的一个示例。
如图15所示，远端装置可执行步骤S73-S78：
S73，接收第一状态信息g1，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S73的实现方式可以参照上文步骤S33的示例，在此不再赘述。
S74，根据第一状态信息g1和至少一个三维模型进行渲染，获得第一渲染图像x11和第一渲染图像的深度信息x11’。
步骤S74的实现方式可以参照上文步骤S34的示例，在此不再赘述。根据第一状态信息g1可以先得到对应于用户左眼的状态信息g11和对应于用户右眼的状态信息g12，第一渲染图像x11可以是根据用户左眼的状态信息g11(或右眼的状态信息g12)得到的。在此以第一渲染图像x11根据用户左眼的状态信息g11得到作为示例。
S75，发送第一渲染图像x11和第一渲染图像的深度信息x11’至终端装置。
步骤S75的实现方式可以参照上文步骤S35的示例，在此不再赘述。
可选地，步骤S75还可以包括S79-S80：
S79，将第一渲染图像和第一渲染图像的深度信息合成渲染帧；S80，发送渲染帧至终端装置。步骤S79、S80的实现方式可以参照上文步骤S4、S5的示例。在此情况下，发送第一渲染图像和第一渲染图像的深度信息至终端装置时，可以以第一码率对渲染帧进行编码，得到渲染帧对应的编码结果后发送该编码结果。
可选地，步骤S75之前，所述方法还包括：
步骤S801,确定发送至终端装置的上一渲染图像与第一渲染图像x11之间间隔指定个数的图像；或者，确定获得第一渲染图像时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，可以设置预设条件，使得远端装置确定在第一渲染图像x11满足预设条件时， 发送第一渲染图像x11和第一渲染图像的深度信息x11’到终端装置。预设条件可以包括：确定发送至终端装置的上一渲染图像与第一渲染图像x11之间间隔指定个数的图像；或者，确定获得第一渲染图像时刻向终端装置传输数据的带宽余量大于或等于带宽阈值。
举例来说，终端装置所在的设备采集的状态信息可以由终端装置直接发送给远端装置，由远端装置判断根据状态信息得到的渲染图像和渲染图像的深度信息是否发送给终端装置。在此情况下，步骤S73接收到第一状态信息g1后，远端装置可以执行步骤S74先得到第一渲染图像x11和第一渲染图像的深度信息x11’，再确定第一渲染图像x11是否满足预设条件，在满足条件时，执行步骤S75，将第一渲染图像x11和第一渲染图像的深度信息x11’发送至终端装置。通过这种方式，使得终端装置不需对发送到远端装置的状态信息作选择，可以降低终端装置的数据处理成本。
S76，接收第二状态信息，第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。
步骤S76的实现方式可以参照上文步骤S59的示例，在此不再赘述。
S77，根据第二状态信息和至少一个三维模型进行渲染，获得第二渲染图像。
步骤S77的实现方式可以参照上文步骤S2的示例，其中，终端装置可根据第二状态信息得到对应于用户左眼的状态信息和对应于用户右眼的状态信息，针对对应于用户左眼的状态信息和对应于用户右眼的状态信息和至少一个三维模型进行渲染，获得的第二渲染图像，可以包括第二状态信息中用户左眼的状态信息对应的渲染图像x21-x61，以及第二状态信息中用户右眼的状态信息对应的渲染图像x22-x62。
S78，以低于第一码率的第二码率将第二渲染图像发送至终端装置。
步骤S78的实现方式可以参照上文步骤S61的示例。
可选地，远端装置还可执行步骤S81-S82：
S81，根据第一状态信息g1和至少一个三维模型进行渲染，获得另一渲染图像x12，第一渲染图像x11对应的虚拟视点和另一渲染图像x12对应的虚拟视点之间距离等于视距。
步骤S81的实现方式可以参照上文步骤S2的示例，其中，另一渲染图像可以是根据第一状态信息中用户右眼的状态信息g12得到的。也即，第一渲染图像x11和另一渲染图像x12，可以分别对应于第一状态信息中用户左眼的状态信息g11和用户右眼的状态信息g12。
S82，以低于第一码率的第二码率将另一渲染图像x12发送至终端装置，第一渲染图像x11是以第一码率发送至终端装置。
以低于第二码率将另一渲染图像发送至终端装置的示例性实现方式，可以参照上文步骤S61中，以第二码率将第二渲染图像发送至终端装置的示例，在此不再赘述。
如图15所示，终端装置可执行步骤S83-S87：
S83，发送第一状态信息g1至远端装置，第一状态信息g1包括终端装置在第一时刻的位置信息和姿态信息。
步骤S83的实现方式可以参照步骤S6的示例，其中终端装置可不设置发送第一状态信息的预设条件。
S84，接收远端装置发送的第一渲染图像x11和第一渲染图像的深度信息x11’的深度信息，第一渲染图像x11是远端装置根据第一状态信息g1和至少一个三维模型进行渲染获得的。
步骤S84的实现方式可以参照步骤S66的示例，其中，第一渲染图像可以是根据第一状 态信息中用户左眼的状态信息得到的。
S85，发送第二状态信息至远端装置，第二状态信息包括终端装置在第一时刻之后的第二时刻的位置信息和姿态信息。
步骤S85的实现方式可以参照步骤S27的示例，其中终端装置可不设置发送第二状态信息的预设条件。
S86，接收第二渲染图像，第二渲染图像是远端装置根据第二状态信息和至少一个三维模型进行渲染获得的。
步骤S86的实现方式可以参照步骤S68的示例。其中，第二渲染图像可以包括第二状态信息中用户左眼的状态信息对应的渲染图像，以及第二状态信息中用户右眼的状态信息对应的渲染图像。
S87，根据第一渲染图像、第一渲染图像的深度信息、第二状态信息和第二渲染图像，获得多帧渲染图像。
第一渲染图像x11是对应于第一状态信息g1中用户左眼的状态信息g11的图像，可以直接在终端装置所在的设备的左眼对应的显示区域上进行显示。由上文描述可知，远端装置还以较低的第二码率将对应于第一状态信息g1中用户右眼的状态信息g12的的另一渲染图像x12发送至终端装置，由于较低的第二码率对渲染图像的质量影响较大，可能不适于直接在终端装置所在的设备的显示区域进行显示。
可选地，步骤S87可包括：
接收远端装置发送的另一渲染图像，另一渲染图像是远端装置根据第一状态信息和应用进行渲染获得的，第一渲染图像和另一渲染图像对应的虚拟视点之间距离等于视距。根据第一渲染图像和第一渲染图像的深度信息，生成点云数据。根据第一状态信息和视距，对点云数据进行渲染，获得第三中间渲染图像。利用远端装置发送的另一渲染图像对第三中间渲染图像进行填补，获得第一状态信息对应的另一渲染图像。其实现方式可以参照上文步骤S53的相关描述。
可选地，步骤S87可包括：
根据第一状态信息和视距确定虚拟视点的位置；截取虚拟视点与远裁剪面之间的连线在近裁剪面和远裁剪面之间的线段，确定多个二维线段。将多个二维线段投影至所述虚拟视点对应的视平面，得到多个二维坐标，其中，每一二维线段对应多个二维坐标。获得多个二维坐标的深度信息。将多个二维坐标的深度信息与第一渲染图像的深度信息进行比较，确定深度值相同的坐标，其中，所述深度信息包括各坐标对应的深度值。根据深度值相同的坐标在所述第一渲染图像中的纹理信息，获得所述第一状态信息对应的第四中间渲染图像。利用远端装置发送的另一渲染图像对第四中间渲染图像进行填补，获得所述第一状态信息对应的另一渲染图像xx12。其示例性实现方式可以参照上文步骤S53的相关描述。
由上文描述可知，终端装置对于第二渲染图像以较低的第二码率发送至终端装置，由于较低的第二码率对渲染图像的质量影响较大，可能不适于直接在终端装置所在的设备的显示区域进行显示。
可选地，步骤S87可包括S88-S90：
S88，根据第一渲染图像x11和第一渲染图像的深度信息x11’，生成点云数据。其实现方式可以参照上文步骤S70的示例，在此不再赘述。
S89，根据第二状态信息对所述点云数据进行渲染，获得第一中间渲染图像。其实现方式可以参照上文步骤S71的示例，在此不再赘述。
S90，利用第二渲染图像对第一中间渲染图像进行填补，获得第二状态信息对应的第二渲染图像。其实现方式可以参照上文步骤S72的示例，在此不再赘述。第二状态信息对应的渲染图像可以包括渲染图像xx21-xx61和渲染图像xx22-xx62。
可选地，步骤S69可包括：
根据第二状态信息和视距确定虚拟视点的位置；截取虚拟视点与远裁剪面之间的连线在近裁剪面和远裁剪面之间的线段，确定多个二维线段；将多个二维线段投影至所述虚拟视点对应的视平面，得到多个二维坐标，其中，每一二维线段对应多个二维坐标；获得多个二维坐标的深度信息；将多个二维坐标的深度信息与第一渲染图像的深度信息进行比较，确定深度值相同的坐标，其中，深度信息包括各坐标对应的深度值；根据深度值相同的坐标在第一渲染图像中的纹理信息，获得第二状态信息对应的第二中间渲染图像；利用第二渲染图像对第二中间渲染图像进行填补，获得第二状态信息对应的渲染图像。其实现方式可以参照上文步骤S69的相关描述。第二状态信息对应的渲染图像可以包括渲染图像xx21-xx61和渲染图像xx22-xx62。
终端装置所在的设备可以按照状态信息g1-g6的采集顺序，在终端装置所在的设备的显示区域，显示第一渲染图像x11、第一状态信息对应的另一渲染图像xx12和第二状态信息对应的渲染图像xx21-xx61、xx22-xx62，例如在左眼对应的显示区域显示x11、xx21-xx61、在右眼对应的显示区域显示xx12、xx22-xx62，使得用户能够通过显示区域观察到三维空间的虚拟世界。其中，对应于相同状态信息的图像可以同时显示。
通过这种方式，使得显示区域显示的图像与用户的动作变化导致的状态信息变化相符合，提高用户体验。直接使用第一渲染图像进行显示，可以降低VR设备的数据处理成本。
在一种可能的实现方式中，对于图8、13、15的各示例，远端装置下行传输到终端装置的第一渲染图像的编码结果和第二渲染图像的编码结果是串行传输，但由于第二渲染图像的编码结果数据量小，因此第二渲染图像的编码结果的解码速度相对于第一渲染图像的编码结果的解码速度要快，本来在远端装置上晚于第一渲染图像得到的第二渲染图像，在终端装置上可能早于第一渲染图像解码获得。如果解码得到第二渲染图像时，第一渲染图像尚未解码得到，则可以先保存该第二渲染图像，并等待对应的第一渲染图像获取完成后，再根据第一渲染图像进行渲染，以及根据第二渲染图像进行渲染图像的增强。如果解码得到第一渲染图像时，其对应的5个第二渲染图像中的全部或部分尚未解码得到，则可以先保存第一渲染图像，并等待获取到其对应的第二渲染图像中的至少一个后，再根据第一渲染图像进行渲染，以及根据第二渲染图像进行渲染图像的增强，或者，也可以先完成渲染得到第二状态信息对应的渲染图像，等待对应的第二渲染图像解码完成后进行图像效果增强，本领域人员应理解，只要使得用于效果增强的一帧第二渲染图像，与进行效果增强的图像对象(第二状态信息对应的渲染图像)为同一时刻采集即可，本申请对使用各第二渲染图像进行效果增强的具体顺序不作限制。这一机制同样适用于上文中包括另一渲染图像的相关描述的各示例(例如图8、11、12、13、15)。
本申请实施例在满足云VR/AR系统的功能基础上，带来了降低网络带宽、降低时延和提高帧率的效益。在原有云VR/AR系统中，采用高码率视频流方式传输内容，将占用较高的网 络带宽，网络带宽的高消耗对云VR/AR的普及带来挑战。在本申请中，通过以第一码率传输第一渲染图像，以第二码率传输第二渲染图像和/或另一渲染图像，第一码率的第一渲染图像大小与视频帧相同，第二码率的第二渲染图像和/或另一渲染图像为视频帧的1/5-1/6，可以为云VR/AR降低60％左右的带宽消耗，将极大降低云VR/AR的运营成本，推动云VR/AR的广泛应用。
图16示出根据本申请实施例的渲染方法的示例性示意图。
在一种可能的实现方式中，本申请实施例提供一种渲染方法，所述方法应用于远端装置，所述远端装置所在的渲染系统还包括终端装置，所述远端装置运行应用的服务端，所述应用包括多个三维模型，所述方法包括：S3310，接收第一状态信息，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；S3320，根据所述第一状态信息和至少一个三维模型进行渲染，获得第一渲染图像和所述第一渲染图像的深度信息；S3330，发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置。
其中，远端装置可以参照上文中云端服务器的示例，终端装置可以参照上文中VR/AR设备的示例，应用可以参照上文中的VR/AR应用的示例，应用的服务端可以参照上文中的服务端程序的示例，应用包括的多个三维模型可以参照上文中的VR/AR应用中预先设置的、物体或者角色的三维模型的示例。
步骤S3310的示例性实现方式可以参照上文步骤S1、S16、S33、S43、S56、S73的相关描述。步骤S3320的示例性实现方式可以参照上文步骤S2、S17、S34、S44、S57、S74的相关描述。步骤S3330的示例性实现方式可以参照上文步骤S3、S18、S35、S45、S58、S75的相关描述。
根据本申请实施例的渲染方法，远端装置上运行应用的服务端，且应用包括多个三维模型，使得远端装置能够使用至少一个三维模型进行渲染；远端装置根据至少一个三维模型以及接收到的第一状态信息，可以渲染得到第一渲染图像和第一渲染图像的深度信息，远端装置将第一渲染图像和第一渲染图像的深度信息至终端装置，使得终端装置可以使用第一渲染图像和第一渲染图像的深度信息得到可以展示给用户的多帧图像。远端装置只需传输一次第一渲染图像和第一渲染图像的深度信息，即可在终端装置上对应得到多帧图像，使得远端装置传输第一渲染图像和第一渲染图像的深度信息的频率得以降低，远端装置对应云端服务器、终端装置对应VR/AR设备时，能够降低云端与VR/AR设备之间数据传输的带宽消耗，使得数据传输成本降低。
第一渲染图像可以包括一帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云AR场景下远端装置能够输出第一渲染图像，以便终端装置接收到第一渲染图像时能够作出响应以实现显示区域的图像显示，并能够使得在云VR场景下远端装置能够输出第一渲染图像，以便终端装置接收到第一渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域中的至少一个显示区域的图像显示。
第一渲染图像可以包括两帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云VR场景下远端装置能够输出第一渲染图像，以便终端装置接收到第一渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域的图像显示。
在一种可能的实现方式中，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置；所述接收第一状态信息后，所述方法还包括：接收第二状态信息，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；根据所述第二状态信息和所述至少一个三维模型进行渲染，获得第二渲染图像；以低于所述第一码率的第二码率将所述第二渲染图像发送至所述终端装置。
其中，以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置，可以参照上文步骤S18、S58、S75的示例。接收第二状态信息的实现方式可以参照上文步骤S19、S59、S76的示例。根据所述第二状态信息和所述至少一个三维模型进行渲染，获得第二渲染图像的实现方式，可以参照上文中步骤S20、S60、S77的示例。以低于所述第一码率的第二码率将所述第二渲染图像发送至所述终端装置的实现方式，可以参照上文步骤S21、S61、S78的示例。
第二状态信息包括所述终端装置在第一时刻之后的第二时刻的位置信息和姿态信息，且状态信息还用于得到第二渲染图像，因此第二渲染图像是对应于第二时刻的渲染图像，通过以较低的第二码率传输第二渲染图像，使得第二渲染图像具有较小的数据量，终端装置在获得对应于第一时刻的渲染图像的基础上，还可以获得对应于第二时刻的渲染图像，在保证用户使用终端装置具有较好的体验的同时，可以降低远端装置的下行传输数据成本。
第二渲染图像可以包括一帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云AR场景下远端装置能够输出第一渲染图像以及第二渲染图像，以便终端装置接收到第一渲染图像以及第二渲染图像时能够作出响应以实现对应用户左眼和右眼的一个显示区域的图像显示，并能够使得在云VR场景下远端装置能够输出第一渲染图像以及第二渲染图像，以便终端装置接收到第一渲染图像以及第二渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域中的至少一个显示区域的图像显示。
在一种可能的实现方式中，所述第二渲染图像包括两帧渲染图像，所述两帧渲染图像对应的虚拟视点之间距离等于视距。
第二渲染图像可以参照上文图13、15的相关描述中的第二渲染图像的示例。
第二渲染图像可以包括两帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云VR场景下远端装置能够输出第一渲染图像以及第二渲染图像，以便终端装置接收到第一渲染图像以及第二渲染图像时能够作出响应，以实现分别对应用户左眼和右眼的两个显示区域的图像显示，同时保证显示区域显示的图像中与第二状态信息对应的图像的质量。
在一种可能的实现方式中，所述方法还包括：根据所述第一状态信息和所述至少一个三维模型进行渲染，获得另一渲染图像，所述第一渲染图像对应的虚拟视点和所述另一渲染图像对应的虚拟视点之间距离等于视距；以低于第一码率的第二码率将所述另一渲染图像发送至所述终端装置，所述第一渲染图像是以所述第一码率发送至所述终端装置。
根据所述第一状态信息和所述至少一个三维模型进行渲染，获得另一渲染图像，可以参照上文步骤S46、S81的示例。以低于第一码率的第二码率将所述另一渲染图像发送至所述终端装置，可以参照上文步骤S47、S82的示例。
通过这种方式，使得执行本申请实施例的渲染方法能够使得云VR场景下，远端装置能够输出第一渲染图像以及另一渲染图像，以便终端装置接收到第一渲染图像以及另一渲染图像时能够作出响应，以实现分别对应用户左眼和右眼的两个显示区域的图像显示，同时保证显示区域显示的图像中与第一状态信息对应的图像的质量。
在一种可能的实现方式中，所述以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置之前，所述方法还包括：确定发送至所述终端装置的上一渲染图像与所述渲染图像之间间隔指定个数的图像；或者，确定获得所述渲染图像时刻向所述终端装置传输数据的带宽余量大于或等于带宽阈值。
其示例性实现方式可以参照步骤S24、S50、S64、S801的相关描述。
可以在执行所述渲染方法前先根据远端装置向终端装置传输数据的带宽上限以及带宽阈值预设好指定个数，使得间隔指定个数的图像发送第一渲染图像，不会使得远端装置向终端装置传输数据的带宽余量小于带宽阈值。在此情况下，通过先确定发送至所述终端装置的上一渲染图像与所述渲染图像之间间隔指定个数的图像，再以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置，使得远端装置不会在带宽余量过小时输出第一渲染图像和第一渲染图像的深度信息，可以保持远端装置向终端装置传输数据信息的顺畅度，并且终端装置上的图像显示不会因第一渲染图像和和第一渲染图像的深度信息未及时接收到而出现延迟。
通过先确定获得所述渲染图像时刻向所述终端装置传输数据的带宽余量大于或等于带宽阈值，再以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置，使得远端装置不会在带宽余量过小时输出第一渲染图像和第一渲染图像的深度信息，可以保持远端装置向终端装置传输数据信息的顺畅度，并且终端装置上的图像显示不会因第一渲染图像和第一渲染图像的深度信息未及时接收到而出现延迟。
在一种可能的实现方式中，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：将所述第一渲染图像和所述第一渲染图像的深度信息合成渲染帧；发送所述渲染帧至所述终端装置。
将所述第一渲染图像和所述第一渲染图像的深度信息合成渲染帧的实现方式，可以参照上文步骤S4、S22、S36、S48、S62、S79的示例。发送所述渲染帧至所述终端装置的实现方式，可以参照上文的步骤S5、S23、S37、S49、S63、S80的示例。
通过将第一渲染图像和第一渲染图像的深度信息合成后传输，避免了位数较高的深度信息单独传输，可以降低远端装置的数据传输成本。
图17示出根据本申请实施例的渲染方法的示例性示意图。
在一种可能的实现方式中，本申请还提供一种渲染方法，所述方法应用于终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述方法包括：S3410，发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；S3420，接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述 第一状态信息和至少一个三维模型进行渲染获得的；S3430，根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像。
步骤S3410的实现方式可以参照上文步骤S6、S38、S51的示例。步骤S3420的实现方式可以参照上文步骤S7、S39、S52的示例。步骤S3430的实现方式可以参照上文步骤S8、S40、S53的示例。
根据本申请实施例的渲染方法，终端装置发送包括终端装置在第一时刻的位置信息和姿态信息的第一状态信息到远端装置，使得远端装置根据至少一个三维模型以及接收到的第一状态信息，可以渲染得到第一渲染图像和第一渲染图像的深度信息，并输出给终端装置，进而使得终端装置可以接收到并使用第一渲染图像和第一渲染图像的深度信息，得到多帧渲染图像。终端装置只需传输一次状态信息，且远端装置只需传输一次第一渲染图像和第一渲染图像的深度信息，即可在终端装置上对应得到多帧渲染图像，使得远端装置传输第一渲染图像和第一渲染图像的深度信息的频率得以降低，远端装置对应云端服务器、终端装置对应VR/AR设备时，能够降低云端与VR/AR设备之间数据传输的带宽消耗，使得数据传输成本降低。
第一渲染图像可以包括一帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云AR场景下，终端装置接收到第一渲染图像时能够作出响应以实现显示区域的图像显示，并能够使得在云VR场景下远端装置能够输出第一渲染图像，以便终端装置接收到第一渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域中的至少一个显示区域的图像显示。
第一渲染图像可以包括两帧图像，在此情况下，执行本申请实施例的渲染方法能够使得在云VR场景下，终端装置接收到第一渲染图像时能够作出响应以实现分别对应用户左眼和右眼的两个显示区域的图像显示。
在一种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。其实现方式可以参照上文步骤S8、S40、S53的示例。
根据第二状态信息可以获得多帧渲染图像中，第二状态信息对应的渲染图像，且第二状态信息包括终端装置在第一时刻之后的第二时刻的位置信息和姿态信息，因此，存在多个不同的第二状态信息时，可以获得多个不同的第二状态信息对应的渲染图像，若多个不同的第二状态信息与第一状态信息是符合用户动作的状态信息，使得根据多个不同的第二状态信息对应的渲染图像，可以满足呈现与用户动作相符合的渲染图像给用户，保证用户体验。
在一种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，获得所述第一状态信息对应的另一渲染图像，所述第一渲染图像和所述另一渲染图像对应的虚拟视点之间距离等于视距。其实现方式可以参照上文步骤S40、S53的示例。
通过这种方式，使得执行本申请实施例的渲染方法能够使得云VR场景下，终端装置接收到第一渲染图像时能够作出响应，以实现使用第一渲染图像和另一渲染图像分别对应用户左 眼和右眼的两个显示区域的图像显示，同时保证显示区域显示的图像中与第一状态信息对应、且与第一渲染图像不同的另一渲染图像的质量。
在一种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第二状态信息和所述点云数据进行渲染，获得所述第二状态信息对应的渲染图像。
根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据的实现方式，可以参照上文步骤S9、S41、S54的示例。根据所述第二状态信息和所述点云数据进行渲染，获得所述第二状态信息对应的渲染图像的实现方式，可以参照上文步骤S10、S42、S55的示例。
点云数据可以包括多个三维的点，可以表示第一渲染图像中的物体在三维场景中的位置、形状等信息。结合第二状态信息，可以得到在第二状态信息下，用户观察三维场景时可以看到的点，根据得到的点以及第一渲染图像，可以得到在第二状态信息下，用户观察三维场景时可以看到的图像。从而能得到用于展示给用户的图像。
图18示出根据本申请实施例的渲染方法的示例性示意图。
本申请实施例还提供一种渲染方法，所述方法应用于终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述方法包括：S3510，发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；S3520，接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；S3530，发送第二状态信息至所述远端装置，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；S3540，接收第二渲染图像，所述第二渲染图像是所述远端装置根据所述第二状态信息和所述至少一个三维模型进行渲染获得的；S3550，根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像。
步骤S3510的实现方式可以参照上文步骤S25、S65、S83的示例。
步骤S3520的实现方式可以参照上文步骤S26、S66、S84的示例。
步骤S3530的实现方式可以参照上文步骤S27、S67、S85的示例。本领域技术人员应理解，步骤S3530也可能在步骤S3520之前执行，只要满足步骤S3530晚于步骤S3510执行即可，本申请对于步骤S3530与步骤S3520的执行顺序不作限制。
步骤S3540的实现方式可以参照上文步骤S28、S68、S86的示例。
步骤S3550的实现方式可以参照上文步骤S29、S69、S87的示例。
根据本申请实施例的渲染方法，终端装置发送包括终端装置在第一时刻的位置信息和姿态信息的第一状态信息到远端装置，使得远端装置根据至少一个三维模型以及接收到的第一状态信息，可以渲染得到第一渲染图像和第一渲染图像的深度信息，并输出给终端装置，终端装置发送包括终端装置在第二时刻的位置信息和姿态信息的第一状态信息到远端装置，使得远端装置根据至少一个三维模型以及接收到的第二状态信息，可以渲染得到第二渲染图像并输出给终端装置，进而使得终端装置可以接收到并使用第一渲染图像、第一渲染图像的深 度信息、第二渲染图像以及使用终端装置上的第二状态信息，得到多帧渲染图像。通过这种方式，可以保证得到的多帧渲染图像的质量较好，将多帧渲染图像展示给用户时，可以提升用户体验。
在一种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第二状态信息对所述点云数据进行渲染，获得第一中间渲染图像；利用所述第二渲染图像对所述第一中间渲染图像进行填补，获得所述第二状态信息对应的第二渲染图像。
根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据的实现方式可以参照上文步骤S30、S70、S88的示例。根据所述第二状态信息对所述点云数据进行渲染，获得第一中间渲染图像可以参照上文步骤S31、S71、S89的示例。利用所述第二渲染图像对所述第一中间渲染图像进行填补，获得所述第二状态信息对应的第二渲染图像可以参照上文步骤S32、S72、S90的示例。
点云数据可以包括多个三维的点，可以表示第一渲染图像中的物体在三维场景中的位置、形状等信息。结合第二状态信息，可以得到在第二状态信息下，用户观察三维场景时可以看到的点，根据得到的点以及第一渲染图像，可以得到在第二状态信息下，用户观察三维场景时可以看到的图像，即第一中间渲染图像。第二渲染图像也是根据第二状态信息得到的，基于第二渲染图像可以对第一中间渲染图像进行处理，得到第二状态信息对应的第二渲染图像。从而能得到用于展示给用户的图像。
图19示出根据本申请实施例的远端装置的示例性结构示意图。
在一种可能的实现方式中，本申请提供一种远端装置，所述远端装置所在的渲染系统还包括终端装置，所述远端装置运行应用的服务端，所述应用包括多个三维模型，所述装置包括：第一接收模块701，用于接收第一状态信息，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；第一渲染模块702，用于根据所述第一状态信息和至少一个三维模型进行渲染，获得第一渲染图像和所述第一渲染图像的深度信息；第一发送模块703，用于发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置。
在一种可能的实现方式中，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置；第一接收模块之后，所述装置还包括：第二接收模块，用于接收第二状态信息，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；第二渲染模块，用于根据所述第二状态信息和所述至少一个三维模型进行渲染，获得第二渲染图像；第二发送模块，用于以低于所述第一码率的第二码率将所述第二渲染图像发送至所述终端装置。
在一种可能的实现方式中，所述第二渲染图像包括两帧渲染图像，所述两帧渲染图像对应的虚拟视点之间距离等于视距。
在一种可能的实现方式中，所述装置还包括：第三渲染模块，用于根据所述第一状态信 息和所述至少一个三维模型进行渲染，获得另一渲染图像，所述第一渲染图像对应的虚拟视点和所述另一渲染图像对应的虚拟视点之间距离等于视距；第三发送模块，用于以低于第一码率的第二码率将所述另一渲染图像发送至所述终端装置，所述第一渲染图像是以所述第一码率发送至所述终端装置。
在一种可能的实现方式中，第一发送模块之前，所述装置还包括：第一确定模块，用于确定发送至所述终端装置的上一渲染图像与所述渲染图像之间间隔指定个数的图像；或者，确定获得所述渲染图像时刻向所述终端装置传输数据的带宽余量大于或等于带宽阈值。
在一种可能的实现方式中，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：将所述第一渲染图像和所述第一渲染图像的深度信息合成渲染帧；发送所述渲染帧至所述终端装置。
图20示出根据本申请实施例的终端装置的示例性结构示意图。
在一种可能的实现方式中，本申请提供一种终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述装置包括：第四发送模块704，用于发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；第四接收模块705，用于接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；第一图像获得模块706，用于根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像。
在一种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。
在一种可能的实现方式中，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，获得所述第一状态信息对应的另一渲染图像，所述第一渲染图像和所述另一渲染图像对应的虚拟视点之间距离等于视距。
在一种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第二状态信息和所述点云数据进行渲染，获得所述第二状态信息对应的渲染图像。
图21示出根据本申请实施例的终端装置的示例性结构示意图。
在一种可能的实现方式中，本申请提供一种终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述装置包括：第五发送模块707，用于发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；第五接收模块708，用于接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；第六发送模块709，用于发送第 二状态信息至所述远端装置，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；第六接收模块710，用于接收第二渲染图像，所述第二渲染图像是所述远端装置根据所述第二状态信息和所述至少一个三维模型进行渲染获得的；第二图像获得模块711，用于根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像。
在一种可能的实现方式中，所述根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像包括：根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；根据所述第二状态信息对所述点云数据进行渲染，获得第一中间渲染图像；利用所述第二渲染图像对所述第一中间渲染图像进行填补，获得所述第二状态信息对应的第二渲染图像。
本申请的实施例提供了一种计算设备集群，包括至少一个计算设备，每个计算设备包括处理器和存储器，其中，所述至少一个计算设备的处理器被配置用于执行上述方法。
本申请的实施例提供了一种终端设备，包括：处理器以及用于存储处理器可执行指令的存储器；其中，所述处理器被配置为执行所述指令时实现上述方法。
本申请的实施例提供了一种非易失性计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现上述方法。
本申请的实施例提供了一种计算机程序产品，包括计算机可读代码，或者承载有计算机可读代码的非易失性计算机可读存储介质，当所述计算机可读代码在电子设备的处理器中运行时，所述电子设备中的处理器执行上述方法。
图22示出根据本申请实施例的计算设备集群的示例性结构示意图。
本申请还提供了一种计算设备集群，如图20所示，该计算设备集群包括至少一台计算设备2000，每个计算设备2000包括总线2001、处理器2002、通信接口2003和存储器2004。处理器2002、存储器2004和通信接口2003之间通过总线2001通信。不同计算设备2000之间可以通过通信网络建立通信通路。每个计算设备2000上运行第一接收模块701、第一渲染模块702以及第一发送模块703中的任意一个或多个。任一计算设备2000可以为云环境中的计算设备，或边缘环境中的计算设备，或终端环境中的计算设备。
图23示出根据本申请实施例的终端设备的示例性结构示意图。
如图23所示，终端设备可以包括桌面型计算机、膝上型计算机、手持计算机、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、个人数字助理(personal digital assistant，PDA)、增强现实(augmented reality，AR)设备、虚拟现实(virtual reality，VR)设备、人工智能(artificial intelligence,AI)设备、可穿戴式设备、车载设备、智能家居设备、或智慧城市设备、服务器设备中的至少一种。本申请实施例对该计算设备集群的具体类型不作特殊限制。
终端设备可以包括处理器110，存储器121。可以理解的是，本申请实施例示意的结构并不构成对终端设备的具体限定。在本申请另一些实施例中，终端设备可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit， GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
处理器110可以根据指令操作码和时序信号，产生操作控制信号，完成获取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器可以为高速缓冲存储器。该存储器可以保存处理器110用过或使用频率较高的指令或数据。如果处理器110需要使用该指令或数据，可从该存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
存储器121可以用于存储计算机可执行程序代码，该可执行程序代码包括指令。存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序等。存储数据区可存储终端设备使用过程中所创建的数据等。此外，存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。处理器110通过运行存储在存储器121的指令，和/或存储在设置于处理器中的存储器的指令，执行终端设备的各种功能方法或上述渲染方法。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(Random Access Memory，RAM)、只读存储器(Read Only Memory，ROM)、可擦式可编程只读存储器(Electrically Programmable Read-Only-Memory，EPROM或闪存)、静态随机存取存储器(Static Random-Access Memory，SRAM)、便携式压缩盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、数字多功能盘(Digital Video Disc，DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。
这里所描述的计算机可读程序指令或代码可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本申请操作的计算机程序指令可以是汇编指令、指令集架构(Instruction Set Architecture，ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意 种类的网络—包括局域网(Local Area Network，LAN)或广域网(Wide Area Network，WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或可编程逻辑阵列(Programmable Logic Array，PLA)，该电子电路可以执行计算机可读程序指令，从而实现本申请的各个方面。
这里参照根据本申请实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本申请的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本申请的多个实施例的装置、系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。
也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行相应的功能或动作的硬件(例如电路或ASIC(Application Specific Integrated Circuit，专用集成电路))来实现，或者可以用硬件和软件的组合，如固件等来实现。
尽管在此结合各实施例对本发明进行了描述，然而，在实施所要求保护的本发明过程中，本领域技术人员通过查看所述附图、公开内容、以及所附权利要求书，可理解并实现所述公开实施例的其它变化。在权利要求中，“包括”(comprising)一词不排除其他组成部分或步骤，“一”或“一个”不排除多个的情况。单个处理器或其它单元可以实现权利要求中列举的若干项功能。相互不同的从属权利要求中记载了某些措施，但这并不表示这些措施不能组合起来产生良好的效果。
以上已经描述了本申请的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术的改进，或者使本技术领域的其它普通技术人员 能理解本文披露的各实施例。

Claims (20)

  1. 一种渲染方法，其特征在于，所述方法应用于远端装置，所述远端装置所在的渲染系统还包括终端装置，所述远端装置运行应用的服务端，所述应用包括多个三维模型，所述方法包括：
    接收第一状态信息，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；
    根据所述第一状态信息和至少一个三维模型进行渲染，获得第一渲染图像和所述第一渲染图像的深度信息；
    发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置。
  2. 根据权利要求1所述的方法，其特征在于，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：
    以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置；
    所述接收第一状态信息后，所述方法还包括：
    接收第二状态信息，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；
    根据所述第二状态信息和所述至少一个三维模型进行渲染，获得第二渲染图像；
    以低于所述第一码率的第二码率将所述第二渲染图像发送至所述终端装置。
  3. 根据权利要求2所述的方法，其特征在于，所述第二渲染图像包括两帧渲染图像，所述两帧渲染图像对应的虚拟视点之间距离等于视距。
  4. 根据权利要求1至3中任一所述的方法，其特征在于，所述方法还包括：
    根据所述第一状态信息和所述至少一个三维模型进行渲染，获得另一渲染图像，所述第一渲染图像对应的虚拟视点和所述另一渲染图像对应的虚拟视点之间距离等于视距；
    以低于第一码率的第二码率将所述另一渲染图像发送至所述终端装置，所述第一渲染图像是以所述第一码率发送至所述终端装置。
  5. 根据权利要求2至4中任一所述的方法，其特征在于，所述以第一码率将所述第一渲染图像和所述第一渲染图像的深度信息发送至所述终端装置之前，所述方法还包括：
    确定发送至所述终端装置的上一渲染图像与所述第一渲染图像之间间隔指定个数的图像；或者，
    确定获得所述第一渲染图像时刻向所述终端装置传输数据的带宽余量大于或等于带宽阈值。
  6. 根据权利要求1至5中任一所述的方法，其特征在于，所述发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置包括：
    将所述第一渲染图像和所述第一渲染图像的深度信息合成渲染帧；
    发送所述渲染帧至所述终端装置。
  7. 一种渲染方法，其特征在于，所述方法应用于终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述方法包括：
    发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；
    接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；
    根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像。
  8. 根据权利要求7所述的方法，其特征在于，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：
    根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息。
  9. 根据权利要求7或8所述的方法，其特征在于，所述根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像包括：
    根据所述第一渲染图像和所述第一渲染图像的深度信息，获得所述第一状态信息对应的另一渲染图像，所述第一渲染图像和所述另一渲染图像对应的虚拟视点之间距离等于视距。
  10. 根据权利要求8所述的方法，其特征在于，所述根据所述第一渲染图像、所述第一渲染图像的深度信息和第二状态信息，获得所述第二状态信息对应的渲染图像包括：
    根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；
    根据所述第二状态信息和所述点云数据进行渲染，获得所述第二状态信息对应的渲染图像。
  11. 一种渲染方法，其特征在于，所述方法应用于终端装置，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述方法包括：
    发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；
    接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；
    发送第二状态信息至所述远端装置，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；
    接收第二渲染图像，所述第二渲染图像是所述远端装置根据所述第二状态信息和所述至少一个三维模型进行渲染获得的；
    根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二 渲染图像，获得多帧渲染图像。
  12. 根据权利要求11所述的方法，其特征在于，所述根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像包括：
    根据所述第一渲染图像和所述第一渲染图像的深度信息，生成点云数据；
    根据所述第二状态信息对所述点云数据进行渲染，获得第一中间渲染图像；
    利用所述第二渲染图像对所述第一中间渲染图像进行填补，获得所述第二状态信息对应的第二渲染图像。
  13. 一种远端装置，其特征在于，所述远端装置所在的渲染系统还包括终端装置，所述远端装置运行应用的服务端，所述应用包括多个三维模型，所述装置包括：
    第一接收模块，用于接收第一状态信息，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；
    第一渲染模块，用于根据所述第一状态信息和至少一个三维模型进行渲染，获得第一渲染图像和所述第一渲染图像的深度信息；
    第一发送模块，用于发送所述第一渲染图像和所述第一渲染图像的深度信息至所述终端装置。
  14. 一种终端装置，其特征在于，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述装置包括：
    第四发送模块，用于发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；
    第四接收模块，用于接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；
    第一图像获得模块，用于根据所述第一渲染图像和所述第一渲染图像的深度信息获得多帧渲染图像。
  15. 一种终端装置，其特征在于，所述终端装置所在的渲染系统还包括远端装置，所述终端装置运行应用的客户端，所述应用包括多个三维模型，所述装置包括：
    第五发送模块，用于发送第一状态信息至所述远端装置，所述第一状态信息包括所述终端装置在第一时刻的位置信息和姿态信息；
    第五接收模块，用于接收所述远端装置发送的第一渲染图像和所述第一渲染图像的深度信息，所述第一渲染图像是所述远端装置根据所述第一状态信息和至少一个三维模型进行渲染获得的；
    第六发送模块，用于发送第二状态信息至所述远端装置，所述第二状态信息包括所述终端装置在所述第一时刻之后的第二时刻的位置信息和姿态信息；
    第六接收模块，用于接收第二渲染图像，所述第二渲染图像是所述远端装置根据所述第二状态信息和所述至少一个三维模型进行渲染获得的；
    第二图像获得模块，用于根据所述第一渲染图像、所述第一渲染图像的深度信息、所述第二状态信息和所述第二渲染图像，获得多帧渲染图像。
  16. 一种计算设备集群，其特征在于，包括至少一个计算设备，每个计算设备包括处理器和存储器，所述至少一个计算设备的处理器用于执行权利要求1-8中任一项所述的方法。
  17. 一种终端设备，其特征在于，包括：
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器被配置为执行所述指令时实现权利要求7-10任意一项所述的方法。
  18. 一种终端设备，其特征在于，包括：
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器被配置为执行所述指令时实现权利要求11-12任意一项所述的方法。
  19. 一种非易失性计算机可读存储介质，其上存储有计算机程序指令，其特征在于，所述计算机程序指令被处理器执行时实现权利要求1-6中任意一项所述的方法。
  20. 一种计算机程序产品，包括计算机可读代码，或者承载有计算机可读代码的非易失性计算机可读存储介质，其特征在于，当所述计算机可读代码在数据处理装置中运行时，所述数据处理装置中的处理器执行权利要求1-6中任意一项所述的方法。
PCT/CN2022/109107 2021-09-03 2022-07-29 渲染方法、远端装置、计算设备集群、终端装置及设备 WO2023029845A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111032167.2 2021-09-03
CN202111032167.2A CN115761096A (zh) 2021-09-03 2021-09-03 渲染方法、远端装置、计算设备集群、终端装置及设备

Publications (1)

Publication Number Publication Date
WO2023029845A1 true WO2023029845A1 (zh) 2023-03-09

Family

ID=85332455

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/109107 WO2023029845A1 (zh) 2021-09-03 2022-07-29 渲染方法、远端装置、计算设备集群、终端装置及设备

Country Status (2)

Country Link
CN (1) CN115761096A (zh)
WO (1) WO2023029845A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170078447A1 (en) * 2015-09-10 2017-03-16 EEVO, Inc. Adaptive streaming of virtual reality data
CN107767437A (zh) * 2017-09-15 2018-03-06 欧昊 一种多层混合异步渲染方法
CN111586391A (zh) * 2020-05-07 2020-08-25 中国联合网络通信集团有限公司 一种图像处理方法、装置及系统
CN112738495A (zh) * 2019-10-28 2021-04-30 阿里巴巴集团控股有限公司 虚拟视点图像生成方法、系统、电子设备及存储介质
CN112738553A (zh) * 2020-12-18 2021-04-30 深圳市微网力合信息技术有限公司 一种基于网络通信质量的自适应云渲染系统及方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170078447A1 (en) * 2015-09-10 2017-03-16 EEVO, Inc. Adaptive streaming of virtual reality data
CN107767437A (zh) * 2017-09-15 2018-03-06 欧昊 一种多层混合异步渲染方法
CN112738495A (zh) * 2019-10-28 2021-04-30 阿里巴巴集团控股有限公司 虚拟视点图像生成方法、系统、电子设备及存储介质
CN111586391A (zh) * 2020-05-07 2020-08-25 中国联合网络通信集团有限公司 一种图像处理方法、装置及系统
CN112738553A (zh) * 2020-12-18 2021-04-30 深圳市微网力合信息技术有限公司 一种基于网络通信质量的自适应云渲染系统及方法

Also Published As

Publication number Publication date
CN115761096A (zh) 2023-03-07

Similar Documents

Publication Publication Date Title
US11024083B2 (en) Server, user terminal device, and control method therefor
JP4214976B2 (ja) 擬似立体画像作成装置及び擬似立体画像作成方法並びに擬似立体画像表示システム
US20170359570A1 (en) Multi-View Interactive Digital Media Representation Lock Screen
CN113099204B (zh) 一种基于vr头戴显示设备的远程实景增强现实方法
US11119719B2 (en) Screen sharing for display in VR
WO2019238114A1 (zh) 动态模型三维重建方法、装置、设备和存储介质
US11436787B2 (en) Rendering method, computer product and display apparatus
KR102441514B1 (ko) 하이브리드 스트리밍
US10861213B1 (en) System and method for automatic generation of artificial motion blur
US12100106B2 (en) Stereoscopic rendering of virtual 3D objects
WO2022151972A1 (zh) 视频编码方法、装置、设备及存储介质
WO2023029845A1 (zh) 渲染方法、远端装置、计算设备集群、终端装置及设备
US20230196658A1 (en) Enclosed multi-view visual media representation
Pintore et al. Deep scene synthesis of Atlanta-world interiors from a single omnidirectional image
TWI817335B (zh) 立體影像播放裝置及其立體影像產生方法
US20230052104A1 (en) Virtual content experience system and control method for same
Van Bogaert et al. Novel view synthesis in embedded virtual reality devices
TWM630947U (zh) 立體影像播放裝置
Pintore et al. Deep synthesis and exploration of omnidirectional stereoscopic environments from a single surround-view panoramic image
WO2022224964A1 (ja) 情報処理装置及び情報処理方法
TW202325016A (zh) 生成立體光場資料的電子裝置及方法
KR20160034742A (ko) 초다시점 영상 렌더링 장치 및 방법
CN106231281A (zh) 一种显示转换方法和装置
JP2016081396A (ja) 画像処理装置及び画像処理方法、プログラム、記憶媒体

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22862986

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22862986

Country of ref document: EP

Kind code of ref document: A1