CN113473180A - 5G or wireless-based Cloud XR data transmission method and device, storage medium and electronic device - Google Patents

5G or wireless-based Cloud XR data transmission method and device, storage medium and electronic device Download PDF

Info

Publication number
CN113473180A
CN113473180A CN202110905918.0A CN202110905918A CN113473180A CN 113473180 A CN113473180 A CN 113473180A CN 202110905918 A CN202110905918 A CN 202110905918A CN 113473180 A CN113473180 A CN 113473180A
Authority
CN
China
Prior art keywords
client
picture
cloudxr
displayed
operation control
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110905918.0A
Other languages
Chinese (zh)
Other versions
CN113473180B (en
Inventor
祁国良
刘兴龙
黄智勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Boliang Shenghe Technology Co ltd
Original Assignee
Beijing Boliang Shenghe Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Boliang Shenghe Technology Co ltd filed Critical Beijing Boliang Shenghe Technology Co ltd
Priority to CN202110905918.0A priority Critical patent/CN113473180B/en
Publication of CN113473180A publication Critical patent/CN113473180A/en
Application granted granted Critical
Publication of CN113473180B publication Critical patent/CN113473180B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23412—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/005—General purpose rendering architectures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/61—Network physical structure; Signal processing
    • H04N21/6106—Network physical structure; Signal processing specially adapted to the downstream path of the transmission network
    • H04N21/6131—Network physical structure; Signal processing specially adapted to the downstream path of the transmission network involving transmission via a mobile phone network

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Graphics (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The application discloses a 5G or wireless-based Cloud XR data transmission method and device, a storage medium and an electronic device. The method comprises the steps of receiving operation control information and spatial position information input through a virtual peripheral control module of a Cloudxr client based on a Cloud XR technology; obtaining a rendering file according to a preset three-dimensional program, the operation control information and the space position information; and encoding the rendering file to obtain a picture to be displayed, and streaming the picture to be displayed to the Cloudxr client. The application solves the technical problems that the transmission efficiency is not high and the transmission is limited by wired transmission. Through the method and the device, the use operation of the user is simplified, and the cost of the terminal equipment is reduced. In addition, the method and the device provide the pictures with convenient and high decoding precision content, and improve the user experience.

Description

5G or wireless-based Cloud XR data transmission method and device, storage medium and electronic device
Technical Field
The application relates to the field of image information and data processing, in particular to a method and a device for transmitting Cloud XR data based on 5G or wireless, a storage medium and an electronic device.
Background
With the increasing computing power of GPU image processing services, three-dimensional models are more vivid and include virtual digital people, three-dimensional models, VR/AR games and the like.
In the related art, content is rendered in a GPU server, and the rendered picture is transmitted to a PCVR head display or other display device through an HDMI cable. However, due to network transmission limitation, the cost of the terminal device is high, and the terminal device is easily interfered by the HDMI cable, so that the user experience is poor.
Aiming at the problems of limited wired transmission and low transmission efficiency in the related technology, no effective solution is provided at present.
Disclosure of Invention
The present application mainly aims to provide a method and an apparatus for data transmission, a storage medium, and an electronic apparatus, so as to solve the problem of low transmission efficiency due to the limitation of wired transmission.
To achieve the above object, according to one aspect of the present application, there is provided a method of 5G or wireless based Cloud XR data transmission.
The data transmission method comprises the following steps: receiving operation control information and spatial position information input through a virtual peripheral control module of a Cloudxr client based on a Cloud XR technology; obtaining a rendering file according to a preset three-dimensional program, the operation control information and the space position information; and encoding the rendering file to obtain a picture to be displayed, and streaming the picture to be displayed to the Cloudxr client.
Further, encoding the rendered file to obtain a to-be-displayed picture, wherein the to-be-displayed picture is transmitted to the Cloudxr client in a streaming manner, and the method further includes: and transmitting the current frame of the picture to be displayed obtained after the rendering file is coded to the Cloudxr client through a wireless network, wherein the wireless network at least comprises one of the following components: 5G, WI-FI.
Further, the transmitting the current frame to the Cloudxr client through the wireless network further includes: and transmitting pictures within the Cloudxr client Fov and with Fov + preset adjustable display scale in the process of transmitting pictures to be displayed based on a dynamic Fov transmission mode.
Further, encoding the rendered file to obtain a to-be-displayed picture, and after the to-be-displayed picture is streamed to the Cloudxr client, the method further includes: and decoding the picture to be displayed at the Cloudxr client and displaying the picture.
Further, decoding the video stream based on H.264 or H.265/HEVC on the picture to be displayed at the Cloudxr client.
Further, the obtaining a rendering file according to a preset three-dimensional program, the operation control information, and the spatial position information includes: calling different interface modules according to different types of files to be rendered, wherein the types of the files to be rendered comprise one of the following types: VR, AR, MR, XR; and obtaining a rendering file by combining the operation control information and the spatial position information through a preset interface provided by an interface module according to a preset three-dimensional program.
Further, the preset three-dimensional program is used for communicating with a Directx interface or an OpenGL interface.
To achieve the above object, according to another aspect of the present application, there is provided an apparatus for data transmission.
The device for data transmission according to the application comprises: the communication module is used for receiving operation control information and spatial position information input by a virtual peripheral control module of the Cloudxr client based on the Cloud XR technology; the rendering module is used for obtaining a rendering file according to a preset three-dimensional program, the operation control information and the space position information; and the streaming transmission module is used for coding the rendering file to obtain a picture to be displayed, and the picture to be displayed is transmitted to the Cloudxr client in a streaming mode.
In order to achieve the above object, according to yet another aspect of the present application, there is provided a computer-readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the method when executed.
In order to achieve the above object, according to yet another aspect of the present application, there is provided an electronic device comprising a memory and a processor, the memory having a computer program stored therein, the processor being configured to execute the computer program to perform the method.
In the embodiment of the application, the method and the device for transmitting the Cloud XR data based on 5G or wireless, the storage medium and the electronic device adopt a mode of receiving the operation control information and the spatial position information which are input through the virtual peripheral control module of the Cloud XR client based on the Cloud XR technology, obtaining a rendering file according to a preset three-dimensional program, the operation control information and the spatial position information, achieving the purpose of obtaining a picture to be displayed after the rendering file is coded, and transmitting the picture to be displayed to the Cloudxr client in a streaming manner, thereby realizing the technical effects of simplifying the use operation of the user, reducing the cost of the terminal equipment and providing the user experience of the content with convenient and high decoding precision, and the technical problem that the picture to be displayed is obtained after the rendering file is coded and is transmitted to the Cloudxr client in a streaming mode is solved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, serve to provide a further understanding of the application and to enable other features, objects, and advantages of the application to be more apparent. The drawings and their description illustrate the embodiments of the invention and do not limit it. In the drawings:
fig. 1 is a schematic diagram of a hardware system structure of a 5G or wireless based Cloud XR data transmission method according to an embodiment of the present application;
fig. 2 is a schematic flow chart of a 5G or wireless based Cloud XR data transmission method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of an apparatus for data transmission according to an embodiment of the present application;
fig. 4 is a schematic diagram illustrating an implementation principle of a Cloud XR data transmission method based on 5G or wireless according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical terms involved in the examples of the present application are as follows:
augmented Reality (English name: Augmented Reality, abbreviation: AR)
Virtual Reality technology (English name: Virtual Reality, abbreviated as VR)
Augmented reality (English name: Extended reality, abbreviated as XR)
Mixed Reality (English name: mixed Reality, abbreviated MR)
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
The embodiment of the application provides a data transmission method, belongs to a wireless streaming technology, realizes cordless by rendering content of a GPU server and wirelessly transmitting the content to a thin client terminal, provides convenient, low-cost and high-quality pictures for users, and improves user experience.
The inventor finds that, in the aspect of a data transmission interface, in the related art, only the server and the thin client are connected through a hardware cable, so that the application range of a user is narrow and the user is inconvenient to move in the using process. Meanwhile, different thin client interaction modes are also different, different interaction modes are not subjected to standardization processing, and a client needs to develop a corresponding interaction mode by himself, so that development becomes complex.
As shown in fig. 1, a hardware system structure diagram of the data transmission method in the embodiment of the present application includes a user side and a server side. The client side at least comprises a thin client side (the thin client side also comprises a system for displaying), and the server side at least comprises a GPU server. The thin Client is configured with a Cloudxr Client program. The GPU server is configured with a Cloudxr server-side program. The GPU server is also configured with a preset three-dimensional program. Also, a Wi-Fi device is included. The user side is connected with the server side through a router or WI-FI equipment.
As shown in fig. 2, the method includes steps S201 to S203 as follows:
step S201, receiving operation control information and spatial position information input by a virtual peripheral control module of a Cloudxr client based on the Cloud XR technology;
step S202, a rendering file is obtained according to a preset three-dimensional program, the operation control information and the space position information;
step S203, encoding the rendering file to obtain a picture to be displayed, and streaming the picture to be displayed to the Cloudxr client.
From the above description, it can be seen that the following technical effects are achieved by the present application:
the method comprises the steps of obtaining a rendering file by receiving operation control information and spatial position information input through a virtual peripheral control module of a Cloudxr client based on a Cloud XR technology and according to a preset three-dimensional program, the operation control information and the spatial position information, achieving the purpose of encoding the rendering file to obtain a picture to be displayed and transmitting the picture to be displayed to the Cloudxr client in a streaming mode, thereby achieving the technical effects of simplifying the use operation of a user, reducing the cost of terminal equipment and providing the user experience of convenient and high-decoding-precision content, and further solving the technical problem that the picture to be displayed is obtained after encoding the rendering file and is transmitted to the Cloudxr client in a streaming mode.
In step 201 above, wireless VR and AR may be implemented across 5G and Wi-Fi networks based on Cloud XR technology. In addition, with NVIDIA GPU virtualization software, Cloud XR can achieve full scalability for data centers and edge networks.
Receiving operation control information and spatial location information input through a virtual peripheral control module of a Cloudxr client based on the Cloud XR technology. Namely, the Cloudxr client side carries out standardized processing on the control data through the virtual peripheral control module through the peripheral input devices including but not limited to a handle, a keyboard and a mouse, gesture recognition, voice input and the like, and then transmits the control data to the Cloudxr server side through the established network. That is, the operation control information and the spatial location information input through the virtual peripheral control module of the Cloudxr client are received at the Cloudxr server.
As an alternative embodiment, the transmission is made at the time of use based on the cloudbr SDK accompanying the installer for the server component.
As an optional implementation mode, XR augmented reality content is streamed to an Android or Windows client based on an OpenVR application program. In addition, the XR terminal comprises one or more cross-terminal devices such as a mobile phone, a tablet, a web, a VR head display all-in-one machine and the like.
In the step 202, a rendering file is obtained according to a preset three-dimensional program, the operation control information, and the spatial position information. That is, after receiving the control and spatial location information at the Cloudxr server, the operational information is communicated to the three-dimensional program, with the modules possibly called being different depending on the content/type.
As an alternative implementation, if VR content needs to be rendered, the operation information data needs to be communicated with OpenXR runtime (steamvr).
As a preference in this embodiment, the preset three-dimensional program is used for communicating with a Directx interface or an OpenGL interface. The preset three-dimensional program needs to communicate with a Directx interface or an OpenGL interface.
In the step 203, the rendered file needs to be encoded to obtain a to-be-displayed picture, and the to-be-displayed picture is transmitted to the Cloudxr client in a streaming manner, so that a data wireless transmission process of audio and video coding, wireless data transmission and thin client audio and video decoding of an image processing server is realized.
Based on the data transmission method, the thin client can display experience three-dimensional content with low time delay, high precision and high image quality through wireless streaming.
As a preferred embodiment of the present invention, encoding the rendered file to obtain a to-be-displayed picture, where the to-be-displayed picture is transmitted to the Cloudxr client in a streaming manner, and the method further includes: and transmitting the current frame of the picture to be displayed obtained after the rendering file is coded to the Cloudxr client through a wireless network, wherein the wireless network at least comprises one of the following components: 5G, WI-FI.
As a preferred option in this embodiment, the transmitting the current frame to the Cloudxr client via a wireless network further includes: and transmitting pictures within the Cloudxr client Fov and with Fov + preset adjustable display scale in the process of transmitting pictures to be displayed based on a dynamic Fov transmission mode.
In the specific implementation, Fov rendering data transmission is considered, that is, in order to save more bandwidth resources when the coded picture is transmitted through a wireless network. In the embodiment of the present application, based on the above, a dynamic Fov transmission mode is provided, that is, only the Fov + 10% screen is transmitted in the thin client Fov during the screen transmission process.
As a preferable preference in this embodiment, encoding the rendered file to obtain a to-be-displayed picture, and after the to-be-displayed picture is streamed to the Cloudxr client, the method further includes: and decoding the picture to be displayed at the Cloudxr client and displaying the picture.
As a preference in this embodiment, the video stream is decoded on the basis of h.264 or h.265/HEVC on the picture to be displayed by the Cloudxr client.
In specific implementation, after the client receives the video transmitted to Fov, the H.264 or H.265/HEVC video stream is decoded by calling a local hardware chip of the client to display a picture in front of the thin client, the delay of the whole system is within 60ms, and a user cannot distinguish local rendering or cloud rendering, so that the low-delay, high-precision and high-image-quality display of three-dimensional content of a GPU (graphics processing unit) server under the thin client is realized.
As a preferable preference in this embodiment, the obtaining a rendering file according to a preset three-dimensional program, the operation control information, and the spatial position information includes: calling different interface modules according to different types of files to be rendered, wherein the types of the files to be rendered comprise one of the following types: VR, AR, MR, XR; and obtaining a rendering file by combining the operation control information and the spatial position information through a preset interface provided by an interface module according to a preset three-dimensional program.
Based on the method, a high-utilization-rate bandwidth use mode is provided, and low-delay user experience of an end-to-end is realized.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
According to an embodiment of the present application, there is also provided an apparatus for implementing the above-mentioned data transmission method, as shown in fig. 3, the apparatus for an image processing server includes:
a communication module 301, configured to receive operation control information and spatial location information input through a virtual peripheral control module of the Cloudxr client based on the Cloud XR technology;
the rendering module 302 is configured to obtain a rendering file according to a preset three-dimensional program, the operation control information, and the spatial position information;
the streaming transmission module 303 is configured to obtain a to-be-displayed picture after encoding the rendered file, and stream the to-be-displayed picture to the Cloudxr client.
Wireless VR and AR may be implemented across 5G and Wi-Fi networks based on Cloud XR technology in the communication module 301. Additionally, with NVIDIA GPU virtualization software, CloudXR can achieve full scalability for data centers and edge networks.
Receiving operation control information and spatial location information input through a virtual peripheral control module of a Cloudxr client based on the Cloud XR technology. Namely, the Cloudxr client side carries out standardized processing on the control data through the virtual peripheral control module through the peripheral input devices including but not limited to a handle, a keyboard and a mouse, gesture recognition, voice input and the like, and then transmits the control data to the Cloudxr server side through the established network. That is, the operation control information and the spatial location information input through the virtual peripheral control module of the Cloudxr client are received at the Cloudxr server.
As an alternative embodiment, the transmission is made at the time of use based on the cloudbr SDK accompanying the installer for the server component.
As an optional implementation mode, XR augmented reality content is streamed to an Android or Windows client based on an OpenVR application program. In addition, the XR terminal comprises one or more cross-terminal devices such as a mobile phone, a tablet, a web, a VR head display all-in-one machine and the like.
And obtaining a rendering file in the rendering module 302 according to a preset three-dimensional program, the operation control information and the spatial position information. That is, after receiving the control and spatial location information at the Cloudxr server, the operational information is communicated to the three-dimensional program, with the modules possibly called being different depending on the content/type.
As an optional implementation, if VR content needs to be rendered, the operation information data needs to be communicated with OpenXR runtime (steammr).
As a preference in this embodiment, the preset three-dimensional program is used for communicating with a Directx interface or an OpenGL interface. The preset three-dimensional program needs to communicate with a Directx interface or an OpenGL interface.
And in the streaming transmission module 303, the rendered file needs to be encoded to obtain a picture to be displayed, and the picture to be displayed is transmitted to the Cloudxr client in a streaming manner, so that a data wireless transmission process of audio and video coding, wireless data transmission and thin client audio and video decoding of an image processing server is realized.
The device based on the data transmission can support the thin client to display experience three-dimensional content with low time delay, high precision and high image quality through wireless streaming.
It will be apparent to those skilled in the art that the modules or steps of the present application described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may alternatively be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, or fabricated separately as individual integrated circuit modules, or fabricated as a single integrated circuit module from multiple modules or steps. Thus, the present application is not limited to any specific combination of hardware and software.
In order to better understand the above method flow of data transmission, the following explains the above technical solutions with reference to the preferred embodiments, but the technical solutions of the embodiments of the present invention are not limited.
In the embodiment of the application, the GPU image processing server renders content based on the Cloud XR technology, and the content is wirelessly transmitted to the thin client terminal, so that the technical effects of simplifying the use operation of a user, reducing the cost of terminal equipment and providing the user experience of convenient and fast content with high decoding precision are achieved. The method in the embodiment of the application can support the wireless streaming, so that the thin client can experience three-dimensional content with low time delay, high precision and high image quality. In addition, the application provides a high-utilization-rate bandwidth use mode, and end-to-end low-delay user experience is realized.
As shown in fig. 4, the flowchart of the data transmission method in the embodiment of the present application is schematic, and mainly relates to a server audio and video encoding technology, a wireless data transmission technology, a thin client audio and video decoding technology, and an interaction technology, and the specific process of implementation includes the following steps:
and S1, starting the three-dimensional program and the Cloudxr server end program as a streaming server end by the GPU image processing server end.
S2, the user starts the Cloudxr client program as the streaming client and establishes network connection with the server.
The specific connection process is as follows:
s21 the server opens the monitor port to monitor the connection request from the client;
s22 client end sends a TCP packet applying for connection to the monitoring port of server end;
s23, after the server monitors that the port receives the TCP packet of the connection request of the client, the server analyzes the request connection packet to obtain the IP address and the port number of the client;
s24 server opens a port, uses TCP protocol to connect the IP and port of the client;
after the connection is completed in S25, the server and the client will ensure whether the connection between the end and the end is normal by sending heartbeat packets regularly. After the heartbeat packet is over time, the connection between the end and the end is considered to be disconnected, so that the client side can resend the connection request, and the process is repeated until the server side is reconnected.
S3, in the Cloudxr rendering process, the thin client includes a virtual peripheral control and a sky ball composition, the thin client carries out standardized processing on the control data through a virtual peripheral control module through a self-contained input peripheral such as a handle, a keyboard and a mouse, gesture recognition, voice input and the like, and then transmits the data to the Cloudxrserver end through an established network. I.e. to perform the input of scene control data.
Then, after receiving the control and spatial location information, the Cloudxrserver end communicates the operation information with the three-dimensional program, and the modules may be called differently according to the content. Such as VR content, the operation information data needs to be communicated with OpenXR runtime.
Finally, three-dimensional programs, in general, need to communicate with Directx or OpenGL. And after receiving the control information, the three-dimensional program renders a picture according to content logic, sends the rendered next frame of picture to a video memory, calls the existing picture by the Cloudxrerver program, performs picture coding by using H.264, H.265/HEVC and sends the coded data to the thin client.
S4, Fov rendering data are transmitted, and when the coded pictures are transmitted through a wireless network, in order to save more bandwidth resources, the invention provides a dynamic Fov transmission mode, namely, only the thin client Fov and Fov + 10% pictures are transmitted in the picture transmission process.
S5, after the client receives the video transmitted to Fov, the client locally calls a hardware chip to decode H.264 or H.265/HEVC video stream to display a picture in front of the thin client, the delay of the whole system is within 60ms, and a user cannot distinguish local rendering or cloud rendering, so that the low-delay, high-precision and high-image-quality three-dimensional content of the GPU server is displayed on the thin client.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects.
The embodiments of this specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The described embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment. In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of an embodiment of the specification. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
The above description is only an example of the present specification, and is not intended to limit the present specification. Various modifications and variations to the embodiments described herein will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present specification should be included in the scope of the claims of the embodiments of the present specification.

Claims (10)

1. A method for transmitting 5G or wireless Cloud XR data is characterized by comprising the following steps:
receiving operation control information and spatial position information input through a virtual peripheral control module of a Cloudxr client based on a Cloud XR technology;
obtaining a rendering file according to a preset three-dimensional program, the operation control information and the space position information;
and encoding the rendering file to obtain a picture to be displayed, and streaming the picture to be displayed to the Cloudxr client.
2. The method according to claim 1, wherein the picture to be displayed is obtained after the render file is encoded, and the picture to be displayed is streamed to the Cloudxr client, further comprising:
and transmitting the current frame of the picture to be displayed obtained after the rendering file is coded to the Cloudxr client through a wireless network, wherein the wireless network at least comprises one of the following components: 5G, WI-FI.
3. The method of claim 2, wherein transmitting the current frame to the Cloudxr client over a wireless network further comprises:
and transmitting pictures within the Cloudxr client Fov and with Fov + preset adjustable display scale in the process of transmitting pictures to be displayed based on a dynamic Fov transmission mode.
4. The method according to claim 1, wherein encoding the rendered file to obtain a picture to be displayed, and after streaming the picture to be displayed to the Cloudxr client, the method further comprises:
and decoding the picture to be displayed at the Cloudxr client and displaying the picture.
5. The method of claim 4, wherein:
and decoding the video stream on the basis of H.264 or H.265/HEVC for the picture to be displayed at the Cloudxr client.
6. The method according to claim 1, wherein obtaining a rendering file according to a preset three-dimensional program, the operation control information, and the spatial location information comprises:
calling different interface modules according to different types of files to be rendered, wherein the types of the files to be rendered comprise one of the following types: VR, AR, MR, XR;
and obtaining a rendering file by combining the operation control information and the spatial position information through a preset interface provided by an interface module according to a preset three-dimensional program.
7. The method of claim 6, wherein the predetermined three-dimensional program is configured to communicate with a Directx interface or an OpenGL interface.
8. A data transmission apparatus for an image processing server, the apparatus comprising:
the communication module is used for receiving operation control information and spatial position information input by a virtual peripheral control module of the Cloudxr client based on the Cloud XR technology;
the rendering module is used for obtaining a rendering file according to a preset three-dimensional program, the operation control information and the space position information;
and the streaming transmission module is used for coding the rendering file to obtain a picture to be displayed, and the picture to be displayed is transmitted to the Cloudxr client in a streaming mode.
9. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the method of any of claims 1 to 7 when executed.
10. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 7.
CN202110905918.0A 2021-08-06 2021-08-06 Wireless-based Cloud XR data transmission method and device, storage medium and electronic device Active CN113473180B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110905918.0A CN113473180B (en) 2021-08-06 2021-08-06 Wireless-based Cloud XR data transmission method and device, storage medium and electronic device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110905918.0A CN113473180B (en) 2021-08-06 2021-08-06 Wireless-based Cloud XR data transmission method and device, storage medium and electronic device

Publications (2)

Publication Number Publication Date
CN113473180A true CN113473180A (en) 2021-10-01
CN113473180B CN113473180B (en) 2024-01-30

Family

ID=77867670

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110905918.0A Active CN113473180B (en) 2021-08-06 2021-08-06 Wireless-based Cloud XR data transmission method and device, storage medium and electronic device

Country Status (1)

Country Link
CN (1) CN113473180B (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019126293A1 (en) * 2017-12-22 2019-06-27 Magic Leap, Inc. Methods and system for generating and displaying 3d videos in a virtual, augmented, or mixed reality environment
CN111179437A (en) * 2019-12-30 2020-05-19 上海曼恒数字技术股份有限公司 Cloud VR connectionless streaming system and connection method

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019126293A1 (en) * 2017-12-22 2019-06-27 Magic Leap, Inc. Methods and system for generating and displaying 3d videos in a virtual, augmented, or mixed reality environment
CN111179437A (en) * 2019-12-30 2020-05-19 上海曼恒数字技术股份有限公司 Cloud VR connectionless streaming system and connection method

Also Published As

Publication number Publication date
CN113473180B (en) 2024-01-30

Similar Documents

Publication Publication Date Title
US11303881B2 (en) Method and client for playing back panoramic video
CN111882626B (en) Image processing method, device, server and medium
CN113457160B (en) Data processing method, device, electronic equipment and computer readable storage medium
WO2022257699A1 (en) Image picture display method and apparatus, device, storage medium and program product
CN111803940A (en) Game processing method and device, electronic equipment and computer-readable storage medium
JPWO2014020641A1 (en) Data providing system, providing device, execution device, control method, program, and recording medium
CN110290398B (en) Video issuing method and device, storage medium and electronic equipment
CN114666225B (en) Bandwidth adjustment method, data transmission method, device and computer storage medium
CN110856005B (en) Live stream display method and device, electronic equipment and readable storage medium
CN111447504B (en) Three-dimensional video processing method and device, readable storage medium and electronic equipment
CN113986177A (en) Screen projection method, screen projection device, storage medium and electronic equipment
CN111464828A (en) Virtual special effect display method, device, terminal and storage medium
CN113973224B (en) Media information transmission method, computing device and storage medium
CN114554277B (en) Multimedia processing method, device, server and computer readable storage medium
CN114268779A (en) Image data processing method, device, equipment and computer readable storage medium
CN111381787A (en) Screen projection method and equipment
WO2022089713A1 (en) Apparatus, method, and computer program for providing service level for extended reality application
CN114268626A (en) Window processing system, method and device
CN113473180B (en) Wireless-based Cloud XR data transmission method and device, storage medium and electronic device
US20220239920A1 (en) Video processing method, related apparatus, storage medium, and program product
CN116248889A (en) Image encoding and decoding method and device and electronic equipment
CN116264619A (en) Resource processing method, device, server, terminal, system and storage medium
CN116546262A (en) Data processing method, device, equipment and storage medium
CN113596583A (en) Video stream bullet time data processing method and device
CN115134647A (en) Video streaming transmission method based on cloud service and related equipment thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant