CN116489410A - Method for processing cloud VR video stream, cloud-edge collaboration system and storage medium - Google Patents

Method for processing cloud VR video stream, cloud-edge collaboration system and storage medium Download PDF

Info

Publication number
CN116489410A
CN116489410A CN202310539806.7A CN202310539806A CN116489410A CN 116489410 A CN116489410 A CN 116489410A CN 202310539806 A CN202310539806 A CN 202310539806A CN 116489410 A CN116489410 A CN 116489410A
Authority
CN
China
Prior art keywords
decompression
task
video
edge server
cloud server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310539806.7A
Other languages
Chinese (zh)
Inventor
张宗帅
邹文浩
田霖
黄家莹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Institute of Computing Technology of CAS
Original Assignee
Institute of Computing Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Computing Technology of CAS filed Critical Institute of Computing Technology of CAS
Priority to CN202310539806.7A priority Critical patent/CN116489410A/en
Publication of CN116489410A publication Critical patent/CN116489410A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2181—Source of audio or video content, e.g. local disk arrays comprising remotely distributed storage units, e.g. when movies are replicated over a plurality of video servers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H04N19/436—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation using parallelised computational arrangements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
    • H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention provides a method for processing cloud VR video streams, an edge cloud cooperative system and a storage medium, wherein the method comprises the following steps: the cloud server receives a video acquisition task uploaded by the edge server, acquires corresponding video resources according to the video acquisition task, and obtains a decompression task; the cloud server determines decompression proportions divided to the edge server and the cloud server according to the cloud server and the computing resources distributed to the decompression tasks by the edge server, and divides the decompression tasks; and the cloud server and the edge server decompress the decompression task according to the divided decompression ratio respectively, and send decompression results to the VR terminal. The invention effectively reduces the transmission delay of the video and improves the user experience while ensuring the image quality.

Description

Method for processing cloud VR video stream, cloud-edge collaboration system and storage medium
Technical Field
The invention relates to the technical field of mobile streaming media, in particular to a method for processing cloud VR video streams, an edge cloud collaboration system and a storage medium.
Background
With the advent of the 5G era, VR (Virtual Reality) industry is further developed by virtue of the characteristics of large bandwidth, low delay and wide connection of the 5G, and scenes such as VR live broadcast, VR social interaction, virtual concert and the like are developed again.
Compared with the common 3D application, the VR application has higher requirement on the mixed rendering capability of the device, and the VR application fuses pictures, videos, 3D digital models, dynamic simulation effects and the like. As shown in fig. 1, VR video encoding and VR video transmission are key links in the transmission process of VR video streams. Video coding requires high compression rate of the image, not only to remove temporal, spatial, coding and visual redundancy, but also low distortion rate, i.e. the decoded image is as consistent as possible with the original image in all aspects. Video transmission requires low transmission delay, short time, low packet loss rate, and higher video frame rate, i.e. requires more information to be successfully transmitted. The current VR use in various industries relies on powerful computing hardware or other external devices (e.g., body tracking, hand tracking, head-mounted devices) that cannot be used for long periods of time due to heavy devices. In addition, when performing computing tasks such as ultra-high definition image rendering, the computing power of the VR device terminal is obviously insufficient, for example, the computing power of most of the integrated VR head displays is only one twentieth of that of a desktop GPU (graphics processing unit, graphics processor), and it is difficult to complete the task within a limited time. In a typical video rendering and transmission method, a VR terminal is connected to a high-performance PC (Personal Computer, computer) and a picture is displayed through the VR terminal in a streaming manner, and a wired connection between the VR terminal and the computer often limits a movement range of a user.
Aiming at the problems, the problems caused by the dependence on hardware computing power are generally reduced in a cloud rendering mode based on a cloud computing technology at present. The VR terminal uploads the demands of resources and computing tasks to the cloud server, the cloud server accelerates the speed of computing tasks such as image rendering and the like by means of strong computing capacity and storage space, and then the video stream is returned to the VR terminal. The user does not need to perform local rendering by the high-end PC any more, and the VR terminal device directly downloads the rendered video image from the cloud server; and uploading the interaction information of the user to the cloud end, so that the user can move in a relatively wide range. The method can effectively reduce the cost of hardware equipment, improve the image quality and the visual quality, enlarge the moving range of a user and effectively improve the user experience. However, the cloud server is usually far away from the user, the data size of the ultra-high definition and high frame rate video image is large, which often causes higher delay, and VR video smoothness directly affects the experience of the user, so how to reduce the video image transmission delay while improving the quality of the transmitted image is an urgent problem to be solved.
Disclosure of Invention
In order to solve the problems, the invention provides a method for processing a cloud VR video stream, an edge cloud cooperative system and a storage medium, which can disassemble a decompression task, and the task is decompressed by a cloud server and an edge server according to a proportion respectively, and a VR terminal is only responsible for user interaction and image display, so that the effects of effectively reducing the transmission delay of the video and improving the user experience while ensuring the image quality are realized.
According to a first aspect of the present invention, there is provided a processing method for a cloud VR video stream including edge cloud collaboration of a cloud server and an edge server, including: the cloud server receives a video acquisition task uploaded by the edge server, acquires corresponding video resources according to the video acquisition task, and obtains a decompression task; the cloud server determines decompression proportions divided to the edge server and the cloud server according to the cloud server and the computing resources distributed to the decompression tasks by the edge server, and divides the decompression tasks; and the cloud server and the edge server decompress the decompression task according to the divided decompression ratio respectively, and send decompression results to the VR terminal.
Preferably, the determining the decompression ratio divided into the edge server and the cloud server according to the computing resources allocated to the decompression task by the edge server and the edge server includes: determining the optimal computing resources allocated to the decompression task by an edge server and the optimal computing resources allocated to the decompression task by a cloud server according to the data size of the decompression task, the priority of the decompression task, the number of CPU cycles of the unit data of the decompression task and the cooperative transmission communication capacity; determining the decompression proportion of the edge server according to the CPU cycle number, the cooperative transmission communication capacity, the optimal computing resource allocated to the decompression task by the edge server and the optimal computing resource allocated to the decompression task by the cloud server; and obtaining the decompression proportion of the cloud server according to the decompression proportion of the edge server.
Preferably, the determining, according to the data size of the decompression task, the priority of the decompression task, the number of CPU cycles of the unit data for decompressing the decompression task, and the cooperative transmission communication capacity, the optimal computing resource allocated to the decompression task by the edge server and the optimal computing resource allocated to the decompression task by the cloud server includes: constructing a constraint optimization problem relation according to the data size of the decompression task, the priority of the decompression task, the number of CPU cycles of the unit data of the decompression task and the cooperative transmission communication capacity; and solving the relation by using a Lagrangian multiplier method to obtain the optimal computing resource allocated to the decompression task by the edge server and the optimal computing resource allocated to the decompression task by the cloud server.
Preferably, the obtaining a decompression task according to the video obtaining task to obtain a corresponding video resource includes: searching the corresponding video resource in a local database of a cloud server according to the video acquisition task: if the video resource can be searched, using the searched video resource as the decompression task; if the search result is not found or the search result is required to be rendered, rendering processing is carried out according to the task information in the video acquisition task, so that corresponding video rendering resources are obtained, and the video rendering resources are compressed to obtain a decompression task.
Preferably, the compressing the video resource as a decompression task includes: and compressing the video rendering resource according to the H.265 standard to obtain a decompression task.
Preferably, the cloud server and the edge server decompress the decompression task according to the divided decompression ratio, and send a decompression result to the VR terminal, including: the cloud server performs partial decompression on the decompression task according to the decompression ratio of the cloud server to obtain the decompression task with partial decompression completed; the cloud server adopts a network transmission KCP protocol to send the decompression task for completing partial decompression to the edge server; the edge server performs all decompression on the decompression tasks which finish partial decompression, and decompressed video resources are obtained; and the edge server adopts a network transmission KCP protocol to send the decompressed video resource to the VR terminal.
Preferably, before the cloud server receives the video acquisition task uploaded by the edge server, the method includes: the edge server receives an instruction task and judges whether the instruction task is a task for requesting to acquire video image resources or not: if not, processing the instruction task and returning a processing result; if yes, searching in a local database of the edge server: if the video image resources can be searched, all the searched video image resources are decompressed, and all the decompressed video image resources are returned to the VR terminal; and if the video is not searched, taking the instruction task as a video acquisition task, and sending the video acquisition task to the cloud server.
Preferably, before the edge server receives the instruction task, the method includes: the VR terminal receives interaction information of a user; and converting the interaction information into an instruction task and sending the instruction task to the edge server.
According to a second aspect of the present invention, there is provided an edge cloud collaboration system adapted to a cloud VR video stream processing method, including: cloud server, edge server and VR terminal; the cloud server is used for receiving a video acquisition task uploaded by the edge server, acquiring a corresponding video resource according to the video acquisition task, and obtaining a decompression task; according to the computing resources allocated to the decompression tasks by the edge server and the edge server, determining the decompression proportion divided to the edge server and the cloud server, and dividing the decompression tasks; partial decompression is carried out according to the decompression proportion of the cloud server; the edge server is used for completely decompressing the decompressed tasks after partial decompression to obtain a decompressed result; and the VR terminal user is used for displaying the decompression result.
According to a third aspect of the present invention there is provided a storage medium having stored therein computer executable instructions which when loaded and executed by a processor implement the steps of the method according to the first aspect of the present invention.
Compared with the prior art, the invention has the advantages that:
in the invention, the cloud server and the edge server connected with the cloud server are cooperated to understand the compression task, the cloud server disassembles the decompression task, the cloud server decompresses one part of the decompression task, and the edge server decompresses the other part of the decompression task. The VR terminal is only responsible for user interaction and image display, and the edge server cooperates with the cloud server to complete all decompression tasks, rather than being independently processed by the cloud server or the edge server in the prior art. The invention effectively reduces the transmission delay of the video and improves the user experience while ensuring the image quality.
Drawings
Fig. 1 is a schematic flow chart of basic steps required in a VR video stream transmission process.
Fig. 2 is a schematic diagram of a processing and transmitting method of a cloud VR video stream in the prior art.
Fig. 3 is a cloud-edge collaborative architecture diagram according to an embodiment of the present invention.
Fig. 4 is a flowchart of a cloud VR video stream processing method according to an embodiment of the present invention.
Detailed Description
Specific embodiments of the invention will be described in detail below, it being noted that the examples herein are for illustrative purposes only and are not intended to limit the invention. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that: no such specific details are necessary to practice the invention. In other instances, well-known procedures, materials, or methods have not been described in detail so as not to obscure the present invention.
In order to achieve processing and transmission of ultra-high definition, high frame rate, low latency cloud VR video streams, the prior art generally includes the following two types of methods, and the applicable architecture is shown in fig. 2: the method is a calculation unloading technology of mobile cloud calculation, high quality of video images is guaranteed by means of high calculation power and large storage of the cloud calculation, program division is performed on the basis of an offline heuristic algorithm, an approximation algorithm of polynomial time, a Markov decision method and the like so as to reduce average delay of tasks, but defects that the cloud calculation is far away from a user still exist, and delay cannot be sufficiently reduced; the other type is a calculation unloading technology of mobile edge calculation, which exerts the characteristic that the edge calculation is closer to a user, the user does not need to send a request to a cloud end, but sends the request to an edge server, and the edge server directly returns the request to the user after processing, for example, the weighting and delay of all devices can be minimized through an optimal closed data segmentation strategy, but for processing ultra-high definition and high frame rate video images, the computing power of the edge server cannot realize the high quality of the images and the shortening of the rendering time at the same time. Therefore, the existing cloud VR video stream processing method is difficult to simultaneously exert the advantages of the cloud server and the edge server and give consideration to time delay and image quality, so that the actual experience of a user is not ideal.
Fig. 3 illustrates a cloud-edge collaborative architecture diagram according to an embodiment of the present invention, as shown in fig. 3, where the embodiment is mainly applied to cloud VR video streaming with edge-cloud collaboration, and a cloud server (i.e., a central cloud) is used for performing edge management, video rendering, compression and decompression, request processing, and so on. The edge management is to cooperatively process edge servers (namely edge clouds) connected in a range; video rendering is to perform video rendering according to the request task; the compression and decompression is to compress and partially decompress the video image according to the compression and decompression standard; request processing is to translate requests from edge servers into rendering tasks or resource searches. The edge server mainly performs decoding decompression, potential state sensing and user interaction information processing, wherein the decoding decompression is used for processing video images sent by the cloud server or existing locally; the potential state perception is to collect and process network routes of VR terminals and the like; the user interaction information processing is to convert the interaction information of the user into instruction processing or image rendering tasks. The VR terminal is mainly used for receiving interaction information of a user, converting the interaction information into an instruction task and sending the instruction task; and the method is also used for displaying the decompressed video image resources after receiving the decompressed video image resources.
Fig. 4 is a flowchart of a processing method of a cloud VR video stream according to an embodiment of the present invention, and the processing method of the cloud VR video stream according to the embodiment of the present invention is described in detail below with reference to fig. 3 and 4.
In one embodiment of the present invention, the VR terminal obtains the interaction information according to the operation of the user, converts the interaction information into a task set (i.e., an instruction task set) of a request instruction or an operation instruction, and uploads the instruction task set to the edge server. After receiving the instruction task set, the edge server maps the instruction tasks in sequence according to the priority of the instruction tasks in the instruction task set and the preset mapping relation. Preferably, the mapping result of the command task mapped according to the preset mapping relation includes but is not limited to an authorized task, a state change task and a task for obtaining video image resources. If the mapping result is the task type of a small amount of information exchange such as authorization, state change and the like, processing the instruction task and returning a processing result to the VR terminal; if the video image resources are acquired, searching is carried out in a local database of the edge server, and if the video image resources can be searched, all the searched video image resources are decompressed and then sent to the VR terminal; if the task cannot be searched, the instruction task is put into a new set (namely a video acquisition task set), and after the instruction task set is completely traversed, the video acquisition task set is uploaded to a cloud server. After receiving the video acquisition task set, the cloud server processes each video acquisition task in the video acquisition task set: searching video resources (including images, videos and the like) required by the video acquisition task in a local database of the cloud server, and if the video resources can be searched, putting the searched video resources into a decompression task set as a decompression task. If the searched resource cannot be searched or the searched resource still needs to be rendered, rendering and compressing are carried out according to the specific information of the task to obtain the video resource, and then the video resource is used as a decompression task and is put into a decompression task set. Wherein, the rendering technology is a prior art known in the art, and the embodiments of the present invention are not described herein. (step S401)
In some embodiments, the process of compressing rendered resources according to the h.265 standard is as follows: including intra-frame coding and inter-frame coding. Intra-frame coding: dividing the rendered video image resources into block areas to obtain block original image frames, carrying out intra-frame estimation and intra-frame coding by using an encoder, and then carrying out intra-frame prediction to obtain an intra-frame prediction result; subtracting the intra-frame prediction result from the original image frame to obtain an intra-frame residual signal, performing linear transformation, coefficient scaling and quantization on the intra-frame residual signal, performing entropy coding with the intra-frame coding signal, and outputting the entropy coding signal. Inter-frame coding: the method comprises the steps of inputting a block original image frame into an encoder, carrying out inter-frame coding by combining an upper image frame and a lower image frame in a prediction result, carrying out inter-frame prediction on inter-frame coded signals to obtain an inter-frame prediction result, subtracting the inter-frame prediction result from the block original image frame to obtain inter-frame residual signals, carrying out linear transformation, coefficient scaling and quantization on the inter-frame residual signals, carrying out entropy coding together with the inter-frame coded signals, and outputting the inter-frame residual signals.
According to one embodiment of the invention, for each decompression task, the cloud server needs to disassemble the decompression task, and the cloud server decompresses one part of the decompression task first and then decompresses the other part of the decompression task. Therefore, the cloud server needs to determine the optimal computing resources allocated to the decompression task by the cloud server and the edge server according to the computing resources of the current cloud server and the edge server which are in butt joint with the cloud server, the current network environment and the data size of the decompression task, and the decompression proportion required to be completed by the edge server for the decompression task and the decompression proportion required to be completed by the cloud server for the decompression task are obtained. And the cloud server and the edge server decompress the decompression task according to the decompression proportion. (step S402)
Preferably, the edge server and cloud server are assigned to the decompression task r i Should minimize the decompression delay to accomplish the decompression task. The larger the data of the decompression task is, the stronger the calculation power of the cloud server and the edge server connected with the cloud server is, the more corresponding allocated calculation resources are needed, otherwise, the calculation resources can be reduced; if the current network environment is poor, the computing resource allocation proportion of the edge server can be increased so as to reduce the decompression proportion of the cloud server and the data transmission time from the cloud server to the edge server.
In some embodiments, an edge server is assigned to the decompression task r i Is allocated to the decompression task r with a cloud server i Can be calculated according to the equation (1) (2):
wherein, the liquid crystal display device comprises a liquid crystal display device,representing an ith decompression task for a jth edge server connected to the cloud server, the jth edge server allocating optimal computing resources for the decompression task; />Representing an ith decompression task for a jth edge server, the cloud server allocating optimal computing resources for the decompression task; i=1, 2, …, I j ，I j The number of decompression tasks of the jth edge server is j=1, 2, …, J being the number of edge servers; alpha j,i Is a weight factor, representing the firstThe priority of the ith decompression task of the j edge servers is in the range of 0-1 in the embodiment of the invention, and the value range of the ith decompression task can be freely set in some embodiments; d (D) j,i The data size of the ith decompression task of the jth edge server (it should be noted that, here, the data size of the decompression task before decompression); t (T) j,i Representing the CPU cycle number of the 1bit data of the ith decompression task of the jth edge server; w (W) j Representing the capacity of the cloud server to transmit communication to the jth edge server, namely the reciprocal of the time required for the backhaul link to transmit 1bit of data; />And->The best value of the lagrangian multiplier that satisfies the computational resource constraint using the lagrangian multiplier method is represented. Here, the ith decompression task is processed by the cloud server according to the command task uploaded by the jth edge server, that is, the cloud server and the edge server cooperate to decompress the decompression task when the decompression task is from which edge server.
Preferably, the computational constraint of formula (1) (2) is formula (3) (4), i.e. the sum of the best computational resources required to meet the allocation to all decompression tasks is equal to the available computational resources of the current cloud server and the edge servers connected thereto:
wherein, the liquid crystal display device comprises a liquid crystal display device,representing availability of a jth edge server connected to a cloud serverComputing resources, C c Representing the available computing resources of the current cloud server.
To solve the formula (1) (2)And->The KKT (Karush-Kuhn-Tucker, carlo Rev-Coulomb-Take conditions) condition is first used to solve +.>And->The following is a preferred solving step:
under the condition that constraint condition formulas (3) and (4) are met, the time for the decompression tasks in the decompression task set to complete all decompression according to the priority is minimized, and the constraint optimization problem is constructed according to specific formulas (5) to (7):
wherein, the liquid crystal display device comprises a liquid crystal display device,representing the ith decompression task coordinated for the jth edge server, the time it takes to complete the full decompression is specifically equation (8):
wherein, the liquid crystal display device comprises a liquid crystal display device,representing an ith decompression task for a jth edge server, computing resources allocated to the decompression task by the jth edge server; />Representing an ith decompression task for the jth edge server, the cloud server allocates computing resources for the decompression task.
By Lagrangian multiplier pairsAnd->And (3) carrying out solving:
defining a Lagrangian function:
wherein, gamma i Andrepresenting the lagrangian multiplier.
Solving a system of equations:
solving for gamma obtained by (10) i 、Namely +.>Substituting (1) and (2) and solving to obtain +.>And->
After the optimal computing resources allocated to the decompression tasks by the cloud server and the edge servers are determined, the cloud server starts to disassemble the ith decompression task of the jth edge server, and the decompression proportion of the cloud server and the edge servers to the decompression tasks is planned. Along withThe larger (i.e., the more computing resources that the edge server can allocate), or W j The smaller (i.e., the slower the transmission speed), the greater the decompression rate allocated to the edge server, and the smaller the decompression rate allocated to the cloud server.
In some embodiments, the resulting product may be usedAnd->Substituting into the formula (11), calculating the ith decompression task aiming at the jth edge server, and the decompression proportion completed by the jth edge server>
As can be seen from the formula (11), for the ith decompression task of the jth edge server, the cloud server completes the decompression ratio of
In some embodiments, the process of decompressing data (i.e., the decoding process) is as follows: including intra-frame decoding and inter-frame decoding. Intra-frame decoding: performing inverse entropy coding on the received bit stream to obtain a decoding coefficient and a decoding flag bit, performing inverse quantization on the decoding coefficient to obtain an inverse quantized coefficient, performing loop filtering on the inverse quantized pixel to remove blocking effect and artifacts, performing adaptive compensation on the filtered pixel to further reduce artifacts and noise, performing inverse transformation on the inverse quantized coefficient to obtain a decoded residual signal, performing intra-frame prediction and reconstruction, namely performing intra-frame prediction on the decoded residual signal according to an intra-frame prediction mode used during coding to obtain a prediction result, and adding the prediction result with the decoded residual signal to obtain a result. Inter-frame decoding: performing inverse entropy coding on the received bit stream to obtain a decoding coefficient and a decoding flag bit, performing inverse quantization on the decoding coefficient to obtain an inverse quantized coefficient, performing loop filtering on the inverse quantized pixel to remove blocking effect and artifacts, performing adaptive compensation on the filtered pixel to further reduce artifacts and noise, performing inverse transformation on the inverse quantized coefficient to obtain a decoded residual signal, and performing inter-frame prediction and reconstruction, namely performing inter-frame prediction on the decoded residual signal according to an inter-frame prediction mode used during coding to obtain a prediction result, and adding the prediction result with the decoded residual signal to obtain a result.
The cloud server calculates the decompression ratio according to the decompression ratioAfter partial decompression of the decompression tasks in the decompression task set is completed, the partial decompressed decompression task data is sent to the corresponding edge server through KCP, and the method comprises the steps of first data sending, receiving, reading and confirming, and subsequent data sending, receiving, reading and confirming.
In some embodiments, the process of the cloud server sending the decompressed task data of the partial decompression to the corresponding edge server through the KCP is as follows: 1) The method comprises the steps of firstly sending data, reading the data by a sending end (cloud server), dividing the data into packets, firstly sending the packets to move the data packets at the head of a sending queue into a sending buffer, then defining the data instruction type, pushing the data and sending the data packets. 2) After receiving data for the first time, a receiving end (edge server) analyzes a packet header and checks validity, records a sequence number sn and a time stamp ts of the data packet, adds the data packet into a receiving buffer, checks whether the first data packet is available, and if so, moves to a receiving queue to send a confirmation data packet, wherein the data packet comprises confirmation information, a next data sequence number expected to be received and a receiving window size. 3) The method comprises the steps that data confirmation is carried out for the first time, a receiving end (a cloud server) receives a confirmation packet, firstly, UNA confirmation is carried out, all packets with confirmation numbers smaller than the value of UNA are received (UNA represents the next receivable serial number, which indicates that all messages before the serial number are received, for example, UNA is 11, which indicates that sn=10 and the previous data packet are received); an ACK acknowledgement is then made, acknowledging that the data packet with the value ACK has been received. 4) And (5) subsequent data transmission, wherein a transmitting end (cloud server) updates a congestion window, and continuously transmits single or multiple data packets according to the size of the congestion window. 5) And (3) subsequent data receiving, wherein the receiving end (edge server) stores the received data packets into a receiving buffer queue, confirms that the received data packets are available data packets, stores the data packets into the receiving queue, updates the next data sequence number expected to be received and the size of a receiving window, and adds confirmation data to be sent back to the transmitting end as a confirmation packet. 6) And (3) subsequent data confirmation, wherein the receiving end (cloud server) receives the confirmation packet, performs UNA confirmation, and confirms that all packets before the number of UNA are received, and then performs ACK confirmation, and confirms that the data packet with the number of ACK is received.
The edge server receives the data and then performs all decompression, and the specific decompression process is described in the previous embodiment, which is not described here again. After all decompression is completed, the edge server sends all data obtained after decompression to the VR terminal, and the terminal directly displays images. (step S403)
In some embodiments, the process of the edge server sending all decompressed data to the VR terminal through KCP is as follows: 1) The method comprises the steps of firstly sending data, reading the data by a sending end (an edge server), dividing the data into packets, firstly sending the packets to move the data packets at the head of a sending queue into a sending buffer, then defining the data instruction type, pushing the data and sending the data packets. 2) After receiving data for the first time, a receiving end (VR terminal) receives the data, performs packet header analysis and validity check, records the sequence number sn and the time stamp ts of the data packet, adds the data packet into a receiving buffer, then checks whether the first data packet is available, if so, moves to a receiving queue, and sends a confirmation data packet, wherein the data packet comprises confirmation information, the next data sequence number expected to be received and the size of a receiving window. 3) The first time of data confirmation, the receiving end (edge server) receives the confirmation packet, carries out the UNA confirmation, and all packets with confirmation numbers smaller than the value of UNA are received; an ACK acknowledgement is then made, acknowledging that the data packet with the value ACK has been received. 4) And (4) subsequent data transmission, wherein a transmitting end (edge server) updates a congestion window, and single or multiple data packets are continuously transmitted according to the size of the congestion window. 5) And (3) subsequent data receiving, wherein the receiving end (VR terminal) stores the received data packets into a receiving buffer queue, confirms that the received data packets are available data packets, stores the data packets into the receiving queue, updates the next data sequence number expected to be received and the size of a receiving window, and adds confirmation data to be sent back to the transmitting end as a confirmation packet. 6) Subsequent data confirmation, the receiving end (edge server) receives the confirmation packet, firstly performs UNA confirmation, all packets before the confirmation number of UNA are received, then performs ACK confirmation, and confirms that the data packet with the confirmation number of ACK is received.
In summary, according to the cloud VR video stream processing method based on the edge cloud cooperation, the division of the cloud server and the edge server can be refined through the edge cloud cooperation, and the adjustment is performed according to different characteristics and the current situation, so that the video transmission speed is further improved, and on the premise that the image quality is not reduced, the delay is reduced by about 23% compared with the cloud VR method. The VR terminal is only responsible for user interaction and image display, and the edge server cooperates with the cloud server to complete all decompression tasks, so that the video transmission delay can be effectively reduced while the image quality is ensured.
According to an embodiment of the present invention, there is also provided an edge cloud collaboration system suitable for the above cloud VR video stream processing method, including: cloud server, edge server and VR terminal; the cloud server is used for receiving the video acquisition task uploaded by the edge server, acquiring corresponding video resources according to the video acquisition task, and obtaining a decompression task; according to the computing resources distributed to the decompression tasks by the edge server and the edge server, determining the decompression proportion divided to the edge server and the cloud server, and dividing the decompression tasks; partial decompression is carried out according to the decompression proportion of the cloud server; the edge server is used for fully decompressing the partial decompressed tasks to obtain a decompressed result and transmitting the decompressed result to the VR terminal user; and the VR terminal is used for displaying the decompression result.
The embodiments of the present disclosure are described in a progressive manner, and each embodiment illustrates a difference from other embodiments or implementations, so that the same or similar parts between the embodiments of the present disclosure may be referred to each other, and the implementation principles and the technical effects around the inventive concept may be referred to each other, which is not repeated herein. The various embodiments or implementations of the invention may be combined with one another without conflict.
It should be noted that, although the steps are described above in a specific order, it is not meant to necessarily be performed in the specific order, and in fact, some of the steps may be performed concurrently or even in a changed order, as long as the required functions are achieved.
The present invention may be a system, method, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present invention.
The computer readable storage medium may be a tangible device that retains and stores instructions for use by an instruction execution device. The computer readable storage medium may include, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing.
The foregoing description of embodiments of the invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the technical improvements in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (10)

1. A method of processing a cloud VR video stream for edge cloud collaboration including a cloud server and an edge server, comprising:
the cloud server receives a video acquisition task uploaded by the edge server, acquires corresponding video resources according to the video acquisition task, and obtains a decompression task;
the cloud server determines decompression proportions divided to the edge server and the cloud server according to the cloud server and the computing resources distributed to the decompression tasks by the edge server, and divides the decompression tasks;
and the cloud server and the edge server decompress the decompression task according to the divided decompression ratio respectively, and send decompression results to the VR terminal.
2. The method of claim 1, wherein determining the decompression ratio divided into the edge server and the cloud server based on the computing resources allocated to the decompression task by the edge server and itself comprises:
determining the optimal computing resources allocated to the decompression task by an edge server and the optimal computing resources allocated to the decompression task by a cloud server according to the data size of the decompression task, the priority of the decompression task, the number of CPU cycles of the unit data of the decompression task and the cooperative transmission communication capacity;
determining the decompression proportion of the edge server according to the CPU cycle number, the cooperative transmission communication capacity, the optimal computing resource allocated to the decompression task by the edge server and the optimal computing resource allocated to the decompression task by the cloud server;
and obtaining the decompression proportion of the cloud server according to the decompression proportion of the edge server.
3. The method according to claim 2, wherein determining the best computing resources allocated to the decompression task by the edge server and the best computing resources allocated to the decompression task by the cloud server according to the data size of the decompression task, the priority of the decompression task, the number of CPU cycles per unit data for decompressing the decompression task, and the cooperative transmission communication capacity, comprises:
constructing a constraint optimization problem relation according to the data size of the decompression task, the priority of the decompression task, the number of CPU cycles of the unit data of the decompression task and the cooperative transmission communication capacity;
and solving the relation by using a Lagrangian multiplier method to obtain the optimal computing resource allocated to the decompression task by the edge server and the optimal computing resource allocated to the decompression task by the cloud server.
4. The method according to claim 1, wherein the obtaining a decompression task according to the video obtaining task obtaining a corresponding video resource includes:
searching the corresponding video resource in a local database of a cloud server according to the video acquisition task:
if the video resource can be searched, using the searched video resource as the decompression task;
if the search result is not found or the search result is required to be rendered, rendering processing is carried out according to the task information in the video acquisition task, so that corresponding video rendering resources are obtained, and the video rendering resources are compressed to obtain a decompression task.
5. The method of claim 4, wherein said compressing said video asset as a decompression task comprises:
and compressing the video rendering resource according to the H.265 standard to obtain a decompression task.
6. The method of claim 1, wherein the cloud server and the edge server decompress the decompression task according to the divided decompression ratio, and send a decompression result to a VR terminal, respectively, including:
the cloud server performs partial decompression on the decompression task according to the decompression ratio of the cloud server to obtain the decompression task with partial decompression completed;
the cloud server adopts a network transmission KCP protocol to send the decompression task for completing partial decompression to the edge server;
the edge server performs all decompression on the decompression tasks which finish partial decompression, and decompressed video resources are obtained;
and the edge server adopts a network transmission KCP protocol to send the decompressed video resource to the VR terminal.
7. The method of claim 1, wherein prior to the cloud server receiving the video acquisition task uploaded by the edge server, comprising:
the edge server receives an instruction task and judges whether the instruction task is a task for requesting to acquire video image resources or not:
if not, processing the instruction task and returning a processing result;
if yes, searching in a local database of the edge server: if the video image resources can be searched, all the searched video image resources are decompressed, and all the decompressed video image resources are returned to the VR terminal; and if the video is not searched, taking the instruction task as a video acquisition task, and sending the video acquisition task to the cloud server.
8. The method of claim 7, comprising, prior to the edge server receiving an instruction task:
the VR terminal receives interaction information of a user;
and converting the interaction information into an instruction task and sending the instruction task to the edge server.
9. An edge cloud collaboration system suitable for a cloud VR video stream processing method, comprising: cloud server, edge server and VR terminal;
the cloud server is used for receiving a video acquisition task uploaded by the edge server, acquiring a corresponding video resource according to the video acquisition task, and obtaining a decompression task; according to the computing resources allocated to the decompression tasks by the edge server and the edge server, determining the decompression proportion divided to the edge server and the cloud server, and dividing the decompression tasks; partial decompression is carried out according to the decompression proportion of the cloud server;
the edge server is used for completely decompressing the decompressed tasks after partial decompression to obtain a decompressed result;
and the VR terminal user is used for displaying the decompression result.
10. A storage medium having stored therein computer executable instructions which when loaded and executed by a processor perform the steps of the method according to any of claims 1 to 8.
CN202310539806.7A 2023-05-12 2023-05-12 Method for processing cloud VR video stream, cloud-edge collaboration system and storage medium Pending CN116489410A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310539806.7A CN116489410A (en) 2023-05-12 2023-05-12 Method for processing cloud VR video stream, cloud-edge collaboration system and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310539806.7A CN116489410A (en) 2023-05-12 2023-05-12 Method for processing cloud VR video stream, cloud-edge collaboration system and storage medium

Publications (1)

Publication Number Publication Date
CN116489410A true CN116489410A (en) 2023-07-25

Family

ID=87219550

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310539806.7A Pending CN116489410A (en) 2023-05-12 2023-05-12 Method for processing cloud VR video stream, cloud-edge collaboration system and storage medium

Country Status (1)

Country Link
CN (1) CN116489410A (en)

Similar Documents

Publication Publication Date Title
US11563974B2 (en) Method and apparatus for video decoding
US10979663B2 (en) Methods and apparatuses for image processing to optimize image resolution and for optimizing video streaming bandwidth for VR videos
US11172220B2 (en) Video encoding method, and storage medium thereof
US7142599B2 (en) Method and device for selecting a transcoding method from a set of transcoding methods
CN1901676B (en) Streaming image system and method
US20240098298A1 (en) Segmentation-based parameterized motion models
US20130155075A1 (en) Information processing device, image transmission method, and recording medium
WO2013125220A1 (en) Moving image distribution server, moving image playback device, control method, program, and recording medium
US11115678B2 (en) Diversified motion using multiple global motion models
US20180184089A1 (en) Target bit allocation for video coding
CN112954398B (en) Encoding method, decoding method, device, storage medium and electronic equipment
KR101742258B1 (en) Apparatus and method for providing contents using web-based virtual desktop protocol
CN111432213A (en) Adaptive tile data size coding for video and image compression
US10225573B1 (en) Video coding using parameterized motion models
US9819958B2 (en) Drawing system, information processing apparatus and drawing control method which switch drawing from still image data to video image data
US20230082784A1 (en) Point cloud encoding and decoding method and apparatus, computer-readable medium, and electronic device
CN116489410A (en) Method for processing cloud VR video stream, cloud-edge collaboration system and storage medium
JP5956316B2 (en) Subjective image quality estimation apparatus, subjective image quality estimation method, and program
WO2021223577A1 (en) Video processing method, related apparatus, storage medium, and program product
US20210281850A1 (en) Spatial Layer Rate Allocation
CN115209185A (en) Video frame insertion method and device and readable storage medium
CN101159869A (en) Providing representative image information
Tang et al. Hybrid remote display protocol for mobile thin client computing
CN106982374A (en) A kind of SPICE video transmission methods and device based on HEVC optimized algorithms
CN105120261A (en) Method and device for generating multi-code-rate videos

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination