CN114727122A - Transcoding and transmitting method for live VR service - Google Patents
Transcoding and transmitting method for live VR service Download PDFInfo
- Publication number
- CN114727122A CN114727122A CN202111577955.XA CN202111577955A CN114727122A CN 114727122 A CN114727122 A CN 114727122A CN 202111577955 A CN202111577955 A CN 202111577955A CN 114727122 A CN114727122 A CN 114727122A
- Authority
- CN
- China
- Prior art keywords
- video stream
- terminal
- base station
- transcoding
- live
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 239000000872 buffer Substances 0.000 claims abstract description 86
- 238000012545 processing Methods 0.000 claims description 34
- 230000005540 biological transmission Effects 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 21
- 238000005457 optimization Methods 0.000 claims description 18
- 238000004891 communication Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 11
- 238000009877 rendering Methods 0.000 claims description 11
- 230000001360 synchronised effect Effects 0.000 abstract description 4
- 230000003139 buffering effect Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 25
- 230000000694 effects Effects 0.000 description 7
- 238000003860 storage Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 6
- 238000011160 research Methods 0.000 description 4
- 239000000126 substance Substances 0.000 description 4
- 239000000835 fiber Substances 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005265 energy consumption Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 239000002355 dual-layer Substances 0.000 description 1
- 239000010410 layer Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
- H04N21/43076—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of the same content streams on multiple devices, e.g. when family members are watching the same movie on different devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/06—Selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS]; Services to user groups; One-way selective calling services
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Databases & Information Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
The embodiment of the application provides a transcoding and transmitting method of live broadcast VR service, which comprises the following steps: receiving a video stream sent by a cloud server, and transcoding the video stream according to the terminal and the computing resources of the edge server; determining a delivery strategy of the video stream according to the size of the terminal buffer area; and issuing the delivery strategy to the sub-base station, deciding self multicast content by the sub-base station according to the delivery strategy, and delivering the transcoded video stream to the terminal based on the self multicast content. According to the embodiment of the application, the multicast content is decided according to the video buffering condition of the terminal, so that synchronous and smooth VR video playing is provided for users.
Description
Technical Field
The application relates to the technical field of communication, in particular to a transcoding and transmitting method for live VR service.
Background
With the innovation of wireless technology, a fifth generation new wireless 5G + (5th generation) network has become a foundation for the development of novel streaming media services such as panoramic video, virtual Reality/Augmented Reality/mixed Reality (VR-virtual Reality/AR-Augmented Reality/MR-media Reality) and the like by virtue of the characteristics of ultra-large bandwidth, ultra-low delay, ubiquitous access, mobile computing support and the like. According to the latest industry reports from the Grand Research of the international famous consulting agency (Grand View Research), it was shown that by 2020, the global VR market size will exceed $ 103.2 billion, with a 6-fold increase in market size predicted in 2027. However, the inherent demand for ultra-high quality, ultra-low latency services for VR live broadcasting, and the current high heterogeneity of network environments and user devices, present a significant challenge to providing high quality VR live broadcasting services for large-scale users.
By unloading the computing task to the network edge, the edge computing shortens the distance between the resource and the user, and can effectively make up for the defect that the cloud computing is far away from the user in the geographic position. In addition, due to the endogenous support of multicast by the live broadcast service and the wireless network, multicast becomes a very potential transmission technology in the live broadcast service. In addition, how to design a multicast strategy to provide high-quality VR services by utilizing overlapping fields of view among VR service users is another research hotspot in the field. To improve the efficiency of computing resource usage, researchers have provided many solutions based on edge computing, including methods based on deep reinforcement learning, distributed content rendering schemes, and scalable multi-layer VR stitching techniques.
Then, current research is dedicated to reducing system overhead and transmission delay, while immersive service experience of the VR service is ignored, and it is not possible to provide immersive VR experience for users with different geographic locations and different device and network configurations on a wireless network with limited resources.
Disclosure of Invention
Because the existing method has the problems, the embodiment of the application provides a transcoding and transmission method for live VR service.
Specifically, the embodiment of the present application provides the following technical solutions:
in a first aspect, an embodiment of the present application provides a transcoding and transmitting method based on VR live broadcast service, which is applied to a base station and includes:
receiving a video stream sent by a cloud server, and transcoding the video stream according to the computing resources of a terminal and an edge server;
determining a delivery strategy of the video stream according to the size of the terminal buffer area; and issuing the delivery strategy to a sub-base station, deciding self multicast content by the sub-base station according to the delivery strategy, and delivering the transcoded video stream to a terminal based on the self multicast content.
Optionally, determining the delivery policy of the video stream according to the size of the terminal buffer area includes:
determining the current residual playing buffer area capacity of the terminal according to the change situation of the terminal buffer area size along with time;
and determining the delivery strategy of the video stream according to the current residual playing buffer capacity of the terminal.
Optionally, determining a delivery policy of the video stream according to the current remaining play buffer capacity of the terminal includes:
determining the minimum value mean value of the buffer areas of all the terminals according to the current residual playing buffer area capacity of the terminals;
optimizing the minimum mean value of the buffer areas of all the terminals based on a random optimization model to obtain a video stream delivery strategy based on the maximum minimum value of the buffer areas;
wherein the stochastic optimization model is:
wherein u ═ vi，fj，sn) The request identification indicates that the resolution of the video stream content of the terminal in the range of the base station n is j;
hu[t]representing the channel state of the terminal u at the time t; gamma rayu[t]Indicating whether the base station executes a multicast strategy to the terminal u at the time t; t represents a time slot set;
represents a mean function;
representing a joint decision set of indexes of different terminal clusters;
representing that the virtual reality VR live library contains C different online VR contents;
indicating that each VR content contains M different field-of-view slices and has K levels of video resolution;
representing N sub base stations.
In a second aspect, an embodiment of the present application further provides a transcoding and transmitting method based on a VR live broadcast service, which is applied to a cloud server, and includes:
and receiving the video stream uploaded by the user provider, rendering the video stream and then sending the video stream to the base station.
In a third aspect, an embodiment of the present application further provides a transcoding and transmitting method based on VR live broadcast service, which is applied to a terminal and includes:
and receiving the transcoded video stream sent by the base station, and providing virtual reality VR live video service according to the transcoded video stream and/or video streams obtained from other terminals based on device-to-device D2D communication.
In a fourth aspect, an embodiment of the present application further provides a base station, including:
the first processing module is used for receiving a video stream sent by a cloud server and transcoding the video stream according to the computing resources of a terminal and an edge server;
the second processing module is used for determining the delivery strategy of the video stream according to the size of the terminal buffer area; and the delivery strategy is issued to the sub-base station, the sub-base station decides self multicast content according to the delivery strategy, and delivers the transcoded video stream to a terminal based on the self multicast content.
In a fifth aspect, an embodiment of the present application further provides a cloud server, including:
and the third processing module is used for receiving the video stream uploaded by the user provider, rendering the video stream and then sending the video stream to the base station.
In a sixth aspect, an embodiment of the present application further provides a terminal, including:
and the fourth processing module is used for receiving the transcoded video stream sent by the base station and providing virtual reality VR live video service according to the transcoded video stream and/or video streams obtained from other terminals based on device-to-device D2D communication.
In a seventh aspect, an embodiment of the present application further provides a base station device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the following steps when executing the computer program:
receiving a video stream sent by a cloud server, and transcoding the video stream according to the computing resources of a terminal and an edge server;
determining a delivery strategy of the video stream according to the size of the terminal buffer area; and issuing the delivery strategy to a sub-base station, deciding self multicast content by the sub-base station according to the delivery strategy, and delivering the transcoded video stream to a terminal based on the self multicast content.
In an eighth aspect, an embodiment of the present application further provides a cloud server device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the following steps when executing the computer program:
and receiving the video stream uploaded by the user provider, rendering the video stream and then sending the video stream to the base station.
In a ninth aspect, an embodiment of the present application further provides a terminal device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the following steps when executing the computer program:
and receiving the transcoded video stream sent by the base station, and providing virtual reality VR live video service according to the transcoded video stream and/or video streams obtained from other terminals based on device-to-device D2D communication.
In a tenth aspect, this application embodiment further provides a processor-readable storage medium, which stores a computer program for causing a processor to execute the steps of the transcoding and transmitting method for VR live broadcast service according to the first aspect, the second aspect, or the third aspect.
According to the transcoding and transmission method of the live broadcast VR service, the video stream sent by the cloud server is received, the video stream is transcoded according to the terminal and the computing resources of the edge server, the delivery strategy of the video stream is determined according to the size of the terminal buffer area, the delivery strategy is issued to the sub base station, the sub base station decides the multicast content of the sub base station according to the delivery strategy, and the transcoded video stream is delivered to the terminal based on the multicast content of the sub base station, so that the multicast content can be decided according to the video buffering condition of the terminal, synchronous and smooth VR video playing is provided for a user, and the computing resources of the 5G heterogeneous network edge server and the ubiquitous access user equipment are cooperatively utilized, so that flexible online transcoding for the VR content is achieved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart illustrating steps of a transcoding and transmitting method applied to a VR live broadcast service of a base station according to an embodiment of the present application;
fig. 2 is a system framework diagram of transcoding and transmission of a VR live broadcast service provided by an embodiment of the present application;
fig. 3 is a schematic diagram illustrating evolution of a queue length of a terminal buffer provided in an embodiment of the present application;
fig. 4 is a flowchart illustrating steps of a transcoding and transmitting method applied to a VR live broadcast service of a cloud server according to an embodiment of the present application;
fig. 5 is a flowchart illustrating steps of a transcoding and transmitting method applied to a VR live service of a terminal according to an embodiment of the present application;
fig. 6 is a block diagram of a transcoding and transmitting apparatus for VR live broadcast service applied to a base station according to an embodiment of the present disclosure;
fig. 7 is a block diagram of a transcoding and transmitting apparatus applied to a VR live broadcast service of a cloud server according to an embodiment of the present application;
fig. 8 is a block diagram of a transcoding and transmitting apparatus applied to a VR live broadcast service of a terminal according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of a base station device according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of a cloud server device provided in an embodiment of the present application;
fig. 11 is a schematic structural diagram of a terminal device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Fig. 1 is a flowchart illustrating steps of a transcoding and transmitting method for VR live broadcast service applied to a base station according to an embodiment of the present application, fig. 2 is a schematic diagram of a system framework for transcoding and transmitting VR live broadcast service according to an embodiment of the present application, and fig. 3 is a schematic diagram illustrating evolution of a queue length of a terminal buffer provided by an embodiment of the present application. With reference to fig. 1 to fig. 3, a method for transcoding and transmitting a VR live broadcast service provided in an embodiment of the present application is explained and explained in detail below, and as shown in fig. 1, a method for transcoding and transmitting based on a VR live broadcast service provided in an embodiment of the present application includes:
step 101: receiving a video stream sent by a cloud server, and transcoding the video stream according to the computing resources of a terminal and an edge server;
in this step, the content provider can continuously produce the fragmented VR video and upload the content to the cloud server, and the cloud server performs unified management on the video streams, and delivers and allocates the video streams to the macro base station and the small base station cluster in a unified manner together with processing tasks such as video rendering. After receiving the video stream sent after processing by the cloud server, the base station transcodes the video stream based on a multicast and task offload (MATO) joint optimization algorithm deployed at the base station side and according to the terminal and the computing resources of the edge server.
Step 102: determining a delivery strategy of the video stream according to the size of the terminal buffer area; and issuing the delivery strategy to a sub-base station, deciding self multicast content by the sub-base station according to the delivery strategy, and delivering the transcoded video stream to a terminal based on the self multicast content.
In this step, it should be noted that, after the base station receives the video stream, the base station performs relevant processing on the video stream. In particular, the base stations will cooperatively perform the assigned video stream processing tasks. The macro base station, as a controller for content delivery, will be responsible for managing the global content delivery. In the process, the macro base station determines a self delivery strategy preferentially according to the size of the terminal buffer area, then informs the strategy to the small base station, the small base station decides self multicast behaviors and contents under the condition of the macro base station delivery strategy, and delivers the transcoded video stream to the terminal based on the self multicast contents.
According to the transcoding and transmission method of the VR live broadcast service, the video stream sent by the cloud server is received, the video stream is transcoded according to the terminal and the computing resources of the edge server, the delivery strategy of the video stream is determined according to the size of the terminal buffer area, the delivery strategy is issued to the sub base station, the sub base station decides the multicast content of the sub base station according to the delivery strategy, and the transcoded video stream is delivered to the terminal based on the multicast content of the sub base station, so that the multicast content can be decided according to the video buffering condition of the terminal, synchronous and smooth VR video playing is provided for a user, and the computing resources of the 5G heterogeneous network edge server and the ubiquitous access user equipment are cooperatively utilized, so that flexible online transcoding for the VR content is achieved.
Based on the content of the foregoing embodiment, in this embodiment, determining the delivery policy of the video stream according to the size of the terminal buffer includes:
determining the current residual playing buffer area capacity of the terminal according to the change situation of the terminal buffer area size along with time;
and determining the delivery strategy of the video stream according to the current residual playing buffer capacity of the terminal.
In this embodiment, it should be noted that, in order to characterize the synchronization attribute of the end user playing, the change of the buffer size of each end user with time needs to be considered, that is, the problem of playing buffer evolution. Therefore, in the embodiment of the application, in the process of determining the delivery strategy of the video stream according to the size of the terminal buffer area, the change situation of the size of the terminal buffer area along with time is considered, and the delivery strategy of the video stream is determined according to the change of the size of the terminal buffer area along with time, so that synchronous and smooth VR video playing is provided for a user.
Based on the content of the foregoing embodiment, in this embodiment, determining a delivery policy of the video stream according to the current remaining play buffer capacity of the terminal includes:
determining the minimum value mean value of the buffer areas of all the terminals according to the current residual playing buffer area capacity of the terminals;
optimizing the minimum mean value of the buffer areas of all the terminals based on a random optimization model to obtain a video stream delivery strategy based on the maximum minimum value of the buffer areas;
wherein the stochastic optimization model is:
wherein u ═ vi，fj，sn) The request identification represents that the resolution of the terminal to the video stream content is i and j within the range of the base station n;
hu[t]representing the channel state of the terminal u at the time t; gamma rayu[t]Indicating whether the base station executes the multicast strategy on the terminal u at the time t; t represents a time slot set;
represents a mean function;
representing a joint decision set of indexes of different terminal clusters;
representing that the virtual reality VR live library contains C different online VR contents;
indicating that each VR content contains M different field-of-view slices and has K levels of video resolution;
representing N sub base stations.
In this embodiment, as shown in fig. 3, when the multicast transmission video stream is successful, the corresponding play-out buffer is charged to τ, and the dot in fig. 3 is the minimum value based on the maximized buffer area. It should be noted that, in the VR live broadcast system, increasing the minimum value point of the buffer area can effectively avoid playing pause, and improve the video service quality. Therefore, the embodiment of the application is based on a random optimization model, and the minimum mean value of the buffer area of all the terminals is optimized, so that a video stream delivery strategy based on the maximum minimum value of the buffer area can be obtained, and the video service quality is improved while the playing is blocked.
As shown in fig. 4, a flowchart of steps of a transcoding and transmitting method for VR live broadcast service applied to a cloud server provided in an embodiment of the present application is shown, where the method includes the following steps:
step 201: and receiving the video stream uploaded by the user provider, rendering the video stream and then sending the video stream to the base station.
In this step, it should be noted that the content provider may continuously produce the fragmented VR video and upload the content to the cloud server, and the cloud server performs unified management on the video streams, and delivers and allocates the video streams to the macro base station and the small base station cluster in a unified manner together with processing tasks such as video rendering.
As shown in fig. 5, a flowchart of steps of a transcoding and transmitting method for VR live broadcast service applied to a terminal provided in the embodiment of the present application is shown, where the method includes the following steps:
step 301: and receiving the transcoded video stream sent by the base station, and providing virtual reality VR live video service according to the transcoded video stream and/or video streams obtained from other terminals based on device-to-device D2D communication.
In this step, it should be noted that the terminal receives the transcoded video stream sent by the base station, and provides a virtual reality VR live video service according to the transcoded video stream. In addition, the user may also access content from other nodes in the multicast cluster. Therefore, when the network condition between the user and the base station is not good, the transcoded video stream can be obtained from the cluster through the D2D communication, and the virtual reality VR live video service is provided according to the transcoded video stream.
The present application will be specifically described below with reference to specific examples.
The first embodiment:
in this embodiment, as shown in fig. 2, under a dual-layer 5G heterogeneous network, a live VR system framework based on end user group assistance. The framework comprises a plurality of types of nodes, including a live content providing node set
Requesting a set of end user clusters
Macro base station s0And N small base stations defined as
Defining the VR live library to contain C different online VR contents, expressed as:
and the content popularity obeys a popularity index of S0Zipf distribution of (a). Each VR content contains M different field slices and has K levels of video resolution, denoted as:
defining the highest resolution as f1 and the lowest resolution fK as f1>f2>…>fK, the original video stream slice can be transcoded to a plurality of low resolutions.
In this embodiment, the VR live frame mainly includes three parts: a collaborative video transcoding, collaborative content delivery and buffer evolution model, for a collaborative video transcoding process: first, the content provider will continuously produce fragmented VR videos and upload the content to the cloud server. Then, the cloud server performs unified management on the video streams, and delivers and distributes the video streams and processing tasks such as video rendering to the macro base station and the small base station cluster in a unified manner. Thus, once the data streams arrive at the base station, they are processed and then transmitted to the end user. Base station
The assigned video processing tasks will be performed cooperatively and the processed video with the appropriate resolution will be delivered to end users with different device configurations. For the collaborative content delivery process: since the base station generally transmits the content to the end users in a multicast manner, the end users can also communicate directly via a Device-to-Device (D2D) link, and in the system framework, the macro base station, as a controller for content delivery, will be responsible for managing the global content delivery. In the process, the macro base station preferentially determines the self delivery strategy, then informs the small base station of the strategy, the small base station decides the self multicast behavior under the condition of the macro base station delivery strategy, and finally enhances the whole content delivery process through D2D communication.
Second embodiment:
in this embodiment, in order to characterize the synchronization attribute of the end user playing, the change of the buffer size of each end user with time needs to be considered, that is, the problem of playing buffer evolution. Definition of
Multicasting a joint decision set of cluster indices to different end users using u ═ c(υi，fj，sn) Indicating the request identification of the end user group for the content i resolution j within the range of the base station n. Thus, the base station can be connected
Multicast strategy of
Expressed as:
wherein the content of the first and second substances,
indicating all multicast user groups within the base station n. The following steps are used:
indicating a base station
Set of radio channel conditions at time t, hu[t]Indicating the channel state of the group u of end users at time t.
In addition, define χu[t]The remaining playout buffer capacity for the group u of end users at time t. The second diagram shows χu[t]One specific example of evolution over time. Buffer χ for each successful multicast completion by the end user groupu[t]It will be charged to tau, otherwise the buffer will drop by 1. Thus, a buffer evolution process expression is obtained:
χu[t+1]＝χu[t]-1+(τ-χu[t])γu[t]hu[t]
when the multicast transmission is successful, the corresponding play-out buffer is filled to τ. And defining the minimum value point of the buffer area as the red point identification position in the second graph. The average of the buffer minimum point for the user group u under the long-term view can be expressed as:
wherein the content of the first and second substances,
is a mean function, defining the mean of the minimum values of the buffers of all user nodes in the system as
The third embodiment:
in this embodiment, in the VR live broadcast system, increasing the minimum value point of the buffer area can effectively avoid playing the video without pause, thereby improving the video service quality. Therefore, the optimization problem of defining the minimum value of the buffer area is
And aims to maximize without violating resource constraints
Thus, the problem of maximizing the average of the buffer minima points can be expressed as:
wherein phi isuIs an auxiliary variable
It should be noted that the multicast-aware transcoding offload problem determines a task offload policy that minimizes the resource consumption cost of all base stations during the entire operation process. Thus, the problem can be formalized as:
wherein the content of the first and second substances,
in order to be a weight factor, the weight factor,
first term of objective function for total transmission overhead of all base stations under strategy pi
Representing the transcoding resource consumption of base station n. The constraints ensure that the transcoding workload of base station n does not exceed the node capacity.
The fourth embodiment:
in the embodiment, a joint optimization algorithm strategy is further provided to jointly optimize multicast content transmission and online video transcoding. The specific strategy comprises the following processes:
1. joint optimization problem for multicast and transcoding task offloading
Define epsilon as the expectation of a random process gamma t h t. The mean maximization problem for the buffer minima points can be re-expressed as:
wherein the content of the first and second substances,
the optimality of the h-only strategy is influenced by epsilonuInfluence, definition of the result γ from the h-only strategy*To the above problems
The optimal solution of (a).
2. Multicast and task offload (MATO) joint optimization algorithm deployed on base station side
As can be seen from the above embodiments, the optimization of the buffer minima is only decided by the multicast decision, and the energy consumption is affected by multicast and transcoding offload. However, the over-play katton (χ [ t ] < 0) is generally less acceptable than the increase in power consumption for real-time VR services. Therefore, the effect of buffer minimum optimization is preferentially considered in algorithm design, and total energy consumption is further reduced through task offloading. The detailed pseudo-code of the algorithm is as follows:
the embodiment of the application also provides a crowd-assisted delivery algorithm deployed on the terminal, and the algorithm describes a crowd-assisted delivery process. This is deployed at the terminal. For each end user group, parameters are selected and initial values for buffer minimum and queue length are set to 0. Based on the multicast process, the end user cluster will update the queue length qu[t]. Wherein [. ]]+Max {, 1 }. Terminal user x according to buffer levelu[t]The resolution of the tiles within the field of view is selected. In addition, the end user may also access content from other nodes in the multicast cluster. Thus, when the user and base areWhen the network conditions between the stations are not good, the content can also be obtained from the cluster through D2D communication, the algorithm is as follows:
in addition, as shown in fig. 6, a block diagram of a transcoding and transmitting apparatus for VR live broadcast service applied to a base station according to an embodiment of the present application is provided, where the apparatus includes:
the first processing module 1 is configured to receive a video stream sent by a cloud server, and transcode the video stream according to computing resources of a terminal and an edge server;
the second processing module 2 is configured to determine a delivery policy of the video stream according to the size of the terminal buffer; and issuing the delivery strategy to a sub-base station, deciding self multicast content by the sub-base station according to the delivery strategy, and delivering the transcoded video stream to a terminal based on the self multicast content.
Based on the content of the foregoing embodiment, in this embodiment, the first processing module is specifically configured to:
determining the current residual playing buffer area capacity of the terminal according to the change condition of the size of the terminal buffer area along with time;
and determining the delivery strategy of the video stream according to the current residual playing buffer capacity of the terminal.
Based on the content of the foregoing embodiment, in this embodiment, the first processing module is further specifically configured to:
determining the minimum value mean value of the buffer areas of all the terminals according to the current residual playing buffer area capacity of the terminals;
optimizing the minimum mean value of the buffer areas of all the terminals based on a random optimization model to obtain a video stream delivery strategy based on the maximum minimum value of the buffer areas;
wherein the stochastic optimization model is:
wherein u ═ vi，fj，sn) The request identification represents that the resolution of the terminal to the video stream content is i and j within the range of the base station n;
hu[t]representing the channel state of the terminal u at the time t; gamma rayu[t]Indicating whether the base station executes the multicast strategy on the terminal u at the time t; t represents a time slot set;
represents a mean function;
representing a joint decision set of indexes of different terminal clusters;
representing that the virtual reality VR live library contains C different online VR contents;
indicating that each VR content contains M different field-of-view slices and has K levels of video resolution;
representing N sub base stations.
It should be noted that, the apparatus can implement all the method steps of the transcoding and transmission method embodiments of the VR live broadcast service applied to the base station and achieve the same technical effect, and details are not repeated herein.
In addition, as shown in fig. 7, a block diagram of a transcoding and transmitting apparatus for VR live broadcast service applied to a cloud server according to an embodiment of the present application is provided, where the apparatus includes:
and the third processing module 3 is configured to receive a video stream uploaded by a user provider, render the video stream, and send the rendered video stream to the base station.
It should be noted that the present apparatus can implement all the method steps of the transcoding and transmission method embodiments of the VR live broadcast service applied to the cloud server and can achieve the same technical effect, and details are not repeated herein.
In addition, as shown in fig. 8, a block diagram of a transcoding and transmitting apparatus for VR live broadcast service applied to a terminal according to an embodiment of the present application is provided, where the apparatus includes:
and the fourth processing module 4 is configured to receive the transcoded video stream sent by the base station, and provide a virtual reality VR live video service according to the transcoded video stream and/or a video stream obtained from another terminal based on the device-to-device D2D communication.
It should be noted that, the apparatus can implement all the method steps of the transcoding and transmission method embodiments of the VR live broadcast service applied to the terminal and can achieve the same technical effect, and details are not repeated herein.
Fig. 9 is a schematic structural diagram of a base station device according to an embodiment of the present application, and includes a memory 920, a transceiver 900, and a processor 910.
Wherein in fig. 9, the bus architecture may include any number of interconnected buses and bridges, with one or more processors, represented by processor 910, and various circuits, represented by memory 920, being linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface. The transceiver 900 may be a number of elements including a transmitter and a receiver that provide a means for communicating with various other apparatus over a transmission medium including wireless channels, wired channels, fiber optic cables, and the like. The processor 910 is responsible for managing the bus architecture and general processing, and the memory 920 may store data used by the processor 910 in performing operations.
The processor 910 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or a Complex Programmable Logic Device (CPLD), and may also have a multi-core architecture.
A memory 920 for storing a computer program; a transceiver 900 for transceiving data under the control of the processor; a processor 910 for reading the computer program in the memory and performing the following operations:
receiving a video stream sent by a cloud server, and transcoding the video stream according to the computing resources of a terminal and an edge server;
determining a delivery strategy of the video stream according to the size of the terminal buffer area; and issuing the delivery strategy to a sub-base station, deciding self multicast content by the sub-base station according to the delivery strategy, and delivering the transcoded video stream to a terminal based on the self multicast content.
Based on the content of the foregoing embodiment, in this embodiment, determining the delivery policy of the video stream according to the size of the terminal buffer includes:
determining the current residual playing buffer area capacity of the terminal according to the change situation of the terminal buffer area size along with time;
and determining the delivery strategy of the video stream according to the current residual playing buffer area capacity of the terminal.
Based on the content of the foregoing embodiment, in this embodiment, determining a delivery policy of the video stream according to the current remaining play buffer capacity of the terminal includes:
determining the minimum mean value of the buffer areas of all the terminals according to the current capacity of the residual playing buffer areas of the terminals;
optimizing the minimum mean value of the buffer areas of all the terminals based on a random optimization model to obtain a video stream delivery strategy based on the maximum minimum value of the buffer areas;
wherein the stochastic optimization model is:
wherein u ═ vi，fj，sn) The request identification represents that the resolution of the terminal to the video stream content is i and j within the range of the base station n;
hu[t]representing the channel state of the terminal u at the time t; gamma rayu[t]Indicating whether the base station executes the multicast strategy on the terminal u at the time t; t represents a time slot set;
represents a mean function;
representing a joint decision set of indexes of different terminal clusters;
representing that the virtual reality VR live library contains C different online VR contents;
indicating that each VR content contains M different field-of-view slices and has K levels of video resolution;
representing N sub base stations.
It should be noted that the base station device provided in the embodiment of the present application can implement all method steps of the transcoding and transmission method embodiment for VR live broadcast service applied to the base station device and can achieve the same technical effect, and details are not repeated herein.
Fig. 10 is a schematic structural diagram of a cloud server device provided in an embodiment of the present application, and includes a memory 1020, a transceiver 1000, and a processor 1010.
Where in fig. 10, the bus architecture may include any number of interconnected buses and bridges, with one or more processors represented by processor 1010 and various circuits of memory represented by memory 1020 being linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface. The transceiver 1000 may be a number of elements including a transmitter and a receiver that provide a means for communicating with various other apparatus over a transmission medium including wireless channels, wired channels, fiber optic cables, and the like. The processor 1010 is responsible for managing the bus architecture and general processing, and the memory 1020 may store data used by the processor 1010 in performing operations.
The processor 1010 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or a Complex Programmable Logic Device (CPLD), and may also have a multi-core architecture.
A memory 1020 for storing a computer program; a transceiver 1000 for transceiving data under the control of the processor; a processor 1010 for reading the computer program in the memory and performing the following operations:
and receiving the video stream uploaded by the user provider, rendering the video stream and then sending the video stream to the base station.
It should be noted that the cloud server device provided in the embodiment of the present application can implement all method steps of the transcoding and transmission method embodiment of the VR live broadcast service applied to the cloud server device and can achieve the same technical effect, and details are not repeated here.
Fig. 11 is a schematic structural diagram of a terminal device according to an embodiment of the present application, and includes a memory 1120, a transceiver 1100, and a processor 1111.
In FIG. 11, among other things, the bus architecture may include any number of interconnected buses and bridges with one or more processors, represented by the processor 1111, and various circuits of memory, represented by the memory 1120, being linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. The bus interface provides an interface. The transceiver 1100 may be a plurality of elements including a transmitter and a receiver that provide a means for communicating with various other apparatus over a transmission medium including wireless channels, wired channels, fiber optic cables, and the like. The processor 1111 is responsible for managing the bus architecture and general processing, and the memory 1120 may store data used by the processor 1111 in performing operations.
The processor 1111 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or a Complex Programmable Logic Device (CPLD), and may also adopt a multi-core architecture.
A memory 1120 for storing a computer program; a transceiver 1100 for transceiving data under the control of the processor; a processor 1111 for reading the computer program in the memory and performing the following operations:
and receiving the transcoded video stream sent by the base station, and providing virtual reality VR live video service according to the transcoded video stream and/or video streams obtained from other terminals based on device-to-device D2D communication.
It should be noted that the terminal device provided in the embodiment of the present application can implement all method steps of the transcoding and transmission method embodiment of the VR live broadcast service applied to the terminal device and can achieve the same technical effects, and details are not repeated here.
On the other hand, the embodiment of the present application further provides a processor-readable storage medium, where a computer program is stored, and the computer program is configured to enable the processor to execute the method described in the foregoing embodiment.
The processor-readable storage medium can be any available medium or data storage device that can be accessed by a processor, including, but not limited to, magnetic memory (e.g., floppy disks, hard disks, magnetic tape, magneto-optical disks (MOs), etc.), optical memory (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor memory (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND FLASH), Solid State Disks (SSDs)), etc.
As can be seen from the above embodiments, a processor-readable storage medium stores a computer program for causing the processor to perform the steps of the above methods of transcoding and transmitting VR live broadcast services.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These processor-executable instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A transcoding and transmission method of a live broadcast VR service is applied to a base station and comprises the following steps:
receiving a video stream sent by a cloud server, and transcoding the video stream according to the computing resources of a terminal and an edge server;
determining a delivery strategy of the video stream according to the size of the terminal buffer area; and issuing the delivery strategy to a sub-base station, deciding self multicast content by the sub-base station according to the delivery strategy, and delivering the transcoded video stream to a terminal based on the self multicast content.
2. The method of transcoding and transmitting of VR live broadcast service of claim 1, wherein determining the delivery policy for the video stream based on the terminal buffer size comprises:
determining the current residual playing buffer area capacity of the terminal according to the change situation of the terminal buffer area size along with time;
and determining the delivery strategy of the video stream according to the current residual playing buffer area capacity of the terminal.
3. The method of claim 2, wherein determining a delivery policy for the video stream based on a current remaining play-out buffer capacity of the terminal comprises:
determining the minimum mean value of the buffer areas of all the terminals according to the current capacity of the residual playing buffer areas of the terminals;
optimizing the minimum mean value of the buffer areas of all the terminals based on a random optimization model to obtain a video stream delivery strategy based on the maximum minimum value of the buffer areas;
wherein the stochastic optimization model is:
wherein u-vi，fj，sn) The request identification represents that the resolution of the terminal to the video stream content is i and j within the range of the base station n;
hu[t]representing the channel state of the terminal u at the time t; gamma rayu[t]Indicating whether the base station executes the multicast strategy on the terminal u at the time t; t represents a time slot set;
representing a mean function;
representing a joint decision set of indexes of different terminal clusters;
representing that the virtual reality VR live library contains C different online VR contents;
indicating that each VR content contains M different field-of-view slices and has K levels of video resolution;
representing N sub base stations.
4. The VR live broadcast service transcoding and transmission method of claim 1, applied to a cloud server, comprising:
and receiving the video stream uploaded by the user provider, rendering the video stream and then sending the video stream to the base station.
5. The method for transcoding and transmitting the VR live service of claim 1 applied to a terminal and comprising:
and receiving the transcoded video stream sent by the base station, and providing virtual reality VR live video service according to the transcoded video stream and/or video streams obtained from other terminals based on device-to-device D2D communication.
6. The VR live service transcoding and transmitting method of claim 1, comprising: a kind of base station is provided, which has a base station,
the base station comprises a first processing module, a second processing module and a third processing module, wherein the first processing module is used for receiving a video stream sent by a cloud server and transcoding the video stream according to the terminal and the computing resources of an edge server;
the second processing module is used for determining the delivery strategy of the video stream according to the size of the terminal buffer area; and issuing the delivery strategy to a sub-base station, deciding self multicast content by the sub-base station according to the delivery strategy, and delivering the transcoded video stream to a terminal based on the self multicast content.
7. The VR live service transcoding and transmitting method of claim 1, comprising: a cloud server comprises a third processing module and is used for receiving a video stream uploaded by a user provider, rendering the video stream and then sending the video stream to a base station.
8. The VR live service transcoding and transmitting method of claim 1, comprising: a terminal comprises a fourth processing module and is used for receiving a transcoded video stream sent by a base station and providing Virtual Reality (VR) live video service according to the transcoded video stream and/or video streams obtained from other terminals based on device-to-device (D2D) communication.
9. The VR live service transcoding and transmitting method of claim 1, comprising: an electronic device comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of the transcoding and transmission method of the VR live broadcast service applied to a base station, the transcoding and transmission method of the VR live broadcast service applied to a server or the transcoding and transmission method of the VR live broadcast service applied to a terminal.
10. The method for transcoding and transmitting of the VR live service of claim 3, wherein the computer program when executed by the processor implements the method for transcoding and transmitting of the VR live service applied to the base station, or the method for transcoding and transmitting of the VR live service applied to the server, or the method for transcoding and transmitting of the VR live service applied to the terminal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111577955.XA CN114727122A (en) | 2021-12-22 | 2021-12-22 | Transcoding and transmitting method for live VR service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111577955.XA CN114727122A (en) | 2021-12-22 | 2021-12-22 | Transcoding and transmitting method for live VR service |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114727122A true CN114727122A (en) | 2022-07-08 |
Family
ID=82235296
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111577955.XA Pending CN114727122A (en) | 2021-12-22 | 2021-12-22 | Transcoding and transmitting method for live VR service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114727122A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116033189A (en) * | 2023-03-31 | 2023-04-28 | 卓望数码技术（深圳）有限公司 | Live broadcast interactive video partition intelligent control method and system based on cloud edge cooperation |
-
2021
- 2021-12-22 CN CN202111577955.XA patent/CN114727122A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116033189A (en) * | 2023-03-31 | 2023-04-28 | 卓望数码技术（深圳）有限公司 | Live broadcast interactive video partition intelligent control method and system based on cloud edge cooperation |
| CN116033189B (en) * | 2023-03-31 | 2023-06-30 | 卓望数码技术（深圳）有限公司 | Live broadcast interactive video partition intelligent control method and system based on cloud edge cooperation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | QoE-driven mobile edge caching placement for adaptive video streaming | |
| US10271112B2 (en) | System and method for dynamic adaptive video streaming using model predictive control | |
| Xing et al. | A real-time adaptive algorithm for video streaming over multiple wireless access networks | |
| US10601731B2 (en) | Broadcast services platform and methods for use therewith | |
| CN106792835B (en) | Control method and system for wireless equipment sharing bandwidth | |
| Zheng et al. | MEC-enabled wireless VR video service: A learning-based mixed strategy for energy-latency tradeoff | |
| CN106230550B (en) | With the coding and transmission method of caching capabilities under car networking environment | |
| CN104104973A (en) | Group bandwidth management optimization method applied to cloud media system | |
| CN112543357B (en) | Stream media data transmission method based on DASH protocol | |
| Meng et al. | A utility-based resource allocation scheme in cloud-assisted vehicular network architecture | |
| US20200136926A1 (en) | Dynamic adaptive network | |
| CN114727122A (en) | Transcoding and transmitting method for live VR service | |
| US8140702B2 (en) | System and method of maximizing utility in media delivery network | |
| CN113938394B (en) | Monitoring service bandwidth allocation method and device, electronic equipment and storage medium | |
| CN110913239B (en) | Video cache updating method for refined mobile edge calculation | |
| WO2022062981A1 (en) | Resource scheduling method and system, electronic device, and computer-readable storage medium | |
| CN107465654B (en) | System for distributing and safely searching and killing contents among service subnets of broadcast station | |
| CN105721901A (en) | Video transcoder adaption method, device and system | |
| Bukhari et al. | To transcode or not? A machine learning based edge video caching and transcoding strategy | |
| Yang et al. | Adaptive configuration of cloud video transcoding | |
| Madarasingha et al. | Edge assisted frame interpolation and super resolution for efficient 360-degree video delivery | |
| Liu et al. | Joint optimization of content placement and request redirection in mobile-CDN | |
| KR102648108B1 (en) | Method for providing collaborative http adaptive streaming using wifi network based on software-defined networking and apparatus thereof | |
| Zhao et al. | Optimizing Quality of Experience of Free-Viewpoint Video Streaming with Markov Decision Process | |
| KR101078213B1 (en) | Method for managing CPU load of contents transmitting device and contents transmitting device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20220708 |
|
| WD01 | Invention patent application deemed withdrawn after publication |