CN113329266B - Panoramic video self-adaptive transmission method based on limited user visual angle feedback - Google Patents

Panoramic video self-adaptive transmission method based on limited user visual angle feedback Download PDF

Info

Publication number
CN113329266B
CN113329266B CN202110634899.2A CN202110634899A CN113329266B CN 113329266 B CN113329266 B CN 113329266B CN 202110634899 A CN202110634899 A CN 202110634899A CN 113329266 B CN113329266 B CN 113329266B
Authority
CN
China
Prior art keywords
video
panoramic video
branch
gof
code rate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110634899.2A
Other languages
Chinese (zh)
Other versions
CN113329266A (en
Inventor
黎洁
韩玲
李奇越
张聪
王枭
王慧宇
陈勇
彭涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hefei University of Technology
Original Assignee
Hefei University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hefei University of Technology filed Critical Hefei University of Technology
Priority to CN202110634899.2A priority Critical patent/CN113329266B/en
Publication of CN113329266A publication Critical patent/CN113329266A/en
Application granted granted Critical
Publication of CN113329266B publication Critical patent/CN113329266B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146—Data rate or code amount at the encoder output
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234309—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 or from Quicktime to Realvideo
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234381—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities

Abstract

The invention discloses a panorama self-adaptive transmission method based on limited user visual angle feedback, which comprises the following steps: 1. the server carries out block processing on the original panoramic video and encodes each block of video into different quality grades; 2. the server extracts the salient features of the panoramic video and predicts the view angles of all users by combining with limited user view angle feedback information; 3. the server selects a quality grade for each video block according to the predicted view angle of all users and the bandwidth information of the downlink channel, and then transmits each video block to the client. The invention can better improve the resource utilization rate and improve the experience (QoE) of multiple users under the condition of bandwidth limitation.

Description

Panoramic video self-adaptive transmission method based on limited user visual angle feedback
Technical Field
The invention relates to the field of multimedia video transmission, in particular to a streaming media self-adaptive transmission method for panoramic video.
Background
A panoramic video adaptive transmission method based on DASH (publication number: CN108235131A) is invented by Chongqing, Cone, et al, at the university of post and telecommunications, and comprises the following steps: the method comprises the steps of establishing a mapping relation model of a three-dimensional panoramic video and a two-dimensional panoramic video, carrying out region priority division on a VR panoramic video based on human vision and motion characteristics, slicing the panoramic video by a server, predicting available bandwidth by a client bandwidth estimation module through a Kalman filtering algorithm, carrying out smoothing processing on the available bandwidth by a client video cache module based on a cache area state, predicting a user window by a client user window perception module based on motion inertia, and carrying out self-adaptive video transmission by the client decision module comprehensively considering the states of the user window, a network environment and the cache area. However, the method does not consider the role of QoE in panoramic video transmission, lacks quality of experience (QoE) indexes of users in the transmission process, and cannot reflect the experience change condition of users in the transmission process.
The Panyuxuan et al of Beijing post and telecommunications university invents an reinforcement learning-based adaptive panoramic video transmission method (publication number: CN112584119A), which is characterized by comprising the following steps: the remote video server analyzes the video content, and obtains the motion speed and the depth of field of the video content by using an optical flow method to obtain a numerical result of the video quality; the remote video server carries out tile segmentation according to the video quality, adopts a two-dimensional clustering algorithm to spatially divide the video into a specified number of tiles with different sizes, and carries out coding of different quality grades on the tiles to obtain coding results of a plurality of quality versions; the remote video server trains a deep learning model according to the stored bandwidth data and the coding results of the multiple quality versions, and the deep learning model is used as a tile self-adaptive quality selector; downloading and locally running a deep learning model by a client, collecting panoramic video watching information from user equipment, obtaining a tile range contained in a future watching view field region of a user through viewpoint prediction, and requesting and obtaining corresponding video content from a remote server according to a quality selection result of the deep learning model; and after the client side obtains the video content, decoding, tile splicing and rendering are carried out on the video content, and the picture is presented to the user. But it ignores the guiding function of QoE in panoramic video transmission and cannot improve the experience of the user in the transmission process. And the prediction complexity by using reinforcement learning is high, and the time consumed by operation is long.
The landlord et al at the university of fuzhou invented a GAN-based panoramic video adaptive streaming method (publication No. CN112616014A), which comprises the following steps: step S1, constructing a time domain similarity graph; step S2, constructing a total network including an encoding network E, a generating network G and a judging network D; step S3, constructing and generating a joint cost function of the code rate and the reconstruction quality of the network G; step S4: inputting the obtained time domain similarity graph into a network, and performing model training to obtain a trained overall network; step S5: at the encoder end, compressing odd frames, extracting the latent codes of even video frames as auxiliary information, combining the latent codes with the compressed odd frame video by using an Mpeg-DASH protocol, and performing dynamic self-adaptive transmission; step S6: at the decoder side, a generator of GAN combines the latent codes of the odd and even video frames to reconstruct the even video frames. However, this method is only considered from the video itself, and does not consider the effect of QoE in panoramic video transmission at all.
People of Beijing post and telecommunications university, such as Wangyumei, have invented a panoramic video adaptive streaming media transmission method and system (publication number: CN112822564A) based on viewpoint, the method includes: the server divides the panoramic video into different tiles in space, encodes a plurality of tile videos into a panoramic video stream file, and packages and slices the panoramic video stream file; and the client predicts the future view field area of the user by using a linear regression method according to the historical viewpoint information of the user watching the panoramic video, and plays the predicted panoramic video. The invention properly enlarges the visual field area according to the deviation degree of the historical prediction, selects high code rate for the tiles in the visual field area, selects low code rate for the tiles in the non-visual field area, and dynamically selects different code rates for different tiles according to the change of network conditions, thereby improving the video definition in the visual field area, reducing the occurrence of the jam condition and effectively improving the watching quality of a user. However, this method is not suitable for multi-view prediction and has a short prediction range, and does not consider the role of QoE in panoramic video transmission.
Disclosure of Invention
In order to avoid the defects existing in the prior art, the invention provides the VR video self-adaptive transmission method based on the visual angle feedback of the limited user, so that the resource utilization rate can be better improved, and the QoE (quality of experience) of multiple users can be improved under the condition of bandwidth limitation.
The invention adopts the following technical scheme for solving the technical problems:
the invention relates to a panoramic video self-adaptive transmission method based on limited user visual angle feedback, which is characterized in that the method is applied to a network environment consisting of a panoramic video server and N clients; the panoramic video server is used for predicting the view angles of all users; transmitting the panoramic video between the panoramic video server and the client through a downlink; and a feedback channel from the client to the panoramic video server is contained in the downlink; the panoramic video self-adaptive transmission method comprises the following steps:
step one, converting the panoramic video from an ERP format into a cube projection format, performing saliency detection on the converted panoramic video to obtain a saliency heat map, converting the saliency heat map into a rectangular projection format, and recording the saliency heat map as a { S }1,S2,…,St,…,St max}；StA saliency heat map representing time t; tmax represents the duration of the panoramic video; t is within [1, t max ∈]；
Step two, generating a small amount of r user views by Gaussian distribution according to historical visual angle information fed back by N users through a feedback channel, and recording the user views as
Figure BDA0003105222030000021
Figure BDA0003105222030000022
The view angle view of the feedback of the R-th user is shown, R is more than or equal to 1 and less than or equal to R and less than or equal to N;
step three, using the method of area covariance to carry out t + t0Video saliency heat map of moments
Figure BDA0003105222030000023
And a small number r of user heatmaps
Figure BDA0003105222030000024
Processing to obtain t + t0Total user perspective of time prediction, written
Figure BDA0003105222030000025
t0Representing the time interval, t + t0∈[1,t max]；
Step four, according to the predicted view angle of the whole user
Figure BDA0003105222030000031
Establishing an objective function with the maximum sum of the quality experience QoE of N clients to be a total utility value, and setting corresponding constraint conditions so as to establish a panoramic video self-adaptive transmission model;
solving the panoramic video self-adaptive transmission model by using a KKT condition and a mixed branch and bound method to obtain a downlink transmission decision variable in the network environment;
step six, the panoramic video server performs blocking processing on the complete panoramic video in time to obtain K frame groups GoF, blocks each frame group GoF in space to obtain M video blocks, and records the M video blocks as { M }1,1,M1,2,...,Mk,m,...,MK,M},Mk,mRepresenting the mth video block of any kth GoF, wherein K is more than or equal to 1 and less than or equal to K; m is more than or equal to 1 and less than or equal to M;
the panoramic video server is the t video block M of the kth frame group GoFk,mD code rate selections are provided for coding processing, so that coded video blocks with D different code rate levels are obtained and recorded as
Figure BDA0003105222030000032
Figure BDA0003105222030000033
T-th video block M representing k-th frame group GoFk,mIs subjected to encoding processingD is more than or equal to 1 and less than or equal to D of the obtained compressed video block with the D-th code rate grade;
seventhly, the panoramic video server predicts the view angles of all the users according to the prediction
Figure BDA0003105222030000034
And the t video block M of the k frame group GoF of the downlink transmissionk,mD-th code rate level decision variable χk,t,dIs selected for any nth client, the tth video block M of the kth frame group GoF is selectedk,mD code rate level of
Figure BDA0003105222030000035
And transmitting to the nth client through a downlink; so that the nth client receives the compressed video blocks with the d code rate grades of the M video blocks;
and step eight, the nth client decodes, maps and renders the compressed video blocks of the M received video blocks at the corresponding code rate levels, thereby synthesizing the panoramic video with the optimized QoE.
The panoramic video adaptive transmission method is also characterized in that the third step is carried out according to the following process:
step 3.1, calculate t + t0Video saliency heat map of moments
Figure BDA0003105222030000036
Each pixel point of
Figure BDA0003105222030000037
A plurality of characteristic values of;
randomly selecting partial feature values to combine, thereby constructing a feature vector phi (I (x, y), x, y) by using the formula (1):
Figure BDA0003105222030000038
step 3.2, heat map of video significance
Figure BDA0003105222030000039
Divided into blocks of Γ × Ψ pixels, denoted as
Figure BDA00031052220300000310
Figure BDA00031052220300000311
Representing a significant heatmap
Figure BDA00031052220300000312
Of the ith pixel block, each pixel block comprising
Figure BDA00031052220300000313
1, i is more than or equal to 1 and less than or equal to gamma multiplied by psi;
structure of utility formula (2)
Figure BDA0003105222030000041
Ith pixel block of pixel
Figure BDA0003105222030000042
Covariance matrix of
Figure BDA0003105222030000043
Figure BDA0003105222030000044
In the formula (2), the reaction mixture is,
Figure BDA0003105222030000045
representing video saliency heat maps
Figure BDA0003105222030000046
The ith pixel block
Figure BDA0003105222030000047
And has:
Figure BDA0003105222030000048
step 3.3, heat map a small number of r users
Figure BDA0003105222030000049
All the pixel blocks are calculated according to the step 3.2 to obtain the ith pixel block in a small number of r user heat maps
Figure BDA00031052220300000410
Of the covariance matrix
Figure BDA00031052220300000411
Sum mean value
Figure BDA00031052220300000412
Figure BDA00031052220300000413
Representing the ith pixel block in the r view
Figure BDA00031052220300000414
The covariance matrix of (a) is determined,
Figure BDA00031052220300000415
representing the ith pixel block in the r view
Figure BDA00031052220300000416
And (c) average of (a);
Figure BDA00031052220300000417
Figure BDA00031052220300000418
construction of video saliency heat map using equation (6)
Figure BDA00031052220300000419
The ith pixel block
Figure BDA00031052220300000420
And a small number r of user heatmaps
Figure BDA00031052220300000421
The ith pixel block
Figure BDA00031052220300000422
Similarity between them
Figure BDA00031052220300000423
Figure BDA00031052220300000424
Step 3.4, similarity
Figure BDA00031052220300000425
Normalized to
Figure BDA00031052220300000426
Construction of t + t Using equation (7)0Temporal predicted total user views
Figure BDA00031052220300000427
Figure BDA00031052220300000428
The fourth step is carried out according to the following processes:
step 4.1, obtaining the quality of experience QoE of the nth client by using the formula (8)n：
Figure BDA0003105222030000051
In the formula (8), θk,m,dA code rate of a video block m representing a kth GoF with a quality level d; thetak,m,DA code rate indicating when the video block m of the kth GoF is transmitted at the highest quality level D;
Figure BDA0003105222030000052
representing video blocks covered within the FoV of the nth client; when xk,m,dWhen the rate is 1, the mth video block representing the kth GoF is transmitted to the client through a downlink at the d code rate levelk,m,dWhen the coding rate is 0, the mth video block representing the kth GoF is not transmitted to the client through a downlink at the d code rate level;
the objective function is constructed using equation (9):
Figure BDA0003105222030000053
formula (9) represents the total utility value;
and 4.2, constructing constraint conditions by using the formulas (10) to (11):
Figure BDA0003105222030000054
Figure BDA0003105222030000055
equation (10) indicates that when the mth video block of any kth GoF is transmitted to the client through the downlink, the transmitted video block can only select one code rate level;
equation (11) represents that the total code rate of all video blocks transmitted does not exceed the bit rate which can be provided by the bandwidth in the whole downlink channel; b iskIndicating the bandwidth of the kth GoF in the downlink channel.
The fifth step is carried out according to the following processes:
step 5.1, carrying out transmission decision variable χ in the panoramic video self-adaptive transmission modelk,m,dPerforming a relaxation operation to obtain [0,1 ]]Continuously transmitting decision variables within the range;
step 5.2 according to the formula(10) -the constraint of formula (11) is
Figure BDA0003105222030000056
Is expressed as a function h (χ)k,t,d) (ii) a Will be provided with
Figure BDA0003105222030000061
Is expressed as a function g (χ)k,m,d) (ii) a Thereby, the lagrangian function L (χ) of the relaxed panoramic video adaptive transmission model is calculated by equation (12)k,m,d,λ,ω)：
Figure BDA0003105222030000062
In equation (12), λ represents the lagrangian coefficient under the inequality constraint in equations (10) and (11), ω represents the lagrangian coefficient under the inequality constraint in equations (10) and (11), and λ represents the function h (χ)k,m,d) Represents the function g (χ) by ωk,m,d) Lagrange coefficients of (a).
Step 5.3, Lagrangian function L (x) according to formula (10)k,m,dλ, ω), obtaining KKT condition of relaxed panoramic video adaptive transmission model as shown in equation (13) -equation (18):
Figure BDA0003105222030000063
g(χk,m,d)≤0 (14)
h(χk,m,d)＝0 (15)
λ≠0 (16)
ω≥0 (17)
ωg(χk,m,d)＝0 (18)
solving the formula (13) -formula (18) to obtain the optimal solution chi of the relaxed panoramic video self-adaptive transmission modelrelaxAnd an optimal total utility value Grelax(ii) a Wherein, the optimal solution χrelaxIs a transmission decision variable χk,t,dThe relaxation optimal solution of (a);
step 5.4, according to the optimal solution chirelaxAnd an optimal total utility value GrelaxAs the initial input parameter of the mixed branch-and-bound method;
step 5.5, defining the branching times in the mixed branch-and-bound method as z, defining the lower bound of the optimal total utility value in the mixed branch-and-bound method as L, and defining the upper bound of the optimal total utility value in the mixed branch-and-bound method as U;
step 5.6, initializing z to be 0;
step 5.7, initializing that L is 0;
step 5.8, initialize U ═ Grelax；
Step 5.9, use chizRepresenting the optimal solution of the z-th branch and recording the corresponding optimal total utility value as GzThen, will xrelaxIs given as XzAnd the optimal solution χ of the z-th branchzAs a root node;
step 5.10, judge χzIf there is a solution not meeting the constraint condition of 0-1, if there is a solution, the X iszThe optimal solution of the relaxation in (1) is divided into a solution satisfying the 0-1 constraint condition and a solution χ not satisfying the 0-1 constraint conditionz(0,1)And executing step 5.12; otherwise, the table is χzThe optimal solution of the non-relaxation panoramic video self-adaptive transmission model;
step 5.11, randomly generating a random number epsilon of the z-th branch in the range of (0,1)zAnd judging that 0 is more than xz(0,1)＜εkWhether the result is true or not; if true, then the constraint "χ" is appliedz(0,1)Adding the obtained value to a non-relaxation panoramic video self-adaptive acquisition and transmission model to form a sub-branch I of the z-th branch; otherwise, the constraint of "χ" is appliedz(0,1)Adding the result 1 into a non-relaxation panoramic video self-adaptive transmission model to form a subbranch II of a z-th branch;
step 5.12, solving relaxation solutions of the sub-branch I and the sub-branch II of the z-th branch by using the KKT condition, and taking the relaxation solutions as the optimal solution chi to the z + 1-th branchz+1And an optimal total utility value Gz+1Wherein x isz+1The method comprises the following steps: (II) relaxation of subbranch I and subbranch II of the z +1 th branch;
step 5.13, judgeOptimum solution χ of branch z +1z+1Whether the constraint condition of 0-1 is met or not, if yes, the optimal total utility value G is obtainedz+1Find the maximum value and assign it to L, and χz+1E {0,1 }; otherwise, from the optimal total utility value Gz+1Find the maximum value and assign it to U, and χz+1∈(0,1)；
Step 5.14, judge Gz+1If < L is true; if yes, cutting off the optimal solution χ of the z +1 th branchz+1Assigning z +1 to z in the branch, and returning to the step 5.10; otherwise, executing step 5.15;
step 5.15, judge Gz+1If L is more than true, assigning z +1 to z, and returning to the step 5.10; otherwise, executing step 5.16;
step 5.16, judge Gz+1If the two-dimensional model is true, the optimal solution x of the z +1 th branch is the optimal solution x of the non-relaxation panoramic video adaptive transmission modelz+1And will be xz+1Assigning an optimal solution χ to a non-relaxed panoramic video adaptive transmission model0-1Will Xz+1Corresponding Gz+1Assigning an optimal total utility value G to a non-relaxed panoramic video adaptive transmission model0-1(ii) a Otherwise, after z +1 is assigned to z, the procedure returns to step 5.10.
Compared with the prior art, the invention has the beneficial effects that:
1. the invention predicts the FoV of all users by extracting video salient features and a small amount of user visual angle feedback information through the FoV prediction method of the panoramic video, and optimizes the QoE of multiple users in the panoramic video transmission process, thereby better improving the QoE of the total users in the panoramic video transmission process.
2. The invention combines the streaming media self-adaptive transmission of the panoramic video with the QoE of multiple users, and provides a method for optimizing the streaming media transmission of the panoramic video by taking the total QoE of the system as a transmission guide factor, thereby better guiding and optimizing the streaming media transmission process of the panoramic video.
3. The method solves the proposed panoramic video self-adaptive transmission model by applying the KKT condition and the mixed branch and bound method, improves the efficiency and the accuracy of the solution, and further improves the high efficiency of the panoramic video visual angle prediction and transmission method.
Drawings
Fig. 1 is an application scene diagram of a streaming media adaptive transmission method of panoramic video proposed in the present invention;
fig. 2 is a system structure diagram of the adaptive transmission method proposed in the present invention.
Detailed Description
In this embodiment, a self-adaptive transmission method based on limited user perspective feedback is applied to a multi-user network scene, as shown in fig. 1, where a panoramic video server and N clients exist in the network scene. The panoramic video server and the user side are transmitted through a downlink; a feedback channel from a user end to the panoramic video server is contained in a downlink; the feedback channel can feed back the real-time visual angle information and the downlink bandwidth information of the user to the server to help the server to carry out acquisition and transmission work. As shown in fig. 2, the method specifically includes the following steps:
step 1, converting the format of the panoramic video from ERP format to cube projection format, and performing significance detection on the converted panoramic video to obtain significance heat map and convert the significance heat map to rectangular projection format, and recording the significance heat map as { S }1,S2,…,St,…,St max}；StA saliency heat map representing time t; tmax represents the duration of the panoramic video; t is within [1, t max ∈]；
Step 2, generating a small amount of r user views by Gaussian distribution according to the visual angle information fed back by the N users through the feedback channel, and recording the user views as
Figure BDA0003105222030000081
Figure BDA0003105222030000082
The view angle view of the feedback of the R-th user is shown, and R is more than or equal to 1 and less than or equal to R and less than or equal to N;
step 3, using the method of area covariance to carry out t + t0Video saliency of momentsSexual heat map
Figure BDA0003105222030000083
And a small number r of user heatmaps
Figure BDA0003105222030000084
Processing to obtain t + t0The total user perspective of the time prediction is recorded as
Figure BDA0003105222030000085
t0Representing the time interval, t + t0∈[1,t max]；
Step 3.1, calculate t + t0Video saliency heat map of moments
Figure BDA0003105222030000086
Each pixel point of
Figure BDA0003105222030000087
A plurality of characteristic values of;
randomly selecting partial characteristic values to combine so as to construct a characteristic vector by using the formula (1)
Figure BDA0003105222030000088
Figure BDA0003105222030000089
Step 3.2, heat map of video significance
Figure BDA00031052220300000810
Divided into blocks of Γ × Ψ pixels, denoted as
Figure BDA0003105222030000091
Figure BDA0003105222030000092
Representing a significant heatmap
Figure BDA0003105222030000093
Of the ith pixel block, each pixel block comprising
Figure BDA0003105222030000094
1, i is more than or equal to 1 and less than or equal to gamma multiplied by psi;
structure of utility formula (2)
Figure BDA0003105222030000095
Ith pixel block of pixel
Figure BDA0003105222030000096
Covariance matrix of
Figure BDA0003105222030000097
Figure BDA0003105222030000098
In the formula (2), the reaction mixture is,
Figure BDA0003105222030000099
representing video saliency heat maps
Figure BDA00031052220300000910
The ith pixel block
Figure BDA00031052220300000911
And has:
Figure BDA00031052220300000912
step 3.3, heat map a small number of r users
Figure BDA00031052220300000913
All the calculation steps are the same as the step 3.2, and the ith pixel block in a small quantity of r user heatmaps is obtained
Figure BDA00031052220300000914
Covariance matrix of
Figure BDA00031052220300000915
Sum mean value
Figure BDA00031052220300000916
Figure BDA00031052220300000917
Representing the ith pixel block in the r view
Figure BDA00031052220300000918
The covariance matrix of (a) is determined,
Figure BDA00031052220300000919
representing the ith pixel block in the r view
Figure BDA00031052220300000920
And (c) average of (a);
Figure BDA00031052220300000921
Figure BDA00031052220300000922
construction of video saliency heat maps using equation (4)
Figure BDA00031052220300000923
The ith pixel block
Figure BDA00031052220300000924
And a small number r of user heatmaps
Figure BDA00031052220300000925
The ith pixel block
Figure BDA00031052220300000926
Similarity between them
Figure BDA00031052220300000927
Figure BDA00031052220300000928
Step 3.4, similarity
Figure BDA00031052220300000929
Normalized to
Figure BDA00031052220300000930
Construction of t + t Using equation (5)0Temporal predictive universal user perspective
Figure BDA00031052220300000931
Figure BDA0003105222030000101
Step 4, according to the predicted view angle of the whole user
Figure BDA0003105222030000102
Establishing an objective function with the maximum sum of the quality experience QoE of N clients to be a total utility value, and setting corresponding constraint conditions so as to establish a panoramic video self-adaptive transmission model;
step 4.1, obtaining the quality of experience QoE of the nth client by using the formula (5)n：
Figure BDA0003105222030000103
In the formula (5), θk,m,dA code rate of a video block m representing a kth GoF with a quality level d; theta.theta.k,m,DA code rate indicating when the video block m of the kth GoF is transmitted at the highest quality level D;
Figure BDA0003105222030000104
is shown asVideo blocks covered within the FoV of the n clients; when xk,m,dWhen the rate is 1, the mth video block representing the kth GoF is transmitted to the client through a downlink at the d code rate levelk,m,dWhen the coding rate is 0, the mth video block representing the kth GoF is not transmitted to the client through a downlink at the d code rate level;
the objective function is constructed using equation (6):
Figure BDA0003105222030000105
formula (6) represents the total utility value;
and 4.2, constructing constraint conditions by using the formulas (6) to (7):
Figure BDA0003105222030000106
Figure BDA0003105222030000107
equation (7) indicates that when the mth video block of any kth GoF is transmitted to the client through the downlink, the transmitted video block can only select one code rate level;
the expression (8) shows that the total code rate of all video blocks transmitted does not exceed the bit rate which can be provided by the bandwidth in the whole downlink channel; bkIndicating the bandwidth of the kth GoF in the downlink channel.
Step 5, solving the panoramic video self-adaptive transmission model by using a KKT condition and a mixed branch and bound method to obtain a downlink transmission decision variable in a network environment;
step 5.1, carrying out transmission decision variable χ in the panoramic video self-adaptive transmission modelk,m,dPerforming a relaxation operation to obtain [0,1 ]]Continuously transmitting decision variables within the range;
step 5.2, according to the constraint conditions of the formula (6) to the formula (7), adding
Figure BDA0003105222030000111
Is expressed as a function h (χ)k,t,d) (ii) a Will be provided with
Figure BDA0003105222030000112
Is expressed as a function g (χ)k,m,d) (ii) a Thereby, the lagrangian function L (χ) of the relaxed panoramic video adaptive transmission model is calculated by equation (10)k,m,d,λ,ω)：
Figure BDA0003105222030000113
In the formula (8), λ represents a lagrangian coefficient under the inequality constraint condition in the formula (6) to the formula (7), μ represents a lagrangian coefficient under the inequality constraint condition in the formula (6) to the formula (7), and λ represents a function h (χ)k,m,d) Represents the function g (χ) by ωk,m,d) Lagrange coefficients of (d).
Step 5.3, Lagrangian function L (x) according to the formula (8)k,m,dλ, ω), obtaining KKT condition of relaxed panoramic video adaptive transmission model as shown in formula (10) -formula (15):
Figure BDA0003105222030000114
g(χk,m,d)≤0 (12)
h(χk,m,d)＝0 (13)
λ≠0 (14)
ω≥0 (15)
ωg(χk,m,d)＝0 (16)
the Lagrangian function L (χ) is expressed by the equations (11) and (12)k,m,dλ, ω) is taken as a necessary condition for the extremum; the expressions (13) and (14) represent the function h (χ)k,m,d),g(χk,m,d) The constraint of (2); equation (15) represents a constraint condition of the lagrangian coefficient λ, ω; the formula (16) represents a complementary relaxation condition.
Solving the formula (11) -formula (16) to obtain the optimal solution chi of the relaxed panoramic video self-adaptive transmission modelrelaxAnd an optimal total utility value Grelax(ii) a Wherein, the optimal solution χrelaxIs a transmission decision variable χk,m,dThe relaxation optimal solution of (a);
step 5.4, according to the optimal solution chirelaxAnd an optimal total utility value GrelaxAs the initial input parameter of the mixed branch-and-bound method;
step 5.5, defining the branching times in the mixed branch-and-bound method as z, defining the lower bound of the optimal total utility value in the mixed branch-and-bound method as L, and defining the upper bound of the optimal total utility value in the mixed branch-and-bound method as U;
step 5.6, initializing z to be 0;
step 5.7, initializing that L is 0;
step 5.8, initialize U ═ Grelax；
Step 5.9, use chizRepresenting the optimal solution of the z-th branch and recording the corresponding optimal total utility value as GzThen, will xrelaxIs given as XzAnd taking the optimal solution χ of the z-th branchzAs a root node;
step 5.10, judge χzIf there is a solution not meeting the constraint condition of 0-1, if there is a solution, the X iszThe optimal solution of the relaxation in (1) is divided into a solution satisfying the 0-1 constraint condition and a solution χ not satisfying the 0-1 constraint conditionz(0,1)And executing step 5.12; otherwise, the table is χzThe optimal solution of the non-relaxation panoramic video self-adaptive transmission model;
step 5.11, randomly generating a random number epsilon of the z-th branch in the range of (0,1)zAnd judging that 0 < χz(0,1)＜εkWhether the result is true; if true, then the constraint "χ" is appliedz(0,1)Adding the obtained value to a non-relaxation panoramic video self-adaptive acquisition and transmission model to form a sub-branch I of the z-th branch; otherwise, the constraint of "χ" is appliedz(0,1)Adding the result 1 into a non-relaxation panoramic video self-adaptive transmission model to form a subbranch II of a z-th branch;
step 5.12, solving relaxation solutions of the sub-branch I and the sub-branch II of the z-th branch by using the KKT condition, and taking the relaxation solutions as the optimal solution chi to the z + 1-th branchz+1And an optimal total utility valueGz+1Wherein x isz+1The method comprises the following steps: (II) relaxation of subbranch I and subbranch II of the z +1 th branch;
step 5.13, judging the optimal solution chi of the z +1 th branchz+1Whether the constraint condition of 0-1 is met, if so, the optimal total utility value G is obtainedz+1Find the maximum value and assign it to L, and χz+1E {0,1 }; otherwise, from the optimal total utility value Gz+1Find the maximum value and assign it to U, and χz+1∈(0,1)；
Step 5.14, judge Gz+1If < L is true; if true, the optimal solution χ of the z +1 th branch is clippedz+1Assigning z +1 to z in the branch, and returning to the step 5.10; otherwise, executing step 5.15;
step 5.15, judge Gz+1If L is more than true, assigning z +1 to z, and returning to the step 5.10; otherwise, executing step 5.16;
step 5.16, judge Gz+1If the two-dimensional model is true, the optimal solution x of the z +1 th branch is the optimal solution x of the non-relaxation panoramic video adaptive transmission modelz+1And will be xz+1Assigning an optimal solution χ to a non-relaxed panoramic video adaptive transmission model0-1Will Xz+1Corresponding Gz+1Assigning an optimal total utility value G to a non-relaxed panoramic video adaptive transmission model0-1(ii) a Otherwise, after z +1 is assigned to z, the step 5.10 is returned.
And 6, the panoramic video server performs block processing on the complete panoramic video in time to obtain K frame groups GoF, and performs block processing on each frame group GoF in space to obtain M video blocks which are marked as { M1,1,M1,2,...,Mk,m,...,MK,M},Mk,mRepresenting the mth video block of any kth GoF, wherein K is more than or equal to 1 and less than or equal to K; m is more than or equal to 1 and less than or equal to M;
the panoramic video server is the t video block M of the k frame group GoFk,mD code rate selections are provided for coding processing, so that coded video blocks with D different code rate levels are obtained and recorded as
Figure BDA0003105222030000131
Figure BDA0003105222030000132
T-th video block M representing k-th frame group GoFk,mD is more than or equal to 1 and less than or equal to D of the compressed video block with the D-th code rate grade obtained after the coding processing;
step 7, the panoramic video server predicts the view angles of all users according to the predicted view angles
Figure BDA0003105222030000133
And the t video block M of the k frame group GoF of the downlink transmissionk,mD type code rate grade decision variable xk,t,dIs selected for any nth client, the tth video block M of the kth frame group GoF is selectedk,mD code rate level of
Figure BDA0003105222030000134
And transmitting to the nth client through a downlink; so that the nth client receives the compressed video block with the d code rate grade of the M video blocks;
and 8, the nth client decodes, maps and renders the compressed video blocks of the M received video blocks at the corresponding code rate levels, so as to synthesize the panoramic video with the optimized QoE.

Claims (3)

1. A panoramic video self-adaptive transmission method based on limited user visual angle feedback is characterized by being applied to a network environment consisting of a panoramic video server and N clients; the panoramic video server is used for predicting the view angles of all users; transmitting the panoramic video between the panoramic video server and the client through a downlink; and a feedback channel from the client to the panoramic video server is contained in the downlink; the panoramic video self-adaptive transmission method comprises the following steps:
step one, converting the panoramic video from an ERP format into a cubic projection format, and performing saliency detection on the converted panoramic video to obtain a saliency heat mapConverted into rectangular projection format, denoted as { S1,S2,…,St,…,Stmax}；StA significance heatmap representing time t; tmax represents the duration of the panoramic video; t ∈ [1, tmax)]；
Step two, generating a small amount of r user views by Gaussian distribution according to historical visual angle information fed back by N users through a feedback channel, and recording the user views as
Figure FDA0003493397270000011
Figure FDA0003493397270000012
The view angle view of the feedback of the R-th user is shown, R is more than or equal to 1 and less than or equal to R and less than or equal to N;
step three, using the method of area covariance to process t + t0Video saliency heat map of moments
Figure FDA0003493397270000013
And a small number r of user views
Figure FDA0003493397270000014
Processing to obtain t + t0Total user perspective of time prediction, written
Figure FDA0003493397270000015
t0Representing the time interval, t + t0∈[1,tmax]；
Step 3.1, calculate t + t0Video saliency heat map of moments
Figure FDA00034933972700000122
Each pixel point of
Figure FDA0003493397270000016
A plurality of characteristic values of;
randomly selecting partial feature values to combine, thereby constructing a feature vector phi (I (x, y), x, y) by using the formula (1):
Figure FDA0003493397270000017
step 3.2, heat map of video significance
Figure FDA00034933972700000121
Divided into blocks of Γ × Ψ pixels, denoted as
Figure FDA0003493397270000018
Figure FDA0003493397270000019
Representing a significant heatmap
Figure FDA00034933972700000110
Of the ith pixel block, each pixel block comprising
Figure FDA00034933972700000111
1, i is more than or equal to 1 and less than or equal to gamma multiplied by psi;
structure of utility formula (2)
Figure FDA00034933972700000112
Ith pixel block of pixel
Figure FDA00034933972700000113
Covariance matrix of
Figure FDA00034933972700000114
Figure FDA00034933972700000115
In the formula (2), the reaction mixture is,
Figure FDA00034933972700000116
representing video saliency heat maps
Figure FDA00034933972700000117
The ith pixel block
Figure FDA00034933972700000118
And has:
Figure FDA00034933972700000119
step 3.3, heat map a small number of r users
Figure FDA00034933972700000120
All the pixel blocks are calculated according to the step 3.2 to obtain the ith pixel block in a small number of r user heat maps
Figure FDA0003493397270000021
Covariance matrix of
Figure FDA0003493397270000022
Sum mean value
Figure FDA0003493397270000023
Figure FDA0003493397270000024
Representing the ith pixel block in the r view
Figure FDA0003493397270000025
The covariance matrix of (a) is determined,
Figure FDA0003493397270000026
representing the ith pixel block in the r view
Figure FDA0003493397270000027
And (c) average of (a);
Figure FDA0003493397270000028
Figure FDA0003493397270000029
construction of video saliency heat map using equation (6)
Figure FDA00034933972700000210
The ith pixel block
Figure FDA00034933972700000211
And a small number r of user heatmaps
Figure FDA00034933972700000212
The ith pixel block
Figure FDA00034933972700000213
Similarity between them
Figure FDA00034933972700000214
Figure FDA00034933972700000215
Step 3.4, similarity
Figure FDA00034933972700000216
Normalized to
Figure FDA00034933972700000217
Construction of t + t Using equation (7)0Temporal predictive universal user perspective
Figure FDA00034933972700000220
Figure FDA00034933972700000218
Step four, according to the predicted all-user visual angle
Figure FDA00034933972700000219
Establishing a target function with the maximum sum of the quality experience QoE of N clients to be a total utility value, and setting corresponding constraint conditions, thereby establishing a panoramic video self-adaptive transmission model;
solving the panoramic video self-adaptive transmission model by using a KKT condition and a mixed branch and bound method to obtain a downlink transmission decision variable in the network environment;
step six, the panoramic video server performs blocking processing on the complete panoramic video in time to obtain K frame groups GoF, blocks each frame group GoF in space to obtain M video blocks, and records the M video blocks as { M }1,1,M1,2,...,Mk,m,...,MK,M},Mk,mRepresenting the mth video block of any kth GoF, wherein K is more than or equal to 1 and less than or equal to K; m is more than or equal to 1 and less than or equal to M;
the panoramic video server is the t video block M of the kth frame group GoFk,mD code rate selections are provided for coding processing, so that coded video blocks with D different code rate levels are obtained and recorded as
Figure FDA0003493397270000031
Figure FDA0003493397270000032
T-th video block M representing k-th frame group GoFk,mD is more than or equal to 1 and less than or equal to D of the compressed video block with the D-th code rate grade obtained after the coding processing;
seventhly, the panoramic video server predicts the view angles of all the users according to the predicted view angles
Figure FDA0003493397270000033
And the t video block M of the k frame group GoF of the downlink transmissionk,mD type code rate grade decision variable xk,t,dFor any nth client, select the tth video block M of the kth frame group GoFk,mD code rate level of
Figure FDA0003493397270000034
And transmitting to the nth client through a downlink; so that the nth client receives the compressed video blocks with the d code rate grades of the M video blocks;
and step eight, the nth client decodes, maps and renders the compressed video blocks of the M received video blocks at the corresponding code rate levels, thereby synthesizing the panoramic video with the optimized QoE.
2. The adaptive transmission method for panoramic video according to claim 1, wherein the fourth step is performed as follows:
step 4.1, obtaining the quality of experience QoE of the nth client by using the formula (8)n：
Figure FDA0003493397270000035
In the formula (8), θk,m,dA code rate of a video block m representing a kth GoF with a quality level d; thetak,m,DA code rate indicating when the video block m of the kth GoF is transmitted at the highest quality level D;
Figure FDA0003493397270000036
representing video blocks covered within the FoV of the nth client; when xk,m,dWhen the rate is 1, the mth video block representing the kth GoF is transmitted to the client through a downlink at the d code rate levelk,m,dWhen the coding rate is 0, the mth video block representing the kth GoF is not transmitted to the client through a downlink at the d code rate level;
the objective function is constructed using equation (9):
Figure FDA0003493397270000037
formula (9) represents the total utility value;
and 4.2, constructing constraint conditions by using the formulas (10) to (11):
Figure FDA0003493397270000041
Figure FDA0003493397270000042
equation (10) indicates that when the mth video block of any kth GoF is transmitted to the client through the downlink, the transmitted video block can only select one code rate level;
equation (11) indicates that the total code rate of all video blocks transmitted does not exceed the bit rate that the bandwidth in the whole downlink channel can provide; bkIndicating the bandwidth of the kth GoF in the downlink channel.
3. The adaptive panoramic video transmission method according to claim 1, wherein the fifth step is performed as follows:
step 5.1, carrying out transmission decision variable χ in the panoramic video self-adaptive transmission modelk,m,dPerforming a relaxation operation to obtain [0,1 ]]Continuously transmitting decision variables within the range;
step 5.2, according to the constraint conditions of the formula (10) to the formula (11), the
Figure FDA0003493397270000043
Is recorded as a function h (χ)k,t,d) (ii) a Will be provided with
Figure FDA0003493397270000044
Is expressed as a function g (χ)k,m,d) (ii) a Thereby calculating the relaxation after the relaxation by the equation (12)Lagrange function L (χ) of panoramic video adaptive transmission modelk,m,d,λ,ω)：
Figure FDA0003493397270000045
In equation (12), λ represents the lagrangian coefficient under the inequality constraint in equations (10) and (11), ω represents the lagrangian coefficient under the inequality constraint in equations (10) and (11), and λ represents the function h (χ)k,m,d) Represents the function g (χ) by ωk,m,d) Lagrange coefficients of (a);
step 5.3, Lagrangian function L (x) according to formula (10)k,m,dλ, ω), obtaining KKT condition of relaxed panoramic video adaptive transmission model as shown in equation (13) -equation (18):
Figure FDA0003493397270000046
g(χk,m,d)≤0 (14)
h(χk,m,d)＝0 (15)
λ≠0 (16)
ω≥0 (17)
ωg(χk,m,d)＝0 (18)
solving the formula (13) -formula (18) to obtain the optimal solution chi of the relaxed panoramic video self-adaptive transmission modelrelaxAnd an optimal total utility value Grelax(ii) a Wherein the optimal solution χrelaxIs a transmission decision variable χk,t,dThe relaxation optimal solution of (a);
step 5.4, according to the optimal solution chirelaxAnd an optimal total utility value GrelaxAs the initial input parameter of the mixed branch-and-bound method;
step 5.5, defining the branching times in the mixed branch-and-bound method as z, defining the lower bound of the optimal total utility value in the mixed branch-and-bound method as L, and defining the upper bound of the optimal total utility value in the mixed branch-and-bound method as U;
step 5.6, initializing z to be 0;
step 5.7, initializing that L is 0;
step 5.8, initialize U ═ Grelax；
Step 5.9, use chizThe optimal solution of the z-th branch is represented, and the corresponding optimal total utility value is recorded as GzThen, will xrelaxIs given as XzAnd taking the optimal solution χ of the z-th branchzAs a root node;
step 5.10, judge χzIf there is a solution not meeting the constraint condition of 0-1, if there is a solution, the X iszThe optimal solution of the relaxation in (1) is divided into a solution satisfying the 0-1 constraint condition and a solution χ not satisfying the 0-1 constraint conditionz(0,1)And executing step 5.12; otherwise, the table is χzThe optimal solution of the non-relaxation panoramic video self-adaptive transmission model;
step 5.11, randomly generating a random number epsilon of the z-th branch in the range of (0,1)zAnd judging that 0 < χz(0,1)＜εkWhether the result is true or not; if true, then the constraint "χ" is appliedz(0,1)Adding the obtained value to a non-relaxation panoramic video self-adaptive acquisition and transmission model to form a sub-branch I of the z-th branch; otherwise, the constraint of "χ" is appliedz(0,1)Adding the result 1 into a non-relaxation panoramic video self-adaptive transmission model to form a subbranch II of a z-th branch;
step 5.12, solving relaxation solutions of the sub-branch I and the sub-branch II of the z-th branch by using the KKT condition, and taking the relaxation solutions as the optimal solution chi to the z + 1-th branchz+1And an optimal total utility value Gz+1Therein xz+1The method comprises the following steps: (II) relaxation of subbranch I and subbranch II of the z +1 th branch;
step 5.13, judging the optimal solution chi of the z +1 th branchz+1Whether the constraint condition of 0-1 is met, if yes, the optimal total utility value G is obtainedz+1Finding outMaximum value is given and assigned to L, and χz+1E {0,1 }; otherwise, from the optimal total utility value Gz+1Find the maximum value and assign it to U, and χz+1∈(0,1)；
Step 5.14, judge Gz+1If < L is true; if true, the optimal solution χ of the z +1 th branch is clippedz+1Assigning z +1 to z in the branch, and returning to the step 5.10; otherwise, executing step 5.15;
step 5.15, judge Gz+1If L is more than true, assigning z +1 to z, and returning to the step 5.10; otherwise, executing step 5.16;
step 5.16, judge Gz+1If the two-dimensional model is true, the optimal solution x of the z +1 th branch is the optimal solution x of the non-relaxation panoramic video adaptive transmission modelz+1And will be xz+1Assigning an optimal solution χ to a non-relaxed panoramic video adaptive transmission model0-1Will Xz+1Corresponding Gz+1Assigning an optimal total utility value G to a non-relaxed panoramic video adaptive transmission model0-1(ii) a Otherwise, after z +1 is assigned to z, the step 5.10 is returned.
CN202110634899.2A 2021-06-08 2021-06-08 Panoramic video self-adaptive transmission method based on limited user visual angle feedback Active CN113329266B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110634899.2A CN113329266B (en) 2021-06-08 2021-06-08 Panoramic video self-adaptive transmission method based on limited user visual angle feedback

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110634899.2A CN113329266B (en) 2021-06-08 2021-06-08 Panoramic video self-adaptive transmission method based on limited user visual angle feedback

Publications (2)

Publication Number Publication Date
CN113329266A CN113329266A (en) 2021-08-31
CN113329266B true CN113329266B (en) 2022-07-05

Family

ID=77419934

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110634899.2A Active CN113329266B (en) 2021-06-08 2021-06-08 Panoramic video self-adaptive transmission method based on limited user visual angle feedback

Country Status (1)

Country Link
CN (1) CN113329266B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114640870B (en) * 2022-03-21 2023-10-03 陕西师范大学 QoE-driven wireless VR video self-adaptive transmission optimization method and system
CN114827750B (en) * 2022-05-31 2023-12-22 脸萌有限公司 Viewing angle prediction method, device, equipment and storage medium
CN115022546B (en) * 2022-05-31 2023-11-14 咪咕视讯科技有限公司 Panoramic video transmission method, device, terminal equipment and storage medium
CN116996661B (en) * 2023-09-27 2024-01-05 中国科学技术大学 Three-dimensional video display method, device, equipment and medium

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110266714A (en) * 2019-06-28 2019-09-20 合肥工业大学 VR video under a kind of QoE driving adaptively acquires and transmission method
CN111866601A (en) * 2020-07-21 2020-10-30 中国科学技术大学 Cooperative game-based video code rate decision method in mobile marginal scene

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108235131B (en) * 2018-01-30 2020-07-03 重庆邮电大学 Panoramic video self-adaptive transmission method based on DASH
US10735778B2 (en) * 2018-08-23 2020-08-04 At&T Intellectual Property I, L.P. Proxy assisted panoramic video streaming at mobile edge
WO2020069976A1 (en) * 2018-10-01 2020-04-09 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Concepts for improved head motion prediction and efficient encoding of immersive video
CN110166850B (en) * 2019-05-30 2020-11-06 上海交通大学 Method and system for predicting panoramic video watching position by multiple CNN networks
CN112468806B (en) * 2020-11-12 2022-07-26 中山大学 Panoramic video transmission optimization method for cloud VR platform
CN112616014B (en) * 2020-12-09 2022-03-15 福州大学 GAN-based panoramic video adaptive streaming transmission method
CN112822564B (en) * 2021-01-06 2023-03-24 鹏城实验室 Viewpoint-based panoramic video adaptive streaming media transmission method and system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110266714A (en) * 2019-06-28 2019-09-20 合肥工业大学 VR video under a kind of QoE driving adaptively acquires and transmission method
CN111866601A (en) * 2020-07-21 2020-10-30 中国科学技术大学 Cooperative game-based video code rate decision method in mobile marginal scene

Also Published As

Publication number Publication date
CN113329266A (en) 2021-08-31

Similar Documents

Publication Publication Date Title
CN113329266B (en) Panoramic video self-adaptive transmission method based on limited user visual angle feedback
CN110266714B (en) QoE-driven VR video self-adaptive acquisition and transmission method
CN110225341B (en) Task-driven code stream structured image coding method
US7830961B2 (en) Motion estimation and inter-mode prediction
US4791486A (en) Recursive image encoding/decoding using interpolation within variably sub-divided picture areas
CN110310343B (en) Image processing method and device
CN101584215B (en) Integrated spatial-temporal prediction
EP2343878A1 (en) Pixel prediction value generation procedure automatic generation method, image encoding method, image decoding method, devices using these methods, programs for these methods, and recording medium on which these programs are recorded
KR20220145405A (en) An image encoding/decoding method and apparatus for performing feature quantization/inverse quantization, and a recording medium for storing a bitstream
JP2005294977A (en) Two-path video encoding method and system using sliding window
KR102012906B1 (en) Systems and methods for processing a block of a digital image
CN113141526B (en) Point cloud video self-adaptive transmission method for joint resource allocation under QoE (quality of experience) drive
CN111556315A (en) Data transmission method and device
CN116803082A (en) Device and method for point cloud processing
JP4151963B2 (en) Image server device, client device, moving image distribution method, program, and information recording medium
WO2021092821A1 (en) Adaptively encoding video frames using content and network analysis
EP3284255A1 (en) Encoding a plurality of signals with data rate target of a signal depending on complexity information
US20230281881A1 (en) Video Frame Compression Method, Video Frame Decompression Method, and Apparatus
JPH10200894A (en) Image signal transformation method
US20210176477A1 (en) Electronic apparatus and control method thereof
JP6055268B2 (en) Color conversion device, color restoration device, and program thereof
CN107888917B (en) Image coding and decoding method and device
CN116996661B (en) Three-dimensional video display method, device, equipment and medium
CN117440162B (en) Multimedia interactive teaching method and system
Zhao et al. High efficient VR video coding based on auto projection selection using transferable features

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant