CN114422868A - Cloud VR video playing system based on edge computing and storing - Google Patents

Abstract

The invention discloses a Cloud VR video playing system based on edge computing storage, which comprises hardware equipment and a control system, wherein the hardware equipment comprises a VR helmet, an edge computing storage server and a central computing Cloud storage server (a client host), and the control flow of the control system is as follows: s1, an edge calculation storage server performs mapping calculation on an ERP panorama by adopting a rectangular pyramid projection mode, S2, a client host acquires and dynamically predicts three-dimensional motion data of a user VR helmet in real time and calculates user visual angle information, and finally, the user visual angle information is transmitted to the edge calculation storage server, and a Cloud VR transmission scheme based on visual angle and edge storage is adopted, so that the network transmission bandwidth can be reduced by more than 60% under the same resolution, and the resolution of a video received by a user can be improved by nearly three times under the condition of a certain bandwidth.

Description

Cloud VR video playing system based on edge computing and storing

Technical Field

The invention relates to the technical field of VR (virtual reality), in particular to a Cloud VR (virtual reality) video playing system based on edge computing and storing.

Background

The evolution of VR to Cloud VR has become a necessary trend, Cloud VR supporting Cloud rendering has great value in the aspects of improving user experience, reducing VR cost consumed by users, protecting VR content copyright, popularizing VR business scenes and the like, and after Cloud VR carries out calculation rendering cloudization, due to the huge data volume of VR, a transmission part becomes the bottleneck of the whole system, so that the data volume transmission is reduced.

The invention provides a Cloud VR video playing system based on edge computing storage, which has the advantages that great redundancy exists in the completely transmitted panoramic image, but the viewing experience is greatly influenced only by the fact that the image in the transmission view field is delayed or is shielded black due to network delay and jitter.

Disclosure of Invention

The invention aims to provide a Cloud VR video playing system based on edge calculation and storage, so as to solve the problem that images in a transmission view field are delayed or blacked due to network delay and jitter, which is proposed in the background art.

In order to achieve the purpose, the invention provides the following technical scheme:

a Cloud VR video playing system based on edge computing storage comprises hardware equipment and a control system, wherein the hardware equipment comprises a VR helmet, an edge computing storage server and a central computing Cloud storage server (client host), and the control flow of the control system is as follows:

s1, an edge calculation storage server performs mapping calculation on an ERP panorama by adopting a rectangular pyramid projection mode;

s2, the client host acquires and dynamically predicts three-dimensional motion data of the VR helmet of the user in real time, calculates user visual angle information, and finally transmits the user visual angle information to an edge calculation storage server;

and S3, the edge computing server selects a corresponding version video GOP according to the user visual angle information and sends the video GOP back to the client host, and the client host computes a visual angle switching corresponding viewpoint position movement vector and transmits the image to the VR helmet.

As a preferred embodiment of the present invention, S1 includes that when the ERP panorama is projected by a pyramid, the generated graph is a square panorama.

As a preferred embodiment of the present invention, the S2 includes the following steps:

s21, acquiring a view matrix of the VR helmet, and setting rotation only and inversion transposition processing;

s22, establishing a space coordinate system according to a right-hand system, and calculating a three-dimensional direction vector of a current playing frame visual angle;

the S23, Fov direction vectors are passed to the edge compute storage server.

As a preferred embodiment of the present invention, the S3 includes the following steps:

s31, calculating the corresponding version number according to the Fov direction vector;

s32, sending the next video GOP to the client host according to the serial number and the current playing pointer, and sending the video GOP in a format that each GOP has 10 frames;

and S33, judging whether the viewpoint version switching occurs in the front and back frames.

Compared with the prior art, the invention has the beneficial effects that:

the Cloud VR video playing system can reduce the required bandwidth by more than 60%, meanwhile, the viewing quality in the visual field range is not damaged, and the situations of image lag caused by network delay and image loss caused by network jitter are effectively avoided.

Drawings

FIG. 1 is a three-dimensional right-hand coordinate system diagram of the present embodiment;

FIG. 2 is a three-dimensional coordinate transformation diagram according to the present embodiment;

FIG. 3 is a flow chart of the present invention;

FIG. 4 is a schematic diagram illustrating view-based dynamic prediction according to the present embodiment;

FIG. 5 is a mapping of a pyramid/pyramid according to the present embodiment;

FIG. 6 is a view point division diagram of the present embodiment 30;

FIG. 7 is a diagram of an edge calculation storage frame according to the present embodiment;

fig. 8 is a diagram of the transmission scheme Fov in this embodiment.

Fig. 9 is a schematic diagram of a rectangular pyramid/pyramid map according to the present embodiment.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Examples

Referring to fig. 1-9, the present invention provides a technical solution:

a Cloud VR video playing system based on edge computing storage comprises hardware equipment and a control system, wherein the hardware equipment comprises a VR helmet, an edge computing storage server and a central computing Cloud storage server (a client host), the system is described by dividing by taking longitude and latitude of the earth as an example, a viewpoint is selected from 0 degree on the equator at intervals of 30 degrees, and 12 viewpoint versions are arranged on the equator; selecting a viewpoint every 60 degrees from 0 degrees at the 30 degrees of the north and south dimensions, wherein 6 viewpoints are respectively arranged at the 30 degrees of the north and south dimensions; selecting a viewpoint every 180 degrees from 0 degrees on the 90 degrees of the north and south dimensions, wherein 2 viewpoints are respectively arranged on the 90 degrees of the north and south dimensions; finally, the north and south poles are respectively a viewpoint, and thus 12+6+6+2+ 1+1 is composed of 30 different viewpoint versions, the viewpoint versions are numbered and divided, as shown in fig. 6, 0,1, 2 are numbered sequentially from 0 longitude to the right-handed system direction from the north pole to the south pole, and the edge storage frame of the invention is a server-client one-to-one form, but is not limited to this, the number of clients can be increased to form a one-to-many form, a plurality of edge computing memories can be arranged on the basis to communicate with the clients, and a central server is used for controlling the operation flow of the control system as follows:

s1, an edge calculation storage server performs mapping calculation on an ERP panorama in a rectangular pyramid projection mode, each panorama is divided into 30 different FOV versions according to viewpoint projection and stored, namely, each panoramic video has 30 video versions with different FOVs in the edge storage server, a square panorama is generated when the ERP panorama performs rectangular pyramid projection, the projection effect is shown in figure 5, namely, the rectangular panorama is mapped into the square panorama, and the process is as follows:

s11, judging the current plane for any pixel point G (m, n) of the plane image;

s12, determining a corresponding mapping coordinate according to which face a pixel belongs to, and converting a two-dimensional coordinate G (m, n) into a three-dimensional coordinate P (x, y, z) in a cubic space;

s13, obtaining a spherical coordinate M of the point in a spherical coordinate system according to a mapping formula;

s14, reversely deducing the position N of a point in the spherical coordinate system in the image according to a formula generated by the spherical panorama;

and S15, judging whether the coordinate of the N is an integer, directly obtaining the pixel value of the N point and assigning the pixel value to the cubic plane image pixel point if the coordinate of the N is the integer, or obtaining the pixel value of the N point by utilizing bidirectional linear interpolation and assigning the pixel value to the cubic plane image pixel point.

S2, a client host acquires and dynamically predicts three-dimensional motion data of a VR helmet of a user in real time and calculates user visual angle information, finally, the user visual angle information is transmitted to an edge calculation storage server to perform dynamic visual angle prediction based on machine learning, a navigation position estimation method is adopted to retain the viewpoint three-dimensional data of the previous frame and perform linear prediction on the viewpoint three-dimensional data of the next frame with the viewpoint three-dimensional data of the current frame, and the specific flow is as follows:

s21, obtaining a view matrix (m _ mat4HMDPose) of the VR helmet, and performing corresponding rotate-only and related inversion and transposition processing, wherein the processing form is as follows:

m_mat4HMDPose.setColumn(3,Vector4(0,0,0,1))；

m_mat4HMDPose.invert()；

m_mat4HMDPose.transpose()；

the view matrix is used for converting a world coordinate system of the object into a camera coordinate system, and the view matrix is a rotation matrix and a translation matrix;

the computer internal rendering establishes a spatial right-hand coordinate system, namely the right side is an x axis, the back side is a z axis, the upper side is a y axis, and a view matrix under right-hand system/left-hand system is as follows:

(Right-hand lower view matrix)

(left hand series lower view matrix)

eye (eyeX, eyeY, eyeZ) is the viewpoint, i.e. the position of the camera in the world coordinate system;

at (atX, atY, atZ) is the observation point, i.e., the observed target point, pointing in the direction of the camera;

up (upX, upY, upZ) is top-facing, i.e., determines which direction the camera is directly above;

zaxis is the direction vector from the viewpoint, i.e. the direction vector from at to eye;

yaxis and Xaxis are the direction vectors of the y and x direction axes of the world coordinate system;

the view matrix is rotated-only, so that the camera only performs rotation operation and does not perform operations such as translation and zooming, and as can be seen from the above, the right-hand system only needs to set (0,0, 0,1) to the ground 4 columns of the view matrix, the right-hand system needs to perform transposition on the view matrix, and the left-hand system does not need to perform the operation;

s22, a space coordinate system is established according to a right-hand system, a three-dimensional direction vector of a current playing frame view angle is calculated, according to c2, a Zaxis vector is the three-dimensional direction vector of the view angle required to be obtained, the operation of firstly negating and then multiplying by (0,0,1) is carried out on a view matrix, and the first three components of the third row of the lower view matrix of the right-hand system are taken out to form a new three-dimensional Fov direction vector: vector3 v3(0,0,1), Fov ═ m _ mat4HMDPose) × v 3;

and S23, transmitting the Fov direction vector to an edge calculation storage server, acquiring a view matrix of the current viewpoint in real time, and performing dynamic view prediction based on machine learning by adopting a navigation position prediction method.

S3, the edge calculation server selects a corresponding version video GOP according to the user visual angle information and sends the video GOP back to the client host, the client host calculates a visual angle switching corresponding viewpoint position movement vector and transmits an image to the VR helmet, the visual angle direction of the previous frame and the viewpoint center of the current frame are obtained first, and the viewpoint center of the current frame is moved to be overlapped with the visual angle direction of the previous frame, and the specific flow is as follows:

s31, calculating corresponding version numbers according to Fov direction vectors, wherein the latitude degrees of south and north are 15 degrees, 45 degrees and 75 degrees are limits, the longitude line is limited by the longitude line in the middle of two horizontal viewpoints, the viewpoint moves within 15 degrees of south and north dimensions and within 15 degrees of left and right longitude, viewpoint switching operation does not occur, and if the viewpoint moves from 0 longitude dimension 0 to 16 longitude dimension 0, the viewpoint should be switched to the version with 30 longitude dimension 0 as the viewpoint center;

s32, as shown in fig. 8, sending the next video GOP to the client host according to the number and the current play pointer, dividing each video stream into GOPs of one second, and implementing an adaptive bit rate stream related to the view for playback, each second (or less), determining the stream to be acquired next according to the network condition of the user, the view port direction and the time of the next GOP, running multiple iterative tests in a controlled network environment, recording the angular distance from the view port center to the viewer direction, the time spent in each view port, the view port resolution and switching delay, the buffer time, and the network bandwidth.

S33, judging whether viewpoint version switching occurs on the front frame and the rear frame, if the viewpoint version switching occurs, calculating a viewpoint position movement vector corresponding to the viewpoint switching, moving the viewpoint of the current frame in a corresponding direction, and finally transmitting an image to the VR helmet, wherein the process is as shown in FIG. 2, three-dimensional data is subjected to model matrix transformation, projection matrix transformation and view matrix transformation and is finally rendered on the VR helmet, wherein in FIG. 2:

ModelViewmatrix is used as a model matrix;

ProjectionMatrix is a projection matrix;

the viewport transformation is a view matrix;

model view matrix.

The specific flow of d3 is as follows:

1. firstly, acquiring the view angle direction of a previous frame and the viewpoint center of a current frame;

frameViewpoint.phi＝(90-frameViewPointTable[viewPointIndex][0])*M_PI/180；

frameViewpoint.theta＝frameViewPointTable[viewPointIndex][1]*M_PI/180；

2. and moving the viewpoint center of the current frame to be coincident with the view angle direction of the previous frame.

float sin_phi＝sin(frameViewpoint.phi)；

float cos_phi＝cos(frameViewpoint.phi)；

float sin_theta＝sin(frameViewpoint.theta)；

float cos_theta＝cos(frameViewpoint.theta)；

#define M(row,col)mModel[col*4+row]

M(0,0)＝cos_theta；

M(0,1)＝sin_theta*sin_phi；

M(0,2)＝sin_theta*cos_phi；

M(0,3)＝0.0；

M(1,0)＝0.0；

M(1,1)＝cos_phi；

M(1,2)＝-sin_phi；

M(1,3)＝0.0；

M(2,0)＝-sin_theta；

M(2,1)＝cos_theta*sin_phi；

M(2,2)＝cos_theta*cos_phi；

M(2,3)＝0.0；

M(3,0)＝0.0；

M(3,1)＝0.0；

M(3,2)＝0.0；

M(3,3)＝1.0；

#undef M

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (4)

1. A Cloud VR video playback system based on edge computing storage, comprising: the system comprises hardware equipment and a control system, wherein the hardware equipment comprises a VR helmet, an edge computing storage server and a central computing cloud storage server (a client host), and the control flow of the control system is as follows:

s1, an edge calculation storage server performs mapping calculation on an ERP panorama by adopting a rectangular pyramid projection mode;

s2, the client host acquires and dynamically predicts three-dimensional motion data of the VR helmet of the user in real time, calculates user visual angle information, and finally transmits the user visual angle information to an edge calculation storage server;

and S3, the edge computing server selects a corresponding version video GOP according to the user visual angle information and sends the video GOP back to the client host, and the client host computes a visual angle switching corresponding viewpoint position movement vector and transmits the image to the VR helmet.

2. The Cloud VR video playback system based on edge computing storage as claimed in claim 1, wherein: and S1, when the rectangular pyramid projection is carried out on the ERP panorama, the generated graph is a square panorama.

3. The Cloud VR video playback system based on edge computing storage as claimed in claim 1, wherein: the S2 includes the following steps:

s21, acquiring a view matrix of the VR helmet, and setting rotation only and inversion transposition processing;

s22, establishing a space coordinate system according to a right-hand system, and calculating a three-dimensional direction vector of a current playing frame visual angle;

the S23, Fov direction vectors are passed to the edge compute storage server.

4. The Cloud VR video playback system based on edge computing storage as claimed in claim 1, wherein: the S3 includes the following steps:

s31, calculating the corresponding version number according to the Fov direction vector;

s32, sending the next video GOP to the client host according to the serial number and the current playing pointer, and sending the video GOP in a format that each GOP has 10 frames;

and S33, judging whether the viewpoint version switching occurs in the front and back frames.

Images