CN106296819A - A kind of panoramic video player based on Intelligent set top box - Google Patents

Abstract

The invention discloses a kind of playing panoramic video on Intelligent set top box or intelligent television, by using each two field picture of panoramic video stream as the texture of OpenGL, be then mapped on a sphere, then this sphere is carried out matrixing project on screen.Using the incoming angle information of remote controller of built-in gyroscope, this angle information is converted into spin matrix, is then engage in matrixing, the video pictures that thus can control on screen moves, it is achieved that the multi angle view of panoramic video.Viewing panoramic video need not wear virtual implementing helmet in this way, provides cost savings, and has the advantages such as light, the applicable many people viewing of operation.

Description

A kind of panoramic video player based on Intelligent set top box

Technical field

The present invention relates to a kind of panoramic video player based on Intelligent set top box, belong to computer graphics and Android multimedia technology field.

Background technology

Panoramic video, also known as 360 degree of videos, as it is shown in figure 1, static panoramic pictures is converted into dynamic video figure by it Picture, panoramic video can about shooting angle upper and lower 360 degree arbitrarily watch dynamic video, make people have one truly Sensation on the spot in person, and do not limited by time, space and region.Panoramic video is mainly used in virtual reality neck at present Territory, does not the most apply on Intelligent set top box or on intelligent television, but viewing panoramic video is compared on mobile phone on TV See with virtual reality glasses, although lacked sensation on the spot in person, but there is the advantage of some uniquenesses, such as need not wear heaviness Virtual reality glasses, screen is bigger, the details seen is more rich, be available for many people viewing etc. simultaneously.

Blender is a cross-platform animation soft increased income, and can be modeled, makes animation, material, regard The software of editing frequently, compares with the three-dimensional animation Rendering software of the specialty of 3dMax, and Blender is the smallest and the most exquisite, but Blender Function the most powerful, the basic work that renders the most enough meets the demand of three-dimensional animation production fan.The design Player used in 3D sphere model will use Blender draw.

Intelligent set top box, refers generally to the Set Top Box equipped with Android operation system, and it can install third party's Android should Use program.Connect on TV, it is possible to achieve the function that android system is abundant, as listened music, seeing video, surf the Net, play game Deng.

With the remote controller of gyroscope, gyroscope is a kind of angular-rate sensor, and the action rotated, deflect can be done very by it Good measurement, thus can judge the actual act of user with Accurate Analysis, then realizes corresponding operation.For convenience Mutual with user, the Intelligent TV remote controller of the most many manufacturers is the most built-in gyroscope, in also having many on market simultaneously Put the universal remote control of gyroscope, various Intelligent set top box, intelligent television etc. can be supported.

Panoramic video is mainly used in field of virtual reality at present, is watched by virtual reality glasses on smart mobile phone.This Although kind of a mode can bring sensation on the spot in person, but there is many defects, such as need to put on virtual implementing helmet, for a long time Viewing is the most tired, dizziness；After additionally putting on the helmet, user cannot touch mobile phone again, there is mutual difficult problem.

Summary of the invention

Present invention aim at: overcome the deficiencies in the prior art, panoramic video is applied on TV, is not limited solely to intelligence Energy mobile phone, saves the added burden of mobile phone terminal virtual reality glasses viewing panoramic video.The present invention provides a kind of by intelligence Set Top Box watches the solution of panoramic video on (intelligent) TV, and player is arranged on Intelligent set top box or intelligent television On, then use the remote controller of band gyroscope to control the movement of panoramic video.Viewing panoramic video need not be worn in this way Wear virtual implementing helmet, provide cost savings, and there is the advantages such as light, the applicable many people viewing of operation.

The purpose of the present invention is achieved through the following technical solutions:

Due to panoramic video be omnibearing angel shooting, therefore can using each two field picture of panoramic video as The texture of OpenGL is mapped on a sphere, then this sphere is carried out matrixing projects on screen.Then band top is used The incoming angle information of remote controller of spiral shell instrument controls the movement of panoramic video.

Described each two field picture of panoramic video stream is mapped on a sphere as the texture of OpenGL, then by this ball Face carries out matrixing and projects to being implemented as follows on screen:

(1) by MediaPlayer and SurfaceView with the use of forming video flowing；

(2) receive video flowing with SurfaceTexture, and each two field picture in video flowing is mapped to as texture Sphere；

(3) the vertex position coordinate of sphere being carried out matrixing, the vertex position obtaining being eventually displayed on screen is sat Mark.

Being implemented as follows of the movement of the remote controller incoming angle information control panoramic video of use band gyroscope:

(1) angle information incoming for gyroscope remote controller is converted into the form of matrix:

(2) this matrix is participated in matrixing as spin matrix, change final transformation matrix in real time；

(3) projection of spheroid is controlled by final transformation matrix, it is achieved the real-time of video pictures moves.

The scheme that the present invention is concrete is described as follows:

(1) first have to play video flowing, used here as the MediaPlayer class in android, MediaPlayer class Be mainly used in play audio frequency, video to be play, also need with Surface class with the use of.After but Surface receives video flowing Can directly show on screen, it has not been convenient to video is carried out relevant treatment, to use another class SurfaceTexture here, It need not be directly displayed on screen after receiving video flowing, and it can obtain picture frame as OpenGL ES from video flowing Texture, texture content can be updated to picture nearest in video flowing, Jing Guoyi by its updateTexImage () method After kind processing, then SurfaceTexture and Surface is associated, is finally shown on screen.

(2) after acquisition picture frame is as texture, by texture and TextureUnit (the texture list in OpenGL rendering pipeline Unit) carry out binding association.TextureUnit can read texture and pass to shader (tinter), and such texture could be Show on screen.

(3) 3 d modeling software Blender is used to create a spheroid, as shown in Figure 2.The 3D sphere model modeled Ultimately form is the file of an obj form, and when loading this model file in a program, loading is a heap data, These data mainly contain vertex information and the texture information of model, and its form is as follows:

v 1.950948-17.654135-2.919814

……

vt 0.654635 0.062500

……

vn -0.062400 0.771500 0.633200

……

f 214/209/198 229/224/198 228/225/198

……

Ellipsis therein represents that data below is the data that the data type that several rows are identical is constituted, and the main distinction is Data are different, and data type referred herein refers to the letter of every a line foremost." v " beginning is expressed as Vertex, the most just Apex coordinate, after three data separated with space be the X on summit, Y, Z coordinate；" vt " beginning is expressed as VertexTexture, it is simply that texture coordinate, data are respectively S, T coordinate；" vn " beginning is expressed as VertexNormal, data For the normal vector on the summit component on each coordinate axes；" f " beginning is expressed as Face, and namely a face, is previously mentioned In OpenGL ES, each face is made up of 3 summits, and therefore this data line is made up of three groups of data, separates with space, each The group packet information containing a summit, the tables of data of the inside is shown as the line number of data directory, such as: f 214/209/214 229/224/229 228/225/228 are expressed as the apex coordinate constituting three summits in a face from representing apex coordinate 214,229 and 228 row in data row (" v " starts), texture coordinate is from representing texture coordinate data row (" vt " starts) 209,224 and 225 row, vertex scheme vectorial coordinate is from 241,229 and the 228 of representation vectorial coordinate data row (" vn " starts) OK.When reading apex coordinate and texture coordinate, it is necessary to use the index information in face, then go to deposit apex coordinate and texture is sat Target data pick up coordinate and stores, finally sending into OpenGL ES rendering pipeline.

(4) 2 d texture is mapped to sphere.When using the texture mapping function of OpenGL ES, vertex shader (Vertex Shader) sends into the vertex attribute of rendering pipeline and mainly includes vertex position coordinate and texture coordinate, wherein, position Coordinate is usually three-dimensional coordinate (X, Y, Z), and texture coordinate is two-dimensional coordinate (S, T).In OpenGL ES, three-dimensional coordinate uses Be right-handed coordinate system, as shown in a of accompanying drawing 3, texture coordinate direction be level to the right for S, be T vertically downward, such as accompanying drawing 3 Shown in b.

When triangle sticker texture in a of accompanying drawing 3 to be, first (X, Y, Z) and (S, T) feeding summit on three summits Tinter and sheet unit tinter, then picture whole in the b of accompanying drawing 3 is sent into texture cell as texture, OpenGL ES renders pipe Line can be mapped in triangular texture interior in the b of accompanying drawing 3 in the triangle in a of accompanying drawing 3.If the size of two trianglees is not Time the same, OpenGL ES rendering pipeline can carry out stretching scaling and process, and programmer needs that does simply to arrange stretching in a program With filtering mode, reasonably stretching and filtering operation can make up some patches caused due to the size defect of texture itself The decline of figure effect.

What the present invention built is a three-dimensional sphere model, but spliced panoramic picture be still that one two-dimentional Image, will be mapped to a two dimensional image on sphere, need to carry out topological transformation.First is the face of cylinder spherical transform, then Carry out topological transformation again, by the face of cylinder along it high line cut formed a rectangle.Carry out the figure after topological transformation, top Connecting line attribute between number and the summit of point is all not changed in, and only these limit dough-making powder has been done translation, has rotated Or zoom operations.So, after a spherical transform is a rectangular planes, texture maps will seem very simple , texture coordinate is the most very clear.

(5) before final display, being also performed to the conversion on some visual angles, because spheroid is three-dimensional, screen is two dimension , needing to project on screen by spheroid, this is realized by a series of matrixing.Become when three-dimensional scenic or object are done When changing, mainly the vertex position describing this object is done corresponding conversion, then OpenGL ES is sent on new summit and render Pipeline, when OpenGL ES rendering pipeline carries out frame renewal when, can draw new three-dimensional scenic or object.Typical matrix Convert schematically as follows:

It is abbreviated as Q=MP

The homogeneous coordinates of Q point after the leftmost side is conversion in above-mentioned linear algebra expression formula, centre for 4 × 4 conversion square Battle array, right side is the homogeneous coordinates of the front P point of conversion.When the element in matrix M takes suitable value, it is special that equation Q=MP just has it Different geometric meaning.Such as: the some P in three dimensions can be translated, rotates or zoom to a Q.The concrete letter of these conversion Breath just leaves in matrix M, and the most commonly referred to as matrix M is transformation matrix.When needs continuously perform a series of conversion, successively Transformation matrix is multiplied by the homogeneous coordinates vector representing some position.Such as translation transformation:

$\left(\begin{array}{c}{Q}_x\\ Q_y\\ Q_z\\ 1\end{array}\right)=\left(\begin{array}{cccc}1& 0& 0& m_x\\ 0& 1& 0& m_y\\ 0& 0& 1& m_z\\ 0& 0& 0& 1\end{array}\right)\×\left(\begin{array}{c}{P}_x\\ P_y\\ P_z\\ 1\end{array}\right)=\left(\begin{array}{c}{P}_x+m_x\\ P_y+m_y\\ P_z+m_z\\ 1\end{array}\right)$

It practice, OpenGL ES is when converting, being not to do for object to convert, conversion is coordinate system in fact, Incoming rendering pipeline remain original apex coordinate.This is one method the most cleverly, if sat all of summit Mark does matrix calculus, it will produce the hugest amount of calculation, if converted coordinate system, it is only necessary to a step calculates.

(6) sensor is used to change visual angle.In order to be controlled the movement of video by gyroscope, need to register one and rotate arrow Quantity sensor, obtain is numerically data coordinate axes and angle hybrid obtained, and it is mainly used to expression and sets The direction of standby rotation.Three output data of rotating vector can be shown by following formula:

$\begin{array}{ccc}X\×sin\left(\frac{\mathrm{\θ}}{2}\right)& Y\×sin\left(\frac{\mathrm{\θ}}{2}\right)& Z\×sin\left(\frac{\mathrm{\θ}}{2}\right)\end{array}$

Now rotating vector size isThe direction of rotating vector is consistent with the direction of rotating shaft.

Three output results of rotating vector are actually rear three expression formulas of quaternary number.The output valve of rotating vector It is without unit.Referenced coordinate system is defined as an orthonormal basis (such as accompanying drawing 4).This coordinate system has following spy Levy:

1, Y value is the tangent value that equipment is presently in position and magnetic north.

2, X value is defined as the vector product of Y Yu Z value.I.e. equipment be presently in the ground of position and east to tangent value.

3, Z value is towards sky direction, perpendicular to the ground crossing.

Next rotating vector data are converted into the form of matrix, namely spin matrix, typical spin matrix Schematically as follows:

$\left(\begin{array}{cccc}cos\mathrm{\θ}+(1-\cos \mathrm{\θ})u_x^2& (1-\cos \mathrm{\θ})u_y{u}_x-{\mathrm{sin\θu}}_z& (1-cos\mathrm{\θ})u_z{u}_x+{\mathrm{sin\θu}}_y& 0\\ (1-cos\mathrm{\θ})u_x{u}_y+{\mathrm{sin\θu}}_z& cos\mathrm{\θ}+(1-cos\mathrm{\θ})u_y^2& (1-cos\mathrm{\θ})u_z{u}_y-{\mathrm{sin\θu}}_x& 0\\ (1-cos\mathrm{\θ})u_x{u}_z-{\mathrm{sin\θu}}_y& (1-\cos \mathrm{\θ})u_y{u}_z+{\mathrm{sin\θu}}_x& cos\mathrm{\θ}+(1-cos\mathrm{\θ})u_z^2& 0\\ 0& 0& 0& 1\end{array}\right)$

Above-mentioned matrix represents around axial vector u, the some P that specifies is rotated θ degree, and expression u therein vector is in X, Y, Z axis Component.

Then this spin matrix is also assisted in matrixing, finally by the matrix that obtains and ball position coordinate to Amount is multiplied, and has just obtained final position on screen, the summit.So by sensor real-time update data, also with regard to real-time update Transformation matrix, the effect obtained is exactly that panoramic video is moved on screen.

The present invention having the beneficial effects that compared with prior art:

(1) direct viewing panoramic video on TV, eliminates heavy expensive virtual implementing helmet, has saved cost；

(2) utilize remote controller to control the movement of panoramic video picture, and without rotary head, the very comfortable that operates is convenient； Long-time viewing there will not be spinning sensation, feeling of fatigue；

(3), on the TV of giant-screen, many people can watch together, heighten the enjoyment；

(4) use the remote controller can quick mobile video picture, it is simple to check specific details in video, can apply Some special occasions, such as checking monitoring video recording.

Accompanying drawing explanation

Fig. 1 is panoramic video schematic diagram；

Fig. 2 is OpenGL three-dimensional sphere schematic diagram；

Fig. 3 is that texture maps schematic diagram；A is object plane, and b is texture picture；

Fig. 4 is the coordinate system of OpenGL ES；

Fig. 5 panoramic video player workflow diagram.

Detailed description of the invention

Below according to summary of the invention, in conjunction with Figure of description, the present invention is specifically described:

Step one: use blender to create a spheroid.

OpenGL only supports graphical pointv, line and triangle, so the multiaspect that spheroid is made up of a series of triangular apex Body, when these summits are abundant, this polyhedron is approximately a spheroid.After using blender to create sphere model, will Its obj file is placed under the Resource TOC of Android project, reads data therein when of use.

Step 2: create and initialize player.

One MediaPlayer of instantiation, uses its setDataSource () method to obtain video source, and video source is permissible It is Internet video, or local video.Then arranging Surface is video container, hides title bar and arranges player and be Full frame.

Step 3: obtain video stream image frame as texture.

Create, generate texture ID, instantiation SurfaceTexture, receive video flowing with it, then obtain in video flowing Picture frame as texture, be tied on the texture ID that generates, use GL_TEXTURE_EXTERNAL_OES as texture mesh Mark, then arranges texture filtering mode.Finally with the SurfaceTexture of instantiation as parameter, create what previous step used Surface。

Step 4: update texture.

Video owing to playing is not picture, so needing to update texture.Use SurfaceTexture's Texture content is updated to picture nearest in video flowing by updateTexImage () method, and thus can make in video is each Two field picture all as texture, is rendered by the rendering pipeline of OpenGL after treatment, finally shows.

Step 5: registration gyro sensor, arranges spin matrix.

First create a sensor manager (SensorManager), it generate rotating vector sensor, then make With function getRotationMatrixFromVector () by sensor sensing to rotating vector change information be converted into four The form of dimension matrix, i.e. spin matrix, this matrix will be for spinning ball model.

Step 6: carry out matrix projection conversion, more New Century Planned Textbook.

Need the ModelMatrix (model matrix, coordinate system is centered by object) used, ViewMatrix (visual angle square Battle array, coordinate system is centered by eyes or video camera), RotationMatrix (spin matrix is generated by sensor values), and ProjectionMatrix (projection matrix, the projection of three dimensions to two-dimensional space).Wherein:

Visual angle matrix V iewMatrix arrange function be Matrix.setLookAtM (mViewMatrix, 0, eyeX, eyeY,eyeZ,lookX,lookY,lookZ,upX,upY,upZ)；

The function that arranges of projection matrix ProjectionMatrix is Matrix.frustumM (mProjectionMatrix,0,left,right,bottom,top,near,far)；

Spin matrix RotationMatrix is generated by sensing data.

By these matrix multiples, just obtain final transformation matrix: MVPMatrix.This matrix is for sitting summit Mark converts.So, when the numerical value real-time update of sensor, final effect have updated transformation matrix exactly MVPMatrix。

Step 7: compiling link coloring process.

Vertex Shader (vertex shader) program write due to OpenGL ES shading language and Fragment Shader (fragment shader) program can not be performed by java procedure identification, and therefore two kinds of coloration program can be placed on another In resource folder, getVertexShader () and getFragmentShader () method is used to come from this resource folder Read code, save as String form.After having read code, use glCompileShader () function to compile, then make By one rendering program program of glCreateProgram () function creation, then will compile by glAttachShader () method The coloring process and the program that translate bind, and finally link this program by glLinkProgram () method.

Step 8: read data from obj model file.

GetResources () the .openRawResource () method using Android SDK to provide is opened and reads money In the data of source file to inlet flow, next resolve this inlet flow.First distribute memory space, be respectively created Store the Dynamic Array of the character string type of vertex point coordinate information, texture coordinate information dough-making powder index information, then according to data Type is stored in corresponding Dynamic Array respectively.

Split (" ") function of the String class in Java can be split character string according to specific character, point Data after cutting will be stored in array the most in order, such as: row data are 21,4/2,09/,198 22,9/2,24/,198 228/ 225/198, the data after segmentation are placed in array faceComponent, then first split by space, store in array j, j [0]=214/209/198, here it is form the line index information on first summit in this face, then j [0] is split by "/", that FaceComponent [0]=214, faceComponent [1]=209, then first apex coordinate is at 214 row, texture Coordinate is at 209 row.It is necessary to remove corresponding local taking-up coordinate after obtaining index information, first use parseInt () function word Symbol string is converted to numeral, and the vertex coordinates data taken out is 1.950948-17.654135-2.919814, then enters by space , there are three numerals in array in row segmentation, finally uses parseFloat () function to be converted into float type variable i.e. in order Can.Finally, also the data of acquisition are fed in internal memory, because rendering of OpenGL ES completes in GPU, but GPU cannot obtain data from program main thread, and GPU can obtain the data of needs from internal memory, so we need Data are sent in order in internal memory and use for GPU.

Step 9: broadcast data to tinter.

Reading obj model data to after internal memory, following GPU to use these data.First statement uses program to make For coloring process, obtain vertex coordinates data, texture, texture coordinate data and transformation matrix (MVPMatrix) isoparametric Handle, then use glVertexAttribPointer () function and glEnableVertexAttribArray () function to The data of Vertex Shader these parameters incoming.Dependent variable data is passed to Fragment by Vertex Shader again Shader, Fragment Shader can carry out rendering pixel-by-pixel with the method for linear interpolation.

Step 10: play video, drawing image.

The start () method using MediaPlayer plays video, after glDrawArrays () the method drafting of use renders Each frame video image, and show on screen.

Finally use gyroscope remote controller to control panoramic video to move up and down.Thus achieve panoramic video at machine Broadcasting on top box and control, the workflow of player is as shown in Figure 5.

Claims (3)

1. a panoramic video player based on Intelligent set top box, it is characterised in that:

(1) each two field picture of panoramic video stream is mapped on a sphere as the texture of OpenGL, then this sphere is entered Row matrix conversion projects on screen；

(2) the incoming angle information of remote controller using band gyroscope controls the movement of panoramic video.

Panoramic video player based on Intelligent set top box the most according to claim 1, it is characterised in that: described step (1) each two field picture of panoramic video stream is mapped on a sphere as the texture of OpenGL, then this sphere is carried out square Battle array converts and projects to being implemented as follows on screen:

(11) by MediaPlayer and SurfaceView with the use of forming video flowing；

(12) receive video flowing with SurfaceTexture, and each two field picture in video flowing is mapped to ball as texture Face；

(13) the vertex position coordinate of sphere is carried out matrixing, obtain the vertex position coordinate being eventually displayed on screen.

Panoramic video player based on Intelligent set top box the most according to claim 1, it is characterised in that: described step (2) it is implemented as follows:

(21) angle information incoming for gyroscope remote controller is converted into the form of matrix:

(22) this matrix is participated in matrixing as spin matrix, change final transformation matrix in real time；

(23) projection of spheroid is controlled by final transformation matrix, it is achieved the real-time of video pictures moves.