CN107835436B - A kind of real-time virtual reality fusion live broadcast system and method based on WebGL - Google Patents
A kind of real-time virtual reality fusion live broadcast system and method based on WebGL Download PDFInfo
- Publication number
- CN107835436B CN107835436B CN201710872854.2A CN201710872854A CN107835436B CN 107835436 B CN107835436 B CN 107835436B CN 201710872854 A CN201710872854 A CN 201710872854A CN 107835436 B CN107835436 B CN 107835436B
- Authority
- CN
- China
- Prior art keywords
- video
- model
- virtual reality
- scene
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/643—Communication protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8543—Content authoring using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML]
Abstract
A kind of real-time virtual reality fusion live broadcast system and method based on WebGL proposed by the present invention, the system and method realize a kind of effective ways of real-time virtual reality fusion at the end Web using WebGL interface.The system is made of 5 modules: Video Model generation module, net cast module, GIS service module, virtual reality fusion module and scene executor module.The system realizes the display effect of the real-time virtual reality fusion of multiple video strems on the end Web, improves the matching accuracy that virtual reality fusion is shown, has demand resource few, cross-platform compatibility is good, the strong advantage of scalability.
Description
Technical field
The invention belongs to the technical field of virtual reality in computer vision, relate generally to a kind of based on the real-time of WebGL
Virtual reality fusion live broadcast system and method.
Background technique
With the development of computer graphics techniques and Internet technology, traditional two dimensional image is not able to satisfy modern gradually
To the needs for showing and understanding scene, and the appearance of virtual reality technology compensates for this disadvantage.Therefore, in recent years, virtually
Reality technology and its related fields are quickly grown, and are widely used in simulated training, video monitoring, city roaming, military affairs demonstration, scape
The fields such as area's displaying.
Virtual three-dimensional scene can allow user to obtain more three-dimensional and true experience.When virtual scene environmental information with
True environment information is more similar, and the sense of reality that user obtains is higher, and virtual scene is able to use family more preferably, more freely
Corresponding real scene is solved, the feeling of immersion and experience sense of scene are increased.But since the model in three-dimensional virtual scene can only
Indicate the stationary state at some moment, user can not recognize this by observing three-dimensional scene models static in this way
The dynamic change of scene.It is just the opposite, although two-dimensional video image cannot show the three-dimensional feeling that threedimensional model has, energy
Enough actual changes than more accurately recording scene in a period of time, people are recognized that the dynamic of scene becomes by video
Change.
If three-dimensional virtual environment to be combined to foundation enhancing virtual environment with two-dimensional video image, three can either be promoted
The information content that dimension module contains, making threedimensional model more has the variation of real world, the richer sense of reality, and can give expression to than view
The more wide spatial dimension of frequency image allows users to the model appearance using virtual scene, spatial distribution, picture dynamic etc.
Information further understands the content of video image and the time-space relationship of each video image, mitigates the cognition pressure of user.It is early
In 1996, Moezzi was (referring to Moezzi S, Katkere A, Kuramura D Y, et al.Reality modeling
and visualization from multiple video sequences[J].Computer Graphics and
Applications, IEEE, 1996,16 (6): 58-63) etc. just propose the concept that shows of three-dimensional scenic fusion video, they make
The object in movement is captured with the camera at multiple and different visual angles, and it is rebuild, then dynamically by the Model Fusion of reconstruction
Into virtual scene.The Neumann of University of Southern California in 2003 is (referring to NeumannU, et al.Augmented Virtual
Environments(AVE):for Visual-ization of Dynamic Imagery[C].IEEE Virtual
Reality 2003.2003:61-67) et al. be put forward for the first time enhancing virtual environment (Augmented Virtual
Environment, AVE) this concept, has further developed video source modeling virtual scene technology, has obtained virtual reality fusion and shown
Effect.Neumann et al. realizes the enhancing virtual environment of multiple campus scenes, then the video data of acquisition is projected to pair
In the buildings model and landform answered, the dynamic 3 D model with image change is formd.Chen (the ginseng of Taiwan Univ. in 2012
See Chen S C, Lee C Y, Lin C W, et al.2D and 3D visualization with dual-resolution
for surveillance.Proceedingsof IEEE Computer Society Conference on Computer
Vision and Pattern Recognition Workshops, Providence, 2012.23-30) etc. establish use
The Visualization Framework of GIS auxiliary, the view of multiple and different resolution cameras is merged with threedimensional model, realizes more points
The monitoring strategies of resolution.
At the same time, it is played an increasingly important role based on the Internet technology of Web in all trades and professions, wherein Web
The trend for merging even more current Technological research of technology and virtual reality technology: virtual three dimensional field is realized by Internet technology
Scape allows users to access virtual scene by browser, can not only break through the limitation of regional space, additionally it is possible to more convenient
It quickly brings and stays indoors and virtual scene on the spot in person experience to user.2011, multimedia technology standard group
It knits Khronos and formally issues WebGL standard criterion, it can be only by html script, not by any browser plug-in
In the case where, realize the drafting of Web interactive three-dimensional scene；By what is unified, standard, cross-platform OpenGL interface, benefit
The acceleration of figure rendering is carried out with the graphic hardware of bottom, enhancing virtual environment may be implemented in desktop browsers in WebGL technology
The even drafting of mobile phone terminal, the succinct of the end Web efficiently also greatly facilitate the enhancing propagation of virtual environment and practical, this is exactly empty
One important trend in quasi- reality technology future.
But in general, the multiple video strems virtual reality fusion method of current main-stream is in the prevalence of picture distortion, syncretizing effect
The problems such as difference and excessive actual situation alignment cost.And since the available resource in the end Web is limited, good syncretizing effect is realized more
It is not able to achieve the end lightweight Web so existing virtual reality fusion the relevant technologies are mostly based on desktop client realization for difficulty
Virtual reality fusion system.The generally existing system requirements of current virtual reality fusion system is excessively high, Video Rendering expend resource excessively and
The problems such as difficult in maintenance, limits virtual reality fusion system in the extensive use of browser or even mobile terminal.
Summary of the invention
Technology of the invention solves the problems, such as: overcoming the deficiencies of the prior art and provide a kind of real-time void based on WebGL technology
Real fusion live broadcast system and method, overcome the excessive problem of current multiple video strems virtual reality fusion cost, improve virtual reality fusion
The cross-platform compatibility of method.
Technical solution of the invention: a kind of real-time virtual reality fusion live broadcast system based on WebGL, the actual situation are melted
Conjunction live broadcast system by holding offline, server end and client are constituted, and affixes one's name to Video Model generation module in offline end；In server
End deploys net cast module and GIS service module；Client deployment virtual reality fusion module and scene executor module:
Video Model generation module: reading the real time monitoring video image or local video image of monitoring camera acquisition,
The file of binary format description is generated using single width image modeling technolog, the binary format file includes Video Model top
Point coordinate data and camera parameter information；Using its camera parameter information calculate Video Model texture projective transformation matrix and
Watch the best camera view pose of the Video Model, then the Video Model relevant information combination Video Model that will be obtained
The conversion of WebGL rendering parameter, the Video Model file for being stored as the identifiable JSON file format of client browser.Described
Video Model file by Video Model vertex coordinates data, Video Model camera posture information, initial texture pictorial information and
WebGL spatial cue composition；The WebGL spatial cue includes that Video Model vertex data format, projection matrix, video are dynamic
State texture information and coloration program；Video Model generation module and server end are asynchronous, generate Video Model text offline
Part, finally by the Video Model of generation for the calling of GIS service module；
Net cast module: the real time monitoring video image or local video image of monitoring camera acquisition are received, and will
It handles and stores；To the corresponding view of net cast module request when client virtual reality fusion module draws Video Model
When frequency image is as dynamic texture, net cast module to client virtual reality fusion module forwards, supplies corresponding video image
It is used；If virtual reality fusion module request is local video, net cast module directly forwards local video；If virtual reality fusion
Module request is real time monitoring video flowing, and net cast module receives pushing away in real time for network monitoring camera head using RTMP agreement
Stream, and transcoding, fragment processing are carried out to live video stream, ultimately generate the video profile of m3u8 format and the view of ts format
Frequency slicing files are pushed to client browser finally by http protocol and use for virtual reality fusion module；
GIS service module: providing and manages entire virtual reality fusion contextual data, and the virtual reality fusion contextual data includes view
The Video Model that frequency model generation module generates, the three-dimensional building model in virtual reality fusion scene, the three-dimensional building model by
Real building modeling generates, and the environment of entire scene, when client browser issues the HTTP request of access, GIS service
Module is responsible for Video Model and three-dimensional building model needed for transmitting to virtual reality fusion system；It is also mentioned simultaneously to client browser
For a GIS-Geographic Information System (Geographic Information System, GIS) as virtual reality fusion scene carrier and
Environment, Video Model and buildings model are located on digital earth by real-world geographical coordinate system, that is, latitude and longitude coordinates, real
Entire scene and the accurate relative position of each model are showed, the GIS-Geographic Information System is a three-dimensional digital earth, is had
Terrain information and satellite base map, realize the true reappearance of entire scene environment；
Virtual reality fusion module: reading Video Model file, calls WebGL interface to realize the rendering of Video Model, and use
To the corresponding video stream data of the net cast module request Video Model, data pass through http protocol and pass the Tag label of HTML5
It is defeated, finally obtain the real-time video flow data of fragment；Using real-time video flow data as texture, the mode wash with watercolours projected using texture
Dye is drawn, and the Video Model virtual reality fusion effect with video dynamic texture is obtained；
Scene executor module: a series of interactive operations on client web interface are provided for user, are enabled a user to
It is enough to be free to navigate through in three-dimensional enhancing virtual environment, including dummy scene roaming, scene information are shown, video texture controls and VR
Mode these fourth types function can allow user to have better experience sense and feeling of immersion for enhancing virtual scene；The virtual field
Scape roaming function can allow user to select to access pre-set important scenes node, or carry out scene from animation along projected route
Trip；The scene information display function can choose corresponding buildings model or Video Model in user's click to obtain to the mould
The introduction of type details；The video texture control function allows user to operate, control to the Video Model in scene
System, allows users to operate interested video, and the operation includes pause, broadcasting, playback, F.F., synchronous behaviour
Make；The VR mode capabilities can allow user to obtain VR display effect when using VR equipment browse client Web.
A kind of real-time virtual reality fusion live broadcasting method based on WebGL of the invention realizes that steps are as follows:
(1) when Video Model generation step: when obtaining real time monitoring video image or local video image input, using list
Monitored picture is generated Video Model vertex data by photos modeling technique, while recording camera parameter information, finally will be upper
It states data summarization and generates the storage of binary format file；The texture projective transformation of Video Model is calculated using camera parameter information
Matrix and the best view pose for watching the Video Model, by obtained data summarization and be converted to JSON format description glTF
Obtained result is supplied to GIS service step and called by file, including former binary file content and WebGL rendering parameter；
(2) it net cast step: receives when client virtual reality fusion step draws Video Model and applies for corresponding video
Request of the image as dynamic texture, if virtual reality fusion steps request is local video, net cast step directly forwards this
Ground video；If virtual reality fusion steps request is real time monitoring video flowing, net cast step is received in real time using RTMP agreement
The real-time plug-flow of network monitoring camera head, and transcoding, fragment processing are carried out to live video stream, ultimately generate the view of m3u8 format
The video slicing file of frequency configuration file and ts format is pushed to client browser for virtual reality fusion finally by http protocol
Step uses；
(3) GIS service step: the Video Model file that Video Model generation step provides is received, actual situation is forwarded it to
Fusion steps use；The buildings model that storage modeling generates, and forward buildings model to supply when receiving client request
Client virtual reality fusion step is called；The HTTP request that client browser issues access is received, it is clear to client according to requesting
Device of looking at provides a GIS-Geographic Information System (Geographic Information System, GIS) and is used as entire virtual reality fusion field
The carrier of scape, the GIS-Geographic Information System that GIS server provides are a three-dimensional digital earths, and three-dimensional digital earth is believed comprising landform
The latitude and longitude coordinates of three-dimensional digital earth and Video Model, buildings model are sent to void by breath and satellite base map, GIS service step
Real fusion steps, for realizing entire scene and the accurate relative position of each model；
(4) virtual reality fusion step: reading the rendering for calling WebGL interface to realize Video Model after Video Model file, and
Using the Tag label of HTML5 to the corresponding video stream data of the net cast steps request Video Model, data pass through HTTP and assist
View transmission, finally obtains the real-time video flow data of fragment；Using real-time video flow data as texture, the side projected using texture
Formula rendering is drawn, and the Video Model virtual reality fusion effect with video dynamic texture is obtained；Virtual reality fusion step reads three simultaneously
Buildings model is tieed up, WebGL interface is called to render buildings model in the three-dimensional digital earth that GIS service step provides together,
Realize the complete color applying drawing of entire scene；
(5) scene executor step: interactive operation of the user on client web interface is received and parsed through, change is passed through
Camera pose, display corresponding informance and change render mode method meet the need that user is free to navigate through in virtual reality fusion scene
It asks, including dummy scene roaming, scene information are shown, video texture controls and VR mode these fourth types function, to be mentioned for user
For the interactive experience of virtual reality fusion scene.
The advantages of the present invention over the prior art are that:
(1) present invention directly reconstructs Video Model from original monitored picture using single width image modeling technolog, makes simultaneously
The mode for using texture to project realizes that video flowing as the dynamic texture of Video Model, solves Most current virtual reality fusion method
Actual situation is directed at the low problem of cost prohibitive, accuracy, realizes preferable virtual reality fusion effect.
(2) present invention improves virtual reality fusion method by calling the interface of WebGL and HTML5 to realize virtual reality fusion
Efficiency reduces demand of this method to system resource, and finally realizes virtual reality fusion system at the end Web, improves actual situation
The cross-platform compatibility of emerging system is conducive to this method and more broadly propagates.
(3) present invention realizes net cast module by building streaming media server, enables the system to support and realize
Storage, forwarding and the projective textures textures of multichannel live video stream, ensure that the real-time of entire virtual reality fusion system, improve
Compatibility of the system to different model monitoring camera, it is ensured that the scalability of whole system.
Detailed description of the invention
Fig. 1 is system structure diagram of the invention；
Fig. 2 is glTF file structure schematic diagram of the invention；
Fig. 3 is Video Model rendering effect schematic diagram；Wherein (a) is not textured Video Model；It (b) is textured
Video Model；It (c) is the Video Model under best view；
Fig. 4 is that schematic diagram is broadcast live in streaming media server live video stream of the invention；
Fig. 5 is m3u8 file playing principles schematic diagram of the invention；
Fig. 6 is texture projection process schematic diagram of the invention.
Specific implementation method
For a better understanding of the technical solution of the present invention, doing and further chatting in detail below in conjunction with attached drawing and implementation example
It states.
The invention proposes a kind of real-time virtual reality fusion live broadcast system and method based on WebGL, as shown in Figure 1, this hair
The bright virtual reality fusion live broadcast system by holding offline, server end and client are constituted, raw in offline end administration Video Model
At module；Net cast module and GIS service module are deployed in server end；Client deployment virtual reality fusion module and
Scene executor module.
It is entire to realize that process is as follows:
(1) when Video Model generation module obtains real time monitoring video image or local video image inputs, video screen module
Monitored picture is generated Video Model vertex data using single width image modeling technolog by type generation module, while recording camera ginseng
Above-mentioned data summarization is finally generated the storage of binary format file by number information；Video screen module is calculated using camera parameter information
The texture projective transformation matrix of type and the best view pose for watching the Video Model.By obtained data summarization and be converted to
The glTF file of JSON format description, including former binary file content and WebGL rendering parameter.Obtained result is supplied to
GIS service module is called.
(2) net cast module, which receives when client virtual reality fusion module draws Video Model, applies for corresponding video figure
As the request as dynamic texture, if virtual reality fusion module request is local video, net cast module directly forwards local
Video；If virtual reality fusion module request is real time monitoring video flowing, net cast module receives net using RTMP agreement in real time
The real-time plug-flow of network monitoring camera, and transcoding, fragment processing are carried out to live video stream, ultimately generate the video of m3u8 format
The video slicing file of configuration file and ts format is pushed to client browser for virtual reality fusion mould finally by http protocol
Block uses.
(3) GIS service module receives the Video Model file that Video Model generation module provides, and forwards it to actual situation and melts
Block is molded to use；The buildings model that storage modeling generates, and forward buildings model for visitor when receiving client request
Family end virtual reality fusion module is called；The HTTP request that client browser issues access is received, according to request to Client browse
Device provides a GIS-Geographic Information System (Geographic Information System, GIS) and is used as entire virtual reality fusion scene
Carrier.The GIS-Geographic Information System that GIS server provides is mainly a three-dimensional digital earth, and it includes terrain informations and satellite
Base map.The latitude and longitude coordinates of three-dimensional digital earth and Video Model, buildings model are sent to virtual reality fusion mould by GIS service module
Block, for realizing entire scene and the accurate relative position of each model.
(4) virtual reality fusion module calls WebGL interface to realize the rendering of Video Model after reading Video Model file, and makes
With the Tag label of HTML5 to the corresponding video stream data of the net cast module request Video Model, data pass through http protocol
Transmission, finally obtains the real-time video flow data of fragment；Using real-time video flow data as texture, the mode projected using texture
Rendering is drawn, and the Video Model virtual reality fusion effect with video dynamic texture is obtained.Virtual reality fusion module reads three-dimensional simultaneously
Buildings model calls WebGL interface to render buildings model in the three-dimensional digital earth that GIS service module provides together, real
The now complete color applying drawing of entire scene.
(5) scene executor module receives and parses through interactive operation of the user on client web interface, by changing phase
The methods of seat in the plane appearance, display corresponding informance and change render mode meet the need that user is free to navigate through in virtual reality fusion scene
It asks, including dummy scene roaming, scene information are shown, video texture controls and VR mode these fourth types major function.To be use
Family provides the interactive experience of the virtual reality fusion scene for this system.
Above-mentioned realization process specific implementation principle and method are as follows:
1. end Video Model generation module cardinal principle and method are as follows offline:
The initial three-dimensional Video Model file format that the present invention uses be BJ University of Aeronautics & Astronautics's virtual reality technology with
A kind of threedimensional model file format SIBM of system National Key Laboratory design definition, the type file are built using single width photo
Mould technology generates.SIBM file format is binary file, storage be model binary data, user can to its into
Row quickly read-write.The information for including in SIBM file is the version information of SIBM file first, and different SIBM versions is corresponding
Different model vertices coordinate systems is suitable for the rendering coordinate system requirement of different engines, followed by the integer type of 4 bytes
Number of vertices information, followed by the three-dimensional coordinate information on these vertex, the type of coordinate are three-dimensional floating point vectors；It is three later
The camera parameter information of dimension module, the position including camera, visual angle, the parameters such as focal length；It is finally the initial texture of threedimensional model
Pictorial information, i.e. the binary stream data of picture.
Video Model generation module outputs it after reading in original SIBM file data and can identify and make for WebGL engine
GlTF formatted file.GlTF (GL Transmission Format) is total to by Microsoft and Khronos 3D company
With the threedimensional model file format released., using the data information of the framework descriptive model based on JSON, data structure is efficient for it,
The high efficiency for having the characteristics that the timeliness of transmission and parsing.Meet very much requirement of the end Web for speed and terseness.
As shown in Fig. 2, the basic framework of glTF can substantially be divided into four parts, wherein the glTF module of top layer is one
A JSON framework describes the node level of model, material, camera, the interrelated logics structure such as animation；Bin module describes
The specific vertex data information of object described by glTF module；Glsl module describes the tinter of rending model；Png, jpg mould
Block describes the texture maps of model.
The camera video flow image data of acquisition is to be drawn by way of projective textures with 3 D video Model Fusion.
Projective textures mapping purpose is mapped on a texture to object with the mode of projection.This method does not need to refer in the application
Determine vertex texture coordinate, but texture coordinate is calculated by viewpoint matrix and projection matrix in vertex shading program.Cause
The apex coordinate of this threedimensional model read from SIBM file needs to carry out matrixing, could be on two-dimensional display just
Really show.
In glTF file, by defining glsl coloration program and defining view transform matrixes and projection matrix come real
Existing projective textures.Wherein view transform matrixes and projection matrix are calculated by the camera parameter in SIBM file.Calculate step
It is rapid as follows:
[input] location, forward, up, focus, width, height, plane (near), remote cutting are closely cut
Plane (far)
The 4x4 matrix of [output] view transform matrixes and projection matrix
Step 1: forward, side, up base vector and visual coordinate system eye are found out.Location is visual coordinate
It is eye, does not need additional operation.Side and up base vector is needed by being calculated, as follows:
Side=cross (forward, up)
Up=cross (side, forward)
Step 2: three obtained base vector is standardized.
Step 3: constituting matrix R using side, up, forward base vector, constitutes matrix T using eye, as follows:
Step 4: view transform matrixes are found out using matrix R and matrix T, operation method is as follows:
Step 5: left, right, bottom, top parameter are found out.
Left (l)=(- ratio) * near/Focus
Right (r)=- left
Bottom (b)=(- near)/Focus
Top (t)=bottom
Step 6: projection matrix is found out.
Result in the identifiable Video Model file of WebGL.The rendering effect of this document is as shown in figure 3, wherein (a)
For not textured Video Model；It (b) is textured Video Model, but not due to observation viewpoint and original monitoring camera position
Coincidence causes video dynamic texture to have certain twisted phenomena；(c) be the Video Model under best view, can get at this time compared with
Good visual effect.As can be seen that the dynamic texture for using video as scene may be implemented by Video Model, and best
Relatively good virtual reality fusion effect can be obtained at viewpoint.
Video Model generation module and server end are asynchronous, generate Video Model file offline, finally by generation
Calling of the Video Model for net cast module.The Video Model file is by Video Model vertex coordinates data, video screen module
Type camera posture information, initial texture pictorial information and WebGL spatial cue composition；The WebGL spatial cue includes
Video Model vertex data format, projection matrix, video dynamic texture information and coloration program.
2. server end net cast module cardinal principle and method are as follows:
The present invention is furthermore achieved network monitoring on the basis of realizing local video as 3 D video model and takes the photograph
As the live video stream of head acquisition is as texture.Thus may be implemented real time monitoring on the 3 D video model of scene with
Live streaming.
The method signal of live video stream live streaming is as shown in figure 4, streaming media server safeguards a message queue to receive
The request of data of browser, and required video stream data is sent to browser.If user browser requests live video stream, stream
Media server can pull video stream data from IP Camera by special module, send out after unified coded treatment
Give the browser for issuing request.In this way, streaming media server can enable virtual reality fusion system preferably to support
Different type, different coding camera data without regard to specific camera model and driving, so that it is new to system addition
Camera and data it is more convenient, ensure that the scalability of system.If what user requested is the video of local cache, matchmaker is flowed
Body server then directly finds corresponding cache file and is sent to user browser.
Wherein, the present invention uses Nginx server as streaming media server for receiving client requirements.According to user
It is required that Nginx streaming media server calls FFmpeg to obtain the real-time view that RTMP protocol code is transmitted from network monitoring camera head
Frequency flow data.RTMP real-time messages transport protocol be Adobe Systems company be Flash player and streaming media server it
Between video/audio transmission and develop agreement, be a kind of network protocol for carrying out real-time data communication.The compatibility of RTMP
Property is poor, and real-time is preferable.So carry out uploaded videos stream using this agreement, that is, pushing video streaming is to Nginx server
The delay in video capture transmission procedure can be reduced to greatest extent.
During Nginx streaming media server is transferred to real-time virtual reality fusion client, in order to meet HTML5 to reality
When stream mobile terminal compatibility, present invention uses HLS protocol transmitting video-frequency flow data.
So-called HLS protocol, briefly entire real-time streams are divided into it is small one by one, based on the file of HTTP come under
It carries, only downloads every time, m3u8 file therein is namely based on HLS protocol, stores the file of video stream metadata.Each
M3u8 file respectively corresponds several ts files, these ts files are only the data of really storage video, and m3u8 file is only deposited
The configuration information and introductory path of some ts files are put.As shown in Fig. 5, when video playing .m3u8 is that dynamic changes
, video label can parse this file, and find corresponding ts file to play.Video label is thereby realized to reality
When the support flowed, recall model rendering drafting module and color applying drawing carried out to real-time streams, thus desktop end and mobile terminal all
The dynamic texture that real-time streams make Video Model is realized, the live streaming of live video stream is realized.
3. server end GIS service module cardinal principle and method are as follows:
In the present invention, the exact position of camera and model is described using geographic coordinate system, that is, latitude and longitude coordinates, so
It can help more effectively to manage this large scene using GIS service module.This Department of Geography based on real world coordinates
System can allow scene and real world to be preferably mapped.
GIS service module safeguards that a message queue receives the HTTP request that client browser issues access, parsing pair
It should request and provide a GIS-Geographic Information System (Geographic Information System, GIS) to client browser
Carrier as entire virtual reality fusion scene.One three-dimensional digital earth of the GIS-Geographic Information System major maintenance, three dimension
The word earth has terrain information and satellite base map.The satellite map includes 13 class precisions in total, and full accuracy can realize street
The base map of rank is shown.Video Model and buildings model in virtual reality fusion scene are located in digital earth by geographic coordinate system
On, accurate relative position is realized, can allow user when going sight-seeing in virtual reality fusion scene just like placing oneself in the midst of the true earth
Environmentally, the feeling of immersion of user is improved.
4. client virtual reality fusion module cardinal principle and method are as follows:
Model especially 3 D video model used in render process HTML5 multimedia new features and with
The corresponding interface of WebGL engine.
HTML5 is that one under the kernel language of WWW, standard generalized markup language applies hypertext markup language
(HTML) the 5th material alteration.It formulates and completes in October, 2014.Its original intention designed is exactly in order on the mobile apparatus
Support multimedia.
In virtual reality fusion module of the invention, local or real-time video fluxion are read by<video>label of HTML5
According to, and the browser as used in user is decoded storage, and the image file that video stream data resolves into a frame frame is stored in number
In group, then by WebGL engine obtained data are handled, is generated pair according to the mode of the projective textures of glsl document definition
The texture answered finally constantly carries out rendering update to the texture of Video Model by the rendering engine of a real-time update, finally exists
A continually changing projective textures have been obtained on 3 D video model.3 D video mould is thus realized in three-dimensional scenic
Type merges drafting and display with video flowing data texturing.
As shown in fig. 6, virtual reality fusion module realizes that virtual reality fusion rendering and the process drawn are main after obtaining corresponding data
There are 4 stages:
(1) posture information of the camera in true environment, the posture information being converted into three-dimensional space are read first；
(2) obtained camera posture information then can be used to calculate the Model-View matrix M of cameramvWith projection matrix Mp；
(3) necessary processing done to model after, such as the cutting invisible model of camera, calculation amount is reduced with this
Accelerate the process of fusion；
(4) rendering acceleration is finally carried out to Video Model according to related rendering parameter using WebGL interface, in video card into
The veining of row piece member and coloration operation, piece member are ultimately converted to pixel seen on screen after rasterization operation.
Due to having used HTML5 as optimized integration, so the present invention can not only support that desktop end is all kinds of clear well
Look at device access and rendering show, also achieve the high efficiency of transmission and display in mobile terminal.
5. scene executor module cardinal principle and method are as follows:
Present system is after realizing that three-dimensional scene models and live video stream data fusion are drawn, by scene executor mould
Block provides a series of interactive operations on web interface, allows users to be free to navigate through in three-dimensional enhancing virtual environment, wrap
Include dummy scene roaming, scene information is shown, video texture controls and VR mode these fourth types major function.To allow user for
The enhancing virtual scene of this system has better experience sense and feeling of immersion.
(a) dummy scene roaming: this function is primarily to allow user that can more freely experience three-dimensional enhancing virtually
Environment.It is that user is able to access that pre-set several important scenario nodes first.In the present system, modal position is by ground
Managing coordinate system indicates, i.e., is indicated by longitude and latitude.Therefore, user can select pre-set important scenes by drop-down menu column
Node obtains corresponding node camera longitude and latitude and elevation information, and wherein camera attitude angle is indicated with Eulerian angles.Again by corresponding to
Camera control module finally makes user access the scenario node according to obtained camera posture information control camera flight.From
Dynamic scene walkthrough principle is similar, and a series of camera node pose parameters are stored in advance in scene control, and correspondence is pressed when the user clicks
When button, camera carries out automatic roaming along the projected route.
(b) scene information is shown: when user goes sight-seeing virtual scene, can also click on corresponding three-dimensional building
Model or 3 D video model obtain the introduction to the model details, so that user be allowed to have more scene and model
Deep understanding.The information of threedimensional model is stored in MySQL database by this system, then using being deployed in Web server
On PHP script page access MySQL database, inquire corresponding form data, these form datas finally returned into use
Family interface, to realize inquiry of the user to model information in three-dimensional scenic.Wherein three-dimensional model information includes pattern number, mould
The information such as type title, building time, building function brief introduction, geographical location, 3 D video model information include pattern number, model
The information such as title, video flowing source, camera parameter, geographical location.
Other than inquiring three-dimensional model information, this system also supports the actual geographic information of current mouse position,
Including longitude and latitude and elevation information.There is user for specific location of the current scene in the three-dimensional earth more to feel
The understanding of property.
(c) video texture controls: this function allows user to operate, control to the Video Model in scene.So that with
Family can operate its interested video, including the operation such as pause, broadcasting, playback, F.F., so as to play back over
The video of time, moreover it is possible to it is synchronous with practical clock at any time, obtain current live video stream.This function combination real-time live broadcast module
User can be helped preferably to monitor scene.
(d) VR mode: VR mode can be entered by clicking VR mode button, in VR mode, system is by scenic picture wash with watercolours
Dye two width picture of right and left eyes, two width pictures are substantially similar, and angle slightly has difference, by artificially simulate right and left eyes parallactic angle to
Realize that VR effect is shown.The function experiences this system in VR equipment well, allows user that can preferably enjoy enhancing empty
Quasi- scene, obtains effect more true to nature.
What the present invention did not elaborated partly belongs to those skilled in the art's well-known technique.
The above is only a preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications are also answered
It is considered as protection scope of the present invention.
Claims (2)
1. a kind of real-time virtual reality fusion live broadcast system based on WebGL, it is characterised in that: the virtual reality fusion live broadcast system by
Offline end, server end and client are constituted, and affix one's name to Video Model generation module in offline end；Video is deployed in server end
Module and GIS service module is broadcast live；Client deployment virtual reality fusion module and scene executor module:
Video Model generation module: the real time monitoring video image or local video image of monitoring camera acquisition are read, is used
Monitored picture is generated Video Model vertex data by single width image modeling technolog, while recording camera parameter information, finally will
Above-mentioned data summarization generates the storage of binary format file, and the binary format file includes Video Model vertex coordinates data
And camera parameter information；The texture projective transformation matrix of Video Model is calculated using its camera parameter information and watches the video
The best camera view pose of model, then the Video Model relevant information combination Video Model WebGL rendering parameter that will be obtained
Convert, be stored as the Video Model file of the identifiable JSON file format of client browser；The Video Model file
By Video Model vertex coordinates data, Video Model camera posture information, initial texture pictorial information and WebGL spatial cue
Composition；The WebGL spatial cue include Video Model vertex data format, projection matrix, video dynamic texture information and
Coloration program；Video Model generation module and server end are asynchronous, generate Video Model file offline, will finally generate
Video Model for GIS service module calling；
Net cast module: the real time monitoring video image or local video image of monitoring camera acquisition are received, and is located
It manages and stores；To the corresponding video figure of net cast module request when client virtual reality fusion module draws Video Model
When as being used as dynamic texture, net cast module to client virtual reality fusion module forwards, makes corresponding video image for it
With；If virtual reality fusion module request is local video, net cast module directly forwards local video；If virtual reality fusion module
Request is real time monitoring video flowing, and net cast module receives the real-time plug-flow of network monitoring camera head using RTMP agreement,
And transcoding, fragment processing are carried out to live video stream, ultimately generate the video profile of m3u8 format and the video of ts format
Slicing files are pushed to client browser finally by http protocol and use for virtual reality fusion module；
GIS service module: providing and manages entire virtual reality fusion contextual data, and the virtual reality fusion contextual data includes video screen module
The Video Model that type generation module generates, the three-dimensional building model in virtual reality fusion scene, and the environment of entire scene, it is described
Three-dimensional building model is modeled by real building and is generated；When client browser issues the HTTP request of access, GIS service module is negative
It blames to Video Model and three-dimensional building model needed for the transmission of virtual reality fusion system；One also is provided to client browser simultaneously
Carrier and environment of the GIS-Geographic Information System (Geographic Information System, GIS) as virtual reality fusion scene,
Video Model and buildings model are located on digital earth by real-world geographical coordinate system, that is, latitude and longitude coordinates, are realized whole
A scene and the accurate relative position of each model, the GIS-Geographic Information System are a three-dimensional digital earths, are believed with landform
Breath and satellite base map, realize the true reappearance of entire scene environment；
Virtual reality fusion module: reading Video Model file, calls WebGL interface to realize the rendering of Video Model, and use HTML5
Tag label to the corresponding video stream data of the net cast module request Video Model, data are transmitted by http protocol, most
The real-time video flow data of fragment is obtained eventually；Using real-time video flow data as texture, is rendered, drawn using the mode that texture projects
System obtains the Video Model virtual reality fusion effect with video dynamic texture；
Scene executor module: a series of interactive operations on client web interface are provided for user, are allowed users to
It is free to navigate through in three-dimensional enhancing virtual environment, including dummy scene roaming, scene information are shown, video texture controls and VR mode
These four types of functions can allow user to have better experience sense and feeling of immersion for enhancing virtual scene；The virtual scene is unrestrained
Trip function can allow user to select to access pre-set important scenes node, or carry out scene automatic roaming along projected route；
The scene information display function can choose corresponding buildings model or Video Model in user's click to obtain to the model
The introduction of details；The video texture control function allows user to operate, control to the Video Model in scene,
It allows users to operate interested video, the operation includes pause, broadcasting, playback, F.F., simultaneously operating；
The VR mode capabilities can allow user to obtain VR display effect when using VR equipment browse client Web.
2. a kind of real-time virtual reality fusion live broadcasting method based on WebGL, it is characterised in that: realize that steps are as follows:
(1) when Video Model generation step: when obtaining real time monitoring video image or local video image input, being shone using single width
Monitored picture is generated Video Model vertex data by piece modeling technique, while recording camera parameter information, finally by above-mentioned number
The storage of binary format file is generated according to summarizing；The texture projective transformation matrix of Video Model is calculated using camera parameter information
And the best view pose of the Video Model is watched, by obtained data summarization and the glTF text for being converted to the description of JSON format
Obtained result is supplied to GIS service step and called by part, including former binary file content and WebGL rendering parameter；
(2) it net cast step: receives when client virtual reality fusion step draws Video Model and applies for corresponding video image
As the request of dynamic texture, if virtual reality fusion steps request is local video, net cast step directly forwards local regard
Frequently；If virtual reality fusion steps request is real time monitoring video flowing, net cast step receives network using RTMP agreement in real time
The real-time plug-flow of monitoring camera, and transcoding, fragment processing are carried out to live video stream, the video for ultimately generating m3u8 format is matched
The video slicing file for setting file and ts format is pushed to client browser for virtual reality fusion step finally by http protocol
It uses；
(3) GIS service step: the Video Model file that Video Model generation step provides is received, virtual reality fusion is forwarded it to
Step uses；The buildings model that storage modeling generates, and forward buildings model for client when receiving client request
Virtual reality fusion step is held to call；The HTTP request that client browser issues access is received, according to request to client browser
GIS-Geographic Information System (Geographic Information System, GIS) is provided as entire virtual reality fusion scene
Carrier, the GIS-Geographic Information System that GIS server provides is a three-dimensional digital earth, three-dimensional digital earth include terrain information and
The latitude and longitude coordinates of three-dimensional digital earth and Video Model, buildings model are sent to actual situation and melted by satellite base map, GIS service step
Step is closed, for realizing entire scene and the accurate relative position of each model；
(4) it virtual reality fusion step: calls WebGL interface to realize the rendering of Video Model after reading Video Model file, and uses
To the corresponding video stream data of the net cast steps request Video Model, data pass through http protocol and pass the Tag label of HTML5
It is defeated, finally obtain the real-time video flow data of fragment；Using real-time video flow data as texture, the mode wash with watercolours projected using texture
Dye is drawn, and the Video Model virtual reality fusion effect with video dynamic texture is obtained；Virtual reality fusion step reads three-dimensional and builds simultaneously
Model is built, WebGL interface is called to render buildings model in the three-dimensional digital earth that GIS service step provides together, is realized
The complete color applying drawing of entire scene；
(5) scene executor step: receiving and parsing through interactive operation of the user on client web interface, by changing camera
Pose, display corresponding informance and change render mode method meet the needs of user is free to navigate through in virtual reality fusion scene, wrap
Include dummy scene roaming, scene information is shown, video texture controls and VR mode these fourth types function, to provide actual situation for user
Merge the interactive experience of scene.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710872854.2A CN107835436B (en) | 2017-09-25 | 2017-09-25 | A kind of real-time virtual reality fusion live broadcast system and method based on WebGL |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710872854.2A CN107835436B (en) | 2017-09-25 | 2017-09-25 | A kind of real-time virtual reality fusion live broadcast system and method based on WebGL |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107835436A CN107835436A (en) | 2018-03-23 |
| CN107835436B true CN107835436B (en) | 2019-07-26 |
Family
ID=61644048
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710872854.2A Active CN107835436B (en) | 2017-09-25 | 2017-09-25 | A kind of real-time virtual reality fusion live broadcast system and method based on WebGL |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107835436B (en) |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109165270A (en) * | 2018-07-02 | 2019-01-08 | 武汉珞珈德毅科技股份有限公司 | A kind of three-dimensional GIS platform architecture system |
| CN109045694B (en) * | 2018-08-17 | 2020-01-10 | 腾讯科技（深圳）有限公司 | Virtual scene display method, device, terminal and storage medium |
| CN109842811B (en) * | 2019-04-03 | 2021-01-19 | 腾讯科技（深圳）有限公司 | Method and device for implanting push information into video and electronic equipment |
| CN110349254B (en) * | 2019-07-11 | 2022-12-06 | 东北大学 | C/S architecture-oriented adaptive medical image three-dimensional reconstruction method |
| CN110418127B (en) * | 2019-07-29 | 2021-05-11 | 南京师范大学 | Operation method of pixel template-based virtual-real fusion device in Web environment |
| CN110738721B (en) * | 2019-10-12 | 2023-09-01 | 四川航天神坤科技有限公司 | Three-dimensional scene rendering acceleration method and system based on video geometric analysis |
| CN111225191B (en) * | 2020-01-17 | 2022-07-29 | 华雁智能科技(集团)股份有限公司 | Three-dimensional video fusion method and device and electronic equipment |
| CN111464818B (en) * | 2020-03-20 | 2022-04-19 | 新之航传媒科技集团有限公司 | Online live broadcast exhibition hall system |
| CN112437276B (en) * | 2020-11-20 | 2023-04-07 | 埃洛克航空科技（北京）有限公司 | WebGL-based three-dimensional video fusion method and system |
| CN112584254A (en) * | 2020-11-30 | 2021-03-30 | 北京邮电大学 | RTSP video stream loading method and device based on Cesium |
| CN112584060A (en) * | 2020-12-15 | 2021-03-30 | 北京京航计算通讯研究所 | Video fusion system |
| CN112584120A (en) * | 2020-12-15 | 2021-03-30 | 北京京航计算通讯研究所 | Video fusion method |
| CN112687012A (en) * | 2021-01-08 | 2021-04-20 | 中国南方电网有限责任公司超高压输电公司南宁监控中心 | Island information fusion method based on three-dimensional visual management and control platform |
| CN113099204B (en) * | 2021-04-13 | 2022-12-13 | 北京航空航天大学青岛研究院 | Remote live-action augmented reality method based on VR head-mounted display equipment |
| CN115686182B (en) * | 2021-07-22 | 2024-02-27 | 荣耀终端有限公司 | Processing method of augmented reality video and electronic equipment |
| CN114047821A (en) * | 2021-11-18 | 2022-02-15 | 中国人民解放军陆军装甲兵学院士官学校 | Virtual teaching method |
| CN114494563B (en) * | 2022-02-14 | 2022-10-11 | 北京清晨动力科技有限公司 | Method and device for fusion display of aerial video on digital earth |
| CN114885147B (en) * | 2022-07-12 | 2022-10-21 | 中央广播电视总台 | Fusion production and broadcast system and method |
| CN115695841B (en) * | 2023-01-05 | 2023-03-10 | 威图瑞(北京)科技有限公司 | Method and device for embedding online live broadcast in external virtual scene |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104376596A (en) * | 2014-11-28 | 2015-02-25 | 北京航空航天大学 | Method for modeling and registering three-dimensional scene structures on basis of single image |
| CN105872496A (en) * | 2016-07-01 | 2016-08-17 | 黄岩 | Ultrahigh-definition video fusion method |
| CN106373148A (en) * | 2016-08-31 | 2017-02-01 | 中国科学院遥感与数字地球研究所 | Equipment and method for realizing registration and fusion of multipath video images to three-dimensional digital earth system |
-
2017
- 2017-09-25 CN CN201710872854.2A patent/CN107835436B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104376596A (en) * | 2014-11-28 | 2015-02-25 | 北京航空航天大学 | Method for modeling and registering three-dimensional scene structures on basis of single image |
| CN105872496A (en) * | 2016-07-01 | 2016-08-17 | 黄岩 | Ultrahigh-definition video fusion method |
| CN106373148A (en) * | 2016-08-31 | 2017-02-01 | 中国科学院遥感与数字地球研究所 | Equipment and method for realizing registration and fusion of multipath video images to three-dimensional digital earth system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107835436A (en) | 2018-03-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107835436B (en) | A kind of real-time virtual reality fusion live broadcast system and method based on WebGL | |
| MacIntyre et al. | The Argon AR Web Browser and standards-based AR application environment | |
| CN110379010A (en) | Three-dimensional geographic information method for visualizing and system based on video fusion | |
| CN114401414B (en) | Information display method and system for immersive live broadcast and information pushing method | |
| CN113891117B (en) | Immersion medium data processing method, device, equipment and readable storage medium | |
| CN105989623B (en) | The implementation method of augmented reality application based on handheld mobile device | |
| Ekong et al. | Teacher-student vr telepresence with networked depth camera mesh and heterogeneous displays | |
| Che et al. | Reality-virtuality fusional campus environment: An online 3D platform based on OpenSimulator | |
| Cao | Development and design case function comparison of panoramic roaming system of virtual museum based on Pano2VR | |
| CN114981822A (en) | Referencing of neural network models for adaptation of 2D video streaming to heterogeneous client endpoints | |
| Lu et al. | Webvrgis: Webgis based interactive online 3d virtual community | |
| CN117390322A (en) | Virtual space construction method and device, electronic equipment and nonvolatile storage medium | |
| Liu et al. | A 2d and 3d indoor mapping approach for virtual navigation services | |
| Priestnall et al. | Virtual geographic environments | |
| KR102598603B1 (en) | Adaptation of 2D video for streaming to heterogeneous client endpoints | |
| Hudson-Smith | Digital urban-the visual city | |
| JP7472298B2 (en) | Placement of immersive media and delivery of immersive media to heterogeneous client endpoints | |
| CN115174954A (en) | Video live broadcast method and device, electronic equipment and storage medium | |
| Tao | A VR/AR-based display system for arts and crafts museum | |
| Tan et al. | Architecture design of Jing Gangshan virtual tourism system based on WebVR | |
| Qian et al. | Design of Smart Tourism System in Urban Scenic Spots Based on Computer Virtual Reality Technology | |
| CN110662099A (en) | Method and device for displaying bullet screen | |
| Li et al. | Design of Intangible Cultural Heritage Virtual Roaming System Based on Veb VAR Stereoscopic Panoramic Technology | |
| US11943271B2 (en) | Reference of neural network model by immersive media for adaptation of media for streaming to heterogenous client end-points | |
| Stirbu et al. | Open and decentralized platform for visualizing web mash-ups in augmented and mirror worlds |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |