CN111901616B - H5/WebGL-based method for improving multi-view live broadcast rendering - Google Patents

H5/WebGL-based method for improving multi-view live broadcast rendering Download PDF

Info

Publication number
CN111901616B
CN111901616B CN202010679364.2A CN202010679364A CN111901616B CN 111901616 B CN111901616 B CN 111901616B CN 202010679364 A CN202010679364 A CN 202010679364A CN 111901616 B CN111901616 B CN 111901616B
Authority
CN
China
Prior art keywords
player
playing
pictures
picture
rendering
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010679364.2A
Other languages
Chinese (zh)
Other versions
CN111901616A (en
Inventor
王嘉炀
洪玉先
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
E Surfing Video Media Co Ltd
Original Assignee
E Surfing Video Media Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by E Surfing Video Media Co Ltd filed Critical E Surfing Video Media Co Ltd
Priority to CN202010679364.2A priority Critical patent/CN111901616B/en
Publication of CN111901616A publication Critical patent/CN111901616A/en
Application granted granted Critical
Publication of CN111901616B publication Critical patent/CN111901616B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21Server components or server architectures
    • H04N21/218Source of audio or video content, e.g. local disk arrays
    • H04N21/2187Live feed
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21Server components or server architectures
    • H04N21/218Source of audio or video content, e.g. local disk arrays
    • H04N21/21805Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23412Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23424Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65Transmission of management data between client and server
    • H04N21/658Transmission by the client directed to the server
    • H04N21/6587Control parameters, e.g. trick play commands, viewpoint selection

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Studio Circuits (AREA)

Abstract

The invention provides an improved multi-view live broadcast rendering method based on H5/WebGL.A plurality of cameras are adopted to shoot a scene, pictures shot in the scene are merged through merging equipment, a playing stream address is transmitted back to an H5 player after transcoding is carried out through a stream media, a code stream is played through an H5 player label, the played pictures are transmitted to the WebGL to be processed, the player pictures are used as chartlet textures, frame-by-frame rendering is carried out on the playing pictures of the player, a canvas is combined with a self-defined playing control bar integrated in the player to finally display the pictures to a user, and the player displays and matches different playing interactions according to different scenes. The invention can realize scene switching rendering display of a single visual angle of the live broadcast image which combines a plurality of visual angle images together, and realize live broadcast rendering play of multiple machine positions, multiple scenes and free visual angles.

Description

H5/WebGL-based method for improving multi-view live broadcast rendering
Technical Field
The invention belongs to the field of video rendering processing and development, and particularly relates to a method for improving multi-view live broadcast rendering based on H5/WebGL.
Background
Ultra high definition video (4K, 8K) displays contain larger data volumes, require faster information transfer speeds, and therefore present certain challenges to hardware facilities. With the arrival of 5G, one of the changes is high bandwidth and low time delay, which can just solve the transmission problem further, drive the upgrade of the whole acquisition, production and playing content, further promote the development of ultra-high definition video application scenes, and more live broadcast events are also adopting the output of 5G live broadcast +4K/8K pictures.
However, at the browser end of the mobile phone, although the H5-video player can realize the playing of the live code stream (H264), because of the limitation of the browser (for example, only one code stream can be played in 1 page), at present, the method for switching playing of multiple views at the H5 end generally realizes the playing of a switching view according to the view and one stream at each view, which is actually the playing of a switching code stream. However, when live broadcast pictures are switched, the live broadcast pictures cannot be completely synchronized, and a transition effect of code stream loading and blocking is generated, so that the viewing experience is influenced.
WebGL (full-write Web Graphics Library) is a 3D drawing protocol, the drawing technical standard allows JavaScript and OpenGL ES 2.0 to be combined together, and by adding one JavaScript binding of OpenGL ES 2.0, WebGL can provide hardware 3D accelerated rendering for HTML5Canvas, so that Web developers can more smoothly display 3D scenes and models in a browser by means of a system display card, and complicated navigation and data visualization can be created.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides an improved multi-view live broadcast rendering method based on H5/WebGL, which is used for realizing scene switching rendering display of a single view angle on a live broadcast picture obtained by combining a plurality of view angle pictures at an H5 end, realizing live broadcast rendering play of multiple machine positions, multiple scenes and free view angles, and achieving the capabilities of switching live broadcast pictures in seconds and completely synchronizing audio.
According to a first aspect of the present invention, a method for improving multi-view live rendering based on H5/WebGL is provided, the method comprising the following steps:
s1, shooting on site: shooting the scene by adopting a plurality of cameras;
s2, merging pictures: combining the pictures shot on site through combining equipment;
s3, screen transmission to H5 Player: after transcoding is carried out through streaming media, a playing stream address is transmitted back to an H5 player;
s4, video playing: playing the code stream through the H5video label, and transmitting the played picture to WebGL for processing;
s5, WebGL rendering: taking the video picture as a chartlet texture, and rendering the video playing picture frame by frame;
s6, canvas presents: finally displaying the picture for the user by combining a user-defined playing control bar integrated in the player;
s7, displaying a playing picture: the player displays according to different scenes and matches different playing interactions.
In one aspect, step S2 is followed by the steps of:
after transcoding is carried out through the streaming media, the playing streaming address is transmitted back to the front-end page.
In one aspect, step S3 is preceded by the step of:
and the front-end page is configured through the player JS-SDK.
In one aspect, in step S5, the single scene picture in the bitstream playing picture is intercepted according to the parameter control configured by the SDK, and the intercepted picture is displayed through the canvas tag of the H5 player.
In one aspect, in step S7, the different scenes are free views, and the switching of the view pictures is performed by sliding left and right; or the different scenes are multi-scene visual angles, and the pictures with different visual angles can be switched by clicking the pictures.
In one aspect, the front-end page is a mobile phone page end, and the image captured on site is spliced, cut and transcoded and then transmitted back to the mobile phone page end.
In one aspect, the mobile phone webpage end receives the live broadcast code stream, and the rendering mode is configured through the JS-SDK.
The method and the device can basically realize the picture rendering of 24-60 frames by utilizing the hardware 3D acceleration (GPU) rendering of webGL, can basically well support the rendering of 4K pictures, can realize the scene switching rendering display of a single visual angle of a live broadcast picture which combines a plurality of visual angle pictures together at the end of H5, can realize the live broadcast rendering playing of a plurality of machine positions, a plurality of scenes and free visual angles, and can achieve the capability of switching the live broadcast picture in seconds and completely synchronizing audio. And the multi-scene live broadcast pictures are combined into one path of code stream for output, so that a large amount of broadband can be saved and more hardware bearing resources can be saved compared with the single code stream output of each picture.
Drawings
In order to more clearly illustrate the technical solutions of the present invention and the prior art, the drawings needed to be used are briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
FIG. 1 is a flow chart of a method of the present invention;
FIG. 2 illustrates a free-view camera arrangement according to the present invention;
FIG. 3 is a free-view interaction diagram of the present invention;
FIG. 4 is a schematic diagram of a multi-scene perspective switching interaction of the present invention;
FIG. 5 is a process flow of the present invention.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
As shown in fig. 1-4, the method for improving multi-view live rendering based on H5/WebGL of the present invention satisfies the usage of different live application scenes by outputting JS-SDK. The JS-SDK integrates the capabilities of playing code streams, intercepting scene pictures, switching and interacting visual angle pictures and the like, and a user can realize the required multi-visual angle picture display form only through simple configuration.
The method comprises the following steps:
s1, shooting on site: shooting the scene by adopting a plurality of cameras;
a plurality of cameras are arranged on the shooting site for shooting so as to obtain pictures with different visual angles. For example, fig. 2 shows a free view angle form, in which a plurality of cameras are placed around a stage, and shooting is performed in a certain angle range according to specific situations, so as to obtain shot pictures at different angles. The view angle may also be a multi-scene picture shooting as shown in fig. 3, and pictures of different shooting angles are obtained by placing cameras at different shooting positions for shooting.
S2, merging pictures: combining the pictures shot on site through combining equipment;
1. after a shooting scene obtains a shooting picture through a camera, merging the picture shot at the scene through merging equipment, transcoding through streaming media, and returning a transcoded playing stream address to a front-end page, wherein the front-end page is configured through a player JS-SDK.
S3, screen transmission to H5 player: after transcoding is carried out through streaming media, a playing streaming address is transmitted back to an H5 player;
after the shot pictures are transcoded by streaming media, the playing streaming addresses are transmitted back to the H5 player, so that data transmission from the shot pictures to the H5 player is realized.
S4, video playing: and playing the code stream through the H5video label, and transmitting the played picture to WebGL for processing.
S5, WebGL rendering: taking the video picture as a chartlet texture, and rendering the video playing picture frame by frame;
and taking the video picture as a chartlet texture through WebGL, rendering the video playing picture frame by frame, intercepting a single scene picture in the code stream playing picture according to parameter control configured by the SDK, and displaying the intercepted picture through a canvas label of H5.
S6, canvas presents: finally displaying the picture for the user by combining the self-defined playing control bar integrated in the player;
the canvas tag is responsible for the final picture presentation after the WebGL rendering processing, and is finally displayed to the user by combining with a self-defined playing control bar integrated in the player.
S7, displaying a playing picture: the player displays according to different scenes and matches different playing interactions.
The inside of the player can be displayed according to different scenes, and different playing interactions are matched. For example, the free view shown in fig. 3, can support the user to slide left and right to switch the view picture, for example, to switch to a different view picture by rotating the picture through sliding left and right. For example, as shown in fig. 4, the original screen has a plurality of different viewing angles, and the user can switch the screen with different viewing angles by clicking different screens.
As shown in fig. 5, in a specific operation process, a live program is required to output a one-way stream containing multiple pictures, a live site is required to view the live site through a plurality of ultra-high-definition camera positions in a surrounding manner, captured picture images are spliced, cut and transcoded, and finally transmitted back to a user mobile phone page end. And at the mobile phone page end, the received live broadcast code stream is subjected to configuration of a rendering mode through js-sdk of the invention, so that the display of the specified live broadcast mode can be realized. The specific processing flow comprises the following steps: playing a live broadcast code stream through an h5-video player; capturing each frame of live broadcast picture played by the video, and performing frame-by-frame rendering through WebGL; and rendering the picture in real time according to the rendering parameters configured by the js-sdk, and intercepting the required view picture for full-screen presentation.
The invention combines the multi-scene live broadcast pictures into one path of code stream for output, and can save a large amount of broadband and save more hardware bearing resources compared with the single code stream output of each picture; through rendering, live rendering playing with multiple machine positions, multiple scenes and free visual angles can be achieved, and the capabilities of switching live pictures in seconds and synchronizing audio completely can be achieved.
The above is a detailed description of the technical solution proposed by the present invention. In the description of the present specification, reference to the description of "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples", etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
The foregoing examples merely illustrate several embodiments of the present invention, and the principles and embodiments of the present invention are described herein, and the above description is only for the purpose of facilitating understanding the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.

Claims (5)

1. A method for improving multi-view live rendering based on H5/WebGL, characterized in that the method comprises the following steps:
s1, shooting on site: shooting the scene by adopting a plurality of cameras;
s2, merging pictures: combining the pictures shot on site through combining equipment;
s3, screen transmission to H5 player: after transcoding is carried out through streaming media, a playing streaming address is transmitted back to an H5 player;
s4, playing by the player: playing the code stream through an H5 player label, and transmitting the played picture to WebGL for processing;
s5, WebGL rendering: taking the player picture as a chartlet texture, and rendering the player playing picture frame by frame;
s6, canvas presents: finally displaying the picture for the user by combining a user-defined playing control bar integrated in the player;
s7, displaying a playing picture: the player displays according to different scenes and matches different playing interactions;
in the step S5, a single scene picture in the bitstream playing picture is intercepted according to the parameter control configured by the SDK, and the intercepted picture is displayed through a canvas label of the H5 player;
in the step S7, the different scenes are free views, and left-right sliding is supported to switch views; or the different scenes are multi-scene visual angles, and the pictures with different visual angles can be switched by clicking the pictures;
the method also comprises the step of performing coloring in the GPU thread, wherein the GPU thread interacts with the CPU thread through a GLSL language, and the GPU thread also comprises a vertex shader and a fragment shader.
2. The method according to claim 1, wherein step S2 is followed by the further steps of:
after transcoding is carried out through the streaming media, the playing streaming address is transmitted back to the front-end page.
3. The method according to claim 2, wherein step S3 is preceded by the step of:
and the front-end page is configured through the player JS-SDK.
4. The method of claim 3, wherein the front-end page is a mobile phone page end, and the image captured on site is spliced, cut, transcoded and transmitted back to the mobile phone page end.
5. The method of claim 4, wherein the webpage end of the mobile phone receives a live code stream, and the rendering mode is configured through the JS-SDK.
CN202010679364.2A 2020-07-15 2020-07-15 H5/WebGL-based method for improving multi-view live broadcast rendering Active CN111901616B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010679364.2A CN111901616B (en) 2020-07-15 2020-07-15 H5/WebGL-based method for improving multi-view live broadcast rendering

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010679364.2A CN111901616B (en) 2020-07-15 2020-07-15 H5/WebGL-based method for improving multi-view live broadcast rendering

Publications (2)

Publication Number Publication Date
CN111901616A CN111901616A (en) 2020-11-06
CN111901616B true CN111901616B (en) 2022-09-13

Family

ID=73192771

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010679364.2A Active CN111901616B (en) 2020-07-15 2020-07-15 H5/WebGL-based method for improving multi-view live broadcast rendering

Country Status (1)

Country Link
CN (1) CN111901616B (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114666565B (en) * 2020-12-22 2024-02-27 中国电信股份有限公司 Multi-view video playing method, device and storage medium
CN112929733B (en) * 2021-01-18 2022-06-28 稿定(厦门)科技有限公司 Video preview playing method and device
CN113271396B (en) * 2021-03-30 2022-07-01 江苏青朴科技有限公司 Conveyer for many cameras
CN113286162B (en) * 2021-05-20 2022-05-31 成都威爱新经济技术研究院有限公司 Multi-camera live-broadcasting method and system based on mixed reality
CN113497967B (en) * 2021-05-26 2023-03-14 浙江大华技术股份有限公司 Video frame switching method and device based on browser and storage medium
CN113627790A (en) * 2021-08-11 2021-11-09 国网河北省电力有限公司邢台供电分公司 Visual monitoring method, system and device
CN113852833B (en) * 2021-08-30 2024-03-22 阿里巴巴(中国)有限公司 Multi-device collaborative live broadcast method and device and electronic device
CN117939183A (en) * 2024-03-21 2024-04-26 中国传媒大学 Multi-machine-position free view angle guided broadcasting method and system

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106658212A (en) * 2017-01-20 2017-05-10 北京红马传媒文化发展有限公司 VR online playing method, system and player based on HTML5

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103649904A (en) * 2011-05-10 2014-03-19 Nds有限公司 Adaptive presentation of content
CN106599225A (en) * 2016-12-19 2017-04-26 成都君乾信息技术有限公司 3D specification display system
CN106897108A (en) * 2017-01-20 2017-06-27 华南理工大学 A kind of implementation method of the virtual reality Panoramic Warping based on WebVR
CN106998477A (en) * 2017-04-05 2017-08-01 腾讯科技(深圳)有限公司 The front cover display methods and device of live video
CN110661752A (en) * 2018-06-29 2020-01-07 广州弘度信息科技有限公司 Plug-in-free real-time video playing system and method
CN109934764A (en) * 2019-01-31 2019-06-25 北京奇艺世纪科技有限公司 Processing method, device, terminal, server and the storage medium of panoramic video file
CN111107391A (en) * 2020-01-16 2020-05-05 深圳市信义科技有限公司 Distributed WEB plug-in-free video live broadcast method

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106658212A (en) * 2017-01-20 2017-05-10 北京红马传媒文化发展有限公司 VR online playing method, system and player based on HTML5

Also Published As

Publication number Publication date
CN111901616A (en) 2020-11-06

Similar Documents

Publication Publication Date Title
CN111901616B (en) H5/WebGL-based method for improving multi-view live broadcast rendering
US11087549B2 (en) Methods and apparatuses for dynamic navigable 360 degree environments
WO2021179783A1 (en) Free viewpoint-based video live broadcast processing method, device, system, chip and medium
US11575869B2 (en) Overlay processing method in 360 video system, and device thereof
US10210907B2 (en) Systems and methods for adding content to video/multimedia based on metadata
CN110351592B (en) Animation presentation method and device, computer equipment and storage medium
CN112585978B (en) Generating a composite video stream for display in VR
CN110709892A (en) Method and system for rendering virtual reality content based on two-dimensional ('2D') captured images of a three-dimensional ('3D') scene
CN108989830A (en) A kind of live broadcasting method, device, electronic equipment and storage medium
CN106713942B (en) Video processing method and device
CN113891117B (en) Immersion medium data processing method, device, equipment and readable storage medium
DE112019000271T5 (en) METHOD AND DEVICE FOR PROCESSING AND DISTRIBUTION OF LIVE VIRTUAL REALITY CONTENT
Kim et al. Design and implementation for interactive augmented broadcasting system
CN112019906A (en) Live broadcast method, computer equipment and readable storage medium
WO2019004073A1 (en) Image placement determination device, display control device, image placement determination method, display control method, and program
CN115665461B (en) Video recording method and virtual reality device
TWI778749B (en) Transmission method, processing device, and generating system of video for virtual reality
CN114245228B (en) Page link release method and device and electronic equipment
CN108280882A (en) A kind of method and system being implanted into AR foreground article positions in virtual display space
Reis et al. An open-source tool for distributed viewing of kinect data on the web
EP4256782A1 (en) Real-time multiview video conversion method and system
TW202213990A (en) Live broadcasting system for real time three-dimensional image display
CN113243112A (en) Streaming volumetric and non-volumetric video

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant