WO2018094814A1 - Dispositif et procédé de synthèse de vidéo - Google Patents

Dispositif et procédé de synthèse de vidéo Download PDF

Info

Publication number
WO2018094814A1
WO2018094814A1 PCT/CN2016/112917 CN2016112917W WO2018094814A1 WO 2018094814 A1 WO2018094814 A1 WO 2018094814A1 CN 2016112917 W CN2016112917 W CN 2016112917W WO 2018094814 A1 WO2018094814 A1 WO 2018094814A1
Authority
WO
WIPO (PCT)
Prior art keywords
texture
video frame
window
original video
video
Prior art date
Application number
PCT/CN2016/112917
Other languages
English (en)
Chinese (zh)
Inventor
曾红
Original Assignee
深圳Tcl数字技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳Tcl数字技术有限公司 filed Critical 深圳Tcl数字技术有限公司
Publication of WO2018094814A1 publication Critical patent/WO2018094814A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23424—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234318—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into objects, e.g. MPEG-4 objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44004—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving video buffer management, e.g. video decoder buffer or video display buffer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4

Definitions

  • the present invention relates to the field of communications technologies, and in particular, to a video synthesis method and apparatus.
  • the Internet field has entered the live broadcast era from the era of the text, the era of pictures, and the short video era.
  • Mobile video live broadcasting has become one of the most eye-catching areas of the Internet.
  • the anchor In order to increase the fun of the video and beautify the video effect, the anchor usually adds some decorative effects such as beauty, fun texture or animation during recording, but usually there is a difference between the recording end and the playback device, and the composite video or decorative effect cannot be played. Recording videos cannot be played synchronously, resulting in a poor user experience.
  • the main object of the present invention is to provide a video synthesizing method and device, which aims to solve the technical problem that the recording end and the playback end device are different and cannot play the synthesized video or the decorative effect cannot be played synchronously with the recorded video.
  • a video synthesis method provided by the present invention includes the following steps:
  • the original video frame texture of the drawing window is mixed and encoded with the effect data of the local system, and the synthesized video is generated and output for the receiving end to decode and play the synthesized video.
  • the present invention further provides a video synthesizing apparatus, including:
  • a drawing module configured to create a drawing window, and establish a correspondence between the drawing window and the drawing layer, to draw the original video frame texture of the drawing layer buffer into the drawing window;
  • the encoding module is configured to mix and encode the original video frame texture of the drawing window with the effect data of the local system, and generate and output a composite video for the receiving end to decode and play the synthesized video.
  • the effect data is synthesized into the to-be-sent video at the video transmitting end based on the drawing layer, and the synthesized video is in a common format, so that the received encoded video can be decoded and played regardless of the type of device at the receiving end. It also solves the technical problem that the recorded video and the added decorative effect data are separately transmitted in the prior art and cannot be played synchronously, thereby improving the user experience.
  • FIG. 1 is a schematic flow chart of a first embodiment of a video synthesizing method according to the present invention
  • FIG. 2 is a schematic flowchart of a step of acquiring an original video frame in a second embodiment of a video synthesizing method according to the present invention
  • FIG. 3 is a schematic flowchart of a step of creating a drawing window in the third embodiment of the video synthesizing method according to the present invention.
  • FIG. 4 is a schematic flowchart of a process of encoding an original video frame texture and an effect data in a fourth embodiment of the video synthesizing method according to the present invention
  • FIG. 5 is a schematic diagram of a refinement process of implementing a video synthesis step in a fifth embodiment of a video synthesis method according to the present invention.
  • FIG. 6 is a schematic diagram of functional modules of a first embodiment of a video synthesizing apparatus according to the present invention.
  • FIG. 7 is a schematic diagram of a refinement function module of an acquisition module in a second embodiment of a video synthesizing apparatus according to the present invention.
  • FIG. 8 is a schematic diagram of a refinement function module of a drawing module in a third embodiment of the video synthesizing device of the present invention.
  • FIG. 9 is a schematic diagram of a refinement function module of an encoding module in a fourth embodiment of a video synthesizing apparatus according to the present invention.
  • FIG. 10 is a schematic diagram of a refinement function module of a coding unit in a fifth embodiment of a video synthesizing apparatus according to the present invention.
  • the present invention provides a video synthesis method.
  • the video synthesis method includes:
  • Step S10 acquiring an original video frame of the camera, converting the original video frame into an original video frame texture and buffering the drawing layer generated by the system coding library;
  • the video recording end can obtain the original video frame of the camera in various ways; for example, when the video recording end is a mobile phone, the original video frame can be obtained by the camera provided by the mobile phone, and when the video recording end is a computer, Get the original video frame from an external camera on your computer.
  • the video recording end is a video recording end with a camera
  • the operating system is an Android system.
  • the camera of the video recording end is opened by the Android system, and camera parameters such as aperture value, exposure control, sensitivity, focus distance, and the like are set; the camera preview is opened, and a camera preview window is created (GL). Surface View), while setting the renderer for the camera preview window, the above steps are based on the Open Graphics Library (Open GL) implementation in the Android system.
  • Open GL Open Graphics Library
  • the camera preview window is used to acquire the original video frame of the camera, that is, each time the camera updates a frame of data, the system will call the renderer on once.
  • the Draw Frame (GL10) method draws an image for the camera preview window, and generates an original video frame texture of the open graphics library based on the original video frame data of each frame of the acquired camera.
  • the texture is an image or photo used to add extra detail to the surface of the basic graphic.
  • Each two-bit texture has its own coordinate space, which ranges from (0,0) in one corner to (1,1) in the other corner.
  • you need to apply a texture to a rectangle you need to specify a texture coordinate for each vertex so that Open GL knows which part of that texture needs to be drawn onto the rectangle.
  • the original video frame and texture will be used as the texture, and (0,0), (1,0), (0,1), (1,1) will be used as the texture coordinates of the four vertices. After scaling the entire image, it is completely drawn on the rectangular area specified by the vertex coordinates.
  • the renderer creates a surface texture and a corresponding texture number based on each original video frame acquired, such that the on Draw of the renderer Frame can be created by surface texture (Surface Texure) and texture number (Texureld) get texture data and manipulate texture. That is, getting the original video frame through the renderer on Draw The Frame is converted to the corresponding original video frame texture. And generating, by the Android system code library (Mediacodec), a drawing layer for buffering the original video frame texture, and the code library will obtain the video stream to be encoded from the drawing layer.
  • Mediacodec Android system code library
  • Step S20 creating a drawing window, and establishing a correspondence relationship between the drawing window and the drawing layer, to draw the original video frame texture of the drawing layer buffer into the drawing window;
  • a drawing window is created based on the Android system to draw the original video frame texture of the drawing layer buffer into the drawing window; understandably, a correspondence between the drawing window and the drawing layer needs to be established.
  • an EGL is created by the system function egl Create Window Surface.
  • the function receives the drawing layer generated by the encoding library as a parameter, thus establishing the correspondence between the drawing window and the drawing layer.
  • the Android system encoding thread sets the EGL Surface by modifying "current", in order to make Open GL can draw on the created drawing window, and also needs to call EGL Surface.make through the system encoding thread.
  • the Current method completes the switching between the drawing layer and the drawing window, and draws the original video frame texture of the drawing layer into the drawing window according to the preset drawing parameters and the drawing process.
  • the drawing parameters can be preset vertex coordinates, texture coordinates and vertex shaders, fragment shaders, etc., according to Open
  • the drawing process of GL draws the original video frame texture on the drawing window.
  • vertex coordinates refer to normalized coordinates for specifying a drawing area of a rectangle.
  • all objects that need to be rendered are mapped to the range of [-1, 1] on the x and y axes.
  • the coordinates in this range are called normalized device coordinates, which are independent of the actual size of the screen. Or shape.
  • Open in Android system The GL occupies the entire display, so the range of [-1, 1] for the x-axis and y-axis is mapped to the pixel width and pixel height of the screen, respectively.
  • the vertex data of the original video frame is first read, the vertex shader is executed, and then each vertex can be controlled, and instructions such as rendering, determining vertex position texture coordinate transformation, etc. are executed; when all the attributes of the vertex are determined, the vertex is assembled.
  • Forming a primitive then rasterizing the primitive, receiving the processed power of a primitive, and converting it into fragments, each fragment corresponding to one pixel of the screen; performing a fragment shader, performing texture searching, determining texture coordinates;
  • the original video frame is written to the frame buffer, the surface of the drawing, and finally displayed on the screen.
  • Step S30 mixing and encoding the original video frame texture of the drawing window with the effect data of the local system, and generating and outputting the synthesized video, so that the receiving end decodes and plays the synthesized video.
  • the effect data can be a beauty, a filter, a fun texture, an animation effect, and the like.
  • the effect data may be data pre-stored in the system, or may be current data generated by the user based on the system program and cached in the system memory.
  • the effect data in the system memory is acquired based on the control instruction input by the user, and is mixed and encoded with the original video frame texture of the drawing window, and the new video is synthesized and output to the network push stream module for decoding and playback by the receiving end.
  • the original video frame is converted into the original video frame texture by the original video frame of the camera, and is cached in the drawing layer generated by the system coding library, and then the drawing window is created, and the drawing window and the drawing are established.
  • step S10 includes:
  • Step S11 creating a camera preview window to obtain an original video frame of the camera, and setting a renderer for the camera preview window;
  • Step S12 acquiring a surface texture and a corresponding texture number created by the renderer based on each original video frame, and converting the obtained original video frame into a original video frame texture by using a surface texture and a texture number;
  • step S13 the original video frame texture is cached in the drawing layer generated by the system coding library.
  • an open graphics library based on the Android system (Open GL)
  • Open GL an open graphics library based on the Android system
  • Camera class of Android system set camera parameters
  • open camera preview an open graphics library based on the Camera class of Android system
  • create camera preview window GL Surface View
  • the camera preview window is created to capture the original video frame of the camera.
  • Texture interface passes in the texture; this way, through the renderer's on Draw
  • the frame method converts the obtained original video frame into a original video frame texture by surface texture and texture number, and caches the drawing layer generated by the system coding library.
  • a camera preview window is created to acquire an original video frame of the camera, and a renderer is set for the camera preview window; and a surface created by the renderer based on each original video frame is acquired.
  • the texture and the corresponding texture number, and the obtained original video frame is converted into the original video frame texture by the surface texture and the texture number, and is cached in the drawing layer generated by the system coding library.
  • the camera preview window can be created by using the Android system independent of the window system and the hardware-independent open graphics library. It can be used on various computers running various operating systems to optimize the live video experience and efficiency.
  • the foregoing step S20 includes:
  • Step S21 creating a drawing window, and receiving the original video frame texture of the drawing layer buffer according to the system function, to establish a correspondence relationship between the drawing window and the drawing layer;
  • Step S22 according to the window switching rule in the system coding thread, complete the switching between the drawing layer and the drawing window, and draw the original video frame texture of the drawing layer into the drawing window according to the preset drawing parameters and the drawing process.
  • EGL Context is the thread where the current rendering code is located
  • EGL Surface is through the system function egl Create Window Created by Surface, this function receives the drawing layer generated by the system code library as a parameter to establish the correspondence between the drawing window and the drawing layer.
  • the original video frame texture buffered in the drawing layer needs to be drawn in the drawing window.
  • EGL Through the system encoding thread
  • the Surface.make Current method completes the switch between the drawing layer and the drawing window. Based on the drawing process of the open graphics library, the original video frame texture is drawn on the drawing window.
  • the original video frame texture is drawn, and then the obtained original video frame can be texture-drawn in the drawing window, and the effect data in the local system can also be drawn in the drawing window.
  • the creation of the drawing window is independent of the camera preview window, thereby avoiding the occurrence of a phenomenon such as a camera preview window, and improving the processing efficiency of the system.
  • the foregoing step S30 includes:
  • Step S31 acquiring corresponding effect data from the local system according to the received input instruction
  • Step S32 converting the effect data into a corresponding effect texture based on the system open graphics library, and drawing the effect texture in the drawing window;
  • Step S33 mixing and encoding the original video frame texture and the effect texture in the drawing window, and generating and outputting the synthesized video for the receiving end to decode and play the synthesized video.
  • effect data may be data pre-stored in the system, such as a map, an animation, etc., or may be current data generated by the user based on the system program and cached in the system memory, such as a filter or a beauty image. Wait.
  • the received input instruction first reading the image in the png or jpg format on the system disk, and generating the bitmap bitmap data into the memory, and then generating the effect texture of the two-bit image based on the glTexImage2D function of the open graphics library, and using Corresponding coordinates and shaders draw the effect texture on the drawing window.
  • the original video frame texture and the effect texture on the drawing window are mixed and encoded, and a new composite video is generated and output to the network push stream module for decoding and playback by the receiving end.
  • the processing steps of the effect data are further refined, and the recorded original video frame and the effect data are synthesized and encoded at the video recording end, so that the real-time transmission of the video synthesis is realized, and the playback end only needs to decode and play. Yes, the processing flow of the playback end is reduced, the live broadcast efficiency is optimized, and the user experience is improved.
  • the foregoing step S33 includes:
  • Step S331 mixing the original video frame texture and the effect texture in the drawing window into a new video frame, and outputting to the drawing layer corresponding to the drawing window;
  • Step S332 The current new video frame on the drawing layer is encoded according to a preset format based on the system coding library, and the synthesized video is generated and outputted for the receiving end to decode and play the synthesized video.
  • the original video frame texture and the effect texture on the drawing window are mixed into a new video frame, and the egl Swap in the system is called.
  • the Buffer function submits the current new video frame drawn in the drawing window and outputs it to the drawing layer corresponding to the current drawing window.
  • the system encoding library will use the drawing layer as the output object and the current new video frame on the drawing layer. According to the preset format encoding, it is output to the network push stream module, and the receiving end only needs to decode and play.
  • GL_SRC_ALPHA is the source blending factor sfactor and GL_ONE_MINUS_SRC_ALPHA is the target blending factor dfactor.
  • the original video frame texture and the effect texture in the drawing window are mixed into a new video frame, and output to a drawing layer corresponding to the drawing window; the current layer on the drawing layer is based on the system coding library.
  • the new video frame is encoded according to a preset format, and a composite video is generated and output for the receiving end to decode and play the synthesized video.
  • the processing steps of the effect texture and the original video frame texture are further optimized to achieve an efficient synchronized playback of the recorded video and the effect data.
  • the present invention further provides a video synthesizing apparatus.
  • the video synthesizing apparatus provided by the present invention includes:
  • the obtaining module 10 is configured to obtain an original video frame of the camera, convert the original video frame into an original video frame texture, and cache the image layer generated by the system encoding library;
  • the drawing module 20 is configured to create a drawing window, and establish a correspondence between the drawing window and the drawing layer, to draw the original video frame texture of the drawing layer buffer into the drawing window;
  • the encoding module 30 is configured to mix and encode the original video frame texture of the drawing window with the effect data of the local system, and generate and output a composite video for the receiving end to decode and play the synthesized video.
  • the specific embodiment of the video synthesizing device of the present invention expands the content, and the content corresponding to the video synthesizing method is basically the same, and will not be described again.
  • the effect data in the system memory is acquired based on the control instruction input by the user, and is mixed and encoded with the original video frame texture of the drawing window, and the new video is synthesized and output to the network push stream module for decoding and playback by the receiving end.
  • the original video frame of the camera is obtained by the obtaining module 10
  • the original video frame is converted into the original video frame texture and cached in the drawing layer generated by the system coding library
  • the drawing module 20 creates a drawing window and establishes a drawing window.
  • the encoding module 30 mixes and encodes the original video frame texture of the drawing window with the effect data of the local system.
  • the obtaining module 10 includes:
  • the obtaining unit 11 is configured to create a camera preview window to acquire an original video frame of the camera, and set a renderer for the camera preview window;
  • the converting unit 12 is configured to acquire a surface texture and a corresponding texture number created by the renderer based on each original video frame, and convert the obtained original video frame into a original video frame texture by using a surface texture and a texture number;
  • the buffer unit 13 is configured to cache the original video frame texture in a drawing layer generated by the system encoding library.
  • an open graphics library based on the Android system (Open GL)
  • Open GL an open graphics library based on the Android system
  • Camera class of Android system set camera parameters
  • open camera preview an open graphics library based on the Camera class of Android system
  • create camera preview window GL Surface View
  • the camera preview window is created to capture the original video frame of the camera.
  • Texture interface passes in the texture; this way, through the renderer's on Draw
  • the frame method converts the obtained original video frame into a original video frame texture by surface texture and texture number, and caches the drawing layer generated by the system coding library.
  • the acquiring unit 11 acquires a camera video preview window to acquire an original video frame of the camera, and sets a renderer for the camera preview window; the converting unit 12 passes the acquired original video frame through rendering.
  • the surface texture and texture number created by the device are converted into the original video frame texture, and are cached by the cache unit 13 in the drawing layer generated by the system code library.
  • the camera preview window can be created by using the Android system independent of the window system and the hardware-independent open graphics library. It can be used on various computers running various operating systems to optimize the live video experience and efficiency.
  • the drawing module 20 includes:
  • a drawing window creating unit 21 configured to create a drawing window, and receive an original video frame texture of the drawing layer buffer according to a system function, to establish a correspondence relationship between the drawing window and the drawing layer;
  • the drawing unit 22 is configured to switch between the drawing layer and the drawing window according to the system coding thread window switching rule, and draw the original video frame texture of the drawing layer into the drawing window according to the preset drawing parameters and the drawing process.
  • EGL Context is the thread where the current rendering code is located
  • EGL Surface is through the system function egl Create Window Created by Surface, this function receives the drawing layer generated by the system code library as a parameter to establish the correspondence between the drawing window and the drawing layer.
  • the original video frame texture buffered in the drawing layer needs to be drawn in the drawing window.
  • EGL Through the system encoding thread
  • the Surface.make Current method completes the switch between the drawing layer and the drawing window. Based on the drawing process of the open graphics library, the original video frame texture is drawn on the drawing window.
  • the original video frame texture is drawn, and then the obtained original video frame can be texture-drawn in the drawing window, and the effect data in the local system can also be drawn in the drawing window.
  • the creation of the drawing window is independent of the camera preview window, thereby avoiding the occurrence of a phenomenon such as a camera preview window, and improving the processing efficiency of the system.
  • the encoding module 30 includes:
  • the effect picture obtaining unit 31 is configured to obtain corresponding effect data from the local system according to the received input instruction;
  • the effect texture drawing unit 32 is configured to convert the effect data into a corresponding effect texture based on the system open graphics library, and draw the effect texture into the drawing window;
  • the encoding unit 33 is configured to mix and encode the original video frame texture and the effect texture in the drawing window, and generate and output a composite video for the receiving end to decode and play the synthesized video.
  • effect data may be data pre-stored in the system, such as a map, an animation, etc., or may be current data generated by the user based on the system program and cached in the system memory, such as a filter or a beauty image. Wait.
  • the effect picture obtaining unit 31 first obtains a picture of the system disk png or jpg format, and generates bitmap bitmap data into the memory, and then generates texture data of the two-bit image based on the glTexImage2D function of the open graphics library, and uses corresponding The coordinates and shader draw the effect texture on the drawing window. After the drawing is completed, the original video frame texture and the effect texture on the drawing window are mixed and encoded, and a new composite video is generated and output to the network push stream module for decoding and playback by the receiving end.
  • the processing steps of the effect data are further refined, and the recorded original video frame and the effect data are synthesized and encoded at the video recording end, so that the real-time transmission of the video synthesis is realized, and the playback end only needs to decode and play. Yes, the processing flow of the playback end is reduced, the live broadcast efficiency is optimized, and the user experience is improved.
  • the encoding unit 33 includes:
  • a mixing unit 331, configured to mix the original video frame texture and the effect texture in the drawing window into a new video frame, and output to a drawing layer corresponding to the drawing window;
  • the output unit 332 is configured to encode the current new video frame on the drawing layer according to the system encoding library according to a preset format, and generate and output a composite video for the receiving end to decode and play the synthesized video.
  • the original video frame texture and the effect texture on the drawing window are mixed into a new video frame, and the egl Swap in the system is called.
  • the Buffer function submits the current new video frame drawn in the drawing window and outputs it to the drawing layer corresponding to the current drawing window.
  • the system encoding library will use the drawing layer as the output object and the current new video frame on the drawing layer. According to the preset format encoding, it is output to the network push stream module, and the receiving end only needs to decode and play.
  • GL_SRC_ALPHA is the source blending factor sfactor and GL_ONE_MINUS_SRC_ALPHA is the target blending factor dfactor.
  • the original video frame texture and the effect texture in the drawing window are mixed into a new video frame by the mixing unit 331 and output to the drawing layer corresponding to the drawing window; the output unit 332 passes the system coding library.
  • the current new video frame on the drawing layer is encoded according to a preset format, and a composite video is generated and outputted for the receiving end to decode and play the synthesized video.
  • the processing steps of the effect texture and the original video frame texture are further optimized to achieve an efficient synchronization of the recorded video and the decorative effect.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Image Generation (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Processing (AREA)
  • Studio Circuits (AREA)

Abstract

L'invention concerne un procédé de synthèse de vidéo qui consiste : à acquérir une trame vidéo d'origine d'une caméra, à convertir la trame vidéo d'origine en une texture de trame vidéo d'origine et à stocker temporairement cette dernière dans une couche de dessin générée par une bibliothèque de codage de système ; à créer une fenêtre de dessin et à établir une relation de correspondance entre la fenêtre de dessin et la couche de dessin de sorte à dessiner la texture de trame vidéo d'origine stockée temporairement dans la couche de dessin sur la fenêtre de dessin ; à exécuter un codage hybride sur la texture de trame vidéo d'origine de la fenêtre de dessin et des données d'effet d'un système local de sorte à générer et à produire une vidéo synthétisée pour qu'une extrémité de réception décode et lise la vidéo synthétisée. L'invention concerne en outre un dispositif de synthèse de vidéo. La solution technique fournie par l'invention résout le problème technique selon lequel une vidéo synthétisée ne peut pas être lue parce qu'un terminal d'enregistrement et un terminal de lecture ont différents dispositifs ou selon lequel un effet décoratif et une vidéo enregistrée ne peuvent pas être lus de manière synchrone.
PCT/CN2016/112917 2016-11-28 2016-12-29 Dispositif et procédé de synthèse de vidéo WO2018094814A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201611085102.3 2016-11-28
CN201611085102.3A CN106534880A (zh) 2016-11-28 2016-11-28 视频合成方法及装置

Publications (1)

Publication Number Publication Date
WO2018094814A1 true WO2018094814A1 (fr) 2018-05-31

Family

ID=58354374

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/112917 WO2018094814A1 (fr) 2016-11-28 2016-12-29 Dispositif et procédé de synthèse de vidéo

Country Status (2)

Country Link
CN (1) CN106534880A (fr)
WO (1) WO2018094814A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110782387A (zh) * 2018-07-30 2020-02-11 优视科技有限公司 图像处理方法、装置、图像处理器及电子设备
CN111754607A (zh) * 2019-03-27 2020-10-09 北京小米移动软件有限公司 图片处理方法及装置、电子设备、计算机可读存储介质
CN112383793A (zh) * 2020-11-12 2021-02-19 咪咕视讯科技有限公司 画面的合成方法、装置、电子设备及存储介质
WO2024125328A1 (fr) * 2022-09-06 2024-06-20 北京字跳网络技术有限公司 Procédé et appareil de traitement de trame d'image de diffusion en continu en direct, dispositif, support de stockage lisible et produit

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108449641B (zh) * 2018-03-28 2021-04-27 聚好看科技股份有限公司 播放媒体流的方法、装置、计算机设备和存储介质
CN109922360B (zh) * 2019-03-07 2022-02-11 腾讯科技（深圳）有限公司 视频处理方法、装置及存储介质
CN110278479B (zh) * 2019-06-04 2021-10-29 海信视像科技股份有限公司 一种图片显示方法及电视机
CN110850983B (zh) * 2019-11-13 2020-11-24 腾讯科技（深圳）有限公司 视频直播中的虚拟对象控制方法、装置和存储介质
CN111614906B (zh) * 2020-05-29 2022-02-22 阿波罗智联(北京)科技有限公司 一种图像预处理方法、装置、电子设备及存储介质
CN112118397B (zh) * 2020-09-23 2021-06-22 腾讯科技（深圳）有限公司 一种视频合成的方法、相关装置、设备以及存储介质
CN114390354B (zh) * 2020-10-21 2024-05-10 西安诺瓦星云科技股份有限公司 节目制作方法、装置和系统及计算机可读存储介质
CN112486606B (zh) * 2020-11-19 2022-08-12 湖南麒麟信安科技股份有限公司 一种基于Android系统的云桌面显示优化方法及系统
CN112581575B (zh) * 2020-12-05 2024-05-03 西安翔腾微电子科技有限公司 一种外视频做纹理系统
CN114900736A (zh) * 2022-03-28 2022-08-12 网易（杭州）网络有限公司 视频生成方法、装置和电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130293537A1 (en) * 2011-01-05 2013-11-07 Cisco Technology Inc. Coordinated 2-Dimensional and 3-Dimensional Graphics Processing
CN103677828A (zh) * 2013-12-10 2014-03-26 华为技术有限公司 一种图层绘制方法、绘图引擎及终端设备
CN105204881A (zh) * 2014-06-25 2015-12-30 中兴通讯股份有限公司 在Android系统上运行J2me应用程序的方法及设备
CN105933724A (zh) * 2016-05-23 2016-09-07 福建星网视易信息系统有限公司 视频制作方法、装置及系统

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102447873A (zh) * 2010-10-13 2012-05-09 张明 哈哈视频网络视频聊天娱乐辅助系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130293537A1 (en) * 2011-01-05 2013-11-07 Cisco Technology Inc. Coordinated 2-Dimensional and 3-Dimensional Graphics Processing
CN103677828A (zh) * 2013-12-10 2014-03-26 华为技术有限公司 一种图层绘制方法、绘图引擎及终端设备
CN105204881A (zh) * 2014-06-25 2015-12-30 中兴通讯股份有限公司 在Android系统上运行J2me应用程序的方法及设备
CN105933724A (zh) * 2016-05-23 2016-09-07 福建星网视易信息系统有限公司 视频制作方法、装置及系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
QIAO, SHAOJIE ET AL: "High-speed Image Rendering Method Based on OpenGL", APPLICATION RESEARCH OF COMPUTERS, vol. 25, no. 5, 31 May 2008 (2008-05-31), pages 1590 - 1591 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110782387A (zh) * 2018-07-30 2020-02-11 优视科技有限公司 图像处理方法、装置、图像处理器及电子设备
CN110782387B (zh) * 2018-07-30 2023-09-22 阿里巴巴（中国）有限公司 图像处理方法、装置、图像处理器及电子设备
CN111754607A (zh) * 2019-03-27 2020-10-09 北京小米移动软件有限公司 图片处理方法及装置、电子设备、计算机可读存储介质
CN112383793A (zh) * 2020-11-12 2021-02-19 咪咕视讯科技有限公司 画面的合成方法、装置、电子设备及存储介质
CN112383793B (zh) * 2020-11-12 2023-07-07 咪咕视讯科技有限公司 画面的合成方法、装置、电子设备及存储介质
WO2024125328A1 (fr) * 2022-09-06 2024-06-20 北京字跳网络技术有限公司 Procédé et appareil de traitement de trame d'image de diffusion en continu en direct, dispositif, support de stockage lisible et produit

Also Published As

Publication number Publication date
CN106534880A (zh) 2017-03-22

Similar Documents

Publication Publication Date Title
WO2018094814A1 (fr) Dispositif et procédé de synthèse de vidéo
WO2018034462A1 (fr) Appareil d'affichage d'image, et procédé de commande correspondant
JP5182957B2 (ja) アスペクト比変換のための画像データの可変スケーリング
WO2018018681A1 (fr) Procédé et dispositif de prévisualisation de programme vidéo
WO2015009107A1 (fr) Procédé et appareil permettant de générer une image à afficher ayant une résolution 3k pour l'écran d'un terminal mobile
WO2017088318A1 (fr) Procédé et dispositif de traitement d'affichage d'interface utilisateur
WO2016165556A1 (fr) Procédé, dispositif et système de traitement de données pour un flux vidéo
WO2016091011A1 (fr) Dispositif et procédé de commutation de sous-titres
WO2018192253A1 (fr) Procédé et appareil de transmission de signaux vidéo
WO2020067759A1 (fr) Procédé de commande d'appareil d'affichage et appareil d'affichage l'utilisant
WO2019010926A1 (fr) Procédé et appareil de push (pousser) de publicités, et support de stockage lisible par ordinateur
WO2017206368A1 (fr) Procédé et appareil de commutation d'image de plage dynamique élevée
CN107743637B (zh) 用于处理外围图像的方法和设备
WO2017020649A1 (fr) Procédé de commande de lecture audio/vidéo et dispositif associé
WO2014048239A1 (fr) Procédé de lecture sur une télévision intelligente et télévision intelligente
WO2017107384A1 (fr) Procédé d'affichage d'image de dispositif d'affichage à cristaux liquides, et dispositif d'affichage à cristaux liquides
WO2019125036A1 (fr) Procédé de traitement d'image et appareil d'affichage associé
WO2017113600A1 (fr) Procédé et dispositif de lecture de vidéo
WO2015169209A1 (fr) Procédé, appareil et dispositif électronique de génération de données d'animation
WO2019221423A1 (fr) Dispositif électronique, procédé de commande associé et support d'enregistrement
WO2017190451A1 (fr) Procédé et appareil de poussée d'image
WO2017201893A1 (fr) Procédé et dispositif de traitement de vidéo
WO2020171657A1 (fr) Dispositif d'affichage et procédé d'affichage d'image associé
WO2018190446A1 (fr) Procédé et appareil de traitement d'image
WO2018058831A1 (fr) Procédé et appareil de réglage de luminosité de rétroéclairage d'un écran d'affichage

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16922047

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 15/10/2019)

122 Ep: pct application non-entry in european phase

Ref document number: 16922047

Country of ref document: EP

Kind code of ref document: A1