WO2018094814A1 - 视频合成方法及装置 - Google Patents

视频合成方法及装置 Download PDF

Info

Publication number
WO2018094814A1
WO2018094814A1 PCT/CN2016/112917 CN2016112917W WO2018094814A1 WO 2018094814 A1 WO2018094814 A1 WO 2018094814A1 CN 2016112917 W CN2016112917 W CN 2016112917W WO 2018094814 A1 WO2018094814 A1 WO 2018094814A1
Authority
WO
WIPO (PCT)
Prior art keywords
texture
video frame
window
original video
video
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/112917
Other languages
English (en)
French (fr)
Inventor
曾红
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen TCL Digital Technology Co Ltd
Original Assignee
Shenzhen TCL Digital Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen TCL Digital Technology Co Ltd filed Critical Shenzhen TCL Digital Technology Co Ltd
Publication of WO2018094814A1 publication Critical patent/WO2018094814A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23424—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234318—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into objects, e.g. MPEG-4 objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44004—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving video buffer management, e.g. video decoder buffer or video display buffer
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4

Definitions

  • the present invention relates to the field of communications technologies, and in particular, to a video synthesis method and apparatus.
  • the Internet field has entered the live broadcast era from the era of the text, the era of pictures, and the short video era.
  • Mobile video live broadcasting has become one of the most eye-catching areas of the Internet.
  • the anchor In order to increase the fun of the video and beautify the video effect, the anchor usually adds some decorative effects such as beauty, fun texture or animation during recording, but usually there is a difference between the recording end and the playback device, and the composite video or decorative effect cannot be played. Recording videos cannot be played synchronously, resulting in a poor user experience.
  • the main object of the present invention is to provide a video synthesizing method and device, which aims to solve the technical problem that the recording end and the playback end device are different and cannot play the synthesized video or the decorative effect cannot be played synchronously with the recorded video.
  • a video synthesis method provided by the present invention includes the following steps:
  • the original video frame texture of the drawing window is mixed and encoded with the effect data of the local system, and the synthesized video is generated and output for the receiving end to decode and play the synthesized video.
  • the present invention further provides a video synthesizing apparatus, including:
  • a drawing module configured to create a drawing window, and establish a correspondence between the drawing window and the drawing layer, to draw the original video frame texture of the drawing layer buffer into the drawing window;
  • the encoding module is configured to mix and encode the original video frame texture of the drawing window with the effect data of the local system, and generate and output a composite video for the receiving end to decode and play the synthesized video.
  • the effect data is synthesized into the to-be-sent video at the video transmitting end based on the drawing layer, and the synthesized video is in a common format, so that the received encoded video can be decoded and played regardless of the type of device at the receiving end. It also solves the technical problem that the recorded video and the added decorative effect data are separately transmitted in the prior art and cannot be played synchronously, thereby improving the user experience.
  • FIG. 1 is a schematic flow chart of a first embodiment of a video synthesizing method according to the present invention
  • FIG. 2 is a schematic flowchart of a step of acquiring an original video frame in a second embodiment of a video synthesizing method according to the present invention
  • FIG. 3 is a schematic flowchart of a step of creating a drawing window in the third embodiment of the video synthesizing method according to the present invention.
  • FIG. 4 is a schematic flowchart of a process of encoding an original video frame texture and an effect data in a fourth embodiment of the video synthesizing method according to the present invention
  • FIG. 5 is a schematic diagram of a refinement process of implementing a video synthesis step in a fifth embodiment of a video synthesis method according to the present invention.
  • FIG. 6 is a schematic diagram of functional modules of a first embodiment of a video synthesizing apparatus according to the present invention.
  • FIG. 7 is a schematic diagram of a refinement function module of an acquisition module in a second embodiment of a video synthesizing apparatus according to the present invention.
  • FIG. 8 is a schematic diagram of a refinement function module of a drawing module in a third embodiment of the video synthesizing device of the present invention.
  • FIG. 9 is a schematic diagram of a refinement function module of an encoding module in a fourth embodiment of a video synthesizing apparatus according to the present invention.
  • FIG. 10 is a schematic diagram of a refinement function module of a coding unit in a fifth embodiment of a video synthesizing apparatus according to the present invention.
  • the present invention provides a video synthesis method.
  • the video synthesis method includes:
  • Step S10 acquiring an original video frame of the camera, converting the original video frame into an original video frame texture and buffering the drawing layer generated by the system coding library;
  • the video recording end can obtain the original video frame of the camera in various ways; for example, when the video recording end is a mobile phone, the original video frame can be obtained by the camera provided by the mobile phone, and when the video recording end is a computer, Get the original video frame from an external camera on your computer.
  • the video recording end is a video recording end with a camera
  • the operating system is an Android system.
  • the camera of the video recording end is opened by the Android system, and camera parameters such as aperture value, exposure control, sensitivity, focus distance, and the like are set; the camera preview is opened, and a camera preview window is created (GL). Surface View), while setting the renderer for the camera preview window, the above steps are based on the Open Graphics Library (Open GL) implementation in the Android system.
  • Open GL Open Graphics Library
  • the camera preview window is used to acquire the original video frame of the camera, that is, each time the camera updates a frame of data, the system will call the renderer on once.
  • the Draw Frame (GL10) method draws an image for the camera preview window, and generates an original video frame texture of the open graphics library based on the original video frame data of each frame of the acquired camera.
  • the texture is an image or photo used to add extra detail to the surface of the basic graphic.
  • Each two-bit texture has its own coordinate space, which ranges from (0,0) in one corner to (1,1) in the other corner.
  • you need to apply a texture to a rectangle you need to specify a texture coordinate for each vertex so that Open GL knows which part of that texture needs to be drawn onto the rectangle.
  • the original video frame and texture will be used as the texture, and (0,0), (1,0), (0,1), (1,1) will be used as the texture coordinates of the four vertices. After scaling the entire image, it is completely drawn on the rectangular area specified by the vertex coordinates.
  • the renderer creates a surface texture and a corresponding texture number based on each original video frame acquired, such that the on Draw of the renderer Frame can be created by surface texture (Surface Texure) and texture number (Texureld) get texture data and manipulate texture. That is, getting the original video frame through the renderer on Draw The Frame is converted to the corresponding original video frame texture. And generating, by the Android system code library (Mediacodec), a drawing layer for buffering the original video frame texture, and the code library will obtain the video stream to be encoded from the drawing layer.
  • Mediacodec Android system code library
  • Step S20 creating a drawing window, and establishing a correspondence relationship between the drawing window and the drawing layer, to draw the original video frame texture of the drawing layer buffer into the drawing window;
  • a drawing window is created based on the Android system to draw the original video frame texture of the drawing layer buffer into the drawing window; understandably, a correspondence between the drawing window and the drawing layer needs to be established.
  • an EGL is created by the system function egl Create Window Surface.
  • the function receives the drawing layer generated by the encoding library as a parameter, thus establishing the correspondence between the drawing window and the drawing layer.
  • the Android system encoding thread sets the EGL Surface by modifying "current", in order to make Open GL can draw on the created drawing window, and also needs to call EGL Surface.make through the system encoding thread.
  • the Current method completes the switching between the drawing layer and the drawing window, and draws the original video frame texture of the drawing layer into the drawing window according to the preset drawing parameters and the drawing process.
  • the drawing parameters can be preset vertex coordinates, texture coordinates and vertex shaders, fragment shaders, etc., according to Open
  • the drawing process of GL draws the original video frame texture on the drawing window.
  • vertex coordinates refer to normalized coordinates for specifying a drawing area of a rectangle.
  • all objects that need to be rendered are mapped to the range of [-1, 1] on the x and y axes.
  • the coordinates in this range are called normalized device coordinates, which are independent of the actual size of the screen. Or shape.
  • Open in Android system The GL occupies the entire display, so the range of [-1, 1] for the x-axis and y-axis is mapped to the pixel width and pixel height of the screen, respectively.
  • the vertex data of the original video frame is first read, the vertex shader is executed, and then each vertex can be controlled, and instructions such as rendering, determining vertex position texture coordinate transformation, etc. are executed; when all the attributes of the vertex are determined, the vertex is assembled.
  • Forming a primitive then rasterizing the primitive, receiving the processed power of a primitive, and converting it into fragments, each fragment corresponding to one pixel of the screen; performing a fragment shader, performing texture searching, determining texture coordinates;
  • the original video frame is written to the frame buffer, the surface of the drawing, and finally displayed on the screen.
  • Step S30 mixing and encoding the original video frame texture of the drawing window with the effect data of the local system, and generating and outputting the synthesized video, so that the receiving end decodes and plays the synthesized video.
  • the effect data can be a beauty, a filter, a fun texture, an animation effect, and the like.
  • the effect data may be data pre-stored in the system, or may be current data generated by the user based on the system program and cached in the system memory.
  • the effect data in the system memory is acquired based on the control instruction input by the user, and is mixed and encoded with the original video frame texture of the drawing window, and the new video is synthesized and output to the network push stream module for decoding and playback by the receiving end.
  • the original video frame is converted into the original video frame texture by the original video frame of the camera, and is cached in the drawing layer generated by the system coding library, and then the drawing window is created, and the drawing window and the drawing are established.
  • step S10 includes:
  • Step S11 creating a camera preview window to obtain an original video frame of the camera, and setting a renderer for the camera preview window;
  • Step S12 acquiring a surface texture and a corresponding texture number created by the renderer based on each original video frame, and converting the obtained original video frame into a original video frame texture by using a surface texture and a texture number;
  • step S13 the original video frame texture is cached in the drawing layer generated by the system coding library.
  • an open graphics library based on the Android system (Open GL)
  • Open GL an open graphics library based on the Android system
  • Camera class of Android system set camera parameters
  • open camera preview an open graphics library based on the Camera class of Android system
  • create camera preview window GL Surface View
  • the camera preview window is created to capture the original video frame of the camera.
  • Texture interface passes in the texture; this way, through the renderer's on Draw
  • the frame method converts the obtained original video frame into a original video frame texture by surface texture and texture number, and caches the drawing layer generated by the system coding library.
  • a camera preview window is created to acquire an original video frame of the camera, and a renderer is set for the camera preview window; and a surface created by the renderer based on each original video frame is acquired.
  • the texture and the corresponding texture number, and the obtained original video frame is converted into the original video frame texture by the surface texture and the texture number, and is cached in the drawing layer generated by the system coding library.
  • the camera preview window can be created by using the Android system independent of the window system and the hardware-independent open graphics library. It can be used on various computers running various operating systems to optimize the live video experience and efficiency.
  • the foregoing step S20 includes:
  • Step S21 creating a drawing window, and receiving the original video frame texture of the drawing layer buffer according to the system function, to establish a correspondence relationship between the drawing window and the drawing layer;
  • Step S22 according to the window switching rule in the system coding thread, complete the switching between the drawing layer and the drawing window, and draw the original video frame texture of the drawing layer into the drawing window according to the preset drawing parameters and the drawing process.
  • EGL Context is the thread where the current rendering code is located
  • EGL Surface is through the system function egl Create Window Created by Surface, this function receives the drawing layer generated by the system code library as a parameter to establish the correspondence between the drawing window and the drawing layer.
  • the original video frame texture buffered in the drawing layer needs to be drawn in the drawing window.
  • EGL Through the system encoding thread
  • the Surface.make Current method completes the switch between the drawing layer and the drawing window. Based on the drawing process of the open graphics library, the original video frame texture is drawn on the drawing window.
  • the original video frame texture is drawn, and then the obtained original video frame can be texture-drawn in the drawing window, and the effect data in the local system can also be drawn in the drawing window.
  • the creation of the drawing window is independent of the camera preview window, thereby avoiding the occurrence of a phenomenon such as a camera preview window, and improving the processing efficiency of the system.
  • the foregoing step S30 includes:
  • Step S31 acquiring corresponding effect data from the local system according to the received input instruction
  • Step S32 converting the effect data into a corresponding effect texture based on the system open graphics library, and drawing the effect texture in the drawing window;
  • Step S33 mixing and encoding the original video frame texture and the effect texture in the drawing window, and generating and outputting the synthesized video for the receiving end to decode and play the synthesized video.
  • effect data may be data pre-stored in the system, such as a map, an animation, etc., or may be current data generated by the user based on the system program and cached in the system memory, such as a filter or a beauty image. Wait.
  • the received input instruction first reading the image in the png or jpg format on the system disk, and generating the bitmap bitmap data into the memory, and then generating the effect texture of the two-bit image based on the glTexImage2D function of the open graphics library, and using Corresponding coordinates and shaders draw the effect texture on the drawing window.
  • the original video frame texture and the effect texture on the drawing window are mixed and encoded, and a new composite video is generated and output to the network push stream module for decoding and playback by the receiving end.
  • the processing steps of the effect data are further refined, and the recorded original video frame and the effect data are synthesized and encoded at the video recording end, so that the real-time transmission of the video synthesis is realized, and the playback end only needs to decode and play. Yes, the processing flow of the playback end is reduced, the live broadcast efficiency is optimized, and the user experience is improved.
  • the foregoing step S33 includes:
  • Step S331 mixing the original video frame texture and the effect texture in the drawing window into a new video frame, and outputting to the drawing layer corresponding to the drawing window;
  • Step S332 The current new video frame on the drawing layer is encoded according to a preset format based on the system coding library, and the synthesized video is generated and outputted for the receiving end to decode and play the synthesized video.
  • the original video frame texture and the effect texture on the drawing window are mixed into a new video frame, and the egl Swap in the system is called.
  • the Buffer function submits the current new video frame drawn in the drawing window and outputs it to the drawing layer corresponding to the current drawing window.
  • the system encoding library will use the drawing layer as the output object and the current new video frame on the drawing layer. According to the preset format encoding, it is output to the network push stream module, and the receiving end only needs to decode and play.
  • GL_SRC_ALPHA is the source blending factor sfactor and GL_ONE_MINUS_SRC_ALPHA is the target blending factor dfactor.
  • the original video frame texture and the effect texture in the drawing window are mixed into a new video frame, and output to a drawing layer corresponding to the drawing window; the current layer on the drawing layer is based on the system coding library.
  • the new video frame is encoded according to a preset format, and a composite video is generated and output for the receiving end to decode and play the synthesized video.
  • the processing steps of the effect texture and the original video frame texture are further optimized to achieve an efficient synchronized playback of the recorded video and the effect data.
  • the present invention further provides a video synthesizing apparatus.
  • the video synthesizing apparatus provided by the present invention includes:
  • the obtaining module 10 is configured to obtain an original video frame of the camera, convert the original video frame into an original video frame texture, and cache the image layer generated by the system encoding library;
  • the drawing module 20 is configured to create a drawing window, and establish a correspondence between the drawing window and the drawing layer, to draw the original video frame texture of the drawing layer buffer into the drawing window;
  • the encoding module 30 is configured to mix and encode the original video frame texture of the drawing window with the effect data of the local system, and generate and output a composite video for the receiving end to decode and play the synthesized video.
  • the specific embodiment of the video synthesizing device of the present invention expands the content, and the content corresponding to the video synthesizing method is basically the same, and will not be described again.
  • the effect data in the system memory is acquired based on the control instruction input by the user, and is mixed and encoded with the original video frame texture of the drawing window, and the new video is synthesized and output to the network push stream module for decoding and playback by the receiving end.
  • the original video frame of the camera is obtained by the obtaining module 10
  • the original video frame is converted into the original video frame texture and cached in the drawing layer generated by the system coding library
  • the drawing module 20 creates a drawing window and establishes a drawing window.
  • the encoding module 30 mixes and encodes the original video frame texture of the drawing window with the effect data of the local system.
  • the obtaining module 10 includes:
  • the obtaining unit 11 is configured to create a camera preview window to acquire an original video frame of the camera, and set a renderer for the camera preview window;
  • the converting unit 12 is configured to acquire a surface texture and a corresponding texture number created by the renderer based on each original video frame, and convert the obtained original video frame into a original video frame texture by using a surface texture and a texture number;
  • the buffer unit 13 is configured to cache the original video frame texture in a drawing layer generated by the system encoding library.
  • an open graphics library based on the Android system (Open GL)
  • Open GL an open graphics library based on the Android system
  • Camera class of Android system set camera parameters
  • open camera preview an open graphics library based on the Camera class of Android system
  • create camera preview window GL Surface View
  • the camera preview window is created to capture the original video frame of the camera.
  • Texture interface passes in the texture; this way, through the renderer's on Draw
  • the frame method converts the obtained original video frame into a original video frame texture by surface texture and texture number, and caches the drawing layer generated by the system coding library.
  • the acquiring unit 11 acquires a camera video preview window to acquire an original video frame of the camera, and sets a renderer for the camera preview window; the converting unit 12 passes the acquired original video frame through rendering.
  • the surface texture and texture number created by the device are converted into the original video frame texture, and are cached by the cache unit 13 in the drawing layer generated by the system code library.
  • the camera preview window can be created by using the Android system independent of the window system and the hardware-independent open graphics library. It can be used on various computers running various operating systems to optimize the live video experience and efficiency.
  • the drawing module 20 includes:
  • a drawing window creating unit 21 configured to create a drawing window, and receive an original video frame texture of the drawing layer buffer according to a system function, to establish a correspondence relationship between the drawing window and the drawing layer;
  • the drawing unit 22 is configured to switch between the drawing layer and the drawing window according to the system coding thread window switching rule, and draw the original video frame texture of the drawing layer into the drawing window according to the preset drawing parameters and the drawing process.
  • EGL Context is the thread where the current rendering code is located
  • EGL Surface is through the system function egl Create Window Created by Surface, this function receives the drawing layer generated by the system code library as a parameter to establish the correspondence between the drawing window and the drawing layer.
  • the original video frame texture buffered in the drawing layer needs to be drawn in the drawing window.
  • EGL Through the system encoding thread
  • the Surface.make Current method completes the switch between the drawing layer and the drawing window. Based on the drawing process of the open graphics library, the original video frame texture is drawn on the drawing window.
  • the original video frame texture is drawn, and then the obtained original video frame can be texture-drawn in the drawing window, and the effect data in the local system can also be drawn in the drawing window.
  • the creation of the drawing window is independent of the camera preview window, thereby avoiding the occurrence of a phenomenon such as a camera preview window, and improving the processing efficiency of the system.
  • the encoding module 30 includes:
  • the effect picture obtaining unit 31 is configured to obtain corresponding effect data from the local system according to the received input instruction;
  • the effect texture drawing unit 32 is configured to convert the effect data into a corresponding effect texture based on the system open graphics library, and draw the effect texture into the drawing window;
  • the encoding unit 33 is configured to mix and encode the original video frame texture and the effect texture in the drawing window, and generate and output a composite video for the receiving end to decode and play the synthesized video.
  • effect data may be data pre-stored in the system, such as a map, an animation, etc., or may be current data generated by the user based on the system program and cached in the system memory, such as a filter or a beauty image. Wait.
  • the effect picture obtaining unit 31 first obtains a picture of the system disk png or jpg format, and generates bitmap bitmap data into the memory, and then generates texture data of the two-bit image based on the glTexImage2D function of the open graphics library, and uses corresponding The coordinates and shader draw the effect texture on the drawing window. After the drawing is completed, the original video frame texture and the effect texture on the drawing window are mixed and encoded, and a new composite video is generated and output to the network push stream module for decoding and playback by the receiving end.
  • the processing steps of the effect data are further refined, and the recorded original video frame and the effect data are synthesized and encoded at the video recording end, so that the real-time transmission of the video synthesis is realized, and the playback end only needs to decode and play. Yes, the processing flow of the playback end is reduced, the live broadcast efficiency is optimized, and the user experience is improved.
  • the encoding unit 33 includes:
  • a mixing unit 331, configured to mix the original video frame texture and the effect texture in the drawing window into a new video frame, and output to a drawing layer corresponding to the drawing window;
  • the output unit 332 is configured to encode the current new video frame on the drawing layer according to the system encoding library according to a preset format, and generate and output a composite video for the receiving end to decode and play the synthesized video.
  • the original video frame texture and the effect texture on the drawing window are mixed into a new video frame, and the egl Swap in the system is called.
  • the Buffer function submits the current new video frame drawn in the drawing window and outputs it to the drawing layer corresponding to the current drawing window.
  • the system encoding library will use the drawing layer as the output object and the current new video frame on the drawing layer. According to the preset format encoding, it is output to the network push stream module, and the receiving end only needs to decode and play.
  • GL_SRC_ALPHA is the source blending factor sfactor and GL_ONE_MINUS_SRC_ALPHA is the target blending factor dfactor.
  • the original video frame texture and the effect texture in the drawing window are mixed into a new video frame by the mixing unit 331 and output to the drawing layer corresponding to the drawing window; the output unit 332 passes the system coding library.
  • the current new video frame on the drawing layer is encoded according to a preset format, and a composite video is generated and outputted for the receiving end to decode and play the synthesized video.
  • the processing steps of the effect texture and the original video frame texture are further optimized to achieve an efficient synchronization of the recorded video and the decorative effect.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Generation (AREA)
  • Image Processing (AREA)
  • Studio Circuits (AREA)

Abstract

本发明公开了一种视频合成方法，包括以下步骤：获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层；创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口；将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频。本发明还公开了一种视频合成装置。本发明提供的技术方案解决了录制端与播放端因设备不同而不能播放合成视频或是装饰效果与录制视频不能同步播放的技术问题。

Description

视频合成方法及装置
技术领域
本发明涉及通讯技术领域，尤其涉及一种视频合成方法及装置。
背景技术
随着4G时代的到来，互联网领域已从文字时代、图片时代、短视频时代进入到了直播时代，移动视频直播已经成为当前互联网表现最抢眼的领域之一。为了增加视频的趣味性，美化视频效果，主播通常会在录制时增加一些美颜、趣味贴图或者动画等装饰效果，但通常存在录制端与播放端设备不同而不能播放合成视频或是装饰效果与录制视频不能同步播放的问题，导致用户体验较差。
发明内容
本发明的主要目的在于提供一种视频合成方法及装置，旨在解决录制端与播放端设备不同而不能播放合成视频或是装饰效果与录制视频不能同步播放的技术问题。
为实现上述目的，本发明提供的一种视频合成方法包括以下步骤：
获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层；
创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口；
将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
此外，为实现上述目的，本发明还提供一种视频合成装置，包括：
获取模块，用以获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层；
绘制模块，用以创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口；
编码模块，用以将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
本发明提供的技术方案中，基于绘图层在视频发送端将效果数据合成至待发送视频中，合成视频为通用格式，使得不管接收端是何种类型设备，都能对接收的编码视频解码播放，也解决了现有技术中录制视频与增加的装饰效果数据单独发送而造成的不能同步播放的技术问题，提高了用户体验。
附图说明
图1为本发明视频合成方法第一实施例的流程示意图；
图2为本发明视频合成方法第二实施例中获取原始视频帧步骤的细化流程示意图；
图3为本发明视频合成方法第三实施例中创建绘图窗口步骤的细化流程示意图；
图4为本发明视频合成方法第四实施例中将原始视频帧纹理与效果数据编码步骤的细化流程示意图；
图5为本发明视频合成方法第五实施例中实现视频合成步骤的细化流程示意图；
图6为本发明视频合成装置第一实施例的功能模块示意图；
图7为本发明视频合成装置第二实施例中获取模块的细化功能模块示意图；
图8为本发明视频合成装置第三实施例中绘制模块的细化功能模块示意图；
图9为本发明视频合成装置第四实施例中编码模块的细化功能模块示意图；
图10为本发明视频合成装置第五实施例中编码单元的细化功能模块示意图。
本发明目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
具体实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。
本发明提供一种视频合成方法，参照图1，在一实施例中，该视频合成方法包括：
步骤S10，获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层；
可以理解地，视频录制端可以通过多种方式获取相机的原始视频帧；例如，当视频录制端为手机时，可以通过手机自带的相机获取原始视频帧，当视频录制端为电脑时，可以通过电脑外接摄像头来获取原始视频帧。本实施例中，视频录制端为自带有相机的视频录制端，且操作系统为Android系统。
具体地，通过Android系统打开视频录制端的相机，并设置相机参数，例如光圈值、曝光控制、感光度、对焦距离等参数的设置；打开摄像头预览，并创建相机预览窗口（GL Surface View），同时为相机预览窗口设置渲染器（Render），上述步骤均是基于Android系统中开放图形库（Open GL）实现。
所述相机预览窗口用以获取相机的原始视频帧，也就是说，相机每更新一帧数据，系统就会调用一次渲染器的on Draw Frame（GL10）方法为相机预览窗口绘制图像，根据获取的相机每一帧的原始视频帧数据生成开放图形库的原始视频帧纹理。
需要说明的是，纹理就是一个图像或照片，用于在基本图形的表面加入额外的细节。每个二位的纹理都有其自己的坐标空间，其范围是从一个拐角的（0，0）到另一个拐角的(1，1)。当需要把一个纹理应用于一个矩形的时候，因此需要为每个顶点指定一个纹理坐标，以便Open GL知道需要用那个纹理的哪个部分画到矩形上。Android系统中，将以原始的视频帧和贴图作为纹理，使用(0，0)、(1，0)、(0，1)、(1，1)作为四个顶点的纹理坐标，其效果就是将整个图像缩放后，完整地绘制在顶点坐标指定的矩形区域上。
具体地，所述渲染器基于获取的每一原始视屏帧创建表面纹理和对应的纹理编号，这样，在渲染器的on Draw Frame中就可以通过创建的表面纹理（Surface Texure）和纹理编号（Texureld）获取纹理数据和操作纹理了。也就是说，获取所述原始视频帧通过渲染器on Draw Frame转换为对应的原始视频帧纹理。并通过Android系统编码库（Mediacodec）生成一个绘图层（surface），用以缓存所述原始视频帧纹理，所述编码库将从该绘图层获取待编码的视频流。
步骤S20，创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口；
基于Android系统创建绘图窗口，用以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口；可以理解地，需要建立绘图窗口与绘图层的对应关系。
具体地，为了使用绘图窗口绘图，通过系统函数egl Create Window Surface创建一个EGL Surface，该函数接收编码库生成的绘图层（surface）作为参数，这样就建立了绘图窗口和绘图层的对应关系。
进一步地，Android系统编码线程通过修改“current”来设置EGL Surface，为了使Open GL能够在创建的绘图窗口上绘图，还需要通过系统编码线程调用EGL Surface.make Current方法完成绘图层与绘图窗口之间的切换，并根据预设的绘图参数和绘图流程将绘图层的原始视频帧纹理绘制于绘图窗口。可以理解地，所述绘图参数可以为预设的顶点坐标、纹理坐标和顶点着色器、片段着色器等，根据Open GL的绘图流程，将所述原始视频帧纹理绘制在绘图窗口上。
需要说明的是，所述顶点坐标指的是归一化坐标，用于指定矩形的绘制区域。在Open GL里，需要渲染的一切物体都要映射到x轴和y轴上的[-1,1]的范围内，这个范围内的坐标被称为归一化设备坐标，其独立于屏幕实际的尺寸或形状。Android系统中Open GL占用整个显示屏，因此x轴、y轴的[-1,1]的范围分别映射为屏幕的像素宽和像素高。
可以理解地，图像在绘制之前，需要在Open GL的管道中传递，这就需要称为着色器的子例程，这些着色器会告诉GPU如何绘制数据，有顶点着色器和片段着色器，在绘制任何内容到屏幕之前，需要先定义顶点着色器和片段着色器。
具体地，首先读取原始视频帧的顶点数据，执行顶点着色器，进而可以控制每个顶点，实现渲染、确定顶点位置纹理坐标变换等指令；当顶点的全部属性都被确定后，将顶点组装成图元，然后光栅化图元，负责接收一个图元经过处理的电，并把它转换为片段，每个片段对应屏幕的一个像素；执行片段着色器，进行纹理查找，确定纹理坐标；而后将原始视频帧写入帧缓冲区，即绘图层（surface），最后显示在屏幕上。
步骤S30，将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
可以理解地，效果数据可以是美颜、滤镜、趣味贴图、动画效果等。所述效果数据可以是预存于系统中的数据，也可以是用户基于系统程序而产生的当前数据并缓存于系统内存中。
具体地，基于用户输入的控制指令获取系统内存中的效果数据，并与绘图窗口的原始视频帧纹理混合和编码，合成新的视频，并输出到网络推流模块，以供接收端解码播放。
本发明提供的技术方案中，通过获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层，再创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口，最后将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频，从而基于绘图层在视频发送端将效果数据合成至待发送视频中（即合成视频），合成视频为通用格式，使得不管接收端是何种类型设备，都能对接收的编码视频解码播放，也解决了现有技术中录制视频与增加的装饰效果数据单独发送而造成的不能同步播放的技术问题，提高了用户体验。
进一步地，请参照图2，基于上述实施例一，在本实施例中，上述步骤S10包括：
步骤S11，创建相机预览窗口以获取相机的原始视频帧，并为所述相机预览窗口设置渲染器；
步骤S12，获取所述渲染器基于各个原始视频帧创建的表面纹理和对应的纹理编号，并将获取的所述原始视频帧通过表面纹理和纹理编号转换成原始视频帧纹理；
步骤S13，将原始视频帧纹理缓存于系统编码库生成的绘图层。
具体地，基于Android系统的开放图形库（Open GL），通过Android系统的Camera类，设置相机参数，打开摄像头预览，并创建相机预览窗口（GL Surface View），并为所述相机预览窗口设置渲染器。所述相机预览窗口的创建，能获取相机的原始视频帧。
进一步地，为了获取纹理数据，首先在渲染器中使用开放图形库（Open GL）基于原始视频帧创建表面纹理（Surface Texure）和对应的纹理编号（Texureld），然后通过Camera的set Preview Texture接口传入纹理；这样，通过渲染器的on Draw Frame方法，将获取的所述原始视频帧通过表面纹理和纹理编号转换成原始视频帧纹理，并缓存于系统编码库生成的绘图层。
在本实施例提供的技术方案中，进一步提出了通过创建相机预览窗口以获取相机的原始视频帧，并为所述相机预览窗口设置渲染器；获取所述渲染器基于各个原始视频帧创建的表面纹理和对应的纹理编号，并将获取的所述原始视频帧通过表面纹理和纹理编号转换成原始视频帧纹理，并缓存于系统编码库生成的绘图层。借助Android系统中独立于窗口系统、独立于硬件的开放图形库来实现相机预览窗口的创建，在运行各种操作系统的各种计算机上都可以用，优化视频直播体验和效率。
进一步地，请参照图3，基于上述实施例二，在本实施例中，上述步骤S20包括：
步骤S21，创建绘图窗口，根据系统函数接收绘图层缓存的原始视频帧纹理，以建立绘图窗口与绘图层的对应关系；
步骤S22，根据系统编码线程中的窗口切换规则，完成绘图层与绘图窗口之间的切换，并根据预设的绘图参数和绘图流程，将绘图层的原始视频帧纹理绘制于绘图窗口中。
本发明提供的技术方案中，为了完成录制视频和效果数据的合成，需要创建一个绘图窗口，以完成原始视频帧的绘制和效果数据的绘制。
具体地，为了使用绘图窗口绘图，需要创建一个EGL Context和一个EGL Surface；EGL Context就是当前渲染代码所在的线程，EGL Surface是通过系统函数egl Create Window Surface创建的，该函数接收系统编码库生成的绘图层作为参数，以建立绘图窗口和绘图层的对应关系。
进一步地，当建立了绘图窗口与绘图层之间的对应关系，需要将缓存于绘图层的原始视频帧纹理绘制于绘图窗口。通过系统编码线程调用EGL Surface.make Current方法完成绘图层与绘图窗口之间的切换，基于开放图形库的绘图流程，将原始视频帧纹理绘制在绘图窗口上。
本实施例中，通过创建绘图窗口，以完成原始视频帧纹理的绘制，进而能在绘图窗口对获取的所述原始视频帧进行纹理绘制，并能将本地系统中的效果数据也绘制于绘图窗口，以实现录制视频与效果数据的合成；所述绘图窗口的创建，独立于相机预览窗口，进而能避免相机预览窗口卡顿等现象的发生，提高系统的处理效率。
进一步地，请参照图4，基于上述实施例三，在本实施例中，上述步骤S30包括：
步骤S31，根据接收的输入指令，从本地系统中获取对应的效果数据；
步骤S32，基于系统开放图形库将效果数据转换为对应的效果纹理，并将效果纹理绘制于绘图窗口中；
步骤S33，对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
可以理解地，为了美化视频效果，需要在录制的视频中添加效果数据，且需要将效果数据转换成与原始视频帧纹理相同的格式，以实现将效果数据与原始视频帧纹理的混合。需要说明的是，所述效果数据可以是预存于系统中的数据，例如贴图、动画等，也可以是用户基于系统程序而产生的当前数据并缓存于系统内存中，例如滤镜、美颜图片等。
具体地，根据接收的输入指令，首先读取系统磁盘中png或者jpg格式的图片，并生成bitmap位图数据到内存中，然后基于开放图形库的glTexImage2D函数生成二位图像的效果纹理，并使用相应的坐标和着色器将所述效果纹理绘制于绘图窗口上。绘制完成后，将绘图窗口上的原始视频帧纹理和效果纹理混合编码，生成新的合成视频并输出至网络推流模块，以供接收端解码播放。
本实施例提供的技术方案中，进一步细化了效果数据的处理步骤，将录制的原始视频帧与效果数据在视频录制端合成编码，实现了视频合成的实时传输，播放端只需解码播放即可，减少了播放端的处理流程，优化了直播效率，提高了用户体验。
进一步地，请参照图5，基于上述实施例四，在本实施例中，上述步骤S33包括：
步骤S331，将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；
步骤S332，基于系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
具体地，当完成将效果纹理绘制于绘图窗口的步骤后，将绘图窗口上的原始视频帧纹理和效果纹理混合成新的视频帧，并调用系统中的egl Swap Buffer函数来提交绘制在绘图窗口的当前新的视频帧，输出至与当前绘图窗口对应的绘图层中，系统编码库将以该绘图层作为输出对象，并将绘图层上的当前新的视频帧按照预设的格式编码，输出至网络推流模块，接收端只需解码播放即可。
需要说明的是，本实施例中，通过如下设置以完成将绘图窗口上的原始视频帧纹理和效果纹理的混合：
glEnbale(GL_BLEND)；//启用混合
glBlendFunc(GL_SRC_ALPHA，GL_ONE_MINUS_SRC_ALPHA)；//混合函数。
其中GL_SRC_ALPHA是源混合因子sfactor，GL_ONE_MINUS_SRC_ALPHA是目标混合因子dfactor。使用这个设置，实现了由远及近的绘图，在效果纹理的绘制区域，透明部分将显示原始帧，有像素的部分将显示效果纹理。
本实施例提供的技术方案中，通过将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；基于系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。进一步优化了效果纹理与原始视频帧纹理的处理步骤，以实现录制视频与效果数据的高效同步播放的效果。
本发明还提供一种视频合成装置，参照图6，在一实施例中，本发明提供的视频合成装置包括：
获取模块10，用以获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层；
绘制模块20，用以创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口；
编码模块30，用以将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
本发明视频合成装置的具体实施方式拓展内容，与视频合成方法对应部分内容基本相同，再次不作累述。
具体地，基于用户输入的控制指令获取系统内存中的效果数据，并与绘图窗口的原始视频帧纹理混合和编码，合成新的视频，并输出到网络推流模块，以供接收端解码播放。
本发明提供的技术方案中，通过获取模块10获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层，绘制模块20再创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口，最后编码模块30将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频，从而基于绘图层在视频发送端将效果数据合成至待发送视频中（即合成视频），合成视频为通用格式，使得不管接收端是何种类型设备，都能对接收的编码视频解码播放，也解决了现有技术中录制视频与增加的装饰效果数据单独发送而造成的不能同步播放的技术问题，提高了用户体验。
进一步地，请参照图7，基于上述实施例一，在本实施例中，所述获取模块10包括：
获取单元11，用以创建相机预览窗口以获取相机的原始视频帧，并为所述相机预览窗口设置渲染器；
转换单元12，用以获取所述渲染器基于各个原始视频帧创建的表面纹理和对应的纹理编号，并将获取的所述原始视频帧通过表面纹理和纹理编号转换成原始视频帧纹理；
缓存单元13，用以将原始视频帧纹理缓存于系统编码库生成的绘图层。
具体地，基于Android系统的开放图形库（Open GL），通过Android系统的Camera类，设置相机参数，打开摄像头预览，并创建相机预览窗口（GL Surface View），并为所述相机预览窗口设置渲染器。所述相机预览窗口的创建，能获取相机的原始视频帧。
进一步地，为了获取纹理数据，首先在渲染器中使用开放图形库（Open GL）基于原始视频帧创建表面纹理（Surface Texure）和对应的纹理编号（Texureld），然后通过Camera的set Preview Texture接口传入纹理；这样，通过渲染器的on Draw Frame方法，将获取的所述原始视频帧通过表面纹理和纹理编号转换成原始视频帧纹理，并缓存于系统编码库生成的绘图层。
在本实施例提供的技术方案中，获取单元11通过创建相机预览窗口以获取相机的原始视频帧，并为所述相机预览窗口设置渲染器；转换单元12将获取的所述原始视频帧通过渲染器创建的表面纹理和纹理编号转换成原始视频帧纹理，并通过缓存单元13缓存于系统编码库生成的绘图层。借助Android系统中独立于窗口系统、独立于硬件的开放图形库来实现相机预览窗口的创建，在运行各种操作系统的各种计算机上都可以用，优化视频直播体验和效率。
进一步地，请参照图8，基于上述实施例二，在本实施例中，所述绘制模块20包括：
绘图窗口创建单元21，用以创建绘图窗口，根据系统函数接收绘图层缓存的原始视频帧纹理，以建立绘图窗口与绘图层的对应关系；
绘制单元22，用以根据系统编码线程窗口切换规则，完成绘图层与绘图窗口之间的切换，并根据预设的绘图参数和绘图流程，将绘图层的原始视频帧纹理绘制于绘图窗口中。
本发明提供的技术方案中，为了完成录制视频和效果数据的合成，需要创建一个绘图窗口，以完成原始视频帧的绘制和效果数据的绘制。
具体地，为了使用绘图窗口绘图，需要创建一个EGL Context和一个EGL Surface；EGL Context就是当前渲染代码所在的线程，EGL Surface是通过系统函数egl Create Window Surface创建的，该函数接收系统编码库生成的绘图层作为参数，以建立绘图窗口和绘图层的对应关系。
进一步地，当建立了绘图窗口与绘图层之间的对应关系，需要将缓存于绘图层的原始视频帧纹理绘制于绘图窗口。通过系统编码线程调用EGL Surface.make Current方法完成绘图层与绘图窗口之间的切换，基于开放图形库的绘图流程，将原始视频帧纹理绘制在绘图窗口上。
本实施例中，通过创建绘图窗口，以完成原始视频帧纹理的绘制，进而能在绘图窗口对获取的所述原始视频帧进行纹理绘制，并能将本地系统中的效果数据也绘制于绘图窗口，以实现录制视频与效果数据的合成；所述绘图窗口的创建，独立于相机预览窗口，进而能避免相机预览窗口卡顿等现象的发生，提高系统的处理效率。
进一步地，请参照图9，基于上述实施例三，在本实施例中，所述编码模块30包括：
效果图片获取单元31，用以根据接收的输入指令，从本地系统中获取对应的效果数据；
效果纹理绘制单元32，用以基于系统开放图形库将效果数据转换为对应的效果纹理，并将效果纹理绘制于绘图窗口中；
编码单元33，用以对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
可以理解地，为了美化视频效果，需要在录制的视频中添加效果数据，且需要将效果数据转换成与原始视频帧纹理相同的格式，以实现将效果数据与原始视频帧纹理的混合。需要说明的是，所述效果数据可以是预存于系统中的数据，例如贴图、动画等，也可以是用户基于系统程序而产生的当前数据并缓存于系统内存中，例如滤镜、美颜图片等。
具体地，所述效果图片获取单元31首先获取系统磁盘png或者jpg格式的图片，并生成bitmap位图数据到内存中，然后基于开放图形库的glTexImage2D函数生成二位图像的纹理数据，并使用相应的坐标和着色器将所述效果纹理绘制于绘图窗口上。绘制完成后，将绘图窗口上的原始视频帧纹理和效果纹理混合编码，生成新的合成视频并输出至网络推流模块，以供接收端解码播放。
本实施例提供的技术方案中，进一步细化了效果数据的处理步骤，将录制的原始视频帧与效果数据在视频录制端合成编码，实现了视频合成的实时传输，播放端只需解码播放即可，减少了播放端的处理流程，优化了直播效率，提高了用户体验。
进一步地，请参照图10，基于上述实施例四，在本实施例中，所述编码单元33包括：
混合单元331，用以将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；
输出单元332，用以基于系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
具体地，当完成将效果纹理绘制于绘图窗口的步骤后，将绘图窗口上的原始视频帧纹理和效果纹理混合成新的视频帧，并调用系统中的egl Swap Buffer函数来提交绘制在绘图窗口的当前新的视频帧，输出至与当前绘图窗口对应的绘图层中，系统编码库将以该绘图层作为输出对象，并将绘图层上的当前新的视频帧按照预设的格式编码，输出至网络推流模块，接收端只需解码播放即可。
需要说明的是，本实施例中，通过如下设置以完成将绘图窗口上的原始视频帧纹理和效果纹理的混合：
glEnbale(GL_BLEND)；//启用混合
glBlendFunc(GL_SRC_ALPHA，GL_ONE_MINUS_SRC_ALPHA)；//混合函数。
其中GL_SRC_ALPHA是源混合因子sfactor，GL_ONE_MINUS_SRC_ALPHA是目标混合因子dfactor。使用这个设置，实现了由远及近的绘图，在效果纹理的绘制区域，透明部分将显示原始帧，有像素的部分将显示效果纹理。
本实施例提供的技术方案中，通过混合单元331将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；输出单元332通过系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。进一步优化了效果纹理与原始视频帧纹理的处理步骤，以实现录制视频与装饰效果的高效同步的效果。
以上仅为本发明的优选实施例，并非因此限制本发明的专利范围，凡是利用本发明说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本发明的专利保护范围内。

Claims (20)

  1. 一种视频合成方法，其特征在于，所述视频合成方法包括以下步骤：
    获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层；
    创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口；
    将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  2. 如权利要求1所述的视频合成方法，其特征在于，所述创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口的步骤包括：
    创建绘图窗口，根据系统函数接收绘图层缓存的原始视频帧纹理，以建立绘图窗口与绘图层的对应关系；
    根据系统编码线程窗口切换规则，完成绘图层与绘图窗口之间的切换，并根据预设的绘图参数和绘图流程，将绘图层的原始视频帧纹理绘制于绘图窗口中。
  3. 如权利要1所述的视频合成方法，其特征在于，所述将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频的步骤包括：
    根据接收的输入指令，从本地系统中获取对应的效果数据；
    基于系统开放图形库将效果数据转换为对应的效果纹理，并将效果纹理绘制于绘图窗口中；
    对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  4. 如权利要求3所述的视频合成方法，其特征在于，所述对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频的步骤包括：
    将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；
    基于系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  5. 如权利要求1所述的视频合成方法，其特征在于，所述获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层的步骤包括：
    创建相机预览窗口以获取相机的原始视频帧，并为所述相机预览窗口设置渲染器；
    获取所述渲染器基于各个原始视频帧创建的表面纹理和对应的纹理编号，并将获取的所述原始视频帧通过表面纹理和纹理编号转换成原始视频帧纹理；
    将原始视频帧纹理缓存于系统编码库生成的绘图层。
  6. 如权利要5所述的视频合成方法，其特征在于，所述将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频的步骤包括：
    根据接收的输入指令，从本地系统中获取对应的效果数据；
    基于系统开放图形库将效果数据转换为对应的效果纹理，并将效果纹理绘制于绘图窗口中；
    对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  7. 如权利要求6所述的视频合成方法，其特征在于，所述对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频的步骤包括：
    将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；
    基于系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  8. 如权利要求5所述的视频合成方法，其特征在于，所述创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口的步骤包括：
    创建绘图窗口，根据系统函数接收绘图层缓存的原始视频帧纹理，以建立绘图窗口与绘图层的对应关系；
    根据系统编码线程窗口切换规则，完成绘图层与绘图窗口之间的切换，并根据预设的绘图参数和绘图流程，将绘图层的原始视频帧纹理绘制于绘图窗口中。
  9. 如权利要8所述的视频合成方法，其特征在于，所述将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频的步骤包括：
    根据接收的输入指令，从本地系统中获取对应的效果数据；
    基于系统开放图形库将效果数据转换为对应的效果纹理，并将效果纹理绘制于绘图窗口中；
    对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  10. 如权利要求9所述的视频合成方法，其特征在于，所述对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频的步骤包括：
    将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；
    基于系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  11. 一种视频合成装置，其特征在于，包括：
    获取模块，用以获取相机的原始视频帧，将原始视频帧转换成原始视频帧纹理并缓存于系统编码库生成的绘图层；
    绘制模块，用以创建绘图窗口，并建立绘图窗口与所述绘图层的对应关系，以将所述绘图层缓存的原始视频帧纹理绘制于所述绘图窗口；
    编码模块，用以将绘图窗口的原始视频帧纹理与本地系统的效果数据混合编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  12. 如权利要求11所述的视频合成装置，其特征在于，所述绘制模块包括：
    绘图窗口创建单元，用以创建绘图窗口，根据系统函数接收绘图层缓存的原始视频帧纹理，以建立绘图窗口与绘图层的对应关系；
    绘制单元，用以根据系统编码线程窗口切换规则，完成绘图层与绘图窗口之间的切换，并根据预设的绘图参数和绘图流程，将绘图层的原始视频帧纹理绘制于绘图窗口中。
  13. 如权利要求11所述的视频合成装置，其特征在于，所述编码模块包括：
    效果图片获取单元，用以根据接收的输入指令，从本地系统中获取对应的效果数据；
    效果纹理绘制单元，用以基于系统开放图形库将效果数据转换为对应的效果纹理，并将效果纹理绘制于绘图窗口中；
    编码单元，用以对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  14. 如权利要求13所述的视频合成装置，其特征在于，所述编码单元包括：
    混合单元，用以将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；
    输出单元，用以基于系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  15. 如权利要求11所述的视频合成装置，其特征在于，所述获取模块包括：
    获取单元，用以创建相机预览窗口以获取相机的原始视频帧，并为所述相机预览窗口设置渲染器；
    转换单元，用以获取所述渲染器基于各个原始视频帧创建的表面纹理和对应的纹理编号，并将获取的所述原始视频帧通过表面纹理和纹理编号转换成原始视频帧纹理；
    缓存单元，用以将原始视频帧纹理缓存于系统编码库生成的绘图层。
  16. 如权利要求15所述的视频合成装置，其特征在于，所述编码模块包括：
    效果图片获取单元，用以根据接收的输入指令，从本地系统中获取对应的效果数据；
    效果纹理绘制单元，用以基于系统开放图形库将效果数据转换为对应的效果纹理，并将效果纹理绘制于绘图窗口中；
    编码单元，用以对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  17. 如权利要求16所述的视频合成装置，其特征在于，所述编码单元包括：
    混合单元，用以将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；
    输出单元，用以基于系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  18. 如权利要求15所述的视频合成装置，其特征在于，所述绘制模块包括：
    绘图窗口创建单元，用以创建绘图窗口，根据系统函数接收绘图层缓存的原始视频帧纹理，以建立绘图窗口与绘图层的对应关系；
    绘制单元，用以根据系统编码线程窗口切换规则，完成绘图层与绘图窗口之间的切换，并根据预设的绘图参数和绘图流程，将绘图层的原始视频帧纹理绘制于绘图窗口中。
  19. 如权利要求18所述的视频合成装置，其特征在于，所述编码模块包括：
    效果图片获取单元，用以根据接收的输入指令，从本地系统中获取对应的效果数据；
    效果纹理绘制单元，用以基于系统开放图形库将效果数据转换为对应的效果纹理，并将效果纹理绘制于绘图窗口中；
    编码单元，用以对绘图窗口中的原始视频帧纹理和效果纹理进行混合和编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
  20. 如权利要求19所述的视频合成装置，其特征在于，所述编码单元包括：
    混合单元，用以将绘图窗口中的原始视频帧纹理和效果纹理混合成新的视频帧，并输出至与绘图窗口对应的绘图层；
    输出单元，用以基于系统编码库将绘图层上当前的新的视频帧按照预设格式编码，生成并输出合成视频，以供接收端解码播放所述合成视频。
PCT/CN2016/112917 2016-11-28 2016-12-29 视频合成方法及装置 Ceased WO2018094814A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201611085102.3 2016-11-28
CN201611085102.3A CN106534880A (zh) 2016-11-28 2016-11-28 视频合成方法及装置

Publications (1)

Publication Number Publication Date
WO2018094814A1 true WO2018094814A1 (zh) 2018-05-31

Family

ID=58354374

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/112917 Ceased WO2018094814A1 (zh) 2016-11-28 2016-12-29 视频合成方法及装置

Country Status (2)

Country Link
CN (1) CN106534880A (zh)
WO (1) WO2018094814A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110782387A (zh) * 2018-07-30 2020-02-11 优视科技有限公司 图像处理方法、装置、图像处理器及电子设备
CN111754607A (zh) * 2019-03-27 2020-10-09 北京小米移动软件有限公司 图片处理方法及装置、电子设备、计算机可读存储介质
CN112383793A (zh) * 2020-11-12 2021-02-19 咪咕视讯科技有限公司 画面的合成方法、装置、电子设备及存储介质
CN115563323A (zh) * 2022-09-30 2023-01-03 际络科技(上海)有限公司 目标检测结果的展示方法及装置
WO2024125328A1 (zh) * 2022-09-06 2024-06-20 北京字跳网络技术有限公司 直播图像帧处理方法、装置、设备、可读存储介质及产品

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108449641B (zh) * 2018-03-28 2021-04-27 聚好看科技股份有限公司 播放媒体流的方法、装置、计算机设备和存储介质
CN109922360B (zh) * 2019-03-07 2022-02-11 腾讯科技（深圳）有限公司 视频处理方法、装置及存储介质
CN110278479B (zh) * 2019-06-04 2021-10-29 海信视像科技股份有限公司 一种图片显示方法及电视机
CN110850983B (zh) * 2019-11-13 2020-11-24 腾讯科技（深圳）有限公司 视频直播中的虚拟对象控制方法、装置和存储介质
CN111614906B (zh) 2020-05-29 2022-02-22 阿波罗智联(北京)科技有限公司 一种图像预处理方法、装置、电子设备及存储介质
CN112118397B (zh) * 2020-09-23 2021-06-22 腾讯科技（深圳）有限公司 一种视频合成的方法、相关装置、设备以及存储介质
CN114390354B (zh) * 2020-10-21 2024-05-10 西安诺瓦星云科技股份有限公司 节目制作方法、装置和系统及计算机可读存储介质
CN112486606B (zh) * 2020-11-19 2022-08-12 湖南麒麟信安科技股份有限公司 一种基于Android系统的云桌面显示优化方法及系统
CN112581575B (zh) * 2020-12-05 2024-05-03 西安翔腾微电子科技有限公司 一种外视频做纹理系统
CN114900736A (zh) * 2022-03-28 2022-08-12 网易（杭州）网络有限公司 视频生成方法、装置和电子设备
CN115334265A (zh) * 2022-06-30 2022-11-11 武汉高德智感科技有限公司 一种红外可见光画中画视频录制的方法与电子设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130293537A1 (en) * 2011-01-05 2013-11-07 Cisco Technology Inc. Coordinated 2-Dimensional and 3-Dimensional Graphics Processing
CN103677828A (zh) * 2013-12-10 2014-03-26 华为技术有限公司 一种图层绘制方法、绘图引擎及终端设备
CN105204881A (zh) * 2014-06-25 2015-12-30 中兴通讯股份有限公司 在Android系统上运行J2me应用程序的方法及设备
CN105933724A (zh) * 2016-05-23 2016-09-07 福建星网视易信息系统有限公司 视频制作方法、装置及系统

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102447873A (zh) * 2010-10-13 2012-05-09 张明 哈哈视频网络视频聊天娱乐辅助系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130293537A1 (en) * 2011-01-05 2013-11-07 Cisco Technology Inc. Coordinated 2-Dimensional and 3-Dimensional Graphics Processing
CN103677828A (zh) * 2013-12-10 2014-03-26 华为技术有限公司 一种图层绘制方法、绘图引擎及终端设备
CN105204881A (zh) * 2014-06-25 2015-12-30 中兴通讯股份有限公司 在Android系统上运行J2me应用程序的方法及设备
CN105933724A (zh) * 2016-05-23 2016-09-07 福建星网视易信息系统有限公司 视频制作方法、装置及系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
QIAO, SHAOJIE ET AL: "High-speed Image Rendering Method Based on OpenGL", APPLICATION RESEARCH OF COMPUTERS, vol. 25, no. 5, 31 May 2008 (2008-05-31), pages 1590 - 1591 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110782387A (zh) * 2018-07-30 2020-02-11 优视科技有限公司 图像处理方法、装置、图像处理器及电子设备
CN110782387B (zh) * 2018-07-30 2023-09-22 阿里巴巴（中国）有限公司 图像处理方法、装置、图像处理器及电子设备
CN111754607A (zh) * 2019-03-27 2020-10-09 北京小米移动软件有限公司 图片处理方法及装置、电子设备、计算机可读存储介质
CN112383793A (zh) * 2020-11-12 2021-02-19 咪咕视讯科技有限公司 画面的合成方法、装置、电子设备及存储介质
CN112383793B (zh) * 2020-11-12 2023-07-07 咪咕视讯科技有限公司 画面的合成方法、装置、电子设备及存储介质
WO2024125328A1 (zh) * 2022-09-06 2024-06-20 北京字跳网络技术有限公司 直播图像帧处理方法、装置、设备、可读存储介质及产品
CN115563323A (zh) * 2022-09-30 2023-01-03 际络科技(上海)有限公司 目标检测结果的展示方法及装置

Also Published As

Publication number Publication date
CN106534880A (zh) 2017-03-22

Similar Documents

Publication Publication Date Title
JP5182957B2 (ja) アスペクト比変換のための画像データの可変スケーリング
WO2018034462A1 (en) Image display apparatus and method of operating the same
CN106534880A (zh) 视频合成方法及装置
WO2018018681A1 (zh) 视频节目预览方法及装置
WO2015009107A1 (en) Method and apparatus for generating 3k-resolution display image for mobile terminal screen
WO2017088318A1 (zh) 用户界面显示处理方法及装置
WO2016091011A1 (zh) 字幕切换方法及装置
WO2018192253A1 (zh) 视频信号的传输方法及装置
KR102483680B1 (ko) 주변 이미지를 프로세싱하기 위한 방법 및 디바이스
WO2019010926A1 (zh) 广告的推送方法、装置及计算机可读存储介质
WO2017020649A1 (zh) 音视频播放控制方法及装置
WO2017005062A1 (zh) 图片帧播放方法和装置
WO2017206368A1 (zh) 高动态范围画面切换方法及装置
WO2017190451A1 (zh) 图片推送方法和装置
WO2017107384A1 (zh) 液晶显示器的图像显示方法及液晶显示器
WO2015169209A1 (en) Animation data generating method, apparatus, and electronic device
WO2017113600A1 (zh) 视频播放方法及装置
WO2014048239A1 (zh) 智能电视播放方法及智能电视
WO2017201893A1 (zh) 视频处理方法和装置
WO2019221423A1 (ko) 전자장치, 그 제어방법 및 기록매체
WO2018190446A1 (ko) 영상 처리 방법 및 장치
WO2017107385A1 (zh) 视频播放控制方法及装置
WO2017161697A1 (zh) 多终端节目同步预览方法和装置
WO2018076560A1 (zh) 画面显示方法和装置
WO2024048960A1 (ko) 사용자 얼굴에 기초한 아바타를 제공하는 전자 장치 및 그 동작 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16922047

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 15/10/2019)

122 Ep: pct application non-entry in european phase

Ref document number: 16922047

Country of ref document: EP

Kind code of ref document: A1