CN112218117B

CN112218117B - Video processing method and device

Info

Publication number: CN112218117B
Application number: CN202011047854.7A
Authority: CN
Inventors: 王启明; 包红来
Original assignee: Beijing Zitiao Network Technology Co Ltd
Current assignee: Beijing Zitiao Network Technology Co Ltd
Priority date: 2020-09-29
Filing date: 2020-09-29
Publication date: 2023-01-03
Anticipated expiration: 2040-09-29
Also published as: CN112218117A

Abstract

The embodiment of the disclosure provides a video processing method and a device, wherein the method comprises the following steps: the method comprises the steps of obtaining a first target video frame and a special effect resource corresponding to the first target video frame, executing loading processing on the target video frame and the corresponding special effect resource through a first thread, executing rendering processing on the first target video frame according to the loaded special effect resource through a second thread to obtain a first rendering video frame, and encoding the first rendering video frame through a third thread to obtain an encoding result, wherein the first thread, the second thread and the third thread are processed in parallel. According to the embodiment, the special effect rendering processing is divided into the first thread and the second thread, the two threads are completed, time consumption of the special effect rendering processing can be greatly reduced, and the video synthesis rendering efficiency is improved.

Description

Video processing method and device

Technical Field

The embodiment of the disclosure relates to the technical field of image processing, in particular to a video processing method and device.

Background

With the development of communication technology, more and more video processing platforms emerge, so that users can record drip life by means of pictures and videos. Meanwhile, the video processing platform provides various special effect functions, can synthesize and render the shot original video, is beneficial to releasing more exquisite works for users, and improves the use experience of the users.

In the prior art, for example, an IOS platform is used, during a video synthesis rendering process, an original video frame is usually required to be subjected to special effect rendering processing through a main thread, and encoding processing is performed through an encoding thread.

However, the time consumption of the special effect rendering processing on the main thread is longer than that of the encoding processing of the encoding thread, and the encoding processing is asynchronous output, so that the efficiency of the whole video synthesis rendering is influenced by the special effect rendering link on the main thread. Therefore, how to reduce the time consumption of the special effect rendering link is an urgent problem to be solved for improving the video synthesis rendering efficiency.

Disclosure of Invention

The embodiment of the disclosure provides a video processing method and video processing equipment, which are used for reducing the time consumption of a special effect rendering link and improving the video synthesis rendering efficiency.

In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

acquiring a first target video frame and a special effect resource corresponding to the first target video frame;

executing loading processing on the first target video frame and the corresponding special effect resource through a first thread;

performing rendering processing on the first target video frame through a second thread according to the loaded special effect resource to obtain a first rendered video frame;

coding the first rendered video frame through a third thread to obtain a coding result;

the first thread, the second thread, and the third thread are processed in parallel.

In a second aspect, an embodiment of the present disclosure provides a video processing apparatus, including:

the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a first target video frame and a special effect resource corresponding to the first target video frame;

the first processing module is used for executing the loading processing of the first target video frame and the corresponding special effect resource through a first thread;

the second processing module is used for executing rendering processing on the first target video frame through a second thread according to the loaded special effect resource to obtain a first rendered video frame;

the third processing module is used for coding the first rendered video frame through a third thread to obtain a coding result;

In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

the memory stores computer-executable instructions;

the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the video processing method as set forth in the first aspect above and in various possible designs of the first aspect.

In a fourth aspect, the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored, and when a processor executes the computer-executable instructions, the video processing method according to the first aspect and various possible designs of the first aspect are implemented.

The method includes the steps of firstly obtaining a first target video frame and a special effect resource corresponding to the first target video frame, executing loading processing on the target video frame and the corresponding special effect resource through a first thread, executing rendering processing on the first target video frame according to the loaded special effect resource through a second thread to obtain a first rendered video frame, and encoding the first rendered video frame through a third thread to obtain an encoding result, wherein the first thread, the second thread and the third thread are processed in parallel. According to the embodiment, the special effect rendering processing is divided into the first thread and the second thread, the two threads are completed, time consumption of the special effect rendering processing can be greatly reduced, and the video synthesis rendering efficiency is improved.

Drawings

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present disclosure, and for those skilled in the art, other drawings can be obtained according to the drawings without inventive exercise.

FIG. 1 is a schematic diagram of video processing in the prior art;

fig. 2 is a schematic flowchart of a video processing method according to an embodiment of the disclosure;

fig. 3 is a schematic diagram illustrating an application of video composition rendering according to yet another embodiment of the present disclosure;

fig. 4 is a schematic flowchart of a video processing method according to another embodiment of the present disclosure;

fig. 5 is a schematic diagram of a video processing method according to another embodiment of the disclosure;

fig. 6 is a schematic diagram of a video processing method according to still another embodiment of the disclosure;

fig. 7 is a schematic flowchart of a video processing method according to another embodiment of the disclosure;

fig. 8 is a schematic diagram of a video processing method according to still another embodiment of the disclosure;

fig. 9 is a block diagram of a video processing device according to an embodiment of the present disclosure;

fig. 10 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present disclosure.

Detailed Description

To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.

Fig. 1 is a schematic diagram of video processing in the prior art. As shown in fig. 1, in a conventional video composition rendering process, a main thread (main gl context) and an encoding thread (encoder thread) are generally used to perform special effect rendering and encoding processes, respectively. The main thread is used for loading a special effect resource of a target video frame and calling a rendering instruction, namely adding the target video frame into a rendering queue (effect drawcall), waiting for rendering completion (render), and adding completed rendering data into an encoding queue (encode). The encoding thread is used to perform the task of encoding callbacks (encodings).

In a specific implementation process, the main thread sequentially processes each target video frame, and sequentially executes effect draw, render and encode tasks for each target video frame. After the main thread completes the execution of the three tasks on a certain target video frame, the encoding thread continues to execute the encode callback task on the target video frame.

Therefore, coding processing in the coding thread is asynchronous output, and the time-consuming bottleneck of the whole synthesis rendering processing is the special effect rendering link (effect draw, render and encode tasks) executed by the main thread. Compared with the method that the task is executed only by the main thread in a serial mode, the two threads (the first thread and the second thread) are adopted for parallel processing of partial tasks, time consumption can be greatly saved, and video processing efficiency is improved.

The technical solution of the present disclosure is explained in detail below with specific examples. These several specific embodiments may be combined with each other below, and details of the same or similar concepts or processes may not be repeated in some embodiments.

Referring to fig. 2, fig. 2 is a schematic flowchart of a video processing method according to an embodiment of the disclosure. The method of the embodiment can be applied to electronic equipment such as a mobile phone terminal and a tablet personal computer, and the video processing method comprises the following steps:

201. the method comprises the steps of obtaining a first target video frame and a special effect resource corresponding to the first target video frame.

In this embodiment, the target video may include a plurality of target video frames.

In practical applications, for example, a mobile phone terminal is taken as an example, after video shooting is performed through a camera of the mobile phone terminal, a target video including a plurality of target video frames is obtained, and the mobile phone terminal can receive a special effect selection instruction (for example, a screen touch operation, a voice instruction, and the like) of a user to perform special effect selection so as to select a special effect resource for performing special effect rendering on each target video frame of the target video.

202. And executing loading processing on the first target video frame and the corresponding special effect resource through a first thread.

203. And executing rendering processing on the first target video frame through a second thread according to the loaded special effect resource to obtain a first rendered video frame.

204. And encoding the first rendering video frame through a third thread to obtain an encoding result. Wherein the first thread, the second thread, and the third thread are processed in parallel.

In this embodiment, a first thread and a second thread are adopted to perform parallelization splitting on a task that is executed serially in a main thread, and the two threads execute the task in parallel. Thus saving time. It should be noted that, in order to bear the original main thread task, the number of threads used may be actually adjusted according to the number of tasks of video processing to meet the actual requirement, which is not limited in this embodiment.

Optionally, the first thread may be a main thread, the second thread may be a shared thread, and the third thread may be a coding thread, where the thread names are only examples, and different names may be given according to actual needs, and this embodiment does not limit this.

Specifically, the first thread, the second thread and the third thread are serially processed for a single target video frame, for example, for the first target video frame, first, processing such as loading of special effect resources is performed through the first thread, then, after the processing is completed, rendering processing is performed on the first target video frame through the second thread according to the loaded special effect resources, and finally, coding processing is performed on the first rendered video frame obtained by the second thread through the third thread, so that rendering and synthesizing processing on the first target video frame is completed. However, the first thread, the second thread, and the third thread are processed in parallel for the plurality of target video frames. For example, after the first thread finishes the loading processing of the first target video frame, the first thread may continue to load the second target video frame without waiting until the second thread and the third thread finish the processing of the first target video frame. Similarly, the second thread may perform rendering processing on the second target video frame after performing rendering processing on the first target video frame, that is, while the third thread performs encoding processing on the first target video frame, the second thread starts rendering processing on the third target video frame.

In practical application, as shown in fig. 3, a user opens a special effect shooting APP installed on an electronic device, for example, a mobile phone terminal, or a short video platform, and can select a special effect resource from special effect options such as A, B, C, D through screen touch or voice control, and perform the video processing method of this embodiment on a video to be processed. And obtaining the video after special effect rendering, and then performing operations such as work release and the like.

As can be seen from the above description, in this embodiment, a first target video frame and a special effect resource corresponding to the first target video frame are first obtained, a first thread is used to perform loading processing on the target video frame and the corresponding special effect resource, a second thread is used to perform rendering processing on the first target video frame according to the loaded special effect resource so as to obtain a first rendered video frame, and a third thread is used to perform encoding processing on the first rendered video frame so as to obtain an encoding result, where the first thread, the second thread, and the third thread are processed in parallel. According to the embodiment, the special effect rendering processing is split into the first thread and the second thread, the two threads are completed, time consumption of the special effect rendering processing can be greatly reduced, and the video synthesis rendering efficiency is improved.

In some embodiments, said performing, by the first thread, load processing of the first target video frame comprises: and calling a corresponding load cache through a first thread to obtain a first target video frame, and adding the first target video frame into a rendering queue so as to receive the first target video frame in the rendering queue through a second thread. By sequentially adding each target video frame into the rendering queue, each video frame can be orderly processed.

In some embodiments, the performing, by the second thread, the rendering processing of the first target video frame according to the loaded special effect resource includes: calling a rendering cache through a second thread to receive the first target video frame, and controlling a GPU to perform rendering processing on the first target video frame; the encoding, by the third thread, the first rendered video frame to obtain an encoding result includes: and calling an encoding cache through a third thread to receive a first rendered video frame in the encoding queue, controlling an encoder to encode the first rendered video frame, and calling back an encoding result. By adding the rendered video frames obtained by rendering processing into the coding queue, the ordered coding processing of each rendered video frame can be realized.

Referring to fig. 4, fig. 4 is a flowchart illustrating a method for video processing according to another embodiment of the disclosure. On the basis of the embodiment of fig. 2, this embodiment illustrates the cache use condition in steps 202 to 204, where steps 402 to 404 are specific implementations of steps 202 to 204, and in this embodiment, the method includes:

401. the method comprises the steps of obtaining a first target video frame and a special effect resource corresponding to the first target video frame.

402. And calling the corresponding load cache through the first thread to obtain a first target video frame, and adding the first target video frame into a rendering queue.

403. And calling the rendering cache to receive the first target video frame in the rendering queue through the second thread, controlling the GPU to render the first target video frame, and adding the first rendering video frame into the encoding queue.

404. And calling the coding cache to receive the first rendered video frame in the coding queue through the third thread, controlling the coder to code the first rendered video frame, and calling back the coding result. The main thread, the sharing thread and the encoding thread perform parallel processing on the target video frame.

The following describes an example of the process of rendering and composition in this embodiment, taking a first thread as a main thread, a second thread as a shared thread, and a third thread as an encoding thread as an example:

in practical application, in order to reduce the time consumption of a special effect rendering link on a main thread, tasks (added into a rendering queue, namely, invoking a rendering instruction effect draw call, monitoring a rendering completion render, and completing rendering and adding into a coding queue encode) executed serially by the main thread may be split, as shown in fig. 5, a shared thread is added, the effect draw call task is executed through the original main thread, and the render and encode tasks are executed through the newly added shared thread, because the main thread and the shared thread can be processed in parallel, compared with the scheme of fig. 1 in which three tasks (effect draw call, render, and encode) are executed serially by the main thread, the time consumption can be reduced, and the efficiency can be improved. In a specific implementation process, after the main thread executes the effect drawcall task on the first target video frame, the shared thread can execute the following render and encode tasks on the first target video frame. Meanwhile, the main thread can continue to execute the effect drawcall task on the second target video frame without waiting until the first target video frame finishes executing the render and encode tasks. Therefore, the embodiment parallelizes and disassembles the tasks executed by the original main thread in series by adding the sharing thread, so that time consumption can be greatly saved, and the video synthesis rendering efficiency can be improved.

In an embodiment of the present disclosure, on the basis of the embodiment of fig. 2, in order to manage the call of the load cache to avoid a problem that a load task executed on a main thread is insufficient in cache due to too much call cache, in this embodiment, the method includes:

setting a load cache pool (render resource), wherein the load cache pool comprises: the number of the loading caches is less than that of the target video frames;

in practical applications, a special effect loading task and a rendering command calling task (effect drawcall, which add a target video frame to a rendering queue) executed by a main thread consume a short time compared with a task (render) of monitoring rendering completion and a task (encode) of adding rendering data to an encoding queue in a shared thread. For example, after calling one cache block to execute effect draw call on a first target video frame, the main thread calls another cache block to execute effect draw call on a second target video frame, and after completion, calls the third cache block to execute effect draw call on a third target video frame. As shown in fig. 5, the processing task of 12 target video frames illustrated by the main thread is likely to call 12 cache blocks for respective tasks. Therefore, the time consumption of the task executed by the main thread is too fast, the number of applied caches may be too large, and other programs call the caches.

Based on this, as shown in fig. 6, a load cache pool may be set in this embodiment, where the load cache pool includes at least two load caches. In the following, taking an example that the load cache pool includes a first load cache and a second load cache, the video processing method is illustrated in fig. 6, where a tile filled with diagonal left lines represents the first load cache, and a tile filled with dots represents the second load cache. In the specific implementation process, the first loading cache is called by the main thread to load the first target video frame, and after the loading is completed, a rendering instruction is called to control the GPU to acquire the first target video frame from the first loading cache, render the first target video frame according to the special effect resource corresponding to the first target video frame, and output the rendered data. After rendering is completed, the first load cache is released to be called by other subsequent target video frames (e.g., a third target video frame). After the first target video frame is loaded, the main thread may call the second load cache to load the second target video frame, and after the loading is completed, call a rendering instruction to control the GPU to obtain the second target video frame in the second load cache, perform special effect rendering on the second target video frame according to a special effect resource corresponding to the second target video frame, and after the rendering is completed, release the second load cache to be called by a subsequent target video frame (e.g., a fourth target video frame). And so on. The main thread calls the cache by setting the loading cache pool, so that the condition that the main thread applies for the cache resources in a wireless manner at the stage and the residual amount of the cache is influenced can be limited.

Moreover, it should be noted that, since the render and encode tasks processed by the shared thread or the encode callback (encode callback) processed by the encode thread are time-consuming, especially compared with the effect draw task executed by the main thread, it can be seen that the bottleneck of the time consumption of the whole video synthesis rendering is the tasks executed by the shared thread and the encode thread. That is, in the present embodiment, although the main thread call cache is limited, it does not affect the efficiency of the overall composition rendering. But instead optimizes the condition of cache calls. The problem of buffer bursting caused by infinite application of buffer when the main thread executes the effect drawcall task is avoided.

In an embodiment of the present disclosure, based on the above-mentioned fig. 2 embodiment, in order to manage the invocation of the encoding buffer, it is avoided that the same buffer is used for both the encoding process of one video frame and the rendering process of another video frame, which results in overlap and pollution between the encoded data and the rendering data. In this embodiment, the method further includes:

and rendering cache and coding cache corresponding to the rendering processing and the coding processing performed on the same target video frame are the same cache.

Setting an encoding buffer pool (encoder resource), which may include: a plurality of code buffers. In a specific embodiment, the plurality of code buffers may include two code buffers arranged in tandem.

In practical application, if a first target video frame stored in a first encoding buffer (encoding callback 1) is encoded by an encoder, and a subsequent frame of the first target video frame, that is, a second target video frame is added to a rendering queue and is input to the same buffer, that is, the first encoding buffer, during the encoding process, the encoded data of the previous frame and the first target video frame will be polluted, which will affect the encoded output data. Therefore, in order to ensure that the cached input data can only be the data of the current target video frame before the current target video frame is encoded by the encoder, namely before the encoding call-back, the input of a new target video frame cannot be allowed, and the current target video frame is prevented from being modified by the input of the new target video frame. Based on this, the code cache can be managed. In the embodiment, by setting the encoding buffer pool, different encoding buffers are allocated to different target video frames for storing rendering encoded data, so that pollution between the rendering encoded data of adjacent video frames can be avoided.

Referring to fig. 7, fig. 7 is a schematic flowchart of a video processing method according to another embodiment of the disclosure. On the basis of the above-mentioned embodiment of fig. 6, in order to manage the invocation of the encoding buffer, the same buffer is prevented from being used for both the encoding process of one video frame and the rendering process of another video frame, which results in overlap and pollution between the encoded data and the rendering data. In this embodiment, the method includes:

701. the method comprises the steps of obtaining a first target video frame and a special effect resource corresponding to the first target video frame.

702. And calling a first loading cache in the multiple loading caches through a first thread to obtain a first target video frame, and adding the first target video frame into a rendering queue.

703. And calling a second load cache in the plurality of load caches through the first thread to obtain a second target video frame, and adding the second target video frame into a rendering queue.

704. And after a first rendering video frame corresponding to the first target video frame is obtained through the second thread, releasing the first loading cache, calling the first loading cache through the first thread after the plurality of loading caches are all called, obtaining an Nth target video frame after the first target video frame, and adding the Nth target video frame after the first target video frame into a rendering queue. The first target video frame is a video frame before the second target video frame, N is the number of load caches, and the first load cache is a load cache before the second load cache.

705. And calling a first encoding cache in the plurality of encoding caches through a second thread to receive the first target video frame, and controlling the GPU to render the first target video frame.

706. And calling a second coding cache in the plurality of coding caches through a second thread to receive a second target video frame, and controlling the GPU to perform rendering processing on the second target video frame to obtain a second rendered video frame.

707. And calling the first coding cache to receive the first rendered video frame in the coding queue through a third thread, controlling an encoder to encode the first rendered video frame, calling back a coding result, and releasing the first coding cache, so that the Mth target video frame after the first target video frame is received by calling the first coding cache through the second thread after the plurality of coding caches are all called. The first target video frame is a video frame before the second target video frame, M is the number of coding buffers, and the first coding buffer is a coding buffer before the second coding buffer.

Optionally, M is an integer greater than or equal to 2.

In the following, taking the first thread as a main thread, the second thread as a shared thread, and the third thread as an encoding thread as an example, a process of rendering and compositing in this embodiment is described as an example:

in practical application, if a first target video frame stored in a first encoding buffer (encoding callback 1) is encoded by an encoder, and a subsequent frame of the first target video frame, that is, a second target video frame is added to a rendering queue and is input to the same buffer, that is, the first encoding buffer, during the encoding process, the encoded data of the previous frame and the first target video frame will be polluted, which will affect the encoded output data. Therefore, in order to ensure that the cached input data can only be the data of the current target video frame before the current target video frame is encoded by the encoder, namely before the encoding call-back, the input of a new target video frame cannot be allowed, and the current target video frame is prevented from being modified by the input of the new target video frame. Based on this, the code cache can be managed. As shown in fig. 8, on the basis that the embodiment shown in fig. 6 is provided with a load cache pool (the left diagonal filled tile represents the first load cache (special effect load & join rendering queue 1), and the dotted filled tile represents the second load cache (special effect load & join rendering queue 2)), the embodiment may provide an encoding cache pool, which includes at least two encoding caches. In fig. 8, the diagonal right filled tiles represent the first encoding buffer (e.g., the tile storing render 1+ join encoding queue 1 for rendering data of the first target video frame and encoding callback 1 for encoding data), and the vertical striped filled tiles represent the second encoding buffer (e.g., the tile storing render 2+ join encoding queue 2 for rendering data of the second target video frame and encoding callback 2 for encoding data). Therefore, the second target video frame is taken as an example. In a specific implementation process, the special effect resource of the second target video frame is loaded through the main thread, the second target video frame is loaded in the second loading cache (dotted filling), and a rendering command is called to control the GPU to render and encode the second target video frame which is loaded and cached in the second loading cache according to the special effect resource of the second target video frame. And releasing the second load cache after the second target video frame is encoded, namely the encoding result is called back. As can be seen from fig. 8, in the previous frame of the second target video frame, the data of the first target video frame subjected to rendering and encoding is stored in the first load cache (right diagonal filling), in the next frame of the second target video frame, and in the third target video frame, after the first target video frame is encoded, the first load cache is released, and then the first load cache is invoked. Therefore, no matter between the first target video frame and the second target video frame or between the second target video frame and the third target video frame, the encoding and rendering processing of two adjacent target video frames cannot be overlapped, and the problem of texture string use between the adjacent frames is avoided.

It should be noted that, in this embodiment, the number of the coding buffers included in the coding buffer pool is not limited, although the coding buffer pool in this embodiment uses two coding buffers, in other embodiments, the number of the coding buffers may be set according to actual needs, so as to avoid a situation that all the coding buffers are occupied before the next frame of coded data arrives and the next frame of coded data cannot be received due to an excessively low number of the coding buffers.

Fig. 9 is a block diagram of a video processing device according to an embodiment of the present disclosure, which corresponds to the video processing method according to the foregoing embodiment. For ease of illustration, only portions relevant to embodiments of the present disclosure are shown. Referring to fig. 9, the apparatus 90 includes: an acquisition module 901, a first processing module 902, a second processing module 903 and a third processing module 904.

The acquiring module 901 is configured to acquire a first target video frame and a special effect resource corresponding to the first target video frame;

a first processing module 902, configured to execute, by a first thread, loading processing on the first target video frame and a corresponding special effect resource;

a second processing module 903, configured to execute, by a second thread, rendering processing on the first target video frame according to the loaded special effect resource to obtain a first rendered video frame;

a third processing module 904, configured to perform encoding processing on the first rendered video frame through a third thread to obtain an encoding result;

In the video processing apparatus provided in this embodiment, an obtaining module 901 obtains a first target video frame and a special effect resource corresponding to the first target video frame, a first processing module 902 executes, by using a first thread, loading processing on the first target video frame and the corresponding special effect resource, a second processing module 903 executes, by using a second thread, rendering processing on the first target video frame according to the loaded special effect resource to obtain a first rendered video frame, and a third processing module 904 performs, by using a third thread, encoding processing on the first rendered video frame to obtain an encoding result, where the first thread, the second thread, and the third thread are processed in parallel. According to the embodiment, the special effect rendering processing is split into the first thread and the second thread, the two threads are completed, time consumption of the special effect rendering processing can be greatly reduced, and the video synthesis rendering efficiency is improved.

In an embodiment of the present disclosure, the first processing module 902 is specifically configured to: and calling a corresponding load cache through a first thread to obtain a first target video frame, and adding the first target video frame into a rendering queue so as to receive the first target video frame in the rendering queue through a second thread.

In an embodiment of the present disclosure, the first processing module 902 is specifically configured to: and calling a first loading cache in the multiple loading caches through a first thread to obtain a first target video frame, and adding the first target video frame into a rendering queue. The first processing module 902 is further configured to: calling a second loading cache in the multiple loading caches through the first thread to obtain a second target video frame, and adding the second target video frame into a rendering queue; after a first rendering video frame corresponding to a first target video frame is obtained through a second thread, releasing the first loading cache, calling the first loading cache through the first thread after a plurality of loading caches are all called, obtaining an Nth target video frame after the first target video frame, and adding the Nth target video frame after the first target video frame into a rendering queue; the number of the loading caches is less than that of the target video frames, the first target video frame is a video frame before the second target video frame, N is the number of the loading caches, and the first loading cache is a loading cache before the second loading cache.

In an embodiment of the present disclosure, the second processing module 903 is specifically configured to: and calling a rendering cache through a second thread to receive the first target video frame, and controlling the GPU to render the first target video frame.

The third processing module 904 is specifically configured to: and calling an encoding cache through a third thread to receive a first rendered video frame in the encoding queue, controlling an encoder to encode the first rendered video frame, and calling back an encoding result.

In an embodiment of the present disclosure, the rendering cache and the encoding cache corresponding to the rendering processing and the encoding processing performed successively on the same target video frame are the same cache.

In an embodiment of the present disclosure, the second processing module 903 is specifically configured to: and calling a first encoding cache in the plurality of encoding caches through a second thread to receive the first target video frame, and controlling the GPU to render the first target video frame.

The second processing module 903 is further configured to: calling a second coding cache in the multiple coding caches through a second thread to receive a second target video frame, and controlling a GPU to render the second target video frame to obtain a second rendered video frame;

the third processing module 904 is specifically configured to: calling the first coding cache to receive a first rendered video frame in the coding queue through a third thread, controlling an encoder to encode the first rendered video frame, calling back a coding result, and releasing the first coding cache, so that after the plurality of coding caches are all called, calling the first coding cache through the second thread to receive an Mth target video frame after the first target video frame; the first target video frame is a video frame before the second target video frame, M is the number of coding buffers, and the first coding buffer is a coding buffer before the second coding buffer.

Referring to fig. 10, a schematic structural diagram of an electronic device 1000 suitable for implementing an embodiment of the present disclosure is shown, where the electronic device 1000 may be a terminal device or a server. Among them, the terminal Device may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a Digital broadcast receiver, a Personal Digital Assistant (PDA), a tablet computer (PAD), a Portable Multimedia Player (PMP), a car terminal (e.g., car navigation terminal), etc., and a fixed terminal such as a Digital TV, a desktop computer, etc. The electronic device shown in fig. 10 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 10, the electronic device 1000 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 1001 that may perform various suitable actions and processes according to a program stored in a Read Only Memory (ROM) 1002 or a program loaded from a storage device 1008 into a Random Access Memory (RAM) 1003. In the RAM 1003, various programs and data necessary for the operation of the electronic apparatus 1000 are also stored. The processing device 1001, ROM1002, and RAM 1003 are connected to each other by a bus 1004. An input/output (I/O) interface 1005 is also connected to bus 1004.

Generally, the following devices may be connected to the I/O interface 1005: input devices 1006 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 1007 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage devices 1008 including, for example, magnetic tape, hard disk, and the like; and a communication device 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data. While fig. 10 illustrates an electronic device 1000 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication means 1009, or installed from the storage means 1008, or installed from the ROM 1002. The computer program, when executed by the processing device 1001, performs the above-described functions defined in the methods of the embodiments of the present disclosure.

It should be noted that the computer readable medium of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of Network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a unit does not in some cases constitute a limitation of the unit itself, for example, the first retrieving unit may also be described as a "unit for retrieving at least two internet protocol addresses".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems on a chip (SOCs), complex Programmable Logic Devices (CPLDs), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

In a first aspect, according to one or more embodiments of the present disclosure, there is provided a video processing method, including:

encoding the first rendered video frame through a third thread to obtain an encoding result;

According to one or more embodiments of the present disclosure, the executing, by the first thread, the loading process of the first target video frame includes:

and calling a corresponding load cache through a first thread to obtain a first target video frame, and adding the first target video frame into a rendering queue so as to receive the first target video frame in the rendering queue through a second thread.

According to one or more embodiments of the present disclosure, the obtaining a first target video frame by calling a corresponding load cache through a first thread and adding the first target video frame to a rendering queue includes:

calling a first loading cache in a plurality of loading caches through a first thread to obtain a first target video frame, and adding the first target video frame into a rendering queue;

after the corresponding load cache is called by the first thread to obtain the first target video frame and the first target video frame is added into the rendering queue, the method further comprises the following steps:

calling a second loading cache in the multiple loading caches through the first thread to obtain a second target video frame, and adding the second target video frame into a rendering queue;

after a first rendering video frame corresponding to a first target video frame is obtained through a second thread, releasing the first loading cache, calling the first loading cache through the first thread after a plurality of loading caches are all called, obtaining an Nth target video frame after the first target video frame, and adding the Nth target video frame after the first target video frame into a rendering queue;

the number of the loading caches is less than that of the target video frames, the first target video frame is a video frame before the second target video frame, N is the number of the loading caches, and the first loading cache is a loading cache before the second loading cache.

According to one or more embodiments of the present disclosure, the performing, by the second thread, a rendering process on the first target video frame according to the loaded special effect resource includes:

calling a rendering cache through a second thread to receive the first target video frame, and controlling a GPU to perform rendering processing on the first target video frame;

the encoding, by the third thread, the first rendered video frame to obtain an encoding result includes:

and calling an encoding cache through a third thread to receive a first rendered video frame in the encoding queue, controlling an encoder to encode the first rendered video frame, and calling back an encoding result.

According to one or more embodiments of the present disclosure, the rendering cache and the encoding cache corresponding to the rendering processing and the encoding processing performed successively on the same target video frame are the same cache.

According to one or more embodiments of the present disclosure, the invoking, by the second thread, the render cache to receive the first target video frame, and controlling the GPU to perform a rendering process on the first target video frame include:

calling a first coding cache in the multiple coding caches through a second thread to receive the first target video frame, and controlling a GPU to render the first target video frame;

the invoking, by the second thread, the render cache to receive the first target video frame, and controlling the GPU to perform rendering processing on the first target video frame further includes:

calling a second coding cache in the plurality of coding caches through a second thread to receive a second target video frame, and controlling a GPU to perform rendering processing on the second target video frame to obtain a second rendered video frame;

the calling, by the third thread, the encoding cache to receive the first rendered video frame in the encoding queue, controlling the encoder to encode the first rendered video frame, and calling back an encoding result, including:

calling the first coding cache to receive a first rendered video frame in the coding queue through a third thread, controlling an encoder to encode the first rendered video frame, calling back a coding result, and releasing the first coding cache, so that after the plurality of coding caches are all called, calling the first coding cache through the second thread to receive an Mth target video frame after the first target video frame;

the first target video frame is a video frame before the second target video frame, M is the number of coding buffers, and the first coding buffer is a coding buffer before the second coding buffer.

In a second aspect, according to one or more embodiments of the present disclosure, there is provided a video processing apparatus including:

the second processing module is used for executing rendering processing on the first target video frame according to the loaded special effect resource through a second thread to obtain a first rendered video frame;

the third processing module is used for coding the first rendering video frame through a third thread to obtain a coding result;

According to one or more embodiments of the present disclosure, the first processing module is specifically configured to: and calling a corresponding load cache through a first thread to obtain a first target video frame, and adding the first target video frame into a rendering queue so as to receive the first target video frame in the rendering queue through a second thread.

According to one or more embodiments of the present disclosure, the first processing module is specifically configured to: and calling a first loading cache in the multiple loading caches through a first thread to obtain a first target video frame, and adding the first target video frame into a rendering queue.

The first processing module is further configured to: calling a second loading cache in the multiple loading caches through the first thread to obtain a second target video frame, and adding the second target video frame into a rendering queue; after a first rendering video frame corresponding to a first target video frame is obtained through a second thread, releasing the first loading cache, calling the first loading cache through the first thread after a plurality of loading caches are all called, obtaining an Nth target video frame after the first target video frame, and adding the Nth target video frame after the first target video frame into a rendering queue; the number of the loading caches is less than that of the target video frames, the first target video frame is a video frame before the second target video frame, N is the number of the loading caches, and the first loading cache is a loading cache before the second loading cache.

According to one or more embodiments of the present disclosure, the second processing module is specifically configured to: calling a rendering cache through a second thread to receive the first target video frame, and controlling a GPU to perform rendering processing on the first target video frame;

the third processing module is specifically configured to: and calling an encoding cache through a third thread to receive a first rendered video frame in the encoding queue, controlling an encoder to encode the first rendered video frame, and calling back an encoding result.

According to one or more embodiments of the present disclosure, the second processing module is specifically configured to: and calling a first encoding cache in the plurality of encoding caches through a second thread to receive the first target video frame, and controlling the GPU to render the first target video frame.

The second processing module is further configured to: and calling a second coding cache in the plurality of coding caches through a second thread to receive a second target video frame, and controlling the GPU to perform rendering processing on the second target video frame to obtain a second rendered video frame.

The third processing module is specifically configured to: calling the first coding cache to receive a first rendered video frame in the coding queue through a third thread, controlling an encoder to encode the first rendered video frame, calling back a coding result, and releasing the first coding cache, so that after the plurality of coding caches are all called, calling the first coding cache through the second thread to receive an Mth target video frame after the first target video frame; the first target video frame is a video frame before the second target video frame, M is the number of coding buffers, and the first coding buffer is a coding buffer before the second coding buffer.

In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device including: at least one processor and memory;

the memory stores computer-executable instructions;

In a fourth aspect, according to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium having stored therein computer-executable instructions that, when executed by a processor, implement the video processing method as set forth in the first aspect and various possible designs of the first aspect.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and the technical features disclosed in the present disclosure (but not limited to) having similar functions are replaced with each other to form the technical solution.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A video processing method, comprising:

the first thread, the second thread, and the third thread are processed in parallel;

the executing, by the first thread, the loading process of the first target video frame includes:

2. The method of claim 1, wherein the invoking, by the first thread, the corresponding load cache to obtain the first target video frame and adding the first target video frame to the rendering queue comprises:

3. The method according to claim 1, wherein the performing, by the second thread, the rendering process of the first target video frame according to the loaded special effect resource comprises:

and calling the coding cache to receive the first rendered video frame in the coding queue through the third thread, controlling the coder to code the first rendered video frame, and calling back the coding result.

4. The method according to claim 3, wherein the rendering cache and the encoding cache corresponding to the rendering processing and the encoding processing performed successively on the same target video frame are the same cache.

5. The method of claim 4, wherein the invoking of the render cache by the second thread to receive the first target video frame controls the GPU to render the first target video frame, and comprises:

the invoking, by the second thread, the render cache to receive the first target video frame, and after controlling the GPU to perform rendering processing on the first target video frame, further includes:

6. A video processing apparatus, comprising:

the first thread, the second thread and the third thread are processed in parallel;

the first processing module is specifically configured to:

7. An electronic device, comprising: at least one processor and a memory;

the memory stores computer-executable instructions;

the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the video processing method of any of claims 1 to 5.

8. A computer-readable storage medium having computer-executable instructions stored therein, which when executed by a processor, implement the video processing method of any one of claims 1 to 5.