CN110855908B - Multi-party video screen mixing method and device, network equipment and storage medium - Google Patents

Abstract

Embodiments of the present invention relate to the technical field of communications, and disclose a method for multi-party video screen mixing, the method comprising: acquiring encoded frames of N videos to be mixed and an input frame rate of each video, where N is a natural number greater than 1; Decode the encoded frame of the video whose input frame rate is different from the preset frame rate to obtain the decoded frame, and obtain the feature parameters of the video whose input frame rate is different from the preset frame rate; input the feature parameters into the motion complexity model to obtain the input The motion complexity of the video whose frame rate is different from the preset frame rate; according to the preset frame rate and motion complexity, the decoded frame is inserted or dropped to obtain the processed decoded frame; screen video. Embodiments of the present invention also provide a multi-party video mixing device, a network device, and a storage medium. The multi-party video screen mixing method, device, network device and storage medium provided by the embodiments of the present invention can improve the display effect of mixed screen communication video.

Description

Multi-party video mixing screen method, device, network device and storage medium

technical field

The present invention relates to the field of communication technologies, and in particular, to a method, device, network device and storage medium for multi-party video mixing.

Background technique

In various video communication scenarios, such as multi-party video conferences, it is necessary to mix multiple videos for video communication on one screen to facilitate viewing and interaction of video communication users.

However, since the types of multiple terminals that carry out video communication may be different, the frame rates of multiple videos may also be different accordingly; and the difference in frame rates will cause the videos after the mixed screen to be out of sync, which will affect the mixed screen. The display effect of the communication video.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a method, device, network device and storage medium for multi-screen video mixing, which can improve the display effect of mixed-screen communication video.

In order to solve the above-mentioned technical problems, embodiments of the present invention provide a multi-party video mixing method, which includes the following steps: obtaining the encoded frames of N videos to be mixed and the input frame rate of each video, where N is greater than 1. Natural number; decode the encoded frame of the video whose input frame rate is different from the preset frame rate to obtain the decoded frame, and obtain the feature parameters of the video whose input frame rate is different from the preset frame rate; input the feature parameters into the motion complexity model, Obtain the motion complexity of the video whose input frame rate is different from the preset frame rate; perform frame interpolation or frame drop processing on the decoded frame according to the preset frame rate and motion complexity, and obtain the processed decoded frame; according to the processed decoded frame Composite mixed screen video.

Embodiments of the present invention also provide a multi-party video screen mixing device, comprising: an encoding frame acquisition module, configured to acquire the encoded frames of N videos to be mixed and the input frame rate of each video, where N is greater than 1 Natural number; parameter acquisition module, used to decode the encoded frame of the video whose input frame rate is different from the preset frame rate to obtain the decoded frame, and acquire the characteristic parameters of the video whose input frame rate is different from the preset frame rate; the complexity acquisition module , which is used to input the feature parameters into the motion complexity model to obtain the motion complexity of the video whose input frame rate is different from the preset frame rate; the frame processing module is used to interpolate the decoded frame according to the preset frame rate and motion complexity. Frame or drop frame processing to obtain processed decoded frames; video synthesis module, used to synthesize mixed-screen video according to the processed decoded frames.

Embodiments of the present invention also provide a network device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by at least one processor. One processor executes, so that at least one processor can execute the above-mentioned multi-party video mixing method.

Embodiments of the present invention further provide a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above-mentioned multi-party video mixing method is implemented.

Compared with the prior art, the embodiments of the present invention can make the input frame rate of the processed video the same as the preset frame rate or the same as the preset frame rate by performing frame insertion or frame drop processing on the video whose input frame rate is different from the preset frame rate. Close, so that the frame rate of each video after screen mixing is the same or similar, and the synchronization of the video after screen mixing is improved; at the same time, the motion complexity of the video is obtained through the motion complexity model, and the video is inserted or dropped. Frame processing When inserting or discarding frames in combination with the motion complexity of the video, the inserted or discarded frames can be made more reasonable, the mixed-screen video can be more in line with the real situation, and the display effect of the mixed-screen communication video can be improved. .

In addition, performing frame insertion or frame discarding processing on the decoded frame according to the preset frame rate and motion complexity includes: if the input frame rate is greater than the preset frame rate, performing frame discarding processing on the decoded frame, wherein the discarding frame processing takes priority to The decoded frame whose motion complexity is less than or equal to the first preset value is performed; if the input frame rate is less than the preset frame rate, frame interpolation processing is performed on the decoded frame. The decoding frame of the preset value is carried out.

In addition, acquiring the characteristic parameters of the video whose input frame rate is different from the preset frame rate, specifically: acquiring the macroblock motion vector, the macroblock coding type, and the frame-level global motion vector of the video whose input frame rate is different from the preset frame rate as a feature parameter. Since the video macroblock motion vector, macroblock coding type, and frame-level global motion vector can better reflect the motion information of the video, these are input into the motion complexity model as feature parameters to obtain a more accurate motion complexity.

In addition, before inputting the feature parameters into the motion complexity model, it also includes: acquiring video training samples, and extracting feature parameters of the video training samples; inputting the feature parameters of the video training samples into the deep learning model for training; Learning models as motion complexity models. By inputting the video training samples into the deep learning model for training, the motion complexity model is obtained, so that the motion complexity of the video whose input frame rate is different from the preset frame rate can be obtained according to the motion complexity model. During frame drop processing, inserting or dropping frames combined with the motion complexity of the video can make the mixed-screen video more in line with the actual motion situation and improve the synchronization of the mixed-screen video.

In addition, inputting the feature parameters of the video training samples into the deep learning model for training, specifically: inputting the feature parameters of the video training samples into the open source deep learning framework for training.

In addition, acquiring the encoded frames of the N videos to be mixed and the input frame rate of each video includes: using Kalman filtering to acquire the input frame rate of each video respectively. Since the Kalman filter can obtain the range of the video input frame rate change, the video input frame rate can be fine-tuned according to the range of change, so that the video input frame rate is more in line with the actual situation, which is more conducive to improving the mixed screen video synchronization and fluency.

In addition, synthesizing the mixed-screen video according to the processed decoded frames includes: synthesizing the mixed-screen video according to the processed decoded frames and the encoded frames of the video whose input frame rate is the same as the preset frame rate.

Description of drawings

One or more embodiments are exemplified by the pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments.

1 is a schematic flowchart of a multi-party video screen mixing method provided by a first embodiment of the present invention;

2 is a schematic flowchart of the refinement of S104 in the multi-party video screen mixing method provided by the first embodiment of the present invention;

3 is a schematic flowchart before S103 in the multi-party video mixing method provided by the first embodiment of the present invention;

4 is a schematic structural diagram of a module of a multi-party video screen mixing device provided by a second embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a network device according to a third embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, each embodiment of the present invention will be described in detail below with reference to the accompanying drawings. However, those of ordinary skill in the art can appreciate that, in the various embodiments of the present invention, many technical details are set forth in order for the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be realized.

The first embodiment of the present invention relates to a multi-party video mixing method, by acquiring the encoded frames of N videos to be mixed and the input frame rate of each video; Decode the encoded frame to obtain the decoded frame, and obtain the feature parameters of the video whose input frame rate is different from the preset frame rate; input the feature parameters into the motion complexity model to obtain the motion complexity of the video whose input frame rate is different from the preset frame rate According to the preset frame rate and motion complexity, the decoded frame is inserted or dropped to obtain the processed decoded frame; the mixed-screen video is synthesized according to the processed decoded frame. By performing frame insertion and frame throwing processing on the decoded frames different from the preset frame rate, the mixed-screen video can be displayed with good synchronization; at the same time, the motion complexity of the video can be obtained through the motion complexity model, according to the motion complexity of the video. It can make the inserted or discarded frames more reasonable, make the mixed-screen video more in line with the real motion situation, and improve the fluency of the display.

It should be noted that the implementation subject of the embodiments of the present invention is a server that receives video, wherein the server can be implemented by an independent server or a server cluster composed of multiple servers. The server is used as an example for description below.

The specific process of the multi-party video mixing method provided by the embodiment of the present invention is shown in FIG. 1 , and includes the following steps:

S101: Obtain the encoded frames of N videos to be mixed and the input frame rate of each video, where N is a natural number greater than 1.

The input frame rate refers to the frame rate of the video received by the server. Alternatively, a low-pass filtering method can be used to estimate the input frame rate of each video. Preferably, Kalman filtering can be used to obtain the input frame rate of each video respectively. Among them, the Kalman filter is an efficient recursive filter (autoregressive filter), which can estimate the state of a dynamic system from a series of incomplete and noisy measurements.

It should be understood that due to the instability of the network environment, the input frame rate of the video will be affected by the network environment and the input frame rate will change. The purpose of using the low-pass filtering method is to obtain a relatively stable input frame rate of the video. value. With the Kalman filtering method, in addition to obtaining a relatively stable value of the input frame rate of the video, it is also possible to obtain the variation range of the input frame rate affected by the network environment. The input frame rate of the video should be fine-tuned when the network environment is unstable, such as when it becomes very stuttering. Since the Kalman filter can obtain the range of the frame rate change of the video, the input frame rate of the video can be fine-tuned according to the range of change. For example, if the frame rate value of a relatively stable video is 25, the Adjust the input frame rate of the video to 26, 27, or 24, etc. according to the range of change, so that the input frame rate of the video is more in line with the actual situation, which is more conducive to improving the synchronization and fluency of the mixed-screen video when it is displayed.

Specifically, the server receives the RTP packets of the N videos to be mixed, parses the RTP packets to obtain the encoded frames of the video, and uses the preset low-pass filtering method to estimate the input frame rate of the video to obtain each The input frame rate of the video. For example, after parsing the RTP packet of each video to obtain the encoded frame of the video, the server can obtain the input frame rate of each video by calling the Kalman filtering method.

S102: Decode the encoded frames of the video whose input frame rate is different from the preset frame rate to obtain decoded frames, and acquire characteristic parameters of the video whose input frame rate is different from the preset frame rate.

Among them, the preset frame rate can be specifically set according to the actual situation, which is not limited here. For example, the input frame rates of N videos may be averaged, and the average value may be used as the preset frame rate; or the input frame rate with the largest number of the same number of N videos may be obtained as the preset frame rate, and so on. The feature parameters of the video refer to parameters that characterize the complexity of the video motion. Optionally, obtaining the feature parameters of the video whose input frame rate is different from the preset frame rate refers to obtaining the macroblock motion vector, macroblock coding type and frame of the video. Because the macroblock motion vector of the video, the macroblock coding type and the frame-level global motion vector can better reflect the motion complexity of the video, it can also be other characteristic parameters of the video, which is not limited here.

Optionally, the server first judges the input frame rates of the N videos, and if the input frame rates of the videos are the same as the preset frame rates, no processing is performed; if the input frame rates of the videos are the same as the preset frame rates If the input frame rate is different from the preset frame rate, the encoded frame of the video whose input frame rate is different from the preset frame rate is decoded to obtain the decoded frame. Since the video whose input frame rate is the same as the preset frame rate does not need to perform subsequent frame insertion or frame drop processing, the server only needs to obtain the characteristic parameters of the video whose input frame rate is different from the preset frame rate.

Specifically, the server can use a general video codec open source library to decode the video whose input frame rate is different from the preset frame rate to obtain decoded frames. Among them, the general video encoding and decoding open source library is, for example, ffmpeg, Xvid, X264, or ffdshow. Optionally, the server can also use other video encoding and decoding tools to decode the video, which is not specifically limited here. When decoding the video, the server can obtain various motion information of the video as characteristic parameters of the video.

S103: Input the feature parameters into the motion complexity model to obtain the motion complexity of the video whose input frame rate is different from the preset frame rate.

Among them, the motion complexity model can be obtained by first acquiring the feature parameters of multiple video samples, and then inputting the feature parameters of multiple video samples into a pre-built neural network for training. The neural network can be, for example, a deep neural network, a convolutional neural network, a deep belief network, a recurrent neural network, and the like.

Specifically, the server inputs the acquired feature parameters of the video with the input frame rate different from the preset frame rate into the motion complexity model, so as to obtain the motion complexity of the video with the input frame rate different from the preset frame rate.

S104: Perform frame interpolation or frame drop processing on the decoded frame according to the preset frame rate and motion complexity, to obtain a processed decoded frame.

The frame insertion processing refers to inserting decoded frames to increase the input frame rate of the video, and the frame dropping processing refers to discarding the decoded frames to reduce the input frame rate of the video.

Specifically, the server compares the input frame rate of the video with the preset frame rate. If the input frame rate of the video is greater than the preset frame rate, the video needs to be frame-thrown, so that the input frame of the video after the frame-throwing process is processed. The frame rate is equal to or close to the preset frame rate; if the input frame rate of the video is less than the preset frame rate, it is necessary to perform frame interpolation processing on the video, so that the input frame rate of the video after frame interpolation processing is equal to or close to the preset frame rate .

When performing frame insertion or frame drop processing on the decoded frames of the video, the server performs frame insertion or frame drop processing according to the motion complexity of the video.

In a specific example, in S104, frame interpolation or frame drop processing is performed on the decoded frame according to the preset frame rate and motion complexity, as shown in FIG. 2, and specifically includes the following steps:

S1041: If the input frame rate is greater than the preset frame rate, perform frame throwing processing on the decoded frame, wherein the frame throwing processing is preferentially performed on the decoded frame whose motion complexity is less than or equal to the first preset value.

S1042: If the input frame rate is less than the preset frame rate, perform frame interpolation processing on the decoded frame, wherein the frame interpolation processing is preferentially performed on the decoded frames whose motion complexity is greater than or equal to the second preset value.

Optionally, the motion complexity can be represented by a normalized value, for example, the motion complexity is represented by a value between 0 and 1, and then the motion complexity can be divided into different categories according to the different values, for example, [0, 0.1] stands for static motion, (0.1, 0.5] stands for gentle motion, (0.5, 0.8] stands for more complicated motion, (0.8, 1.0] stands for very complicated motion, and you can set the categories corresponding to different values according to actual needs. Here No specific restrictions are imposed.

In S1041, the first preset value may be set according to the actual situation, for example, it may be set to 0.5, which is not specifically limited here.

Specifically, the server compares the input frame rate of the video with the preset frame rate. If the input frame rate of the video is greater than the preset frame rate, it performs frame throwing processing on the decoded frames of the video. The decoded frames whose motion complexity is less than or equal to the first preset value are discarded. It should be understood that to preferentially discard decoded frames with motion complexity less than or equal to the first preset value means discarding at least some decoded frames with motion complexity less than or equal to the first preset value, so that the input frame rate of the video is reduced. The frame rate is the same as or close to the preset frame rate; if all the decoded frames less than or equal to the first preset value have been discarded, and the input frame rate of the video has not been made the same as or close to the preset frame rate, continue to adjust the motion complexity greater than or equal to the preset frame rate. Or the decoded frame equal to the first preset value is subjected to frame throwing processing. Optionally, when the frame throwing process is performed on the decoded frame whose motion complexity is less than or equal to the first preset value, it can be further subdivided according to the motion complexity of the decoded frame, so that the decoded frame with the lower motion complexity is more Priority is given to frame throwing processing; it is also possible to perform frame throwing processing without further subdivision, but to perform frame throwing processing according to the sequence of decoding frames, which can be set according to actual needs, which is not limited here. Optionally, when continuing to perform frame throwing processing on decoded frames whose motion complexity is greater than or equal to the first preset value, a third preset value can also be set, and try to keep those whose motion complexity is greater than or equal to the third preset value. frames, and preferentially discard frames whose motion complexity is less than or equal to the third preset value. Wherein, the third preset value can be set according to actual needs, for example, set to 0.8, which is not specifically limited here.

In S1042, the second preset value may be set according to actual needs, for example, set to 0.1, which is not specifically limited here.

Specifically, the server compares the input frame rate of the video with the preset frame rate, and performs frame interpolation processing on the decoded frame if the input frame rate of the video is less than the preset frame rate. When performing frame interpolation processing, frame interpolation is preferentially performed on decoded frames whose motion complexity is greater than or equal to the second preset value. It should be understood that to preferentially perform frame interpolation processing on decoded frames whose motion complexity is greater than or equal to the second preset value refers to performing frame interpolation processing on at least part of the decoded frames whose motion complexity is greater than or equal to the second preset value, Make the input frame rate of the video the same as or close to the preset frame rate; if frame interpolation has been performed on the decoded frames whose motion complexity is greater than or equal to the second preset value, the input frame rate of the video has not been adjusted to the preset frame rate. If they are the same or close to each other, continue to perform frame interpolation processing on the decoded frames whose motion complexity is less than the second preset value. Optionally, when performing frame interpolation processing on a decoded frame whose motion complexity is greater than or equal to the second preset value, further subdivisions can be made according to the motion complexity of the decoded frame, so that the decoded frame with higher motion complexity is more The frame insertion processing is given priority, or it may not be further subdivided, and the frame insertion processing is performed according to the sequence of the decoded frames, which can be set according to actual needs, which is not limited here. It should be noted that, when performing frame interpolation processing on the decoded frame whose motion complexity is greater than or equal to the second preset value, the frame interpolation is performed according to the frame before or after the decoded frame whose motion complexity is greater than or equal to the second preset value. The frames are calculated to get the interpolated frames inserted between them, and then the interpolated frames are interpolated. When calculating the inserted frame, it can be calculated according to some characteristic parameters of the video (such as macroblock motion vector), so that the display effect of the video after the inserted frame is inserted is smoother. Optionally, when continuing to perform frame interpolation processing on the decoded frame whose motion complexity is less than the second preset value, a simple copy may be performed on the decoded frame whose motion complexity is less than the second preset value to reduce the amount of calculation, or It can be obtained by the above calculation method, and there is no specific limitation here.

It should be noted that, when the input frame rate of the video is different from the preset frame rate, the frame throwing or interpolation processing is performed, so that the input frame rate of the video is the same as or close to the preset frame rate. The difference between the preset frame rates to determine the number of frames that should be dropped or inserted. For example, if the input frame rate of the video is 27 and the preset frame rate is 25, the number of frames that should be discarded is 27-25=2 frames.

S105: Synthesize a mixed-screen video according to the processed decoded frame.

Wherein, the mixed screen refers to synthesizing the videos to be mixed into one screen.

Specifically, the server mixes the decoded frames after frame-throwing or frame-insertion processing, synthesizes the mixed-screen video, encodes it, and outputs it. It should be understood that since there may be a video with the same input frame rate as the preset frame rate, and the video does not need to be framed or inserted, if there is a video with the same input frame rate as the preset frame rate, the service will The terminal mixes the processed decoded frame and the video together, synthesizes the mixed-screen video, and then outputs it.

Compared with the prior art, the multi-party video mixing method provided by the embodiment of the present invention can make the input frame rate of the processed video be changed by inserting or discarding the video whose input frame rate is different from the preset frame rate. The frame rate is the same or close to the preset frame rate, so that the frame rate of each video after screen mixing is the same or close to improving the synchronization of the video after screen mixing; at the same time, the motion complexity of the video is obtained through the motion complexity model. When inserting or throwing frames, combining the motion complexity of the video to insert or throw frames can make the inserted or dropped frames more reasonable, make the mixed video more realistic, and improve the mixing effect. The display effect of the communication video on the screen.

In a specific example, before S103, that is, before the feature parameters are input into the motion complexity model, as shown in FIG. 3, the following steps are further included:

S201: Obtain a video training sample, and extract feature parameters of the video training sample.

S202: Input the feature parameters of the video training samples into the deep learning model for training.

S203: Use the trained deep learning model as a motion complexity model.

In S201, the video training samples may be composed of video data sources collected during actual calls in various scenarios, or may be videos specially recorded according to training requirements, which are not limited here.

In S202, the deep learning model refers to a learning model constructed based on a deep neural network, which is used to learn the operation information of the video, thereby obtaining the motion complexity of the video. Optionally, the deep learning model can be a general open source deep learning framework, such as Tensorflow, Torch, or Caffe, etc., that is, input the feature parameters of the video training book into the open source deep learning framework for training.

Specifically, the server obtains the video training samples, performs encoding and decoding operations on the video training samples, extracts the feature parameters of the video training samples, and then inputs the feature parameters of the video training samples into the deep learning model for training. After setting the accuracy, the trained deep learning model is used as the motion complexity model.

By inputting the video training samples into the deep learning model for training, the motion complexity model is obtained, so that the motion complexity of the video whose input frame rate is different from the preset frame rate can be obtained according to the motion complexity model. During frame drop processing, inserting or dropping frames combined with the motion complexity of the video can make the mixed-screen video more in line with the actual motion situation, and improve the display synchronization and fluency of the mixed-screen video.

The step division of the above various methods is only for the purpose of describing clearly. During implementation, it can be combined into one step or some steps can be split and decomposed into multiple steps, as long as they contain the same logical relationship, they are all within the protection scope of this patent. ;Adding insignificant modifications to the algorithm or process or introducing insignificant designs, but not changing the core design of the algorithm and process are all within the scope of protection of this patent.

The second embodiment of the present invention relates to a multi-party video mixing device, as shown in FIG.

An encoded frame acquisition module 301, configured to acquire encoded frames of N videos to be mixed and the input frame rate of each of the videos, where N is a natural number greater than 1;

A parameter obtaining module 302, configured to decode the encoded frame of the video whose input frame rate is different from the preset frame rate to obtain a decoded frame, and obtain the characteristic parameters of the video whose input frame rate is different from the preset frame rate;

The complexity obtaining module 303 is used for inputting the feature parameter into the motion complexity model to obtain the motion complexity of the video whose input frame rate is different from the preset frame rate;

a frame processing module 304, configured to perform frame insertion or frame throwing processing on the decoded frame according to the preset frame rate and the motion complexity, to obtain a processed decoded frame;

The video synthesis module 305 is configured to synthesize the mixed-screen video according to the processed decoded frames.

Further, the frame processing module 304 is also used for:

If the input frame rate is greater than the preset frame rate, performing frame throwing processing on the decoded frame, wherein the frame throwing processing is preferentially performed on decoded frames whose motion complexity is less than or equal to a first preset value;

If the input frame rate is less than the preset frame rate, frame interpolation processing is performed on the decoded frame, wherein the frame interpolation processing is performed preferentially on decoded frames with a motion complexity greater than or equal to a second preset value.

Further, the parameter obtaining module 302 is further configured to obtain the macroblock motion vector, the macroblock coding type and the frame-level global motion vector of the video whose input frame rate is different from the preset frame rate as characteristic parameters.

Further, the multi-party video mixing device provided by the embodiment of the present invention further includes a model determination module, wherein the model determination module is used for:

Obtain a video training sample, and extract the characteristic parameters of the video training sample;

The feature parameters of the video training samples are input into the deep learning model for training;

The trained deep learning model is used as the motion complexity model.

Further, the model determination module is further configured to: input the feature parameters of the video training samples into an open source deep learning framework for training.

Further, the coded frame obtaining module 301 is further configured to obtain the input frame rate of each of the videos by using Kalman filtering.

Further, the video synthesis module 305 is further configured to: synthesize the mixed-screen video according to the processed decoded frame and the encoded frame of the video whose input frame rate is the same as the preset frame rate.

It is not difficult to find that this embodiment is a device example corresponding to the first embodiment, and this embodiment can be implemented in cooperation with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and are not repeated here in order to reduce repetition. Correspondingly, the related technical details mentioned in this embodiment can also be applied to the first embodiment.

It is worth mentioning that each module involved in this embodiment is a logical module. In practical applications, a logical unit may be a physical unit, a part of a physical unit, or multiple physical units. A composite implementation of the unit. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

The third embodiment of the present invention relates to a network device, as shown in FIG. 5 , comprising at least one processor 401; and a memory 402 connected in communication with the at least one processor 401; wherein, the memory 402 stores data that can be processed by the at least one processor 401. The instructions are executed by the processor 401, and the instructions are executed by the at least one processor 401, so that the at least one processor 401 can execute the above-mentioned multi-party video mixing method.

The memory 402 and the processor 401 are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors 401 and various circuits of the memory 402 together. The bus may also connect together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface between the bus and the transceiver. A transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a means for communicating with various other devices over a transmission medium. The data processed by the processor 401 is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor 401 .

Processor 401 is responsible for managing the bus and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory 402 may be used to store data used by the processor 401 when performing operations.

A fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program. The above method embodiments are implemented when the computer program is executed by the processor.

That is, those skilled in the art can understand that all or part of the steps in the method of the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium and includes several instructions to make a device (which can be It is a single chip microcomputer, a chip, etc.) or a processor (processor) that executes all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes.

Those skilled in the art can understand that the above-mentioned embodiments are specific examples for realizing the present invention, and in practical applications, various changes in form and details can be made without departing from the spirit and the spirit of the present invention. scope.

Claims (9)

1. A multi-party video screen mixing method is characterized by comprising the following steps:

acquiring coding frames of N videos to be mixed and an input frame rate of each video, wherein N is a natural number greater than 1;

decoding the coded frames of the video with the input frame rate different from the preset frame rate to obtain decoded frames, and acquiring the characteristic parameters of the video with the input frame rate different from the preset frame rate, wherein the characteristic parameters comprise: a macroblock motion vector, a macroblock coding type, and a frame-level global motion vector;

inputting the characteristic parameters into a motion complexity model to obtain the motion complexity of the video with the input frame rate different from a preset frame rate;

performing frame interpolation or frame polishing processing on the decoded frame according to the preset frame rate and the motion complexity to obtain a processed decoded frame; wherein the frame polishing process comprises: the method for processing the frame dropping of the decoded frame with lower motion complexity is characterized by comprising the following steps: preferentially discarding at least part of decoded frames with motion complexity smaller than or equal to a first preset value so as to enable the input frame rate of the video to be the same as or close to the preset frame rate; the frame interpolation processing comprises: the method for performing frame interpolation on the decoded frame with higher motion complexity is characterized by comprising the following steps: preferentially performing frame interpolation processing on at least part of decoded frames with motion complexity greater than or equal to a second preset value so as to enable the input frame rate of the video to be the same as or close to the preset frame rate;

and synthesizing the mixed screen video according to the processed decoded frame.

2. The multi-party video mixing method according to claim 1, wherein the frame interpolation or frame dropping processing on the decoded frame according to the preset frame rate and the motion complexity comprises:

if the input frame rate is greater than the preset frame rate, performing frame polishing processing on the decoded frame, wherein the frame polishing processing is performed on the decoded frame with the motion complexity less than or equal to a first preset value preferentially;

and if the input frame rate is less than the preset frame rate, performing frame interpolation on the decoded frame, wherein the frame interpolation is preferentially performed on the decoded frame with the motion complexity greater than or equal to a second preset value.

3. The multi-party video mixing method according to claim 1, further comprising, before inputting the feature parameters into a motion complexity model:

acquiring a video training sample, and extracting characteristic parameters of the video training sample;

inputting the characteristic parameters of the video training samples into a deep learning model for training;

and taking the trained deep learning model as the motion complexity model.

4. The multi-party video mixing method according to claim 3, wherein the feature parameters of the video training samples are input into a deep learning model for training, specifically:

and inputting the characteristic parameters of the video training samples into an open source deep learning framework for training.

5. The multi-party video mixing method according to claim 1, wherein said obtaining encoded frames of N videos to be mixed and an input frame rate of each of the videos comprises:

and respectively acquiring the input frame rate of each video by adopting Kalman filtering.

6. The multi-party video mixing method according to claim 1, wherein said synthesizing of the mixed screen video according to the processed decoded frames comprises:

and synthesizing the mixed screen video according to the processed decoded frame and the coded frame of the video with the input frame rate being the same as the preset frame rate.

7. A multi-party video mixing device, comprising:

the device comprises a coded frame acquisition module, a frame mixing module and a frame mixing module, wherein the coded frame acquisition module is used for acquiring coded frames of N videos to be mixed and an input frame rate of each video, and N is a natural number greater than 1;

a parameter obtaining module, configured to decode the encoded frames of the video with the input frame rate different from the preset frame rate to obtain decoded frames, and obtain characteristic parameters of the video with the input frame rate different from the preset frame rate, where the characteristic parameters include: a macroblock motion vector, a macroblock coding type, and a frame-level global motion vector;

the complexity obtaining module is used for inputting the characteristic parameters into a motion complexity model to obtain the motion complexity of the video with the input frame rate different from a preset frame rate;

the frame processing module is used for performing frame interpolation or frame polishing processing on the decoded frame according to the preset frame rate and the motion complexity to obtain a processed decoded frame; wherein the frame dropping process comprises: the method for processing the frame dropping in the decoding frame with lower motion complexity is characterized by comprising the following steps: preferentially discarding at least part of decoded frames with motion complexity smaller than or equal to a first preset value so as to enable the input frame rate of the video to be the same as or close to the preset frame rate; the frame interpolation processing comprises: the method for performing frame interpolation on the decoded frame with higher motion complexity is characterized by comprising the following steps: preferentially performing frame interpolation processing on at least part of decoded frames with the motion complexity greater than or equal to a second preset value so as to enable the input frame rate of the video to be the same as or close to the preset frame rate;

and the video synthesis module is used for synthesizing the mixed screen video according to the processed decoded frame.

8. A network device, comprising:

at least one processor; and the number of the first and second groups,

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the multi-party video mixing method of any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the multi-party video mixing method according to any one of claims 1 to 6.

Images