CN110855908B - Multi-party video screen mixing method and device, network equipment and storage medium - Google Patents
Multi-party video screen mixing method and device, network equipment and storage medium Download PDFInfo
- Publication number
- CN110855908B CN110855908B CN201911128504.0A CN201911128504A CN110855908B CN 110855908 B CN110855908 B CN 110855908B CN 201911128504 A CN201911128504 A CN 201911128504A CN 110855908 B CN110855908 B CN 110855908B
- Authority
- CN
- China
- Prior art keywords
- frame
- video
- frame rate
- preset
- decoded
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000012545 processing Methods 0.000 claims abstract description 57
- 238000005498 polishing Methods 0.000 claims abstract description 29
- 230000002194 synthesizing effect Effects 0.000 claims abstract description 13
- 238000012549 training Methods 0.000 claims description 40
- 238000013136 deep learning model Methods 0.000 claims description 17
- 238000001914 filtration Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 6
- 238000013135 deep learning Methods 0.000 claims description 5
- 230000015572 biosynthetic process Effects 0.000 claims description 2
- 238000007517 polishing process Methods 0.000 claims description 2
- 238000003786 synthesis reaction Methods 0.000 claims description 2
- 238000004891 communication Methods 0.000 abstract description 11
- 230000000694 effects Effects 0.000 abstract description 6
- 230000008859 change Effects 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 5
- 238000003780 insertion Methods 0.000 description 5
- 230000037431 insertion Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000033228 biological regulation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/136—Incoming video signal characteristics or properties
- H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/146—Data rate or code amount at the encoder output
- H04N19/149—Data rate or code amount at the encoder output by estimating the code amount by means of a model, e.g. mathematical model or statistical model
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
- H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Algebra (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The embodiment of the invention relates to the technical field of communication, and discloses a multi-party video screen mixing method, which comprises the following steps: acquiring coding frames of N videos to be mixed and an input frame rate of each video, wherein N is a natural number greater than 1; decoding encoded frames of videos with different input frame rates and preset frame rates to obtain decoded frames, and acquiring characteristic parameters of the videos with the different input frame rates and the preset frame rates; inputting the characteristic parameters into a motion complexity model to obtain the motion complexity of the video with the input frame rate different from the preset frame rate; performing frame interpolation or frame polishing processing on the decoded frame according to a preset frame rate and motion complexity to obtain a processed decoded frame; and synthesizing the mixed screen video according to the processed decoded frames. The embodiment of the invention also provides a multi-party video screen mixing device, network equipment and a storage medium. The multi-party video screen mixing method, the multi-party video screen mixing device, the network equipment and the storage medium can improve the display effect of the communication video of the mixed screen.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a multiparty video mixing method and apparatus, a network device, and a storage medium.
Background
In a plurality of video communication scenarios, such as multiparty video conferencing, a plurality of videos for video communication needs to be mixed in one screen to facilitate viewing and interaction of video communication users.
However, since the types of the plurality of terminals performing video communication may be different, the frame rates of the plurality of videos may be different accordingly; and the different frame rates can cause that the videos after the screen mixing cannot be synchronized, and the display effect of the communication video of the screen mixing is influenced.
Disclosure of Invention
The embodiment of the invention aims to provide a multi-party video screen mixing method, a multi-party video screen mixing device, network equipment and a storage medium, which can improve the display effect of a communication video of screen mixing.
In order to solve the above technical problem, an embodiment of the present invention provides a multi-party video mixing method, including the following steps: acquiring coding frames of N videos to be mixed and an input frame rate of each video, wherein N is a natural number greater than 1; decoding the coded frames of the videos with the input frame rate different from the preset frame rate to obtain decoded frames, and acquiring characteristic parameters of the videos with the input frame rate different from the preset frame rate; inputting the characteristic parameters into a motion complexity model to obtain the motion complexity of the video with the input frame rate different from the preset frame rate; performing frame interpolation or frame polishing processing on the decoded frame according to a preset frame rate and motion complexity to obtain a processed decoded frame; and synthesizing the mixed screen video according to the processed decoded frames.
The embodiment of the invention also provides a multi-party video screen mixing device, which comprises: the device comprises a coded frame acquisition module, a frame mixing module and a frame mixing module, wherein the coded frame acquisition module is used for acquiring coded frames of N videos to be mixed and the input frame rate of each video, and N is a natural number greater than 1; the parameter acquisition module is used for decoding the coding frame of the video with the input frame rate different from the preset frame rate to obtain a decoding frame and acquiring the characteristic parameter of the video with the input frame rate different from the preset frame rate; the complexity acquisition module is used for inputting the characteristic parameters into the motion complexity model to obtain the motion complexity of the video with the input frame rate different from the preset frame rate; the frame processing module is used for performing frame interpolation or frame polishing processing on the decoded frame according to a preset frame rate and motion complexity to obtain a processed decoded frame; and the video synthesis module is used for synthesizing the mixed screen video according to the processed decoded frame.
An embodiment of the present invention further provides a network device, including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the multi-party video mixing method.
The embodiment of the invention also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the multi-party video screen mixing method.
Compared with the prior art, the embodiment of the invention has the advantages that the frame interpolation or frame polishing processing is carried out on the videos with the input frame rates different from the preset frame rate, so that the input frame rates of the processed videos are the same as or close to the preset frame rate, the frame rates of the videos after screen mixing are the same as or close to each other, and the synchronism of the videos after screen mixing is improved; meanwhile, the motion complexity of the video is obtained through the motion complexity model, and when the video is subjected to frame interpolation or frame polishing, the frame interpolation or frame polishing is carried out in combination with the motion complexity of the video, so that the inserted frames or the dropped frames are more reasonable, the video after screen mixing is more in line with the real situation, and the display effect of the communication video of the screen mixing is improved.
In addition, the frame interpolation or frame polishing processing is performed on the decoded frame according to the preset frame rate and the motion complexity, and the method comprises the following steps: if the input frame rate is greater than the preset frame rate, performing frame polishing processing on the decoded frame, wherein the frame polishing processing is performed on the decoded frame with the motion complexity less than or equal to a first preset value preferentially; and if the input frame rate is less than the preset frame rate, performing frame interpolation on the decoded frame, wherein the frame interpolation is preferentially performed on the decoded frame with the motion complexity greater than or equal to a second preset value.
In addition, acquiring characteristic parameters of a video with an input frame rate different from a preset frame rate specifically comprises the following steps: and acquiring a macro block motion vector, a macro block coding type and a frame-level global motion vector of a video with an input frame rate different from a preset frame rate as characteristic parameters. Because the macro block motion vector, the macro block coding type and the frame-level global motion vector of the video can reflect the motion information of the video, the motion information can be input into the motion complexity model by taking the macro block motion vector, the macro block coding type and the frame-level global motion vector as characteristic parameters, and more accurate motion complexity can be obtained.
In addition, before inputting the feature parameters into the motion complexity model, the method further includes: acquiring a video training sample, and extracting characteristic parameters of the video training sample; inputting the characteristic parameters of the video training samples into a deep learning model for training; and taking the trained deep learning model as a motion complexity model. The video training samples are input into the deep learning model for training to obtain the motion complexity model, so that the motion complexity of videos with different input frame rates and preset frame rates can be obtained according to the motion complexity model, and when the videos are subjected to frame interpolation or frame polishing, the motion complexity of the videos is combined for frame interpolation or frame polishing, so that the mixed-screen videos can better accord with the actual motion conditions, and the synchronism of the mixed-screen videos is improved.
In addition, the characteristic parameters of the video training samples are input into the deep learning model for training, and the training specifically comprises the following steps: and inputting the characteristic parameters of the video training samples into an open source deep learning framework for training.
In addition, acquiring the encoding frames of the N videos to be mixed and the input frame rate of each video includes: and respectively acquiring the input frame rate of each video by adopting Kalman filtering. The Kalman filtering can acquire the change range of the input frame rate of the video, so that the input frame rate of the video can be finely adjusted according to the change range, the input frame rate of the video is more consistent with the actual condition, and the synchronism and the fluency of the mixed-screen video are more favorably improved.
In addition, the method for synthesizing the mixed screen video according to the processed decoding frame comprises the following steps: and synthesizing the mixed screen video according to the processed decoded frame and the coded frame of the video with the input frame rate being the same as the preset frame rate.
Drawings
One or more embodiments are illustrated by the figures in the accompanying drawings, which correspond to and are not intended to limit the embodiments.
Fig. 1 is a schematic flow chart of a multi-party video mixing method according to a first embodiment of the present invention;
fig. 2 is a schematic flowchart of S104 refinement in the multiparty video mixing method according to the first embodiment of the present invention;
fig. 3 is a schematic flowchart of a multi-party video mixing method according to a first embodiment of the present invention before S103;
fig. 4 is a schematic block diagram of a multi-party video mixing device according to a second embodiment of the present invention;
fig. 5 is a schematic structural diagram of a network device according to a third embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.
The invention relates to a multi-party video screen mixing method, which comprises the steps of acquiring coding frames of N videos to be mixed and the input frame rate of each video; decoding encoded frames of videos with different input frame rates and preset frame rates to obtain decoded frames, and acquiring characteristic parameters of the videos with the different input frame rates and the preset frame rates; inputting the characteristic parameters into a motion complexity model to obtain the motion complexity of the video with the input frame rate different from the preset frame rate; performing frame interpolation or frame polishing processing on the decoded frame according to a preset frame rate and motion complexity to obtain a processed decoded frame; and synthesizing the mixed screen video according to the processed decoded frames. By carrying out frame insertion and frame polishing processing on the decoded frames different from the preset frame rate, the synchronism of the mixed screen video can be good when the mixed screen video is displayed; meanwhile, the motion complexity of the video is obtained through the motion complexity model, frame inserting or frame throwing processing is carried out according to the motion complexity of the video, the inserted frames or the thrown frames can be more reasonable, the mixed screen video can better accord with the real motion situation, and the display smoothness is improved.
It should be noted that the main implementation body of the embodiment of the present invention is a server for receiving a video, where the server may be implemented by an independent server or a server cluster composed of a plurality of servers, and the following description takes the server as an example.
The specific flow of the multiparty video screen mixing method provided by the embodiment of the invention is shown in fig. 1, and comprises the following steps:
s101: acquiring coding frames of N videos to be mixed and an input frame rate of each video, wherein N is a natural number larger than 1.
The input frame rate refers to a frame rate of a video received by the server. Alternatively, the input frame rate of each video may be estimated using a low-pass filtering method. Preferably, kalman filtering may be employed to obtain the input frame rate of each video separately. Kalman filter (Kalman filter) is a highly efficient recursive filter (autoregressive filter) that can estimate the state of a dynamic system from a series of incomplete and noisy measurements.
It should be understood that, due to the instability of the network environment, the input frame rate of the video may be affected by the network environment to change, and the purpose of using the low-pass filtering method is to obtain a relatively stable value of the input frame rate of the video. By adopting the Kalman filtering method, the relatively stable value of the input frame rate of the video can be obtained, and the variation range of the input frame rate caused by the influence of the network environment can also be obtained. When the network environment is unstable, for example, becomes very jammed, the input frame rate of the video should be finely adjusted. The kalman filtering may obtain a change range of the frame rate of the video, so that the input frame rate of the video may be fine-tuned according to the change range, for example, if a relatively stable frame rate value of a certain video is 25, the input frame rate of the video may be fine-tuned to 26, 27, or 24 according to the change range during fine tuning, so that the input frame rate of the video better conforms to an actual situation, and the method is more favorable for improving the synchronization and the fluency when the mixed-screen video is displayed.
Specifically, the server receives RTP packets of N videos to be mixed, analyzes the RTP packets to obtain coding frames of the videos, and estimates the input frame rate of the videos by adopting a preset low-pass filtering method to obtain the input frame rate of each video. For example, after the server analyzes the RTP packet of each video to obtain the encoded frame of the video, the input frame rate of each video can be obtained by calling a kalman filtering method.
S102: decoding the coding frame of the video with the input frame rate different from the preset frame rate to obtain a decoding frame, and acquiring the characteristic parameters of the video with the input frame rate different from the preset frame rate.
The preset frame rate may be specifically set according to an actual situation, and is not limited herein. For example, the input frame rates of N videos may be averaged, and the average value is used as a preset frame rate; or acquiring the input frame rate with the maximum number in the N videos as the preset frame rate, and the like. The characteristic parameter of the video refers to a parameter representing a video motion complexity, and optionally, the characteristic parameter of the video with the input frame rate different from the preset frame rate refers to a macroblock motion vector, a macroblock coding type and a frame-level global motion vector of the video, because the macroblock motion vector, the macroblock coding type and the frame-level global motion vector of the video can reflect the video motion complexity, and may also be other characteristic parameters of the video, which is not limited specifically here.
Optionally, the server firstly judges the input frame rates of the N videos, and if the input frame rate of one of the N videos is the same as the preset frame rate, the server does not perform processing; and if the input frame rate of the video is different from the preset frame rate, decoding the coded frame of the video with the input frame rate different from the preset frame rate to obtain a decoded frame. Because the video with the input frame rate being the same as the preset frame rate does not need to be subjected to subsequent frame interpolation or frame polishing, the server only needs to acquire the characteristic parameters of the video with the input frame rate being different from the preset frame rate.
Specifically, the server may use a universal video coding and decoding open source library to decode a video with an input frame rate different from a preset frame rate to obtain a decoded frame. The general video codec open source library is, for example, ffmpeg, Xvid, X264, or ffdshow. Optionally, the server may also use other video coding and decoding tools to decode the video, which is not limited herein. When the video is decoded, the server can acquire various motion information of the video as the characteristic parameters of the video.
S103: and inputting the characteristic parameters into the motion complexity model to obtain the motion complexity of the video with the input frame rate different from the preset frame rate.
The motion complexity model can be obtained by firstly obtaining the characteristic parameters of a plurality of video samples and then inputting the characteristic parameters of the plurality of video samples into a pre-constructed neural network for training. The neural network may be, for example, a deep neural network, a convolutional neural network, a deep confidence network, a recurrent neural network, and the like.
Specifically, the server inputs the acquired characteristic parameters of the video with the input frame rate different from the preset frame rate into the motion complexity model, so that the motion complexity of the video with the input frame rate different from the preset frame rate can be obtained.
S104: and performing frame interpolation or frame polishing processing on the decoded frame according to the preset frame rate and the motion complexity to obtain the processed decoded frame.
The frame insertion processing refers to inserting a decoding frame to increase the input frame rate of the video, and the frame throwing processing refers to throwing off the decoding frame to decrease the input frame rate of the video.
Specifically, the server compares the input frame rate of the video with a preset frame rate, and if the input frame rate of the video is greater than the preset frame rate, the video needs to be subjected to frame throwing processing, so that the input frame rate of the video subjected to frame throwing processing is equal to or close to the preset frame rate; if the input frame rate of the video is less than the preset frame rate, frame interpolation processing needs to be performed on the video, so that the input frame rate of the video after frame interpolation processing is equal to or close to the preset frame rate.
When the frame interpolation or frame dropping processing is carried out on the decoding frame of the video, the frame interpolation or frame dropping processing is carried out by the server side according to the motion complexity of the video.
In a specific example, the frame interpolation or frame dropping process performed on the decoded frame according to the preset frame rate and the motion complexity in S104 specifically includes the following steps, as shown in fig. 2:
s1041: and if the input frame rate is greater than the preset frame rate, performing frame throwing processing on the decoded frame, wherein the frame throwing processing is preferentially performed on the decoded frame with the motion complexity smaller than or equal to a first preset value.
S1042: and if the input frame rate is less than the preset frame rate, performing frame interpolation on the decoded frame, wherein the frame interpolation is preferentially performed on the decoded frame with the motion complexity greater than or equal to a second preset value.
Alternatively, the motion complexity may be expressed by a normalized numerical value, for example, the motion complexity is expressed by a numerical value between 0 and 1, and then the motion complexity is classified into different categories according to the difference of the numerical value, for example, [0, 0.1] represents motion still, (0.1, 0.5] represents motion smoothness, (0.5, 0.8] represents motion complexity, (0.8, 1.0] represents motion complexity, and the categories corresponding to different numerical values may be set according to actual needs, which is not limited specifically here.
In S1041, the first preset value may be set according to an actual situation, for example, may be set to 0.5, which is not limited specifically herein.
Specifically, the server compares the input frame rate of the video with a preset frame rate, and if the input frame rate of the video is greater than the preset frame rate, the server performs frame throwing processing on the decoded frames of the video, and preferentially throws the decoded frames with the motion complexity less than or equal to a first preset value when performing the frame throwing processing. It should be understood that discarding the decoded frames with the motion complexity less than or equal to the first preset value preferentially means discarding at least some of the decoded frames with the motion complexity less than or equal to the first preset value, so that the input frame rate of the video is the same as or close to the preset frame rate; if all the decoding frames smaller than or equal to the first preset value are discarded and the input frame rate of the video is not the same as or close to the preset frame rate, continuing to perform frame discarding processing on the decoding frames with the motion complexity larger than or equal to the first preset value. Optionally, when frame polishing processing is performed on a decoded frame with a motion complexity less than or equal to a first preset value, further subdivision may be performed according to the motion complexity of the decoded frame, so that the decoded frame with a lower motion complexity is subjected to frame polishing processing preferentially; the frame polishing process may be performed according to the sequence of the decoded frames without further subdivision, and may be specifically set according to actual needs, which is not limited herein. Optionally, when the frame dropping process is continuously performed on the decoded frames with the motion complexity greater than or equal to the first preset value, a third preset value may be further set, so that the frames with the motion complexity greater than or equal to the third preset value are retained as much as possible, and the frames with the motion complexity less than or equal to the third preset value are preferentially dropped. The third preset value may be set according to actual needs, for example, set to 0.8, and is not limited specifically here.
In S1042, the second preset value may be set according to actual needs, for example, set to 0.1, which is not limited herein.
Specifically, the server compares the input frame rate of the video with a preset frame rate, and performs frame interpolation processing on the decoded frame if the input frame rate of the video is less than the preset frame rate. And when the frame interpolation is carried out, preferentially interpolating the decoded frame with the motion complexity larger than or equal to a second preset value. It should be understood that, preferentially performing frame interpolation on decoded frames with motion complexity greater than or equal to a second preset value means performing frame interpolation on at least some decoded frames with motion complexity greater than or equal to the second preset value, so that the input frame rate of the video is the same as or close to the preset frame rate; if the frame interpolation processing is carried out on the decoded frame with the motion complexity being larger than or equal to the second preset value, and the input frame rate of the video is not the same as or close to the preset frame rate, the frame interpolation processing is continuously carried out on the decoded frame with the motion complexity being smaller than the second preset value. Optionally, when the decoded frame with the motion complexity greater than or equal to the second preset value is subjected to frame interpolation processing, the decoded frame with the higher motion complexity may be further subdivided according to the motion complexity of the decoded frame, so that the decoded frame with the higher motion complexity is subjected to frame interpolation processing preferentially, or may not be further subdivided, and the frame interpolation processing is performed according to the sequence of the decoded frames, which may be specifically set according to actual needs, and is not limited herein. It should be noted that, when the frame interpolation processing is performed on the decoded frame with the motion complexity greater than or equal to the second preset value, the interpolated frame inserted between the previous frame and the next frame of the decoded frame with the motion complexity greater than or equal to the second preset value is obtained by calculation, and then the interpolated frame is inserted. When the insertion frame is calculated, the insertion frame can be calculated according to some characteristic parameters (such as a macro block motion vector) of the video, so that the display effect of the video after the insertion frame is inserted is smoother. Optionally, when the frame interpolation processing is continuously performed on the decoded frame with the motion complexity smaller than the second preset value, the decoded frame with the motion complexity smaller than the second preset value may be simply copied to reduce the amount of calculation, and may also be obtained by using the above calculation method, which is not limited specifically here.
It should be noted that when performing frame dropping or frame inserting processing on a video with an input frame rate different from the preset frame rate to make the input frame rate of the video equal to or close to the preset frame rate, the number of frames to be dropped or inserted may be determined according to the difference between the input frame rate of the video and the preset frame rate. For example, if the input frame rate of the video is 27 and the preset frame rate is 25, the number of frames that should be discarded is 27-25 — 2 frames.
S105: and synthesizing the mixed screen video according to the processed decoded frames.
The screen mixing refers to that videos to be mixed are combined in one screen.
Specifically, the server side performs screen mixing on the decoded frames subjected to frame throwing or frame inserting processing, synthesizes screen mixing video, encodes and outputs the screen mixing video. It should be understood that, since there may be a video with the same input frame rate as the preset frame rate, and the video does not need to be processed by frame dropping or frame interpolation, if there is a video with the same input frame rate as the preset frame rate, the server mixes the processed decoded frame with the video, synthesizes the mixed-screen video, and outputs the synthesized mixed-screen video.
Compared with the prior art, the multi-party video screen mixing method provided by the embodiment of the invention has the advantages that the input frame rate of the processed video is the same as or close to the preset frame rate by performing frame interpolation or frame polishing processing on the video with the input frame rate different from the preset frame rate, so that the frame rates of the videos after screen mixing are the same as or close to each other, and the synchronism of the videos after screen mixing is improved; meanwhile, the motion complexity of the video is obtained through the motion complexity model, and when the video is subjected to frame interpolation or frame polishing, the frame interpolation or frame polishing is carried out in combination with the motion complexity of the video, so that the inserted frames or the dropped frames are more reasonable, the video after screen mixing is more in line with the real situation, and the display effect of the communication video of the screen mixing is improved.
In a specific example, before S103, that is, before inputting the feature parameters into the motion complexity model, as shown in fig. 3, the method further includes the following steps:
s201: and acquiring a video training sample, and extracting the characteristic parameters of the video training sample.
S202: and inputting the characteristic parameters of the video training samples into a deep learning model for training.
S203: and taking the trained deep learning model as a motion complexity model.
In S201, the video training sample may be composed of video data sources collected in actual call processes in various scenes, or may be a video recorded specially according to training requirements, which is not limited herein.
In S202, the deep learning model is a learning model constructed based on a deep neural network, and is used for learning the operation information of the video, so as to obtain the motion complexity of the video. Alternatively, the deep learning model may be a general open source deep learning framework, such as tensflow, Torch, or Caffe, that is, the feature parameters of the video exercise book are input into the open source deep learning framework for training.
Specifically, the server side obtains video training samples, carries out coding and decoding operations on the video training samples, extracts characteristic parameters of the video training samples, inputs the characteristic parameters of the video training samples into the deep learning model for training, and takes the trained deep learning model as a motion complexity model after the deep learning model is trained to a preset precision.
The video training samples are input into the deep learning model for training to obtain the motion complexity model, so that the motion complexity of videos with different input frame rates and preset frame rates can be obtained according to the motion complexity model, and when the videos are subjected to frame interpolation or frame polishing, the motion complexity of the videos is combined for frame interpolation or frame polishing, so that the mixed-screen videos can better accord with the actual motion conditions, and the display synchronism and smoothness of the mixed-screen videos are improved.
The steps of the above methods are divided for clarity, and the implementation may be combined into one step or split some steps, and the steps are divided into multiple steps, so long as the steps contain the same logical relationship, which is within the protection scope of the present patent; it is within the scope of the patent to add insignificant modifications to the algorithms or processes or to introduce insignificant design changes to the core design without changing the algorithms or processes.
A second embodiment of the present invention relates to a multi-party video mixing device, as shown in fig. 4, including: an encoded frame acquisition module 301, a parameter acquisition module 302, a complexity acquisition module 303, a frame processing module 304, and a video composition module 305.
The encoding frame acquiring module 301 is configured to acquire encoding frames of N videos to be mixed and an input frame rate of each of the videos, where N is a natural number greater than 1;
a parameter obtaining module 302, configured to decode the encoded frames of the video with the input frame rate different from the preset frame rate to obtain decoded frames, and obtain characteristic parameters of the video with the input frame rate different from the preset frame rate;
a complexity obtaining module 303, configured to input the feature parameter to a motion complexity model, so as to obtain a motion complexity of a video with the input frame rate different from a preset frame rate;
a frame processing module 304, configured to perform frame interpolation or frame polishing on the decoded frame according to the preset frame rate and the motion complexity to obtain a processed decoded frame;
and a video synthesizing module 305, configured to synthesize a mixed-screen video according to the processed decoded frames.
Further, the frame processing module 304 is further configured to:
if the input frame rate is greater than the preset frame rate, performing frame polishing processing on the decoded frame, wherein the frame polishing processing is performed on the decoded frame with the motion complexity less than or equal to a first preset value preferentially;
and if the input frame rate is less than the preset frame rate, performing frame interpolation on the decoded frame, wherein the frame interpolation is preferentially performed on the decoded frame with the motion complexity greater than or equal to a second preset value.
Further, the parameter obtaining module 302 is further configured to: and acquiring a macro block motion vector, a macro block coding type and a frame-level global motion vector of a video with an input frame rate different from a preset frame rate as characteristic parameters.
Further, the multi-party video mixing device provided by the embodiment of the present invention further includes a model determining module, wherein the model determining module is configured to:
acquiring a video training sample, and extracting characteristic parameters of the video training sample;
inputting the characteristic parameters of the video training samples into a deep learning model for training;
and taking the trained deep learning model as the motion complexity model.
Further, the model determination module is further configured to: and inputting the characteristic parameters of the video training samples into an open source deep learning framework for training.
Further, the coded frame acquiring module 301 is further configured to: and respectively acquiring the input frame rate of each video by adopting Kalman filtering.
Further, the video composition module 305 is further configured to: and synthesizing the mixed screen video according to the processed decoded frame and the coded frame of the video with the input frame rate being the same as the preset frame rate.
It should be understood that this embodiment is an example of the apparatus corresponding to the first embodiment, and may be implemented in cooperation with the first embodiment. The related technical details mentioned in the first embodiment are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related-art details mentioned in the present embodiment can also be applied to the first embodiment.
It should be noted that each module referred to in this embodiment is a logical module, and in practical applications, one logical unit may be one physical unit, may be a part of one physical unit, and may be implemented by a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, elements that are not so closely related to solving the technical problems proposed by the present invention are not introduced in the present embodiment, but this does not indicate that other elements are not present in the present embodiment.
A third embodiment of the invention is directed to a network device, as shown in fig. 5, comprising at least one processor 401; and a memory 402 communicatively coupled to the at least one processor 401; the memory 402 stores instructions executable by the at least one processor 401, and the instructions are executed by the at least one processor 401 to enable the at least one processor 401 to execute the multi-party video mixing method.
Where the memory 402 and the processor 401 are coupled by a bus, which may include any number of interconnected buses and bridges that couple one or more of the various circuits of the processor 401 and the memory 402 together. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 401 may be transmitted over a wireless medium via an antenna, which may receive the data and transmit the data to the processor 401.
The processor 401 is responsible for managing the bus and general processing and may provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory 402 may be used to store data used by processor 401 in performing operations.
A fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program. The computer program realizes the above-described method embodiments when executed by a processor.
That is, those skilled in the art can understand that all or part of the steps in the method of the foregoing embodiments may be implemented by a program to instruct related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples of practicing the invention, and that various changes in form and detail may be made therein without departing from the spirit and scope of the invention in practice.
Claims (9)
1. A multi-party video screen mixing method is characterized by comprising the following steps:
acquiring coding frames of N videos to be mixed and an input frame rate of each video, wherein N is a natural number greater than 1;
decoding the coded frames of the video with the input frame rate different from the preset frame rate to obtain decoded frames, and acquiring the characteristic parameters of the video with the input frame rate different from the preset frame rate, wherein the characteristic parameters comprise: a macroblock motion vector, a macroblock coding type, and a frame-level global motion vector;
inputting the characteristic parameters into a motion complexity model to obtain the motion complexity of the video with the input frame rate different from a preset frame rate;
performing frame interpolation or frame polishing processing on the decoded frame according to the preset frame rate and the motion complexity to obtain a processed decoded frame; wherein the frame polishing process comprises: the method for processing the frame dropping of the decoded frame with lower motion complexity is characterized by comprising the following steps: preferentially discarding at least part of decoded frames with motion complexity smaller than or equal to a first preset value so as to enable the input frame rate of the video to be the same as or close to the preset frame rate; the frame interpolation processing comprises: the method for performing frame interpolation on the decoded frame with higher motion complexity is characterized by comprising the following steps: preferentially performing frame interpolation processing on at least part of decoded frames with motion complexity greater than or equal to a second preset value so as to enable the input frame rate of the video to be the same as or close to the preset frame rate;
and synthesizing the mixed screen video according to the processed decoded frame.
2. The multi-party video mixing method according to claim 1, wherein the frame interpolation or frame dropping processing on the decoded frame according to the preset frame rate and the motion complexity comprises:
if the input frame rate is greater than the preset frame rate, performing frame polishing processing on the decoded frame, wherein the frame polishing processing is performed on the decoded frame with the motion complexity less than or equal to a first preset value preferentially;
and if the input frame rate is less than the preset frame rate, performing frame interpolation on the decoded frame, wherein the frame interpolation is preferentially performed on the decoded frame with the motion complexity greater than or equal to a second preset value.
3. The multi-party video mixing method according to claim 1, further comprising, before inputting the feature parameters into a motion complexity model:
acquiring a video training sample, and extracting characteristic parameters of the video training sample;
inputting the characteristic parameters of the video training samples into a deep learning model for training;
and taking the trained deep learning model as the motion complexity model.
4. The multi-party video mixing method according to claim 3, wherein the feature parameters of the video training samples are input into a deep learning model for training, specifically:
and inputting the characteristic parameters of the video training samples into an open source deep learning framework for training.
5. The multi-party video mixing method according to claim 1, wherein said obtaining encoded frames of N videos to be mixed and an input frame rate of each of the videos comprises:
and respectively acquiring the input frame rate of each video by adopting Kalman filtering.
6. The multi-party video mixing method according to claim 1, wherein said synthesizing of the mixed screen video according to the processed decoded frames comprises:
and synthesizing the mixed screen video according to the processed decoded frame and the coded frame of the video with the input frame rate being the same as the preset frame rate.
7. A multi-party video mixing device, comprising:
the device comprises a coded frame acquisition module, a frame mixing module and a frame mixing module, wherein the coded frame acquisition module is used for acquiring coded frames of N videos to be mixed and an input frame rate of each video, and N is a natural number greater than 1;
a parameter obtaining module, configured to decode the encoded frames of the video with the input frame rate different from the preset frame rate to obtain decoded frames, and obtain characteristic parameters of the video with the input frame rate different from the preset frame rate, where the characteristic parameters include: a macroblock motion vector, a macroblock coding type, and a frame-level global motion vector;
the complexity obtaining module is used for inputting the characteristic parameters into a motion complexity model to obtain the motion complexity of the video with the input frame rate different from a preset frame rate;
the frame processing module is used for performing frame interpolation or frame polishing processing on the decoded frame according to the preset frame rate and the motion complexity to obtain a processed decoded frame; wherein the frame dropping process comprises: the method for processing the frame dropping in the decoding frame with lower motion complexity is characterized by comprising the following steps: preferentially discarding at least part of decoded frames with motion complexity smaller than or equal to a first preset value so as to enable the input frame rate of the video to be the same as or close to the preset frame rate; the frame interpolation processing comprises: the method for performing frame interpolation on the decoded frame with higher motion complexity is characterized by comprising the following steps: preferentially performing frame interpolation processing on at least part of decoded frames with the motion complexity greater than or equal to a second preset value so as to enable the input frame rate of the video to be the same as or close to the preset frame rate;
and the video synthesis module is used for synthesizing the mixed screen video according to the processed decoded frame.
8. A network device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the multi-party video mixing method of any one of claims 1-6.
9. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the multi-party video mixing method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911128504.0A CN110855908B (en) | 2019-11-18 | 2019-11-18 | Multi-party video screen mixing method and device, network equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911128504.0A CN110855908B (en) | 2019-11-18 | 2019-11-18 | Multi-party video screen mixing method and device, network equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110855908A CN110855908A (en) | 2020-02-28 |
| CN110855908B true CN110855908B (en) | 2022-09-27 |
Family
ID=69602021
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911128504.0A Active CN110855908B (en) | 2019-11-18 | 2019-11-18 | Multi-party video screen mixing method and device, network equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110855908B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111741303B (en) * | 2020-06-09 | 2022-03-25 | Oppo广东移动通信有限公司 | Deep video processing method and device, storage medium and electronic equipment |
| CN112511768B (en) * | 2020-11-27 | 2024-01-02 | 上海网达软件股份有限公司 | Multi-picture synthesis method, device, equipment and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102082896B (en) * | 2011-03-04 | 2014-04-16 | 中山大学 | Method for treating video of liquid crystal display device |
| CN104660916B (en) * | 2013-11-18 | 2018-09-18 | 杭州海康威视数字技术股份有限公司 | The processing method of screen splicing system and video data stream |
| CN104092974A (en) * | 2014-05-27 | 2014-10-08 | 浙江工业大学 | Multi-channel video clip splicing system and method based on moving object identity |
| CN104363508B (en) * | 2014-11-21 | 2018-03-23 | 浙江宇视科技有限公司 | A kind of image split-joint method and device that prevent video from retracting |
| TW201637432A (en) * | 2015-04-02 | 2016-10-16 | Ultracker Technology Co Ltd | Real-time image stitching device and real-time image stitching method |
| CN105763919A (en) * | 2016-04-14 | 2016-07-13 | 福州瑞芯微电子股份有限公司 | Method and device for display and video synchronization |
| US10873708B2 (en) * | 2017-01-12 | 2020-12-22 | Gopro, Inc. | Phased camera array system for generation of high quality images and video |
-
2019
- 2019-11-18 CN CN201911128504.0A patent/CN110855908B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110855908A (en) | 2020-02-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9872021B2 (en) | Video conversion method and apparatus | |
| CN102625106B (en) | Scene self-adaptive screen encoding rate control method and system | |
| CN104427291B (en) | A kind of image processing method and equipment | |
| CN110855908B (en) | Multi-party video screen mixing method and device, network equipment and storage medium | |
| CN103634556B (en) | Information transmission method, information receiving method and electronic apparatus | |
| CN114007107A (en) | Video self-adaption method, related equipment and storage medium | |
| US20130142251A1 (en) | Syntax extension of adaptive loop filter in hevc | |
| CN110225340B (en) | Control method and device for video coding, computing equipment and storage medium | |
| CN108063973A (en) | A kind of method for decoding video stream and equipment | |
| CN106454348B (en) | A kind of video coding-decoding method and device | |
| CN108600783A (en) | A kind of method of frame rate adjusting, device and terminal device | |
| WO2021057697A1 (en) | Video encoding and decoding methods and apparatuses, storage medium, and electronic device | |
| CN111617466B (en) | Method and device for determining coding format and method for realizing cloud game | |
| CN110708570A (en) | Video coding rate determining method, device, equipment and storage medium | |
| CN105208388B (en) | Dynamically adjustment encodes the method and system of frame per second in video communication | |
| CN114374841B (en) | Optimization method and device for video coding rate control and electronic equipment | |
| KR20160099977A (en) | Data processing apparatus and data processing method for videoconferencing service | |
| CN111954034B (en) | Video coding method and system based on terminal equipment parameters | |
| CN110572677B (en) | Video encoding and decoding method and device, storage medium and electronic device | |
| CN110572672A (en) | Video encoding and decoding method and device, storage medium and electronic device | |
| Martínez-Rach et al. | Quality assessment metrics vs. PSNR under packet lossscenarios in manet wireless networks | |
| CN114222169B (en) | Video streaming method, communication device and computer readable storage medium | |
| CN111327864A (en) | Video call control method and device, terminal equipment and storage medium | |
| CN110636295A (en) | Video encoding and decoding method and device, storage medium and electronic device | |
| CN110572674A (en) | Video encoding and decoding method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |