WO2021082799A1

WO2021082799A1 - Videoconference auxiliary stream processing method and device

Info

Publication number: WO2021082799A1
Application number: PCT/CN2020/116249
Authority: WO
Inventors: 李鹏; 曾庆波; 郭热思
Original assignee: 南京中兴新软件有限责任公司
Priority date: 2019-10-30
Filing date: 2020-09-18
Publication date: 2021-05-06
Also published as: CN112752057A

Abstract

The embodiments of the present application provide a videoconference auxiliary stream processing method and device. The videoconference auxiliary stream processing method comprises: coding a videoconference auxiliary stream by using a coding window of a predetermined length; performing real-time bandwidth detection and traffic limitation on the videoconference auxiliary stream by sliding the coding window; and when the total code amount of a current auxiliary stream video frame in the coding window exceeds a bandwidth limit, performing frame drop on the current auxiliary stream video frame in the coding window, so that the total code amount of the auxiliary stream video frame in the coding window subjected to the frame drop does not exceed the bandwidth limit. The solution of the embodiments of the present application can improve the picture definition of the videoconference auxiliary stream, and improve user experience.

Description

Method and device for processing auxiliary stream of video conference

Technical field

The present invention relates to the field of communication technology, and in particular to a method and device for processing auxiliary streams of video conferences.

Background technique

In a video conference, the video conference terminal sends the main stream and secondary stream to other terminals through the MCU (Multipoint Control Unit). Among them, the main stream refers to the video signal of the camera connected to the video conference terminal, and the secondary stream refers to the signal of the computer connected to the video conference terminal, such as the screen signal output when the computer shares files.

In video conferences, PCs, notebooks, etc. are very common application scenarios as the auxiliary stream access of the video conference terminal. It is often necessary to view Excel or presentation PPT on a PC or notebook. In this application scenario, the auxiliary stream picture is more complicated and contains many lines. When the picture is switched, the video auxiliary stream video frame after the encoding of the video conference terminal will be blurry and occupy Very large bandwidth, the MCU device has strict bandwidth restrictions. When the encoding of the auxiliary stream exceeds the negotiated bandwidth, the MCU device will discard part of the video auxiliary stream data, causing the receiving end of the auxiliary stream to decode abnormally, causing problems such as screen freezes or blurring. .

Therefore, when the video conference is connected through the MCU, the clarity of the auxiliary stream picture is poor, and there are phenomena such as freezing, which affects the user experience.

Summary of the invention

The embodiments of the present application provide a method and device for processing a video conference auxiliary stream, which can improve the picture clarity of the video conference auxiliary stream and improve user experience.

According to the first aspect of the present application, an embodiment of the present application provides a video conference auxiliary stream processing method, including:

Use a coding window of a predetermined length to encode the video conference auxiliary stream;

Perform real-time bandwidth detection and current limiting processing on the auxiliary stream of the video conference by sliding the coding window: when the total amount of coding of the auxiliary stream video frames in the current coding window exceeds the bandwidth limit, the auxiliary stream in the current coding window is The stream video frame is subjected to frame dropping processing, so that the total amount of coding of the auxiliary stream video frame in the coding window after the frame is dropped does not exceed the bandwidth limit.

According to the second aspect of the present application, an embodiment of the present application provides a video conference auxiliary stream processing device, including:

The encoding module is set to encode the auxiliary stream of the video conference by using an encoding window of a predetermined length;

The bandwidth limit module is set to perform real-time bandwidth detection and current limiting processing on the video conference auxiliary stream by sliding the encoding window: when the total amount of encoding of the auxiliary stream video frame in the current encoding window exceeds the bandwidth limit, all The auxiliary stream video frames in the current encoding window are subjected to frame drop processing, so that the total amount of encoding of the auxiliary stream video frames in the encoding window after frame loss does not exceed the bandwidth limit.

According to the third aspect of the present application, an embodiment of the present application provides a video conference auxiliary stream processing device, including:

A memory, a processor, and a video conference auxiliary stream processing program that is stored on the memory and can run on the processor, and the video conference auxiliary stream processing program is executed by the processor to realize the video conference auxiliary stream processing Method steps.

According to the fourth aspect of the present application, an embodiment of the present application provides a computer-readable storage medium having a video conference auxiliary stream processing program stored on the computer-readable storage medium, and the video conference auxiliary stream processing program is executed by a processor When realizing the steps of the above video conference auxiliary stream processing method.

Description of the drawings

FIG. 1 is a flowchart of a method for processing auxiliary streams of a video conference according to Embodiment 1 of the application;

FIG. 2 is a schematic diagram of a video conference auxiliary stream processing apparatus according to Embodiment 2 of the application.

Detailed ways

In order to make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily if there is no conflict.

The steps shown in the flowcharts of the drawings may be executed in a computer system such as a set of computer-executable instructions. And, although a logical sequence is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here.

Example 1

As shown in Figure 1, an embodiment of the present application provides a video conference auxiliary stream processing method, including:

Step S110, encoding the auxiliary stream of the video conference by using an encoding window of a predetermined length;

Step S120: Perform real-time bandwidth detection and current limiting processing on the auxiliary stream of the video conference by sliding the encoding window: when the total amount of encoding of the auxiliary stream video frames in the current encoding window exceeds the bandwidth limit, the current encoding window Frame drop processing is performed on the auxiliary stream video frames inside, so that the total amount of coding of the auxiliary stream video frames in the encoding window after frame loss does not exceed the bandwidth limit;

In an implementation manner, the performing real-time bandwidth detection and current limiting processing on the auxiliary stream of the video conference by sliding the encoding window includes:

When the total amount of coding of the auxiliary stream video frames in the current coding window does not exceed the bandwidth limit, retaining the coding results of all the auxiliary stream video frames in the coding window; sending all the coding frames in the current coding window;

When the total amount of coding of the first n auxiliary stream video frames in the current coding window exceeds the bandwidth limit, the coding results of the first m auxiliary stream video frames in the coding window are retained, and the mth frame after the current coding window is discarded The coding results of all auxiliary stream video frames; sending the first m coding frames in the current coding window;

Wherein, m is the largest integer that can make the total coding amount of the first m auxiliary stream video frames in the current sliding window not exceed the bandwidth limit;

In the foregoing embodiment, the encoding size of the encoded video conference auxiliary stream is detected frame by frame through the sliding window, which can ensure the real-time performance of encoding bandwidth detection. And by abandoning the coding of some frames in the sliding window, the bandwidth limit can be adaptively adapted.

If performing frame dropping processing on the auxiliary stream video frame in the current encoding window, then perform I frame encoding on the auxiliary stream video frame in the next encoding window;

Among them, I-frame encoding refers to intra-frame compression encoding, which compresses, encodes and transmits the full-frame image information, and only uses I-frame data to reconstruct the complete image during decoding;

In an implementation manner, before encoding a video conference auxiliary stream by using an encoding window of a predetermined length, the method further includes:

Set coding parameters for the video conference auxiliary stream;

In an implementation manner, the encoding parameter includes at least one of the following parameters: quantizer parameter (Quantizer Parameter, QP for short), resolution, frame rate, and code rate;

Among them, the quantization parameter reflects the compression of spatial details. The smaller the value, the finer the quantization, the higher the image quality, and the longer the code stream generated. The resolution determines the fineness of the bitmap image details. Generally, the higher the resolution of the image, the more pixels it contains, the clearer the image and the greater the amount of coding. Frame rate (frame rate) refers to the number of frames or images that are shown or displayed per second. If you play more frames per second, the playback will be smoother. Too low frame rate will cause intermittent playback. The code rate is the number of data bits transmitted per unit time during data transmission. The higher the bit rate, the smaller the compressed ratio and the smaller the playback quality loss.

Among them, the setting of coding parameters is usually limited by the bandwidth, the quantization parameter cannot be unlimitedly small, and the resolution cannot be unlimitedly high. By setting appropriate coding parameters, the clarity of the picture can be improved to a certain extent.

In the foregoing embodiment, by setting appropriate coding parameters, the definition of each frame of the video conference auxiliary stream can be improved. Since the video conference auxiliary stream is not sensitive to the smoothness of the picture, it is possible to sacrifice part of the smoothness of the video conference auxiliary stream picture through dynamic frame drop processing, thereby improving the clarity of the overall picture.

In an embodiment, the length of the encoding window is 1 second;

In an embodiment, the sliding distance of the encoding window is 1 frame each time;

In an implementation manner, performing real-time bandwidth detection and current limiting processing on the video conference auxiliary stream by sliding the encoding window includes performing the following steps:

Step 1: Create an array Array[] with a length of M. The value of each element in the array Array[] represents the encoding amount of one frame of video conference auxiliary stream encoding frame. Initially, the value of all elements is set to 0; The size of M is equal to the size of the frame rate;

Step 2: Assign the coding amount of the current coded frame j of the auxiliary stream of the video conference to the i-th element in the array Array[], i=(j-1)%x+1; 1≤i≤M, 1≤j ≤N, N is the total number of frames of the auxiliary stream of the video conference;% is the sign of the remainder;

Step 3: Determine whether the cumulative sum of the values of all elements in the array Array[] is greater than the preset bit rate of the video conference auxiliary stream. If yes, give up the coding result of the j-th frame of the video conference auxiliary stream, and add the array Array[] The value of the i-th element of is modified to 0, otherwise the encoding result of the j-th frame of the video conference auxiliary stream is accepted;

Step 4: Determine whether the current encoded frame j is the last frame of the video conference auxiliary stream, that is, determine whether j is equal to N, and if yes, it ends, otherwise the current encoded frame of the video conference auxiliary stream is moved one frame backward, that is, Add 1 to j and return to step two.

In the foregoing embodiment, the encoding size of the encoded video conference auxiliary stream is detected frame by frame by using the array Array[] with a preset length, which can ensure the real-time performance of encoding bandwidth detection. And by discarding the encoding result of the encoding frame corresponding to some elements in the array Array[], the upper limit of the bandwidth can be adaptively adapted.

In the foregoing embodiment, by executing the foregoing video conference auxiliary stream processing method on the MCU, the picture definition of the video conference auxiliary stream can be improved when the video conference is accessed through the MCU, and the user experience can be improved.

Compared with related technologies, the method for processing video conference auxiliary streams provided by the embodiments of the present application uses an encoding window of a predetermined length to encode the video conference auxiliary stream, and performs real-time processing of the video conference auxiliary stream by sliding the encoding window. Bandwidth detection and current limiting processing: when the total amount of encoding of the auxiliary stream video frames in the current encoding window exceeds the bandwidth limit, the auxiliary stream video frames in the current encoding window are subjected to frame loss processing to make the encoding window after frame loss The total amount of encoding of the auxiliary stream video frames does not exceed the bandwidth limit. The technical solution of the embodiment of the present application can improve the picture clarity of the auxiliary stream of the video conference and improve the user experience.

Example 2

As shown in FIG. 2, an embodiment of the present application provides a video conference auxiliary stream processing device, including:

The encoding module 10 is configured to encode the auxiliary stream of the video conference by using an encoding window of a predetermined length;

The bandwidth limiting module 20 is configured to perform real-time bandwidth detection and current limiting processing on the auxiliary stream of the video conference by sliding the encoding window: when the total amount of encoding of the auxiliary stream video frames in the current encoding window exceeds the bandwidth limit, The auxiliary stream video frames in the current encoding window are subjected to frame drop processing, so that the total amount of encoding of the auxiliary stream video frames in the encoding window after frame loss does not exceed the bandwidth limit.

In an implementation manner, the bandwidth limiting module is configured to perform real-time bandwidth detection and current limiting processing on the auxiliary stream of the video conference by sliding the encoding window in the following manner:

Wherein, m is the largest integer that can make the total coding amount of the first m auxiliary stream video frames in the current sliding window not exceed the bandwidth limit.

If the auxiliary stream video frame in the current encoding window is subjected to frame dropping processing, then the auxiliary stream video frame in the next encoding window is subjected to I-frame encoding.

In an implementation manner, the encoding module is further configured to set encoding parameters for the auxiliary stream of the video conference.

In an implementation manner, the encoding parameters include at least one of the following parameters: quantization parameters, resolution, frame rate, and code rate;

In an embodiment, the length of the encoding window is 1 second.

In an embodiment, the sliding distance of the encoding window is 1 frame each time.

Perform the following steps:

Step 4: Judge whether j is equal to N, if yes, end, otherwise add 1 to j and return to step 2.

Compared with related technologies, the video conference auxiliary stream processing device provided by the embodiments of the present application uses an encoding window of a predetermined length to encode the video conference auxiliary stream, and performs real-time on the video conference auxiliary stream by sliding the encoding window. Bandwidth detection and current limiting processing: when the total amount of encoding of the auxiliary stream video frames in the current encoding window exceeds the bandwidth limit, the auxiliary stream video frames in the current encoding window are subjected to frame loss processing to make the encoding window after frame loss The total amount of encoding of the auxiliary stream video frames does not exceed the bandwidth limit. The technical solution of the embodiment of the present application can improve the picture clarity of the auxiliary stream of the video conference and improve the user experience.

Example 3

An embodiment of the application provides a video conference auxiliary stream processing device, including:

A memory, a processor, and a video conference auxiliary stream processing program that is stored on the memory and can run on the processor, and the video conference auxiliary stream processing program is executed by the processor to implement the above-mentioned embodiment 1 The steps of the video conference auxiliary stream processing method described above.

Example 4

An embodiment of the present application provides a computer-readable storage medium having a video conference auxiliary stream processing program stored on the computer-readable storage medium. When the video conference auxiliary stream processing program is executed by a processor, the implementation of the above-mentioned embodiment 1 is The steps of the video conference auxiliary stream processing method.

A person of ordinary skill in the art can understand that all or some of the steps, systems, and functional modules/units in the method of the embodiments of the application in the above text can be implemented as software, firmware, hardware, and appropriate combinations thereof. In the hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may consist of several physical components. The components are executed cooperatively. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software may be distributed on a computer-readable medium, and the computer-readable medium may include a computer storage medium (or a non-transitory medium) and a communication medium (or a transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile data implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Sexual, removable and non-removable media. Computer storage media include but are not limited to RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or Any other medium used to store desired information and that can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media usually contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as carrier waves or other transmission mechanisms, and may include any information delivery media. .

It should be noted that the embodiments of the present application can also have various other embodiments. Without departing from the spirit and essence of the present application, those skilled in the art can make various corresponding changes and changes based on the embodiments of the present application. Deformations, but these corresponding changes and deformations should fall within the protection scope of the appended claims of this application.

Industrial applicability

Encode the video conference auxiliary stream by using an encoding window of a predetermined length, and slide the encoding window to perform real-time bandwidth detection and current limiting processing on the video conference auxiliary stream: when the total amount of encoding of the auxiliary stream video frames in the current encoding window exceeds the bandwidth limit , Perform frame drop processing on the auxiliary stream video frames in the current encoding window, so that the total amount of encoding of the auxiliary stream video frames in the encoding window after frame loss does not exceed the bandwidth limit. It solves the problem of poor clarity and stuttering of the secondary stream, improves the clarity of the secondary stream of the video conference, and improves the user experience.

Claims

A video conference auxiliary stream processing method, including:

Use a coding window of a predetermined length to encode the video conference auxiliary stream;

Perform real-time bandwidth detection and current limiting processing on the auxiliary stream of the video conference by sliding the coding window: when the total amount of coding of the auxiliary stream video frame in the current coding window exceeds the bandwidth limit, the auxiliary stream in the current coding window is The stream video frame is subjected to frame dropping processing so that the total amount of coding of the auxiliary stream video frame in the coding window after the frame is dropped does not exceed the bandwidth limit.
The method of claim 1, wherein:

The performing real-time bandwidth detection and current limiting processing on the auxiliary stream of the video conference by sliding the encoding window includes:

When the total amount of coding of the auxiliary stream video frames in the current coding window does not exceed the bandwidth limit, retaining the coding results of all the auxiliary stream video frames in the coding window; sending all the coding frames in the current coding window;

When the total amount of coding of the first n auxiliary stream video frames in the current coding window exceeds the bandwidth limit, the coding results of the first m auxiliary stream video frames in the coding window are retained, and the mth frame after the current coding window is discarded The coding results of all auxiliary stream video frames; sending the first m coding frames in the current coding window;

Wherein, m is the largest integer that can make the total coding amount of the first m auxiliary stream video frames in the current sliding window not exceed the bandwidth limit.
The method of claim 1 or 2, wherein:

The performing real-time bandwidth detection and current limiting processing on the auxiliary stream of the video conference by sliding the encoding window includes:

If the auxiliary stream video frame in the current encoding window is subjected to frame dropping processing, then the auxiliary stream video frame in the next encoding window is subjected to I-frame encoding.
The method according to claim 1, before encoding the auxiliary stream of the video conference by using an encoding window of a predetermined length, wherein the method further comprises:

Set encoding parameters for the video conference auxiliary stream.
The method of claim 1, wherein:

The length of the encoding window is 1 second.
The method of claim 1, wherein:

Each sliding distance of the encoding window is 1 frame.
A video conference auxiliary stream processing device, including:

The encoding module is set to encode the auxiliary stream of the video conference by using an encoding window of a predetermined length;

The bandwidth limit module is set to perform real-time bandwidth detection and current limiting processing on the video conference auxiliary stream by sliding the encoding window: when the total amount of encoding of the auxiliary stream video frame in the current encoding window exceeds the bandwidth limit, all The auxiliary stream video frames in the current encoding window are subjected to frame drop processing, so that the total amount of encoding of the auxiliary stream video frames in the encoding window after frame loss does not exceed the bandwidth limit.
The device of claim 7, wherein:

The bandwidth limiting module is set to perform real-time bandwidth detection and current limiting processing on the auxiliary stream of the video conference by sliding the encoding window in the following manner:

When the total amount of coding of the auxiliary stream video frames in the current coding window does not exceed the bandwidth limit, retaining the coding results of all the auxiliary stream video frames in the coding window; sending all the coding frames in the current coding window;

When the total amount of coding of the first n auxiliary stream video frames in the current coding window exceeds the bandwidth limit, the coding results of the first m auxiliary stream video frames in the coding window are retained, and the mth frame after the current coding window is discarded The coding results of all auxiliary stream video frames; sending the first m coding frames in the current coding window;

Wherein, m is the largest integer that can make the total coding amount of the first m auxiliary stream video frames in the current sliding window not exceed the bandwidth limit.
A video conference auxiliary stream processing device, including:

A memory, a processor, and a video conference auxiliary stream processing program that is stored on the memory and can run on the processor, and the video conference auxiliary stream processing program is executed by the processor to implement the above claims 1-6 The steps of any one of the video conference auxiliary stream processing methods.
A computer-readable storage medium, wherein a video conference auxiliary stream processing program is stored on the computer-readable storage medium, and when the video conference auxiliary stream processing program is executed by a processor, the above-mentioned any one of claims 1 to 6 is realized. The steps of the video conference auxiliary stream processing method described above.