CN115086779A

CN115086779A - Video transmission system

Info

Publication number: CN115086779A
Application number: CN202210893047.XA
Authority: CN
Inventors: 邓志吉; 张朝阳; 刘明; 叶奇; 钟广海; 李辉
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2021-12-17
Filing date: 2022-07-27
Publication date: 2022-09-20
Anticipated expiration: 2042-07-27
Also published as: CN115086779B; CN114222194A

Abstract

The application relates to a video transmission system, which comprises a coding end, a platform end and a playing end, wherein the coding end is used for acquiring first information and sending the first information to the playing end; the playing end is used for receiving the first information and acquiring the second information; the network congestion control module is further used for determining the network congestion condition within the preset time length based on the first information and the second information; and/or determining the video frame playing condition within the preset time length based on the second information; and sending the network congestion condition and/or the video frame playing condition to the platform end; the platform end is used for receiving the network congestion condition and/or the video frame playing condition and determining a code stream adjustment strategy based on the network congestion condition and/or the video frame playing condition; and the platform end is also used for sending a code stream adjustment strategy to the encoding end and the playing end. The code streams of the encoding end and the playing end are coordinated and controlled through the platform end, end-to-end matching of video encoding, video decoding and video transmission is guaranteed, and quality of video transmission and playing is improved.

Description

Video transmission system

Technical Field

The present application relates to the field of video technologies, and in particular, to a video transmission system.

Background

With the development of wireless network technology and video technology, the use of video playing application is becoming more and more popular. The video encoding end generally forwards a video to the video playing end through the video platform end, so that an uplink network is formed between the video encoding end and the video platform end, and a downlink network is formed between the video playing end and the video platform end. However, the video encoding end cannot sense the network quality of the downlink network and the video playing quality of the playing end, and the video playing end cannot sense the network quality of the uplink network, so that the sensing of the video encoding end and the sensing of the video playing end are isolated, the code stream adjustment strategies of the video encoding end and the video playing end are poor in cooperation, the overall playing experience of the video service is poor, and the problems of playing jam, video missing and the like are prone to occurring.

Therefore, how to provide a video transmission system to improve the video playing quality is a problem that needs to be solved in the present application.

Disclosure of Invention

The embodiment of the invention provides a video transmission system, which is used for improving a code stream adjustment strategy of a video so as to improve the video playing quality and effectively improve the user experience.

The application provides a video transmission system, including encoding end, platform end and broadcast end, wherein:

the encoding end is used for acquiring first information, the first information comprises video frame sending information of the encoding end within a preset time length, and the first information is sent to the playing end;

the playing end is used for receiving the first information and acquiring second information, wherein the second information comprises video frame receiving information and/or video frame playing cache information of the playing end within the preset time length;

the network congestion control module is further configured to determine a network congestion condition within the preset duration based on the first information and the second information; and/or determining the video frame playing condition in the preset time length based on the second information;

and sending the network congestion condition and/or the video frame playing condition to the platform end;

the platform end is used for receiving the network congestion condition and/or the video frame playing condition and determining a code stream adjustment strategy based on the network congestion condition and/or the video frame playing condition;

the platform end is also used for sending the code stream adjustment strategy to the encoding end and the playing end, and the code stream adjustment strategy is used for adjusting the code stream.

In one embodiment, the determining the network congestion condition within the preset time duration based on the first information and the second information includes:

determining a first network status and/or a second network status based on the first information and the second information; the first network state is associated with video frame loss rate, and the second network state is associated with video frame delay;

determining the network congestion condition based on the first network status and/or the second network status.

In one embodiment, the video transmission information includes a video frame transmission number, and the video reception information includes a video frame reception number; determining a first network state based on the first information and the second information, comprising:

determining a video frame loss rate within the preset duration based on the video frame sending number and the video frame receiving number;

if the video frame loss rate is larger than a first threshold value, determining that the first network state is an overload state; or, if the video frame loss rate is a second threshold, determining that the first network state is a normal state; or, if the video frame loss rate is greater than or equal to a second threshold and less than a first threshold, determining that the first network state is a low-load state.

In one embodiment, the video transmission information comprises a video frame transmission interval and the video reception information comprises a video frame reception interval; determining a second network state based on the first information and the second information, comprising:

determining the change trend of the video frame time delay in the preset duration based on the video frame sending interval and the video frame receiving interval;

if the quantized value of the change trend of the video frame time delay is greater than a third threshold, and the overload times of the network bandwidth in the preset time length exceed a fourth threshold and/or the overload time length of the network bandwidth exceeds a fifth threshold, determining that the second network state is an overload state;

or if the quantized value of the change trend of the video frame time delay is smaller than a sixth threshold, determining that the second network state is a low-load state;

or if the quantized value of the change trend of the video frame delay is less than or equal to a third threshold and greater than or equal to a sixth threshold, and the normal times of the network bandwidth in the preset time length exceed a seventh threshold and/or the normal time length of the network bandwidth exceeds an eighth threshold, determining that the second network state is a normal state.

In one embodiment, the third threshold and the sixth threshold are dynamically adjusted according to an absolute value of a difference between a quantized value of the trend of change of the video frame delay and the third threshold or the sixth threshold.

In one embodiment, determining the network congestion condition based on the first network status and/or the second network status comprises:

if the first network state and/or the second network state is an overload state, determining that the network bandwidth within the preset time length is in the overload state;

if the first network state and the second network state are both normal states, determining that the network bandwidth within the preset time length is in a normal state;

and if the first network state is a normal state and the second network state is a low-load state, determining that the network bandwidth within the preset time length is in the low-load state.

In one embodiment, the video receiving information includes a video frame receiving interval, and the video playing buffer information includes a video frame buffer time; determining the video frame playing condition within the preset time length based on the second information, wherein the determining comprises:

determining frame playing jitter time within the preset duration according to the video frame receiving interval and the video frame caching time;

if the frame playing jitter time is less than or equal to a ninth threshold value, determining that the video playing is not blocked in the preset time length; or if the frame playing jitter time is greater than a ninth threshold, determining that the video playing is paused in the preset time length.

counting the frame playing jitter times within the preset duration according to the video frame receiving interval and the video frame caching time;

if the frame playing jitter frequency is less than or equal to a tenth threshold, determining that the video playing is not blocked within the preset time length; or if the frame playing jitter frequency is greater than a tenth threshold, determining that the video playing is paused in the preset time length.

In one embodiment, the receiving the network congestion condition and/or the video frame playing condition, and determining the code stream adjustment policy based on the network congestion condition and/or the video frame playing condition includes:

when the video playing is blocked and/or the network bandwidth is in an overload state within the preset time, the code stream adjusting strategy is used for indicating the playing end to adjust the code stream; alternatively, the first and second electrodes may be,

and when the video playing is not blocked and the network bandwidth is in a normal state within the preset time, the code stream adjusting strategy is used for indicating the playing end to adjust the code stream upwards.

In one embodiment, the platform end includes a video transmission monitoring module and a dynamic rate balancing module, wherein:

the encoding end is also used for sending the uplink network transmission bandwidth information to the platform end;

the playing end is also used for sending the downlink network transmission bandwidth information to the platform end;

the video transmission monitoring module is used for receiving the uplink network transmission bandwidth information and the downlink network transmission bandwidth information;

the video transmission monitoring module is further configured to determine a code stream adjustment strategy based on the uplink network transmission bandwidth information, the downlink network transmission bandwidth information, the network congestion condition and/or a video frame playing condition, and send the code stream adjustment strategy to the dynamic code rate balancing module;

if the code stream adjustment strategy is a down-regulation code stream, the dynamic code rate balance module is used for controlling the playing end to reduce the playing rate based on the code stream adjustment strategy and increasing the capacity of a receiving cache module of the playing end;

the dynamic code rate balancing module is also used for controlling the coding end to increase the capacity of the sending cache module and reduce the coding code rate.

In one embodiment, if the code stream adjustment policy is an up-regulation code stream and the dynamic code rate balancing module sends a code stream adjustment policy of an over-regulation code stream to the encoding end and the playing end, the dynamic code rate balancing module is further configured to determine a first delay time based on the downlink network transmission bandwidth information;

if the first delay time is greater than a preset threshold, the dynamic code rate balancing module is used for controlling the playing end to improve the playing speed based on the code stream adjustment strategy;

and if the first delay time is less than a preset threshold, the dynamic code rate balancing module is used for restoring the playing speed of the playing end, the capacity of the receiving cache module and the capacity of the sending cache module to initial states.

In one embodiment, the platform end comprises a video transmission monitoring module and a key frame peak staggering module, wherein:

the video transmission monitoring module is used for determining whether a key frame collision exists based on the first information and the second information;

if the key frame collision exists, determining a key frame peak staggering strategy, and sending the key frame peak staggering strategy to a key frame peak staggering module;

and the key frame peak staggering module sends the key frame peak staggering strategy to the encoding end and the playing end.

In one embodiment, if there is a key frame collision, the key frame peak staggering module is configured to control the playing end to reduce the playing rate and increase the capacity of a receiving buffer module of the playing end;

the key frame peak staggering module is also used for controlling the encoding end to increase the capacity of the sending buffer module and adjusting the key frame generation time sequence after the preset time.

In one embodiment, the playing end is further configured to send downlink network transmission bandwidth information to the platform end, and after the key frame peak staggering module sends the key frame peak staggering policy to the encoding end and the playing end, the key frame peak staggering module is further configured to determine a second delay time based on the downlink network transmission bandwidth information;

if the second delay time is greater than a preset threshold, the key frame peak staggering module is used for controlling the playing end to improve the playing speed;

and if the second delay time is less than a preset threshold, the key frame peak staggering module is used for restoring the playing speed of the playing end, the capacity of the receiving cache module and the capacity of the sending cache module to initial states.

The video transmission system comprises an encoding end, a platform end and a playing end, wherein: the encoding end is used for acquiring first information, the first information comprises video frame sending information of the encoding end within a preset time length, and the first information is sent to the playing end; the playing end is used for receiving the first information and acquiring second information, wherein the second information comprises video frame receiving information and/or video frame playing cache information of the playing end within the preset time length; the network congestion control module is further configured to determine a network congestion condition within the preset duration based on the first information and the second information; and/or determining the video frame playing condition in the preset time length based on the second information; and sending the network congestion condition and/or the video frame playing condition to the platform end; the platform end is used for receiving the network congestion condition and/or the video frame playing condition and determining a code stream adjustment strategy based on the network congestion condition and/or the video frame playing condition; the platform end is also used for sending the code stream adjustment strategy to the encoding end and the playing end, and the code stream adjustment strategy is used for adjusting the code stream.

In the embodiment of the application, a playing end determines the network congestion condition and/or the video frame playing condition within a preset time length by combining video frame receiving information and/or video frame playing cache information of the playing end per se within the preset time length and video frame sending information of an encoding end within the preset time length; the platform end determines a code stream adjustment strategy based on the video frame playing condition and/or the network congestion condition, and sends the code stream adjustment strategy to the encoding end and the playing end; and then the coding end and the playing end can adjust the code stream based on the code stream adjusting strategy. The code streams of the encoding end and the playing end are coordinated and controlled through the platform end, end-to-end matching of video encoding, video decoding and video transmission is guaranteed, and quality of video transmission and playing is improved.

Drawings

FIG. 1 is a diagram of a system architecture suitable for use with embodiments of the present application;

fig. 2 is a schematic flowchart of a video stream adjustment method according to an embodiment of the present application;

fig. 3 is a schematic flowchart of determining a first network status according to an embodiment of the present application;

fig. 4 is a schematic flowchart of determining a second network status according to an embodiment of the present application;

fig. 5 is a schematic diagram illustrating a principle of video frame interval delay congestion detection according to an embodiment of the present disclosure;

fig. 6 is a schematic flowchart of determining a network congestion situation by combining a first network status and a second network status according to an embodiment of the present application;

fig. 7 is a schematic flow chart of video stuck detection according to an embodiment of the present application;

fig. 8 is a schematic flowchart of determining a code stream adjustment policy according to an embodiment of the present application;

fig. 9 is a schematic diagram of a video transmission system according to an embodiment of the present application;

fig. 10 is a schematic diagram of an execution flow of a code stream adjustment policy provided in an embodiment of the present application;

fig. 11 is a flowchart illustrating a multi-stream adjusting method according to an embodiment of the present application;

FIG. 12 is a schematic diagram of a video stream provided by an embodiment of the present application;

FIG. 13 is a schematic diagram of a video stream provided by an embodiment of the present application;

fig. 14 is a flowchart illustrating a key frame peak shifting method according to an embodiment of the present application;

fig. 15 is a flowchart illustrating a multi-stream adjusting method according to an embodiment of the present application;

FIG. 16 is a schematic diagram of a video stream provided by an embodiment of the present application;

fig. 17 is a schematic diagram of a video stream provided in an embodiment of the present application.

Detailed Description

The present invention will be described in further detail with reference to the attached drawings, and it should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

First, a system architecture diagram applicable to the embodiments of the present application will be described with reference to the accompanying drawings.

Fig. 1 shows a system architecture diagram applicable to the embodiment of the present application, where the system includes a playing side, an encoding side and a platform side. The encoding end and the platform end can be connected through a wireless network, and then the encoding end can forward a video to the playing end through the platform end, so that an uplink network side is formed; the playing end and the platform end can be connected through a wired network or a wireless network, and then the platform end can send videos to the playing end, so that a downlink network side is formed.

It should be understood that the playing end may be a device with display capability, such as a mobile phone, a decoding wall, a smart television, a computer, and the like, and the embodiment of the present application is not particularly limited. The encoding end is a device with video encoding capability and code stream adjustment capability, such as a mobile monitoring device, and the embodiment of the present application is not particularly limited.

In the embodiment of the application, the encoding end is used for acquiring first information, the first information comprises video frame sending information of the encoding end within a preset time length, and the first information is sent to the playing end; the playing end is used for receiving the first information and acquiring second information, wherein the second information comprises video frame receiving information and/or video frame playing cache information of the playing end within a preset time length; the network congestion control module is further used for determining the network congestion condition within the preset time length based on the first information and the second information; and/or determining the video frame playing condition within the preset time length based on the second information; and sending the network congestion condition and/or the video frame playing condition to the platform end; the platform end is used for receiving the network congestion condition and/or the video frame playing condition and determining a code stream adjustment strategy based on the network congestion condition and/or the video frame playing condition; the platform end is also used for sending a code stream adjusting strategy to the encoding end and the playing end, and the code stream adjusting strategy is used for adjusting the code stream. Therefore, the code streams of the encoding end and the playing end are coordinated and controlled through the platform end, the end-to-end matching of video encoding, video decoding and video transmission is ensured, and the quality of video transmission and playing is improved.

Fig. 2 is a schematic flowchart of a video stream adjustment method according to an embodiment of the present invention, where an execution main body of the method takes a play end (i.e., the play end in fig. 1) and an encoding end (i.e., the encoding end in fig. 1) as an example, and the process includes the following steps:

s301: the encoding end transmits the first information. Correspondingly, the playing end receives the first information.

The first information comprises video frame sending information of the encoding end in a preset time length. In the embodiment of the present application, the video frame transmission information includes, but is not limited to, a video frame transmission number, a video frame encoding number, a video frame transmission time, a video frame encoding time, a frame type, a frame number, and a video frame transmission time.

Specifically, after acquiring the first information, the encoding end sends the first information to the platform end, and the platform end forwards the first information to the playing end; or after the encoding end acquires the first information, the first information is directly sent to the playing end through a transmission channel between the encoding end and the playing end.

S302: the playing end obtains the second information.

The second information comprises video frame receiving information and/or video frame playing cache information of the playing end within a preset time length. In the embodiment of the present application, the video frame receiving information includes, but is not limited to, a video frame receiving number, a video frame receiving time, a frame type, and a frame number, and the video frame playing buffering information includes, but is not limited to, a video frame buffering number, a video frame playing number, a frame type, and a frame number.

S303: the playing end determines the network congestion condition within a preset time length based on the first information and the second information; and/or determining the video frame playing condition within the preset time length based on the second information.

It should be noted that there are various implementations of determining the network congestion condition in the embodiment of the present application, including but not limited to the following ways:

in the embodiment 1, the playing end determines the first network state based on the first information and the second information; based on the first network state, a network congestion condition is determined.

In a possible implementation, the first network state may be associated with a video frame loss rate, and if the video transmission information includes a video frame transmission number, the video reception information includes a video frame reception number; the playing end can determine the frame loss rate of the video frames within the preset duration based on the sending number and the receiving number of the video frames; if the video frame loss rate is greater than a first threshold value, determining that the first network state is an overload state; or if the frame loss rate of the video frame is a second threshold value, determining that the first network state is a normal state; or if the video frame loss rate is greater than or equal to the second threshold and smaller than the first threshold, determining that the first network state is the low-load state. When the sending number of the video frames comprises the number of the encoding input video frames, the frame loss rate = (the number of the encoding input video frames-the number of the video receiving frames)/the number of the encoding input video frames in the preset time length.

It is understood that the "overload state" refers to a state in which the network bandwidth is insufficient, the "normal state" refers to a state in which the network bandwidth is normally used, and the "low load state" refers to a state in which the network bandwidth is loaded to a certain extent. Correspondingly, when the first network state is an overload state, the network congestion condition is that the network bandwidth is insufficient; when the first network state is a normal state, the network congestion condition is that the network bandwidth is sufficient; when the first network state is a low load state, the network congestion condition is that the network bandwidth is slightly insufficient.

For example, referring to fig. 3, the first threshold is 10%, the second threshold is 0, and the preset time period is 5 minutes. If the number of the encoded input video frames is 100 frames and the number of the received video frames is 50, determining that the first network state is an overload state, namely the network bandwidth is insufficient, if the frame loss rate of the video frames in 5 minutes = (the number of the encoded input video frames-the number of the received video frames)/the number of the encoded input video frames = (100-50)/100=50%, and the frame loss rate of the video frames is greater than 10%; or, if the number of the encoded input video frames is 100 frames and the number of the received video frames is 100, the frame loss rate of the video frames in 5 minutes = (the number of the encoded input video frames-the number of the received video frames)/the number of the encoded input video frames = (100-)/100 =0, and the frame loss rate of the video frames is 0, then it is determined that the first network state is a normal state, that is, the network bandwidth is sufficient; or, if the number of the encoded input video frames is 100 frames and the number of the received video frames is 95, the frame loss rate of the video frames in 5 minutes = (the number of the encoded input video frames-the number of the received video frames)/the number of the encoded input video frames = (100-95)/100=5%, and the frame loss rate of the video frames is less than 5% and greater than 0 and less than 10%, it is determined that the first network state is a low load state, that is, there is a partial load in the network bandwidth.

In embodiment 2, the encoding end determines the second network state based on the first information and the second information, and determines the network congestion condition based on the second network state.

Referring to fig. 4, in one possible embodiment, the second network status is associated with a video frame delay; the video sending information comprises video frame sending intervals, and the video receiving information comprises video frame receiving intervals; the process that the playing end determines the state of the second network based on the first information and the second information comprises the following steps:

s501: and determining the quantitative value of the change trend of the video frame time delay in the preset time length based on the video frame sending interval and the video frame receiving interval.

Further, determining a quantization value of a change trend of the video frame delay within a preset time length based on the video frame sending interval and the video frame receiving interval, includes:

s5011: a video frame interval delay gradient value is determined based on the video frame transmission interval and the video frame reception interval.

Illustratively, the video frame interval delay gradient value (delta _ ms) = video frame reception interval (recv _ delta _ ms) — video frame transmission interval send _ delta _ ms.

S5012: and determining the quantization value of the change trend of the video frame time delay according to the video frame interval time delay gradient value in the preset time length.

The detection principle based on the video frame group delay gradient congestion detection algorithm is shown in fig. 5, and the encoding end continuously sends the video frames to the playing end. Frame (i) represents the ith video Frame group, Frame (i-1) represents the ith-1 video Frame group, and Frame (i +1) represents the (i +1) th video Frame group, wherein each video Frame group comprises a plurality of video frames, for example, continuous video frames are divided into the 1 st group, the 2 nd group, … …, the i-1 st group and the ith group according to 31 groups. T (i) represents the arrival time of the last frame of the video frame group i, T (i-1) represents the arrival time of the last frame of the video frame group i-1, T (i) represents the transmission time of the last frame of the video frame group i, and T (i-1) represents the transmission time of the last frame of the video frame group i-1. The delay variation d (i) between two frame groups satisfies the following equation:

d(i)=（t(i)–t(i-1)）–(T(i)–T(i-1))；

wherein d (i) is 0, the network is not congested, d (i) is greater than 0, the network is congested, and d (i) is less than 0, the network congestion is recovering.

Furthermore, the playing end can combine a one-time exponential smooth prediction algorithm and a least square linear regression algorithm to accumulate, smooth and linearly regress the video frame interval delay gradient values within a preset time length to obtain a quantitative value of the change trend of the video frame time delay, and the congestion degree of the network is represented by the quantitative value of the change trend.

S502: and determining the second network state based on the quantized value of the variation trend of the video frame time delay.

The playing end can compare the quantized value of the change trend of the video frame time delay with a preset threshold value to determine a second network state; if the quantized value of the change trend of the video frame time delay is larger than a third threshold, and the overload times of the network bandwidth exceed a fourth threshold and/or the overload time of the network bandwidth exceed a fifth threshold in a preset time, determining that the second network state is an overload state; or if the quantized value of the change trend of the video frame time delay is smaller than a sixth threshold value, determining that the second network state is a low-load state; or if the quantized value of the change trend of the video frame time delay is less than or equal to a third threshold value and greater than or equal to a sixth threshold value, and the normal times of the network bandwidth in the preset time length exceed a seventh threshold value and/or the normal time length of the network bandwidth exceeds an eighth threshold value, determining that the second network state is a normal state.

Example 1, the preset time period is 5 minutes, the third threshold is 10%, the fourth threshold is 2 times, and the fifth threshold is 1 minute. For example, if the quantized value of the variation trend of the video frame delay is 15%, the overload time of the network bandwidth is 3 times in 5 minutes, and the overload duration of the network bandwidth is 4 minutes, the quantized value of the variation trend of the video frame delay is greater than a third threshold, and the overload time of the network bandwidth exceeds a fourth threshold and the overload duration of the network bandwidth exceeds a fifth threshold in 5 minutes, it is determined that the second network state is an overload state; for another example, if the quantized value of the change trend of the video frame delay is 15% and the number of times of overloading of the network bandwidth within 5 minutes is 3, the quantized value of the change trend of the video frame delay is greater than the third threshold, and the number of times of overloading of the network bandwidth within 5 minutes exceeds the fourth threshold, it is determined that the second network state is an overload state; for another example, if the quantized value of the variation trend of the video frame delay is 15% and the overload duration of the network bandwidth is 4 minutes, the quantized value of the variation trend of the video frame delay is greater than the third threshold, and the overload duration of the network bandwidth exceeds the fifth threshold within 5 minutes, it is determined that the second network status is an overload status.

Example 2, the preset time period is 5 minutes, and the sixth threshold is 5%. And if the quantized value of the change trend of the video frame time delay is 1%, determining that the second network state is the low load state if the quantized value of the change trend of the video frame time delay is smaller than a sixth threshold value.

Example 3, the preset time period is 5 minutes, the third threshold is 10% as an example, the fourth threshold is 2 times as an example, the fifth threshold is 1 minute as an example, the sixth threshold is 5% as an example, the seventh threshold is 50 times as an example, and the eighth threshold is 3 minutes as an example. For example, if the quantized value of the variation trend of the video frame delay is 8%, the normal times of the network bandwidth in 5 minutes is 55 times, and the normal duration of the network bandwidth is 4 minutes, the quantized value of the variation trend of the video frame delay is smaller than the third threshold and larger than the sixth threshold, and the normal times of the network bandwidth in 5 minutes exceeds the seventh threshold and/or the normal duration of the network bandwidth exceeds the eighth threshold, it is determined that the second network state is the normal state. For another example, if the quantized value of the variation trend of the video frame delay is 8% and the normal number of times of the network bandwidth in 5 minutes is 55 times, the quantized value of the variation trend of the video frame delay is smaller than the third threshold and larger than the sixth threshold, and the normal number of times of the network bandwidth in 5 minutes exceeds the seventh threshold, it is determined that the second network status is a normal status. For another example, if the quantized value of the variation trend of the video frame delay is 8% and the normal duration of the network bandwidth is 4 minutes, the quantized value of the variation trend of the video frame delay is smaller than the third threshold and larger than the sixth threshold, and the normal duration of the network bandwidth exceeds the eighth threshold within 5 minutes, it is determined that the second network status is the normal status.

Correspondingly, when the second network state is an overload state, the network congestion condition is that the network bandwidth is insufficient; when the second network state is a normal state, the network congestion condition is that the network bandwidth is sufficient; when the second network state is a low load state, the network congestion condition is that the network bandwidth is slightly insufficient.

Optionally, after the playing end determines the second network status, S503 may be further executed: and dynamically adjusting the third threshold and the sixth threshold according to the absolute value of the difference value between the quantized value of the change trend of the video frame time delay and the third threshold or the sixth threshold. Therefore, the network congestion detection method based on the video frame delay can be suitable for different application scenes.

Wherein, S503 specifically comprises the following steps:

s5031: and determining the absolute value trend _ deta of the difference value of the quantized value trend of the change trend of the video frame time delay change value and the dynamic threshold value threshold.

S5032: and determining the adjustment value of the dynamic threshold according to the absolute value trend _ deta of the difference.

Illustratively, the adjustment value of the dynamic threshold may be determined according to the following formula.

deta = k (trand _ deta-threshold _) deta _ t; where, deta is a threshold adjustment value, k is an adjustment factor, trend _ deta is an absolute value of a difference between a quantized value trend of a change trend of the video frame delay change value and a dynamic threshold, and deta _ t is a time difference of a last dynamic threshold update. Where deta _ t > =100 milliseconds.

The adjustment factor K is determined according to the absolute value of the difference value trend _ deta between the quantized value trend of the trend and the dynamic threshold value threshold. E.g., trend _ deta < threshold _, then the adjustment factor K = the up-adjustment factor Kup (i.e., increase K); trend _ deta > = threshold _, the adjustment factor K = down-regulation factor Kdown (i.e. decrease K).

S5033: and updating the dynamic threshold according to the adjustment value of the dynamic threshold.

It should be understood that the dynamic threshold refers to the third threshold and the sixth threshold described above.

In the embodiment 3, the playing terminal determines a first network state and a second network state based on the first information and the second information; a network congestion condition is determined based on the first network state and the second network state.

Referring to fig. 6, the process of determining the network congestion condition by the play end based on the first network status and the second network status may be: the method comprises the steps that a playing end obtains a first network state and a second network state, and when the first network state and/or the second network state are/is determined to be an overload state, the network bandwidth within a preset time length is determined to be in the overload state; or when the first network state and the second network state are both normal states, determining that the network bandwidth within the preset time length is in the normal state; or when the first network state is determined to be the low-load state and the second network state is determined to be the normal state, determining that the network bandwidth within the preset time length is in the low-load state. For a specific implementation of determining the first network status and the second network status, please refer to the description of implementation 1 and implementation 2, which is not described herein again.

For example, the preset time duration is 5 minutes, and when the first network state and/or the second network state is an overload state within 5 minutes, it is determined that the network bandwidth within 5 minutes is in the overload state; or when the first network state and the second network state are both normal states within 5 minutes, determining that the network bandwidth within 5 minutes is in a normal state; or when the first network state is determined to be the normal state and the second network state is the low-load state, determining that the network bandwidth within 5 minutes is in the low-load state.

It should be noted that, the playing end determines the playing condition of the video frame within the preset duration based on the second information, and there are various embodiments, including but not limited to the following embodiments:

in the embodiment 1, the video receiving information includes a video frame receiving interval, and the video playing buffer information includes a video frame buffer time; the playing end determines the video frame playing condition within the preset time length based on the second information, and the method comprises the following steps: determining frame playing jitter time within a preset duration according to a video frame receiving interval and video frame caching time; if the frame playing jitter time is less than or equal to a ninth threshold, determining that video playing is not blocked within a preset time length; or, if the frame playing jitter time is greater than the ninth threshold, determining that the video playing is paused in the preset time length. Wherein, frame playing jitter time (jitter time) = video frame buffer time (buffer time) -video frame receiving interval (interval).

Illustratively, the ninth threshold is 2s, the video frame buffer time (buffer time) is 1s, and the video frame reception interval (interval) is 0.6 s. If the frame playing jitter time (jitter time) = video frame buffer time (buffer time) -video frame receiving interval (interval) =0.4s, and if the frame playing jitter time is smaller than a ninth threshold, it is determined that video playing is not blocked within a preset time length.

In embodiment 2, the video receiving information includes a video frame receiving interval, and the video playing buffer information includes a video frame buffer time; the playing end determines the video frame playing condition within the preset time length based on the second information, and the method comprises the following steps: counting the frame playing jitter times within a preset duration according to the video frame receiving interval and the video frame caching time; if the frame playing jitter frequency is less than or equal to a tenth threshold, determining that the video playing is not blocked within a preset time length; or, if the frame playing jitter frequency is greater than the tenth threshold, determining that the video playing is paused in the preset time length.

Illustratively, the tenth threshold is 3 times, the frame playing jitter time within 5 minutes is not 4 times, and the frame playing jitter time is greater than the tenth threshold, it is determined that the video playing is paused within 5 minutes.

In embodiment 3, referring to fig. 7, after the playing end obtains the second information (i.e., the video frame buffer time (buffer time) and the video frame receiving interval (interval)), it determines the frame playing jitter time (jitter time), and if the frame playing jitter time (jitter time) is less than the preset threshold, it determines that the video playing is stuck; further, the frame playing jitter frequency in the preset duration is counted, and when the frame playing jitter frequency is greater than a tenth threshold, the video playing is considered to be paused in the preset duration. Or counting the frame playing jitter duration in the preset duration, considering that the video playing is paused in the preset duration only when the frame playing jitter duration is greater than the ninth threshold, and stopping streaming. Therefore, the video pause detection is more flexible.

S304: and the platform end determines a code stream adjustment strategy based on the video frame playing condition and/or the network congestion condition.

Referring to fig. 8, in a possible implementation, the process of determining the bitstream adjustment policy by the platform based on the video frame playing condition and/or the network congestion condition may be: acquiring a video frame playing condition and a network congestion condition, and when the video playing is blocked and/or the network bandwidth is in an overload state within a preset time length, indicating a code stream regulation strategy at a coding end to regulate the code stream; or when the video playing is not blocked and the network bandwidth is in a normal state within a preset time, the code stream adjustment strategy is used for indicating the encoding end to adjust the code stream upwards; or when the video playing is not blocked and the network bandwidth is in a low-load state within the preset time length, the code stream adjustment strategy indicates that the code stream is not changed. It can be understood that the embodiment in fig. 8 is only an example, and the bitstream adjustment policy in the present invention may be set based on actual scene requirements, and is not limited to the correspondence between the video playing situation and the network congestion situation and the bitstream adjustment policy.

S305: and the platform end sends a code stream adjustment strategy to the encoding end and the playing end. Correspondingly, the encoding end and the playing end receive the code stream adjustment strategy.

S306: and the coding end and the playing end adjust the code stream based on the code stream adjusting strategy.

Illustratively, the code stream adjustment policy indicates that the encoding end adjusts the code stream downward, and then the encoding end adjusts the code stream downward, for example, the code stream is decreased from 2M to 1M. If the code stream adjustment strategy indicates that the code stream is adjusted upwards at the encoding end, for example, the code stream is reduced from 1M to 2M.

In another embodiment, the platform comprises a video transmission monitoring module and a dynamic rate balancing module, wherein: the encoding end is also used for sending the uplink network transmission bandwidth information to the platform end; the playing end is also used for sending the downlink network transmission bandwidth information to the platform end; the video transmission monitoring module is used for receiving uplink network transmission bandwidth information and downlink network transmission bandwidth information; the video transmission monitoring module is also used for determining a code stream adjustment strategy based on the uplink network transmission bandwidth information, the downlink network transmission bandwidth information, the network congestion condition and/or the video frame playing condition, and sending the code stream adjustment strategy to the dynamic code rate balancing module; if the code stream adjustment strategy is the down-regulation code stream, the dynamic code rate balance module is used for controlling the playing end to reduce the playing rate based on the code stream adjustment strategy and increasing the capacity of a receiving cache module of the playing end; the dynamic code rate balancing module is also used for controlling the coding end to increase the capacity of the sending cache module and reduce the coding code rate.

Exemplarily, under wireless environments such as 5G/4G/Wi-Fi and the like, an encoding end such as a large-scale network camera is accessed to a platform end (including a cloud platform, NVR, EVS and the like) through a wireless network, and a video is stored through the platform end and forwarded to a playing end for real-time video playing. When the network is interfered by the environment, the problems of unmatched video encoding, video decoding and video transmission often occur, the large fluctuation of the network bandwidth occurs, the overall transmission efficiency of the video is low, the problems of pause in playing, missing of videos and the like easily occur, and the user video experience is poor.

Specifically, since the video needs to be forwarded by being accessed through the platform, the video encoding end cannot sense the downlink video transmission quality and the video playing quality of the user, and the video playing end cannot sense the uplink video transmission quality of the video encoding end, so that the video encoding end and the video decoding end are sensed and isolated, the video production, the video consumption and the video transmission are completely unmatched, and the strategy matching among the three is poor. When the network bandwidth fluctuates (for example, the downlink transmission network bandwidth is insufficient), the video encoding end cannot sense and still produces the video according to the original encoding rate, and the video transmission is not timely, so that the problems of playing blockage, screen splash and the like at the video decoding end can be caused.

Illustratively, a video transmission monitoring module and a dynamic code rate balancing module are arranged in the platform end in this embodiment, the video transmission monitoring module is configured to detect network transmission quality from the encoding end to the playing end, and the dynamic code rate balancing module is configured to perform code stream adjustment on the uplink transmission network and the downlink transmission network. Specifically, the video transmission detection module acquires uplink network transmission bandwidth information and downlink network transmission bandwidth information, determines video transmission and playing quality of the video transmission system based on the uplink network transmission bandwidth information, the downlink network transmission bandwidth information, network congestion information and a video frame playing condition, and further formulates a corresponding code stream adjustment strategy and sends the corresponding code stream adjustment strategy to the dynamic code stream balancing module. And the dynamic code stream balancing module determines whether the received code stream adjustment strategy is a down-regulation code stream, and if so, controls the playing end to reduce the playing rate, increases the capacity of a receiving cache module of the playing end, controls the coding end to increase the capacity of a sending cache module and reduces the coding code rate.

Referring to fig. 9, fig. 9 is a schematic diagram of a video transmission system according to an embodiment of the present application.

Illustratively, the encoding end and the playing end are accessed to the platform end through protocols such as streaming media access and transmission, so that the encoded code stream of the encoding end can be forwarded to the playing end through the platform end for decoding and playing. The method comprises the following specific steps:

step 1: and an uplink bandwidth detection module of the encoding end reports the current uplink network transmission bandwidth information and video frame sending information. The video frame sending information includes frame encoding time, frame type, frame sequence number, frame sending time consumption, and the like. Specifically, the uplink network transmission bandwidth information is fed back to the video transmission monitoring module at the platform end, and the video frame sending information is directly fed back to the playing end.

Step 2: the playing end detects and evaluates the playing quality and the network transmission quality of the current code stream respectively through a video decoding module and a feedback bandwidth evaluation module according to the video frame sending information, the video frame receiving information and the video playing cache information fed back by the coding end to obtain a video frame playing condition and a network congestion condition, and feeds back the video frame playing condition, the network congestion condition and the detected downlink network transmission bandwidth information to a video transmission monitoring module of the platform end.

And step 3: and a video transmission monitoring module at the platform end receives the uplink network transmission bandwidth information, the downlink network transmission bandwidth information, the network congestion condition and the video frame playing condition, and performs video traffic mean value overload detection based on the information. When the video transmission network can not bear the transmission of the current coding video, the video transmission monitoring module at the platform end appoints a code stream adjustment strategy and sends the code stream adjustment strategy to the dynamic code rate balancing module.

Specifically, the video transmission monitoring module at the platform end acquires the current playing smooth state of the playing end based on the video frame playing condition, and determines the current network bandwidth state from the system coding end to the playing end based on the uplink network transmission bandwidth information, the downlink network transmission bandwidth information and the network congestion condition. If the playing smooth state is monitored to be blocked or the network bandwidth state is monitored to be in an overload state, a code stream adjustment strategy of code stream down regulation is output to the dynamic code rate balancing module; when the smooth playing state is monitored to be smooth and the network bandwidth state is in a normal state, outputting a code stream adjustment strategy for adjusting the code stream to the dynamic code rate balancing module; and when the smooth playing state is monitored to be smooth and the network bandwidth state is in a low-load state, outputting a code stream adjustment strategy with unchanged code stream to the dynamic code rate balancing module, and adjusting the code stream by the dynamic code rate balancing module based on the code stream adjustment strategy. And after the flow is finished, determining whether the monitoring of the code stream needs to be stopped, and if so, ending the code stream adjusting process.

And 4, step 4: and the dynamic code flow balancing module receives the code flow adjusting strategy and adjusts the code flow and the cache of the coding end and the equipment end based on the code flow adjusting strategy. Specifically, if the code stream adjustment strategy is to down-regulate the code stream, the dynamic code rate balancing module is used for controlling the playing end to reduce the playing rate based on the code stream adjustment strategy and increasing the capacity of a receiving cache module of the playing end; the dynamic code rate balancing module is also used for controlling the coding end to increase the capacity of the sending cache module and reduce the coding code rate.

Referring to fig. 10, fig. 10 is a schematic diagram illustrating an execution flow of a code stream adjustment policy according to an embodiment of the present application.

In one embodiment, if the code stream adjustment policy is a down-regulation code stream, the step 4 specifically includes:

step 1: determining a smaller value Bi of the uplink network transmission bandwidth and the downlink network transmission bandwidth based on the uplink network transmission bandwidth information and the downlink network transmission bandwidth information;

step 2: the dynamic code rate balancing module executes a code stream adjustment strategy of code stream down-regulation, reduces the play rate of the video decoding module at the play end, and performs slow play at alpha times of the normal play rate, wherein the alpha value can be set based on the requirement of an actual scene, and the preferred alpha belongs to [0.8, 1 ];

and 3, step 3: the dynamic code rate balancing module increases the cache capacity of a receiving cache module of the playing end, for example, the received video delay data is introduced, and the cache capacity required to be increased is calculated through a flow peak balancing technology;

and 4, step 4: the dynamic code rate balancing module increases the cache capacity of a sending cache module at a coding end, for example, video delay data is introduced and the cache capacity required to be increased is calculated through a flow peak balancing technology;

and 5: after a certain time, the dynamic code flow balancing module informs the coding end to reduce the coding code rate, and controls the coding end to code with a new code rate Bt, wherein Bt = Bi x gamma, the value of gamma can be set based on the requirement of an actual scene, and the preferred gamma belongs to [0.5, 0.8 ].

In this embodiment, a video transmission monitoring module at a platform end determines a code stream adjustment strategy according to uplink network transmission bandwidth information, downlink network transmission bandwidth information, a network congestion condition and/or a video frame playing condition, and sends the code stream adjustment strategy to a dynamic code rate balancing module; the dynamic code flow balancing module controls the playing end to reduce the playing speed based on a code rate adjustment strategy, increases the capacity of a receiving cache module of the playing end, controls the encoding end to increase the capacity of a sending cache module and reduce the encoding code rate of the encoding end, balances the bandwidths of an uplink network and a downlink network by reducing the playing speed of the playing end and the encoding code flow of the encoding end, and dynamically stores video data by increasing the capacities of the receiving cache module and the sending cache module, so that the video encoding code rate is ensured, the video decoding playing is end-to-end matched with the video transmission, and data cannot be lost in the video transmission and video playing processes in the matching process, so that the stability of a video transmission system and the fluency of the video playing are improved, and further the user experience is improved.

In another embodiment, if the code stream adjustment policy is an up-regulation code stream and the dynamic code rate balancing module sends the code stream adjustment policy of an over-regulation code stream to the encoding end and the playing end, the dynamic code rate balancing module is further configured to determine a first delay time based on the downlink network transmission bandwidth information; if the first delay time is greater than a preset threshold value, the dynamic code rate balancing module is used for controlling the playing end to improve the playing speed based on a code stream adjustment strategy; if the first delay time is smaller than the preset threshold, the dynamic code rate balancing module is used for restoring the playing rate of the playing end, the capacity of the receiving cache module and the capacity of the sending cache module to the initial state.

Exemplarily, after the dynamic code rate balancing module sends the code stream adjustment policy of the over-and-under-regulation code stream to the encoding end and the playing end, if the code stream adjustment policy is the up-regulation code stream, the dynamic code rate balancing module continues to adjust the code stream and the cache of the encoding end and the equipment end based on the code stream adjustment policy, including: and determining a first delay time based on the downlink network transmission bandwidth information, wherein the first delay time is the video buffering delay time of a receiving cache module of the playing end. Comparing the first delay time with a preset threshold, if the first delay time is greater than the preset threshold, indicating that the video buffering delay is overlarge, and controlling a playing end to increase the playing speed through a dynamic code stream balancing module so as to reduce the video buffering delay time; when the first delay time is less than the preset threshold, it indicates that the video buffering delay is normal, and at this time, the playing rate of the playing end, the capacity of the receiving buffer module and the capacity of the sending buffer module can be recovered to be normal.

In one embodiment, as shown in fig. 10, the above steps include:

step 1: the dynamic code rate balancing module calculates video buffering delay Td (first delay time) of a receiving buffer module of a playing end based on the downlink network transmission bandwidth information, and compares the video buffering delay Td with a preset threshold Ts2 to determine whether the video buffering delay is too large;

step 2: if the video buffering delay Td is greater than a preset threshold Ts2, the video buffering delay is too large, the dynamic code rate balancing module needs to increase the playing speed of a video decoding module at a playing end and plays the video at a speed which is beta times of the normal playing speed, the beta value can be set based on the requirement of an actual scene, and the optimal beta belongs to [1,1.2 ];

and step 3: if the video buffering delay Td is smaller than a preset threshold Ts2, the video buffering delay is normal, and the dynamic code rate balancing module recovers the playing speed of the video decoding module at the playing end;

and 4, step 4: if the cache of the receiving cache module of the playing end is not restored to the default value, gradually reducing the cache of the receiving cache module;

and 5: and if the cache of the sending cache module of the encoding end is not restored to the default value, gradually reducing the cache of the sending cache module.

Referring to fig. 11, fig. 11 is a flowchart illustrating a multi-stream adjusting method according to an embodiment of the present disclosure.

Optionally, if there are multiple code streams of multiple encoding end devices that need to perform code rate balancing processing, a code stream that preferentially performs code rate balancing operation needs to be determined, and the specific steps are as follows:

step 1: the platform end determines the number M of code streams with overload of the current video flow mean value;

step 2: if the number M of the video flow mean value overloaded code streams exceeds 1, selecting the code stream with the minimum ratio of the uplink network transmission bandwidth to the coding code rate from the M code streams, for example, the uplink network transmission bandwidth of the code stream 1 is 10Mbps, the video coding code rate is 5Mbps, the ratio is 2, the uplink network transmission bandwidth of the code stream 2 is 9Mbps, the video coding code rate is 3Mbps, and the ratio is 3, selecting the code stream 1 to preferentially perform code rate balance operation; or selecting a code stream with higher priority to perform code rate balance operation based on a flow peak balance technology;

and step 3: and carrying out code rate adjustment on the selected code stream according to the code rate balance processing flow of the single code stream.

In this embodiment, if the bit stream adjustment policy is an up-regulation bit stream and the dynamic bit rate balancing module sends the bit stream adjustment policy of an over-regulation bit stream to the encoding end and the playing end, whether to recover the playing rate of the playing end, the capacity of the receiving cache module, and the capacity of the sending cache module is determined by the first delay time, so that the video transmission quality and the playing quality are recovered in time under the condition that the video encoding bit rate, the video decoding playing, and the video transmission are end-to-end matched, and the viewing experience of a user is improved.

In another embodiment, the platform comprises a video transmission monitoring module and a key frame peak shifting module, wherein: the video transmission monitoring module is used for determining whether key frame collision exists or not based on the first information and the second information; if the key frame collision exists, determining a key frame peak staggering strategy, and sending the key frame peak staggering strategy to a key frame peak staggering module; and the key frame peak staggering module sends a key frame peak staggering strategy to the encoding end and the playing end.

Referring to fig. 12-13, fig. 12-13 are schematic diagrams of video streams according to an embodiment of the present application.

Illustratively, the abscissa in fig. 12-13 represents time in milliseconds (ms) and the ordinate represents codestream in bytes per millisecond (KByte/ms). The encoding end transmits video frames in real time at a certain frequency and period, as shown in fig. 12, wherein the amount of video key I frames is much larger (usually more than 10 times) than that of normal P frame data. Traffic characteristics in the network take FPS =25, GOP =50 video as an example, and a video frame is generated every 40ms, and a key I frame is generated every 25 × 50 × 40=2000 ms. When a plurality of encoding terminals simultaneously transmit video frames, the more encoding terminal devices, the more likely a plurality of encoding terminal devices transmit I frames at the same time point (I frame collision). When an I-frame collision occurs, a large amount of data bursts in a short time often occur in a network, thereby causing data congestion, picture delay increase, and blocking, which is particularly obvious in a 5G/4G wireless network environment. As shown in fig. 13, fig. 13 is a schematic diagram of a video stream when two encoding end devices collide with each other.

Illustratively, the platform end further comprises a video transmission monitoring module and a key frame peak shifting module, wherein the video transmission monitoring module determines the time sequence distribution condition of the key frames based on the video frame sending information in the first information and the video frame receiving information in the second information, and further determines whether key frame collision exists. And if the key frame collision exists, the video transmission monitoring module determines a key frame peak error strategy based on the key frame collision condition and sends the key frame peak error strategy to the key frame peak error module. And after the key frame peak error module receives the key frame peak error strategy, the key frame peak error module controls the encoding end and the playing end to execute the peak error strategy.

In one specific embodiment, before performing the key frame collision detection, the first information and the second information are already stored in the cache of the platform end for detection and analysis, for example, the platform end stores the first information and the second information into the cache while forwarding the first information and the second information, or directly sends the first information and the second information to the platform end after acquiring the first information and the second information respectively at the encoding end and the playing end for storing the platform end; in another embodiment, when performing the key frame collision detection, the platform end requests the encoding end and the playing end to send the first information and the second information in real time for detection and analysis.

Optionally, in the key frame peak shifting process, the dynamic buffer balancing module at the platform end controls the decoding adjusting module at the playing end in advance (adjusts the capacity of the receiving buffer module and the playing rate of the video decoding module), and controls the encoding adjusting module at the encoding end in advance (adjusts the capacity of the sending buffer module), so that the playing end can smoothly play the video in the key frame peak shifting process.

In this embodiment, the video transmission monitoring module determines whether a key frame collision exists based on the first information and the second information, determines a key frame peak staggering strategy when the key frame collision exists, and controls the encoding end and the playing end to carry out peak staggering based on the key frame peak staggering strategy by the key frame peak staggering module, so that a flow peak value from the encoding end to the playing end is reduced, bandwidths of an uplink network and a downlink network are balanced, stability of flow from the encoding end to the playing end is ensured, the utilization rate of the bandwidths is improved, fluency of video transmission and video playing is not affected on the premise of not losing video data in an adjustment process, a picture clip segment when the key frame peak is staggered is avoided, and quality of video transmission and playing is ensured.

In another embodiment, if there is a key frame collision, the key frame peak staggering module is used for controlling the playing end to reduce the playing speed and increasing the capacity of the receiving buffer module of the playing end; the key frame peak staggering module is also used for controlling the encoding end to increase the capacity of the sending and caching module and adjusting the key frame generation time sequence after the preset time.

Exemplarily, in the case of key frame collision, the key frame peak staggering module controls the encoding end and the playing end to execute a key frame peak staggering strategy, which specifically includes: controlling the playing end to reduce the playing speed, increasing the capacity of a receiving buffer module of the playing end, controlling the encoding end to increase the capacity of a sending buffer module and adjusting the key frame generation time sequence after the preset time.

Referring to fig. 14, fig. 14 is a flowchart illustrating a key frame peak shifting method according to an embodiment of the present disclosure.

In one embodiment, the key frame peak shifting method comprises:

step 1: the method comprises the steps that a playing end sends downlink network transmission bandwidth information to a platform end, a platform end video transmission monitoring module carries out I-frame flow peak value superposition detection, the receiving time interval Ti of an I frame of a current code stream and the I frame of a previous code stream is determined, and whether I-frame flow peak value superposition occurs is determined by judging the receiving time interval Ti and a preset threshold Ts 1;

step 2: if the receiving time interval Ti is smaller than a preset threshold Ts1, indicating that I frame flow peak value superposition occurs, reducing the playing speed of a video decoding module at a playing end, and playing at alpha times of the normal playing speed, wherein the preferred alpha belongs to [0.8, 1], and the slow-down playing at the moment is not obvious to human eyes;

and 3, step 3: the key frame peak staggering module increases the capacity of a receiving cache module of a playing end, and calculates the size of the cache capacity needing to be increased by introducing the video receiving delay data;

and 4, step 4: the key frame peak staggering module increases the capacity of a sending cache module at the encoding end, and calculates the cache capacity required to be increased by introducing video sending delay data;

and 5: after waiting for the preset time, the key frame peak staggering module informs the encoding end to adjust the I frame generation time sequence of the current code stream, requires the encoding end to regenerate the I frame at the current moment, and performs video encoding according to the appointed video GOP period.

In the embodiment, the key frame peak staggering module is used for controlling the encoding end and the playing end to carry out key frame peak staggering, the playing rate of the playing end, the capacity of the receiving cache module and the like are adjusted in the peak staggering process, so that the end-to-end matching of video encoding code rate, video decoding playing and video transmission is ensured, and the stability of video playing of the video end in the key frame peak staggering process is ensured.

In another embodiment, the playing end is further configured to send downlink network transmission bandwidth information to the platform end, and after the key frame peak staggering module sends the key frame peak staggering policy to the encoding end and the playing end, the key frame peak staggering module is further configured to determine a second delay time based on the downlink network transmission bandwidth information; if the second delay time is greater than the preset threshold, the key frame peak staggering module is used for controlling the playing end to improve the playing speed; and if the second delay time is less than the preset threshold, the key frame peak staggering module is used for restoring the playing speed of the playing end, the capacity of the receiving cache module and the capacity of the sending cache module to the initial state.

Exemplarily, after the key frame peak staggering module sends the key frame peak staggering strategy to the encoder and the playing end and executes the peak staggering strategy, the playing end sends downlink network transmission bandwidth information to the platform end, and the platform end determines the video buffering delay, namely the second delay time, of the playing end receiving the caching module based on the downlink network transmission bandwidth information; comparing the second delay time with a preset threshold, if the second delay time is greater than the preset threshold, the video buffering delay is over-high, and at the moment, the key frame peak staggering module controls the playing end to improve the playing speed so as to reduce the video buffering delay; if the video buffering delay time is smaller than the preset threshold value, the video buffering delay time of the playing end is normal, and the playing speed of the playing end, the capacity of the receiving cache module and the capacity of the sending cache module are restored to the initial state.

In one embodiment, as shown in fig. 14, after the I-frame traffic peak staggering is completed, the key frame peak staggering method further includes:

step 1: the key frame peak staggering module calculates video buffering delay Td (second delay time) of the receiving and caching module of the playing end, and compares the video buffering delay Td with a preset threshold Ts 2;

step 2: if the video buffering delay Td is greater than a preset threshold Ts2, the video buffering delay Td indicates that the video delay is large, the playing speed of a video decoding module at a playing end needs to be increased, and the video decoding module is played at a speed beta times of a normal speed, wherein the preferable beta is within the range of [1,1.2], and at the moment, the accelerated playing is not obvious to human eyes;

and step 3: if the video buffering delay Td is smaller than the preset threshold Ts2, it indicates that the video delay is normal, and the playing rate of the playing end, the capacity of the receiving buffer module and the capacity of the sending buffer module are respectively recovered.

Referring to fig. 15, fig. 15 is a flowchart illustrating a multi-stream adjusting method according to an embodiment of the present disclosure.

Optionally, if there are multiple code streams to be subjected to I-frame traffic peak balancing processing, a code stream to be subjected to balancing processing preferentially needs to be determined first. The method comprises the following specific steps:

step 1: the platform end determines the number N of code streams with superposed I frame flow peak values;

step 2: for the code stream number N of the I frame flow peak value superposition exceeds 1, selecting a code stream with a code stream adjustment strategy evaluated as code stream down-regulation from a plurality of code streams of the I frame peak value superposition as a first priority adjustment code stream, using a code stream with a code stream adjustment strategy evaluated as code stream invariance as a second priority adjustment code stream, and using a code stream with a code stream adjustment strategy evaluated as code stream up-regulation as a third priority adjustment code stream; if a plurality of code streams are required to be adjusted in each priority level, selecting the code stream with smaller ratio of the uplink network transmission bandwidth to the coding code rate as the code stream which is preferentially adjusted; for example, the first preferentially adjusted code stream includes a code stream 1 and a code stream 2, the uplink network transmission bandwidth of the code stream 1 is 10Mbps, the coding code rate is 5Mbps, the ratio is 2, the uplink network transmission bandwidth of the code stream 2 is 9Mbps, the coding code rate is 3Mbps, and the ratio is 3, then the code stream 1 is selected for preferential adjustment;

and step 3: and carrying out I frame time sequence adjustment on the selected code stream according to the I frame flow peak value balance processing flow of the single code stream.

Referring to fig. 16-17, fig. 16-17 are schematic views of video streams according to embodiments of the present disclosure.

As shown in fig. 16 to 17, the abscissa represents time in seconds(s) and the ordinate represents a code stream in bit flip amount/100 milliseconds (Bits/100 ms). The encoding end comprises 8 encoding end devices. By the key frame peak staggering adjustment method in the embodiment, the flow peak value 94Mbps when the I frame collides is reduced to the flow peak value 53Mbps after peak staggering, and the flow peak value is reduced by 43.6%.

After the key frame peak staggering strategy is executed, the playing speed of the playing end, the capacity of the receiving cache module and the capacity of the sending cache module are adjusted through the second delay time, so that the flow peak value when the key frames are overlapped is reduced, the situation of network congestion is relieved, and the bandwidth utilization rate is further improved.

It should be understood that, although the steps in the flowcharts related to the embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the above embodiments may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above may be implemented by hardware that is instructed by a computer program, and the computer program may be stored in a non-volatile computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A video transmission system comprises an encoding end, a platform end and a playing end, and is characterized in that:

2. The system of claim 1, wherein the determining the network congestion condition within the preset time period based on the first information and the second information comprises:

3. The system of claim 2, wherein the video transmission information includes a video frame transmission number and the video reception information includes a video frame reception number; determining a first network state based on the first information and the second information, comprising:

determining a video frame loss rate in the preset duration based on the sending number of the video frames and the receiving number of the video frames;

if the video frame loss rate is greater than a first threshold value, determining that the first network state is an overload state; or if the frame loss rate of the video frame is a second threshold value, determining that the first network state is a normal state; or, if the video frame loss rate is greater than or equal to a second threshold and less than a first threshold, determining that the first network state is a low-load state.

4. The system of claim 2, wherein the video transmission information comprises a video frame transmission interval and the video reception information comprises a video frame reception interval; determining a second network state based on the first information and the second information, comprising:

5. The system of claim 4, wherein the third threshold and the sixth threshold are dynamically adjusted according to an absolute value of a difference between a quantized value of the trend of change of the video frame delay and the third threshold or the sixth threshold.

6. The system of claim 2, wherein determining the network congestion condition based on the first network status and/or the second network status comprises:

7. The system according to any one of claims 1-6, wherein the video reception information comprises a video frame reception interval, and the video playback buffer information comprises a video frame buffer time; determining the video frame playing condition in the preset time length based on the second information, wherein the determining comprises the following steps:

if the frame playing jitter time is less than or equal to a ninth threshold, determining that video playing is not blocked within the preset time length; or if the frame playing jitter time is greater than a ninth threshold, determining that the video playing is paused in the preset time length.

8. The system according to any one of claims 1-6, wherein the video reception information comprises a video frame reception interval, and the video playback buffer information comprises a video frame buffer time; determining the video frame playing condition within the preset time length based on the second information, wherein the determining comprises:

9. The system according to any one of claims 1-6, wherein receiving the network congestion condition and/or the video frame play condition, and determining a codestream adjustment policy based on the network congestion condition and/or the video frame play condition comprises:

10. The system according to any one of claims 1-6, wherein the platform side comprises a video transmission monitoring module and a dynamic rate balancing module, wherein:

if the code stream adjustment strategy is a down-regulation code stream, the dynamic code rate balancing module is used for controlling the playing end to reduce the playing speed based on the code stream adjustment strategy and increasing the capacity of a receiving cache module of the playing end;

11. The system of claim 10, wherein if the stream adjustment policy is an up-modulation stream and the dynamic rate balancing module sends a stream adjustment policy that the stream is down-modulated to the encoding end and the playing end, the dynamic rate balancing module is further configured to determine a first delay time based on the downlink network transmission bandwidth information;

if the first delay time is longer than a preset threshold value, the dynamic code rate balancing module is used for controlling the playing end to improve the playing speed based on the code stream adjustment strategy;

12. The system of claim 1, wherein the platform side comprises a video transmission monitoring module and a key frame peak staggering module, wherein:

the video transmission monitoring module is used for determining whether key frame collision exists or not based on the first information and the second information;

13. The system according to claim 12, wherein if there is a key frame collision, the key frame peak staggering module is configured to control the playing end to reduce the playing rate and increase the capacity of the receiving buffer module of the playing end;

14. The system according to claim 13, wherein the playing side is further configured to send downlink network transmission bandwidth information to the platform side, and after the key frame peak staggering module sends the key frame peak staggering policy to the encoding side and the playing side, the key frame peak staggering module is further configured to determine a second delay time based on the downlink network transmission bandwidth information;