CN116633871A

CN116633871A - Method, device, storage medium and system for real-time transmission of multilink data stream

Info

Publication number: CN116633871A
Application number: CN202310522143.8A
Authority: CN
Inventors: 崔涛; 张鹏洲; 郭翊; 王国夫
Original assignee: Beijing International Cloud Broadcasting Technology Co ltd
Current assignee: Beijing International Cloud Broadcasting Technology Co ltd
Priority date: 2023-05-10
Filing date: 2023-05-10
Publication date: 2023-08-22

Abstract

The disclosure relates to a real-time transmission method, a real-time transmission device, a storage medium and a real-time transmission system for a multi-link data stream. The method for real-time transmission of the multilink data stream comprises the steps of monitoring stable transmission bandwidth, real-time transmission bandwidth, limited transmission state and link delay time difference when a plurality of data transmission links perform data transmission; determining bandwidth proportion weights of the data transmission links when the data transmission links transmit data based on stable transmission bandwidths, real-time transmission bandwidths, transmission limited states and link delay time differences of the data transmission links; based on the bandwidth proportion weight, distributing the data quantity of each data transmission link when data transmission is carried out on a plurality of data transmission links; based on the data quantity distributed to each data transmission link, the data is transmitted in real time, which is beneficial to accurately and reasonably distributing the data quantity of each data transmission link and reducing the delay time during data transmission.

Description

Method, device, storage medium and system for real-time transmission of multilink data stream

Technical Field

The disclosure relates to the technical field of communication, and in particular relates to a real-time transmission method, a transmission device, a storage medium and a real-time transmission system for a multi-link data stream.

Background

With the development of mobile communication, in particular, the popularization of 5G and Wifi 6. The nominal data transmission bandwidth is increasingly larger. Therefore, in the field of audio and video acquisition, especially in the audio and video acquisition and transmission work of a mobile mode, the situation of using a mobile network is more and more increasing. But is limited by the technology of mobile communication itself, and is also limited by the amount of interference of mobile users and other wireless signals in the area, the mobile communication itself is not a stable communication network, and includes unstable bandwidth performance and delay variation. Therefore, a scheme of adopting a plurality of groups of wireless network cards or wireless modules to simultaneously transmit one path of real-time audio and video is presented. Although the overall bandwidth becomes large, there is still a problem of long delay time, and particularly, the problem of long delay time in a weak network environment is obvious.

Disclosure of Invention

In view of this, it is desirable for embodiments of the present disclosure to provide a method for real-time transmission of a multi-link data stream, a device for real-time transmission of a multi-link data stream, a storage medium, and a system for real-time transmission of a multi-link data stream.

The technical scheme of the present disclosure is realized as follows:

in a first aspect, the present disclosure provides a method for real-time transmission of a multilink data stream.

The method for transmitting the multilink data stream in real time provided by the embodiment of the disclosure comprises the following steps:

monitoring stable transmission bandwidth, real-time transmission bandwidth, limited transmission state and link delay time difference when a plurality of data transmission links perform data transmission;

determining bandwidth proportion weights of the plurality of data transmission links when the data transmission is performed based on the stable transmission bandwidths, the real-time transmission bandwidths, the transmission limited states and the link delay time differences of the plurality of data transmission links when the data transmission is performed;

based on the bandwidth proportion weight when the data transmission is carried out by the data transmission links, distributing the data quantity of each data transmission link when the data transmission is carried out on the data transmission links;

based on the amount of data allocated to each data transmission link, data is transmitted in real time.

In some embodiments, the determining the bandwidth proportion weight of the plurality of data transmission links when transmitting data based on the stable transmission bandwidth, the real-time transmission bandwidth, the transmission limited state and the link delay time difference when transmitting data includes:

Determining whether the plurality of data transmission links are in the transmission limited state when data is transmitted;

if all the data transmission links are not in the transmission limited state, determining the link delay time difference when the data transmission links transmit data;

adjusting the stable transmission bandwidth of the data transmission links when the data transmission links transmit data based on the link delay time difference when the data transmission links transmit data;

based on the regulated stable transmission bandwidth, determining the corresponding bandwidth proportion weight when each data transmission link carries out data transmission; or, determining the corresponding bandwidth proportion weight when each data transmission link performs data transmission based on the difference value between the regulated stable transmission bandwidth and the real-time transmission bandwidth;

and if at least one data transmission link is in the transmission limited state when the plurality of data transmission links perform data transmission, determining the corresponding bandwidth proportion weight to be 0 when the data transmission link in the transmission limited state performs data transmission.

In some embodiments, the determining whether the plurality of data transmission links are in the transmission limited state when transmitting data includes:

If at least one of the following situations occurs when the data transmission link performs data transmission, determining that the data transmission link is in the transmission limited state when the data transmission link performs data transmission;

wherein, the transmission limited state condition includes:

the data transmission link is disconnected, the time for the data receiving end to receive the data packet exceeds a preset time length, and the round trip delay of the data transmission link exceeds a preset value.

In some embodiments, the adjusting the stable transmission bandwidth of the plurality of data transmission links when transmitting data based on the link delay time difference when transmitting data by the plurality of data transmission links includes:

if the link delay time difference when the data transmission link performs data transmission is greater than 0, reducing the stable transmission bandwidth when the data transmission link performs data transmission;

if the link delay time difference is equal to 0 when the data transmission link performs data transmission, keeping the stable transmission bandwidth unchanged when the data transmission link performs data transmission;

and if the link delay time difference when the data transmission link performs data transmission is smaller than 0, increasing the stable transmission bandwidth when the data transmission link performs data transmission.

In some embodiments, the allocating the data amount of each data transmission link when the data is transmitted on the plurality of data transmission links based on the bandwidth proportion weight when the data is transmitted on the plurality of data transmission links includes:

sequencing and numbering the data to be transmitted;

distributing the data packets with the sequence numbers to the plurality of data transmission links according to the bandwidth proportion weight for data transmission; wherein the bandwidth proportion weight determines the amount of data allocated to the data transmission link for data transmission.

In some embodiments, the allocating the data packets with the sequence numbers to the plurality of data transmission links according to the bandwidth proportion weight for data transmission includes:

and sequentially distributing the data packets with the sequencing numbers to the plurality of data transmission links for data transmission according to the sequencing of the bandwidth proportion weight from large to small.

In some embodiments, the transmitting data in real time based on the data amount allocated to each data transmission link includes:

when a data receiving end receives a data packet, if the data packet with the sequence number of N+1 is received and the data packet with the sequence number of N is not received after exceeding a preset time length, determining that the data packet with the sequence number of N is lost;

The data receiving end feeds back the packet loss information of the data packet with the sequence number N to the data sending end, and sends a key frame retransmission signaling to the data sending end;

and the data transmitting end transmits the data packet with the sequence number N of the lost packet to the data receiving end according to the key frame retransmission signaling.

In some embodiments, the sending, by the data sending end, the data packet with the sequence number N of the packet loss to the data receiving end according to the key frame retransmission signaling includes:

and when the data transmitting end transmits the data packet with the sequence number N of the lost packet to the data receiving end according to the key frame retransmission signaling, the data transmitting end selects the data transmission link with the largest bandwidth proportion weight or the data transmission link with the smallest round trip delay to transmit data.

In some embodiments, the monitoring the link delay time difference when the plurality of data transmission links perform data transmission includes:

acquiring a receiving time point of a plurality of continuous data packets received by a data receiving end in a preset time and a transmitting time point of the plurality of continuous data packets at a data transmitting end;

according to the receiving time points of the data receiving end for receiving a plurality of continuous data packets, obtaining the difference value of the receiving time points of the data receiving end for receiving any two adjacent data packets;

Obtaining the difference value of the sending time points of any two adjacent data packets according to the sending time points of the plurality of continuous data packets at the data sending end;

according to the receiving time point difference value of any two adjacent data packets and the sending time point difference value of any two adjacent data packets, obtaining the delay time difference of any two adjacent data packets when the data transmission link carries out data transmission;

based on delay time difference of a plurality of adjacent data packets when the data transmission link carries out data transmission, carrying out primary exponential smoothing method prediction formula fitting to obtain a smooth fitting curve;

determining the slope of the smooth fitting curve based on a least square method;

and determining the slope of the smooth fitting curve as a link delay time difference when the data transmission link transmits data in the preset time.

In some embodiments, the monitoring the stable transmission bandwidth of the data transmission of the plurality of data transmission links includes:

obtaining the transmission bandwidth of a data transmission link according to the data length and the transmission time length of data transmitted by a data transmission end when the data transmission link transmits data in a preset time;

According to the data length and the receiving time length received by the data receiving end when the data transmission link carries out data transmission in the preset time, the receiving bandwidth of the data transmission link is obtained;

and determining the minimum value of the sending bandwidth and the receiving bandwidth as the stable transmission bandwidth when the data transmission link performs data transmission in a preset time.

In some embodiments, the monitoring the real-time transmission bandwidth of the plurality of data transmission links during data transmission includes:

based on a Gaussian conjugate prior model, determining the real-time transmission bandwidth at the time t through prior estimation, prior variance, posterior estimation and posterior variance; wherein, the liquid crystal display device comprises a liquid crystal display device,

the method for determining the real-time transmission bandwidth at the time t based on the Gaussian conjugate prior model through prior estimation, prior variance, posterior estimation and posterior variance comprises the following steps:

acquiring the data quantity transmitted in unit time when the data transmission link performs data transmission, and obtaining a first transmission bandwidth serving as the posterior estimation for determining the real-time transmission bandwidth;

based on the posterior estimation and the prior estimation, obtaining a mean square error of the posterior estimation and the prior estimation as the posterior variance for determining the real-time transmission bandwidth;

Acquiring a real-time transmission bandwidth determined based on a Gaussian conjugate prior model at a time t-1 as the prior estimate for determining the real-time transmission bandwidth at the time t;

and dividing the product of the prior variance of the real-time transmission bandwidth and the posterior variance of the real-time transmission bandwidth, which is used for determining the real-time transmission bandwidth, at the time t, by the sum of the prior variance of the real-time transmission bandwidth, which is used for determining the real-time transmission bandwidth at the time t-1, so as to obtain the prior variance of the real-time transmission bandwidth at the time t.

In some embodiments, when the data receiving end receives the data packets, if the number of the data packets which are lost is multiple, the data receiving end feeds back the packet loss information of the multiple data packets to the data sending end;

wherein, the packet loss information comprises the serial number of the packet loss data packet;

the feeding back packet loss information of a plurality of data packets to the data sending end comprises:

if the data packets with continuous sequence numbers exist in the data packets with continuous sequence numbers, feeding back the starting sequence numbers and the ending sequence numbers of the data packets with continuous sequence numbers when the data packets with continuous sequence numbers are fed back with the packet loss information.

If the total bandwidth of the real-time transmission bandwidths of the monitored data transmission links is smaller than the target bandwidth, the coding format of the video to be transmitted in video streaming transmission is adjusted, and the data volume of the video to be transmitted is reduced.

In a second aspect, the present disclosure provides a real-time transmission device for a multi-link data stream, including:

the link monitoring module is used for monitoring stable transmission bandwidth, real-time transmission bandwidth, limited transmission state and link delay time difference when the data transmission is carried out by the plurality of data transmission links;

the bandwidth proportion weight determining module is used for determining bandwidth proportion weights when the data transmission links transmit data based on the stable transmission bandwidth, the real-time transmission bandwidth, the transmission limited state and the link delay time difference when the data transmission links transmit data;

the data quantity distribution module is used for distributing the data quantity of each data transmission link when the data transmission is carried out on the plurality of data transmission links based on the bandwidth proportion weight when the data transmission is carried out on the plurality of data transmission links;

and the data transmission module is used for carrying out data real-time transmission based on the data quantity distributed to each data transmission link.

In some embodiments, the bandwidth proportion weight determining module is configured to

wherein, the transmission limited state condition includes:

In some embodiments, the data volume allocation module is configured to

Sequencing and numbering the data to be transmitted;

In some embodiments, the data volume allocation module is configured to

In some embodiments, the data transmission module is configured to

In some embodiments, the link monitoring module is configured to

In some embodiments, a data transmission module is used for

When a data receiving end receives data packets, if a plurality of data packets are lost, feeding back packet loss information of the plurality of data packets to the data transmitting end;

In some embodiments, the data transmission module is configured to

In a third aspect, the present disclosure provides a computer readable storage medium having stored thereon a multilink data stream real-time transmission program, which when executed by a processor, implements the multilink data stream real-time transmission method of the first aspect.

In a fourth aspect, the present disclosure provides a multi-link data stream real-time transmission system, including:

the data sending end is used for sending the video stream of the video to be transmitted in real time;

the data receiving end is used for receiving the video stream sent by the data sending end;

A plurality of data transmission links connected between the data transmitting end and the data receiving end;

the system comprises a memory, a processor and a multi-link data stream real-time transmission program stored in the memory and capable of running on the processor, wherein the processor realizes the multi-link data stream real-time transmission method according to the first aspect when executing the multi-link data stream real-time transmission program.

The method for transmitting the multi-link data stream in real time according to the embodiment of the disclosure comprises the steps of monitoring stable transmission bandwidth, real-time transmission bandwidth, transmission limited state and link delay time difference when a plurality of data transmission links transmit data; determining bandwidth proportion weights of the data transmission links when the data transmission links transmit data based on stable transmission bandwidths, real-time transmission bandwidths, transmission limited states and link delay time differences of the data transmission links; based on the bandwidth proportion weight when the data transmission is carried out by the data transmission links, distributing the data quantity of each data transmission link when the data transmission is carried out on the data transmission links; based on the amount of data allocated to each data transmission link, data is transmitted in real time. According to the application, when the data transmission is carried out by the plurality of data transmission links, the bandwidth proportion weight of the plurality of data transmission links when the data transmission is carried out is determined by comprehensively considering the stable transmission bandwidth, the real-time transmission bandwidth, the transmission limited state and the link delay time difference, so that the data volume distribution of each data transmission link when the data transmission is carried out on the plurality of data transmission links is facilitated to be more accurate and reasonable, the delay time of the data transmission is reduced as a whole, the data transmission efficiency is further improved, and the condition of data stream transmission blocking is reduced.

Additional aspects and advantages of the disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the disclosure.

Drawings

Fig. 1 is a flow chart illustrating a method of real-time transmission of a multi-link data stream according to an exemplary embodiment;

fig. 2 is a schematic diagram of an application scenario of a method for real-time transmission of a multi-link data stream according to an exemplary embodiment;

fig. 3 is a schematic diagram illustrating signaling processing in a method for real-time transmission of a multi-link data stream according to an exemplary embodiment;

fig. 4 is a schematic diagram illustrating calculation of a delay time difference in a method for real-time transmission of a multi-link data stream according to an exemplary embodiment;

fig. 5 is a schematic diagram of sequence number feedback of packet loss data in a method for real-time transmission of a multi-link data stream according to an exemplary embodiment;

fig. 6 is a schematic diagram of a corresponding ACK in a method for real-time transmission of a multi-link data stream according to an exemplary embodiment;

fig. 7 is a schematic diagram of a multi-link data stream real-time transmission apparatus according to an exemplary embodiment;

fig. 8 is a flow chart illustrating real-time transmission of a multi-link data stream according to an exemplary embodiment.

Detailed Description

Embodiments of the present disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and intended for the purpose of explaining the present disclosure and are not to be construed as limiting the present disclosure.

In view of the above, the present disclosure provides a method for real-time transmission of a multi-link data stream. Fig. 1 is a flow chart illustrating a method for real-time transmission of a multi-link data stream according to an exemplary embodiment. As shown in fig. 1, the method for transmitting the multi-link data stream in real time includes:

step 10, monitoring stable transmission bandwidth, real-time transmission bandwidth, limited transmission state and link delay time difference when a plurality of data transmission links perform data transmission;

Step 11, determining bandwidth proportion weights of the data transmission links when the data transmission links transmit data based on the stable transmission bandwidths, the real-time transmission bandwidths, the transmission limited states and the link delay time differences;

step 12, based on the bandwidth proportion weight when the data transmission is carried out on the plurality of data transmission links, distributing the data quantity of each data transmission link when the data transmission is carried out on the plurality of data transmission links;

and 13, carrying out data real-time transmission based on the data quantity distributed to each data transmission link.

In the present exemplary embodiment, the data stream may be a media video stream. The real-time transmission method of the multilink data stream can be applied to media video live broadcast. Such as real-time transmission of weak network-opposed multi-link media streams, etc. Fig. 2 is a schematic diagram of an application scenario of a method for real-time transmission of a multi-link data stream according to an exemplary embodiment. As shown in fig. 2, the plurality of data transmission links may include communication links such as wired links and/or wireless links. Such as wired networks, wireless networks, wifi networks, and others such as fiber optic networks. Each network may have multiple connections, such as multiple operators' SIM cards in the mobile network, or multiple SIM cards for one operator. The transmitting end and the receiving end establish a physical transmission link between all available links through negotiation links.

The stable transmission bandwidth is the average bandwidth of the whole transmission state when the data is transmitted in the preset time of the data transmission link. Wherein the predetermined time may be 1000ms before the current time;

the real-time transmission bandwidth is the instantaneous bandwidth of the data transmission at the current time of the data transmission link;

the transmission limited state is used for representing a state that data transmission of the data transmission link is blocked and data transmission can not be continued;

the bandwidth proportion weight is a communication bandwidth proportion which can meet the data transmission requirement when data transmission can be carried out among a plurality of data transmission links. For example, the communication bandwidth of the first data transmission link is a; the communication bandwidth of the second data transmission link is B; the communication bandwidth of the third data transmission link is C; and if the communication bandwidth of the fourth data transmission link is D, the bandwidth proportion weight between the first data transmission link and the fourth data transmission link is A: b: c: D. the larger the communication bandwidth which can meet the data transmission requirement when the data transmission link can transmit data, the larger the corresponding bandwidth proportion weight, and the more data quantity is distributed by the data transmission link.

The video stream with transmission can be a video stream after video numbering of video collected by live broadcast equipment. And according to the bandwidth proportion weight, after the data volume of each data transmission link is distributed on the plurality of data transmission links, carrying out real-time data transmission. Namely, a plurality of data transmission links are monitored in real time, the data quantity is adjusted in real time, and the data is transmitted in real time. In the specific transmission, the data transmitting end negotiates all possible feasible communication links with the data receiving end, and cuts the media stream into packets, so that the media packets are transmitted on all possible communication links in real time. The data receiving end gathers all media packets received on the feasible communication links, and outputs the media packets to subsequent processing after combination.

In the present exemplary embodiment, when the data transmission link is in the transmission limited state, which means that the data transmission link is no longer suitable for data transmission, the data transmission link is no longer allocated with data to be transmitted.

In this exemplary embodiment, the determining whether the plurality of data transmission links are in the transmission limited state when transmitting data includes:

wherein, the transmission limited state condition includes:

In the present exemplary embodiment, the disconnection of the data transmission link may refer to a network communication abnormality such as a network disconnection, no network signal, or the like.

When the data receiving end receives the data packet, if the data packet with the sequence number of N+1 is received and the data packet with the sequence number of N is not received after the data receiving end receives the data packet for a preset time length, determining that the data packet with the sequence number of N is lost. Because the data transmission link does not receive the data packet for a long time, the data transmission link is in an abnormal state at this time, so that the data packet is easy to be lost, and the data to be transmitted can not be allocated to the data transmission link at this time. N is 1 or more.

The round trip delay RTT of the data transmission link exceeds a predetermined value (typically 100 ms), which indicates that the data transmission link may have network congestion, and the data transmission link may not be allocated with data to be transmitted.

When the data transmission link does not have the transmission limited state, the data amount distribution can be performed on the data transmission link according to the bandwidth proportion weight.

When the bandwidth proportion weight is determined, firstly, the link delay time difference when the data transmission is carried out through a plurality of data transmission links is used for adjusting the stable transmission bandwidth when the data transmission is carried out through the plurality of data transmission links, and then, based on the adjusted stable transmission bandwidth, the corresponding bandwidth proportion weight when the data transmission is carried out through each data transmission link is determined; or determining the corresponding bandwidth proportion weight of each data transmission link when data is transmitted based on the difference value of the regulated stable transmission bandwidth and the real-time transmission bandwidth; at this time, the adjusted stable transmission bandwidth of the data transmission link is used as a communication bandwidth capable of meeting the data transmission requirement when the data transmission link can perform data transmission, and the bandwidth proportion weight is determined; or taking the difference between the stable transmission bandwidth and the real-time transmission bandwidth when the data transmission link performs data transmission as the communication bandwidth which can meet the data transmission requirement when the data transmission link can perform data transmission, and determining the bandwidth proportion weight.

In the present exemplary embodiment, fig. 3 is a schematic diagram illustrating signaling processing in a method for real-time transmission of a multi-link data stream according to an exemplary embodiment. As shown in fig. 3, in real-time transmission of the multi-link data stream, various signaling processes are involved, including reporting of the receiver fed back by the data receiving end to the data transmitting end, acknowledgement determination, negative acknowledgement, key frame retransmission, congestion control of transmission, acknowledgement determination, and so on. Wherein the Round Trip Time (RTT) can be calculated by means of acknowledgement signaling of acknowledgements, i.e. acknowledgements determining acknowledgements. The main role of acknowledgement signaling for acknowledgements is to calculate the Round Trip Time (RTT) of the link, which RTT is included as important link information in the acknowledgement packet. Firstly, the acknowledgement data packet and the acknowledgement data packet of acknowledgement contain accurate time stamp and acknowledgement sequence number, when the sender transmits the acknowledgement data packet to the receiver, the receiver returns an acknowledgement data packet immediately, and then the sender corresponds the acknowledgement data packet and the acknowledgement data packet of acknowledgement one by one according to the acknowledgement sequence number, and the Round Trip Time (RTT) of the link is obtained by subtracting the time stamps of the acknowledgement data packet and the acknowledgement data packet of acknowledgement.

In this exemplary embodiment, the monitoring the link delay time difference when the plurality of data transmission links perform data transmission includes:

In the present exemplary embodiment, fig. 4 is a schematic diagram illustrating calculation of a delay time difference in a real-time transmission method of a multi-link data stream according to an exemplary embodiment. As shown in fig. 4, when the data transmission link performs data transmission within the predetermined time, the transmission time point difference T between two adjacent data packets when transmitting can be obtained at the data transmitting end _i -T _i-1 Obtaining the difference t of the receiving time points of two adjacent data packets at the data receiving end during the receiving _i -t _i-1 The method comprises the steps of carrying out a first treatment on the surface of the The delay time difference between the ith data packet and the (i-1) th data packet of the data transmission link is d _i ＝t _i -t _i-1 -(T _i -T _i-1 )。

And carrying out primary exponential smoothing method prediction formula fitting based on delay time difference of a plurality of adjacent data packets in a preset time when the data transmission link carries out data transmission, so as to obtain a smooth fitting curve.

The primary exponential smoothing method prediction formula is as follows:

S _t+1 ＝αy _t +(1-α)S _t the method comprises the steps of carrying out a first treatment on the surface of the Which is a kind ofIn S _t+1 S is the primary exponential smoothing trend predictive value at the time t+1 _t A predicted value of primary exponential smoothing trend at the moment t, alpha is an exponential smoothing coefficient, y _t Is the sum of the delay time differences of the actual observer at time t, i.e. time t. When the smooth fitting curve is obtained, the horizontal axis is set as time, the vertical axis is the sum of delay time differences in unit time, and then a discrete line segment can be drawn. The exponential smoothing method is a time series analysis prediction method developed on the basis of the moving average method, and predicts the future of the phenomenon by calculating an exponential smoothing value and matching with a certain time series prediction model. The principle is that the exponential smoothing value in any period is the weighted average of the actual observed value in the current period and the exponential smoothing value in the previous period, so that the method is also a special weighted average method. Thus, a linear exponential smoothing predictive formula can be used to fit the time series of this delay time difference sum, resulting in a smooth fit curve.

And calculating the slope of the curve by a least square method, so that the change trend of the delay time difference can be obtained and the change trend is used for representing the saturation degree of the buffer in the network. I.e. the link delay time difference when the data transmission link according to the above embodiments performs data transmission within the predetermined time. The link delay time difference is used to characterize the saturation level of the buffer in the network.

In the two-point data transmission process based on the IP network, many routers, switches, gateways and other devices are passed through in the middle. For IP packets, these devices can be considered as buffers, and the overall buffer of these devices is limited, and if the buffer is full, serious conditions such as packet loss occur. In real-time communication against weak networks, it is necessary to evaluate whether the buffering of the network species is saturated. The delay time difference characterizes this feature exactly.

If the trend of the delay time difference is greater than 0, the buffer memory representing the physical transmission link is saturated, and the rate of sending data should be monitored so as to reduce the network buffer memory, at this time, the stable transmission bandwidth of the data transmission link during data transmission can be properly reduced, for example, the stable transmission bandwidth is reduced to 95% of the original transmission bandwidth. If the variation trend of the delay time difference is equal to 0, the buffer of the physical transmission link tends to be saturated, and the buffer is not suitable for rapidly increasing data transmission, and at the moment, the stable transmission bandwidth of the data transmission link in data transmission is kept unchanged. If the variation trend of the delay time difference is smaller than 0, the buffer of the physical transmission link is not full, and the method is suitable for rapidly increasing data transmission, and at the moment, the stable transmission bandwidth of the data transmission link during data transmission can be properly increased, for example, the stable transmission bandwidth is increased to 105% of the original transmission bandwidth. The bandwidth proportion weights are then recalculated.

The least square method formula is as follows:

1. let the fitted straight line be y=ax+b;

2. for any sample point (x _i ,y _i ) Taking the error as e=y _i -(ax _i +b)；

3. When (when)The minimum fitting degree is the highest; 4. and (3) respectively obtaining first-order partial derivatives of the a and the b to obtain:

5. let equation (1) and equation (2) equal 0, and have:obtaining a final solution:

wherein an arbitrary sample point (x _i ,y _i ) Sample points on the curve are fitted for smoothing. X is x _i Taking the value y of the moment i in the transverse axis direction of the smooth fitting curve _i And taking the value of the moment i in the vertical axis direction of the smooth fitting curve. And (3) obtaining a slope a of the smooth fitting curve as a link delay time difference when the data transmission link performs data transmission in the preset time through formulas (1) to (4).

sequencing and numbering the data to be transmitted;

In this exemplary embodiment, the allocating the data packets with the ordering numbers to the plurality of data transmission links according to the bandwidth proportion weight for data transmission includes:

In the present exemplary embodiment, the media packets transmitted at the data transmitting end and the data receiving end have consecutive packet sequence numbers. For example, the data to be transmitted is 1G, which can be divided into 1000 packets, and the sequence numbering of 1 to 1024 is performed. The plurality of data transmission links are 4 data transmission links, and comprise a first data transmission link, a second data transmission link, a third data transmission link and a fourth data transmission link, and the corresponding bandwidth proportion weight is 5:2:2:1. at this time, packets with sequence numbers 1 to 500 may be divided into a first group and allocated to a first data transmission link; the packets with sequence numbers 501-700 are divided into a second group and distributed to a second data transmission link; the packets with the sequence numbers 701-900 are divided into a third group and distributed to a third data transmission link; packets with sequence numbers 901 to 1000 are divided into a fourth group and allocated to a fourth data transmission link. In the application, a plurality of allocation modes can be adopted when specific data allocation is carried out. This is by way of example only and is not limiting.

In the present exemplary embodiment, the present application supports the transmission of signaling on each transmission link separately, which is the basis for various bandwidth estimation, code rate allocation, data transmission, and packet loss retransmission. The real-time interactive signaling of the data transmitting end and the data receiving end on each link mainly comprises a report of a receiver sent by the data receiving end to the data transmitting end, a positive response signaling sent by the data receiving end to the data transmitting end, a negative response signaling sent by the data receiving end to the data transmitting end, a key frame retransmission signaling sent by the data receiving end to the data transmitting end, transmission congestion control information sent by the data receiving end to the data transmitting end and a determination response sent by the data receiving end to the data receiving end.

When the data receiving end successfully receives the data packet, a corresponding acknowledgement signaling can be sent to the data sending end;

when the data receiving end fails to receive the data packet, a corresponding negative response signaling can be sent to the data sending end (for example, when the data receiving end receives the data packet with the sequence number of N+1, a timer is set after the data receiving end receives the data packet with the sequence number of N, and the data packet with the sequence number of N is determined to be lost if the data packet with the sequence number of N is not received for longer than a preset time period;

when the data receiving end needs the data transmitting end to retransmit the data packet with a certain sequence number, the corresponding key frame retransmission signaling can be transmitted to the data transmitting end. For example, the data transmitting end transmits the data packet with the sequence number N of the lost packet to the data receiving end according to the key frame retransmission signaling.

The signaling communication module of the data transmitting end calculates and transmits various communication protocols on each physical transmission link independently, and is used for the data transmitting end to calculate the optimal available transmission parameters (including stable transmission bandwidth, real-time transmission bandwidth, limited transmission state and link delay time difference) of each communication link.

And the signaling communication module of the data receiving end sends negative acknowledgement signaling on the RTT optimal link and receives the retransmission packet data packet on the optimal link. The signaling communication module of the data receiving end receives the affirmation acknowledgement of the determinable acknowledgement, further calculates the accurate RTT rapidly, and reports back to the data sending end through the receiver.

In order to prevent the data packet of the negative acknowledgement signaling from being lost in transmission, the signaling communication module of the data receiving end sends the negative acknowledgement signaling with a certain interval period, wherein the negative acknowledgement signaling comprises all lost packet sequence numbers of the interval period. For example, the negative acknowledgement signaling sent in the first period includes sequence numbers 50-100 of packet loss data, and if packet loss occurs in the second period 200-220, the negative acknowledgement signaling sent in the second period includes sequence numbers 50-100 and 200-220.

In this exemplary embodiment, for example, when the bandwidth proportion weight of the first data transmission link is the largest, the first data transmission link is selected to send the packet with the sequence number N of the packet loss to the data receiving end.

In this exemplary embodiment, when the data receiving end receives a data packet, if there are a plurality of data packets that are lost, the data receiving end feeds back packet loss information of the plurality of data packets to the data transmitting end;

In the present exemplary embodiment, the negative acknowledgement signaling includes not only the independent portion Xu Xuhao of the packet loss, but also one continuous packet sequence number region, which can cover the scenario where continuous, large-area packet loss occurs. Fig. 5 is a schematic diagram of sequence number feedback of packet loss data in a method for real-time transmission of a multi-link data stream according to an exemplary embodiment. As shown in fig. 5, the sequence number of the lost packet data may include an independent sequence number of the lost packet, or may include a sequence number of the consecutive lost packet, including a start sequence number of the consecutive lost packet and a stop sequence number of the consecutive lost packet. Meanwhile, in order to prevent the negative acknowledgement signaling data packet from being lost in the reverse transmission, the signaling communication module of the receiving end also sends a periodic negative acknowledgement signaling report at a certain interval, wherein the periodic negative acknowledgement signaling report comprises all lost packet sequence numbers in the period. The data transmitting end only needs to process the missing packet sequence number which is not processed. For example, if the data packets with sequence numbers 50 to 100 are lost, the negative acknowledgement signaling may include only the start sequence number 50 and the stop sequence number 100.

In the present exemplary embodiment, when determining a stable transmission bandwidth when the data transmission link performs data transmission, a minimum value of the transmission bandwidth and the reception bandwidth may be used as the stable transmission bandwidth when the data transmission link performs data transmission for a predetermined time. Thus, the accuracy of the monitoring data is ensured.

In the present exemplary embodiment of the present invention,

gaussian conjugate prior is the most commonly used prior probability distribution in statistics. The gaussian conjugate prior can be used to describe a prior view of the unknown parameters, while it also allows the prior distribution to be adjusted as information is obtained so that it matches the new data.

When calculating the real-time bandwidth of a certain physical link, if the determined response number in unit time is directly used for calculation, the randomness is too large, and the bandwidth can be regarded as being in accordance with Gaussian normal distribution, so that a Gaussian conjugate prior formula can be adopted to estimate the real-time bandwidth.

The real-time bandwidth estimation formula is:

wherein, the liquid crystal display device comprises a liquid crystal display device,

the prior estimation mu is the real-time bandwidth estimated last time (calculated once for 50 ms), and the prior variance sigma ² The variance calculated last time is the current calculated real-time bandwidth, and the posterior estimation x is the posterior variance tau ² The mean square error between the current calculated real-time bandwidth and the prior estimate. In addition, the update of the a priori variance is the post-test variance τ ² A priori variance sigma from last time ² The product of (a) divided by the prior variance sigma of the last time ² And the post-inspection variance tau ² And updating. I.e. the a priori variance is equal to the a posteriori variance τ ² A priori variance sigma from last time ² The product of (a) divided by the prior variance sigma of the last time ² And the post-inspection variance tau ² And the resulting quotient. Real-time bandwidth estimation thus obtainedAs a priori estimate for the next estimation of real-time bandwidth.

For example, the number of the cells to be processed,wherein (1)>For the prior variance when estimating the real-time bandwidth this time, +.>A priori variance for last estimation of real-time bandwidth,/->The posterior variance when the real-time bandwidth is estimated this time.

When the data amount transmitted in the unit time when the data transmission link performs data transmission is obtained, the data amount received by the data receiving end can be determined according to the response amount received in the unit time. In a 99% scenario, the data length of the packet is fixed (i.e., the length of the UDP packet, which typically is around 1400 bytes). When N acknowledgement responses are received, there is a data size of n×1400 bytes, and dividing by unit time is real-time bandwidth.

In this exemplary embodiment, if the total bandwidth of the real-time transmission bandwidths of the monitored plurality of data transmission links during data transmission is smaller than the target bandwidth, if the data transmission is performed according to the original plan at this time, the data amount of the video to be transmitted may be reduced by adjusting the encoding format of the video to be transmitted during video streaming transmission because the total bandwidth of the current real-time transmission bandwidth cannot meet the data amount transmission of the current data to be transmitted. For example, the coding format is adjusted from h.264 to h.265, or the code rate 1080p to 720p, or the like.

The present disclosure provides a real-time transmission device for a multi-link data stream. Fig. 7 is a schematic diagram illustrating a structure of a multi-link data stream real-time transmission apparatus according to an exemplary embodiment. As shown in fig. 7, includes:

the link monitoring module 70 is configured to monitor a stable transmission bandwidth, a real-time transmission bandwidth, a limited transmission state, and a link delay time difference when the plurality of data transmission links perform data transmission;

a bandwidth proportion weight determining module 71, configured to determine a bandwidth proportion weight when the plurality of data transmission links perform data transmission based on the stable transmission bandwidth, the real-time transmission bandwidth, the transmission limited state, and the link delay time difference when the plurality of data transmission links perform data transmission;

A data amount distribution module 72, configured to distribute, based on the bandwidth proportion weights when the plurality of data transmission links perform data transmission, the data amount of each data transmission link when the plurality of data transmission links perform data transmission;

the data transmission module 73 is configured to perform data real-time transmission based on the data amount allocated to each data transmission link.

In the present exemplary embodiment, the data stream may be a media video stream. The real-time transmission method of the multilink data stream can be applied to media video live broadcast. Such as real-time transmission of weak network-opposed multi-link media streams, etc. Fig. 2 is a schematic diagram of an application scenario of a method for real-time transmission of a multi-link data stream according to an exemplary embodiment. As shown in fig. 2, the plurality of data transmission links may include wired links and/or wireless links. Such as wired networks, wireless networks, wifi networks, and others such as fiber optic networks. Each network may have multiple connections, such as multiple operators' SIM cards in the mobile network, or multiple SIM cards for one operator. The transmitting end and the receiving end establish a physical transmission link between all available links through negotiation links.

The stable transmission bandwidth is the average bandwidth of the whole transmission state when the data is transmitted in the preset time of the data transmission link. Wherein the predetermined time may be 1000ms before the current time.

The real-time transmission bandwidth is the instantaneous bandwidth of the data transmission at the current time of the data transmission link.

And the transmission limited state is used for representing a state that the data transmission of the data transmission link is blocked and the data transmission can not be continued.

The multi-link data stream real-time transmission device is used for monitoring stable transmission bandwidth, real-time transmission bandwidth, limited transmission state and link delay time difference when a plurality of data transmission links perform data transmission; determining bandwidth proportion weights of the data transmission links when the data transmission links transmit data based on stable transmission bandwidths, real-time transmission bandwidths, transmission limited states and link delay time differences of the data transmission links; based on the bandwidth proportion weight when the data transmission is carried out by the data transmission links, distributing the data quantity of each data transmission link when the data transmission is carried out on the data transmission links; based on the amount of data allocated to each data transmission link, data is transmitted in real time. According to the application, when the data transmission is carried out by the plurality of data transmission links, the bandwidth proportion weight of the plurality of data transmission links when the data transmission is carried out is determined by comprehensively considering the stable transmission bandwidth, the real-time transmission bandwidth, the transmission limited state and the link delay time difference, so that the data volume distribution of each data transmission link when the data transmission is carried out on the plurality of data transmission links is facilitated to be more accurate and reasonable, the delay time of the data transmission is reduced as a whole, the data transmission efficiency is further improved, and the condition of data stream transmission blocking is reduced.

In the present exemplary embodiment, the bandwidth proportion weight determining module is configured to

wherein, the transmission limited state condition includes:

In the present exemplary embodiment, as shown in fig. 3, in the real-time transmission of the multi-link data stream, various signaling processes are involved, including a report of a receiver fed back from the data receiving end to the data sending end, determining an acknowledgement, a negative acknowledgement, a key frame retransmission, controlling congestion in transmission, determining an acknowledgement, and so on. Wherein the Round Trip Time (RTT) can be calculated by means of acknowledgement signaling of acknowledgements, i.e. acknowledgements determining acknowledgements. The main role of acknowledgement signaling for acknowledgements is to calculate the Round Trip Time (RTT) of the link, which RTT is included as important link information in the acknowledgement packet. Firstly, the acknowledgement data packet and the acknowledgement data packet of acknowledgement contain accurate time stamp and acknowledgement sequence number, when the sender transmits the acknowledgement data packet to the receiver, the receiver returns an acknowledgement data packet immediately, and then the sender corresponds the acknowledgement data packet and the acknowledgement data packet of acknowledgement one by one according to the acknowledgement sequence number, and the Round Trip Time (RTT) of the link is obtained by subtracting the time stamps of the acknowledgement data packet and the acknowledgement data packet of acknowledgement.

In the present exemplary embodiment, the link monitoring module is configured to

In the present exemplary embodiment, as shown in fig. 4, the transmission time point difference T between the transmission time points of two adjacent data packets at the transmission time of the data transmission end can be obtained when the data transmission link performs data transmission within the predetermined time _i -T _i-1 Obtaining the difference t of the receiving time points of two adjacent data packets at the data receiving end during the receiving _i -t _i-1 The method comprises the steps of carrying out a first treatment on the surface of the The delay time difference between the ith data packet and the (i-1) th data packet of the data transmission link is d _i ＝t _i -t _i-1 -(T _i -T _i-1 )。

The primary exponential smoothing method prediction formula is as follows:

S _t+1 ＝αy _t +(1-α)S _t the method comprises the steps of carrying out a first treatment on the surface of the Wherein S is _t+1 S is the primary exponential smoothing trend predictive value at the time t+1 _t A predicted value of primary exponential smoothing trend at the moment t, alpha is an exponential smoothing coefficient, y _t Is the sum of the delay time differences of the actual observer at time t, i.e. time t. When the smooth fitting curve is obtained, the horizontal axis is set as time, the vertical axis is the sum of delay time differences in unit time, and then a discrete line segment can be drawn. The exponential smoothing method is a time series analysis prediction method developed on the basis of the moving average method, and predicts the future of the phenomenon by calculating an exponential smoothing value and matching with a certain time series prediction model. The principle is that the exponential smoothing value in any period is the weighted average of the actual observed value in the current period and the exponential smoothing value in the previous periodThus, a special weighted average method is also used. Thus, a linear exponential smoothing predictive formula can be used to fit the time series of this delay time difference sum, resulting in a smooth fit curve.

The least square method formula is as follows:

1. let the fitted straight line be y=ax+b;

2. for any sample point (x _i ,y _i ) Taking the error as e=y _i -(ax _i +b)；

3. When (when)The minimum fitting degree is the highest;

4. and (3) respectively obtaining first-order partial derivatives of the a and the b to obtain:

/>

In some embodiments, the data volume allocation module is configured to

Sequencing and numbering the data to be transmitted;

In the present exemplary embodiment, the data amount distribution module is configured to

In some embodiments, the data transmission module is configured to

In the present exemplary embodiment, a data transmission module is used for

In some embodiments, the link monitoring module is configured to

In the present exemplary embodiment, the gaussian conjugate prior is the most commonly used prior probability distribution in statistics. The gaussian conjugate prior can be used to describe a prior view of the unknown parameters, while it also allows the prior distribution to be adjusted as information is obtained so that it matches the new data.

The real-time bandwidth estimation formula is:

For example, the number of the cells to be processed,wherein (1)>For the prior variance when estimating the real-time bandwidth this time, +. >A priori variance for last estimation of real-time bandwidth,/->The posterior variance when the real-time bandwidth is estimated this time.

In some embodiments, the data transmission module is configured to

The device for transmitting the multi-link data stream in real time provided by the application comprises a signaling communication module and an input media receiving module which are contained in a data receiving end shown in fig. 2, a reorganization ordering module and a media output module which are contained in the data receiving end besides the link monitoring module 70, the bandwidth proportion weight determining module 71, the data quantity distribution module 72 and the data transmission module 73 which are shown in fig. 7. The data transmitting end comprises, but is not limited to, a link negotiation module, a link monitoring module, a bandwidth proportion weight determining module, a data quantity distributing module, a signaling communication module and an input media receiving module. The data receiving end comprises, but is not limited to, a link negotiation module, a link receiving module, a reorganization ordering module, a signaling communication module and a media output module.

The link negotiation module of the data transmitting end is used for negotiating communication link parameters with the link negotiation module of the data receiving end. The link negotiation module of the data transmitting end negotiates communication link parameters with the link negotiation module of the data receiving end, including but not limited to parameters such as IP addresses, ports, handshake information, link encryption keys and the like of the two ends. And establishing link communication between the data transmitting end and the data receiving end through the negotiated communication link parameters. For example, the first data transmission link communication is established by negotiating parameters such as an IP address, port, handshake information, link encryption keys, etc. associated with the first data transmission link.

The signaling communication module of the data transmitting end is used for carrying out real-time signaling receiving and transmitting communication with the signaling communication module of the data receiving end in real time.

The signaling communication module of the data transmitting end can independently process various communication protocols in each data transmission link, so as to obtain optimal available transmission parameters (at least comprising stable transmission bandwidth, real-time transmission bandwidth, limited transmission state, RTT and link delay time difference) of each data transmission link.

And the signaling communication module of the data transmitting end processes negative response signaling on the RTT optimal link and transmits media data packets needing to be retransmitted on the RTT optimal link. For example, packets with sequence numbers 101 to 500, which require retransmission, are packets with sequence numbers of retransmission packet loss.

The signaling communication module of the data transmitting end processes the data and also comprises an acknowledgement which is immediately returned to the acknowledgement, and the signaling communication module of the data receiving end accurately calculates the RTT of the data transmission link, so that the transmission link with the minimum RTT time is selected, and the data of lost packets is retransmitted.

The link monitoring module of the data transmitting end can obtain each link parameter based on the signaling communication module to detect and estimate the stable transmission bandwidth, the real-time transmission bandwidth, the transmission limited state, the RTT and the link delay time difference of each link in real time.

The real-time transmission bandwidth of the link monitoring module of the data transmitting end adopts acknowledgement signaling information transmitted back based on the communication signaling of the data receiving end to calculate the real-time bandwidth on the link.

The real-time transmission bandwidth of the link estimation module of the transmitting end adopts acknowledgement ACK information transmitted back based on the communication signaling of the receiving end to calculate the real-time bandwidth on the link. Fig. 6 is a schematic diagram of a corresponding ACK in a method for real-time transmission of a multi-link data stream according to an exemplary embodiment. As shown in fig. 6, the acknowledgement ACK information includes a time stamp corresponding to the ACK reception time shown in fig. 6 and a corresponding ACK sequence number.

In the application, the data quantity received by the data receiving end can be determined according to the determined response quantity. In a 99% scenario, the data length of the packet is fixed (i.e., the length of the UDP packet, which typically is around 1400 bytes). When N acknowledgement responses are received, there is a data size of n×1400 bytes, and dividing by unit time is real-time bandwidth.

In the present exemplary embodiment, the signaling communication module of the receiving end processes parameter calculation and protocol packet transmission of communication protocols including, but not limited to, receiver report RR, acknowledgement ACK, negative acknowledgement NACK, key frame retransmission Pli, transport congestion control transport cc, and the like.

In this exemplary embodiment, the signaling communication module of the receiving end sends a negative acknowledgement NACK on the RTT optimal link, and receives the packet data packet that needs to be retransmitted on the optimal link.

In this exemplary embodiment, the bandwidth proportion weight determining module of the data transmitting end allocates streaming media transmission bandwidths to each data transmission link according to the real-time link parameters of all the data transmission links estimated by the link monitoring module of the data transmitting end.

The data quantity distribution module of the data transmitting end distributes the streaming media packet input from the input media receiving module to each data transmission link for transmission according to the streaming media transmission bandwidth distributed to each data transmission link by the bandwidth proportion weight determination module of the data transmitting end.

And the reorganization sequencing module of the data receiving end sequences the media data packets received by the link receiving module of the data receiving end from each data transmission link according to the packet sequence numbers, takes the sequencing result as output, and completes real-time data transmission. For example, the link receiving module receives the data packet with the sequence number of 1-100 transmitted by the first data transmission link, the data packet with the sequence number of 101-500 transmitted by the second data transmission link, the data packet with the sequence number of 501-800 transmitted by the third data transmission link, and the data packet with the sequence number of 801-1000 transmitted by the fourth data transmission link, and then the reorganization ordering module orders the data packets with the sequence numbers of 1-1000 transmitted by the first data transmission link to the fourth data transmission link according to the sequence numbers, so as to complete real-time data transmission. When the link receiving module receives the data packet, there may be a case that the data packet with the sequence number of 501-800 transmitted by the third data transmission link is received first, and then the data packet with the sequence number of 101-500 transmitted by the second data transmission link is received, which prevents the subsequent reorganization and sorting module from sorting according to the packet sequence number. After receiving all the 1-1000 data packets, the reorganization and sorting module can sort according to the sequence numbers to complete real-time data transmission.

In this exemplary embodiment, the reassembly and sorting module of the receiving end sorts the media data packets received by the link receiving module of the receiving end from each communication link according to the packet sequence numbers, and takes the sorting result as output, so as to complete real-time data transmission.

In the present exemplary embodiment, fig. 8 is a flow chart illustrating the real-time transmission of a multi-link data stream according to an exemplary embodiment. As shown in fig. 8, the real-time transmission flow of the multi-link data stream includes:

step (1), the data transmitting end and the data receiving end carry out transmission link negotiation through a link negotiation module, and establish corresponding transmission links, such as transmission links 1 to n in fig. 2;

step (2), bandwidth allocation is carried out according to an input bandwidth allocation rule of an initial setting media packing and bandwidth proportion weight determining module, the input media packing and bandwidth proportion weight determining module distributes the input media to each transmission link for transmission according to the rule;

step (3), a link transmitting module of the data transmitting end transmits a media packet stream to a link receiving module of the data receiving end;

step (4), the signaling communication module of the data receiving end reports the data receiving condition to the data sending end according to the communication protocol based on the media information received by the link receiving module;

Step (5), the signaling communication module of the data transmitting end analyzes the signaling, and provides parameters for the link monitoring module of the transmission link to estimate the bandwidth;

step (6), the link monitoring module of each transmission link provides the respective bandwidth estimation to the bandwidth proportion weight determining module, the bandwidth proportion weight determining module determines the bandwidth based on the information of each link according to the own rule, and provides the result to the input media packing and entry transmission module, wherein the input media packing and entry transmission module comprises a data amount distribution module and a data transmission module shown in fig. 7;

step (7), the data receiving end sends the media packet received by each link to the reorganization and sequencing module;

step (8), if no sequence number is received, the reorganization ordering module informs the sequence number of the packet loss data to a negative response signaling,

step (9), the data receiving end selects the link with the optimal current RTT as a channel, and reports the negative response signaling back to the data transmitting end;

step (c), after the packet loss retransmission module of the data transmitting end receives the negative response signaling, searching the packet in the own packet buffer according to the sequence number of the data packet to be retransmitted;

Step (a)The packet loss retransmission module of the data transmitting end preferentially transmits the lost packet queues to the data receiving end through the RTT optimal transmission link;

step (a)After all data are received by the data receiving end, the received data packets are ordered and output according to the sequence numbersAnd (5) completing real-time data transmission.

The present disclosure provides a computer readable storage medium having stored thereon a multilink data stream real-time transmission program which, when executed by a processor, implements the multilink data stream real-time transmission method described in the above embodiments.

The present disclosure provides a multi-link data stream real-time transmission system, comprising:

the system comprises a memory, a processor and a multi-link data stream real-time transmission program stored in the memory and capable of running on the processor, wherein the multi-link data stream real-time transmission method described in each embodiment is realized when the processor executes the multi-link data stream real-time transmission program.

It should be noted that the logic and/or steps represented in the flowcharts or otherwise described herein, for example, may be considered as a ordered listing of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium upon which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present disclosure. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

In the description of the present disclosure, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings are merely for convenience in describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element being referred to must have a specific orientation, be configured and operated in a specific orientation, and therefore should not be construed as limiting the present disclosure.

Furthermore, the terms "first," "second," and the like, as used in embodiments of the present disclosure, are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated in the present embodiment. Thus, a feature of an embodiment of the present disclosure that is defined by terms such as "first," "second," and the like may explicitly or implicitly indicate that at least one such feature is included in the embodiment. In the description of the present disclosure, the word "plurality" means at least two or more, for example, two, three, four, etc., unless explicitly specified otherwise in the examples.

In this disclosure, unless expressly specified or limited otherwise in the examples, the terms "mounted," "connected," and "secured" and the like as used in the examples are intended to be broadly construed, as for example, the connection may be a fixed connection, may be a removable connection, or may be integral, and as may be a mechanical connection, an electrical connection, or the like; of course, it may be directly connected, or indirectly connected through an intermediate medium, or may be in communication with each other, or in interaction with each other. The specific meaning of the terms in this disclosure will be understood by those of ordinary skill in the art depending on the specific implementation.

In this disclosure, unless expressly stated or limited otherwise, a first feature "up" or "down" a second feature may be the first and second features in direct contact, or the first and second features in indirect contact through an intervening medium. Moreover, a first feature being "above," "over" and "on" a second feature may be a first feature being directly above or obliquely above the second feature, or simply indicating that the first feature is level higher than the second feature. The first feature being "under", "below" and "beneath" the second feature may be the first feature being directly under or obliquely below the second feature, or simply indicating that the first feature is less level than the second feature.

Although embodiments of the present disclosure have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the present disclosure, and that variations, modifications, alternatives, and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the present disclosure.

Claims

1. A method for real-time transmission of a multilink data stream, comprising:

2. The method according to claim 1, wherein determining the bandwidth proportion weight of the plurality of data transmission links when transmitting data based on the stable transmission bandwidth, the real-time transmission bandwidth, the transmission limited state, and the link delay time difference when transmitting data by the plurality of data transmission links comprises:

3. The method for real-time transmission of a multi-link data stream according to claim 2, wherein determining whether the plurality of data transmission links are in the transmission limited state when transmitting data comprises:

wherein, the transmission limited state condition includes:

4. The method for real-time transmission of a multi-link data stream according to claim 2, wherein said adjusting the stable transmission bandwidth of the data transmission of the plurality of data transmission links based on the link delay time difference of the data transmission of the plurality of data transmission links comprises:

5. The method according to claim 1, wherein the allocating the data amount of each data transmission link when data transmission is performed on the plurality of data transmission links based on the bandwidth proportion weight when data transmission is performed on the plurality of data transmission links comprises:

sequencing and numbering the data to be transmitted;

6. The method for real-time transmission of a multi-link data stream according to claim 5, wherein said allocating the data packets with the sequence numbers to the plurality of data transmission links according to the bandwidth proportion weight for data transmission comprises:

7. A method for real-time transmission of a multi-link data stream according to claim 3, wherein said transmitting data in real time based on the amount of data allocated to each data transmission link comprises:

8. The method for real-time transmission of a multi-link data stream according to claim 7, wherein the sending, by the data sending end, the data packet with the sequence number N of the lost packet to the data receiving end according to the key frame retransmission signaling includes:

9. The method for real-time transmission of a multi-link data stream according to claim 1, wherein the monitoring the link delay time difference when the data transmission is performed by the plurality of data transmission links comprises:

10. The method for real-time transmission of a multi-link data stream according to claim 1, wherein said monitoring the stable transmission bandwidth of the data transmission of the plurality of data transmission links comprises:

11. The method for real-time transmission of a multi-link data stream according to claim 1, wherein said monitoring the real-time transmission bandwidth of the data transmission of the plurality of data transmission links comprises:

12. The method according to claim 7, wherein when the data receiving end receives the data packets, if there are a plurality of data packets, the data receiving end feeds back packet loss information of the plurality of data packets to the data transmitting end;

13. The method for real-time transmission of a multi-link data stream according to claim 1, wherein said performing data real-time transmission based on the data amount allocated to each data transmission link comprises:

14. A multi-link data stream real-time transmission device, comprising:

15. The apparatus according to claim 14, wherein the bandwidth proportion weight determining module is configured to

16. The apparatus according to claim 15, wherein the bandwidth proportion weight determining module is configured to

wherein, the transmission limited state condition includes:

17. The apparatus according to claim 15, wherein the bandwidth proportion weight determining module is configured to

18. The apparatus according to claim 14, wherein the data volume allocation module is configured to

Sequencing and numbering the data to be transmitted;

19. The apparatus of claim 18, wherein the data volume allocation module is configured to

20. The apparatus according to claim 16, wherein the data transmission module is configured to

21. The apparatus according to claim 20, wherein the data transmission module is configured to

22. The apparatus of claim 14, wherein the link monitor module is configured to

23. The apparatus of claim 14, wherein the link monitor module is configured to

24. The apparatus of claim 14, wherein the link monitor module is configured to

25. The apparatus of claim 20, wherein the data transmission module is configured to

26. The apparatus according to claim 14, wherein the data transmission module is configured to

27. A computer-readable storage medium, on which a multilink data stream real-time transfer program is stored, which, when executed by a processor, implements the multilink data stream real-time transfer method of any one of claims 1 to 13.

28. A system for real-time transmission of a multi-link data stream, comprising:

a memory, a processor and a multi-link data stream real-time transmission program stored on the memory and executable on the processor, the processor implementing the multi-link data stream real-time transmission method according to any one of claims 1-13 when executing the multi-link data stream real-time transmission program.