CN110943803A - Data transmission control method based on erasure coding - Google Patents
Data transmission control method based on erasure coding Download PDFInfo
- Publication number
- CN110943803A CN110943803A CN201911247578.6A CN201911247578A CN110943803A CN 110943803 A CN110943803 A CN 110943803A CN 201911247578 A CN201911247578 A CN 201911247578A CN 110943803 A CN110943803 A CN 110943803A
- Authority
- CN
- China
- Prior art keywords
- data
- chunk
- sending
- data segment
- transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
- H04L1/0056—Systems characterized by the type of code used
- H04L1/0061—Error detection codes
Abstract
The invention discloses a data transmission control method based on erasure coding, which specifically comprises the following steps: 1. the Chunk is cut and coded with erasure correction, and put into sending buffer; 2. selecting the untransmitted data segments from the sending buffer for transmission according to a given sending rate; 3. placing the received data segment into a receiving cache; 4. performing Chunk recombination and decoding; 5. sending a confirmation message to each received data segment and feeding back the confirmation message; 6. estimating available bandwidth of the network according to the receiving condition of the feedback message of the receiving end so as to update the sending rate of the data segment; the invention ensures the transmission quality of the current network and the effective utilization of the network bandwidth, improves the stability and reduces the time delay.
Description
Technical Field
The invention belongs to the field of data transmission, and particularly relates to a data transmission control method based on erasure coding.
Background
In wireless as well as wired networks, packet loss may occur in data being transmitted due to variations in channel quality or congestion. The underlying transport capabilities provided by the network tend to be unreliable. To achieve reliable data transmission in unreliable network environments, the most widely used design at present is to use the TCP protocol, and use the acknowledgement and retransmission mechanisms used therein to ensure effective delivery of data. The existing reliable data transmission protocols such as TCP are very sensitive to packet loss, which results in low utilization rate of bandwidth in an unreliable network.
Erasure coding techniques enable reliable data transmission in unreliable networks. For original data cut into X segments with the same size, erasure coding technology can code the original data into Y segments and ensure that the original data can be decoded as long as any X + Z segment is obtained. In practice, the value of Z is related to a specific encoding technique, and may be 0, or 1, 2, etc. After erasure coding is adopted, the transmitting and receiving parties do not need to accurately retransmit the lost data segment, and only need to ensure that the receiving end receives enough data. Erasure coding techniques can help the transceiving end not perform complex retransmission control on the lost packet, but face the problem of how to avoid transmission of redundant data and how to reduce transmission delay of data.
Disclosure of Invention
In view of the above problems, the present invention provides a data transmission control method based on erasure coding.
The invention relates to a data transmission control method based on erasure coding, which comprises the following steps:
step 1: chunk segmentation and encoding: inputting a file to be transmitted or streaming data, segmenting the file or the streaming data into a plurality of chunks, encoding each chunk into a plurality of data segments with equal size by using erasure correction codes, and putting the data segments into a sending cache;
step 2: data segment selection and transmission: selecting the untransmitted data segments from the sending buffer for transmission according to a given sending rate;
and step 3: and (3) receiving a data segment: placing the received data segment into a receiving cache;
and 4, step 4: chunk reorganization and decoding: attempting to decode the corresponding chunk after receiving the new data segment, and if the decoding is successful, performing streaming output on the chunk or reconstructing and restoring the chunk into a source data file;
and 5: confirmation and feedback: sending a confirmation message to each received data segment, and feeding back the number of data segments which need to be transmitted for completing decoding and corresponding to chunk;
step 6: bandwidth estimation and transmission control: estimating available bandwidth of the network according to the receiving condition of the feedback message of the receiving end so as to update the sending rate of the data segment; controlling the sending of the chunk residual data according to the number of the data segments required by the chunk fed back by the receiving end, so as to avoid redundant transmission of the data; and removing the data segment left by the transmitted chunk from the sending buffer.
Further, the sending rate of the data segment in step 2 is R, and the value of R is equal to the estimated available bandwidth Bw of the current path; the time interval for sending the data segments is v/R, wherein v is the length of the data segments; when a data segment is transmitted, selecting the chunk with the minimum number value which is not transmitted from the transmission buffer for checking to see whether the data segment needs to be selected for transmission.
The chunk check is as follows:
for chunk i, i represents a chunk number; setting the maximum number of the data segment sent by the current sending end as SiThe maximum data segment number that has received an acknowledgement is Ai(ii) a The total amount of data segments that have been sent by the chunk but have not yet been acknowledged is Si-Ai(ii) a The ratio of successful transmission of the transmission path is estimated as gammaiThen there is γ in these datai(Si-Ai) Can be received by the receiving end.
If chunk still requires OiThe decoding can be completed for each data segment, then at gammai(Si-Ai)<OiThen, selecting the data segment with the minimum number which is not transmitted from chunk i and sending the data segment to the receiving end, and updating Si(ii) a Otherwise, the same data segment selection and control is continued at the next chunk which has not been transmitted.
Further, the control method of the sending rate R is as follows:
(1) the sending rate R defaults to the maximum available bandwidth Bw supported by the network.
(2) The sending end continuously sends M data segments and then pauses; after the sending end receives N confirmation messages (the values of the data M and N are different according to different networks, and M is greater than N), the estimation of the available bandwidth Bw is immediately carried out: setting the duration between the first arrival and the last arrival acknowledgement message to T1; the minimum sending sequence number confirmed by the confirmation messages is A1, and the maximum sequence number is A2; then the available bandwidth of the current path is estimated to be N × v/T1, and the packet loss rate is estimated to be 1-N/(a2-a1+ 1).
(3) If the packet loss rate threshold caused by the non-congestion of the network path is set to be less, if 1-N/(A2-A1+1) > less, the sending end takes N x v/T1 as a new available bandwidth Bw value; otherwise, indicating that the bandwidth is not full, the sending rate may be increased, with N v/T1 θ as the new value of the available bandwidth Bw, where θ >1, representing the increased proportion of the sending rate, is a configurable parameter.
(4) If the N acknowledgement messages are not collected within the maximum acceptance time T, then (2) is returned.
(5) After the Bw finishes the first estimation, the sending end immediately adopts a new sending rate R to send the data segment; thereafter, each time an acknowledgment message is received, the sender performs the same calculations as described above based on the most recently received N acknowledgment messages.
Further, rateless coding or the like may be used as the erasure coding.
Compared with the prior art, the invention has the beneficial technical effects that:
1. the erasure correction coding is utilized to ensure that the system realizes reliable data transmission in an unreliable environment, and the transmission stability is ensured by automatically detecting and adjusting the transmission rate in a network environment with fluctuating bandwidth, so that the occurrence of congestion is avoided, and the transmission quality of the current network is ensured.
2. The available bandwidth in the network can be fully utilized while data transmission is avoided, and effective utilization of the network bandwidth is guaranteed.
3. The sending selection strategy of the data segments among different chunks ensures that the earlier chunk is transmitted with higher priority, reduces the time required for completing transmission of one chunk, and reduces the delay caused by introducing erasure coding technology.
Drawings
Fig. 1 is a schematic diagram of the transmission control operation of the present invention.
Detailed Description
The invention is described in further detail below with reference to the figures and the detailed description.
The design idea of the invention is as follows:
1. for a large file to be transmitted or continuously generated data to be transmitted, a data transmitting end divides the large file to be transmitted into a plurality of chunks, and each chunk is encoded into a plurality of data segments with the same size by adopting an erasure correcting encoding technology; each data segment obtained by coding can be just put into a UDP message for transmission; (by splitting a large file into multiple chunks, on one hand, the time required for erasure coding and decoding can be reduced, and on the other hand, the transmission delay of each data segment can be reduced, which is beneficial to reliable transmission of streaming data, and by splitting one large transmission task into multiple chunks for processing, which can help the sending and receiving ends to utilize the multi-core processing capability of computer hardware, and perform concurrent processing to reduce the required time).
2. The transmitting end and the receiving end adopt UDP to complete the transmission of the coded data, and by utilizing the characteristics of erasure correction coding, the transmitting and receiving parties only need to ensure that a receiver receives enough decoded data quantity, and do not care whether the received data is continuous or not.
3. The receiving end sends a confirmation message to the received data segment and feeds back the number of data segments needed to finish the decoding of the chunk; the transmitting end estimates the available bandwidth in the network according to the arrival rule of the confirmation message, and estimates and adjusts the transmitting rate of the data segment; and the sending end controls the sending of the chunk data segment according to the data volume required by chunk decoding fed back by the receiving end, so that the transmission of excessive redundant data segments is avoided.
4. Before the data segment generated by coding in each chunk is sent, the sending end does not need to retransmit the unacknowledged data segment; after all the data segments are sent, the sending end sends the unacknowledged data segments in sequence, and the receiving end is ensured to receive enough data.
5. In order to fully utilize the available bandwidth in the network, the transmitting end may transmit multiple chunk data segments simultaneously.
To achieve the above design, the present invention provides a data transmission control method based on erasure coding, as shown in fig. 1, specifically:
a: chunk segmentation and encoding: inputting a file to be transmitted or streaming data, segmenting the file or the streaming data into a plurality of chunks, encoding each chunk into a plurality of data segments with equal size by using erasure correction codes, and putting the data segments into a sending cache;
b: data segment selection and transmission: selecting the untransmitted data segments from the sending buffer for transmission according to a given sending rate;
c: and (3) receiving a data segment: placing the received data segment into a receiving cache;
d: chunk reorganization and decoding: attempting to decode the corresponding chunk after receiving the new data segment, and if the decoding is successful, performing streaming output on the chunk or reconstructing and restoring the chunk into a source data file;
e: confirmation and feedback: sending a confirmation message to each received data segment, and feeding back the number of data segments which are not decoded and correspond to chunk and need to be transmitted;
f: bandwidth estimation and transmission control: estimating available bandwidth of the network according to the receiving condition of the feedback message of the receiving end so as to update the sending rate of the data segment; controlling the sending of the chunk residual data according to the number of the data segments required by the chunk fed back by the receiving end, so as to avoid redundant transmission of the data; and removing the data segment left by the transmitted chunk from the sending buffer.
The selection and sending method of the data segment in B comprises the following steps:
in the invention, the data of each chunk generates a plurality of data segments with the same size through erasure coding, and the data segments are assumed to be v; if the available bandwidth of the current path estimated by the F module is Bw, the sending rate is R, and the value of R is equal to the estimated available bandwidth of the current path Bw; and simultaneously, the module B continuously selects data segments from the sending buffer to send to the data receiving end according to the time interval of v/R. In order to fully utilize the remaining bandwidth and avoid excessive redundant transmission, the present invention adopts a novel data segment selection method, and the working principle thereof is as follows.
When a data segment can be sent, the module B selects the chunk with the smallest number value that has not been transmitted from the sending buffer to check, and it is necessary to select the data segment for sending.
The examination contents are as follows:
for chunk i, assume that the maximum number of the data segment that has been sent by the current sender is SiThe maximum data segment number that has received an acknowledgement is Ai(ii) a Then the total amount of data segments that have been sent by the chunk but have not yet been acknowledged is Si-Ai. It is contemplated that these data segments in the transmission may not all be received because of packet loss. If the estimated value of the proportion of successful transmission of the transmission path is gammaiThen there is about γ in these datai(Si-Ai) Can be received by the receiving end.
Further, if the latest feedback message at the receiving end indicates that the chunk still needs to be sent by OiDecoding can be completed by each data segment; then, if γi(Si-Ai)<OiThe B module selects the data segment with the minimum number which is not transmitted from chunk i to send to the receiving end, and updates Si(ii) a Otherwise, the module B continues to perform similar data segment selection and control on the next chunk that has not been transmitted.
The method for estimating the bandwidth (i.e., controlling the transmission rate) in F is designed as follows:
in a dynamic network, the available bandwidth between the sending and receiving ends may change over time. Therefore, the invention estimates the available bandwidth on the transmission path by using the sending and confirming conditions of the data segment, and dynamically adjusts the sending rate R used by the B module.
In the initial state, the transmission rate R defaults to the maximum transmission rate supported by the network.
In order to start the whole process, at the beginning, a sending end continuously sends M data segments, and then pauses;
after the sending end receives N confirmation messages, the estimation of R is immediately carried out: assume that the duration between the first arrival and the last arrival acknowledgement message is T1; the minimum sending sequence number confirmed by the confirmation messages is A1, and the maximum sequence number is A2; then, the available bandwidth of the current path is estimated to be N × v/T1, and the packet loss rate is estimated to be 1-N/(a2-a1+ 1).
Assuming that a packet loss rate threshold caused by non-congestion of a network path is less, if 1-N/(a2-a1+1) > less, a sending end takes N × v/T1 as a new R value; otherwise, indicating that the bandwidth is not full, the sending rate can be increased, taking N x v/T1 x θ as a new R value, where θ >1, representing the increasing proportion of the sending rate, is a configurable parameter;
if N confirmation messages are not collected within the time T2, repeating the process;
after R finishes the first estimation, the sending end immediately adopts the new sending rate to send the data segment; thereafter, each time an acknowledgment message is received, the sender performs a similar calculation as described above based on the most recently received N acknowledgment messages.
At this time, if no new acknowledgement message is received within the time T2, the sender returns to the start phase and starts over.
Claims (6)
1. A data transmission control method based on erasure coding is characterized by comprising the following steps:
step 1: chunk segmentation and encoding: inputting a file to be transmitted or streaming data, segmenting the file or the streaming data into a plurality of chunks, encoding each chunk into a plurality of data segments with equal size by using erasure correction codes, and putting the data segments into a sending cache;
step 2: data segment selection and transmission: selecting the untransmitted data segments from the sending buffer for transmission according to a given sending rate;
and step 3: and (3) receiving a data segment: placing the received data segment into a receiving cache;
and 4, step 4: chunk reorganization and decoding: attempting to decode the corresponding chunk after receiving the new data segment, and if the decoding is successful, performing streaming output on the chunk or reconstructing and restoring the chunk into a source data file;
and 5: confirmation and feedback: sending a confirmation message to each received data segment, and feeding back the number of data segments which need to be transmitted for completing decoding and corresponding to chunk;
step 6: bandwidth estimation and transmission control: estimating available bandwidth of the network according to the receiving condition of the feedback message of the receiving end so as to update the sending rate of the data segment; controlling the sending of the chunk residual data according to the number of the data segments required by the chunk fed back by the receiving end, so as to avoid redundant transmission of the data; and removing the data segment left by the transmitted chunk from the sending buffer.
2. The erasure coding-based data transmission control method according to claim 1, wherein the sending rate of the data segment in step 2 is R, and the value of R is equal to the estimated available bandwidth Bw of the current path; the time interval for sending the data segments is v/R, wherein v is the length of the data segments; when a data segment is transmitted, selecting the chunk with the minimum number value which is not transmitted from the transmission buffer for checking to see whether the data segment needs to be selected for transmission.
3. The erasure coding-based data transmission control method according to claim 2, wherein the chunk check contents are as follows:
for chunk i, i represents a chunk number; setting the maximum number of the data segment sent by the current sending end as SiThe maximum data segment number that has received an acknowledgement is Ai(ii) a The total amount of data segments that have been sent by the chunk but have not yet been acknowledged is Si-Ai(ii) a The ratio of successful transmission of the transmission path is estimated as gammaiThen there is γ in these datai(Si-Ai) The data segment can be received by the receiving end;
if chunk still requires OiThe decoding can be completed for each data segment, then at gammai(Si-Ai)<OiThen, selecting the data segment with the minimum number which is not transmitted from chunk i and sending the data segment to the receiving end, and updating Si(ii) a Otherwise, the same data segment selection and control is continued at the next chunk which has not been transmitted.
4. The method as claimed in claim 2, wherein the sending rate R is controlled by:
(1) the sending rate R defaults to adopt the maximum available bandwidth Bw supported by the network;
(2) the sending end continuously sends M data segments and then pauses; after the sending end receives N confirmation messages, the available bandwidth Bw is estimated immediately: setting the duration between the first arrival and the last arrival acknowledgement message to T1; the minimum sending sequence number confirmed by the confirmation messages is A1, and the maximum sequence number is A2; then the available bandwidth estimation value of the current path is N × v/T1, and the packet loss rate estimation value is 1-N/(a2-a1+ 1);
(3) if the packet loss rate threshold caused by the non-congestion of the network path is set to be less, if 1-N/(A2-A1+1) > less, the sending end takes N x v/T1 as a new available bandwidth Bw value; otherwise, indicating that the bandwidth is not full, the sending rate can be increased, taking nv/T1 θ as a new value of the available bandwidth Bw, where θ >1 represents an increasing proportion of the sending rate, which is a configurable parameter;
(4) if N confirmation messages are not collected in the maximum receiving time T, returning to the step (2);
(5) after the Bw finishes the first estimation, the sending end immediately adopts a new sending rate R to send the data segment; thereafter, each time an acknowledgment message is received, the sender performs the same calculations as described above based on the most recently received N acknowledgment messages.
5. The method as claimed in claim 4, wherein in the control of the sending rate R, values of the number of data segments M and the number of acknowledgement messages N are different according to different networks, and M > N.
6. The method of claim 1, wherein the erasure coding is not limited to a specific coding scheme.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911247578.6A CN110943803B (en) | 2019-12-09 | 2019-12-09 | Data transmission control method based on erasure coding |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911247578.6A CN110943803B (en) | 2019-12-09 | 2019-12-09 | Data transmission control method based on erasure coding |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110943803A true CN110943803A (en) | 2020-03-31 |
| CN110943803B CN110943803B (en) | 2021-10-08 |
Family
ID=69909833
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911247578.6A Active CN110943803B (en) | 2019-12-09 | 2019-12-09 | Data transmission control method based on erasure coding |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110943803B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101651519A (en) * | 2009-09-15 | 2010-02-17 | 中国人民解放军国防科学技术大学 | Method, device and system of self-adaptive dynamic forward error correction coding |
| US20100095193A1 (en) * | 2008-10-15 | 2010-04-15 | Nokia Corporation | System and Method for Pre-calculating Checksums |
| CN101997930A (en) * | 2010-12-24 | 2011-03-30 | 南开大学 | Retransmission-erasure code transmission protocol-based remote mirroring method and system |
| CN103209210A (en) * | 2013-03-04 | 2013-07-17 | 华中科技大学 | Method for improving erasure code based storage cluster recovery performance |
| CN103561057A (en) * | 2013-10-15 | 2014-02-05 | 深圳清华大学研究院 | Data storage method based on distributed hash table and erasure codes |
| US20150100858A1 (en) * | 2012-11-08 | 2015-04-09 | Q Factor Communications Corp. | Method & apparatus for improving the performance of tcp and other network protocols in a communication network |
| CN108111434A (en) * | 2017-12-14 | 2018-06-01 | 四川大学 | A kind of aeronautical Ad hoc networks method for reliable transmission based on reliable UDP and fountain codes |
| CN110324256A (en) * | 2019-05-13 | 2019-10-11 | 西南交通大学 | A kind of Transmitting Data Stream control method |
-
2019
- 2019-12-09 CN CN201911247578.6A patent/CN110943803B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100095193A1 (en) * | 2008-10-15 | 2010-04-15 | Nokia Corporation | System and Method for Pre-calculating Checksums |
| CN101651519A (en) * | 2009-09-15 | 2010-02-17 | 中国人民解放军国防科学技术大学 | Method, device and system of self-adaptive dynamic forward error correction coding |
| CN101997930A (en) * | 2010-12-24 | 2011-03-30 | 南开大学 | Retransmission-erasure code transmission protocol-based remote mirroring method and system |
| US20150100858A1 (en) * | 2012-11-08 | 2015-04-09 | Q Factor Communications Corp. | Method & apparatus for improving the performance of tcp and other network protocols in a communication network |
| CN103209210A (en) * | 2013-03-04 | 2013-07-17 | 华中科技大学 | Method for improving erasure code based storage cluster recovery performance |
| CN103561057A (en) * | 2013-10-15 | 2014-02-05 | 深圳清华大学研究院 | Data storage method based on distributed hash table and erasure codes |
| CN108111434A (en) * | 2017-12-14 | 2018-06-01 | 四川大学 | A kind of aeronautical Ad hoc networks method for reliable transmission based on reliable UDP and fountain codes |
| CN110324256A (en) * | 2019-05-13 | 2019-10-11 | 西南交通大学 | A kind of Transmitting Data Stream control method |
Non-Patent Citations (2)
| Title |
|---|
| ABDERRAHMEN MTIBAA等: "RC-UDP: On Raptor Coding over UDP for Reliable High-Bandwidth Data Transport", 《IEEE》 * |
| 徐盈盈: "纠删码和可靠UDP相结合的无线视频传输技术研究与实现", 《中国优秀硕士学位论文全文数据库（信息科技辑）》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110943803B (en) | 2021-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101061659B (en) | Adaptive forward error correction method and device | |
| CN107888342B (en) | A kind of network real-time video transmission method and device | |
| US20150117468A1 (en) | Apparatus and method for transmitting data | |
| US7584404B2 (en) | Method and apparatus for multimedia communication over packet channels | |
| US8458567B2 (en) | FEC-based reliability control protocols | |
| US7095729B2 (en) | Method for multimedia communication over packet channels | |
| KR102173084B1 (en) | Method and apparatus for transmitting and receiving data packets in a wireless communication system | |
| EP1708400B1 (en) | Loss tolerant transmission control protocol | |
| CN105450357A (en) | Adjustment method of encoding parameters, adjustment device of encoding parameters, processing method of feedback information and processing device of feedback information | |
| US9954752B2 (en) | Transmission terminal, communication system, communication method, and program | |
| CN110391879A (en) | Loss recovery method, device and the computer equipment of data transmission network | |
| KR20090058927A (en) | Method of data block transmitting | |
| US8112688B2 (en) | Data-transmission control method and transmission device | |
| EP1195938B1 (en) | Method for asynchronous incremental redundancy transmission in a communication system | |
| CN113055285A (en) | Self-adaptive data transmission method based on MPTCP and network coding | |
| CN110943803B (en) | Data transmission control method based on erasure coding | |
| WO2015036086A1 (en) | Rateless encoding | |
| US10135576B2 (en) | Rateless encoding | |
| CN109005011B (en) | Data transmission method and system for underwater acoustic network and readable storage medium | |
| CN109690991B (en) | Feedback information sending method, receiving method, device and system | |
| CN115189810B (en) | Low-delay real-time video FEC coding transmission control method | |
| US20230060948A1 (en) | Methods for Reliable Low Latency Data Delivery Using Erasure Codes and Feedback | |
| US9209947B1 (en) | Fault-tolerant data transmission system for networks subject to jamming conditions | |
| CN114337917A (en) | Data transmission method based on forward error correction and short message sending device | |
| CN117220830A (en) | Wireless network data transmission method, transmitting terminal and receiving terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |