WO2021128602A1 - 一种数据传输的方法和装置 - Google Patents

一种数据传输的方法和装置 Download PDF

Info

Publication number
WO2021128602A1
WO2021128602A1 PCT/CN2020/079351 CN2020079351W WO2021128602A1 WO 2021128602 A1 WO2021128602 A1 WO 2021128602A1 CN 2020079351 W CN2020079351 W CN 2020079351W WO 2021128602 A1 WO2021128602 A1 WO 2021128602A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
receiving end
unit
transmitted
packet
Prior art date
Application number
PCT/CN2020/079351
Other languages
English (en)
French (fr)
Inventor
杨宗兴
李政
Original Assignee
网宿科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 网宿科技股份有限公司 filed Critical 网宿科技股份有限公司
Priority to EP20775796.4A priority Critical patent/EP3873067A4/en
Priority to US17/061,517 priority patent/US11303737B2/en
Publication of WO2021128602A1 publication Critical patent/WO2021128602A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
    • H04L69/163—In-band adaptation of TCP data exchange; In-band control procedures
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/133—Protocols for remote procedure calls [RPC]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
    • H04L67/1078—Resource delivery mechanisms
    • H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/141—Setup of application sessions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/565—Conversion or adaptation of application format or content
    • H04L67/5651—Reducing the amount or size of exchanged application data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/14—Multichannel or multilink protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/143—Termination or inactivation of sessions, e.g. event-controlled end of session
    • H04L67/145—Termination or inactivation of sessions, e.g. event-controlled end of session avoiding end of session, e.g. keep-alive, heartbeats, resumption message or wake-up for inactive or interrupted session
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
    • H04L69/166—IP fragmentation; TCP segmentation
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Definitions

  • This application relates to the field of network communication technology, and in particular to a method and device for data transmission.
  • the sender and receiver can be deployed in different geographical locations according to a distributed structure.
  • the sender can call the processing logic of objects in the local application without understanding the underlying Network transmission process, and remotely request service on the receiving end through the network, that is, remotely calling the object on the receiving end.
  • the sender can pass the call request (carrying method and parameter information) to the sender, and the client
  • the call request is packaged such as encoding and encapsulation, and then the packaged call request is sent to the receiving end.
  • the receiving end stub receives the call request, decodes it, and then calls the receiving end function.
  • the receiving end performs processing based on the parameters to obtain the call result.
  • the receiving end can package the call result and return it to the sending end. In this way, the client receives the call result, decodes it, and then provides the final result to the sender.
  • the inventor found that the prior art has at least the following problems: when transmitting large data packets based on the RPC framework, on the one hand, because the data content needs to be loaded into the memory before and after the data is sent and received , It will occupy larger memory resources of the sender and receiver; on the other hand, the transmission process will occupy more bandwidth resources and take a long time, especially in a weak network environment, there will be a higher possibility of packet loss.
  • a data transmission method including:
  • the multiple data units are transmitted to the receiving end through multiple data streams.
  • a data transmission device in a second aspect, includes:
  • connection module is used to establish a communication connection with the receiving end
  • Splitting module used to split the to-be-transmitted data packet into multiple data units
  • the transmission module is used to transmit the multiple data units to the receiving end through multiple data streams
  • a network device in a third aspect, includes a processor and a memory.
  • the memory stores at least one instruction, at least one program, code set, or instruction set. A section of program, the code set or the instruction set is loaded and executed by the processor to implement the data transmission method as described in the first aspect.
  • a computer-readable storage medium stores at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code
  • the set or instruction set is loaded and executed by the processor to implement the data transmission method as described in the first aspect.
  • the sending end and the receiving end establish a communication connection; the data packet to be transmitted is split into multiple data units; through multiple data streams, The receiving end transmits multiple data units.
  • the sender side splits the data packet to be transmitted in multiple layers to obtain data units, and transmits them through multiple data streams in the form of data units.
  • the receiver side receives the data units and assembles the data units into small data packets. , Only a small amount of data units need to be loaded before and after data transmission, without loading all data packets, which can save the memory resources of the sender and the receiver.
  • FIG. 1 is a flowchart of a data transmission method provided by an embodiment of the present application
  • FIG. 2 is a flowchart of a method for connection migration provided by an embodiment of the present application
  • Fig. 3 is a flowchart of a data retransmission method provided by an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of a data transmission device provided by an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of another data transmission device provided by an embodiment of the present application.
  • Fig. 6 is a schematic structural diagram of a network device provided by an embodiment of the present application.
  • the embodiment of the present application provides a data transmission method, which can be implemented by the sending end and the receiving end.
  • Both the sending end and the receiving end can be network devices with data transmission and reception and processing functions, such as servers, Computer etc.
  • An RPC framework can be deployed on the sending end and the receiving end, and the two can realize data transmission and remote function call through the RPC framework.
  • the sender can send a call request to the receiver, and add the function to be called and the parameters required when calling the function in the call request.
  • the sending end can send a call request to the receiving end in the form of a data packet.
  • the sending end can first split the data packet, and then divide it according to the split specifications.
  • the data packet is transmitted to the receiving end.
  • the receiving end After receiving the split data packet, the receiving end can perform assembly processing first, and then perform subsequent processing on the assembled data packet.
  • the functions of the sending end and the receiving end can be interchanged, that is, the receiving end can send a call request to the sending end to realize the remote call of the method function on the sending end.
  • the specific process is related to the sending end. The process of initiating a remote call by the terminal is basically the same, and will not be repeated in this embodiment.
  • Step 101 The sending end and the receiving end establish a communication connection.
  • the sender when the sender under the RPC framework needs to remotely call the method function of the receiver, the sender can first establish a TCP connection with the receiver.
  • the TCP connection can be an on-demand connection, that is, when the sender needs to perform a remote call, it actively initiates the establishment with the receiver, and the sender actively disconnects after the call ends; it can also be a long connection, that is, the sender and the receiver establish a connection After long-term retention, regardless of whether there is a data packet transmission, multiple remote invocation procedures can use the same TCP connection.
  • the sending end and the receiving end can establish multiple data streams on the above TCP connection, where each data stream can correspond to a thread on the sending end and the receiving end, and different threads are used to send or receive on different data streams.
  • data can correspond to a thread on the sending end and the receiving end, and different threads are used to send or receive on different data streams.
  • the above multiple data streams can be two-way streams without interference.
  • the streams are independent of each other. Data transmission blocking and failed retransmissions on a single data stream will not affect the data transmission of other data streams on the same TCP connection.
  • the sending end can send data to the receiving end through the data flow, and vice versa, the receiving end can also send data to the sending end through the above data flow.
  • the sending end and the receiving end can detect the health status of each data stream by periodically sending probing data packets, so as to adjust the data transmission in time according to the detection result.
  • the sending end and the receiving end can establish multiple data streams on the communication connection according to the load information of both parties, and the corresponding processing may be as follows: the sending end sends a connection creation request carrying the load information of the sending end to the receiving end; The receiving end establishes a communication connection with the sending end, and initializes multiple data streams on the communication connection based on the load information of the sending end and the load information of the receiving end.
  • the sender when it sends a connection creation request to the receiver, it can detect the sender's load information such as local memory, bandwidth, and CPU, and add the detected sender's load information to the connection creation request. After receiving the connection creation request from the sending end that carries the load information of the sending end, the receiving end can establish a communication connection with the sending end, and can also collect local memory, bandwidth, CPU and other receiving end load information, and then can be based on the sending end The load information and the receiving end load information initialize multiple data streams on the communication connection.
  • the sender sends a connection creation request to the receiver, it can detect the sender's load information such as local memory, bandwidth, and CPU, and add the detected sender's load information to the connection creation request.
  • the receiving end After receiving the connection creation request from the sending end that carries the load information of the sending end, the receiving end can establish a communication connection with the sending end, and can also collect local memory, bandwidth, CPU and other receiving end load information, and then can be based on the sending end
  • the number of data streams on the communication connection can be equal to the number of CPU cores on the receiving end by default, and then refer to the load of each CPU on the receiving end and the sending end, as well as the memory and bandwidth usage in the last 5 minutes, to calculate the data The number of streams and the size of the traffic window for each data stream.
  • the number and size of data streams are determined based on the load status of the sending end and the receiving end, and the device resources of the sending end and the receiving end can be fully utilized to improve the data transmission efficiency and prevent the data transmission quality from being affected by the overload of the device.
  • Step 102 The sending end splits the data packet to be transmitted into multiple data units.
  • the sending end after the sending end has established multiple data streams based on the communication connection with the receiving end, it can send data packets to the receiving end through the multiple data streams.
  • the sending end may first perform split processing on the data packet to be transmitted to obtain multiple small data packets with a small amount of data. After that, the sender can compress the small data packets, and then continue to split each small data packet into multiple data units with smaller data specifications.
  • the sending end may further split the data unit based on the same principle to obtain a transmission unit with a smaller data volume, thereby alleviating the transient pressure of transmission caused by an excessively large data packet.
  • the sender when the sender unpacks a large data packet, it may determine the unpacking rule based on the size and type of the data packet, and the load of the sender and the receiver. Accordingly, the processing of step 102 may be as follows: send The terminal determines the packet splitting rule based on the size and type of the data packet to be transmitted, as well as the load information of the sender and the load information of the receiver; the sender splits the data packet to be transmitted into multiple small data packets according to the packet splitting rule , And split each small data packet into multiple data units.
  • the sender when the sender performs unpacking processing of the data packet to be transmitted, it may first determine the size and type of the data packet to be transmitted, and then determine the corresponding data packet splitting scheme according to the type and size. At the same time, the sender can also refer to the load information of the receiver and the load information of the sender to determine the packet splitting rule. Furthermore, the sending end may split the to-be-transmitted data packet into multiple small data packets according to the packet splitting rule, and split each small data packet into multiple data units.
  • the load information of the receiving end may be sent from the receiving end to the sending end when the sending end establishes a communication connection with the receiving end. It should be noted that when the load is high, in order to avoid data stream transmission congestion, a smaller number of transmission packets can be selected as much as possible, so the data packets to be transmitted can be split into larger numbers and smaller data packets.
  • the sending end may obtain the unit ID of each data unit configured to indicate specific data content for splitting. Accordingly, after step 102, the following processing may exist: For the data unit, the sender generates the unit ID of the data unit according to the small packet ID of the small data packet to which the data unit belongs and the data offset of the data unit in the small data packet, and adds the unit ID to the data unit.
  • the sending end splits the data packet to be transmitted multiple times, and after obtaining multiple data units, it can add a unit ID to each data unit to mark each data unit, among which, the unit of multiple data units
  • the ID can be incremented sequentially.
  • the sending end can first determine the small data packet to which the data unit belongs, and the data offset of the data unit in the small data packet, and then based on the preset algorithm according to the small packet ID and the small data packet ID and The above data offset generates the unit ID of the data unit, and then the unit ID can be added to the data unit.
  • each data unit can be uniquely identified through different unit IDs, and the specific data content corresponding to each data unit can be quickly determined through the unit ID.
  • the small packet ID can be generated for each small data packet obtained when the sender splits the data packet to be transmitted, and is used to uniquely identify the small data packet, and when the sender generates the small packet ID ,
  • the offset position of the small data packet in the data packet to be transmitted can be further referred to, so that the specific data content corresponding to the small data packet in the data packet to be transmitted can be determined through the small packet ID.
  • Step 103 The sending end transmits multiple data units to the receiving end through multiple data streams.
  • the sending end after the sending end splits the data packet to be transmitted multiple times to obtain the data unit, it can transmit the split data unit to the receiving end through multiple data streams established with the receiving end.
  • one stream can transmit multiple data units, and multiple data units of a small data packet can be transmitted between multiple streams.
  • the sending end can transmit data in different data streams according to the load status of each data stream. Data units of different data sizes are transmitted on the network.
  • Step 104 The receiving end receives and buffers the data units transmitted by the sending end through multiple data streams.
  • the receiving end after the receiving end has established multiple data streams based on communication connections with the sending end, it can use the data receiving threads corresponding to the multiple data streams to receive the data units transmitted by the sending end through the multiple data streams, and then can Buffer the received data unit into a unified buffer.
  • the receiving end may be provided with multiple buffers, and each buffer may be used to buffer data units of different small data packets.
  • Step 105 The receiving end assembles the buffered data units into small data packets, and performs application layer processing on the small data packets.
  • the receiving end can assemble the buffered data units.
  • the receiving end can transmit the assembled small data packet to the application layer for processing, and at the same time delete related data units in the buffer.
  • the processing of step 105 can be specifically as follows: For each data unit in the cache, according to the unit ID carried in the data unit, determine the packet corresponding to the data unit The data offset of the ID and data unit in the small data packet; according to the small packet ID and data offset corresponding to each data unit, the cached data units are assembled into small data packets.
  • the unit ID carried in the data unit can be extracted, and then the assembling process of each small data packet is completed according to the unit ID.
  • the server can restore the small packet ID corresponding to the data unit and the data offset of the data unit in the small data packet from the unit ID carried by the data unit based on a preset algorithm, and then can follow
  • the packet ID separately summarizes the data units belonging to different small data packets, and according to the data offset corresponding to each data unit, all data units corresponding to the same small packet ID can be assembled into small data packets.
  • the sending end after the sending end establishes a data stream with the receiving end, it can send a data stream ID for uniquely identifying each data stream to the receiving end, so as to realize the connection migration between the sending end and the receiving end.
  • the sender Before sending the connection creation request to the receiving end, the sender can generate data stream IDs for multiple data streams, and then add all the data stream IDs to the connection creation request.
  • the receiving end receives the connection creation request and establishes a communication connection with the sending end, the parameter information of the communication connection can be stored in correspondence with the aforementioned data stream ID.
  • the process shown in Fig. 2 may also be included between the sending end and the receiving end.
  • Step 201 When the communication connection with the receiving end is interrupted, the sending end sends a connection re-establishment request carrying the data stream IDs of multiple data streams to the receiving end.
  • the sending end detects that the communication connection with the receiving end is interrupted during the process of sending data to the receiving end, it can obtain the data stream IDs of multiple data streams generated before the communication connection is established, and send it to the receiving end. Send a connection re-establishment request carrying these data stream IDs.
  • Step 202 When receiving the connection reconstruction request carrying the data stream ID sent by the sending end, the receiving end restores the communication connection corresponding to the data stream ID, and restores multiple data streams on the communication connection, and then based on the multiple data streams Continue to transmit data units.
  • the receiving end can receive the connection re-establishment request sent by the sending end, and determine whether the data stream ID is carried in it. If it is carried, the receiving end can restore the communication connection corresponding to the data stream ID according to the stored data stream ID and the parameter information of the communication connection, and then restore multiple data streams on the communication connection, and then continue based on the multiple data streams Transmission data unit. It is not difficult to understand that the above data stream ID has nothing to do with the network (IP address, port and other information) of the sender.
  • the receiver can use the data stream ID to determine the original
  • the communication connection still transmits data through the original data stream and the sending end, which can realize connection migration, avoid multiple creation of connections, reduce unnecessary data retransmissions due to connection reconstruction, and ensure that business logic is not interrupted.
  • the sender may, after splitting the data packet to be transmitted into small data packets, buffer the offset position of each small data packet in the data packet to be transmitted, so as to perform processing based on the offset position and the aforementioned unit ID.
  • the specific process of the data retransmission processing shown in FIG. 3 can be as follows.
  • Step 301 When it is detected that the transmission of the target data unit fails, or the data of the received target data unit is incomplete, the receiving end feeds back a data retransmission request carrying the unit ID of the target data unit to the sending end.
  • the total data volume of each data unit can be determined according to the unit ID of the data unit, and then the total data volume and the received volume of the data unit can be used to determine the received data. Whether the data unit is complete. At the same time, considering that the sender sends data units in order according to the unit ID, the receiving end should receive the data units in order when the data is normally transmitted, so it can be set that the receiving end has received the preset data before receiving the data unit A. If the number of data units whose sending sequence is after data unit A, it is determined that the transmission of data unit A fails.
  • the receiving end when it is detected that the transmission of the target data unit fails or the data of the received target data unit is incomplete, the receiving end can feed back a data retransmission request carrying the unit ID of the target data unit to the sending end, so that The sender resends the target data unit.
  • Step 302 When receiving the data retransmission request carrying the unit ID of the target data unit sent by the receiving end, the sending end determines the target small data packet and the data offset corresponding to the target data unit according to the unit ID.
  • Step 303 The sending end regenerates the target data unit based on the offset position and the data offset of the target small data packet in the data packet to be transmitted, and sends the target data unit to the receiving end.
  • the sender can determine the offset position of the target small data packet in the data packet to be transmitted according to the cache content, and then combine the target data unit in the target small data packet The data offset of, determines the specific data content corresponding to the target data unit, and then can regenerate the target data unit.
  • the sending end can transmit the target data unit to the receiving end through any data stream on the communication connection. In this way, when a data unit needs to be retransmitted, the sender does not need to reload the entire data packet to be transmitted, and can determine and generate the data content that needs to be retransmitted, thereby reducing the consumption of data retransmission equipment resources and improving data retransmission. The efficiency of transmission.
  • the sending end can add priority information to the target data unit, thereby triggering the receiving end to receive and process the target data unit preferentially.
  • the sender can also assign different transmission priorities to different data units according to the data type (such as key data, supplementary data, etc.) that each data unit belongs to, and/or the current load situation of the data stream. Therefore, the sender can preferentially send data units with a higher transmission priority. For example, if the current load of the data stream is relatively high, and only data units with a small amount of data can be transmitted, a higher transmission priority can be set for the data units with a small amount of data, and vice versa.
  • the sender can also decide whether to add or delete data streams based on the data transmission status of each data stream.
  • the corresponding processing can be as follows: The sender sends data streams to the receiver according to the data transmission status of multiple data streams. New request; when receiving a new request for a data stream sent by the sender, the receiver creates a new data stream based on the communication connection; or,
  • the sending end sends a data stream interruption request to the receiving end according to the data transmission conditions and/or device load conditions of the multiple data streams; when receiving the data stream interruption request sent by the sending end, the receiving end interrupts the data stream interruption Request the corresponding target data stream and stop receiving data units on the target data stream.
  • the sending end when it sends data units to the receiving end through multiple data streams on the communication connection, it can monitor the data transmission of multiple data streams in real time, and can determine whether to perform data stream increase processing based on the monitoring results, that is, decide Whether to send a new data stream request to the receiving end. For example, when it is detected that the load of multiple data streams is higher than the preset overload threshold for a long time, the data stream can be increased appropriately, so the sending end can send a new data stream request to the receiving end.
  • the sending end when it sends data units to the receiving end through multiple data streams on the communication connection, it can monitor the data transmission status and device load status of the multiple data streams in real time (including the sender load status and the receiver load status), and According to the monitoring results, it can be judged whether to perform data stream deletion processing, that is, whether to send a data stream interrupt request to the receiving end. For example, when it is detected that the load of multiple data streams has been lower than the preset idle threshold for a long time, the data streams can be appropriately deleted, so the sender can send data streams corresponding to one or more data streams to the receiver.
  • the data stream interrupt request can carry the data stream ID of the data stream that needs to be interrupted; or, when it is detected that the data transmission quality of data stream B is poor, the sending end can send the corresponding data stream B to the receiving end At the same time, you can also send a new data stream request that carries the data stream ID of data stream B to the receiving end, so that the receiving end and the sending end can add data stream B to the new data stream after creating a new data stream. The data is transferred to the new data stream for transmission.
  • the detection of multiple data streams is real-time.
  • the sending end and the receiving end increase or interrupt the data stream.
  • the receiving end when the receiving end receives the data stream addition request sent by the sending end, it can first create a new data stream based on the communication connection between the two, and then according to the carrying status of the stream identifier in the data stream addition request, Determine whether to execute the migration process of data transmission.
  • the receiving end receives the data stream interruption request sent by the sender, it can determine the target data stream corresponding to the stream identifier carried in the data stream interruption request, and then close the target data stream, and stop receiving the sender from subsequently passing the target data Streaming data unit. In this way, by adding/deleting data streams, it is possible to avoid resource idleness, ensure resource utilization, and prevent data stream overload from affecting the quality of data transmission.
  • the sender can set the parameters of the new data stream that needs to be created by adding information such as priority and traffic window size in the data stream new request.
  • the receiving end can add new data streams after receiving the data stream. After the request, a new data stream that meets the corresponding parameters can be created based on the added information and the sender.
  • the receiving end implements load balancing of data transmission among multiple data streams by sending traffic adjustment instructions to the sending end.
  • the corresponding processing may be as follows: The sending end sends a flow adjustment instruction; the sending end receives the flow adjustment instruction sent by the receiving end, and adjusts the flow window size of the data flow according to the flow identifier carried in the flow adjustment instruction and the specified flow threshold.
  • the flow adjustment instruction may carry a flow identifier and a designated flow threshold.
  • the receiving end when the receiving end receives the data unit of the sending end, it can detect the receiving load status corresponding to each data stream, and then according to the receiving load status, send a flow adjustment instruction to the sending end.
  • the flow adjustment instruction can carry Flow identification and specified flow threshold. For example, when the receiving end detects that the data stream is overloaded, it can send a flow adjustment instruction for the data stream to the sending end, so that the sending end can reduce the flow window size of the data stream; when the receiving end detects that the data stream is in In the idle state, a flow adjustment instruction for the data stream can be sent to the sending end, so that the sending end can enlarge the flow window size of the data stream.
  • the sending end can determine the data flow that needs to be adjusted according to the flow identifier carried in the flow adjustment instruction, and then determine the flow window of the data flow based on the specified flow threshold carried in the flow adjustment instruction. The size is adjusted.
  • the flow adjustment instruction can effectively avoid load unevenness among multiple data streams, improve the utilization of network resources in the transmission process, and prevent data transmission failure due to data stream overload.
  • the sending end and the receiving end establish a communication connection; the data packet to be transmitted is split into multiple data units; and the multiple data units are transmitted to the receiving end through multiple data streams.
  • the sender side splits the data packet to be transmitted in multiple layers to obtain data units, and transmits them through multiple data streams in the form of data units.
  • the receiver side receives the data units and assembles the data units into small data packets. , Only a small amount of data units need to be loaded before and after data transmission, without loading all data packets, which can save the memory resources of the sender and the receiver.
  • an embodiment of the present application also provides a data transmission device. As shown in FIG. 4, the device includes:
  • connection module 401 is used to establish a communication connection with the receiving end
  • the splitting module 402 is used to split the to-be-transmitted data packet into multiple data units
  • the transmission module 403 is configured to transmit the multiple data units to the receiving end through multiple data streams.
  • an embodiment of the present application also provides a data transmission device. As shown in FIG. 5, the device includes:
  • the connection module 501 is configured to establish a communication connection with the sending end and multiple data streams based on the communication connection;
  • the receiving module 502 is configured to receive and buffer the data units transmitted by the sending end through the multiple data streams;
  • the processing module 503 is configured to assemble the cached data units into small data packets, and perform application layer processing on the small data packets.
  • an embodiment of the present application also provides a data transmission system.
  • the system includes a sending end and a receiving end, wherein the sending end and the receiving end cooperate to realize the data transmission described in the above embodiment. method.
  • Fig. 6 is a schematic structural diagram of a network device provided by an embodiment of the present application.
  • the network device 600 may have relatively large differences due to different configurations or performances, and may include one or more central processing units 622 (for example, one or more processors) and a memory 632, and one or more storage application programs 642 or
  • the storage medium 630 of the data 644 (for example, one or a storage device with a large amount of storage).
  • the memory 632 and the storage medium 630 may be short-term storage or persistent storage.
  • the program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the network device 600.
  • the central processing unit 622 may be configured to communicate with the storage medium 630, and execute a series of instruction operations in the storage medium 630 on the network device 600.
  • the network device 600 may also include one or more power supplies 629, one or more wired or wireless network interfaces 650, one or more input and output interfaces 658, one or more keyboards 656, and/or, one or more operating systems 641, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.
  • the network device 600 may include a memory and one or more programs. One or more programs are stored in the memory and configured to be executed by one or more processors. The above data transfer instructions.
  • the program can be stored in a computer-readable storage medium.
  • the storage medium mentioned can be a read-only memory, a magnetic disk or an optical disk, etc.

Abstract

本申请公开了一种数据传输的方法和装置，属于网络通信技术领域。所述方法包括：发送端与接收端建立通信连接（101）；发送端将待传输数据包拆分为多个数据单元（102）；发送端通过多个数据流，向所述接收端传输所述多个数据单元（103）。

Description

一种数据传输的方法和装置
交叉引用
本申请引用于2019年12月24日递交的名称为“一种数据传输的方法和装置”的第201911347142.4号中国专利申请，其通过引用被全部并入本申请。
技术领域
本申请涉及网络通信技术领域，特别涉及一种数据传输的方法和装置。
背景技术
在RPC(Remote Procedure Call，远程过程调用)框架下，发送端和接收端可以按照分布式的结构部署在不同地理位置，发送端可以基于调用本地应用程序中对象的处理逻辑，不需要了解底层的网络传输过程，而通过网络远程在接收端上请求服务，即远程调用接收端上的对象。
具体来说，发送端在与接收端建立连接、并完成了寻址处理后，如果需要发起远程过程调用，发送端可以将调用请求(携带有方法、参数信息)交由发送端，客户端对调用请求进行编码和封装等打包处理，再将打包后的调用请求发送到接收端。接收端存根接收调用请求，进行解码，然后调用接收端函数，由接收端基于参数执行处理，得到调用结果。之后，接收端可以对调用结果进行打包并返回至发送端。这样，客户端接收调用结果，并进行解码，再将得到的最终结果提供给发送端。
在实现本申请的过程中，发明人发现现有技术至少存在以下问题：在基于RPC框架进行传输大数据包时，一方面，由于在数据发送前和接收后需要把数据内容全部加载到内存中，会占用发送端和接收端较大的内存资源；另一方面，传输过程中会占用较多的带宽资源，花费较长时间，尤其在弱网络环境下 会存在较高的丢包可能性。
发明内容
为了解决现有技术的问题，本申请实施例提供了一种数据传输的方法、装置和系统，所述技术方案如下。
第一方面，提供了一种数据传输的方法，所述方法包括：
与接收端建立通信连接；
将待传输数据包拆分为多个数据单元；
通过多个数据流，向所述接收端传输所述多个数据单元。
第二方面，提供了一种数据传输的装置，所述装置包括：
连接模块，用于与接收端建立通信连接；
拆分模块，用于将待传输数据包拆分为多个数据单元；
传输模块，用于通过多个数据流，向所述接收端传输所述多个数据单元
第三方面，提供了一种网络设备，所述网络设备包括处理器和存储器，所述存储器中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由所述处理器加载并执行以实现如第一方面所述的数据传输的方法。
第四方面，提供了一种计算机可读存储介质，所述存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由处理器加载并执行以实现如第一方面所述的数据传输的方法。
本申请实施例提供的技术方案带来的有益效果是：本申请实施例中，发送端与接收端建立通信连接；将待传输数据包拆分为多个数据单元；通过多个数据流，向接收端传输多个数据单元。这样，发送端侧对待传输数据包进行多层拆分得到数据单元，并以数据单元的形式通过多个数据流进行传输，接收端侧接收数据单元并将数据单元拼装成小数据包，一方面，在数据发送前和接收 后仅需加载小量的数据单元，无需加载全部数据包，可以节省发送端和接收端的内存资源，另一方面，可以有效避免传输过程中的队头阻塞，并可以降低传输过程中的时延，减少网络因素导致的连接重连、数据重传等造成的影响，从而提高数据传输质量，减少传输过程占用的带宽资源。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的一种数据传输的方法流程图；
图2是本申请实施例提供的一种连接迁移的方法流程图；
图3是本申请实施例提供的一种数据重传的方法流程图；
图4是本申请实施例提供的一种数据传输的装置结构示意图；
图5是本申请实施例提供的另一种数据传输的装置结构示意图；
图6是本申请实施例提供的一种网络设备的结构示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施例作详细描述。
本申请实施例提供了一种数据传输的方法，该方法可以由发送端和接收端共同实现，其中，发送端和接收端均可以是具备数据收发和处理功能的网络设备，如可以是服务器、计算机等。发送端与接收端上可以部署有RPC框架，二者通过RPC框架可以实现数据传输和函数的远程调用。在调用过程中，发送端可以向接收端发送调用请求，并在调用请求中添加需要调用的函数以及调用函数时所需的参数。在本实施例中，发送端可以以数据包的形式向接收端发送 调用请求，当数据包的数据量过大时，发送端可以先对数据包进行拆分，然后以拆分后的规格将数据包传输至接收端。接收端接收到拆分后数据包后，可以先执行拼装处理，然后对拼装得到的数据包进行后续处理。值得一提的是，在不同实施例中，发送端和接收端的功能可以互换，即接收端可以向发送端发送调用请求，以实现对发送端上方法函数的远程调用，其具体过程与发送端发起远程调用的过程基本一致，本实施例不再赘述。
下面将结合具体实施例，对图1所示的处理流程进行详细的说明，内容可以如下。
步骤101，发送端与接收端建立通信连接。
在实施中，在RPC框架下的发送端需要远程调用接收端的方法函数时，发送端可以先与接收端建立TCP连接。该TCP连接可以是按需连接，即发送端需要执行远程调用时主动与接收端发起建立，在调用结束后发送端主动断开连接；也可以是长连接，即发送端和接收端建立起连接之后长期保留，无论是否存在数据包的发送，多次远程调用过程可以使用同一个TCP连接。之后，发送端可以与接收端在上述TCP连接上建立多个数据流，其中，每个数据流可以在发送端和接收端上对应一个线程，不同线程用于在不同的数据流上发送或接收数据。上述多个数据流可以是相互间不存在干扰影响的双向流，流与流之间相互独立，单个数据流上的数据传输阻塞及失败重传不会影响同一TCP连接上其它数据流的数据传输，发送端可通过数据流向接收端发送数据，反之接收端也可通过上述数据流向发送端发送数据。在本实施例中，在建立数据流后，发送端和接收端可以通过周期性发送探活数据包的形式检测每条数据流的健康状态，以根据检测结果对数据传输进行及时调整。
在一个实施例中，发送端和接收端可以根据双方的负载信息在通信连接上建立多个数据流，相应的处理可以如下：发送端向接收端发送携带有发送端负载信息的连接创建请求；接收端与发送端建立通信连接，并基于发送端负载信息和接收端负载信息在通信连接上初始化多个数据流。
在实施中，发送端在向接收端发送连接创建请求时，可以检测本地内存、带宽、CPU等发送端负载信息，并将检测到的发送端负载信息添加至连接创建请求中。接收端在接收到发送端发送的携带有发送端负载信息的连接创建请求之后，可以与发送端建立通信连接，并可以同样采集本地内存、带宽、CPU等接收端负载信息，然后可以基于发送端负载信息和接收端负载信息在通信连接上初始化多个数据流。具体来说，通信连接上数据流的个数可以默认等同为接收端的CPU核数，再参考接收端和发送端上各CPU的负载，以及最近5分钟内的内存和带宽使用状况，计算出数据流的个数和每个数据流的流量窗口大小。这样，基于发送端和接收端的负载状态决定数据流的个数和大小，可以充分利用发送端和接收端的设备资源，以提高数据传输效率，并可以防止由于设备过载而影响数据的传输质量。
步骤102，发送端将待传输数据包拆分为多个数据单元。
在实施中，发送端在与接收端建立了基于通信连接的多条数据流之后，可以通过多条数据流向接收端发送数据包。具体的，发送端可以先对待传输数据包进行拆分处理，以得到多个数据量较小的小数据包。之后，发送端可以对小数据包进行压缩，再将每个小数据包继续拆分为数据规格更小的多个数据单元。当然，在不同实施例中，发送端还可以基于相同原理对数据单元进行更进一步的拆分，以得到数据量更小的传输单元，从而可以缓解数据包过大造成的传输瞬时压力。
在一个实施例中，发送端在对大数据包进行拆包时，可以基于数据包大小、类型，以及发送端和接收端的负载来确定拆包规则，相应的，步骤102的处理可以如下：发送端基于待传输数据包的大小、类型，以及发送端负载信息和接收端负载信息，确定数据包拆分规则；发送端根据数据包拆分规则将待传输数据包拆分为多个小数据包，并将每个小数据包拆分为多个数据单元。
在实施中，发送端在对待传输数据包进行拆包处理时，可以先确定待传输数据包的大小和类型，然后根据类型和大小决定对应的拆分数据包的方案。 同时，发送端还可以参考接收端负载信息和发送端负载信息，确定出数据包拆分规则。进而，发送端可以根据数据包拆分规则将待传输数据包拆分为多个小数据包，并将每个小数据包拆分为多个数据单元。其中，接收端负载信息可以是发送端与接收端建立通信连接时，由接收端发送至发送端的。需要说明的是，负载较高时，为了避免数据流传输阻塞，可以尽量选取传输数量更小的包，故而可以将待传输数据包拆分为数量更多，数据量更小的包。
在一个实施例中，发送端在对数据包进行拆包之后，可以为拆分得到每个数据单元配置成指示具体数据内容的单元ID，相应的，步骤102之后可以存在如下处理：对于任一数据单元，发送端根据数据单元所属小数据包的小包ID和数据单元在小数据包中的数据偏移量，生成数据单元的单元ID，并将单元ID添加至数据单元。
在实施中，发送端将待传输数据包进行多次拆分，得到多个数据单元之后，可以对每个数据单元添加单元ID，用于标记每个数据单元，其中，多个数据单元的单元ID可以依序递增。具体的，对于任一数据单元，发送端可以先确定该数据单元所属的小数据包，以及数据单元在小数据包中的数据偏移量，然后基于预设算法根据小数据包的小包ID和上述数据偏移量，生成该数据单元的单元ID，进而可以将单元ID添加至数据单元中。这样，通过不同的单元ID可以对每个数据单元进行唯一标识，并且通过单元ID可以快速确定每个数据单元所对应的具体数据内容。值得一提的是，小包ID可以是发送端在拆分待传输数据包时，针对每个拆分得到的小数据包生成的，用于唯一标识小数据包，且发送端在生成小包ID时，可以进一步参考小数据包在待传输数据包中的偏移位置，以通过小包ID可以确定小数据包在待传输数据包中对应的具体数据内容。
步骤103，发送端通过多个数据流，向接收端传输多个数据单元。
在实施中，发送端对待传输数据包进行多次拆分得到数据单元后，可以通过与接收端建立的多个数据流，将拆分得到的数据单元传输至接收端。其中，一个流可以传输多个数据单元，一个小数据包的多个数据单元可在多个流之间 传输，在传输过程中，发送端可以根据每个数据流的负载状态，在不同数据流上传输不同数据量大小的数据单元。
步骤104，接收端接收并缓存发送端通过多个数据流传输的数据单元。
在实施中，接收端在与发送端建立了基于通信连接的多个数据流后，可以通过多个数据流对应的数据接收线程，接收发送端通过上述多个数据流传输的数据单元，然后可以将接收到的数据单元缓存至统一的缓冲区。在本实施例中，接收端可以设置有多个缓冲区，每个缓冲区可以用于缓存不同小数据包的数据单元。
步骤105，接收端将缓存的数据单元拼装成小数据包，并对小数据包执行应用层处理。
在实施中，接收端在接收并缓存数据单元的同时，可以对已缓存的数据单元进行拼装处理。当已缓存的多个数据单元可以成功拼装成一个完整的小数据包时，接收端可以将拼装得到的小数据包传输至应用层进行处理，并且同时删除缓冲区内相关的数据单元。
在一个实施例中，基于发送端在数据单元中添加单元ID的处理，步骤105的处理可以具体如下：对于缓存的每个数据单元，根据数据单元中携带的单元ID，确定数据单元对应的小包ID和数据单元在小数据包中的数据偏移量；根据每个数据单元对应的小包ID和数据偏移量，将缓存的数据单元拼装成小数据包。
在实施中，接收端通过多个数据流接收到发送端发送的数据单元并进行缓存后，可以提取数据单元中携带的单元ID，然后根据单元ID完成各个小数据包的拼装处理。上述过程中，对于缓存的每个数据单元，服务器可以基于预设算法由数据单元携带的单元ID还原出数据单元对应的小包ID和数据单元在小数据包中的数据偏移量，然后可以按照小包ID将归属不同小数据包的数据单元分别进行汇总，进而可以根据每个数据单元对应的数据偏移量，将对应同一小包ID的所有数据单元拼装成小数据包。
在一个实施例中，发送端在与接收端建立数据流后，可以向接收端发送 用于唯一标识每个数据流的数据流ID，以用于实现发送端和接收端间的连接迁移，故而发送端在向接收端发送连接创建请求之前，可以生成多个数据流的数据流ID，然后将所有数据流ID添加至连接创建请求中。接收端在接收到连接创建请求，并与发送端建立了通信连接后，可以将该通信连接的参数信息与上述数据流ID对应存储。
而基于上述数据流ID的处理，发送端和接收端间还可以包含如图2所示流程。
步骤201，当与接收端的通信连接中断时，发送端向接收端发送携带有多个数据流的数据流ID的连接重建请求。
在实施中，发送端在向接收端发送数据的过程中，如果检测到与接收端的通信连接中断，则可以获取该通信连接建立前所生成的多个数据流的数据流ID，并向接收端发送携带有这些数据流ID的连接重建请求。
步骤202，当接收到发送端发送的携带有数据流ID的连接重建请求时，接收端恢复数据流ID对应的通信连接，并在该通信连接上恢复多个数据流，再基于多个数据流继续传输数据单元。
在实施中，接收端可以接收发送端发送的连接重建请求，判断其中是否携带有数据流ID。如果携带，接收端则可以根据已存储的数据流ID和通信连接的参数信息，恢复该数据流ID对应的通信连接，然后在该通信连接上恢复多个数据流，再基于多个数据流继续传输数据单元。不难理解，上述数据流ID与发送端所处网络(IP地址、端口等信息)无关，当发送端所处网络发生变化时，数据流ID不变，故而接收端可以使用数据流ID确定原通信连接，仍通过原数据流与发送端进行数据传输，从而可以实现连接迁移，避免连接多次创建，减少因连接重建导致不必要的数据重传，保证业务逻辑不被中断。
在一个实施例中，发送端可以在将待传输数据包拆分为小数据包后，缓存每个小数据包在待传输数据包中的偏移位置，以便基于偏移位置和前述单元ID进行如图3所示的数据重传处理，具体流程可以如下。
步骤301，当检测到目标数据单元传输失败，或接收的目标数据单元的数据不完整时，接收端向发送端反馈携带有目标数据单元的单元ID的数据重传请求。
在实施中，接收端在接收发送端发送的数据单元时，可以根据数据单元的单元ID确定每个数据单元的总数据量，然后可以利用总数据量和数据单元的已接收量判断接收到的数据单元是否完整。同时，考虑到发送端按照单元ID依序发送数据单元，数据正常传输的情况下接收端应当依序接收数据单元，故而可以设定接收端在接收到数据单元A之前，已累计接收到预设数目个发送顺序在数据单元A之后的数据单元，则判定数据单元A传输失败。在本实施例中，当检测到目标数据单元传输失败，或接收的目标数据单元的数据不完整时，接收端可以向发送端反馈携带有目标数据单元的单元ID的数据重传请求，以使发送端重新发送目标数据单元。
步骤302，当接收到接收端发送的携带有目标数据单元的单元ID的数据重传请求时，发送端根据单元ID确定目标数据单元对应的目标小数据包和数据偏移量。
步骤303，发送端基于目标小数据包在待传输数据包中的偏移位置和数据偏移量，重新生成目标数据单元，并向接收端发送目标数据单元。
在实施中，发送端在确定目标数据单元所属的目标小数据包后，可以根据缓存内容判断目标小数据包在待传输数据包中的偏移位置，然后结合目标数据单元在目标小数据包中的数据偏移量，确定出目标数据单元对应的具体数据内容，进而可以重新生成目标数据单元。之后，发送端可以将目标数据单元通过通信连接上的任一数据流传输至接收端。这样，当需要重传某个数据单元时，发送端无需重新加载整个待传输数据包，便可确定并生成需要重传的数据内容，从而可以减少数据重传设备资源的消耗，并且提高数据重传的效率。当然，为了便于接收端尽快完成目标数据单元所属目标小数据包的拼装处理，发送端可以在目标数据单元上添加优先级信息，从而可以触发接收端优先接收并处理目 标数据单元。此外，发送端在发送数据单元的同时，还可以根据各数据单元所属的数据类型(如关键数据、补充数据等)，和/或数据流的当前负载情况，为不同数据单元赋予不同的传输优先级，从而发送端可以优先发送传输优先级较高的数据单元。例如，数据流的当前负载较高，只能传输数据量较小的数据单元，则可以为数据量较小的数据单元设置较高的传输优先级，反之亦成立。
在本实施例中，发送端还可以基于每个数据流的数据传输情况，决定是否增删数据流，相应的处理可以如下：发送端根据多个数据流的数据传输情况，向接收端发送数据流新增请求；当接收到发送端发送的数据流新增请求时，接收端基于通信连接创建新的数据流；或者，
发送端根据所述多个数据流的数据传输情况和/或设备负载状况，向所述接收端发送数据流中断请求；当接收到发送端发送的数据流中断请求时，接收端中断数据流中断请求对应的目标数据流，停止接收目标数据流上的数据单元。
在实施中，发送端通过通信连接上的多个数据流向接收端发送数据单元时，可以实时监测多个数据流的数据传输情况，并且可以根据监测结果，判断是否执行数据流增加处理，即决定是否向接收端发送数据流新增请求。举例来说，当检测到多个数据流的负载长期高于预设过载阈值时，则可以适当增加数据流，故而发送端可以向接收端发送数据流新增请求。
相似的，发送端通过通信连接上的多个数据流向接收端发送数据单元时，可以实时监测多个数据流的数据传输情况和设备负载状况(包括发送端负载状况和接收端负载状况)，并且可以根据监测结果，判断是否执行数据流删除处理，即决定是否向接收端发送数据流中断请求。举例来说，当检测到多个数据流的负载长期低于预设空闲阈值时，则可以对数据流进行适当删减，故而发送端可以向接收端发送对应一个或多个数据流的数据流中断请求，该数据流中断请求中可以携带有需要中断的数据流的数据流ID；或者，当检测到数据流B的数据传输质量较差时，发送端则可以向接收端发送对应数据流B的数据流中断请求，同时，还可以向接收端发送携带有数据流B的数据流ID的数据流新增请求，以 使接收端与发送端在创建了新数据流后，将数据流B上的数据转移至新数据流上进行传输。
需要说明的是，对多个数据流的检测是实时的，当符合增加或减少数据流要求时，发送端与接收端进行数据流的增加或中断。
基于上述发送端的处理，接收端在接收到发送端发送的数据流新增请求时，可以先基于两者间的通信连接创建新的数据流，然后根据数据流新增请求中流标识的携带情况，判断是否执行数据传输的迁移处理。而接收端在接收到发送端发送数据流中断请求时，可以确定数据流中断请求携带的流标识所对应的目标数据流，然后对目标数据流进行关闭操作，并停止接收发送端后续通过目标数据流传输过来的数据单元。这样，通过对数据流的增/删处理，既可以避免资源闲置，保证资源利用率，又可以防止因数据流过载而影响数据的传输质量。
值得一提的是，发送端可以通过在数据流新增请求中添加优先级、流量窗口大小等信息的方式对需要创建的新数据流进行参数设置，这样，接收端在接收到数据流新增请求后，可以基于其中添加的信息与发送端创建符合相应参数的新数据流。
在一个实施例中，接收端通过向发送端发送流量调整指令的方式实现多个数据流间数据传输的负载均衡，相应的处理可以如下：接收端根据多个数据流对应的接收负载情况，向发送端发送流量调整指令；发送端接收接收端发送的流量调整指令，根据流量调整指令携带的流标识和指定流量阈值，调整数据流的流量窗口大小。
其中，流量调整指令可以携带有流标识和指定流量阈值。
在实施中，接收端在接收发送端方的数据单元时，可以检测每个数据流对应的接收负载状态，然后根据该接收负载状态，向发送端发送流量调整指令，该流量调整指令可以携带有流标识和指定流量阈值。例如，当接收端检测到数据流处于过载状态时，则可以向发送端发送针对该数据流的流量调整指令，以 使发送端缩小该数据流的流量窗口大小；当接收端检测到数据流处于空闲状态时，则可以向发送端发送针对该数据流的流量调整指令，以使发送端放大该数据流的流量窗口大小。故而，发送端在接收到接收端发送的流量调整指令后，可以根据流量调整指令携带的流标识，确定需要调整的数据流，再基于流量调整指令携带的指定流量阈值，对数据流的流量窗口大小进行调整。这样，通过流量调整指令可以有效避免多个数据流间负载不均，提高了传输过程中的网络资源利用率，并且可以防止因数据流过载而导致数据传输失败。
本申请实施例中，发送端与接收端建立通信连接；将待传输数据包拆分为多个数据单元；通过多个数据流，向接收端传输多个数据单元。这样，发送端侧对待传输数据包进行多层拆分得到数据单元，并以数据单元的形式通过多个数据流进行传输，接收端侧接收数据单元并将数据单元拼装成小数据包，一方面，在数据发送前和接收后仅需加载小量的数据单元，无需加载全部数据包，可以节省发送端和接收端的内存资源，另一方面，可以有效避免传输过程中的队头阻塞，并可以降低传输过程中的时延，减少网络因素导致的连接重连、数据重传等造成的影响，从而提高数据传输质量，减少传输过程占用的带宽资源。
基于相同的技术构思，本申请实施例还提供了一种数据传输的装置，如图4所示，所述装置包括：
连接模块401，用于与接收端建立通信连接；
拆分模块402，用于将待传输数据包拆分为多个数据单元；
传输模块403，用于通过多个数据流，向所述接收端传输所述多个数据单元。
基于相同的技术构思，本申请实施例还提供了一种数据传输的装置，如图5所示，所述装置包括：
连接模块501，用于与发送端建立通信连接和基于所述通信连接的多个数据流；
接收模块502，用于接收并缓存所述发送端通过所述多个数据流传输的数据单元；
处理模块503，用于将缓存的数据单元拼装成小数据包，并对所述小数据包执行应用层处理。
基于相同的技术构思，本申请实施例还提供了一种数据传输的系统，所述系统包括发送端和接收端，其中，所述发送端和接收端协同实现上述实施例所述的数据传输的方法。
图6是本申请实施例提供的网络设备的结构示意图。该网络设备600可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上中央处理器622(例如，一个或一个以上处理器)和存储器632，一个或一个以上存储应用程序642或数据644的存储介质630(例如一个或一个以上海量存储设备)。其中，存储器632和存储介质630可以是短暂存储或持久存储。存储在存储介质630的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对网络设备600中的一系列指令操作。在本实施例中，中央处理器622可以设置为与存储介质630通信，在网络设备600上执行存储介质630中的一系列指令操作。
网络设备600还可以包括一个或一个以上电源629，一个或一个以上有线或无线网络接口650，一个或一个以上输入输出接口658，一个或一个以上键盘656，和/或，一个或一个以上操作系统641，例如Windows Server，Mac OS X，Unix，Linux，FreeBSD等等。
网络设备600可以包括有存储器，以及一个或者一个以上的程序，其中一个或者一个以上程序存储于存储器中，且经配置以由一个或者一个以上处理器执行所述一个或者一个以上程序包含用于进行上述数据传输的指令。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成，也可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，上述提到的存储介质可以是只读存储器，磁盘或光盘等。
以上所述仅为本申请的部分实施例，并不用以限制本申请，凡在本申请 的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (12)

  1. 一种数据传输的方法，包括：
    与接收端建立通信连接；
    将待传输数据包拆分为多个数据单元；
    通过多个数据流，向所述接收端传输所述多个数据单元。
  2. 根据权利要求1所述的方法，其中，所述与接收端建立通信连接，包括：
    向接收端发送携带有发送端负载信息的连接创建请求，以使所述接收端与发送端建立通信连接，并基于所述发送端负载信息在所述通信连接上初始化多个数据流。
  3. 根据权利要求2所述的方法，其中，所述连接创建请求还携带有发送端生成的所述多个数据流的数据流ID；
    所述通过所述多个数据流，向所述接收端传输所述多个数据单元之后，还包括：
    当与所述接收端的通信连接中断时，向所述接收端发送携带有所述数据流ID的连接重建请求，以使所述接收端根据所述数据流ID恢复所述多个数据流，并基于所述多个数据流继续传输所述多个数据单元。
  4. 根据权利要求1或2所述的方法，其中，所述将待传输数据包拆分为多个数据单元，包括：
    基于待传输数据包的大小、类型，以及发送端负载信息和接收端负载信息，确定数据包拆分规则；
    根据所述数据包拆分规则将待传输数据包拆分为多个小数据包，并将每个所述小数据包拆分为多个数据单元。
  5. 根据权利要求4所述的方法，其中，所述将每个所述小数据包拆分为多个数据单元之后，还包括：
    对于任一数据单元，根据所述数据单元所属小数据包的小包ID和所述数据 单元在小数据包中的数据偏移量，生成所述数据单元的单元ID，并将所述单元ID添加至所述数据单元。
  6. 根据权利要求4所述的方法，其中，所述将待传输数据包拆分为多个小数据包之后，还包括：
    缓存每个所述小数据包在所述待传输数据包中的偏移位置；
    所述方法还包括：
    当接收到所述接收端发送的携带有目标数据单元的单元ID的数据重传请求时，根据所述单元ID确定所述目标数据单元对应的目标小数据包和数据偏移量；
    基于所述目标小数据包在所述待传输数据包中的偏移位置和所述数据偏移量，重新生成所述目标数据单元，并向所述接收端发送所述目标数据单元。
  7. 根据权利要求1所述的方法，其中，所述方法还包括：
    根据所述多个数据流的数据传输情况，向所述接收端发送数据流新增请求，以使所述接收端在接收到所述数据流新增请求后，基于所述通信连接创建新的数据流。
  8. 根据权利要求1所述的方法，其中，所述方法还包括：
    根据所述多个数据流的数据传输情况和/或设备负载状况，向所述接收端发送数据流中断请求，以使所述接收端在接收到所述数据流中断请求后，中断所述数据流中断请求对应的目标数据流，停止接收所述目标数据流上的数据单元。
  9. 根据权利要求1所述的方法，其中，所述方法还包括：
    接收所述接收端发送的流量调整指令，根据所述流量调整指令携带的流标识和指定流量阈值，调整所述数据流的流量窗口大小。
  10. 一种数据传输的装置，包括：
    连接模块，用于与接收端建立通信连接；
    拆分模块，用于将待传输数据包拆分为多个数据单元；
    传输模块，用于通过多个数据流，向所述接收端传输所述多个数据单元。
  11. 一种网络设备，所述网络设备包括处理器和存储器，所述存储器中存 储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由所述处理器加载并执行以实现如下处理：
    与接收端建立通信连接；
    将待传输数据包拆分为多个数据单元；
    通过多个数据流，向所述接收端传输所述多个数据单元。
  12. 一种计算机可读存储介质，所述存储介质中存储有至少一条指令、至少一段程序、代码集或指令集，所述至少一条指令、所述至少一段程序、所述代码集或指令集由处理器加载并执行以实现如下处理：
    与接收端建立通信连接；
    将待传输数据包拆分为多个数据单元；
    通过多个数据流，向所述接收端传输所述多个数据单元。
PCT/CN2020/079351 2019-12-24 2020-03-13 一种数据传输的方法和装置 WO2021128602A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20775796.4A EP3873067A4 (en) 2019-12-24 2020-03-13 DATA TRANSFER METHOD AND DEVICE
US17/061,517 US11303737B2 (en) 2019-12-24 2020-10-01 Method and device for data transmission

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911347142.4A CN111147573A (zh) 2019-12-24 2019-12-24 一种数据传输的方法和装置
CN201911347142.4 2019-12-24

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/061,517 Continuation US11303737B2 (en) 2019-12-24 2020-10-01 Method and device for data transmission

Publications (1)

Publication Number Publication Date
WO2021128602A1 true WO2021128602A1 (zh) 2021-07-01

Family

ID=70519699

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/079351 WO2021128602A1 (zh) 2019-12-24 2020-03-13 一种数据传输的方法和装置

Country Status (4)

Country Link
US (1) US11303737B2 (zh)
EP (1) EP3873067A4 (zh)
CN (1) CN111147573A (zh)
WO (1) WO2021128602A1 (zh)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220191256A1 (en) * 2020-12-16 2022-06-16 Ncr Corporation Agnostic data collection platform
CN113438274A (zh) * 2021-05-26 2021-09-24 曙光网络科技有限公司 数据传输方法、装置、计算机设备及可读存储介质
CN114338227B (zh) * 2022-01-21 2023-04-18 山东大学 基于分割流量的网络流量分析对抗方法及装置
CN115021873A (zh) * 2022-06-13 2022-09-06 浙江大华技术股份有限公司 一种数据重传的方法、装置及电子设备
CN115052044B (zh) * 2022-08-12 2022-11-18 南瑞轨道交通技术有限公司 一种基于双网双链路的海量数据传输方法
CN116264565B (zh) * 2023-05-16 2023-08-04 山东省邱集煤矿有限公司 一种基于语义拆分的煤矿井下数据传输方法、装置及设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102523225A (zh) * 2011-12-22 2012-06-27 浙江国自机器人技术有限公司 分布式松耦合构架中的解耦方法及分布式松耦合构架平台
US20150237060A1 (en) * 2008-10-08 2015-08-20 Cisco Technology, Inc. Target-based smb and dce/rpc processing for an intrusion detection system or intrusion prevention system
CN107948215A (zh) * 2018-01-17 2018-04-20 广州汇智通信技术有限公司 一种基于udp通信的远程调用方法和装置
CN109951260A (zh) * 2018-02-12 2019-06-28 华为技术有限公司 一种数据包发送方法及相关设备
CN109981674A (zh) * 2019-04-04 2019-07-05 北京信而泰科技股份有限公司 一种远程过程调用方法、装置、设备及介质

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5557798A (en) * 1989-07-27 1996-09-17 Tibco, Inc. Apparatus and method for providing decoupling of data exchange details for providing high performance communication between software processes
WO2011022405A2 (en) * 2009-08-17 2011-02-24 Akamai Technologies, Inc. Method and system for http-based stream delivery
DK2661845T3 (da) * 2011-01-04 2014-11-10 Napatech As Apparat og fremgangsmåde til modtagelse og videresendelse af data
CN103457973B (zh) * 2012-06-01 2016-04-27 深圳市腾讯计算机系统有限公司 一种图片上传方法、系统、图片上传客户端及网络服务器
US10904144B2 (en) * 2012-12-27 2021-01-26 Sitting Man, Llc Methods, systems, and computer program products for associating a name with a network path
WO2018211488A1 (en) * 2017-05-18 2018-11-22 Liveu Ltd. Device, system, and method of wireless multiple-link vehicular communication
US10986029B2 (en) * 2014-09-08 2021-04-20 Liveu Ltd. Device, system, and method of data transport with selective utilization of a single link or multiple links
US10397674B2 (en) * 2015-12-01 2019-08-27 Adtran, Inc. PON wavelength bonding for providing higher-rate data services
US10243837B2 (en) * 2016-06-06 2019-03-26 Argela Yazilim ve Bilisim Teknolojileri San. ve Tic. A.S. Enabling split sessions across hybrid public safety and LTE networks
CN108243211A (zh) * 2016-12-24 2018-07-03 华为技术有限公司 一种数据传输方法及装置
KR101983088B1 (ko) * 2017-06-23 2019-05-31 (주)넷비젼텔레콤 다중 경로 환경에서의 udp 패킷 처리 방법
KR102021671B1 (ko) * 2017-12-11 2019-09-16 건국대학교 산학협력단 SDN에서 애플리케이션 레이어의 통신을 위한 노스바운드 API로써의 gRPC
CN110391989B (zh) * 2018-04-17 2023-03-03 网宿科技股份有限公司 一种进行数据传输的方法和装置
CN109474642A (zh) * 2019-01-08 2019-03-15 郑州云海信息技术有限公司 基于以太网udp协议数据传输方法、装置、设备及介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150237060A1 (en) * 2008-10-08 2015-08-20 Cisco Technology, Inc. Target-based smb and dce/rpc processing for an intrusion detection system or intrusion prevention system
CN102523225A (zh) * 2011-12-22 2012-06-27 浙江国自机器人技术有限公司 分布式松耦合构架中的解耦方法及分布式松耦合构架平台
CN107948215A (zh) * 2018-01-17 2018-04-20 广州汇智通信技术有限公司 一种基于udp通信的远程调用方法和装置
CN109951260A (zh) * 2018-02-12 2019-06-28 华为技术有限公司 一种数据包发送方法及相关设备
CN109981674A (zh) * 2019-04-04 2019-07-05 北京信而泰科技股份有限公司 一种远程过程调用方法、装置、设备及介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3873067A4 *

Also Published As

Publication number Publication date
EP3873067A1 (en) 2021-09-01
CN111147573A (zh) 2020-05-12
US11303737B2 (en) 2022-04-12
US20210195000A1 (en) 2021-06-24
EP3873067A4 (en) 2021-09-01

Similar Documents

Publication Publication Date Title
WO2021128602A1 (zh) 一种数据传输的方法和装置
US9781052B2 (en) Virtual machine and application movement over local area networks and a wide area network
US10355997B2 (en) System and method for improving TCP performance in virtualized environments
US8553572B2 (en) Internet protocol optimizer
EP1175065B1 (en) Method and system for improving network performance enhancing proxy architecture with gateway redundancy
CN103916405B (zh) 一种IOS上针对App的TCP/UDP流量引导的方法
WO2022247550A1 (zh) 数据重传处理方法、装置、计算机设备和存储介质
US9516114B2 (en) Data packet transmission method and related device and system
JP2004526218A (ja) 相互接続されたファブリックにおける高度にスケーラブルで高速のコンテンツ・ベース・フィルタリング及び負荷均衡化システム及び方法
US10432530B2 (en) System and method of providing compression technique for jitter sensitive application through multiple network links
US20140317173A1 (en) Data communication system and method
CN104683259A (zh) Tcp拥塞控制方法及装置
US20200220800A1 (en) Data Distribution Method and Distribution Server
WO2017107148A1 (zh) 一种数据传输方法及网络侧设备
US20150055482A1 (en) TCP Extended Fast Recovery and Segment Timing
CN112217735A (zh) 信息同步方法与负载均衡系统
US11044350B1 (en) Methods for dynamically managing utilization of Nagle's algorithm in transmission control protocol (TCP) connections and devices thereof
JP4506430B2 (ja) アプリケーションモニタ装置
CN115277806A (zh) 云桌面连接方法、装置和设备
CN115766605A (zh) 网络拥塞控制方法、装置及系统
CN117813595A (zh) 用于远程直接存储器访问的设备和方法
WO2020154872A1 (zh) 一种传输控制协议加速方法和装置
JP2005011267A (ja) リアルタイムデータ通信システム、リアルタイムデータ通信装置およびリアルタイムデータ通信方法
CN117640530A (zh) 集群网络的低延迟可靠数据传输方法和装置
JP2006087010A (ja) 通信路制御装置およびそれを用いたネットワークシステム

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2020775796

Country of ref document: EP

Effective date: 20201001

NENP Non-entry into the national phase

Ref country code: DE