CN116566973A - File transmission system based on peer-to-peer network - Google Patents

File transmission system based on peer-to-peer network Download PDF

Info

Publication number
CN116566973A
CN116566973A CN202310731255.4A CN202310731255A CN116566973A CN 116566973 A CN116566973 A CN 116566973A CN 202310731255 A CN202310731255 A CN 202310731255A CN 116566973 A CN116566973 A CN 116566973A
Authority
CN
China
Prior art keywords
service node
block
client
module
current
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310731255.4A
Other languages
Chinese (zh)
Other versions
CN116566973B (en
Inventor
蔡维恒
王琪
陈清华
潘海勇
李旭明
杨柳青
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Leadal Technology Development Co ltd
Original Assignee
Beijing Leadal Technology Development Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Leadal Technology Development Co ltd filed Critical Beijing Leadal Technology Development Co ltd
Priority to CN202310731255.4A priority Critical patent/CN116566973B/en
Publication of CN116566973A publication Critical patent/CN116566973A/en
Application granted granted Critical
Publication of CN116566973B publication Critical patent/CN116566973B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/02—Topology update or discovery
    • H04L45/03—Topology update or discovery by updating link state protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/12—Shortest path evaluation
    • H04L45/122—Shortest path evaluation by minimising distances, e.g. by selecting a route with minimum of number of hops
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
    • H04L67/1078—Resource delivery mechanisms
    • H04L67/108—Resource delivery mechanisms characterised by resources being split in blocks or fragments

Abstract

A file transmission system based on peer-to-peer network includes a plurality of service nodes and clients; the service node establishes peer-to-peer network connection through the adjacency module and monitors the connection state with the adjacency service node; the service node detects link information between adjacent service nodes through the detection module and plans the shortest path reaching other service nodes; the client establishes connection with the service node through a second connection module; the source client sends a connection request to the target client; if the target client confirms the connection request, a service node connected with the source client inquires network bandwidth occupation information through a detection module and sends the network bandwidth occupation information to the source client, and a blocking module of the source client blocks a file to be transmitted according to the network bandwidth occupation information; the sending module of the source client sends a block file data packet to the service node; and the service node forwards the block file data packet according to the planned path reaching the service node corresponding to the target client until reaching the target client.

Description

File transmission system based on peer-to-peer network
Technical Field
The invention relates to the technical field of file transmission, in particular to a file transmission system based on a peer-to-peer network.
Background
Along with the development of the internet age, the popularization of network informatization is that the demands for technology and information exchange in various fields are continuously increasing, and the demands for information exchange are also increasing.
From the computer technology perspective, the traditional way of file transfer can be roughly divided into 4 types: FTP, disk, file transfer software, email, etc. In the above file transmission manner, the inventor finds that the conventional file transmission manner is all required to rely on a specific service, that is, the sender uploads the file to be transmitted to the service, and the receiver downloads the file from the service; the specific efficiency of transmission mainly depends on the network of the service, and if the uplink and downlink bandwidths of the network of the service are lower than those of the sender and the receiver, the transmission efficiency is affected, so that the transmission efficiency is low. Particularly, when the number of connections of the transmission service is too large, the amount of data to be transmitted is too large, and the size of data, network conditions, and the like, which are accompanied by CPU, network IO, and memory resources required by the server to process the data, are increased in a nonlinear manner, so that the server is very easy to overflow the memory and the CPU full load, which are caused by backlog of the data, are not always guaranteed in terms of transmission efficiency.
Disclosure of Invention
In view of the above analysis, an embodiment of the present invention is directed to providing a peer-to-peer network based file transfer system, so as to solve the problem of low file transfer efficiency in the prior art.
In one aspect, an embodiment of the present invention provides a peer-to-peer network based file transfer system, including a plurality of service nodes and clients; each of the service nodes includes: the system comprises an adjacent module, a detection module, an analysis module, a forwarding module and a first connection module; each client comprises a second connection module, a blocking module, a sending module and a receiving module;
the service node establishes peer-to-peer network connection with other service nodes through the adjacent module and monitors the connection state with the adjacent service nodes through the heartbeat data packet;
the service node detects link information between adjacent service nodes through a detection module, and plans the shortest path reaching other service nodes based on the link information;
the first connection module is used for establishing connection with the client and maintaining a connection channel with the client;
the client establishes connection with the service node through a second connection module; the source client sends a connection request to the target client through the service node, and the target client receives the connection request through the receiving module and feeds back whether to confirm the connection or not to the source client through the sending module; if the target client confirms the connection request, a service node connected with the source client inquires network bandwidth occupation information through a detection module and sends the network bandwidth occupation information to the source client, and a blocking module of the source client blocks a file to be transmitted according to the network bandwidth occupation information;
The sending module of the source client sends a block file data packet to the target client through the service node;
the analysis module of the service node analyzes the destination address of the received block file data packet, if the current service node is not the service node corresponding to the target client, the forwarding module forwards the block file data packet to the next service node according to the planned shortest path of the current service node to the service node corresponding to the target client until the current service node is the service node corresponding to the target client; and if the current service node is the service node corresponding to the target client, the forwarding module sends the block file data packet to the target client.
Based on further improvement of the method, the adjacent module periodically sends heartbeat data packets to the adjacent service nodes and receives heartbeat response data packets of the adjacent service nodes, if no response data packets of the adjacent service nodes are received within a first threshold time, the sending frequency of the heartbeat data packets is increased, if no response data packets of the adjacent service nodes are received within a second threshold time, the connection between the current service node and the adjacent service nodes is judged to be disconnected, the adjacent module of the current service node updates a node link information table, and the updated node link information table is sent to other service nodes of the peer-to-peer network, wherein the first threshold time is smaller than the second threshold time.
Based on further improvement of the method, the client sends a connection request to the service node through the second connection module, wherein the connection request comprises a client address and a port;
after receiving the connection request, the first connection module of the service node generates an ID for the client, stores the client ID, address and port in a connection information table, returns connection confirmation information to the second connection module of the client, and establishes connection with the client.
Based on the further improvement of the method, the service node connected with the source client is taken as a first node, and the service node connected with the target client is taken as a second node;
a service node connected with a source client queries network bandwidth occupation information through a detection module, comprising:
the detection module of the first service node inquires and acquires a planned shortest path from the first service node to the second service node;
and taking the residual bandwidth of the link with the highest network bandwidth occupancy rate on the shortest path as the network bandwidth occupancy information obtained by inquiry.
Based on the further improvement of the method, the blocking module of the source client blocks the file to be transmitted according to the network bandwidth occupation information, and the method comprises the following steps:
setting a file block threshold according to the network bandwidth occupation information;
Taking the starting position of the file to be transmitted as the current starting position;
the sliding window is arranged at the current starting position;
calculating a fingerprint value of data in the current sliding window, judging whether the sliding window is positioned at a block boundary or not based on the fingerprint value, if not, sliding the window forward until the sliding window is positioned at the block boundary or the end of a file to be transmitted; if the current window is positioned at the block boundary or the end of the transmission file, taking the data from the current starting position to the cut-off position of the current sliding window as the current block;
if the current block is smaller than or equal to the file block threshold value, calculating the fingerprint of the current block, and if the current block is the same as the current block fingerprint, only storing the index value of the same block as the current block as the content of the current block; otherwise, storing the data and the fingerprint of the current block as the content of the current block;
if the current block is larger than the file block threshold, dividing the current block into a plurality of sub-blocks according to the file block threshold, sequentially calculating fingerprints of each sub-block, and if the blocks identical to the sub-block fingerprints exist, only storing index values of the blocks identical to the fingerprints as contents of the sub-blocks; otherwise, storing the data and the fingerprint of the sub-block as the contents of the sub-block;
If the current sliding window is at the end of the file to be transmitted, ending the partitioning; otherwise, taking the next position of the current sliding window as the current initial position; and returning to the step of placing the sliding window at the current starting position.
Based on a further improvement of the method, the following mode is adopted to judge whether the same block as the current block fingerprint exists or not:
calculating the hash value of the current block as a weak fingerprint value of the current block, judging whether the stored block has the same block as the weak fingerprint value of the current block, and if so, judging that the stored block has the same block as the fingerprint of the current block; otherwise, calculating the Md5 value of the current block as the strong fingerprint of the current block, judging whether the stored block has the same block as the strong fingerprint value of the current block, if so, the stored block has the same block as the fingerprint of the current block; otherwise, there is no chunk identical to the current chunk fingerprint.
Based on a further improvement of the method, the detection module adopts the following steps to plan the shortest path to reach other service nodes based on the link information:
the detection module of each service node periodically transmits detection data packets to adjacent service nodes, and detects link information of links between the service nodes and each adjacent service node;
And calculating the shortest path from each service node to other service nodes by adopting a path planning algorithm based on the link information.
Based on the further improvement of the method, the link information comprises network delay, packet loss rate and bandwidth utilization rate of the link;
calculating the shortest path from each service node to other service nodes by adopting a path planning algorithm based on the link information, wherein the method comprises the following steps:
calculating network delay, packet loss rate and bandwidth utilization rate of a link based on the transmission time, transmission number and size of the probe data packets;
calculating the weight of the link based on the network delay, packet loss rate and bandwidth utilization of the link;
and calculating the shortest path from each service node to other service nodes according to the weight of each link.
Based on a further improvement of the method, according to the formula Weight ij ＝α×BL ij +β×Ploss ij +δ×Delay ij Computing Weight for links from service node i to service node j ij Wherein α, β and γ represent weight coefficients, delay ij Representing network delay, ploss, of a link from service node i to service node j ij Packet loss rate, BL, representing link from service node i to service node j ij Representing the bandwidth utilization of the link from serving node i to serving node j.
Based on the further improvement of the method, the Dijkstra algorithm is adopted to calculate the shortest path from each service node to other service nodes according to the weight of each link.
Compared with the prior art, the service node establishes connection through the adjacent module to construct a peer-to-peer network, the service node in the network is peer-to-peer, and the service node only performs connection and forwarding without reading and writing data, so that the problem of memory overflow caused by long-time data in a socket cache area is avoided; the connection state between the heartbeat packet detection and the adjacent node service node is periodically sent, so that a link information table can be updated when the network is abnormal, path information can be updated in time, and the reliability and stability of file transmission are ensured; the blocking module of the client transmits the file data in a blocking mode, so that the forwarding pressure of the service node is effectively reduced, the reliability of file transmission is improved, and the occurrence of the conditions of memory leakage and service full load of the service node is prevented. After receiving the data packet of the block file, the service node forwards the file to the next service node in the planned path according to the planned path, so that the optimal transmission path is selected in real time according to the network condition to perform data transmission, thereby avoiding the condition of network congestion and improving the transmission efficiency.
In the invention, the technical schemes can be mutually combined to realize more preferable combination schemes. Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, like reference numerals being used to designate like parts throughout the drawings;
fig. 1 is a block diagram of a peer-to-peer network based file transfer system according to an embodiment of the present invention.
Reference numerals:
1-a service node; 2-client.
Detailed Description
Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form a part hereof, and together with the description serve to explain the principles of the invention, and are not intended to limit the scope of the invention.
In one embodiment of the present invention, a peer-to-peer network based file transfer system is disclosed, as shown in fig. 1, comprising:
A plurality of service nodes and clients; each of the service nodes includes: the system comprises an adjacent module, a detection module, an analysis module, a forwarding module and a first connection module; each client comprises a second connection module, a blocking module, a sending module and a receiving module;
the service node establishes peer-to-peer network connection with other service nodes through the adjacent module and monitors the connection state with the adjacent service nodes through the heartbeat data packet;
the service node detects link information between adjacent service nodes through a detection module, and plans the shortest path reaching other service nodes based on the link information;
the first connection module is used for establishing connection with the client and maintaining a connection channel with the client;
the client establishes connection with the service node through a second connection module; the source client sends a connection request to the target client through the service node, and the target client receives the connection request through the receiving module and feeds back whether to confirm the connection or not to the source client through the sending module; if the target client confirms the connection request, a service node connected with the source client inquires network bandwidth occupation information through a detection module and sends the network bandwidth occupation information to the source client, and a blocking module of the source client blocks a file to be transmitted according to the network bandwidth occupation information;
The sending module of the source client sends a block file data packet to the target client through the service node;
the analysis module of the service node analyzes the destination address of the received block file data packet, if the current service node is not the service node corresponding to the target client, the forwarding module forwards the block file data packet to the next service node according to the planned shortest path of the current service node to the service node corresponding to the target client until the current service node is the service node corresponding to the target client; and if the current service node is the service node corresponding to the target client, the forwarding module sends the block file data packet to the target client.
The service nodes establish connection through the adjacent modules to construct a peer-to-peer network, the service nodes in the network are peer-to-peer, the service nodes only perform connection and forwarding, data reading and writing are not needed, and the problem of memory overflow caused by long-time data in a socket cache area is avoided; the connection state between the heartbeat packet detection and the adjacent node service node is periodically sent, so that a link information table can be updated when the network is abnormal, path information can be updated in time, and the reliability and stability of file transmission are ensured; the blocking module of the client transmits the file data in a blocking mode, so that the forwarding pressure of the service node is effectively reduced, the reliability of file transmission is improved, and the occurrence of the conditions of memory leakage and service full load of the service node is prevented. After receiving the data packet of the block file, the service node forwards the file to the next service node in the planned path according to the planned path, so that the optimal transmission path is selected in real time according to the network condition to perform data transmission, thereby avoiding the condition of network congestion and improving the transmission efficiency.
When the method is implemented, a plurality of service nodes are connected with other service nodes through the adjacent modules to build a peer-to-peer network structure. Each service node maintains a node link information table and a connection information table.
The node link information table is used for recording node information of service nodes in the current network and link information among the service nodes, and the node information comprises an ID, an address, a port and an adjacent service node ID of each service node; the link information includes network delay, packet loss rate, broadband occupancy of the link, and shortest path from each service node to other service nodes.
The connection information table is used for recording the client information connected with the current service node, and comprises the ID, the address and the port of the client.
The adjacent module of the service node periodically sends heartbeat data packets to the adjacent service node and receives heartbeat response data packets of the adjacent service node, if the response data packets of the adjacent service node are not received within a first threshold time, the sending frequency of the heartbeat data packets is increased, if the response data packets of the adjacent service node are not received within a second threshold time, the connection between the current service node and the adjacent service node is judged to be disconnected, the adjacent module of the current service node updates a node link information table of the current service node, and the updated node link information table is sent to other service nodes of the peer-to-peer network.
The service node monitors the connection relation through the heartbeat data packet, if the response data packet of the adjacent service node is not received within the first threshold time, the network abnormality is determined, whether the connection is disconnected is judged by increasing the sending frequency, if the response data packet of the adjacent service node is not received within the second threshold time, the connection with the adjacent node is judged to be disconnected, the node link information table of the current service node is updated, and the updated node link information table is sent to other service nodes of the peer-to-peer network, so that the whole network is updated.
Wherein the first threshold time is less than the second threshold time. The first threshold time and the second threshold time may be set according to network stability requirements.
In the implementation, in order to more accurately detect the network connection condition and improve the transmission efficiency, the detection module adopts the following steps to plan the shortest path reaching other service nodes:
s01, a detection module of each service node periodically transmits a detection data packet to adjacent service nodes, and detects link information of links between the service nodes and each adjacent service node;
s02, calculating the shortest path from each service node to other service nodes by adopting a path planning algorithm based on the link information.
Specifically, the link information includes network delay, packet loss rate, and bandwidth utilization of the link.
Each service node periodically (or as needed) transmits a probing data packet to its neighboring service node to probe the link information between the current service node and its neighboring service node. In practice, the probe packet includes a destination address, a type of probe packet (e.g., ICMP Echo request), and a size of the probe packet.
The service node that sends the probe packet records the time stamp at the time of sending, the service node that receives the probe packet records the time stamp at the time of receiving, and returns a response packet, e.g., an ICMP Echo response, to the sending service node that records the time stamp at the time of receiving the response packet. The service node transmitting the probe packet can calculate RTT (Round-Trip Time), i.e., round Trip Time, which is the Time it takes for the probe packet to be transmitted from the service node to the neighboring node and back to the service node, based on the transmitted and received Time stamps. Meanwhile, the service node transmitting the probe data packet records the number of the transmitted probe data packets and the number of the received response packets. The real-time traffic rate of the link between the current serving node and the neighboring node can be obtained by probing the size and return time of the packet.
Specifically, step S02 calculates, by using a path planning algorithm, a shortest path from each service node to other service nodes based on the link information, including:
s021, calculating network delay, packet loss rate and bandwidth utilization rate of a link based on the sending time, sending number and size of the detection data packets;
in practice, according toCalculating network Delay from service node i to service node j ij ；
According toCalculating packet loss rate Ploss between service node i and service node j ij ；
According toCalculating bandwidth utilization BL between service node i and service node j ij 。
Wherein, the liquid crystal display device comprises a liquid crystal display device,indicating the transmission time of the kth probe packet from service node i to service node j,/-, and>nps indicating the time when service node i receives service node j's response packet to the kth probe packet ij Indicating the number of probing packets sent from serving node i to serving node j, npr ij Representing the number of response packets received by service node j by service node i, +.>Real-time traffic rate, AB, representing the kth probe of service node i to service node j ij Representing the total bandwidth of the links from service node i to service node j, and K represents the number of probing packets sent from service node i to service node j.
S022, calculating the weight of the link based on the network delay, the packet loss rate, and the bandwidth utilization of the link.
Specifically, according to the formula Weight ij ＝α×BL ij +β×Ploss ij +δ×Delay ij Computing weights for links from service node i to service node j
According to the formula Weight ij ＝α×BL ij +β×Ploss ij +δ×Delay ij Computing Weight for links from service node i to service node j ij 。
Wherein, alpha, beta and gamma represent weight coefficients, which can be set according to the requirement, and the sum of the three weight coefficients is 1.
And each service node periodically updates a node link information table according to the detected link information, stores network delay, packet loss rate, bandwidth utilization rate and weight of each link in the node link information table, and calculates the shortest path from each service node to other service nodes according to the weight of each link.
It should be noted that, after each service node calculates the network delay, packet loss rate, bandwidth utilization rate and weight information of each link between the service node and the adjacent node, the information is broadcasted to other service nodes, so that service node information in the network is consistent.
In implementation, the Dijkstra algorithm is adopted to calculate the shortest path from each service node to other service nodes according to the weight of each link.
In practice, the client is connected to the peer-to-peer network by any one of the service nodes in the peer-to-peer network through the second connection module. For example, the source client and the target client establish a connection with the first service node and the second service node, respectively. It should be noted that any client may be used as a source client and a target client, which is only for distinguishing different file processing parties and is not a fixed client. The source client and the target client may be connected to different service nodes or may be connected to the same node, and thus, the first service node and the second service node may be different service nodes or may be the same service node.
Specifically, the client establishes a connection with the service node by:
the client sends a connection request to the service node through the second connection module, wherein the connection request comprises a client address and a port, the service node generates an ID for the client after receiving the connection request, stores the ID, the address and the port of the client in a connection information table, and returns connection confirmation information to the client, so that the client and the service node establish connection.
After establishing connection with the service node, the source client sends a file transmission request to the target client through the service node. The file transfer request includes a source client ID, a destination client ID, a file name, a file size, and an MD5 check value.
When the method is implemented, the service node further comprises a verification module, after the first service node connected with the source client receives the file transmission request, the verification module is used for verifying the file transmission request data packet first, judging whether the data packet is a legal data packet, for example, whether the client is connected with the service node in a TCP (transmission control protocol) mode or not is judged, if so, the file transmission request data packet is further analyzed, and if not, the file transmission request data packet is discarded.
If the data packet is legal, the first service node analyzes the legal file transmission request data packet through an analysis module to obtain the ID of the target client. The first node queries the service node corresponding to the target client ID, i.e., the second node service node to which the target client ID is connected, by sending a broadcast query packet to the neighboring service nodes.
If the target client is also connected to the first service node, that is, the second service node and the first service node are the same node, the forwarding module of the first service node directly sends the file transmission request data packet to the target client.
If the first service node and the second service node are different nodes, the detection module of the first service node queries the shortest path from the first service node to the second service node according to the node link information table stored by the detection module of the first service node, adds the ID of the second service node into the file transmission request data packet, and forwards the file transmission request data packet to the next service node in the queried shortest path through the forwarding module.
And each service node receiving the file transmission request data packet firstly checks the data packet through a check module, wherein the check process is the same as that of the first service node, if the data packet is legal, the data packet is analyzed to obtain a second service node ID and a target client ID, and otherwise, the received data packet is discarded. If the current service node is the second service node, namely the service node corresponding to the target client, the file transmission request data packet is sent to the target client, otherwise, the shortest path from the current service node to the second service node is queried according to the node link information table stored by the current service node, and the file transmission request data packet is forwarded to the next service node in the shortest path.
And after receiving the file transmission request, the target client feeds back connection response information to the source client through the second service node. After receiving the connection response information, the first service node queries network bandwidth occupation information and sends the network bandwidth occupation information to the source client side by a detection module of the service node (the first service node) connected with the source client side if the response information is confirmation connection, and the detection module is used for partitioning the file to be transmitted by the source client side.
In implementation, the source client and the target client can also be connected in a passive connection mode. The source client generates a key by adopting an encryption algorithm based on the file name and the file size to be transmitted, and the target client passively connects the source client with the target client by sending key information to the source client.
A service node connected with a source client queries network bandwidth occupation information through a detection module, comprising:
the detection module of the first service node inquires and acquires a planned shortest path from the first service node to the second service node;
and taking the residual bandwidth of the link with the highest network bandwidth occupancy rate on the shortest path as the network bandwidth occupancy information obtained by inquiry.
The detection module detects and updates the network bandwidth occupation information regularly, so that the file block size can be adjusted according to the dynamic network bandwidth, the size of each data packet can fully utilize the current network bandwidth, and the occurrence of network congestion and data loss is avoided.
And the service node (namely the first service node) connected with the source client transmits the residual bandwidth to the source client, and the blocking module of the source client blocks the file to be transmitted according to the received residual bandwidth of the link with the highest bandwidth occupancy rate.
Specifically, the blocking module of the source client performs blocking on the file to be transmitted according to the network bandwidth occupation information, and the method comprises the following steps:
s21, setting a file block threshold according to network bandwidth occupation information;
s22, taking the starting position of the file to be transmitted as the current starting position.
S23, placing the sliding window at the current starting position.
S24, calculating a fingerprint value of data in the current sliding window, judging whether the sliding window is positioned at a block boundary or not based on the fingerprint value, and if not, sliding the window forward until the sliding window is positioned at the block boundary or the end of a file to be transmitted; if the current window is positioned at the block boundary or the end of the transmission file, taking the data from the current starting position to the cut-off position of the current sliding window as the current block;
s25, if the current block is smaller than or equal to the file block threshold value, calculating the fingerprint of the current block, and if the current block is the same as the current block fingerprint, only storing the index value of the same block as the current block as the content of the current block; otherwise, storing the data and the fingerprint of the current block as the content of the current block;
If the current block is larger than the file block threshold, dividing the current block into a plurality of sub-blocks according to the file block threshold, sequentially calculating fingerprints of each sub-block, and if the blocks identical to the sub-block fingerprints exist, only storing index values of the blocks identical to the fingerprints as contents of the sub-blocks; otherwise, storing the data and the fingerprint of the sub-block as the contents of the sub-block;
s26, if the current sliding window is at the end of the file to be transmitted, ending the blocking; otherwise, taking the next position of the current sliding window as the current initial position; returning to step S23.
When the method is implemented, the source client uses gamma-RB as a file block threshold to block the file to be transmitted. Where γ represents a reduction factor, for example, a value of 0.9, and rb represents the remaining bandwidth of the link with the highest bandwidth occupancy.
After determining the blocking threshold, sliding the file to be transmitted by adopting a sliding window, and judging the boundary of the blocking. In practice, the sliding window typically takes a value of 64kb or 128kb.
At first, the starting position of the file to be transmitted is used as the starting position of the first block, the sliding window is arranged at the starting position of the first block, the fingerprint of the data of the file to be transmitted in the sliding window is calculated first, and whether the current window is positioned at the boundary of the block is judged based on the calculated fingerprint.
In practice, a hash value of the data within the computation window may be employed as the fingerprint value.
Modulo F of fingerprint values W % D, judging whether the calculation result is equal to a preset value R, if so, locating the current window at the boundary of the block, otherwise, not locating the current window at the boundary of the block. Wherein F is W The fingerprint value representing the current window data, D and R are preset integers, R<D。
If the current window is not the boundary of the block, continuing to slide the window downwards, calculating the fingerprint value of the data in the window by adopting the mode, and judging whether the window is at the boundary of the block or not until the current window is at the boundary of the block. In practice, the sliding step may be set to a fixed size, such as 8kb or 16kb.
If the current sliding window is judged to be at the boundary of the block, taking the starting position of the block to the ending position of the current sliding window as a block to obtain a first block.
If the first block is smaller than or equal to the file block threshold, the first block can be sent as a file block, and then a block with an index of 1 is obtained. The fingerprint value of the first chunk is calculated and the index, fingerprint value and data content of the first chunk are stored. And taking the next position of the end position of the current sliding window as the starting position of the second block, placing the sliding window at the starting position of the second block, and finding the boundary of the second block according to the same process.
If the first block is greater than the file block threshold, network congestion may occur, and therefore the first block needs to be further partitioned according to the file block threshold. Dividing the first sub-block into s sub-blocks according to the gamma-RB of each sub-block, obtaining blocks with indexes of 1 to s from the first sub-block to the s sub-block, sequentially calculating the fingerprint value of each sub-block, and storing only the index value of the same block as the stored fingerprint value of the current block as the content of the current block if the fingerprint value of the current block is the same as the stored fingerprint value of the stored block, otherwise, storing the data and the fingerprint of the current block as the content of the current block. The fingerprints are the same, meaning that the data are the same, so if the current block is the same as the previous block fingerprint, the data of the current data packet are not required to be transmitted, and only the block indexes of the same fingerprint are transmitted, thereby further reducing network burden and improving transmission efficiency.
And taking the next position of the end position of the current sliding window as the starting position of the s+1th block, placing the sliding window at the position of the start of the s+1th block, and finding the boundary of the s+1th block according to the same process.
And the like, realizing the blocking of the file to be transmitted.
By blocking the file to be transmitted according to the network bandwidth information, network congestion caused by the blocking file in transmission is avoided, and transmission efficiency is improved.
When the method is implemented, the first service node can also send the ID of the service node connected with the target client to the source client, and the source client can add the ID of the service node into the transmission file data packet when generating the transmission file data packet, so that the service node can conveniently and rapidly carry out file transmission, and the transmission efficiency is improved.
Specifically, in step S25, it is determined whether there is a block identical to the current block fingerprint in the following manner:
calculating the hash value of the current block as a weak fingerprint value of the current block, judging whether the stored block has the same block as the weak fingerprint value of the current block, and if so, judging that the stored block has the same block as the fingerprint of the current block; otherwise, calculating the Md5 value of the current block as the strong fingerprint of the current block, judging whether the stored block has the same block as the strong fingerprint value of the current block, if so, the stored block has the same block as the fingerprint of the current block; otherwise, there is no chunk identical to the current chunk fingerprint.
In the implementation, in order to further judge whether a data block with the same fingerprint exists, for the current data block, the hash value of the block can be calculated as a weak fingerprint value first, whether the block with the same weak fingerprint value exists is judged, if so, the fingerprint value of the current block is considered to be the same as the stored fingerprint value of the block, otherwise, the Md5 value of the current block is calculated as a strong fingerprint, whether the block with the same strong fingerprint value exists is judged, if so, the fingerprint value of the current block is considered to be the same as the stored fingerprint value of the block, otherwise, the block with the same fingerprint value as the current block does not exist. Thus accurately judging whether the same data blocks exist or not.
And after the source client-side blocking module blocks the file to be transmitted, sequentially sending the blocking file data packet to the target client-side through the service node according to the blocking index.
The analysis module of the service node analyzes the destination address of the received block file data packet, if the current service node is not the service node corresponding to the target client, planning is carried out according to the planned path of the current service node reaching the service node corresponding to the target client, and the forwarding module forwards the block file data packet to the next service node until the current service node is the service node corresponding to the target client; and if the current service node is the service node corresponding to the target client, the forwarding module sends the block file data packet to the target client.
Specifically, the first service node firstly receives the block file data packet, firstly checks the data packet through a checking module, and after the checking is passed, analyzes the block file data packet to obtain a destination address, wherein the destination address comprises a service node (namely a second service node) connected with the target client and the target client. And the first service node judges whether the first service node is a second service node or not, if so, the first service node inquires a target client in the connection information table, and if the target client exists, the forwarding module sends the block file data packet to the target client. And if the target client does not exist, returning sending failure information to the source client. After the target client receives the block file data packet, analyzing the block file data packet to obtain a file name, a file index and a file block, and feeding back the file index and the receiving confirmation to the source client through the service node. And the source client receives the feedback information of the target client and then continues to send the next block file data packet.
If the first service node is not the target service node, forwarding the block file data packet to the next service node according to the planned shortest path forwarding module, judging whether the next service node is the second service node according to the same method, and if not, forwarding the block file data packet to the next service node according to the planned path until the block file data packet is transmitted to the second service node. And the second service node queries the target client in the connection information table, and if the target client exists, the forwarding module sends the block file data packet to the target client.
For example, the first service node obtains a next-hop service node a on the shortest path according to the shortest path that the first service node reaches the second service node (the service node corresponding to the target client), forwards the data packet to the next-hop service node a, the service node a judges whether the data packet is the second service node or not, if not, obtains a next-hop service node B on the shortest path according to the shortest path that the first service node reaches the second service node, forwards the data packet to the next-hop service node B, and so on until the data packet is forwarded to the second service node.
Each service node is only used as a connection point and forwards, data is not required to be read and written, and the problem of memory overflow caused by long-time data in a socket cache area is avoided; in the file transmission process, file data are transmitted in a segmented mode, and after the target client confirms that the n-th segment packet is successfully received, the source client transmits the n+1th segment packet, so that the forwarding pressure of the service node is effectively reduced, the reliability of file transmission is improved, and the probability of transmitting memory leakage and service full load conditions by the service node is prevented.
After receiving the block file data packet, the receiving module of the target client reorganizes the blocks according to the file index to obtain a complete file, and if the file deletion is determined to occur, the receiving module of the target client can send the corresponding index to the source client to transmit the corresponding file block.
In practice, the data transmission range can also be extended by adding service nodes.
The specific process is as follows:
s51, connecting a new service node with any service node in a peer-to-peer network through an adjacent module, and adding the new service node into the peer-to-peer network;
s52, the service node connected with the new service node sends the node information of other service nodes of the peer-to-peer network and the link information between the service nodes to the new service node;
And S53, the service node connected with the new service node detects the link information of the link between the service node and the new service node by sending a detection data packet to the new service node, and sends the node information of the new service node and the detected link information to other service nodes of the peer-to-peer network, wherein the other service nodes update their own node link information tables.
When a new service node joins the network, it can join the network by establishing a connection with any service node in the network. And the service node connected with the newly added service node calculates the link information between the service node and the newly added service node according to the method by sending the detection data packet, broadcasts the node information of the newly added service node and the detected link information to other service nodes in the network, and updates the node link information tables of the other service nodes.
The service node connected with the newly added service node greatly transmits the node link information table stored by the service node to the newly added service node, so that the newly added service node acquires the node and link information in the network, and the subsequent path planning is facilitated.
The invention can conveniently and rapidly increase the service nodes and rapidly expand the network, thereby increasing the data transmission range and improving the reliability and efficiency of transmission. And each service node can be either an access node or a forwarding node, so that no special access node or forwarding node is required to be arranged, resources and network construction cost are saved, and the network construction process is simplified.
Those skilled in the art will appreciate that all or part of the flow of the methods of the embodiments described above may be accomplished by way of a computer program to instruct associated hardware, where the program may be stored on a computer readable storage medium. Wherein the computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory, etc.
The present invention is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present invention are intended to be included in the scope of the present invention.

Claims (10)

1. A file transfer system based on a peer-to-peer network, comprising a plurality of service nodes and clients; each of the service nodes includes: the system comprises an adjacent module, a detection module, an analysis module, a forwarding module and a first connection module; each client comprises a second connection module, a blocking module, a sending module and a receiving module;
the service node establishes peer-to-peer network connection with other service nodes through the adjacent module and monitors the connection state with the adjacent service nodes through the heartbeat data packet;
The service node detects link information between adjacent service nodes through a detection module, and plans the shortest path reaching other service nodes based on the link information;
the first connection module is used for establishing connection with the client and maintaining a connection channel with the client;
the client establishes connection with the service node through a second connection module; the source client sends a connection request to the target client through the service node, and the target client receives the connection request through the receiving module and feeds back whether to confirm the connection or not to the source client through the sending module; if the target client confirms the connection request, a service node connected with the source client inquires network bandwidth occupation information through a detection module and sends the network bandwidth occupation information to the source client, and a blocking module of the source client blocks a file to be transmitted according to the network bandwidth occupation information;
the sending module of the source client sends a block file data packet to the target client through the service node;
the analysis module of the service node analyzes the destination address of the received block file data packet, if the current service node is not the service node corresponding to the target client, the forwarding module forwards the block file data packet to the next service node according to the planned shortest path of the current service node to the service node corresponding to the target client until the current service node is the service node corresponding to the target client; and if the current service node is the service node corresponding to the target client, the forwarding module sends the block file data packet to the target client.
2. The peer-to-peer network based file transfer system of claim 1 wherein the adjacency module periodically sends heartbeat packets to an adjacency service node and receives heartbeat response packets from the adjacency service node, increases the frequency of sending heartbeat packets if no response packets from the adjacency service node are received within a first threshold time, determines that the connection between the current service node and the adjacency service node has been broken if no response packets from the adjacency service node are received within a second threshold time, and updates the node link information table and sends the updated node link information table to other service nodes of the peer-to-peer network, wherein the first threshold time is less than the second threshold time.
3. The peer-to-peer network based file transfer system of claim 1 wherein the client sends a connection request to the service node via the second connection module, the connection request including a client address and a port;
after receiving the connection request, the first connection module of the service node generates an ID for the client, stores the client ID, address and port in a connection information table, returns connection confirmation information to the second connection module of the client, and establishes connection with the client.
4. The peer-to-peer network based file transfer system according to claim 1, wherein the service node connected to the source client is a first node, and the service node connected to the destination client is a second node;
a service node connected with a source client queries network bandwidth occupation information through a detection module, comprising:
the detection module of the first service node inquires and acquires a planned shortest path from the first service node to the second service node;
and taking the residual bandwidth of the link with the highest network bandwidth occupancy rate on the shortest path as the network bandwidth occupancy information obtained by inquiry.
5. The peer-to-peer network based file transfer system of claim 1 wherein the blocking module of the source client blocks the file to be transferred according to the network bandwidth occupancy information, comprising:
setting a file block threshold according to the network bandwidth occupation information;
taking the starting position of the file to be transmitted as the current starting position;
the sliding window is arranged at the current starting position;
calculating a fingerprint value of data in the current sliding window, judging whether the sliding window is positioned at a block boundary or not based on the fingerprint value, if not, sliding the window forward until the sliding window is positioned at the block boundary or the end of a file to be transmitted; if the current window is positioned at the block boundary or the end of the transmission file, taking the data from the current starting position to the cut-off position of the current sliding window as the current block;
If the current block is smaller than or equal to the file block threshold value, calculating the fingerprint of the current block, and if the current block is the same as the current block fingerprint, only storing the index value of the same block as the current block as the content of the current block; otherwise, storing the data and the fingerprint of the current block as the content of the current block;
if the current block is larger than the file block threshold, dividing the current block into a plurality of sub-blocks according to the file block threshold, sequentially calculating fingerprints of each sub-block, and if the blocks identical to the sub-block fingerprints exist, only storing index values of the blocks identical to the fingerprints as contents of the sub-blocks; otherwise, storing the data and the fingerprint of the sub-block as the contents of the sub-block;
if the current sliding window is at the end of the file to be transmitted, ending the partitioning; otherwise, taking the next position of the current sliding window as the current initial position; and returning to the step of placing the sliding window at the current starting position.
6. The peer-to-peer network based file transfer system of claim 5 wherein the determination of whether there is a chunk that is the same as the current chunk fingerprint is made by:
calculating the hash value of the current block as a weak fingerprint value of the current block, judging whether the stored block has the same block as the weak fingerprint value of the current block, and if so, judging that the stored block has the same block as the fingerprint of the current block; otherwise, calculating the Md5 value of the current block as the strong fingerprint of the current block, judging whether the stored block has the same block as the strong fingerprint value of the current block, if so, the stored block has the same block as the fingerprint of the current block; otherwise, there is no chunk identical to the current chunk fingerprint.
7. The peer-to-peer network based file transfer system of claim 1, wherein the probe module employs the following steps to plan shortest paths to other serving nodes based on the link information:
the detection module of each service node periodically transmits detection data packets to adjacent service nodes, and detects link information of links between the service nodes and each adjacent service node;
and calculating the shortest path from each service node to other service nodes by adopting a path planning algorithm based on the link information.
8. The peer-to-peer network based file transfer system of claim 7 wherein the link information includes network delay, packet loss rate, and bandwidth utilization of the link;
calculating the shortest path from each service node to other service nodes by adopting a path planning algorithm based on the link information, wherein the method comprises the following steps:
calculating network delay, packet loss rate and bandwidth utilization rate of a link based on the transmission time, transmission number and size of the probe data packets;
calculating the weight of the link based on the network delay, packet loss rate and bandwidth utilization of the link;
and calculating the shortest path from each service node to other service nodes according to the weight of each link.
9. The peer-to-peer network based file transfer system as claimed in claim 6 wherein Weight is based on the formula ij ＝α×BL ij +β×Ploss ij +δ×Delay ij Computing Weight for links from service node i to service node j ij Wherein α, β and γ represent weight coefficients, delay ij Representing network delay, ploss, of a link from service node i to service node j ij Packet loss rate, BL, representing link from service node i to service node j ij Representing the bandwidth utilization of the link from serving node i to serving node j.
10. The peer-to-peer network based file transfer system of claim 8 wherein the Dijkstra algorithm is used to calculate the shortest path from each serving node to the other serving nodes based on the weight of each link.
CN202310731255.4A 2023-06-20 2023-06-20 File transmission system based on peer-to-peer network Active CN116566973B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310731255.4A CN116566973B (en) 2023-06-20 2023-06-20 File transmission system based on peer-to-peer network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310731255.4A CN116566973B (en) 2023-06-20 2023-06-20 File transmission system based on peer-to-peer network

Publications (2)

Publication Number Publication Date
CN116566973A true CN116566973A (en) 2023-08-08
CN116566973B CN116566973B (en) 2023-11-07

Family

ID=87486310

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310731255.4A Active CN116566973B (en) 2023-06-20 2023-06-20 File transmission system based on peer-to-peer network

Country Status (1)

Country Link
CN (1) CN116566973B (en)

Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050007964A1 (en) * 2003-07-01 2005-01-13 Vincent Falco Peer-to-peer network heartbeat server and associated methods
CN101183323A (en) * 2007-12-10 2008-05-21 华中科技大学 Data stand-by system based on finger print
CN101599887A (en) * 2008-06-05 2009-12-09 华为技术有限公司 The transmission method of data and equipment in the peer-to-peer network
US20100049868A1 (en) * 2008-08-19 2010-02-25 Cisco Technology, Inc. Maintaining protocol adjacency state with forwarding failure
US20110199939A1 (en) * 2008-10-27 2011-08-18 Huawei Technologies Co., Ltd. Path computation method, node device and path computation element
CN102223257A (en) * 2011-07-25 2011-10-19 莫雅静 Heartbeat interval adjusting method, server, client terminal and communication system
CN102821316A (en) * 2012-05-14 2012-12-12 南京邮电大学 Improved video on demand (VOD) transmission method based on peer-to-peer computing core algorithm
CN102868617A (en) * 2011-07-05 2013-01-09 中兴通讯股份有限公司 P2P (peer-to-peer) network routing method and system
CN106789537A (en) * 2017-01-20 2017-05-31 网宿科技股份有限公司 A kind of VPN construction method and system
US20180012281A1 (en) * 2016-07-08 2018-01-11 Sergio Salani Virtual, Location-Based Connection Tool for Service Providers and Users
CN109922161A (en) * 2019-04-10 2019-06-21 山东师范大学 Content distribution method, system, equipment and the medium of dynamic cloud content distributing network
WO2019169874A1 (en) * 2018-03-09 2019-09-12 南京邮电大学 Wireless mesh network opportunistic routing algorithm based on quality of service assurance
CN111191277A (en) * 2019-12-10 2020-05-22 北京中宏立达科技发展有限公司 Hidden encryption and confidentiality method for optical disk file and optical disk medium
CN111447290A (en) * 2020-06-12 2020-07-24 支付宝(杭州)信息技术有限公司 Communication method and service data transmission method in block chain network
WO2021035084A1 (en) * 2019-08-20 2021-02-25 Vapor IO Inc. Distributed processing for determining network paths
WO2021134928A1 (en) * 2019-12-31 2021-07-08 深圳市网心科技有限公司 Method, device, system and apparatus for p2p network-based data transmission, and medium
CN115914074A (en) * 2021-09-30 2023-04-04 数界(深圳)科技有限公司 Point-to-point network message transmission method, routing equipment and computer storage medium

Patent Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050007964A1 (en) * 2003-07-01 2005-01-13 Vincent Falco Peer-to-peer network heartbeat server and associated methods
CN101183323A (en) * 2007-12-10 2008-05-21 华中科技大学 Data stand-by system based on finger print
CN101599887A (en) * 2008-06-05 2009-12-09 华为技术有限公司 The transmission method of data and equipment in the peer-to-peer network
US20100049868A1 (en) * 2008-08-19 2010-02-25 Cisco Technology, Inc. Maintaining protocol adjacency state with forwarding failure
US20110199939A1 (en) * 2008-10-27 2011-08-18 Huawei Technologies Co., Ltd. Path computation method, node device and path computation element
CN102868617A (en) * 2011-07-05 2013-01-09 中兴通讯股份有限公司 P2P (peer-to-peer) network routing method and system
CN102223257A (en) * 2011-07-25 2011-10-19 莫雅静 Heartbeat interval adjusting method, server, client terminal and communication system
CN102821316A (en) * 2012-05-14 2012-12-12 南京邮电大学 Improved video on demand (VOD) transmission method based on peer-to-peer computing core algorithm
US20180012281A1 (en) * 2016-07-08 2018-01-11 Sergio Salani Virtual, Location-Based Connection Tool for Service Providers and Users
CN106789537A (en) * 2017-01-20 2017-05-31 网宿科技股份有限公司 A kind of VPN construction method and system
WO2019169874A1 (en) * 2018-03-09 2019-09-12 南京邮电大学 Wireless mesh network opportunistic routing algorithm based on quality of service assurance
CN109922161A (en) * 2019-04-10 2019-06-21 山东师范大学 Content distribution method, system, equipment and the medium of dynamic cloud content distributing network
WO2021035084A1 (en) * 2019-08-20 2021-02-25 Vapor IO Inc. Distributed processing for determining network paths
CN111191277A (en) * 2019-12-10 2020-05-22 北京中宏立达科技发展有限公司 Hidden encryption and confidentiality method for optical disk file and optical disk medium
WO2021134928A1 (en) * 2019-12-31 2021-07-08 深圳市网心科技有限公司 Method, device, system and apparatus for p2p network-based data transmission, and medium
CN111447290A (en) * 2020-06-12 2020-07-24 支付宝(杭州)信息技术有限公司 Communication method and service data transmission method in block chain network
CN115914074A (en) * 2021-09-30 2023-04-04 数界(深圳)科技有限公司 Point-to-point network message transmission method, routing equipment and computer storage medium

Also Published As

Publication number Publication date
CN116566973B (en) 2023-11-07

Similar Documents

Publication Publication Date Title
CN113709057B (en) Network congestion notification method, proxy node, network node and computer equipment
JP5740449B2 (en) Data transmission method and apparatus
KR101506849B1 (en) A generalized dual-mode data forwarding plane for information-centric network
EP2813059B1 (en) Method and apparatus for adaptive forwarding strategies in content-centric networking
US7644182B2 (en) Reconfiguring a multicast tree
US11558302B2 (en) Data transmission method and apparatus
CN116471283B (en) File transmission method
KR101887835B1 (en) Apparatus and method for data delivery in delay-tolerant network(dtn)
US20150334630A1 (en) Method and apparatus for transmission of data over a plurality of networks
US9548921B2 (en) Compression of routing information exchanges
US20210315028A1 (en) Method and network node of setting up a wireless connection
JP2008205932A (en) Node device and band control method
CN116566973B (en) File transmission system based on peer-to-peer network
Ahdan et al. Adaptive Forwarding Strategy in Named Data Networking: A Survey
CN113810349B (en) Data transmission method, device, computer equipment and storage medium
Marandi et al. A comparative analysis of bloom filter-based routing protocols for information-centric networks
Pu Adaptive forwarding strategy based on MCDM model in named data networking
AT&T
CN103138873A (en) Obtaining method and device for two-layer data package time delay
Wang Caching, routing and congestion control in a future information-centric internet
Shen et al. Prediction-based redundant data elimination with content overhearing in wireless networks
JP4828555B2 (en) Node device and bandwidth control method
US7787368B1 (en) In-network per packet cashes
CN109995724B (en) Communication method, client and communication system
Salehi et al. Content Delivery Networks: Interactions between storage/delay/cost

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant