WO2012142844A1

WO2012142844A1 - P2p-based file distribution system and slicing method

Info

Publication number: WO2012142844A1
Application number: PCT/CN2011/084210
Authority: WO
Inventors: 季健
Original assignee: 中兴通讯股份有限公司
Priority date: 2011-04-22
Filing date: 2011-12-19
Publication date: 2012-10-26
Also published as: CN102170475A

Abstract

The present invention provides a P2P-based file distribution system and a slicing method. In a file distribution process, the adaptive slicing method slices a file to be downloaded, a slice in a last level being a transmission-level slice. The size of the transmission-level slice is dynamically adjusted according to a network resource condition between a local node and a serving node, so as to ensure that the transmission performance of the file distribution system is adaptive to the change of the network bandwidth, thereby improving the data transmission efficiency and expanding the application scope of the file distribution system.

Description

P2P-based file distribution system and fragmentation method

The invention belongs to the peer-to-peer (P2P) technology, and specifically relates to a P2P-based file distribution system and a fragmentation method. Background technique

A P2P network is a network architecture in which resources are distributed and shared. When communicating, each node is peer-to-peer, that is, each node can be either a client or a server. The P2P network solves the problems of network congestion and server bandwidth bottlenecks by distributing the service functions originally completed on the centralized server to the client. P2P file distribution uses a fragmentation method for data interaction, that is, dividing a file into a plurality of data blocks of the same length or different length, and then transmitting the data in units of data blocks. The sharding is a calculation method. According to the calculation method, the file distribution system can calculate which position of the first shard of a file is in the file, so that when the data is replied, the location can be calculated at the calculated position. The fragmented data of the required size is read out for reply. The size of the fragmentation of the current P2P download system is fixed, and the fragmentation size of the file distribution system is also fixed. In the file distribution system, the fragmentation method adopts a three-level fragmentation method, and the specific fragmentation method is as follows:

First, a file is divided into segments, and the segment is a first-level fragmentation. The size of the current system middle segment is fixed, which can be 4M. Secondly, the segment is further fragmented, which is a secondary fragment, and the secondary fragment is a file. The unit shared by the distribution system client, that is, only the file distribution system client downloads a complete secondary fragment, and this part of the data can be shared and shared. Currently, the size of the secondary fragment in the system is fixed. It is 16KB; then, based on the secondary fragmentation, three-level fragmentation is performed. The third-level fragmentation is the transmission unit of the file distribution system client when performing data interaction. Currently, the size of the system's three-level fragmentation is fixed. Specifically, it can be 1KB. The file distribution system also establishes corresponding index information while establishing the fragmentation, and the index information indicates the fragmentation of a file. When the file distribution system client performs file data sharing, the shared parties interact through the index information to know whether the peer has the data it needs. The index information is binary data. Each bit of data indicates whether the corresponding fragment exists. A value of 1 indicates that the corresponding fragment exists, and a value of 0 indicates that the corresponding fragment does not exist.

If the fragmentation is too small, it is beneficial to the data sharing between the service nodes, but the index information occupies more bytes. Excessive index information will consume a certain amount of bandwidth when transmitted between networks; If the fragmentation is relatively thick, it will affect the data sharing between the service nodes, because only the corresponding bit of the index information is 1, the data can be shared externally. In some applications, such as the content synchronization of the Electronic Program Guide (EPG), the main download is a large number of small files, and in the distribution of media files of the Content Delivery Network (CDN), the main download is It is a relatively large video file; using a fixed fragmentation method for files of different sizes will reduce network utilization.

In the system of the file distribution system, the data transmission uses the UDP protocol. Due to the limitation of the MTU, the current data transmission unit of the system is 1 KB, so that the fixed-size transmission unit cannot adapt to the dynamic change of the network bandwidth. When the network bandwidth is relatively good, The size of 1KB is suitable, and in the case of large fluctuations in network bandwidth, the size of 1KB may cause packet loss or congestion of network transmission, which reduces transmission efficiency. Summary of the invention

The technical problem to be solved by the present invention is to provide a fragmentation method for a P2P-based file distribution system, which can dynamically adjust the size of a data transmission level fragment according to a change in network bandwidth, since the transmission level fragmentation is The unit of transmission when data is exchanged, thus improving the efficiency of data transmission.

The invention provides a P2P-based file distribution system, which can dynamically adjust the size of a data transmission unit according to changes in network bandwidth, thereby improving data transmission efficiency. The present invention provides a method for sharding a P2P-based file distribution system, the method comprising: sharding a file to be downloaded in a file distribution process, where the shard includes a shared shard and a transport shard. And the last level fragment is a transport level fragment, wherein the size of the transport level fragment is dynamically adjusted according to the network resource status between the local node and the service node during transmission.

In the above solution, when the file to be downloaded is fragmented, the size of the shared-level fragment is determined according to the file size range corresponding to the size of the file to be downloaded.

In the above solution, the network resource status between the local node and the serving node includes: one or more pieces of information of the amount of interaction data, the packet loss rate, and the delay size between the local node and the serving node in a period of time.

The present invention also provides a file downloading method of a P2P-based file distribution system, the method comprising the following steps:

When the P2P file is downloaded, the local node fragments the file to be downloaded, and establishes index information of the file to be downloaded, and the index information of the file to be downloaded includes index information of each fragment obtained by the fragment, and each level index The information includes the fragmentation location information of the corresponding segment, where the fragment includes a shared-level fragment and a transport-level fragment, and the initial size of the transport-level fragment is a default value;

After the local node requests the server to download the node information, the local node determines the download node corresponding to the shared-level fragment of the file to be downloaded;

The local node sends a download request for the shared-level shard to the determined download node, where the index information of the shared-level shard that needs to be downloaded from the download node and the size of the determined transport-level shard are included, and are The network resource status between the service nodes dynamically adjusts the size of the transport level fragment;

The local node saves the file data after receiving the data replied by the download node.

In the above solution, the network resource status between the local node and the service node includes: the amount of interaction data, the packet loss rate, and the delay size between the local node and the service node in a period of time. One or more messages.

The invention also provides a P2P-based file distribution system, the system comprising:

The local node performs the P2P file downloading, and the file to be downloaded is fragmented, and the index information of the file to be downloaded is established, and the index information of the file to be downloaded includes index information of each fragment obtained by the fragment, each level. The index information includes fragmentation location information of the corresponding segment, the fragment includes a shared-level fragment and a transport-level fragment, and the size of the transport-level fragment is an initial default value; and then the server is requested to download the node information, and after receiving the reply, Sending a download request for the shared-level shard to the determined download node, where the download request includes the index information of the shared-level shard that needs to be downloaded from the download node, and the size of the determined transport-level shard, during the transmission process. Dynamically adjusting a size of the transport level fragment according to a network resource status between the serving node; receiving data replied by the download node;

a server, providing index information and data information to the local node; after receiving the download node request of the local node, providing download node information to the local node;

The download node provides file data to the local node according to the download request for the shared-level shard sent by the local node.

In the above solution, when the local node fragments the file to be downloaded, the size of the shared-level fragment is determined according to the file size range corresponding to the size of the file to be downloaded.

In the above solution, the index information is saved in the local node and the server. In the above solution, the network resource status between the local node and the service node includes: one or more pieces of information about the amount of interaction data, the packet loss rate, and the delay size between the local node and the serving node in a period of time.

In the above solution, the download node is a service node or a server.

The download node is a server or a service node. The P2P-based file distribution system and the fragmentation method provided by the invention can dynamically adjust the size of the data transmission unit according to the change of the network bandwidth during the downloading process, so as to ensure that the transmission performance of the file distribution system can adapt to the change of the network bandwidth, thereby Improve the efficiency of data transmission and expand the scope of application of file distribution systems. DRAWINGS

1 is a schematic diagram of a three-level adaptive fragmentation method according to an embodiment of the present invention;

2 is a schematic diagram of a four-level adaptive fragmentation method according to an embodiment of the present invention;

3 is a flowchart of a file downloading method of a P2P-based file distribution system according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of the overall architecture of a file distribution system according to an embodiment of the present invention. detailed description

The present invention provides a method for sharding a P2P-based file distribution system, the method comprising: sharding a file to be downloaded in a file distribution process, where the shard includes a shared shard and a transport shard. And the last shard is a transport shard, wherein the size of the transport shard is dynamically adjusted according to the network resource status between the local node and the serving node during the transmission; specifically, the local node and the service The network resource status between the nodes includes: one or more pieces of information such as the amount of interaction data, the packet loss rate, and the delay size between the local node and the service node in a period of time, when the file to be downloaded is initially fragmented, according to The file size range corresponding to the size of the download file determines the size of the shared-level shard, and the size of the transport-level shard is initially the default.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

As a first embodiment of the present invention, as shown in FIG. 1, a three-level fragmentation method is used to determine the size of the first and second fragments according to the file size, and the first and second fragments are shared-level fragments. , where the second-level fragment is used as the data sharing unit for data interaction between nodes, and the third-level fragment is The transmission level fragmentation is the transmission unit when the data is transmitted between the nodes. The size of the third-level fragment is the default value, which can be 1 KB. During the transmission process, the size of the third-level fragment can be dynamically changed according to the network status. Adjustment. The specific fragmentation method is as follows:

The first level of fragmentation and the second level of fragmentation are as follows:

When the file size is greater than 12 MB, the size of the segment 11 into which the first-stage fragment is divided is 4M, and the size of the block 12 into which the second-stage fragment is divided into 16 KB is 16 KB;

When the file size is between 4MB and 12MB, the segment size is 3MB, and the block size is 16KB.

When the file size is between 2MB and 4MB, the segment size is 2MB, and the block size is 8KB.

When the file size is between 1MB and 2MB, the size of the segment 11 is 1MB, and the size of the block 12 is 4KB.

When the file size is less than 1MB, the segment 11 size is 1MB, and the block 12 size is 2KB. The size of the third-level fragment 13 is an integer multiple of 64 bytes, and may be 1 KB or 512 bytes.

256 bytes, 128 bytes, 64 bytes, etc.; Since the three-stage fragmentation is the transmission unit for data interaction, the transmission unit is up to 1 KB and the minimum is 64 bytes.

Correspondingly, the index information corresponding to the file is established at the same time as the three-level fragmentation of the file, and the index information is also divided into three levels, and the information of each level indicates the position information of the corresponding level of the fragment as the present invention provides In the second embodiment, as shown in FIG. 2, a four-level fragmentation method is adopted. The fragmentation method determines the size of the first, second, and third fragments according to the file size, and the first, second, and third-level fragments are shared. Fragmentation, where the secondary fragment is used as the data sharing unit for data exchange between nodes, the fourth fragment is the transmission level fragmentation, and the fourth fragment is the transmission unit when the data is transmitted between the nodes. The size of the fourth-level fragment is the initial default value, which can be 1 KB. The size of the fourth-level fragment can be dynamically adjusted according to the network status during the transmission process. Next:

When the file size is greater than 12 MB, the size of the segment 21 into which the first-stage fragment is divided is 4M, and the size of the block 22 into which the second-level fragment is divided into 16 KB is 16 KB;

When the file size is between 4MB and 12MB, the segment 21 size is 3MB, and the block 22 size is 16KB.

When the file size is between d and 2MB to 4MB, the size of segment 21 is 2MB, and the size of block 22 is 8KB.

When the file size is between 1MB and 2MB, the size of the segment 21 is 1MB, and the size of the block 22 is 4KB.

When the file size is less than 1MB, the segment 21 size is 1MB, and the block 22 size is 2KB. The size of the third stage fragment 23 is fixed to 1 KB;

The fourth-level fragment 24 is further subdivided on the basis of the transmission unit of the third-level fragment 1 KB, and its size is an integer multiple of 64 bytes, which may be 1 KB, 512 bytes, 256 bytes, 128 bytes. , 64 bytes, etc.; Since the four-level fragmentation is the transmission unit for data interaction, the transmission unit is up to 1 KB and the minimum is 64 bytes.

After the file is fragmented, the index information corresponding to the file is established, and the index information is also divided into four levels, which respectively represent the location information of the fragment of the corresponding level, that is, how many segments are divided into the file, and each segment is divided into How many blocks, the size of the last piece of the last piece, and so on.

For example, for a file having a size of 20.51 MB, according to the adaptive fragmentation method of the present invention, the same index information is maintained in the file distribution system and the file distribution system DTS-S-SP; since the file is larger than 12 MB, the level is The fragment size is divided into 4 MB. It can be seen that the file is divided into 6 segments in the index information, and the block size is divided into 16 KB. The first 5 segments of the index information respectively contain 256 blocks. The last segment contains 33 blocks, and the last block of the last segment has a size of 10.14 KB. The present invention also provides a file downloading method of a P2P-based file distribution system. As shown in FIG. 3, the method includes the following steps:

Step 301: start;

Step 302: The local node obtains a download instruction.

Step 303: The local node applies to the server for the file attribute information of the file to be downloaded, and the file attribute information may be the CID and the file size information of the file. After obtaining the attribute information of the file, the file to be downloaded is fragmented according to the file to be downloaded. The size of the file size corresponding to the size determines the size of the shared-level shard, the default setting of the size of the transport-level shard, and the corresponding index information is saved;

Specifically, the sharing level fragmentation may be performed according to the size of the file to be downloaded, as described in the first embodiment, that is, the sharing level fragment may be implemented as the first level fragment and the second layer fragment described in the first embodiment. The third-level fragment described in the first example is a transport-level fragment, and its initial default value is set to 1 KB, which can be adjusted according to the network state during transmission;

The sharing level fragmentation may be performed according to the size of the file to be downloaded, as described in the second embodiment, that is, the sharing level fragment may be the first level fragment, the second level fragment, and the third level described in the second embodiment. Fragmentation, and the four-stage fragment described in the second embodiment is a transport-level fragment, and its initial default value is set to 1 KB, which can be adjusted according to the network state during transmission.

Step 304: After the local node applies to the server to download the node information, the download node corresponding to the shared-level fragment of the file to be downloaded is determined, where the shared-level fragment may be the second-level fragment described in the first embodiment. The two-stage fragmentation described in the second embodiment may be used. Specifically, the download node may be a service node or a server.

Step 305: The local node sends a download request for the shared-level fragment to the determined download node, and determines a transmission unit size, that is, a size of the transport-level fragment, according to a network resource status between the local node and the download node.

Specifically, the local node requests the file CID, the segment number, when requesting data from the download node. Block number, slice number, offset from the slice, the number of bytes to be read, the number of bytes is specifically

An integer multiple of 64 bytes;

The network resource status refers to one or more pieces of information about the amount of interaction data, the packet loss rate, and the delay size between the local node and the download node in a period of time; preferably, the period of time may be 5s; The initial default size of the shard is 1 KB, so the initial default value of the transmission unit is 1 KB. When the network condition deteriorates, resize the transmission shard to be decremented by a multiple of 2. The size of the transmission shard is always 64-byte integer multiple, the minimum is 64 bytes, the maximum is 1KB; When adjusting the size of the transport-level fragment, the corresponding index information is modified at the same time, so that the file fragment size and the actual file fragment size in the index information are always Specifically, the transmission level fragmentation may be the third level fragment described in the first embodiment, or may be the fourth level fragment described in the second embodiment.

Step 306: The download node sends a data reply to the local node. Specifically, when replying the data, the file CID, the segment number, the block number, the fragment number, the offset with respect to the fragment, the number of bytes of the shared data, and Actual file data;

Step 307: After receiving the data replied by the download node, the local node writes the actual file data into the corresponding memory location, and sets the corresponding index information. When the lowest level index information is filled in, Then set the upper level index;

Step 308: If the corresponding data is received, the corresponding location index is set. If the shared-level data block index is set to 0, then step 305 is continued until the lowest-level fragment data index of the shared-level fragment is set to 1 o'clock, continue to step 309;

For example, when the lowest level index is the four level index information described in the second embodiment, when the local node receives the reply and writes the actual file data to the corresponding memory location, the fourth level index is set, that is, The corresponding four-level index position is set to 1. When the 1KB data is filled, the three-level index position is set until the secondary fragment data is filled, and the secondary index is set until the second level. When the index position is set to 1, the data transmission ends, the data The blocks can be shared externally.

Step 309: End the data transmission.

As shown in FIG. 4, the present invention also provides a P2P-based file distribution system, which includes:

The local node performs the P2P file downloading, and the file to be downloaded is fragmented, and the index information of the file to be downloaded is established, and the index information of the file to be downloaded includes index information of each fragment obtained by the fragment, each level. The index information includes fragmentation location information of the corresponding segment, the fragment includes a shared-level fragment and a transport-level fragment, and the size of the transport-level fragment is an initial default value; and then the server is requested to download the node information, and after receiving the reply, Sending a download request for the shared-level shard to the determined download node, where the download request includes the index information of the shared-level shard that needs to be downloaded from the download node, and the size of the determined transport-level shard, during the transmission process. Dynamically adjusting the size of the transport level fragment according to a network resource condition between the serving node; receiving data replied by the download node.

a server, providing index information and data information to the local node; after receiving the download request of the local node, providing the download information to the local node;

Specifically, the server may include the following three modules:

The content management module DTS-S-SM monitors related directories on the library, monitors the addition and deletion of files in real time, and notifies the node of the index module file changes;

The node indexing module DTS_S_RM provides node index information, that is, which download nodes the files to be downloaded are respectively, and the download node includes a server and a service node. Specifically, the node indexing module is configured to provide CID and file size information of the file to be downloaded according to the request of the local node, and provide the service node with a shared-level fragment corresponding to the file to be downloaded according to the request of the local node. Download node information;

The static node module DTS_S_SP receives the online message of the node indexing module and returns the data to the local node according to the download request for the shared-level fragment sent by the local node. The downloading node provides the file data to the local node according to the download request for the shared-level fragment sent by the local node. Specifically, the download node may be a service node or a static node module DTS-S-SP.

The fragmentation method of the P2P-based file distribution system proposed by the present invention can adaptively perform fragmentation adjustment according to the size of the file to be downloaded, so that files of different sizes can obtain similar P2P distribution performance, and can be downloaded. During the process, the size of the data transmission unit is dynamically adjusted according to the change of the network bandwidth, so as to ensure that the transmission performance of the file distribution system can adapt to changes in the network bandwidth, thereby improving the performance of data transmission.

It is apparent that those skilled in the art can make various modifications and variations to the invention without departing from the spirit and scope of the invention. Thus, it is intended that the present invention cover the modifications and modifications of the invention

Claims

Claim

A fragmentation method for a P2P-based file distribution system, the method comprising:

In the file distribution process, the file to be downloaded is fragmented, and the fragment includes a shared-level fragment and a transport-level fragment, and the last-level fragment is a transport-level fragment, wherein the local-layer fragment is transmitted according to the local The network resource status between the node and the serving node dynamically adjusts the size of the transport level fragment.

The method according to claim 1, wherein when the file to be downloaded is sliced, the size of the shared-level slice is determined according to a file size range corresponding to the size of the file to be downloaded.

3. The method according to claim 1, wherein the network resource status between the local node and the service node comprises: an amount of interaction data between the local node and the service node, a packet loss rate, and a period of time. One or more pieces of information in the delay size.

A file downloading method for a P2P-based file distribution system, characterized in that the method comprises the following steps:

The local node sends a download request for the shared-level shard to the determined download node, where the index information of the shared-level shard that needs to be downloaded from the download node and the size of the determined transport-level shard are included, and are The network resource status between the service nodes dynamically adjusts the size of the transport level fragmentation; The local node receives the data replied by the download node.

The file downloading method according to claim 4, wherein the network resource status between the local node and the serving node includes: the amount of interaction data and the packet loss rate between the local node and the serving node in a period of time And one or more pieces of information in the delay size.

The file downloading method according to claim 4, wherein when the file to be downloaded is fragmented, the size of the shared-level fragment is determined according to a file size range corresponding to the size of the file to be downloaded.

A P2P-based file distribution system, the system includes: a local node, when a P2P file is downloaded, the file to be downloaded is fragmented, and the index information of the file to be downloaded is established. The index information of the download file includes index information of each fragment obtained by the fragment, and the index information of each level includes fragmentation location information of the corresponding fragment, and the fragment includes a shared-level fragment and a transport-level fragment, and the transport-level fragment is included. The size is the initial default value; then the server is requested to download the node information, and after receiving the reply, the download request for the shared-level shard is sent to the determined download node, where the download request includes the share level that needs to be downloaded from the download node. The index information of the fragment and the size of the determined transport level fragment, dynamically adjusting the size of the transport level fragment according to the network resource status with the service node during the transmission; receiving the data replied by the download node;

The download node provides file data to the local node according to the download request for the shared-level fragment sent by the local node.

The system according to claim 7, wherein when the local node fragments the file to be downloaded, the size of the shared-level fragment is determined according to the file size range corresponding to the size of the file to be downloaded.

9. The system according to claim 7, wherein the index information is saved in the present The ground node and the server.

10. The system according to claim 7, wherein the network resource status between the local node and the serving node comprises: the amount of interaction data, the packet loss rate, and the delay between the local node and the serving node in a period of time. One or more messages in size.

The system according to claim 7, wherein the download node is a service node or a server.